Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report FINAL PERFORMANCE EVALUATION Cold Chain Bangladesh Alliance (CCBA) Final Report June 30, 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Abelardo Rodríguez, John D. Willsie, Hasan Iqbal, and Nasirul Islam for the Accelerating Capacity for Monitoring and Evaluation (ACME) activity held by International Business & Technical Consultants, Inc. (IBTCI). Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report Final Performance Evaluation Final Report COLD CHAIN BANGLADESH ALLIANCE (CCBA) PROJECT Revised Submission: November 7, 2016 Original Submission: June 30, 2016 AID-388-C-14-00001 (USAID/Bangladesh Accelerating Capacity for Monitoring and Evaluation) DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Cover Photo: Collection Center in Jessore district, where farmers aggregate, select, pack, and ship high-quality vegetables to wholesale markets. (Photo: Nazim Chawdry) Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) i ACKNOWLEDGMENTS The CCBA Evaluation Team would like to thank the many people who were willing to meet with the Team to share information and offer their guidance and recommendations. The staff within the USAID/Bangladesh Mission in Dhaka provided excellent direction and support to the Team. The staff within the USAID/ACME project could not have been more helpful and hospitable. The information they shared with the Team at the outset of the evaluation set the stage for productivity and efficiency. The IBTCI support staff in the U.S. were continually ready to provide whatever assistance the Team required. This included the security team, who continually maintained contact with us. Also, Travel Tours worked tirelessly to ensure that the Team arrived on time and returned home safely. The staff within the CCBA project made every effort to accommodate the Team’s schedule and we appreciated their efforts. The senior staff within the USAID/Agricultural Inputs Project (AIP) and the USAID/Agricultural Value Chain (AVC) projects gave generously of their time and knowledge and provided valuable information and insights. The Team deeply appreciated the candid discussions with staff members at Golden Harvest, which will be useful for future Public-Private Partnership (PPP) initiatives. In addition, meetings with senior management at Rahimafrooz (Agora Supermarkets) and Bengal Meat were extremely helpful to the Team in formulating future recommendations. The staff at the Bangladesh Agricultural Development Corporation provided valuable perspectives for the Team, especially in envisioning a path forward within the context of Bangladesh. The support the Team received from project staff in the USAID/CCBA project during travel throughout the country made each trip productive, efficient, and informative. During field visits, numerous additional individuals shared their experiences and knowledge with the Team. This included local experts (who provided training on behalf of the USAID/CCBA project), note takers, translators, and drivers—all of whom helped the Team achieve productive field visits. The Team particularly appreciates the contributions of the dozens of beneficiaries involved in wholesale and retail food production, including vegetable, dairy, and poultry producers; women with backyard gardens; transporters; Collection Center managers; local service providers; and value chain stakeholders. All gave their time freely to discuss their CCBA experiences and offer suggestions for future donor interactions. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) ii CONTENTS Acronyms....................................................................................................................................................iii Executive Summary ................................................................................................................................... 1 Project Introduction.................................................................................................................................. 7 The Development Problem and USAID’s Response ......................................................................... 9 Evaluation Purpose and Evaluation Questions.................................................................................. 12 Evaluation Purpose.......................................................................................................................... 12 Evaluation Questions...................................................................................................................... 12 Evaluation Methods and Limitations.................................................................................................... 13 Findings and Conclusions ....................................................................................................................... 16 Question 1. Objective and reduction of PH loss..................................................................... 16 Question 2. Most suitable and economically viable commodities........................................ 18 Question 3. GDA and leveraging private investments............................................................ 20 Question 4. Constraints and opportunities for sustainability ............................................... 21 Question 5. Gender integration................................................................................................... 23 Question 6. Approaches to inform design and implementation........................................... 24 Recommendations ................................................................................................................................... 27 Question 1. Objective and reduction of PH loss..................................................................... 27 Question 2. Most suitable and economically viable commodities........................................ 27 Question 3. GDA and leveraging private investments............................................................ 28 Question 4. Constraints and opportunities for sustainability ............................................... 29 Question 5. Gender integration................................................................................................... 29 Question 6. Approaches to inform design and implementation........................................... 30 Lessons Learned....................................................................................................................................... 30 Figure 1. Cold Chain Bangladesh Alliance Intervention Sites 14 Figure 2. Usage of Financial Resources, Q1 ending Sep. 2013 to Q11 ending Mar. 2016 22 ANNEXES 33 Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) iii ACRONYMS ACME Accelerating Capacity for Monitoring and Evaluation BADC Bangladesh Agricultural Development Corporation BARI Bangladesh Agricultural Research Institute BASA Bangladesh Association for Social Advancement CCBA Cold Chain Bangladesh Alliance CDCS Country Development Cooperation Strategy CIP Country Investment Plan CNFA Cultivating New Frontiers in Agriculture COP Chief of Party COR Contracting Officer´s Representative CREL Climate Resilient Ecosystems and Livelihoods CP Collection Point/Center DAE Department of Agriculture and Extension DEC Development Experience Clearinghouse DO Development Objective DOC Day-Old Chick DQA Data Quality Assessment FGD Focus Group Discussion FtF Feed the Future GAP Good Aqua/Agricultural Practices GDA Global Development Alliance GH Golden Harvest GPRA Government Performance and Results Act HACCP Hazard Analysis Critical Control Point HQP High-Quality Products HVC High-Value Commodity HVP High-Value Product HVV High-Value Vegetable IR Intermediate Result KII Key Informant Interview ISO International Standards Organization LOE Level of Effort LOP Life of Project LSP Local Service Provider MOU Memorandum of Understanding MPC Marketing and Planning Committees M&E Monitoring and Evaluation NRM Natural Resource Management PH Post-Harvest PIRS Project Indicator Reference Sheet PITT Performance Indicator Tracking Table PMEP Performance Monitoring and Evaluation Plan PMP Performance Management Plan PPP Public-Private Partnership REAP Rural Enterprise Poverty Alleviation SME Small and Medium Enterprises SMKK Sheba Menab Kallyan Kendra SOW Statement of Work SUS Shatkhira Unnayan Shongstha TOT Training of Trainers Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) iv USAID U.S. Agency for International Development USG U.S. Government WFLO World Food Logistics Organization ZOI Zone of Influence Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 1 EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS The purpose of the Cold Chain Bangladesh Alliance (CCBA) project final performance evaluation is to assess the extent to which the project has achieved its overall goal: “to establish Bangladesh’s first integrated cold chain to reduce post-harvest (PH) losses and deliver high-value agricultural products to market.” The evaluation will: • Review, analyze, and evaluate the effectiveness of the CCBA project in achieving the program objectives and contributing to USAID/Bangladesh’s efforts to increase agricultural productivity and increase private sector investment in agricultural activities and PH management. • Provide specific recommendations and lessons learned on strategies and approaches that USAID/Bangladesh should continue in its other relevant Feed the Future (FtF) activities and in any future cold chain development programs. USAID developed six evaluation questions: 1. To what extent was the CCBA project successful in achieving its objective to minimize PH loss? 2. Which commodities were found to be most suitable and economically viable for marketing using cold chain systems? 3. As CCBA is part of the Global Development Alliance (GDA), how successful was the activity in leveraging private sector investments and in developing public-private partnerships? 4. What have been the major constraints and opportunities with respect to sustainability of the interventions? What measures should be taken to enhance sustainability? 5. How effectively has CCBA integrated or incorporated gender in the interventions? What have been the challenges and opportunities, if any? 6. What are the best approaches from the CCBA project that could inform the design and implementation of similar activities in Bangladesh? This is a final performance evaluation of a pilot project. The findings are not necessarily meant to improve the management of the project but will help inform the USAID Mission in Bangladesh regarding its program design and development. Other missions where FtF is being implemented may also find the evaluation helpful. In addition, CCBA management under Winrock International (Winrock) and its private sector partner, Golden Harvest (GH), may benefit from a retrospective view of the project. PROJECT BACKGROUND In Bangladesh, once a crop leaves the farm on its journey to the consumer's table, estimated food waste—due to deficient PH practices—can be as much as 40 percent.1 To help prevent this waste, Winrock proposed a pilot project to USAID that would establish the first integrated cold chain in the country. The project was designed to reduce PH losses and deliver high-value agricultural products to market. 1 Winrock International (2013). Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 2 The three-year project incorporated four components that would collectively contribute to this overarching objective and that also would improve the availability of locally produced nutritious food. It also, ultimately, would improve the livelihoods of farmers engaged in selected subsectors of agriculture in the country. The project components are: (1) increased capacity of small and marginal farmers to grow high-value agricultural products; (2) improved agricultural market efficiency and planning; (3) increased private sector investment and capacity in cold chain management; and (4) increased compliance with international food safety standards. The project was designed through a PPP with GH, a Bangladeshi food processing firm. Winrock was tasked with bringing global expertise to manage the first three components of the project while the World Food Logistics Organization (WFLO) took the lead in the fourth component with CCBA supervision and assistance. Through a subsector analysis, high-value vegetables (HVVs), poultry, and dairy were selected as the focus of the project. In component one, CCBA with management from GH, was expected to provide assistance to 14,400 small and marginal farmers, the majority of whom were already HVV producers residing primarily within the FtF zone of influence (ZOI). However, the project suffered a major setback at the outset when GH displayed no desire to enter the frozen vegetable export business and invest in marketing ice cream and poultry. This unexpected exit created an implementation challenge: how could CCBA assist these beneficiaries as initially intended? In addition to this upset, component one also suffered from repeated hartals (strike actions) that included road blockages and, consequently, delayed project implementation schedules. This also financially impacted producers who were unable to market their crops, milk, and poultry. Activities related to the second component got off to a slow start and gained some traction in the midpoint of Year 2 when CCBA added a marketing staff member who was well-qualified in Bangladesh food marketing. The Evaluation Team (“the Team”) did find some success under component two, because groups of farmers organized and managed the PH process and were able to ship their produce to wholesalers. Because GH, for all practical purposes, was not engaged in project activities during the course of project implementation, little progress was made under component three. Under component four, even with the assistance from WFLO, activities were limited during the first half of Year 2 and continued into Year 3. The CCBA’s Project Monitoring and Evaluation Plan (PMEP) includes a total of 11 indicators to assess performance during the course of the project; two of these are custom indicators. Two standard indicators: (1) Increased Cold Storage Capacity and (2) Value of Private Sector Investment, were directly related to the anticipated GH involvement. No verifiable results have been demonstrated with regard to these indicators. In addition, CCBA was to inspire the creation of 21,600 jobs during the life of the project (LOP). The only jobs that CCBA recorded were held by new employees of GH, which were not related to CCBA activities or objectives. As a result, the Team did not find any jobs created. During early July 2015 and again in March 2016, USAID and Accelerating Capacity for Monitoring and Evaluation (ACME) staff members conducted Data Quality Assessments (DQAs) and corresponding field trips to CCBA sites. On these trips, participants analyzed evidence related to the project indicators. The DQA field trip reports suggested that the CCBA project had no systematic methodology for measuring PH loss. After an exhaustive analysis and extensive document review along with interviews with dozens of stakeholders and beneficiaries, the Team found that the depth and duration of training could be improved. Also, more thorough record-keeping would allow more effective follow-up of trainees. EVALUATION QUESTIONS, DESIGN, METHODS, AND LIMITATIONS This evaluation looks at 11 quarters of the CCBA project, from July 2013 to March 2016: five quarters in Year 1 (July 2013 to September 2014); four quarters in Year 2 (October 2014 to September 2015); and two quarters in Year 3 (October 2015 to March 2016). The Team interviewed USAID officers and CCBA management to clarify roles and fine-tune the design of Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 3 interview questions for four groups of stakeholders: (1) CCBA (as the implementing partner), the Government of Bangladesh (specifically the Bangladesh Agricultural Development Corporation, Bangladesh Agricultural Research Institute, and Department of Agriculture and Extension), and GH, the private sector partner; (2) nongovernmental organizations (NGOs); (3) retailers and wholesalers, local service providers (LSPs), farmers growing High Value Products (HVPs), and backyard female farmers; and (4) Collection Center managers and transporters. The Team used the quantitative PMEP indicators provided by CCBA in the progress reports submitted to USAID, but relied on qualitative tools or instruments to uncover the nuances of cold chain development. Knowing what works and why will enhance informed managerial decision-making and program development. Through key informant interviews (KIIs) and focus group discussions (FGDs), interviewees received special attention to glean what the mechanisms or practices were that aided or hindered cold chain development. From May 16 to June 7, 2016, the Team conducted interviews with stakeholders from seven districts (Barisal, Comilla, Dhaka, Gazipur, Jessore, Khulna, and Sylhet). The Barisal, Khulna and Jessore districts are in the ZOI. The Team interviewed women involved in backyard gardening. One hundred and twenty-four individuals participated in the KIIs or FGDs, The Team conducted nine FGDs and 24 KIIs, with 42 percent women participants. The Team analyzed the results of the DQAs performed in 2015 and 2016 and their corresponding field reports. The Team conducted data collection and analysis systematically by triangulating across stakeholder categories and interview methods (KIIs, FGDs, and on-site observations) to ensure the reliability and validity of the findings. Lack of verification of GH-related indicators remains an information gap. The Team could not access aggregated cost figures for each of the four components during implementation to assess results-based disbursements. FINDINGS AND CONCLUSIONS Q1. The Team found no evidence of a cold chain in Bangladesh, but did find evidence of a frozen food chain not linked to any of the project’s subsectors (fresh vegetables, poultry, or dairy). During Year 1, numerous trainings were conducted to minimize PH losses in HVVs and the project reached 9,630 producers (including 540 poultry producers) in PH best practices and production technologies. However, the Team did not find evidence of an improvement of PH losses. During Year 2, WFLO provided expertise to assess PH losses in four key vegetables. According to the CCBA report, 2015b: “Despite several efforts and interventions through various organizations and approaches, no significant improvement has been observed in the rate of PH loss in Bangladesh.” In the first quarter of Year 3, the project trained 115 beneficiaries in PH practices while claiming a 50 percent PH reduction. The calculations presented were not in line with the methodology proposed by the WFLO specialist. Thus, the reported baseline percentages of PH reduction do not appear credible. In the second quarter of Year 3, a total of 1,853 beneficiaries were trained in production and PH losses of HVV. While this constituted a 16-fold increase in the numbers trained (compared to the previous quarter), the CCBA did not explain how it was able to sustain the quality of training. The project did not achieve its primary objective of minimizing PH losses through the first integrated cold chain in Bangladesh. The cold chain links required to properly remove field heat were not evident in the locations where CCBA conducted interventions: regular refrigerated transport to serve beneficiaries was not readily available, and cold storage to producers or retailers was lacking. Finally, there was no coordination with retailers to ensure that a cold chain would be maintained from the field to the retail shelves. The Team concurs with the conclusion of the DQA regarding the lack of a systematic approach to documenting PH losses among beneficiaries. The Team found no documentation related to PH losses for any of the target crops. Numerous trainings on production and PH reduction were carried out, but the Team (through KIIs and a review of materials) noticed a limited capacity of the trainers and in the depth of the training manuals. They also found that the trainings were of short duration (1 day or less) with an apparent lack of follow-up with trainees. Q2. The Evaluation Team did not find evidence of economic viability in marketing vegetables through a cold chain system after three years of operating the pilot project. No crop budgets or Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 4 balance sheets were found as evidence or as a tool for decision-making to achieve economic viability. Winrock’s technical proposal (that led to the USAID/CCBA project) assumed that GH would be a substantial player in the fresh winter HVV business using a vegetable cold chain. However, senior GH management maintained that the company had never expressed an interest in becoming involved with the business of fresh vegetables in Bangladesh. Based on Winrock’s assumption that GH would be involved in HVVs, CCBA focused all its resources on the production of HVVs instead of exploring cold chain options with GH. During Year 2, the project conducted three trial shipments to emulate the real-world cold chain practice. The results were unsatisfactory and deterred further “pilot” experimentation. Prior to submission of the CCBA Technical Proposal, there was no study to determine whether there was a segment of the marketplace in Bangladesh willing to pay for the costs of providing HVVs managed through a properly developed cold chain. The Team was unable to find food business representatives who could appreciate the monetary value of vegetables with longer shelf life—the result of a well-maintained cold chain. Project documentation did not adequately address the extra costs involved in operating a proper cold chain (e.g., the cost of chill rooms at collection centers; plastic crates or fiberboard cartons for packing; shipping in refrigerated trucks; cold storage in major cities; cold chain displays at retail outlets; and backup generators at all levels). If these costs had been considered and included in a pilot small business plan, it’s more likely that a cold chain development agenda could have been advanced. Q3. Attempts to leverage private sector investment were unsuccessful. There is no evidence that GH was geared toward engaging in a cold chain related to fresh HVVs; their objective was to invest in a frozen food chain to support a local brand of ice cream. GH intended to use its cold chain investment for frozen food products, not cold chain food products. The public-private partnership between CCBA and GH was inadequately conceived and did not advance from the signing of the Memorandum of Understanding (MOU) in 2013. Apparently, USAID and Winrock had the mistaken impression that GH planned to be an active participant in the fresh HVV chain based on production within the FtF ZOI. GH said it perceived itself in a very different role, with poultry and dairy being their areas of business interest. The gap between CCBA and GH grew wider during the LOP due to the lack of communication and planning between the parties. For the CCBA project, the common goals of the alliance were not evident. While the financial commitment from GH fit the GDA norms, without a mutual understanding of common goals, the parties were, in effect, moving in different directions. In retrospect, actions could have been taken early in the project to mitigate the lack of a strong relationship. Winrock could have been more cognizant of implementation considerations and challenges prior to drafting a technical proposal. After the project was underway, Winrock also could have been aware of the growing gap between CCBA and GH and should have been more proactive. Q4. The sustainability of the CCBA intervention faced two major constraints. One was the compounding effect of the short duration of the pilot project (three years on paper) with the hartals (strike actions), and the unstable political environment that affected market activities. Pilot projects in agricultural marketing require more than five years to bloom. With shorter timeframes the measures of performance are unable to capture behavioral change, creativeness in the market environment, and the ability of beneficiaries to adapt their crops, products, or services according to market signals in Bangladesh’s volatile markets. The value of a pilot project is the identification of a sustainable pathway to change. This result may be diluted when activities cover a wide geographic area or when project themes become too diverse. Rather than looking back to its original constituency of well-organized farmers in Jessore district, the project expanded horizontally in the last two quarters, mostly outside the ZOI. This negatively impacted the potential sustainability of project interventions and missed the opportunity to ensure sustainable vegetable value chains that included small and marginal farmers. To help ensure a strong and sustainable platform, funding could have been allocated to increase training in marketing and managerial economics in the original seven Collection Centers. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 5 The other constraint to sustainability of CCBA was Winrock’s agreement to meet the USAID￾specified target indicators. As a result, the pilot appeared to be a captive of indicator targets that are more typically found in a development project rather than in a pilot project. It’s also possible that CCBA did not express options or concerns with USAID’s COR regarding the sustainability of the project or discuss ways to utilize the budget on activities with a higher probability of success. Q5. CCBA incorporated gender in its project interventions on a limited scale. Performance indicators show that women took 447 jobs in the first two years, equivalent to 42 percent of the jobs created by the project (1,055 in total). However, all jobs created were related to GH investments outside the ZOI and have not been verified. None of the agricultural activities contributed to job creation as interpreted by the CCBA M&E Unit under this standard indicator. Furthermore, less than one percent of the number of hectares that received U.S. Government (USG) support or that received support in improved technologies in landholdings were owned or managed by women (124 ha out of 15,463 ha). This is related to Bangladesh’s cultural traditions and land property rights. Gender was included in the subsector analysis as backyard gardening and poultry. The Team did not find evidence of scholarships, grants, or internships provided to women. However, the Team verified the positive effect of project activities on household hygiene and nutrition, in self-esteem, and in strengthening the cohesion of women´s groups in the districts visited (50 women in five FDGs). Cultural barriers do continue to limit women’s access to land and capital in Bangladesh. Women continue to be relegated to the home with fewer options for personal development. However, male interviewees from Gazipur district mentioned that women have more opportunities to engage in business, chiefly poultry, because of their close proximity to Dhaka. The indicator for job creation does not take into account women’s (or men’s) labor in agricultural or marketing activities. If a major thrust of the project is based on agricultural labor, it does not make sense to account for jobs created by GH only; custom indicators could possibly have captured this better. Q6. The project included a study document (CCBA 2014f, without authorship, dated April 2014) that was somewhat comprehensive, but it did not assess the demand for fresh vegetables by upscale retailers and consumers. The references to demand for fresh vegetables, poultry, or dairy are from secondary information without specifying whether a cold chain system was included. Furthermore, the Team found no evidence of interaction between the CCBA project and the USAID/AVC project, which had conducted a market study in 2014 to choose commodities. CCBA management was aware of that project but did not develop a working relationship with that project. The CCBA project followed a value chain approach to select and analyze the agricultural subsectors. With vegetables, poultry, and dairy subsectors selected, the project took a bottom-up facilitative approach to economic growth (adopting a more advisory role than that of implementer). The project emphasized production more than marketing, with 18,500 small and marginal farmers identified as the main beneficiaries. A PPP was put in motion with a low level of understanding between CCBA and GH. Complicating matters further: the two partners did not appear to be communicating. While the project was meant to benefit farmers predominantly in the ZOI, GH was primarily focused on dairy and poultry development using a cold chain for the domestic market. Also, CCBA realized during the first year of implementation that the success of the cold chain was unlikely because the market of urban consumers did not appreciate safe and nutritious fresh vegetables and, therefore, were unwilling to pay premium prices for high-quality products. The Evaluation Team could find no documented actions within the project to ensure that 2,000 to 2,500 farmers in Jessore and Khulna districts could or would receive more training to embrace the markets in a sustainable manner in order to capitalize on their already-acquired knowledge in production, post-harvesting, and marketing. Based on a review of the training manuals (Annex X) and on direct observations and KIIs and FGDs in four districts, the Team found that the depth and duration of training could be improved. Similarly, more thorough record-keeping would allow for follow-up with the trainees. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 6 The PPP between GH and CCBA did not include a thorough understanding of each other’s interests, and no provisions were made to address possible changes in their target sectors due to fluctuations in the Bangladesh market environment. If CCBA management had looked back at its own achievements, it may have been able to further enhance the capacity of small and marginal farmers in Jessore district to sustainably contribute to food security and income generation while making the vegetable value chain more efficient. Instead, CCBA opted to expand its geographical coverage rather than improving the knowledge and capacity of farmers already trained in the ZOI. CCBA may have expanded coverage to meet USAID’s LOP targets. Based on the findings of the Evaluation Team, whether higher-income urban consumers are likely to demand higher-quality produce, including fresh vegetables in Bangladesh, remains to be seen. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 7 PROJECT INTRODUCTION According to a study from the International Finance Corporation, South Asia Enterprise Development Facility, on agribusiness opportunities and constraints in high-value agriculture in Bangladesh: “as a result of sustained economic growth, rising incomes, and rapid urbanization, widespread shifts have occurred in consumer food demand. In particular, consumers are buying more high-value foods such as fish, meat, fruit, vegetables, and processed products.”2 It was estimated that by 2020 Bangladesh could need an additional 6.5 million metric tons of vegetables. The study recommended “special attention to ensure that smallholders are effectively linked to higher-value markets so that they are not excluded from the growing opportunities. The efforts of the private and public sectors will need to be closely aligned and well-coordinated in order to make the most of new opportunities.” In 2013, Winrock began implementation of a USAID cooperative agreement titled “Cold Chain Bangladesh Alliance” (CCBA). The primary objective of the CCBA pilot project (or untested concept) was “to establish Bangladesh’s first integrated cold chain to reduce PH losses and deliver high-value agricultural products to market.” Four components were incorporated in the CCBA project to collectively contribute to its overarching objective during the three-year LOP. Additionally, the project was expected to improve the availability of locally produced nutritious food and ultimately improve the livelihoods of farmers engaged in the production of selected subsectors of agriculture. The project components are: (1) increased capacity of small and marginal farmers to grow high-value agricultural products; (2) improved agricultural market efficiency and planning; (3) increased private sector investment and capacity in cold chain management; and (4) increased compliance with international food safety standards. The project was designed under the USAID GDA concept via a PPP with GH, a Bangladeshi food processing firm. According to the project design, Winrock would bring its global expertise as donor development implementers to manage the first three components, while WFLO would lead in the fourth component with CCBA supervision and assistance. Shortly after the project began operations, a preliminary agricultural sector analysis was conducted. As a result, HVVs, poultry, and dairy were selected as the focal points of the project. CCBA was tasked with providing assistance to 14,400 small and marginal farmers, the majority of whom were already HVV producers and were residing primarily within the FtF ZOI. The CCBA project, however, suffered a major setback at the outset when private partner GH decided to exit the frozen vegetable export business to market ice cream and poultry.3 The entire investment originally was designed to support a cold chain for HVVs but was instead (reportedly) invested in support of the GH ice cream brand Bloop. The exit of GH from the vegetable subsector created a significant challenge: how could CCBA assist the target beneficiaries as initially intended? In addition to the private sector partner changing course at the beginning of the project, CCBA was further challenged during the project’s first two years due to repeated hartals (strike actions) that included road blockages. This delayed project implementation schedules and also financially impacted producers who were unable to market their crops, milk, and poultry. 2 IFC-South Asia Enterprise Development Facility, 2008. 3 CCBA, First Annual Report, September 2014. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 8 With the loss of GH participation in HVVs, CCBA largely concentrated available financial and human resources in the first component. Activities related to the second component were largely ineffective until the midpoint of Year 2 when CCBA added a marketing staff member who was well￾qualified in Bangladesh food marketing. The Evaluation Team did find some success under component two, however, because groups of farmers organized to manage the PH process and were able to ship their produce to wholesalers. Because GH, for all practical purposes, sat “on the sidelines” during the course of project implementation, little progress was made under component three. Under component four: even with the assistance from WFLO, activities were limited during the first half of Year 2 and continued into Year 3. The CCBA’s Project Monitoring and Evaluation Plan (PMEP) includes a total of 11 indicators to assess performance during the course of the project; two of these are custom indicators. Two standard indicators (Increased Cold Storage Capacity and Value of Private Sector Investment) were directly related to the anticipated GH involvement. No verifiable results have been demonstrated concerning these indicators. In addition, CCBA was to inspire the creation of 21,600 jobs during the LOP. The only jobs that CCBA recorded were held by new employees of GH, and none of these positions were related to CCBA activities or objectives. As a result, the Team documented zero jobs created. During early July 2015 and again in March 2016, USAID and ACME staff members conducted Data Quality Assessments (DQAs) and corresponding field trips to CCBA sites. On these trips, participants analyzed evidence related to the indicators. The DQA field trip reports suggested no documentation related to PH losses for the target crops. After an exhaustive analysis involving extensive document review and interviews with dozens of stakeholders and beneficiaries, the Team found that the depth and duration of training could be improved. Also, more thorough record￾keeping would allow for more effective follow-up of trainees. In the following sections, the development problem and USAID’s response to address it are outlined. Also included are the strategy and project point of entry, the purpose of the evaluation, and the methodology design, findings, and conclusions for each of the evaluation questions. This is followed by the corresponding recommendations prioritized for each question. Finally, lessons learned are stated in the last section. The Team’s recommendations and lessons learned are intended to be used by USAID in its program design in Bangladesh, with possible application in other missions where FtF is being implemented. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 9 THE DEVELOPMENT PROBLEM AND USAID’S RESPONSE THE DEVELOPMENT PROBLEM Bangladesh is the world’s eighth-most populous country, home to over 168 million people living on just 147,570 km² of land. Over the past 20 years, Bangladesh has accelerated its economic growth and the percentage of the population living in poverty has declined significantly. Despite these successes, more than 50 million people still live in poverty and a large proportion of the population is underemployed. 4 Against this backdrop, evidence shows that food is going to waste due to several factors. According to Winrock,5 the majority of high-value agricultural products grown in Bangladesh, including vegetables, meat, and fish, are sold to consumers without proper handling, packing, or refrigeration, resulting in losses exceeding 40 percent. Products reaching the market do not meet consumer demands for quality. Other factors contributing to food waste include the following: • At the farm level, producers lack improved inputs and lack knowledge in market requirements and in the production and PH practices needed for food quality. • Small and marginal farmers are unable to achieve product volume required to enter commercial markets. • Women, due to cultural traditions, are excluded from income-generating opportunities and may be food-insecure, even in households with access to a supply of nutritious food. • Poor integration and coordination among value chain business interests—as well as corruption and dishonesty—inhibit the transmission of market information and lead to missed opportunities for value addition. • Deficient infrastructure, particularly roads and energy, compounds the limitations. USAID’S RESPONSE USAID’s response to this problem was the creation of a PPP between Winrock and GH to install the first cold chain in Bangladesh (CCBA).6 The CCBA development hypothesis asserted that sustainable increases in the availability, access, and use of domestically produced and nutritious foods would lead to reduced poverty and hunger in Bangladesh if integrated improvements take place through: • Increased income among small and marginal farmers due to improved sustainable, high-value agriculture • Improved agricultural market efficiency and planning • Increased private sector investment and capacity in cold chain management • Increased compliance with international food safety standards 4 Consultative Group on International Agricultural Research (n. d.) (https://ccafs.cgiar.org/bangladesh#.V2QZXo5akQE) accessed on June 15, 2016. 5 Winrock International (2013). 6 The cold chain refers to the process of preserving the quality of perishable, consumable goods from the point of harvest to the point of consumption (ACDI VOCA, 2005, “Cold Chain Association Development Project”). Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 10 Specifically, CCBA would leverage $10.2 million in new investments in cold storage infrastructure and operational capacity while linking 18,000 small and marginal farmers to the inputs, knowledge, and markets required to grow high-value agricultural products and to increase food security. High￾value agriculture is one of the best opportunities for farmers because many products can be sold for profit after a single cropping cycle on marginal land. Past Winrock projects have demonstrated that farmers who adopt high-value agricultural production are able to increase their incomes by 100 percent in the first year when they implement improved practices (such as multi-cropping and Good Aqua/Agricultural Practices, GAP). Increased production of safe, traceable, high-quality, well￾preserved, and properly handled products will stimulate new investments in PH processing, storage, and transportation—essential value chain functions that create off-farm jobs (CCBA, 2013). The proposed strategy began with market demand: • To increase production linked to markets, CCBA would consider agro-ecological conditions, infrastructure, and market/consumer demand through a subsector analysis and mapping exercise completed by CCBA staff in consultation with GH. • Selection of locations and products: A preliminary assessment carried out by Winrock and GH identified Chittagong, Sylhet, Khulna, and Dhaka as high-potential divisions for production of poultry for meat, fresh vegetables, and dairy. • Selection of groups of small and marginal farmers: Within these divisions, CCBA would identify production areas—clusters of approximately 200 smallholders—to match product demand with local capabilities. • Working with well-organized groups, such as the Bangladesh Fruits, Vegetables, and Allied Products Exporters Association, or newly created groups, CCBA would establish Marketing and Planning Committees (MPCs) consisting of farmers within a production area and local traders. • The MPCs would play an important role in organizing, improving, and managing small-scale Collection Centers to ensure seamless and timely delivery of products and improve buyer￾seller transaction efficiency. Proposed entry point and mechanisms: • Collection Centers would be the entry point for small and marginal farmers into a pilot model of food collection and distribution that would be the centerpiece of GH’s $10.2 million investment. GH had prepared a business plan that outlined the steps needed to establish an International Standards Organization (ISO)-compliant cold chain that ultimately would serve 22 districts and span all seven divisions of Bangladesh. • The network may include a 60,000-ft3 central depot in Dhaka, four divisional hubs of 48,000- ft3 each (Chittagong, Sylhet, Khulna, and Dhaka), and a network of rural hubs of 24,000-ft3 each (at tentative locations in Khulna, Jessore, Faridpur, Barisal, Chuadanga, Sylhet, and Chittagong). • GH planned to gradually put in place a fleet of 50 small and large temperature-controlled refrigerated trucks and a computerized information management system. Through direct supply of products, use of backhaul, and the rental of a warehouse and transport facilities to allied industries, CCBA would optimize operating costs and profitability while increasing food security through higher incomes for producers. According to Winrock, the CCBA approach was scalable. Once the pilot cold chain was established, additional hubs could be refurbished or established to support the hub-and-spoke scheme in Khulna, Jessore, Faridpur, Barisal, Chuadanga, Sylhet, and Chittagong. GH’s involvement in all aspects of implementation would enable it to assume ownership for interventions aimed at integrating small and marginal farmers into the value chain. Similar to its role on Winrock projects in Indonesia and Pakistan, WFLO would design and provide support, training, and specialized interventions on cold chain development, association-building, and international standards and best practices for cold chain associations and enterprises. The governance of the project: Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 11 A steering committee comprised of USAID, GH, Winrock, and WFLO would guide the Alliance, and could also include representatives of interested business associations. CCBA would work in close coordination with USG food security and economic growth programs, such as “Poverty Reduction by Increasing the Competitiveness of Enterprises” (PRICE). It also would work through planned investments in the horticulture and aquaculture value chains. This would help ensure that CCBA investments in infrastructure provided opportunities for producers in high-value, export-oriented sectors such as aquaculture and domestic markets for meat and dairy. The project would look at the cold storage potential for products, potentially under a warehouse receipts system, to take advantage of price fluctuations and reduce the sale of products at unfavorable market rates. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 12 EVALUATION PURPOSE AND EVALUATION QUESTIONS EVALUATION PURPOSE The purpose of the CCBA final performance evaluation is to assess the extent to which the CCBA project has achieved its overall goal: “to establish Bangladesh´s first integrated cold chain to reduce PH losses and deliver high-value agricultural products to market.” The evaluation will: • Review, analyze, and evaluate the effectiveness of the CCBA project in achieving the program objectives and in contributing to USAID/Bangladesh’s efforts to increase agricultural productivity and increase private sector investments in agricultural activities and PH management. • Provide specific recommendations and lessons learned on strategies and approaches that USAID/Bangladesh should continue in its other relevant FtF activities and in any future cold chain development programs. EVALUATION QUESTIONS 1. To what extent was the CCBA project successful in achieving its objective to minimize PH loss? 2. Which commodities were found to be most suitable and economically viable for marketing using cold chain systems? 3. As CCBA is part of the GDA, how successful was the activity in leveraging private sector investment and developing public-private partnerships? 4. What have been the major constraints and opportunities with respect to sustainability of the interventions? What measures should be taken to enhance sustainability? 5. How effectively has CCBA integrated or incorporated gender in the interventions? What have been the challenges and opportunities, if any? 6. What are the best approaches from the CCBA project that could inform the design and implementation of similar activities in Bangladesh? Four project components are the backdrop to respond to these questions: (See SOW in Annex I for more details.) • Increased capacity of small and marginal farmers to grow high-value products: capacity￾building in production and post-harvesting. • Improved agricultural market efficiency and planning: empowering organized farmers through two key institutions: Collection Centers and their MPCs. • Increased private-sector investment and capacity in cold chain management: USAID leveraging private sector investments that would link small and marginal farmers with markets. • Increased compliance with international food safety standards: capacity-building for the private sector to meet standards needed for international trade. This final performance evaluation is special because the pilot project was meant to test an innovative idea, i.e., cold chain development in Bangladesh through a PPP. Thus, the good practices and lessons learned are not for the CCBA management but to help the USAID Mission enhance its program design and streamline strategic partnerships with the private sector. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 13 EVALUATION METHODS AND LIMITATIONS This evaluation looks at 11 quarters of the CCBA project, from July 2013 to March 2016: Year 1 includes five quarters (July 2013 to September 2014); Year 2 includes four quarters (October 2014 to September 2015); and Year 3 includes two quarters (October 2015 to March 2016). The last quarter of the project, April to June 2016, is still ongoing and is not included in this evaluation. The Team used quantitative and qualitative tools to assess the final performance of the CCBA project. The Evaluation Design Matrix contains the six key evaluation questions, the information sought, information sources, data analysis methods, and limitations associated with the methods (Annex II). Prior to conducting stakeholder consultations and interviews, the Team interviewed USAID officers and CCBA management to clarify roles, fine-tune the design of questionnaires for different stakeholders, and prepare a fieldwork schedule. (See Annex III.) Four groups of stakeholders were identified: (1) CCBA (as the implementing partner), the Government of Bangladesh (specifically the Bangladesh Agricultural Development Corporation, Bangladesh Agricultural Research Institute, and the Department of Agriculture and Extension), and GH, the private sector partner; (2) nongovernmental organizations (NGOs); (3) retailers and wholesalers, local service providers (LSPs), farmers growing HVPs (questions asked: 1, 2, 4, and 5); and female backyard farmers; and (4) collection center managers and transporters (questions asked: 1, 4, and 5). The Team used the quantitative PMEP indicators provided by CCBA in the progress reports submitted to USAID. Most importantly, however, the evaluation relied on qualitative tools or instruments to uncover the nuances of cold chain development. Knowing what works and why will enhance informed managerial decision-making and program development. The Team interviewed beneficiaries: small and marginal farmers who had received assistance and training in high-value crop production and PH practices; collection center managers who had received assistance in cool and cold storage, marketing and distribution; and entrepreneurs of small and large marketing outlets. The Team also interviewed the implementing partner, private sector partner and government officers. Interviewees, through either KIIs or FGDs, received special attention to glean what mechanisms or practices induced or hindered cold chain development. (See data collection instruments in Annex IV. Sources of secondary information are provided in Annex VIII.) From May 16 to June 7, 2016, the Team conducted interviews with stakeholders from seven districts (Barisal, Comilla, Dhaka, Gazipur, Jessore, Khulna, and Sylhet) (See Figure 1). The districts of Barisal, Khulna, and Jessore are in the ZOI. The Team selected the sites in consultation with CCBA management, who also organized the interviews with stakeholders on site. There was no time allowance for random selection of stakeholders to be interviewed. However, it was possible for the Team to obtain plenty of information regarding project accomplishments, verification, and remedial actions present or absent. Had the Team randomized the selection of interviewees, the findings might likely be “less positive.” The Chief of Party (COP) attempted to be involved in the evaluation meetings and it was necessary to keep him absent in order that the team select and discuss the project objectively with stakeholders. The Team interviewed women who are involved in backyard gardening and at different levels in the value chain to understand their limitations and opportunities in being promoting to managers or decision-makers. One hundred and twenty-four individuals participated in the KIIs or FGDs. The Team held nine FGDs and conducted 24 KIIs, with 42 percent women participants. (See Evaluation Coverage in Annex V. See also Annex IX for the list of people contacted.) The Team used performance indicators to ascertain CCBA’s achievements against its expected and agreed milestones. This entailed computation of proportions or percentages. (See Quantitative Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 14 Assessment and the Performance Indicator Tracking Table (PITT) in Annexes VI and VII, respectively.) Qualitative analysis of information gathered through direct observation, semi￾structured interviews, and FGDs were based on a guide with open-ended questions related to each of the six evaluation questions that were linked to the 11 indicators in the PMEP. The Evaluation Team assembled field notes to identify key information on topics relevant to assessing the evaluation questions. Only responses provided by at least three participants representing at least two stakeholder groups were considered valid evidence for a finding. An exception to that was Question 3 wherein answers from government officials, the private sector, or the implementing agency were used (all in the first group of stakeholders) to triangulate information. The CCBA provided the Team with lists of stakeholders, but due to time constraints, the Team was unable to randomly select stakeholders from these lists in different districts. PMEP indicators may represent a measure of performance, but they are entirely dependent on whether they properly represent the complexity of the processes being assessed. There was no need to quantify margins of error associated with questions related to processes rather than those associated with quantifiable outcomes. Such a process question would be, for example: How, and to what extent livelihoods or agricultural practices have changed as a result of USAID support? Only in the case of percentage of PH losses did the Team ask for numbers, but the results contrasted with those captured by the project. (See section on Question 1 and Annex VI to ascertain how some quantitative figures provided by the project were not considered credible.) The Team analyzed the results of the DQAs performed in 2015 and 2016 and also looked at the corresponding field notes and the indicators that USAID uses for accountability purposes. The Team used qualitative methods to offset what the performance indicators did not explain. Relying extensively on qualitative data, however, also has limits and risks relying on anecdotal evidence that may not fairly represent results. To compensate for this potential limitation, the Team conducted the data collection and analysis systematically by triangulating across stakeholder categories and interview methods (KIIs, FDGs, and on-site observations) to ensure the reliability and validity of findings. Lack of verification of GH-related indicators remains an information gap. Information relevant to the project prior to its implementation was largely absent. The lack of a proper project library of documents and data limited the quality of information compiled. Moreover, crop budgets were nonexistent. The Team could not access aggregated cost figures for each component during project implementation. Therefore, it was not possible to assess results-based disbursements. Likewise, classified or confidential restrictions impeded the Team’s access to information that would have explained how the project operates. Moreover, no business plans were made available. No matter how meticulous the quantitative and qualitative assessments were, restricted access to information severely limited the reach of this evaluation. The Team presented its preliminary findings, conclusions, and recommendations to USAID on June 8, 2016, to verify that the results of this evaluation were congruent with their perceptions of the project and that the recommendations were actionable and useful to inform decision-making and Figure 1. Cold Chain Bangladesh Alliance Intervention Sites Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 15 program development. Observations and recommendations from this process are integrated in this report and delivered as scheduled. (See Annex XI.) Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 16 FINDINGS AND CONCLUSIONS Question 1. Objective and reduction of post-harvest (PH) loss To what extent was the CCBA project successful in achieving its objective to minimize PH loss? Findings The Team noted the following statement in the Year 1 Annual Report: This statement suggests that there was already a cold chain in operation after the first 12 months of the project. However, the Team could not find evidence of this either at the beginning or at the conclusion of the project. CCBA fielded two international consultants to assess the cold chain system in Bangladesh. . . . From the assessment, it was revealed that additional investments would be required . . . to establish an effective nationwide cold chain for fresh produce in Bangladesh.7 The Team realized that there is an active frozen food chain8 in Bangladesh, but found no cold chain applicable to fresh vegetables.9 The institutional factors (governance) related to the absence of a cold chain in the country are addressed in the following questions. The CCBA project conducted numerous trainings aimed at minimizing PH loss in HVVs. In addition, CCBA organized a Training of Trainers (TOT) on the “Post-Harvest Handling of High-Value Crops” in February 2014, as reported in the Year 1 annual report. During the same year, CCBA reportedly trained 9,630 producers (including 540 poultry producers) in PH best practices and production technologies. However, at the end of the first project year, CCBA was unable to document an improvement in PH losses. At the conclusion of Year 210, activities reportedly continued in an effort to mitigate PH losses. During the year, WFLO provided expertise to assess PH losses of HVV and, in turn, conducted a seminar on the subject. The WFLO intervention included a comprehensive analysis of PH losses for four key HVV crops. The analysis included the following comment about past attempts to mitigate the PH-loss problem: 7 CCBA (2015b). 8 Golden Harvest http://www.goldenharvestbd.com/ ; this was also verified during interviews with GH management. 9 Jany et. al. (2008) or any of our interviews with relevant stakeholders. 10 CCBA (2015b). “Despite several efforts and interventions through various organizations and approaches, no significant improvement has been observed in the rate of post-harvest loss in Bangladesh.” “The project also attempted to address post-harvest losses, which are very high in Bangladesh, by using an efficient cold chain management system consisting of a coordinated and series of uninterrupted, temperature-controlled links in a logistics chain, where several actors play specific roles.” Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 17 Following the PH analysis, CCBA held a workshop in Jessore district with a broad range of stakeholders specifically to address PH losses. As the project moved into its third year of activities, according to the first quarterly report (CCBA 2016a), PH training continued with 115 total beneficiaries attending the course over four sessions during the first quarter. At this juncture of the project, CCBA claimed credit for a significant reduction in PH losses due to project interventions. Without identifying specific crops, CCBA (2016a) claimed a 50 percent reduction in PH loss (6 percent down to 3 percent). However, an associated chart accompanying this statement suggests a baseline of 4 percent PH loss down to 3 percent. These sets of numbers are not in agreement with each other. All of the calculations utilized in this report regarding PH losses conflict with the methodology and conclusions reported by Dr. C. Hell, WFLO Consultant (WFLO & WI, 2014), in the Year 2 annual report. During the second year of the project, Dr. Hell analyzed PH losses at the farm level for four of the vegetable crops that were the subject of CCBA training and guidance. For instance, reduction in PH loss in Chili was analyzed and found to achieve only 0.5 to 1.0 percent PH loss at the farm level. The other crops were found to have PH losses at the point of harvest between 7 percent and 15 percent. Therefore, the reported baseline percentages for PH loss suggested in the first quarter of Year 3 were questionable due to conflicting information. Furthermore, the PITT in Annex VII shows reductions of 3 percent PH losses for both Year 2 and Year 3, relative to a 15 percent target. M&E specialists from the project could not explain the lack of congruence between indicators in the PITT and the narratives in the project reports because they were not involved in reporting. CCBA conducted another TOT session during the first quarter of Year 3 that included, among many topics, PH-loss reduction. During the second quarter of Year 3, a total of 1,858 beneficiaries reportedly received training on production and PH losses of HVVs. This increased the number trained 16-fold (from 115 to 1859 beneficiaries), but the Team found no mention of project steps taken to ensure a high-quality of training in a (short) three-month timespan. Project managers wanted to meet the indicators in the PMEP in the last year of the approved program. The Evaluation Team did not have time to assess those trained but did talk to trainers in the field and those in the project headquarters. The Team did not assess the differences between the impact of 155 and 1859 participants but has serious reservations about the quality of training reported in the last two quarters and analyzed in this evaluation. A report from the USAID/CCBA DQA field trip in March 2016 stated: The Team could not assess the quality of information after the DQA in 2016 because the second quarterly report of 2016 was not available. Thus, the first quarter of 2016 (October through December 2015) was the last quarter that the Team analyzed for this evaluation. The discussion presented in the quantitative analysis, PPTT, suggests, however, that some problems were not fixed in the last quarter of the LOP. This includes, for example the indicators pertaining to the number of jobs attributed to FTF; the total increase in installed storage capacity, the value of new private sector investment, or the food chain leveraged by FTF implementation. (See Annex VI.) The Team perceived an attitude that the completion of the DQA was not significant beyond what had already taken place because the project was coming to an end in a few weeks. “Regarding the reduction of post-harvest loss, no significant evidence was presented on improvements towards CCBA systematically collecting data to measure post-harvest loss.” Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 18 Despite the lack of documentation, the numerous trainings and the focus on PH losses logically should have resulted in a change of behavior in the areas where CCBA was operational. However, the team observed that: • The training materials were generic and often did not incorporate a cold chain concept.11 • The capability of the trainers interviewed was limited. The Team interviewed a few trainers from an NGO towards the end of its implementation plan, when the project was disbanding. • The timeframe of training sessions was too short.12 • There was a lack of consistent follow-up with trainees.13 Conclusions Due to technical factors, the project did not achieve the primary objective of minimizing PH losses through the first integrated cold chain in Bangladesh. The cold chain links required to properly remove field heat were not evident in the locations where CCBA conducted interventions: regular refrigerated transport to serve beneficiaries was not readily available, and cold storage to producers or retailers was lacking. In addition, there was no coordination with retailers to ensure that a cold chain would be maintained from the field to retail shelves. The Team concurs with the conclusion of the DQA noted above regarding the lack of a systematic approach to document PH losses among CCBA beneficiaries. The Team found a variety of general statistics reported with respect to PH losses in Bangladesh. Other than the analysis conducted by Dr. Hell (which was limited in scope: too short in time, with few locations and crops), the Team could find no documented evidence related to PH losses specifying the crops targeted for intervention by CCBA. Question 2. Most suitable and economically viable commodities Which commodities were found to be most suitable and economically viable for marketing using cold chain systems? Findings To assess commodities to be selected by CCBA, the project conducted a “Selection and Analysis of Agricultural Subsectors” study (CCBA, 2014f). The document was written under the assumption that, to a large degree, CCBA activities would be tied to GH. However, due to the reality of GH’s business plans this basic assumption was incorrect. For example, during a KII and a FGD with GH staff, the Evaluation Team was informed that poultry would be frozen rather than chilled after processing. Concerning which fresh vegetables would be suitable and economically viable for marketing using a cold chain system, the Team could find no evidence within any CCBA project reports to provide information on this matter. In addition, no crop budgets or balance sheets were found as evidence or as a tool for decision-making by CCBA project staff. The revised Technical Proposal put forth by Winrock in 2013 (that led to the USAID/CCBA project) depended on GH becoming a substantial player in the fresh winter HVV business employing a cold chain for fresh vegetables—an assumption that proved to be inaccurate (CCBA, 2015b). Frequent references appear in the Year 1 work plan, implementation plan, and quarterly and annual reports suggesting that GH, in partnership with CCBA, would become a powerhouse in fresh 11 See Assessment of training manuals in Annex X. 12 CCBA (2015b). 13 USAID/ACME (2015a, 2015b). Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 19 vegetable marketing via a cold chain. However, senior GH staff maintain that the company had never expressed any interest in becoming involved with the business of fresh vegetables in Bangladesh. The GH staff interviewed did mention engaging in a limited manner in supplying frozen vegetables for export via a third party that sells the inventory. From a business perspective, however, GH must have concluded that there was no profit in sourcing vegetables from the beneficiaries of CCBA and consistently asserted that it had never implied interest in the business of fresh vegetables other than serving as an occasional transporter. As the initial year of the project progressed it became evident to the COP that GH as the private sector partner within CCBA would not (as initially anticipated) be creating a cold chain that would focus on vegetables. Therefore, the project manager decided to use available project resources to specifically improve the production capabilities of HVV producers. During Year 2 of the project, CCBA conducted three trial shipments intending to emulate a real￾world cold chain exercise. Based on the CCBA Year 2 annual report, the unsatisfactory results actually served as a deterrent to further pilot experimentation.14 During a field trip to the FtF ZOI, the Team visited one of the CCBA Collection Centers with the most highly trained producers and active MPCs. (See cover photo of this report.) This operation was located in the middle of a major bitter gourd production area. At this location the producers were collectively preparing an order for a client. Sorting was evident, although there was no effort to collect the rejected material for any secondary use. The client (probably sourced by CCBA staff) insisted on packing 15 kg of bitter gourd into plastic crates that could hold only 13 kg without damaging the produce. This almost certainly resulted in significant additional PH loss. The price being paid was only 0.0875 USD/kg—well below its break-even price in the FtF ZOI which was just under 0.188 USD/kg at that time (DAI, 2014). The Evaluation Team could find no documentation or rationale for operating where cost of production was higher than what the product could be sold for. Based on the documentation made available to the Team and on FGDs with beneficiaries, the Team found little evidence to suggest that producers of vegetables had changed substantially to new vegetable crops. The Team did see some reports on experimentation with new crops and on the adoption of new hybrid vegetable varieties, as well as an acceptance of new technologies.15 Interviews with a small group of dairy producers trained through CCBA revealed that prior to the training they had been producing milk, but this was at low production levels and under limited stock and feeding conditions. Based on a poultry FGD, the Team had the impression that many of the poultry producers were new to this activity. Milk production and distribution, as envisioned by Winrock (in 2013), required a cold chain system. The Team found that CCBA’s private partner, GH, was investing in a first-class cold chain to supply milk to its Bloop ice cream brand. Producers trained by CCBA in the Sylhet district anticipated being suppliers to GH; however, the Team saw no evidence that these producers would be sourced by GH to supply the volume of milk required to meet the Bloop demands. Conclusions What essentially remains unknown is whether there are retailers in Bangladesh and, in turn, consumers of fresh vegetables, who will pay for the expense involved with maintaining a cold chain 14 CCBA (2015f). 15 CCBA (2015b). Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 20 for HVVs. Nevertheless, the primary objective of the CCBA project was: To establish Bangladesh’s first integrated cold chain to reduce PH losses and deliver high-value agricultural products to market. Prior to the submission of the CCBA Technical Proposal, it was not determined whether a segment of the marketplace in Bangladesh would pay for the costs related to providing any type or variety of HVVs that had been managed via a proper cold chain. In addition to testing the value to the marketplace for cold chain vegetables, future market tests should also assess if retailers and consumers will be willing to pay extra for the guaranteed absence of “mystery” chemicals, such as formalin, and ensure that all products have been properly graded. During the evaluation mission in the field, the Team was unable to find real-world food business representatives who understood the monetary advantage of vegetables “enjoying” a longer shelf life due to a well-maintained cold chain. In addition, the Team found that CCBA documents that encouraged a cold chain for vegetables did not adequately address the extra costs involved to maintain a proper cold chain. The following are the major costs/expenditures to consider (GCCA, 2015): • Chill rooms at Collection Centers powered by a generator or a generator standby • Shipping in plastic crates—adds a 40 percent increase in costs to any transportation method • Shipping in well-designed fiberboard cartons adds even more packaging expense • Shipping in refrigerated trucks adds substantially to transportation costs • Cold storage facilities costs in Dhaka or other major cities • Adding proper cold chain displays at retail outlets • Cold storage at food service and institutional facilities • Back-up generators at all levels The Evaluation Team asserts that if these costs had been investigated and incorporated into a pilot small business plan, the project would have been more successful in advancing a cold chain development agenda. Question 3. GDA and leveraging private investments As this is part of the GDA, how successful was the activity in leveraging private sector investment and developing a public-private partnership? Findings Late in 2013, private sector partner GH and CCBA entered into a Memorandum of Understanding (MOU)16 in which GH agreed to invest $10.2 million in temperature-controlled trucks and warehouses. The MOU did not specify which markets would be targeted as part of the PPP, but did mention a focus on “poultry, vegetables, fruits, and dairy.” Although GH claims that it did purchase the cold chain rolling stock and refrigerated storage, as agreed, the company also asserts that it never intended to engage in a cold chain related to fresh HVVs. Rather, its objective was to invest in a frozen food chain to support Bloop ice cream. All of GH’s cold chain investment was intended for frozen food products, not for cold chain food products. The PPP agreement in the CCBA project was inadequately conceived and did not advance over time. From the signing of the MOU in 2013, USAID and Winrock had the mistaken impression that GH 16 GH and Winrock. Signed by M.S. Choudhury, Dec. 4, 2013, and by David Norman, VP E&A, Oct. 14, 2013). Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 21 planned to be an active participant in the fresh HVV chain based on production within the FtF ZOI. GH claims that it had a very different relationship, with poultry and dairy being their areas of business interest. Based on interviews with GH and CCBA staff, the chasm between CCBA and GH grew wider during the LOP due to lack of communication and planning between the parties. As reported via the FtF/USAID website (https://www.feedthefuture.gov/resource/global￾development-alliance-gda-information), the typical criteria for successful GDAs includes: • At least 1:1 leverage (cash and in-kind) of USAID resources • Common goals defined for all partners • Jointly defined solutions to a social or economic development problem • Nontraditional resource partners (companies, foundations, etc.) • Shared resources, risks, and results • Innovative, sustainable approaches to development Although the CCBA project was not successful from the hoped-for PPP perspective, the presence and activities of the project did spur interest among small and medium enterprises (SMEs). “The business links with various stakeholders resulting from attending a CCBA workshop opened new opportunities for my agricultural input business. As a result, I increased my production of Vermicompost, enhanced my factory size, invested in equipment and employed new staff. As a result of CCBA training, my sales volume has literally increased from 63 USD to 940 USD.” - Abdul Ahad Shahin, LSP, Sylhet Conclusions In the case of the CCBA project, as evidenced by the MOU between GH and Winrock, the common goals were ill-defined. While the financial commitments from the private partner fit the GDA norms, without a mutual understanding of common goals, the parties were in effect moving in separate directions. GDAs in developing nations like Bangladesh are most successful when the private sector can provide human resources with knowledge that benefits local stakeholders. In the case of CCBA and GH, instead of being capable of offering skilled personnel, the private sector partner needed training and services to assist the beneficiaries of the project (CCBA 2015b). In turn, FtF is dedicated to assisting small and marginal farmers. The result was lack of common interests. In retrospect, actions could have been taken to strengthen the weak relationship early in the project. Winrock could have been more cognizant of the reality of the situation prior to drafting a technical proposal. After the project was underway, Winrock could have been aware of the growing chasm between CCBA and GH and been more proactive. Question 4. Constraints and opportunities for sustainability What have been the major constraints and opportunities with respect to sustainability of the interventions? What measures should be taken to enhance sustainability? Findings The sustainability of the CCBA intervention, measured as the probable continuity of processes initiated by the pilot project, faced two major constraints. The first constraint was related to the Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 22 compounding effect of a short time17 duration of the pilot (three years on paper), “but in reality one-and-a-half years,” as stated by GH management), in addition to hartals and an unstable political environment that negatively affected market activities. The second constraint was CCBA’s commitment to meet the USAID approved target indicators. The CCBA pilot project achieved a good level of organization in linking producers with markets in the Jessore district during the first two years (CCBA, 2015b). In the last two quarters of the third year, however, the pilot more than tripled its reach to 35 Collection Centers in nine districts (CCBA, 2016a) to meet the indicator targets in the PMEP. Rather than looking back to its original constituency of stakeholders, the project expanded horizontally. This negatively impacted the potential sustainability of project interventions and missed the opportunity to ensure sustainable HVV chains in the ZOI. The Team found no documentation reflecting concerns from CCBA or USAID about the very high quantitative expectations in the PMEP. It seems, rather, that CCBA and USAID had set high expectations for the relationship and interactions between CCBA and GH (as Winrock had envisioned in 2013) in order to meet the PMEP indicator targets. (See Annex VI and Annex VII.) Figure 2: Usage of financial resources, Q1 ending Sep. 2013 to Q11 ending Mar. 2016 During each of the last two quarters in Year 3, the project used up more than half-a-million U.S. dollars, more than twice the average of the previous quarters (Figure 2). However, the Team was unable to identify how the use of financial resources corresponded to each of the four components of the project. The Team acknowledges that in the last three quarters the project managed to develop the interests of high-end retailers to reduce PH losses and to obtain certification for proper cold chain management (KII; CCBA, 2016a). However, there is no evidence to suggest that all 35 Collection Centers will continue to function when the project comes to an end. It has taken two years for the Collection Centers in Jessore to be organized and functional (albeit subject to improvement), but the 28 newly created centers are unlikely to have achieved a similar level of organization and functionality in less than half a year. In the 11 quarters evaluated, there were three different 17 Gates Foundation (2013). 0 500 1000 1500 2000 2500 3000 0 2 4 6 8 10 12 Thousands of USD Quarter Cumulative Expenditures CCBA Project Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 23 marketing specialists; this poses concerns about the project’s ability to serve more than the 15,000 farmers reported to benefit directly from CCBA. Conclusions Agricultural marketing pilots require more than five years to bloom, and the measures of performance do not capture behavioral change, creativeness in the market environment, and the ability of beneficiaries to adapt their crops, products, or services according to market signals in Bangladesh’s volatile markets. The surge in expenditures during the past two quarters could have been reallocated to conduct more training in marketing and managerial economics in the seven Collection Centers. This would help to ensure a strong and sustainable platform of 2,000 to 2,500 farmers that eventually could be incorporated into ongoing projects on value chain development or for expansion of input/service procurements. The value of a pilot project is the identification of a sustainable pathway to change and not its widespread geographic or thematic dissemination. The project appeared to be held captive by the indicator reporting; CCBA, evidently, felt responsible to deliver targets typical of a development project (such as the creation of 20,000 jobs and expansion outside of the ZOI) instead of those suited to a pilot project. Thus, CCBA did not express options or concerns regarding the sustainability of the project to USAID’s COR and could have refocused its budget to improve the skills of small and marginal farmers in the ZOI with a higher probability of success. The project could have benefitted more from a “quality over quantity” approach. Question 5. Gender integration How effectively has CCBA integrated or incorporated gender in the interventions? What have been the challenges and opportunities, if any? Findings The CCBA project incorporated gender in its interventions on a limited scale. The PITT shows that women took 447 jobs in Year 1 and Year 2, equivalent to 42 percent of the jobs created by the project (1,055). Nevertheless, all the jobs created, including those taken by women, were associated with GH investments outside the ZOI, which the Team was unable to verify. None of the agricultural activities accrued to job creation as interpreted by the CCBA M&E Unit under the standard indicator. However, less than one percent of the number of hectares that received USG support or that received support in improved technologies in landholdings were owned or managed by women (124 ha out of 15,463 ha). This is related to Bangladesh’s cultural traditions and land property rights. Gender training was provided at all the sites visited by the Team. Gender was included in the subsector analysis as backyard gardening and poultry. The Team did not find evidence of scholarships, grants, or internships provided to women. However, the Team verified the positive effect of project activities on household hygiene and nutrition, in self-esteem, and in strengthening the cohesion of women’s groups in the districts visited. Below are some gender issues addressed by CCBA under the four different components. • Increased capacity of small and marginal producers to grow high-value agricultural products: The Team did not find evidence of entrepreneurial training for women to increase their income through high-value agricultural products. However, the Team found evidence of enthusiastic groups of women (five out of five FDG including 50 women in Jessore, Khulna, Barisal, and Sylhet districts) preparing eggplant pickles, a new product line, with positive responses from their families and limited income generation. In some areas, men have been sensitized about women’s involvement in different activities. The project has helped create all-women producer groups for backyard gardening and poultry. Men in an FDG from Gazipur Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 24 mentioned that women have more opportunities to engage in the poultry business because they are closer to the city of Dhaka which has a less-traditional culture. • Improved market efficiency and planning: The presence of women in the MPCs was very low or nonexistent. However, the project has introduced value-added processing on a limited scale, predominantly in Jessore and Barisal. • Strengthened capacity and new investment in cold chain facilities and operational management: The Evaluation Team found no evidence of women receiving training in cold chain internships or managerial skills in agriculture, marketing, or management. Likewise, they found no evidence of women’s involvement in WFLO/Global Cold Chain Alliance (GCCA) training, or in accessing grants for university degrees, or for short courses in PH management. CCBA does not have a designated gender group, and all gender-related activities are carried out by collaborating NGOs. Conclusions Cultural barriers in Bangladesh continue to limit women’s access to land and capital. Women continue to be relegated to the homestead with fewer options for personal development. On the other hand, male interviewees from Gazipur district mentioned that because of the close proximity to Dhaka, women have more opportunities to engage in businesses, chiefly in poultry. Women in Barisal, Jessore, Khulna, and Sylhet districts are very enthusiastic about the training received by CCBA (50 women in five FGDs). They mentioned their increased self-esteem and willingness to take some risks to borrow small amounts of capital for potential businesses or revolving funds. It was evident to the Team that both literacy and numeracy are two factors that limit women’s access to higher-level positions. -“My children now get more nutritious food and their health-related problems have been reduced significantly. What else do you want as a mother? And I am attending this meeting with national and international men. My husband encouraged me to come here. Is that not an indication of my freedom, my self-esteem or my empowerment?” - Safia, Women Lead Farmer; Village: Arjikalikapur, Union: Chandpasa, Upazilla: Babugonj The indicator for job creation does not take into account women’s (or men’s) labor in agricultural or marketing activities. If a major thrust of the project is based on agricultural labor, it does not make a lot of sense to account for jobs created only by GH. Custom indicators could be used to capture women´s employment. The CCBA project has added gender equity as an objective throughout this project—“from project management to all points of the value chain.” In fact, with the exception of the poultry sector, the Evaluation Team found little evidence of “gender equity.” Question 6. Approaches to inform design and implementation What are the best approaches from the CCBA project that could inform the design and implementation of similar activities in Bangladesh? Findings As stated by Winrock International (2013), the CCBA project followed a value chain approach18 to assess the agricultural subsectors. With vegetables, poultry, and dairy subsectors selected, the project took a bottom-up facilitative approach to economic growth (adopting a more advisory role 18 USAID Microlinks Value Chain Wiki http://www.microlinks.org/using-value-chain-development-wiki. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 25 than that of implementer). The project emphasized production more than marketing, with 18,500 small and marginal farmers identified as the main beneficiaries. Applying the value chain approach to subsectors, CCBA identified opportunities in the vegetable subsector in Gazipur, Narshingdi, Comilla, and Jessore districts (eggplant, bitter gourd, cauliflower, and cucumber, among other vegetables) and the poultry subsector in Gazipur and Narshingdi districts (live chicks and fresh and frozen poultry). CCBA offered an economic analysis for GH in collecting vegetables and poultry from collection points and delivering them to the customers and included a generic cold chain system (CCBA, 2014f, 67). However, the project did not assess the demand for specific vegetables and poultry as part of an integrated cold chain system. The Team did not find evidence that CCBA had talked to upscale retail owners and consumers regarding the demand for fresh, quality produce. As a result, the study (cited above) had a limited contribution to enrich the ideas proposed by Winrock International (2013) and that knowledge gap was not filled by CCBA in a subsequent report (CCBA, 2015f). The PPP was put in motion with a low level of understanding of both parties. While the project was meant to benefit farmers predominantly in the ZOI (CCBA, 2103), it became clear, as reported in the first annual report, that GH, as the private sector partner, was primarily interested in the milk and poultry value chains (CCBA, 2014b). Originally, the CCBA envisioned implementing some activities in the northeastern part of the country (Sylhet district), outside of the ZOI, while still keeping a substantial level of effort in the Jessore and Khulna districts that have a large presence of small and marginal vegetable farmers. It appears that the project was following GH’s investments and activities outside ZOI to benefit from the GH infrastructure. The CCBA project managers realized from the beginning of the project that the cold chain was not feasible because of the combination of: (1) a market with urban consumers that do not appreciate safe and nutritious fresh vegetables and, therefore, are unwilling to pay premium prices for high￾quality produce, and (2) the private sector partner shifting gears from the outset of the project from fresh vegetables to ice cream. After this realization, the two partners did not communicate to air out their differences and to propose remedial actions or a formal separation. The decision of the private sector to focus on poultry and dairy induced CCBA to expand its reach to other farmers in regions with less-marginal farmers (larger landholdings). Vegetable production and marketing became solely the responsibility of CCBA, but the project continued to support GH in cold chain development in the poultry and dairy subsectors. In 2015, CCBA acknowledged that GH was not working with vegetable farmers and “introduced a pilot concept” called Collection Centers that would emphasize production, PH practices, and marketing. This expanded approach would reach out to more than 18,000 farmers in nine districts through 31 collection points (Work Plan of Year 3). This concept was not new; it had already been used in the first two years of the project. The Team found no evidence that CCBA had considered constraints, opportunities, or intervention design issues to engage small and marginal farmers in other locations outside of the ZOI. Moreover, CCBA had taken no action to ensure that 2,000 to 2,500 farmers in Jessore and Khulna districts would receive more training to help them embrace their markets in a sustainable manner and empower them to capitalize on their previously acquired knowledge in production, post-harvesting, and marketing. The Team, based on the training manuals (Annex X), direct observations, and KIIs and FGDs in Jessore, Khulna, Barisal, and Sylhet districts, found that the depth and duration of training could be improved. Similarly, as suggested by the DQA 2105 (USAID/ACME 2015a, 2105b) more thorough record-keeping would have allowed for better follow-up with trainees. Conclusions The value chain, bottom-up, facilitative approach that CCBA used did not consider the demand for fresh, quality produce (using a cold chain system) by upscale retailers and consumers. This knowledge gap could have been avoided with more active monitoring and communication among the institutions (the Steering Committee comprised of USAID, GH, Winrock, and WFLO) involved in Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 26 the governance of the project. (See also “The Development Problem and USAID’s Response” on page 9.) The PPP between CCBA and GH did not rely on a thorough understanding of their interests and there were no provisions to address possible changes in their target sectors due to the fluctuations in the Bangladesh market environment. The CCBA project team recognized at the beginning of the project, that the private sector partner was not interested in vegetable cold chain development. The project, therefore, missed an opportunity to adapt to sustainable vegetable value chain development in the remaining LOP. CCBA management did not look back at its own achievements to further enhance the capacity of small and marginal farmers in Jessore to contribute sustainably to food security and income generation while making the vegetable value chain more efficient. CCBA opted for quantity rather than quality in human capacity-building, very possibly to meet USAID’s LOP targets. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 27 RECOMMENDATIONS Question 1. Objective and reduction of PH loss To what extent was the CCBA project successful in achieving its objective to minimize PH loss? 1. High-value vegetable producers in the initial Collection Centers established by CCBA as of the end of Year 2 of the project should continue to be assisted by the FtF Bangladesh AVC Program, particularly related to the second component of the CCBA project. 2. Direct linkage from the Collection Center-MPCs to the two major grocery chains headquartered in Dhaka should be orchestrated by the AVC. In addition, these grocery chains should continue to receive training on the third and fourth components of the CCBA project. 3. Pilot projects should be limited in scope in order to verify or prove a concept or theory: “A pilot really is just trying to look at a proof of concept. . . . What makes a pilot different is the intention to learn something, and the intention to both prove the concept and to improve the model in a formal way.” (DevEx, November 9, 2015.) 4. PPP projects, including pilot projects in agriculture, should be a minimum of five years in duration.19 5. Beneficiaries left to fend for themselves after very limited training typically become discouraged and return to their previous habits and methods. One-day training exercises are not sufficient to bring about true behavioral change and long-term sustainability. The publication, “Concepts and Practices in Agricultural Extension in Developing Countries: A Sourcebook,” makes a strong case for regular and sustainable training by competent trainers.20 6. Apply commonsense principles of business economics—developing and using crop and commodity budgets and balance sheets for decision-making, using the concepts of break￾even prices and risks in production and marketing. This was missing in CCBA training and should be present in all future projects involving agricultural value chains.21 Question 2. Most suitable and economically viable commodities Which commodities were found to be most suitable and economically viable for marketing using cold chain systems? 1. Despite the lack of progress in cold chain development in the CCBA project, there must be a continued effort by international donors to encourage cold chain implementation in Bangladesh. According to Dr. Lisa Kitinoja, the founding member and president of Postharvest Education Foundation, “Few other interventions can so dramatically maintain 19 Global Donor Platform for Rural Development (2016). 20 Ponniah et. al. (2008). 21 USAID Microlynks Value Chain Wiki, http://www.microlinks.org/using-value-chain-development-wiki. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 28 the visual quality and nutritional value, and increase shelf life and the ultimate market value of fresh foods as much as simply holding the foods at a lower temperature.”22 2. The initial focal point of the interest in further cold chain development in Bangladesh should be focused on horticultural crops, such as HVVs and mango. 23 3. The USAID/AVC Program should be tasked with developing a close, direct working relationship between the HVV producers (connected with the seven initial CCBA Collection Centers) and the two upscale grocery chains headquartered in Dhaka (Agora and Meena) that expressed interest in working directly with the CCBA Collection Centers. The relationship should be based on shared values as described in the 2013 “Strategic Analysis of Supermarket PPPs in Central America.”24 There is need for a demand baseline survey and options to improve current demand for fresh produce result of an integrated cold chain system. 4. Via the USAID/AVC project, an investment should be made to set up a real-world branded marketing trial in order to test the high-end market outlets, such as grocery chains and hotels, to determine the value (if any) of cold chain horticulture products versus traditional fresh produce to retailers and consumers. The marketing message related to this trial should emphasize that “the higher the temperature, the faster the natural degradation processes will occur, leading to loss of color, flavor, nutrients and texture changes.”25 5. Future market tests should assess whether retailers and consumers would be willing to pay extra for the guaranteed absence of “mystery” chemicals, such as formalin, and for assurances that all products have been properly graded. Question 3. GDA and leveraging private investments As this is part of the GDA, how successful was the activity in leveraging private sector investment and developing public-private partnerships? 1. USAID should continue to invest in PPPs related to cold chain development for high-value crops in Bangladesh. Although the Team could not find any evidence of the current market demand for high-value crops, the global need is well-documented26 and certainly applicable to Bangladesh. This will result in improved access to nutritious foods, increased food security, reduction in waste of foodstuffs, and increased profits for small and marginal producers. 2. Although the most successful GDA activities involve a single private sector firm,27 the Team encourages exploration of a multi-donor project designed around a cluster approach and tied to a grant program. This may result in more sustainable outcomes.28 3. Future cold chain initiatives also would be well-advised to involve the Global Cold Chain Alliance (GCCA), which represents the warehousing, construction, and transport industries engaged in temperature-controlled logistics. WFLO, a partner on the CCBA project, is also 22 Global Cold Chain Alliance (2015). 23 CCBA (2014f). 24 Davachi et. al. (2013). 25 Kotinoja (2013). 26 Rockefeller Foundation (2013). 27 USAID (n. d .b). 28 Institute for Strategy and Competitiveness, Harvard Business School (n. d.). Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 29 one of the four core members of the GCCA29 and a member of the Steering Committee of the project with expertise in moving commodities under controlled-temperatures. Question 4. Constraints and opportunities for sustainability What have been the major constraints and opportunities with respect to sustainability of the interventions? What measures should be taken to enhance sustainability? 1. Interruptions in movement of commerce (including fresh produce) due to hartals, political unrest, and violent weather conditions can be expected as a fact of life in the future in Bangladesh. Critical for the project: the ability to extend the shelf life of HVVs at Collection Centers should be part of a strategy to cope with such external factors. Addressing financial management decisions also should be part of the strategy. Several options that have been employed, as described in the “Kenya Cold Chain Assessment,” should be considered: a. Identify and implement alternative energy sources for cold storage facilities, including liquefied natural gas, which is particularly important in Bangladesh for areas far off the power grid. b. Use solar power for farm-level and collection-level cold storage. c. Improve infrastructure and management for optimal results in the cold chain. d. Insulated panels rather than concrete are the optimal choice of building material for small, cold storage facilities. e. Make available low-cost technologies, such as Coolbot and solar power, for local farmers to provide cold rooms. Coolbot technologies provide low-cost cooling for horticultural products while solar-powered coolers can provide cold storage for many commodities. 30 f. Farmers, Collection Center managers and local service providers should be empowered to make commonsense managerial economic decisions. 2. Continued donor support to assist in the development of producer organizations will be the single most effective tool available for achieving sustainability. 31 3. Certifications at global standards related to the fourth component of the CCBA SOW remained a work in progress at the end of the project and a continued to constrain producers. A concept worthy of consideration is “Smallholders’ Access to Markets for Certified Sustainable Products” (SAMCERT), a partnership among the International Fund for Agricultural Development (IFAD), the Sustainable Commodity Initiative (SCI), and the Institute for Ethical and Environmental Certification (ICEA) that was launched at the beginning of 2012 with a strong focus on PPPs.32 Question 5. Gender integration How effectively has CCBA integrated or incorporated gender in the interventions? What have been the challenges and opportunities, if any? 1. Despite cultural traditions, USAID projects should strive to incorporate women in higher￾level positions in agriculture, processing, or the provision of services. This may require 29 Global Cold Chain Alliance (2015). 30 Global Cold Chain Alliance (2015). 31 Global Forum on Food Security and Nutrition (2012). 32 International Fund for Agricultural Development (n. d.). Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 30 training in literacy and numeracy to enable women access to positions above manual labor in the fresh vegetable, poultry, and dairy value chains. 2. “At USAID, we believe that gender equality and women’s empowerment isn’t a part of development but the core of development.”33 Future FtF projects in Bangladesh should attempt to design projects that include women as the ideal conduit to induce behavioral change in nutrition, health, and intra-household decision-making. This could be achieved more easily by including women trainers (as much as possible) in all thematic or disciplinary areas. 3. In the case of pilot projects, USAID should consider the tradeoff between the pilot test and gender-equity issues. Favoring gender equity will not necessarily ensure a successful pilot test. However, a successful pilot test would provide a more conducive environment for gender equity. Question 6. Approaches to inform design and implementation What are the best approaches from the CCBA project that could inform the design and implementation of similar activities in Bangladesh? 1. The 2016 Global Donor Platform for Rural Development states: “PPPs need to be focused and designed according to both partner and value chain needs and opportunities.”34 Future GDA proposal implementers should research carefully to ensure a complete and common understanding of mutual objectives among all parties. Conducting more thorough research and consultation between the public and private entities can reduce the risk of sudden changes by any of the parties involved. All partners will benefit from knowing the “business plan” or “strategic plan” for the next five or ten years and this will help create an MOU that will last throughout project implementation. Private-public partnerships are not immune to differences in approaches and interests among the partners as a result of changes in the market environment. When there is an obvious disconnect between the partners, the donor partner should evaluate the relationship in the interest of archiving the initial goals and objectives of the project. 2. Since the launch of the USAID PPP program in 2001, there have been approximately 1,600 PPP projects.35 Almost all PPP activities have reportedly been well-structured and preceded by sufficient vetting of all partners. There also typically has been sufficient research and a complete shared analysis of the project objectives and goals. Clearly the proposal put forth by Winrock leading to the USAID/CCBA project required additional vetting and analysis. The Team would encourage the USAID Mission in Bangladesh to reconsider whether they took advantage of the worldwide USAID experience in preparation for the PPP, especially related to FtF programs. Furthermore, pilot projects deserve special consideration since they cannot be measured with traditional indicators. LESSONS LEARNED When Winrock presented USAID/Bangladesh with the Revised Technical Proposal (March 13, 2013) titled “Cold Chain Bangladesh Alliance (CCBA) Program,” this well-respected donor-development 33 USAID (n. d. a). 34 Global Donor Platform for Rural Development (2016). 35 Ingram et. al. (2016). Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 31 implementing organization brought relevant experience gained via recent USAID/GDA projects in Pakistan36 and the Philippines,37 both widely acclaimed as very successful interventions.38 However, based on the information available, these two successful projects primarily focused on high-value commodities such as perishable fruits and seafood. In contrast, the geographic area USAID targeted for FtF under CCBA intervention within Bangladesh was known for vegetable production. There is little evidence to suggest that Winrock or its private sector partner GH researched the current market demand for vegetables managed via a cold chain prior to finalizing a cooperative agreement with USAID to “establish Bangladesh’s first integrated cold chain to reduce PH losses and deliver high-value agricultural products to market.”39 Future PPP intervention proposals must be preceded by proper value chain and market analysis with the goals of all partners in mind. Regarding the CCBA project, a disconnect concerning the objectives of each partner was present from the beginning of the project and continued throughout the LOP. The USAID 2013 publication, “Strategic Analysis of Supermarket PPPs in Central America,”40 noted the following change of course by USAID: “For many years, USAID approached partnerships as a source of leverage, i.e., outside resource contributions. Leverage, though, is merely an input, and its results—if any—are not always clear. More recently, USAID has begun to focus its attention on the value of public-private collaboration of achieving development results as well as addressing private sector business interests.” Clearly, from the very inception, the CCBA project failed to heed this recent direction concerning USAID-involved PPP activities. The major lesson learned as a result of the CCBA project was that the overall marketplace for popular HVVs in Bangladesh does not perceive any current economic value in maintaining a consistent (farm-to-fork) cold chain for the crops selected for intervention through the CCBA project.41 Due to a number of factors beyond the control of the implementer,42 in reality the CCBA project only functioned for approximately 18 months. Donor-funded agricultural interventions in developing countries must be longer than five years in duration, especially pilot projects intended to find alternative pathways to market sustainability through encouraging a dramatic change in how a value chain could function. The Bill and Melinda Gates Foundation has developed well-researched criteria on the duration of interventions and has concluded that “most funding cycles are not long enough to permit such learning and transference.”43 The Organisation for Economic Co-operation and Development (OECD) also has devoted considerable attention to the length of time required for donor￾development projects to achieve sustainable results. It concluded that: “Empowerment processes are essentially about envisioning and achieving change. Therefore, donor initiatives that support them must be designed and managed to accommodate new proposals, possibilities and capabilities.”44 A significant aspect of the CCBA project was devoted to gender. The project gender activities reportedly provided training to over 8,000 women regarding family nutrition, home food processing, 36 Pakistan Agricultural Cold Chain Development Project (n.d.) http://www.cawnagency.com 37 Philippines Cold Chain Project (n.d.) www.winrockpccp.org. 38 http://www.ccaphils.org/news-list/philippine-cold-chain-project andhttp://www.gcca.org. 39 CCBA (2015a). 40 Davachi et. al. (2013). 41 CCBA (2015c). 42 CCBA (2015b). 43 Gates Foundation (2013). 44 OECD (2012). Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 32 and production of backyard vegetables.45 However, the CCBA inception document states that the “CCBA has added gender equity as an objective throughout this project—from project management to all points of the value chain.”46 For the CCBA project, gender access was limited to backyard agriculture or laborers supporting male heads of households. In the opinion of the Team, the CCBA project design regarding gender neither advanced the primary objective of the project nor met the stated objective of creating gender equity. Future FtF pilot projects with a definitive focus, such as CCBA, should gain awareness of social and cultural conditions at the design stage and incorporate gender elements in line with the actual objective of the project. The project could have delivered higher-quality results if it had not found itself compelled to deliver the milestones stipulated in the PMEP. In fact, the indicators commandeered the pilot. Pilot projects are intended to explore alternative pathways. For the CCBA, the task was to determine whether a cold chain for crops would result in reduced post-harvest loss, improved market efficiency, sustainability, and improved food security for the population of Bangladesh. The performance indicators should have focused on the complexity of the project finding the pathway (qualitative features) rather than using figures of employment or incremental sales that will cease when the project has ended. The design of pilot projects should be focused on a singular objective with indicators and include monitoring and evaluation plans structured to reflect the goal of the project. Of the four components tasked to the CCBA, the Team found positive impacts related to the first component—increased capacity of small and marginal farmers to grow high-value products. However, the obligation to meet all project indicators negatively impacted the sustainability of results in this component. In the opinion of the Team, if the CCBA had been able to focus on the FtF ZOI throughout the life of the project, the results would have been far more sustainable. The initial Collection Centers within the FtF ZOI and the related MPCs that were formed represent an excellent beginning toward producers being responsible stakeholders in the value chain for high￾value agricultural production. “The Collection Center brings us a huge bonus in terms of efficient farm management through better techniques of harvesting our produce, helps extend the shelf life of our produce, and reduces post-harvest waste and transportation costs as well.” - Toriqul Islam, HVC Farmer: Lebutala, Birnarayanpur, Jessore With training to reduce PH losses, along with a disciplined approaches to grading and packing, producers can be successful partners with the emerging supermarket chains in Bangladesh that serve higher-income clientele. The USAID 2013 publication, “Strategic Analysis of Supermarket PPPs in Central America,”47 noted the importance in “shared value” as a key to sustainability: “If a partnership proves its capacity to generate shared value, then these impacts are sustained even after USAID support is withdrawn.” The design of the CCBA project took a “broad brush” approach to PH losses and to the shelf life of agricultural commodities, in particular winter vegetables. The 2008 report, “Effect of Storage Conditions on Quality and Shelf Life of Selected Winter Vegetables,”48 along with numerous later studies, support the WFLO analysis that verifies the dramatic difference between agricultural commodities (in particular vegetables) as they relate to PH losses and shelf life. Cold chain management may be profitable for producers, retailers, and consumers for many commodities. 45 CCBA (2016a). 46 Cold Chain Bangladesh (CCB) Alliance (2013). 47 Davachi et. al. (2013). 48 Jany et. al. (2008). Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 33 However, in a country like Bangladesh with a large population within a relatively small geographic area, where agricultural production from virtually any area of the nation can be delivered to wholesale markets in urban areas in less than 24 hours, future donor-funded interventions relating to cold chain management must be designed to reflect solid research and be target-oriented. Improved project management could have benefited the project. The Team could not find evidence of a delegation of responsibilities in the analysis of data and reporting; work plans and budgets were not shared; and interaction among the team players was not encouraged through seminars, presentations, or brainstorming sessions to discuss progress in any of the project components. Interaction with other USAID-funded projects was kept to a minimum, with no commitment to share information or have joint activities, even though there were topics in common within the ZOI. The deteriorating relationship with GH was not explicitly shared with USAID to take remedial actions or to agree to refocus the project. For future pilot projects, closer communication is vital to success even more so than in a conventional development project since the donor has to budget for the necessary time. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 34 ANNEXES TABLE OF CONTENTS Annex I: Evaluation Statement of Work.................................................................................... 35 Annex II: Evaluation Design Matrix............................................................................................. 46 Annex III: Planned Field Work Schedule.................................................................................... 49 Annex IV: Data Collection Instruments..................................................................................... 50 Annex V: Evaluation Coverage..................................................................................................... 52 Annex VI. Quantitative Assessment ........................................................................................... 55 Annex VII. PITT............................................................................................................................... 58 Annex VIII: Sources of information............................................................................................. 59 Annex IX: People Contacted........................................................................................................ 64 Annex X: Assessment of Training Materials............................................................................. 73 Annex XI. Gantt Chart for the CCBA Final Performance Evaluation................................. 76 Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 35 Annex I: Evaluation Statement of Work Scope of Work (SOW) for the Cold Chain Bangladesh Alliance (CCBA) Project Final Performance Evaluation Point of Contact (POC) USAID/Bangladesh Economic Growth Office Mohammad Sayed Shibly Agreement Officer’s Representative, CCBA Tel:(+880) 5566-2339 Email: mshibly@usaid.gov Farzana Yasmeen Contracting Officer’s Representative, ACME Tel:(+880) 5566-2381 Email: fyasmeen@usaid.gov PROJECT TO BE EVALUATED Project Name Cold Chain Bangladesh Alliance (CCBA) Contract Number Contract Number: AID-388-A-13-00004 Original Project Date July 1, 2013 - June 30, 2016 Original Funding $4,975,233 Implementing Partner Winrock International I. BACKGROUND USAID/Bangladesh awarded Contract Number: AID-388-A-13-00004 to Winrock International to implement the Cold Chain Bangladesh Alliance (CCBA) project on June 30, 2013. CCBA is a three￾year project and aims to establish Bangladesh's first integrated cold chain in order to reduce post￾harvest losses, deliver high value agricultural products to the market, and improve the availability of locally produced nutritious foods to improve livelihoods of farmers.This project is part of the Global Development Alliance (GDA), USAID’s model for public-private partnerships. USAID/Bangladesh and Winrock International are partnered with Golden Harvest, a local food processing and transportation company, to implement this project. The majority of high-value agricultural products grown in Bangladesh, including vegetables, meat, and fish, are sold directly to consumers without proper handling, packaging or refrigeration. This results in loss rates that exceed 40%. Products that do reach market often do not meet consumer demands for quality. At the farm level, producers lack improved inputs and knowledge of market requirements and production and post-harvest practices essential to product quality. Farmers typically have small land holdings and are unable to achieve product volume required to enter commercial markets. Women in particular, due to their traditional status in the household, are excluded from income generating opportunities and may be food insecure even in households with sufficient access to and availability of nutritious foods. Poor integration and coordination among value chain business interests—as well as corruption and dishonesty—inhibit the transmission of market information and lead to missed opportunities for value addition. As a result, the ability of Bangladesh to produce sufficient high-quality and nutritious food for its growing population is severely compromised. USAID/Bangladesh has a specific commitment to support the Global Hunger and Food Security Presidential Initiative goal to “sustainably reduce global poverty and hunger.” Its 2011-2016 Country Development Cooperation Strategy (CDCS) is oriented around four development objectives (DOs). DO2 for FY 2011–2016: “Food Security and Poverty Status Improved” is aligned with the GOB’s 6th Five-Year Plan and the FTF-supported Country Investment Plan (CIP) for agriculture, food security, and nutrition through three inter-related intermediate results (IR): 1) Increased agricultural productivity; 2) increased income generation access to affordable food; and 3) improved nutrition and dietary diversity. CCBA Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 36 contributes to IR 1 and IR 2 of the DO2 results framework, which are shown in Figure1 below. Figure1: Illustrative CCBA Results Framework The CCBA development hypothesis asserts that if integrated improvements take place in: (1) increased income among small and marginal farmers from improved sustainable, high-value agriculture; (2) improved agricultural market efficiency and planning; (3) increased private sector investment and capacity in cold chain management; and (4) increased compliance with international food safety standards, then sustainable increases in the availability, access, and use of domestically produced and nutritious foods will lead to reduced poverty and hunger in Bangladesh. USAID/Bangladesh has identified the following three intermediate results (IR) and six outcomes, which comprise the CCBA project. The Results Framework depicted above is a graphical depiction of the Theory of Change. Please see Annex 1 for CCBA M&E plan for more details. The CCBA is also aligned with GoB's 6th Five-Year Plan and FTF-supported Country Investment Plan (CIP) for agriculture, food security, and nutrition. It will: • Achieve impact by raising the incomes of 14,400 small and marginal farmers through gender-sensitive training on agricultural production and access to markets. • Ensure reach by establishing scalable cold chain infrastructure, reducing post-harvest losses and opening marketing channels for thousands of additional farmers. • Ensure efficiency through rapid launch of proven approaches that leverage and complement existing USAID/Bangladesh food security programs. • Achieve effectiveness by building the capacity of the private sector to take ownership for cold chain investments and activities that contribute to food security. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 37 II. PROJECT COMPONENTS The CCBA has four primary and integrated components with specific objectives. Component 1: Increased capacity of small and marginal farmers to grow high-value products The constraints to achieving greater volume, quality, and consistency in agricultural value chains in Bangladesh are inappropriate or low quality inputs; lack of knowledge about production and post-harvest practices; and lack of access to market information and collection centers (the latter is discussed under Component 2). CCBA will address the constraints through proven market-based approaches that match demand-driven training with private sector provision of inputs and services. Men and women will increase their income after adapting improved technologies and management practices. Building on training methods established under Winrock programs including REAP All, CCBA will organize on-farm training for farmers to enable them to unlock theft competitive advantages and create shorter, more efficient, and profitable routes to high-end markets. Component 2: Increased agricultural market efficiency and planning Low-volume production and farmer disaggregation result in low incomes for producers and high transaction costs for the private sector. CCBA will improve market reach and efficiency by expanding the successful model focused on production areas developed under Winrock food security programs. Farmers in a production area will supply collection centers or rural hubs directly. Rural hubs will be linked to divisional hubs operated by Golden Harvest, the GDA partner, or secondary markets at the division level and ultimately to a central refrigerated depot or wholesale market. The "hub-and-spoke" model will reduce market volatility through coordinated production planning, improved quality control methods, better tracking and access to storage facilities, aggregation of increasingly larger volumes for scale transactions, and initial value-addition through sorting, grading, and proper packing and transportation to ensure product integrity. Component 3: Increased private sector investment and capacity in cold chain management To ensure sustainable investments in agricultural production and marketing, CCBA will work with Golden Harvest and other agro-processors to 1) support new investments in cold chain infrastructure including storage and transportation and 2) increase the capacity of the private sector to maintain and trace product quality from farm gate to table. Component 3 will focus on reducing post-harvest losses and getting more products to market by promoting greater integration of businesses operating along the value chain. Strengthened business relationships will enable new efficiencies in the scale and volume of products reaching market. Showcasing Golden Harvest as an "industry standard," CCB will offer training and exposure visits to prepare the private sector for investment opportunities that harness production potential. Component 4: Increased compliance with international food safety standards World Food Logistics Organization (WFLO) will lead the design and implementation of Component 4 to increase compliance with international food safety standards through firm￾level Hazard Analysis Critical Control Point (HACCP) and Good Aqua/Agricultural Practices (GAP) training for interested private sector actors and capacity building of local research and training institutions to provide training and certification courses. International food safety standards now demand "zero tolerance' on pathogens, contaminants, adulterations, banned Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 38 substances, and antibiotics. Adopting this approach, CCBA will advocate and assist local firms in adhering to established food safety standards in domestic food production. III. EVALUATION OBJECTIVES The purpose of the CCBA final performance evaluation is to assess the extent to which the CCBA project has achieved its overall goal. The findings and recommendations of the evaluation will be used in the implementation of other relevant EG activities. With the exclusion of procurement sensitive sections, USAID intends to disseminate the report widely to stakeholders such as USAID/Bangladesh, USAID/Washington, USAID implementing partners, GOB ministry partners (see CREL Implementing Partners box above), other sector￾specific donors, and the Development Experience Clearinghouse (DEC) The evaluation will – 1. Review, analyze and evaluate the effectiveness of CCBA project in achieving the program objectives and contributing to USAID/Bangladesh’s efforts to increase agricultural productivity and increase private sector investment in agricultural activities and post￾harvest management 2. Provide specific recommendations and lessons learned on strategies and approaches USAID/Bangladesh should continue in its other relevant FTF activities and any future cold chain development programs. IV. EVALUATION QUESTIONS: 1. To what extent was the CCBA project successful in achieving its objective and to minimize post-harvest loss? 2. Which commodities were found to be most suitable and economically viable for marketing using cold chain systems? 3. As this is part of Global Development Alliances (GDAs), how successful the activity was in leveraging private sector investment and developing public-private partnership? 4. What have been the major constraints and opportunities with respect to sustainability of the interventions? What measures should be taken to enhance sustainability? 5. How effectively CCBA has integrated or incorporated gender in the interventions? What have been the challenges and opportunities, if any? 6. What are the best approaches from CCBA project that could inform the design and implementation of similar activities in Bangladesh? V. EVALUATION METHODOLOGY The M&E contractor for the USAID Bangladesh Economic Growth (EG) office, Accelerating Capacity for Monitoring and Evaluation (ACME), will carry out this evaluation. Based on the SOW from USAID, ACME will recruit and manage the evaluation team. The evaluation work plan and evaluation protocol drafted by the evaluation team will be reviewed and approved by USAID. The detailed methodology of the evaluation will be designed by the evaluation team in the work plan; this will include presentation of a data collection matrix that will explicitly link evaluation questions to particular data collection approaches and data sources. It is suggested that the evaluation team use a mixed- method approach utilizing both qualitative (including non-experimental design) and quantitative analysis (key informant interviews, stakeholder interviews/mini-surveys, and focus group discussions). The evaluation team should develop the best evaluation design methodology in light of the evaluation questions, timeframe, budget, data collection requirements, quality of existing data sources, and potential biases. The evaluators should utilize several different, yet complementary and inter-related forms of gathering information / data. Suggestions are provided below: Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 39 Document Review: Evaluation team members will review documents throughout the evaluation process including program reports, relevant studies and evaluations and Bangladesh Energy Regulatory Commission (BERC) documents to ensure that comprehensive and grounded best practices will be identified. Key Informants Interview: The team will conduct one-on-one interviews with a variety of stakeholders including and various stakeholders, and other projects supporting cold chain sector development. Self-assessment: The IPs will respond to a self-assessment either through a questionnaire or standard interview checklist put together by the evaluation team and approved by USAID before use. Expert Opinion Survey: Utilizing expert opinion is a technique used increasingly in the market system and cold chain sector. The Evaluation team, with approval of USAID, can apply this method as well. Participatory Rural Appraisal (PRA): This form of survey method will be applied to seek views and opinion of the local community on the issues related to cold chain development. This qualitative survey requires sufficient triangulation.49 Focus Group Discussions (FGD): FGD (small group of 6 to 10 people) will be used to lead open discussion through a skilled moderator to gather semi-structured qualitative data. The preselected participants will discuss issues and concerns based on a list of key theme drawn up by the moderator. No more than 10 questions will be addressed by a group. These sessions will encourage free flowing discussion about the activity. Mini Survey: This type of survey is small (30-40 participants) and can be performed rapidly in the field without analytical software such as SPSS or a large questionnaire. The sample size is not statistically significant but this type of analysis can be used to triangulate with other methods. Because of the small sample size this type of survey can be implemented quickly when time and resources are constrained. Social and Economic Analysis: This type of analysis might require sample survey (e.g. cold chain network sustainability). Due to time and resource constraints, it is neither possible nor desirable to carry out a complete census for social and economic analysis. So, a sample size needs to be determined, based on the project area of focus, for the survey, which is statistically sound in terms of representativeness of the sample, and which is most widely used for inferential analysis. Regardless of data collection and analysis methods, USAID requires qualitative and quantitative data disaggregated by gender.50 Methodological limitations and challenges for this evaluation are expected to include: • Ensuring that samples of interview sources are sufficient to support evaluation findings; 49 Davis (Davis, A.C.S , ‘Participatory Rural Appraisal’, Rural Travel and Transport Program, 2001, 5.6.a, TRL Limited) reveals that PRA triangulation is often carried out in groups of at least 3 (three) to increase the credibility of each survey technique. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 40 • Taking systematic actions to counter any biases in (a) reporting by data collection sources and (b) interpretations of collected data by the evaluation team; and ensuring “actual” results can be measured, which will only be possible if data can be gathered and analyzed beyond respondent perceptions. All the methodological strengths and weaknesses should be explicitly described in the evaluation report. VI. EXISTING SOURCES OF INFORMATION The evaluation team should consult a broad range of background documents apart from project documents provided by USAID/Bangladesh. USAID and CCBA project will provide the assessment team with a package of briefing materials which will include the documents indicated below: • Project Description Document (Attachment B - Program Description; AID-388-A-13-0000, p.17-69) • CCBA M&E Plan • Project Quarterly and Annual reports, work plans, and management review • DQA reports • USAID/Bangladesh DO2 Performance Management Plan • USAID/Bangladesh Country Development Cooperation Strategy 2011 – 2016 (Public version) The evaluation team must also research any other relevant documents from GOB, Donors and other international organizations pertinent to the sector that CCBA serves. VII. EVALUATION TEAM COMPOSITION The evaluation team should consist of three consultants: two international and one national. One international consultant will work as Team Leader who will be assisted by the other two experts. The following skills and qualifications are expected from the evaluation team: 1. Team Leader/ USAID Evaluation Expert Duties and Responsibilities The Evaluation Team Leader/Evaluation Expert will provide overall leadership for the team, and s/he will finalize the evaluation design, coordinate activities, arrange periodic meetings, consolidate individual input from the other team member, and coordinate the process of assembling the final findings and recommendations into a high quality document. S/he will design the methodology for the evaluation, and also lead the preparation and presentation of the key evaluation findings and recommendations to the USAID/Bangladesh team and other major partners. Skills and Qualifications: • Minimum Master’s degree or higher degree in a relevant discipline such as Economics, Agriculture, Business Administration, Management and Marketing • Experience in designing, implementing and evaluating USAID projects aiming at capacity development especially in private sector, cold chain or value added agriculture enterprises or agriculture credit sector • Significant experience in program evaluation including knowledge of USAID Evaluation Policy • At least 15 years of experience in implementing and evaluating relevant programmes and/or projects for USAID or other international development agencies; • Knowledge of USAID regulations and systems, performance monitoring guidance, evaluation policy, gender policy, USAID Global Development Alliance (GDA), annual reporting etc. is essential • Experience working with government, public administration improvements and infrastructure financing Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 41 • Relevant experience in Bangladesh or South Asia preferred • Strong analytical skills; • Excellent communications and writing skills in English; • Excellent coordination skills; • Knowledge of Bangla is an asset 2. International Cold Chain and Value Chain Expert The International Agricultural Value Chain/Cold Chain Expert will provide technical guidance to the team leader and the team. Utilizing their in depth knowledge of agricultural value chains and the cold chain business, they will work closely with the Team Leader to design the methodology for the evaluation and the approaches that are best needed to answer the evaluation questions. Skills and Qualifications: • Minimum Master’s degree or higher degree in a relevant discipline such as Food Technology, Food Science, Agricultural Economics or Business Administration • Experience in designing, implementing and evaluating projects aiming at capacity development especially in private sector, cold chain or value added agriculture enterprises or agriculture credit sector • Significant practical experience in the field, in managing/implementing development projects • Experience working in a private sector development program, preferably with a focus on market development, public-private partnership, business development services, or value chain improvement; • At least 15 years of experience in implementing and evaluating relevant programmes and/or projects for USAID or other international development agencies; • Knowledge of USAID regulations and systems, performance monitoring guidance, evaluation policy, gender policy, USAID Global Development Alliance (GDA), annual reporting etc. is essential • Experience working with government, public administration improvements and infrastructure financing • Knowledge and experience of agriculture production, post-harvest management and marketing processes from grassroots to highest levels is a plus; • Relevant experience in Bangladesh or South Asia preferred • Strong analytical skills; • Excellent communications and writing skills in English; • Excellent coordination skills; • Knowledge of Bangla is an asset 3. National Agricultural Value Chain/Cold Chain Expert Duties and Responsibilities: The National Cold Chain Expert will be responsible for collection of background materials upon request by the evaluation team leader. S/he will actively participate in the desk review of materials and assist the team leader in developing methodologies, work plans and report outlines. The National Expert will assist the Team Leader in setting and conducting interviews with relevant stakeholders and actively take part in these. S/he will participate in team meetings, site visits, and draft the sections of the report relevant to his/her expertise. S/he will also participate in presenting the report to USAID or other stakeholders and be responsible for addressing pertinent comments provided by USAID/Bangladesh or other stakeholders. Skills and Qualifications: • Master’s degree in Agricultural Science, Economics, Business Administration, Management and Marketing, or other related fields; • 10+ years of experience working in the areas of agronomy and production, post-harvest technologies and management, commercialization of new and improved technologies, and/or private sector development Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 42 • Knowledge of stakeholder engagement; developing partnerships and managing relationships with donors, NGOs, private sector and government • Experience with market systems approach and value chain projects, preferably of USAID, is a strong asset; Experience in public-private partnership management is desirable • Strong analytical skills; • Strong oral communications and writing skills in English; • Strong knowledge and understanding of monitoring and evaluation Conflict of Interest All evaluation team members will provide a signed statement attesting to a lack of conflict of interest, or describing an existing conflict of interest relative to the project being evaluated. USAID will provide the conflict of interest forms. VIII.DELIVERABLES All deliverables are internal to USAID and the Evaluation Team unless otherwise instructed by USAID. Evaluation deliverables are indicated below: Evaluation Team Planning Meeting: During the meeting, the team should review and discuss the SOW in its entirety, clarify team member roles and responsibilities, prepare the work plan, develop data collection methods, review and clarify any logistical and administrative procedures for the assignment and instruments, and prepare for the in-brief with USAID/Bangladesh. Work Plan: Prior to initiation of the evaluation activities, the evaluation team will provide a detailed initial work plan to the ACME COR. The ACME COR will provide any necessary feedback or edits to the work plan, after which the evaluation team will have three days to submit a final version of the document. The initial work plan will include (a) a task timeline, (b) a description of the methodology to answer each evaluation question, (c) team responsibilities, (d) document review process, (e) key informant and stakeholder meetings, (f) site visits, and (g) draft and final report writing. The work plan will be submitted to the ACME COR at USAID/Bangladesh for approval no later than the 5th day after commencement of evaluation. In-briefing Meeting: The evaluation team will meet with USAID/Bangladesh within two working days of the International Team Leader’s arrival in country. Evaluation Design Matrix: A table that lists each evaluation question and the corresponding information sought, information sources, data analysis methods, and limitations. The matrix should be finalized and shared with USAID/Bangladesh before evaluation field work starts. It should also be included as an annex in the evaluation report. Data Collection Instruments: Development and submission of data collection instruments to USAID/Bangladesh during the design phase and after the evaluation is completed. Regular Updates: The Evaluation Team Leader will brief the ACME COR and any other designated evaluation POC on progress with the evaluation on at least a weekly basis, in person or by electronic communication. Any delays or communications must be quickly communicated to USAID/Bangladesh to allow quick resolution and to minimize any disruptions to the evaluation. Emerging opportunities to strengthen the evaluation should also be discussed with USAID/Bangladesh as they arise. Preliminary Draft Evaluation Report: The evaluation team will submit a Preliminary Draft Evaluation Report to the ACME COR five working days before the Mission debriefing. Within three working days after receipt, USAID staff will provide preliminary comments prior to the Mission debriefing. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 43 Debriefing with USAID: The evaluation team will present the major evaluation findings to USAID/Bangladesh through a PowerPoint presentation before the team’s departure from country. The debriefing will include a discussion of achievements and issues as well as any preliminary recommendations. The team will consider USAID comments and incorporate them in the Draft Evaluation Report. Draft Evaluation Report: A draft report on the findings and recommendations should be submitted to USAID/Bangladesh within 10 business days after departure of international team leader. The written report should clearly describe findings, conclusions, and recommendations. The draft report must be of high quality with no grammatical errors or typos. A report is high quality when it represents a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why. The draft report must have well-constructed sentences that are presented in a way that clearly presents findings, conclusions and recommendations. The report should answer all the evaluation questions and the structure of the report should make it clear how the questions were answered. The draft report must meet the criteria set forth under the Final Report section below. USAID will provide comments on the draft report within 10 working days of submission. Final Evaluation Report: The evaluation team will submit a Final Evaluation Report that incorporates Mission comments and suggestions no later than 10 working days after USAID/Bangladesh provides written comments on the Draft Evaluation report. The format of the final report is provided below. The report will be submitted electronically in English. The total pages of the final report, excluding references and annexes, should be no more than 30 pages. A second version of the evaluation report, excluding any potentially procurement-sensitive information, will be submitted and disseminated among implementing partners and other stakeholders within 10 days following approval from USAID. All quantitative data, if gathered, should be (1) provided in an electronic file in easily readable format; (2) organized and fully documented for use by those not fully familiar with the project or the evaluation; (3) owned by USAID and made available to the public barring rare exceptions. A thumb drive with all the data could be provided to the ACME COR. The final report will be edited and formatted by the evaluation team and provided to USAID/Bangladesh 5 working days after the Mission has reviewed the content and approved thefinal revised version of the report. IX. REPORTING REQUIREMENTS: The total pages, excluding references and annexes, should not be more than 30 pages. The following content (and suggested length) should be included in the report: Table of Contents List of Acronyms Executive Summary – concisely state the project purpose and background, key evaluation questions, methods, most salient findings and recommendations (2-3 pp.); 1. Introduction – country context, including a summary of any relevant history, demography, socio-economic status, etc. (1 pp.); 2. The Development Problem and USAID’s Response – brief overview of the development problem and USAID’s strategic response, including design and implementation of the CCBA activity and any previous USAID activities implemented in response to the problem, (2-3 pp.); 3. Purpose of the Evaluation – purpose, audience, and synopsis of task (1 pp.); Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 44 4. Evaluation Methodology – describe evaluation methods, including strengths, constraints, and gaps (1 pp.); 5. Findings and Conclusions – describe and analyse findings for each evaluation question using graphs, figures, and tables, as applicable, and also include data quality and reporting system that should present verification of spot checks, issues, and outcomes. Conclusions should be credible and should be supported by the findings (12-15 pp.); 6. Recommendations – prioritized for each evaluation question; should be separate from conclusions and be supported by clearly defined set of findings and conclusions. Include recommendations for future project implementation or relevant program designs and synergies with other USAID projects and other donor interventions as appropriate (3-4 pp). 7. Lessons Learned – provide a brief of key technical and/or administrative lessons on what has worked, not worked, and why for future project or relevant program designs (2-3 pp.); 8. Annexes – to include statement of work, documents reviewed, bibliographical documentation, evaluation methods, data generated from the evaluation, tools used, interview lists, meetings, FGDs, surveys, and tables. The Evaluation Design Matrix must be presented as an annex to the report. Annexes should be succinct, pertinent, and readable. Should also include if necessary, a statement of differences regarding significant unresolved difference of opinion by funders, implementers, or members of the Evaluation Team on any of the findings or recommendations. The report format should be restricted to Microsoft products and 12-point type font should be used throughout the body of the report, with page margins one-inch top/bottom and left/right. X. TIMELINE AND LEVEL OF EFFORTS: Work will be carried out over a period of 13 weeks, during April-June, 2016. The evaluation team should plan for holidays, such as Eid-al-Adha, and the possibility of hartals, which can disrupt travel within Bangladesh. Below is an estimate of the evaluation level of effort (LOE): A. Preparatory Work Evaluation Team Leader Int’l Subj. Matter Expert National Experts (2) Comprehensive document collection and review. 5 days 5 days 5 days Travel to Bangladesh 2 days 2 days 0 days Team planning meeting and meeting with USAID/Bangladesh. 1 day 1 day 1 day Development of evaluation work plan (concurrent with document review and initial meetings). 2 days 2 days 2 days Develop preliminary interview instruments and begin scheduling key interviews. 3 days 3 days 3 days B. Data Gathering In-country information and data collection. Includes interviews with key informants (stakeholders and USAID staff) and site visits. 18 days 18 days 18 days C. Data Analysis/Drafting Report Data analysis in preparation for presentations 3 days 3 days 3 days In-country discussion with USAID and presentation of preliminary analysis and draft of final report. 1 day 1 day 1 day Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 45 Presentation of preliminary results and recommendations to the USAID/Bangladesh Mission and relevant stakeholders 1 day 1 day 1 day Depart Bangladesh 1 1 0 Analysis of data and draft of final evaluation report. The evaluation team will submit the draft within 10 working days to the IBTCI HQ after the departure of international team. HQ will review for quality control and final revisions and submit to USAID within 3 working days. 10 days 10 days 7 days Evaluation team has ten days to update and finalize final evaluation report and five working days for HQ Quality Control review and final revisions. 5 days 3 days 3 days TOTAL 52 50 44 XI. SCHEDULING AND LOGISTICS: Funding and Logistical Support USAID/Bangladesh’s ACME project will be responsible for all off-shore and in-country administrative and logistical support, including identification and fielding appropriate local staff. They will take care of arranging and scheduling meetings, international and local travel, hotel bookings, working/office spaces, computers, printing, and photocopying.51 A local administrative assistant/coordinator may be hired to arrange field visits, local travel, hotel, and appointments with stakeholders and provide translation services. Scheduling Work is to be carried out over a period of approximately 13 weeks, beginning in April 2016 with field work completed in the same month and final report and close out concluding o/a / June 2016. A six-day work week (Saturday-Thursday) is authorized for the evaluation team while in Bangladesh. The evaluation team will submit a work plan as part of the evaluation methodology proposal with timeline and develop a GANTT chart displaying the time periods during which activities occur. Pre-departure arrangements should include: travel approval; airline tickets; visa; lodging; work facility and vehicle transport arrangements; dates for meetings with USAID/Bangladesh EG staff and key contacts; in-country travel agenda; and accommodations. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 46 Annex II: Evaluation Design Matrix Evaluation Question and PMEP related Indicator Information sought Information sources Data analysis methods Limitations 1. To what extent was the CCBA project successful in achieving its objective and to minimize post-harvest loss? Measures of reduction in post-harvest losses Assessments or reviews on post-harvest loss reduction How relevant stakeholders measure reduction in post￾harvest losses? PIRS description related to reduction of post-harvest loss. Quantitative assessment of indicator(s) Inception document for CCBA Group of stakeholder (1, 2, 3 and 4) DQAs (2015 and 2016) Literature review Direct observations, open ended questions for stakeholders, interviews and focus groups Three years is a short time to determine successful reduction in post-harvest losses but data may show a trend over 11 quarters Changing preferences (opportunities) for different commodities, due to changing market conditions, can affect systematic measurements of the same commodities. 2. Which commodities were found to be most suitable and economically viable for marketing using cold chain systems? How did you determine which commodities were the best options for marketing using cold chain system? Was there a baseline work to guide you? If so, could you describe it or provide the reference document? CDS Bangladesh 2012-2016 USAID Annual and quarterly reports DQAs (2015 and 2016) Group of stakeholder (1, 2 and 3) Sector analysis performed by CCBA during the first six months of project implementation Agric. Value Chain project (USAID Bangladesh) reports Literature review of the state of the art on HVC agriculture Direct observations, open ended questions for stakeholders, interviews and focus groups Incomplete information and criteria for commodity selection Lack of strategic thinking or policy framework Interest and needs of small and marginal farmers not fully understood 3. As this is part of Global How was leveraging of Annual and quarterly Direct observations, Contributions from the private Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 47 Evaluation Question and PMEP related Indicator Information sought Information sources Data analysis methods Limitations Development Alliances (GDAs), how successful the activity was in leveraging private sector investment and developing public￾private partnership? investment measured? Under what conditions it was considered that USAID intervention leveraged private investments? reports Documentation that verifies that leveraging took place DQAs (2015 and 2016) Inception document for BBCA Group of stakeholder (1) Memorandum(s) of understanding between CCBA and Golden Harvest open ended questions for stakeholders, interviews and focus groups Detailed analysis of timings in investment decisions will be used to determine the attribution of leveraging investments sector could have been the result of their own planning that coincided with USAID´s intention to invest in cold chains (i.e., the investment of the private sector was the counterfactual of USAID´s intervention) 4. What have been the major constraints and opportunities with respect to sustainability of the interventions? What measures should be taken to enhance sustainability? Define the interventions, specify actors involved and beneficiaries. How are the production and marketing processes likely to continue after USAID project comes to an end? Do stakeholders have a plan of action post-CCBA that keeps the (cold) chain alive? Annual and quarterly reports Annual and quarterly reports DQAs (2015 and 2016) Group of stakeholder (1, 2, 3 and 4) Memorandum(s) of understanding between CCBA and Golden Harvest Direct observations, open ended questions for stakeholders, interviews and focus groups Farmers continue to produce and sell in organized groups The linkage of organized farmers with wholesalers or retailers remain There may not be a plan of action or contingency post CCBA. 5. How effectively CCBA How has the project helped Annual and quarterly Direct observations, Cultural barriers and lack of Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 48 Evaluation Question and PMEP related Indicator Information sought Information sources Data analysis methods Limitations has integrated or incorporated gender in the interventions? What have been the challenges and opportunities, if any? women to be involved in production, selection, packing, shipping, agribusiness management and investment decision￾making? How has the project helped women in backyard farming? Has the project contributed to more availability of nutritious food? reports Indicators disaggregated by gender Group of stakeholder (1, 2, 3 and 4) open ended questions for stakeholders, interviews and focus groups resources (land and capital) limit women to prosper in high value agriculture; their access is limited to backyard agriculture or as laborers supporting male household heads. The role of women is seldom acknowledged and it is even less acknowledged if they play a role as managers. 6. What are the best approaches from CCBA project that could inform the design and implementation of similar activities in Bangladesh? Definitions of approaches to cold chain development (direct vs indirect intervention, top-down vs bottom-up, inclusive vs non-inclusive, production vs market oriented, mechanisms to break barriers to access markets) Performance of the four project components separately and jointly will enable distillation of lessons learned that will be used for future project design and implementation. Inception document for CCBA Annual and quarterly reports DQA 2015 and 2016 Group of stakeholder (1 and 2) Direct observations, open ended questions for stakeholders, interviews and focus groups The selected high value commodities have a high expected demand and profitability with high potential for cold chain development in Bangladesh. This requires suitable infrastructure, produce quantity and quality, and access to markets, as well as the commitment from all actors to support the (cold) chain. As a pilot project, CCBA, is more a learning experience than a project that is disseminating its benefits. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 49 Annex III: Planned Field Work Schedule Stakeholders Districts Dhaka Comilla Sylhet Jessore Khulna Barisal Implementing partner X X X X X Government X X Priv. Sector Partner X X Retailers X X X X Wholesalers X X NGOs X X X X Collection Point Managers X X X Transporters X X X Local Service Provider X X X X X Farmers (backyard) X X X X Farmers (HVP) X X X X X Dates May16-17 May 18 May 19 May 24 May 25 May 26 Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 50 Annex IV: Data Collection Instruments Data Collection tools – Questionnaire used for KII and FGD Sl Question 01 To what extent was the CCBA Project successful in achieving its objective and to minimize harvest losses? • How did you come to know about this project? • Have you benefitted financially as a result of the CCBA project? • Are you aware of any increased market efficiency as a result of the project? • Have you noted any increase in private sector investment especially related to the cold chain development? • Are you aware of any changes in food safety standards as a result of the project? • One of the objectives of the CCBA project was to reduce the percentage of harvest losses; how successful has been the program achieving this objective? 02 Which commodities were found to be most suitable and economically viable for marketing using cold chain systems? • What commodities are the most appropriate for cold chain systems? • Are the commodities most appropriate for a cold chain system also the most economically viable • What commodities would benefit more from a marketing program that has been supported by a cold chain system? (very perishable; proper selection, packing, adequate and speedy transportation) 03 As this is part of the Global Development Alliance (GDA), how successful the activity was in leveraging private sector investment and development public-private partnerships? • Under what conditions it was considered that USAID intervention leveraged private investments? 04 What have been the major constraints and opportunities with respect to sustainability of the interventions? What measures should be taken to enhance sustainability? • Has the CCBA project created opportunities for you? • Have there been challenges which have created difficulties for you to benefit from the CCBA Project? (hartels, transportation, logistics, market opportunities) • The CCBA Project will end up shortly, what sustainability do you envision as a result of the interventions? 05 How effectively CCBA has integrated or incorporated gender in the interventions? What have been the challenges and opportunities, if Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 51 any? • Has women´s participation increased due to the CCBA project compared to the time prior to the its existence? • Based on your knowledge of the project have women enjoyed opportunities that were not available to them prior to the beginning of the project? • In what areas do you think women can be more involved? o Challenges o Production o Post-harvest (sorting, packing, logistics, etc.) o Marketing o Cultural barriers removed • [only for women farmers] How has the diet of your household members changed due to the project activities? Is this due to: a) higher disposable income, b) more availability of backyard vegetables, c) more availability of vegetables from the household plot. 06 What are the best approaches from CCBA project that could inform the design and implementation of similar activities in Bangladesh? • If you were to be involved in the design of a new venture between private and public sectors in agriculture, what should be included and how would you go about it? • The overarching objective of this project was to develop a cold chain, was the country ready for this endeavor? Were there any components absent in this experience? Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 52 Annex V: Evaluation Coverage Summary of stakeholders meetings (geographical coverage vs stakeholder groups) District No of interviews conducted (field notes) No of interviews with different groups Female respondents Male respondents Total respondents Group 1 (IP, PSP & Government /Other) Group 2 (NGOs) Group 3 (LSP, Retailer & Wholesaler, HVC & Backyard) Group 4 (Collection Point Managers & Transporter) Jessore 05 01 - 03 01 03 16 19 Khulna 04 01 01 02 - 12 03 15 Barisal 04 01 - 03 - 15 12 27 Comilla 03 - - 03 - 12 07 19 Sylhet 05 01 - 04 - 08 09 17 Gazipur 01 - - 01 - - 04 04 Dhaka 13 10 01 02 - 02 21 23 Total 35 14 02 18 01 52 72 124 Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 53 Summary of stakeholder meetings (Geographical Coverage VS Different Stakeholders) Distric t No of interviews Cond. (field notes) No. of interviews conducted with different groups / stakeholders & (corresponding number of participants) Female Respondents Male Respondents Total Respondents Group 1 Group 2 Group 3 Group 4 Imple m￾enting Partne r Privat e Sector Partne r GoB / Other s NGOs Retail er Whole saler Local Service Provide r Farmer s – Back Yard Farmer s -HVC Transp orter Collecti on Point Manag er Jessore 05 - - 01 (02) - - 01 (02) - 01 (03) 01 (10) 01 03 16 19 (01) (01) Khulna 04 - - 01 (01) 01 (01) - 01 (01) - 01 (12) - - - 12 03 15 Barisal 04 01 - - - 01 (01) 01 (15) 01 (09) - - 15 12 27 (01) - (01) Comill a 03 - - - - - - 01 (01) 01 (12) 01 (06) - - 12 07 19 Sylhet 05 - - 01 (01) - - - 02 (02) 01 (08) 01 (06) - - 08 09 17 Gazipu r 01 - - - - 01 - - - 04 04 (01) (01) - - (02) Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 54 Dhaka 13 02 (03) 04 (06) 04 (08) 01 (03) 01 (01) - 01 (02) - - - - 02 21 23 Total 35 14 02 (04) 18 01 (04) (06) (13) (02) (04) (06) (50) (33) (01) (01) 52 72 124 (23) (95) (02) Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 55 Annex VI. Quantitative Assessment Quantitative Assessment of Performance Indicators Below is a narrative of nine standard and two custom indicators used to assess the performance of the CCBA Project during 11 quarters, from July 2013 to March 2016. Year 1 includes five quarters, from July 2013 to September 2014; Year 2 includes four quarters, from October 2014 to September 2015, and Year 3, includes two quarters, from October 2015 to March 2016. The last quarter of the project, April to June 2016, is still ongoing and it is not included in this assessment. Standard indicators (USAID No.) 1. (4.5.2-7) Number of individuals who have received USG supported short-term agricultural sector productivity or food security training- During eleven quarters 31,144 individuals have received support towards productivity or food security training, exceeding the Life of Project (LOP) target by 25%. Relative to the target LOP, 26% was achieved in Year 1, 62% in Year 2, and 37 % in the first two quarters in Year 3 (October 2015- March 2016)52. Women comprised 26% of the total of individuals receiving USG support or training (8,070 out of 31,144). 2. (4.5.2-5) Number of farmers and others who have applied improved technologies or management practices as a result of USG assistance- Farmers applying improved technologies or management practices in eleven quarters added 15,463 farmers, of whom only 124 were women, representing less than one percent. The LOP target was exceeded by 3%. Only 14% of the LOP target was achieved during Year 1, 68% in Year 2, and 42% during the last two quarters of Year 3. 3. (4.5.2-42) Number of private enterprises, Service Providers, Farmer Groups, producers organizations, water user associations, women’s groups, trade and business associations and community-based organizations (CBOs) that applied improved technologies or management practices as a result of USG assistance- The project overachieved the LOP target by 37%, 16% in Year 1, 30% in Year 2, and 91% in Year 3. 4. (4.5.2-2) Number of hectares under improved technologies or management practices as a result of USG assistance- The project brought a total 1,448 hectares of land under improved technologies or management practices, overachieving the LOP target by 11%. However, out of this area only 20 hectares were managed by women, or less than 1% of the LOP target. In Year 1 only 20% of the LOP target was achieved; 58% was achieved in Year 2, and 33% in Year 3. 5. (4.5.2-13) Number of rural households benefiting directly from USG interventions- The reached 29,026 rural households, representing 61% above of the LOP target. A total 6,333 women were also reached by the project (22%). The project reached 26% of the LOP target households, 85% in Year 2, and 50% in Year 3. 52 The addition of the percentages of LOP target above/below 100%, reflect an overall over/under performance. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 56 6. (4.5-2) Number of jobs attributed to FtF implementation- These are jobs related to Golden Harvest interventions as explained by the CCBA M&E group. This project created 1,055 jobs, or 5% of this project LOP target (21,600). In Year 1 it created 3% of jobs in the LOP target and 2% in Year 2. No jobs were created in Year 3. Despite the low number of total jobs created, 42% of them were taken by women. According to the PIRS, “CCBA will track and report from Golden Harvest and other partners´ records and reports. CCBA staff will review data on a quarterly basis for inclusion in quarterly reports and annual progress reports. At least 10% will be verified by the Manager.” There is no evidence of these verifications or how the jobs reported relate to the project. 7. (4.5.2-23) Value of incremental sales (collected at farm-level and outlets) attributed to FtF implementation- The project increased sales by USD 12,094,179 in the eleven quarters evaluated, exceeding the LOP target (USD 7,200,000) by 68%. In Year 1 it achieved 12% of the LOP target, 67% in Year 2, and 89% in Year 3. 8. (4.5-10) Total increase in installed storage capacity (m3) (GH): The project increased the storage capacity by 2,571 (m3) in Year 1, representing 43% of the LOP target, 2 m3 in Year 2 and nothing in Year 3. The main private sector partner, Golden Harvest, changed its focus from vegetables to ice cream at the end of Year 153 9. (4.5.2-38) Value of new private sector investment in the agricultural sector or food chain leveraged by FtF implementation- The value of new private sector investment leveraged by FtF was USD 9,233,800 over eleven quarters, or 91% of the LOP target. Most of this investment (89%) occurred in Year 1, 2% in Year 2 and nothing in Year 3. Custom Indicators 10. Post-harvest losses (% of total)- During the first year no reductions were noted. There was a 6% reduction in post-harvest losses out of the 15% target LOP, representing only a 40% achievement; these reductions were evenly distributed between Year 2 and Year 3. 11. Number of certifications for compliance with international food handling standards awarded to local: No result was reported during eleven months. In sum, out of eleven indicators, the project outperformed six in the eleven quarters considered. However, the project underperformed in one indicator relative to the LOP target, post harvest losses, and had not delivered certifications for compliance with international food handling standards awarded to local firms, as of March 2016. Figures for three indicators were provided for (as shown in the PITT) • 4.5-2- Number of jobs attributed to FTF implementation- 5% of LOP target • 4.5-10- Total increase in installed storage capacity (m3)- 43% of LOP target • 4.5.2-38- Value of new private sector investment in the agricultural sector or food chain leveraged by FTF implementation- 91% of LOP target 53 Annual Report July 2013 – September 2014, page 22. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 57 However, none of these figures were verified (see DQA 2015 and 2016). No performance was accounted for or documented. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 58 Annex VII. Performance Indicator Tracking Table (PITT) Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 59 Annex VIII: Sources of Information VIII.I: References and consulted documents Ahmed, S. & Monirul, H. (2013). Need Assessment of Postharvest Handling and Storage for Selected Vegetable Crops in Bangladesh. Consultancy for the Asian Vegetable Research and Development Center, Dhaka. Akhter U. A., Hernandez, R. & Quabili, W. (2015). Poverty Assessment for the FTF ZOI in Bangladesh: Change from 2011/12 baseline to 2015 midline. International Food Policy Research Institute, September, Dhaka. Chandni, J.F. (2014). Demand analysis of Golden Harvest Agro Industries Ltd. Thesis submitted to the BRAC Business School, BRAC University, Dhaka. Chowdry, H.M. (2012). Cold chain business in Bangladesh to enhance the economic growth of the country. Intl. Journal of Supply Chain Management, Vol. 1. No. 3. ISSN:2050-7399 (online). Consultative Group for International Agricultural Research. (n.d). Bangladesh, Research Program on Climate Change, Agriculture and Food Security, https://ccafs.cgiar.org/bangladesh#.V2QZXo5akQE) accessed on June 15, 2016. Davachi, C., Swimerton, K. & Bolick, C. (2013). Strategic Analysis of Supermarket Public Private Partnerships in Latin America (LAC PPP) Program. AID-OAA-M-12-00014 April 1, 2013 https://ssg￾advisors.com/files/2013/12/Strategic_Analysis_of_Supermarket_PPPs_in_Central_America.pdf Development Alternatives International (DAI). (2014). Value Chain Selection Report, End Market & Value Chain Analysis. Agricultural Value Chains (AVC) Project in Bangladesh AID-388-C-13- 00003, Dhaka. Export Promotion Bureau. (2015). Status of export of vegetables in Bangladesh. Marketing Wing of the Hortex Foundation, Dhaka. Gates Foundation (2013). Innovations in Feedback and Accountability Systems for Agricultural Development (Round 12), http://gcgh.grandchallenges.org/challenge/innovations-feedback-and￾accountability-systems-agricultural-development-round-12 Global Cold Chain Alliance. (2015). Kenya Cold Chain Assessment. www.gcca.org/wp-content/uploads/2016/02/KenyaColdChainAssessment2016_02_23.pdf Global Donor Platform for Rural Development. (2016). www.donorplatform.org/...454&return=aHR0cHM6Ly93d3cuZG9ub3JwbGF0Zm9ybS5vcmcvY29iY Wx0 Global Forum on Food Security and Nutrition. (2012). Enabling rural cooperatives and producer organizations to thrive as sustainable business enterprises http://www.fao.org/fsnforum/sites/default/files/files/82_cooperatives/SUMMARY_82_EN_rural_c ooperatives.pdf Golden Harvest and Winrock International (GH & WI), 2013, Memorandum of Understanding between Golden Harvest Agro Industries and Winrock International Institute for Agricultural Development. Dhaka, Bangladesh and Little Rock, AK. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 60 Gulati, A., Minot, N., Delgado, C. & Bora, S. (2005). Growth in high-value agriculture in Asia and the emergence of vertical links with farmers. Paper presented at the workshop “Linking Small-Scale Producers to Markets: Old and New Challenges,” The World Bank, Washington, DC Gustavsson, J., Cederberg, C. & Sonesson, U. (2011). Global Food Losses and Food Waste. The Swedish Institute for Food and Biotechnology. PowerPoint presentation to the Save Food Congress, Düsseldorf. Ingram, G.M., Johnson, A.E. & Moser, H. (2016). USAID’S Public-Private Partnerships: A data picture and review of business engagement. Global Economy and Development, Working Paper 94. http://www.brookings.edu/research/papers/2016/02/usaid-public-private-partnerships-ingram￾johnson-moser Institute For Strategy & Competitiveness, Harvard Business School. (n.d.). “Understanding and analyzing competitiveness”. Accessed on June 15, 2016 http://www.isc.hbs.edu/competitiveness-economic-development/frameworks-and-key￾concepts/Pages/default.aspx International Finance Corporation-South Asia Enterprise Development. (2008). High-Value Agriculture in Bangladesh: An Assessment of Agro-businesses Opportunities and Constraints. Bangladesh Development Series Paper No. 21. http://siteresources.worldbank.org/BANGLADESHEXTN/Resources/publications/384630- 1222304825031/BDAgricultureReport21.pdf International Food Policy Research Institute. (2015). Value Chains and Nutrition: A Framework to Support the Identification, Design, and Evaluation of Interventions. Discussion Paper 01413, Washington, DC, IFPRI. International Food Policy Research Institute. (2012). Bangladesh Household Survey Results, June 2012, Feed the Future Zone of Influence Baseline Report. Dhaka, IFPRI. International Fund for Agricultural Development. (n.d.). Smallholders´Access to Markets for Certified Sustainable Products (SAMCERT), accessed on June 15, 2016. https://www.ifad.org/documents/10180/184e7cbc-a136-48ed-aa28-ff7f37e7f7a9 Jany, M.N.H., Sarker, C., Mazumder, M.A.R. & Shikder, M.F.H. (2008), Effect of storage conditions on quality and shelf life of selected winter vegetables. J. Bangladesh Agril. Univ. 6(2):391-400. Kitinoja, L. (2013). Use of cold chains for reducing food losses in developing countries. White Paper No.13-03. The Postharvest Education Foundation, La Pine, Oregon. www.postharvest.org/...0of%20cold%20chains%20PEF%20white%20paper%2013-03%20final.pdf Kotinoja, L. & Thompson, J.F. (2010). Pre-cooling systems for small-scale producers. Stewart Postharvest Review 2:2, Online ISSN:1945-9656 Minten, B., Shafiqul Alam, A.Z.M., Deb, U.K., Kabir, A.Z.K., Laborde, D., Hassanullah, M. & Murshid, K.A.S. (2010). Agricultural marketing, price stabilization, value chains, and global/regional trade. Prepared for the Bangladesh Food Security Investment Forum. Mittal, S. (2007). Strengthening backward and forward linkages in horticulture: some successful initiatives. Agricultural Economics Research Review, Vol. 20 (Conference Issue), pp. 457-469. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 61 Narrod, C., Roy, D., Okello, J., Avendaño, B., Rich, K. & Thorat, A. (2007). The role of Public-Private Partnerships and Collective Action in Ensuring Smallholder Participation in High Value Fruit and Vegetable Supply Chains.” CAPRI Working Paper No. 70, Washington, DC, IFPRI. Organisation for Economic Cooperation and Development (OECD). (2012). Empowerment of poor rural people through initiatives in agriculture and natural resource development. (https://www.oecd.org/dac/povertyreduction/50157503.pdf ). Pakistan Agricultural Cold Chain Development Project. http://www.cawnagency.com Philippines Cold Chain Project (PCCP). www.winrockpccp.org. Poko, L. (2015). Conduct Cold Chain Study Tour for Project in Bangladesh. World Food Logistics Organization-Global Cold Chain Alliance. Ponniah, A., Puskur, R. Workneh, S. & Hoekstra, D. (2008). Concepts and practices in agricultural extension in developing countries: A source book. International Research Institute. http://www.cida-ecco.org/IPMS/IPMS_SourceBook/IPMS_SourceBook.pdf Prato, B. & Longo, R. (2012). Empowerment of Poor Rural People Through Initiatives in Agriculture and Natural Resource Management. In: Organization for Economic Cooperation and Development, Poverty reduction and Pro-Poor Growth: the Role of Empowerment. https://www.oecd.org/dac/povertyreduction/50157503.pdf Reardon, T., Chen, K., Minten, B. & Adriano, L. (2012). The Quiet Revolution in Staple Food Value Chains: Enter the Dragon, the Elephant and the Tiger. Asian Development Bank & International Food Policy Research Institute. http://www.adb.org/sites/default/files/publication/30063/quiet￾revolution-staple-food-value-chains.pdf Rockefeller Foundation. (2013). Waste and Spoilage in the Food Chain Decision Intelligence Document. https://www.rockefellerfoundation.org/app/uploads/Waste-and-Spoilage-in-the￾Food-Chain.pdf Sheba Manab Kallyan Kendra (SMKK). (n.d.). Ongoing projects – CCBA; funded by USAID. www.smkk.org/project/project.doc USAID. (2015). Local Private Sector Partnership Cold Chain Bangladesh Alliance - USAID’S Assets Amplify Local Partner Investments. https://www.usaid.gov/sites/default/files/documents/15396/ccba-draft_edited.pdf USAID. (2011). Bangladesh – Country Development Cooperation Strategy FY 2011-2016. https://www.usaid.gov/sites/default/files/documents/1860/Bangladesh_CDCS.pdf USAID. (n.d.a). Gender Equality and Women's Empowerment. https://www.usaid.gov/what-we-do/gender-equality-and-womens-empowerment USAID. (n.d.b). Evaluating Global Development Alliances – USAID. http://www.bsr.org/reports/Evaluation_Book.pdf USAID Microlynks, Value Chain Wiki. https://www.microlinks.org/using-value-chain-development￾wiki. Accessed, June 15, 2016. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 62 Winrock International, 2013, Revised Technical Proposal: Cold Chain Bangladesh Alliance, RFP No. RFA-388-13-000001, under the FY2012 GDA APS-OAA-12-000003, submitted to USAID/Bangladesh, WI, Little Rock, Arkansas. VIII.II: Project Specific Documents Annual Implementation Plans Cold Chain Bangladesh Alliance (CCBA), 2015a, Annual Implementation Plan Year 3: October 2015 – June 2016, AID-388-A-13-00004, USAID, Dhaka. Cold Chain Bangladesh Alliance (CCBA), 2014a, Annual Implementation Plan Year 2: October 2014 – September 2015, AID-388-A-13-00004, USAID, Dhaka. Cold Chain Bangladesh Alliance (CCBA), 2013, Annual Implementation Plan Year 1: July 2013 – September 2014, AID-388-A-13-00004, USAID, Dhaka. Annual Progress Reports Cold Chain Bangladesh Alliance (CCBA), 2015b, Annual Progress Report for CCBA Project October 2014 – September 2015, AID-388-A-13-00004, USAID, Dhaka. Cold Chain Bangladesh Alliance (CCBA), 2014b, Annual Progress Report for CCBA Project July 2013 – September 2014, AID-388-A-13-00004, USAID, Dhaka. Quarterly Reports Cold Chain Bangladesh Alliance (CCBA), 2016a, Quarterly Report January – March 2016, AID-388- A-13-00004, USAID, Dhaka. Cold Chain Bangladesh Alliance (CCBA), 2016b, Quarterly Report October – December 2015, AID￾388-A-13-00004, USAID, Dhaka. Cold Chain Bangladesh Alliance (CCBA), 2015c, Quarterly Report April – June 2015, AID-388-A-13- 00004, USAID, Dhaka. Cold Chain Bangladesh Alliance (CCBA), 2015d, Quarterly Report January – March 2015, AID-388- A-13-00004, USAID, Dhaka. Cold Chain Bangladesh Alliance (CCBA), 2015e, Quarterly Report October – December 2014, AID￾388-A-13-00004, USAID, Dhaka. Cold Chain Bangladesh Alliance (CCBA), 2014c, Quarterly Report April – June 2014, AID-388-A-13- 00004, USAID, Dhaka. Cold Chain Bangladesh Alliance (CCBA), 2014d, Quarterly Report January –March 2014, AID-388- A-13-00004, USAID, Dhaka. Cold Chain Bangladesh Alliance (CCBA), 2014e, Quarterly Report October – December 2013, AID￾388-A-13-00004, USAID, Dhaka. Others Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 63 Cold Chain Bangladesh Alliance (CCBA), 2016, Report on Cold Chain Expo, May 2016, CCBA Project, Dhaka. Cold Chain Bangladesh Alliance (CCBA), 2015f, Study on the Effectiveness of Refrigerated Trucks for Transportation & Marketing of Fresh Produce, CCBA Project, Dhaka. Cold Chain Bangladesh Alliance (CCBA), 2014f, Selection and Analysis of Agricultural Subsectors, CCBA Project, Dhaka. Cold Chain Bangladesh Alliance (CCBA), 2014g, Monitoring & Evaluation (M&E) Report, July 2014, Cooperative Agreement Number AID-388-A-13-00004, USAID, Dhaka. Cold Chain Bangladesh (CCB) Alliance, 2013, Attachment B – Program Description. AID-388-A-13- 00004. USAID/ACME, 2016a, Data Quality Assessment Report of the Cold Chain Bangladesh Alliance (CCBA), April, ACME, Dhaka. USAID/ACME, 2016b, CCBA Site Visit Report, 4 April 2016. ACME, Dhaka. USAID/ACME, 2015a, Data Quality Assessment Report of the Cold Chain Bangladesh Alliance (CCBA), September, ACME, Dhaka. USAID/ACME, 2015b, Field Visit Report, 7-9 July 2015. ACME, Dhaka. World Food Logistics Organization & Winrock International, 2014, Post Harvest Loss Assessment in Vegetable Systems & Recommendations, December, Dhaka Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 64 Annex IX: People Contacted Final list of people contacted # Date Time Name F M Title Organizati on Location 1 May 9 11:00 am Md. Nazrul Islam  Sr. M & E Specialist ACME ACME, Gulshan , Dhaka 2 Nasirul Islam  Evaluation Coordinator ACME Do 3 May 10 2:30 pm Mark Tegenfeldt  Deputy Director, FTF, EG Office USAID USAID Baridhara, Dhaka 4 Mohammad Sayed Shibly  Project Mgt. Specialist, EG Office, COR, CCBA USAID Do 5 M Nazmul Hossain Bhuiyan  Project Mgt. Coordinator, EG Office, POC, ACME USAID Do 6 Matt Curtis  Ag. Development Officer, EG Office USAID Do 7 11:00 am All Project staffs   ACME ACME, Gulshan, Dhaka 8 May 11 9:30 am James Phillips  Acting COP ACME Do 9 3:00 pm Amirul Islam  Former Marketing CCBA Do 10 May 12 10:00 am A B Siddiqui  COP CCBA Lake Shore Hotel 11 S S Bakht Hindole  M & E Specialist CCBA Do 12 May 16 10:30 am Dr. Anisur Rahman  DCOP CCBA CCBA, Gulshan, Office 13 A B Siddiqui  COP CCBA Do 14 May 17 2:30 pm Jahangir Alam  Lead Farmer Metholmar, Burichor, Comilla Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 65 15 Ruhul Amin  Farmer Do 16 Mannan  Do Do 17 Delwar Hossain  Do Do 18 Jashim Khan  Do Do 19 Kamal Mia  Do Do 20 3:30 pm Khursheda  Lead Woman Farmer Syedpur, Bashbari, Comilla 21 Rubi Akter  Woman farmer Do 22 Shahin Aktar  Do Do 23 Kamrun Nahar  Do Do 24 Rezia  Do Do 25 Khadiza  Do Do 26 Anowara  Do Do 27 Sitara Khanam  Do Do 28 Jahanara  Do Do 29 Bilkis Begum  Do Do 30 Jharna Das  Do Do 31 Shefali  Do Do 32 4:30 pm Rafiqul Islam  LSP Abidpur Bazar, Mokam, Burirchar, Comilla Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 66 33 May 18 2:30 pm Muksedur Rahman  Asst. Field Coordinator CCBA Fatehpur, Ponchom Khondo, Goyenghat, Sylhet 34 3:00 pm Hazi Belal  Lead Farmer Do 35 Azmaul Hossain  Farmer Do 36 Islam Uddin  Do Do 37 Shefatur Rahman  Do Do 38 Moina Miah  Do Do 39 Mohibul Islam  Do Do 40 4:30 pm Faisal Ahmed  LSP Do 41 May 19 9:00 am Ruma Rani  Lead Woman Farmer Kurigram, South Surma, Kamal Bazar, Sylhet 42 Bala Rani Das  Woman farmer Do 43 Sonju Rani Das  Do Do 44 Mukti Rani  Do Do 45 Rubi Begum  Do Do 46 Shikha Rani Das  Do Do 47 Sabiya  Do Do 48 Fateha Begum  Do Do 49 10:30 am Abdul Ahad Shahin  LSP Khaler Mukh Bazar, South Surma, Sylhet 50 11:30 am Dr. M. I. Nazrul  Sr. S. O. Training Resource BARI BARI Office, Sylhet Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 67 51 May 22 11:30 am Zakir Hai  COO, Food Division Golden Harvest Golden Harvest, Shanta Tower, Level 5, Link Road, Tejgaon, Dhaka 52 3:30 pm Niaz Rahim  Managing Director Agora Rahimafrooz Ltd, Mohakhali, Gulshan, Dhaka 53 May 24 10:00 am Md. Sahadat Hossain  Field Co-Ordinator CCBA, Jessore Birnarayanpur, Jessore 54 May 24 10:30 am Sahed Hossain  Whole Salers /Trader Lebutala, Birnarayanpur, Jessore 55 Mohon Ghosh  Whole Salers / Trader Do 56 11:00 am Jahangir Hossain  Transporter Do 57 Md. Nazim Uddin,  President, MC, Collection Point, Lead Farmer Do 58 11:30 am Monirul,Alim  Farmer Do 59 Torikul islam  Farmer Do 60 Mofijul biswas,  Farmer Do 61 Sukumar ghos,  Farmer Do 62 Nijamuddin,  Farmer Do 63 Rustom Ali  Farmer Do 64 Abul Kasem,  Farmer Do 65 Mofijul mollah  Farmer Do 66 Kashem  Farmer Do 67 12:30 pm Monirul Islam  S. O. T Training resource BARI RARS, Jessore Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 68 68 Dr. Md. Altaf Hossain  Sr. S.O. BARI RARS, Jessore 69 3:30 pm Salina Akter-  Lead Woman farmer Woman Group Hurgati, Vojgati, Monirampur, Jessore CCBA Office, Harinath Dutta Lane, Jessore 70 Tahmina Begum  Woman Farmer Do 71 Khadija Begum-  Woman Farmer Do 72 Jashim  Farmer Do 73 May 25 9.30 am Md. Moslem Uddin  Field Coordinator, & Regional Manager, Khulna office. CCBA Hotel City Inn, Khulna 74 10:00 am Mrinal Sarkar  Project Coordinator, Tala, Satkhira SUS Do 75 11:00 am Md. Jakir Hossain Sarder  Commission agent, whole seller, proprietor Whole saler Paikari kacha Bazar, Sonadanga Truck Terminal, Khulna 76 12:30 pm 12 Women Farmers  Women Farmer Group Rangpur, Dumuria, Khulna 77 2:30 pm Nazrul Islam  UAO, and resource person DAE, Dumuria Upazila, Khulna Khukra, Dumuria, Khulna 78 May 26 9:30 am Safia  Woman Lead Farmer Arjikalikapur, Union: Chandpasa, Upazilla: Babugonj, District: Barisal 79 Rubi Akter  Farmers Do 80 Sahina Begum  Do Do 81 Nasrin  Do Do 82 Nehar  Do Do Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 69 83 Salma  Do Do 84 Taslima  Do Do 85 Mukti  Do Do 86 Minara  Do Do 87 Khukumoni  Do Do 88 Sefali  Do Do 89 Soma  Do Do 90 Bubli  Do Do 91 Jhumur  Do Do 92 Ferdousi  Do Do 93 11:30 am Md. Jasim Uddin  LSP Arjikalikapur, Union: Chandpasa, Upazilla: Babugonj, District: Barisal 94 12:30 pm Aftab  Lead Farmer Do 95 Dulal Khan  Farmer Do 96 Aminul  Do Do 97 Samsul  Do Do 98 Chan Mia  Do Do 99 Delowar  Do Do 100 Siddique  Do Do 101 Lal Mia  Do Do Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 70 102 Habib  Do Do 103 3:00 pm Md. Ziaur Rahman,  Scientific Officer BARI BARI, Rahamatpur, Barisal 104 Ujjal Kumar Roy  Field Coordinator CCBA Barisal 105 May 29 10:00 am Dr. AHM Monirul Haque  Manager- Production & Post Harvest CCBA CCBA office, Gulshan , Dhaka 106 12:00 pm Mustabshira Jannat  Monitoring Officer CCBA Do 107 S S Bakht Hindole  M & E Specialist CCBA Do 108 Nizam Chowdhury  Head Marketing CCBA Do 109 May 31 11:00 am AKM Shirajul Islam  ED BASA BASA office, Mohakhali, DOHS, Dhaka 110 Dulal Chandra Das  Program Coordinator BASA Do 111 Liton Kumar Dutta  Deputy GM BASA Do 112 2:30 pm AFM Asif  CEO Bengal Meat Do 113 June 1 2:30 pm Nasir Uddin:  Quality Assurance Manager BRAC Hotel Lake Shore 114 Abul Hossain  Farmer Gazipur Group Do 115 Aktar Hossain  Lead Farmer Do Do 116 Muksed  Whole Salers / Trader Do Do 117 June 2 9:00 am Bani amin  DCOP AVC AVC, Gulshan Office, dhaka 118 11:30 am Mustafa Salam  Joint Director BADC BADC office Motijheel, Dhaka 119 Md. Jasim Uddin  Project Director BADC Do Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 71 120 Mahmood Hossain  Ex DG BADC Do 121 2:00 pm Allexis Ellicott  COP CNFA CNFA Office, Gulshan, Dhaka 122 3:30 pm Dr.Anisur Rahman  Ex DCOP CCBA Hotel Lake Shore, Dhaka 123 Mr. Moazzem Hossain  Ex Grant Manager Do Do 124 June 4 12:00 pm S.M. Jahangir Hossain  President, BFVAPEA BFVAPEA office, Motijheel, Dhaka 125 Md. Shafique Ullah  Secretary Do Do 126 June 5 11:00 am Samad Choudhury  Director Golden Harvest Golden Harvest, Tejgaon, Dhaka 127 Md. Mohsin Uddin  Business Development Manager Do Do 128 2:30 pm Abdus Salam  M & E Manager, BAGH Project; Ex M & E Manager CCBA Hotel Lake Shore 129 June 7 11:00 am Dr. Abdul Jalil Bhuyan  Managing Director Hortex Foundation Horetx Foundation, Manik Mia, Dhaka 130 Md. Rafiqul Islam  AGM, GAP Do Do 131 Mitul K. Saha  AGM, VC, R & D Do Do 132 1:00 pm Sonia Akter  Business Training Coordinator SGS SGS, Karwan Bazar, Dhaka 133 Anisur Rahman  Sr. Asst. Manager Do Do 134 June 8 9:00 am Mohammad Sayed Shibly  Project Mgt. Specialist, EG Office, COR, CCBA USAID USAID Baridhara, Dhaka 135 Mark Tegenfeldt  Deputy Director, FTF, EG Office USAID Do Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 72 136 Shayan Shafi  Project Mgt. Specialist, EG Office USAID Do 137 Motasim Billah  M & E Specialist Program Office Do Do 138 Fareen Khurrum  Sr. M & E Specialist Program Office Do Do 139 Shahidur Bhuiyan  Sr. Ag Economist EG Office Do Do Blue shaded numbers represent 35 Field Reports Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 73 Annex X: Assessment of Training Materials CCBA Training Module Contents and Relevance to Project Objectives Sl Training Module/Manual Major Contents Remarks 01 Food Safety Management Systems (FSMS) • Introduction to FSMS • ISO 22000-2005-Overview and Clause 4 • Food safety hazards and control measures • ISO 22000-2005-Overview and Clause 5,6 • ISO 22000-2005-Overview and Clause 7,8 • This manual contains relevant topics which seem to be of very high standard in terms of contents & extent. • SGS developed this module 02 Food Safety begins on the farm – A grower’s guide • This is for GAP for fresh fruits and vegetables • Introduction – food borne diseases, consumer concerns, record keeping, potential sources of on farm contamination • Factors for Minimizing risks starts before planting • Factors for Minimizing risks during production • Factors for Minimizing risks at harvest • Factors for Minimizing risks during post harvest • Food safety is everyone’s responsibility • Business Plan • This publication is supported by the cooperative state research education & extension services, USDA & USFDA • This guide book looks good and specific and handy for the users. Important how these were translated / customized for the farmers • A business plan (4 pages) is also attached at the end though not reflected in table of contents 03 Production of High value Vegetables/ Crops, Harvest & Post Harvest Technology • Improved Technique to Produce Seeds & Seedling • Improved Technique /GAP of production, harvest & post harvest for Egg Plant, Tomato, Bitter Gourd, Chili, Onion, Beans, Carrot, Cauliflower, Cucumber • The technical part covered here is thorough and specific • No indication /evidence of cold chain concept 04 Post Harvest Management of Horticulture Crops • Traditional & better practices on post harvest management technique used by farmers of Tomato, Cauliflower, Onion, Bitter Gourd, Beans, Cucumber & Egg Plant • Food processing Techniques of – - Egg Plant Pickles • The technical part covered here is thorough and specific (each crops given 3- page pictorial description) • No indication /evidence of cold chain concept Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 74 - Cauliflower Pickles - Preservation of Cauliflower, Carrot in Brine - Tomato Ketch up • Steps for processing activities mentioned in brief but specific way 05 Integrated Pest Management (IPM) • Pest Management Technique for crops with special focus on Egg plant & Bitter Gourd • Details of different crop specific pests – symptoms, damage & its control • Management of Soil Nutrition • The technical part covered here addresses important aspects, though the number of pictorial descriptions is minimum • No mention of the cold chain concept 06 Marketing & Distribution of Agricultural Crops/Products • Theoretical description of principles of marketing, distribution • Basic demand & supply mechanism • Agro Products value chain & actors • Basics of Production cost, profit & loss calculations • Concept of combined distribution & Collection Center • Introduction to crops plucking, grading, cleaning, sorting, packing etc. • Basics of distribution of perishable crops with an introduction of Cold chain • Market linkages – buyer-seller meet technique • Basic of cold chain concept (2 page) introduced here • Basics of demand – supply for ag. Products • Pricing policy & introduction to profit & loss introduced (1 page) • Management of collection points (CP), requirements for efficient operations of this CP (3 page) • Market linkage (half page) introduced 07 Local Service Provider (LSP) & Agro Inputs Sellers • Introduction of CCBA project & its objective & incentives for the participants • Pre training tests • Management of seed, soil & fertilizers • IPM • Basics of retailing, creative selling, effective communication, customer services • Awareness of using pesticides • Introduction to Inventory management, balance sheet • Post training tests • Pictorial presentation for pests, seed, etc. • Brief orientation to inventory management, balance sheet (I page/30 mins) • Brief on market information, ethical business and customer services (30 min session) • No indication of cold chain concept 08 Balanced & adequate Diet for Better Health • Advantages of taking adequate balanced diet • Nutrition data base table for various crops/ items & age wary demand table for nutrition • Food pyramid included • No indication of cold chain concept • In the concluding part of this manual a brief of CCBA project is attached where it says Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 75 CCBA is primarily responsible for linking farmers with market system & its different catalysts to make sure that farmers get right price / value for it’s products. Another important aspect of CCBA project is to make villagers aware of nutrition, balanced diet and good health 09 Vegetables Nutrition & Gender Equality • Techniques, requirements & importance of preparing nutritious food/meals • Introduction to gender equality & its importance in agriculture development • Importance of nutrition, sources of nutrients, preparation for nutritious food, related diseases in lack of nutrients introduced (7 pages) • Raising awareness of gender, it’s equity & importance, women empowerment, importance of gender equity in agriculture, society and family. Source: CCBA Project (manuals in Bangla assessed by Hasan Iqbal) General Comments: • In most modules there is no such agricultural Economics / Business topics covered in depth. • In spite of being a cold chain project, the cold chain concept was not introduced the way it must be in any of the modules and only slight touch in one of them • Pre & post training test concept included in case of LSP & Input suppliers • The ultimate quality and effectiveness of these trainings (usefulness of training modules) depend a lot on the trainers who eventually developed power point presentations not based on these modules. • Module 1 (FSMS) was conducted by SGS and received by partners such as Agora, GH, PNGO etc for certification purposes and so it is expected that training was effective • Two Power point presentations were shared with the Team and found the content for production technology of Tomato and Bitter Gourd to be satisfactory. However, no cold chain concept or agro-economic perspective were included. Accelerating Capacity for Monitoring and Evaluation (ACME) AID-388-C-14-00001 Cold Chain Bangladesh Alliance (CCBA) Final Performance Evaluation Final Report International Business & Technical Consultants, Inc. (IBTCI) 76 Annex XI. Gantt Chart for the CCBA Final Performance Evaluation U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 USAID/Bangladesh Economic Growth Office Madani Avenue Dhaka, Bangladesh