March 20, 2015 This publication was produced for review by the United States Agency for International Development. It was prepared by Mr. Michael Viola, Dr. Sahr Lebbie, Ms. Paula Higgins, Mr. Jean de Dieu Kampayana, Mr. Ildephonse Musafiri, Ms. Jeanne Francoise Umuzigambeho, and Mr. Greg Norfleet on behalf of Development & Training Services, Inc. (dTS). MIDTERM PERFORMANCE EVALUATION OF THE RWANDA DAIRY COMPETITIVENESS PROGRAM II (RDCP II) Prepared for the United States Agency for International Development, USAID Contract Number AID￾RAN-I-00-09-00015, Task Order Number < AID-696-TO-15-00002>, Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program. Implemented by: Development & Training Services, Inc. (dTS) 4600 North Fairfax Drive, Suite 402 Arlington, VA 22203 Phone: +1 703-465-9388 Fax: +1 703-465-9344 www.onlinedts.com ACKNOWLEDGEMENTS The dTS evaluation team would like to express its most sincere thanks to the entire Land O’Lakes team, whose cooperation, assistance, and patience made this evaluation possible. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II ii MIDTERM PERFORMANCE EVALUATION OF THE USAID/RWANDA DAIRY COMPETITIVENESS PROGRAM II (RDCP II) March 20, 2015 This publication was produced for review by the United States Agency for International Development. It was prepared Development & Training Services, Inc. (dTS). DISCLAIMER The authors' views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II iii TABLE OF CONTENTS ACKNOWLEDGEMENTS................................................................................................................. ii Contents ....................................................................................................................................iv Acronyms ................................................................................................................................ vii Executive Summary................................................................................................................... 1 Project Background ....................................................................................................................................... 1 Purpose and Evaluation Questions............................................................................................................. 1 Purpose...................................................................................................................................................... 1 Questions.................................................................................................................................................. 1 Evaluation Design, methods and limitations............................................................................................. 2 Data Collection Methods........................................................................................................................ 2 Data Analysis Methods........................................................................................................................... 3 Strengths.................................................................................................................................................... 3 Limitations................................................................................................................................................ 3 Findings and Conclusions............................................................................................................................. 3 Summary OF RECOMMENDATIONS .................................................................................................. 4 Land O’ Lakes.......................................................................................................................................... 4 Government of Rwanda ......................................................................................................................... 5 Future Evaluators .................................................................................................................................... 6 Program Background................................................................................................................. 7 Context ............................................................................................................................................................ 7 Evaluation Purpose and Questions ........................................................................................... 9 Evaluation Purpose........................................................................................................................................ 9 Evaluation Questions.................................................................................................................................... 9 Evaluation Methods and Limitations......................................................................................11 Data Collection Methods......................................................................................................................11 Data Analysis Methods.........................................................................................................................12 Data Warehousing Statement ..............................................................................................................12 Team Composition................................................................................................................................12 Strengths and Limitations of the Evaluation.....................................................................................12 Findings, Conclusions, and Recommendations......................................................................14 Question 1: To what extent has the project contributed to improved standards and quality of Rwandan dairy products in the Zone of Influence (ZOI) and the expansion of investments in dairy processing and marketing in the ZOI? ................................................................................................................................14 Findings...................................................................................................................................................14 Conclusions.............................................................................................................................................21 Recommendations................................................................................................................................. 22 Question 2: To what extent have new management approaches (staffing structures) used to implement development of financial products by INSPIRE, increased investment, improvement of quality milk/Seal of Quality contributed to achievement of RDCP II results?................................................................24 Findings...................................................................................................................................................24 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II iv Conclusions.............................................................................................................................................25 Recommendations................................................................................................................................. 25 Question 3: Based on return on investment at MCC level and service providers, what would be the most cost-effective management approach to implement scaling up technology?.....................................26 Findings...................................................................................................................................................26 Conclusions.............................................................................................................................................26 Recommendations................................................................................................................................. 26 Question 4: Based on RDCP II approach to reach out to high number of rural and vulnerable farmers through MCCs and service providers through new technologies, to what extent is the project reaching the intended population and the targeted number of beneficiaries in terms of knowledge and skills (adoption of technology), and the targeted number of beneficiaries in terms of diet behavior change as a result of the milk consumption campaign? ....................................................................................................................27 Findings...................................................................................................................................................27 Conclusions.............................................................................................................................................28 Recommendations................................................................................................................................. 28 Question 5: What opportunities for new partnerships (not yet explored) are available to scale improved practices in the ZOI?...................................................................................................................................29 Findings...................................................................................................................................................29 Conclusions.............................................................................................................................................29 Recommendations................................................................................................................................. 29 Question 6: What has been the influence at the institutional level of the RDCP II Seal of Quality program in terms of increased milk quality and compliance to dairy products (including raw and pasteurized milk, yogurt, and cheese) standards, on GOR agencies (Rwanda Agriculture Livestock Inspection Services, Kigali City, Rwanda Bureau of Standards, relevant districts, if necessary, etc.) and the dairy industry? 31 Findings...................................................................................................................................................31 Conclusions.............................................................................................................................................31 Recommendations................................................................................................................................. 32 Question 7: Considering RDCP II approach for the different components and activities, what are the ones that have potential to be more sustainable after RDCP II closes and what would be the recommended change to foster sustainability?..................................................................................................................33 Findings...................................................................................................................................................33 Conclusions.............................................................................................................................................33 Recommendations................................................................................................................................. 33 Question 8: Using the WEAI framework, did the RDCPII approach to gender equality and equity ensure balanced involvement of women and men and particularly address constraints faced by women in the industry? ........................................................................................................................................................34 Findings...................................................................................................................................................34 Conclusions.............................................................................................................................................37 Recommendations................................................................................................................................. 38 Summary of Recommendations...............................................................................................39 Land O’ Lakes........................................................................................................................................39 Government of Rwanda .......................................................................................................................40 Future Evaluators ..................................................................................................................................40 APPENDICES......................................................................................................................... 41 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II v Appendix A. USAID Evaluation Policy...................................................................................................41 Appendix B. Statement of Work...............................................................................................................42 Appendix C. Getting to Answers Table...................................................................................................54 Appendix D. Sources of Information.......................................................................................................65 Appendix E. Work Plan..............................................................................................................................76 Appendix F. Findings, Conclusions, and REcommendations..............................................................79 Appendix G. Copies of Instruments........................................................................................................94 Appendix H. Conflict of Interest Disclosures......................................................................................121 Appendix I. Statement of Differences Table ........................................................................................ 126 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II vi ACRONYMS AI Artificial Insemination AGDP Agricultural Gross Domestic Product BDS Business Development Services BRD Banque Rwandaise de Development (Rwanda Development Bank) CEZONYI Cooperative des Eleveurs de la Zone Nyiragikokora CODERU Cooperative del Eleveurs de Rubavu COMESA Common Market for Eastern and Southern Africa DCOP Deputy Chief of Party dTS Development and Training Services, Inc. FGD Focus Group Discussions FtF Feed the Future GOR Government of Rwanda IAKIB Ihuza Aborozi ba Kijyambere Bafatanyije INSPIRE Firm contracted by LOL to develop prospectuses on starting dairy businesses KII Key Informant Interview LOL Land O’Lakes LOP Life-of-Project KCB Kenya Commercial Bank MCC Milk Collection Center M&E Monitoring and Evaluation MF Model Farmer MFI Micro-finance Institution MINAGRI Ministry of Agriculture and Livestock NDS National Dairy Strategy NGO Non-Governmental Organization PSF Private Sector Federation RDCP II Rwanda Dairy Competitiveness Program II RNDP Rwanda National Dairy Platform ROI Return on Investment SACCO Savings and Credit Cooperative Organization SOQ Seal of Quality SP Service Provider TOT Training of Trainers UC Davis University of California Davis UOB Urwego Opportunity Bank USAID United States Agency for International Development USAID/R United States Agency for International Development in Rwanda WEAI Women Empowerment in Agriculture Index ZOI Zone of Influence Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II vii EXECUTIVE SUMMARY PROJECT BACKGROUND This is the Final Report of a Midterm Performance Evaluation of the Rwanda Dairy Competitiveness Program II (RDCP II), managed by Land O’Lakes (LOL) and funded by USAID/Rwanda. The evaluation was carried out by Development and Training Services, Inc. (dTS) during the period February 21-March 29, 2015 by a six-person team: three (3) Rwandan nationals and three (3) expatriates. The RDCP II was originally conceptualized and designed as a “value chain” program, the implementation of which would result in a reduction in poverty due to increased employment and income generated by the increased production and marketing of quality raw milk, the term “quality” referring to milk that meets the food-safety standards of COMESA. The RDCP II goal is: Rwandan dairy products competitive in regional markets. PURPOSE AND EVALUATION QUESTIONS PURPOSE The rationale for the evaluation is fourfold: 1. Assess the achievement of the RDCP II to date from the standpoints of its overall performance, coverage and outreach approach, efficiency, sustainability, and gender equality and equity; 2. Verify and ground-truth, to the extent feasible, RDCP II results reported by Land O’Lakes (LOL); 3. Generate data and information to allow USAID/RWANDA and LOL to make informed decisions concerning the best use of RDCP II resources over the remaining life of the program; and, 4. Assist USAID/RWANDA to understand the mechanics and methodologies involved in the conduct of performance evaluations. QUESTIONS 1. To what extent has the project contributed to improved standards and quality of Rwandan dairy products in the Zone of Influence (ZOI) and the expansion of investments in dairy processing and marketing in the ZOI? 2. To what extent have new management approaches (staffing structures) used to implement development of financial products by INSPIRE, increased investment, improved quality milk/Seal of Quality contributed to achievement of RDCP II results? 3. Based on return on investment at MCC level and service providers, what would be the most cost-effective management approach to implement scaling up technology? 4. Based on the RDCP II approach to reach out to a high number of rural and vulnerable farmers through MCCs and service providers through new technologies, to what extent is the project reaching the intended population and the targeted number of beneficiaries in terms of knowledge and skills (adoption of technology), and the targeted number of beneficiaries in terms of diet behavior change as a result of the milk consumption campaign? Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 1 5. What opportunities for new partnerships (not yet explored) are available to scale improved practices in the ZOI? 6. What has been the influence at the institutional level of the RDCP II Seal of Quality program in terms of increased milk quality and compliance to dairy products (including raw and pasteurized milk, yogurt, and cheese) standards, on GOR agencies (Rwanda Agriculture Livestock Inspection Services, Kigali City, Rwanda Bureau of Standards, relevant districts, if necessary, etc.) and the dairy industry? 7. Considering the RDCP II approach for the different components and activities, what are the ones that have potential to be more sustainable after RDCP II closes and what would be the recommended change to foster sustainability? 8. Using the WEAI framework, did the RDCPII approach to gender equality and equity ensure balanced involvement of women and men and particularly address constraints faced by women in the industry? EVALUATION DESIGN, METHODS AND LIMITATIONS DATA COLLECTION METHODS The evaluation employed the following data collection methods: Secondary Research Document review Soon after being awarded the contract, the dTS team conducted a literature review focused on the performance of the Rwandan Dairy Sector. Documents reviewed included previous evaluations published by USAID and implementing partners, annual and quarterly reports provided by Land O’ Lakes, project design documents, and economic analyses from international organizations such as the World Bank. Appendix D provides a complete list of documents used in this evaluation. Primary Research The team’s fieldwork utilized both quantitative and qualitative methods in order to triangulate results. The team completed household surveys along with questionnaires to quantitatively investigate the impact of the RDCP II on individuals and institutions. This data was supplemented with qualitative investigations. Surveys and Questionnaires Surveys1 were conducted at 132 households (27 women, 105 men); and questionnaires were completed at seven (7) financial institutions. In total, 90 participants and 42 non-participants completed surveys in the areas around MCCs visited by the team. Focus Group Discussions and Key Informant Interviews Focus Group Discussions (FGDs) were organized at fourteen (14) Milk Collection Centers (MCCs) and Key Informant Interviews (KIIs) were held with eighteen (18) Model Farmers and eleven (11) Service Providers 1 The sampling strategy was based upon a convenience sample of households located near MCCs visited by the team. There is a possibility of selection bias because both participants and non-participants were given the opportunity to take part in the program leading to the possibility that the two groups have some unmeasured difference that may have impacted the results (e.g., the MCCs have a membership fee so participants may be more affluent). Despite this limitation, the results provide a useful glimpse into the impact of the RDCP II program and are illustrative of the program’s actual effect. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 2 (SPs) in the Rwandan dairy value chain, and with government and private sector actors in the field and in Kigali. Field Travel A considerable portion of the ZOI was visited by the evaluation team, including travel to all (4) milk sheds – Eastern, North/West, Southern, and Kigali - and ten (10) of the seventeen (17) districts in which the RDCP II is operating. Field travel occurred during the periods March 4, March 5-7, March 8-11, and March 12, a total of nine (9) days; a copy of the field travel schedule is attached in Appendix E. DATA ANALYSIS METHODS The evaluation employed the following data analysis methods: Descriptive Statistics Numerical data was analyzed using measures of central tendency and are reported based on percentages throughout the paper. The evaluation team took extra care to disaggregate gender where data were person￾focused. Content Analysis Recurring themes were identified through the KIIs and FGDs which lead to the general conclusions reported throughout the evaluation. The evaluation team provided special focus to investigating issues related to access for women where applicable. STRENGTHS The team was comprised of an international group with varied expertise that provided a number of different perspectives on how to approach the evaluation. The team was able to investigate the efficacy of the project through technical, financial and gendered lenses. LIMITATIONS The contract for this evaluation was awarded later than initially expected, which led to a compressed timeline for the study. The evaluation team began work immediately and were in the field three weeks after the contract was awarded and were only able to spend twenty-seven days in the field. Although the team performed admirably under the circumstances, there is a potential that more time in the field would have yielded more complete analysis. FINDINGS AND CONCLUSIONS The RDCP II program overall has had an extremely positive impact on the dairy industry in Rwanda. The program can be credited with improving the availability and quality of milk as well as the improving the financial viability of dairy producing operations throughout the agricultural value chain. The relationships that the program has formed with various governmental and private sector actors should serve as a model for similar programs and have allowed the formation of a common sense of purpose and pride throughout the industry as should the dedication shown to inclusion of women into the program. The data suggests that participation in the program is associated with higher incomes for dairy farmers and is well received by participants. The most cost-effective aspects of the program are the Seal of Quality program along with investments in technology and business development services. One interesting finding was that Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 3 neighbors within a particular MCC’s ZOI may not have been aware of the services being offered, the exact cause of this lack of awareness is surprising given the extent of outreach programs conducted by LOL. The most important factor in the program’s success is the incredible network of program champions throughout the agricultural value chain and these relationships are critical for the sustainability of the program. Appendix F provides a full findings, conclusions and recommendations table. SUMMARY OF RECOMMENDATIONS LAND O’ LAKES 1. For the remaining life of the project, the RDCP II should continue implementing activities directed at increasing the production and improving the quality of raw milk, but the focus of the program should be shifted to: a. Increasing local demand for Rwandan milk and dairy products and their marketing in neighboring countries b. The enhancement of the dairy processing component of the value chain in order to accommodate increased raw milk production and the demand created by increased local consumption and export marketing of Rwandan dairy products 2. Consideration should be given to a follow-on dairy value chain activity that will seek to preserve the achievements of the RDCP II and address a new phase of dairy value chain programming by: a. Strengthening the management and marketing capability of the MCCs b. Assuring the readiness and capacity of the RNDP to assume responsibility and take on the challenges associated with supporting and advocating for the Rwandan dairy industry c. Promoting local consumption of Rwandan milk and dairy products and their export to neighboring countries d. Further strengthening of the capacity to process Rwandan milk and dairy products. 3. Land O’ Lakes should investigate the relationship between higher quality offerings vis a vis their increased availability and impact on demand2. 4. Land of Lakes and USAID should reduce, where possible, the number of indicators associated with this study. a. Additional indicators associated with a nutrition objective and the USAID Nutrition policy should not be inserted into the program at this time. The possibility exists for adding a nutritional dimension to future follow-on studies. b. If additional indicators are added, care should be exercised so that LOL managers and staff do not change their approach – the value chain and dairy business should remain their focus. 5. The $5,000,000 target in FY 2015 in new private-sector investment in the dairy value chain be revised in light of the current climate for investment. 2 See Statement of Differences Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 4 6. Building on findings in the Rwanda Dairy Strategy and Deep Dive Investment Analysis, a formal and concerted effort should be undertaken to explore the actual potential for the capitalizing on the demand for Rwandan dairy products in neighboring countries. 7. LOL should limit the scope of responsibility for the full-time M&E specialist to purely activities associated with monitoring and evaluation, especially dealing with data quality assurance protocols. 8. The reported regular and ongoing unavailability of semen and hormones and privatization of that service should be taken up with the GOR by Land O Lakes and other key players in the industry including RDFA. 9. Artificial Insemination (AI), mastitis control, and technologies generally associated with milk quality improvement should be scaled up by implementing partner organizations. 10. The Shisha Wumva milk consumption education outreach campaign should build upon success of reaching a large number of potential participants. Implementing partners would be wise to incorporate behavioral change communications and strategies into the program. 11. LOL should continue to nurture the relationship between RDCP II and the GOR, which has significantly contributed to the overall effectiveness of the SOQ. 12. The relationships that have been formed through RDCP II should continue to be a point of emphasis for LOL. 13. LOL should continue efforts to improve business knowledge within MCCs. 14. Land O’Lakes and the MINAGRI should continue outreach activities, especially encouraging women in positions of leadership within cooperatives. 15. The LOL gender mainstreaming strategy, given its age, should be reviewed and updated to accommodate changes in the program environment and any shift in program focus that might occur during the remaining life span of the program. 16. The LOL gender specialist should incorporate the WEAI framework into the RDCP II framework. GOVERNMENT OF RWANDA 17. The GOR along with LOL should investigate ways to strengthen collaboration between various actors in monitoring compliance with the Ministerial Instructions. 18. The channeling of RDCP II funds through financial institutions should be explored as a means to create relationships between formal financial institutions and program beneficiaries. . A check off system should be considered in this regard. 19. The GOR must enforce the Ministerial Instructions on the Preservation, Collection, Transportation and Selling of Milk and establish a regulatory and inspection system to ensure the high quality of Rwandan milk and dairy products. 20. Land O’Lakes and the GOR should continue outreach activities, especially encouraging women in positions of leadership within cooperatives. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 5 FUTURE EVALUATORS 21. The lack of control over bank accounts for some women in the sample should be further investigated in a larger study. 22. Future evaluators should further investigate the reasons for a lack of awareness with the ZOI of several MCCs participating in RDCPII. This should inform future outreach strategies to identify potentially hard to reach populations and to inform outreach strategies. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 6 PROGRAM BACKGROUND CONTEXT The government of Rwanda (GOR) has set out a course to become a middle income country within the next five years with its Vision 2020 program. The program is based upon the six pillars of (1) good governance, (2) an efficient state, (3) skilled human capital, (4) a strong private sector, (5) infrastructure development, and (6) a productive agricultural sector. The dairy sector in Rwanda has been identified as a strategic commodity that offers a potential path out of poverty for large numbers of households within the country. The dairy sector represents 15 percent of Rwanda’s agricultural gross domestic product (AGDP) and 6 percent to overall gross domestic product (GDP). The Rwanda Dairy Competitiveness Program II (RDCP II) is a $15 million, 5 year activity implemented by Land O’ Lakes, Inc (LOL) and funded by USAID. The project is designed to reduce poverty through expanded marketing of quality milk. The theory of change of the program is that marketing of quality milk and complying with standards the industry will generate employment and income, improve nutrition in rural households, and improve its overall competitiveness. RDCP II intends to meet these goals by linking new and existing small dairy producers to expand market demand in Rwanda and the surrounding region, driven by improved quality of milk production and effective dairy operations, reduced transaction costs and increased investment all along the dairy value chain. To achieve this, the program provides a number of activities to increase adoption of new technologies and practices throughout Rwanda’s Dairy Sector while also providing business development services and consumption campaign facilitation to stimulate demand. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 7 The program is expected to achieve the following life-of-project (LOP) results: • 60% increase in the volume and value of dairy products sold; • 30 new products awarded with the Rwanda Seal of Quality; • 60% increase in net household income; • 7,500 new dairy-related jobs created; • US$20 million in non-US government resources leveraged; and, • 9 policy reforms advocated for or enacted. As of September 2014, the program has already reported: • A net increase in dairy income of $5,018,533.24 in a single year • Seal of Quality kits have been provided to over 30 dairy cooperatives and 5 milk processors • A total of 9,152 jobs (738 female and 8,414 men) created directly associated with RCDP II activities. 91% of which were taken by individuals ages 18-35. • A LOL survey found that farmers in the program had increased their gross margin by 17.7% directly attributable to an increase in the price of milk and increased demand. • Exports of $14.9 million in a single calendar year, with $1.6 million worth of dairy products exported in August 2014 alone. • 228 private enterprises, milk porducers, women’s groups community based organizations have applied technologies provided by the program • Three national policies developed – including the National Dairy Strategy (NDS), the national milk cerification plan and the restructuring of the Rwanda National Dairy Platform (RNDP), Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 8 EVALUATION PURPOSE AND QUESTIONS EVALUATION PURPOSE In accord with the Task Order plus discussions with the USAID/Rwanda Program Office, the rationale for the evaluation is fourfold: 1. Assess the achievement of the RDCP II to date from the standpoints of its overall performance, coverage and outreach approach, efficiency, sustainability, and gender equality and equity. 2. Verify and ground-truth, to the extent feasible, RDCP II results reported by Land O’ Lakes (LOL) 3. Generate data and information to allow USAID/Rwanda and LOL make informed decisions concerning the best use of resources over the remaining life of the program and to inform future programming. 4. Assist USAID/Rwanda in understanding the mechanics and methodologies involved in conducting performance evaluations. EVALUATION QUESTIONS In order to respond to these objectives, a series of eight overarching evaluation questions were developed in collaboration with USAID. 1. To what extent has the project contributed to improved standards and quality of Rwandan dairy products in the Zone of Influence (ZOI) and the expansion of investments in dairy processing and marketing in the ZOI? a What are the major contributing factors and constraints to achieving these results? b What corrective measures may be necessary to ensure objectives are achieved? c To what extent are project beneficiaries (farmers, traders, milk collection centers) experiencing expected changes in their skills, knowledge, attitudes, behaviors, yield, income, etc.? i Is the project on track to meeting the currently set targets, and should current targets be modified? ii To what extent have new technologies introduced by the project improved animal feeding, artificial insemination, hygiene, and disease control and contributed to increased farmer income? Considering RDCP II costs, what is the project return on investment (ROI) of each of the technology transfer activities at farm level (Artificial Insemination, animal feeding, mastitis control), financial products development, Seal of Quality, cheese-making related activities, and business development services to MCCs and milk kiosks? iii Which of the above technologies transferred as part of the RDCP II program resulted in high adoption and high return on investments for a farmer? What should be the top three technologies to promote in the next phases? iv What are the major constraints hindering achievement of the main project goals? What are the major changes in the project environment that affected implementation or achievement of intended results? Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 9 2. To what extent have new management approaches (staffing structures) used to implement development of financial products by INSPIRE, increased investment, improvement of quality milk/Seal of Quality contributed to achievement of RDCP II results? 3. Based on return on investment at MCC level and service providers, what would be the most cost￾effective management approach to implement scaling up technology? 4. Based on RDCP II approach to reach out to high number of rural and vulnerable farmers through MCCs and service providers through new technologies, to what extent is the project reaching the intended population and the targeted number of beneficiaries in terms of knowledge and skills (adoption of technology), and the targeted number of beneficiaries in terms of diet behavior change as a result of the milk consumption campaign? 5. What opportunities for new partnerships (not yet explored) are available to scale improved practices in the ZOI? 6. What has been the influence at the institutional level of the RDCP II Seal of Quality program in terms of increased milk quality and compliance to dairy products (including raw and pasteurized milk, yogurt, and cheese) standards, on GOR agencies (Rwanda Agriculture Livestock Inspection Services, Kigali City, Rwanda Bureau of Standards, relevant districts, if necessary, etc.) and the dairy industry? 7. Considering RDCP II approach for the different components and activities, what are the ones that have potential to be more sustainable after RDCP II closes and what would be the recommended change to foster sustainability? 8. Using the WEAI framework, did the RDCPII approach to gender equality and equity ensure balanced involvement of women and men and particularly address constraints faced by women in the industry? Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 10 EVALUATION METHODS AND LIMITATIONS The team leader assigned responsibility for data collection and analysis to team members in accordance with their individual skill sets, i.e. the dairy industry and livestock development specialists interviewed and interacted with dairy farmers, processors, and service providers, while the financial services specialist visited banks and financial institutions. As a conceptual framework for the evaluation, the team loosely adopted the four pillars that LOL uses to organize management of the RDCP II, i.e. Milk Production, Milk Quality, Policy and Enabling Environment, and End Market (Nutrition), because working through those pillars, or implementation areas, required addressing or touching upon all aspects of the dairy value chain and, therefore, the areas in which the RDCP II is engaged. A full Getting to Answers Table is included in Appendix C. DATA COLLECTION METHODS The evaluation employed the following data collection methods: Secondary Research Document review Soon after being awarded the contract, the dTS team conducted a literature review focused on the performance of the Rwandan Dairy Sector. Documents reviewed included previous evaluations published by USAID and implementing partners, annual and quarterly reports provided by Land O’ Lakes, project design documents, and economic analyses from international organizations such as the World Bank. Appendix D provides a complete list of documents used in this evaluation. Primary Research The team’s fieldwork utilized both quantitative and qualitative methods in order to triangulate results. The team completed household surveys along with questionnaires to quantitatively investigate the impact of the RDCP II on individuals and institutions. This data was supplemented with qualitative investigations. Surveys and Questionnaires Surveys3 were conducted at 132 households (27 women, 105 men); and questionnaires were completed at seven (7) financial institutions. In total, 90 participants and 42 non-participants completed surveys in the areas around MCCs visited by the team. Focus Group Discussions and Key Informant Interviews Focus Group Discussions (FGDs) were organized at fourteen (14) Milk Collection Centers (MCCs) and Key Informant Interviews (KIIs) were held with eighteen (18) Model Farmers and eleven (11) Service Providers (SPs) in the Rwandan dairy value chain, and with government and private sector actors in the field and in Kigali. 3 The sampling strategy was based upon a convenience sample of households located near MCCs visited by the team. There is a possibility of selection bias because both participants and non-participants were given the opportunity to take part in the program leading to the possibility that the two groups have some unmeasured difference that may have impacted the results (e,g, the MCCs have a membership fee so participants may be more affluent). Despite this limitation, the results provide a useful glimpse into the impact of the RDCP II program and are illustrative of the program’s actual effect. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 11 Field Travel A considerable portion of the Zone of Influence (ZOI) was visited by the evaluation team, including travel to all (4) milk sheds – Eastern, North/West, Southern, and Kigali - and ten (10) of the seventeen (17) districts in which the RDCP II is operating. Field travel occurred during the periods March 4, March 5-7, March 8-11, and March 12, a total of nine (9) days; a copy of the field travel schedule is attached in Appendix E. DATA ANALYSIS METHODS The evaluation employed the following data analysis methods: Descriptive Statistics Numerical data was analyzed using measures of central tendency and are reported based on percentages throughout the paper. The evaluation team took extra care to disaggregate gender where data were person￾focused. Content Analysis Recurring themes were identified through the KIIs and FGDs which lead to the general conclusions reported throughout the evaluation. The evaluation team provided special focus to investigating issues related to access for women where applicable. DATA WAREHOUSING STATEMENT All original data was transmitted in person to the USAID/Rwanda Mission by a designated carrier. Digital copies of the results are available from dTS and will be uploaded into the Development Data Library (http://www.usaid.gov/data) TEAM COMPOSITION The evaluation team was comprised of six development professionals led by Mr. Michael Viola (team lead). He was supported by 3 Rwandan nationals (Ms. Jeanne Francois Umuzigambeho – Gender Specialist; Mr. Ildephonse Musafiri – Economist; and Mr. Jean de Dieu Kampayana – Financial Services Expert) and 2 expatriates (Dr. Sahr Lebbie – Technical Expert and Ms. Paula Higgins – Technical Expert) skilled in evaluation, livestock and dairy value chain and industry development, finance and investment analysis, economics, and gender. Gregory Norfleet and Rada Lankina provided analytical and methodological support from the dTS home office. STRENGTHS AND LIMITATIONS OF THE EVALUATION Strengths The team was an international group with varied expertise that provided a number of different perspectives on how to approach the evaluation. The team was able to investigate the efficacy of the program through technical, financial and gendered lenses. Limitations The contract for this evaluation was awarded significantly later than initially expected, which led to a compressed timeline for the study. The evaluation team began work immediately and were in the field three weeks after the contract was awarded and were only able to spend twenty-seven days in the field. Although the team performed admirably under the circumstances, there is a potential that more time in the field would have yielded a richer analysis. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 12 The major consequence of the shortened timeline (along with the large number of evaluation questions) was that the sample for the survey cannot be considered statistically representative of the entire population of beneficiaries. The total population of beneficiaries has been estimated at 15,000 people; in order to generate a statistically representative sample with a confidence interval of 5% or less the team would have had to interview a total of 375 participants and 375 non-participants; in total the team was able to distribute 92 surveys to participants and 40 to non-participants. As a result, the results for this section should be viewed as suggestive rather than authoritative. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 13 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS QUESTION 1: TO WHAT EXTENT HAS THE PROJECT CONTRIBUTED TO IMPROVED STANDARDS AND QUALITY OF RWANDAN DAIRY PRODUCTS IN THE ZONE OF INFLUENCE (ZOI) AND THE EXPANSION OF INVESTMENTS IN DAIRY PROCESSING AND MARKETING IN THE ZOI? FINDINGS Based upon the evaluation team’s review of relevant documents and confirmed through interviews, it appears that RDCP II has facilitated a number of positive outcomes related to the implementation of standards in the dairy sector within Rwanda. First, technologies have been transferred to farmers and are being adopted into practice on a wide scale which has drastically impacted the availability and quality of raw milk. Further, the team has found through interviews that the level of awareness by consumers of the benefits of milk consumption and the importance of the cleanliness of milk and dairy products is increasing and it is reasonable to believe this is directly attributable to the RDCP II-sponsored Seal of Quality and the Shisha Wumva education campaign. The project has also been responsible for the drafting of a National Dairy Strategy and Ministerial Instructions for the proper collection, handling, and transporting of raw milk and has worked collaboratively with a new private entity, the Rwanda National Dairy Platform (RNDP), has been created and is poised to provide support to and advocate for the dairy industry when the RDCP II comes to an end in January, 2017. Improvement in Quality of Raw Milk The quality of raw milk produced in the ZOI has significantly improved as measured by the amount or volume of poor quality raw milk rejected by MCCs and processors. At the MCCs visited by the evaluation team, the amounts of raw milk rejected prior to RDCP II interventions were found to be significantly greater than the amounts rejected after RDCP II-supported technology transfer, training, and grants. Records at thirteen (13) of the fourteen (14) MCCs visited indicated that the amount of poor quality milk rejected dropped from an average of 200 liters per week before the RDCP II to 41 liters per week after RDCP II investment in milk-testing equipment; an improvement of 79.5%. As shown in Table 1, this decrease in rejected milk has been accompanied by a nearly 50% gain in the production of milk at the average MCC. Period Liters of Milk Rejected /week Liters of Milk Produced / Week Baseline 35 6,116 Current Survey 7 9,124 Percentage Change -80% 49% Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 14 Despite these positive results with the amount of milk rejected by MCCs diminished and the quality of the milk definitely higher, much of the milk produced still does not meet COMESA standards (with the average MCC rejecting 364 liters of milk a year). In addition, the extremely low rejection rates found at certain MCCs – one MCC reported zero (0) rejection – raise a concern in that producers of low quality milk may be bypassing the MCCs and selling their milk on the alternative market. Both the National Dairy Strategy produced by MINAGRI and the Deep Dive program have established that demand exists for dairy products throughout Rwanda, Burundi, and the Democratic Republic of Congo and that currently this demand is being met by low-quality producers on alternative markets. Expansion of investment in dairy processing and marketing There are some indications that investment may increase. Of the seven (7) financial institutions visited, four (4) had entered into Memorandums of Understanding (MOUs) process with the RDCP II with the aim of lending to dairy chain actors. Of those, two (2), Urwego Opportunity Bank (UOB) and Kenya Commercial Bank (KCB), while their lending to dairy began only in late 2014, had already developed portfolios, within six months, totaling RWF 83,700,000 (US$119,571), with both banks expecting to quadruple their portfolios by the end of 2015. So far, the loans’ purposes have been for cattle acquisition, and mobile coolers. The maximum dairy loan period is 4 years, with an interest rate between 15%-20%, and monthly repayment after a grace period based on the cash-flow. The average loan size is Rwfs 400,000 (US$570). The general impression gained from discussions with commercial bank officials and loan officers is that the dairy industry is currently not considered an attractive investment opportunity; not because of the investment cost analysis, but rather a poor perception of the business and individual borrowers. One reason for that is a past history of non-performance of construction and equipment loans in the amount of RWF 210,000,000 (US$300,000) made to MCCs by the BRD (Rwanda Development Bank) in 2012. Those loans have been restructured and repayment should begin within the next three months, but their prior poor performance, due to what the banks consider poor management by the MCCs, has created an element of caution, if not fear, in the investment community. The other constraints as stated by financial institutions include but not limited to potential leakage of the contract dairy market and poor impressions of the financial expertise of within MCCs. What are the major contributing factors and constraints to achieving these results? Contributing Factors for Success Based upon the results from the team’s mixed methods approach, the RDCP II program has done a praise￾worthy job of involving multiple stakeholders throughout the agricultural value chain including their outreach activities specific to farmers, service providers and MCCs. The team has identified willing partners within relevant agencies of MINAGRI and the GOR which has created a collaborative relationship that has benefited all parties. The Seal of Quality (SOQ) set of best practices and standards and the training and capacity-building in best practices of handling raw milk carried out by RDCP II have led to an impressive level of awareness of the program throughout the country and seems to have been effective in getting ideas into practice. Constraints The biggest constraint to the achievement of RDCP II goals is the obvious one: the scheduled termination of the program in January 2017. The amount of time from the present – March 2015 – until the planned closeout date, in approximately twenty-two (22) months, will probably be inadequate to accomplish some Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 15 major program objectives and the stated program goal - competitiveness of Rwandan dairy products in regional markets – at least, insofar as the team understand that goals, and given the relatively heavy reliance on investment upon which the goal is posited. Other constraints to the achievement of the program are objective: • Although RDCP has reported significant progress of the past three years, the business acumen of several participants points to a need for targeted training. • In the case of cooperatives: the absence of services, (e.g. training, equipment, seed, transport, and other services) that could attract more members and generate income for the cooperative at the same time improving the capacity of the members; • Lack of working capital at the MCCs to be used to facilitate (i.e. speed up) payments to farmers and producers of raw milk. Based on observations from the team and a review of financial documents, the amount of cash generated by the MCCs is sizeable, but it appears it could be managed more efficiently and leveraged. • Lack of reinvestment in the MCCs by the members due to the absence of monthly or annual membership subscriptions4 and other mechanisms designed to create cash reserves for the improvement of MCC infrastructure and the enhancement of services provided to the members. • KIIs with farmers and MCCs also identified a lack of incentive to produce high-quality milk absent the enforcement of hygienic production and handling practices and standards, the upshot being that a producer of low-quality milk can bypass or circumvent the system by selling his/her milk in the alternative market at the same price, or possibly more, than that received by a producer of high-quality raw milk. This has been observed by the team as raw milk and dairy products have been transported over the border to the Democratic Republic of Congo and Burundi. Corrective measures Based upon the KIIs and FGDs, stakeholders believe that it is of the upmost importance to strengthen the MCCs in terms of their overall business and marketing acumen. Stakeholders identified a number of potential measures including: • Installation of professional managers trained in the workings of the dairy industry paid for by the MCCs. In this scenario, a number of actors would act collectively to hire an educated individual to oversee several MCCs; allowing them to offer a competitive salary to attract quality talent. • Continued expansion of mechanisms aimed at the generation of reserves or working capital at the MCCs. • Building upon ongoing efforts through AgPRO in the training of MCC managers in business planning and the development of a business and annual operational plans at each MCC. • A system of enforcement of the forthcoming Ministerial Instructions on the collection, transporting and handling of milk; and, development of incentives through which producers of high quality milk are rewarded and producers of low-quality milk punished. 4 Although the team acknowledged that the presence of a membership fee may have biased our sample, incorporating a monthly or annual fee based upon how often members used services provided by the MCCs would potentially address inequal access to the services. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 16 To what extent are project beneficiaries (farmers, traders, milk collection centers) experiencing expected changes in their skills, knowledge, attitudes, behaviors, yield, income, etc.? Changes in skills, knowledge, yield, and income Findings based on amounts reported by farmers and farm records corroborate RDCP II reporting of increases in milk yield and raw milk production of upwards of 30%, increased income of more than 40% by participating farmers, and a significant improvement in the quality of milk (see above), all directly attributable to the technology transfer, training, and capacity building provided by the RDCP II. Period Milk Rejection at MCCs (L) Milk Production (L) Milk Sales Income (US$) Production Cost (US$) Net Dairy Income(US$) Baseline 35 6,116 1,676.73 232.95 1,443.78 Current Survey 7 9,124 3,008.61 350.83 2,657.77 % Change (=/-) -80 49 79 50 84 The team also found that the Seal of Quality has had a major influence on the awareness of the standards from producers, processors, government officials, Service Providers (SPs), and other stakeholders. Interviews and discussions at all levels in the field indicated that fully 100% of farmers and local authorities were aware of the SOQ. Changes in behavior Interviews with eighteen (18) Model Farmers indicate they are sharing their new knowledge, technologies, and skills with other farmers; the Model Farmers have trained 1,837 farmers over the past 1½ years, of which 832 were male and 1,004 were female. One farmer told us that prior to working with the RDCP II, he “didn’t know that water was important for his cows and for milk production”; another was building a new, improved cattle shed based on RDCP II recommendations; others spoke about the importance of keeping the milk shed clean and wearing clean clothes while milking. The team heard many similar stories. Is the project on track to meet targets? The RDCP II has thirty (30) indicators: fifteen (15) Feed the Future (FtF), twelve (12) custom, i.e. Land O’Lakes indicators, and three (3) indicators for the measurement of gender equity and equality. Some indicators are more “important” than others in terms of measuring project achievement, but each indicator has a target. Each indicator adds a level of complexity in both data collection and analysis; and in this case data quality assurance is a big issue because of multiple actors are involved in both generating and collecting data. There are discussions currently underway regarding additional indicators targeting nutritional outcomes. The team expressed concern that nutritional indicators would require a large sample and significantly longer timeline in order to detect any effect in the population while also adding additional complexity to an already complicated program. Based on our analysis, the vast majority of currently tracked RDCP II targets, particularly those dealing with milk producers, MCCs, persons and organizations adopting technologies are achievable. That said, the a number of key informants including the COP for RDCP II have questioned the achievability of the target of US$5,000,000 in FY 2015 in new private-sector investment in the dairy value chain. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 17 To what extent have new Technologies contributed to Positive Outcomes? The team’s research and discussions with dairy farmers and MCCs indicates that production in the ZOI has steadily risen from 16 million liters in 2012 (value: US$4.78 million) to 25.2 million in 2013 (US$7.52 million) to 36.5 million liters in 2014 (US$10.89 million), a more than doubling of output in two years (average price of a liter of milk = RWF209.00/RWF700.00 = US$1.00). Return on Investment (ROI) ROI is, normally, a relatively simple economic calculation when dealing with discrete investments having specific values; but, that is not the case with the RDCP II and the wide array of technologies and services it offers. It was not possible to calculate the ROI for individual RDCP II activities. First, LOL does not make or account for expenditures in accordance with technologies or activities. Program expenditures are disbursed and accounted for, instead, in accordance with direct program interventions or sub-activities, e.g. training workshops, development and dissemination of behavior change educational materials; and, therefore, the true cost of each activity can’t be determined. Furthermore, many project investments (on business development services for example) have a long payback period (more than 10 years) and have not yet reached sufficient capacity to generate returns. This is especially true for the technologies transferred for the purpose of improving livestock health and hygiene and increasing the quantity of raw milk, e.g. artificial insemination (AI), mastitis control, feed and forage production, and related technologies. Even if the amount invested in the transfer of individual technologies were known, the technologies all come together in the cow and, along with other factors, including the cow’s handling and environment, result in the production of more milk. It is not possible, therefore, to single out one particular technology, assign it a value or cost, and calculate its ROI. That said, the team understands USAID/Rwanda’s desire to learn the value of individual program activities in order to make informed decisions for program implementation in the future. Accordingly, based on the financial records of certain cooperatives and information the team were able to obtain on the amounts of production and sale of raw milk, the team did attempt to make limited ROI calculations and found the following: Seal of Quality and Milk Testing Kits The Seal of Quality and the investment in milk testing kits seems to have represent a solid deal for MCCs. Records at thirteen (13) of the fourteen (14) MCCs visited showed that that the amount of spoiled milk dropped on average from 200 liters per week before RDCP II to 41 liters per week after the introduction of RDCP II provided milk-testing equipment. This represents an estimated weekly savings of 159 liters per MCC. Although, these numbers are not definitive as some MCCs do not have detailed financial information the team conducted a limited ROI analysis for two of the MCCs where data was available. Figure 1: Everything comes together in the cow Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 18 MCC Total value of invest in Milk testing kit (FRW) Depreciation period Value used in 2014 FRW Net Profit 2014 ROI (Cost benefit ratio) Abahuzamugambi Ryabega 554,894 8 years 69,362 519,246 7.49 Kiziguro Dairy Cooperative 1,051,240 8 years 131,405 78,332 0.60 Based upon this analysis, the benefits do not seem to be equally distributed, with the Abahuzamugambi Ryabega showing an incredible return-on-investment and the Kiziguro Dairy Cooperative actually showing a loss. This could potentially be explained by the large, upfront investment into the technology by Kiziguro which significantly increases their depreciation costs. The possibility also exists that some unmeasured environmental factor has impacted the quality of the milk created at Kiziguro compared to Abahuzamugambi which would be supported by the proven industry standards that show 70% of cattle performance is determined by their feed, housing and vet care. Attempts to calculate ROI on cheese making, financial product development, and business development services (BDS) proved almost impossible due to the reasons cited above and because the MCCs and other relevant institutions either do not maintain records that show the cost of the activity or, in the case of some MCCs, records are not precise or sufficiently detailed to carry out a robust economic analysis. Financial Impact of Technological adoption A simple comparison of the average income for farmers who have adopted these technologies and those who have not is shown in the table below5. The adopters of the technologies earned higher incomes overall except in regard to farm made concentrates where farmers experienced loses. The failure of homemade concentrates to improve cattle performance is most likely due to two main factors, namely, the high cost of ingredients and poor feed formulation that results in poor performance of the animals and therefore poor returns on investment. Type of technologies Average income (FRW) Difference (Adopters minus Non Adopters) Non-Adopters Adopters Milk quality improvement 434,630 491,217 56,858 AI services 305,009 611,299 306,290 Deworming 476,396 482,460 6,064 Mastitis control 457,702 499,336 41,634 Vaccination 406,660 498,810 92,150 Farm made concentrates 556,263 449,518 -106,745 Which of the identified technologies resulted in high adoption and high ROI for farmers? Which three should be promoted in future phases? 5 It should be stressed that, due to potential sampling bias and the lack of a baseline, we cannot exclude the possibility that these 2 groups varied before the intervention took place. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 19 Based on a variety of methodologies including KIIs, FGDs and financial analysis, as well as a broad understanding of the word “technology” to include novel activities undertaken as part of the overall study, the team believes that the top three RDCP II activities, in terms of their return on investment, application, and effectiveness, are the Seal of Quality, the transfer of on-farm technologies broadly, and business development services. The Seal of Quality The SOQ can be viewed in two distinct ways: a) in the traditional sense, as a discernible mark on a product or product package indicating superior quality; and, b) as it currently exists in Rwanda: as a paradigm, the aim of which is the increased awareness of dairy value chain actors of the need for hygienic procedures in the handling of raw milk, and by consumers of the importance of the cleanliness of the milk and dairy products they purchase and consume. There is no doubt that the Seal of Quality is the RDCP II activity most responsible for heightened awareness of the cleanliness of milk and dairy products throughout the country, and that the Seal of Quality must be pursued with an aim to making it a true seal that informs consumers of a superior product. The transfer of on-farm technologies Technologies including AI, mastitis control which is being implemented in collaboration with University of Rwanda and UC Davis, and other best practices in animal hygiene, milk collection and handling have been widely adopted by program beneficiaries and they many offer a substantial return on investment. Farmers are very concerned with death loss due to disease in their improved cattle and are therefore willing to invest in health care which accounts for the high rate of adoption of technologies directly related to health and veterinary care. 36% 36% 73% 53% 73% 61% 37% 78% 82% 81% 57% 58% 81% 83% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Adoption frequency Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 20 Though AI (the ROI of which has discussed earlier) appears to have a low adoption rate when compared to the other technological packages (see Fig 1), our interviews with KIIs show the low adoption rate is largely due to an ineffective delivery system constrained largely by the inconsistent availability of the essential inputs of quality semen and heat synchronizing hormones, which are currently under the control of government, not an unwillingness of farmers to adopt this technology. Proven industry standards show that cattle performance is based 30% on genetics and 70% on environment (feed, housing, vet care). Cattle that are genetically inferior will not perform well regardless of their environment and cattle with superior genetics will not perform at their genetic potential in a poor quality environment. Both elements must be present for cattle to excel in performance – good genetics and a quality environment. It must be noted the sustainability of the dairy industry in terms of improved milk production is highly dependent on the wide uptake of this technology. Business development services (BDS) As discussed elsewhere throughout the study, the level of business acumen is a serious inhibiting factor for the future success of RDCP II and related initiatives. Numerous interviews conducted by members of the evaluation team among famers and other individuals involved in the production or transportation of milk products all pointed to a fundamental lack of understanding of foundational business practices. Considering the success the program has already achieved, it seems reasonable to believe that basic managerial training could significantly increase the impact. What are the major changes in the project environment that affected implementation or achievement of intended results? We did not discern changes in the project environment that affected RDCP II implementation or achievement. That said, there are areas that bear watching: a) the relatively high cost of dairy products for the average Rwandan consumer, particularly the packaging and prohibition of the use of cheap packaging materials made of plastic, and the concomitant need to bring down the costs of production and transport; b) the need for and potential shortage of trained veterinarians and other livestock experts; c) the availability of sufficient processing capacity to accommodate increased local demand for processed milk and dairy products; and, the increased risk of cattle disease given the increasing size of the herd. CONCLUSIONS • From a number of standpoints, the RDCP II is performing at a high level of effectiveness, with planned targets being reached and, in many instances, exceeded • Progress has been made in improving the quality of raw milk being produced under RDCP II. However, if the assumption that producers of low quality milk may be selling their products on alternative markets is correct, this practice will require policing through the forthcoming Ministerial Instructions, if the high quality of Rwandan milk and dairy products is to be upheld. • Investment in the dairy value chain is occurring, but at a rate and volume less than what was perhaps envisioned or desired. MCCs have reinvested in their operations but they are relatively few in number and the amount of investment is relatively low. This has implications for gender, because, traditionally, men have more access to banks and formal financial institutions than women. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 21 • RDCP II seems to have a positive effect on skills of locals while influencing both behavior and attitude. • Although the program is generally successful, improvements in business and marketing acumen throughout the entire agricultural value chain could increase the impact of the program. • There is a regional demand for Rwandan Dairy products • In at least some districts, the milk testing kits distributed through the Seal of Quality Initiative generate a high ROI. That said investment costs need to be controlled as ROI is not a necessarily linear measurement. RECOMMENDATIONS 1. For the remaining life of the project, the RDCP II should continue implementing activities directed at increasing the production and improving the quality of raw milk, but the focus of the program should be shifted to: a. Increasing local demand for Rwandan milk and dairy products and their marketing in neighboring countries b. The enhancement of the dairy processing component of the value chain in order to accommodate increased raw milk production and the demand created by increased local consumption and export marketing of Rwandan dairy products 2. Consideration should be given to a follow-on dairy value chain activity that will seek to preserve the achievements of the RDCP II and address a new phase of dairy value chain programming by: a. Strengthening the management and marketing capability of the MCCs b. Assuring the readiness and capacity of the RNDP to assume responsibility and take on the challenges associated with supporting and advocating for the Rwandan dairy industry c. Promoting local consumption of Rwandan milk and dairy products and their export to neighboring countries d. Further strengthening of the capacity to process Rwandan milk and dairy products. 3. Land O’ Lakes should investigate the relationship between higher quality offerings vis a vis their increased availability and impact on demand6. 4. Land of Lakes and USAID should reduce, where possible, the number of indicators associated with this study. a. Additional indicators associated with a nutrition objective and the USAID Nutrition policy should not be inserted into the program at this time. The possibility exists for adding a nutritional dimension to future follow-on studies. b. If additional indicators are added, care should be exercised so that LOL managers and staff do not change their approach – the value chain and dairy business should remain their focus. 5. The $5,000,000 target in FY 2015 in new private-sector investment in the dairy value chain be revised in light of the current climate for investment. 6 See Statement of Differences Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 22 6. Building on findings in the Rwanda Dairy Strategy and Deep Dive Investment Analysis, a formal and concerted effort should be undertaken to explore the actual potential for the capitalizing on the demand for Rwandan dairy products in neighboring countries. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 23 QUESTION 2: TO WHAT EXTENT HAVE NEW MANAGEMENT APPROACHES (STAFFING STRUCTURES) USED TO IMPLEMENT DEVELOPMENT OF FINANCIAL PRODUCTS BY INSPIRE, INCREASED INVESTMENT, IMPROVEMENT OF QUALITY MILK/SEAL OF QUALITY CONTRIBUTED TO ACHIEVEMENT OF RDCP II RESULTS? FINDINGS RDCPII hired INSPIRED to research the dairy value chain in Rwanda’s four main milk sheds and to recommend potential feasible financial products that would underpin the active engagement of Rwanda’s financial institutions in this sub-sector. Financial institutions revealed that the INSPIRE timeframe was too short and much too ambitious to leave space for contextualization and adoption of the model. This short time frame resulted in poor and reluctant participation of the initial targeted financial institutions. The Agri-Relationship Manager from the Banque Populaire du Rwanda (BPR Ltd) said their MoU with INSPIRE was only recently drafted and they are still waiting for INSPIRE to further develop the agreement. UOB and KCB’s heads of agriculture finance identified the lack of a participatory approach with INSPIRE as a barrier to forming a cooperative relationship. Fortunately, UOB Ltd and KCB ltd have been able to develop the dairy financing product with direct partnership of Land O’ Lakes (though this activity is not really linked to INSPIRE work) backed by guarantee loans facilities from other sources. Contribution of New Management Structures to Investment and Improvement in Milk Quality We discussed “new management approaches” with LOL who note that they have made minor adjustments to program staffing, but nothing major in that regard. Contribution of Seal of Quality to program achievement Through our financial analysis as well as KIIs with relevant stakeholders, we found the SOQ, including mastitis control and the dissemination of “best practices” in the handling and transport of raw milk, to be the single most important program activity or component in terms of the improvement of the quality of the raw milk being produced. Does the project have the right mix/balance of staff/funding given the activity priorities? What would be the recommended management structure and staff needs for the next two and half years of implementation? The team looked at the LOL management structure and staffing from two standpoints: a) in terms of their overall approach to managing the different aspects of the program; and, b) in terms of the “end-game”, i.e. the scheduled close of the program in January 2017. In more than one interview with key individuals inside of MCCs and within financial institutions, it was mentioned that the LOL approach, i.e. socially concerned but with a business or profit attitude and frame of mind was an effective approach. The team discussed staffing and approach with LOL and learned that, in preparation for the close of the program, they are gradually moving from a “push” to “facilitation” mode. Accordingly, they are focusing on strengthening RDCP II partners (i.e. GOR, SPs, dairy processors, and the RNDP) and attempting to heighten the awareness and desire of program partners to preserve and carry forward the work of the program after it ends. One example is that of the Milk Shed Working Groups, in the service of which the RDCP Deputy Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 24 Chief of Party (DCoP) previously spent a considerable amount of time organizing, and encouraging the membership to meet and discuss issues of concern. According to interviews with individuals from LOL, those Working Groups are becoming increasingly aware, analytical, and active and are now coming together on their own volition on a regular basis to discuss issues and solve problems. This localized “facilitation” approach is in line with the USAID Forward Initiative. One area of concern, however, deals with the overall complexity of the program and the staffing being currently used to ensure data quality assurance. As previously mentioned, RDCP II has a large number of indicators and evaluation data is collected by a wide variety of actors which leads to concerns over the integrity of the data. LOL has dedicated an M&E specialist to oversee data collection, however, currently the position also has a number of other responsibilities in the office. CONCLUSIONS • The present staffing structure is adequate to the management and monitoring of the program in its present configuration and operational modality. Staff changes will depend on the program focus over the next twenty-two (22) months. • The complexity of the project necessitates a dedicated individual or team specifically concerned with maintaining data integrity. RECOMMENDATIONS 1. LOL should limit the scope of responsibility for the full-time M&E specialist to purely those activities associated with monitoring and evaluation, especially dealing with data quality assurance protocols. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 25 QUESTION 3: BASED ON RETURN ON INVESTMENT AT MCC LEVEL AND SERVICE PROVIDERS, WHAT WOULD BE THE MOST COST-EFFECTIVE MANAGEMENT APPROACH TO IMPLEMENT SCALING UP TECHNOLOGY? FINDINGS Implementation approach to scaling-up technology Sixty-two percent of farmers participating in RDCP II interviewed (62%) are using AI services and “super￾semen,” which based on our analysis is associated with an increase in income of 306,290 RWF per year. Mastitis control is a major factor contributing to the improved quality of the raw milk produced; and, the RDCP II collaboration with the University of Rwanda and the University of California/Davis (UC/Davis) is showing great promise in terms of its outreach and effectiveness. Due to the shortage of available land and zero grazing policies in some places, MCCs will be required to provide high quality feed to farmers7; however, farmers need to be provided technical assistance in the production of high-quality feed and forage as well as the skills necessary to produce their own silage and hay and instruction in the use of such crop residues as feed. The team notes, however, that AI is dependent on the availability of affordable high-quality semen and hormones, the supply of which is currently controlled by the GOR. There is a considerable body of evidence that semen and hormones are regularly unavailable, hindering the ability of farmers to obtain AI services and constraining the use of this technology, let alone its scaling-up. CONCLUSIONS • The technologies that have the greatest potential benefit to increase and sustain gains from RDCP II if scaled-up are Artificial Insemination (AI), mastitis control, and technologies generally associated with milk quality improvement. • The uncertain supply of high-quality semen and hormones could potentially limit the ability of RDCP II to effectively scale up AI. RECOMMENDATIONS 1. The reported regular and ongoing unavailability of semen and hormones and privatization of that service should be taken up with the GOR by Land O Lakes and other key players in the industry including RDFA. 2. Artificial Insemination (AI), mastitis control, and technologies generally associated with milk quality improvement should be scaled up by implementing partner organizations. 7 See Statement of Differences Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 26 QUESTION 4: BASED ON RDCP II APPROACH TO REACH OUT TO HIGH NUMBER OF RURAL AND VULNERABLE FARMERS THROUGH MCCS AND SERVICE PROVIDERS THROUGH NEW TECHNOLOGIES, TO WHAT EXTENT IS THE PROJECT REACHING THE INTENDED POPULATION AND THE TARGETED NUMBER OF BENEFICIARIES IN TERMS OF KNOWLEDGE AND SKILLS (ADOPTION OF TECHNOLOGY), AND THE TARGETED NUMBER OF BENEFICIARIES IN TERMS OF DIET BEHAVIOR CHANGE AS A RESULT OF THE MILK CONSUMPTION CAMPAIGN? FINDINGS Adoption of technology The increase in raw milk production and the improvement in its quality strongly suggests that the technologies offered by RDCP II are being adopted by program beneficiaries. This suggestion is backed up by our survey of farmers, in which between 39.1 (n=36) and 89.1% (n=82) of participants reported using each included technology with each technology recording an average of 69.3% (n=64) of participants. Diet Behavior Change Based on interviews with industry actors and GOR officials, there is an across-the-board heightening of awareness of the importance of cleanliness and quality in milk produced and consumed due to the Seal of Quality and the Shisha Wumva campaigns. The Shisha Wumva campaign seems to be everywhere – on billboards, radio, TV, and in the form of a calendar on the desk of most of the people interviewed; and, is reportedly reaching large numbers of people. Its effectiveness in terms of diet behavior change, however, is relatively unmeasured. The team included a question on milk consumption in the household survey and found a correlation in households that had heard Shisha Wumva messages and increased milk consumption. Those households consumed an average of 35% more milk than those who hadn’t heard the messages. The continued patronization by consumers of rural milk kiosks and other informal channels indicates that much work remains to be done in terms of educating consumers on the importance of purchasing high-quality, unadulterated milk and dairy products. The team was surprised to find that dairy farmers in the immediate vicinity of some MCCs had no knowledge of the RDCP II, so the possibility to reach a larger beneficiary population within the ZOI may be possible. Technology % Adopters (N=92) Silage 39.1 Hay 41.3 Crop Residues 78.3 Legumes hay 64.1 Farm made concentrate 75.0 Purchased concentrates 69.6 Molasses 42.4 Spraying against ticks 83.7 Deworming 87.0 Vaccination 85.9 Mastitis control 65.2 AI Services (gutera intanga) 63.0 Milk hygiene 85.9 Milk quality improvement 89.1 Average 69.3 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 27 CONCLUSIONS • The technologies, overwhelmingly, are reaching and being used by the target population of the program. • The Shisha Wumva outreach campaign has reached an impressive section of the population, though its impact is still unclear because it has yet to be evaluated. • It was not possible to investigate why non-participants were not aware of the program being implemented in their community. RECOMMENDATIONS 1. The Shisha Wumva milk consumption education outreach campaign should build upon success of reaching a large number of potential participants. Implementing partners would be wise to incorporate behavioral change communications and strategies into the program. 2. Future evaluators should further investigate the reasons for a lack of awareness with the ZOI of several MCCs participating in RDCPII. This should inform future outreach strategies to identify potentially hard to reach populations and to inform outreach strategies. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 28 QUESTION 5: WHAT OPPORTUNITIES FOR NEW PARTNERSHIPS (NOT YET EXPLORED) ARE AVAILABLE TO SCALE IMPROVED PRACTICES IN THE ZOI? FINDINGS Land O’Lakes The evaluation team did not uncover opportunities for new partnerships, mainly due to time constraints for data collection and analysis. That said, Land O’Lakes is discussing a collaborative effort with USAID-funded Global Communities (GC) through which the latter would use informational materials on best practices in milk handling produced by RDCP II, specifically the pamphlet “Ubuziranenge Bw’amata,” which outlines best practices in comic book form. GC is working with vulnerable dairy farmers in four (4) districts, two (2) of which overlap with the RDCP II ZOI, and two (2) that do not, implementing literacy, community health, and kitchen garden projects (nutrition). A partnership with GC, whereby LOL supplies the educational materials on milk collection best practices and GC disseminates those materials would allow for the expanded outreach of the materials over a wider area to more and more vulnerable people at relatively low cost. One potentially problematic issue that came up in an interview with a member of the Rwandan Standards Board related to the project. The point was made that they are only consulted in “finished” dairy projects and were not involved in monitoring the inputs. LOL and USAID has made the point that this was an intentional delineation of responsibilities with numerous actors having different responsibilities Financial Institutions In interviews with financial institutions, the point was made that channeling funds through local financial institutions could potentially create relationships between formal financial institutions and program beneficiaries. Furthermore, the increased contact between the aforementioned entities might serve to generate other business and would relieve RDCP II of that responsibility. A model for this sort of relationship may be a “check off” system, in which an MCC would house a store which would allow members to take agricultural inputs on credit and to repay their debts using the profits after the sale of their goods. CONCLUSIONS • Land O’Lakes has done an incredible job forming working relationships with actors throughout the agricultural value chain and should be commended. • The possibility for a relationship between Land O’ Lakes and Global Communities could expand the reach of the program to marginalized populations in the ZOI. • While the current responsibilities of the RSB were created intentionally, there may be a need to address their expected role in the MI directly. • The possibility of forming relationships between institutions and program beneficiaries shows promise and support on the part of financial institutions. RECOMMENDATIONS 1. The GOR along with LOL should investigate ways to strengthen collaboration between various actors in monitoring compliance with the Ministerial Instructions. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 29 2. The channeling of RDCP II funds through financial institutions should be explored as a means to create relationships between formal financial institutions and program beneficiaries. A check off system should be considered in this regard. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 30 QUESTION 6: WHAT HAS BEEN THE INFLUENCE AT THE INSTITUTIONAL LEVEL OF THE RDCP II SEAL OF QUALITY PROGRAM IN TERMS OF INCREASED MILK QUALITY AND COMPLIANCE TO DAIRY PRODUCTS (INCLUDING RAW AND PASTEURIZED MILK, YOGURT, AND CHEESE) STANDARDS, ON GOR AGENCIES (RWANDA AGRICULTURE LIVESTOCK INSPECTION SERVICES, KIGALI CITY, RWANDA BUREAU OF STANDARDS, RELEVANT DISTRICTS, IF NECESSARY, ETC.) AND THE DAIRY INDUSTRY? FINDINGS At the institutional level, the RDCP II can legitimately claim a large degree of responsibility for initiating and implementing several key milestones or markers in the strengthening of the Rwandan dairy value chain and industry. The Seal of Quality program component itself, the development of the National Dairy Strategy (NDS), the creation with the Private Sector Federation (PSF) and nurturing of the Rwanda National Dairy Platform (RNDP), the Shisha Wumva milk consumption education and behavioral change program component and campaign, and, the forthcoming Ministerial Instructions on the Preservation, Collection, Transportation, and Selling of Milk8, approved by MINAGRI and, at the time this report was being drafted, awaiting signature in the office of the Prime Minister represent extraordinary accomplishments. The Seal of Quality is now universally recognized by all actors interviewed throughout the Rwandan dairy value chain and industry as the driver and impetus for the safety and cleanliness of Rwandan milk and dairy products and, therefore, critical to the export marketing of high-quality Rwandan dairy products and their increased local consumption. The Seal of Quality, along with the transfer of on-farm technology to dairy farmers, again, more than other program components, have had the most profound effect on the value change and spurred change in the handling and cleanliness of raw milk. This achievement is not readily quantifiable, but becomes apparent when discussing the RDCP II with actors throughout the dairy value chain and industry. In addition, the evaluation team interviewed scores of farmers and members and staff of MCCs outside Kigali and to individuals from both the private sector and the government of Rwanda (GOR) in Kigali. A full list of contacted parties is available in Appendix D. Within the GOR, the SOQ was consistently a significant part of the conversation in terms of recognition of its importance in not only assuring the cleanliness of milk and dairy products for local consumption, but as an indicator of the high quality of Rwandan dairy products and the significance of high quality for export marketing. It is the SOQ that differentiates Rwandan products from those from other countries; and, the importance of the SOQ has not been lost on the GOR. CONCLUSIONS At the institutional level, both RDCP II and the SOQ have been well received by the GOR and other stakeholders. The GOR has shown itself to be an enthusiastic partner in the implementation and institutionalization of both the program and the SOQ. 8 The Ministerial Instructions will serve as official guidance governing milk in Rwanda and will carry the full force of the GOR in terms of enforcement, including the establishment of an inspection system with violators being prohibited from selling or otherwise handling raw milk and dairy products Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 31 RECOMMENDATIONS 1.) LOL should continue to nurture the relationship between RDCP II and the GOR, which has significantly contributed to the overall effectiveness of the SOQ. 2.) The GOR must enforce the Ministerial Instructions on the Preservation, Collection, Transportation and Selling of Milk and establish a regulatory and inspection system to ensure the high quality of Rwandan milk and dairy products. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 32 QUESTION 7: CONSIDERING RDCP II APPROACH FOR THE DIFFERENT COMPONENTS AND ACTIVITIES, WHAT ARE THE ONES THAT HAVE POTENTIAL TO BE MORE SUSTAINABLE AFTER RDCP II CLOSES AND WHAT WOULD BE THE RECOMMENDED CHANGE TO FOSTER SUSTAINABILITY? FINDINGS As for continuation of the RDCP II approach, the MCC’s Training of Trainers (TOT) and Model Farmers activities appear to have the most promise as implementation modalities or vehicles once the program has ended. On the institutional level, the Private Sector Foundation (PSF) and LOL have formed the Rwanda National Dairy Platform (RNDP), which already has offices and equipment provided by a grant from the RDCP II, and for which there is currently ongoing a search for a permanent director. The RNDP is designed to bring together the key players in the Rwandan dairy value chain and industry and provide the advocacy and support necessary for the continued strengthening and advancement of the industry. LOL has worked collaboratively with PSF to create a 5 year strategic plan to handle the transition between RDCP II and RNDP, the roll out of which started last year. The team’s interviews with key actors in the dairy value chain uncovered a universal concern for the preservation of both the services and benefits provided by the RDCP II once the project comes to an end in January 2017. The team visited the Agira Gitereka MCC and found that although they had capacity for 7,500 liters of milk daily, they were only producing 400 liters. Model Farmers within the MCC claimed that, although the technological investments could be effective, farmers had lost trust in the MCC and were instead selling their milk on alternative markets. This points to a serious threat to sustainability of the gains of RDCP II; the entropy of the productive relationships that have been formed as part of the program thus far9. As discussed in previous sections, the team believes that business development services are integral to the continued (and potentially increased) success of RDCP II. Competent management would undoubtedly contribute to the sustainability of the program. CONCLUSIONS • Among the most important accomplishments of RDCP II is the facilitation of cooperative relationships throughout the agricultural value chain. • The transition between RDCP II and RNDP in January 2017 is a source of anxiety for some stakeholders in the agricultural value chain RECOMMENDATIONS 1. The relationships that have been formed through RDCP II should continue to be a point of emphasis for LOL. 2. LOL should continue efforts to improve business knowledge within MCCs. 9 LOL has made it clear that they believe that SPs MCCs, milk zones will all exists and continue to implement RDCP II practices after the program transitions into the RNDP. However, due to the observation of a limited number of currently underperforming or defunct MCCs and based upon interviews with stakeholders in the value chain, the team believes the concern bears mentioning. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 33 QUESTION 8: USING THE WEAI FRAMEWORK, DID THE RDCPII APPROACH TO GENDER EQUALITY AND EQUITY ENSURE BALANCED INVOLVEMENT OF WOMEN AND MEN AND PARTICULARLY ADDRESS CONSTRAINTS FACED BY WOMEN IN THE INDUSTRY? FINDINGS The Women’s Empowerment in Agriculture Index (WEAI) framework was developed as part of the Feed the Future Initiative and represents the first “comprehensive and standardized measure to directly capture women’s empowerment and inclusion in the agricultural sector.10” The baseline report of the framework was published last year and Rwanda scored a respectable .91 rating (out of 1.00), which was the highest score reported in Africa and the second highest overall. The study estimates that 70 percent of women in the agricultural sector in Rwanda have achieved adequate employment and 73 percent have achieved gender parity. The evaluation team was unable to fully utilize the WEAI framework by replicating the baseline specifically for the dairy sector analysis because such an investigation would require a much larger sample than was feasible to access under the short timeline of the study. WEAI has also not yet been integrated into the RDCP II framework, although the gender specialist for LOL has expressed interest in incorporating these indicators. The analysis therefore speaks to directly observable consequences of empowerment and inclusion that could be ascribed to RDCP II. Gender mainstreaming has been a major consideration throughout RDCP II and LOL has done an admirable job promoting female participation in the dairy sector, which in many societies is a male dominated field. Types of Work Available Common Constraints for Women in Dairy Jobs along the agricultural value chain are often gendered11. Women are frequently limited to work in what are deemed feminine tasks because of gendered divisions of labor such as the cleaning or food preparation, while men are given responsibility for tending the cattle and the transportation of goods to the market. These jobs not only vary in the amount compensated, but in many cases control over the handling of money. The Rwandan Dairy Sector Context Gender equality and equity is a right enshrined in the Rwanda constitution and the government has made significant efforts to improve the plight of women in agriculture. According to MINAGRI’s Agricultural Gender Strategy, up to 86% of work completed in the agricultural sector as a whole is completed by low income women, who are often limited to subsistence farming in part due to low levels of literacy (this concern is shared by LOL). Within the dairy sector, this has often meant that women are tasked with traditionally gendered positions and have lacked control over finances. 10 Measuring Progress Toward Empowerment. Women’s Empowerment in Agriculture Index: Baseline Report http://www.ifpri.org/sites/default/files/publications/weaireport2013.pdf 11 See Promoting Gender Equitable Opportunities in Agricultural Value Chains. Prepared by dTS under contract no. GEW-I-00-02-00018-00, Task Order No. 02. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 34 Our sample seemed to be in line with previous findings regarding the control over income within Rwandan families with over two-thirds (68%) of families responding that men and women collaboratively made financial decisions. Responsibility for Household Decisions Percentage Only men 3 Only women 14 Both men and women 68 Whole family 12 Missing data 3 Total 100 Participants spoke of a difference between access to a bank account and control over the funds within. The comment was made that while our female participants have access to formal financial institutions, did not have full control over funds. Access to Technology Based upon data provided in LOL’s FY2014 Q4 report, women make up 35.6% of beneficiaries of AIs, which as previously established appears to offer one of the highest ROIs of available technologies. This is roughly in line with reported gender participation in the program. Service Providers DISTRICT Target AIs AIs completed Beneficiaries Number of farmers trained by AI SPs under the AI grant Male Female Total Male Female Total AGROJOY Ltd Gatsibo 300 300 244 54 298 402 109 511 HOBAS Ltd Kayonza 300 150 99 51 150 333 230 563 BARICE Ltd Rwamagana 670 655 381 274 655 487 596 1083 Nyamata Agri-inputs Ltd Bugesera 300 300 134 136 270 388 205 593 Kicukiro 280 168 175 98 273 160 149 309 COOVIGI Gasabo 300 300 164 163 327 286 230 516 Gicumbi 500 323 228 95 323 676 329 1005 A.T.VET Ltd Musanze 300 300 192 108 300 356 417 773 Rulindo 300 300 184 116 300 246 262 508 UPROCENYA Nyabihu 300 275 183 92 275 270 261 531 BAIR Rubavu 300 225 141 84 225 405 202 607 SPAD Ltd Gisagara 300 294 214 80 294 225 409 634 Huye 300 300 198 95 293 366 240 606 NIR HOPE Ltd Kamonyi 300 300 229 71 300 384 223 607 Nyanzza Agro Center Ltd Nyanza 300 121 69 52 121 207 107 314 Ruhango 300 300 174 126 300 416 196 612 5350 4611 3009 1695 4704 5607 4165 9772 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 35 Access to Training LOL has made a concerted effort to involve women in training programs and based on our document review have been remarkably successful in encouraging not only participation in training but leadership within cooperatives. The table below shows that a full 40% of participants in reported training sessions were female and encouragingly in the districts of Musanze and Rulindo women actually make up the majority of participants. Participants in KIIs expressed a great deal of satisfaction with the outreach activities conducted by GOR as part of RDCP II and have acknowledged that the cooperatives serve as a platform in which they are able to freely express their views. At the end of FY 2014, LOL reported that 38 percent of cooperative members were women and that the percentage has been growing. According to LOL, 76% of women participating in these training sessions report increased self-efficacy. SERVICE PROVIDERS DISTRICT Male Female Total Number % Number % AGROJOY Ltd Gatsibo 402 79% 109 21% 511 COOVIGI Kicukiro 286 55% 230 45% 516 COOVIGI Gicumbi 679 67% 329 33% 1,008 A.T.VET Ltd Musanze 356 46% 417 54% 773 Rulindo 246 48% 262 52% 508 UPROCENYA Nyabihu 270 51% 261 49% 531 BAIR Rubavu 445 66% 231 34% 676 Samuel Kamugundu Huye 766 62% 474 38% 1,240 NIR HOPE LTD Kamonyi 384 63% 223 37% 607 Nyanza Agro Center Ltd Nyanza 207 66% 107 34% 314 Total 4,041 60% 2,643 40% 6,684 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 36 Also of interest, 55 percent of trainees for Model Farmer courses conducted under RDCP II were women. Nearly a third (5/17) of trainers were women and of courses trained by women 37% of trainees were male; suggesting that men are willing to learn from female trainers. This assertion is supported by LOL’s finding that 84% of participants in the program believe that women should have equal access to economic, social, and political opportunities. Model Farmer (MF) Sex of MF No. Farmers Trained No. of Male Farmers Trained No of Female Farmers Trained % Female Farmers Trained. Patrice Habarugira Male 20 10 10 50 Innocent Ngabo Male 10 6 4 40 Raban Hagenimana Male 34 23 11 32 Theogene Munyansanga Male 33 25 8 40 Cdenis Bakamwe Male 3 3 0 0 Angelique Wihigora Female 18 13 5 35 Thomas Ndagijimana Male 40 15 25 63 Innocent Rutaremana Male 3 1 2 67 Eraste Nsengiyumva Male 22 10 12 55 Anasthasie Murekeyimana Female 9 6 3 30 Jean Ntimpirangeza Male 86 58 28 33 Gregoire Kabiligi Male 120 90 49 41 Didacienne Nyirahabimana Female 620 220 400 65 Heralie Usabyamahoro Female 310 110 200 65 Emmanuel Havugimana Male 417 200 217 52 Jessica Candari Female 25 15 10 40 Mutsinzi Abdounur Male 47 27 20 43 Total 1817 832 1004 55 CONCLUSIONS • Though our sample may not have been representative of the entire population, some participants believed that women had less control over funds in formal institutions than men. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 37 • Land O’Lakes and the GOR should be commended for the attention paid to gender equity and equality under RDCP II. That said, women are still only make up a third of beneficiaries and there is room for increased participation. • Training programs undertaken by SPs and MFs have done an admirable job of ensuring equal participation between men and women though there is still room for improvement in terms of women’s participation. RECOMMENDATIONS 1. The lack of control over bank accounts for some women in the sample should be further investigated in a larger study. 2. Land O’Lakes and the MINAGRI should continue outreach activities, especially encouraging women in positions of leadership within cooperatives. 3. The LOL gender mainstreaming strategy, given its age, should be reviewed and updated to accommodate changes in the program environment and any shift in program focus that might occur during the remaining life span of the program. 4. The LOL gender specialist should incorporate the WEAI framework into the RDCP II framework. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 38 SUMMARY OF RECOMMENDATIONS LAND O’ LAKES 1. For the remaining life of the project, the RDCP II should continue implementing activities directed at increasing the production and improving the quality of raw milk, but the focus of the program should be shifted to: a. Increasing local demand for Rwandan milk and dairy products and their marketing in neighboring countries b. The enhancement of the dairy processing component of the value chain in order to accommodate increased raw milk production and the demand created by increased local consumption and export marketing of Rwandan dairy products 2. Consideration should be given to a follow-on dairy value chain activity that will seek to preserve the achievements of the RDCP II and address a new phase of dairy value chain programming by: a. Strengthening the management and marketing capability of the MCCs b. Assuring the readiness and capacity of the RNDP to assume responsibility and take on the challenges associated with supporting and advocating for the Rwandan dairy industry c. Promoting local consumption of Rwandan milk and dairy products and their export to neighboring countries d. Further strengthening of the capacity to process Rwandan milk and dairy products. 3. Land O’ Lakes should investigate the relationship between higher quality offerings vis a vis their increased availability and impact on demand12. 4. Land of Lakes and USAID should reduce, where possible, the number of indicators associated with this study. a. Additional indicators associated with a nutrition objective and the USAID Nutrition policy should not be inserted into the program at this time. The possibility exists for adding a nutritional dimension to future follow-on studies. b. If additional indicators are added, care should be exercised so that LOL managers and staff do not change their approach – the value chain and dairy business should remain their focus. 5. The $5,000,000 target in FY 2015 in new private-sector investment in the dairy value chain be revised in light of the current climate for investment. 6. Building on findings in the Rwanda Dairy Strategy and Deep Dive Investment Analysis, a formal and concerted effort should be undertaken to explore the actual potential for the capitalizing on the demand for Rwandan dairy products in neighboring countries. 7. LOL should limit the scope of responsibility for the full-time M&E specialist to purely activities associated with monitoring and evaluation, especially dealing with data quality assurance protocols. 12 See Statement of Differences Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 39 8. The reported regular and ongoing unavailability of semen and hormones and privatization of that service should be taken up with the GOR by Land O Lakes and other key players in the industry including RDFA. 9. Artificial Insemination (AI), mastitis control, and technologies generally associated with milk quality improvement should be scaled up by implementing partner organizations. 10. The Shisha Wumva milk consumption education outreach campaign should build upon success of reaching a large number of potential participants. Implementing partners would be wise to incorporate behavioral change communications and strategies into the program. 11. LOL should continue to nurture the relationship between RDCP II and the GOR, which has significantly contributed to the overall effectiveness of the SOQ. 12. The relationships that have been formed through RDCP II should continue to be a point of emphasis for LOL. 13. LOL should continue efforts to improve business knowledge within MCCs. 14. Land O’Lakes and the MINAGRI should continue outreach activities, especially encouraging women in positions of leadership within cooperatives. 15. The LOL gender mainstreaming strategy, given its age, should be reviewed and updated to accommodate changes in the program environment and any shift in program focus that might occur during the remaining life span of the program. 16. The LOL gender specialist should incorporate the WEAI framework into the RDCP II framework. GOVERNMENT OF RWANDA 17. The GOR along with LOL should investigate ways to strengthen collaboration between various actors in monitoring compliance with the Ministerial Instructions. 18. The channeling of RDCP II funds through financial institutions should be explored as a means to create relationships between formal financial institutions and program beneficiaries. . A check off system should be considered in this regard. 19. The GOR must enforce the Ministerial Instructions on the Preservation, Collection, Transportation and Selling of Milk and establish a regulatory and inspection system to ensure the high quality of Rwandan milk and dairy products. FUTURE EVALUATORS 20. The lack of control over bank accounts for some women in the sample should be further investigated in a larger study. 21. Future evaluators should further investigate the reasons for a lack of awareness with the ZOI of several MCCs participating in RDCPII. This should inform future outreach strategies to identify potentially hard to reach populations and to inform outreach strategies. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 40 APPENDICES Appendix A. USAID Evaluation Policy Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 41 Appendix B. STATEMENT OF WORK SECTION C – DESCRIPTION / SPECIFICATIONS/STATEMENT OF WORK USAID/Rwanda – Dairy Competitiveness Sector Program Performance Evaluation C.1 Purpose The overall objective of the USAID/Rwanda Dairy Competitiveness Evaluation is to: (1) determine the extent to which the Rwanda Dairy Competitiveness Sector Project II implementing partners have been accountable to USAID, UASID and other local stakeholders, including GOR agencies, dairy industries and relevant civil society organizations; (2) draw lessons from the performance and results of the project, that can be applied and inform future dairy project design and operations in Rwanda. Findings from the performance evaluation will help identify opportunities to expand identified activities that demonstrated economic, institutional and social impacts on targeted beneficiaries in the Feed the Future Zone of influence. Expected results from the performance evaluation will include removal of activities that have not shown substantial impact and low financial returns (CBA) and expansion of activities that have demonstrated real potential for scale through private and public sector and local NGOs and having higher impact on income increase and improving nutrition outcomes. The results of the evaluation also will inform broader audiences relevant to related activities, including the Government of Rwanda’s (GOR) One Cow per poor family and the Communal cattle management style: Igikumba cy’umudugudu. C.2 Background a. Context The dairy sub-sector is important to the economic development of Rwanda, and dairy offers a pathway out of poverty for large numbers of households keeping livestock and for those who provide services and value addition throughout the supply chain. The 2012 "farm gate" value of milk is approximately Rwf 79.7 billion (US$129.70 million). Dairy subsector contributes 15 percent to agricultural gross domestic product (AGDP) and 6 percent to gross domestic product (GDP). Dairy's contribution to GDP is likely underestimated when considering ancillary products that can be attributed to dairy, e.g. hides, meat, traction/carting and manure. Dairy is a strategic commodity for Rwanda. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 42 The Government of Rwanda (GOR) in its Vision 2020 set the country on a course in 2000 to become a middle income country by the year 2020. The country achieved significant positive growth since 2001 with 7 to 8 percent average annual rates of growth in GDP. Vision 2020 has six pillars13 and three cross-cutting issues, and dairy supports each pillar and cross-cutting issue directly or indirectly with its contribution to GDP, household income, and job creation just to mention a few of the impacts. b. Current status The cattle inventory is 1.33 million and 28 percent are improved dairy cows generating 82 percent of the milk produced. Rwanda produces approximately 445,000,000 l/yr (per simple calculation this is an average of 1,115 liter per day for 300 days of lactation per dairy cow) of milk with an estimated farm value of Rwf 70 trillion (US$ 115.3 million). National dairy productivity is low. The informal market captures the largest percentage of the milk sold beyond the farm gate. Current consumption of milk is approximately 40 l/person/day (l/p/dy). Overall consumption is not keeping pace with natural expansion in milk production, and the outlook is for the growth in demand to remain relatively flat. The reasons for low milk consumption are the low purchasing power of consumers relative to the price of milk, and therefore low access to diverse good quality dairy products like cheese, yogurt, fermented milk, powder milk, etc. lack of accessibility to milk products. c. History of the project The Rwanda Dairy Competitiveness Program II (RDCP II) is a $15 million, USAID funded 5-year (January 2012 to January 2017) activity managed by Land O’Lakes Inc. International Development. The project is designed to reduce poverty through expanded marketing of quality milk that generates income and employment, and improved nutrition of rural households. Specifically, quality of milk is defined as milk with an acceptable level of bacteria accounts and of density and acidity level based on agreed upon standards (EAC standards in the case of Rwanda). By improving quality of dairy products and complying with standards, the industry, including smallholders’ farmers will improve its competitiveness, ensure significant market expansion and exports in the region and reduce production costs. The RDCPII aims at achieving its goals as mentioned above by linking existing and new smallholder dairy producers to expanding market demand in Rwanda and within the region, driven by improved quality milk production and effective dairy players operations, reduced transaction and production costs and increased 13 Vision six pillars are the following: (1) Good Governance and Capable State, (20 Human Resources development and a knowledge based economy, private sector led development, infrastructure development, productive high value and market oriented agriculture and regional and international integration. Vision 2020 crosscutting issues are gender equality, Natural Resources and Environment and Science, technology and ICT. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 43 investment all along the dairy value chain. To achieve this, the program provides a number of activities to increase adoption of new technologies and practices, including use of Artificial insemination, cattle disease control, better management of forage for dry season, animal feeding, and hygienic practices at milk aggregation or collection centers. In addition to this, appropriate transport logistics and cold storage are key infrastructure that helps maintain a quality milk product. On the policy side, RDCPII supports the industry to positively influence policy reforms and the business enabling environment in the dairy. The project was instrumental in facilitating policy dialogue to attract the private sector and also supported the development of the National Dairy Strategy. Also through a partnership with the Private Sector Federation (the sole private sector umbrella organization), the project supported a restructuring and establishment of an inclusive advocacy forum which include all dairy actors grouped in different clusters. To ensure sustainability of project interventions and stimulate demand of dairy products overall, the project provides a range of business development services and consumption campaign facilitation to stimulate demand of dairy products. The project supports the distribution network mainly made of popular milk kiosks and small shops in urban centers to expand domestic market and link them with financial services institutions to enhance access to finance. Here is below a graphic that present the activity Rwanda Dairy Competitiveness Project result framework: Narratives on the various RDCPII components will be attached to this SOW. The activity’s theory of change stipulates that supply side interventions like producers and MCCs accessing technologies and services and financing would result in increased availability of quality raw milk that would be attractive to investors as well as meet growing milk demand in non-processed milk value chain. The activities expected results are listed as follows: 1. 60 percent increase in the volume and value of dairy products sold; 2. 30 new products awarded with the Rwanda Seal of Quality; 3. 60 percent increase in net household income; 4. 7,500 new dairy-related jobs created; 5. $20 million(USD) in non US government resources leveraged; and Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 44 6. 9 policy reforms advocated for or enacted. RDCPII achievements up to December 2013 are as follows: • 11487 individuals trained in animal health, reproduction and feeding and milk hygiene; • 341 private enterprises, producer orgs, water users associations, women’s groups, trade and business associations and CBOs have applied new technologies or management practices as a result of USG assistance; • 3 Policies / regulations / administrative procedures currently in development. These include the National Dairy Strategy in process of validation, the National Milk certification plan and launch of Rwanda National Dairy Platform in restructuring process under Private Sector Federation umbrella (since one year); • 440 new dairy-related jobs created against a target of 1,000; • $1,038,437 new private sector investment in the dairy sector against a target of $4,000,000; • No new milk quality assessment technologies/protocols have been adapted or implemented which imply that no new products have been awarded and market with the Rwanda Seal of Quality; and • Only 15 dairy enterprises with upgraded production facilities resulting from financing applications against a target of 500. The project activities are located in 17 of the 30 districts in the country (Gasabo, Nyabihu, Rubavu, Nyagatare, Gatsibo, Bugesera, Nyanza, Huye, Ruhango, Gisagara, Kamonyi, Rulindo, Musanze, Kicukiro, Gicumbi, Rwamagana, and Kayonza). Here are attached the last 2 annual reports and last quarterly report. The reports also included detailed results on indicators related to increased margin, incremental sales, trainings and technology application, value of loans and investment increase as a result of the activity’s interventions, results on new jobs created, etc. d. Evolving change to activities implementation as a result of policy environment As mentioned above, RDCPII has been designed to be a value chain project. During implementation, the policy environment for dairy sector has changed positively with an emphasis of its potential contribution to reduce poverty by linking RDCPII with the One Cow per Poor Family GOR program that targets vulnerable farmers with one or two cows) and increasingly its contribution to avail more nutritious foods especially among the children under 2 years and lactating women. The last 2011 DHS revealed indeed that stunting level declined from 2006 to 2011 by 51% to 44%. As a result, RDCPII had to develop a milk consumption campaign and facilitate with the industry a strategy for dairy products consumption. Currently, the Mission is in negotiation with RDCP II management leadership to expand nutrition sensitive activities including enriching milk consumption campaign messaging using existing and locally available nutrition education and messaging curriculum. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 45 e. Institutional background: key decision to be made The goal of the National Dairy Strategy (NDS) is to double milk consumption to 80 l/p/yr by 2020, matching the goal of Rwanda becoming a middle income country. Based on population growth estimates, 1,161 million liters of milk will be required in 2020. Milk production will have to increase 13 percent per year to meet this target. At the same time milk intake by current consumers will have to increase, as well as attract new consumers. A balanced dual-approach to the dairy sub-sector is needed. C.3 Services and Tasks Required The evaluation team shall perform the following tasks: 1. Desk review of key program documents including the activity baseline report, RDCPII annual and quarterly reports which include detailed performance plan reports will constitute an important source of information to evaluators; 2. Develop detailed evaluation plan with clear evaluation study design and methods, explaining sampling details, data collection tools and data analysis plan; 3. Train enumerators when needed, conduct pre-test data collection tools and collect data, and ensuring standard and rigorous data quality control; 4. Develop data entry mask and enter quantitative and qualitative data; 5. Develop PowerPoint presentation of preliminary results to USAID-Rwanda; 6. Revision of final report based on USAID staff comments. C.4 Evaluation Critical Questions This evaluation shall address the following questions: a. Overall RDCPII performance 1. Referring to the theory of change as described above, to what extent has the project contributed to improved standards and quality of Rwanda dairy products; and expansion of investments in dairy processing and marketing in the zone of influence of the project?, what are the major contributing factors and constraints to achieving these results, and what corrective measures may be necessary to ensure objectives are achieved? To what extent are project beneficiaries (farmers, traders, milk collection centers) experiencing expected changes in their skills, knowledge, attitudes, behaviors, yields, income, etc.? Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 46 i. Is the project on track to meeting the currently set targets and should current targets be modified? ii. To what extent have new technologies introduced by the project improved animal feeding, artificial insemination, hygiene disease control, and contributed to increased farmer income? Considering RDCPII costs, what is the project return on investment (ROI) of each of the technology transfer activities at farm level (AI, animal feeding, mastitis control), financial products development, Seal of Quality, Cheese making related activities, business development services to MCC and milk kiosks? iii. Which of the above technologies transferred as part of the RDCPII program resulted into high adoption and high return on investments for a farmer? What should be the top three technologies to promote in the next phases? iv. What are the major constraints hindering achievement of the main project goals? What are the major changes in the project environment that affected implementation or achievement of intended results? 2. To what extent have new management approaches(staffing structures) used to implement development of financial products by INSPIRE, business development services, increased investment and improvement of quality milk/Seal of Quality, contributed to achievement of RDCPII results? i. Does the project have the right mix/balance of staff/funding given the activity priorities? What would be the recommended management structure and staff needs for the next two and half years of implementation? b. Efficiency 3. Based on return on investment at MCC level and the service providers, what would be the best cost effective management approach to implement scaling up of technologies? c. Coverage and outreach approach 4. Based on RDCPII approach to reach out to high number of rural and vulnerable farmers through MCCs and services providers with new technologies, to what extent is the project reaching the intended population and the targeted number of beneficiaries, in terms of knowledge and skills (adoption of technology), diet behavior change as result of milk consumption campaign? d. Sustainability Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 47 5. What opportunities for new partnerships (not yet explored) do you see to scale improved practices in the ZOI? 6. What has been the influence at the institutional level of the RDCPII Seal of Quality (SOQ) program in terms of increased milk quality and compliance to dairy products (including raw and pasteurized milk, yogurt, and cheese) standards, on GOR agencies (Rwanda Agriculture Livestock Inspection Services, Kigali City, Rwanda Bureau of Standards, relevant districts if necessary, etc) and the dairy industry? 7. Considering RDCPII approach for the different components and activities, what are the ones that have potential to be more sustainable after RDCPII closes and what would be the recommended change to foster sustainability? e. Gender equality and equity 8. Using on the WEIA framework, did the RDCPII approach to gender equality and equity ensure balanced involvement of women and men and particularly address constraints faced by women in the industry? C.5 Evaluation Design, Data collection, and Analysis This evaluation team will use a mixture of quantitative and qualitative approaches to gain insight into the impact of RDCP II activities and the processes that led to those impacts. A variety of methods and approaches will be used to collect and analyze information relevant to the evaluation objectives and questions outlined in the Scope of Work. The activity baseline report, RDCPII annual and quarterly reports which include detailed performance plan reports will constitute an important source of information to evaluators. Data available will include info linked to adoption of new technologies at different stages of the value chain, increased income and productivity at milk aggregation level and collection, increased investments and loans to the sector, introduction of the Seal of Quality program and impact on gender mainstreaming. The team will work together to conduct the evaluation using various methods including documents content analysis, surveys, Key Informant Interviews, Focus Group Discussions, and field/on-site observations. The Qualitative survey will focus on questions related to the RDCPII management, sustainability, and gender and project efficiency. A field survey will be required to collect some of the qualitative and quantitative data. The evaluation team will develop an appropriate evaluation questionnaire and design a survey using sampling methods that will provide a statistically valid result, especially informing and providing data on increase of income at farm, MCC level and kiosks level, and cow yields. A random or purposive sampling of beneficiaries and key stakeholders will be used to draw the sample of key informant and focus group participants. The sampling method can be decided based on discussion with the RCDPII project team on how best to identify a representative but relatively non-biased sample. The findings are expected to reflect, as appropriate, aggregation by activities, gender and location. Since multiple data sources will be utilized in the evaluation, it Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 48 is expected that triangulation will be used to synthesize the findings. Recommended evaluation methods should use sex-disaggregated data and incorporate attention to gender relations in all relevant areas. The evaluation firm is expected to provide quantitative and qualitative information using graphs and tables and specify the type of data they will collect. The evaluation data quality as per the ADS 578 and by reference 203 should apply. Bidders should submit a statement of work including proposed appropriate data collection and analysis methods for quantitative and qualitative information. Based on information available in the Statement of Work, the offeror is encouraged to elaborate an evaluation design matrix showing the way the evaluation questions will be addressed, related indicator/assessment criteria, collection methods, criteria and method. Once in Rwanda, the Evaluation Team Leader will work out the details of the evaluation protocol with USAID/Rwanda. USAID staff and GOR staff may participate in “one on one” meetings and focus groups discussions as appropriate. This participation will be determined in consultation with the Evaluation Team leader. C.6 General Program Implementation Guidelines C.6 (a) Assessment Team Composition The Evaluation Team shall be comprised of a six-person team including: one international consultant, who will be the Evaluation Team Leader; two international or regional technical experts covering the areas of cattle management, and dairy value chain expert; one local or regional financial services expert; one local gender expert; and one local economist. The following composition and expertise is required to perform this contract: The Team Leader The Team Leader will oversee the overall drafting of the assessment framework, including methodology determinations; organization of calendar/travel/meetings; coordinating the desk study, interview, survey and other data collection; and analyzing the data with input from team members and USAID to draft an evaluation report. In the field, the Team Leader will be responsible for day-to-day direction of team members. All evaluation team members should have defined roles and know in advance an outline of the report and the portion they are expect to draft. The Team Leader will be assisted by consultants with the following expertise: Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 49 1. International or Regional Technical Experts (2 people): 2. Financial Services Expert (1 person): 3. Gender Expert (1 person): 4. Economist (1 person, local): USAID team members will be able to provide guidance related to the Feed the Future initiative. C.6 (b) Schedule and Logistics The evaluation is expected to be conducted over a period of 8 weeks. The team will coordinate logistical arrangements with the Task Order COR. The contractor shall be responsible for the administrative support and logistics required to fulfill this task. These shall include all travel arrangements, appointment scheduling, secretarial services, report preparations services, printing, and duplicating. The USAID shall assist the contractor to get program documents and contacts necessary to fulfill the task. The contractor will propose the work plan and timeline for the evaluation task for USAID approval. The Task Order COR will provide strategic direction and guidance throughout the evaluation process, including the development of the evaluation plan, any data collection tools, and evaluation report outline, approach, and content. C.7 Deliverables The following deliverables are required. All written documentation for submission by the Contractor to USAID must be in English. C.7 (a) Evaluation In-Briefing and Out-briefing Presentations C.7 (a)(i) In-Briefing Presentation The Contractor shall provide an in-brief presentation to USAID on the methodologies to be used in the evaluation, including any survey questionnaires and instruments to be used within 5 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 50 days of arrival in-country, a presentation to USAID. The presentation will have to address all the questions mentioned in section C.4 of the evaluation scope of work. It should include detailed methodology to be used and highlight expected type of findings. C.7 (a)(ii) Out-briefing Presentation The Contractor shall provide an out-briefing PowerPoint presentation on preliminary results within 3 days after completion of field work to USAID/Rwanda and activity stake holders that will: • Review and assess RDCPII performance based on the topic questions and the main questions (in bold) reported above. The core of the report should align with the main topics of this evaluation: activity performance, efficiency, sustainability, geographic coverage and outreach approach, gender equity and equality; • Provide the methodology and approach used to evaluate the project; • Provide analysis and conclusions based on activity performance, efficiency, sustainable institutional change and outreach approach and assessment of the RDCPII contribution to gender equity; and • Provide sources of Information. C.7 (b) Draft Performance Evaluation Report The Contractor shall provide a draft narrative report (in MS Word) not later than 5 days after presentation of the draft results, but before the expatriate evaluation team members leave Rwanda. The draft report should not exceed 40 pages, excluding the cover page and table of content. Annexes should include the questionnaire used for surveys, focus group discussion; Key Informant Interviews and the list of contacts persons met, with their contacts (email addresses and phone numbers). C.7 (c) Final Performance Evaluation Report The Contractor shall submit a final performance evaluation report no later than 7 days after USAID/Rwanda provides written comments on the draft evaluation report. The report shall meet the USAID evaluation quality standards specified in Attachment J.3 of this RFTOP. The report must follow the requirements set forth below: • The evaluation report should represent a thoughtful, well-researched and organized effort to objectively evaluate what worked in the project, what did not and why. • The report must address all evaluation questions included in the statement of work as well as meeting the objective and specific objectives of the evaluation. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 51 • The report should address all limitations to the evaluation, including limitations associated with the evaluation methodology • Evaluation findings should be presented as analyzed facts, evidence and data, and should not be based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. • Recommendations need to be supported by a specific set of findings, and should be action-oriented, practical and specific. The format of the Performance Evaluation Report is recommended to include: 1. Title page; 2. Table of Contents; 3. Acronyms; 4. List of tables, or charts (if any); 5. Executive summary (not exceeding 3 pages); 6. Introduction (not exceeding 5 pages) shall include: a. Description of the project including goals and objectives; b. Evaluation rationale, including a list of the main evaluation questions; and c. Description of the evaluation design, methodology and limitations. 7. Findings – Describe the findings related to each of the evaluation questions; 8. Conclusions; 9. Recommendations – Based on the evaluation’s purpose and the findings, describe what remains to be done; what changes can be made in program design or implementation to result in more effective and/or efficient execution and improved results; identify potential new solutions to problems the project has faced; identify adjustments/corrections that need to be made; and recommend actions and/or decisions to be taken by management; 10. Lessons Learned - in terms of program implementation, coordination, and beneficiary satisfaction; 11. Annexes: a. Statement of Work; b. List of places visited, people interviewed, including contact information; c. Copies of all survey instruments and questionnaires; d. Electronic copy of data sets; e. List of background documents reviewed; and f. Copies of background documents which were used by Contractor but which were not provided by USAID-Rwanda. Note: Items 6 and 7 combined should not exceed 25 pages. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 52 As per the USAID Evaluation policy, the Contractor must submit completed or approved evaluations to the agency’s Development Experience Clearinghouse (DEC) and a coversheet attached indicating the type of evaluation conducted and design. Each completed evaluation must include a 5-10 page summary of the purpose, background of the project, main evaluation questions, methods, findings, conclusions, recommendations and lessons learned (if applicable) of the evaluation. In general, this performance evaluation should comply with USAID Evaluation policy requirements for evaluations available in the following link: http://www.usaid.gov/sites/default/files/documents/2151/USAIDEvaluationPolicy.pdf END OF SECTION C Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 53 Appendix C. GETTING TO ANSWERS TABLE Evaluation Question Data Collection Methods Sources of Information Sampling/ Selection Criteria Responsible Party Potential Constraints a. Overall RDCPII Performance 1. SOW Q: Referring to the theory of change (TOC) to what extent has the project contributed to improved standards and quality of Rwanda dairy products; and expansion of investments in dairy processing and marketing in the zone of influence of the project? Document review • Define original TOC • Analyze deviations from original design • KIIs  CDCS  FtF Multiyear Strategy  Project Design Documents  World Bank economic analysis  GOR economic analysis  Annual and Quarterly Activity Reports  USAID staff  IP staff  GOR Ministry of Agriculture  Rwandan Dairy Umbrella Organizations  Individuals in banks, credit institutions, and cooperatives  Project design documents and programming  Agreements for program interventions  USAID staff who designed project and current managers  IP management and staff; any staff key to design who may have moved on S. Lebbie/ P. Higgins 1. Re: the TOC, to what extent has the project contributed to improved standards and quality of Rwandan dairy products in the ZOI? Team analyzes current dairy standards vis-à-vis baseline + attribution to RDCP II Absence of baseline data; standards undocument ed 1. Re: the TOC, to what extent has the project contributed to the expansion of investments in dairy processing and marketing in the ZOI? Team analyzes current dairy standards vis-à-vis baseline + attribution to RDCP II Absence of baseline data; Reluctance of investors to disclose financial information 1. What are the major contributing factors and Team confirms anticipated RDCP II results w/ LoL Inadequate or no record￾keeping; Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 54 Evaluation Question Data Collection Methods Sources of Information Sampling/ Selection Criteria Responsible Party Potential Constraints constraints to achieving these results? & USAID/R: (pg. 5-6 of TO) reluctance to disclose financial information; inadequate monitoring 1. What corrective measures may be necessary to ensure objectives are achieved? Team dairy expert(s) analyze dairy industry environment Insufficient knowledge of Rwandan market forces et al 1.i. SOW Q: Is the project on track to meeting the currently set targets and should current targets be modified? Analysis of project implementation progress through reports, observation, & contacts with key RDCP II & industry actors  Annual Targets and Actuals (2012- 14)  Annual and Quarterly Activity Reports  PMP  Baseline Study  Gender Studies  WEIA Baseline (if available)  Disaggregate by sex, milk shed, MCC, and farmers  Synthesis of all data S. Lebbie/ M. Viola 1.ii. SOW Q: To what extent have new technologies introduced by the project improved animal feeding, artificial insemination, hygiene, and disease control? 1. ii. To what extent have new technologies introduced by the project contributed to increased farmer income? KIIs, FGDs, surveys  Annual and Quarterly Activity Reports  White papers on topical issues  Local agricultural extension agents  IP regional staff  Local veterinarians  Lead farmers  Farmers  Agri-business owners  Local government officials  Farmers  MCC  Agribusinesses/mil k kiosks  Quarterly and annual progress reports received for all interventions.  Representativ e sample of individuals in each category  Sampling of beneficiary farmers to include representative cross-section of project sites and beneficiary groups, with gender balance  Probability sampling using multi￾stage cluster S. Lebbie/ P. Higgins Inadequate or no record￾keeping; reluctance to disclose information; insufficient resources & time to conduct valid survey Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 55 Evaluation Question Data Collection Methods Sources of Information Sampling/ Selection Criteria Responsible Party Potential Constraints design stratified for gender representation  Random sample of villages in each Milk Shed receiving interventions  One village per milk shed not receiving interventions  Sample of households per village  Household Hunger Scale Module  representative sample of  MCC staff , by milk shed  Representativ e sample of agribusinesses , by milk shed 1.ii. SOW Q: Considering RDCPII costs, what is the project return on investment (ROI) of each of the technology transfer activities at farm level (AI, animal feeding, mastitis control)? Economic analysis  Annual and Quarterly Activity Reports  White papers on topical issues  Local agricultural extension agents  IP regional staff  Local veterinarians  Lead farmers  Business financial records (before and after)  Quarterly and annual progress reports received for all interventions.  Representativ e sample of individuals in each category  Representativ e sample of participating businesses J. de D. Kampayana/ I. Musafiri Same as above 1.ii. Economic analysis Same as above Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 56 Evaluation Question Data Collection Methods Sources of Information Sampling/ Selection Criteria Responsible Party Potential Constraints Considering RDCPII costs, what is the project return on investment (ROI) of financial products development? 1.ii. Considering RDCPII costs, what is the project return on investment (ROI) of Seal of Quality? Economic analysis Same as above 1.ii. Considering RDCPII costs, what is the project return on investment (ROI) of cheese-making related activities? Economic analysis Same as above 1.ii. Considering RDCPII costs, what is the project return on investment (ROI) of business development services to MCC and milk kiosks? Economic analysis Same as above 1. iii. SOW Q: Which of the above technologies transferred as part of the RDCPII program resulted into high adoption and high return on investments KIIs; economic analysis Analysis of all technologies offered; Household Survey  Farmers  Stakeholders along the value chain: processors, traders, financial service providers, input suppliers  Lead farmers  Agribusiness  Association representatives  Sample of stakeholders along the value chain  Sampling of beneficiary farmers to include representative cross-section of project sites and beneficiary J. de D. Kampayana/ I. Musafiri Clear understanding of the term “adoption;” reluctance by farmers to share financial information Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 57 Evaluation Question Data Collection Methods Sources of Information Sampling/ Selection Criteria Responsible Party Potential Constraints for a farmer? 1.iii. What should be the top three technologies to promote in the next phases? groups  Specific questions on technologies adopted  Technologies needed  Representativ es from each category (snowball sampling) 1.iv. SOW Q: What are the major constraints hindering achievement of the main project goals? 1.iv. SOW Q: What are the major changes in the project environment that affected implementatio n or achievement of intended results? Observation; and analysis of achievement of overall project objectives  USAID staff  IP staff  Policy leaders at national and regional levels GOR Ministry of Finance and Economic Planning, Ministry of Agriculture M. Viola/S. Lebbie/ P. Higgins Limited resources & time Understanding of project environment through interviews & study of project documentation Limited time given extent of evaluation questions 2. SOW Q: To what extent have new management approaches (staffing structures) used to implement development of financial products by INSPIRE? 2. To what extent have Review & analysis of current staffing vis-à-vis achievement of RDCP II objectives  USAID documents  IP documents  USAID staff  Business service providers  Rwanda Development Board  Rwanda Milk Sellers’ Association  Private Sector Federation Umbrella  Rwanda National Dairy Platform  USAID staff  Additional individuals after consultation with USAID, IOs, IPs, other key informants & secondary research  USAID staff who designed project and current managers M. Viola/S. Lebbie/ P. Higgins/ J. de D. Kampayana/ I. Musafiri Lack of clarity of organizational structures; limited time Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 58 Evaluation Question Data Collection Methods Sources of Information Sampling/ Selection Criteria Responsible Party Potential Constraints new management approaches (staffing structures) used to implement business development services contributed to achievement of RDCPII results? 2. To what extent have new management approaches (staffing structures) used to implement increased investment contributed to achievement of RDCPII results? 2. To what extent have new management approaches (staffing structures) used to implement improvement of quality milk/Seal of Quality contributed to achievement of RDCP II results?  IP management and staff; any key design staff 2.i. Does the project have the right mix/balance of Same as above Same as above Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 59 Evaluation Question Data Collection Methods Sources of Information Sampling/ Selection Criteria Responsible Party Potential Constraints staff/funding given the activity priorities? 2.i. What would be the recommended management structure and staff needs for the next two and half years of implementatio n? Evaluation team understanding of extent of scale-up & revised targets Time & resource allocations b. Efficiency 3. Based on return on investment at MCC level and service providers, what would be the best cost effective management approach to implement scaling up technology? Analysis of rates of return on RDCP II services & technologies offered; & clear understanding of staffing & scaling￾up requirements. • Current budget of the MCC project directed towards this • Current costs of the service providers • Current management approach • Costs anticipated for scaling up • Management challenges to scaling up • Representativ e sample of the full range of service providers P. Higgins/ I. Musafiri Clarity re: major elements of the management approach Cost Effectiveness analysis (KIIs) • Current management approach • Costs anticipated for scaling up • Management challenges to scaling up • Representativ e sample of the full range of service providers Service providers c. Coverage and outreach approach 4. Based on RDCP II approach to reach out to high number of rural and vulnerable farmers through MCCs and service providers through new Study of RDCP II progress reports; results of LoL annual survey; evaluation survey, KIIs, FGDs  Strategy documents to review targets  FtF Indicators tables for targets vs. actual  Internal project analyses and/or white papers  Farmers male and female, with at least two per Milk Shed with only women  Project design documents and programming received and being reviewed  IP partner documents and agreements for on-going M. Viola/ P. Higgins Inadequate record￾keeping; insufficient time to implement survey; lack of a definition & understandin g of “adoption” Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 60 Evaluation Question Data Collection Methods Sources of Information Sampling/ Selection Criteria Responsible Party Potential Constraints technologies, to what extent is the project reaching the intended population and the targeted number of beneficiaries in terms of knowledge and skills (adoption of technology)? 5. Based on RDCPII approach to reach out to high number of rural and vulnerable farmers through MCCs and service providers through new technologies, to what extent is the project reaching the intended population and the targeted number of beneficiaries in terms of diet behavior change as a result of the milk consumption campaign? Same as above  Farmers/household s  MCC staff  Business service providers  Milk sellers and exporters interventions  IP local implementing staff  Additional individuals after consultation with USAID, IOs, IPs, other key informants & secondary research  Representativ e sample  Probability sampling using multi￾stage cluster design stratified for gender representatio n for household survey  Representativ e sample of MCC staff, service providers, milk sellers, exporters Same as above; definition & understandin g of “vulnerability ” d. Sustainability 5. SOW Q: What opportunities for new partnerships Analysis of dairy industry & environment in the ZOI  Key actors in value chain  GOR National, regional, local officials  Representativ e sample from each group S. Lebbie/ P. Higgins Time limitations Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 61 Evaluation Question Data Collection Methods Sources of Information Sampling/ Selection Criteria Responsible Party Potential Constraints (not yet explored) do you see to scale improved practices in the ZOI?  Dairy associations at all levels 6. SOW Q: What has been the influence at the institutional level of the RDCPII Seal of Quality program in terms of increased milk quality and compliance to dairy products (including raw and pasteurized milk, yogurt, and cheese) standards, on GOR agencies (Rwanda Agriculture Livestock Inspection Services, Kigali City, Rwanda Bureau of Standards, relevant districts if necessary, etc.) and the dairy industry? Analysis of SoQ activity at various institutional levels through KIIs & available documentation  Owners of agribusiness  Gov’t of Rwanda officials (from Economics/Financ e and Agriculture)  Farm management and staff  Regional gov’t staff  National gov’t staff  Owners of agribusiness  Farmers  Rwanda Development Board  Rwanda Milk Sellers’ Association  Private Sector Federation Umbrella  Rwanda National Dairy Platform  Representativ e sample from each group S. Lebbie/ P. Higgins Same as above 7. SOW Q: Considering RDCPII approach for the different components and activities, what are the ones that have potential to be more sustainable Review & analysis of RDCP II sub￾activities vis-à-vis sustainability; review of policy changes by GoR concerned agencies; KIIs Same as above  USAID documents  IP documents  Synthesis of all interview data  Comparison of original design with actual results achieved S. Lebbie/ P. Higgins Same as above Same as Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 62 Evaluation Question Data Collection Methods Sources of Information Sampling/ Selection Criteria Responsible Party Potential Constraints after RDCPII closes and what would be the recommended change to foster sustainability? 7. Considering RDCPII approach for the different components and activities, what would be the recommended change to foster sustainability? above e. Gender equality and equity 8. SOW Q: Using the WEIA framework, did the RDCPII approach to gender equality and equity ensure balanced involvement of women and men and particularly address constraints faced by women in the industry? Analysis of LoL and USAID/RWAND A gender strategy; analysis of gender activities vis-à-vis WEIA; survey  Project documents that have the targets and reported results  Gender Strategy to identify social norms, policies and practices pre project intervention  Results of WEIA baseline study, if conducted  Analyses/evaluation s of project activities with a gender focus  All Quarterly and annual progress reports received for all interventions.  All gender strategy documents and accompanyin g reports  Final WEIA report J. Umuzigambe ho Time limitations to conduct survey KIIs  Women owners of agribusinesses  Women leaders/members of dairy associations and unions (regional and national)  Women focused org  Representativ e sample from each respondent category FGDs  Farmers  Women owners of  Representativ e sampling of Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 63 Evaluation Question Data Collection Methods Sources of Information Sampling/ Selection Criteria Responsible Party Potential Constraints agribusinesses  Service providers to women and men all respondent categories, with some all women, some mixed male and female Household Surveys  Women Farmers WEIA Categories:  role in household decision-making  access to productive capital  Income  Leadership  labor time allocations  Probability sampling using multi￾stage cluster design stratified for gender representatio n Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 64 Appendix D. SOURCES OF INFORMATION Documents Cited 1. Ministry of Agriculture and Animal Resources. Agriculture Gender Strategy . Kigali: MINAGRI, November 2010. 2. RDCPII_BASELINE_REPORT_FINAL (1) 3. USAID Evaluation Policy, January 2011. 4. USAID Dairy Competitiveness Project; End of Project Evaluation, Dermot Cassidy and Alistair Paterson, March 2011. 5. Promoting Gender Equitable Opportunities in Agricultural Value Chains. Prepared by dTS under contract no. GEW-I-00-02-00018-00, Task Order No. 02, 2011 6. Rwanda Dairy Competitiveness Program II Final Report: November 15, 2007- November 15, 2011, Land O’Lakes, Inc., November 2011. 7. USAID/Rwanda Task Order, January 2012. 8. Rwanda Dairy Competitiveness Program II, Annual and Quarterly Report: January 6 – September 30, 2012; Quarterly Report: July 1, 2012 – September 30, 2012, Land O’Lakes, Inc., October 2012. 9. Rwanda Dairy Competitiveness Program II Quarterly Report: July 1– September 30, 2013, Land O’Lakes, Inc., October 2013. 10. Bamusiime, A. (2012). Gender Mainstreaming Strategy. Kigali, Rwanda: Land O' Lakes, Inc. Feed the Future. (n.d). Women Empowerment in Agriculture Index. USAID . Land O' Lakes, 2013. 11. Rwanda Dairy Competitiveness Program II Annual Report, Land O’Lakes, Inc., October 2013. 12. Ministry of Agriculture and Animal Resources. Rwanda National Diary Project. Kigali: MINAGRI, 2013. 13. Ministry of Agriculture and Animal Resources. Rwanda National Diary Strategy. Kigali: MINAGRI, April 2013. 14. Rwanda Dairy Competitiveness Program II Quarterly Report: July 1 – September 30, 2014, Land O’Lakes, Inc., October 2014. 15. Rwanda Dairy Competitiveness Program II Annual Report, Land O’Lakes, Inc., October 2014. 16. Rwanda Dairy Competitiveness Program II Annual Report, Land O’Lakes, Inc., 1st Qtr. Progress Report, October 1 – December 31, 2014. 17. Deep Dive Regional Demand Study - Final Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 65 18. (Rwanda) Dairy Competitiveness Project - Fact Sheet (2015) 19. USAID, International Food Policy Research Institute, Oxford Poverty and Human Development Initiative: Measuring progress toward empowerment; Women’s empowerment in agriculture index: baseline report, The Women Empowerment Index in Agriculture (WEIA) Framework, 2014. 20. Wakala EA Consulting Services Ltd. (n.d). Rwanda Dairy Competitiveness Program II: Baseline Survey Report. Kigali: USAID Rwanda, 2015. 21. LOL Gender Mainstreaming Strategy Work Plan. 22. RDCP II List of Indicators. 23. RDCP Performance Management Plan (PMP). 24. LOL Annual PPR Reporting. 25. Annual Survey Questionnaires and LOL survey protocols 26. Close Out Report final draft- INSPIRED, February 2015 27. 2014_WP_WhiteGold-DairyEastAfrica FINAL June 2014 28. Rwanda’s Dairy Value Chain from a Financier’s Perspective—analysis and opportunities-INSPIRED INTERNATIONAL, November 2012 29. RwandaDairyCompetitivenessProjectII-2014 PPR Indicator Narratives (RDCPII done) Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 66 Key Informants Interviewed NAME TITLE TEL. NO.# EMAIL Dr. Cyubahiro Mark Bagabe Director General, Rwanda Standards Board 0788 304 197 mark.bagabe@rsb.gov.rw Charles Bizimana M&E Specialist, RDCP II, Land O'Lakes 250 280 252 555 charles.bizimana@idd.landolakes.co m Dr. Livingstone Byamungu Private Sector Advisor, Private Sector Federation 0788 302 863 drbyamungu@gmail.com Dr. Charles Lawanga Gakwaya Managing Director, Inyange Industries 0788 301 533 clgakwaya@inyangeindustries.com Ajay Gupta Deputy Managing Director, Inyange Industries 0788 389 666 anaresh@inyangeindustries.com Dr. Christine Kanyandekwe, Director General, Rwanda Agricultural Board 0788 590 435 k_chris2005@yahoo.fr Dennis Karamuzi Deputy Chief of Party, RDCP II, Land O'Lakes 0788 305 014 dennis.karamuzi@idd.landolakes.co m Antoinette Mbabazi National Certification Division Manager, Rwanda Standards Board antoinette.mbabazi@rsb.gov.rw Eusebe Muhikira Agricultural & Trade Manager, Rwanda Development Board 0788 513 764 eusebe.muhikira@rdb.rw Harriet Mutoni Post￾Harvest/Marketing Specialist, Livestock Infrastructure Support Project/MINAGRI 0738 455 330 brjanny@yahoo.com Dr. Michel Ngarambe Program Coordinator, Livestock Infrastructure Support Project/MINAGRI 0788 508 082 ngarambemic2000@yahoo.fr Euphrasie Nyirazikwiye In-Charge: Animal Production Certification, Rwanda Agricultural Livestock Inspection Service 078884477 8 nzikwiye@yahoo.fr Frank O'Brien Chief of Party, RDCP II, Land O'Lakes 0786 112 468 frank.obrien@idd.landolakes.com Dr. Theogene Rutagwenda Director General, MINAGRI 0788 303 309 rutagwendat2006@yahoo.com Florence Umurungi Chairperson, Rwanda 0788 302 umurungip@yahoo.com Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 67 National Dairy Platform 613 Beatrice Uwumukiza Director General, Rwanda Agricultural Livestock Inspection Service 0788 848 410 buwumukiza@gmail.com Mr. Juvenal Kalema Director of Credit Department, Banque Rwandaise de Development 250 788 301 848 j.kalema@brd.rw Mr. Alexis Bizimana Agriculture Finance Manager, Kenya Commercial Bank 250 782 596 747 abizimana@rw.kcbbankgroup.com Mr. Espoire Serugo Head of Agriculture Finance, Urwego Opportunity Bank 250 788 821 810 eserugo@uob.rw Mrs. Hyacinthe Kabandana Marketing Manager, Duterimbere IMF Ltd. 250 788 465 637 Mrs. Ruth Nyinawumuntu Sacco Manager, SACCO Kinazi Amizero 250 788 566 256 Mr. Christophe Musafiri Loan officer, Sacco Ibonemo Gacurabwenge 250 788 403 807 Farmers Interviewed KARIMBA MATIAS KARIMBA MATIAS 788757084 AUTHUR ELIYAZARI AUTHUR ELIYAZARI 784386342 ANDREW BAHIZI ANDREW BAHIZI DUSABE CONSOLE DUSABE CONSOLE MUSESITA THANASE MUSESITA THANASE 788416020 KAYIRANGA JONATH KAYIRANGA JONATH 781207100 MUNYENSANGA THEOGENE MUNYENSANGA THEOGENE 785761258 NYIRAMURUTA BERTHILDA NYIRAMURUTA BERTHILDA 786308766 NZABAMWITA NAFANIEL NZABAMWITA NAFANIEL NSHIMIYIMANA SEBASTIEN NSHIMIYIMANA SEBASTIEN 782571799 5 NTIMPIRANGIZA JOHN NTIMPIRANGIZA JOHN 788447326 UTAGIRUWE JEAN PAUL UTAGIRUWE JEAN PAUL 788634843 SIBOMANA PASCAL SIBOMANA PASCAL UWABABYEYI UWABABYEYI 725329832 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 68 MARIE ROSE MARIE ROSE UFITINSHUTI EMILE UFITINSHUTI EMILE 788771047 TUGIRAMAHORO PACIFIC TUGIRAMAHORO PACIFIC 783103595 SEKAMANA ALEX SEKAMANA ALEX 786199259 NKURIKIYINKA PIERRE NKURIKIYINKA PIERRE 788522212 TWIZEYIMANA ILDEPHONSE TWIZEYIMANA ILDEPHONSE 788658225 MWIKARAGO RUGEMANYIKO MWIKARAGO RUGEMANYIKO 788448013 MUSHIRARUNGU ANGELIQUE MUSHIRARUNGU ANGELIQUE ZINKWABANZE ANGELIQUE ZINKWABANZE ANGELIQUE 785697940 MUKANOHERI ANASTASIE MUKANOHERI ANASTASIE 783797366 RUKEREZA SARASTIEN RUKEREZA SARASTIEN KAGIRANEZA ELIAS KAGIRANEZA ELIAS 789596870 HASHAKIMANA RUBERA HASHAKIMANA RUBERA 786573351 SEKANYANA ALPHONSE SEKANYANA ALPHONSE 788418193 NYIRAKAZIBO DEBORAH NYIRAKAZIBO DEBORAH MUTAMBUKA EVALISTE MUTAMBUKA EVALISTE 784555358 UWINGABIRE THIRPHONIA UWINGABIRE THIRPHONIA 782921082 MUJAWUMUREMYI MUJAWUMUREMYI 782204502 MUSABYIMANA OLIVE MUSABYIMANA OLIVE 786600023 KAYITEGERE CHARLOTTE KAYITEGERE CHARLOTTE MUTEGARUGORE ANSILLA MUTEGARUGORE ANSILLA 783354151 KAREMERA DICK KAREMERA DICK 788442329 RUZIBIZA STEVEN RUZIBIZA STEVEN 788855583 AKIMANA ANGELIQUE AKIMANA ANGELIQUE 788680125 HAKIZIMANA EMMANUEL HAKIZIMANA EMMANUEL 788598592 MUKARWEGO ZILPA MUKARWEGO ZILPA 788471100 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 69 NTAWIHA JOHN NTAWIHA JOHN 783876154 SENYANA JOB SENYANA JOB 785010099 KUBWAYO ALPHONSE KUBWAYO ALPHONSE 783710896 BYUKUSENGE BYUKUSENGE 788733346 MUKOBWAJANA SERAPHINE MUKOBWAJANA SERAPHINE 783368510 NGENDAHIMANA AUGUSTIN NGENDAHIMANA AUGUSTIN 788396520 NDAGIJIMANA GASPARD NDAGIJIMANA GASPARD 783710896 NGENDAHIMANA AUGUSTIN NGENDAHIMANA AUGUSTIN 788396520 RWAZIGA ANASTASIE RWAZIGA ANASTASIE 788461133 SEKABIBI VICENT SEKABIBI VICENT 788509617 NIYINTEZE ERIC NIYINTEZE ERIC 787867527 UFITINEMA EMILE UFITINEMA EMILE S MUGARURA JOHN BAPTIST MUGARURA JOHN BAPTIST 722506607 788937853 KAZUBA BYIRINGIRO KAZUBA BYIRINGIRO 783881074 KAZUNGU PAUL KAZUNGU PAUL 782473314 MUSOHOKE RWARINDA MUSOHOKE RWARINDA 782473314 GASASIRA ZOGEYE GASASIRA ZOGEYE 788557412 AGASHATI NZATATIRA AGASHATI NZATATIRA 787111147 KAZARWA UWAMWEZI DATIVA KAZARWA UWAMWEZI DATIVA 788593646 GASOFERO CONSTASTAIN GASOFERO CONSTASTAIN 783418849 NDINDABAHINZI INNOCENT NDINDABAHINZI INNOCENT 788219335 RWABUZISONI LEONARD RWABUZISONI LEONARD 788512023 HABIMANA THANASE HABIMANA THANASE 783787783 BAGIRAWUBUSA JEAN BAPTIST BAGIRAWUBUSA JEAN BAPTIST 723471300 MUHAWIMANA LEONILE MUHAWIMANA LEONILE 788603089 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 70 BAYINGANA JEAN MARIE VIANNEY BAYINGANA JEAN MARIE VIANNEY 783578036 MUKANDORIMAN A LOUISE MUKANDORIMAN A LOUISE 726894698 MUKAYUHI JEAN MUKAYUHI JEAN 788607761 UWIHANGANYE EUGENE UWIHANGANYE EUGENE 784231155 BUHINGIRO MARTIN BUHINGIRO MARTIN 788536139 KAYISINGA CALLIXTE KAYISINGA CALLIXTE 784141846 KANANI AUGUSTIN KANANI AUGUSTIN 789767171 HAVUGIMANA THEOGENE HAVUGIMANA THEOGENE 789806809 HAKIZIMANA GASIANE HAKIZIMANA GASIANE 786616736 NIYOMUGABO YOTAMU NIYOMUGABO YOTAMU NTIRUSHWA INNOCENT NTIRUSHWA INNOCENT 727833867 RUTEMBESA PRUDENCE RUTEMBESA PRUDENCE 725465183 RWINKESHA ROGINE RWINKESHA ROGINE 788894870 UWANTEGEKA CASILDA UWANTEGEKA CASILDA HATEGEKIMANA FRANCOIS HATEGEKIMANA FRANCOIS 788784442 UWAMAHORO VIANNEY UWAMAHORO VIANNEY 783166257 NIBASABE JOSEPHINE NIBASABE JOSEPHINE 788827451 DUSENGEMUREMY I LEOSTACHE DUSENGEMUREM YI LEOSTACHE 726271106 DUSENGEMUREMY I JEAN BAPTISTE DUSENGEMUREM YI JEAN BAPTISTE 783141122 RUBAMBO LUIS RUBAMBO LUIS 788562923 NGARAMBE NIRORA NGARAMBE NIRORA 788293263 RUDACENGA MICHEAL RUDACENGA MICHEAL 783565367 NKWIYINKA KAGIRANEZA NKWIYINKA KAGIRANEZA 786320409 KABERUKA KABERUKA NYIRAMUTARUTW A NYIRAMUTARUTW A SEBIKARI SAMUEL SEBIKARI SAMUEL 788858383 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 71 SEBAHUTU ETIENNE SEBAHUTU ETIENNE 783388321 RUKOYOYO FERESITA RUKOYOYO FERESITA GAKWAYA THEODOMILE GAKWAYA THEODOMILE 788265873 UWIRANGIYE SERAPHINE UWIRANGIYE SERAPHINE 782041010 RUTABURANGONG A INNOCENT RUTABURANGONG A INNOCENT 789131583 MUHAYIMANA SLYVESTRE MUHAYIMANA SLYVESTRE 788890478 NIZEYIMANA FAUSTIN NIZEYIMANA FAUSTIN 786016573 NSHYIMIYUKIZA NASTASE NSHYIMIYUKIZA NASTASE 788740989 MUKANKWIRO CECILE MUKANKWIRO CECILE 782912412 MUHIMPUNDU THERESE MUHIMPUNDU THERESE 788413497 MUKAMABANO EPIPHANIE MUKAMABANO EPIPHANIE 783053584 TURIKUMANA J.M.V TURIKUMANA J.M.V 783185350 RUHETESHA DONAT RUHETESHA DONAT 788754721 MPAMYABIGWI MPAMYABIGWI KADUSHI NYIRIMBUGA KADUSHI NYIRIMBUGA 783098231 GATOTO TITUS GATOTO TITUS 783230403 BUSINDA BIGIRIMANA BUSINDA BIGIRIMANA 788865660 KANAMUGIRE ATHANASE KANAMUGIRE ATHANASE 783010656 HAKIZABERA AZARIAS HAKIZABERA AZARIAS 788895613 HABUMUGISHA MICHAEL HABUMUGISHA MICHAEL 788430700 KANYANKORE RUJUGIRO KANYANKORE RUJUGIRO 788537949 NYAKARE THEOGENE NYAKARE THEOGENE 783085163 INTARAMIRWA ESPERANCIE INTARAMIRWA ESPERANCIE 788248556 KABAGWIRA BEATRICE KABAGWIRA BEATRICE 781618517 GAHIMA ALPHONSE GAHIMA ALPHONSE 788303575 MUKARETA MUKARETA 725116516 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 72 IMMACULE IMMACULE KAYIGAMBA KAYIGAMBA 788653827 TWAGIRUMUREMY I JOAS TWAGIRUMUREMY I JOAS 0 MUKIZA JEAN RUKEMA MUKIZA JEAN RUKEMA 783541676 NSABIMANA UZIEL NSABIMANA UZIEL 788470181 HAKUZIMANA JACKON HAKUZIMANA JACKON 783560976 BARITONDA BAPTISTE BARITONDA BAPTISTE 726144055 MUKAHIGIRO CECILIA MUKAHIGIRO CECILIA UWIHOREYE JEAN LUC UWIHOREYE JEAN LUC 726321427 UMURERWA SOLANGE UMURERWA SOLANGE 788960584 HABIMANA VINCENT HABIMANA VINCENT 728530131 NZABAKURIKIZA TELESPHORE NZABAKURIKIZA TELESPHORE 782679447 NZABONIMANA ANTOINE NZABONIMANA ANTOINE 788366263 MUKABAGEMA DOROTHE MUKABAGEMA DOROTHE 728412704 HABIYAKARE JEAN DAMASCENE HABIYAKARE JEAN DAMASCENE NYIRABAKIGA REGINE NYIRABAKIGA REGINE Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 73 RDCP II SEVICE PROVIDERS VISITED AND INTERVIEWED Service Providers DISTRICT AGROJOY Ltd Gatsibo COOVIGI Gasabo COOVIGI Gicumbi A.T.VET Ltd Musanze Rulindo UPROCENYA Nyabihu BAIR Rubavu Samuel Kamugundu) Huye NIR HOPE LTD Kamonyi Nyanza Agro Center Ltd) Nyanza Frederic Mutiganda Kicukiro Emmanuel Robero Kicukiro Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 74 RDCP II MODEL FARMERS VISITED AND INTERVIEWED Model Farmer (MF) Sex of MF Patrice Habarugira Male Innocent Ngabo Male Raban Hagenimana Male Theogene Munyansanga Male Cdenis Bakamwe Male Angelique Wihigora Female Thomas Ndagijimana Male Innocent Rutaremana Male Eraste Nsengiyumva Male Anasthasie Murekeyimana Female Jean Ntimpirangeza Male Gregoire Kabiligi Male Didacienne Nyirahabimana Female Heralie Usabyamahoro Female Emmanuel Havugimana Male Jessica Candari Female Mutsinzi Abdounur Male Evrethere Twagirimana Male TOTAL 18 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 75 Appendix E. WORK PLAN Implementation Calendar EVENT PERSON(S) RESPONSIBLE DATE(S) Phase 1: Preparations and Desk Review February 2 – 13 Pre-Travel: Review background materials & study Task Order et al All members February 16 – 23 Study evaluation questions w/aim to determining team division of labor & formation of mini-teams All members Establish preliminary contact w/ Jim Yazman M. Viola/P. Higgins Develop & disseminate agenda for 1st team meeting M. Viola Establish lines of communication between team members M. Viola/all members Discuss w/ aim to preliminary planning of surveys & development of guides to KIIs & FGDs M. Viola/S. Lebbie Phase 2: Design and Work Plan February 20-22 Team travel to Rwanda February 20-21 1st Team Meeting: (Deliverable: In￾Briefing): Introduce members & discus skill sets vis-à￾vis evaluation questions and report All members February 22 Establish local communication parameters All members Assign team members to evaluation questions & report sections, incl. formation of mini￾teams M. Viola/S. Lebbie Identify gaps in background materials, esp. gender All members Define team approach to meetings w/ LoL and USAID/R M. Viola/S. Lebbie Confirm team transport & related logistics, e.g. fuel, associated costs, monitoring, etc. I. Musafiri Continue planning of surveys, KIIs & FGDs & begin development of interview & discussion guides Phase 3: Rwanda Field Work & Data Collection February 23 – March 20 Meeting w/ Land Introduce team members; explain roles of members & objective(s) of evaluation All members February 23 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 76 O’Lakes: Review evaluation questions w/ LoL team M. Viola/S. Lebbie Identify & obtain any missing background documentation, esp. gender & TORs of sub￾contractors All members Obtain results of surveys conducted by LoL & methodologies applied M. Viola/S. Lebbie Establish lines of communication between LoL & team; determine counterparts, if any All members Determine sites for visits & discuss local travel & related logistics All members Obtain an updated overview of project status & achievement, incl. major constraints & scaling-up planning All members In-Briefing at USAID/R: Introduce team members & explain members’ roles M. Viola February 23 Review USAID/RWANDA expectations and anticipated results of the evaluation All members Determine USAID/RWANDA participation in meetings and/or site visits, etc. M. Viola/S. Lebbie De-brief re: security & other relevant logistical matters All members Review initial work plan; discuss format & content of in-brief presentation M. Viola/S. Lebbie 2nd Team Meeting Identify potential constraints to data collection & analysis & develop mitigation strategies M. Viola/S. Lebbie February 23 Finalize overall strategic approach to data collection & analysis M. Viola/S. Lebbie Define KIIs, FGDs, data collection tools, and sampling frames and methodologies for all evaluation questions All members Develop outline for In-brief Presentation to USAID/R M. Viola Prepare a detailed data collection & analysis plan M. Viola/S. Lebbie Define the precise responsibilities of the financial specialist and economist vis-à-vis evaluation questions & report M. Viola/S. Lebbie/J. Kampayama/I. Musafiri Develop an action plan for the gender & gender analysis aspects of the evaluation & report, incl. existing materials, surveys & data collection & analysis tools required M. Viola/S. Lebbie/P. Higgins/J. Umuzigambeho Prepare for Data Collection: Interview four (4) finalists for enumerator positions February 24 Contract team of enumerators Train enumerators Develop surveys & guides to KIIs & FGDs Prepare In-brief Obtain inputs from team members M. Viola/S. February 24-25 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 77 Presentation: Lebbie Flesh-out presentation outline M. Viola/S. Lebbie In-brief Presentation to USAID/R Make presentation M. Viola/All team members M. Viola/S. Lebbie February 26 (subject to confirmation by USAID/R Refine work plan in accordance with USAID/RWANDA feedback Data Collection (to be determined & planned) All team members February 24- March 12 Data Analysis (to be determined & planned) All team members March 12-15 Report Drafting: (Deliverable: Out￾briefing) Report drafting to commence early on in the evaluation process M. Viola/All team members Ongoing through March 19 Out-briefing to USAID/R: (Deliverable: draft report) Prepare presentation based on consultations w/ USAID/R M. Viola/S. Lebbie/All members March 17 (subject to confirmation Make presentation M. Viola/S. by USAID/R Lebbie Draft Evaluation Report Submit 1st draft of evaluation report M. Viola March 20 Depart Rwanda M. Viola/S. Lebbie/P. Higgins March 21 Phase 4: Final Analysis and Reporting March 23-29 Final Report Submit Final Report to USAID/R M. Viola March 29 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 78 Appendix F. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Overarching Recommendations The team’s qualitative research identified a number of factors that have contributed to the success of RDCP II including: • The Seal of Quality set of best practices and standards propagated by the RDCP II; • Training and capacity-building in “best practices” of handling raw milk carried out by the RDCP II; • The approach taken on the ground by LOL and the close working relationship the RDCP II enjoys with farmers, MCCs, and Service Providers (SPs); and, • The collaboration and unity of purpose and harmonious working relationship between the RDCP II and relevant agencies of the MINAGRI/GOR. Although the program is generally successful, improvements in business and marketing acumen throughout the entirety of the agricultural value chain could increase the impact and sustainability of the program. Land O’ Lake should revisit the Theory of Change in light of changes in the program environment, i.e. increased production and quality. The Theory of Change should read something akin to: “the increased availability of raw milk along with the increased local consumption and cross-border export sales of Rwandan milk and dairy products will result in the development of new dairy products and increased investment to the Rwandan dairy value chain and industry.” the team’s research indicates that it is demand that will drive the program; and, if demand is created, investment will follow. Revisit and reformulate the program Results Framework: given the addition of a nutrition objective and the USAID Nutrition Policy, the Results Framework should now be re-conceptualized with nutrition as a high-level program objective; and, agricultural activities with nutrition results should be added to the program implementation plan. Care should be exercised so that LOL managers and staff do not change their approach – the value chain and dairy business should remain their focus – however, the importance of nutrition as a high-level program objective must be understood and implanted. SPECIFIC Recommendations Q1 Contribution of program to standards, quality, and investment Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 79 Standards Based upon the evaluation team’s review of relevant documents and confirmed through interviews, it appears that RDCP II has facilitated a number of positive outcomes related to the implementation of standards in the dairy sector within Rwanda. First, technologies have been transferred to farmers and are being adopted into practice on a wide scale which has drastically impacted the availability and quality of raw milk. Further, the team has found that the level of awareness by consumers of the benefits of milk consumption and the importance of the cleanliness of milk and dairy products is increasing and is directly attributable to the RDCP II-sponsored Seal of Quality and the Shisha Wumva education campaign. The project has also been responsible for the drafting of a National Dairy Strategy and Ministerial Instructions for the proper collection, handling, and transporting of raw milk and has worked collaboratively with a new private entity, the Rwanda National Dairy Platform (RNDP), has been created and is poised to provide support to and advocate for the dairy industry when the RDCP II comes to an end in January, 2017. From a number of standpoints, the RDCP II is performing at a high level of effectiveness, with planned targets being reached and, in many instances, exceeded. Based upon the KIIs and FGDs, stakeholders believe that it is of the upmost importance to strengthen the MCCs in terms of their overall business and marketing acumen. Stakeholders identified a number of potential measures including: • Installation of professional managers trained in the workings of the dairy industry paid for by the MCCs. In this scenario, a number of actors would act collectively to hire an educated individual to oversee several MCCs; allowing them to offer a competitive salary to attract quality talent. • Continued expansion of mechanisms aimed at the generation of reserves or working capital at the MCCs. • Building upon ongoing efforts through AgPRO in the training of MCC managers in business planning and the development of a business and annual operational plans at each MCC. • A system of enforcement of the forthcoming Ministerial Instructions on the collection, transporting and handling of milk; and, development of incentives through which producers of high quality milk are rewarded and producers of low-quality milk punished. Improvement in Quality of Raw Milk The quality of raw milk produced in the ZOI has significantly improved as measured by the amount or volume of poor quality raw milk rejected by MCCs and processors. At the MCCs visited by the Progress has been made in improving the quality of raw milk being produced under RDCP II. However, if the assumption that producers of low quality milk may be selling their products on alternative markets is correct, this practice will require policing Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 80 evaluation team, the amounts of raw milk rejected prior to RDCP II interventions were found to be significantly greater than the amounts rejected after RDCP II￾supported technology transfer, training, and grants. Records at thirteen (13) of the fourteen (14) MCCs visited indicated that the amount of poor quality milk rejected dropped from an average of 200 liters per week before the RDCP II to 41 liters per week after RDCP II investment in milk￾testing equipment; an improvement of 79.5%. As shown in Table 1, this decrease in rejected milk has been accompanied by a nearly 50% gain in the production of milk at MCCs. through the forthcoming Ministerial Instructions, if the high quality of Rwandan milk and dairy products is to be upheld. Expansion of investment in dairy processing and marketing There are some indications that investment may increase. Of the seven (7) financial institutions visited, four (4) had entered into Memorandums of Understanding (MOUs) process with the RDCP II with the aim of lending to dairy chain actors. Of those, two (2), Urwego Opportunity Bank (UOB) and Kenya Commercial Bank (KCB), while their lending to dairy began only in late 2014, had already developed portfolios, within six months, totaling RWF 83,700,000 (US$119,571), with both banks expecting to quadruple their portfolios by the end of 2015. So far, the loans purposes have been for cattle acquisition, and mobile coolers. The maximum dairy loan period is 4 years, interest rate between 15%-20%, monthly repayment after a grace period based on the cash-flow. The average loan size is Rwfs 400,000 (US$570). The general impression gained from discussions with commercial bank officials and loan officers is that the dairy industry is currently not considered an attractive investment opportunity; not because of the investment cost analysis, but rather a poor perception of the business and individual borrowers. One reason for that is a past history of non-performance of construction and equipment loans in the amount of RWF 210,000,000 (US$300,000) made to MCCs by the BRD (Rwanda Development Investment in the dairy value chain is occurring, but at a rate and volume less than what was perhaps envisioned or desired. MCCs have reinvested in their operations but they are relatively few in number and the amount of investment is relatively low. This has implications for gender, because, traditionally, men have more access to banks and formal financial institutions than women. There is a regional demand for Rwandan Dairy products. In at least some districts, the milk testing kits distributed through the Seal of Quality Initiative generate a high ROI. For the remaining life of the project, the RDCP II should continue implementing activities directed at increasing the production and improving the quality of raw milk, but the focus of the program should be shifted to: a) increasing local demand for Rwandan milk and dairy products and their marketing in neighboring countries; and, b) the enhancement of the dairy processing component of the value chain in order to accommodate increased raw milk production and the demand created by increased local consumption and export marketing of Rwandan dairy products; and Consideration should be given to a follow-on dairy value chain activity that will seek to preserve the achievements of the RDCP II and address a new phase of dairy value chain programming by: a) strengthening the management and marketing capability of the MCCs; b) assuring the readiness and capacity of the RNDP to Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 81 Bank) in 2012. Those loans have been restructured and repayment should begin within the next three months, but their prior poor performance, due to what the banks consider poor management by the MCCs, has created an element of caution, if not fear, in the investment community. The other constraints as stated by financial institutions include but not limited to potential leakage of the contract dairy market and poor impressions of the financial expertise of within MCCs. That said investment costs need to be controlled as ROI is not a necessarily linear measurement. assume responsibility and take on the challenges associated with supporting and advocating for the Rwandan dairy industry; c) promoting local consumption of Rwandan milk and dairy products and their export to neighboring countries; and, d) further strengthening of the capacity to process Rwandan milk and dairy products. Land O’ Lakes should investigate the relationship between higher quality offerings vis a vis their increased availability and impact on demand. The $5,000,000 target in FY 2015 in new private-sector investment in the dairy value chain be revised in light of the current climate for investment. Building on findings in the Rwanda Dairy Strategy and Deep Dive Investment Analysis, a formal and concerted effort should be undertaken to explore the actual potential for the capitalizing on the demand for Rwandan dairy products in neighboring countries. The RDCP II has thirty (30) indicators: fifteen (15) Feed the Future (FtF), twelve (12) custom, i.e. Land O’Lakes indicators, and three (3) indicators for the measurement of gender equity and equality. Some indicators are more “important” than others in terms of measuring project achievement, but each indicator has a target. Our opinion is that the vast majority of RDCP II targets, particularly those dealing with milk producers, MCCs, persons and organizations adopting technologies are achievable. There has been discussion regarding additional indicators and, therefore, targets for the “nutrition” objective of the program. Whenever those indicators and targets are formulated, they should address and measure “behavior change” and not “nutrition” because: a) increased milk consumption and dairy products by itself will probably not show improved nutritional status of participants; and, b) because results demonstrating improved nutritional status due to increased consumption of milk and dairy products, even if Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 82 they were to show improved nutritional status, would require a period of time to manifest themselves much longer than that remaining in the life of the program. Land of Lakes and USAID should reduce, where possible, the number of Feed the Future indicators associated with this study. a. Additional indicators associated with a nutrition objective and the USAID Nutrition policy should not be inserted into the program at this time. The possibility exists for adding a nutritional dimension to future follow-on studies. b. If additional indicators are added, care should be exercised so that LOL managers and staff do not change their approach – the value chain and dairy business should remain their focus. Change experienced by beneficiaries Changes in skills, knowledge, yield, and income Findings based on amounts reported by farmers and farm records corroborate RDCP II reporting of increases in milk yield and raw milk production of upwards of 30%, increased income of more than 40% by participating farmers, and a significant improvement in the quality of milk (see above), all directly attributable to the technology transfer, training, and capacity building provided by the RDCP II. The team also found that the Seal of Quality has had a major influence on producers, processors, government officials, Service Providers (SPs), and other stakeholders. Interviews and RDCP II seems to have a positive effect on skills of locals while influencing both behavior and attitude. Whenever additional indicators and targets are formulated, they should address and measure “behavior change” and not “nutrition” as stated above. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 83 discussions at all levels in the field indicated that fully 100% of farmers and local authorities were aware of the SOQ. Changes in behavior Interviews with eighteen (18) Model Farmers indicate they are sharing their new knowledge, technologies, and skills with other farmers; the Model Farmers have trained 1,837 farmers over the past 1½ years, of which 832 were male and 1,004 were female. One farmer told us that prior to working with the RDCP II, he “didn’t know that water was important for his cows and for milk production”; another was building a new, improved cattle shed based on RDCP II recommendations; others spoke about the importance of keeping the milk shed clean and wearing clean clothes while milking. The team heard many similar stories. Changes in attitude The team found that the Seal of Quality has had a major influence on producers, processors, government officials, Service Providers (SPs), and other stakeholders. Interviews and discussions at all levels in the field indicated that fully 100% of farmers and local authorities were aware of the SOQ. QUESTION 3: Contribution of New Management Structures to Investment and Improvement in Milk Quality RDCPII hired INSPIRED to research the dairy value chain in Rwanda’s four main milk sheds, recommend potential feasible financial products that would underpin the active engagement of Rwanda’s financial institutions in this sub-sector. Financial institutions revealed that INSPIRE timeframe was too squeezed and much ambitious to leave space for the model contextualization and adoption. This resulted into poor and reluctance The present staffing structure is adequate to the management and monitoring of the program in its present configuration and operational modality. Staff changes will depend on the program focus over the next twenty-two A concerted effort should be undertaken to strengthen MCC managers in terms of their capability and skills in dairy as a business, and in marketing. The formation of a cooperative, union, or umbrella organization of the MCCs to advocate for and provide skill training to MCCs lacking the wherewithal to hire professional staff and management should be Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 84 participation of the initial targeted financial institutions. The Agri-Relationship Manager from the Banque Populaire du Rwanda (BPR Ltd) said their MoU with INSPIRE was only drafted and they’re still waiting for INSPIRE for more development. UOB and KCB’s heads of agriculture finance identified the lack of a participatory approach with INSPIRE as a barrier to forming a cooperative relationship. Fortunately, UOB Ltd and KCB ltd have been able to develop the dairy financing product with direct partnership of Land O’ Lakes (though this activity is not really linked to INSPIRE work) backed by guarantee loans facilities from other sources. Contribution of New Management Structures to Investment and Improvement in Milk Quality We discussed “new management approaches” with LOL who assert that they have made minor adjustments to program staffing, but nothing major in that regard. Contribution of Seal of Quality to program achievement We found the SOQ, including mastitis control, and the dissemination of “best practices” in the handling and transport of raw milk, to be the single most important program activity or component in terms of the improvement of the quality of the raw milk being produced. (22) months. The complexity of the project necessitates a dedicated individual or team specifically concerned with maintaining data integrity. explored. LOL should limit the scope of responsibility for the full-time M&E specialist to purely activities associated with monitoring and evaluation, especially dealing with data quality assurance protocols. A behavior change communications specialist should be added to staff to guide and monitor an intensified Shisha Wumva education campaign; and, last, expertise in marketing and dairy as a business should be brought in to assess and develop plans for cross-border trade and to provide training to improve the business and marketing skills of MCC managers. Initiate discussions with the Rwanda Standards Board (RSB) to elicit their participation in the program in terms of enforcement of the Ministerial Instructions and the inclusion of “finished” dairy products in the commodities they monitor. Question 3: Implementation approach to scaling-up technology Sixty-two percent of farmers participating in RDCP II interviewed (62%) are using AI services and “super-semen,” which based on our analysis is associated with a difference in income of 306,290 RWF per year. Mastitis control is a major factor contributing to the improved quality of the raw milk produced; and, the RDCP II collaboration with the University of Rwanda and the University of California/Davis (UC/Davis) is showing great promise in terms of its outreach and effectiveness. Due to the shortage of The technologies that have the greatest potential benefit to increase and sustain gains from RDCP II if scaled-up are Artificial Insemination (AI), mastitis control, and technologies generally associated with milk quality improvement. The uncertain supply of Program monitoring should be strengthened due to the large beneficiary population, the high number of indicators, and the large quantity of results data being generated and handled by a variety of external entities, among them MCCs and SPs; and, verification of data points should be installed along the data chain to insure the accuracy and purity of the data. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 85 available land and zero grazing policies in some places, MCCs will be required to provide high quality feed to farmers; however, farmers need to be provided technical assistance in the production of high-quality feed and forage as well as the skills necessary to produce their own silage and hay and instruction in the use of such crop residues as feed. The team notes, however, that AI is dependent on the availability of affordable high-quality semen and hormones, the supply of which is currently controlled by the GOR. There is a considerable body of evidence that semen and hormones are regularly unavailable, hindering the ability of farmers to obtain AI services and constraining the use of this technology, let alone its scaling-up. high-quality semen and hormones could potentially limit the ability of RDCP II to effectively scale up AI. The possibility of scaling up the number of beneficiaries within the ZOI should be explored and, if feasible, outreach techniques and strategies should be developed and employed in that regard. The reported regular and ongoing unavailability of semen and hormones and privatization of that service should be taken up with the GOR by Land O Lakes and other key players in the industry including RDFA. Artificial Insemination (AI), mastitis control, and technologies generally associated with milk quality improvement should be scaled up. Question 4: Beneficiaries reached by adoption of technology and diet behavior change Adoption of technology The increase in raw milk production and the improvement in its quality strongly suggests that the technologies offered by RDCP II are being adopted by program beneficiaries. This suggestion is backed up by our survey of farmers, in which between 39.1 (n=36) and 89.1% (n=82) of participants reported using each included technology with each technology recording an average of 69.3% (n=64) of participants. Diet Behavior Change Based on interviews with industry actors and GOR officials, there is an across-the￾board heightening of awareness of the importance of cleanliness and quality in milk produced and consumed due to the Seal of Quality and the Shisha Wumva campaigns. The Shisha Wumva campaign seems to be everywhere – on billboards, radio, TV, and in the form of a calendar on the desk of most of the people interviewed; and, is reportedly reaching large numbers of people. Its effectiveness in terms of diet behavior change, however, is relatively unmeasured. The technologies, overwhelmingly, are reaching and being used by the target population of the program. The Shisha Wumva outreach campaign has reached an impressive section of the population, though its impact is still unclear because it has yet to be evaluated. It was not possible to investigate why non￾participants were not aware of the program being implemented in their community. The Shisha Wumva milk consumption education outreach campaign should build upon success of reaching a large number of potential participants. Implementing partners would be wise to incorporate behavioral change communications and strategies into the program. Future evaluators should further investigate the reasons for a lack of awareness with the ZOI of several MCCs participating in RDCPII. This should inform future outreach strategies to identify potentially hard to reach populations and to inform outreach strategies. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 86 The team included a question on milk consumption by the household in the survey and found a correlation in households that had heard Shisha Wumva messages and increased milk consumption. Those households consumed an average of 35% more milk than those who hadn’t heard the messages. The continued patronization by consumers of rural milk kiosks and other informal channels indicates that much work remains to be done in terms of educating consumers on the importance of purchasing high-quality, unadulterated milk and dairy products. The team was surprised to find that dairy farmers in the immediate vicinity of some MCCs had no knowledge of the RDCP II, so the possibility to reach a larger beneficiary population within the ZOI may be possible. Question 5: Opportunities for new partnerships Land O’Lakes The evaluation team did not uncover opportunities for new partnerships, mainly due to time constraints for data collection and analysis. That said, Land O’Lakes is discussing a collaborative effort with USAID-funded Global Communities (GC) through which the latter would use informational materials on best practices in milk handling produced by RDCP II, specifically the pamphlet “Ubuziranenge Bw’amata,” which outlines best practices in comic book form. GC is working with vulnerable dairy farmers in four (4) districts, two (2) of which overlap with the RDCP II ZOI, and two (2) that do not, implementing literacy, community health, and kitchen garden projects (nutrition). A partnership with GC, whereby LOL supplies the educational materials on milk collection best practices and GC disseminates those materials would allow for the expanded outreach of the materials over a wider area to more and more vulnerable people at relatively low cost. Land O’Lakes has done an incredible job forming working relationships with actors throughout the agricultural value chain and should be commended. The possibility for a relationship between Land O’ Lakes and Global Communities could expand the reach of the program to marginalized populations in the ZOI. While the current responsibilities of the RSB were created intentionally, there may Land O’Lakes should investigate ways to strengthen collaboration between various actors in monitoring compliance with the Ministerial Instructions. The channeling of RDCP II funds through financial institutions should be explored as a means to create relationships between formal financial institutions and program beneficiaries. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 87 One potentially problematic issue that came up in an interview with a member of the Rwandan Standards Board related to the project. The point was made that they are only consulted in “finished” dairy projects and were not involved in monitoring the inputs. LOL and USAID has made the point that this was an intentional delineation of responsibilities with numerous actors having responsibilities Financial Institutions In interviews with financial institutions, the point was made that channeling funds through local financial institutions could potentially create relationships between formal financial institutions and program beneficiaries. Furthermore, the increased contact between the aforementioned entities might serve to generate other business and would relieve RDCP II of that responsibility. A model for this sort of relationship may be a “check off” system, in which an MCC would house a store which would allow members to take agricultural inputs on credit and to repay their debts using the profits after the sale of their goods. be a need to address their expected role in the MI directly. The possibility of forming relationships between institutions and program beneficiaries shows promise and support on the part of financial institutions. A check￾off system should be considered in this regard. Question 6: Influence of the RDCP II and Seal of Quality at the institutional level At the institutional level, the RDCP II can legitimately claim a large degree of responsibility for initiating and implementing several key milestones or markers in the strengthening of the Rwandan dairy value chain and industry. The Seal of Quality program component itself, the development of the National Dairy Strategy (NDS), the creation with the Private Sector Federation (PSF) and nurturing of the Rwanda National Dairy Platform (RNDP), the Shisha Wumva milk consumption education and behavioral change program component and campaign, and, the forthcoming Ministerial Instructions on the Preservation, Collection, Transportation, and Selling of Milk, approved by MINAGRI and, at the time this report was being drafted, awaiting At the institutional level, both RDCP II and the SOQ have been well received by the GOR and other stakeholders. The GOR has shown itself to be an enthusiastic partner in the implementation and institutionalization of both the program and the SOQ. LOL should continue to nurture the relationship between RDCP II and the GOR, which has significantly contributed to the overall effectiveness of the SOQ. The GOR must enforce the Ministerial Instructions on the Preservation, Collection, Transportation and Selling of Milk and establish a regulatory and inspection system to ensure the high quality of Rwandan milk and dairy products. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 88 signature in the office of the Prime Minister represent extraordinary accomplishments. The Seal of Quality is now universally recognized by actors throughout the Rwandan dairy value chain and industry as the driver and impetus for the safety and cleanliness of Rwandan milk and dairy products and, therefore, critical to the export marketing of high-quality Rwandan dairy products and their increased local consumption. The Seal of Quality, along with the transfer of on-farm technology to dairy farmers, again, more than other program components, have had the most profound effect on the value change and spurred change in the handling and cleanliness of raw milk. This achievement is not readily quantifiable, but becomes apparent when discussing the RDCP II with actors throughout the dairy value chain and industry. In addition, the evaluation team spoke to scores of farmers and members and staff of MCCs outside Kigali and to individuals from both the private sector and the government of Rwanda (GOR) in Kigali. A full list of contacted parties is available in Appendix D. Within the GOR, the SOQ was consistently a significant part of the conversation in terms of recognition of its importance in not only assuring the cleanliness of milk and dairy products for local consumption, but as an indicator of the high quality of Rwandan dairy products and the significance of high quality for export marketing. It is the SOQ that differentiates Rwandan products from those from other countries; and, the importance of the SOQ has not been lost on the GOR. Question 7: What RDCP II activities have the potential for sustainability after the program closes As for continuation of the RDCP II approach, the MCC’s Training of Trainers (TOT) and Model Farmers activities appear to have the most promise as implementation modalities or vehicles once the program has ended. On the institutional Among the most important accomplishments of RDCP II is the facilitation of The relationships that have been formed through RDCP II should continue to be a point of emphasis for LOL. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 89 level, the Private Sector Foundation (PSF) and LOL have formed the Rwanda National Dairy Platform (RNDP), which already has offices and equipment provided by a grant from the RDCP II, and for which there is currently ongoing a search for a permanent in-charge. The RNDP is designed to bring together the key players in the Rwandan dairy value chain and industry and provide the advocacy and support necessary for the continued strengthening and advancement of the industry. LOL has worked collaboratively with PSF to create a 5 year strategic plan to handle the transition between RDCP II and RNDP, the roll out of which started last year. The team’s discussions with key actors in the dairy value chain uncovered a universal concern for the preservation of both the services and benefits provided by the RDCP II once the project comes to an end in January 2017. The team visited the Agira Gitereka MCC and found that although they had capacity for 7,500 litres of milk daily, they were only producing 400 litres. Model Farmers within the MCC claimed that, although the technological investments could be effective, farmers had lost trust in the MCC and were instead selling their milk on alternative markets. This points to a serious threat to sustainability of the gains of RDCP II; the entropy of the productive relationships that have been formed as part of the program thus far. As discussed in previous sections, the team believes that business development services are integral to the continued (and potentially increased) success of RDCP II. Competent management would undoubtedly contribute to the sustainability of the program. cooperative relationships throughout the agricultural value chain. The transition between RDCP II and RNDP in January 2017 is a source of anxiety for some stakeholders in the agricultural value chain LOL should continue efforts to improve business knowledge within MCCs. The Ministerial Instructions on the Preservation, Collection, Transportation and Selling of Milk must be enforced and a regulatory and inspection system to ensure the high quality of Rwandan milk and dairy products should be put in place. Question 8: Gender equity and equality The Women’s Empowerment in Agriculture Index (WEIA) framework was Though our sample may not have been The lack of control over bank accounts for some women in the Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 90 developed as part of the Feed the Future Initiative and represents the first “comprehensive and standardized measure to directly capture women’s empowerment and inclusion in the agricultural sector.” The baseline report was published last year and Rwanda scored a respectable .91 rating (out of 1), which was the highest score reported in Africa and the second highest overall. The study estimates that 70 percent of women in the agricultural sector in Rwanda have achieved adequate employment and 73 percent have achieved gender parity. The evaluation team was unable to fully utilize the WEIA framework by replicating the baseline specifically for the dairy sector analysis because such an investigation would require a much larger sample than was feasible to access under the short timeline of the study. The analysis therefore speaks to directly observable consequences of empowerment and inclusion that could be ascribed to RDCP II. Gender mainstreaming has been a major consideration throughout RDCP II and LOL has done an admirable job promoting female participation in the dairy sector, which in many societies is a male dominated field. Common Constraints for Women in Dairy Jobs along the agricultural value chain are often gendered. Women are frequently limited to work in what are deemed feminine tasks because of gendered divisions of labor such as the cleaning or food preparation, while men are given responsibility for tending the cattle and the transportation of goods to the market. These jobs not only vary in the amount compensated, but in many cases control over the handling of money. The Rwandan Dairy Sector Context Gender equality and equality is a right enshrined in the Rwanda constitution and the government has made significant efforts to improve the plight of women in representative of the entire population, some participants believed that women had less access to formal institutions then men. Land O’Lakes and the GOR should be commended for the attention paid to gender equity and equality under RDCP II. That said, women are still only make up 1/3 of beneficiaries and there is room for increased participation. Training programs undertaken by SPs and MFs have done an admirable job of ensuring equal participation between men and women. sample should be further investigated in a larger study. Land O’Lakes and the GOR should continue outreach activities, especially encouraging women in positions of leadership within cooperatives. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 91 agriculture. According to MINAGRI’s Agricultural Gender Strategy, up to 86% of work completed in the agricultural sector as a whole is completed by low income women, who are often limited to subsistence farming in part due to low levels of literacy (this concern is shared by LOL). Within the dairy sector, this has often meant that women are tasked with traditionally gendered positions and have lacked control over finances. Our sample seemed to be in line with previous findings regarding the control over income within Rwandan families, although woman-headed households are underrepresented (the Agricultural Gender Strategy suggests that 30% of households are female-headed while our sample only shows 15%). Perhaps due in part to the composition of our sample, the team identified themes in KIIs that suggested that women were not satisfactorily engaged in household decisions. Participants spoke of a difference between access to a bank account and control over the funds within. The comment was made that our female participants did not have access to formal financial institutions and so when payment passed through the banks, it became the purview of the males in the household. Access to Technology Based upon data provided in LOL’s FY2014 Q4 report, women make up 35.6% of beneficiaries of AIs, which as previously established appears to offer one of the highest ROIs of available technologies. This is roughly in line with reported gender participation in the program. Access to Training LOL has made a concerted effort to involve women in training programs and based on our document review have been remarkable successful in encouraging not only participation in training but leadership within cooperatives. The table below shows that a full 40% of participants in Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 92 reported training sessions were female and encouragingly in the districts of Musanze and Rulindo women actually make up the majority of participants. Participants in KIIs expressed a great deal of satisfaction with the outreach activities conducted by GOR as part of RDCP II and have acknowledged that the cooperatives serve as a platform in which they are able to freely express their views. At the end of FY 2014, LOL reported that 38 percent of cooperative members were women and that the percentage has been growing. According to LOL, 76% of women participating in these training sessions report increased self-efficacy. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 93 Appendix G. COPIES OF INSTRUMENTS DEVELOPMENT & TRAINING SERVICES, INC. (dTS) PERFORMANCE EVALUATON OF THE RWANDA DAIRY COMPETITIVENESS PROGRAM II (RDCP II) FINANCIAL INSTITUTIONS QUETIONNAIRE We are from dTS, an organization that has been asked to perform an evaluation of the performance of Rwanda Dairy Competitiveness Program II (RDCP II), financed by the United States Agency for International Development (USAID) and managed by Land O’Lakes, Inc. This questionnaire will help dTS understand how farmers supplying milk to MCCs have benefited from the activities of RDCP II. The information collected will then be used to provide feedback to Land o’ Lakes and USAID on the progress of RDCP II to the present, and will help RDCP II to plan activities and make the best use of project resources for the time remaining to the project. Please answer the questions as accurately and honestly as possible so that future activities will accurately address the current state of the dairy sector in Rwanda and challenges faced by dairy farmers like yourself. The information collected will be used by dTS and will remain strictly confidential. Are you willing to participate? Yes / No. Thank you Respondent No: …………… Name of Interviewer Signature Date Interview Completed Supervisor’s Name Signature Date Checked Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 94 Baseline interview Questionnaire: Financial Institutions. Name of the respondent: ………………………… Position of the Respondent within the Financial Institution: ………………………………………………. Service: …………………………………………………….. Office: Head Office/Branch/Sub-Branch/Outlet Location: Province …………………..and District: --------------------------------------- SECTION A: BASIC INFORMATION. A1. Name of the Financial Institution: ………………………………………… A2. Status of the Financial Institution: Sacco……………..Micro-Finance ………………. Bank: ………………………………… A3. Do you have a dairy loans portfolio? A4. If yes, when did the portfolio begin? A5. Was there a partnership with RDCP II in the development or implementation of your dairy portfolio? A6. If yes, what was the nature of the partnership? Policy development? ................... Capacity building? ……………. Grant? ................... A7. How do you value RDCP II contribution to the development of your dairy portfolio? SECTION B: DAIRY LOAN PRODUCT PROFILE Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 95 B1. How many clients do you have in your dairy loan portfolio? ------------------------- Please provide number on males clients ………………… and females ----------------- B2. What is the total value of your dairy loan portfolio in terms of Rwfs ? B3. How do you reach-out with your dairy loan product? (e.g. through farmer cooperatives, MCCs, Land o’ Lake, dairy farmer umbrella, financial institution marketing activities, any other channel?) B4. What is the profile of the dairy loan product: Maximum loan period? ……………………. Repayment frequency? --------------------------- interest rate?------------------- B5. Please provide the volumes of dairy portfolio during the following end periods (in Rwfs)? 2011 2012 2013 2014 B6. What are the dairy portfolio risks that you are facing to date? B7. If yes, do you have a mechanism to address those risks? B8. What is the loan delinquency rate, if any ---------------- B9. If yes, what the non-performing loan (NPL) rate? .............. SECTION C: DAIRY FINANCE MANAGEMENT C1. What are the positions of staff managing this dairy financing?: Customer care officers? -------- loan officers? ……… Commercial/business officers? -------- Senior managers? -------------- Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 96 C2. How many staff members manage your dairy loan portfolio? SECTION D: DAIRY FINANCE PROSPECTS D1. Please project the volumes of your dairy portfolio volumes (Rwfs) for the following three years? 2015: 2016: 2017: D2. What are your the opportunities to increasing dairy financing; and what, if any, are the obstacle to increasing dairy financing? D3. Do you plan on expanding your dairy loan program in the dairy industry; and what do you see the role of RDCP II? THANK YOU VERY MUCH FOR YOUR TIME AND COOPERATION. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 97 DEVELOPMENT & TRAINING SERVICES, INC. (dTS) PERFORMANCE EVALUATION OF THE RWANDA DAIRY COMPETITIVENESS PROGRAM II (RDCP II) FOCUS GROUP DISCUSSION (FGD) QUESTIONNAIRE We are from dTS, an organization that has been asked to perform an evaluation of the performance of Rwanda Dairy Competitiveness Program II (RDCP II), financed by the United States Agency for International Development (USAID) and managed by Land o’ Lakes, Inc. This questionnaire will help dTS understand how farmers supplying milk to MCCs have benefited from the activities of RDCP II. The information collected will then be used to provide feedback to Land o’ Lakes and USAID on the progress of RDCP II to the present, and will help Land o’ Lakes and USAID to plan activities and make the best use of project resources for the time remaining to the project. Please answer the questions as accurately and honestly as possible so that future activities will accurately address the current state of the dairy sector in Rwanda and challenges faced by dairy farmers like yourself. The information collected will be used by dTS and will remain strictly confidential. Are you willing to participate? Yes / No. Thank you Focus Group No: …………… Name of Interviewer (s) Signature Date Interview Completed Supervisor’s Name Signature Date Checked SECTION 1. MEMBERSHIP IN MCC 1.1 What motivated you to become a member of this MCC? • Financial Gains • Loans • Grants • Business development services (BDS) • AI Service • Feeds and supplements • Health (vet services) • Market and Marketing Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 98 • Technical assistance? • Training • Quality control • Farm management • Material assistance • Milk cans • Transportation • Feeds • Vet products (drugs etc.) • Other (Specify) 1.2 Have your expectations been met as a member? Yes----------- No -------- 1.3 If not, why? SECTION 2. PARTICIPATION LEVEL AT MCC 1.1 Has decision-making concerning the management and functioning of the MCC changed due to the intervention of the RDCP II? If so, how? • Election of Board members • Milk price determination • Investment 1.2 How will you rate your level of participation in the MCC affairs before the RDCP II: 1.3 Has your level participation changed since the introduction of the RDCP II project? Yes------ No------. If Yes, How? 1.4 Tell us about women and men’s decision making powers: Before the project: After the project: 1.5 In what ways has the project influenced women and men’s access, ownership, and control over income? 1.6 How do you compare the role of men and women’s social networks and community leadership before and after the project? 1.7 How has the work load of women and men changed? Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 99 RWANDA DAIRY COMPETITIVE PROGRAM II (RDCP II) dTS DAIRY FARMERS HOUSEHOLD SURVEY QUESTIONNAIRE ______________________________________________________________________________ ______________________________________________________________________________ _______ We are from dTS, an organization that has been asked to perform an evaluation of the performance of Rwanda Dairy Competitiveness Program II (RDCP II), financed by the United States Agency for International Development (USAID) and managed by Land O’Lakes, Inc. This questionnaire will help dTS understand how farmers supplying milk to MCCs have benefited from the activities of RDCP II. The information collected will then be used to provide feedback to Land o’ Lakes and USAID on the progress of RDCP II to the present, and will help RDCP II to plan activities and make the best use of project resources for the time remaining to the project. Please answer the questions as accurately and honestly as possible so that future activities will accurately address the current state of the dairy sector in Rwanda and challenges faced by dairy farmers like yourself. The information collected will be used by dTS and will remain strictly confidential. Are you willing to participate? Yes / No. Respondent No: …………… Name of Enumerator Signature Date Interview Completed Supervisor’s Name Signature Date Checked SECTION I: IDENTIFICATION PARTICULARS I .1. Province…………………………………………………………… Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 100 I .2. District………………………………… Sector …………………………………. I .3. Cell …………………………………… Umudugudu …………………… I.4 Names of Respondent …………………………………………………… Are you the farm owner? Yes No If no what’s the name of the owner? ……………………………………… Farmer contact phone: ……………………………………………. Sex: Male Female Is the farmer married? Yes No If yes, do you live with your spouse? Yes No I.5. Type of household: Female no male ------------------------------------------- Male no female ------------------------------------------ Child no adults ------------------------------------------- I.6. Number of People in the household <18 _______ >18_____ I.7. Age of Farmer: <35------------ >35---------------- 1.8. How are decisions reached in the household on resource allocation? Income Use 1=Husband only___ 2=Wife only____ 3=Husband and Wife____ 4=whole Family_______ I.9. Wealth category of the household as assigned by Ubudehe wealth categorization Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 101 1=umutindi nyakujya/umutindi; the very poor/the poor. 2 =Umukene/umukene wifashije resourceful poor, 3 = umukungu jumba, money rich, 4 = umukire 1.10. Are you a member of a MCC? Yes __________ No ____________ 1.11 If yes, since when? Month_______ Year ___________ Name of the MCC ________________________________ What are the membership requirements for the MCC? 1.12 Were you trained by RDCP II? Yes No If yes, answer the following: # Type of training Venue When? (month/year) How long was it? (number of hours/days) How many times? Was it valuable? 1=Yes; 0=No Are you using it now? 1=Yes; 0=No If No, why? 1 2 3 4 5 6 7 8 SECTION II: NUTRITION AND MILK CONSUMPTION II.1 Are you aware of milk benefits for human body? Yes No If yes, which ones (tick those mentioned): 1. Child development 2. Mental development 3. Full of proteins 4 Full of minerals (calcium, phosphorous, magnesium, etc.) 5 Other, specify ______________________________________ Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 102 II.2 Has your household consumption of milk (or other dairy products) increased in the past 6 months Yes No If yes, how many liters of milk does the household consume on a given day? ----------------------------------- II.3 Have you heard about “Shisha Wumva” program? Yes No If yes, by which means of communication? Radio Television News paper Exhibition Meeting Fellow farmer Other, Specify _____________ SECTION III: DAIRY PRODUCTION III.1 Milk Production and sales III.1.1. With reference to the last 16 months (since January 2014), please provide information related to your farm on the following: Number of cows / Milk production Rainy season Dry season Rainy Season Dry Season Mar-May 2014 Jun-Aug 2014 Oct-Dec 2014 Jan-Feb 2015 Total number of cows 1. Number of cows in milk ( inka zikamwa) Ankole (Inyarwanda) Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 103 Cross-breed (ivanze) Frisian (frisonne) Jersey (jerise) Other Total number of cows in milk 2. Liters of milk produced ( per day: morning and evening) Ankole Cross-breed Frisian Jersey Other Total Liters produced 3. Liters of milk consumed at home or given for free to neighbors Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 104 ( per day) 4. Liters of milk Given to calf ( per day)_Ayahawe inyana 5. Liters of milk home processed to make ghee ( average per day) 6. Liters of milk spoiled (yangiritse) Quantity of milk sold ( per day: morning & evening) to: Rainy season Dry season Rainy Season Dry Season 1. Neighbors 2. MCC (ikusanyirizo) 3. Traders (abacuruzi, transporters) 4. Kiosks 5. Private / public institutions 6. Other: specify Congo / Burundi / Uganda_______________ ___ Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 105 TOTAL III.1.2 What was the price of sold milk per liter? Price of milk given to farmers by (per liter): From Oct 2013 onward Rain season Oct –Dec 13 / Mar-May 14 Dry Season Jan-Feb & Jun-Aug 14 1. Neighbors 2. MCC 3. Traders 4. Kiosks 5. Private /public institutions 6. Other: specify Congo, Burundi, Uganda, Tanzania III.1.3 Other sources of income at farm level (See Musafiri Table Source of income Estimate (Rwf) Per month Estimate (Rwf) Per quarter Total Estimate (Rwf) Per year Write farmers general comment 1. Sale of cow manure (ifumbire) Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 106 2. Sale of cattle 3. Bull rent for natural breeding (gukodesha imfizi) 4. Sale of fodder seeds ( imbuto z’ubwatsi) 5. Sale / rent of fodder plots (gukodesha umurima w’ubwatsi) 6. Other III.2. Cost of production III.2.1. Feeding What type of feeds do you give to your cows and what was/is the cost for each type/cow or per herd? Type of feed Used? If yes write “1” If no leave blank Unit of measurement Quantity fed per month Unit price/cost (Rwf) Total (Rwf) Month Total (Rwf) Quarter Total (Rwf) Year 1. Forage (ubwatsi,.) 2. Legumes / leguminous (ibinyamisogwe) 3. Grain (ibinyampeke) 4. Farm-made concentrates Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 107 (imvange y’ibinyampeke) 5. Water 6. Molasses (umushongi) 7. Urea for animal feeding 8. Minerals (imyunyu) 9. Vitamins 10. Brewers’ waste ( ibisigazwa byo mu ruganda..) III.2.1. Other costs Please indicate, where appropriate, the costs incurred on the following items? Cost item Used? If yes write “1” If no leave blank Unit price Estimate per month (Rwf) (if applicable) (Rwf) Estimate per Quarter(Rwf) (if applicable) (Rwf) Total Estimate per year (Rwf) A. Vet services 1. Spraying against ticks (koza, kuhagira, gufuhirira…. 2. Deworming (kugabura umuti w’inzoka) 3. Vaccination (gukingira inka…) 4. Drugs and vet products ( imiti, ….) Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 108 B. AI services (gutera intanga) C. Transport D. Dairy Labor E. Other services Section IV: Technology Application IV.1 Applying improved technologies and management practices Which technologies were you able to implement due to RDCP II intervention (after training, technical support, in-kind support, staff advises, etc…..) Introduced Technologies (Ikoranabuhanga, ubumenyi bwo mu mahugurwa, cg inama yahawe…., ubumenyi bushya, etc…., ibyo yungutse akabishyira mu bikorwa…….) Use “ 1 ” for yes Leave “blank” for No Write farmers general comment for not implementing 1. Silage ( guhunika ubwatsi bubisi..) 2. Hay ( kubika ubwatsi bwumye….) Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 109 3. Crop residues (ibisigazwa by’imyaka..ibigorigori, ibishogoshogo……) 4. Legumes hay 5. Farm made concentrates ( kuvanga ibyo kurya by’amatungo) 6. Purchased concentrates 7. Molasses ( umushongi) 8. Spraying against ticks 9. Deworming umuti w’inzoka 10. Vaccination a. gukingira 11. Mastitis control (ifumbi…) 12. AI services Gutera intanga 13. Milking hygiene ( isuku y’amata..) 14. Milk quality improvement ( Kongera ubuziranenge bw’amata…….(SoQ_Seal of Quality…..) 15. Record keeping ( kubika inyandiko….) 16. Financial services (loans) IV.2 Additional information on application of technologies IV.2.1. Cow Breeding Performance Average lactation period (Igihe imara Type of cows Before Intervention After Intervention (Project) Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 110 yonsa) (Project) Ankole (Inyarwanda) Cross-breed (ivanze) Frisian (frisonne) Jersey (jerise) Other Calving Interval Gukurikiza) Ankole Cross-bred Frisian Jersey Other Age at First Breeding (Ibanguriwe) Ankole Cross-bred Frisian Jersey Other Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 111 IV.2.2. Artificial Insemination Do you use AI? Yes___ No___ If not why? SECTION V: ASSISTANCE FROM MCC. V.1 Did you receive any assistance the MCC? If yes, state what Type of assistance Value (RwF) 2012 VALUE(FRW) 2013 Value (FRW) 2014 Value (FRW) 2015 Transport Milk Cans Structure Totals SECTION VI: JOB CREATION Job creation ( record employee name as many times as he/she was hired at different times, if applicable) Name of employee (Optional) Sex ( M, F) Age Type of job (short description Date starting job (from Oct 2012 onward) Date end of job Estimate of job duration in months ( job lasting Payment per month (in Rwf) Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 112 ) at least 1 month) THANK YOU VERY MUCH FOR YOUR TIME AND COOPERATION. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 113 DEVELOPMENT & TRAINING SERVICES, INC. (dTS) PERFORMANCE EVALUATION OF THE RWANDA DAIRY COMPETITIVENESS PROGRAM II (RDCP II) MILK COLLECTION CENTER (MCC) QUESTIONNAIRE We are from dTS, an organization that has been asked to perform an evaluation of the performance of Rwanda Dairy Competitiveness Program II (RDCP II), financed by the United States Agency for International Development (USAID) and managed by Land O’Lakes, Inc. This questionnaire will help dTS understand how farmers supplying milk to MCCs have benefited from the activities of RDCP II. The information collected will then be used to provide feedback to Land o’ Lakes and USAID on the progress of RDCP II to the present, and will help Land o’ Lakes and USAID to plan activities and make the best use of project resources for the time remaining to the project. Please answer the questions as accurately and honestly as possible so that future activities will accurately address the current state of the dairy sector in Rwanda and challenges faced by dairy farmers like yourself. The information collected will be used by dTS and will remain strictly confidential. Are you willing to participate? Yes / No. Thank you Respondent No: …………… Name of Enumerator Signature Date Interview Completed Supervisor’s Name Signature Date Checked Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 114 Province: ------------------------------------------------------------- Name of Milk Collection Center (MCC): ---------- -------------------------------------------------------------------------------------------------------------------- Position of the Respondent: (i) Owner/Manager/Chairman -------- (ii) Secretary --------- (iii) Others (Specify) ---- Location/Umidugudu (Village): --------------------------------------- SECTION A: BASIC INFORMATION. A1. Date of establishment/creation of MCC---------------------------------- A2. Number of active members --------------- Males ---------Females A3. Do your pay a membership fee? Yes-------------- No--------- If Yes, How much? ---------------------- A4. Do your members pay an annual subscription? Yes ------ No------- If Yes, how much? ------------------------------ A5. Do you have employees at the MCC? Yes------ No--------- If Yes, what is the number of paid employees: (i) Permanent employees ------------ Male------------Female ------ (ii) Temporary/casual employees --------- Male ---------- Female -------- A6. Please indicate number and value of assets owned by the MCC: ASSET NUMBERS CURRENT VALUE RDCP II contribution Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 115 (rwfs) (Rwfs) Building (s) Vehicles Cooling tank(s) Milking cans A7. What was the source of the initial capital for: Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 116 (a) Buildings? (i) Members’ contributions-------- (ii) Grant (specify) -------- (iii) Individual money lenders ------ (iv) Bank --------(v) Others (Specify) ------------ (b) Equipment? (i) Members’ contributions ------------- (ii) Grant (specify) ------------------ (iii) Individual money lenders ------------------ (iv) Bank & MFI------------ (v) Others (Specify) --------------- SECTION B: MILK COLLECTION, STORAGE AND QUALITY CONTROL. B1. Please indicate quantities of raw milk collected and prices paid to farmers during the rainy and dry season. SEASON Total milk quantity collected % of milk collected by grades (if applicable) Price paid (RwF) Price paid by grade (if applicable) (RwF) Total Sales (RwF) A B Lower grade A B Lower grade Rainy/wet 2012 Dry 2012 Rainy/wet 2013 Dry 2013 Rainy/wet 2014 Dry 2014 B2. What is the capacity of your milk collection/storage/cooling facilities per day? --------------------------------- -liters/day B3. Do you incur losses due to spoilage of milk? Yes ------ No. -------- Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 117 If Yes, how many liters of milk are wasted due to spoilage? Liters/per day? ---------- Liters/week? -------------- SECTION C: MILK SELLING. C1. Please indicate quantities of milk sold and prices received from your buyers/outlets in the rainy and dry season B uy er Quantity sold (l) Price received (RwF) Income (RwF) 2012 2013 2014 2015 2012 2012 2013 2 0 1 4 2 0 1 5 2012 2013 2014 2015 W et D r y W et D r y W et D r y W et D r y W et D r y W et D r y W et D r y W et D ry W et D r y W et D r y W et D r y W et D r y C2. Which buyer(s)/outlets buy the must milk from your MCC? -------------------------------- C3. Do you have contractual agreements with buyers of your milk? Yes ----- No. - If Yes to C3, what is the nature of the agreement? : (i) Informal agreement based on mutual understanding --- ------------------ii) Formal written agreement with legal implications ----------- C4. What are the terms of the agreement? : (i) Specify milk price ------------- (ii) Specify quantity of milk ------- ------- (iii) Insist on quality of milk--------------, (iv) Specify frequency of supply-------------------- C5. How far is your furthest buyer from your milk collection point? ------------- Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 118 C6. What is the average distance to your buyers? …………. C7. Do you deliver milk to your buyers/customers or buyers/customers collect milk from the MCC? (i) MCC deliver milk to buyers--------------- (ii) Buyers collect milk from MCC----------- (iii) Both (i) and (ii) ----------------------- C8. If you deliver milk to your customers, what means of transport do you use? (i) Own vehicle -----------(ii)- Hired vehicle ------------- (iii)-Public transport ---------- (iv)-Others (Specify)------------------- SECTION D: COSTS AND CHALLENGES. D1. What is the major operational cost of your MCC? ---------------------- D2. Please indicate quantity and costs incurred for the following items? The same as above: it can be useful to get this info for 2012, 2013, and 2014 COSTS ITEM QUANTITY PER DAY OR MONTH COST PER UNIT (RWF) TOTAL COST (RWF) Milk purchase Labor Maintenance cost Electricity Water Transport Taxes Others (Specify) D3. What do you consider to be the main challenges/constraints facing your MCC in terms of costs? Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 119 D4. Did you receive a grant from RDCP II? If yes how much? Type of grant Value (FRW) 2012 VALUE(FRW) 2013 Value (FRW) 2014 Value (FRW) 2015 Machinery Car Building Totals E. PARTICIPANTS EXPECTATIONS E.1 Did the project meet your expectations? Yes---------- No------------ If no, why not? E.2 What would you like the RDCP II to do for you in order to better meet your expectations in milk sector? 1._______________________________________ 2._______________________________________ 3._______________________________________ 4.--------------------------------------------------------------- THANK YOU VERY MUCH FOR YOUR TIME AND COOPERATION. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 120 Appendix H. CONFLICT OF INTEREST DISCLOSURES Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 121 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 122 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 123 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 124 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 125 Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 126 Appendix I. STATEMENT OF DIFFERENCES TABLE Statement of Differences Table Implementing Partners Evaluation Team Drivers of Demand The Implementing partners have identified a lack of consumer preference within the dairy market in Rwanda. They support this through the existence of secondary markets and the amount of low￾quality milk products transported across the border. The evaluation team argues that the demand for higher quality dairy products has not yet been evaluated and that the possibility that the availability of higher quality options could create demand above and beyond what already exists. Role of MCCs The Implementing partners recognize MCCs exclusively as conduits for the sale of dairy products The evaluation team envisions an expanded role for the MCCs in which they will act as “hubs” for supplies and training for local dairy producers. Performance Evaluation of the USAID/Rwanda Dairy Competitiveness Program, RDCP II 1