EVALUATION Performance Evaluation of the Integrated Improved Livelihoods Program (IILP) in Rwanda [March 2015] This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Mendez, England & Associates. PERFORMANCE EVALUATION OF THE INTEGRATED IMPROVED LIVELIHOODS PROGRAM (IILP) Final Report March 2015 Prepared under RFTOP No. SOL-696-15-000001 Submitted to: USAID/Rwanda Submitted by: Colin Maclean, Team Leader Didier Nkubito, Gender and Nutrition Specialist Elie Nsabimana, Financial Services Expert Olivier Habimana, Data Analyst and Logistics Coordinator Contractor: Mendez England & Associates 4300 Montgomery Avenue, Suite 103 Bethesda, MD 20814 Tel: 301- 652 -4334 www.mendezengland.com DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government CONTENTS Executive Summary ........................................................................................................... i Evaluation Purpose and Evaluation Questions .................................................................................................i Project Background ................................................................................................................................................i Evaluation Design, Methods and Limitations ....................................................................................................i Findings, Conclusions and Recommendations .................................................................................................i Lessons learned ....................................................................................................................................................vii 1.0 Evaluation Purpose, Objective & Questions .......................................................... 1 1.1 Evaluation Purpose .................................................................................................................................... 1 1.2 Evaluation Objectives ............................................................................................................................... 1 1.3 Evaluation Questions ................................................................................................................................ 1 2.0 Project Background .................................................................................................. 2 3.0 Evaluation Methods and Limitations ...................................................................... 3 3.1 Evaluation Methodology........................................................................................................................... 3 3.1.1 Quantitative Research and Analysis ........................................................................................ 4 3.1.2 Qualitative Research and Analysis ........................................................................................... 4 3.2 Evaluation Limitations ............................................................................................................................... 5 4.0 Findings, Conclusions & Recommendations .......................................................... 6 4.1 Group A: Monitoring and Reporting .................................................................................................... 6 4.1.1 Question 1 .................................................................................................................................... 6 4.1.1.1 Findings ....................................................................................................................................... 6 4.1.1.2 Conclusions ............................................................................................................................... 7 4.1.1.3 Recommendations .................................................................................................................... 7 4.2 Group B: Efficiency and Effectiveness ................................................................................................... 7 4.2.1 Question 1 .................................................................................................................................... 7 4.2.1.1 Findings ....................................................................................................................................... 7 4.2.1.2 Conclusions ................................................................................................................................ 8 4.2.1.3 Recommendations .................................................................................................................... 9 4.2.2 Question 2 .................................................................................................................................... 9 4.2.2.1 Findings ....................................................................................................................................... 9 4.2.2.2 Conclusions .............................................................................................................................13 4.2.2.3 Recommendations ................................................................................................................14 4.2.3 Question 3 .................................................................................................................................. 15 4.2.3.1 Findings .....................................................................................................................................16 4.2.3.2 Conclusions .............................................................................................................................18 4.2.3.3 Recommendations ................................................................................................................18 4.2.4 Question 4 .................................................................................................................................. 18 4.2.4.1 Findings .....................................................................................................................................19 4.2.4.2 Conclusions .............................................................................................................................20 4.2.4.3 Recommendations ..................................................................................................................20 4.3 Group C: Impact and Relevance .......................................................................................................... 20 4.3.1 Question 1 .................................................................................................................................. 20 4.3.1.1 Findings .....................................................................................................................................21 4.3.1.2 Conclusions .............................................................................................................................23 4.3.1.3 Recommendations ..................................................................................................................23 4.3.2 Question 2 .................................................................................................................................. 23 4.3.2.1 Findings .....................................................................................................................................24 4.3.2.2 Conclusions .............................................................................................................................26 4.3.2.3 Recommendations..................................................................................................................27 4.3.3 Question 3 .................................................................................................................................. 27 4.3.3.1 Findings .....................................................................................................................................27 4.3.3.2 Conclusions .............................................................................................................................29 4.3.3.3 Recommendations ..................................................................................................................29 4.4 Group D: Sustainability .......................................................................................................................... 30 4.4.1 Question 1 .................................................................................................................................. 30 4.4.1.1 Findings .....................................................................................................................................30 4.4.1.2 Conclusions .............................................................................................................................32 4.4.1.3 Recommendations ..................................................................................................................32 5.0 Lessons learned ...................................................................................................... 32 ANNEXES Annex A: Statement of Work Annex B: List of Interviewees Annex C: List of Documents Reviewed Annex D: Overall Monitoring and Evaluation Matrix Annex E: Annual Performance Matrix Annex F: Illustrative Interviewee Questionnaires Annex G: Household Survey Methodology Annex H: Household Survey Questionnaire FIGURES Figure 1: Economy and Efficiency Considerations Related to Value for Money Figure 2: Budget Expenditures by Program Element Figure 3: Budget Expenditures by RPO Figure 4: Increase in Income Resulting from IILP Figure 5: ISLG Contribution to Savings and Access to Small Loans Figure 6: Entrepreneurship Related to IILP Interventions Figure 7: Average Annual Production of Maize Before and After IILP Intervention Figure 8: Increases in Production by Influencing Factor Figure 9: Annual Income Increases in Relation to Production in Kilograms (kg) Figure 10: Food Shortage Causes Figure 11: Common Household Diseases Figure 12: Disaggregated Data Male / Female Activities ACRONYMS ADEPR Association des Eglises des Pentecotes au Rwanda AEE African Evangelical Enterprise AFR Access to Finance Rwanda AMIR Association of Microfinance Institutions BCC Behavior Change and Communication BCV Be the Change Volunteer BDS Business Development Services COR Contracting Officer’s Representative CRS Catholic Relief Services DUHAMIC-ADRI Duharanire amajyambere y’icyaro EDPRS-2 Second Economic Development and Poverty Reduction Strategy ET Evaluation Team FFS Farmer Field School FGD Focus Group Discussion FtF Feed the Future FY Fiscal Year GoR Government of Rwanda HHS Household Survey IGA Income Generating Activities IILP Integrated Improved Livelihood Program IPAR Institute of Policy Research and Analysis Rwanda ISLG Integrated Savings and Lending Groups KCB Kenya Commercial Bank KIIs Key Informant Interviews LOP Life of Project M&E Monitoring and Evaluation ME&A Mendez England and Associates MFI Micro-Finance Institution MINAGRI Ministry of Agriculture and Animal Resources MIS Management Information System MOU Memorandum of Understanding MSME Micro, Small and Medium Enterprises NGO Non-Governmental Organization NISR National Institute of Statistics Rwanda PAT Poverty Assessment Tool PIRS Performance Indicator Reference Sheets PMP Performance Monitoring Plan RAB Rwanda Agriculture Board RFA Request for Application RPO Rwanda Partner Organization Rwf Rwandan Franc SACCO Saving and Credit Cooperative SME Small and Medium Enterprise SMS Short Message Service SOW Statement of Work STC Save the Children TOR Terms of Reference UNDP United Nations Development Programme USAID United States Agency for International Development USG United States Government WB World Bank WFP World Food Programme WHO World Health Organization i EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS This is a report on the Performance Evaluation of the Integrated Improved Livelihoods Program (IILP) funded by the United States Agency for International Development (USAID) Mission in Rwanda. The project is being implemented by Global Communities (formerly CHF International) from July 2011 through July 2016. The evaluation was conducted by a team assembled by Mendez England & Associates (ME&A). According to the Statement of Work (SOW) for this assignment (see Annex A), the main purpose of the evaluation was to provide USAID/Rwanda with an objective external assessment of the management and performance of IILP’s activities from July 2011 to July 2014. PROJECT BACKGROUND According to the original Request for Applications (RFA) issued by USAID in February 2011, IILP was designed to address the needs of Rwanda’s poorest rural communities, a substantial proportion of whom remained unbanked and dependent on subsistence agriculture for survival. In addition, IILP would take account of the Government of Rwanda’s (GoR) Vision 2020 and its second Economic Development and Poverty Reduction Strategy (EDPRS) 2013-2018. For more details, see Section 2.0 of this report. EVALUATION DESIGN, METHODS AND LIMITATIONS Quantitative Research and Analysis To conduct the evaluation of IILP, the Evaluation Team (ET) reviewed a wide range of project-related materials from various sources, including those received from USAID/Rwanda. The ET also conducted a household survey (HHS) of 264 beneficiaries in both the Western and Southern Provinces. Qualitative Research and Analysis The ET elicited feedback and commentary from as representative a body as possible with direct or indirect knowledge or experience of IILP throughout the period in question and conducted face-to-face interviews and site visits in Western and Southern Provinces (see Annex B for a complete list of interviewees). Evaluation Limitations The ET encountered few limitations during its fieldwork in Rwanda. Some of the more relevant were recall bias (respondents having difficulty in recollecting events) and halo bias (whether respondents actually reveal true opinions). To mitigate the above limitations, the ET gathered data by employing various data collection methods and used triangulation to cross-validate results and findings. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS A. Monitoring and Reporting A.1 Finding While it is clear from Focus Group Discussion (FGD) participants’ comments and HHS responses that IILP’s various interventions have influenced recipients to some degree, the project does not utilize a methodology capable of tracking behavior changes over time other than gathering subjective opinions of interviewees. A.2 Conclusion Currently, IILP reports mainly on outputs rather than outcomes1 , suggesting more of a weakness in the project’s design rather than its implementation as all indicators are effectively based on quantitative data gathering. While this does not mean that changes in behavior have not occurred since the beginning of the project, there is no existing methodology as defined in, for example, USAID’s Performance Indicator Reference Sheets (PIRS), to which IILP can refer. A.3 Recommendation 1 The ET carefully considered all IILP indicators but did not consider that they were in fact measureable in terms of outcome. ii The methodology to track behavior change should incorporate the Knowledge, Attitudes and Practices (KAPs). It should be conducted quarterly by Rwanda Partner Organization (RPO) field officers and reported to the project team. Appropriate tools should be developed and harmonized with all RPOs and there needs to be inputs from the project that could be directly linked to changes in habits and/or attitudes. B. Efficiency and Effectiveness B.1 Findings Planned Activities IILP’s contribution to health, nutrition, and literacy, as defined by indicators and targets, has been successful and is confirmed unanimously by beneficiaries in all sectors. In particular, the project’s nutrition team, in collaboration with the Regional Nutrition Advisor of Save the Children (STC), reinforced nutrition activities in cooperatives and Integrated Savings and Lending Groups (ISLGs) by establishing kitchen gardens aimed at supporting beneficiaries to consume a more diversified diet. The fact that the majority of beneficiaries adopted kitchen garden technology can be regarded as a project success. On the other hand, the greatest area of weakness appears to be in the financial sector where loans remain restricted. Program Elements Adult Literacy: Evidence, supported by FGDs and the HHS, suggests that with its allocated budget accounting for 16% of the total program element budget,2 the Adult Literacy program element (basic and financial literacy) provided value for money. Data gathered from FGDs clearly indicates a 100% level of satisfaction with the project’s training programs from the literacy groups in all 24 sectors. Health and Nutrition: Targets, related to beneficiaries being trained in child health and nutrition, eating balanced and diversified diets, and creating kitchen gardens, have largely been met or are likely to be met by September 2015. IILP’s Fiscal Year (FY)14 Annual Report draws attention to the fact that while 87% of beneficiaries had knowledge of balanced and diversified diets, only 26% indicated actually consuming such a diet – significantly under the 75% target for FY14. On the positive side, over 11,000 kitchen gardens have already been established with IILP’s support. Financial Services: The results achieved appear to belie actual events in the field. For example, progress to date appears to have been rather minimal for ISLG-microfinance institution (MFI) linkages aimed at facilitating borrowing for individuals, in particular, and for small businesses3 , in general, although IILP expects that in time ISLGs will develop full working relationships with Saving and Credit Cooperatives (SACCOs) and/or MFIs. The ISLGs have been very successful in facilitating savings and loans with groups as evidenced by FGD feedback. Nevertheless, financial service interventions generally require more funds to support implementation than other project interventions4. Agriculture: Key activities included training farmers in agriculture extension as well as support for micro, small and medium enterprises (MSMEs) and farmers cooperatives or farmer field schools (FFS) focusing on value chain development. Reported performance indicators show that most targets related to the introduction of new technologies and management practices for farmers accessing agriculture extension services, in particular how to deal with of pests and post-storage handling and distribution, have been achieved or will be achieved by September 2015. Rwanda Partner Organizations CARITAS: CARITAS is involved in most of IILP’s activities and, although it is difficult to calculate a specific contribution regarding outcomes, there appears to be no need to change its implementation strategy or delivery mechanism. FGD participants from districts where CARITAS operates expressed overall favourable satisfaction with the support received. 2 16% for January 2012 – September 2014 ($147,555) and 1% for October 2014 – September 2015 ($15,932). 3 The ET acknowledges that the creation of ISLGs and developing them to maturity takes time. IILP has supported this initiative with financial literacy training in order for group members and the groups themselves to embrace a savings culture. 4 Due to the nature of activities involved such as market research and financial product development. iii Duharanire amajyambere y’icyaro (DUHAMIC-ADRI): DUHAMIC-ADRI’s budget accounts for nearly one third of the total RPO budget across both periods. Although its contribution was similar to that of CARITAS, FGD participants reported that its contribution to tackling extension related issues – pest and disease control, seasonal planning, seed selection, and integrated use of fertilizers – on behalf of beneficiaries was not strong. African Evangelical Enterprise (AEE): AEE operates in 3 districts in the Southern Province mainly in financial matters and business development. In the first period, AEE’s budget accounted for 15% of the total spent by RPOs. Due to the decrease in AEE’s FY15 budget (no figures available) and a lack of feedback from beneficiaries regarding AEE’s activities, the ET is unable to comment further on its activities or on its performance in terms of value for money. Association des Eglises des Pentecotes au Rwanda (ADEPR): AEDPR operates in both provinces and in all districts. Its budget for the current year was reduced from 33% of RPO expenditure in the first period to 15%. Given that ADEPR is exclusively involved in literacy activities5 and that literacy targets across the board have largely been met or will be met by September 2015, this reduction seems sensible and justified. Behavior Change and Communication (BCC) Although not directly related to any project performance indicators, IILP allocated a significant budget for promotional activities, primarily radio broadcasts, to reach targeted beneficiaries with messages aimed at motivating behaviour changes on a range of issues, including breastfeeding, literacy, and financial products. However, as no data was available for the ET to review, it concluded that IILP has no real way of knowing the extent to which its radio messaging initiatives reach or influence their targeted audience. Entrepreneurship and Incomes Among Beneficiaries 63% of HHS respondents confirmed that IILP’s interventions contributed to increasing incomes,6 especially for women. This increase was mostly attributed to the organization of ISLGs, which contributed significantly to savings and the ability of group members to access affordable small loans for production. 95% of FGD participants learned financial literacy on how to save money and reported that borrowing from informal self-help savings and loan groups was their only real alternative to accessing credit. Linkages Between Beneficiaries and MFIs Linkages between beneficiaries and MFIs are not significant. 95% of FGD participants were unable to conduct business with commercial banks or MFIs due to high interest rates, insurance requirements, compulsory deposits, lack of collateral, or realistic business plans. However, 80% of IILP beneficiaries earn less than $1.25 per household per day and are not target customers of formal financial institutions. B.2 Conclusions Planned Activities While most targets are being met and, in all probability, end-of-project targets will be achieved, several activity areas are cause for concern. Given IIPL’s current portfolio of activities, loans made by MFIs, farmers who applied new technologies or management practices, and percent of beneficiaries consuming a diet with increased diversification, will neither meet final indicator targets nor have a sustainable impact on income earning potential. Program Elements Adult Literacy: IILP delivered adult literacy initiatives efficiently within financial parameters, therefore providing value for money. Health and Nutrition: IILP has made significant progress in achieving results, or being able to do so by the project’s end, for activities related to child health, nutrition, and kitchen gardens. However, IILP does not possess an effective means of measuring the extent to which consuming a diet with increased diversification has occurred in reality. 5 ADEPR also work in financial services, health and nutrition and agriculture and is the only RPO working in all 4 elements. 6 Average annual increase for all income categories = 64%, avg. increase for men = 52.19%, avg. increase for women = 72.67%. iv Financial Services: Access to finance for project beneficiaries appears to have had mixed results. On one hand, establishment of ISLGs encouraged beneficiaries to save and gave them the opportunity to acquire funds to meet their needs, leading to an increase in incomes and assets as supported by the project’s own program surveys. On the other hand, evidence suggests that linkages between small businesses and MFIs remain rather weak requiring further review and action from IILP to make further progress. Agriculture: With more than a quarter of the budget supporting agricultural activities and from beneficiary feedback regarding pests and diseases resulting in crop losses, it is possible to conclude that while targets are generally being met, there appears to be an absence of follow-up from IILP in determining the outcomes of their trainings. Value for money, therefore, remains uncertain. RPOs CARITAS: With the budget allocated to it and results achieved to date, CARITAS has successfully fulfilled its obligations as an RPO. DUHAMIC-ADRI: Given that its budget accounts for around one third of the total RPO allocation and that it operates in only 5 districts, DUHAMIC-ADRI needs to conduct more fieldwork addressing extension issues and liaising more closely with Be the Change Volunteers (BCVs) and local agronomists. AEE: The ET had insufficient information, including little or no FGD feedback, to make an informed judgement on AEE’s performance relative to existing and planned expenditures. ADEPR: With the budget allocation and results achieved to date, ADEPR has successfully fulfilled its obligations as an RPO. Behavior Change and Communication In the absence of any meaningful feedback from radio stations delivering IILP’s messages, and given the responses from FGD participants, it is unclear why over $200,000 has already been spent, and another $30,000 is planned for the current year, on media promotion. In the absence of indicators or results to be achieved, the expenditure of funds without data on whether target audiences are being reached and influenced to change their behavior does not suggest an effective use of those funds or value for money7 . Entrepreneurship and Incomes Among Beneficiaries IILP has performed well through the organization and strengthening of ISLGs for poor rural families to access finance. This success is attributable to intensive financial literacy training and beneficiary ownership of ISLGs (setting regulations to facilitate borrowing). However, collateral requirements and high interest rates demanded by MFIs and SACCOs continue to limit borrowing and income potential. Linkages Between Beneficiaries and MFIs ISLG-MFI linkages remain relatively weak and represent a very small intervention outcome in actual project results; i.e. still lacking is a systemic lending approach for project beneficiaries through MFIs to ensure expansion of financial services for the rural poor to improve small-scale enterprise performance. B.3 Recommendations Planned Activities IILP should conduct an immediate review of its training programs related to MFI linkages to groups and extension services, including kitchen gardens, in order to help alleviate some beneficiary concerns, such as farmers acquiring more practical information related to extension services. IILP should also conduct a needs assessment to determine practical actions required to ensure the achievement of its objectives. Program Elements Adult Literacy: Given IILP’s success in delivering planned results with its current activities and available budget, no specific recommendations are envisaged for this component. Health and Nutrition: IILP’s strategy of concentrating efforts on the creation of 240 model kitchen gardens appears appropriate in acting as a catalyst for others in rural areas to emulate. No specific recommendation other than IILP should immediately review its agriculture-related activities to include 7 See 4.3.2.1 Findings below for more detailed explanation regarding BCC and messaging in particular. v more effective extension services, such as preventative pest and disease control in collaboration with Rwanda Agriculture Board (RAB). Financial Services: It is strongly recommended that IILP take the initiative in establishing real links between individual, small businesses and MFIs. This could include hosting local workshops in the 8 districts with representatives of self-help groups (ISLGs, cooperatives, etc.), as well as BCVs, RPOs and project staff. Agriculture: In the absence of expertise or experience in dealing with this issue by those tasked with advising affected farmers, including local agronomists, BCVs, and RPO representatives, an alternative approach is needed to alleviated the problem. RPOs CARITAS: No recommendation. Activities and results meet indicators and beneficiaries are satisfied. DUHAMIC-ADRI: Given its concentration of effort on agriculture activities, including extension of services, it would be advisable for the project to take stock of activities currently being implemented and those planned for the rest of the current year. Focus should be on upgrading the capacity of field officers to be able to offer advice to beneficiaries on matters related to pest and disease control. AEE: No specific recommendation in the absence of up-to-date financial statistics on expenditures. AEE representatives state that they are involved in several program element activities but were unable to comment on the success or effectiveness of their contributions in supporting beneficiaries. IILP would be well advised to follow-up on AEE’s activities for clarification. ADEPR: No recommendation. IILP's activities implemented through ADEPR on adult literacy have been largely successful. With a reduced budget for the current year, there is no need for any adjustments. Behavior Change and Communication The project team should immediately review its media expenditure and reconsider the validity of spending money on an activity with no means of determining the extent of its reach or impact. Entrepreneurship and Incomes Among Beneficiaries IILP should consider creating a credit guarantee fund for ISLGs and upgrade capacity building measures to improve business planning and project management. A formal cooperation agreement between IILP and MFIs is highly recommended in order to address lending limitations for the project’s beneficiaries. Linkages Between Beneficiaries and MFIs The project should seriously consider focusing on the development of business clusters through partnerships between ISLGs, MFIs and local services providers based on high potential business areas. Small-scale farmers and their emerging enterprises require local services that help them integrate into value chains (chain empowerment) and services (value chain finance). C. Impact and Relevance C.1 Findings 1. HHS data revealed that a majority of the 75% of beneficiaries interviewed who took up loans, irrespective to loan source (ISLGs or Banks/MFIs), used the money for agricultural purposes with a view to increasing food production. However, despite the positive trend in production, 41.9% of respondents still faced food shortages throughout the year due mainly to variable climatic conditions (15.2%), low production (9.8%), limited income (6.4%), post-harvest losses due to pest infestations (6.4%), and market shortages (6.1%). 2. The percentage of the households with severely and mildly malnourished children has decreased, in some cases by 100%, according to Cynika Health Centre’s nutrition office. Although nutrition messaging approaches appear to have had little effect on the decrease in malnutrition, STC’ nutrition education activities successfully focused on teaching the most vulnerable rural farming households.8 3. FGDs and the HHS show that women not only account for 70% of all group members and 70% of kitchen garden owners but are also more than twice as likely to borrow money from within their 8 Particularly on diet, nutrition and hygiene and especially within the critical 1000days of newborn children and mothers. vi own self-help groups; they fared less successfully than men only in borrowing loans from MFIs. 92.8% of women recognized that being a member of an ISLG increased their level of savings and thus their confidence to use the money to improve their livelihoods. C.2 Conclusions 1. IILP’s financial services initiatives, especially related to ISLGs, are partially contributing to increased food security but a significant proportion of households face food insecurity due to the low rate of farmers adopting agricultural technology practices, lack of access to extension services (only 26,499 of the 58,994 farmers were able to access them9 ), and continued lack of marketing opportunities for farmers to sell excess crops in order to increase income and thus allow the possibility to buy more diversified foods including dairy. 2. The level of malnutrition among project beneficiaries has decreased, although IILP’s literature lacks information on the extent or how this is measured. Nutrition education also played a key role in increasing knowledge and awareness in the community. 3. Although it has no targets to meet, IILP has succeeded in reaching and supporting a significant percentage of women in its target areas, with no apparent disadvantage impacting their daily lives compared with their male counterparts. C.3 Recommendations 1. IILP should consider expanding its income strengthening work on orienting both on/off farm initiatives to improve market access and increase the potential for households to access supplementary foods (meat, milk, fish, eggs, fruit) from markets. The project’s extension model should also be upgraded to increase the number of farmers actually implementing in practice what they have learned in theory, having in mind limited availability of funds for extension inputs. 2. IILP should continue its strategy of creating model kitchen gardens and consider options aimed at helping more vulnerable families obtain loans to buy or be provided with chickens or goats (possibly from project budgets) in order to deal with the issue of protein deficiency. The project should also revise its radio messaging strategy and consider partnering with Urunana.10 3. No specific recommendations for activities related to women in IILP’s targeted communities; activities adequately support women with significant advances in improving and understanding of nutritional issues. D. Sustainability D.1 Finding IILP’s strategy regarding sustainability largely rests with BCVs continuing to play an integral role in local communities post-project. However, FGDs with BCVs revealed that many believed they are ill-equipped to continue offering support to beneficiaries in their communities due to lack of relevant expertise or knowledge on some issues, such as preparing a ‘bankable’ business plan or offer practical advice related to pest and disease control. Some BCVs suggested they would not continue volunteering post-IILP. D.2 Conclusion Relying on BCVs, RPO systems and structures, and MFIs and SACCOs as the mediums through which sustainability is ensured, is not fully supported by the evidence. In short, there appears to be an element of disconnect between what IILP and RPOs say is being delivered and reported regarding support for beneficiaries and the views expressed by representatives of groups to which those beneficiaries belong, such as ISLGs, cooperatives, and nutrition groups, together with local officials and authorities. As this information does not seem to be transmitted to RPOs or IILP’s management, it can be concluded that this is more of a project systematic issue. D.3 Recommendation 9 Reasons provided by farmers included not having sufficient funds to buy input supplies such as seeds, tools or equipment and also that on occasion they didn’t fully understand what they had been taught or understand instructions in manuals provided. 10 A local popular radio program. vii IILP should urgently conduct a review to assess the competence of BCVs and identify gaps in their skills and knowledge. IILP should also consider options to help BCVs convert into business organizations, empowering them to offer services and, potentially, borrow funds to support their activities. LESSONS LEARNED 1. At the project design stage, careful consideration should be given when introducing indicator/result areas that are based on partial productivity measures such as yield (e.g. output or gross margin per unit of land) as these can be difficult to calculate in the absence of verifiable data from farmers (especially in poor, rural areas). Planned interventions could involve supporting the creation of local agriculture extension centres, which could advise farmers on the provision of extension supplies. 11 2. Project requests for qpplications (RFAs) or requests for task order proposals (RFTOPs) should avoid use of indicators where there is a requirement to calculate the project’s contribution towards, for example, a change in behavior, attitude, or performance in relation to an expected outcome because there is no direct correlation between a project intervention that can exclusively be attributed to the project, due to the existence of other contributing factors, that can influence outcomes. 3. The more project partners and implementing intermediaries, the greater the risk of diluting the chain-of-command for gathering and reporting data in real time to support informed decision-making at project management level. Future project designs should consider this issue during preparation. 4. To improve project reporting (quarterly, annually and work plans), future project RFAs or RFTOPs should require tenderers to indicate the type of management information system (MIS) they propose for tracking project performance and for monitoring and evaluation (M&E). This should be mandatory and highlighted in all technical proposal submissions. 5. When including sustainability as a factor in project design, it is important to consider the need for including a specific definition of what is intended, as it could be misinterpreted or mean different things to different people. Sustainability usually refers to post-project activities, attitudes or behavior, which will not, by definition, be followed up at a later date. 6. At the project design stage, consider including the requirement of a needs assessment to be conducted prior to implementation in order to identify ‘real’ needs or gaps to be addressed (which could potentially differ from those originally identified in RFAs or RFTOPs). Also, review who among the international community might be planning to implement the same to avoid duplication. 7. In RFAs or RFTOPs for evaluations, when raising issues related to cost-benefit (analysis or comparisons), clarify if what is intended is a ‘value for money’ exercise as opposed to comparing how funds could have been more effectively spent elsewhere. Inter-component comparisons would be valid in terms of efficiency and effectiveness analysis. 8. In RFAs or RFTOPs consider whether there is a concentration of effort on achieving quantitative results (outputs) relative to the impact of planned results (outcomes). This distinction is relevant in IILP-type projects as the former is easily measurable while the latter is more complicated to assess. 11 Including plowing, planting, cultivating and harvesting 1 1.0 EVALUATION PURPOSE, OBJECTIVE & QUESTIONS 1.1 EVALUATION PURPOSE This is a report on the Performance Evaluation of the Integrated Improved Livelihoods Program (IILP) funded by the United States Agency for International Development (USAID) Mission in Rwanda. IILP is being implemented by Global Communities (formerly CHF International) from July 2011 through July 2016. The evaluation was conducted during the period January – February 2015 by a team assembled by Mendez England & Associates (ME&A). The Evaluation Team (ET) comprised four key experts: international expert, Mr. Colin Maclean (Team Leader), and local experts, Mr. Didier Nkubito (Gender and Nutrition Specialist), Mr. Olivier Habimana (Data Analyst and Logistics Coordinator), and Mr. Elie Nsabimana (Financial Services Expert). According to the Statement of Work (SOW) for this assignment (see Annex A), the main purpose of the evaluation is to provide USAID/Rwanda with an objective external assessment of the management and performance of IILP’s activities from July 2008 to July 2011. 1.2 EVALUATION OBJECTIVES According to the SOW, the primary overall objectives of this evaluation are to “assess the progress of the USAID Ejo Heza12 program over its first three years of implementation towards achieving Intermediate Results for each Program Element and cross cutting objectives, to test the effect of the used integrated approach on the beneficiaries’ behavior towards program key elements and to provide recommendations that guide the program team to make necessary course corrections or adjustments in order to fully achieve USAID Ejo Heza’s goal of improving the livelihoods of Rwanda’s very poor by the end of the program.” 1.3 EVALUATION QUESTIONS Per the SOW, the ET was to answer a specific set of evaluation questions. For ease of understanding and to assist the ET in its investigation and deliberations, the evaluation questions as highlighted in the SOW have been grouped into four main headings as outlined below (the original grouping of questions are found in the SOW in Annex A of this report): A. Monitoring and Reporting 1. How can USAID Ejo Heza better track the outcomes of behavior change activities on beneficiaries? B. Efficiency and Effectiveness 1. To what extent planned activities for IILP enable achieving program expected results, purpose and goal? 2. How well IILP resources and activities are converted into results, in terms of quality and time based on cost-benefit comparisons? 3. How and to what extent IILP-support financial services increased entrepreneurship (new, profitable SMEs) and incomes among the beneficiaries? 4. How and to what extent IILP’s agricultural extension services contributed to household crop production and incomes? C. Impact and Relevance 1. How and to what extent has IILP’s work in promoting and strengthening income generating activities contributed to household food security? 2. How and to what extent IILP nutrition education activities contributed to the improved household nutrition status, especially for women and children? Are the used kitchen garden and nutrition messaging approaches effective to reduce malnutrition or need for improvement and/or other alternatives? 12 The IILP project is known locally as Ejo Heza and reference is periodically made to this name throughout this report. 2 3. To what extent are the activities implemented by IILP contributing to increased participation, improved income and overall “empowerment” of women in the targeted communities? What are women’s perceptions or feeling on their current income situation? D. Sustainability 1. To what extent are district authorities and community volunteers collaborating in the implementation of project activities, as a strategy for ensuring sustainability of the project results? What can we learn from this process to inform efforts to further improve collaboration as a sustainability strategy for continued program benefits in targeted communities? 2.0 PROJECT BACKGROUND In designing the original Request for Applications (RFA) for IILP, USAID took into consideration the findings of two contemporary surveys that influenced the geographic focus of IILP’s activities – the USAID/Rwanda-sponsored “Assessment of Economic Opportunities for Low-Income Women and Very Poor Households” conducted by the OFT Group, and the “Comprehensive Food Security and Vulnerability Analysis and Nutrition Survey” led by the World Food Program (WFP). Analysis of these assessments, together with a review13 of provinces already appearing to have adequate activity support coverage, led USAID to identify the Southern and Western Provinces as high priority for the program objectives discussed below. In total, 8 districts, which were later defined and elaborated in the project’s four main components, were chosen to be the focus of IILP’s interventions14 . Further, analysis of the WFP’s 2012 “Comprehensive Food Security and Vulnerability Analysis and Nutrition Survey” report, led by the Ministry of Agriculture and Animal Resources (MINAGRI) and the National Institute of Statistics Rwanda (NISR) followed two similar exercises in 200615 and 2009. The study, which was “specifically geared towards producing evidence-based support for targeting (including at district level) of social protection and other assistance aimed at eliminating food insecurity and malnutrition in Rwanda” estimated that in March/April 2012 one in five Rwandan households (21%) had unacceptable food consumption and therefore could be considered as food insecure. The Southern and Western provinces were deemed to be the worst affected by food security, with little progress noted since the findings of the 2009 survey. It is against this background that IILP’s raison d’être – to address the needs of Rwanda’s poorest rural communities, a substantial proportion of whom remain unbanked and dependent on subsistence agriculture for survival – can be placed in perspective given that much work remains to improve the livelihoods of Rwanda’s poorest household farmers. While evidence suggests that some progress has been made in recent years to address the primary causes of malnutrition and low household incomes, the road ahead remains challenging, especially if the Rwanda Vision 2020 goal of “replacing subsistence farming by a fully monetized, commercial agricultural sector by 2020” is to be realized. According to the original RFA issued by USAID in February 2011, IILP was designed to address the needs of Rwanda’s poorest rural communities. In order to achieve its aim “to improve livelihoods of Rwanda’s very poor, particularly women, through expansion of responsible and appropriate access to finance, increased economic opportunities and incomes, and enhanced food consumption,” IILP was designed to contribute to the United States Government’s (USG’s) Feed the Future (FtF) initiative in Rwanda and take into account the Government of Rwanda’s (GoR) Vision 2020 and its Second Economic Development and Poverty Reduction Strategy (EDPRS-2) 2013-2018. To provide targeted beneficiaries with the means to purchase more nutritious foods to improve diets and reduce malnutrition, IILP’s activities were intended to focus on supporting and encouraging the adoption of sustainable practices and engagement in activities related to agriculture value chains 13 USAID also took account of information from the Urwego Opportunity Bank as a proxy in the absence of detailed information from microfinance providers such as microfinance institutions (MFIs), non-governmental organizations (NGOs) and Savings and Credit Cooperatives (SACCOs). Data was also extracted from USAID’s Higa Ubeho and SAFI projects to provide perspective. 14 See map in Annex B – Work Plan & Design Methodology 15 USAID refers to this survey in its RFA (February 2011) 3 that could increase earning potential. In short, IILP’s integrated interventions were designed to positively impact the overall economic, health, and general quality of life well-being of all members of rural household families.16 In its implementation, IILP partnered with local organizations, international organizations, GoR Ministries and Departments, and financial institutions, including: Rwanda Partner Organizations (RPOs): Four local non-profit organizations – CARITAS, Duharanire Amajyambere y’icyaro (DUHAMIC-ADRI), African Evangelical Enterprise (AEE), and Association des Eglises des Pentecotes au Rwanda (ADEPR) – partnered with IILP to implement activities within the project’s key program elements of agriculture, adult literacy, financial services, and health and nutrition. RPO activities, including training workshops for local beneficiaries and working in collaboration with Be the Change Volunteers (BCVs), are directly associated with behavior change. BCVs are individuals elected by their local communities to provide representation and ground support. Save the Children (STC). As IILP’s international partner mainly concerned with health and nutrition project elements, STC works closely with RPOs to create new nutrition groups, prepare kitchen gardens, and provide training workshops on food preparation, diets, hygiene, and sanitation. STC plays an important role in supporting IILP’s overall objectives in improving the livelihoods of its targeted beneficiaries. Government Ministries and Departments: The Ministries of Finance and Economic Planning, Agriculture and Animal Resources, and Health, along with the Rwanda Agriculture Board (RAB) are IILP’s stakeholders. Financial Institutions: Financial institutions, including microfinance institutions (MFIs), Association of Microfinance Institutions in Rwanda (AMIR), Access to Finance Rwanda (AFR), and Saving and Credit Cooperatives (SACCOs), are related to a major part of IILP’s mandate of contributing towards improving access to finance for individuals and businesses in rural communities. 3.0 EVALUATION METHODS AND LIMITATIONS 3.1 EVALUATION METHODOLOGY To design the methodology for this evaluation, the ET took into account a number of key factors, including quantitative and qualitative aspects of the mission, which could have influenced or placed limitations on its strategy for answering the evaluation questions. The more important of these factors are outlined below: 1. Key Factor: IILP was tasked with meeting a number of indicators and quantitative targets as highlighted in the original Cooperative Agreement and subsequent modifications. Issue: The ET’s mandate was to review the project’s activities from July 2011 to July 2014 (3 years). In that period, IILP’s results, as defined in its various quarterly and annual reports and annual work plans, were approved by USAID/Rwanda. Therefore, there was no need for the ET to verify reported data and, in any event, there were no means available to conduct such an exercise. Outcome: Based on the above, the only requirement incumbent on the ET was to report on IILP’s data gathering and reporting methodology in order to determine its efficiency and relevance from an operational point of view and thus, the extent to which it contributed to the effectiveness of the project’s internal monitoring and evaluation (M&E) system. 2. Key Factor: Since July 2014, a further 6 months had passed during which IILP continued with its core activities while the evaluation was underway in January / February in Rwanda. Issue: Should the ET consider project activities and performance between July 2014 and February 2015? Outcome: While not required to review the IILP’s performance after July 2014, the ET 16 See Annex E – Overall Monitoring and Evaluation Matrix for details. 4 believed that the feedback from beneficiaries and stakeholders during this period of time would serve to inform the team with respect to possible recommendations for the remaining 18 months of the project and thereafter in terms of sustainability following the end of the IILP in July 2016. Comments to that effect are therefore reflected in this report. 3. Key Factor: The ET was tasked with commenting on two inter-linked questions regarding, activities, resources, results and cost-benefit comparisons, which have been grouped under ‘Efficiency and Effectiveness’ in this report. Issue: Can a true cost-benefit analysis be conducted in the absence of an alternative use of available resources towards achieving similar or better results, i.e. spending money relative to what? Outcome: In order to satisfactorily address both these questions, the ET found it necessary to interpret cost-benefit in terms of ‘value for money,’ i.e. if project activities or actions are not achieving results then they are not value for money. In addition, we consider the quality of the project’s outcomes as fundamental to understanding whether something is providing value. In short, we have reflected on comparisons between components in terms of expected outputs and outcomes and whether there are any discernable differences in performance between components in their delivery mechanisms. 3.1.1 Quantitative Research and Analysis The ET reviewed a wide range of project-related materials from various sources, including those received from USAID/Rwanda at the pre-mobilization stage and from IILP in particular on arrival at their offices in Kigali. These documents included, among others, the original RFA for IILP,17 the Cooperative Agreement between USAID and CHF International18 and three subsequent modifications to the agreement, Annual Work Plans, Quarterly and Annual Reports, Performance Monitoring Plans (PMPs), and selected IILP commissioned survey reports and papers. These reports constituted the main reference point for all relevant statistical data, including financial, from IILP’s origins in July 2011 through July 2014.19 From this material, the ET tracked IILP’s performance over time with respect to the various pre-determined indicator targets established at the start of the project (see Annex D and Annex E for a breakdown of IILP’s quantitative targets and results). The verification of statistical results actually achieved by IILP at the time of the evaluation, however, did not fall within the remit of this evaluation. The nature and extent of IILP’s activities in terms of reaching and providing assistance and support to its targeted beneficiaries was reviewed by cross￾referencing quantitative achievements with evidence gathered by the ET from its qualitative data. Household Survey (HHS) The ET conducted a HHS of 246 beneficiaries20 to assess how nutrition activities contributed to improved household nutrition status and determine the extent to which IILP’s agriculture extension services contributed to household crop production and incomes. Data was gathered by a contracted local Rwandan firm, STRATDEVER, through a combination of direct interviews with selected households using structured questionnaires and focus group discussions (FGDs). It was then analyzed and tabulated so the ET could extract relevant information and statistics (see Annexes G and H for more information on the HHS). 3.1.2 Qualitative Research and Analysis The ET elicited feedback and commentary from as representative a body as possible with direct or indirect knowledge and/or experience of IILP throughout the period in question.  62 face-to-face interviews conducted with a cross-section of key stakeholders with direct knowledge of IILP from both the public and private sector (see Annex B for a complete list).  Site visits were made to Western Province (Ngororero, Rutsiro and Karongi) and Southern Province (Nyamagabe, Nyanza, Huye, Nyaruguru and Gisagara). 17 RFA No. USAID-RWANDA-RFA-696-11-000002 (7 February 2011) 18 Cooperative Agreement No. AID-696-A-11-0008 (13 July 2011) 19 It should be noted here that although the ET was tasked with only reviewing IILP’s performance for this period, it was deemded relevant to include available data and information up to September 2014, as the project in fact reports from October to September on an anuual basis. Reference to this inclusion is reflected elsewhere in this report where relevant and in Annexes E & F. 20 See Annex G – Household Survey Methodology for details 5  40 FGDs were held in the project’s 8 districts (24 sectors) and were attended by a total of 483 beneficiaries representing the various groups supported by the project: ISLGs, nutrition, literacy, cooperatives, and BCVs. Separate sessions were held for each group. Each FGD had an average of 12 participants. From the FGDs, the ET was able to gather a substantial amount of information on IILP’s activities and provide insight on participant behaviors and attitudes towards the project’s interventions. The results of the FGDs were cross-correlated with the results of the HHS which, in both quantitative and qualitative terms, corroborated the ET’s findings. The focus of these interviews and meetings was to acquire an in-depth understanding of the extent to which IILP’s activities contributed to meeting its overall objectives as defined by specific project indicators and targets. While the views, opinions, and observations expressed by interviewees are largely subjective, their perceptions regarding IILP clearly provided valuable feedback on the project’s activities and shed light on how they assessed IILP’s current and anticipated impact on its intended beneficiaries (for a complete list of interviewees, see Annex B). Below is a brief summary of targeted respondents that the ET had met with during data gathering activities. RPOs: The ET interviewed representatives of all four RPOs partnering with IILP – CARITAS, DUHAMIC-ADRI, AEE, and ADEPR. Information gathered was highly relevant to the evaluation and provided a useful insight into the connection between IILP and RPOs, especially regarding reporting structures and meeting targets. STC: STC provided the ET with relevant reports, including work plans and quarterly reports, and informative and useful feedback from the Country Director and Program Director. The ET also interviewed STC’s IILP in-house representative who operates from the project’s office as an embedded Senior Nutrition Specialist. Government Ministries and Departments: The ET met with key individuals in the Ministries of Finance and Economic Planning, Agriculture and Animal Resources, and Health, along with the RAB. Interviewees were forthcoming about the roles of their respective departments in relation to IILP. As official government bodies, they all have some measure of oversight of IILP’s activities so that they fit into the departments’ longer term plans, for example, with GoR’s Vision 2020. Their perspectives and observations were vital to the ET’s understanding of IILP’s mission and objectives. Financial Institutions: The ET met with commercial banks, MFIs, AMIR, and AFR to clarify its understanding of the role they play with respect to saving and lending, especially among Rwanda’s impoverished and disadvantaged communities who are generally considered to be unbankable by traditional financial institutions such as banks and MFIs. The ET sought to understand the extent of lack of funds for potential entrepreneurs, farmers, processors, and women. During its field visit to the provinces, the ET also interviewed district-level SACCO representatives who provided feedback on their activities related to savings and loans. International Community and Selected Stakeholders: The ET held interviews with various international donors and stakeholders in Kigali who are involved in international development work that touches on health and nutrition and income generation for farming communities, including World Food Programme (WFP), SPARK (a Dutch NGO), and Catholic Relief Services (CRS). Their views and perceptions of IILP provided a different perspective to those more closely associated with the project and its activities. In summary, the ET accumulated sufficient data and information from existing sources and field investigations to be able to provide satisfactory and meaningful answers to USAID’s questions outlined in the SOW. Furthermore, a number of interviewees representing several important and relevant stakeholders in connection with IILP’s operations, provided the ET with additional material, including in-house publications and/or other papers/documents which, in turn, provided useful insight into project-related activities. 3.2 EVALUATION LIMITATIONS The ET encountered few limitations during its fieldwork in Rwanda. Some of the more relevant are listed below: 1. Recall Bias: Since IILP project activities were launched in July 2011, recall bias could not be excluded. Some respondents had difficulty to accurately recall events that occurred several 6 years previously and were therefore unable to offer meaningful comment from a comparative point of view with more recent events, particularly in the fields of access to finance, business planning, and some issues related to loss of produce caused by pests and diseases. 2. Halo Bias: The extent to which respondents are prepared to reveal their true opinions might have varied for some questions that called upon them to assess the performance of their colleagues or people on whom they depend upon for the provision of services. 3. Evaluation Period: The evaluation has a contractual mandate to review IILP activities from July 2011 to July 2014. However, as mentioned in 3.1.1 above, it did in fact consider data and information through September 2014. In addition, as a further six months have passed since then during which additional information is available to the team regarding IILP’s performance, reference will be made to such information and commented upon with regard to sustainability issues post-IILP in July 2016. Given these time parameters and with approximately 16 months remaining for the project, the ET’s recommendations have been restricted to a number of practical actions that can be implemented by the project within this period – these are highlighted in the content of the relevant questions below. The report will also offer a number of suggestions for USAID with regard to lessons learned. To mitigate the above limitations, the ET gathered data employing various data collection methods and used triangulation to cross-validate results and findings. In addition, it endeavored at all times to confirm confidentiality and anonymity with respect to interviewee responses, in order to increase the probability of unbiased comments. Moreover, FGDs were conducted among peer groups to encourage expression and the development of ideas that might not otherwise be accepted outside of these groups. In summary, while important, the above limitations did not prevent the ET from gathering sufficient information and data needed to draw conclusions and make recommendations in response to the specific questions it was tasked with answering. Following is a summary of the main findings, conclusions and recommendations within each of the four main categories. 4.0 FINDINGS, CONCLUSIONS & RECOMMENDATIONS 4.1 GROUP A: MONITORING AND REPORTING 4.1.1 Question 1 How can USAID Ejo Heza better track the outcome of behavior change activities on beneficiaries? SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources & Methods Used Data Collection Method & Sample Size Limitations 1. How can USAID Ejo Heza better track the outcomes of behavior change activities on beneficiaries?  Description – based on content analysis of project documentation  KIIs with IILP key personnel  Review & analysis of IILP reports  Review/analysis of project commissioned studies/surveys/ other relevant published reports  ET to elicit feedback from key project team members involved in delivery of planned interventions  None encountered 4.1.1.1 FINDINGS The ET found this an extremely difficult question to answer due to the complexities involved in defining and measuring behavior change. Only one of IILP’s indicators – the percentage of 7 beneficiaries consuming a diet with increased diversification – which is under Program Element 5/IR3, is directly related to behavior change. Real behavior change insinuates a permanent change in habits and practices, which is not possible to track in the absence of time-based attitude surveys that measure behavior changes over a given time period. While it is clear from FGD participant comments and HHS responses that recipients of IILP’s various interventions have been influenced by what they learned, IILP does not employ a methodology capable of tracking changes in behavior over time, other than subjective opinions made by those interviewed. Their feedback does, however, provide sufficient evidence to believe that attitudes towards breastfeeding, hygiene and sanitation, and balanced diets have changed for the better as a result of IILP-supported initiatives. Unfortunately, the project only records and reports on outputs rather than outcomes, meaning that its understanding of real behavior change remains largely anecdotal and limited. 4.1.1.2 CONCLUSIONS IILP currently reports on only outputs, not outcomes. This is more a weakness in the project’s design rather than a fault of its implementation as all indicators are effectively based on quantitative data gathering and not on tracking outcomes. While this does not mean that behavior changes have not occurred because of the project’s activities, there is no existing methodology as defined in, for example, USAID’s Performance Indicator Reference Sheets (PIRS), to track the changes. 4.1.1.3 RECOMMENDATIONS In order to track behavior changes, an attitude survey should be conducted to measure project inputs that could directly influence behavior and attitude change, as well as measureable parameters, preferably exclusively related to project inputs, such as children’s weight, number of school days missed, or number of days sick. Unfortunately, however, such a survey is complicated and may prove beyond the project’s ability to introduce during its remaining implementation period. 4.2 GROUP B: EFFICIENCY AND EFFECTIVENESS 4.2.1 Question 1 To what extent planned activities for IILP enable achieving program expected results, purpose and goal? SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources & Methods Used Data Collection Method & Sample Size Limitations 1. To what extent planned activities for IILP enable achieving program expected results, purpose and goal?  Description – based on content analysis of expert opinions & review of relevant documentation  KIIs with IILP key personnel  Review & analysis of IILP reports  Review & analysis of project commissioned studies / surveys / other relevant published report  Feedback from project partners & key stakeholders  ET to meet with IILP team members actively involved in delivery of interventions  Potential interviewees to be identified by ET and USAID on arrival of experts in country  No specific limitations or restrictions to gathering data were encountered during the in￾country mission 4.2.1.1 FINDINGS IILP’s activities are essentially implemented by the four RPOs, STC, and locally based BCVs – whose collective task is to deliver results in-line with agreed, pre-determined performance indicators and targets. They, in turn, work closely with the project’s field officers who oversee data gathered for reporting purposes. IILP has a comprehensive range of activities within its four program elements – financial services, agriculture, literacy, and nutrition – as well as several sub-elements, which feed into USAID’s Economic Opportunities Results Framework,21 as outlined in the original RFA issued in 21 Defined as Assistance Objective 7 – Expanded Economic Opportunities in Rural Areas and supported by IR7.1 – Improved Agriculture Productivity (sub IR7.1.1 & IR7.1.2) and IR7.3 – Increased Access to Agrciultural Finance (sub IR7.3.1,IR7.3.2 & IR7.3.3). 8 February 2011. These activities included providing advice and support for IILP’s beneficiaries on how to: improve agriculture techniques, improve personal and household hygiene habits to reduce illness, read and write, save and borrow money, prepare more balanced diets to reduce malnutrition, and create kitchen gardens to increase intake of diversified foods. Individually and collectively, these activities have enabled the achievement of program goals, particularly ‘expanding access to finance and financial services, empowering rural poor with advanced and/or alternative livelihood skills and increasing food consumption.’ IILP’s reporting focuses on achieving results related to indicators and targets on a quarterly and annual basis (see Annex D for performance results relative to Life of Project (LOP) expected targets). Indicators and sub-indicators within IR7.1 are defined and measured in numerical, quantitative terms, i.e., number of beneficiaries accessing financial services. Those within IR7.3 are similarly quantitative, with the exception of three which require percentage calculations, i.e., the percent of beneficiaries consuming a more diversified diet. Information highlighted in Annex D summarizes performance for the evaluation period and comments on the extent to which each indicator target will be met by the end of the project in July 2016. It is against this background that the following assessment can be made in response to the evaluation question. The ET analyzed all available IILP reporting data22 to understand how it is gathered, how frequently, and the extent to which it is verifiable by those who gather it. Annex E shows annual performance by indicator, clearly demonstrating how each indicator has performed and continued to perform at the time of this evaluation. While reported quantitative results indicate that targets are generally being met, FGD feedback suggests the existence of critical gaps in some activity areas across program elements, for example, lack of integrated savings and loan group (ISLG) linkages to MFIs, extension services not adequately addressing pest control issues and diseases affecting plots and kitchen gardens, lack of evidence supporting the ability to track real behavior change results, and BCVs lacking relevant skills and knowledge to offer practical advice to beneficiaries across a range of issues, including business plan preparation. FGD participants provided the ET with a substantial amount of information on IILP’s activities, including insight into their behaviors and attitudes towards various interventions. There was almost universal agreement from FGD participants that borrowing capabilities remain minimal due to high interest rates, lack of collateral, and lack of ‘bankable’ business plans. In addition, all respondents felt that extension services were limited and, in particular, did not address the pressing pest and disease problems which, during the dry season from September to February, account for significant crop losses of up to 100% (cassava being in most danger). This issue was most prevalent in the Western Province, especially in Rutsiro and Karongi. Although all respondents in all sectors agreed that their lives and those of their families have improved as a result of IILP-supported dietary consumption interventions, including advice on issues such as how to prepare meals using eggs and milk, they could not indicate the extent of improvement. Establishing the true extent of dietary changes resulting from consuming more diversified foods remains problematic and difficult to measure as IILP has no real means of verifying whether reported changes (based on subjective comments) in behavior are temporary or permanent. This explains the low cumulative 26% for IR3 (see Annex D). Based on information from FGDs, it is apparent that personnel from neither RPO nor BCVs possessed the relevant knowledge, skills, and abilities to offer practical advice on how to deal with pests or improve individual farmer borrowing potential (according to MFIs, groups could more easily borrow than individuals due to their lower default rate). 4.2.1.2 CONCLUSIONS Beneficiaries from all sectors unanimously agreed that IILP’s contribution to health, nutrition, and literacy, as defined by indicators and targets, has been successful. The project has been successful in creating kitchen gardens, which are contributing to results in line with expected results and overall goals. They are also prone to crop losses especially during the dry season, which can have an adverse effect on household diets. This issue, therefore, needs to be reviewed to determine a 22 IILP Quarterly Reports, Annual Reports, Work Plan and PMPs as well as various reports from RPOs and STC. 9 strategy to help alleviate this problem. The greatest area of weakness appears to be the financial sector, where loans remain restricted to business development. Although most of IILP’s targets are being met (see Question 4.2.2 below for an in-depth analysis of reported results) and the project will likely meet its end-of-project targets, several activity areas are cause for concern. Given IILP’s current activity portfolio, activities related to MFI loans, farmers applying for new technologies and management practices, and beneficiaries consuming a more diversified diet, will neither meet their final indicator targets nor have a sustainable impact on income earning potential unless addressed within the remaining lifetime of the project. 4.2.1.3 RECOMMENDATIONS To alleviate beneficiary concerns, IILP should immediately review its training programs related to group MFI linkages and extension services, such as kitchen gardens. It would be useful for IILP to conduct a needs assessment to determine what practical actions could be taken to achieve objectives during its remaining implementation period. IILP should consider supporting post-harvest and access-to-market activities, as well as introducing a credit guarantee facility or finance fund for those still unable to access loans from banks or MFIs. A brief feasibility study could be undertaken immediately to determine the cost-effectiveness of such an initiative compared to providing additional training to project partners and implementers in order to facilitate lending to beneficiaries. However, this may not yield the required results in IILP’s remaining time given the financial sector’s frequent lack of urgency to change working practices. 4.2.2 Question 2 How well IILP resources and activities are converted into results, in terms of quality and time, based on cost-benefit comparisons? SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources & Methods Used Data Collection Method & Sample Size Limitations 2. How well IILP resources and activities are converted into results, in terms of quality and time, based on cost-benefit comparisons?  Description – based on content analysis of project documentation  Description – based on content analysis of expert opinions  Review of IILP work plans, quarterly & annual reports & other relevant documents  Feedback from KIIs and project team  All available documents – no exclusions  Selection of key stakeholders - to be identified by ET  Interviews with key project personnel  None encountered  Reliance on anecdotal evidence in support of comments  None encountered 4.2.2.1 FINDINGS It was agreed between USAID/Rwanda and the ET that this question would be interpreted as assessing the efficiency of program implementation by looking at whether money spent is worth the results achieved, rather than a cost-benefit analysis of whether money might have been more effectively spent elsewhere. This proved to be a complicated exercise due to how funds are allocated between program elements and distributed among RPOs, and led to a review of whether differences exist between program elements themselves, in terms of achieving planned results and RPOs as delivery mechanisms of IILP’s main activities. In other words, the question concerns cost-effectiveness defined as “a method of analysis where monetizing outcomes is not possible” (as is the case here where real behavior change regarding health and nutrition cannot be assessed due to other factors that influence behavior and the fact that such changes in behavior must be tracked over time). For purposes of this evaluation: 1) efficiency is defined as “increasing output for a given input or minimizing input for a given output;” and 2) economy as “reducing the cost of resources used for an activity.” Taken together, the question is about value for money defined as “the right balance between economy, efficiency and effectiveness and can’t be assessed through some of these dimensions in isolation.” These distinctions are highly important in answering this question, particularly given the complexity of budget allocations to the RPOs where funds are drawn 10 down annually (from the project) per fiscal year ending in September. To elaborate further and put the entire ‘value for money’ concept into perspective, Figure 1 identifies the real questions the ET needed to address. Figure 1: Economy and Efficiency Considerations Related to Value for Money Source: Extracts from OECD – ‘Value for money in international development’, May 2012 By focusing attention on the above considerations, the ET addressed this question based on the premise that if it does not achieve results, it is not value for money. Upon ET’s request, the IILP Project Team prepared an Excel sheet depicting a breakdown of the project’s implementation budget by program element, key activities, and time period (January 2012 –September 2014 and fiscal year 2015 (FY15) October 2014 – September 2015). This detailed sheet also included: expenditures for commissioned contracts, a breakdown of funds already spent by the RPOs and those planned for the year ending September 2015, and details of funds allocated for behavior change and communication (BCC) activities since January 2012. Following analysis of this data, the ET found the following: Program Elements From January 2012 – September 2014, a total of $923,347 was spent on activities across the four main program elements. 40% of this amount was spent implementing financial services, 27% on agriculture, and the remainder almost equally between adult literacy and health and nutrition. Closer analysis reveals that most of the funds were spent on various training activities for farmers, promotional campaigns, cooperatives, and supporting SMEs. In FY2015 (October 2014–September 2015) $144,923 has been allocated for all four elements with a reduced input of the same activities except kitchen gardens where only 240 model gardens will be supported. In terms of budget allocations, 42% will be used to support health and nutrition while agriculture accounts for 37%. Adult literacy and financial services have an almost equal proportion of the remaining budget. Rwanda Partner Organizations In terms of RPO expenses and sub-awards, the total spent in the period January 2012–September 2014 amounted to $2,349,845 of which 33% was accounted for by ADEPR in delivering literacy￾related activities (in 8 districts), with 30% accounted for by DUHAMIC-ADRI (in 5 districts) in delivering activities related to supporting BCVs and Farmer Field Schools (FFSs). CARITAS (in 3 districts) spent 23% of the total with AEE (in 3 districts) spending 14%. For FY15 (October 2014 – September 2015) a planned budget of $1,049,083 is anticipated for the four RPOs. For this period DUHAMIC-ADRI will account for 31% of the total, AEE 28%, CARITAS 26% and ADEPR 15%. Behavior Change and Communication (BCC) A total of $203,798 was spent on weekly radio programs aimed at promoting the project’s health and nutrition messages. A further $30,086 has been allocated for FY15 (October2014- September2015) and also an additional $53,900 for FY16 (October2015-June2016). These are the financial statistics provided by IILP’s management team and which formed the basis of the ET’s assessment of ‘value for money’ in terms of results achieved to date and those anticipated 11 until September 2015. They also provide important information on how IILP plans to allocate funds by project element and by RPO. In examining this data and having in mind the above questions in Figure 1, the ET was able to draw conclusions relative to each finding, based in part on feedback from interviews, FGDs, and from the HHS conducted by the team. These are listed below and elaborate on the main issue of whether budget spent was worth results achieved and in particular with regard to the Efficiency Considerations highlighted in Figure 1 above. According to the data provided by IILP in its FY15 Annual Report and detailed in Annex E: Overall Monitoring and Evaluation Matrix, most of the project’s performance indicators and targets have been met or likely to be met by September 2015. Notable exceptions are loans made by MFIs (to beneficiaries), number of MSMEs receiving US Government (USG) assistance to access bank loans, and percentage of beneficiaries consuming a diet with increased diversification. If applying the premise of value for money, then it is clear that where results have been achieved, the project has been efficient in implementing its core activities. This applies to outputs (measureable quantitative indicators) and not outcomes (real changes in behavior which are more complicated to track and measure).23 For the purposes of this exercise, the ET has focused on the former. For comparative purposes, Figure 2 below highlights expenditures by program element24 as provided by the IILP project team: Figure 2: Budget Expenditures by Program Element Program Element January 2012 – September2014 FY 2015 (October 2014- September 2015) Budget Used $US % of Total Budget Projected $US % of Total Agriculture $251,013 27% $54,045 37% Adult Literacy 147,555 16% 15,932 11% Financial Services 368,418 40% 14,946 10% Health & Nutrition 156,361 17% 60,500 42% TOTAL $923,347 100% $145,423 100% Source: IILP statistics prepared for the ET The planned FY 2015 budget of $1,049,083 is anticipated to be split between the four RPOs with allocations of 31% to DUHAMIC-ADRI, 28% to AEE, 26% to CARITAS, and 15% to ADEPR. Program Elements Adult Literacy: The evidence suggests that with its allocated budget accounting for 16% of the total program element budget, the Adult Literacy program element (basic and financial literacy) has provided value for money, given that past and ongoing results have been met or are likely to be met through delivery of its current activities in the field. In support of this assertion are the findings from the FGDs, in particular, and from the HHS. Data gathered from the FGDs clearly indicated a 100% level of satisfaction from the literacy groups in all 24 sectors with respect to the project’s training programs. Health and Nutrition: Quantitative targets, related to beneficiaries trained in child health and nutrition, balanced and diversified diets, and creating kitchen gardens, have largely been or are likely to be met by September 2015. However, it should be noted that kitchen gardens are often small and prone to seasonal losses due to pest infestations and/or diseases, which can therefore have an impact on the ability of households to consume balanced diets. With regard to consuming a diet with increased diversification, there is no mention of this in IILP’s FY14 Annual Report although only 26% of the 75% target for that year has been reached. Given the comments above regarding the project’s inability to measure real behavior change, this lack of progress is not unsurprising. In USAID’s PIRS for Feed the Future (FtF) reviewed by the ET there is no reference to indicators directly related to 23 IILP has no present means of determining the extent to which outputs e.g. advice received by beneficiaries in IILP￾supported training sessions on issues such as diets, have led to lasting outcomes, in terms of whether such advice has resulted in a permament change in lifestyle habits or customs by those beneficiaries. 24 Expenditures in the chart do not include RPO implementation costs – see Figure 3 below for details. 12 nutrition and, therefore, no way of assessing IILP’s efficiency (other than results actually achieved so far) with respect to those nutrition indicators, in the absence of any data collection method. On the positive side, over 11,000 kitchen gardens have already been established with IILP’s support, which has exceeded the project’s end-of-project target, indicating a significant achievement in terms of value for money and use of resources for this activity. Financial Services: Accounted for 40% of the program element budget covering a range of activities, including the creation of ISLGs and training beneficiaries in financial matters such as savings and loans. However, here the results achieved appear to contradict actual events in the field. For example, with regard to ISLG – MFI linkages aimed at facilitating borrowing for individuals and small businesses – progress has been minimal. Data gathered from the FGDs indicated that of the 242 groups represented (4669 individual members in total) only 13% of those members were able to borrow from an MFI as opposed to 86% who borrowed from their ISLG. Reasons provided for this anomaly are varied but tend to focus on a fear of applying for loans from MFIs in the event they might default. High interest rates and lack of collateral also featured highly as reasons for not seeking loans from MFIs. Agriculture: This program element included training farmers in agriculture extension and providing support for MSMEs and farmer cooperatives, such as FFSs, to focus on value chain development. Reported performance indicators, specifically those concerning the introduction of new technologies and management practices for farmers and access to agricultural extension services, show that most targets have been or will be met by September 2015. However, while quantitative targets have been met, FGD feedback brought to light a number of important issues to be addressed by IILP on beneficiary perceptions of project activities. Establishing kitchen gardens can be regarded as an important result for the project, even though there is insufficient evidence that they actually provide sufficient crops to raise household intake of nutritious foods. While numeric indicators have been met and therefore can be regarded as a success for the project, there appears to be little in the way of follow-up by the project in dealing with seasonal crop failure. Most beneficiaries interviewed in all sectors mentioned that their crops fail annually because of pests and diseases which, in some cases, result in worse financial well-being for those who had previously borrowed money to purchase tools or seeds and subsequently defaulted. In response to why no one had provided assistance to deal with this problem, it became clear that RPOs, BCVs, and local agronomists were insufficiently qualified to offer practical advice. Beneficiaries mentioned this as a major weakness in IILP’s support. Other beneficiaries discussed how some training sessions on specific subjects were too complicated for them to understand or implement, i.e. the type of seeds to use, how to use fertilizers and chemicals, and guess or estimate productivity improvements due to lack of knowledge on the calculations. ISLGs, cooperatives, and literacy and nutrition groups, particularly in Western Province (Ngororero and Karongi being the worst affected), believe that the apparent lack of professional input from the project’s implementers is, to some extent, restricting real improvements in household livelihoods. RPOs Given the complexity of budget allocations to the four RPOs, actual financial numbers and percentages need to be matched with proportional budget expenditures for each program element. By comparing both budgets it is possible to draw conclusions related to results achieved thus far and to those planned for FY15. Figure 3 below highlights this data for comparative purposes. Figure 3: Budget Expenditures by RPO RPO January 2012 – September2014 FY15 (October 2014- September 2015) No. of Districts Operating In Budget Used $US % of Total Budget Projected $US % of Total CARITAS $535,664 23% $272,440 26% 3 DUHAMIC-ADRI 695,371 30% 326,989 31% 5 AEE 344,096 15% 295,842 28% 3 ADEPR 774,713 33% 153,812 15% 8 TOTAL $2,349845 100% $1,049,083 100% Source: IILP financial reporting statistics prepared for ET 13 From the above information, and with reference to RPO findings above, the ET found the following in considering value for money: CARITAS: Operates in the Western Province (3 districts). Its previous and planned budgets remain similar at around a quarter of total spent for RPOs and projected for the current year. CARITAS is involved in most of IILP’s activities on the ground and although it is difficult to calculate its specific contribution regarding outcomes, there appears no need for any change in its implementation strategy or delivery mechanism. This assertion is also supported by feedback from the FGDs in the districts where CARITAS operates (Ngororero, Rutsiro, Karongi) and where overall satisfaction with its support appears favorable. DUHAMIC-ADRI: Operates in the Southern Province (5 districts) mainly in agriculture related activities, including extension of well off-farm activities such as business development. Its budget accounts for nearly one third of RPOs total budget across both time periods. Its contribution retains similar features to CARITAS although its lack of practical support in dealing with pests and diseases and extension in general remains suspect. AEE: Operates in the Southern Province (3 districts) mainly in financial matters and business development. In the first period, AEE’s budget accounted for 15% of the total RPOs spent. For the current year, FY15 (October 2014-September 2015), its budget has almost doubled. From IILP literature it is not possible to determine the rationale for this increase given that the budget for financial services (including support for ISLGs) has proportionately been reduced to 10% of the total allocated for all four program elements, down from 40% for the first period25 (see Figure 2 above). Given this proportionate reduction in budget allocation for financial services where AEE is most visible, it is hard to conclude that by doubling its allocation, this will yield better results in the 3 districts in which it operates. ADEPR: Operates in both provinces and in all districts. From accounting for 33% of RPOs expenditure in the first period, ADEPR’s budget for the current year has been reduced by more than half to 15% of the total. Given that they are exclusively involved in literacy activities and that literacy targets across the board have largely been met or will be by September 2015, this reduction seems sensible and justified. BCC Although not directly related to any project performance indicators, IILP has allocated a significant budget for promotional activities in order to reach targeted beneficiaries with messages aimed to motivate behavior change on a range of issues including breastfeeding, literacy and financial products. These messages, related to all four program elements, are primarily spread via the Gira Ejo Heza weekly radio programs as well as publication distributions. The ET met with IILP’s media representative responsible for coordinating the pre-recorded radio programs. While the radio stations (three are used for greater coverage – Radio Salus, Radio Isangano, Radio Huguka) involved do not conduct listenership surveys, the media representative advised that field surveys of targeted beneficiaries be conducted once a month to acquire feedback on the programs. These surveys are essentially group interviews. Since no data from the surveys was available to the ET, it seems that IILP has no real way of knowing the extent to which its radio messaging initiative is reaching or influencing the targeted audience. Additionally, most FGD participants have either never listened to the Gira Ejo Heza program or have listened only once. Reasons for not listening included, among others, poor radio signals and that the programs are aired at inconvenient times [Radio Salus at 18.45pm on Saturday evenings, Radio Isangano at 19.45pm (when participants weren’t home), and Radio Huguka at 20.00pm]. 4.2.2.2 CONCLUSIONS Program Elements Adult Literacy: IILP’s adult literacy initiatives have been delivered efficiently within their financial parameters (accounting for only 16% of the total program element budget) and therefore provided value for money. 25 The 40% covers a period of approximately 32 months. 14 Health & Nutrition: Significant progress has been made in achieving results related to child health and nutrition, and kitchen gardens. However, IILP lacks an effective means of measuring and verifying the extent to which beneficiaries have actually consumed a diet with increased diversification because learning to consume more nutritious and balanced diets is not the same as actual consumption. This is a project design weakness inhibiting lack of performance measurements. Therefore, health and nutrition activities can only be regarded as partially representing value for money. Financial Services: Access to finance for the IILP’s beneficiaries has not improved sufficiently to have had any meaningful impact on their standards of living or livelihoods. The ET found weaknesses showing that the project has not been as efficient as it could have been in advancing the cause of enabling beneficiaries to borrow funds to upgrade or expand their agriculture ventures. Agriculture: With more than a quarter of the budget supporting agricultural activities and from beneficiary feedback, it is evident that while targets are generally being met there is a lack of follow￾up from IILP to determine the outcomes of their trainings. Value for money is therefore questionable. RPOs CARITAS: Evidence from the ET’s findings supports the conclusion that CARITAS is meeting the obligations and targets within its remit. DUHAMIC-ADRI: As an implementing organization, DUHAMIC-ADRI appears not to possess the necessary agriculture-based skills to make a real impact assisting beneficiary households. In terms of value for money, its real contribution remains inconclusive as it is also involved in other activities related to groups such as ISLGs and cooperatives. AEE: Given that a significant proportion of AEE’s work is related to finance, its contribution in terms of providing value for money remains questionable given the weaknesses of finance-related activities. ADEPR: It is realistic to conclude that in terms of value for money, ADEPR has efficiently used resources to reach its targeted audience. BCC In the absence of any meaningful feedback from the radio stations that air IILP’s BCC messages, and given the responses from FGD participants, it is unclear why over $200,000 has already been spent in media promotion and another $30,000 is planned for the current year. Even in the absence of an indicator or results to be achieved, the expenditure of such funds without data on whether target audiences are being influenced to change their behavior or, at a minimum, being reached, does not suggest an efficient use of funds or value for money. 4.2.2.3 Recommendations Program Elements Adult Literacy: Given IILP’s success in delivering planned results with its current activities and available budget, no specific recommendations are envisaged for this component. Health and Nutrition: The project’s strategy of concentrating its efforts on the creation of 240 model kitchen gardens appears appropriate in acting as a catalyst for other rural households to emulate. The project should conduct an immediate review of its agriculture-related activities to include more effective extension services including preventative pest and disease control. This should include a more robust working relationship with RAB [perhaps refining their memorandum of understanding (MOU) with IILP] to develop a long-term strategy to tackle pests and diseases, and provide further training of BCVs and local agronomists. Inaction is not an option due to the detrimental economic impact of lost crops (attention to be paid to Season A in particular). With a budget accounting for 42% of all program elements, this allocation needs to be reviewed to determine if it represents real value for money or whether alternative uses could be found in-line with the above recommendation. It is also worth considering whether a more cost-effective option would be to engage the services of a professional agronomist on a short-term fixed contract to develop a pest and disease control strategy to be implemented for the remainder of the project. Financial Services: It is strongly recommended that IILP take the initiative to establish real links between individuals, small businesses, and MFIs. This could include hosting local workshops in the 8 districts to be attended by representatives from the self-help groups (ISLGs, cooperatives etc.) as 15 well as BCVs, RPOs, and project staff. These workshops should help clarify what activities should be implemented that will benefit all concerned parties. In addition, IILP needs to communicate with AMIR, who has oversight of all MFIs in Rwanda and who currently works with OXFAM on horticulture value chains linking beneficiaries with MFIs in the country. During discussions with AMIR’s Executive Secretary, it was apparent that MFIs lack the management capacity to deal with loans and, in most cases, do not actually understand cash-flow management. IILP could cooperate with AMIR to help resolve this weakness and improve loans for project beneficiaries. Agriculture: The project should conduct an immediate review of its agriculture-related activities to include more effective extension services including preventative pest and disease control. In addition, support to agriculture activities, which includes supporting MSMEs and farmers cooperatives, needs to be clarified regarding whether it is directly related to the weaknesses associated with limited beneficiary access to finance, identified above, as more of the same inputs are unlikely to help alleviate the problem. More intensive training of BCVs could represent a more effective use of resources given that they lack the skills and knowledge to offer practical advice on financial matters. RPOs CARITAS: No recommendation. The delivery of activities and results in its districts and sectors meets result indicators and has the satisfaction of beneficiaries. DUHAMIC-ADRI: Given its concentration of efforts on agriculture activities, including extension, it is advisable for the project to take stock of what is currently being implemented and what is planned for the remainder of the current year. Focus should be on pest and disease control, and upgrading the capacity of DUHAMIC-ADRI to be able to offer advice to client beneficiaries. This should also apply to any involvement they have with group linkages to MFIs. AEE: IILP should revisit AEE’s activities and review the near doubling of its budget from the previous period as a proportion of the total program element budget. IILP should also determine whether this expenditure is justified given that AEE only operates in 3 districts and 6 sectors. AEE representatives mentioned that they are involved in several program element activities but were unable to comment on how successful or effective their contributions have been in supporting beneficiaries. IILP would be advised to follow this up for clarification given the proposed allocation of funds for the current period. ADEPR: No recommendation. IILP’s adult literacy activities implemented through ADEPR have largely been successful. As its budget has been reduced accordingly for the current year there is no need for any adjustment. BCC The Project Team should conduct an immediate review of its media expenditures and re-consider whether spending money on an activity with no means to determine the extent or reach of its impact is justified. IILP should also seriously contemplate alternative uses of the funds allocated for the remainder of the current year and in future years. 4.2.3 Question 3 How and to what extent IILP-support financial services increased entrepreneurship (new, profitable SMEs) and incomes among the beneficiaries? Are linkages between beneficiaries and MFIs really taking place? SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources & Methods Used Data Collection Method & Sample Size Limitations 16 SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources & Methods Used Data Collection Method & Sample Size Limitations 3. How and to what extent IILP￾support financial services increased entrepreneurship (new, profitable SMEs) and incomes among the beneficiaries? Are linkages between beneficiaries and MFIs really taking place?  Description – based on content analysis of project documentation  Description – based on content analysis of expert opinions  KIIs with IILP key personnel  Review & analysis of IILP reports  KIIs with representatives of relevant financial institutions  Feedback from FGDs based on completion of structured questionnaires relevant to direct beneficiaries All available documents – no exclusions ET to identify and select relevant interviewees (input from IILP team members will assist in process) – especially MFIs To be organized by ET in sample regions – target = max. of 12 per focus group session including representatives from all categories of participants especially women None envisaged MFIs may not gather and record data that is sufficiently robust enough to inform ET regarding business performance or income generation of ultimate beneficiaries Probability that participants do not record data that is verifiable regarding income – responses likely to be anecdotal or basic observations Reliance on anecdotal evidence 4.2.3.1 FINDINGS Entrepreneurship and Incomes Among Beneficiaries Two of IILP’s important interventions are increasing access to financial services and fostering entrepreneurship development for impoverished rural families in target districts and sectors in Western and Southern Provinces. These are achieved through the organization and capacity building of ISLGs, financial literacy training, and creation of sustainable linkages between ISLGs and MFIs to acquire access to high level services, including increased loans to group members to ensure they are able to make sound investments. Furthermore, IILP was designed to identify financial partners with an interest in targeting project beneficiaries as potential clients, and to collaborate with them to expand financial services to rural areas through the development of relevant new services. According to IILP’s work plan, it was expected that 48,420 beneficiaries would be trained in financial literacy, 70% of ISLGs would be created and connected to financial institutions, 40,000 beneficiaries would have access to financial services, and 3,600 MSMEs would receive USG assistance to access banks loans. The ET found that by September 2104, 34,148 beneficiaries had received training in financial literacy, 68% of ISLGs were created and connected to financial institutions, 31,391 beneficiaries had access to financial services, and 2,298 MSMEs were able to access bank loans. Feedback from FGDs, which were comprised of representatives from all 242 groups with a total of 4,623 members, indicated that 13% borrowed from an MFI and the rest from an ISLG. Reasons for this mix include high interest rates and inability to provide collateral. ISLGs provide the opportunity for project beneficiaries to graduate from an informal to a more traditional financial system. However, the connection between self-help groups and MFIs remains tenuous, with individual members still preferring to borrow from their ‘own’. At the same time, groups themselves can sometimes borrow from MFIs on behalf of their members, as MFIs consider them less likely to default due to the internal pressures that group members place upon one other. 17 63% of HHS respondents confirmed that IILP’s interventions26 contributed to an increase of income, especially for women. This increase in income was mainly attributed to the organization of ISLGs, which contributed significantly to savings and the ability of group members to access affordable small loans for production. The average income increase was 63.99%, with women’s increasing by an average of 72.67% and men’s by 52.19%. 197 of the 264 HHS respondents reported that they were able to access a loan. Loans were used for various purposes27, including expansion of agricultural land, farm renovation, purchase of agricultural inputs including tools, purchase of livestock, investing in off-farm activities, paying school fees, house renovations, investing in off-farm among activities and health insurance among others. Figure 4 details the percentage of respondents reporting changes in income as a result of IILP. Figure 4: Increase in Incomes Resulting from IILP Response % Reported by Respondents Increase in Income 91.3% No Increase in Income 8.0% No Change in Income 0.8% (Note: Table reports on number of people who have increased their incomes not how much income has increased) Feedback from FGDs also indicated that 95% of attendees learned about how to save money and reported that borrowing from ISLGs was the only real alternative for them to access credit. Additional findings confirmed that 86% of respondents obtained credit from internal lending through ISLGs while only 13.30% of respondents received loans from MFIs. Figure 5 details the percentage of respondents reporting whether ISLGs contributed to their savings and access to small loans. Figure 5: ISLG Contribution to Savings and Access to Small Loans Response % Reported by Respondents Did Contribute 92.8% Contributed Less 4.5% Did Not Contribute 2.7% In response to the question of whether IILP’s interventions have helped increase entrepreneurship and new profitable SMEs, 15% of respondents confirmed having started a new business (36% of men and 64% of women), while 65.5% of respondents invested in small commercial agriculture to expand agriculture land, farm renovation, purchase agriculture inputs, tools, and livestock, and extend markets. As shown in Figure 6 below, off-farm business activities comprised the majority of new businesses started, with technical activities comprising far less. Figure 6: Entrepreneurship Related to IILP Interventions Type of New Business Started % Reported by Respondents Off-Farm Business Activities Small Retail Shop 30.77% Sorghum Beer Making 23.08% Manufacturing Banana Juice or Beer 20.51% Purchasing and Wholesaling Crops 15.38% Technical Activities Hairdressing and Sewing 10.26% Total 100% Linkages Between Beneficiaries and MFIs Linkages between beneficiaries and MFIs are not significant. FGDs feedback indicated that 95% of respondents were unable to conduct business with commercial banks or MFIs due to high interest rates, insurance requirements, compulsory deposits, lack of collateral, or unrealistic business plans. 26 Important to note that interventions are based on demand so that not all beneficiaries need or receive / participate in all interventions which is the underlying concept of the IILP. 27 65.5% of all loans were used for agricultural purposes. 18 Interviewed SACCO staff confirmed that interest rates ranged from 24% to 30% per year and admitted that the majority of loan applicants did not meet MFIs’ criteria of being viable and capable of generating sales or cash flow to repay loans. Only 12% of all ISLG individual members had access to small loans from MFIs: 18% in Western Province and 9% in Southern Province, with 48% granted to women. 4.2.3.2 CONCLUSIONS Entrepreneurship and Incomes among Beneficiaries IILP has performed well in organizing and strengthening ISLGs as a means of access to finance for poor rural families. However, the collateral requirements and high interest rates demanded by MFIs continue to limit borrowing and income potential. ISLGs provide the opportunity for the project’s beneficiaries to graduate from an informal to a more traditional financial system. However, the connection between self-help groups and MFIs remains tenuous, with individual members still preferring to borrow from their ‘own’. At the same time, groups themselves can sometimes borrow from MFIs on behalf of all their members, as MFIs consider them less likely to default due to internal pressures that group members place upon one other. Linkages Between Beneficiaries and MFIs The outreach of ISLG-MFI linkages is weak and represents a very small intervention outcome in the actual project results. In other words, a systemic lending approach to project beneficiaries through MFIs, in order to ensure the expansion of financial services among the rural poor to improve the performance of small-scale agricultural enterprises, is still lacking. 4.2.3.3 RECOMMENDATIONS Entrepreneurship and Incomes among Beneficiaries IILP should consider creating a credit guarantee fund for ISLGs and upgrading capacity building measures to improve business planning and project management. All interviewed MFIs expressed interest in the possibility of working with a credit guarantee scheme, currently unavailable to them, to reduce the lending risks for larger projects. A formal cooperation agreement between IILP and MFIs is highly recommended in order to address lending limitations for the project’s beneficiaries (see 4.2.2 Question 2 above which is connected to this question). Linkages Between Beneficiaries and MFIs The project should seriously consider focusing on the development of business clusters through partnerships between ISLGs, MFIs, and local service providers based on pre-selected, high potential business areas. Small-scale farmers and their emerging enterprises require local services to help them integrate into value chains (chain empowerment) and relate to other chain actors (traders, processors) and services (value chain finance). The potential benefits of business clusters for smallholder farmers include increased entrepreneurial knowledge, better market linkages, enhanced access to credit, and better marketing opportunities. 4.2.4 Question 4 How and to what extent IILP’s agricultural extension services contributed to household crop production and incomes? SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources & Methods Used Data Collection Method & Sample Size Limitations 4. How and to what extent IILP’s agricultural extension services contributed to household crop production and incomes?  Description – based on content analysis of project documentation  Description – based on content analysis of expert opinions  KIIs with IILP key personnel  Review/analysis of IILP reports & other relevant publications  KIIs with representatives of extension service businesses All available documents – no exclusions ET to identify and select interviewees (input from IILP will assist the process)  None encountered  Limited means of correlation between IILP’s agriculture extension services and real￾time improvement in household crop 19 SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources & Methods Used Data Collection Method & Sample Size Limitations production and incomes 4.2.4.1 FINDINGS Feedback from FGDs indicated that IILP’s trainings in farming technology, such as the integrated use of fertilizers, planting in rows, seed selection, and harvesting, have contributed to improved agricultural practices and to increased production and incomes. This is further supported by the HHS which showed, for example, that the average annual maize production increased from 144.55 kg to 285.98 kg per farmer, realizing an average increase of 96.45% (see Figure 7) over a three year period (2012-2014). Males, with an average increase of 109.78%, realized a much higher increase than females, of 87.15%. Figure 7: Average Annual Production of Maize Before and After IILP Intervention Amount in kg Male Female Total Current Previous Current Previous Current Previous # % # % # % # % # % # % <50 2 1.90 22 21.4 5 3.10 49 30.40 7 2.70 71 26.90 50-100 24 23.30 36 35.00 35 21.70 60 37.30 59 23.30 96 36.40 100-250 32 31.10 24 23.30 66 41.00 27 16.80 98 37.10 51 19.30 250-500 21 20.40 15 14.60 30 18.60 14 8.70 51 19.30 29 11.00 500-750 7 6.80 3 2.90 10 6.20 5 3.10 17 6.40 8 3.00 750-1000 9 8.70 3 2.90 4 2.50 6 2.50 13 4.90 7 2.70 >1000 8 7.80 0 0.00 11 6.80 2 1.20 19 7.20 2 0.80 Average Current Income 322.82 153.88 262.42 140.22 285.98 145. 55 Increase 168.93 122.20 140.44 % Increase 109.78 87.15 96.49 As described in Figure 8 below, the increase in crop production was largely influenced by project trainings. Of the major contributions IILP has made to farm production, 32.2% of the HHS respondents attributed the increase in production to the proper use of agricultural inputs, such as improved seeds, integrated use of fertilizers, and combination of organic and synthetic fertilizers; 26.5% attributed it to proper farm management; and 13.3% to improved agro-techniques, including planting in rows. It is worth noting that, at 0.80%, the fewest number of respondents attributed an increase in production to kitchen gardens. Although most respondents agreed that production had increased, such increases were offset by the significant and regular crop loss that all interviewed farmers experienced. Farmers reported losing between 20% and 100% of their crops during the growing or post-harvest seasons due to pest infestations, and lack of inputs, equipment, and storage facilities. Respondents felt that none of the BCVs, RPOs, or local agronomists were suitably qualified to deal with these issues, possibly because of their lack of relevant training. The ET found no reference to this issue in any of the project’s reporting literature. Figure 8 shows the number and percentage of individuals who attributed each factor to increases in production. 20 Figure 8: Increases in Production by Influencing Factor Factor Male Female Total # of Responses Percent # of Responses Percent # of Responses Percent Good Weather 6 5.80% 20 12.40% 26 9.80% Exetnsion Services 4 3.90% 12 7.50% 16 6.10% Equipment 3 2.90% 4 2.50% 7 2.70% Proper Farm Management 26 25.20% 44 27.30% 70 26.50% Proper Input Usage 35 34.00% 50 31.10% 85 32.20% Increased Land 4 3.90% 6 3.70% 10 3.80% Promising Market 6 5.80% 3 1.90% 9 3.40% Improved Agro Techniques 16 15.50% 19 11.80% 35 13.30% Kitchen Gardens 1 1% 1 0.60% 2 0.80% Figure 9, below, depicts income increases which are closely aligned with production. The average annual income per household increased from 163,087 Rwf to 267,443 Rwf – an increase of 104,356 Rfw equivalent to 63.99% of the total. It is notable that female incomes increased proportionately more than those of males, demonstrating that the project has not only contributed to an increase in household income but also to female empowerement. This can be attributed to the fact that IILP specifically encouraged women to become more involved and active in implementing farming techniques (especially planting in rows rather than the more traditional scattering of seeds). In Figure 9 below, the decrease in numbers at the lower end of the current kg range indicates that individual farmers are increasing their annual production. Figure 9: Annual Income Increases in Relation to Production in Kilograms (kgs) Amount in kg Male Female Total Current Previous Current Previous Current Previous # of People % # of People % # of People % # of People % # of People % # of People % <10,000 0 0.0% 2 20.0% 1 100% 8 80.0% 1 100% 10 100% 10,000-20,000 0 0.0% 10 40.0% 0 0% 12 60.0% 0 0% 20 100% 20,000-30,000 4 57.1% 2 20.0% 3 42.9% 8 80.0% 7 100% 10 100% 30,000-40,000 3 42.9% 7 38.9% 4 57.1% 11 61.1% 7 100% 18 100% 40,000-50,000 5 33.3% 13 43.3% 10 66.7% 17 56.7% 15 100% 30 100% 50,000-100,000 13 43.3% 16 32.7% 17 56.7% 33 67.3% 30 100% 49 100% 100,000-500,000 49 35.3% 45 42.1% 90 64.7% 62 57.9% 139 100% 107 100% >500,000 29 44.6% 10 50.0% 36 55.4% 10 50.0% 65 100% 20 100% Average Income 93,738 RWF 273,350 179,612 263,665 152,702 267,443 163,087 Increase RWF 93,738 110,963 % of Increase 51.2% 72.7% 4.2.4.2 CONCLUSIONS FGD and HHS responses indicate a satisfactory level of crop production resulting from IILP’s training programs. However, the project’s package of extension services does not adequately address serious issues such as pest and disease as well as post-harvest losses. The current follow-up monitoring of extension services is not sufficient to inform remedial measures and remaining at status quo is not an option. 4.2.4.3 RECOMMENDATIONS IILP should revise its MOU with RAB in order to provide essential technical assistance related to pest and disease management, and should consider contracting out an external consultant to advise on a strategy to tackle this problem 4.3 GROUP C: IMPACT AND RELEVANCE 4.3.1 Question 1 How and to what extent has work in promoting and strengthening income generating activities contributed to household food security? 21 SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources & Methods Used Data Collection Method & Sample Size Limitations 1. How and to what extent has work in promoting and strengthening income generating activities contributed to household food security?  Description – based on content analysis of project documentation  Description – based on content analysis of expert opinions  Review of IILP work plans, quarterly & annual reports & other relevant documents  Feedback from KIIs  Feedback from Project Team  Feedback from FGDs on completion of structured questionnaires to particular participants – i.e. direct beneficiaries or intermediaries  Feedback from HHS  All available documents – no exclusions  Selection of key stakeholders – to be identified by ET  Interviews with key project personnel  To be organized by ET in sample regions – target = max. of 12 per focus group session including representatives from all categories of participants especially women  None envisaged  Reliance on anecdotal evidence in support of comments  None envisaged  Limited availability of sex disaggregated data  Unavailability of participants  Reliance on anecdotal evidence 4.3.1.1 FINDINGS The ET interpreted this question as the extent to which IILP’s activities supported increased agriculture production, which resulted in increased income opportunities for targeted beneficiaries in rural farming communities and, as a consequence, contributed to food security. The ET used the World Health Organization28 (WHO) definition of food security as its point of reference. Since its inception in July 2011, IILP has addressed the demand-side issues that limit the use of financial services to improve livelihoods and food consumption by promoting savings and use of credit, financial literacy, and basic literacy and numeracy, while also expanding access to business development and agricultural extension services. Activities related to increasing demand for financial services are conducted through ISLGs, while activities in agriculture are conducted through cooperatives that focus on the three main value chains of maize, beans, and dairy. Activities are implemented in partnership with and through three RPOs – AEE, CARITAS, and DUHAMIC ADRI – through training-of-trainer initiatives and coaching BCVs. All of IILP’s activities have contributed to households improving their food security to some degree, although not in equal measure. For rural farmers, food security can be increased in two ways: 1) producing more of what they currently produce, and/or cultivating diversified crops for more nutritious, balanced, and healthy diets; or 2) earning enough income from selling excess production from existing plots and/or starting a new business, either agriculture related or not, to increase overall income potential. From the evidence gathered, IILP’s more important contribution to increasing income has resulted from the creation of ISLGs which, according to FGDs, resulted in 4,017 members (2,768 women and 1,249 men) borrowing money from their own ISLG. Although it was not possible to determine exactly where the loans were spent, some of the money was used for farm activities, such as upgrading existing facilities, leasing land, or purchasing tools and agriculture inputs such as chemicals and fertilizers, and for non-farm activities, such as opening a local grocery store and paying school fees. The HHS revealed that a majority of the 75% of beneficiaries interviewed who took loans, irrespective of the source (ISLG or banks/MFI), used the money for agricultural purposes in order to increase food production and ultimately food security. 28 The World Health Organisation defines food security as existing ‘when all people at all times have access to sufficient, safe, nutritious food to maintain a healthy and active life’. 22 In attempting to compare agricultural production, the ET found that average production per farm from the start of the project to September 2014, increased significantly. The percentage of farms with an annual output of less than 50kg decreased from 26.9% (2011) to 2.7% (2014), in the same period, the percentage of farms with an annual output in the 50-100 kg range decreased from 36% to 22.3%, and the percentage of farms with an annual output in the 100,000-500,000kg and over 500,000kg ranges both increased significantly and saw proportionate increases in income (see Figure 9 above for a breakdown of statistics per kg category). In other words, production output per kg has progressively increased from lower levels of output to 20,000kgs per year and above with respective increases in income. This means that larger scale operations are becoming more prevalent and provide evidence that farmers have benefited from IILP’s support in its first 3 years of implementation (and according to interview comments). Increased production and the ability to market excess output have enabled some farmers to increase their incomes, which has allowed them to buy additional food such as eggs and milk (often for the first time) and, therefore, contributing to food security. Despite this positive production trend, 41.9% of respondents mentioned that they still face food shortages throughout the year due to a variety of reasons detailed in Figure 10, below29. The ET found it difficult to compare data for food shortages before and at the mid-term stage of the project as the baseline30 (pp10-37) only provides the narrative description of source of food among wealth Categories 1 – 5 and does not express statistics in terms of percentage of households facing food shortages before the project. It highlights instead that wealth Categories 1 - 3 are short of food at some points of the year. It is therefore difficult to assess to what extent the 41.9% represents an increase or a decrease in households facing food shortages. Additionally, the fact that the high response on the cause of food shortage is due to variable climatic conditions followed by low production is an indication that project beneficiaries are still relying on agriculture as their main source of food and not using household income to purchase adequate food from markets throughout the year which contradicts the project’s theory of change, which states that, ‘expansion of appropriate access to finance, increased economic opportunities and incomes will enhance food consumption by people living in the eight most poor and food insecure districts of South Western Rwanda’. The link between increased agricultural production and increased income from sales (off and on-farm), and their combined contribution to food security highlighted by this theory, seems to be lacking. The increase of income level of 63.9% (as reported by 91.3% of households) does not tally with the level of expenditure on food; the HHS demonstrated that only 29% of HH income was spent on food purchases. The three most food insecure districts, with the highest percentage of households only able to have one meal per day are: 22.9% in Karongi, 12.5% in Gisagara, and 10% in Huye. The contribution of kitchen gardens to diversified food was rated as important by only 29% of respondents. Furthermore, dairy value chain contribution to food security of the targeted households was found to be non-existent, highlighting a challenge remaining for IILP in its final stages. Figure 10: Food Shortage Causes Cause % Reported by Respondents Variable Climatic Conditions 15.2% Low Production 9.8% Limited Income 6.4% Post-Harvest Losses 6.4% Market Shortages 6.1% Overall, respondents viewed loans largely as a means to improve their livelihoods, and thus food security, through increased plot production. However, although this development can be viewed as positive in terms of using loans, the same FGD respondents indicated that while IILP’s support in 29 It is important to differentiate between the percentage of income increases and the number of people reporting that their income increase. Earlier in the report it mentions 63% which is the percentage of increase in income not the number of people claiming to have increased their incomes. 30 Reference – ‘Ejo Heza Livelihoods Baselines: Southern and Wetsern Provinces’ report commissioned by IILP. 23 creating kitchen gardens contributed to diversified crops for household consumption, it did not lead to income-generating opportunities with the exception of a few cases. This was due to the small size of kitchen gardens and the on-going problem associated with pests and diseases, as yet unaddressed by the project. Feedback on IILP’s extension services remains weak and more related to the provision of advice than the implementation of practical and affordable measures to improve productivity and output. Furthermore, it was clear from FGD participants’ comments that they did not have enough knowledge on how to market their products, determine prices that would allow a profit, assess the size of their potential customer base, and organize distribution, especially from remote locations. 4.3.1.2 CONCLUSIONS IILP’s initiatives regarding the demand for financial services, especially from ISLGs, are contributing partially to increased food security. However, there remain a significant proportion of households facing food insecurity due to: 1. The low rate of farmers implementing extension services. By September 2014, 58,994 farmers had accessed extension services through IILP’s training programs although only 26,499 were able to adopt any of the agriculture technologies on which they had been instructed. Feedback from FGD participants indicated that this was due to a number of reasons including not being able to afford seeds, tools or equipment or in fact not understanding what they had been taught. 2. Higher consumption of own produced food at the farm level and, irrefutably, the absence of both strong market orientation (for increased income and diversified food) and dairy-related interventions. Low maize and bean outputs from farms and vegetables from kitchen gardens do not offer diversity in the preparation of nutritious meals with balanced diets. 4.3.1.3 RECOMMENDATIONS IILP should consider expanding its income strengthening work to orient both on- and off-farm initiatives to improve market access and increase the potential for households to access supplementary foods, such as meat, milk, fish, eggs, and fruit, from markets. The project’s extension model should be upgraded to increase the number of farmers actually putting into practice what they have learned in theory; keeping in mind the limited availability of funds for extension inputs (supplies of raw materials, tools and machinery) for farmers. This entails providing more intensive training in farming-for-business, change in agro-entrepreneurship mind-set, grouping ISLGs to adhere to agricultural cooperatives, and promotion of farmer-to-farmer technology transfer within the cooperatives. This will require IILP to consider, on the one hand, having RPOs’ technical and field staff provide intensive proximity coaching for BCVs and model farmers, and on the other, subcontracting business development service (BDS) providers with expertise in business coaching and market linkages. 4.3.2 Question 2 How and to what extent IILP nutrition education activities contributed to the improved household status, especially for women and children? Are the used kitchen garden and nutrition messaging approaches effective to reduce malnutrition or need for improvement and/or other alternatives? SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources & Methods Used Data Collection Method & Sample Size Limitations 2. How and to what extent IILP’s nutrition education activities contributed to the improved household status, especially for  Description – based on content analysis of project documentation  KIIs with IILP key personnel & review of IILP documents  Review & analysis of IILP reports & other relevant publications including published studies & survey findings from Ejo Heza (KAP Study) &  All available documents – no exclusions  All available documents – no exclusions (specifically designed questionnaires will be employed to include questions related height / weight pre- and post-  None envisaged  None envisaged  None envisaged subject to availability of 24 SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources & Methods Used Data Collection Method & Sample Size Limitations women and children? Are the used kitchen garden and nutrition messaging approaches effective to reduce malnutrition or need for improvement and/or other alternatives? other bodies  Interviews with representatives of STC, Ministry of Health, nutrition groups & media organizations  Design & implementation of HHS (format and structure to be decided) targeted at IILP primary beneficiaries  Interviews with media organizations involved in promoting nutrition messages  Discussions with direct beneficiaries & direct observation of household nutrition project intervention)  ET to organize face-to￾face meetings with identified interviewees (IILP to assist with suggested individuals  ET to discuss with IILP team and USAID re: type, format and timing of survey  ET to discuss with IILP project team to identify relevant media publishers or radio presenters  Conduct a HHS & FGDs with beneficiaries to capture their perceptions potential interviewees  Identification & agreement on sample size & planned methodology to reach to reach targeted audience. Time constraint considerations to gather & analyze results  None encountered  None encountered 4.3.2.1 FINDINGS Nutrition Education Activities In answering this question the ET needed first to define a few key terms to be able to comment meaningfully on the extent to which women and children have been influenced by IILP’s nutrition education activities. It was decided that: 1) a household31 would be treated as “all persons who occupy a housing unit,” as per the United States Census Bureau; 2) status32 would be considered “the role of women in the household relative to their male counterparts, primarily in terms of their income generating abilities and potential as the primary area of investigation.” The ET also decided that status regarding children would focus more on the extent to which illnesses and malnutrition had been reduced. Feedback from the FGDs and HHS served to inform the team on both matters although it cannot be assumed that there is in fact a direct correlation between the household status of both and the project’s interventions, due to other influencing factors. This was also taken into account by the ET when commenting on the project’s contribution. In short, it is possible to determine the how but not necessarily the extent, which will by default restrict findings to the former. The extent will be limited to a balanced assessment by the ET on feedback from stakeholders and field visit activities. The ET found that all of IILP’s nutrition activities are directly linked with both education and extension services. The main aim of this approach was to ensure that vulnerable households had greater food security and understanding of diets and hygiene, especially during the first 1,000 days of life. More specifically, nutrition is incorporated into adult literacy classes, training on agricultural production, storage and handling and establishing kitchen gardens. The project has four performance-related indicators within IR3-Improved Nutrition Among the Poor: 31 The US Cencus defines a household as: ‘A household includes all the persons who occupy a housing unit. A housing unit is a house, an apartment, a mobile home, a group of rooms, or a single room that is occupied (or if vacant, is intended for occupancy) as separate living quarters. Separate living quarters are those in which the occupants live and eat separately from any other persons in the building and which have direct access from the outside of the building or through a common hall.” 32 In this context status refers mainly to the position of women in the family environment with emphasis on the economic role they play as income earners. The status of children refers to their general wellbeing. 25 1. Beneficiaries consuming a diet with increased diversification 2. Beneficiaries with increased knowledge about balanced and diversified diet 3. People trained in child health and nutrition 4. Kitchen gardens established In reviewing the project’s reported results regarding these indicators, all targets were met or surpassed in FY14 (October 2013-September 2014), with the exception of the first regarding consumption of diets with increased diversification, which has already been noted as problematic in this report. Specific activities undertaken by the project in FY14 fall into the following categories: 1. Incorporating nutrition into adult literacy education Trainings within this activity include: exclusive breastfeeding, food recipes, general and personal hygiene and kitchen gardens. 2. Improving food production, post-harvest storage and handling The project worked with DUHAMIC-ADRI on the construction of drying sheds in Huye and Gisagar districts, by supplying construction materials (stones, timber, cement), with members of local cooperatives supplying labor. This has resulted in some beneficiaries being able to sell excess maize yields for milling. 3. Establishing kitchen gardens Creation of kitchen gardens basically inform households on how grow nutritious crops, especially vegetables (lettuce, cabbage, carrots) in confined spaces. Although all targets have been met regarding kitchen gardens more needs to be done in terms of irrigation and storage to ensure all-year-round production as a contributing factor leading to food security. In addition, the project undertook activities related to promoting better utilization of food. An assessment of nutrition activities was conducted with the help of CARITAS and DUHAMIC-ADRI to identify strengths and weaknesses of the groups and to assess technical support needs. A mapping exercise was also conducted to determine if nutrition groups still existed and were conducting health and nutrition activities in line with RPOs reported data. This exercise found that some nutrition groups were graduating to ISLGs, a positive development, and that some BCVs were not up to speed in receiving training in health and nutrition. The ET has not conducted an in-depth analysis of how this was followed up as it does not fall within the evaluation timeframe. However, it was noted that for the remaining project time period it would focus on the construction of model kitchen gardens as a strategy of encouraging neighbors to emulate them and thus increase participation among communities from sector to cell level. Findings from FGDs and data obtained from the health centers visited during the evaluation field visits revealed that there have been improvements in the nutrition status of beneficiaries. All nutrition group participants in all 24 sectors visited agreed without exception that IILP’s support activities had a direct influence on changing their behavior in line with the advice and assistance received. All literacy groups similarly commented that being able to read and write allowed them to understand written instructions in leaflets and pamphlets on issues related to nutrition. BCVs also indicated that they felt more informed on nutrition issues not only on their own behalf but in terms of the quality of support they could offer to the beneficiaries. The HHS provided solid data on malnutrition affecting children. It recorded that the percentage of households with severe and mild malnourished children had decreased (see Figure 11 below), as had those with other common conditions such as diarrhea, skin infections, and digestive diseases associated with poor diets and malnutrition. Figure 11: Common Household Diseases Disease Current Prevalence Previous Prevalence # of Individuals % of individuals # of individuals % of individuals Malnutrition 8 3.10% 18 6.82% Diarrhea 63 23.90% 73 27.65% Skin Infections 35 13.40% 28 10.61% Digestive Diseases 11 4.20% 19 7.20% Other 65 24.60% 71 26.89% Note: The numbers relate to individual households with or without children 26 According to household interviewees, these decreases were attributable to the project’s nutrition education activities, which included training in the preparation of a balanced diet, exclusive 1,000 days of breastfeeding, proper child and mother care, and proper hygiene and sanitation practices. In addition, the creation of kitchen gardens has increased food varieties on the plate through an increased consumption of green leafy vegetables, carrots, cabbage, beetroots, and onions. This has contributed to a decrease in the incidence of micronutrient deficiencies associated with vitamin A and zinc. To date, more than 11,000 kitchen gardens have been established. However, it should be noted that the decreased rates of malnutrition cannot be exclusively attributed to IILP’s interventions because the project has no way of determining the nature or extent of other factors influencing household behavior towards dealing with illnesses and diseases due to their medical nature which is not within the project’s mandate. There appears to be a correlation between children’s health and wellbeing and household status. Women’s status has also improved, especially with regard to kitchen gardens, which are essentially their exclusive domain. Gardens that have been able to produce excess crops for sale, and thus extra income, have not only added to women’s self-esteem but have also sufficiently impressed male household members (including heads of households) whose respect for these women (wives) has positively shifted. This is an important development in rural and often remote areas where traditional habits and attitudes are difficult to change. Kitchen Garden and Nutrition Messaging Approaches The ET could find no specific activities related to kitchen garden messaging other than those delivered through formal training sessions with BCVs and the groups. Model gardens were designed to show neighbors what could be cultivated and how, in order to benefit their health and wellbeing, as well as offering the potential to sell excess output to increase income. In rural communities, word of mouth messaging is often more effective than other mediums as it is based on trust and personal knowledge. IILP’s strategy in relation to nutrition messaging has focused on radio as the medium to disseminate information. Various themes have already been broadcast on health and nutrition, breastfeeding, and the importance of milk, as well as related topics on literacy and balanced diets. However, the radio stations that broadcast the messages did not have a media coverage strategy or means of obtaining feedback on listener responses to the radio programs and their messages. What are known are the responses from the FGD participants who, almost unanimously, agreed that the broadcast timing was problematic (Wednesday and Saturday evenings) as they were not home. Some had listened to a program once but had not been inspired to listen again, either due to not understanding the message or to disinterest in the subject matter. In the absence of verifiable audience data and responses to structured attitude surveys post-program, it cannot be said that IILP’s nutrition messages are having a major impact on their intended audience nor that such approaches have any impact on the household status of women and children within the abovementioned definitions. Nutrition messaging approaches seem to have had little effect on the decrease in malnutrition. This medium appears not to have effectively reached the population and most of the messages focused on group savings and increasing agriculture production – there was no mention of proper nutrition in these radio programs. Moreover, over half of respondents from the HHS and FGDs claimed not to have listened to these radio programs – 49.2% of respondents listened only once to the program, while the remaining 50.8% had never listened at all. The respondents who reported listening at least once said that they had difficulties following this program regularly due to the inconvenient broadcasting time and inadequate programs content. 4.3.2.2 CONCLUSIONS Nutrition Education Activities It is apparent that the level of malnutrition among the project’s beneficiaries has decreased considerably. Nutrition education has played a role in increasing knowledge and awareness in households while kitchen gardens have contributed to an increase in household food varieties and quantities. However, these interventions lack a livestock-crop combination to provide beneficiaries with a guaranteed source of animal products to increase levels of protein consumption. There is no verifiable evidence indicating that IILP’s radio programs, with their various messages including health 27 and nutrition, have any major impact on household behavior or attitudes, at least according to feedback form FGD participants. Kitchen Garden and Nutrition Messaging Approaches There is no verifiable evidence from FGD participants indicating that IILP’s radio programs, with their various messages including health and nutrition, have any major impact on household behavior or attitudes. The project is continuing to spend part of its budget on these programs, which should now be reviewed as to purpose and impact. 4.3.2.3 RECOMMENDATIONS Nutrition Education Activities Continue with the strategy of creating model kitchen gardens and add a component to help vulnerable families access loans to buy chickens or goats to reduce the protein gap. Kitchen Garden and Nutrition Messaging Approaches The project should revise its radio messaging strategy to consider partnering with local popular drama programs (e.g. Urunana – message inclusion in script) or review its entire media strategy (public relations and advertising) as an effective means of reaching and/or influencing targeted beneficiaries. 4.3.3 Question 3 To what extent are the activities implemented by IILP contributing to increased participation, improved income and overall “empowerment” of women in the targeted communities? What are women’s perceptions or feelings on their current situation? SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources & Methods Used Data Collection Method & Sample Size Limitations 3. To what extent are the activities implemented by the IILP contributing to increased participation, improved income and overall “empowerment ” of women in the targeted communities? What are women’s perceptions or feelings on their current situation?  Description – based on content analysis of project documentation  Description – based on content analysis of expert opinions  KIIs with IILP key personnel  Review & analysis of IILP reports & other relevant publications  KIIs especially with representative s of women’s groups / organizations such as Rwanda Women Network All available documents – no exclusions ET to discuss with IILP / USAID optimum method of reaching & gathering data – to include FGDs, direct interviews using structured questionnaires &/or conducting survey  None envisaged  Determining size of sample if survey conducted in order to gather & analyze responses within time￾frame of evaluation mission in country  Limited means of correlating women ‘empowerme nt’ with IILP activities 4.3.3.1 FINDINGS According to the CHF International/USAID Cooperative Agreement, “Ejo Heza will improve the livelihoods and food consumption of Rwanda’s very poor, particularly women, by building the capacity of low income households to position themselves effectively to access the appropriate financial services necessary to gwo their enterprises.” Furthermore, in order to ensure sustainaibility of the project’s 28 interventions it will, ‘promote gender equity by ensuring women are proportionally represented in groups supported by Ejo Heza and resulting learning opportunities and that they are utilized as trainers within their communities’. The project’s frame of reference for all gender related interventions was to be based on the findings of two studies commissioned by IILP following start-up: 1) Ejo Heza Libelihood Baselines: Southern and Western Provinces;33 and 2) Poverty Assessment Tools Survey (PAT).34 As a cross-cutting issue, IILP has no indicators/results that need to be met. Instead, analysis would be made where possible regarding women’s access to resources like land, livestock, credit, education and health care, although this does not mean that they are necessarily poorer than male-headed households in terms of income poverty. This is largely due to the fact that women are more likely to use scarce resources more productively than males or to diversify their income-generating activities more than males. The participation of women in the project’s activities is self-evident from the data gathered in the field. For example, of the 483 FGD attendees, 64% were women, considerably more than men, with a significant majority in both nutrition and literacy groups. Further comments from women in the groups, especially literacy and nutrition, indicated that they felt more confident at home with an increase in self-esteem and more respect from their male family members, who were beginning to appreciate their wive’s increasing contributions to the family’s wellbeing and improved wealth. Although the project promoted the need for women in groups, the evidence suggests that women would have in any case predominated many of the groups due to cultural attitudes where men would traditionally not be involved, for example, in nutrition or health matters. Similarly, kitchen gardens are an area that men would traditionally participate in, leaving it to their female family members to run. Improved income potential for women from either kitchen gardens, expanding exisiting plots, or diversifying into other off-farm activities has already been discussed elsewhere in the report. During the evaluation mission, the Institute of Policy Research and Analysis Rwanda (IPAR) held its annual conference, a significant part of which focused on the fact that 74% of women do not participate in formal banking compared with 64% of men. Speakers touched on the urgent need to sensitize, educate and link women to financial institutions so that they can save and borrow money to fund income generating activities (IGAs). Other issues discussed were the discouragement of consumption loans in favor of productive activities and low access to finance by women which is continuing to hinder the eradication of extreme poverty and hunger. What was required were new initiatives to boost women entrepreneurship, job creation and inclusive growth opportunities. It is against this background that the ET reviewed IILP’s contribution to women’s empowerment. IILP integrated two main outputs: 1) expanded responsible access to financial services by women (demand and supply side); and 2) improved entrepreneurship/microenterprise management skills in both on and off-farm sectors through three RPOs (AEE, CARITAS, and DUHAMIC ADRI). To proactively mitigate the effect of female illiteracy on their adoption of the project’s inputs, the project partnered with ADEPER to develop and roll-out a functional literacy and numeracy program tailored to beneficiaries’ needs evolving around the project’s interventions in agriculture, nutrition, and saving and loans schemes. However, during its investigations the ET found that project documentation did not specifically report on women empowerment at the household or community level, or disaggregate indicators by gender. It was therefore deemed necessary to obtain feedback from the FGDs and the HHS to shed light on the matter. Discussiosn with key informants revealed that the project has supported two financial institutions, Kenya Commercial Bank (KCB) and Duterimbere IMF, to develop two new women tailored financial products. However, feedback from female FGD participants indicated that they had not been specifically targeted to facilitate access to these products. It is possible that women in other non-interviewed groups accessed these products, so it is therefore not possible to make a general statement on the subject without further information. 33 Usually referred to as Household Economy Approach (HEA) – Authors: S.Browne and N. Lecumberri – Publication date unknown. 34 Author: Arthur Byabagambi – October 2013 29 Figure 12 summarizes FGD feedback which was analyzed and tabulated for reference purposes. Relevant information related to women was extracted and presented below based on data gathered: Figure 12: Disaggregated Data on Male / Female Activities Gender No. of Groups Group Type No. of Members in all Groups Kitchen Garden Owners Access to Finance Loan ISLG Loan MFI Male 1,370 1,287 1,249 314 Female 3,299 2,983 2,768 281 Total 242 4,669 4,270 4,017 605 Source: Data extracted from FGD findings From this information, it is evident that women only accounted for 70% of all group members and 70% of kitchen garden owners, but were more than twice as likely to borrow money from within their own self-help groups. It should be noted that men did have more access to MFI loans than women. Women reported that this was because they felt less confident about paying back loans than men. This is also in line with IILP’s assessment of female versus male participants in the project’s support programs in basic literacy, nutrition, financial literacy, and savings and lending, where females represented 68% of the total attendees. When asked about the extent to which IILP had contributed to their empowerment, 92.8% of women recognised that being a member of an ISLG had increased their level of savings and thus the confidence to use the money for purposes that would improve their livelihoods. Furthermore, comments from major stakeholders interviewed universally supported the belief that there was no discernible gender imbalance in Rwanda with respect to women in terms of their involvement in rural farming activities (notwithstanding the inability to borrow from MFIs). Female FGD participants indicated their preference for further support in accesssing loans from MFIs and SACCOs and starting a business, and more information and training in agriculture including pest and disease management, in addition to further support for BCVs in all of these areas. 4.3.3.2 CONCLUSIONS Despite having no targets to meet, IILP has clearly succeeded in reaching and supporting a significant percentage of women in its target areas, with no apparent disadvantage impacting their daily lives relative to their male counterparts which can therefore justifiably be viewed as a positive development for the project. Women have benefited from instruction in both cases which, in turn, has led to an increase in feeling that they now have more influence within their families in terms of how, where, and when limited funds should be spent. At the same time, an important effect of this development is that household males, who were previously the main income earners, now have more respect for their female relatives as equals due to the latter’s financial contributions to the household. However, the support to KCB and Duterimbere IMF in product development has not empowered targeted women to access bank loans to be used in creating new or expanding current IGAs and thus increase income – at least not among those interviewed by the ET. The project does not appear to offer enough support to women with a view to developing their existing farms or kitchen gardens, facilitate loans for business development or increase the knowledge and skills of BCVs to offer relevant or practical advice on all related matters. 4.3.3.3 RECOMMENDATIONS In reality, and taking into account female FGD participant responses, there is little value in the project allocating a great deal more time to literacy or nutrition initiatives (for the remainder of the project). What was apparent from discussions with the women (and supported by the men) was more focus on teaching them how to move beyond improving subsistence farming, instruction on dietary habits and general issues related to individual and household health training. Significant emphasis was placed on making kitchen gardens more productive as well as less vulnerable to pests and diseases. There were also many who felt that extension services should be upgraded to offer (via BCVs and local agronomists) much needed practical assistance with seed selection, appropriate tools needed, post-harvest storage and distribution, and marketing (including sales) (see 4.2.3 for details regarding extension services obtained from the HHS). 30 Facilitating loans from MFIs and SACCOs also remained a high priority in order to improve growth potential for those women who wanted to progress to becoming entrepreneurs or help those already in business to move forward. Active support from the project in these areas should be considered sooner rather than later in order to improve the likelihood of sustainability in project results after the end of the mandate in July 2016. In addition, for the financial service providers to really impact women, the project should consider scaling up of the two financial products developed for and piloted by the two financial institutions to a wider spectrum of SACCOs with whom the majority of ISLGs are banking. This could be achieved by employing specialised BDSs in product development to identify successes and failures of current financial with a view to adapting them for the most promising SACCOs and women. 4.4 GROUP D: SUSTAINABILITY 4.4.1 Question 1 To what extent are District Authorities and Community Volunteers collaborating in the implementation of project activities, as a strategy for ensuring sustainability of the project results? What can we learn from this process to inform efforts to improve further collaboration as a sustainability strategy for continued program benefits in targeted communities? SUMMARY OF EVALUATION DESIGN AND METHODS Evaluation Question Type of Analysis Conducted Data Sources & Methods Used Data Collection Method & Sample Size Limitations 1. To what extent are District Authorities and Community Volunteers collaborating in the implementation of project activities, as a strategy for ensuring sustainability of the project results? What can we learn from this process to inform efforts to improve further collaboration as a sustainability strategy for continued program benefits in targeted communities?  Description – based on content analysis of project documentation  Description – based on content analysis of expert opinions  KIIs with IILP key personnel  Review & analysis of IILP reports & other relevant publications  KIIs with representatives from district authorities and BCVs  All available documents – no exclusions  ET to identify potential interviewees (assistance from IILP  None encountered  Reliance on anecdotal evidence  Clarification of definition of ‘collaborating’ – formal / informal arrangement / agreement 4.4.1.1 FINDINGS District Authorities and BCVs To answer this question the ET needed to define the term ‘sustainability’ in order to develop a strategy for gathering information that would contribute towards its understanding of what would happen post-project to IILP’s beneficiaries. After some debate it was agreed that investigations should focus on measures the project has already or plans to put in place that are specifically designed to ensure sustainability. From this the team would be able to assess the level of probability that such measures would in fact lead to continuity of support services and facilities with beneficiaries effectively as clients. Here it is worth noting what the original CHF/USAID Cooperative 31 Agreement had to say on the subject – ‘sustainability will be achieved through leveraging proven local partnerships through utilization of local organizations and structures’. It adds further that, ‘ownership and sustainability...will occur...through localized cost share’. Here, cost share will be provided through linkages facilitated by Ejo Heza including RPOs’ existing capacities and systems, for example, ADEPR literacy centers, trainers, and time donated by community members to support forming new ISLGs. In the evaluation SOW, the following statement adds to the debate. ‘In order for Ejo Heza activities to be sustainable, Be the Change Volunteers are used to facilitate grass root program activities through training and mentoring – use of BCs is a sustainability strategy that assures continuity even after USAID Ejo Heza ends’. This set the tone for the ET’s investigations. When preparing and organizing the FGDs in the field, considerable care was taken to ensure that representatives from all IILP-supported groups were included, as feedback from them would provide invaluable information on which of the above-mentioned support mechanisms were in fact in place to ensure sustainability. Specifically, these were: BCVs, ISLGs, Cooperatives, and Literacy and Nutrition groups. The ET also planned to meet with local authorities and agronomists whose views would also be highly relevant. Feedback from FGDs indicated that, without continued support post￾IILP, it was unlikely that they would be able to continue due to limited resources. These views were particularly prevalent in the Western Province (Rutsiro and Karongi). Sectors visited by the ET were in remote and almost inaccessible locations where support would clearly still be needed well into the future given the implicit lack of opportunities to develop and improve livelihoods. What was interesting here was that BCVs themselves believed they were ill￾equipped to continue offering support not least as volunteers without pay (some small amounts were provided for transport but this would cease after the project ends). Although most BCVs felt a moral duty to continue their work post-project, when pressed, they agreed that it would be unlikely in many cases and that they would prefer to be re-trained in other potential income-earning activities, such as vocational work (joiners, builders, tailors). Local district officials and agronomists advised that they were essentially struggling to survive in their respective sectors and gave no impression that matters would improve post-IILP. Resources remain scarce and ‘goodwill’ often stretched in the absence of funds and other means to continue supporting the poorest of farmers. Considerable discussion centered around the problem of pests and diseases and the effect this had on farming communities with no apparent help or support from anyone. Comments were made that geographic remoteness and isolation resulted in a lack of official interest. There was little feedback on the input of RPOs as the BCVs tended to be the frontline points of contact. It was therefore difficult to gain a clear understanding of the RPOs existing structures other than their use of field officers reporting back to head offices in Kigali. Linkages with MFIs and SACCOs were also investigated and, as reported earlier, they have not developed to any great extent, further restricting the possibility of sustainability if individual farmers still have difficulty in accessing loans. From the ET’s observations in the field, it is clear that both district officials and the community volunteers (BCVs) have collaborated with IILP since their respective involvements began. However, given the prevailing view that they are required to work with limited resources and often in remote areas, this will present problems in terms of sustainability, if some of the more pressing issues are not addressed, including recompense for the BCVs to cover expenses. The district officials play an important role as a focal point for the communities that they serve, offering advice and support to local household farmers, for example, on loan applications to MFIs or advice on business development including horticulture or animal husbandry. For those involved in health centers, dealing with nutrition, feedback from those interviewed suggests that collaboration with IILP remains sound with a focus on conducting, for example, training-of-trainer (ToT) sessions for health center staff. These ToT trainings are conducted in collaboration with STC who is IILP’s international partner responsible for delivering the nutrition component. Once trained, participants go into the field and work with households putting theory into practice, for example, in connection with kitchen gardens. Collaboration as a Sustainable Strategy Further to the above findings, the ET found that the existing collaboration between the project, local officials and BCVs, is a strategy that has enabled significant progress to be made in reaching out to 32 local farming communities, especially in matters related to health and nutrition. From the feedback obtained from interviews, FGDs, and the HHS, there is little doubt that the project’s interventions, delivered through its intermediaries (RPOs and STC), have contributed towards achieving the project’s aims in line with meeting pre-determined indicators and results. Conversely, without this support, it is unlikely that IILP’s targeted beneficiaries would have benefited to the same extent. Other non-project related initiatives by WFP and CRS among others also have an influence via their respective programs but ET found that USAID maintains the higher profile in those areas where it operates. 4.4.1.2 CONCLUSIONS District Authorities and BCVs IILP has succeeded in establishing links with district authorities and the BCVs in its target area of 8 districts and 24 sectors. All parties concerned have worked towards supporting the project’s beneficiaries in line with overall objectives. Whether in the absence of the project post-July 2016 these local parties continue to function as effectively without input, for example, from the RPOs (all four responded that their continued support could not be guaranteed due to the potential costs involved) remains to be seen. District officials have to work within civil service parameters while BCVs have commented that for many of them, continuing to support local communities without some sort of reasonable remuneration is unlikely. Collaboration as a Sustainable Strategy Collaboration has without doubt been a necessary and important aspect of IILP’s implementation strategy. Involving local officials is always going to be an essential part of any support activities for local communities, as they are often the first point of contact for beneficiaries in terms of being advised of what projects such as IILP intend and what they have to offer in terms of benefits. 4.4.1.3 RECOMMENDATIONS District Authorities and BCVs As a matter of urgency, the project should conduct a review to assess the competence of BCVs and to identify gaps in their skills and knowledge. It would also be constructive to consider options to help BCVs convert into business organizations, empowering them to offer services and potentially borrow funds to support their activities. It is also arguable that as part of its exit strategy IILP considers whether or not additional funds could be allocated to BCVs in the remaining time period, at least to cover transport costs (which many of them say are not fuly covered at the moment). Volunteers give of their time willingly but they also have to earn a living as the BCVs are part of the communities they serve and in fact have been promoted as BCVs by their peers. To lose their support post-project would not serve the project’s aims regarding sustainability. Collaboration as a Sustainable Strategy There is no specific recommendation other than to continue supporting the district authorities with whom the project collaborates and also the BCVs in line with above mentioned recommendations. 5.0 LESSONS LEARNED Below are a number of suggestions for USAID to consider for future reference in particular with regard to project design and development of indicators and result areas. They are not intended for consideration for the remainder of IILP’s implementation period: 1. At the project design stage, careful consideration should be given when introducing indicator/result areas that are based on partial productivity measures such as yield (e.g. output or gross margin per unit of land) as these can be difficult to calculate in the absence of verifiable data from farmers (especially in poor, rural areas). It is also difficult to attribute increases in productivity to a particular input due to the probable existence of other influencing factors. In agriculture, increases in productivity and output are generally associated with the introduction of mechanization, fertilizers, pesticides, irrigation or higher yield crops. Planned interventions should therefore focus on measures that can be taken by the project to contribute towards supporting productivity improvement, for example, in extension services. This could involve supporting the creation of local agriculture extension centers which could 33 advise farmers on plowing, planting, cultivating and harvesting as well as providing extension supplies or creation of post-harvesting storage facilities aimed at supporting marketing and distribution as well helping reduce wastage. In short, this approach focuses on supporting farmers’ ability to improve output on an ongoing basis and thus contribute towards sustainability. Indicators and results could then be developed in accordance with this type of support. 2. Similarly, it is suggested that project RFAs or RFTOPs avoid use of indicators where there is a requirement to calculate the project’s contribution towards, for example, a change in behavior, attitude or performance in relation to an expected outcome. 3. The greater the number of project partners and implementing intermediaries, the greater the risk of diluting the ‘chain of command’ in terms of gathering and reporting data in ‘real time’ in support of informed decision-making at project management level. It is suggested that future project designs consider this issue during preparation. 4. Following from the above and in order to improve project reporting (quarterly, annually and work plans), future project RFAs or RFTOPs should include the need for a management information system (MIS) for M&E purposes – this should be mandatory and specifically highlighted in all technical proposal submissions. It will ensure that real time reporting and decision-making are accessible by all relevant parties with an input to project reporting based on password-protected access. 5. When including ‘sustainability’ as a factor in project design, consider the need for the inclusion of a specific definition of what is intended, as it could be misinterpreted or mean different things to different people. 6. At the project design stage, consider including the requirement of a needs assessment to be conducted prior to implementation in order to identify real needs or gaps to address (which could potentially differ from those originally identified in RFAs or RFTOPs). Also review who else might be planning to do the same to avoid duplication (among international donors). 7. In RFAs or RFTOPs for evaluations, when raising issues related to cost-benefit (analysis or comparisons) clarify if what is intended is a ‘value for money’ exercise as opposed to comparing how funds could have been more effectively spent elsewhere. Inter-component comparisons would be valid in terms of efficiency and effectiveness analysis. 8. In RFAs or RFTOPs, consider whether there is a concentration of effort on the achievement of quantitative results (outputs) relative to the impact of planned results (outcomes). This distinction is highly relevant in IILP-type projects as the former is easily measurable while the latter is much more complicated to assess, if not impossible, in the absence of a defined methodology as per those to be found in USAID’s PIRS. ANNEXES ANNEX A: STATEMENT OF WORK Performance Evaluation of the USAID/Rwanda Integrated Improved Livelihoods Program RFTOP No.: SOL-696-15-000001 SECTION C – DESCRIPTION / SPECIFICATIONS/STATEMENT OF WORK USAID/Rwanda - Integrated Improved Livelihoods Program Performance Evaluation C.1 Purpose The overall objective of the USAID Ejo Heza Performance Evaluation is to assess the progress of the USAID Ejo Heza program over its first three years of implementation towards achieving Intermediate Results for each Program Element and cross cutting objectives, to test the effect of the used integrated approach on the beneficiaries’ behavior towards program key elements and to provide recommendations that guide the program team to make necessary course corrections or adjustments in order to fully achieve USAID Ejo Heza’s goal of improving the livelihoods of Rwanda’s very poor by the end of the program. C.2 Background The Integrated Improved Livelihoods Program locally known as USAID Ejo Heza (Kinyarwanda for “Brighter Future”), is a five-year project (July 2011 to June 2016) USAID funded livelihoods program under the Feed the Future initiative. USAID Ejo Heza aims to improve the livelihoods and food consumption of 75,000 of Rwanda’s very poor, particularly women, by building the capacity of low income households to themselves position effectively to access the appropriate financial services necessary to grow their enterprises. Simultaneously, the program works with financial service providers to recognize and approach rural market opportunities, while increasing their capacity to foster linkages between opportunities and products. This approach promotes effective and sustainable partnerships for economic growth and is market driven, cost effective and sustainable. Finally, USAID Ejo Heza improves rural people’s health and nutrition by increasing their understanding on how to access and fully utilize locally available, familiar foods through the integration of nutrition messaging throughout its program activities. To date, Ejo Heza has successfully reached and surpassed the majority of program targets set forth in Cooperative Agreement AID-696-A-11-00008. Through the locally driven work with literacy centers, nutrition groups and agricultural cooperatives, among others, the program has built critical momentum for continued program success, having reached over 38,617 rural Rwandan households. Ejo Heza is led by Global Communities, working with four local Rwandan Partner Organizations (RPOs): CARITAS, DUHAMIC-ADRI (Duharanire Amajyambere y’Icyaro), African Evangelical Enterprise (AEE) and Association Des Eglises des Pentecote au Rwanda (ADEPR), and one International Organization, Save the Children, which leads nutrition activities. Global Communities and Save the Children provide technical leadership on the program, transferring knowledge, skills, and tools to assure consistent, quality services to program beneficiaries across the area of operation. The local Rwanda Partner Organizations (RPOs) implement the program on the ground and work directly with local authorities and community volunteers to deliver services through the program entry points: Literacy classes, Integrated Savings and Lending Groups, Cooperatives and Nutrition groups. The RPOs mobilize and organize the community members into these various groups for entry and participation in program activities. USAID Ejo Heza covers eight districts of the Southern and Western Provinces of Rwanda: Nyaruguru, Nyanza, Huye, Nyamagabe, Gisagara in the Southern Province and Rutsiro, Karongi and Ngororero in the Western Province as per the map shown below in figure 1. Fig.1. IILP USAID-Ejo Heza targeted district map I.1. Project Approach and Implementation Fig.2: USAID Ejo Heza integrated approach In line with the Feed the Future’s global mandate to sustainably reduce global hunger and the Government of Rwanda Economic Development and Poverty Reduction Strategy (EDPRS II) and Strategic Plan for the Transformation of Agriculture in Rwanda (PSTA III), USAID Ejo Heza is being implemented in an integrated manner, linking the four program elements illustrated in the chart presented in the figure 2 as shown above. USAID Ejo Heza uses an approach that targets benefits to individuals and families based on their specific needs rather than having pre-determined “packages” of services. This allows resources to be used more efficiently to create the desired impact on income/welfare.” This assumes that not all individuals will be reached through all interventions. Participation is on a “self-selecting” basis. Opportunities for participation in the full range of activities are widely disseminated in the target communities in cooperation with the local authorities. Individuals, producer groups and cooperatives contact the RPO or local authorities to sign-up to participate in the activities that best suit their interests. One key element of the program strategy outlined in the initial program document was the collection and utilization of local data. As a baseline, USAID Ejo Heza contracted the Food Economy Group (FEG) to conduct an assessment utilizing the Household Economy Approach (HEA). In addition to this detailed assessment into the economic welfare of targeted communities, USAID Ejo Heza also conducted a Knowledge, Attitudes and Practices (KAP) survey, leading to the creation of the Behavior Change Strategy, and a Value Chain Assessment and Financial Sector Mapping Exercise. Based on the above approach, and the activities described below, USAID Ejo Heza contributes to: • Increased rural household income at a level that allows those households to successfully weather shocks. • Improved agricultural practices and farm management leading to sustainable increases in food production on rural farms. • Increased rural farmer access to and utilization of financial services. • Improved outreach by financial service providers delivering appropriate products • Improved literacy, numeracy and financial literacy that contributes to improved farm management and access to loans. • Improved access to agriculture extension and business development services. • Increased consumption of balanced and diversified diet through purchase and/or production • Statistically significant increase in the number of women providing agriculture extension services. The actual program indicators are outlined below for each of four program elements: The program goal is to improve livelihoods of Rwanda’s very poor and key Feed the Future and goal level indicators include: • Prevalence of poverty: % of people living on less than $1.25 per day • % change in income of targeted population • Number of rural households benefiting directly from USG interventions C.3 Project Activities Increased agricultural productivity and demand for financial services USAID Ejo Heza addresses the demand-side issues that limit the use of financial services to improve livelihoods and food consumption by promoting savings and use of credit, financial literacy, basic literacy and numeracy, while expanding access to business development and agricultural extension services. Activities around increasing demand for financial services are conducted through Integrated Savings and Lending Groups (ISLG’s). These are savings groups composed of about 25 members who voluntarily come together to save and borrow among themselves. Since the beginning of the program, USAID Ejo Heza successfully formed 1,525 integrated savings and loan groups (ISLGs) out of 2,000 targeted by the end of the program, furthermore, program monitoring results show that 65% of these groups have opened a bank account. Adult learners are taught how to read and write through 400 literacy centers organized across the 8 districts. ADEPR is the lead implementing organization in literacy and now has 800 teachers committed to the program across the literacy centers. Each center averages 50 learners per intake with the program facilitating two intakes per year. Basic literacy incorporates learning around all the program components; agriculture, nutrition, financial education and financial services. Practical lessons such as cell phone literacy are also incorporated. Activities in agriculture are conducted through cooperatives focused on the three main value chains of maize, beans and dairy. Currently the program has 44 cooperatives participating in these value chains.Farmer Field Schools (FFS) that average about 30 members each are found within each cooperative. FFS is a method used to provide practical demonstration to the community on improved agriculture and extension techniques. Key Feed the Future and program level Indicators under this component include: • Number of hectares under improved technologies or management practices as a result of USG assistance • Gross margin per unit of land, kilogram, or animal of selected product • Number of farmers and others who have applied new technologies or management practices as a result of USG assistance • Number of new technologies or management practices in one of the following phases of development: • Number of farmers accessing agricultural extension services • Number of MSMEs receiving business development services from USG assisted sources • Number of private enterprises, producers organizations, water users associations, trade and business associations, and community-based organizations (CBOs) receiving USG assistance • Number of people trained in one or more forms of literacy, including basic literacy, basic numeracy, financial literacy Increase supply of financial services USAID Ejo Heza seeks out, engages and provides targeted support to those financial service providers, including Microfinance Institutions (MFIs), Saving and Credit Cooperative (SACCOs) and potentially commercial banks, with the interest and capacity to expand their outreach and develop products that meet the needs of the rural market. A “Mapping the Financial Sector Landscape“ study was done, which has assisted in the quick scaling up of program activities. USAID Ejo Heza supports the increased supply of rural finance with the following types of activities. • Supporting linkages between ISLGS and MFIs: USAID Ejo Heza supports linkages between ISLGs and cooperatives and formal financial institutions, including MFIs and SACCOS. • Facilitating financial sector innovation: USAID Ejo Heza provides technical assistance to support the development and expansion of products that address the needs of rural sector households. • Supporting Capacity Building: USAID Ejo Heza supports the provision of targeted assistance to strengthen the institutional capacity of financial service providers so that they are better able to sustainably deliver services. • Supporting the Expansion of Mobile Money: USAID Ejo Heza works with telecommunication companies and financial institutions to expand this market. Key Feed the Future and program level Indicators under this component include: • Value of agricultural and rural loans • % of created and/or supported ISLGs with linkages to financial institutions • Number of New financial products developed • Number of Loans made by MFIs • Number of beneficiaries accessing financial services • Number of MSMEs receiving USG assistance to access bank loans Behavior Change and Marketing Behavior change and social marketing approaches sensitize the population on the benefits of financial services, improving production practices, working cooperatively and improving access to health and nutrition services. This is achieved through model farmers, Integrated Savings and Lending Groups (ISLG) and cooperative members as Be the Change Volunteers (BCV), already known and respected within their communities. In order for USAID Ejo Heza activities to be sustainable, BCVs are used to facilitate grass root program activities. BCVs assist in training individual beneficiaries and mentoring them around the four program areas of nutrition, agriculture, financial services and literacy. BCVs are typically community members who are volunteers and self-selected to take a leading role in facilitating change within the communities. All BCVs are selected from these community groupings and are then facilitated to support their groups in training and mentorship. Use of BCV’s is a sustainability strategy that assures continuity even after USAID Ejo Heza program ends since skills and knowledge are already transferred to capable community members. To date over 2,000 BCVs have been trained. Health and Nutrition USAID Ejo Heza implements complementary short-term and long-term actions to ensure that the most vulnerable populations attain skills and knowledge that lead to greater food security and greater understanding of diet, nutrition and hygiene—especially within the critical first 1,000 days of newborn children and their mothers. Activities in nutrition are organized through nutrition groups. These groups provide a channel through which community members can learn about better methods of food utilization and cooking for healthier diets.They use the knowledge within their own households and share it with their neighbors through establishment of kitchen gardens. Each nutrition group has on average 30 members. All nutrition activities, coordinated by Save the Children, are linked closely with both education and extension services delivered by Rwandan partner organizations (RPOs) as well as Government of Rwanda priorities. Over 38,926 Rwandans accessed agricultural extension services, and 8,983 kitchen gardens were established by the end of the second year of the program. Key Feed the Future and program level Indicators under this component include: • % of beneficiaries consuming a diet with increased diversification • % of beneficiaries with increased knowledge about balanced & diversified diet • Number of people trained in child health and nutrition through USG supported programs • Number of kitchen gardens established C.4 Services and Tasks Required The evaluation team shall perform the following tasks: 1. Desk review of key program documents; 2. Develop an evaluation plan comprising evaluation methodology with sampling method; data collection tools and data analysis plan; 3. Pre- test data collection tools; 4. Collect data, ensuring standard and rigorous data quality control; 5. Develop data entry matrix and enter quantitative and qualitative data; 6. Presentation of key findings to USAID-Rwanda; 7. Develop draft evaluation report; and 8. Revise and submit final evaluation report; C.5 Evaluation Questions, Design and Methodology C.5 (a) Evaluation Questions The Performance Evaluation should also provide recommendations based on the following evaluation questions and these are formulated based on specific objectives outlined above. The evaluation will assess project implementation since July 2011 to July 2014. The evaluation will make conclusions based on findings, identify opportunities, challenges and formulate recommendations for improvements, across all four program elements and cross cutting issues especially gender parity and women empowerment. The key evaluation questions are: 1. To what extent are activities implemented by IILP contributing to increased agricultural productivity; increased demand for, and supply of financial services; and improved food security, nutrition and health, in targeted communities? In answering this questions, the offeror should address the following: a. To what extent planned activities for IILP enable achieving program expected results, purpose and goal? b. How well IILP resources and activities are converted into results, in terms of quality, quantity and time based on cost-benefit comparisons? c. How and to what extent has IILP’s work in promoting and strengthening incomegenerating activities contributed to household food security? d. How can USAID Ejo Heza better track the outcomes of behavior change activities on beneficiaries? e. How and to what extent IILP-support financial services increased entrepreneurship (new, profitable SMEs) and incomes among the beneficiaries? Are linkages between beneficiaries and Microfinances really taking place? f. How and to what extent IILP nutrition education activities contributed to the improved household nutrition status, especially for women and children? Are the used kitchen garden and nutrition messaging approaches effective to reduce malnutrition or need for improvement and/or other alternatives? g. How and to what extent IILP’s agricultural extension services contributed to improved household crop production and incomes? 2. To what extent are the activities implemented by IILP contributing to increased participation, improved income, and overall “empowerment” of women in the targeted communities? In answering this question, the following should also be addressed: a. What are women perceptions or feelings on their current income situation? 3. To what extent are District authorities and Community volunteers collaborating in the implementation of project activities, as a strategy for ensuring sustainability of the project results? What can we learn from this process to inform efforts to further improve collaboration as a sustainability strategy for continued program benefits in targeted communities? C.5 (b) Evaluation Design and Methodology The team will carry out the evaluation using designs such as before and after, time series, panel designs and others that could answer the proposed questions. The evaluation will include qualitative and quantitative data collection. The proposal should give a description of specific data collection methods such as individual or focus group and structured interviews, checklists and other tools. Each evaluation design can also be negotiated with the evaluation team upon further reflection and discussion.. Key stakeholders to be interviewed will include government representatives, field staff and managerial staff of Global Communities and other implementing partners, community volunteers and beneficiaries. Any sampling for this evaluation should be designed to be economical while representative and should be inclusive of all districts, RPOs and beneficiaries representative of the four program components (sample of key activities for each component). The contractor shall propose a methodology for review and approval by USAID-Rwanda that should include but not be limited to: 1. Review of key program documents; these will be made available and include: Cooperative Agreement with Global Communities, USAID Ejo Heza Annual Work Plans, PMP, Sub Agreements, Progress Reports, USAID Rwanda Feed The Future Strategy, PSTA III (Strategic Plan for the Transformation of Agriculture in Rwanda) from Ministry of Agriculture, DHS 2010 (The Rwanda Demographic and Health Survey) from Ministry of Health, CFSVA 2012 from WFP, EDPRS II (Economic Development And Poverty Reduction Strategy), EICV III , Surveys conducted by USAID Ejo Heza among others. 2. Key informant interviews with Global Communities staff, Partner Organization staff, local government officials and other program stakeholders 3. Focus group discussions with community volunteers and program beneficiaries bearing in mind that 2 and 3 groups close to program interventions might be biased in responses as they tend to show expected/planned results as achieved while there are still progress to be made. 4. Review of results of any surveys made, interviews and other tools used for on-going project monitoring and evaluation 5. Review of any M&E data and reports being tracked through PRPMS, FTFMS or AIDTracker + reporting systems and Global Communities’ automated Program Tracker System as well. 6. Data Analysis: emphasis to be put on analysis of trends towards achieving program targets and analysis of outcomes for males and females and factors affecting any differential outcomes. 7. In case limitations of proposed evaluation design and methodology are found, including data quality issues, lack of baseline data on some indicators, the offeror is requested to propose ways of mitigating/addressing these limitations. 8. Based on information available in the Statement of Work, the offeror is encouraged to elaborate an evaluation design matrix showing the way the evaluation questions will be addressed, related indicator/assessment criteria, collection methods, criteria and method. C.6 General Program Implementation Guidelines C.6 (a) Assessment Team Composition The Evaluation Team shall be comprised of a four-person team including one international consultant, who will be the Evaluation Team Lead; one local financial services expert; one local gender expert, and one data analyst/logistics coordinator. The following composition and expertise is required to conduct this analysis: 1. Team Leader (Rural Livelihoods Expert), International The Team Leader’s main role will be to coordinate all evaluation tasks, guide other team members and make sure that all deliverables are qualitatively and timely realized. She/he should also fulfill the following conditions: • Master’s degree from a recognized university in one of the following fields: agriculture, rural development, nutrition and health, economic development and program evaluation or other related fields. • Expertise in rural livelihoods including small-holder agriculture/household nutrition, behavior change and social marketing and/or micro-finance particularly desirable. • 5-10 years of experience in leading evaluations, particularly related to rural livelihoods integrated programs addressing rural finance, agriculture, nutrition and entrepreneurship issues • Experience in conducting operational research studies, and both quantitative and qualitative evaluation methodologies • Experience in conducting evaluations in Africa (required); having worked in Rwanda would be an advantage • Excellent computer, data analysis and reporting skills. • Proven ability to lead teams and work well/interact with USAID; prior experience as an evaluation team leader for a USAID evaluation is a plus. The Team Leader will be assisted by three local consultants with the following expertise: 2. Local Financial Services Expert (1 person) • Bachelor from a recognized university in one of the following fields: agribusiness, rural development, economics and program evaluation or other related fields. Master’s degree is desirable. • Expertise in small-holder agriculture/household economics and/or micro-finance particularly desirable. • Experience in conducting operational research studies, and both quantitative and qualitative evaluation methodologies • Demonstrated knowledge in designing and conducting evaluations using different data collection tools such as rapid rural appraisal, using focus group discussions, key informant interviews of beneficiaries, surveys, project staff and other stakeholders • Past experience in participating in independent external evaluations of development projects highly desirable. • Excellent computer, data analysis and reporting skills. 3. Local Nutrition & Gender Expert (1 person) • Bachelor from a recognized university in one of the following fields: agriculture, rural development, nutrition and health, program evaluation or other related fields. Master’s degree is desirable. • Expertise in small-holder agriculture/household nutrition and gender integration particularly desirable. • Experience in conducting operational research studies, and both quantitative and qualitative evaluation methodologies • Demonstrated knowledge in designing and conducting evaluations using different data collection tools such as rapid rural appraisal, using focus group discussions, key informant interviews of beneficiaries, surveys, project staff and other stakeholders • Past experience in participating in independent external evaluations of development projects highly desirable. • Excellent computer, data analysis and reporting skills. 4. Data Analyst/Logistics Coordinator (1 person) • Bachelor from a recognized university in one of the following fields: agriculture, rural development, nutrition and health, economic development and statistics. Master’s degree is desirable. • Expertise in small-holder agriculture/household nutrition and/or micro-finance particularly desirable. • Experience in conducting operational research studies, and both quantitative and qualitative evaluation methodologies. • Demonstrated knowledge in use of different computer-based data analysis tools/programs. • Experience in evaluation management and logistics coordination. • Past experience in participating in independent external evaluations of development projects highly desirable. C.6 (b) Schedule and Logistics The assignment is expected to be approximately seven weeks, with approximately 5 weeks in Rwanda. The team will coordinate logistical arrangements with the Task Order COR. The contractor shall be responsible for the administrative support and logistics required to fulfill this task. These shall include all travel arrangements, appointment scheduling, secretarial services, report preparations services, printing, and duplicating. USAID shall assist the contractor to get program documents and contacts necessary to fulfill the task. The contractor will propose the evaluation timeline for the evaluation task for USAID approval. The Task Order COR will provide strategic direction and guidance throughout the evaluation process, including the development of the evaluation plan, any data collection tools, and evaluation report outline, approach, and content. C.7 Deliverables The following deliverables are required. All written documentation for submission by the Contractor to USAID must be in English. C.7 (a) Evaluation In-Briefing and Out-briefing Presentations C.7 (a)(i) In-Briefing Presentation The Contractor shall provide an in-brief to the task order COR on the methodologies to be used in the evaluation including any survey questionnaires and instruments to be used within 5 days of arrival in-Rwanda. The contractor shall develop an evaluation plan comprising evaluation methodology with sampling method, data collection tools and data analysis plan C.7 (a)(ii) Out-briefing Presentation The Contractor shall provide an out-briefing presentation with USAID/Rwanda of the key findings and recommendations within 3 days after completion of field work. C.7 (b) Draft Performance Evaluation Report The Contractor shall provide a draft performance evaluation report with details of methodology followed, key findings to evaluation questions, conclusion and recommendations after 10 days after completion of field work. USAID/Rwanda will provide comments to the contractor within 6 working days of receipt of the draft. C.7 (c) Final Performance Evaluation Report The Contractor shall submit a final performance evaluation report no later than 5 days after USAID/Rwanda provides written comments on the draft evaluation report. The report, not exceeding 50 pages (excluding appendices) must meet the USAID evaluation quality standards specified in Attachment J.3 of this RFTOP. The Performance Evaluation report must meet the following criteria: • The evaluation report should represent a thoughtful, well-researched and organized effort to objectively evaluate what worked in the project, what did not and why. • The report must address all evaluation questions included in the statement of work as well as meeting the objective and specific objectives of the evaluation. • The report should address all limitations to the evaluation, including limitations associated with the evaluation methodology • Evaluation findings should be presented as analyzed facts, evidence and data, and should not be based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. • Recommendations need to be supported by a specific set of findings, and should be actionoriented, practical and specific. The format of the Performance Evaluation Report is recommended to include: 1. Title page; 2. Table of Contents; 3. Acronyms; 4. List of tables, or charts (if any); 5. Executive summary (not exceeding 3 pages); 6. Introduction (not exceeding 5 pages) shall include: a. Description of the project including goals and objectives; b. Evaluation rationale, including a list of the main evaluation questions; and c. Description of the evaluation design, methodology and limitations. 7. Findings – Describe the findings related to each of the evaluation questions; 8. Conclusions; 9. Recommendations – Based on the evaluation’s purpose and the findings, describe what remains to be done; what changes can be made in program design or implementation to result in more effective and/or efficient execution and improved results; identify potential new solutions to problems the project has faced; identify adjustments/corrections that need to be made; and recommend actions and/or decisions to be taken by anagement; 10. Lessons Learned - in terms of program implementation, coordination, and beneficiary satisfaction; 11. Annexes: a. Statement of Work; b. List of places visited, people interviewed, including contact information; c. Copies of all survey instruments and questionnaires; d. Electronic copy of data sets; e. List of background documents reviewed; and f. Copies of background documents which were used by Contractor but which were not provided by USAID-Rwanda. Note: Items 6 and 7 combined should not exceed 25 pages. As per the USAID Evaluation policy, the Contractor must submit completed or approved evaluations to the agency’s Development Experience Clearinghouse (DEC) and a coversheet attached indicating the type of evaluation conducted and design. Each completed evaluation must include a 5-10 page summary of the purpose, background of the project, main evaluation questions, methods, findings, conclusions, Recommendations and lessons learned (if applicable) of the evaluation. In general, this performance evaluation should comply with USAID Evaluation policy requirements for evaluations available in the following link: http://www.usaid.gov/sites/default/files/documents/2151/USAIDEvaluationPolicy.pdf C.8 Evaluation Use and Users The audience of this evaluation includes the USAID/Rwanda Mission, Economic Growth Feed the Future Team, but also Health and Education Teams. Other users are Global Communities and Rwandan Partner Organizations (RPOs) that collaborate to implement the Program, as well as the Government of Rwanda, especially Ministry of Agriculture and Animal Resources (MINAGRI) and District authorities. Best practices and lessons learned from evaluation will also be shared with program beneficiaries. END OF SECTION C ANNEX B: LIST OF INTERVIEWEES CONTACT PERSON POSITION ORGANISATION LOCATION DATE Adriana Hayes (Tel: 0252 451588) Program Office Director USAID US Mission, Kigali 20/01/15 Bamwesigye Jackson (Tel: 0783 451588) M & E Specialist USAID US Mission, Kigali 20/01/15 Przemek Praszczalek (Tel: 0252 596400) Deputy Program Officer USAID US Mission, Kigali 20/01/15 Patrice Hakizimana (Tel: 0783 009366) Agriculture Specialist USAID US Mission, Kigali 20/01/15 John Ames + Team (Tel: 0727 700160) COP Global Communities Kigali 20/01/15 Jeanne d’Arc Nyirajyambere (Tel: 0788 387455) Senior Nutrition Specialist Save the Children (Global Communities) Kigali 20/01/15 Andrew Kagabo (Tel: 0788 305015) National Coordinator –‘ One￾cow-per –family’ Program Rwanda Agricultural Board Kigali 21/01/15 Fr. Emmanuel Rubagumya (Tel: 0784391290) Dep. Secretary General CARITAS Kigali 21/01/15 Prosper Sebagenzi (Tel: 0788760662) Program Manager CARITAS Kigali 21/01/15 Clement Kayitare (Tel: 574344) M & E Advisor CARITAS Kigali 21/01/15 Faustin Gatera Tel: 574344) Project Coordinator CARITAS Kigali 21/01/15 John Ames (Tel: 0727 700160) COP Global Communities Kigali 21/01/15 Jean-Pierre Mbagurire (Tel: 0727 700106) M & E Specialist Global Communities Kigali 21/01/15 Emmanuel Niyongira (Tel: 0788 524971) Agriculture Specialist Global Commnities Kigali 21/01/15 Innocent Benineza (Tel: 0788 305329) Executive Secretary Duhamic-Adri Kigali 22/01/15 Alexis Dushimimana (Tel: 0788 471103) Progam Co-ordinator Duhamic-Adri Kigali 22/01/15 Janvier Ugeziwe (Tel: 0788305329) Rural Development Expet Duhamic-Adri Kigali 22/01/15 John Kelenzi (Tel: 0788 303675) Team Leader African Evangelistic Enterprise Kigali 22/01/15 Charles Magezi (Tel: 0788 862563) Program Director African Evangelistic Enterprise Kigali 22/01/15 Venuste Niyitanga (Tel: 0788 303675) Project Coordinator African Evangelistic Enterprise Kigali 22/01/15 Lorien Jyambere (Tel: 0788 404460) Rural Livelihoods, TL Global Communities Kigali 22/01/15 Karuhije Janvier (Tel: 0788 893542) Behaviour Change Comunication Officer Global Communities Kigali 22/01/15 Jeanne d’Arc Nyirajyambere (Tel: 0788 387455) Senior Nutrition Specialist Save the Children (Global Communities) Kigali 22/01/15 Judith Aguga Acon (Tel: 0782 507751) Technical Director Access to Finance Rwanda (AFR) Kigali 23/01/15 Rita Ngarambe (Tel: 0782 889582) Executive Secretary Association of Microfinance Institutions in Rwanda (AMIR) Kigali 23/01/15 André Rurangangabo (Tel: 0788 408353) Adult Literacy Officer Global Communities Kigali 23/01/15 Laurien Kubwirana (Tel: 0727 700093) Coordinator of Projects ADEPR Kigali 23/01/15 Dorothee Munyana (Tel: 0727 600205) Supervsior of Projects (ADEPR/Ejo Heza) ADEPR Kigali 23/01/15 Narcene Kamuhisha (Tel: 0727 600205) Head of Operations ADEPR Kigali 23/01/15 Edouard Dutaganira (Tel: 0727 600205) M & E (ADEPR/Ejo Heza) ADEPR Kigali 23/01/15 Oswald Mutuyeyezu (Tel: 0788 805855) Radio Presenter (Consultant to Ejo Heza) City Radio (Consultant to Ejo Heza) Kigali 23/01/15 Bijou Kuzimbu (Tel: 0788 381435) USAID Higa Ubeho TL Catholic Relief Services Kigali 23/01/15 Cecile Teteli (Tel: 0788 383234) Nutrition TL Catholic Relief Services Kigali 23/01/15 Frank O’Brien (Tel: 0786 112468) COP Land O’Lakes – RDCP II Project Kigali 26/01/15 Jean D`Amour Kubwimana Deputy Country Manager Harvest Plus Kigali 26/01/15 Esperance Mukarugwiza (Tel: 0788 532061) Agri-hub coordinator Agri Profocus Rwanda Kigali 26/01/15 Serene Nterinanziza (Tel: 0782 464557) Capacity Building Specialist IFDC Kigali 26/01/15 Jeannette Kayirangwa (Tel: 252 5877611) National Facilitator ONE UN Joint Programming for Nutrition (WFP) Kigali 26/01/15 Geoffrey Mugisha (Tel: 0733 134478) Country Director Save the Children Kigali 26/01/15 Elisa Radisone (Tel: 9738 303686) Programme Director Save the Children Kigali 26/01/15 Eric Rwigamba (Tel: 0788 308025) Director General Ministry of Finance & Economic Planning Kigali 26/01/15 Augustine Ntaganda (Tel: 0788 357176) Capacity Building & Financial Education Expert Ministry of Finance & Economic Planning Kigali 26/01/15 Eric Musizana (Tel: 0788 560429) Programming Officer SPARK Kigali 26/01/15 Dr. Fidele Ngabo (Tel: 0788 304750) Head of Child Community Health Division Ministry of Health Kigali 26/01/15 George Odhiambo (Tel: 0788 384546) Head of Business Develop’t & Client Services Kenya Commercial Bank (KCB) Kigali 26/01/15 Odile Mukayiranga (Tel: 0732 187000) Micro Banking Manager Kenya Commercial Bank (KCB) Kigali 26/01/15 Alexis Bizimana (Tel: 0782 596747) Agribusiness Relationship Manager Kenya Commercial Bank (KCB) Kigali 26/01/15 Delphin Ngumije (Tel: 0788 305651) Managing Director Duterimbere IMF Kigali 26/01/15 Catherine Dugein (Tel: 577458) Head of Community Health Unit Ministry of Health Kigali 27/01/15 Alexis Mucunbitsi (Tel: 577458) In Charge of Nutrition Ministry of Health Kigali 27/01/15 Francisca Mukakarangwa (Tel: 0788 624136) Head SACCO coordination Unit Rwanda Cooperative Agency Kigali 27/01/15 Alice Uwingabiye (Tel: 0788 796547) Financial Analyst Rwanda Cooperative Agency Kigali 27/01/15 Raphael Rurangwa (Tel: 0788 301498) DG Planning & Policy Ministry of Agriculture & Animal Resources Kigali 27/01/15 Francisca Mukakarangwa Head SACCO Coordination Unit Rwanda Cooperative Agency Kigali 27/01/15 Alice Uwingabiye Financial Analyst Rwanda Cooperative Agency Kigali 27/01/15 Interviews in Ngororero Paulin Twagirayezu Sector Agronomist District Authority Matyezo 30/01/15 Interviews in Nyanza Francis Nkurunziza (Tel: 0788 303292) Vice Mayor in charge of economy District of Nyanza Rwabicuma 30/01/15 Jeanne d’ Arc Byukusenge (Tel: 0788 671401) JADF Officer District of Nyanza Rwabicuma 30/01/15 Viateur Karerangabo (Tel: 0783 634555) Loans manager SACCO Nyanza Rwabicuma 30/01/15 Interviews in Rutsiro Evodu Dushbrirmawa Branch Manager Coopec Inkunga Gihango 04/02/15 Interviews in Huye Concessa Dusabemariya (Tel: 0788 680316) Nutrition and vaccination officer Mbazi Health Center MBAZI 04/02/15 Interviews in Karongi Jean-Babtiste Mbarushimana Community Health Worker Supervisor Mubuga Sector Office Mubuga 07/02/15 Interviews in Gisagara Peruth Niyonsaba (Tel: 0784 653845) Sector Agronomist Distict of Nyanza SAVE 07/02/15 Yves MUGISHA (Tel: 0783348380) Loan Officer District of Gisagara MUKINGO 07/02/15 Interviews in Nyamagabe Dorothe Uwamahirwi Nutrition Officer Cyanika Health Centre Cyanika 11/02/15 Collixte Bizumuremyi Manager Cyanika SACCO Cyanika 11/02/15 Interviews in Nyaruguru Jean Marie Niyonsenga Manager Zamuka Ngoma SACCO Ngoma 11/02/15 ANNEX C: LIST OF DOCUMENTS REVIEWED IILP Documentation Work Plans Name of Document Ref. to PMP Form Submission Date Year 1 – Work Plan (July 2011 – June 2012) WP Appendix Printed Document January 2012 Year 1 – Work Plan (July 2011 – June 2012) Work Plan Printed Document Aug 19, 2011 Work Plan for LOP 2013-2016 & FY2013 No text Excel Sheet Not Known Work Plan (October 2013 – September 2014) No text Excel Sheet Not Known Work Plan (October 2014 – September 2015) No text Excel Sheet Not Known Annual Reports Name of Document Ref. to PMP Form Submission Date USAID Ejo Heza – Phase 1 (First 18 months) No reference Printed Document Not Known Annual Report FY2013 In body of text Printed Document October 2013 Annual Report FY2014 In body of text Printed Document October 2014 Quarterly Reports Name of Document Ref. to PMP Form Submission Date Quarterly Report (October – December 2011) Annex B in report Printed Document January 2012 Quarterly Report (January – March 2012) No reference Printed Document April 2012 Quarterly Report (April – June 2012) No reference Printed Document Not Known Quarterly Report (July – September 2012) In body of report Printed Document October 2012 Quarterly Report (October – December 2012) n/a Within 2012 Annual Report n/a Quarterly Report (January – March 2013) In body of report Printed Document April 2013 Quarterly Report (April – June 2013) In body of report Printed Document Not Known Quarterly Report (July – September 2013) n/a Within 2013 Annual Report n/a Quarterly Report (October – December 2013) In body of report Printed Document January 2014 Quarterly Report (January – March 2014) In body of report Printed Document April 2014 Quarterly Report (April – June 2014) In body of report Printed Document July 2014 Other Reports, Papers, Documents Name of Document Publisher / Author Form Publication Date IILP M & E Plan FY14-1 IILP Printed Document n/a Performance Indicator Reference Sheets (PIRS) n/a Printed Document n/a Rwanda Baseline Feed the Future Global Report Printed Document n/a FTF Feedback – Population-Based Survey Rwanda Detra Robinson Printed Document September 2013 Rwanda – Feed the Future FY2011-2015 Multi-Year Strategy US Government Printed Document February 2011 Social & Behaviour Change Communication Strategy IILP Printed Document September 2012 Voluntary, Education and Target Savings Product Development Duterimbere IMF Printed Document April 2013 Mapping the Financial Services Landscape in Rwanda AMIR / REMC Printed Document April 2012 Poverty Assessment Tools Survey Research Moguls Ltd Printed Document October 2013 Ejo Heza: Livelihood Baselines: Southern & Western Provinces IILP Printed Document Not Known Knowledge, Attitudes and Practices of USAID-EJO HEZA beneficiaries Emmanuel Rigira Printed Document August 2012 USAID/Ejo Heza Value Chain Development Strategy Noel Ujeneza Printed Document August 2012 USAID/Ejo Heza – Yr 13 Narrative Report IILP Printed Document December 2013 IILP 2014 Indicators Report IILP Printed Document Not Known Yr13 Indicator Report IILP Printed Document December 2013 M.O.U between Global Communities and the Rwanda Agriculture Board (RAB) n/a Printed Document July 2013 Strategic Plan 2014- 2016 Access to Finance Rwanda AFR) Printed Document n/a Benchmarking Report 2008-11 Association of Microfinance Institutions in Rwanda (AMIR) Printed Document n/a Rwanda Dairy Competitiveness Program II – USAID – Quarterly Report – July-Sept 2014 Land O-Lakes Printed Document October 2014 Rwanda Vision 2020 Government of Rwanda Accessed online n/a Comprehensive Food Security and Vulnerability Analysis and Nutrition Survey World Food Program (WFP) / MINAGRI Accessed online 2012 Rwanda Nutrition Stakeholder Mapping Ministry of Health Provided electronically by WFP 2012 National Food and Nutrition Strategic Plan (NFNSP):– 2013 - 2018 Ministry of Health Provided electronically by WFP n/a Nationa Food and Nutrition Policy (NFNP) Ministry of Health Provided electronically by WFP n/a Q1 FY2015 - Quarterly Report – Oct-Dec 2014 Save the Children (STC) Provided electronically by STC January 2015 FY2015 Work Plan – Q2 Jan-Mar 2015 Save the Children Provided electronically by STC n/a MEMO of Kitchen Garden Price Assessment – from IILP to STC IILP Provided electronically by STC November 2014 Integrated Pest Management – Steps 2013/2014 Good Agriculture Practices (GAP) & Organic Farming Centre for the Development of Enterprise (EU) PowerPoint Presentation n/a ANNEX D: OVERALL MONITORING & EVALUATION MATRIX Overall Monitoring and Evaluation Matrix Period of Performance: Oct 2011 – Sept 2014* 35 S = Strong Likelihood M = Medium Likelihood W = Weak Possibility 36 Removed as an indicator Indicators Life of Project (LOP) Targets Actual cumulative as of Sept 2014 Program Element Source(s) of Verification Probability of LOP targets being meet by July 2016 (end-of￾project) S M 35 W 1 No. of rural households benefitting directly from USG interventions 104,350 70,834 n/a Annual ReportsFY13 & FY14, PMP Progress Report (Jan15)  IR 1 Improved agriculture productivity 2 No. of hectares under improved technologies or management practices as a result of USG assistance 1,176 847 1 Annual & Quarterly Reports, Work Plans, M & Plan FY14  3 Gross margin per unit of land, kilogram or animal of selected product Maize 82 211 n/a Annual & Quarterly Reports, Work Plans  Beans 402 595 n/a Annual & Quarterly Reports, Work Plans  Dairy 318 315 n/a Annual & Quarterly Reports, Work Plans  IR 1.1 Improved access to production and processing practices 4 No. of farmers and others who have applied new technologies or management practices as a result of USG assistance 52,500 26,499 1 Annual & Quarterly Reports, Work Plans  5 No. of new technologies or management practices in one of the following phases of development 15 17 ** 1 Annual & Quarterly Reports, Work Plans  36 6 No.of farmers accessing agriculture extension services 75,000 *** 58,994 *** 1 Annual ReportsFY13 & FY14, PMP Progress Report (Jan15)  IR 1.2 Increased access to business development services 7 No. of MSMEs receiving business development services from USG assisted sources 2,000 1,584 1 Annual ReportsFY13 & FY14, PMP Progress Report (Jan15)  IR 2 Increased access to agricultural finance * The evaluation team was tasked with reviewing performance from July 2011 to July 2014 (3 years). However, as IILP reporting is from October to September annually it has been decided to analyse the project’s performance from October 2011 to September 2014 having in mind the LOP final targets. 8 Value of agriculture and rural loans $174,376 $137,846 2 PMP Progress Report (Jan15)  IR 2.1 Improved institutional capacity of service providers 9 %age of created and/or supported ISLGs with linkages to financial institutions 70% 68% 2 Annual ReportsFY13 & FY14, PMP Progress Report (Jan15)  10 No. of private enterprises, producers organisations, water users associations, trade & business associations & community based organisations receiving USG assistance 2,000 1,555 1 Annual ReportsFY13 & FY14, PMP Progress Report (Jan15)  11 No. of new financial products developed 5 2 2 Annual ReportsFY13 & FY14, PMP Progress Report (Jan15)  12 No. of loans made by MFIs 3,600 1,581 2 Annual ReportsFY13 & FY14, PMP Progress Report (Jan15)  IR 2.2 Improved capacity of borrowers to access and manage financial services 13 No. of beneficiares accessing financial services 40,000 31,391 2 Annual ReportsFY13 & FY14, PMP Progress Report (Jan15)  14 No. of MSMEs receiving USG assistance to access banks loans 3,600 2,298 2 Annual ReportsFY13 & FY14, PMP Progress Report (Jan15)  15 No. of people trained in financial literacy 48,420 34,148 1 Annual ReportsFY13 & FY14, PMP Progress Report (Jan15)  16 No. of people trained in basic literacy 28,000 25,223 1 Annual ReportsFY13 & FY14, PMP Progress Report (Jan15) IR 3 Improved nutrition among rural poor 17 %age of beneficiaries consuming a diet with increased diversification 85% 26% 5 Annual ReportsFY13 & FY14, PMP Progress Report (Jan15)  18 %age of beneficiaries with increased knowledge about balanced and diversfied diet 90% 87% 5 Annual ReportsFY13 & FY14, PMP Progress Report (Jan15)  19 No. of people trained in child health & nutrition through USG supported programs 125,000 97,334 5 Annual ReportsFY13 & FY14, PMP Progress Report (Jan15)  20 No. of kitchen gardens established 11,000 11,416 5 Annual ReportsFY13 & FY14, PMP Progress Report (Jan15)  ** LOP target later removed from indicator – (see Annual Performance Matrix). *** Differs from figures in FY14 Annual Report i.e. 45,000 and 51,361 respectively. ANNEX E: ANNUAL PERFORMANCE MATRIX Annual Performance Matrix Timeframe of Review: Oct 2011 – Sept 2014 (includes ongoing data and LOP targets for reference) PROJECT INDICATORS FY12 (Oct11- Sept12) FY13 (Oct12- Sept13) FY14 (Oct13- Sept14) FY (Oct14-Sept15) FY16 Planned Target Actual Planned Target Actual Planned Target Actual Planned Target Actual (ongoing) Planned Target Goal: Improve Livelihoods of Rwanda’s very poor 1 No. of rural households benefitting directly from USG interventions 3,409 15,272 28,592 33,314 42,250 70,834 100,750 87,431 104,350 IR 1 Improved agriculture productivity 2 No. of hectares under improved technologies or management practices as a result of USG assistance 159 196 500 526 830 847 1,176 1,029 1,176 Gross margin per unit of land, kilogram or animal of selected product Maize n/a n/a n/a n/a 82 211 219 211 222 3 Beans n/a n/a n/a n/a 402 595 619 595 625 Dairy n/a n/a n/a n/a 318 315 328 315 331 IR 1.1 Improved access to production and processing practices 4 No. of farmers and others who have applied new technologies or management practices as a result of USG assistance 1875 1874 10,500 14,104 28,125 26,499 40,500 29,618 52,500 5 No. of new technologies or management practices in one of the following phases of development 10 8 15 12 16 15 n/a n/a removed 6 No.of farmers accessing agriculture extension services 17,500 22,565 35,000 34,367 42,000 58,994 70,000 66,431 75,000 IR 1.2 Increased access to business development services 7. No. of MSMEs receiving business development services from USG assisted sources 30 0 400 529 1,200 1,584 1,800 1,834 2,000 IR 2 Increased access to agricultural finance 8 Value of agriculture and rural loans $5,000 $32,847 $285,200 $82,736 $110,000 $137,846 $151,631 $32,562 $174,376 IR 2.1 Improved institutional capacity of service providers 9 No. of institutions / organisations that, as a result of USG assistance are in one of these five stages of improved institutional capacity 15 79 n/a n/a n/a n/a n/a n/a removed 10 %age of created and/or supported ISLGs with linkages to financial institutions 35% 44% 50% 65% 60% 68% 70% 51.5% 70% 11 No. of private enterprises, producers organisations, water users associations, trade & business associations & community based organisations receiving USG assistance 500 417 1,200 1,248 1,500 1,555 1,800 2,001 2,000 12 No. of new financial products developed 0 0 2 1 2 2 2 0 5 * Changed to number of people trained in financial literacy ** Number of people trained in basic literacy (reported separately from financial literacy since FY14) *** Only model kitchen gardens will be promoted in FY15 and FY16 Note: Although this evaluation is tasked with reviewing performance from July 2011 – July 2014 it is more realistic to review activities between October 2011 – September 2014 as this is referred to as FY12 to FY14 in IILP reporting documentation. Detailed information is already available for this period to which the Evaluation Team had access. 13 No. of loans made by MFIs 0 0 500 421 2,400 1,581 3,000 1,738 3,600 IR 2.2 Improved capacity of borrowers to access and manage financial services 14 No. of beneficiares accessing financial services 3,000 9,530 24,000 21,931 30,000 31,391 36,000 41,232 40,000 15 No. of MSMEs receiving USG assistance to access banks loans 6 0 500 1,150 2,400 2,298 3,000 2,353 3,600 16 No. of people trained in one or more forms or literacy, basic numeracy & financial literacy 7,500 8,576 36,870 37,466 30,420 34,148 48,420 36,956 48,420 17 No. of people trained in basic literacy ** n/a n/a n/a n/a 18,000 25,223 30,000 25,223 35,000 IR 3 Improved nutrition among rural poor 18 %age of beneficiaries consuming a diet with increased diversification 55% 24% 70% 24% 75% 26% 80% 85% 19 %age of beneficiaries with increased knowledge about balanced and diversfied diet 80% 84% 80% 84% 85% 87% 90% 90% 20 No. of people trained in child health & nutrition through USG supported programs 18,750 20,063 37,500 42,123 55,397 97,334 125,000 100,849 125,000 21 No. of kitchen gardens established 2,000 3,215 5,000 7,233 8,000 11,416 240*** 11,540 11,000 ANNEX F: ILLUSTRATIVE INTERVIEWEE QUESTIONNAIRES INTERVIEWEE QUESTIONNAIRE FOR IILP PROJECT TEAM Background Information of Interviewee 1 Name of interviewee 2 Position of person interviewed SOW Evaluation Questions: All Q.1 When did you become part of IILP and what is your exact role? (From the beginning in July 2011 or later) A.1 Q.2 Since you became involved how would you describe your main responsibilities, how they are carried out and reported? A.2 Q.3 How would you describe your relationship with project partners with whom you are involved? (Positive / negative)? Can you elaborate on either view? A.3 Q.4 Do you work mostly on your own or as part of a team? What are your views on either position in terms of being able to carry out your brief? A.4 Q.5 What reporting mechanism do you use to record your activities (daily, weekly, monthly, quarterly) and to whom provided? Do you receive feedback on your activities or performance? A.5 Q.6 Which aspects of your work could be improved, changed or removed for you to perform more effectively? A.6 Q.7 In terms of the project in general, can you suggest any areas of improvement that would contribute towards its overall objectives being accomplished more effectively? A.7 Q.8 Can you briefly describe your relationship with the project’s beneficiaries in as much as you have direct dealings with them, especially with regard to women? A.8 INTERVIEWEE QUESTIONNAIRE FOR FINANCIAL INSTITUTIONS Background Information of Interviewee 1 Name of interviewee 2 Name of institution 3 Main activities of institution 4 Position of person interviewed 5 Province and district of the institution 6 Legal status of institution 7 When institution was established 8 Nature of institutions relationship with IILP SOW Evaluation Questions: 1a and 1e Q.1 How and when did you become officially involved with IILP project and in what capacity? A.1 Q.2 How would you assess the current situation regarding issues concerning ‘access to finance’ for entrepreneurs and/or SMEs in your jurisdiction? To what extent has IILP intervention influenced / impacted the playing field? A.2 Q.3 To your knowledge has there been an increase in the number of loans made in the communities where you operate? A.3 Q.4 To what extent has IILP’s intervention influenced / impacted the playing field regarding the financial services sector in Rwanda? A.4 Q.5 What new products have been developed for low-income entrepreneurs that were not previously available (i.e. prior to IILP’s involvement) A.5 Q.6 What progress has been made in recent years in the numbers of women being able to atke out loans for their businesses? To your knowledge, has IILP’s interventions influence ‘access to finance’ for women individually or as a group? A.6 Q.7 One of IILP’s mandates is to support the expansion of mobile money in Rwanda. What is (has been) your involvement in this endeavour (if any) and do you believe the use of cell technology is relevant to managing a samll business in Rwanda today? A.7 Q.8 In your view, how would you describe the link between MFI’s and beneficiary? Have matters improved following IILP’s intervention in this field? A.8 Q.9 To your knowledge, has IILP support for financial services increased entrepreneurship and/or incomes among the project’s beneficiaries? A.9 INTERVIEWEE QUESTIONNAIRE FOR SAVINGS GROUPS (ISLG) Background Information of Interviewee 1 Name of interviewee 2 Name of institution 3 Main activities of institution 4 Position of person interviewed 5 Province and district of the institution 6 Legal status of institution 7 When institution was established 8 Nature of institutions relationship with IILP SOW Evaluation Questions: 1a , 1e, 2 Q.1 How and when did you become officially involved in the IILP project? A.1 Q.2 In general terms, what has been you experience in collaborating with IILP project? A.2 Q.3 How would you describe your relationship today with formal financial institutions such as MFIs, commerical banks? A.3 Q.4 Has you ISLG been able to secure a bank account with a commercial bank. If so, can you provide details of when? A.4 Q.5 What do members of your ISLG you loans for? Can you provide one or two examples? A.5 Q.6 Generally speaking, do your members repay their loans on time? A.6 Q.7 To your knowledge, what do your members use their savings for? Can you provide one or two example? A.7 Q.8 Since your involvement with the IILP project has access to loans improved for your members’ businesses? A.8 INTERVIEWEE QUESTIONNAIRE FOR IILP PARTNERS (CARITAS, AEE, ADEPR, DUHAMIC-ADRI) Background Information of Interviewee 1 Name of interviewee 2 Name of institution 3 Main activities of institution 4 Position of person interviewed 5 Region and district of the institution 6 Legal status of institution 7 When institution was established 8 Nature of institutions relationship with IILP SOW Evaluation Questions: 1c, 1d, 1f, 1g, 2,3 Q.1 How and when did you become officially involved in the IILP project? A.1 Q.2 In general terms, what has been you experience in collaborating with IILP project – positive and/or negative experience? Provide examples of either. A.2 Q.3 How successful has IILP been in reaching its intended beneficiaries in particular with their ability to access business loans and investing savings? A.3 Q.4 In your view, to what extent are IILP project activities contributing towards increased participation, improved income and overall ‘empowerment of women’ in targeted communities? A.4 Q.5 As far as you are aware, to what extent are local authorities and community volunteers collaborating in the implementation of project activities? A.5 Q.6 Have you noticed any significant improvement in the livelihoods of the project’s beneficiaries as a result of IILP interventions? A.6 Q.7 In your opinion, is there any aspect of the project’s activities that could be conducted differently in terms of the way in which it is being implemented? A.7 Q.8 What reporting mechanism do you employ to provide IILP with information and data on your own activities? A.8 INTERVIEWEE QUESTIONNAIRE FOR OFFICIAL BODIES (GOVERNMENT MINISTRIES, LOCAL AUTHORITIES) Background Information of Interviewee 1 Name of interviewee 2 Name of institution 3 Main activities of institution 4 Position of person interviewed 5 Region and district of the institution 6 Legal status of institution 7 When institution was established 8 Nature of institutions relationship with IILP SOW Evaluation Questions: All Q.1 How and when did you become officially involved in the IILP project? What is the precise nature of your involvement? A.1 Q.2 In general terms, what has been you experience in collaborating with IILP project – positive and/or negative experience? Provide examples of either. A.2 Q.3 How successful has IILP been in reaching its intended beneficiaries in particular with their ability to access business loans and investing savings? A.3 Q.4 In your view, to what extent are IILP project activities contributing towards increased participation, improved income and overall ‘empowerment of women’ in targeted communities? A.4 Q.5 As far as you are aware, to what extent are local authorities and community volunteers collaborating in the implementation of project activities? A.5 Q.6 Have you noticed any significant improvement in the livelihoods of the project’s beneficiaries as a result of IILP interventions especially for women? A.6 Q.7 In your opinion, is there any aspect of the project’s activities that could be conducted differently in terms of the way in which it is being implemented? A.7 Q.8 In general terms, what impact would you say IILP has had on local economies regarding improved household income and nutrition? A.8 INTERVIEWEE QUESTIONNAIRE FOR IILP STAKEHOLDERS (World Bank, EU, DfID, CIDA etc) Background Information of Interviewee 1 Name of interviewee 2 Name of institution 3 Main activities of institution 4 Position of person interviewed 5 Region and district of the institution 6 Legal status of institution 7 When institution was established 8 Nature of institutions relationship with IILP SOW Evaluation Questions: All Q.1 How and when did you become aware of USAID’s IILP project? A.1 Q.2 What is your understanding of the project’s objectives and how it is (has been) performing since you became aware of its existence? A.2 Q.3 From your perspective, what feedback have you had regarding IILP’s activities and have you formed any lasting impressions regarding its activities? A.3 Q.4 Have you or your organisation had any direct dealings with the project? If so, can you advise in what capacity? A.4 Q.5 IILP has a mandeate to ‘empower’ women in Rwanda. What is your assessment of the country’s prevailing attitude towards women, in particular female farmers or other type of entrepreneur? A.5 Q.6 In your view, is the Government of Rwanda doing enough to support rural communities and in particular ‘access to finance’ for entrepreneurs / SMEs? A.6 Q.7 Do you know of any other international players in Rwanda today who are involved in any way in rural communities regarding food security and/or promoting nutritional issues? A.7 Q.8 Are there any aspects in Rwanda’s economy that current projects such as IILP are not so far addressing? If so, can you elaborate? A.8 FOCUS GROUP DISCUSSION QUESTIONNAIRE - NUTRITION Province District Sector Service Group Nutrition Q.1 When did you become members this group ? (before / after July 2011)? How many groups are represented here today? Q.2 What kind of support have you received since joining this group & from whom? Training, services etc. Be specific. Q.3 Who provided this support and how regularly? (RPO, BCV, local agronomist, other?) Q.4 Since being trained do you feel you have acquired sufficient knowledge to be able to practically help the project’s beneficiaries (household farmers)? Q.5 How many of these household farmers have started kitchen gardens as a result of training & advice? i.e within this group and/or the other groups? Q.6 How many of them have taken out loans to start a business (i.e. become entrepreneurial)? Ask for info. re: loans from within self-help group or MFI/SACCO Date Q.7 In general, since your involvement with the IILP project, have you noticed a difference in the incomes of poor farmers and in their children’s health? (Any concrete examples) Q.8 Which aspects of your day-to-day activities could be improved, changed or removed altogether for you to earn more income? Specify biggest problems you have to deal with and how you deal with them now. (Prompt on pest control / disease management) – estimate losses. Q.9 In July 2016 the Ejo Heza project ends. How likely is it that you will continue with the activities you have been trained to deliver to the project’s beneficiaries and/or affect your own livelihoods? Q.10 Are you a member of any other group? Radio? FOCUS GROUP DISCUSSION QUESTIONNAIRE - LITERACY Province District Sector Service Group Literacy Q.1 When did you become members this group ? (before / after July 2011)? Q.2 What kind of support have you received since joining this group & from whom? Training, services etc. Be specific. Q.3 Who provided this support and how regularly? Q.4 Since being trained do you feel you have acquired sufficient knowledge to be able to save, apply for loans, start a business? Q.5 How many of you have started kitchen gardens as a result of training & advice? Q.6 How many of you have taken out loans to start a business (i.e. become entrepreneurial)? Date Q.7 In general, since your involvement with the IILP project, have you noticed a difference in your incomes and in your children’s health? (Any concrete examples) Q.8 Which aspects of your learning could be improved, changed or removed altogether for you to be able to put what you have learned into practice? Specify biggest problems you have to deal with and how you deal with them now. (Prompt on pest control / disease management) Q.9 In July 2016 the Ejo Heza project ends as well as the support from the RPOs. What effect wil this have on you and your families? Q.10 Are you a member of any other group? FOCUS GROUP DISCUSSION QUESTIONNAIRE - ISLGs Province District Sector Service Group ISLGs Q.1 When did you become members this group ? (before / after July 2011)? Q.2 What kind of support have you received since joining this group & from whom? Training, services etc. Be specific. Q.3 Who provided this support and how regularly? Q.4 Since being trained with knowledge regarding financial services how many of you have since gone on to borrow money from a bank or MFI? Q.5 Those of you who have borrowed money have you since paid back the loan or in the process of paying it back? Are you experiencing any difficulty with repayments? Q.6 How have these additional funds helped you with your business activities? Date Q.7 Do you now see an improvement in your household income as a result of being able to borrow money for your business activities? Q.8 Do you feel that your families and children in particular are healthier now as a result of your improved incomes? Q.9 In July 2016 the Ejo Heza project ends together with continuing support from the RPOs. What effect will this have on you individually and on the group? Q.10 Are you a member of any other group ? FOCUS GROUP DISCUSSION QUESTIONNAIRE – CO￾OPs Province District Sector Service Group CO-OPs Q.1 When did you become members this group ? (before / after July 2011)? Q.2 What kind of support have you received since joining this group & from whom? Training, services etc. Be specific. Q.3 Who provided this support and how regularly? Q.4 Since being trained do you feel have acquired sufficient knowledge to be able to practically help the project’s beneficiaries (household farmers)? Q.5 How many of these household farmers have started kitchen gardens as a result of training & advice? Q.6 How many of them have been involved with agriculture extension services and to what effect? Date Q.7 In general, since your involvement with the IILP project, have you noticed a difference in the incomes of poor farmers and in their children’s health? (Any concrete examples) Q.8 Which aspects of your work could be improved, changed or removed altogether for you to be able to perform your duties more effectively? Specify biggest problems you have to deal with and how you deal with them now. (Prompt on pest control / disease management) Q.9 In July 2016 the Ejo Heza project ends. How likely is it that you will continue with the activities you have been trained to deliver to the project’s beneficiaries? Q.10 Are you a member of any other group? FOCUS GROUP DISCUSSION QUESTIONNAIRE - BCVs Province District Sector Service Group BCVs Q.1 When did you become members this group ? (before / after July 2011)? Q.2 What kind of support have you received since joining this group & from whom? Training, services etc. Be specific. Q.3 Who provided this support and how regularly? Q.4 Since being trained do you feel have acquired sufficient knowledge to be able to practically help the project’s beneficiaries (household farmers)? Q.5 How many of these household farmers have started kitchen gardens as a result of training & advice? Q.6 How many of them have taken out loans to start a business (i.e. become entrepreneurial)? Date Q.7 In general, since your involvement with the IILP project, have you noticed a difference in the incomes of poor farmers and in their children’s health? (Any concrete examples) Q.8 Which aspects of your work could be improved, changed or removed altogether for you to be able to perform your duties more effectively? Specify biggest problems you have to deal with and how you deal with them now. Q.9 In July 2016 the Ejo Heza project ends. How likely is it that you will continue with the activities you have been trained to deliver to the project’s beneficiaries? Q.10 Are you a member of any other groups? ANNEX G: HOUSEHOLD SURVEY METHODOLOGY Household Survey Methodology Choice of sampling methods The Household Survey (HHS) was aimed at providing feedback on the socio-economic situation of IILP beneficiaries in 8 districts, to complement information collected through FGDs and Key Informant Interviews. In this case a non-probability based sampling, the purposive sampling methods, was used. Sample size Considering the total population of 62,404 beneficiaries of the project, the sample size was calculated using two formulas below. First the sample size was calculated from an infinite population - 264 respondents were be obtained by this formula. Then the sample size from the infinite population was used to calculate the sample size for 62,404 direct beneficiaries of the project. The final sample size using this approach came to 264 respondents. Formula for the infinite population Formula for finite population Calculation Final sample size Target Population: 62,404 beneficiaries sample size 264 Male=103 (39%) Female 131 (61%) x D Where: SS= Sample Size for infinite population Z = Z value equal to 1.96 for 95% confidence level P = population proportion 30% (0.3) M = Margin of Error at 6% (0.06) D=Design effect of 1.2 Where: SS= Sample Size for Infinite Population Ss =Sample size for finite population Pop = Population of 62,404 beneficiaries Quantitative data collection Qualitative data collection Districts # questionnaires Number of FGDs Number of interviews NGORORERO 30 5 1 RUTSIRO 30 5 1 KARONGI 42 5 1 NYAMAGABE 30 5 1 NYANZA 30 5 1 HUYE 30 5 1 GISAGARA 30 5 1 NYARUGURU 42 5 1 Total 264 40 8 Collection of Primary Data (HHS) 1. Pre- test questionnaires and feedback Prior to the start of the survey a pilot test of the data questionnaire was conducted in 4 selected provinces to check the validity of tools and methodology as well as to assess the challenges for enumerators in accessing and collecting information. Each enumerator completed three questionnaires. All pre-tested questionnaires were reviewed and entered on computer to test the practicability of data entry and necessary adjustments were made accordingly. 2. The survey A full scale data collection followed the pilot test; primary data were collected in the three districts (Ngororero, Rutsiro, Karongi) of Western Province and five districts (Nyamagabe, Nyanza, Huye, Gisagara, Nyaruguru) of Southern Provinces targeted by the project for a period of 5 days. Five (4) teams composed of four (4) enumerators under the supervision of one team leaders each, covered each districts selected one by one successively. The questionnaires were used to cover quantitative information key variables in socio economic, production, income, health and nutrition, savings, women empowerment, and marketing. Qualitative data including perceptions and relevance of the project, key challenges and opportunities were collected by means of Focus Group Discussions (FGDs) and in-depth interviews with selected beneficiaries. Completed questionnaires were collected and checked on a daily basis by the team leader and corrections was made whether necessary. 3. Quality assurance The quality assurance during primary the HHS was guaranteed through:  Providing data collection teams with training and guidelines on data collection tools.  Ensure that local actors are prepared to receive data collection teams  Day-wise field activity plan for every field team to be recorded in field log  Monitor progress vis-à-vis key dates in work plan  Debriefing after completion of field research  Review and assess the data as soon as data analysis allowed 4. Data entry and Analysis Editing the data was carried out progressively during and after the field survey. Data derived from FGDs was sorted according to key themes and sub-themes. A double data entry system was used in order to control errors. A data entry matrix was developed in SPSS. The final data spreadsheet was prepared and analyzed using Statistical Package for Social Sciences (SPSS) software and MS Excel. Frequency tables, cross-tabulations and descriptive statistics, different types of graphs and histograms was produced and used in reporting and interpreting the survey data. The analysis techniques employed tracked general trends, points of consensus and strongly held opinions. Detailed Work Plan for Field Evaluation (HHS) Locations Jan-15 Feb-15 Beneficiaries Involved 28 29 30 2 3 4 5 6 Districts Sectors Morning (10:00-12:00) Afternoon (14:00-16:00) NGORORERO MATYAZO 18 beneficiaries 12 beneficiaries RUTSIRO MUSHUBATI 18 beneficiaries 12 beneficiaries KARONGI RUBERENGERA 12 beneficiaries 12 beneficiaries KARONGI MUTUNTU 12 beneficiaries 12 beneficiaries NYAMAGABE KIBILIZI 12 beneficiaries 12 beneficiaries NYANZA RWABICUMA 18 beneficiaries 12 beneficiaries HUYE KIGOMA 18 beneficiaries 12 beneficiaries GISAGARA MUSHA 12 beneficiaries 12 beneficiaries NYARUGURU NYAGISOZI 12 beneficiaries 12 beneficiaries NYARUGURU CYAHINDA 12 beneficiaries 12 beneficiaries ANNEX H: HOUSEHOLD SURVEY QUESTIONNAIRE HOUSEHOLD SURVEY QUESTIONNAIRE For: Performance Evaluation of the USAID/Rwanda Integrated Improved Livelihoods Program (IILP) DATE OF INTERVIEW (DD/MM /YY) /___/___/2014 ENUMERATOR’S NAME ________________________________________________________________________ RESPONDENT’S NAME _____________________________________________________________________ PROVINCE DISTRICT [CODE 0.0] SECTOR [CODE 0.1] SERVICE GROUPS [CODE 0.2] [CODE 0.0] 1=KARONGI 2=RUTSIRO 3=NGORORERO 4=NYAMAGABE 5= NYANZA 6= HUYE 7= GISAGARA 8=NYARUGURU [CODE 0.1] 1=MATYAZO 2=MUSHUBATI 3=RUBERENGERA 4=MUTUNTU 5=KIBILIZI 6=RWABICUMA 7=KIGOMA 8=MUSHA 9=NYAGISOZI 10=CYAHINDA [CODE 0.2] 1=Farmers group (Cooperative) 2=Nutrition 3= ISLG 4=Literacy Section 0-Respondent Identification Sex: 1 = Male, 2 = Female Age [code 0.2] Position of the respondent in the household [CODE 0.3] What is your primary occupation [code 0.5] What is your secondary occupation [code 0.5] What is your tertiary Occupation [code 0.5] Occupation of Spouse [code 0.5] [code 0.2] 1= under 16 2=16-25 3=26-35 4=36-45 5=46-55 6=56-65 7=66-75 8=above 75 [code 0.3] 1=Head 2=Spouse 3=Son 4=Daughter 5=Hired worker 6=family relative 7= non family relative [Code 0.4] 1=Informal Education 2= Primary-incomplete 3=Primary complete 4=Secondary-incomplete 5=Secondary complete 6=Vocational Training 7=University 8=Other [Specify] [code 0.5] 1=Agricultural labor 2=Non-ag labor 3=Skilled labor 4=Student 5= Mason 5=Formal Employee 6=Not working 7=Other [Specify] Section 1-Agricultural Production 1. What is the average size of your agricultural land? [including the land you lend] 2. What is the current average production (Kg) per season 3. What was the average production (Kg) per season over the past three years 4. How did the agricultural production changed over the last three years? 2= unchanged, 1=increased, 0=decreased 5. What do you think was the main reasons for this change in production (increase)? [CODE 1.1] 6. What do you think was the main reasons for this change in production (decrease)? [CODE 1.2] [code 1.1] 1= Climate condition 2= extension services 3= equipment’s 4= proper farm management 5= proper use of fertilizers and pesticides 6= increase in land 7= Promising market 8= improved agricultural techniques 9= Kitchen garden 10= Extensions services and training 10= Other [specify] [code 1.2] 1= Bad climate condition 2= Limited extensions services 3= Limited equipment’s 4= decreased land 5= lack of agricultural inputs 6= Weed and pests 7= lack of market 8= decrease in the price 9 = loss of soil fertility 10= diseases 11= Other [specify] Animals Number of Traditional breed Number of Cross breed Number of Improved breed Current Three years ago Current Three years ago current Three years ago Cattle Goats Sheep Poultry Pigs Rabbits Guniea pigs When you need extension services or advice about agricultural activities, who do you go?[ please fill in the table below] Services Types of Service providers [code 1.4] Name of institution s Method of access to services [code 1.5] Location ( in Km from homestead) Amount paid Frequency per year Agronomy Farm management Livestock Agricultural inputs Tools and equipment Production Processing Storage Marketing Quality Assurance Financing Business Development Entrepreneurship Other [Specify] [code 1.4] 0=Ejo-Heza Project 1 = Government agencies [specify] 2 = Other Farmers 3 = Private entrepreneurs [specify] 4 = NGOs, [Specify]] 5 = Cooperative 6 = A research organization, [specify] 7 = A learning institution 9=Sector or district veterinary/Agronomist 8 = Other [specify [code 1.5] 1 = Farmer/group to farmer 2 = Extension services 3 = Media (Radio, Print, TV) 4 = Field days, demos etc. 5 = Training workshops, seminars 6 =Churches/Religious 7= Poster/Banners 8 = Other [specify] Section 2-Food Security, Nutrition and health 1) What are different food available in you region? [CODE 2.1] 2) Which type of food did you eat this week in your own house?[CODE 2.1 ] 3) In the past 12 month, where there months in which you did not have enough food for HH? 1=Yes, 0=No 4) If yes which months you didn’t have enough food to meet your HH needs?[CODE 2.2] 5) What were the main reasons of not meeting the household meals need? [CODE 2.3] 6) Has your household received any food aid in the past 2 years? 1 = Yes, 0 = No 7) If yes from which source did you receive food? [ CODE 2.4] 8) For your households, what are the two main sources of food [CODE 2.5] 9) How many meals do you have per day? 1=one, 2=Two, 3= Three, 4=More than three 10)Do you think kitchen garden have improved the nutrition status in your household? 1=Yes, 0=No 11)If yes How have kitchen garden contributed to improve you household nutrition status [Code 2.6] 12)Have you ever listened to Ejo-Heza radio programs? 1=Yes, 0=No 13)If yes how frequent per mont? ( number of times per month) 14)If yes How ejo heza radio program contributed to improve you household livelihood [Code 2.6] [code 2.2] 1=January 2=February 3=March 4=April 5=May 6=June 7=July 8=August 9=September 10=October 11=Novembe r 12=Decembe r [code 2.3] 1=Low food production 2=Limited income 3=Bad climate 4=Lack of food at the market 5=Fail in loss in other business 6=Other[Specify] [code 2.4] 1=Family and relatives 2=Friends 3=Neighbors 4=government 5=NGO 6=WFP 7=FAO 8= Ejo-Heza 9=Other [specify] [code 2.5] 1 = Own production 2 = Purchase 3 = Traded goods or services 4 = Borrowed 5 = Food aid 6 = Other [Specify [code 2.6] 1=it increased awareness on nutrition 2= reduced children malnutrition 3=increased the varieties 4= increased the quantity 5= increased the consumption of balanced diet 6= increased the income 7= reduced nutritional related diseases 8= improved children growth 9= it helped me saving more 10= it helped me knowing how to access loans 11= other [specify] Food Consumption Level Food names [codes 2.1] Quantity consumed per day for the whole household Frequency of consumption per month Source 1=Own production 2=Purchased Unit cost per Kg Current production per Kg 1= Maize grain 2= Maize flour 3= Sorghum 4= Wheat flour 5= Rice 6= Cassava (Fresh) 7= Cassava flour 8= Cassava leaves 9= Sweet potato 10= Irish potato 11= Plantain 12= Beans 13= Ground nuts 14= Peas 15= Soybeans 16= Pork 17= Chicken 18= Beef 19= Mutton 20= Goat 21= Fish 22= Rabbit 23= Dairy products 24= Vegetable (any) 25= Fruits (any) 26= Other (specify): ____ Health Which diseases or sickness is often seen in your household? [code 2.7] Which sicknesses often affect children under 5 years? [code 2.7] Which sicknesses often affect adults? [code 2.7] When is the last time a household member fallen sick? [in month] What was the sickness? How many times in year a member of the household fall sick [code 2.8] How many times one child has missed class because of sickness? [code 2.8] How many times a household member has not gone to the health center when he/she was sick? [code 2.8] Please fill the table below for the health center services usage? Name of organization Types of the organization [code 2.9] Distance between the homestead Help provided [code 2.10] Do you have health insurance? 1=Yes, 0=No If no health insurance why? [code 2.11] [code 2.9] 1=Health center 2=Dispensary 3= Hospital 4=Other [Specify] [code 2.10] 1=provide drugs/medicines 2=health mobilization 3=information on nutrition 4=family planning 5=HIV/AIDS related services 6= Maternal health 7= Other [specify] [code 2.8] 1= once in a week 2=once in a month 3=once in three month 4= once in six month 5=once in a year 6= very rare [code 2.7] 1=malaria 2=diarrhea 3= skin infections 4= pulmonary diseases 5=tuberculosis 6= worm 7= digestives diseases 8= Protein energy malnutrition [kwashiorkor/marasmus] 9= Micronutrients [iron, Vitamin a, zinc deficiencies] [specific] 10=HIV/AIDS [code 2.11] 1=poverty 2=very expensive 3=don’t need it 4=not available 5= other specify Section 3 –Literacy 1) Can you read and write? 1=Yes, 0=No 2) Where did you learn to read and write? [code 3.1] 3) Since when did you learn to read and write? [ __ __ __ __] year 4) If you know to read and write, what is the Level of education [code 3.2] 5) What have EJO-HEZA adult literacy contributed to your lifestyle [code 3.3] 6) Other member of the household, the level of literacy and occupations Household members by age Number Member who can read and write Male by Occupation Female by occupation Male Female Male Level of education Female Level of education Number Occupation [code 3.4] Number Occupation [code 3.4] 6-10 11-15 16-25 26-35 36-45 46-55 56-65 66-75 75 [code 3.1] 1= Formal School ( Primary, Secondary, University) 2= Ejo-Heza Project 3= Other projects [specify] 4= Somewhere else [Specify] [code 3.2] 1=Informal Education 2= Primary-incomplete 3=Primary complete 4=Secondary-incomplete 5=Secondary complete 6=Vocational Training 7=University 8=Other [Specify] [code 3.3] 1=self confidence 2= increase my participation in leadership places 3= opened door to more activities 4=help planning 5= Access to loan 6= better farm management 7= Increased business profit 8= Nothing 9=Other Occupation 1=Agricultural labor 2=Non-ag labor 3=Skilled labor 4=Student 5= Mason 5=Formal Employee 6=Not working 7=Other [Specify] Section 4- women empowerment (to be asked to women) 1) What are your main income generating activities? [code 4.1] 2) What was the main income generating activities before three years ago [code 4.1] 3) do you think you income has increased or decreased? 2=Didn’t change, 1=Increased, 0=Decreased 4) If it increased what do you think may be the reasons for that? [code 4.2] 5) How do you think Ejo-Heza project contributed to this increase of your income? 2=contributed nothing, 1=contributed less, 1=contributed much 6) If yes how [code 4.3] 7) Please fill in the table below to show your level of participation in social groups Name of social group Position in the group [code4.4] How month been a member Your contribution to the social Group [4.5] Advantage you get from social group [4.6] If No what are the reasons of not being in social group [4.7] Other household member of the group 1= Farmer group 2= credit and saving group 3= women group 4= church group 5= self-help group 6= community opinion leaders 7= HIV/AIDS association 8= youth group 9= health group 10=Business group 11=School committee 12=Other[specify] what do you think have been the contribution of EJO-HEZA in increasing your participation in social group [CODE 4.3] Do you ever share your opinion or be involved in decision making in your social group? 1=Yes, 0=N0 How do you think your opinion is considered and accepted by the group members? [code 4.8] Have you even been considered as the final decision maker in your social group? 1=Yes, 0=No [Code 4.1] 1= from Food crops 2= from cash crops 3= from livestock activities 4= e from handicraft 5= from renting out land 6= from small business 7= from wages/Salaries/Pension 8= from selling food aid sales 9= from begging 10= from remittances 11= from other sources [specify] [Code 4.2] 1=Better management 2=increased agricultural production 3=Diversification of income generating activities 4=Switching to off farm activities 5= increase in level of literacy [code 4.3] 1= increased my level of participation in the society 2= increased my professional skills and knowledge 3=increased my level of saving and accessing loans 4= increased my capacity of investing in income generating activities 5= decreased health related expenses due to decreased diseases 6=Other [Specify] [code 4.4] 1= Simple Member 2= Member in executive committee 3= the leader of the group 4= simple labor 5=Other [Specify] [code 4.5] 1= cash (amount) 2=Kind 3=participation in meetings 4=provide the place of meeting 5=participation in leadership bodies 6= provide extension services 7= others [code 4.6] 1= Training 2= Coaching 3= Funds 4= Place of meeting and sharing experience 5= Accessing processing infrastructures 6= Easy access to loan 7= Other [specify] [code 4.7] 0= my husband cant allow 1= Lack of information 2= Not motivated 3= Ineffectiveness of social groups 4= Not interested 5= Absence of social groups 6= Long distance 7= Membership Fees 8= Other [specify] [code 4.8] 1= Never considered 2= need support from male member to be considered 3= sometimes considered 4=always considered Section 5-Household Income What is your household poverty category level [code 5.2]? SOURCES OF INCOME Three years ago Current situation RWF/year Who participate Rwf/year Who participate [code 5.1] 1) Income from food crops 2) Income from cash crops 3) Income from livestock activities 4) Income from handicraft 5) Income from renting out land 6) Income from small business 7) Income from wages/Salaries/Pension 8) Income from food aid sales 9) Income from begging 10) Income from remittances 11) Income from other sources [specify] 12) In which of the categories do you estimate your total monthly household income, from all the above mentioned source of income. RWF per year Three years ago Current situation < 10,000 10,001- 20,000 20,001- 30,000 30,001- 40,000 40,001- 50,000 50,001-100,000 100,001-500,000 >500,000 In which social life the income coming from your activities help you in the following life needs Three years ago [Rwf/month] Current situation [Rwf/month] 1) Buy food 2) Purchase home materials 3) Buy clothes 4) House rent 5) Transportation 6) Pay school related fees 7) Savings 8) Pay health related cost including health insurance 9) Reinvest in other business 10) Celebrations/ social events/ Leisure 11) Construction of houses 12) Purchase of agricultural inputs 13) Others [Specify] [Code 5.1] 1=Head of household alone 2= The spouse of the household alone 3=Both the head and the spouse of household 4= children alone 5= all members of household including parents and children [code 5.2] 1=Category 1 2=Category 2 3=Category 3 4=Categiry 4 5=Category 5 6= Category 6 Section 6 –Finance (saving and lending) 1. What is the current level of savings of your household [Rwf/account] 2. Do you have a Bank/MFI account 1=Yes, 0=No 3. If yes what is your Bank/MFI Name 4. When did you open the bank or MFI account? 5. When did you start saving in general regardless it was in bank or somewhere else? 6. Where did you start saving [Code 6.1] 7. If you don’t have a Bank/MFI account what are the reasons [CODE 6.2] 8. If you don’t have a bank account where do you currently save? 9. Where do you use to save 3 year ago? 10. How many savings groups are you a member of? 11. How has ISLG contributed to you level of saving? [Code 6.3] 12. Can you name any three financing institutions do you know in the region 1. 2. 3. [code 6.1] 0= Bank/MFIs 1=At home 2=NGO 3=Community organization 4=Government institution 5=Relative/friend 6= Saving and lending group 7= Other [Specify] [code 6.2] 1 = Not aware of the operations of financial institutions 2= Have no enough money to save in banks/MFIs 3= No trust to banks and MFIs operations 4= Keep saving elsewhere (indicate where) 5= Religious reasons 6=Other [Specify] [code 6.3] 1= Contributed a lot 2= Contributed less 3= Contributed nothing 1. Have you ever obtained credit (loans) either formal or informal way? 1=Yes, 0=No 2. Did you obtain credit as individual or as a group? 1= Individual 2= Group 3. How do you find the process of getting loan from Bank or MFIs currently and three years ago? [code 6.4] 4. If No credit obtained why [CODE 6.5] 5. If yes, indicates which needs credit was obtained, when and from what credit source Use of credit [CODE 6.6] Year Source of credit [CODE 6.7] In cash =1 or in kind =2 Equivalent of credit in money (RWF) Annual interest rate [ code 6.4] 1=Very bad 2=Bad 3=Nothing changed 4= Good 5= Very good 6=Don’t know [code 6.5] 1 = Credit requested but didn't get 2 = No available financing institutions 3 = Interest rate is too expensive 4 = Lack of collateral (“mortgage”) 5 = Fear of being unable to pay 6 = not aware / do not have information 7 = Never thought of it 8 = Does not need credit/have enough capital 9= Procedures are too complicated 10 = Other [specify] [code 6.6] 1= Starting capital for farm activities 2= Increase the agricultural land 3= House construction 4= Extend the market 5= Farm renovation 6= Purchase of inputs agriculture production 7= Purchase of equipment or tools for agricultural activities 8= Purchase of livestock 9= Purchase of food 10= To meet health needs 11= For house rent 12= School fees 13 = Investing in Other Off farm activities [specify] 14= for home materials 15= Other [Specify] [code 6.7] 1 = Government bank/agency [Specify] 2 = Commercial bank [Specify] 3 = Informal lenders [Specify] 4 = Cooperatives [Specify] 5= Friends/Relatives [Specify] 6=Traders/Merchants [Specify] 7=Project/NGO[Specify] 8=Church [Specify] 9=Saving and lending groups[Specify] 9=Other [specify]