FINAL PERFORMANCE EVALUATION GRADUATION WITH RESILIENCE TO ACHIEVE SUSTAINABLE DEVELOPMENT (GRAD) ACTIVITY March 2017 This publication was produced for review by the United States Agency for International Development/Ethiopia. It was prepared independently by Social Impact, Inc. GRADUATION WITH RESILIENCE TO ACHIEVE SUSTAINABLE DEVELOPMENT (GRAD) ACTIVITY Final Performance Evaluation Contracted under, AID-663-C-16-000010 - EPMES Ethiopia Performance Monitoring and Evaluation Service (EPMES) for USAID/Ethiopia Project DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Graduation with Resilience to Achieve Sustainable Development (GRAD) Activity Final Performance Evaluation ABSTRACT Background: The performance evaluation of Graduation with Resilience to Achieve Sustainable Development (GRAD) Activity assessed the activity’s results at higher and intermediate levels, the extent to which the activity met its goals, the overall effectiveness of partnerships and identified the strengths and challenges in implementation. Six evaluation questions were investigated to assess the: (1) Technical areas of GRAD, which have and have not exhibited sizable results; (2) Extent of beneficiary households’ asset and income changes over the life-of-the activity; (3) Extent of households’ resiliency improvement during periods of shock; (4) Level of GRAD partners’ influence in the beneficiary households’ graduation process; (5) Extent of the activity’s contribution to gender equity and women empowerment; and (6) Effectiveness of GRAD’s collaboration/complementarity with other feed the future activities. Methods: A mixed methods approach was used, involving secondary data review, household survey, key informant interviews (KIIs) and focus group discussions (FGDs). The endline evaluation samples included: 1,602 GRAD households; 55 KIIs and 314 participants in 41 FGDs. Results: Nearly 80% of GRAD households graduated from the government Productive Safety Net Program (PSNP) food support. GRAD participants realized an increase in household income from $418 at baseline to $771 at endline. Additionally, average household savings increased from $12 at baseline to $141 at endline. The Activity organized participating households into small groups through which technical support and value chain advice for diversifying livelihood opportunities was delivered. GRAD created village economic savings associations (VESA) to promote savings and access to loans. In line with the GRAD target, participating households saved at least $1 per day. The VESA model successfully promoted savings, increased access to loans and recovered re-payments from borrowers. Agro-dealers and cooperatives were the most important suppliers of inputs to GRAD households. The Activity also made positive contributions to gender equity. Women made up 51% of the executives in VESAs, 36% of VESA members and 38% of value chain groups. Participants in KIIs and FGDs noted that changes had been observed in women’s roles in decision-making, particularly in household decision-making. These results are corroborated by the quantitative findings which show an increase in the involvement of women in: household decisions from 8.7% to 60.7%; production and livelihoods decisions from 5.4% 54%; and financial decisions from 7.1% to 56%. Women’s financial contribution to the household was notably mentioned as having given women strong social capital within their families. Conclusion: GRAD improved households’ ability to cope with climate change through improving financial safety nets and improved inputs. The activity was also successful in increasing the incomes of participating households and in contributing towards creating space for women and men to explore and discuss gender issues that resulted in the advancement of gender equity and women’s empowerment and a general improvement of families’ nutritional awareness and dietary practices. Graduation with Resilience to Achieve Sustainable Development (GRAD) Activity Final Performance Evaluation CONTENTS Abstract..........................................................................................................................................iii Tables …………………………………………………………………………………………………...vi Figures ...........................................................................................................................................vii Acronyms.....................................................................................................................................viii Acknowledgements....................................................................................................................... 1 Executive Summary........................................................................................................................i Findings, Conclusions, and Recommendations by Evaluation Question ........................................................ii EQ 1: Key technical areas..................................................................................................................................ii EQ 2: Change in household income and assets...........................................................................................vi EQ 3: Crisis modifier, benefits and resiliency .............................................................................................vii EQ 4: Influence of GRAD in household graduation process..................................................................viii EQ 5: Gender equity and women’s empowerment .................................................................................viii EQ 6: Effectiveness of consortium and project collaboration ................................................................ ix Introduction................................................................................................................................... 1 Purpose......................................................................................................................................................................... 2 Evaluation Questions................................................................................................................................................. 2 Methodology................................................................................................................................................................ 3 Evaluation Findings ....................................................................................................................... 6 Overall performance of GRAD on outcome indicators................................................................................... 6 EQ 1: Key technical areas........................................................................................................................................ 6 EQ 1a: Household graduation rate from PSNP........................................................................................... 6 EQ 1b: Household income improvement...................................................................................................10 EQ 1c: Technical value chain advice.............................................................................................................13 EQ 1d: Access to finance................................................................................................................................15 EQ 1e: Extension services..............................................................................................................................17 EQ 1f: Input supplies and market linkages..................................................................................................19 EQ 1g: Nutrition...............................................................................................................................................21 EQ 1h: Climate change adaptation ...............................................................................................................23 EQ 1i: Innovations............................................................................................................................................24 EQ 2: Change in household income and assets................................................................................................25 Findings................................................................................................................................................................25 Conclusions........................................................................................................................................................26 Recommendations............................................................................................................................................26 EQ 3: Crisis modifier, benefits and resilience ...................................................................................................27 Findings................................................................................................................................................................27 Conclusions........................................................................................................................................................28 Recommendations............................................................................................................................................29 Graduation with Resilience to Achieve Sustainable Development (GRAD) Activity Final Performance Evaluation EQ 4: Influence of GRAD in household graduation process.........................................................................29 Findings................................................................................................................................................................29 Conclusions........................................................................................................................................................29 Recommendations............................................................................................................................................29 EQ 5: Gender equity and women’s empowerment.........................................................................................30 Findings................................................................................................................................................................30 Conclusions........................................................................................................................................................31 Recommendations............................................................................................................................................31 EQ 6: Effectiveness of consortium and project collaboration.......................................................................31 Findings................................................................................................................................................................31 Conclusions........................................................................................................................................................32 Recommendations............................................................................................................................................32 Conclusions .................................................................................................................................. 33 Recommendations ...................................................................................................................... 35 Annexes........................................................................................................................................ 37 Annex 1: Statement of Work................................................................................................................................37 Annex 2: Places visited, list of organizations, and people interviewed (plus GRAD Evaluation contact details).........................................................................................................................................................................48 Annex 3: GRAD Evaluation Design Matrix........................................................................................................53 Annex 4: GRAD Activity Endline Quantitative Household Survey Questionnaire..................................55 Annex 5: GRAD Final Performance Evaluation Qualitative Questions Guide Matrix.............................67 Annex 6: Bibliography of critical background documents..............................................................................70 Annex 7: Semi-structured Interview Guides (SIGs), 1–9 templates............................................................71 Annex 8: Coded Summary Sheets (SSs) for KIIs & FGDs with stakeholders............................................89 Annex 9: Crisis Modifier Summary Sheet ..........................................................................................................92 Annex 10: Reference tables of GRAD survey results.....................................................................................94 Annex 11: Regression Output Tables...............................................................................................................100 Annex 12: Maps of the GRAD activity survey woredas...............................................................................104 Annex 13: GRAD Final Performance Evaluation Team CVs .......................................................................105 Graduation with Resilience to Achieve Sustainable Development (GRAD) Activity Final Performance Evaluation TABLES Table 1. GRAD operational areas (regions and woredas)....................................................................................... 1 Table 2. GRAD Activity Results Framework.............................................................................................................. 1 Table 3. Number of kebeles and respondents at baseline and endline, with effect size detectable, by woreda ........................................................................................................................................................................................ 4 Table 4. Key informant interviews. ............................................................................................................................... 5 Table 5. Gender participation in KIIs and FGDs and members in their organizations..................................... 5 Table 6: GRAD performance on outcome indicators.............................................................................................. 6 Table 7. GRAD's cumulative graduations, target and actual ................................................................................... 7 Table 8. Average household income by woreda, 2015–16....................................................................................11 Table 9. Mean savings amounts reported by sample households in 2012 and 2016, in USD........................16 Table 10. Percentage of households receiving training in 2015–16 by training type.......................................18 Table 11. Mean household expenditures by gender and woreda, 2012 and 2016...........................................25 Table 12. Asset counts at baseline and endline........................................................................................................26 Table 13. Months of food security by woreda, baseline and endline ..................................................................27 Table 14. Food security and weather-related crop loss, baseline and endline .................................................27 Table 15. Percent of households with weather-related crop loss by woreda, baseline and endline ..........28 Table 16. GRAD quantitative and qualitative data collection ...............................................................................94 Table 17. Key result areas considered in GRAD's Results Framework.............................................................94 Table 18. GRAD operational areas (regions and woredas) ..................................................................................94 Table 19. Key informant interviews............................................................................................................................94 Table 20. Gender participation in KIIs and FGDs and members in their organizations.................................95 Table 21. GRAD's cumulative graduations, target and actual...............................................................................95 Table 22. Average household income by woreda, 2015–16 .................................................................................95 Table 23. Mean savings amounts reported by sample households in 2012 and 2016, in ETB ......................95 Table 24. Mean amounts of loans households accessed in 2012 and 2016, by source...................................96 Table 25. Percentage of households receiving training in 2015–16 by training type.......................................96 Table 26. Mean household expenditures by gender and woreda, 2012 and 2016 (amounts in ETB).........96 Table 27. Percent of households that borrowed from various sources by woreda, 2016............................97 Table 28. Percent of households involved in Value Chain Activities by woreda, 2016..................................97 Table 29. Mean annual value chain derived income by gender and woreda, 2016 (amounts in ETB).........97 Table 30. Average value of household-held livestock by woreda (amounts in ETB) ......................................98 Table 31. Average values of household assets by woreda (amounts in ETB)...................................................98 Table 32. Average households' productive assets value by woreda (amounts in ETB)..................................98 Table 33. Mean annual household expenditures by gender and woreda in 2012 and 2016 (amounts in ETB) ......................................................................................................................................................................................98 Table 34. Average months food needs met between November 2015 to October 2016 ............................99 Table 35. Diet diversity score ....................................................................................................................................100 Table 36. Expenditures.................................................................................................................................................100 Table 37. Total number of months’ food needs met............................................................................................101 Table 38. Household graduation from PSNP..........................................................................................................101 Table 39. Household hunger scale.............................................................................................................................101 Table 40. Household income......................................................................................................................................102 Table 41. Total household savings.............................................................................................................................102 Table 42. Number of months with all food needs met ........................................................................................102 Graduation with Resilience to Achieve Sustainable Development (GRAD) Activity Final Performance Evaluation FIGURES Figure 1. GRAD intervention areas............................................................................................................................... 1 Figure 2. Maps of surveyed GRAD woredas............................................................................................................... 4 Figure 3. Graduation rate by woreda ........................................................................................................................... 7 Figure 4. Graduation rate by woreda and gender of household head.................................................................. 7 Figure 5. Livelihoods shocks by graduation status and woreda.............................................................................. 8 Figure 6. Months of food security by graduation status and woreda.................................................................... 8 Figure 7. Income by graduation status and woreda.................................................................................................11 Figure 8. Livelihoods shocks and income, by woreda.............................................................................................12 Figure 9. Loan sources....................................................................................................................................................13 Figure 10. VESA program components......................................................................................................................14 Figure 11. Loan sources at baseline versus endline.................................................................................................15 Figure 12. Loan size by source, baseline and endline..............................................................................................16 Figure 13. Access to loans by purpose, baseline and endline................................................................................16 Figure 15. Suppliers of agricultural inputs..................................................................................................................20 Figure 16. Sale outlet use by woreda..........................................................................................................................21 Figure 17. Diet diversity scores, baseline versus endline.......................................................................................22 Figure 18. Food shortage coping mechanisms, baseline versus endline .............................................................23 Figure 19. Asset counts at baseline and endline, by woreda.................................................................................26 Graduation with Resilience to Achieve Sustainable Development (GRAD) Activity Final Performance Evaluation ACRONYMS ADS Automated Directives System AGP-AMDE The Agricultural Growth Program – Agribusiness and Market Development ASE Agri Service Ethiopia AT+ AID Tracker Plus BoA Bureau of Agriculture CDCS Country Development Cooperation Strategy CF Community Facilitator CIP International Potato Center CLA Collaborating, Learning, and Adaptation CM Crisis Modifier COP Chief of Party CP Community Promoter CRS Catholic Relief Services DA Development Agent DO Development Objective DQA Data Quality Assessment ENGINE Empowering New Generations to Improve Nutrition and Economic Opportunity EPMES Ethiopia Performance Monitoring and Evaluation Service EQ Evaluation Question ETB Ethiopian birr (currency) FEMA Farmers’ Economic and Market Association FGD Focus Group Discussion FTC Farmer Training Center FTF Feed the Future FY Fiscal Year GIS Geographic Information System GoE Government of Ethiopia GPS Graduation Prediction System GRAD Graduation with Resilience to Achieve Sustainable Development HABP Household Assets Building Program HEW Health Extension Worker HH Household HQ Headquarters ICT Information and Communication Technology IEE Initial Environmental Examination IGA Income Generating Activity IP Implementing Partner IPTT Indicator Performance Tracking Table KII Key Informant Interview LSPs Local Service Providers M&E Monitoring and Evaluation MCS Meki Catholic Secretariat MECS Monitoring and Evaluations Capacity Strengthening MFI Microfinance Institution NGO Non-Governmental Organization NSA Nutrition-Sensitive Agriculture OCA Organizational Capacity Assessment OFSP Orange-Fleshed Sweet Potato ORDA Organization for Rehabilitation and Development in Amhara PICS Purdue Improved Crop Storage Graduation with Resilience to Achieve Sustainable Development (GRAD) Activity Final Performance Evaluation PMP Performance Management Plan PSNP Productive Safety Net Program REST Relief Society of Tigray RuSACCO Rural Savings and Credit Cooperative SART Sub Saharan African Research and Training Center SI Social Impact, Inc. SIG Semi-structured Interview Guide SME Small and Medium Enterprise SNNPR Southern Nations, Nationalities, and Peoples Region SNV The Netherlands Development Organization SOW Scope of Work TA Technical Assistance USAID United States Agency for International Development USD United States Dollar USDA United States Department of Agriculture USG United States Government VC Value Chain VESA Village Economic and Social Association VSLA Village Savings and Loans Association ACKNOWLEDGEMENTS The evaluation team first wishes to thank Mr. Awoke Tilahun, Contracting Officer’s Representative for USAID’s Ethiopia Performance Monitoring and Evaluation Service Activity (EPMES), for his close guidance and cordial assistance to us throughout the evaluation process. Thanks also go to members of the EMPES Field Office, specifically, Francis Ogojo Okello, Chief of Party, and Worku Ambelu, Evaluation Methods Specialist, for their time spent on this evaluation. We particularly appreciate their extra effort in helping the evaluation team to strengthen the quantitative and qualitative data analysis and for suggesting ways to improve this report. Among the Social Impact (SI) project management team in Washington, D.C., Tasneem Nahar (Project Manager) and Lisa Anderberg (Project Assistant) provided consistent backstopping on managerial support, including organizational procedures and ethics. Also, our thanks to James Fremming for his comments on the draft report. Likewise, thanks to many people on the SI Field Office team, especially the financial administration staff, including Solomon, Melat, Mariam, and Dawit. They played pivotal roles in supporting the evaluation team by punctually arranging travel, hotels, and appointments. The evaluation team would also like to thank the Graduation with Resilience to Achieve Sustainable Development (GRAD) Coordination Section, the agriculture and food security specialists within USAID Ethiopia and CARE-Ethiopia, and CRS-Ethiopia senior staff for providing not only informative sessions to the team but also for providing valuable reference documents on GRAD-related activities. Furthermore, our thanks go to the implementing partner (IP) staff of the Organization for Rehabilitation and Development in Amhara (ORDA) in Bahir Dar and the Relief Society of Tigray (REST) in Mekele, who were with the team throughout field work and facilitated key informant interviews (KIIs) and focus group discussions (FGDs). Tadesse Zerihun Team Leader, GRAD Final Performance Evaluation Team December 22, 2016, Boston, MA Source: Aspire, Building Resilience in the Ethiopian Highlands, CARE Ethiopia. https://www.usaid.gov/sites/default/files/documents/1860/ASPIRE.pdf Note: The map only shows woredas visited by the evaluation team (those listed in Orange). GRAD INTERVENTION AREAS Regions ORANGE Woredas Visited by Evaluation Team REDDISH￾BROWN Woredas in GRAD-Project but not visited by Evaluation Team BLACK Tigray Region 1. Endemehoni 2. Raya Azebo 3. Ofla 4. Alamata Amhara Region 5.Libo Kemkem 6.Lay Gayint SNNP Region/Guraghe Zone 7.Meskan 8.Mareko Oromia Region 9. Adami Tulu Jido Kombolcha 10. Zeway Dugda 11. Arsi Negelle 12. Shalla SNNP Region/Sidama Zone 13. Hawassa Zuria 14. Hawela Tula 15. Shebedino 16. Loka Abaya Figure 1. GRAD intervention areas i EXECUTIVE SUMMARY Purpose. Social Impact, Inc. (SI) conducted this final performance evaluation of the Graduation with Resilience to Achieve Sustainable Development (GRAD) activity, as a part of the services it provides to USAID Ethiopia under the Ethiopia Project Monitoring and Evaluation Service (EPMES) Activity. The main purpose was to examine GRADs development outcomes or results at higher and intermediate levels and the extent to which GRAD met its goals, and to determine the overall effectiveness of its partnerships and any strengths and challenges during implementation. The recommendations from this evaluation will help improve USAID’s programming decisions for future similar livelihoods and food security programs. Project background. GRAD was a five-year, USAID-funded activity that ran from December 2011 to December 2016. GRAD was designed to support and enhance livelihood options of chronically food insecure households by promoting and supporting on-and off-farm income generating activities, facilitating output and input market linkages, and increasing access to microfinance services. The strategic objective of GRAD was to graduate at least 50,000 chronically food-insecure households from the Government of Ethiopia’s (GoE’s) Productive Safety Net Program (PSNP) food support in 16 targeted woredas, and to increase each household’s income by $365 by the activity’s fifth year in 16 woredas of Tigray, Amhara, Oromia, and Southern Nations, Nationalities, and Peoples Region (SNNPR). CARE served as the primary implementer of GRAD, working with a consortium of partners, including: Catholic Relief Services (CRS) with Meki Catholic Secretariat (MCS), Organization for Rehabilitation and Development in Amhara (ORDA), the Relief Society of Tigray (REST), and Agri-Service Ethiopia (working in Mareko areas). GRAD’s technical partners included the Netherlands Development Organization (SNV) and the Feinstein International Center of Tufts University. Evaluation design. This final performance evaluation addresses six evaluation questions (EQs), which are listed in the following section along with key findings, conclusions, and recommendations. The evaluation team applied a mixed methods approach involving four approaches: secondary data review, household survey, key informant interviews (KIIs), and focus group discussions (FGDs). The approach is summarized in an evaluation design matrix showing the source of data, method of data collection, and the tool being used to answer each of the evaluation questions (Annex 3). All data collected and presented in this report are disaggregated, as appropriate, by sex and woreda. The quantitative method involved conducting household interviews with a representative sample of GRAD beneficiaries selected at random. The evaluation team determined the minimum number of respondents (1,602) required to conduct a two-sample test of proportions (baseline and endline), using independent random samples from the GRAD-targeted households. The results on key indicators were compared with the baseline earlier conducted during the activity inception. Additionally, secondary analysis was conducted on selected indicators from the routine monitoring date to analyze trends. Qualitative data was collected from KIIs and FGDs in Addis Ababa and in four survey woredas selected from Amhara, Oromia, SNNPR, and Tigray Regions, just as at baseline. A total of 1,602 household surveys and 99 KIIs/FGDs were conducted (see Tables 3, 4, and 5). Limitations. The team faced several limitations related to the quantity and quality of data available. First, fieldwork was scheduled for November 2016, during harvest season, which meant farmers had limited time to participate in interviews and FGDs. Second, GRAD baseline data was not compiled in an organized dataset, but rather, was provided across several Excel worksheets, making it difficult to merge with the endline data. Additionally, parts of the baseline data were not found in the baseline data files. Therefore, the fractured state of baseline data lead to limitations in the design and data quality of the endline household survey data and the subsequent comparison of the baseline and endline results for some variables of interest, such as income data. ii Findings, Conclusions, and Recommendations by Evaluation Question EQ 1: Key technical areas Which of the following key technical areas of GRAD exhibited sizable results and which one(s) have not? Evaluation Question (EQ) 1a. Graduate 50,000 chronically food-insecure households from PSNP food support in targeted woredas. Key findings: The quantitative survey provides representative information on household participation in PSNP. Nearly all respondents (96%) indicated that their households had participated in PSNP, and 89% have had at least one household member participate in a new value chains activity since joining GRAD. Of these, 53% have graduated from PSNP, somewhat below GRAD's reported graduation rate of 39,306 households out of a targeted 65,000 who participate in direct service delivery, or 60% of those engaged. Those who experienced livelihoods shocks in the past year were less likely to have graduated. Overall, 37.2% of female-headed households reported graduating, versus 58.2% of male-headed households. The gender divide was most pronounced in Lay Gayint and Endamehoni, while Hawassa Zuria saw no gender gap. KIIs differed in their views regarding the practicality of the benchmarks set for graduation. Some believed the flexibility and transparency to be a strength of the graduation process. Others viewed the graduation process’s emphasis on flexibility, which gives local authorities a great deal of discretion in making graduation decisions, to sometimes lead to a lack of clarity or perception that the process was politicized. KIIs also critiqued the use of simple income and expenditure measures as benchmarks for graduation, which they felt did not adequately capture the complexities faced by GRAD-targeted households. Conclusions: Given that GoE, not GRAD, ultimately decides on graduation and the flaws identified in the process, this is not an ideal benchmark for a USAID project. Recommendations: 1. While graduation plays an important role in household access to food and therefore should continue to be tracked, it is not a strong project indicator because of the many external influences on graduation. Future USAID-funded projects should not have graduation as a key indicator of success if the decisions about graduation are outside the control of USAID’s activity. It might serve well as a proxy indicator, since the USAID activity only contributes towards the graduation outcome. 2. Gender of household heads needs to be more explicitly taken into consideration in future project design to ensure that female-headed households’ unique barriers to livelihoods opportunities are addressed. 3. In the upcoming review of the graduation process and benchmarks planned by the GoE , USAID should use lessons learned from the GRAD activity to advocate for evidence-based graduation and an appropriate, responsive graduation system. 4. Given the vulnerability of Ethiopia to weather-related shocks, future USAID-funded food security projects should consider more explicitly addressing drought resilience, and strengthening linkages with drought resilience activities of other stakeholders. EQ 1b. Targeted households’ income improvement to reach the $365 increase at the end of the life of activity Key findings: On average, GRAD beneficiary households had annual income of $771 at the endline survey, an increase of $353 since CARE’s baseline calculation of $418 in annual income. Average income varied by woreda. Endamehoni and Lay Gayint had relatively high average incomes, at $1,039 and $739 respectively (Table 8). In the Southern woredas, average income was lower: $684 in Ziway Dugda and iii $539 in Hawassa Zuria. In Lay Gayint and Endamehoni, graduation was associated with higher income. Overall, livelihoods shocks have a negative association with income. Conclusions: GRAD households on average appeared to increase their income by the targeted $1 per day. However, this still leaves many households in extreme poverty, and climate shocks leave many in a continued state of vulnerability. Northern woredas were better off compared to southern woredas. Recommendation: 5. See Recommendation 4 (EQ1a) EQ 1c. Participating households are organized into small groups and supported with technical value chain advice to take advantage of diversified livelihood opportunities. Key findings: In all four woredas, FGD and KII participants expressed satisfaction with the technical Value Chain (VC) advice received from GRAD facilitators. Most Village Economic and Social Association (VESA) members attach more value to the knowledge and skills they obtained from GRAD than changes in assets or income that they may have experienced. Microfinance Institutions (MFIs) and VESAs were the most common loan sources. In the quantitative survey, 41.1% of respondent’s report having ever taken a loan from a MFI, while 39.3% report having taken a loan from a VESA. Conclusions: The overall model of the VESA appears to have been quite successful. Enrollment and use of the VESAs were widespread, and reports of participants’ experiences with the VESAs were positive. The training and increased knowledge were considered the most valuable part of the VESAs. Recommendation: 6. The VESA model of combining knowledge with access to credit should be continued through future programming. EQ1d. Access to finance for beneficiaries is increased to fund livelihoods activities. Key findings: Household savings increased in all woredas. In all woredas, VESAs were the most common platform for saving; 77% of respondents reporting having some savings in VESAs, compared to 22% with savings in banks, the next most common place for savings. Overall savings amounts in banks tended to be higher, ranging from $59 in Hawassa Zuria to $773 in Endamehoni, whereas VESA savings amounts ranged from $26 in Ziway Dugda to $49 in Endamehoni. At baseline, 41% of respondents had taken out a loan in the past year, which increased slightly to 44.6% at endline, a marginally significant change (p=.059). The composition of these loans changed substantially between baseline and endline (Figure 11). At baseline, the most common loan source was private lenders. This shifted to VESAs at endline, while loans from private lenders had declined. The average loan amount from all sources increased 89% from baseline to endline, rising from $58.28 to $110.09. Respondents reported the largest loan sizes from MFIs, with average loans from MFIs at $227.99 and Rural Savings and Credit Cooperatives (RuSACCOs) following at $178.10. VESAs offered loans that were an average of $63.63. Access to loans for all purposes increased, though this was most pronounced for livestock and agriculture. Conclusions: GRAD households increased their savings from an average of $12 to $141. VESAs were a common place for households to save their money. GRAD households did not substantially increase their access to loans—the difference in households who had taken loans in the past year was not statistically significant—but the composition of loans did change. GRAD households were relying less on private lenders at endline, and more on VESAs, MFIs, and RuSACCOs. Average loan size has increased, especially through MFIs and RuSACCO. Recommendations: 7. Improve awareness of beneficiaries and partners to the rationale for initial small loan sizes and iv relatively short loan periods. 8. Emphasize savings as a key component of future USAID resilience projects. 9. Explore the possibility of registration of VESAs as a next step in their maturation. EQ1e. Extension services are strengthened. Key findings: The GRAD endline household survey results in Table 10 show that 52% of the surveyed households received training in crop production at some point in the year prior to the survey; this was highest in Lay Gaying (71%) and lowest in Endamehoni (29%). About one-fifth of development agents (DAs) are female, and one-quarter of community facilitators (CFs) are female. Project beneficiaries, on the other hand, ranged from 44-48% female during Years 4 and 5 (GRAD’s Indicator Performance Tracking Table (IPTT)). In FGDs, GRAD households expressed that extension trainings had changed their willingness to adopt new short season varieties, conservation practices, vegetable production, and beekeeping. Beneficiaries reported that working with VESAs had been a key part of the success of the extension trainings, as VESAs were able to raise awareness of best practices with farmers and building VESA’s VC capacity enabled farmers to better access the improved inputs necessary to take advantage of best practices learned in trainings. FGDs and KIIs report that DA capacity related to value chains is still weak, partly due to a lack of opportunity to practice the skills they learn during training. KIs both from the project and the GoE described challenges working together at the beginning. Issues included the fact that DAs had many responsibilities outside of the project’s work, and were therefore disinterested in taking on additional work that they did not initially see as connected to their mandate. The lack of per diems for government workers further exacerbated early DA disinterest in the project. However, KIs typically reported that this had improved over the course of the project. During FGDs and KIIs, GoE officials mentioned their involvement and that of field staff in substantive and meaningful ways made if more likely that activities would continue after GRAD ended. Conclusions: GRAD effectively strengthened extension services through continuous collaboration and partnership with the GoE. This relationship is key to the likelihood of the GRAD extension activity’s sustainability. The gender balance of government and project extension workers does not reflect the gender balance of GRAD beneficiaries; more men are extension workers, whereas the activity beneficiaries had a high proportion of women. Recommendations: 10. Set standards for per diem or related payments for staff who participate in donor-funded project activities; create awareness of the standard and maintain it. 11. Balance the gender of staff and community volunteers to reflect the gender balance of project participants, and advocate with GoE to achieve a similar gender balance among their extension workers. 12. Provide increased opportunity for hands-on practical learning/training in all DA and other extension workers’ capacity building efforts. 13. Develop a model of extension collaboration based on the GRAD experience and distribute the model to upcoming projects like GRAD and to others in community development for replication. GoE collaboration should be a key element of future projects. EQ1f. Development of input supplies and market linkages Key findings: Cooperatives are the most common source of agricultural inputs. In all four survey woredas, implementing partners (IPs), and farmer groups reported the importance of primary cooperatives and cooperative unions in providing the bulk of inputs and market services for VESAs, cooperatives, and VCs. Agro-dealers are also an important source of herbicide and pesticide. GRAD beneficiaries shared many success stories of improved access to agriculture inputs with the evaluation v team. Agro-dealers’ access to improved inputs meant that, in turn, farmers found it easier to purchase these inputs rather than traveling the long distance to large towns to buy them. Local markets were the largest sale outlet. Cooperatives were a comparatively low channel, though Lay Gayint was an exception. In FGDs, VC farmers generally reported selling their honey, livestock, and crops individually rather than collectively, through group negotiations with traders, processors, or primary cooperative buyers. Given the short time that they had been operating as a group, VC farmers did not yet have the capacity to expand their collective marketing; they considered this a later stage goal. Conclusions: Agro-dealers and cooperatives were the most important suppliers of inputs to GRAD households. Farmers’ Economic and Market Associations (FEMAs) were spoken of positively but were never major suppliers of inputs or channels of marketing. Some exceptional FEMAs made the transition to cooperatives. Recommendations: 14. In future USAID projects, support the transition of community groups like FEMAs to become registered as cooperatives and further help them establish linkages to local, regional, and international markets. 15. Value Chain Groups should improve their ability to negotiate as groups on matters pertaining to collection, storage, processing, and market handling services, as well as prices for their crops and livestock. EQ1g. Nutrition interventions Key findings: CFs and GoE Health Extension Workers (HEWs) conducted nutrition education in target communities, especially on breastfeeding and balanced diets for children: 15,147 households were trained on dietary diversity and 16,366 households were trained in food preparation for children,1 increasing nutritional awareness. In addition to awareness raising, GRAD and its partners conducted nutritional skills training such as cooking demonstrations and provided seeds for micro-gardens. A few DAs reported participating in the nutrition activities, but generally HEWs were the GoE focal points for this component of GRAD. In Hawassa Zuria, GRAD collaborated with Empowering New Generations to Improve Nutrition and Economic Opportunity (ENGINE) to provide nutrition training to VESA households with PLW and children under two. FGDs there mentioned the success of GRAD’s poultry activity as making more eggs available for household consumption. Quantitative results demonstrate change in nutrition practices in most of the sampled areas. Using the Food and Nutrition Technical Assistance Household Diet Diversity Score standard2 to calculate GRAD household’s diet diversity scores, overall diet diversity increased from 3.7 to 5.0 between baseline and endline. Conclusions: Generally, families’ nutritional awareness and dietary practices improved. Collaboration with health-focused projects such as ENGINE and with GoE HEWs were key to effectively implementing this series of activities. Though some DAs received nutrition-related training, for the most part GRAD worked with HEWs on nutrition practices and DAs on agriculture practices. There was a missed opportunity for better integration of agriculture and nutrition activities with GoE extension staff. Recommendations: 16. Train DAs and their supervisors in nutrition-sensitive agriculture (NSA) to build their capacity to include nutrition-related messages and activities in their everyday work. This work should be supported through the provision of nutrition education materials and funds for nutrition-related 1 http://www.care.org/sites/default/files/documents/GRAD-Brief_0.pdf 2 http://www.fantaproject.org/sites/default/files/resources/HDDS_v2_Sep06_0.pdf vi demonstrations and follow-up to VESA households (including promoting nutrient-dense vegetable production and production diversification). EQ1h. Climate change adaptation activities Key findings: Reported weather-related loss of crops was high both at baseline (86.6%) and endline (90.3%), and FGDs frequently reported the impact of weather on livelihoods. Droughts were the major shocks in all woredas and were particularly extreme in 2015–2016. A majority of VESA participants reported that their households lost their entire crop, however, GRAD beneficiaries reported that they were better able to cope with the effects of climate change because of the increased savings, access to credit, and diversified livelihoods gained through project activities. This is upheld by the quantitative data, which demonstrates increases in the percentage of households reporting using savings (10.0% at baseline to 38.4% at endline) or borrowing money (37.1% to 52.2%) to cope with decreased food availability. Though decreasing food intake declined as a coping mechanism, it was still the most common coping mechanism available to households. Despite increases in reliance on savings, income generation, and improved agricultural practices to withstand drought, some amount of crop loss was still widespread (90.3% of households), and decreasing food consumption remained a common household reaction. Conclusions: GRAD effectively improved households’ abilities to cope with climate change through improving financial safety nets. GRAD beneficiaries were also appreciative of improved inputs that have the potential to make their crops more resilient to climate impacts, though the effects of these inputs on decreasing crop loss are not yet apparent—perhaps because of the extreme drought of 2015–2016. Despite these gains, climate-related crop loss remains a major threat to household food security. Recommendation: 17. Climate-resilient agriculture and livelihoods resilience activities should remain a key focus of future USAID activities. EQ1i. Contribution of GRAD “innovations,” such as micro-franchising and the agro-dealer strategy Key findings: Micro-franchising initiatives focused on encouraging women entrepreneurs. Women took training in conducting door-to-door sales and GRAD linked these women to wholesalers to supply the goods they are selling. The agro-dealer strategy is discussed throughout this report, particularly in EQ1f. GRAD worked with agro-dealers in target areas to improve their supply of seeds, animal feed and supplements and agrochemicals so that target households could, in turn, have better access to these important agricultural inputs. Agro-dealers stated in KIIs that they found the trainings valuable in improving their ability to offer quality products to consumers. Community members spoke of the agro￾dealer intervention as improving access to crop inputs, particularly in Endamehoni. Agro-dealers were credited with being “instrumental in the supply of quality value chain commodities,” links to markets, and extension services. Conclusions: GRAD’s micro-franchising and agro-dealer innovations both appeared to be effective at increasing incomes and access to agriculture inputs. Recommendation: 18. Replicate and scale up the micro-franchise and agro-dealer approaches. EQ 2: Change in household income and assets To what extent have the assets and income of beneficiary households (considering both the total value of assets and income and the nature/composition of asset holdings and sources of income) changed over the activity’s duration? vii Key findings: Both the quantitative and qualitative data suggest that GRAD increased income at levels close to its target of an additional $1 per day per household. In addition to data on income, the evaluation team also collected data on expenditure and assets, which proxy for household economic well-being and are often more accurate than income data. Increases in expenditures are typically indicative of increase in income. The evaluation found that the expenditure rose in GRAD households from an average of $329 to $617, except in Hawassa Zuria where male-headed households had no statistically significant expenditure change and female-headed households’ expenditure decreased (Table 11). Female-headed households have on average 1.2 fewer household members (4.8 household members versus 6.0 in male-headed households). Controlling for household size decreases the expenditure gap between male- and female-headed households, but female-headed households still reported on average $77 less in annual income when controlling for household size, woreda, and survey period (Annex 11). In 2015–16, the household income from value chain activities constituted 38 percent of the total annual income in male-headed households and 31 percent in female-headed households. In focus groups, respondents associated participation in these activities with increased income from the sales they generated through value chain improvements. The survey also measured three types of assets: livestock assets, productive assets, and household assets. For all types, assets increased between baseline and endline (Table 12, counts only assets that were on both the baseline and endline survey). Conclusions: GRAD households’ income, expenditures, and assets all rose between baseline and endline. Qualitative findings illustrate that GRAD has played a role in improving households’ economic well-being through increasing access to value chains. Hawassa Zuria saw either slower improvements or declines across household economic indicators, likely due to the severe effect of drought in that woreda. Male￾and female-headed households both saw growth, though male-headed households’ expenditures grew at faster rates and maintained higher levels than female-headed households, demonstrating that female￾headed households may face special barriers that are not fully addressed by project interventions. Recommendations: 19. See Recommendation 2 (from EQ1a). 20. See Recommendation 4 (from EQ1a, EQ1b). EQ 3: Crisis modifier, benefits and resiliency To what extent have beneficiary households become resilient and benefited from the Crisis Modifier during periods of shock? Key findings: Findings elsewhere in this report have demonstrated that GRAD-targeted households have experienced increases in income and assets (EQ1b and EQ2), access to savings and credit (EQ1d), access to improved agricultural inputs (EQ1f), and diversification of coping mechanisms to deal with food shortages (EQ1h). These outcomes are in turn, according to GRAD’s theory of change, expected to enable vulnerable households to meet their food security needs in times of livelihoods shocks. Quantitative data also supports this. The average number of months where households had enough food to meet their household needs has increased on average since baseline, but not in Hawassa Zuria and Ziway Dugda, two woreda that suffered disproportionately from the drought in 2015-2016. Overall, households without any weather-related crop loss improved their months of food security from baseline to endline from 7.3 to 10.0, while households with weather-related crop loss saw no statistically significant change in crop loss. What cannot be determined from the data without a comparison group is whether or not fewer households suffered weather-related crop loss due to GRAD’s agricultural extension work, and whether those who did have crop loss suffered fewer months of food insecurity than they would have without GRAD’s resilience programming. GRAD responded to the drought conditions with a Crisis Modifier (CM) activity that addressed targeted households’ resilience through provision of emergency animal feed, viii seed for crops, and additional access to finance. FGD respondents were generally appreciative of these activities, particularly the emergency input provision. Conclusions: Many of the safety nets and coping mechanisms that typically support resilience improved under GRAD. Overall, months of food security in the past year also increased, a sign that the existence of these strategies is helping households cope with shocks. The evidence suggests that GRAD’s interventions were most effective in improving resilience in areas that experienced moderate and normal weather-related shocks, but beneficiaries were not equipped to meet the crisis-level drought that Hawassa Zuria and Ziway Dugda experienced. The emergency inputs GRAD provided seem to have played an important role in household survival during this period, when more incremental improvements in resilience the activity could reasonably have been expected to promote were insufficient to cope with the full extent of the drought. Recommendations: 21. Crisis modifiers in future programs will probably be necessary and are a flexible emergency approach the GoE ought to consider to complement donor interventions. 22. Continue to measure change over time to better measure the effect of USAID programming on resilience in less extreme circumstances. 23. See Recommendation 4 (from EQ1a, EQ1b, EQ2). EQ 4: Influence of GRAD in household graduation process What level of influence did the GRAD partners have in the beneficiary households’ graduation process? Key findings: GRAD’s partners fell into three broad groups: partners at the local and community level, IPs and the GoE. The majority of project activities relied on the partners at the local and community level, including MFIs, VESAs, and agro-dealers. FGD participants considered MFIs and VESAs to be fundamental to the ability for households to graduate, as these sources enabled households to access loans they needed to begin new income-generating activities. VESAs particularly enabled households to build savings, which they relied on when weather-related shocks decreased their income. GRAD IPs’ technical expertise in crops production and marketing and initial business-to-business linkages contributed to food-insecure households’ opportunities to enhance their livelihoods. IPs also helped GRAD form external partnerships to link communities to partners and resources that would last beyond the project such as Mekele University, private traders, and livestock markets. Overall, most KIIs spoke favorably of the collaboration between project and GoE staff, however some households perceived the graduation process as politicized. Conclusions: GRAD partners, given the constraints of the 2015–2016 drought, influenced households to increase their incomes and safety nets through improved access to savings and credit, improved income generating opportunities, and improved agricultural techniques. However, KIIs, FGDs, and IP representatives consistently argued that the graduation process was directly controlled by the GoE and therefore that the influence that a given project can have on graduation is limited. Recommendations: 24. See Recommendation 1 (from EQ1a) 25. See Recommendation 3 (from EQ1a) EQ 5: Gender equity and women’s empowerment To what extent has this activity contributed to gender equity and women’s empowerment, specifically in addressing the role of gender in decision-making on use of resources? To what extent has GRAD addressed gender gaps identified among women, men, girls, and boys? ix Key findings: GRAD targeted both men and women for engagement in project activities, as well as explicitly including training regarding women’s empowerment as part of its activities. According to project data, women made up 51% of executive leaders in VESAs, 36% of VESA members and 38% of VC groups. The household survey asked respondents about women’s involvement in fourteen common household decisions related to production and livelihoods, finances, and personal matters and housekeeping. Respondents scored how much influence the wife or main female in the household had on these decisions on a scale 1-4. For male-headed households, this score was 2.9 at baseline, which grew to 3.5 at endline (p=.000). KIIs and FGDs also frequently mentioned changes in women’s roles in decision-making, attributing the increased contribution of women to household decision-making to their increased ability to contribute financially to the household, which gave them more social capital within their families. As findings from EQ1a and EQ1b discuss, female-headed households saw less economic growth than male￾headed households. Some MFIs also reported that the majority of their borrowers were still men. FG and KII respondents indicated that GRAD considerably increased women’s access to loans and financial literacy skills through VESAs, allowing them to participate in VCs, develop income generating activities (IGAs) and earn income. KII and FGD respondents also discussed GRAD’s gender training as key to advancing gender equity and women’s empowerment through raising awareness of gender issues and encouraging dialogue among household members and community. Conclusions: GRAD's influence on gender was reported as positive. One of the most important contributions GRAD made to advancing gender equity and women’s empowerment was by creating a safe space for husbands and wives, other men and women, and the community in general to begin exploring and discussing gender issues. Some of these changes, however, go against deeply held attitudes and traditions. Although VESA savings groups have provided women with access to financial resources, women still struggle against unequal access to financial capital and resources. Female-headed households, particularly, have seen slower gains than male-headed households. Recommendations: 26. See Recommendation 2 (from EQ1a, EQ2). 27. Include a women’s economic empowerment approach, coupled with explicit gender sensitization, in future USAID projects. 28. Conduct a sustainability study of gender relationships to identify challenges to maintaining changes after the end of project support. EQ 6: Effectiveness of consortium and project collaboration How effective was collaboration/complementarities with other Feed the Future projects, as part of testing the push-pull hypothesis and the effectiveness of project management through consortium arrangement? What lessons and experiences could be drawn? Key findings: KIIs reported positively on GRAD consortium collaboration. Collaboration focused mainly on M&E follow-up tasks, provision of joint training efforts and field visits, sharing information with GoE, and sharing lessons learned and best practices. Each quarter, consortium members hold a technical working group meeting where members cross-fertilize both lessons learned and best practices adopted by presenting specific thematic areas for discussion. Key informants also pointed out some weaknesses in coordination including clearer communication of issues important to staff like per diems and benefits, delays in procurement, improving communication and understanding of roles and responsibilities among IPs, and the need for some flexibility in models and strategies. Conclusions: IPs and donors reported positive but limited examples of cross-project collaboration. The consortium proved productive, with a few areas for improvement that were not a significant cause of concern. GRAD made a strong start at leveraging collaboration both within and outside the consortium. Recommendations: x 29. Review and carry out timely procurement management procedures and clarify specific roles and responsibilities of each IP within the consortium. 30. Enhance USAID project partnerships through regular coordination and collaboration, both within USAID and between implementing partners, beginning at the design phase and continuing through implementation and evaluation. 1 INTRODUCTION The Graduation with Resilience to Achieve Sustainable Development (GRAD) was a five-year, USAID￾funded activity that ran from December 2011 to December 2016. GRAD was designed to support and enhance livelihood options of chronically food insecure households by promoting and supporting on- and off-farm income generating activities, facilitating output and input market linkages, and increasing access to microfinance services. GRAD’s activities complement Ethiopia’s Productive Safety Net Program (PSNP) to accelerate the graduation of targeted beneficiaries from PSNP. CARE served as the primary implementer of GRAD, working with a consortium of partners, including: Catholic Relief Services (CRS) with Meki Catholic Secretariat (MCS), Organization for Rehabilitation and Development in Amhara (ORDA), the Relief Society of Tigray (REST), and Agri-Service Ethiopia (working in Mareko areas). GRAD’s technical partners included the Netherlands Development Organization (SNV) and the Feinstein International Center of Tufts University. The approach followed by GRAD implementing partners (IPs) was based on a global theory of change that addresses the underlying causes of poverty and particularly women’s exclusion in agriculture. GRAD’s theory of change was founded on three pillars, namely: (1) enhancing livelihood options, (2) improving household and community resilience, and (3) strengthening enabling environment to increase GRAD’s impact and sustainability. Actually, GRAD complemented the GoE’s PSNP by supporting village-level savings/lending and market-driven agribusiness value chains (VCs) for selected commodities. The strategic objective of GRAD was to graduate at least 50,000 chronically food-insecure households from PSNP food support in 16 targeted woredas and to increase each household’s income by $365 by the activity’s fifth year in 16 woredas of Tigray, Amhara, Oromia, and SNNPR (Table 1). Table 1. GRAD operational areas (regions and woredas) Region Woreda Tigray (4) Raya Azebo, Alamata, Offla, Enda Mehoni Amhara (2) Libo Kemkem, Lay Gayint Oromia (4) Zeway Dugda, Arsi Negelle, Shala, Adami Tulu SNNP (6) Meskan, Mareko, Loka Abaya, Hawela Tula, Hawassa Zuria, Shebedino GRAD’s activities were intended to support all four core investment areas of USAID/Ethiopia’s Feed the Future strategy to achieve food security and nutrition objectives: (1) improve agricultural productivity of staple crops and livestock; (2) reduce transaction costs and increase market linkages; (3) increase the purchasing power of poor consumers by promoting alternative livelihoods; and (4) maximize efficient utilization of food. To support these objectives, GRAD employed strategies that allowed poor and chronically food insecure households to gradually assume more productive roles in value chains and income generating activities that are appropriate for their conditions. GRAD’s Activity Results Framework (Table 2) further outlines the anticipated results of GRAD activities. Table 2. GRAD Activity Results Framework Strategic Objective: Graduate 50,000 chronically food insecure households from PSNP food support in 16 targeted woredas and increase each household’s income by $365 at the fifth year of the activity’s lifetime. Result 1: Enhanced Livelihood Options of Chronically Food Insecure Households in Highland Areas IR 1.1: On- and off-farm economic opportunities, inclusive value chains, and market access for targeted households stimulated IR 1.2: An inclusive financial sector promoted and access to a range of financial products and services expanded 2 IR 1.3: Extension services upgraded Result 2: Improved Household and Community Resilience IR 2.1: Women’s resilience and access to inputs, services, and information increased IR 2.2: Nutritional status of infants, children, and reproductive-age women improved IR 2.3: Climate change adaptation improved IR 2.4: Promote aspirations for graduation among targeted PSNP households and enhance enablers of graduation Result 3: Strengthened Enabling Environment to Promote Scale-up and Sustainability IR 3.1: Collaboration among stakeholders consolidated to promote joint learning and scale up IR 3.2: Enabling environment improved Purpose Social Impact, Inc., (SI) was requested to conduct this final performance evaluation of GRAD, as a part of the services it provides to USAID/Ethiopia under the Ethiopia Project Monitoring and Evaluation Service (EPMES) activity. The main purpose of this final performance evaluation was to examine its development outcomes or results at higher and intermediate levels and the extent to which GRAD met its goals and to determine the overall effectiveness of its partnerships and any strengths and challenges during implementation. Specifically, the evaluation examined: 1. The key components of GRAD in achieving objectives and targets, including the cross-cutting gender objective 2. The effectiveness of the livelihood diversification (on- and off-farm), nutrition, climate change adaptation, and gender-related activities promoted by GRAD 3. The major types and values of assets developed by the GRAD beneficiaries and their level of access to financial services and markets 4. The strengths and weaknesses in project management (planning, implementation, and monitoring) and effectiveness in delivery of the desired services and inputs to the targeted beneficiaries and communities. Since GRAD has ended, recommendations of this evaluation look forward to similar projects and provides USAID/Ethiopia with findings, conclusions, and recommendations to inform assessments of its overall program and the design of a new Country Development Cooperation Strategy. Lessons learned may guide similar activity development and collaboration with GoE and other donors’ policies, strategies, and programs to reduce chronic malnutrition in rural Ethiopia. Evaluation Questions This evaluation addresses the following evaluation questions (EQs): EQ1. Which of the following key technical areas of GRAD exhibited sizable results and which one(s) did not? a. Graduate 50,000 chronically food-insecure households from PSNP food support in targeted woredas b. Targeted households’ income improvement to reach the $365 increase by the end of GRAD c. Participating households are organized into small groups and supported with technical value chain advice to take advantage of diversified livelihood opportunities d. Access to finance for beneficiaries is increased to fund livelihoods activities; e.g., guarantee fund scheme in facilitating access to finance e. Extension services are strengthened f. Development of input supplies and market linkages 3 g. Nutrition interventions h. Climate change adaptation and Crisis Modifier (CM) activities i. Contribution of GRAD “innovations,” such as micro-franchising and the agro-dealer strategy EQ2. To what extent have the assets and incomes of beneficiary households (considering both the total value of assets and income and the nature/composition of asset holdings and sources of income) changed over the activity’s duration? EQ3. To what extent have beneficiary households become resilient and benefited from the Crisis Modifier during periods of shock? EQ4. What level of influence did the GRAD partners have in the beneficiary households’ graduation process? EQ5. To what extent has this activity contributed to gender equity and women’s empowerment, specifically in addressing the role of gender in decision-making on use of resources? To what extent has GRAD addressed gender gaps identified among women, men, girls, and boys? EQ6. How effective was collaboration/complementarities with other Feed the Future projects, as part of testing the push-pull hypothesis and the effectiveness of project management through consortium arrangement? What lessons and experiences could be drawn? Methodology The evaluation team applied a mixed methods approach involving qualitative and quantitative data collection and analysis tools. The approach is summarized in an evaluation design matrix showing the source of data, method of data collection, and the tool being used to answer each of the evaluation questions (Annex 3). All data collected and presented in this report are disaggregated, as appropriate, by sex and woreda. Quantitative method. The endline household survey sampled beneficiaries of GRAD that the evaluation team designed to mirror the baseline survey of GRAD household beneficiaries conducted by the Feinstein International Center of Tufts University in 2012. The endline surveyed households in the same villages as the baseline and used the same data collection tool, with minor refinements to improve the flow and clarity of questions. The evaluation team piloted the refined survey in Buta Girra, Oromia Region, and the final instrument can be found in Annex 4. SART, EPMES’s data collection partner, collected the endline data from in November 2016, with oversight from the evaluation team. Enumerators used handheld electronic devices to collect data in the field. Sample size determination. The evaluation used a multi-stage cluster sampling design. GRAD implementing partners (IPs) purposively chose woredas at baseline to include one woreda per region and implementing partner. These woredas were Endamehoni in Tigray region, Lay Gayint in Amhara, Ziway Dugda in Oromia, and Hawassa Zuria in SNNPR (Figure 2). The baseline research team constructed their sample using a simple random sampling approach from household registers. At endline, the evaluation team used the baseline kebeles as clusters from which to randomly sample household respondents, using GRAD’s kebele-level beneficiary lists to identify the final sample. The baseline sample size was 1,584 households, and the endline sample size was 1,602. Table 3 below presents the breakdown of number of kebeles and respondents by woreda, as well as the change detectable at the woreda level based on the sample size. This sample size is sufficient to detect a statistically significant minimum change in proportions at the woreda level ranging from 9.2 percent to 10.6 percent, and, in the whole sample, of 5.0 percent. 3 3 Effect size calculated using a calculation for the power of two proportions, alpha=.05, power=.8, p1=.5. 4 Table 3. Number of kebeles and respondents at baseline and endline, with effect size detectable, by woreda Woreda Baseline Endline Effect Size Detectable # kebeles n # kebeles n Lay Gayint 9 385 9 384 10.0% Endamehoni 5 401 5 418 9.7% Hawassa Zuria 11 345 11 339 10.6% Ziway Dugda 18 452 18 461 9.2% Totals 43 1,584 43 1,602 5.0% The evaluation team exported the endline data into SPSS for cleaning then merged the Microsoft Excel￾based baseline data and endline data into STATA for further comparison. The team carried out frequencies analysis, categorical data analysis, means analysis, and statistical significance tests to investigate the existence of statistically significant differences. Where appropriate, the evaluation team used multivariate regressions to control for key factors that might be influencing results. Regression results cited in the report can be found in Annex 11. Figure 2. Maps of surveyed GRAD woredas Source: Resilience Assessment for the Graduation with Resilience to Achieve Sustainable Development (GRAD) Project, May 2016. Note: The November 2016 GRAD endline survey was held in Endamehoni Woreda, while the CM assessment was held in Raya Azebo (both areas highlighted in red above) in the Tigray Region. Source: Resilience Assessment for the Graduation with Resilience to Achieve Sustainable Development (GRAD) Project, May 2016. Note: The November 2016 GRAD endline survey was held in Lay Gayint Woreda (highlighted in red above) in the Amhara Region. Source: Resilience Assessment for the Graduation with Resilience to Achieve Sustainable Development (GRAD) Project, May 2016. Note: The November 2016 GRAD endline survey was held in Ziway Dugda Woreda (highlighted in red above) in the Oromia Region. Source: Resilience Assessment for the Graduation with Resilience to Achieve Sustainable Development (GRAD) Project. May 2016. Note: The November 2016 GRAD endline survey was held in Hawassa Zuria Woreda, while the CM assessment was held in Shebe Dino Woreda (both areas highlighted in red above). 5 Qualitative methods. The evaluation team conducted key informant interviews (KIIs) and focus group discussions (FGDs) in the same woredas where quantitative data was collected. The team consulted with the GRAD IPs and informed USAID about the final site selection process. The team also conducted a desk review of relevant documents, listed in Annex 6. These included GRAD proposals, monitoring and evaluation plans, baseline studies and assessments, periodic performance reports, the midterm evaluation, case studies, and other GRAD-related documents. Annex 8 includes a summary of qualitative methods applied in collecting and coding responses to the evaluation questions. The team conducted 55 KIIs at the federal and woreda level. KIIs fell into one of four categories, as outlined in Table 4 below: Table 4. Key informant interviews. KIIs by Category # USAID, GoE, and other donor officers 6 GRAD implementing partner staff 30 Regional and woreda representatives, public extension, Development Agents 6 Microfinance, cooperatives/unions, agro-dealers, and food processors 13 Total 55 As shown in Table 5 below, 7 of 55 (12.7%) KII interviewees and 128 of 314 (40.8%) FGD participants were women. Of the total of 383 members active in VESA, VC, and cooperative group activities, 39 percent were women, so the evaluation’s representation of beneficiaries by gender is in proportion to participation in project activities. Table 5. Gender participation in KIIs and FGDs and members in their organizations Category Total M F % Female KII 55 48 7 13 FGD 41 186 128 41 21-VESAs, 16-VC Groups & 5-Coop members 42 222 161 37 Totals 138 456 296 39 Source: November 2016 Endline Qualitative Interviews Results Forty-one FGDs were selected from GRAD-targeted households, and several GRAD stakeholders participated. For each KII and FGD session, as a moderator, the evaluation team member took the lead and facilitated by obtaining consent from participants, setting ground rules, asking questions, and managing group dynamics. The Evaluation Matrix, presented in Annex 3, relates data sources, data collection tools, evaluation questions, and data analysis methods. 6 EVALUATION FINDINGS Overall performance of GRAD on outcome indicators Table 6 provides an overall description of the performance of GRAD on the Activity outcome indicators. These indicators are described further under the respective evaluation questions. Table 6: GRAD performance on outcome indicators Key Indicators Results Number of GRAD Supported beneficiary graduating from PSNP 39,306 (79% of the set target of 50,000) Average Increased Household Income $771 (an increase of $353 from the baseline value $418.) Percent of men and women reporting meaningful participation of women in decision making regarding productive resources and income and increased access to productive resources4 . Production and Livelihoods Decisions (53.9% from the baseline 5.4%) Financial Decisions (56% from the baseline 7.1%) Household Decisions (60.7% from the baseline 8.7%) Average number of Income sources of vulnerable households receiving USG assistance 2.7* Average value of assets of GRAD supported households USD 53* Percent of USG supported PSNP households selling productive assets during periods of shock 50.1% from the baseline 9.7% Percent of women and men reporting increase in women's influence over household decision making5 (Measured as a score) 3.5 score from the baseline 3.2 *Baseline values could not be established from the data received by the evaluation team EQ 1: Key technical areas Which of the following key technical areas of GRAD exhibited sizable results and which one(s) have not? EQ 1a: Household graduation rate from PSNP Graduate 50,000 chronically food insecure households from PSNP food support Findings To reach its target, GRAD activities engaged 50,000 chronically food insecure households in 16 woredas, disaggregated by male- and female-headed households. Of these, 39,306 households (79% of the target) graduated. Table 7 below illustrates the activity’s cumulative progress over time toward this target. 4 “Meaningful participation” is defined as giving a score of 3 or 4 on half or more of the decisions for production/livelihoods and financial decisions 5 This indicator was measured differently—as a score—to be consistent with the way the baseline question was designed. Therefore, the figures reported here represent the change in the average score for women’s influence on household decisions between the baseline and the endline. And it was measured using gender-based decision-making attributes that were scored by respondents on a four-point scale. The results in indicator 3 provide a proxy to measuring this indicator. 7 Table 7. GRAD's cumulative graduations, target and actual GRAD Year Cumulative Graduations Target Graduated % of Target 1 0 0 – 2 0 0 – 3 10,000 11,500 115 4 27,000 21,923 81 5 50,000 39,306 79 Source: GRAD’s Indicator Performance Tracking Table (IPTT) The quantitative survey provides representative information on household participation in PSNP. Ninety￾six percent of respondents indicated that their household had participated in PSNP, and 89 percent have had at least one household member participate in a new value chains activity since joining GRAD. Of these, 53 percent have graduated, somewhat below GRAD's reported graduation rate of 39,306 households out of a targeted 65,000 who participate in direct service delivery, or 60 percent of those engaged. A breakdown of graduation rates by woreda is in Figure 3 below, and a further breakdown by woreda and gender of household head is in Figure 4. Overall, 37.2 percent of female-headed households reported graduating, versus 58.2 percent of male-headed households. Hawassa Zuria had the highest rate of graduation, 77.6 percent, compared to a mean of 52.6 percent. Hawassa Zuria also had almost no gap in the graduation rate of male-headed households versus female-headed households (the difference between the means, 77.4% and 78.2%, is not statistically significant). This graduation rate is despite 66.4 percent of respondents reporting that they had experienced a food- or income-related shock, primarily weather￾related. In Endamehoni, on the other hand, graduation rates were below the average, at only 29.4 percent. Key informants attributed this to a moratorium on graduations in that woreda since 2014. Figure 3. Graduation rate by woreda Figure 4. Graduation rate by woreda and gender of household head In Lay Gayint and Endamehoni, gender of household head related to differences in the graduation rates (the differences in the other two woredas were not statistically significant); 55.6 percent and 38.9 percent of male-headed households graduated in Lay Gayint and Endamehoni, respectively, compared to 23.9 49.7% 29.4% 77.6% 57.7% 52.6% Overall, 52.6% of households graduated from PSNP. Hawassa Zuria's rate, 77.6%, was particularly high. 55.6% 38.9% 77.4% 59.0% 58.2% 23.9% 17.4% 78.2% 52.7% 37.3% Lay Gayint Endamehoni Hawassa Zuria Ziway Dugda Total There was an overall gender divide in household graduation, but it was most pronounced in Lay Gayint and Endamehoni. Hawassa Zuria saw no gender gap. % Male-Headed HHs Graduating % Female-Headed HHs Graduating 8 percent and 17.4 percent of female-headed households. Key informants attributed these differences to women’s household productive and reproductive roles, limited access to financial services, and traditional restrictions on women’s mobility. Livelihoods shocks, including weather, crop diseases, and livestock mortality, are related to graduation within woredas, though the near universal experience (98.2% of households) of weather-related livelihoods shocks in Ziway Dugda means that the overall average does not reflect this trend (Figure 5). In Hawassa Zuria and Ziway Dugda, where 61.7 percent and 98.2 percent of households respectively reported weather-related livelihoods shocks in the quantitative survey, KIIs and FGDs also indicated that the severe drought negatively affected the graduation rate overall. Controlling for woreda and other variables, experiencing a livelihoods shock was associated with 9.5 percent fewer respondents graduating, compared to those respondents who did not experience a livelihoods shock (see regression output in Annex 11). According to project documents, some families that had graduated have “fallen back” into vulnerability because of climate change-related shocks that caused crop failure, depletion of savings, stored food, and other assets, and decreases in livestock. A Ziway Dugda official said: “Due to the recurrent drought in this woreda, most beneficiaries are back to a food insecure situation, retarding the graduation process.” Graduation is supposed to signify that households are food secure without support from the GoE. One indicator of this is the number of months in the past year that households had enough food. There is no statistically significant relationship between the number of months of food security that households experienced and graduation: graduated households experienced an average of 8.7 months of food security during the past year, while non-graduated households had 8.8 months of food security. This varied, however, by woreda: graduated households in Lay Gayint and Endamehoni experienced more months of food security than non-graduated households, while the other two woredas did not have a statistically significant difference between graduated and non-graduated households (Figure 6). 35.8% 32.9% 73.7% 99.0% 54.5% 23.0% 22.8% 66.4% 99.0% 59.8% Those who experienced livelihoods shocks in the past year were less likely to have graduated. The exception was Ziway Dugda, where shocks were almost universal. Did not graduate, experienced shock Graduated, experienced shock 8.4 10.5 8.0 7.2 8.8 9.8 11.2 8.4 7.2 8.7 Lay Gayint Endamehoni Hawassa Zuria Ziway Dugda Total In Lay Gayint and Endamehoni, graduated household experienced more months of food security. Non-graduated Graduated p=.00 p=.00 p=.15 p=.96 p=.54 No. of months of food security in the last year Figure 5. Livelihoods shocks by graduation status and woreda Figure 6. Months of food security by graduation status and woreda 9 The qualitative interviews shed further light on the dynamics between food security and graduation. While over two-thirds of the FGD participants had already graduated, most of them believed they were not ready to graduate. In Ziway Dugda, where 57.7 percent of households have graduated, FGD participants representing VESA and FEMA groups stated that in the year before participating in GRAD, they had had 4.5 months’ worth of food stored; now, at the end of GRAD, they had only 3.5 months of food stored. Members of four VESA and FEMA groups in Hawassa Zuria, on the other hand, indicated that on average they had increased their food stores by the equivalent of two additional months after participating in GRAD. FGD respondents saw food security as variable from year to year and highly related to the weather. The comment of one focus group participant illustrates the situation: “When the weather is good we have food, but when it is bad we don’t.” -VESA member in Ziway Dugda. KIIs differed in their views regarding the practicality of the benchmarks set for graduation. Some believed the flexibility and transparency to be a strength of the graduation process. Others viewed the graduation process’s emphasis on flexibility, which gives local authorities a great deal of discretion in making graduation decisions, as sometimes leading to a lack of clarity or a perception that the process was politicized. KIIs also critiqued the use of simple income and expenditure measures as benchmarks for graduation, which they felt did not adequately capture the complexities faced by GRAD-targeted households. Donors reported that the GoE has developed a new Graduation Prediction System (GPS) to address weaknesses related to the graduation process. The Food Security Coordination Directorate officials stated that the GPS will be further tested following approval by the Regions. Conclusions Though GRAD did not achieve its target of 50,000 graduated households, it made substantial progress: the 39,306 households that graduated constitute 78.6 percent of this target. Graduation is not fully within GRAD’s control, and two major external factors affected the achievement of the target. First, the severe drought in 2015 was a major livelihoods shock to households throughout the country, negatively affecting household food security and therefore the preparedness of households to graduate in the final year of the activity. GRAD’s response to this drought is discussed in EQ3. It is possible that GRAD’s intervention improved resilience in the face of the drought, though there is no data from non-GRAD areas to make this comparison. This possibility is also explored further in EQ3. Second, graduation is not a completely objective process. The GoE ultimately decides whether to graduate households and can lay a flat moratorium on graduation, as it did in Endamehoni. The benchmarks and procedures for graduation are subject to criticism and are currently under review. Households reported continued food insecurity in all woredas, though the extent varied. The two northern woredas, Endamehoni and Lay Gayint, had the highest levels of food security, as measured by months in the past year with sufficient food. They also had fewer households experiencing livelihoods shocks, though the 23.0 percent and 18.8 percent of households, respectively, that did experience a livelihoods shock in those two woredas still represents a substantial plurality. Their relatively lower graduation rate is partly due to government policy, which placed a moratorium on graduations in Endamehoni. The fact that non-graduated households in these two woredas did have fewer months of food security in the past year than graduated households implies that the food support provided through PSNP to those households was still necessary. Gender also played a strong role in graduation in these two woredas: traditional barriers to women’s livelihoods activities and the difficulty of diversifying livelihoods with fewer adults in the household appear to have limited the ability of female-headed households to graduate. In the central and southern woredas, Ziway Dugda and Hawassa Zuria, the drought severely affected household food security, and data from the survey and focus groups both indicate continued need for food support in these areas, despite the relatively high graduation rates. Recommendations 10 1. While graduation plays an important role in household access to food and therefore should continue to be tracked, it is not a strong project indicator because of the many external influences on graduation. Future USAID-funded projects should not have graduation as a key indicator of success if the decisions about graduation are outside the control of USAID's activity. It might serve well as a proxy indicator, since the USAID activity merely contributes toward graduation. 2. Gender of household heads needs to be more explicitly taken into consideration in future project design to ensure that female-headed households’ unique barriers to livelihoods opportunities are addressed. 3. In the upcoming review the GoE is planning of the graduation process and benchmarks, USAID should use lessons learned from the GRAD activity to advocate for evidence-based graduation and an appropriate, responsive graduation system. 4. Given the vulnerability of Ethiopia to weather-related shocks, future USAID-funded food security projects should consider more explicitly addressing drought resilience, or else link to and build on drought resilience projects. EQ 1b: Household income improvement Targeted households’ income improvement to reach the $365 increase at the end of the life of activity Findings On average, GRAD beneficiary households had annual income of $771 at the endline survey. Using project data, this represents an increase of $3536 since CARE’s baseline calculation of $418 in annual income. This more than doubling of household income was close to the activity’s tracking of income, which demonstrated a $367 increase over the activity’s life compared to a target of a $365 increase. The comparison of the endline survey data to CARE’s baseline has several limitations. First, because the evaluation team did not have access to the raw data or the methodology for collecting income data, we cannot assess how comparable the calculation of the indicator and the methods of collecting the data were. Secondly, because CARE’s baseline draws on the whole project area, while the endline survey sampled households from four purposively selected woredas, the endline data is not representative of the same population as the activity’s data. KIIs and FGDs support this magnitude of increase. In Endamehoni, qualitative interviews indicated that most households increased their incomes by at least $1 a day, the basis for the activity target, and attributed this gain to GRAD’s support of crop and livestock sales as well as income-generating activities (IGAs). A representative of REST in Endamehoni claimed that “GRAD beneficiaries surpassed the financial requirement benchmark set for possible graduation. Today they earn a minimum of USD $1 per day. Since their income has improved, they aspire to self-graduate, be on their own, and continue to maximize their household income.” KIIs attributed increases in household income to increased marketing of short season crops, better access to farm inputs, and commodities sales facilitated by GRAD-supported VESAs, VCs, and cooperatives. Average income varied by woreda. Endamehoni and Lay Gayint had relatively high average incomes, at $1,039 and $739, respectively (Table 8). In the Southern woredas, average income was lower: $684 in Ziway Dugda and $539 in Hawassa Zuria. These regional differences are reflected in the qualitative data as well. In Lay Gayint, a Farmer’s Cooperative Union member said, “because the union facilitates market linkages to its primary coop societies and selected value chains, the PSNP 4 and GRAD-targeted households’ income has increased drastically.” Conversely, an MFI respondent in Ziway Dugda reflected 6 Income data were captured in Ethiopian birr. Currency values are converted to USD using the exchange rate of .0448 birr to 1 dollar. This report uses the same exchange rate for both 2012 and 2016. 11 that “very few households were resilient to last year’s [2015] drought, and some were unable to repay their loans from MFIs.” Consequently, few households in that woreda saw increases in income. Table 8. Average household income by woreda, 2015–16 Source: GRAD Endline Household Survey, November 2016 Note: n = number of households surveyed In Lay Gayint and Endamehoni, graduated households also had higher incomes than non￾graduated households (Figure 7). This trend was reversed in Hawassa Zuria, while Ziway Dugda reported no income difference between graduated and non-graduated households. When controlling for woreda, graduation is associated with a $125 increase in income (Annex 11). While the average income is $771, this average includes outliers with higher incomes; the overall median income was $561, meaning that half the households surveyed had income below this value. Without the raw baseline data, it is not possible to further explore how income has changed for individuals at different relative levels of poverty. Woreda n Average income (USD) Lay Gayint 385 739 Endamehoni 418 1,083 Hawassa Zuria 339 539 Ziway Dugda 460 684 Total 1,602 771 $- $200 $400 $600 $800 $1,000 $1,200 $1,400 In Lay Gayint and Endamehoni, graduation was associated with higher income. Non-Graduated HH Income Graduated HH Income Figure 7. Income by graduation status and woreda 12 Income was associated negatively with livelihoods shocks, the vast majority of which are weather-related. Figure 8illustrates this trend broken down by woreda (in Ziway Dugda, only five households reported not experiencing livelihoods shocks, so the relationship is not statistically significant; in Endamehoni, it is marginal when not controlling for other variables). When controlling for woreda and graduation, livelihoods shocks are associated with an overall decrease of $285 in annual income (Annex 11); in this version of the analysis, only Endamehoni has an average income statistically significantly different from the other woredas. As discussed in EQ1a, Hawassa Zuria and Ziway Dugda reported much more widespread livelihoods shocks than Lay Gayint and Endamehoni. FGD participants reported that households used income to repay loans and to save, typically through their VESAs. Other uses included purchasing livestock to fatten and sell, building houses, starting merchandising shops, renting land to produce, sending children to school (one woman mentioned sending her husband to college), buying clothes and food, and taking care of family health costs. One VESA member who participated in the FGD proudly said: “I buy goats and fatten and sell them and then I buy a cow and now my family is drinking milk.” -VESA member in Ziway Dugda. Conclusions Given the limitations of the quantitative data, the evaluation team cannot give a precise estimate of the level of increase in income since baseline; however, comparisons to project data and triangulation with qualitative findings indicate that GRAD participants’ income may have increased to the level targeted by the activity. The close correspondence of these data sources gives the evaluation team a moderate degree of confidence in presenting this comparison. There is a strong indication that GRAD met or nearly met the activity target of increasing annual household income by $365. Income within woredas varied by region, with the northern woredas faring relatively better. There is no evidence that this is linked to graduation. As discussed in EQ1, these woredas also had lower graduation rates. However, graduated households in these areas had higher income than non-graduated households; we would expect the reverse, or no difference between the two groups, if the slower rate of graduation in the north was causing the income differences by woredas by allowing non-graduated households to supplement their incomes with food support. The drought appears to have had a profound influence on income. The regression analysis including graduation, woredas, and livelihoods shocks shows livelihoods shocks as being the most significant and largest single factor affecting income. Focus groups upheld this result in their frequent citations of the impact the drought had on household income. Recommendation 5. See Recommendation 4 (from EQ1a). $- $200 $400 $600 $800 $1,000 $1,200 Livelihoods shocks have a negative association with income. Household Income, No Livelihoods Shocks Household Income, Livelihoods Shocks p=.44 p=.00 p=.10 p=.00 p=.00 Figure 8. Livelihoods shocks and income, by woreda 13 EQ 1c: Technical value chain advice Participating households are organized into small groups and supported with technical value chain advice to take advantage of diversified livelihood opportunities. Findings GRAD organized households into VESAs to help poor households access financial services and accumulate savings. VESAs underwent an initial nine months of training to become member-managed groups that would not need project supervision to continue operating, with the expectation that these groups could then continue operating following project end. Training topics included savings and loans, business management skills, financial literacy, and value chains (VCs). VESA members selected which VC activities they felt were most appropriate for their communities, and GRAD tailored the VC approach to each woreda. For example, in two of the surveyed woredas, GRAD assisted beekeeping households to obtain credit from MFIs for purchases of improved hives and accessories. Figure 10 illustrates how the VESA model is the central piece of GRAD’s multi-sectoral interventions. Figure 9. Loan sources 41.1% 39.3% 27.4% 21.4% 1.9% 0.6% MFI VESA Rusacco Money Lender, Trader or Neighbor Other Bank Question 7A: Have you ever borrowed money from any of the following loan sources? MFIs and VESAs were the most common loan sources. 14 Figure 10. VESA program components In all four woredas, FGD and KII participants expressed satisfaction with the technical VC advice received from GRAD facilitators. Most VESA members attach more value to the knowledge and skills they obtained from GRAD than changes in assets or income that they may have experienced. Respondents mentioned the development of a savings culture, improved household farming techniques, and access to improved agriculture and livestock inputs. Use of VESAs for loans was relatively common. In the quantitative survey, 39.3 percent of respondents report having ever taken a loan from a VESA, and 55.0 percent of these had done so within the last year. Of the VESA loans reported, 49.1 percent were for productive purposes such as investments in livestock, small businesses, or agriculture; 82.5 percent of people who have taken a VESA loan report at least one loan as for a productive purpose. Qualitative data showed myriad success stories of individuals who had taken loans from VESAs, invested them in livelihoods activities in combination with the knowledge they gained via GRAD, and improved their income. For example, several groups reported that access to VESA or MFI loans enabled greater investment in livestock forage and thereafter increased livestock sales. FGDs also mentioned potato seed and malt barley production as successes; the latter benefitted from the communities’ proximity to local breweries. Conclusions The overall model of the VESA appears to have been quite successful. Enrollment and use of the VESAs were widespread, and reports of participants’ experience with the VESAs were positive. The training and increased knowledge were considered the most valuable part of the VESAs. Tech Training & Advice Savings & Loan Value Chains IGAs Financial Literacy Gender Self￾Monitoring Climate Change Social & Community Issues Nutrition Marketing 15 Recommendation 6. The VESA model of combining knowledge with access to credit should be continued through future programming. EQ 1d: Access to finance Access to finance for beneficiaries is increased to fund livelihoods activities. Findings MFIs provided loans to GRAD-affiliated VESAs and agro-dealers, with the intention that this would help food-insecure households engage effectively in agricultural value chains through access to capital for investing in IGAs. VESAs extended small loans to their members as an entry point for GRAD-targeted households. GRAD also facilitated access to loans from MFIs for poor households through a guarantee fund scheme, which substituted poor households’ lack of collateral. According to KIIs and FGDs, the guarantee fund scheme enabled poor households to either take their first loan or obtain increased loan amounts for relatively longer repayment periods. At baseline, 41 percent of respondents had taken out a loan in the past year, which increased slightly to 44.6 percent at endline, a marginally significant change (p=.059). The composition of these loans changed substantially between baseline and endline (Figure 11). At baseline, the most common loan source was private lenders. This shifted to VESAs at endline, which provided loans to one￾fifth of the sample, while loans from private lenders had declined. Respondents taking loans from RuSACCOs, another kind of rural savings cooperative, had also increased, from 2.4 percent to 7.4 percent of the sample. The average loan amount from all sources increased 89 percent from baseline to endline, rising from $58.28 to $110.09. Respondents reported the largest loan sizes from MFIs, with average loans from MFIs at $227.99 and RuSACCOs following at $178.10. VESAs offered loans that were an average of $63.63 (Figure 12). At baseline, the most common reported purpose for a loan was food; 17 percent of respondents had taken out a loan for food in the past twelve months. Food remained an important reason for taking loans at endline, with 19 percent of respondents taking out food loans. However, livestock became the most common reason for taking loans by endline, with 22 percent of respondents having taken a loan for that purpose. The third most common reason for taking loans at endline, at 11 percent, was for agricultural inputs. Repayment rates were generally high, according to key informants. For example, KIIs in Lay Gayint reported that more than 80 percent of the VESA loans given were collected and repaid. In areas where repayment rates were low, most of the defaulters encountered challenges like extended drought, animal diseases, market information gaps, and other social defaults that hindered them from loan repayments. Although the aim of MFIs is to supply clients with seed money to start up small business, most FGDs 10.6% 11.7% 0.5% 21.5% 2.4% 7.4% 24.6% 18.3% 4.2% 1.1% Baseline Endline VESA use grew the most since baseline, while private loans decreased. VESA Private lender MFI RuSACCO Other Q7a: Have you ever borrowed money from any of the following loan source? Q7b: When was it borrowed? Last 12 months. Figure 11. Loan sources at baseline versus endline 16 complained that the loan was not large enough to provide adequate capital for their IGAs. They suggested the MFIs should increase the size of loans in proportion to the business size and type of activities. GRAD also provided limited funds to some agro-dealers so that agro-dealers could expand their services in remote villages. The team observed agro-dealer shops in rural towns selling farm inputs, including vegetable and fruit seeds, pesticides, and feed supplements. These shops, started with seed capital through GRAD, made improved farm inputs available to rural households closer to their communities. Figure 12. Loan size by source, baseline and endline Figure 13. Access to loans by purpose, baseline and endline Household savings increased in all woredas. Households in Endamehoni saved more money than their counterparts in the other three woredas at both baseline and endline and increased their savings by a much larger value over the time period. Households in Endamehoni also had higher incomes than the other woredas, but controlling for income (as well as gender of household head and woreda) still shows that households in Endamehoni had much higher savings (Annex 11). FGDs and KIIs revealed that households in Endamehoni had more experience in savings and credit programs promoted by the Dedebit Microfinance Institution before GRAD. Table 9. Mean savings amounts reported by sample households in 2012 and 2016, in USD Baseline Endline Lay Gayint $4 $104 Endamehoni $30 $325 Hawassa Zuria $8 $42 Ziway Dugda $4 $65 Totals $12 $141 Source: GRAD Endline Household Survey, November 2016 In all woredas, VESAs were the most common platform for saving; 77 percent of respondents reporting having some savings in VESAs, compared to 22 percent with savings in banks, the next most common $- $50 $100 $150 $200 $250 Average loan size has increased, especially through MFIs and RuSACCO. Baseline Endline 0% 10% 20% 30% Food Livestock Medical Needs Businesses Agriculture Education Other Home Improvement Social Obligations Other Payments % of people taking loans for each purpose Access to loans for all purposes increased, though this was most pronounced for livestock and agriculture. Baseline Endline 17 place for savings. Overall savings amounts in banks tended to be higher, ranging from $59 in Hawassa Zuria to $773 in Endamehoni, whereas VESA savings amounts ranged from $26 in Ziway Dugda to $49 in Endamehoni. Conclusions GRAD effectively contributed to expanded access to credit and savings for households in targeted areas. VESAs and MFIs are two of the primary ways that GRAD-area households access loans, and reliance on private lenders has decreased. Growth in loans for livelihoods investments—livestock, businesses and agriculture—has grown faster than loans for non-livelihoods purposes. Savings also increased across the board, and it appears that VESAs played an important role in the increased number of respondents saving. While VESA savings amounts tended to be more modest, if we hypothesize that VESAs are catering to those with less disposable income that might not have been saving at baseline, then we would expect this relatively lower value of VESA savings. Past household experience in saving and credit and the lending capacity of MFIs and RuSACCo influenced the overall achievements. This suggests that building the foundation for saving and lending now will allow continued growth of savings and loan sizes in other areas; while loans may have been too small at the beginning of the activity to invest in IGAs, they are likely to grow over time in tandem with VESAs’ capacities to manage larger funds and individuals’ abilities to manage larger IGAs. VESAs are informal (not legally registered) entities with defined mandates, like RuSACCOs, that function in accordance with rural cooperative law. This puts them at a disadvantage when considering sustainability compared to other entities like RuSACCOs, which have a formal status and are supported by the government. Lacking this formal status, VESAs’ continued sustainability following the activity’s end is questionable. Recommendations 7. Improve sensitization of beneficiaries and partners on the rationale for initial small loan sizes and relatively short loan periods. 8. Emphasize savings as a key component of future USAID resilience projects. 9. Explore the possibility of registration of VESAs as a next step in their maturation. EQ 1e: Extension services Extension services are strengthened. Findings GRAD used a pluralistic extension services model, which engages a wide range of service providers, approaches, funding streams, and sources of information available to farmers and clients in order to service the diverse needs of beneficiaries and take advantage of diverse market opportunities available.7 GRAD’s model included Community Promoters (CPs) and Community Facilitators (CFs) who were employed by the activity, volunteer model farmers and promoters in communities, agro-dealers who would demonstrate the improved farm products they sold, and government Development Agents (DAs). FGDs and KIIs report that DA capacity related to value chains is still weak, partly owing to a lack of opportunity to practice the skills they learn during training. GRAD CFs and GoE field-level DAs work together to jointly deliver training and follow-up to beneficiaries. GoE extension staff members from the Bureau of Agriculture (BoA) sometimes, though not consistently, jointly monitored GRAD activities with GRAD staff. Key informants stated that DAs typically conducted technical production-oriented training to beneficiaries, while CFs mainly conducted training and follow-up 7 INGENAES. 2016. “Extension and Advisory Services: Terminology and Glossary.” Urbana, IL: University of Illinois. 18 related to financial matters such as VESA savings, loans, and business planning, and project monitoring. Some VESA members confirmed that they tend to work most closely with GRAD’s staff. The GRAD endline household survey results in Table 10 show that 52 percent of the surveyed households received training in crop production at some point in the year prior to the survey; this was highest in Lay Gayint (71%) and lowest in Endamehoni (29%). Livestock production training ranged from 30 percent in Hawassa Zuria to 62 percent of respondents in Ziway Dugda, for a total of 48 percent of respondents participating. Twenty-nine percent received training in financial literacy, ranging from 8 percent in Hawassa Zuria to 33 percent in Endamehoni. Table 10. Percentage of households receiving training in 2015–16 by training type Woreda n Crop Production Training (%) Livestock Production Training (%) Financial Literacy Training (%) Lay Gayint 384 71 46 30 Endamehoni 418 29 51 33 Hawassa Zuria 339 39 30 8 Ziway Dugda 461 67 62 42 Totals 1,602 52 48 29 Source: GRAD Endline Household Survey, November 2016 Note: Households may receive more than one type of training, so row percentages may total to greater than 100 percent. Model farmers provided some extension services, and their farms were reportedly used for demonstration purposes. Yet, demonstrations as an extension method were seldom discussed in FGDs and KIIs, except for several unsuccessful gardens installed at FTCs. Government DAs frequently spoke of the advantages of working with GRAD because of the ongoing relationship of trust that the activity had with target communities, particularly at the woreda and kebele levels. For example: “If we (extension workers) go by ourselves to visit farmers and we try to introduce new practices, they are often not accepted by farmers because they don’t have the inputs to adopt the new practices. If we go with GRAD, then GRAD facilitates farmers getting the inputs they need, and extension can do the technical training.” -Key informant, Hawassa Zuria Government KIIs also frequently spoke appreciatively of GRAD’s communication with them regarding their training plans, reporting on activities completed, and consultation prior to beginning work with communities. During FGDs and KIIs, GoE officials mentioned their involvement and that of field staff in substantive and meaningful ways made it more likely that activities would continue after GRAD ended. A Lay Gayint Woreda Office of Agriculture representative advised that his administration had already formed a committee and designed a strategy to sustain the VESAs’ operation after GRAD’s termination. He advised that “our office will take up the continuation of extension service provision.” Key informants did raise issues with collaboration at the activity during interviews, particularly in Hawassa Zuria. KIs both from the activity and the GoE described challenges working together at the beginning, especially due to a lack of common understanding between DAs and CFs in the beginning, which fostered disagreements. Issues included the fact that DAs had many responsibilities outside of the activity’s work and were therefore uninterested in taking on additional work that they did not initially see as connected to their mandate. The lack of per diems for government workers further exacerbated early DA disinterest in the activity. However, KIs typically reported that this had improved over the course of the activity. 19 About one-fifth of DA staff are female, and one-quarter of CFs are female. Project beneficiaries, on the other hand, ranged from 44–48 percent female during Years 4 and 5 (GRAD IPTT). This conflicts with existing literature that suggests that women farmers typically prefer to work with female extension agents,8 though this concern was not explicitly raised during fieldwork. In FGDs, GRAD households expressed that extension trainings had changed their willingness to adopt new short season varieties, conservation practices, vegetable production, and beekeeping. Beneficiaries reported that working with VESAs had been a key part of the success of the extension trainings, as VESAs were able to raise awareness of best practices with farmers, and building VESAs’ VC capacities enabled farmers to better access the improved inputs necessary to take advantage of best practices learned in trainings. Conclusions GRAD strengthened the pluralistic extension system in target areas to respond to VESA and FEMA needs. Improving producers’ access to such services was part of the “push” strategy in GRAD’s causal model and has resulted in increased willingness of GRAD beneficiaries to try new agricultural practices. GRAD kept the GoE transparently updated on its activities and involved in meaningful ways as trainees, as trainers of GRAD beneficiaries, and in joint monitoring and evaluation (M&E) activities. This has resulted in a high level of trust between the GoE and GRAD and strengthened technical capacity of GoE staff, which in turn resulted in GoE extension workers’ ability to deliver higher-quality extension services to GRAD and non-GRAD beneficiaries. Challenges faced in the activity’s relationship with the GoE mainly took place early in implementation and were overcome. The gender balance both of GoE extension workers and GRAD staff was not optimal for working with the high level of female beneficiaries that the activity targeted. Recommendations 10. Set standards for per diem or related payments for staff who participate in donor-funded project activities; create awareness of the standard and maintain it. 11. Balance the gender of staff and community volunteers to reflect the gender balance of project participants, and advocate with GoE to achieve a similar gender balance among their extension workers. 12. Provide increased opportunity for hands-on practical learning/training in all DA and other extension workers’ capacity-building efforts. 13. Develop a model of extension collaboration based on the GRAD experience and distribute the model to upcoming projects like GRAD and to others in community development for replication. GoE collaboration should be a key element of future projects. EQ 1f: Input supplies and market linkages Development of input supplies and market linkages Findings GRAD beneficiaries shared many success stories of improved access to agriculture inputs with the evaluation team. Agro-dealers’ access to improved inputs meant that, in turn, farmers found it easier to purchase these inputs rather than traveling the long distance to large towns to buy them. An internal GRAD study found that agro-dealers had served 30,000 local farmers by October 2016. 9 IPs and agro￾dealers reported that agro-dealers’ margins are fixed through an agreement reached with GRAD, keeping 8 FAO. 2011. “Gender differences in assets.” ESA Working Paper No. 11-12, March 2011. Rome: FAO. 9 http://www.snv.org/update/agro-dealership-market-based-solution-improve-access-agricultural-inputs 20 prices reasonable, and that agro-dealers’ affiliation with GRAD also provides some quality assurance: buyers are encouraged to return seeds that do not germinate. Agro-dealers have established linkages with wholesalers so that they can continue to source inputs after the end of GRAD. In all four survey woredas, IPs and farmer groups reported the importance of primary cooperatives and cooperative unions in providing the bulk of inputs and market services for VESAs, cooperatives, and VCs. Figure 14 shows that cooperatives were the respondents’ primary source for all major agriculture inputs and were by far the most important source for fertilizer and improved seeds. It is not clear, however, to what extent these cooperatives represent preexisting or independent bodies, versus project FEMAs that have converted to cooperatives. Agro￾dealers also were a significant source of herbicides and pesticides. FEMAs represented 2.2 percent to 5.7 percent of input purchases. Beekeeping had a few special constraints, including a twice yearly (October and May) marketing schedule. Beekeepers also expressed the need for higher quality bee protection clothing and accessories. Mekele has better, more modern supplies than what is available locally in Endamehoni, where smokers are not as good, bee brushes could be better, and there is insufficient information on colony management. To develop market linkages, GRAD selected, trained, and supported private wholesale traders that provide marketing services to VESAs and FEMAs. GRAD then promoted off-farm market opportunities at community meetings and in mini-workshops on market days and by introducing farmer groups to potential local and regional traders, cooperatives, and other individual buyers. For example, GRAD assisted wholesale buyers in developing linkages with export abattoirs/slaughterhouses in Mojo. As a result, VESAs in Hawassa Zuria and its surrounding SNNPR woredas regularly sell fattened shoats to a wholesaler who sells them as meat exporters. GRAD supported the formation of FEMAs and linked selected ones with cooperatives or unions, some selling vegetables to international markets, like the Hawassa Zuria Potato VC FEMA and, in Ziway Dugda, a fattened shoats collection center. The Oromia Cooperative Agency and GRAD were working together to transform FEMAs from transitional status to registered cooperative, but it was not clear whether this agreement applied to all regions. It is not clear whether the GRAD-initiated cooperatives will continue to exist alongside the cooperatives functioning under the GoE system when GRAD support is no longer available. The cases seem to be exceptions. Figure 15 demonstrates mixed use of cooperatives: in Lay Gayint, cooperatives were the second-most common sale outlet, with 56.8 percent of households reporting using them. In the other three woredas, however, sales through cooperatives were reported by fewer than 10 percent of households. In all areas, local markets were the most common sale outlet: 83.7 percent of households reported selling products through local markets. 0% 20% 40% 60% 80% 100% Fertilizer Improved Seeds Herbicide Pesticide Cooperatives are the most common source of agricultural inputs. Agro￾dealers are also an important source of herbicide and pesticide. FEMA Agro-dealers Cooperatives Other Q6e: From which supplier did you purchase each of the following inputs? Figure 14. Suppliers of agricultural inputs 21 Figure 15. Sale outlet use by woreda In the focus groups, as well, VC farmers generally reported selling their honey, livestock, and crops individually rather than collectively, through group negotiations with traders, processors, or primary cooperative buyers. Given the short time that they had been operating as a group, VC farmers did not yet at this stage have the capacity to expand their collective marketing. They considered this a later stage goal. Conclusions Agro-dealers were successfully providing inputs to farmers in target communities, particularly herbicides and pesticides. They appear to have been making it easier for farmers to access improved inputs. GRAD created an enabling environment for this success, for instance by requiring agro-dealers to limit their margins on certain inputs, that is not guaranteed to persist past the activity’s end. Agro-dealers did not, however, surpass the importance of cooperatives in providing farmers with improved seeds and fertilizer. While there were examples of FEMAs successfully becoming cooperatives and promoting off-farm value addition and marketing, use of FEMAs to purchase inputs and sell products was uncommon. VC members lack the capacity to negotiate buy-sell agreements, sell in bulk, and engage in value-added activities such as processing and packaging their products. Recommendations 14. In future USAID projects, support the transition of community groups like FEMAs to become registered as cooperatives and further help them establish linkages to local, regional, and international markets. 15. Value Chain groups should improve their abilities to negotiate as groups on matters pertaining to collection, storage, processing, and market handling services, as well as prices for their crops and livestock. EQ 1g: Nutrition Nutrition interventions Findings CFs and GoE Health Extension Workers (HEWs) conducted nutrition education in target communities, especially on breastfeeding and balanced diets for children. 15,147 households were trained on dietary 0% 20% 40% 60% 80% 100% Lay Gayint Endamehoni Hawassa Zuria Ziway Dugda Total FEMA Cooperative Private Aggregator Agro-dealers Local market 22 diversity and 16,366 households were trained in food preparation for children,10 increasing nutritional awareness. In addition to awareness raising, GRAD and its partners conducted nutritional skills training such as cooking demonstrations and provided seeds for micro-gardens. A few DAs reported participating in the nutrition activities, but generally HEWs were the GoE focal points for this component of GRAD. Nutrition trainees remembered and discussed the specifics of the best practices learned in trainings during FGDs. Participants stated that GRAD households’ dietary awareness and training helped them balance daily nutrition and that GRAD helped them start micro kitchen gardens to supplement their household consumption with additional vegetables. FGD participants reported that households consumed the vegetables from these gardens and sold the surplus at the local market. In Hawassa Zuria, GRAD collaborated with Empowering New Generations to Improve Nutrition and Economic Opportunity (ENGINE) to provide nutrition training to VESA households with pregnant women and children under two. FGDs in Hawassa Zuria mentioned the success of GRAD’s poultry activity in making more eggs available for household consumption. This intervention increased the supply of eggs and addressed local taboos against egg consumption while pregnant through nutrition education. A volunteer community promoter said: “GRAD has given training on nutrition twice, which changed the attitude of people towards nutrition. . . . now all pregnant women eat eggs, so people have started poultry production to make eggs available to eat, and also as an income source.” -KII with a community promoter in Hawassa Zuria Quantitative results demonstrate changes in nutrition practices in most of the sampled areas. Using the Food and Nutrition Technical Assistance Household Diet Diversity Score standard11 to calculate GRAD households’ diet diversity scores, overall diet diversity increased from 3.7 to 5.0 between baseline and endline. The exception was in Hawassa Zuria, where diet diversity declined slightly, from 4.0 to 3.8. This decline was not statistically significant (p=.22). Controlling for survey period (baseline or endline) and woreda, livelihoods shocks are correlated with a .2 decline in diet diversity (see Annex 11). Conclusions Generally, families’ nutritional awareness and dietary practices improved. Collaboration with health￾focused projects such as ENGINE and with GoE HEWs were key to effectively implementing this series of activities. Hawassa Zuria did not see the same improvements in dietary outcomes in the quantitative survey, likely in part because of the toll weather-related shocks took on household consumption during the endline year. However, qualitative findings demonstrate changes in dietary practices in Hawassa Zuria that are similar to other woredas, so overall the nutrition interventions appear to have resulted in behavioral change across all target areas. 10 http://www.care.org/sites/default/files/documents/GRAD-Brief_0.pdf 11 http://www.fantaproject.org/sites/default/files/resources/HDDS_v2_Sep06_0.pdf Figure 16. Diet diversity scores, baseline versus endline 0.0 1.0 2.0 3.0 4.0 5.0 6.0 Baseline Endline Diet Diversity Score Diet diversity increased everywhere except Hawassa Zuria. Hawassa Zuria Total 23 Though some DAs received nutrition-related training, for the most part GRAD worked with HEWs on nutrition practices and DAs on agriculture practices. There was a missed opportunity for better integration of agriculture and nutrition activities with GoE extension staff. Recommendations 16. Train DAs and their supervisors in nutrition-sensitive agriculture (NSA) to build their capacity to include nutrition-related messages and activities in their everyday work. This work should be supported through the provision of nutrition education materials and funds for nutrition-related demonstrations and follow-up to VESA households (including promoting nutrient-dense vegetable production and production diversification). EQ 1h: Climate change adaptation Climate Change Adaptation Activities Findings Reported weather-related loss of crops was high both at baseline (86.6%) and endline (90.3%), and FGDs frequently reported the impact of weather on their livelihoods. Droughts were the major shocks in all woredas and were particularly extreme in 2015–2016. A majority of the participants in VESAs reported that their households lost their entire crop. GRAD responded to this particularly extreme shock with the Crisis Modifier (CM) intervention, discussed further under EQ3 below. GRAD beneficiaries reported that they were better able to cope with the effects of climate change because of project activities. These took two main forms. First, FGD respondents reported, particularly in the north, that improved savings through VESAs and access to credit helped them withstand livelihoods shocks by giving them funds necessary to buy food when their crops failed. This is upheld by the quantitative data, which demonstrates increases in the percent of households reporting using savings (10.0% at baseline to 38.4% at endline) or borrowing money (37.1% to 52.2%) in order to cope with decreased food availability (Figure 17). FGD participants also reported increasing their income generation to deal with shocks, which also appeared in the quantitative data as a common coping mechanism. More people reported engaging in supplementary labor (32.7% to 42.6%) or selling firewood (2.5% to 18.0%) to earn additional income to deal with food shortages. FGD respondents also reported that the improved agricultural inputs and techniques from GRAD helped them cope with weather changes. For example, respondents in the north reported that short-season variety crops helped them cultivate crops that were better adapted to drought. One VESA member said: “After the crop failure last year, GRAD helped us with wheat and maize seed, and we were able to save the lives of our children.” 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Baseline Endline More households used improved access to finances and increased income generation to cope with food shortages. Though decreasing food intake declined as a coping mechanism, it was still the most common coping mechanism available to households. Reduced meals Smaller portions Sold livestock Borrowed money Used savings Engaged in labor Sold firewood Figure 17. Food shortage coping mechanisms, baseline versus endline 24 FGDs also appreciatively mentioned soil conservation activities like contouring, planting trees, learning small-scale conservation practices, and new types of insecticide that worked better in the current climate conditions. Despite increases in reliance on savings, income generation, and improved agricultural practices to withstand drought, some amount of crop loss was still widespread (90.3% of households), and decreasing food consumption remained a common household reaction. Households reducing the number of meals consumed decreased from 81.3 percent to 66.5 percent, and households eating less at each meal decreased from 82.8 percent to 66.7 percent. Conclusions GRAD effectively improved households’ abilities to cope with climate change through improving financial safety nets. GRAD beneficiaries were also appreciative of improved inputs that have the potential to make their crops more resilient to climate impacts, though the effects of these inputs on decreasing crop loss are not yet apparent—perhaps because of the extreme drought of 2015–2016. Despite these gains, climate-related crop loss remains a major threat to household food security. Recommendation 17. Climate-resilient agriculture and livelihoods resilience activities should remain a key focus of future USAID activities. EQ 1i: Innovations Contribution of GRAD “innovations,” such as micro-franchising and the agro-dealer strategy Findings Micro-franchising initiatives focused on encouraging women entrepreneurs. Women took training in conducting door-to-door sales, and GRAD linked these women to wholesalers to supply the goods they are selling. In Hawassa Zuria and Ziway Dugda FGDs, women reported that this IGA was effectively increasing their incomes. Others in communities voiced appreciation for this model, saying that it offered them easier access to goods that they want to buy and that the goods women were selling were affordable. The agro-dealer strategy is discussed throughout this report, particularly in EQ1f. GRAD worked with agro-dealers in target areas to improve their supply of seeds, animal feed, and supplements and agrochemicals so that target households could, in turn, have better access to these important agricultural inputs. In each of the GRAD activity woredas, two agro-dealers were identified and trained in agribusiness entrepreneurship, commodity business skills, record keeping, customer handling, and financial management. Agro-dealers stated in KIIs that they found the trainings valuable in improving their ability to offer quality products to consumers. Community members spoke of the agro-dealer intervention as improving access to crop inputs, particularly in Endamehoni. The quantitative results presented in EQ1f show that agro-dealers were particularly important suppliers of herbicides and pesticides. Agro-dealers were credited with being instrumental in the supply of quality value chain commodities, links to markets, and extension services. In addition, key informants referenced the success of some of the new technologies GRAD introduced. Rope and washer pumps for irrigating crops are effective and affordable. Improved beehives, made using local materials, and bee flora, distributed to beneficiaries, have added income from the sale of honey, a product also well-received in the home. Conclusions GRAD’s two main innovations, micro-franchising and agro-dealers, both proved successful in improving household livelihoods. 25 Recommendation 18. Replicate and scale up the micro-franchise and agro-dealer approaches. EQ 2: Change in household income and assets To what extent have the assets and income of beneficiary households changed over the activity’s duration? Findings The findings under EQ1b discuss changes in incomes for GRAD beneficiaries in detail. Both the quantitative and qualitative data suggest that GRAD increased income at levels close to its target of an additional $1 per day per household, though the evaluation team had access to the baseline data only through GRAD’s Indicator Performance Tracking Table (IPTT) and therefore cannot attest to the data’s quality. In addition to data on income, the evaluation team also collected data on expenditures and assets, which proxy for household economic well-being and are often more accurate than income data. Increases in expenditures are typically indicative of increases in income, under the assumption that income is the primary constraint on household spending (this may not be accurate if access to loans is also changing). The evaluation found that expenditures rose for GRAD households from an average of $329 to $617, except in Hawassa Zuria, where male-headed households had no statistically significant expenditure change and female-headed household expenditures decreased (Table 11). This closely reflects the change in income discussed in EQ1b. Female-headed households have on average 1.2 fewer household members (4.8 household members versus 6.0 in male-headed households). Controlling for household size, which directly impacts the amount of food a household buys and other daily expenditures, decreases the expenditure gap between male- and female-headed households, but female-headed households still reported on average $77 less in annual income when controlling for household size, woreda, and survey period (Annex 11). Table 11. Mean household expenditures by gender and woreda, 2012 and 2016 Woreda Male-headed households % Female-headed households % 2012 2016 Change 2012 2016 Change Endamehoni $340 $841 147% $308 $615 100% Lay Gayint $216 $712 230% $167 $457 173% Ziway Dugda $381 $617 62% $298 $477 60% Hawassa Zuria $497 $500 1% $403 $271 -33% Totals (mean) $352 $661 88% $292 $495 69% Source: GRAD Endline Household Survey, November 2016 In 2015–16, the household income from value chain activities constituted 38 percent of the total annual income in male-headed households and 31 percent in female-headed households. Farmers were most commonly engaged in sheep fattening (64 percent), malt barley production (58 percent), and improved potato seed production (48 percent). In focus groups, respondents associated participation in these activities with increased income from the sales they generated through value chain improvements. Household asset counts are another common proxy for household economic well-being, as assets both demonstrate the wealth a household has accumulated and can be sold in times of need. The survey measured three types of assets: livestock assets, productive assets, and household assets. For all types, assets increased between baseline and endline (Table 12 counts only assets that were on both the baseline and endline survey). There was some variation by woreda, with assets in Hawassa Zuria generally increasing least (Figure 19). 26 Table 12. Asset counts at baseline and endline Baseline Endline # Livestock Assets 5.0 9.3 # Productive Assets 7.5 9.0 # Household Assets 3.3 6.5 Figure 18. Asset counts at baseline and endline, by woreda Consistent with the overall increase in ownership of assets among GRAD households, evaluation findings also show an increase in the percentage of households that sold an asset last year—from 9.7% at baseline to 50.1% at endline (see Table 6). The increased sale of assets could have been influenced by the fact that GRAD households had marketable assets (shown in Table 12 and Figure 19), which they could rely on during the severe drought period. From this perspective, the increased sale of assets by GRAD households is a positive outcome. Conclusions GRAD households’ incomes, expenditures, and assets all increased between baseline and endline. Though the baseline data available for household income were not available to the evaluation team in raw form for analysis and quality assurance, expenditures rose in similar proportions to what the baseline and endline data suggest. Though there is no counterfactual through which to attribute these gains specifically to the activity, qualitative findings illustrate that GRAD has played a role in improving households’ economic well-being through increasing access to value chains. Hawassa Zuria saw slower improvements or declines in household economic indicators, likely because of the severe effect of drought in that woreda. Male- and female-headed households both saw growth, though male-headed households’ expenditures grew at faster rates and maintained higher levels than female-headed households, demonstrating that female-headed households may face special barriers that are not fully addressed by project interventions; these are discussed in section 1b. The growth in assets enabled households to cope with the hardships of climate change as demonstrated by the increase in the percentage of households selling an asset. Recommendations 19. See Recommendation 2 (EQ1a). 20. See Recommendation 4 (EQ1a, EQ1b). Livestock Assets Productive Assets Household Assets 0 2 4 6 8 10 12 14 Baseline Endline Baseline Endline Baseline Endline Baseline Endline Lay Gayint Endamehoni Hawassa Zuria Ziway Dugda # assets While assets grew overall, households in Hawassa Zuria had declining numbers of assets. 27 EQ 3: Crisis modifier, benefits and resilience To what extent have beneficiary households become resilient and benefited from the Crisis Modifier during periods of shock? Findings USAID defines resilience as “the ability of people, households, communities, countries and systems to mitigate, adapt to, and recover from shocks and stresses in a manner that reduces chronic vulnerability and facilitates inclusive growth.”12 GRAD’s resilience programming includes women’s empowerment, access to credit, climate change adaptation, nutrition support, and modernization of agricultural extension. Findings elsewhere in this report have demonstrated that GRAD-targeted households have experienced increases in incomes and assets (EQ1b and EQ2), access to savings and credit (EQ1d), access to improved agricultural inputs (EQ1f), and diversification of coping mechanisms to deal with food shortages (EQ1h). These outcomes are, in turn, according to GRAD’s theory of change, expected to enable vulnerable households to meet their food security needs in times of livelihoods shocks. As the table below demonstrates, the average number of months where households have had enough food to meet their household needs has increased on average since baseline, but not in all woredas. Hawassa Zuria saw no statistically significant change in months of food security, and Ziway Dugda households reported a decrease in the number of months of food security. Table 13. Months of food security by woreda, baseline and endline # of months of food security in the past year Lay Gayint Endamehoni Hawassa Zuria Ziway Dugda Total Baseline 7.7 6.9 8.1 7.6 7.6 Endline 9.1 10.7 8.3 7.2 8.8 p 0.000 0.000 0.187 0.006 0.000 As discussed throughout this report, Hawassa Zuria and Ziway Dugda also suffered disproportionately from the drought in 2015–2016. As the table below demonstrates, households without any weather￾related crop loss improved their months of food security from baseline to endline from 7.3 to 10.0, while households with weather-related crop loss saw no statistically significant change in crop loss. Table 14. Food security and weather-related crop loss, baseline and endline # of months of food security in the past year Weather-related crop loss No Weather-related crop loss Baseline 7.6 7.3 Endline 7.7 10.0 p 0.652 0.000 What cannot be determined from the data without a comparison group is whether fewer households suffered weather-related crop loss because of GRAD’s agricultural extension work and whether those who did have crop loss suffered fewer months of food insecurity than they would have without GRAD’s resilience programming. Fewer households were reporting any weather-related crop loss at endline than were at baseline, despite the severe drought from 2015–2016; Ziway Dugda was an exception (Table 15). Controlling for weather-related crop loss and woreda, participating in an agriculture value chains activity had no statistically significant relationship with months of food security (Annex 11). 12 https://www.usaid.gov/resilience. Also, see a similar definition by the Department for International Development (DFID), 2011, “Defining Disaster Resilience: A DFID Approach Paper,” London. 28 Table 15. Percent of households with weather-related crop loss by woreda, baseline and endline % households with weather-related crop loss Lay Gayint Endamehoni Hawassa Zuria Ziway Dugda Total Baseline 95.8% 61.3% 90.4% 98.5% 86.7% Endline 18.8% 23.0% 61.6% 98.3% 51.8% p 0.000 0.000 0.000 0.825 0.000 In focus groups, GRAD beneficiaries in Lay Gayint reported improved household resilience due to increased savings, access to credit, and income-generating mechanisms. GRAD-targeted households said that they can now feed themselves without seeking extra help for at least a year through the money they have been able to save in VESAs. However, FGDs in Hawassa Zuria and Ziway Dugda reported that the shocks from the drought were too severe for the coping mechanisms available, though they also frequently mentioned the importance of having savings for mitigating shocks. GRAD responded to the drought conditions with a CM activity that addressed targeted households’ resilience through provision of emergency animal feed, seed for crops, and additional access to finance. Longer-term efforts include water conservation/floodwater capture, further diversification of livelihood streams, and enhanced disaster risk management. FGD respondents were generally appreciative of these activities, particularly the emergency input provision. Comments included: “No rain came because of climate change, we were very frustrated, but then we received seed support from GRAD, including haricot bean, maize seed, sweet potato seed, and potato seed, and we survived through that.” “Because there was no rain, we could not produce anything, but then we got the seed and we survived.” Household members received vouchers for these seeds from CARE, which they then procured themselves from agro-dealers. For women, the vouchers had their own name on them rather than their husbands’ names, which they found empowering. They reported that they then had a “good harvest” that enabled them to feed their children. Conclusions As discussed under the other evaluation questions, many of the safety nets and coping mechanisms that typically support resilience improved under GRAD: households had more income and assets (EQ1b and EQ2), better access to credit and more savings (EQ1d), more robust agricultural inputs and practices (EQ1f), and reported using these as coping mechanisms to provide food in times of need (EQ1h). Overall, months of food security in the past year also increased, a sign that the existence of these strategies is helping households cope with shocks. However, there is not much evidence that GRAD’s programming specifically improved food security in areas that were affected by the drought. Because the drought that occurred from 2015–2016 was much more severe than weather-related shocks that were likely to have occurred during the baseline period, seeing no change in food security for households with weather￾related crop loss between baseline and endline does not necessarily mean that GRAD had no effect on these households. The evidence suggests that GRAD’s interventions were most effective in improving resilience in areas that experienced moderate and normal weather-related shocks but could not meet the crisis-level drought that Hawassa Zuria and Ziway Dugda experienced. The emergency inputs GRAD provided seem to have played an important role in household survival during this period, when more incremental improvements in resilience the activity could reasonably have promoted would have been insufficient to cope with the full extent of the drought. 29 Recommendations 21. Crisis modifiers in future programs will likely be necessary and are a flexible emergency approach the GoE ought to consider to complement donor interventions. 22. Continue to measure change over time to better measure the effect of USAID programming on resilience in less extreme circumstances. 23. See Recommendation 4 (from EQ1a, EQ1b, EQ2) EQ 4: Influence of GRAD in household graduation process What level of influence did GRAD partners hold? Especially, in the beneficiary households’ graduation process? Findings GRAD’s partners fell into three broad groups: partners at the local and community levels, implementing partners, and the GoE. Most project activities relied on the partners at the local and community level, including MFIs, VESAs, and agro-dealers. FGD participants considered MFIs and VESAs to be fundamental to the ability of households to graduate, as these sources enabled households to access loans they needed to begin new income￾generating activities. VESAs particularly enabled households to build savings, which they relied on when weather-related shocks decreased their incomes. GRAD IPs’ technical expertise in crops production and marketing and initial business-to-business linkages contributed to food-insecure households’ opportunities to enhance their livelihoods. VC activities promoted by IPs included beekeeping and honey, livestock, pulses, red peppers, onions, potatoes, and malt barley. IPs also helped GRAD form external partnerships to link communities to partners and resources that would last beyond the activity such as Mekele University, private traders, and livestock markets. As discussed in EQ1e, GRAD also worked closely with GoE extension staff to reach out to target communities. KIIs spoke favorably of the collaboration between the activity and GoE staff. However, as discussed in EQ1a, the GoE was the ultimate arbitrator of graduation. In Endamehoni, the government placed a moratorium on graduation. Some households perceived the graduation process as politicized. In other areas, the GoE had a quota system for graduation that sometimes superseded graduation benchmark criteria. Conclusions GRAD partners, given the constraints of the 2015–2016 drought, influenced households to increase their incomes and safety nets through improved access to savings and credit, improved income generating opportunities, and improved agricultural techniques. However, KIIs, FGDs, and IPs’ representatives consistently argued that the graduation process was directly controlled by the GoE and therefore that the influence that a given project can have on graduation is limited. Recommendations 24. See Recommendation 1 (from EQ1a). 25. See Recommendation 3 (from EQ1a). 30 EQ 5: Gender equity and women’s empowerment To what extent has GRAD activity contributed to gender equity and women’s empowerment, specifically in addressing the role of gender in decision-making on use of resources? To what extent has GRAD addressed gender gaps identified among women, men, girls, and boys? Findings GRAD targeted both men and women for engagement in project activities and explicitly included training regarding women’s empowerment as part of its activities. According to project data, women made up 51 percent of executive leaders in VESAs, 36 percent of VESA members, and 38 percent of VC groups. FG and KII respondents indicated that GRAD considerably increased women’s access to loans and financial literacy skills through VESAs, allowing them to participate in VCs, develop IGAs, and earn income. Earning income allowed them to contribute to their households and provided them with some economic independence. Women also reported that contributing financially to community social funds, a part of VESA social funds for those in need, helped build their social capital and confidence. The household survey asked respondents about women’s involvement in fourteen common household decisions related to production and livelihoods, finances, and personal matters and housekeeping. Respondents scored how much influence the wife or main female in the household had on these decisions on a scale from 1 to 4. For male-headed households, this score was 2.9 at baseline, which grew to 3.5 at endline (p=.000). As shown in Table 6, the perception that women meaningfully participate in decision￾making was virtually universal—96.4%, among women, and 94.8% among men. Women’s participation in all parameters measured increased since the baseline period. Respondents reported women’s participation (see Table 6) in key decisions as follows: • In production and livelihoods decisions—increased from 5.4% to 53.9%n • In financial decisions—increased from 7.1% to 56.0% • In household decisions—increased from 8.7% to 60.7%. KIIs and FGDs also frequently mentioned changes in women’s roles in decision-making, attributing the increased contribution of women to household decision-making to their increased ability to contribute financially to the household, which gave them more social capital within their families. As findings from EQ1a and EQ1b discuss, female-headed households saw less economic growth than male￾headed households. Some MFIs also reported that most their borrowers were still men. KII and FG respondents also discussed GRAD’s gender training as key to advancing gender equity and women’s empowerment through raising awareness of gender issues and encouraging dialogue among household members and the community. This finding confirms similar findings reported in the GRAD Gender Outcome Mapping report, which indicates that women identify training as the most important factor enabling women’s empowerment. 13 One FGD participant said: “There is change in my husband's attitude and acceptance of gender equality after attending gender awareness creation conducted by GRAD. As a result, he allowed me to participate in the project activities, working together, and consults in decision-making on household resources” -Ziway Dugda VESA member FGDs and KIIs gave many examples indicating that men and women, and boys and girls, shared more of the household roles traditionally seen as “women’s work.” A man commented, “because of GRAD 13 Conrad, et al. 2016. “Grad Gender Outcome Mapping Report.” CARE Ethiopia. 31 training, if my wife goes to market, I will prepare things like the firewood for when she returns.” However, these gains were not universal; there is still community disapproval in some rural communities of men who take on tasks traditionally prescribed for women. Participants in several FGDs discussed how their changed attitudes about gender roles and emphasized the importance of sending both boys and girls to school to them: “We now send both our boys and girls equally to school; in the past we kept them home to watch cattle and do household chores.” And a man said: “Our children know better than we do about equality between girls and boys.” At the woreda level, GRAD has worked with the Bureaus of Women and Children’s Affairs, providing them with training and other capacity development activities. The two institutions were collaborating to enhance transformative change pertaining to women’s leadership and empowerment, girls’ education and gender equality, and improving poor nutritional status of women. Conclusions GRAD's influence on gender was reported as positive. GRAD built a solid foundation to maintain and support changes through women’s economic empowerment. One of the most important contributions GRAD made to advancing gender equity and women’s empowerment was by creating a safe space for husbands and wives, other men and women, and the community in general to begin exploring and discussing gender issues. This, coupled with gender training, created a situation conducive to change in gender relationships. Some of these changes, however, go against deeply held attitudes and traditions. Time will tell whether changes are sustainable after GRAD support ends, as many families struggle to survive with few assets. Although VESA savings groups offer women access to financial resources, women still struggle against unequal access to financial capital and resources. Female-headed households, particularly, have seen slower gains than have male-headed households. Recommendations 26. See Recommendation 2 (from EQ1a, EQ2) 27. Include a women’s economic empowerment approach, coupled with explicit gender sensitization, in future USAID projects. 28. Conduct a sustainability study of gender relationships to identify challenges to maintaining changes after the end of project support. EQ 6: Effectiveness of consortium and project collaboration How effective was collaboration/complementarities with other Feed the Future projects, as part of testing the push-pull hypothesis and the effectiveness of project management through consortium arrangement? What lessons and experiences could be drawn? Findings GRAD forged a close relationship with ENGINE, a flagship nutrition program for USAID, in seven GRAD woredas throughout 2015 and 2016 and collaborated with ENGINE in conducted nutrition education and skills-building trainings (see EQ1g). GRAD also coordinated with The Agricultural Growth Program – Agribusiness and Market Development (AGP-MADE), a Feed the Future project, to implement its push￾pull strategy. Though originally the two intended to work in the same areas and focus on different aspects of the push-pull strategy, in practice the activities’ approaches were better suited to different woredas, 32 with GRAD focusing on poorer areas and AGP-ADME on those with more developed value chain capacities.14 GRAD created multi-stakeholder platforms (MSPs) in communities to exchange information, give updates on progress, and discuss critical issues pertaining to production and sale of specific commodities. The MSPs engaged stakeholders including farmers, cooperatives, government, and private sector; each MSP typically met biannually. MSP participants interviewed during the evaluation reported that collaboration was effective in creating joint plans and addressing issues that arose. KIIs reported positively on GRAD consortium collaboration. Collaboration focused mainly on M&E follow-up tasks, provision of joint training efforts and field visits, sharing information with GoE, and sharing lessons learned and best practices. Each quarter, consortium members hold a technical working group meeting where members cross-fertilize both lessons learned and best practices adopted by presenting specific thematic areas for discussion. Key informants pointed out some weaknesses in coordination including the clarity of communication on issues important to staff like per diems and benefits, delays in procurement, improving communication, and understanding of roles and responsibilities among IPs, and the need for some flexibility in models and strategies. In particular, partners in Hawassa Zuria did not appear to be consistently leveraging their different areas of expertise. Conclusions IPs and donors reported positive but limited examples of cross-project collaboration. The consortium proved productive, with a few areas for improvement that were not a significant cause of concern. GRAD made a reasonable start at leveraging collaboration both within and external to the consortium. Recommendations 29. Review and carry out timely procurement management procedures and clarify specific roles and responsibilities of each IP within the consortium. 30. Enhance USAID project partnerships through regular coordination and collaboration, both within USAID and between implementing partners, beginning at the design phase and continuing through implementation and evaluation. 14 GRAD MTE. 33 CONCLUSIONS The following are brief conclusions for each of the six evaluation questions: EQ 1a. 39,306 of the targeted 50,000 households graduated from PSNP food support. Given that GoE, not GRAD, ultimately decides on graduation and the flaws identified in the process, this is not an ideal benchmark for a USAID project. EQ 1b. GRAD households on average appeared to increase their income by the targeted $1 per day. However, this still leaves many households in extreme poverty, and climate shocks leave many in a continued state of vulnerability. Northern woredas were better off compared to southern woredas. EQ 1c. The overall model of the VESA appears to have been quite successful. Enrollment in and use of the VESAs were widespread, and reports of participants’ experience with the VESAs were positive. The training and increased knowledge were considered the most valuable part of the VESAs. EQ 1d. GRAD households increased their savings from an average of $12 to $141. VESAs were a common place for households to save their money. GRAD households did not substantially increase their access to loans—the difference in households who had taken loans in the past year was not statistically significant—but the composition of loans did change. GRAD households were relying less on private lenders at endline and more on VESAs, MFIs, and RUSACCOs. EQ 1e. GRAD effectively strengthened extension services through continuous collaboration and partnership with the GoE. This relationship is key to the likelihood of GRAD extension work’s sustainability. The gender balance of government and project extension workers does not reflect the gender balance of GRAD beneficiaries: more men are extension workers, whereas the activity beneficiaries had a high proportion of women. EQ 1f. Agro-dealers and cooperatives were the most important suppliers of inputs to GRAD households. FEMAs were spoken of positively, but were never major suppliers of inputs or channels of marketing. Some exceptional FEMAs made the transition to cooperatives. EQ 1g. Generally, families’ nutritional awareness and dietary practices improved. Collaboration with health-focused projects such as ENGINE and with GoE HEWs were key to effectively implementing this series of activities. Though some DAs received nutrition-related training, for the most part GRAD worked with HEWs on nutrition practices and DAs on agriculture practices. There was a missed opportunity for better integration of agriculture and nutrition activities with GoE extension staff. EQ 1h. GRAD effectively improved households’ abilities to cope with climate change through improving financial safety nets. GRAD beneficiaries were also appreciative of improved inputs that have the potential to make their crops more resilient to climate impacts, though the effects of these inputs on decreasing crop loss are not yet apparent, perhaps because of the extreme drought of 2015–2016. Despite these gains, climate-related crop loss remains a major threat to household food security. EQ 1i. GRAD’s micro-franchising and agro-dealer innovations both appeared to be effective at increasing incomes and access to agriculture inputs. EQ 2. GRAD households’ incomes, expenditures, and assets all rose between baseline and endline. Qualitative findings illustrate that GRAD has played a role in improving households’ economic well-being through increasing access to value chains. Hawassa Zuria saw slower improvements or declines in household economic indicators, likely because of the severe effect of drought in that woreda. Male- and female-headed households both saw growth, though male-headed households’ expenditures grew at faster rates and maintained higher levels than female-headed households, demonstrating that female-headed households may face special barriers that are not fully addressed by project interventions. 34 EQ 3. Many of the safety nets and coping mechanisms that typically support resilience improved under GRAD. Overall, months of food security in the past year also increased, a sign that the existence of these strategies is helping households to cope with shocks. The evidence suggests that GRAD’s interventions were most effective in improving resilience in areas that experienced moderate and normal weather￾related shocks but could not meet the crisis-level drought that Hawassa Zuria and Ziway Dugda experienced. The emergency inputs GRAD provided seem to have played an important role in household survival during this period, when more incremental improvements in resilience the activity could reasonably have promoted would be insufficient to cope with the full extent of the drought. EQ 4. GRAD partners, given the constraints of the 2015–2016 drought, influenced households to increase their incomes and safety nets through improved access to savings and credit, improved income generating opportunities, and improved agricultural techniques. However, KIIs, FGDs, and IPs’ representatives consistently argued that the graduation process was directly controlled by the GoE and therefore that the influence that a given project can have on graduation is limited. EQ 5. GRAD's influence on gender was reported as positive. One of the most important contributions GRAD made to advancing gender equity and women’s empowerment was by creating a safe space for husbands and wives, other men and women, and the community in general to begin exploring and discussing gender issues. Some of these changes, however, go against deeply held attitudes and traditions. Although VESA savings groups have women’s access to financial resources, women still struggle against unequal access to financial capital and resources. Female-headed households, particularly, have seen slower gains than male-headed households. EQ 6. IPs and donors reported positive but limited examples of cross-project collaboration. The consortium proved productive, with a few areas for improvement that were not a significant cause of concern. GRAD made a reasonable start at leveraging collaboration both within and outside the consortium. 35 RECOMMENDATIONS 1. While graduation plays an important role in household access to food and therefore should continue to be tracked, it is not a strong project indicator because of the many external influences on graduation. Future USAID-funded projects should not have graduation as a key indicator of success if the decisions about graduation are outside the control of USAID’s activity. It might serve well as a proxy indicator, since the USAID activity merely contributes toward graduation. 2. Gender of household heads needs to be more explicitly taken into consideration in future project design to ensure that female-headed households’ unique barriers to livelihoods opportunities are addressed. 3. In the upcoming review the GoE is planning of the graduation process and benchmarks, USAID should use lessons learned from the GRAD activity to advocate for evidence-based graduation and an appropriate, responsive graduation system. 4. Given the vulnerability of Ethiopia to weather-related shocks, future USAID-funded food security projects should consider more explicitly addressing drought resilience, or else link to and build on drought resilience projects. 5. The VESA model of combining knowledge with access to credit should be continued through future programming. 6. Improve awareness of beneficiaries and partners to the rationale for initial small loan sizes and relatively short loan periods. 7. Emphasize savings as a key component of future USAID resilience projects. 8. Explore the possibility of registration of VESAs as a next step in their maturation. 9. Set standards for per diem or related payments for staff who participate in donor-funded project activities; create awareness of the standard and maintain it. 10. Balance the gender of staff and community volunteers to reflect the gender balance of project participants, and advocate with GoE to achieve a similar gender balance among their extension workers. 11. Provide increased opportunity for hands-on practical learning/training in all DA and other extension workers’ capacity building efforts. 12. Develop a model of extension collaboration based on the GRAD experience and distribute the model to upcoming projects like GRAD and to others in community development for replication. GoE collaboration should be a key element of future projects. 13. In future USAID projects, support the transition of community groups like FEMAs to become registered as cooperatives and further help them establish linkages to local, regional, and international markets. 14. Value Chain Groups should improve their ability to negotiate as groups on matters pertaining to collection, storage, processing, and market handling services, as well as prices for their crops and livestock. 15. Train DAs and their supervisors in nutrition-sensitive agriculture (NSA) to build their capacity to include nutrition-related messages and activities in their everyday work. This work should be supported through the provision of nutrition education materials and funds for nutrition-related demonstrations and follow-up to VESA households (including promoting nutrient-dense vegetable production and production diversification). 16. Climate-resilient agriculture and livelihoods resilience activities should remain a key focus of future USAID activities. 17. Replicate and scale up the micro-franchise and agro-dealer approaches. 18. Crisis modifiers in future programs will probably be necessary and are a flexible emergency approach the GoE ought to consider to complement donor interventions. 19. Continue to measure change over time to better measure the effect of USAID programming on resilience in less extreme circumstances. 20. Include a women’s economic empowerment approach, coupled with explicit gender sensitization, in future USAID projects. 36 21. Conduct a sustainability study of gender relationships to identify challenges to maintaining changes after the end of project support. 22. Review and carry out timely procurement management procedures and clarify specific roles and responsibilities of each IP within the consortium. 23. Enhance USAID project partnerships through regular coordination and collaboration, both within USAID and between implementing partners, beginning at the design phase and continuing through implementation and evaluation. 37 ANNEXES Annex 1: Statement of Work STATEMENT OF WORK FINAL PERFORMANCE EVALUATION OF GRADUATION WITH RESILIENCE TO ACHIEVE SUSTAINABLE DEVELOPMENT (GRAD) (AID-663-12-000001) I. INTRODUCTION The Graduation with Resilience to Achieve Sustainable Development (GRAD) was designed to support and enhance livelihood options of the chronically food insecure households by promoting and supporting on-and off-farm income generating activities, facilitating output and input market linkages, and increasing access to microfinance services. The activity is meant to support the Government of Ethiopia’s safety net and food security programming. GRAD complements Ethiopia’s Productive Safety Net Program (PSNP) to accelerate the graduation of targeted beneficiaries from PSNP by supporting selected commodity value chains. The activity promotes alternative livelihood options through greater access to inputs and credit and saving services, while reducing transaction costs and strengthening market linkages in chronically food insecure areas. This five-year activity is a USAID investment valued at over $25 million, and has been using market-driven approaches to work in food insecure and climate-vulnerable areas to diversify livelihoods, build assets, raise income and enable them to withstand climate and other shocks. Table 1: GRAD Budget and Timeline Award# Start Date End Date LOP Budget AID-663-12-000001 12/5/2011 12/4/2016 25,587,133 The objective of GRAD is to directly link with all four core investment areas of USAID/Ethiopia’s Feed the Future strategy that achieves food security and nutrition objectives. These are (1) improve agricultural productivity of staple crops and livestock; (2) reduce transaction costs and increase market linkages; (3) increase the purchasing power of poor consumers by promoting alternative livelihoods; and (4) maximize efficient utilization of food. GRAD employs strategies that allow poor and chronically food insecure households to gradually assume more productive roles in value chains and income generating activities that are appropriate for their conditions. The strategic objective of GRAD is to graduate at least 50,000 HHs from PSNP and increase the beneficiary households’ (HHs’) 38 income on average by $365by the end of the fifth year in 16 Woredas of Tigray, Amhara, Oromia, and SNNPR as depicted in the below table. Table 2: GRAD Operational Areas (Regions and Woredas) Regions Woredas Tigray (4) Raya Azebo, Alamata, Offla, Endamehoni Amhara (2) LiboKemkem, Lay Gayint Oromia (4) Zeway Dugda, ArsiNegelle, Shala, Adami Tulu SNNP (6) Meskan, Mareko, Loka Abaya, Hawela Tula, Hawassa Zuria, Shebedino CARE Ethiopia leads the GRAD consortium, which includes the Relief Society of Tigray, the Organization for Rehabilitation and Development in Amhara, Catholic Relief Services and its local partner Meki Catholic Secretariat (MCS) in Oromia, and CARE itself in SNNP. The Netherlands Development Organization (SNV) provides technical support on value chain development and extension. Tufts University's Feinstein International Center was to lead GRAD’s impact evaluation efforts through year 4 of the activity but Tufts is not doing any impact evaluation as their contract has ended earlier than expected. Externally, GRAD collaborates with multiple agencies of the GoE of Ethiopia and many of USAID's Feed the Future projects. USAID’s evaluation policy encourages independent external evaluation to increase accountability to inform those who develop programs and strategies, and to refine designs and introduce improvements into future efforts. It is with this objective that this evaluation is planned to review and evaluate the performance of the USAID-funded GRAD activity implemented by CARE Ethiopia and its consortium members. Social Impact, Inc., through Ethiopia Performance Monitoring and Evaluation Services (EPMES) activity, will conduct the final performance evaluation of GRAD to assess program results and document lessons learned, and will hereafter be referred to as the “Team” in this document. This task is considerably part of the activities managed under the EPMES award. The evaluation will focus on assessing the activity’s five-year performance in achieving its program goal, objectives, and results. II. BACKGROUND Building on its National Food Security Strategy, the Government of Ethiopia (GoE) launched a major consultation process with development partners in 2003 that led to the design of the Food Security Program (FSP). The FSP aimed to exit households from the emergency relief system, and enabling them to ‘graduate’ to sustainable food security. Under the FSP, the GoE launched the Productive Safety Net Program to provide a multi-annual, predictable resources transfer (usually a mix of cash and food) to chronically food insecure households in a way that prevents depletion of household assets while creating public assets within communities. The PSNP was designed with the idea that other components such as household credit packages, other food security programs and longer-term development actions implemented by the GoE and other development partners would be in place and complement the safety net so households could move toward food security. PSNP was successful in meeting immediate food needs of the beneficiaries, but it did not provide a sufficient strategy to enable chronically food insecure families to move out of food insecurity. It is now acknowledged that graduation of the chronically poor from PSNP and eventually from food insecurity can only be achieved if PSNP is linked to livelihood promotion programs. The Graduation with Resilience to Achieve Sustainable Development (GRAD) was designed to support and enhance livelihood options of the chronically food insecure households by promoting and supporting on-and off-farm income generating activities, facilitating output and input market linkages, increasing access to microfinance services, improving nutritional status of communities and assisting communities to adapt to climate change. The complementarities of GRAD to the GoE led PSNP was needed to ensure households were able to move out of food insecurity and into a sustainable future. GRAD supports efforts to strengthen linkages of livelihood activities with PSNP by supporting and enhancing asset creation at household level through focused interventions. These linkages help beneficiaries graduate to sustainable food security. The program complements PSNP to accelerate the graduation of targeted beneficiaries by supporting selected commodity value chains. The activity builds on extensive lessons and experiences of previous USAID funded livelihoods programs that targeted chronically poor rural households in PSNP implementing Woredas. The program was designed within the general framework outlined by the previous GoE’s Household Assets Building Program (HABP) launched in February 2010 and also aims to raise 39 income of food insecure households, protect their assets and improve their resiliency. GRAD has been implemented through a consortium of local and international NGOs led by CARE and includes Tufts University, Netherlands Development Organization, Catholic Relief Services, Organization for Rehabilitation and Development in Amhara, Relief Society of Tigray, Ethiopian Catholic Church, and Agriservice Ethiopia. The GRAD program has an integrated logical framework designed to achieve its outcomes and ultimately contribute to the goal of the GoE’s Food Security Program. As per Table 3 indicated in Annex 11, the framework outlined objectively verifiable indicators and means of verification at output, outcome and goal levels. Table 3: GRAD Activity Results Framework Strategic Objective: Graduate 50,000 chronically food insecure HHs from PSNP food support in 16 targeted Woredas and increase each HH’s income by $365 at the fifth year of the activity’s lifetime. Result 1—Enhanced Livelihood Options of Chronically Food Insecure Households in Highland Areas IR 1.1 On- and off-farm economic opportunities, inclusive value chains and market access for targeted HHs stimulated. IR 1.2: An inclusive financial sector promoted and access to a range of financial products and services expanded: IR 1.3: Extension services upgraded Result 2 – Improved Household and Community Resilience IR 2.1: Women’s resilience and access to inputs, services and information increased IR 2.2: Nutritional status of infants, children and reproductive age women improved IR 2.3: Climate change adaptation improved IR 2.4: Promote aspirations for graduation among targeted PSNP HHs and enhance enablers of graduation Result 3 – Strengthened Enabling Environment to Promote Scale-up and Sustainability IR 3.1: Collaboration among stakeholders consolidated to promote joint learning and scale up IR 3.2: Enabling environment improved Source: SOW III. PROJECT GOALS AND OBJECTIVES The GRAD activity supports asset creation and adoption of market-led livelihood options for the chronically poor households through sustainable linkages to output and inputs markets and access to microfinance services. The activity identified and supported viable on-farm commodity value chains to be implemented by targeted beneficiaries. Other off-farm income generating activities selected by the beneficiaries themselves and pertinent to the area of operation will also be supported. GRAD has been supporting capacity building activities for the stakeholders to enable successful management and implementation of program activities. GRAD expected to achieve the following specific objectives: a) Reduce the status of food insecurity in targeted households; b) Improve the nutritional status of the households by increasing production/income and promote greater utilization; c) Increase economic growth opportunities and diversify livelihoods by promoting on-and off-farm income generating activities; d) Demonstrate market-driven approaches for targeted food-insecure households; e) Improve access to microfinance services through a graduated assistance program; f) Create a sustainable demand for extension services; and g) Through Feed the Future, link the GRAD program with other projects in both pastoral and productive Ethiopia. 40 The GRAD activity consists of five major components: 1) Viable On-and Off-Farm Income Generation: To graduate from food insecurity, households must invest in the improvement of existing production systems and/or developing new income-generating enterprises. GRAD expects that beneficiaries might prefer to identify the Income Generation Activities (IGAs) themselves, depending on their agro￾ecological zones and the market demand of the area. 2) Access to Financial Services: GRAD promotes an inclusive financial sector by fostering cooperation and coordination among key actors while building the capacity of service providers to introduce appropriate products and of service recipients to enter into financial markets. 3) Market Access for Inputs and Outputs: GRAD focuses on facilitating the linkages between the suppliers and users of inputs and recognizes that a successful design fosters engagement and integration with the market economy and existing private sector actors. 4) Nutrition: Nutrition also plays a critical role in the GRAD activity. GRAD aims to capitalize and link with other Feed the Future programs to ensure that the beneficiaries (especially women) have opportunities to improve their livelihoods, and indirectly, the nutritional status of the household. The activity encourages innovative and creative ways in which synergies and coordination can be created and strengthened to ensure that households receive the maximum health and nutrition benefits. 5) Adaptation to Climate Change: Climate change adaption activities are an integral part of USAID/Ethiopia’s Feed the Future strategy. With the high levels of vulnerability found in chronically food insecure areas, GRAD builds the resilience of communities through supporting climate change adaptation activities. IV. PURPOSE AND USE OF THE EVALUATION The main purpose of this final performance evaluation of GRAD is to examine its development outcomes or results at higher and intermediate levels and the extent to which GRAD has met its goals, and determine the overall effectiveness of the partnerships, and any strengths and challenges in the courses of implementation. Specifically, the evaluation will assess or identify: i. The key components of GRAD with regard to achieving the objectives and targets, including the cross-cutting gender objective; ii. The effectiveness of the livelihood diversification (on and off-farms), nutrition, climate change adaptation and gender related activities promoted by GRAD iii. The major types and values of assets developed by the GRAD beneficiaries and their level of access to financial services and markets. iv. The strengths and weaknesses in project management (planning, implementation and monitoring) and effectiveness in delivery of the desired services and inputs to the targeted beneficiaries and communities. The ultimate purpose of this evaluation is to make use of the evaluation results to demonstrate the effectiveness of the program, disseminate results and lessons learned to similar type of programs and share with a broader development community. The recommendations from this evaluation would help to improve the USAID’s programming of livelihoods and food security oriented programs. 41 EVALUATION QUESTIONS The evaluation is expected to provide answers to the following key questions: 1. Which of the following key technical areas of GRAD exhibited sizable results and which one(s) have not? a. Graduate 50,000 chronically food insecure HHs from PSNP food support in targeted Woredas. b. Targeted households’ income improvement to reach the $365 increase at the end of the life of activity. c. Participating HHs are organized into small groups (with typical GRAD group VESAs comprise of 20 to 30 HHs) and supported with technical value chain advice to take advantage of diversified livelihood opportunities. d. Access to finance for beneficiaries is increased to fund livelihoods activities; e.g. guarantee fund scheme in facilitating access to finance. e. Extension services are strengthened. f. Development of input supplies and market linkages. g. Nutrition interventions. h. Climate change adaptation activities. i. The contribution of GRAD ‘innovative’ interventions such as micro-franchising initiative and the agro-dealer strategy. 2. To what extent have the assets and income of beneficiary households (considering both the total value of assets and income, and the nature/composition of asset-holdings and sources of income) changed over the course of the activity duration? 3. To what extent have beneficiary households become resilient during periods of shock? 4. What level of influence did the GRAD partners in the beneficiary households’ graduation process? 5. To what extent has this activity contributed to gender equity and women empowerment, specifically in addressing the role of gender in decision making on use of resources? To what extent has GRAD addressed gender gaps identified among women, men, girls, and boys?” 6. The effectiveness of collaboration/complementarities with other Feed the Future Projects as part of testing the push￾pull hypothesis and the effectiveness of project management through consortium arrangement and what lessons and experiences could be drawn? V. EVALUATION METHODS The evaluation team will be responsible for developing an evaluation strategy and methodologies that include a mix of qualitative and quantitative data collection and analysis approaches. The team should present an evaluation questions matrix showing the source of data, method of data collection and also the tool to be used to answer each of the evaluation questions. The methodology will be presented as part of the draft work plan as outlined in the deliverables below and included in the final report. The evaluation team will have access to a variety of program implementation documents for their analysis and reports. The strengths and weaknesses of proposed methodology/ies should be identified as well as measures taken to address those weaknesses. All data collected and presented in the evaluation report must be disaggregated, as appropriate, by sex and geography (E.g. Woreda level). The suggested methodology should include, but is not limited to: 6.1 Quantitative Method: i, Assessing the activity performance using indicators: Comparison of current indicator values to baseline data for those selected output and outcome indicators. Note: Baseline data collected and analyzed in October 2012 is available. Table 3: Key Indicators proposed for Final Performance Evaluation Strategic Objective: Chronically food insecure households graduate from food support Graduation: # of GRAD Supported beneficiary graduating from PSNP Income: Income/expenditure per household Equity: % of men and women reporting meaningful participation of women in decision making regarding productive resources and income and increased access to productive resources. Result 1: Enhanced livelihood options for chronically food insecure households 1.1 Average number of Income sources of vulnerable households receiving USG assistance 1.2 Average value of assets of GRAD supported households 42 Result #2: Improved community and household resilience 2.1% Percent of USG supported PSNP households selling productive assets during periods of shock 2.2 % of women and men reporting increase in women's influence over household decision making i. Population-based Quantitative End-line Survey: The evaluation team is required to design and execute all aspects of a representative, stratified population-based household survey. These include developing a sampling procedures, questionnaires, and field procedure manuals for enumerators and supervisors; training enumerators and supervisors to administer the questionnaires; piloting and refining questionnaires; arranging logistics for field work; pre-testing the survey rollout; supervising data collection; and ensuring data entry, cleaning, tabulation, and analyses. 6.2 Qualitative Methods: The following are list of indicative qualitative methods that might be employed: i. Desktop review of relevant documentation: includes the implementing partner’s proposals, monitoring and evaluation plans, baseline studies, periodic performance reports, midterm evaluation report, case studies, and other related documents as necessary. ii. Site visits: Evaluation team members shall visit selected GRAD sites in the four regions. The evaluation team in collaboration with USAID will choose a strategy to select sites. The evaluation team will prepare and use adequate check list for site visits and consultations. iii. Key informant interviews: The evaluation team will have interviews with the following stakeholders (note that this list is not exhaustive): ● Relevant USAID offices and other USG offices in Ethiopia and USAID/Washington; ● Implementing partner’s staff at both prime and sub partner level; ● GRAD Beneficiaries, community leaders etc.; ● Key GoE of Ethiopia representatives at both national and local levels (i.e. regional and woreda level representatives); ● IV. Focus group discussions: the evaluation team in consultation with ALT office or GRAD implementing partners will decide on who will be the focus group discussants. VI. EXISTING PERFORMANCE INFORMATION SOURCES The evaluation team will review the following documents: a) Program Descriptions and Modifications b) Annual Work Plan over the five years c) Quarterly Reports d) Annual Reports e) PMP and other M&E documents f) GRAD performance data g) Project-generated assessments (e.g. Intermediate Results Assessments, Cost Benefit Analysis) h) GRAD Mid Term Evaluation i) Feed the Future Mid Term Evaluation j) Relevant external evaluations from other sources (e.g., PSNP Impact Evaluation Reports) k) GoE FSP (PSNP/HABP) performance data (if available) l) Activity’s baseline survey report VII. TEAM COMPOSITION, QUALIFICATION AND RESPONSIBILITIES The evaluation team shall consist of three independent international experts (with one serving as the team lead and primary coordinator with USAID’s contractor, EPMES) as well as two high level Ethiopian experts, at least one of whom can also serve as an interpreter. The international experts should be senior-level evaluation analysts specialized in areas such as Agriculture, Food Security and Livelihoods and gender issues. All international experts must be fluent in English and have strong writing skills and at least one team member have credentials and experience in evaluation design and methods. In similar fashion, the Ethiopian experts should have experience with Agriculture, Food Security and Livelihoods, and monitoring and evaluation in 43 Ethiopia. The Ethiopian experts should also be proficient in English and Amharic. Knowledge of one or more of other major local Ethiopian languages is desired. A statement of potential bias or conflict of interest (or lack thereof) is required of each team member. The multi-disciplinary team should also be capable of assessing most or all dimensions of GRAD including technical areas (agriculture, agribusiness, financial inclusion, nutrition, climate change adaptation and gender) and operational areas (management, partnerships, staffing, reporting systems, M&E, etc.). USAID may propose internal staff members from USAID/Washington or other Missions to accompany the team during site visits or participate in key parts of the evaluation (specific event participation to be determined in conjunction with the contractor and the team leader), and they are expected to provide written inputs to the draft report. Team Leader (one): The team leader will take the overall responsibility of the evaluation. S/he will be the primary point of contact between USAID and the evaluation team and have responsibility for submission of all deliverables. The team leader should be an evaluation expert. Substantial food security background and experience in Ethiopia are desirable. The incumbent must meet the following criteria: ● Master’s or Ph.D. that included high-level coursework or professional continuing-education training in food security, social sciences, statistics, project evaluation, or another field relevant to evaluation. ● At least 10-year work experience in particular with practical evaluation experience in Ethiopia and /or other African countries and who has led at least two similar evaluation/studies. He/she will be responsible for overall management of the evaluation, including coordinating and packaging the deliverables as well as ensuring quality and timeliness ● Strong team management skills, and sufficient experience with evaluation standards and practices to ensure a credible product; ● Directly managed the design and implementation of at least two large-scale household surveys with complex designs; ● Has excellent organization and writing skills and a demonstrated ability to deliver a quality written product in English (e.g., evaluation report and PowerPoint presentations); ● Has excellent oral communication, presentation, and inter-personal skills; and ● Has the technical and management skills to manage budget resources (dollars and staff) for the study, as well as assist and support the team with field logistics, e.g., coordinating with USAID and/or a GoE ministry and project stakeholders to set up initial appointments for interviews. International Experts (Livelihoods/Food Security Specialist)- (Two): – These experts will provide technical guidance related to livelihoods and food security. The incumbents must meet the following criteria: ● Should have at least eight (8) year work experience in managing and evaluating food security projects in Ethiopia and/or other African countries. One of these experts should be an expert in Value Chain/Agribusiness. ● Master’s or PhD degree in a field of study related to Agricultural Economics, livelihoods and/or food security or agriculture; ● Strong knowledge of livelihoods and food security indicators, agricultural extension, and the Productive Safety Nets and Household Assets Building Programs in Ethiopia; ● Must be fluent in English and have strong writing and organization skills; ● Excellent oral communication, presentation, and inter-personal skills; ● Excellent analytical and technical skills; and ● Strong knowledge of USAID’s programming, experience on past food security and livelihoods baseline surveys or final evaluations would be a plus. Local Ethiopian Experts (Two): The Ethiopian experts, with at least five years of work experience in monitoring and evaluation, should be a mix of Agricultural Economist/Food Security Specialist, Climate Change/Adaptation Specialist, and Gender Specialist. The Ethiopian experts should also be proficient in English and Amharic. Knowledge of one or more of other major local Ethiopian languages is desired. As appropriate, the contractor may also hire other lower level evaluation team members who can serve as data collectors, field work supervisors, etc. 44 VIII. EVALUATION SCHEDULE The estimated time period for undertaking this evaluation is 56 calendar days, of which at least 50 days should be spent in Ethiopia. The ideal available time for the evaluation team is September 25-November 30, 2016, however, the date will be finalized between USAID and the EPMES Contractor. The evaluation team is required to work six days a week, but with no premium payment for the sixth day. The team is required to travel to select Woredas in each region where program activities are being implemented. At least 40% of the consultants’ time will be spent to conduct interviews with project staff, GoE partners, and project beneficiaries both in Addis and outside Addis. The evaluation team will prepare an exit briefing and presentation of the findings to USAID staff before the consultants depart Ethiopia. Also, the evaluation team will submit a draft report 24 hours in advance of the exit briefing for review and comments by USAID. Comments from USAID will be incorporated before the submission of the final draft. As GRAD will end on December 04, 2016 the evaluation should be completed at least one month before the activity’s completion to provide adequate time for sharing of the evaluation findings and recommendations to the implementing partners, other stakeholders and concerned parties who have been involved in the courses of the activity implementation. Illustrative LOE is provided below but detailed LOE will be worked out and included in the work plan by the contractor for USAID review and approval. Table 3: Level of Effort (LOE) in person days Activity Expat Team Leader [1] Expat Expert [2] Ethiopian Experts [2] Travel to/from Ethiopia 4 8 - In-briefing with USAID 1 2 2 Document review, Initial work plan submission (methodology, draft questions, data analysis plan, suggested list of interviewees, finalized questions for the survey) 5 10 10 Submission of final work plan 2 4 4 Interviews in Addis Ababa 3 6 6 Interviews or survey work in project/activity areas 20 40 40 Mid-term briefing and interim meetings with USAID 1 2 2 Data analysis, preliminary report and presentation preparation 10 20 20 Draft evaluation report 3 6 6 Final exit presentation to USAID (with PowerPoint presentation and draft evaluation report) 1 2 2 Final evaluation report & one page briefer 10 10 8 Totals 56 112 112 IX. MANAGEMENT Social Impact, Inc., the Contractor managing the Ethiopia Monitoring and Evaluation Service (EPMES) activity will identify and hire the evaluation team, pending the Contracting Officer’s Representative (COR’s) and relevant technical office’s concurrence, assist in facilitating the work plan, and arrange meetings with key stakeholders identified prior to the initiation of the fieldwork. The evaluation team will organize other meetings as identified during the evaluation, in consultation with EPMES’s Contractor and USAID/Ethiopia. The EPMES Contractor is responsible for all logistical support required for the evaluation team, including arranging accommodation, security, office space, computers, Internet access, printing, communication, and transportation. The evaluation team will officially report to the Ethiopia Monitoring and Evaluation Service (EPMES) Contractor, Social Impact, 45 Inc. The EPMES Contractor is responsible for all direct coordination with the USAID/Ethiopia Program Office through the EPMES COR. From a technical management perspective, the evaluation team will work closely with Reta Assegid, Senior Livelihoods Advisor and AOR for GRAD and Endale Lemma, Senior Program Management Specialist/M&E POC in the Assets and Livelihoods in Transition (ALT) Office. In order to maintain objectivity, all final decisions about the evaluation will be made by the USAID/Ethiopia’s Program Office. X. LOGISTICS The contractor will be responsible for all travel and logistics associated with conducting the evaluation. XI. REPORTING REQUIREMENTS AND DELIVERABLES A. DESCRIPTION AND TIMELINE OF DELIVERABLES 1. In-briefing: Within 48 hours of the availability of the evaluation team in the EPMES’s Contractor Office, the evaluation team will have an in-brief meeting with USAID/Ethiopia’s Program Office and ALT Office for introductions; presentation of the team’s understanding of the assignment; initial assumptions; review of the evaluation questions, survey instruments, and initial work plan; and adjustment of the SOW, if necessary. 2. Evaluation Work Plan: Within five working days following the in-brief, the Evaluation Team Leader shall provide a detailed initial work plan to USAID/Ethiopia’s Program Office and ALT Office. The initial work plan will include: (a) the overall evaluation design, including the proposed methodology, data collection and analysis plan, and data collection instruments; (b) a list of the team members and their primary contact details while in-country, including the e-mail address and mobile phone number for the team leader; (c) the team’s proposed schedule for the evaluation; and (d) the estimated cost for the evaluation. USAID offices and relevant stakeholders are asked to take up to three working days to review and consolidate comments through the EPMES COR. Once the evaluation team receives the consolidated comments on the initial work plan, they are expected to return with a revised work plan within two working days. The revised work plan shall include the list of potential interviewees and sites to be visited. USAID Offices send their final comments/say on the Contractor’s re-submitted documents/work plan within two working days of receipts of the revised documents/work plan from the Contractor and the Contractor proceeds accordingly. 3. Mid-term Briefing and Interim Meetings: Schedule a mid-term briefing with USAID to review the status of the evaluation’s progress, with a particular emphasis on addressing the evaluation’s questions and a brief update on potential challenges and emerging opportunities. The team will also provide the COR for EPMES and AOR for GRAD with periodic written briefings and feedback on the team’s findings. If desired or necessary, weekly briefings by phone can be arranged with the Program Office and the Assets and Livelihoods in Transition Office to provide updates on field progress and any problems encountered. 4. PowerPoint and Final Exit Presentation to USAID and relevant partners that will include a summary of key findings and key conclusions as these relate to the evaluation’s questions and recommendations to USAID. To be scheduled as agreed upon in advance of the in-briefing, and five days prior to the evaluation team’s departure from Addis Ababa. A copy of the PowerPoint file will be provided to the Program Office prior to the final exit presentation. 5. Draft Evaluation Report: The content of the draft evaluation report is outlined in Section XII.B, below, and all formatting shall be consistent with the USAID branding guidelines. The focus of the report is to answer the evaluation questions and may include factors the team considers to have a bearing on the objectives of the evaluation. Any such factors can be included in the report only after consultation with USAID. The draft evaluation report will be submitted by the evaluation team leader to the Program Office 24 hours in advance of the exit briefing for review and comments by USAID. USAID’s Program Office and ALT Office will have five working days in which to review and comment and the Program Office shall submit all comments to the evaluation team leader. The evaluation team will then have 10 working days to make appropriate edits and revisions to the draft and re-submit the revised final draft report to USAID. Assets and Livelihoods Transition and the Program Office will have five working days after the submission of the second revised draft to again review and send any final comments. 6. Final Evaluation Report will incorporate final comments provided by the Program Office and ALT. The length of the final evaluation report should not be more than 30 pages, not including Annexes and Executive Summary. The final report should be submitted to the Program Office within three days of receipt of comments by the evaluation team leader. All project data and records will be submitted in full and shall be in electronic form in easily readable format; organized and fully document for use by those not fully familiar with the project or evaluation; and owned by USAID and made available to the public, barring rare exceptions, on the USAID Development Experience Clearinghouse (http://dec.usaid.gov ). 7. One-page briefer on key qualitative and quantitative findings and conclusions relative to the key evaluation questions included in the evaluation’s scope—to be given to the appropriate GoE counterpart(s) so that they have the opportunity to review evaluation findings and share them with the larger community. Each briefer will be reviewed by the Program Office and ALT prior to distribution. 46 B. FINAL REPORT CONTENT The evaluation report shall include the following: 1. Title Page 2. Table of Contents (including Table of Figures and Table of Charts, if needed) 3. List of Acronyms 4. Acknowledgements or Preface (optional) 5. Executive Summary (3-5 pages) The executive summary should succinctly capture the evaluation purpose and evaluation questions; project background; evaluation design, methods; and limitations; and the findings, conclusions, and recommendations. 6. Introductory Chapter a. A description of the project evaluated, including goals and objectives. b. Brief statement on purpose of the evaluation, including a list of the main evaluation questions. c. Brief statement on the methods used in the evaluation such as desk/document review, interviews, site visits, surveys, etc. d. Explanation of any limitations of the evaluation—especially with respect to the methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.)—and how these limitations affect the findings. 7. Findings: This section should include findings relative to the evaluation questions. The information shall be organized so that each evaluation question is a sub-heading. 8. Conclusions: This section must answer the evaluation questions based upon the evidence provided through the Findings section. The information shall be organized so that each evaluation question is a sub-heading. 9. Recommendations: Based on the conclusions, this section must include actionable statements that can be included into future program design. Recommendations are only valid when they specify who does what, and relate to activities over which the USAID program has control. For example, recommendations describing GoE action is not valid, as USAID has no direct control over GoE actions. Alternatively, the recommendation may state how USAID resources may be leveraged to initiate change in GoE behavior and activities. It should also include recommended future objectives and types of specific activities based on lessons learned. The information shall be organized so that each evaluation question is a sub-heading. 10. Annex: The annexes to the final evaluation report should be submitted as separate documents—with appropriate labels in the document file name (e.g., Annex 1 – Evaluation SOW), and headers within the document itself—and may be aggregated in a single zipped folder. a. Evaluation Statement of Work b. Places visited; list of organizations and people interviewed, including contact details. c. Evaluation design and methodology. d. Copies of all tools such as survey instruments, questionnaires, discussions guides, checklists. e. Bibliography of critical background documents. f. Meeting notes of all key meetings with stakeholders. g. “Statement of Differences” h. Evaluation Team CV’s C. REPORTING GUIDELINES ● The format of the report shall be consistent with the USAID branding guidelines. ● The evaluation report should represent a thoughtful, well-researched and well- organized effort to objectively evaluate what worked in the project over the given time period, what did not, and why. ● Evaluation reports shall address all evaluation questions included in the statement of work. ● The evaluation report should include the statement of work as an annex. All modifications to the statement of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline need to be agreed upon in writing by the Program Office. ● Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an annex in the final report. ● Evaluation findings will assess outcomes and impact on males and females, and data will be disaggregated by gender, age group, and geographic area wherever feasible. ● Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). ● Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong 47 quantitative and/or qualitative evidence. ● Sources of information, including any peer-reviewed or grey literature, will be properly identified and listed in an annex. ● Recommendations will be supported by a specific set of findings. They will also be action-oriented, practical, and specific, with defined responsible parties for each action. 48 Annex 2: Places visited, list of organizations, and people interviewed (plus GRAD Evaluation contact details) 1. Key Informants (KIIs) and Focus Groups Discussions (FGDs) by Organization and Title in Addis Ababa, North, Central and South Regions, Oct.-Nov. 2016 2. Addis Ababa, and North (Tigray, Amhara) Regions 3. Central (Oromia) and South (SNNPR) Regions Organization Name Position/Title Addis Ababa ASE Country Representative/Program Manager CARE GRAD Chief of Party CARE DCOP CARE LDM Manager (FSF+) CARE M&E Specialist CARE Program or Country Director CARE Former Gender Specialist CARE VC, Livestock, Bees Experts CARE VESA & MFI Experts CARE Sr. Child Health & Nutrition Advisor CARE Sidama Zone Zone POC CRS Country Rep./Program Manager DONOR - DFID Rural Dev. And Livelihoods Expert DONOR - ECCO-CIDA Canadian FSF Food Sufficiency for Farmers DONOR - EU Rural Dev. And Food Security Expert DONOR - The World Bank Livelihoods Spec. - PSNP Donors Coordinator DONOR - USAID Ethiopia Assets & Livelihoods in Transition Food Security Program Coordinator/Senior Livelihoods Advisor DONOR - USAID Ethiopia Program Officer DONOR - USAID Ethiopia M&E Specialist DONOR - USAID Ethiopia FTF Coordinator DONOR - USAID Ethiopia Health or Nutrition Officer DONOR - USAID Ethiopia GRAD AOR Min. of Agriculture Minister or Deputy Minister 49 Min. of Agriculture Director/Food Security Coordination Min. of Agriculture Coordinator, Federal Livelihoods Coordination Unit ORDA Country Representative/Program Manager REST Country Representative/Program Manager SNV GRAD Coordinator TIGRAY (North) Agro-dealer, Mekoni Owner Agro-dealer, Maichew Owner CM VESA Tesfa Raya Men and women farmers CM VESAS in Raya Azebo Men and women farmers CM VESAS in Raya Azebo Men and women farmers Coops Promotion Agency Director Cooperative Union Gen. Manager ENGINE/Save the Children Regional Director GoE/Min. of Agriculture Regional Agricultural Dev. Officer Office of Women's Affair Gender Specialist Primary Cooperative Maichew Farmers Private Meat Processor Marketing, Supply, & Processing Mgrs. REST Executive Director, REST REST Dept. Head, Environ. Rehab. & Agric. Dev. REST Planning, M&E Expert REST REST GRAD Coordinator REST REST Head of Gender Unit REST VC Honey & Livestock Specialist REST Nutrition Specialist REST Regional Micro-Finance Sales Agent for Spent Grain Agro-dealer VC Honey, Misawit, Endamehoni Men and women farmers VC Livestock Farmer Groups, Endamehoni Men and women farmers VC Potato farmer group Men and women farmers VESA Endamahoni Men and women farmers 50 AMHARA (North) ACSI Regional MFI Deputy CEO Agriculture & Marketing Cooperative Model farmers, Private Traders, Experts and Team Leaders Agro-dealer, Amhara Coop member owner Coop. Promotion Agency Director Cooperative Promotion Agency Regional Director Cooperative Saving/Loan Union CEO ENGINE/Save the Children Regional Director GoE Regional President GoE Woreda Administrator GoE/Min. of Agriculture Woreda Agricultural Director Officers GoE/Min. of Health Woreda Health a/o Nutrition Officers Multi-Purpose Farmers’ Cooperative Union – Nefas Mewchia, Lay Gayint Men and women farmers Multi-Stakeholder Platforms (MSPs) Beneficiaries ORDA Regional Director and Food Sec. Coord. ORDA Regional GRAD COORD Specialist ORDA Lay Gayit Woreda Coordinator VC Livestock Farmer Groups Men and women farmers VC Malt Barley Men and women farmers VC Potato Men and women farmers VESA, Endamehoni Men and women farmers OROMIA (Central) Ag & NR Dev Office, Ziway Dugda Ext Agronomist Ag & NR Dev Office, Ziway Dugda Vice Head of Ag (Deputy) Ag & NR Dev Office, Ziway Dugda Natural Resources Expert (Extension) Agro-dealer, Ziway Dugda Owner Agro-dealer, Ziway Dugda Owner Bureau of Women and Children’s Affairs, Olgocho Town, Ziway Dugda Vice Head (Deputy) Woreda Level Bureau of Women and Children’s Affairs, Olgocho Town, Ziway Dugda Projects Expert 51 Bureau of Women and Children’s Affairs, Olgocho Town, Ziway Dugda Projects Expert Burka Lamafo, Kebele, Ziway Dugda Micro-franchiser Coop OCSS, Ziway Dugda Branch Manager Coop promotion, Ziway Dugda Head Cooperative Promotion Agency, Input Supply Core Process, Ziway Dugda Vice Head Cooperative Promotion Agency, Ziway Duga Head FEMA Bari, Ziway Dugda Farmers FEMA Kabele Hallo, Ziway Dugda Farmer/FEMA Secretary Food Security Task Force, Ziway Dugda Different GoE Kabele Burka Lemafo Development Agent (DA) GoE Kabele Burka Lemafo Development Agent (DA) GoE Kabele Hallo Development Agents (DAs) GoE Kabele Sheledgoto Development Agent (DA) GoE Kabele Sheledgoto, Ziway Dugda Development Agent (DA) GRAD MCS, Olgocho Town, Ziway Dugda Lead Community Facilitator (LCF) MCI/GRAD, Meki, Oromia Value Chain Expert MCI/GRAD, Meki, Oromia Gender Expert MCI/GRAD, Meki, Oromia GRAD Project Manager MCI/GRAD, Meki, Oromia M&E Expert MCI/GRAD, Meki, Oromia MFI Expert MCS, Meki, Ziway Dugda GRAD, Nutrition Expert MCS/GRAD Animators, Ziway Dugda Animators Metemame MFI, Ziway Dugda Branch Manager MFI Metemamen, Ziway Dugda Manager Regional Oromia Bureau of Ag & NRM, Addis Livelihoods Regional Oromia Bureau of Ag & NRM, Addis Ag Dev Officer Trade & Market Dev. Office, Ziway Dugda Head VESA Burka Lamafo Kebele, Ziway Dugda Model farmers VESA Halo Kebele, Ziway Dugda Farmers VESA Kenanisa, Ziway Dugda Farmers 52 VESA Misoma, Ziway Dugda VESA Group VESA Ubo Barichakebele, Ziway Dugda Farmers VESA, Kusaye, Ziway Dugda VESA Group SNNPR (South) Agric. & Natural Resource Dev. Office, Agric. Ext. Department, Hawassa Zuria Ag Extension, Crop Protection, Crop Production, Hort. Experts Agric. & Natural Resource Management Office, Hawassa Zuria GoE Development Agents (DAs) Agro-dealer, Dore Bafano, Hawassa Zuria Owner, manager CARE/GRAD, Hawassa Zuria Community Facilitators Doyo Chale Coop, Doyo Chale, Hawassa Zuria Coop Chairman Doyo Chale, Hawassa Zuria IP Promoters Doyo Chale, Hawassa Zuria Model farmers GRAD VESA, Hawassa Zuria Farmers Kabele Doyo Chale, Woreda Hawassa Zuria FEMA Groups (3) that joined to become a Cooperative Livestock buyer, Shebedino, SNNPRS Owner Lopho VESA, KabeleJara Gelalicha, Hawassa Zuria Men and women farmers Micro-franchiser group, Hawassa Zuria Member MINJA VESA, Hawassa Zuria Men and women farmers Salam VESA, Kabele Doyo Chale, Hawassa Zuria Men and women farmers Sidama MFI, Hawassa Zuria General Manager Woreda Ag. Office, Hawassa Zuria Food security coordinator 53 Annex 3: GRAD Evaluation Design Matrix Evaluation Question Data Sources Data Collection Tools Data Analysis Method 1. Which of the following key technical areas of GRAD exhibited sizable results and which one(s) have not? For further detail refer to GRAD Qualitative Semi￾Structured Interview Guide included in this report. a. Graduate 50,000 chronically food insecure HHs from PSNP food support in targeted Woredas. Project Documents (last quarterly report); Woreda Food Security Task Force (WFSTF) Records; GRAD HH. Household (HH) Survey, Focus Groups (FG) using semi￾structured interview guides, Key Informant Interviews (KII) using semi-structured interview guides. Descriptive /categorical comparisons/ Analysis b. Targeted households’ income improvement to reach the $365 increase at the end of the life of activity. Project Documents; GRAD HHs (which includes VESA/FEMA groups and their leadership, promoters, and members); reports of WFSTF. HH Survey, FG, KII. Descriptive /categorical comparisons/ Analysis c. Participating HHs are organized into small groups (with typical GRAD group VESAs comprise of 20 to 30 HHs) and supported with technical value chain advice to take advantage of diversified livelihood opportunities. Project M&E and other project resources; Implementing Partners (IP); GRAD HHs; Agro￾dealers. HH Survey, FG, KII. Descriptive /categorical comparisons/ Analysis d. Access to finance for beneficiaries is increased to fund livelihoods activities; e.g. guarantee fund scheme in facilitating access to finance. Project records; GRAD HH, financial services/products providers (e.g., MFIs, VESAs). HH Survey, FG, KII. Descriptive /categorical comparisons/ Analysis e. Extension services are strengthened. Extension Workers; GRAD HH; IPs; multi-stakeholder platform members; other Stakeholders (donors, similar projects, farmer cooperatives/unions). HH Survey, KII, FG, mini-survey of extension Workers at FG. Descriptive /categorical comparisons/ Analysis f. Development of input supplies and market linkages. Project records; private sector (e.g., input suppliers, traders, agro-dealers); cooperatives/Unions; GRAD HH; IP actors facilitating market linkages; multi-sector platform participants; other projects (NGO/contractor led). KII, FG, Observation. Descriptive /categorical comparisons/ Analysis g. Nutrition interventions. GRAD HH, providers of nutrition interventions (e.g., extensionists, local GoE, other projects ENGINE, health extension workers). HH Survey, FG, KII. Descriptive /categorical comparisons/ Analysis h. Climate change adaptation activities. GRAD HHs, IPS, Woreda/Kabele/ level staff (e.g., Bureau of Agriculture Staff/SMS, DA). HH Survey, FG, KII. Descriptive /categorical comparisons/ Analysis i. The contribution of GRAD ‘innovative’ interventions such as micro-franchising initiative and the agro-dealer strategy. Project records, IPs, GRAD HH, micro-franchisers, agro￾dealers, promoters. KII, FG. Descriptive /categorical comparisons/ Analysis 2. To what extent have the assets and income of beneficiary households (considering both the total value of assets and income, Project records; IPs; GRAD HH; Woreda/Kabele/ level staff (e.g., Bureau of Agriculture Staff/SMS, DA). HH Survey, FG, KII. Descriptive /categorical comparisons/ Analysis 54 and the nature/composition of asset-holdings and sources of income) changed over the course of the activity duration? 3. To what extent have beneficiary households become resilient during periods of shock? Project records (M&E reports); GRAD HH; Woreda/Kabele level staff (e.g., Bureau of Agriculture Staff/SMS, DAs/). HH survey. FG. In-depth interview based on positive deviance. Descriptive /categorical comparisons/ Analysis 4. What level of influence did the GRAD partners in the beneficiary households’ graduation process? GRAD HH; IPs; Woreda /Kabele/ level staff (e.g., Bureau of Agriculture Staff/SMS, DAs). KII, FG. Descriptive /categorical comparisons/ Analysis 5.To what extent has this activity contributed to gender equity and women empowerment, specifically in addressing the role of gender in decision making on use of resources? To what extent has GRAD addressed gender gaps identified among women, men, girls, and boys?” GRAD HHs; Project reports; Extension workers; Secondary data (Gender Assessments); Woreda /Kabele/ level staff (e.g., Bureau of Agriculture Staff/SMS, DAs, gender officers). HH Survey, FG, KII, Mini-Case Studies. Descriptive /categorical comparisons/ Analysis 6.The effectiveness of collaboration/ complementarities with other Feed the Future Projects as part of testing the push-pull hypothesis and the effectiveness of project management through consortium arrangement and what lessons and experiences could be drawn? Other FTF project staff; IPs; other stakeholders. KII, FG, SWOT Analysis. Descriptive /categorical comparisons/ Analysis NOTE: The appropriateness of carrying-out Focus Groups will be assessed by location and in light of the location-based security context. 55 Annex 4: GRAD Activity Endline Quantitative Household Survey Questionnaire [Tools, survey instruments, questionnaires, discussions guides, checklists: GRAD Questionnaire for HH Survey] Introduce yourself: Good morning/afternoon. My name is _____. I represent Social Impact, Inc. through Sub-Saharan Africa Research and Training Center. We are conducting a performance evaluation of the USAID-funded Graduation with Resilience to achieve Sustainable Development (GRAD) project. The purpose of the evaluation is to examine the results of GRAD and determine its effectiveness. We are interested in your response for the different questions I have about GRAD. The information we receive from you will be used only in evaluating the GRAD project. The interview will take about one hour and the interview is based on your full willingness. You can stop anywhere or you may not answer some of the questions that you do not want to. But we appreciate your participation in the interview and would be grateful. Is it all right if we begin now? 1=Yes, 0 =No. (If ‘No’ stop the interview). Thank you! Registered Name of GRAD-HH ----------------------------- Questionnaire# -------------------------------------- Name of Interviewer ----------------------------------- Day: ----------- Month: ------------- Year 2016 Region Woreda Peasant Association (Kebele) Village/Cluster 1. Household Socio-Demographics (Enter number or Circle Appropriate Box) A B Gender (respondent) Female Male C Age (respondent) Yrs D Marital Status (respondent) Never married Currently married Divorced Separated Widowed E Literacy (Can s/he read & write?) No YES E1 If yes, number of years in education F Number of Household (HH) members G Number of literate HH members over the age of 16 (who can read & write) H Respondents Relationship to HH head Head Spouse Child Parent Grandchild Grandparent Other If the respondent is also the Head of the household skip to section 2 I J Gender of HH head 1=male 2=Female J1 Age in years K Marital Status HH head Never married Currently married Divorce d Separated Widowed 56 PSNP and GRAD Participation A Number of (active) working adults in the HH (Ages between 15 to 65 years old) B Has your HH been participating in the PSNP? Yes NO→ C How many years has your HH been participating in the PSNP? D Has your HH already graduated from PSNP? YES NO→ skip to d2 D1 How many years do you expect it will take for you to graduate from the PSNP? D2 Are you or any member of the HH engaged in at least one new Value Chain activity since joining GRAD project? YES NO D3 How long have you participated in GRAD project? Filter: check the answer for D2 and if it is No go to 3 otherwise continue below E. in which of the following GRAD value a chain is your HH registered/participating in? Type of Value Chain Participating/Registered Yes =1 No=0 Number of HH members participating Cattle fattening Sheep fattening Goat rearing Malt Barley production Beans production Apiculture Vegetables 2. Livelihoods Shocks a. In the past year, has your household experienced any food or income related shocks? 1=Yes 2=No →skip to Q4 a1. What type of shock did you experience? Shock/Event (1=Yes 0=No) Weather related crop loss (drought rain failure, flood etc.) Disease or pest related crop loss (specify) Livestock disease or mortality Other unexpected shock (human illness, death etc.) b. Ask the participant what impact these shocks had on their livelihoods? Impact/Outcome (1=Yes 0=No) Crop Loss Income Loss Livestock Loss (mortality & stress sales) Food Shortage Livestock Feed Shortage Labor Loss/shortage Other (specify) c. What strategies has your household employed to cope with the impact of these shocks? Coping Strategies (1=Yes 0=No) (1=Yes 0=No) Reduced the number of meals Collected/sold firewood or charcoal Ate less (smaller portions) Sent children to stay with relatives 57 Borrowed food or money Withdrew children from school Sold livestock or other productive assets Sent children to work Engaged in labor activities Household members migrated to find work Used own saving Used grain reserve Other (specify) 4. Household Expenditure a. In the past 12months–how much did your Household (HH) spend on the following items? (Indicators1-­­15only–if nothing put zero) Major Expenditures ETB 1 Land renting 2 Farming inputs (seeds/fertilizers/pesticides/tools) 3 Livestock or poultry purchases 4 Livestock inputs (livestock feed/water/vaccines/treatments etc.) 5 Investments in other production/income generating activities (e.g. beehives) 6 Education/schooling (fees/uniforms/rent/transport) 7 Health/medical expenditures 8 Clothes (including shoes/blankets/gabis etc.) 9 Home improvements (construction) 10 Household Items (furniture/cooking utensils) 11 Social obligations/ceremonies (weddings/funerals/Iddir – other contributions) 13 Water & Transportation 14 Loan or debt repayment 15 Other major nonfood expenses over 100 Birr (specify) 16 Cereals for consumption (teff/sorghum/wheat/maize/barley/rice etc.) 17 All other food items (meat/fruit/vegetables/oil/salt/sugar etc.) 18 Other (specify) TOTAL ETB 3. Income Sources a. Crop Sales Ask the respondent to list each type of crop that they produced and sold the past 12 months. Only include crops that were sold for cash income. List each crop mentioned in the table below and count the total number of different crops sold. Types of crops sold Types of crop sold Types of crop sold Total # of different crops sold b. Other Income Sources Where possible, try to summarize each of the income sources into the categories provided in the following table. If a corresponding income source has been mentioned, enter 1 into the appropriate column for that income source. 58 c. Now using the following table as a reference, ask the respondent if she/he or any other household member was involved in any of the following income generating activities last year. The objective of this exercise is to make sure they have not missed any income sources. First ask the respondent spontaneously and enter1 for the product responded spontaneously and secondly probe for the remaining products that the respondent could not answer and enter 1 if yes otherwise enter zero. Livestock & Livestock Products Spontaneously answered 1=yes, 0=No Answered by probing 1=Yes 0=No Crafts /Small Industry/ Spontaneously answered 1=yes 0=No Answered by probing 1=Yes 0=No Selling Fattened Shoats (meat) Making baskets or mats Selling Fattened Cattle (meat) Spinning or weaving cloth (cotton or wool) Rearing & selling ‘un-­­fattened’ animals (cattle, shoats, camels, donkeys etc.) Making repairing clothes (embroidery, tailoring) Dairy sales (milk, cheese, yogurt) Making traditional utensils or farm tools Selling animal skins/hides/dung Pottery Poultry rearing and sales (chickens, eggs) Blacksmithing/metal work or carpentry Beekeeping (selling honey, bees-wax, beehives or colonies) Other craft/small industry (specify) Renting Oxen for farming Services Renting pack animals for transport Water carrier, Porter Other livestock related income (specify) Barber or Hairdresser Renting land Musician (drummer, singer, dancer) Employment Traditional healer Salaried job (specify) Midwife or birth attendant Public works (PSNP cash for work) Counselor (disputes, marriage) Agricultural/farming work Other services (specify) Non-agricultural work (e.g. construction) Food & Drink Processing Domestic work (e.g. house maid) Selling tea, coffee or beverages (e.g. Tejj) Military service Selling cooked food Other employment (specify) Other Income Sources Trading & Retail Specify Trading in crops (grains, pulses, vegetables, chat, coffee) Specify Trading in livestock and livestock products Specify Trading in other commodities (not petty trading go to next Specify 59 row) Petty trading (selling cake, bread, soft drinks, candy, tobacco etc.) Specify Sale of Natural Products Specify Selling firewood or charcoal Specify Selling water Specify Selling grass or fodder (for livestock) Specify Selling construction materials (poles, sand etc.) Specify Selling wild fruits, bush meat etc. Specify Selling other natural products (specify) Specify TOTAL 4. Access to Input and Output Markets a. Are you a member of Farmers Economic and Marketing Association? Yes=1 No=0 b. Through which channel did you sell your products in the past two years? (circle all that apply) Type of Marketing Agency/aggregato rs Selling products Yes=1 No=0 How often in past two years? 1=always 2=Most of the time 3= Rarely FEMA Cooperatives Private aggregator Selling Through Agro- dealers Local market /consumers Others (specify) c. In your opinion is the price you are paid for your products when selling through the following marketing channels fair Type of Marketing Agency/aggregators 0=No 1= Yes 2= Not sure FEMA Cooperatives Private aggregator Selling Through Agro- dealers Local market /consumers Others (specify) d. Did you buy any of the following agricultural inputs for your production in the last 12 months? Inputs purchased 0= No 1= Yes Fertilizer Improved seed Herbicides 60 Pesticides Other1 (specify) Other2 (specify) e. Location 1=in village store 2=Kebele store 3=Woreda town 4=other (specify)Please ask the respondent from which supplier and from where the respondent purchased the inputs (Multiple response is possible) Inputs purchased Suppliers Location Agro￾dealer 0=No, 1=Yes FEMA 0=No, 1=Yes Coops 0=No, 1=Yes Other 0=No, 1=Yes Within Kebele 0=No, 1=Yes Other kebele 0=No, 1=Yes Wored a town 0=No, 1=Yes Other 0=No, 1=Yes Fertilizer Improved seed Herbicides Pesticides Other1 (specify) Other2 (specify) f. Please ask the respondent about affordability and quality of the following inputs Inputs purchased Affordable? 0=No 1= Yes Quality of inputs 1= low 2= medium 3= high Fertilizer Improved seed Herbicides Pesticides Other1 (specify) Other2 (specify) 7. Credit & Savings a. Have you ever borrowed money from any of the following loan source? Loan Source 0= No 1= Yes Microfinance Institution Bank Rusacco Village economic & Social Association (VESA) Money Lender/Trader/Neighbor Other (specify) Filter2: Check the answers for “a” above and if No to all skip to d b. When did the last time the household borrowed loan from the following institutions, what was the amount borrowed and the repayment status? Loan Source When was it borrowed? Amount Loan & Interest Repaid (YES=1 No = 0) 61 Last 12 months 0=No 1=Yes 2 to 3 years ago 0=No 1=Yes Before 3 years 0=No 1=Yes Microfinance Institution Bank Rusacco Village economic & Social Association (VESA) Money Lender/Trader/Neighbor Other (specify) c. How did your household use this loan? Loan Utilization Amount ETB 1 Food purchases 2 Medical costs 3 Education/schooling (fees/uniforms/rent) 4 Land rent/property or home improvements (corrugated roofing etc.) 5 Purchase livestock or poultry 6 Invested in petty trade/retail or other business 7 Farming inputs (animal treatment/seeds/fertilizers/pesticides/tools) 8 Social obligations/ceremonies (weddings/funerals other contributions) 9 Pay taxes/debts/loans 10 Other (specify) d. Household Savings Does the household (any member) currently have any savings? (circle) 1=YES 0= NO →skip to Q8 If the answer is yes, ask where the money is saved (circle all that apply) Bank/MFI 0=No, 1=Yes Home 0=No, 1=Yes VESA 0=No, 1=Yes Other 0=No, 1=Yes Ask them how much money they have saved by source (ETB) 62 7.0 Project Outcomes and Benefits 8.1 Value Chain Income a. In the past year (2015) have you sold any fattened livestock? (put zero if none were sold) Livestock type Sold any livestock 0=No, 1=yes Number Sold Total Income ETB Oxen Sheep Goats Other (specify) b. In the past year (2015) what quantity of honey has your household produced, consumed and sold? Produced (Kg) Consumed (Kg) Sold (Kg) Total Income ETB c. In the past year (2015) what quantity of the following products has your household produced, consumed and sold? Crop Type Produced (Kg) Consumed (Kg) Sold (Kg) Total Income ETB Malt barley Beans Tomato Onion Other VC TBD VC TBD Other VC TBD Other VC TB 8.2. Extension Services &Training: a. In the past year, have you or any member of your household received training related to farming from Agricultural extension workers, NGO or other experts in ?1= Yes 0= No→ skip to c b. Please specify the type of training given; the provider; whether the skill acquired is applied; and your rating about usefulness of the training received in the table below. a. Training Type 1=Yes 0=No b. Specific field of training (e.g. fattening, crop protection, queen rearing, etc.) c. Provider (e.g. ET￾Gov, NGO, others d. Did you apply it? 1=Yes 0=No e. How useful was it? Score 0- 5 (0=not at all; 5= very useful) Crop Production Livestock Production Beekeeping Value Addition/Marketing Health Nutrition (Dietary Diversity) Financial Literacy Business Skills (IGA) Climate change adaptation * Other (specify) *Such as risk reducing practices/actions to improve resilience to climate change c. Did you receive any other agricultural extension services during the past two years? 1= YES0= NO→ Skip to 8. d. How satisfied are you with services you received (0 =not satisfied, 1= somehow satisfied 2 = satisfied). 63 e. Who provided you these agricultural extension services? 1= GoE 2= NGO 3=others. 8.2 Assets a. Land Holding Land Holdings Hectares How much land does your household (HH) currently own? * How much land did your HH cultivate last year (including rented land)? *Include land they rent to others b. Asset Inventory15 Livestock No Productive Assets No Household Assets No Oxen/Bulls Axes Blankets/Gabis Cows Machete Chairs Steers Sickles Tables Heifers Spade Cupboard Calves Hoe Mats Sheep Bucket Lantern Goats Grain-mill Flashlight (torch) Donkeys Plow yoke Watch/clocks Poultry Plow beam Kerosene stove Mules Plow share Radio/cassette player Horses Traditional beehive Mobile phone Camels Modern beehive Bicycle TOTAL 9 Food Security and Nutrition a. You will now ask the respondent to give an assessment of her/his household food security for the past production year (2015. Using the calendar as a visual aid – you will provide the respondent with 30 counters and ask her/him to score each month using the following criteria: 0 = Not enough food (food insecure) 1 = Just enough food (food sufficient) 2 = Plenty of food/surplus (food secure) 2015 2016 Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Total # Months food needs met (all months with a score of 1 or 2) 15Asset values will be collected separately through focus groups and key informant interviews 64 b. If the HH met all 12 months of their food needs, ask them what quantity of cereals (kg) they had in storage right before the onset of the Kremt-rains (reference month June). Then ask them to estimate how many months this would have lasted the household16. It may be useful here to refer to the number of HH members (1.F) to ensure that the estimates make sense. Quantity of cereals in storage Kg Number of months’ household food needs covered Months Number of HH members (reference) 9 Household Dietary Diversity a. Informtherespondentthatyouwouldliketoaskaboutthetypesoffoodeateninthehouseholdyesterdayduring the day and night. However, if the previous day was a fasting day ask them about the day before (or last non-­­fasting day). Then read the following list of foods and place a ONE in the box if the food in question was eaten by a household￾member or place a ZERO in the box if the food was not eaten by a household member. List of Foods Consumed by Household Members Yesterday (1=Yes 0=No) A Any local foods such as (Lafiso. Towlo, etc.), bread, rice noodles, biscuits, cookies or any other foods made from Teff, millet, sorghum, maize, rice, wheat (or other locally available grain)? B Any white potatoes, manioc, cassava, or any other foods made from roots and tubers? C Any pumpkin, carrot, squash, or sweet potato that are yellow or orange inside? D Any dark, green, leafy vegetables such as cassava leaves, bean leaves, kale, spinach, pepper leaves, taro leaves, and amaranth leaves? E Any other vegetables? F Any ripe mangoes, ripe papayas or any other locally available fruit rich in vitamin A (insert)? G Any other fruits? H Any beef, lamb, goat, chicken, duck, other birds, liver, kidney, heart or other organ meats? I Any eggs? J Any fresh or dried fish? K Any foods made from beans, peas or lentils? L Any cheese, yogurt, milk, or other milk products? M Any foods made with oil, fat or butter? N Any sugar or honey? O Any processed foods such as chips, pastry, cakes, chocolates, sweets or candies, soda, fruit juices or drinks? P Any other foods, such as condiments, coffee, tea? b. Household Hunger Scale No 1 Question Response Code In the past month, was there ever no food to eat of any kind in your house because of lack of resources to get food? Yes = 1 2 = No (skip to 3) 2 How often did this happen in the past month? 1=Rarely (1-2 times) 2=Sometimes (3­10 times) 3=Often (more than 10 times) 16It may be useful here to refer to the number of HH members (1.F) to ensure that the estimates make sense. As a rough estimate, each HH member would require 0.5 kg of cereal/day or family of 6 would need 3 kg/day or 90 Kg/month. This assumes that approximately 1,750 Kcal of the 2100 kcal daily energy requirements are being met through cereals. 65 3 In the past month, did you or any household member go to sleep at night hungry because there was not enough food? Yes =1 2= No (skip to 5) 4 How often did this happen in the past month? 1=Rarely (1­2 times) 2=Sometimes (3­10 times) 3=Often (more than 10 times) 5 In the past month, did you or any household member go a whole day and night without eating anything at all because there was not enough food? Yes =1 2=No (end of module) 6 How often did this happen? 1=Rarely (1-2 times) 2=Sometimes (3-10 times) 3=Often (more than 10 times) 11. Women’s Empowerment: 11.1 Decision Making If the respondent is female, ask her if she is the only adult living in her household. If the answer is ‘Yes’, skip to the next exercise. Similarly, if the respondent is male and there are no adult females in the household, skip to the next exercise and write NA in large letters next to the table below On a scale of 1-4, ask the respondent how much influence the wife or main female in the household has over decisions on the following aspects of Household life. 1= No influence at all 3 = A medium amount of influence If the participant is more comfortable using proportional 2 = A little influence 4 = A lot of influence scoring to assess her influence over decisions you can use proportional piling with 4 counters but be sure to explain that more counters equal more influence. a. Production & Livelihoods Decisions SCORE 1 What crops to grow 2 What farming inputs to use or buy (fertilizer, improved seeds etc.) 3 What crops to take to the market to sell and when to sell them 4 What livestock production activities to engage in (rearing/fattening/dairy etc.) 5 What livestock to purchase or sell 6 What business or Income Generating Activities to engage in b. Financial Decisions 7 Major household expenditures (such as clothes purchases, furniture etc.) 8 Minor household expenditures (such as food for daily consumption) 9 Borrowing money (loan amount/source and utilization) 10 Lending (loaning) money to relatives or friends c. Household Decisions 8 Food and meals 9 Children’s education 10 Household construction/maintenance 11 Family planning 12. Supplementary PAT Module Poverty Assessment Tool 66 Number 1 How many rooms does your household’s dwelling unit have? 2 What is the main construction material of your dwelling’s roof? Corrugated iron sheet …………………..….1 Reed and bamboo.... 4 Clay .....................5 Other ..................6 Thatch and grass ……………………….....…2 Wood and mud …….……..………………….3 3 What is the main source of lighting for your dwelling? Kerosene ………………………..…….…....…1 Firewood ............4 Candle.................5 Other...................6 Electricity (private) …………………………2 Electricity (shared) ……………….…………3 4 What is the main source of cooking fuel for your dwelling? Mainly collected firewood ………..1 Electricity ............6 Mainly purchased firewood ….… 2 Crop residue........7 Charcoal ……………………..…..3 Don’t use cooking fuel.........8 Kerosene ………..……………… 4 Other....................................9 Butane gas--------------------------- 5 5 What is your household’s main source of drinking water in the rainy season? Tap inside the house …..…..…………….1 Protected well/spring....... 5 Unprotected well/spring... 6 Rain water........................ 7 River, lake, pond, etc....... 8 Tap in compound (private) ………..….…2 Tap in compound (shared) ………..….….3 Tap outside the compound (shared) …4 For questions 6-11 refer to the asset inventory to confirm whether the HH-owns any cattle, axes, or gabis and the number of these items they own – then enter this information in the following table: Number or Circle 6 Does your household currently own cattle? (0=No 1=Yes) 0 1 7 How many cattle does your household own? 8 Does your household currently own any axes (gejera)? (0=No 1=Yes) 0 1 9 How many axes does your household own? 10 Does your household currently own any blankets (gabis)? (0=No 1=Yes)0 1 11 How many blankets does your household own? 12 Does your household own a radio (0 =No 1=Yes) 0 1 13 Does your household own a television (0 =No 1=Yes) 0 1 14 Does your household own a video deck (0 =No 1=Yes) 0 1 13. Climate Change Risk Reducing Practices In the past year, have you or your household employed any of the following risk reduction practices to improve your resiliency to climate change? Type 1= Yes 0=No Agriculture – practices to increase predictability and or productivity of agriculture anticipating Climate related variability or shocks Water – practices or actions to improve water quality, supply and efficient use Health – practices or actions to prevent or control disease incidences and outcomes Disaster Risk Management – practices or actions to reduce the negative impact of extreme events. 67 Annex 5: GRAD Final Performance Evaluation Qualitative Questions Guide Matrix Evaluation Questions FEDERAL GRAD IPS, GoE (MOA, MOH), DONORS REGIONAL & WOREDA GoE & COUNTERP ARTS e.g. EXTENSION WOREDA/ KABELLE/ IP & GoE FIELD STAFF WOREDA GRAD Targeted HHs & WOMEN LEADERS WOREDA AGRO￾DEALERS WOREDA MFI LEADERS WOREDA / KABELLE / VOLUNTEER PROMOTERS WOREDA VESA, FEMA, FOCUS GROUPS 1. Which of the following key technical areas of GRAD exhibited sizable results and which one(s) have not? a. Graduate 50,000 chronically food insecure HHs from PSNP food support in targeted Woredas. X X b. Targeted households’ income improvement to reach the $365 increase at the end of the life of activity. X X c. Participating HHs are organized into small groups (with typical GRAD group VESAs comprise of 20 to 30 HHs) and supported with technical value chain advice to take advantage of diversified livelihood opportunities. X x X X X X d. Access to finance for beneficiaries is increased to fund livelihoods activities; e.g. guarantee fund scheme in facilitating access to finance. X X X X X e. Extension services are strengthened. X X X X X X X f. Development of input supplies and market linkages. X X X X X g. Nutrition X X X X X X 68 interventions. h. Climate change adaptation activities X X X X X i. The contribution of GRAD ‘innovative’ interventions such as micro - franchising initiative and the agro -dealer strategy. X X X X X X To what extent have the assets and income of beneficiary households (considering both the total value of assets and income, and the nature/compositi on of asset - holdings and sources of income) changed over the course of the activity duration? X X X X X To what extent have beneficiary households become resilient during periods of shock? X X X X What level of influence did the GRAD partners have in the beneficiary households’ graduation process? X X X X To what extent has this activity contributed to gender equity and women empowerment, specifically in addressing the role of gender in decision making on use of resources? To what extent has GRAD addressed gender gaps identified among women, men, girls, and boys?” X X X X X X X X The effectiveness of collaboration/ complementaritie X X X 69 s with other Feed the Future Projects as part of testing the push - pull hypothesis and the effectiveness of project management through consortium arrangement and what lessons and experiences could be drawn? 70 Annex 6: Bibliography of critical background documents 1. GRAD Documents a. Program Descriptions and Modifications b. Annual Work Plan over the five years c. Quarterly Reports d. Annual Reports e. PMP and other M&E documents f. GRAD performance data g. Project-generated assessments (e.g. Intermediate Results Assessments, Cost Benefit Analysis) h. GRAD Mid Term Evaluation i. Baseline Survey Report j. CARE, 2016: GRAD Gender Mapping 2. Dessalegn Rahmato, 2004: Searching for tenure security?: the land system and new policy initiatives in Ethiopia, Discussion Paper 12, Addis Ababa: Forum for Social Studies 3. DFID, 2013: Support to the Humanitarian Response Fund (HRF) and OCHA in Ethiopia: Business Case and Intervention Strategy 2014-2016. 4. Feed the Future Mid Term Evaluation - http://www.agri-learning-ethiopia.org/wp￾content/uploads/2015/08/FtF-Ethiopia-Midterm-Evaluation-report.pdf 5. Feed the Future Gender Integration Review, 2016 6. GoE FSP (PSNP/HABP) performance data and Impact Evaluation Reports 7. IFPRI, 2015: Patterns of Agricultural Production among Male and Female Holders: Evidence from Agricultural Sample Surveys in Ethiopia. REAP Research for Ethiopia’s Agriculture Policy. 8. LEAP, 2012: Livelihoods, Early Assessment and Protection: Innovative Climate Risk Management for Food Security Partnership between the World Food Program and the GoE of Ethiopia. May 2012. 9. OCHA, 2014: Humanitarian Response Fund Ethiopia. Annual Report 2013: Office for the Coordination of Humanitarian Affairs. 10. Petersen, T. C., P. A. Stott and S. Herring (eds.), 2012: Explaining extreme events of 2011 from a climate perspective. American Meteorological Society, pp. 1041-1067. 11. USAID, 2016: Resilience at USAID: USAID definition of “resilience." Or more briefly the capacity to withstand shocks.” https://scms.usaid.gov/sites/default/files/documents/1867/06.30.2015%20- %20Resilience%20Fact%20Sheet.pdf. 71 Annex 7: Semi-structured Interview Guides (SIGs), 1–9 templates SIG 1 FEDERAL LEVEL Final Template. Semi-structured Interview Guide (SIG) FOR KII: Complete the table below. CASE ____________ DATE & Interview Team LOCATION (Kabele, Woreda, Region) ORGANIZATI ON TITLE (OR ROLE) OF INTERVIEWEE M F Introduce yourself and those with you. Good morning/afternoon. My name is _____. I represent Social Impact, Inc. We are conducting a performance evaluation of the USAID-funded Graduation with Resilience to achieve Sustainable Development (GRAD) project. The purpose of the evaluation is to examine the results of GRAD and determine its effectiveness. We are very interested in your ideas and opinions about GRAD. While we will appreciate talking with you today we want you to know your participation today is based solely on your willingness to talk to us. You may stop or leave at any time. Your name will not be linked to this discussion and the information we receive from you will be confidential and used only in evaluating the GRAD project. The discussion will take about 1 to 1.5 hours. We would like to write down notes today and ask if that is OK with you? Yes__ No__. Is it alright if we begin now? Yes_ No_. Thank you! 1. Please describe the areas of collaboration between your (organization) and GRAD. Was the collaboration useful? How? Suggestions for ways of improving? (RECORD IF FTF OR OTHER PROJECT/ORGANIZATION). (6) 2. Please tell us about the PSNP graduation process. Probe: Strengths/weaknesses, targeting, inclusion/exclusion of families. (1a) 3. Was your organization involved in strengthening extension? If so, who did you work with primarily (public, private, GRAD ext., Das, CFs,). What specifically was done to strengthen extension? Probe: What was accomplished? Problematic issues? Your recommendations for improvement. (1e) 4. What changes have occurred in input supply and market linkages as a result of GRAD activity? A. Have you observed changes in the availability of inputs? If so, which changes have you observed and which resulted from GRAD intervention? B. Have you observed changes in market linkages? If so, which changes have you observed and which resulted from GRAD intervention? (1f) 5. Was your organization involved with nutrition interventions? Probe: If so, who delivered, what to whom, when, through which channels? How effective were interventions? What was successful/not so successful. 6. What “innovative” mechanisms were used in GRAD? (e.g., agro-dealers, micro￾franchising, other?) To what extent were they successful/not so successful? (1i) 7. What factors do you think promote resilience? Did GRAD apply any of these? To what 72 extent have households become resilient? (3) 8. In what, if any, ways did GRAD partners influence participating households to move forward to graduation? Probe: What was most critical in moving households forward to graduation. (4) 9. Has Grad had any influence on gender equity and women’s empowerment? If yes, how? Do you think these changes are positive? Probe for reasons. If not, probe for reasons. (5) 10. Relative to GRAD what is your understanding of the push-pull model? Which push and which pull strategies do you think have been effective? Which less so? (6) 11. IF PART OF THE CONSORTIUM CONTINUE, IF NOT SKIP AND CLOSE. a. As part of the consortium, what do you see as the strengths/weaknesses of the consortium? Please provide some specific examples. What is the added value/drawbacks? (6) b. As an individual, and at the organizational level, what did you - your organization- learn from participating in the consortium. (6) Thank you for your participation in this interview. Your input is very valuable and we appreciate your willingness to take the time to share your thoughts and opinions! 73 SIG 2 REGIONAL/WOREDA LEVEL Final Template. Semi-structured Interview Guide (SIG) for Regional GoE Officials (e.g., Bureau of Ag, Cooperatives) and Collaborators (MSP, Others) FOR KII: Complete the table below. CASE ___________________ DATE & Interview Team LOCATION (Kabele, Woreda, Region) ORGANIZATIO N TITLE (OR ROLE) OF INTERVIEWEE M F Introduce yourself and those with you. Good morning/afternoon. My name is _____. I represent Social Impact, Inc. We are conducting a performance evaluation of the USAID-funded Graduation with Resilience to achieve Sustainable Development (GRAD) project. The purpose of the evaluation is to examine the results of GRAD and determine its effectiveness. We are very interested in your ideas and opinions about GRAD. While we will appreciate talking with you today we want you to know your participation today is based solely on your willingness to talk to us. You may stop or leave at any time. Your name will not be linked to this discussion and the information we receive from you will be confidential and used only in evaluating the GRAD project. The discussion will take about 1 to 1.5 hours. We would like to write down notes today and ask if that is OK with you? Yes__ No__. Is it alright if we begin now? Yes_ No_. Thank you! 1. Please describe the areas of collaboration between your (organization) and GRAD. Was the collaboration useful? How? Suggestions for ways of improving? (RECORD IF FTF OR OTHER PROJECT/ORGANIZATION). (6) 2. Please tell us about the PSNP graduation process. Probe: Strengths/weaknesses, targeting, inclusion/exclusion of families. (1a) 3. Was your organization involved with VESAs and providing technical advice (extension). If so, what was done and how effective was this? (1c, 1e) 4. Was your organization involved in strengthening extension? If so, who did you work with primarily (public, private, GRAD extension, Das, CFs). What specifically was done to strengthen extension? Probe: What was accomplished? Problematic issues? Your recommendations (1e) 5. Was your organization involved in any multi-stakeholder platforms (MSP)? If so, what was your role and what was the added value of MSP involvement? (1e) 6. What changes have occurred in input supply and market linkages as a result of GRAD activity? A. Have you observed changes in the availability of inputs? If so, which changes have you observed and which resulted from GRAD intervention? B. Have you observed changes in market linkages? If so, which changes have you observed and which resulted from GRAD intervention? 7. Was your organization involved with nutrition interventions? Probe: If so, what was your role, who delivered, what to whom, when, through which channels? How effective were interventions? What was successful/not so successful. (1g) 8. Climate change is widely-discussed these days. Was your organization involved with climate change adaptation activities and if so how and what was accomplished, successful/less successful? (1h) 9. What factors do you think promote resilience? What shocks occurred in your region and what was GRADs response? To what extent do you think GRAD households have become resilient? (3) 10. What changes have occurred in input supply and market linkages as a result of GRAD activity? A. Have you observed changes in the availability of inputs? If so, which changes have you observed and which resulted from GRAD intervention? B. Have you observed changes in market linkages? If so, which changes have you observed and 74 which resulted from GRAD intervention? (4) 11. Has Grad had any influence on gender equity and women’s empowerment? If yes, how? Do you think these changes are positive? Probe for reasons. If not, probe for reasons (5) 12. Has GRAD had an indirect effect on closing gender gaps between women and men, boys and girls? If so, what changes do you see. (5) 13. Relative to GRAD what is your understanding of the push-pull model? Which push and which pull strategies do you think have been effective? Which less so? (6) Agro-dealers’ role? 14. IF PART OF THE CONSORTIUM CONTINUE, IF NOT SKIP AND CLOSE. a. As part of the consortium, what do you see as the strengths/weaknesses of the consortium? Please provide specific examples. What is the added value/drawbacks? (6) b. As an individual, and at the organizational level, what did you - your organization- learn from participating in the consortium. (6) Thank you for your participation in this interview. Your input is very valuable and we appreciate your willingness to take the time to share your thoughts and opinions! 75 SIG 3 WOREDA/KABELE LEVEL Final Template. Semi-structured Interview Guide (SIG) for WOREDA/ KABELE IPs (e.g., TL, CFs), NGOs & GoE FIELD STAFF (e.g., DAs) FOR KII: Complete the table below. CASE___________________ DATE & Interview Team LOCATION (Kabele, Woreda, Region) ORGANIZATIO N TITLE (OR ROLE) OF INTERVIEWEE M F Introduce yourself and those with you. Good morning/afternoon. My name is _____. I represent Social Impact, Inc. We are conducting a performance evaluation of the USAID-funded Graduation with Resilience to achieve Sustainable Development (GRAD) project. The purpose of the evaluation is to examine the results of GRAD and determine its effectiveness. We are very interested in your ideas and opinions about GRAD. While we will appreciate talking with you today we want you to know your participation today is based solely on your willingness to talk to us. You may stop or leave at any time. Your name will not be linked to this discussion and the information we receive from you will be confidential and used only in evaluating the GRAD project. The discussion will take about 1 to 1.5 hours. We would like to write down notes today and ask if that is OK with you? Yes__ No__. Is it alright if we begin now? Yes_ No_. Thank you! 1. Please describe the areas of collaboration between your (organization) and GRAD. Was the collaboration useful? How? Suggestions for ways of improving? (RECORD IF FTF OR OTHER PROJECT/ORGANIZATION). (6) 2. Was your organization involved with VESAs and providing technical advice (extension). If so, what was done and how effective was this? (1c, 1e) 3. For the people in GRAD households who have had access to finance, what difference do you think that made in their lives? (1d) 4. (a) Was your organization involved in strengthening extension? If so, who did you work with primarily (public, private, GRAD extension, Das, CFs?). What specifically was done to strengthen extension? Probe: What was accomplished? Problematic issues? Your recommendations for improvement. (b) If GoE., what did GRAD do to strengthen the extension system of your organization? Please give specifics. Probe: How effective do you think this was? What would you have recommended? (1e) 5. What changes have occurred in input supply and market linkages as a result of GRAD activity? A. Have you observed changes in the availability of inputs? If so, which changes have you observed and which resulted from GRAD intervention? B. Have you observed changes in market linkages? If so, which changes have you observed and which resulted from GRAD intervention? (4) 6. What, if any, changes have occurred in input supply and market linkages as a result of GRAD activity? Probe: If changes, at what level are changes noticeable: federal, regional, zonal, woredas, kabele. (1f) 7. Was your organization involved with nutrition interventions? Probe: If so, what was your role, who delivered, what to whom, when, through which channels? How effective were interventions? What was successful/not so successful. (1g) 76 8. Climate change is widely-discussed these days. Was your organization involved with climate change adaptation activities and if so how and what was accomplished, successful/less successful? (1h) 9. What factors do you think promote resilience? What shocks occurred in your region and what was GRADs response? To what extent do you think GRAD households have become resilient? (3) 10. In what, if any, ways did GRAD partners influence participating households to move forward to graduation? Probe: What was most critical in moving households forward to graduation. (4) 11. A. Has Grad had any influence on gender equity and women’s empowerment? If yes, how? Do you think these changes are positive? Probe for reasons. If not, probe for reasons (5) B. Has GRAD had an indirect effect on closing gender gaps between women and men, boys and girls? If so, what changes do you see. (5) 12. Relative to GRAD what is your understanding of the push-pull model? Which push and which pull strategies do you think have been effective? Which less so? (6) 13. IF PART OF THE CONSORTIUM CONTINUE, IF NOT SKIPAND CLOSE. a. As part of the consortium, what do you see as the strengths/weaknesses of the consortium? Please provide some specific examples. What is the added value/drawbacks? (6) b. As an individual, and at the organizational level, what did you - your organization- learn from participating in the consortium. (6) Thank you for your participation in this interview. Your input is very valuable and we appreciate your willingness to take the time to share your thoughts and opinions! 77 SIG 4 KII GRAD BENEFICIARY FEMALE/MALE LEADERS Final Template Semi-structured Interview Guide (SIG) FOR KII: Complete the table below. CASE _____________________ DATE & Interview Team LOCATION (Kabele, Woreda, Region) ORGANIZATION (OR GROUP) Interviewee # Members in ORG. or GROUP M F M F Introduce yourself and those with you. Good morning/afternoon. My name is _____. I represent Social Impact, Inc. We are conducting a performance evaluation of the USAID-funded Graduation with Resilience to achieve Sustainable Development (GRAD) project. The purpose of the evaluation is to examine the results of GRAD and determine its effectiveness. We are very interested in your ideas and opinions about GRAD. While we will appreciate talking with you today we want you to know your participation today is based solely on your willingness to talk to us. You may stop or leave at any time. Your name will not be linked to this discussion and the information we receive from you will be confidential and used only in evaluating the GRAD project. The discussion will take about 1 to 1.5 hours. We would like to write down notes today and ask if that is OK with you? Yes__ No__. Is it alright if we begin now? Yes_ No_. Thank you! KI Female/Male 1. How many of your members have graduated from PSNP food support? (1a) 2. If we compare average annual household income among your members in 2016 to 4 years ago, would you say it has increased? Y/N If yes, by approximately how many Birr for the whole year? (1b) 3. Which value chains are your organization working in that are supported by GRAD? What other opportunities to earn income are supported by GRAD? (1c) 4. What sources of finance/credit did your organization/group have before GRAD? What effect has GRAD had on your member's access to funds for livelihoods activities? Have funds been guaranteed? (1d) 5. What contact do you have with GoE extension services? Which ones are most helpful? How could they be improved? (1e) 6. For the crops and products you and your members sell, where do sell them? Who are the main buyers? Do you sell individually or as a group? Is it easy to find buyers? When in the year is it easy and when is it difficult to find buyers, and why? (1f) 7. What is the main food security problem your families face? What has GRAD done to help improve families’ nutrition? (1g) 8. What does climate change mean for you? How has climate change affected your community and group members? How have they coped? Have you been involved in any climate change activities? Y/N If Yes, please describe. (1h) 9. What new techniques, practices or services has GRAD demonstrated or provided? What results have agro-dealers or micro-franchising activities had for your members? What benefits? (1i) 78 10. How did GRAD activity affect for households' assets? How have families' annual incomes changed and why? (2) 11. What shocks, e.g. draughts, have you experienced in the past 4 years? How has GRAD helped your organizations prepare for future shocks? (3) 12. How are decisions about resources made by members of your organization? What role do women have in making these decisions? What are the areas [gaps] that were improved by GRAD? (5) 13. What GoE or other projects/programs support your group? What are examples of GRAD collaborating with other project? How did they help to improve food availability, access, and utilization in your households. Were you able to increase household incomes from GRAD supported activities. Several different organizations collaborated with CARE to implement GRAD; which ones supported your organization? What activities were most effective? Which ones should be improved in future projects? (6) Thank you for your participation in this interview. Your input is very valuable and we appreciate your willingness to take the time to share your thoughts and opinions! 79 SIG 5 KIIs AGRODEALERS Final Template Semi-Structured Interview Guide (SIG) FOR KII: Complete the table below. CASE __________________ DATE & Interview Team LOCATION (Kabele, Woreda, Region) ORGANIZATION TITLE (OR ROLE) OF INTERVIEWEE Introduce yourself and those with you. Good morning/afternoon. My name is _____. I represent Social Impact, Inc. We are conducting a performance evaluation of the USAID-funded Graduation with Resilience to achieve Sustainable Development (GRAD) project. The purpose of the evaluation is to examine the results of GRAD and determine its effectiveness. We are very interested in your ideas and opinions about GRAD. While we will appreciate talking with you today we want you to know your participation today is based solely on your willingness to talk to us. You may stop or leave at any time. Your name will not be linked to this discussion and the information we receive from you will be confidential and used only in evaluating the GRAD project. The discussion will take about 1 to 1.5 hours. We would like to write down notes today and ask if that is OK with you? Yes__ No__. Is it alright if we begin now? Yes_ No_. Thank you! 1. What improved inputs do you sell to farmers in your Woreda? If applied correctly, what increases in yields would be expected? If improved inputs were available and applied with resultant increase in crop yields, by how much (in Birr equivalent) would net revenue increase per hectare? (1b) 2. Which inputs do you sell to GRAD farmers or farmer groups? How has GRAD supported your role in value chains? In addition to selling inputs what other opportunities to earn income are supported by GRAD? (1c) 3. What access to finance/credit did you have before GRAD? How has GRAD helped you to access credit to purchase improved inputs and value added (processing) equipment? Have funds been guaranteed? Y/N (1d) 4. What contact do you have with GoE agricultural extension services? Which activities have been most helpful for your business? How could they be improved? (1e) 5.For the crops and products in your area (village?) what inputs are in biggest demand? Are you also a buyer of crops and products from farmers and villagers? Do you provide any processing, storage or marketing services? (1f) 6. How do you see climate change affecting agricultural and livestock in your area? How has climate change affected your community and group members? How have they coped? (1h) 7. What would you GRAD provided in terms of new inputs, equipment, processes, or activities that supported your agro-dealership? How has it worked or benefited your agro-dealership? (1i) 8. What would you say were the main results from GRAD support in terms of your agro￾dealership assets? Have you observed any changes in the availability of improved inputs, (e.g. seeds) and if so, what changes? Which of those changes resulted from GRAD interventions. I’d repeat the same for market linkages (2) 9. What shocks, e.g. draughts, have you experienced in the past 4 years? What preparation will help prepare for future shocks? (3) 10. What organizations provided support to your agro-dealership to help provide better inputs/marketing services to farmers in your area? How did GRAD help? (4) 11. If you belong to an agro-dealer association, how many members are women____ and men____? How does your organization make decisions about the use of resources? What role do women have in making these decisions? What are the areas [gaps] that were improved by GRAD? What 80 still need to be improved? (5) 12. What GoE or other projects/programs support your group? What are examples of GRAD collaborating with other project? How did they help to improve food availability, access, and utilization in your households. Were you able to increase household incomes from GRAD supported activities. Several different organizations collaborated with CARE to implement GRAD; which ones supported your organization? What activities were most effective? Which ones should be improved in future projects? (6) Thank you for your participation in this interview. Your input is very valuable and we appreciate your willingness to take the time to share your thoughts and opinions! 81 SIG 6 FINANCIAL FOCUS KIIs with WOREDA / RuSACCO (private) / Semi-structured Interview Guide (SIG) FOR KII: Complete the table below. CASE ___________________ DATE & Interview Team LOCATION (Kabele, Woreda, Region) ORGANIZATION TITLE (OR ROLE) OF INTERVIEWEE M F Introduce yourself and those with you. Good morning/afternoon. My name is _____. I represent Social Impact, Inc. We are conducting a performance evaluation of the USAID-funded Graduation with Resilience to achieve Sustainable Development (GRAD) project. The purpose of the evaluation is to examine the results of GRAD and determine its effectiveness. We are very interested in your ideas and opinions about GRAD. While we will appreciate talking with you today we want you to know your participation today is based solely on your willingness to talk to us. You may stop or leave at any time. Your name will not be linked to this discussion and the information we receive from you will be confidential and used only in evaluating the GRAD project. The discussion will take about 1 to 1.5 hours. We would like to write down notes today and ask if that is OK with you? Yes__ No__. Is it alright if we begin now? Yes_ No_. Thank you! 1. How do you think loans from your institution have affected the incomes of GRAD beneficiaries, so that they may “graduate”? (1a) If group can’t pay? 2. If you have estimates (from loan applications) of GRAD borrowers’ average annual household incomes, would you say it has increased in 2016, compared to 4 years ago? Y/N If yes, by approximately how many Birr for the whole year? (1b) 3. Which GRAD franchises and/or value chains have benefited from loans from your institution? What new opportunities have proved profitable for GRAD beneficiaries that took out loans? (1c) 4. How has GRAD activities affected the number of borrowers and/or number of loans per year for livelihoods activities? (1d) They can’t specify/estimate. 5. Is provision of (or access to) extension services a condition for borrowers’ eligibility? Y/N For activities that rely on extension services who normally provides the services to the borrowers? (1e) 6. What is the approximate return rate for borrowers, that is those who borrow year after year? What is the overall default rate for GRAD borrowers, and how does that compare to other, non-GRAD borrowers (1f)? 7. Does your institution have a lending policy for financing community or group climate change activities? Y/N If Yes, please describe. (1h) 8. What GRAD initiated opportunities for income generation would you consider innovative for your GRAD borrowers? 9. How would you rank (1 to 5, with 1 being best) the following types of borrowers in terms of loan risks: VESA members _____ micro franchises____, FEMA VCs____, Cooperative members _______, agro-dealers____? (1f) 82 10. Are GRAD groups able to borrow from your institution for capital investments, such as irrigation equipment? If Yes, what are most common investments by GRAD beneficiaries? (2) 11. How have shocks such as drought affected borrowers risk and repayments of loans? (3) 12. Have GRADHH organizations borrowed for food processing equipment to make nutritious foods for home consumption and/or for sale.? (4) GRAD HH are too poor. “Colo” processing? Minimum investment to produce. 13. How many employees in your institution? ___ Approximately, how many are women? _____ How many women are in decision-making positions? _____ (5) 14. What GoE or other projects/programs support your institution? (6) Thank you for your participation in this interview. Your input is very valuable and we appreciate your willingness to take the time to share your thoughts and opinions! 83 SIG 7 PROMOTERS/ANIMATORS Final Template. Semi-structured Interview Guide (SIG) for Woreda/Kebele Level: Promoters/Animators FOR FG: Complete the table below. CASE _______________________ DATE & Interview Team LOCATION (Kabele, Woreda, Region) ORGANIZATIO N (OR GROUP) # Interview Participants # Members in GROUP M F M F FOR KII: Complete the table below. DATE & Interview Team LOCATION (Kabele, Woreda, Region) ORGANIZATION TITLE (OR ROLE) OF INTERVIEWEE M F Introduce yourself and those with you. Good morning/afternoon. My name is _____. I represent Social Impact, Inc. We are conducting a performance evaluation of the USAID-funded Graduation with Resilience to achieve Sustainable Development (GRAD) project. The purpose of the evaluation is to examine the results of GRAD and determine its effectiveness. We are very interested in your ideas and opinions about GRAD. While we will appreciate talking with you today we want you to know your participation today is based solely on your willingness to talk to us. You may stop or leave at any time. Your name will not be linked to this discussion and the information we receive from you will be confidential and used only in evaluating the GRAD project. The discussion will take 1 to 1.5 hours. We would like to write down notes today and ask if that is OK with you? Yes__ No__. Is it all right if we begin now? Yes_ No_. Thank you! 1. What does a promoter/animator do? How do you feel about being a promoter/animator? Probe: What are the benefits, since when, specific activities carried-out, with whom, what groups? (1c) 2. Do you think you have contributed to the graduation of households in your VESA? If so, how? (1a) 3. Have you and members of your group been able to access finance from MFI or your VESA for livelihood activities? How has this changed your/their lives? (1d) 4. What has GRAD done to help you be a good promoter? Probe: Training, other inputs, working side-by-side with DAs. Do you think you will continue to work with your group after the project closes? Why/why not? (1e) 5. What if any changes have occurred in input supply and then market linkages as a result of GRAD activity? Probe: What are these changes? (1f) 84 6. Was nutrition discussed in your group? If so, what do you think was the most helpful information received and from whom? Were there any problems with this? (1g) 7. We hear a lot about climate change. What do you think it is? Has it impacted on your community in the past? Have you learned ways to adapt to climate change? Probe: What? From whom? Did you take this information to your group? (1h) 8. Have you been involved with agro-dealers and/or micro-franchisers? How? Probe: Do you think these activities were successful or not? (1i) 9. What do you think resilience is? Do you think you are more resilient than you were before GRAD? Probe: Why? How many members in your group do you think are now resilient? (3) 10. For households who have graduated, what were the three most important things GRAD did to help them reach the level of graduation? (4) 11. Since GRAD, do you think there have been changes in how men and women make decisions about their livelihood? If changes, what are these changes? Probe: How did GRAD contribute to those changes? Describe specifics. (5) 12. Do you think women have more influence on decisions about production (and then assets) than they did before GRAD? What makes you think this/What proof do you have of this? (5) Thank you for your participation in this interview. Your input is very valuable and we appreciate your willingness to take the time to share your thoughts and opinions! 85 SIG 8 VESA/FEMA Coops Micro-Franchise VC Groups Final Template Semi-structured Interview Guide (SIG) FOR FG: Complete the table below. CASE _____________________ DATE & Interview Team LOCATION (Kabele, Woreda, Region) ORGANIZATION (OR GROUP) # Interview Participants # Members in GROUP M F M F Introduce yourself and those with you. Good morning/afternoon. My name is _____. I represent Social Impact, Inc. We are conducting a performance evaluation of the USAID-funded Graduation with Resilience to achieve Sustainable Development (GRAD) project. The purpose of the evaluation is to examine the results of GRAD and determine its effectiveness. We are very interested in your ideas and opinions about GRAD. While we will appreciate talking with you today we want you to know your participation today is based solely on your willingness to talk to us. You may stop or leave at any time. Your name will not be linked to this discussion and the information we receive from you will be confidential and used only in evaluating the GRAD project. The discussion will take about 2 hours. We would like to write down notes today and ask if that is OK with you? Yes__ No__. Is it all right if we begin now? Yes_ No_. Thank you! 1. What effect, if any has GRAD had on your food security (food availability, access, a/o utilization)? How many of your members have graduated from PSNP food support? (1a) 2. If we compare average annual household income in 2016 to 4 years ago, would you say it has increased? Y/N If yes, by approximately how many Birr for the whole year? (1b) 3. Which value chain(s) is/are your organization supported by GRAD? What other opportunities to earn income are supported by GRAD? (1c) 4. What access to finance/credit did your organization/group have before GRAD? How has GRAD affected your member's access to credit? Have funds been guaranteed? (1d) 5. What contact do you have with GoE extension services? Which ones are most helpful? How could they be improved? (1e) 6. For the crops and products your members sell, where do you sell them? Who are the main buyers? Do you sell individually or as a group? (1f) 7. What is the main food security problem your families face? What has GRAD done to help improve families’ nutrition? (1g) 8. What does climate change mean for your group? How has climate change affected your community and group members? How have they coped? Have you been involved in any climate change activities? Y/N If Yes, please describe. (1h) 9. What were new activities and interventions by GRAD for your members? How has it worked and benefited your group? (1i) 10. What were GRAD results in terms of households' assets? How have families' annual incomes changed and why? (2) 11. What shocks, e.g. draughts, have you experienced in the past 4 years? How are you responding/preparing for future shocks? (3) 12. What organizations have provided support to your households to help them become more food secure and to have better nutrition? How did GRAD help? [Scale 1 to 5?] (4) 86 13. How does your organization make decisions about the use of resources? What role do women have in making these decisions? How are men and women, girls and boys able to work together? What are the areas [gaps] that were improved by GRAD? That still need to be improved? (5) 14. What GoE or other projects/programs support your group? What are examples of GRAD collaborating with other project? How did they help to improve food availability, access, and utilization in households? Were you able to increase household incomes from GRAD supported activities. Several different organizations collaborated with CARE to implement GRAD; which ones supported your organization? What activities were most effective? Which ones should be improved in future projects? (6) Thank you for your participation in this interview. Your input is very valuable and we appreciate your willingness to take the time to share your thoughts and opinions! 87 SIG 9 KIIs and FGDs on CRISIS MODIFIER Final Template Semi-structured Interview Guide (SIG) BENEFICIARY_____________________________ SERVICE PROVIDER____________________________ SHAPE THE QUESTIONS ACCORDING TO WHETHER INTERVIEWEE IS BENEFICIARY OR SERVICE PROVIDER FOR KII: Complete the table below. CASE ___________________ DATE & Interview Team LOCATION (Kabele, Woreda, Region) ORGANIZATION TITLE (OR ROLE) OF INTERVIEWEE M F FOR FGD: Complete the table below. DATE & Interview Team LOCATION (Kabele, Woreda, Region) ORGANIZATI ON (OR GROUP) # Interview Participants # Members in GROUP M F M F Introduce yourself and those with you. Good morning/afternoon. My name is _____. I represent Social Impact, Inc. We are conducting a performance evaluation of the USAID-funded Graduation with Resilience to achieve Sustainable Development (GRAD) project. USAID has requested that the evaluation include an assessment of the Crisis Modifier (CM) implemented within GRAD. We are very interested in your experiences and opinions about the CM in your area. While we will appreciate talking with you today we want you to know your participation today is based solely on your willingness to talk to us. You may stop or leave at any time. Your name will not be linked to this discussion and the information we receive from you will be confidential and used only in evaluating the GRAD project. The discussion will take about 1 hour. We would like to write down notes today and ask if that is OK with you? Yes__ No__. Is it alright if we begin now? Yes_ No_. Thank you! 7. How did Crisis Modifier (or whatever it is called in the field) affect you (organization) as beneficiary OR as provider of CM services? Probe: How did your Crisis Modifier (CM) activities support emergency activities that addressed the risks and/or localized crisis that would otherwise hinder or derail the broader resilience-building objectives of GRAD? 7. What effect did your institution’s CM activities have on: a. saving lives and protecting livelihoods and b. protecting GRAD and HH investments in value chains (VCs) in income generating activities? 7. Your CM activities supported the protection and recovery of GRAD beneficiary communities in response to which of the following external crisis and shocks (check all that apply): a. drought or erratic rainfall_____ 88 b. flooding/heavy rain/hail_____ c. livestock diseases______ 7. The GRAD program identified six potential responses to the three major shocks. Please check all that your CM activities included: 1. Emergency Seed Distribution _____ _____ _____ _____ 2. De-Stocking/Re-Stocking of Livestock 3. Distribution of Emergency Animal Feed 4. Vaccination Campaigns and Veterinary Services 5. Cash for Work (CFW) for a. water harvesting purpose _____ b. flood control _____ _____ _____ _____ _____ c. market roads d. re-vegetation/re-forestation e. check-dam construction f. gully reclamation 6. Provide Bee Feed Substitute for honey VC _____ 5. What were the key considerations your CM applied in selecting the responses checked above, particularly in terms of: a) Areas of Potential Need among the GRAD woredas you were working in? b) Triggers and Target Population? c) CM Methods applied? 6. What GoE or other projects/programs support your CM activities? 6. What recommendations would you make to improve future CM Management? Thank you for your participation. Your input is very valuable and we appreciate your willingness to take the time to share your thoughts and opinions! 89 Annex 8: Coded Summary Sheets (SSs) for KIIs & FGDs with stakeholders SIG #: _________CASE #: ____0______ (North: 1 – 100; South 101 – 200) Date of KII:______11--16_________________ Target Group of KII/FGD: □ GoE (detail) _____________□IP □VESA □FEMA □Agro-Dealer □Micro￾Franchiser □MFI □Private Sector □Model Farmer/Promoter □Other Type Project Beneficiary □ Private Sector □ Donor □Other __________ Location: Kabele_______________Woreda_____________Region____________ Name(s) of Interviewer(s): ___________RW_______________________ Name (s) of Summary Sheet Authors: ____RW______________________ Informant(s) Gender: # Male ______________ # Female______________________ PUT NA IF NOT APPLICABLE TO THE KII/FGD. EQ1:Which of the following key technical areas of GRAD exhibited sizable results and which one(s) have not? a. Graduate 50,000 chronically food insecure HHs from PSNP food support in targeted Woredas. b. Targeted households’ income improvement to reach the $365 increase at the end of the life of activity. c. Participating HHs are organized into small groups (with typical GRAD group VESAs comprise of 20 to 30 HHs) and supported with technical value chain advice to take advantage of diversified livelihood opportunities. d. Access to finance for beneficiaries is increased to fund livelihoods activities; e.g. guarantee fund scheme in facilitating access to finance. e. Extension services are strengthened. f. Development of input supplies and market linkages. g. Nutrition interventions. h. Climate change adaptation activities. i. The contribution of GRAD ‘innovative’ interventions such as micro-franchising initiative and the agro-dealer strategy. EQ2: To what extent have the assets and income of beneficiary households (considering both the total value of assets and income, and the nature/composition of asset￾holdings and sources of income) changed over the course of the activity duration? ASSETS ● Types INCOMES ● On-farm ● Off-farm 90 OTHER EQ3: To what extent have beneficiary households become resilient during periods of shock? TYPES OF SHOCKS ● Drought ● Floods ● Late/early rains ● Death in family ● Sickness ● OTHER Reaction to shocks (before GRAD) Reaction to shocks (after GRAD) Other EQ4: What level of influence did the GRAD partners in the beneficiary households’ graduation process? BUILT ASSETS BUILT SOCIAL CAPITAL HELD ASPIRATION MEETINGS OTHER EQ5A: To what extent has this activity contributed to gender equity and women empowerment, specifically in addressing the role of gender in decision making on use of resources? GENDER EQUITY ● Respondent examples of: ● Other WOMEN’S EMPOWERMENT ● Examples ● Other DECISIONS ON USE OF RESOURCES ● Types ● Other EQ5B: To what extent has GRAD addressed gender gaps identified among women, men, girls, and boys? ● GENDER GAPS ● Which ones? Examples? ● Directly address ● Indirectly address ● Other EQ6A: The effectiveness of collaboration/complementarities with other Feed the Future Projects as part of testing the push-pull hypothesis. 91 MEANING OF PUSH-PULL ASSISTANCE FROM OTHER PROJECTS Type OTHER EQ6B: The effectiveness of project management through consortium arrangement and what lessons and experiences could be drawn? CONSORTIUM EFFECTIVENESS IN PROJECT MANAGEMENT STRENGTHS ● Joint implementation of training ● Joint planning ● Joint M&E ● Joint demonstrations ● Other ● WEAKNESSES ● LESSONS LEARNED ● EXPERIENCES ● OTHER 92 Annex 9: Crisis Modifier Summary Sheet The following completed sheet contains response by five Focus Group Discussions related to the Crisis Modifier in Ethiopia, implemented under USAID/Ethiopia’s GRAD activity. 1. How did Crisis Modifier affect you...? Tigray - Raya Azebo Cases 31, 33 & 37 (two FGDs done together) GRAD CFs organized 300 GRAD HHs into VESAs and helped them benefit from the GRAD-Crisis Modifier for the 2015 drought crisis. They helped these HHs to take out loans for sheep, fattened and sold to restaurants and urban consumers in Mekhoni town. Others raised chicken for sale to consumers. These coping mechanisms helped VESA-members maintain their previously owned assets and remain resilient during the 2015 drought shock and the 2016 re-adjustment of the PSNP￾initiatives. All HHs re-paid their initial loans on time. Male VESA participant stated that he was a GRAD beneficiary of Climate Change (CC) activities, since beginning of GRAD. “They learned to adapt to shocks”. The CM provided them with seeds, feed concentrate, and molasses. Group savings and petty trade promoted by GRAD and followed up by CFs and experts, plus intensive training. SNNPRS - ShebeDinno Case 118 & 119 ● After La Nina 2015 drought GRAD assisted with free seeds for second season crops: maize seed, haricot bean and potato, often intercropped. ● Seed distributed with financial assistance from the CM, and from European Commission (ECO) through a voucher system which respondent advises went very well. ● GRAD, along with a community task force, made up of kabele officials, DAs, made decision regarding who would receive shoats (young hogs). Of 19 kebeles in Shebe Dinno, selected beneficiaries in 11 kebeles were given shoats. 2. What effect did your institution’s CM activities have on: a. Saving lives and protecting livelihoods and b. Protecting GRAD and HH investments in value chains (VCs) in IGAs? 3. Your CM activities supported the protection and recovery of GRAD beneficiary communities in response to which of the following external crisis and shocks: RA (2) HZ - ZD Shebe Dinno a. drought or erratic rainfall X XXX b. flooding/heavy rain/hail c. livestock diseases 4. The GRAD program identified six potential responses to the three major shocks. Please check all that your CM activities included: RA - HZ - ZD Shebe Dinno 1. Emergency Seed Distribution _X_ _X_ _X_ __X__ ● Fast growing, short season crops, e.g. sorghum and Tef seeds ● Backyard gardens of cabbage and Swiss chard (new for them), given watering cans ● Moisture conserving activities, e.g. short season varieties ● Buy seeds with loan or savings from the BOA’s Farmer Training Center that also multiplies seed. ● Woman mentioned their “cooperative for women” that grows/sells sweet potato, papaya, and mangos. Agro-dealers channeled seed distribution in Shebe Dinno: a good example of knock-on effect of GRAD activity). _ 2 De-Stocking/Re-Stocking of Livestock __X___ ● Loans for animals and grain trading. ● Learned to minimize the # of animals (from 10-15 to 5-7 animals), because of draught and to adjust feed. X 93 _____ X ● Sept/October 2016, CM began a restocking program (shoats) for selected HHs who had been particularly hard-hit by the drought. 3. Distribution of Emergency Animal Feed _X__ X ● Feed concentrate, and molasses 4. Vaccination Campaigns and Veterinary Services None 5. Cash for Work (CFW) for a. water harvesting purpose b. flood control c. market roads d. re-vegetation/re-forestation e. check-dam construction f. gully reclamation Some benefited from Cash for Work (CFW) for water diversion trenches from highland and to collect and apply compost. Others said they graduated from taking cash, because activities were on, and benefited, individual farms. X X IP had not previously used voucher system so this provided an opportunity to learn about and use the system. Use of vouchers was very effective and catapulted the private dealers and the primary cooperatives that were providing seed. 6. Provide Bee Feed Substitute for honey VC __X __ Most participants are in livestock VC. Few raise bees and they got CM bee feed substitute. 5. What were the key considerations your CM applied in selecting the responses checked above, particularly in terms of: a) Areas of Potential Need among the GRAD Woredas you were working in? b) Triggers and Target Population? c) CM Methods applied? 6. What GoE or other projects/programs support your CM activities? In Tigray - Raya Azebo: ● Agro-forestry - 4 of 5 women participated as PSNP beneficiaries in annual reforestation with CC-friendly tree varieties, especially on hillsides. Men dug holes and women planted trees. ● GoE soil and water conservation campaign in January every year ● CM-provided improved stoves and solar lanterns 7. What (Challenges) recommendations would you make to improve future CM Management? ● “They started to change since the project began, and now graduated – where are we going?” ● There are many who need inclusion. ● They gave up time from a very busy day to participate in this FGD. And they appreciate the opportunity. They hope GRAD will continue. ● Distribution requires significant management. GRAD used CFs to manage the distribution activity, which significantly increased their workload. They also hired one extra CF, while that helped, additional staff time was needed. In future, plans should include increasing staff to correspond with increased workload. ● Finding seed was challenging because everyone was looking to purchase seed for distribution. The price increased. Finding shoats also has been a challenge. 94 Annex 10: Reference tables of GRAD survey results Table 16. GRAD quantitative and qualitative data collection Woreda/Region Household Survey KIIs FGDs Addis Ababa/Federal – 11 – Endamehoni/Tigray 418 13 10 Lay Gayint/Amhara 384 18 13 Ziway Dugda/Oromia 461 6 11 Hawassa Zuria/SNNPR 339 7 10 Totals 1,602 55 44 Table 17. Key result areas considered in GRAD's Results Framework Strategic Objective: Graduate 50,000 chronically food insecure households from PSNP food support in 16 targeted Woredas and increase each household’s income by $365 at the fifth year of the activity’s lifetime. Result 1 Enhanced Livelihood Options of Chronically Food Insecure Households in Highland Areas Result 2 Improved Household and Community Resilience Result 3 Strengthened Enabling Environment to Promote Scale-up and Sustainability Source: Evaluation SOW (Annex I) Note: Indicators from the GRAD PMEP were used by the evaluation team to inform survey design. Table 18. GRAD operational areas (regions and woredas) Region No. of Woredas Name of Woredas Tigray 4 Raya Azebo, Alamata, Offla, Enda Mehoni Amhara 2 Libo Kemkem, Lay Gayint Oromia 4 Zeway Dugda, Arsi Negelle, Shala, Adami Tulu SNNP 6 Meskan, Mareko, Loka Abaya, Hawela Tula, Hawassa Zuria, Shebedino Source: Evaluation SOW (Annex I). Table 19. Key informant interviews KIIs by Category # USAID, GoE, and other donor officers 6 GRAD implementing partner staff 30 Regional and Woreda representatives, public extension, Development Agents 6 Microfinance, cooperatives/unions, agro-dealers, and food processors 13 Total 55 Source: Qualitative Survey Results, November 2016 95 Table 20. Gender participation in KIIs and FGDs and members in their organizations Category Total M F % Female KII 55 48 7 13 FGD 41 186 128 41 21-VESAs, 16-VC Groups & 5-Coop members 42 222 161 37 Total 138 456 296 39 Source: November 2016 Endline Qualitative Interviews Results Table 21. GRAD's cumulative graduations, target and actual GRAD Year Cumulative Graduations Target Graduated % of Target 1 0 0 – 2 0 0 – 3 10,000 11,500 115 4 27,000 21,923 81 5 50,000 39,306 79 Source: November 2016 Endline Qualitative Interviews with CARE Senior Staff and secondary sources Table 22. Average household income by woreda, 2015–16 Woreda n Average income (USD) Lay Gayint 385 739 Endamehoni 418 1,083 Hawassa Zuria 339 539 Ziway Dugda 460 684 Total 1,602 771 Source: GRAD Endline Household Survey, November 2016 Note: n = number of households surveyed Table 23. Mean savings amounts reported by sample households in 2012 and 2016, in ETB Woreda Male-headed households Average increment at endline Female-headed households Average increment at endline 2012 n=1,079 2016 n=1,176 2012–16 2012 n=504 2016 n=426 2012–16 Endamehoni 494.14 7235.66 6741.52 815.46* 7298.94 6483.48 Lay Gayint 92.10 2573.30 2481.2 93.54 1180.07** 1086.53 Ziway Dugda 102.52 1606.42 1503.9 59.50* 816.76** 757.26 Hawassa Zuria 193.24 1022.75 829.51 129.51 632.79* 503.28 Total (mean) 194.38 2911.20 2716.82 416.20 3824.54 3408.34 Total households saved (n) 991 999 466 361 Source: GRAD Endline Household Survey, November 2016 Notes: * indicates savings differences between Male-headed households and Female-headed households were significant at p<0.05 in the same survey period. ** mean differences between Male-headed households and Female-headed households were significant at p<0.01in the same survey period. 96 Table 24. Mean amounts of loans households accessed in 2012 and 2016, by source Sources MALE-HEADED HOUSEHOLD Average increment at endline FEMALE-HEADED HOUSEHOLD Average increment at endline 2012 2016 2016–12 2012 2016 2016–12 MFIs 2486.53 4895.96 2409.43 2397.54 5776.66 3379.12 Bank 1000.00 - 2100.00 - RuSACCO 2997.00 4131.29 1134.29 1922.17 3288.64 1366.47 VESA 672.50 1573.08 900.58 200.00 1105.56* 905.56 Private 878.65 1580.14 701.49 530.06 896.76** 366.7 Other 117.55 2310.71 2193.16 74.19 865.00 790.81 Total mean 1311.88 3215.88 2409.43 1397.48 2386.91** 989.43 Total households borrowed (n) 457 580 195 210 Source: GRAD Endline Household Survey, November 2016 Notes: * indicates differences between Male-headed households and Female-headed households were significant at p<0.05 in the same survey period. ** indicates that differences between Male-headed households and Female-headed households were significant at p<0.01in the same survey period. Table 25. Percentage of households receiving training in 2015–16 by training type Woreda n Crop production % Livestock production % Financial literacy % Lay Gayint 384 71 46 30 Endamehoni 418 29 51 33 Hawassa Zuria 339 39 30 8 Ziway Dugda 461 67 62 42 Total 1,602 52 48 29 Source: GRAD Endline Household Survey, November 2016 Note: Households may receive more than one type of training, so row percentages may total to greater than 100 percent. Table 26. Mean household expenditures by gender and woreda, 2012 and 2016 (amounts in ETB) Woreda Male-headed households % Female￾headed households % 2012 2016 Change 2012 2016 Change Endamehoni 7,594 18,770 147 6,870 13,721 100 Lay Gayint 4,814 15,899 230 3,731 10,196 173 Ziway Dugda 8,511 13,770 62 6,651 10,641 60 Hawassa Zuria 11,086 11,150 1 8,998 6,043 -33 Total (mean) 7,862 14,750 88 6,524 11,055 69% Source: GRAD Endline Household Survey, November 2016 97 Table 27. Percent of households that borrowed from various sources by woreda, 2016 Credit Source 2016 Lay Gayint (n=384) Endamehoni n=418 Hawassa Zuria n=339 Zeway Dugda n=461 Total n=1602 MFIs 38.4 73.9 30.1 21.5 41.1 Bank 0.0 2.2 0.0 0.0 0.6 Rusacco 34.5 47.8 7.7 17.4 27.4 VESA 41.0 61.7 14.7 35.4 39.3 Private Money Lender 4.9 10.5 31.9 37.4 21.4 Other 0.8 3.8 0.0 2.8 2.0 Source: GRAD Endline Household Survey, November 2016 Note: Percentages can add up more than 100% as more than one finance source is possible. Table 28. Percent of households involved in Value Chain Activities by woreda, 2016 Value Chain Activities 2016 Lay Gayint n=317 Endamehoni n=399 Hawassa Zuria n=294 Zeway Dugda n=420 Total n=1430 Cattle fattening 7.9 29.1 21.1 17.4 19.3 Sheep fattening 64.0 69.4 20.4 73.1 59.2 Goat rearing 7.9 11.0 50.7 84.0 39.9 Malt barley production 57.7 0.3 0.3 3.3 13.9 Beans production 17.0 0.3 44.6 28.6 21.4 Apiculture 0.0 2.8 0.3 2.6 1.6 Vegetables 48.3 12.3 36.4 10.7 24.8 Source: GRAD Endline Household Survey, November 2016 Note: Percentages can add up more than 100% as more than one finance source is possible. Table 29. Mean annual value chain derived income by gender and woreda, 2016 (amounts in ETB) Woreda Male-headed households N=1176 Female￾headed households N=426 Combined N=1602 Endamehoni 7125.33 4153.25** 5817.04 Lay Gayint 8140.65 4165.42** 7405.64 Ziway Dugda 4038.71 2266.48** 3681.19 Hawassa Zuria 3454.29 2687.14 3275.71 Total Mean Income 5614.93 3473.29** 5045.43 Source: GRAD Endline Household Survey, November 2016 **The Mean differences of value chain income between male household and female household are significant at p<0.01. 98 Table 30. Average value of household-held livestock by woreda (amounts in ETB) Woreda Average Value, 2016 N=1602 Endamehoni 14152.77 Lay Gayint 18327.13 Ziway Dugda 12797.36 Hawassa Zuria 8876.40 Total Mean Value 13646.79 Source: GRAD Endline Household Survey, November 2016 Table 31. Average values of household assets by woreda (amounts in ETB) Woreda Average Value, 2016 N=1602 Endamehoni 1638.29 Lay Gayint 1660.28 Ziway Dugda 741.17 Hawassa Zuria 794.28 Total Mean Value 1206.80 Source: GRAD Endline Household Survey, November 2016 Table 32. Average households' productive assets value by woreda (amounts in ETB) Woreda Average Value, 2016 N=1602 Endamehoni 1552.01 Lay Gayint 903.94 Ziway Dugda 634.88 Hawassa Zuria 394.80 Total Mean Value 887.87 Source: GRAD Endline Household Survey, November 2016 Table 33. Mean annual household expenditures by gender and woreda in 2012 and 2016 (amounts in ETB) Woreda Male-headed households Female-headed households 2012 2016 2012 2016 Endamehoni 7,593.9 18770.35 6,870.3 13720.86 Lay Gayint 4814.4 15898.99 3731.2 10195.66 Ziway Dugda 8511.3 13770.27 6650.60 10641.42 Hawassa Zuria 11086.3 11150.27 8998.40 6043.17 Total (mean) 7862.3 14750.28 6524.3 11055.2 Source: GRAD Endline Household Survey, November 2016 99 Table 34. Average months food needs met between November 2015 to October 2016 Woreda Ave # of Months N* Lay Gayint 9.06 384 Endamehoni 10.69 418 Hawassa Zuria 8.32 339 Ziway Dugda 7.17 461 Total 8.79 1,602 Source: GRAD Endline Household Survey, November 2016 * N = number of households in endline survey 100 Annex 11: Regression Output Tables Note on how to read the Annex Tables: The tables in this annex document the regression output supporting findings that are referenced within the text, and the interpretation of specific tables can be found in those sections. The column headers show the dependent variables, while the row labels show the independent variables. Many of the tables present the same dependent variable as a function of a different combination of independent variables so that readers can understand how the significance of independent variables changes depending on the variables controlled for—for example, in one iteration we might see that which woreda a respondent is in has a significant correlation with their likely outcome, but when livelihoods shocks are added into the equation, we see that geography no longer has a statistically significant effect. In the tables that present the effect of woreda on the results, Lay Gayint is the comparison township and so is left out of the regression output (Lay Gayint is instead represented by the constant). Table 35. Diet diversity score Diet Diversity Score Diet Diversity Score Endline 1.369 (23.38) ** 1.248 (19.32) ** Endamehoni 0.410 (4.95) ** 0.357 (4.27) ** Hawassa Zuria -0.321 (3.70) ** -0.258 (2.93) ** Ziway Dugda 0.403 (4.98) ** 0.548 (6.30) ** Weather-related crop loss -0.353 (4.45) ** _cons 3.526 (53.12) ** 3.789 (42.64) ** R 2 0.17 0.18 N 3,185 3,184 *p<0.05; ** p,0.01 Table 36. Expenditures Expenditure in USD Expenditure in USD Endline 241.428 (11.41) ** 274.192 (12.04) ** Endamehoni 132.274 (4.08) ** 135.714 (4.23) ** Hawassa Zuria 22.969 (0.69) -6.928 (0.21) Ziway Dugda 39.583 (1.28) -26.016 (0.82) Gender of household head -133.252 (5.45) ** -77.799 (3.08) ** Number of household members 45.584 (7.70) ** _cons 339.823 (12.59) ** 82.182 (1.92) R 2 0.06 0.08 N 3,185 3,185 *p<0.05; **p,0.01 101 Table 37. Total number of months’ food needs met Total # Months food needs met (all months with a score of 1 or 2) Endamehoni 1.583 (8.24) ** Hawassa Zuria -0.182 (0.86) Ziway Dugda -0.877 (3.96) ** Income in USD 0.000 (2.06) * 3.A In the past year, has your household experienced any food or income related -1.460 (8.46) ** Constant 9.397 (59.93) ** R2 0.23 N 1,602 *p<0.05; ** p<0.01 N Table 38. Household graduation from PSNP 2.D Has your household already graduated from PSNP? Endamenhoni -0.184 (5.44) ** Hawassa Zuria 0.314 (8.66) ** Ziway Dugda 0.136 (3.56) ** Gender of Household Head -0.142 (5.24) ** 3.A In the past year, has your household experienced any food or income related crop loss? -0.0.95 (3.22) ** 2.D2 Are you or any member of the HH engaged in at least one new Value Chain activity? 0.138 (3.63) ** Constant 0.438 (10.66) ** R2 0.14 1,602 *p<0.05: ** p<0.01 Table 39. Household hunger scale Household Hunger Scale Income in USD -0.000 (2.36) * Constant 0.143 (9.38) ** R2 0.00 N 1,602 102 *p<0.05; **p<0.01 Table 40. Household income Income in USD Income in USD 2.D Has your household already graduated from PSNP? 125.232 (2.34) ** 103.768 (1.94) Endamehoni 368.771 (5.12) ** 365.772 (5.11) ** Hawassa Zuria -235.692 (3.08) ** -124.337 (1.55) Ziway Dugda -65.626 (0.94) 134.288 (1.63) 3.A In the past year, has your household experienced any food or income related crop loss due to weather? -285.235 (4.47) ** Constant 676.812 (11.69) ** 771.425 (12.57) ** R2 0.04 0.05 N 1,602 1,602 *p,0.05; **p<0.01 Table 41. Total household savings Total Savings in USD Gender of Household Head 32.323 (1.05) Endamehoni 134.276 (3.63) ** Hawassa Zuria -39.952 (0.93) Ziway Dugda -30.353 (0.90) Income in USD 0.200 (16.20) ** Constant -50.771 (1.87) R2 0.20 N 1,3620 *p<0.05; **p<0.01 Table 42. Number of months with all food needs met Total number of months with all food needs met Total number of months with a food needs met Experienced weather￾related crop loss -1.375 (7.50) ** -1.308 (7.38) ** Endamehoni 1.658 (8.60) ** 2.294 (9.94) ** Hawassa Zuria -0.162 0.292 103 (0.74) (1.17) Ziway Dugda -0.827 (3.48) ** -0.721 (2.61) ** Participated in agriculture VC activity 0.879 (1.53) 0.149 (0.27) Participated in cattle fattening VC activity 0.478 (2.65) ** Participated in sheep fattening VC activity 0.828 (5.63) ** Participated in goat fattening VC activity 0.600 (3.43) ** Participated in malt barley VC activity 1.072 (4.06) ** Participated in bean production VC activity 0.488 (2.69) ** Participated in vegetable VC activity 0.998 (5.69) ** _cons 9.328 (65.30) ** 7.831 (39.48) ** R2 0.22 0.28 N 1,602 1,602 * p<0.05; ** p<0.01 104 Annex 12: Maps of the GRAD activity survey woredas Source: Resilience Assessment for the Graduation with Resilience to Achieve Sustainable Development (GRAD) Project, May 2016. Note: The November 2016 GRAD endline survey was held in Endamehoni Woreda, while the CM assessment was held in Raya Azebo (both areas highlighted in red above) in the Tigray Region. Source: Resilience Assessment for the Graduation with Resilience to Achieve Sustainable Development (GRAD) Project, May 2016. Note: The November 2016 GRAD endline survey was held in Lay Gayint Woreda (highlighted in red above) in the Amhara Region. Source: Resilience Assessment for the Graduation with Resilience to Achieve Sustainable Development (GRAD) Project, May 2016. Note: The November 2016 GRAD endline survey was held in Ziway Dugda Woreda (highlighted in red above) in the Oromia Region. Source: Resilience Assessment for the Graduation with Resilience to Achieve Sustainable Development (GRAD) Project. May 2016. Note: The November 2016 GRAD endline survey was held in Hawassa Zuria Woreda, while the CM assessment was held in Shebe Dino Woreda (both areas highlighted in red above). 105 Annex 13: GRAD Final Performance Evaluation Team CVs 1. Tadesse Zerihun, Team Leader 2. Vickie A. Sigman, International Livelihoods & Food Security Specialist 3. Raymond Waldron, International Expert 4. Raya Abagodu, Local M&E Specialist 5. Kedir Nuri, Local Expert 6. Mazengia Abera Birra, Logistician Tadesse Zerihun - Team Leader Dr. Tadesse Zerihun is an agricultural economics professional with more than 20 years of experience in international development. Dr. Zerihun has extensive regional experience in African and Middle Eastern agricultural development projects, including project design, implementation, and evaluation. Dr. Zerihun specializes in agribusiness value chain analysis and smallholder livelihoods, excelling in the facilitation of multi-stakeholder projects and promotion of capacity building in teams and projects. Dr. Zerihun holds a Ph.D. in Agricultural Economics from Swedish University of Agricultural Sciences and is fluent in Amharic and English. PROFESSIONAL EXPERIENCE 2016 Institutional Capacity Building Advisor, International City/County Management Association (ICMA), USAID-Capacity Building & Change Management Program II – (CBCMP-II) Ministry of Agriculture, Irrigation & Livestock (MAIL) (Afghanistan) ● Provided technical advice on and facilitated in building and promoting the overall project team’s capacity to work in a technology (IT) driven, learning environment to expand successful project activities and communicating best practices to relevant parties within 11 of the MAIL Directorates. ● Provided technical Advice to a team of senior technical Change Management Specialists (CMSs) within the project and coordinated various Technical Working Groups representing the 11-MAIL Directorates and technical partners within the USAID agricultural unite. ● Facilitated the project initiatives for implementing standard operating procedures and work across the implementation teams to ensure project objectives and targets are met while promoting the capacity building and learning efforts. 2015-2016 Rural Development Program Senior Advisor, Swedish Committee for Afghanistan (SCA) (Afghanistan) ● Mapped villages and districts of each project’s operations and cross comparison of different programmatic presence in different geographic locations; ● Conducted community Based Sustainability Study and Analysis (covering identification and analysis of degree of presence or absence of critical factors that impact on sustained benefits delivery to targeted villagers; ● Strengthened potentially viable local institutions through providing support for function-based and result￾oriented organizational change management processes for facilitating CSOs and CBOs at community level. ● Supported Community Governance and Livelihoods Projects designed for demand-driven vocational and employment skills training programs for the rural youth and women, including carrying out assessments of occupational standards and facilitating the development of short-term training schemes. 2014 Contract Consultant, PNG Sustainable Development Program (Papa New Guinea) ● Reviewed progress and development reports and provided feedback on final completion report & manuals. ● Edited project proposal drafts prepared for the Programs’ inputs quality control. ● Edited papers prepared for the PNG Sustainable Development Program based on M4P approach ● Recommended procedural changes for improved SME-performance and greater profitability. 106 2014 Team Leader, Agricultural Extension and Nutrition Assessment, Modernizing Extension and Advisory Services (MEAS), USAID/Malawi (Malawi) ● Led and technically-guided an interdisciplinary team of five (expatriates and nationals) to collaboratively assess the effectiveness and capacity of Malawi’s agricultural extension, nutrition education, and integrated agriculture-nutrition outreach systems across public, private, and civil society sector service providers ● Developed the assessment methodology and guided design of instruments ● Managed and participated in data collection and analysis, and led the writing of a comprehensive report detailing results, including results of assessing related Information and Communication Technology (ICT) initiatives and opportunities 2012-2013Team Leader Institutional Development Project at Ministry of Agriculture, Irrigation, and Livestock (MAIL), GRM￾International(Afghanistan) ● Facilitated initiatives that Improve rural and agricultural conditions by turning knowledge into practice through developing agricultural reform policies for Ministry of Agriculture Irrigation & Livestock (MAIL) agencies. ● Managed a pool of international technical experts working on MAIL’s reform plans and proposals as part of the Organizational Development Framework. ● Managed the M&E team at MAIL in terms of supporting departmental and directorate heads for effective organizational development resources’ use and specific inputs as required. ● Designed and implemented projects pertinent to enhancing Irrigated (Kareez) Agriculture, Rangeland Management, Adaptation to Climate Change in Agriculture, Soil and Water Management, and Pastoral Livelihoods in arid- and semi-arid Kuchi pastoral areas within Central and Southern Afghanistan. 2011 Economic Growth & Employment Expert, UN-FAO Iraq. (Jordan) ● Identified, analyzed, and proposed key points for policy formulation processes, and priority strategies, especially as related to new donor programs, that will allow the operational offices to meet regulatory oversights. ● Involved in the development and dissemination of the Logical Framework for project cycle management. ● Produced mid-term project evaluation report covering the value chain analysis, mapping out entry points; assessing value chain governance; and making framework for agribusiness firm capabilities; and prepared training manual for designing and implementing projects pertaining to employment venues in Iraq. 2010 Food Security Officer / Team Leader / of Food Security Unit, UN-FAO-Nairobi, Kenya ● Evaluated the relevance, effectiveness, efficiency, sustainability, impact, coherence, and value addition of agricultural products processing enterprises and firm capabilities of selected projects. ● Designed projects that integrate marginalized farmers into mainstream agricultural markets based on village-level organization and sector development, and evaluation of research-based organizations in Nairobi. ● Designed and implemented projects pertinent to enhancing Irrigated Agriculture, Rangeland Management, Adaptation to Climate Change in Agriculture, Soil and Water Management, and Pastoral Livelihoods in arid￾and semi-arid pastoral areas of Central and Northern Kenya. ● Facilitated capacity building of small-scale farmers & pastoral communities on: (i) Integrated Food Security Phase Classification (IPC) protocol; and (ii) in livestock restocking and drought resistant seeds multiplication. 107 2008-2009 Senior Consultant - Food Security and Livelihoods Program Evaluation, GIFITA Development Studies Ltd (Kenya) ● Developed a new strategy based on M4P-approach with an emphasis on small-scale agribusinesses. Value chains stimulating systemic change in key sectors’ staff capacity building. ● Worked on the design, management and monitoring of cash-based interventions within Northern Kenya. ● Made M&E on managing the agricultural and food security sector by following the ‘SWOT Approach’; reviewed the Strengths, Weaknesses, Opportunities and Threats facing the agricultural sector; and the coffee growers’ capabilities in terms of production, processing and marketing. ADDITIONAL SELECTED ROLES: ● Team Leader /Regional Coordinator, Cardno-Emerging Markets Ltd. UK, EU - funded Northern Uganda Rehabilitation Program (NUREP), Uganda (2007-2008) ● Team Leader, Cardno-Emerging Markets Ltd. UK, EU - funded Small-Scale Projects (SSPs), Liberia (2004- 2006) ● Early Warning Data Analyst & Emergency Program Coordinator, Save the Children USA, Ethiopia (2003) EDUCATION Ph.D. Agricultural Economics, Swedish University of Agricultural Sciences, Uppsala, Sweden, 1996 M.S. Agricultural Economics, Swedish University of Agricultural Sciences, Uppsala, Sweden, 1983 B.A. International Trade and Economics, Stockholm University, Sweden, 1979 LANGUAGE English – Fluent, Amharic – Fluent, Swedish – Professional Proficiency 108 Vickie A. Sigman - International Livelihoods & Food Security Specialist Dr. Vickie Sigman is an agriculture and extension education professional with more than 30 years of experience in international development. Beginning as a Peace Corps Volunteer in Guatemala, Dr. Sigman has since worked in diverse agricultural development programs and projects in various countries, primarily in Africa but also in Indonesia, Iraq, Afghanistan, Tajikistan, and Guatemala. Over the past 30 plus years her work has focused on improving food security and agriculturally-based livelihoods. Dr. Sigman has provided short-term technical and management assistance to several USAID Missions in Africa including revising USAID/Nigeria’s Food Security Strategy. Dr. Sigman holds a Ph.D. in International Agricultural Extension Education from University of Illinois. PROFESSIONAL EXPERIENCE 2015-2016 Senior Agricultural Extension Advisor, Modernizing Extension and Advisory Services (MEAS), University of Illinois (Guatemala) ● Researched and wrote a technical recommendation paper to inform USAID/Mozambique decisions on FTF agricultural innovation systems (AIS) project design Participated in implementation of field-based data collection ● Analyzed extension efforts of the Buena MILPA project, a USAID Feed the Future (FTF) food security project through key informant interviews and secondary research 2015 Senior Agricultural Extension Policy Specialist, Modernizing Extension and Advisory Services (MEAS), University of Illinois (Ghana) ● Designed, led the implementation team, and reported on a high-level event “Ghana Agricultural Extension Policy Forum” to review Ghana’s existing Agricultural Extension Policy (2005) ● Convened by partner Ministry of Food and Agriculture, Directorate of Agricultural Extension Services (DAES), the Forum was attend by over 60 senior stakeholders from the public, private, and civil society sectors. Forum deliberations resulted in (1) consensus that DAES would lead the policy revision process; (2) key recommendations to guide the process; and (3) establishment of two volunteer groups to champion moving the process forward 2014 Senior Agricultural Development Specialist, USAID Africa Bureau Surge Team/University of Missouri. USAID/West Africa (Ghana) ● Provided short-term technical assistance to the Economic Growth Office, USAID/West Africa. Analyzed and substantially revised the design of a five-year agricultural research program to be implemented by the West African Council for Research and Development ● Aligned the program with USAID emphasis on scaling-up and with continental and regional initiatives, institutions, and policies (e.g., Comprehensive Africa Agriculture Development Program, Economic Community of West African States [ECOWAS] and its Agricultural Policy) ● Wrote the various procurement documents, including the Project Description, to move the new program forward to award. Advised on the work plan for the ECOWAS Joint Sector Review 2014 Team Leader, Agricultural Extension and Nutrition Assessment, Modernizing Extension and Advisory Services (MEAS), USAID/Malawi (Malawi) ● Led and technically-guided an interdisciplinary team of five (expatriates and nationals) to collaboratively assess the effectiveness and capacity of Malawi’s agricultural extension, nutrition education, and integrated agriculture-nutrition outreach systems across public, private, and civil society sector service providers ● Developed the assessment methodology and guided design of instruments ● Managed and participated in data collection and analysis, and led the writing of a comprehensive report 109 detailing results, including results of assessing related Information and Communication Technology (ICT) initiatives and opportunities 2013 Senior Agricultural Extension Specialist, MEAS/Farmer Advisory Services Tajikistan (FAST)/USAID (USA, Tajikistan) ● Reviewed an existing concept, revised, and wrote-up the overall design of a food-security and livelihoods￾focused agricultural extension and advisory services (EAS) system to reach approximately 30,000 smallholder farm families in Tajikistan’s USAID FTF Zone of Influence ● Designed and managed pre-testing of EAS system components ● Wrote an action work plan detailing activities required to move EAS system forward 2013 Acting Team Leader, USAID Africa Bureau Surge Team/University of Missouri. USAID/Mali ● Provided short-term technical assistance to the Accelerated Economic Growth Office, USAID/Mali. Guided and managed office activities including building sense of team spirit, developing and implementing tools to track progress/problems/needed actions, responding to Regional Inspector General audit, moving existing food security project procurements forward, providing input into project impact evaluation, supporting various budget and financial analyses, and advising on office staffing patterns 2012 Agriculture Consultant, USAID Africa Bureau Surge Team/University of Missouri. USAID/Nigeria ● Led and facilitated a Ministry-appointed team of senior MOA Headquarter staff and representatives of USAID/Liberia and USAID agriculture project contractors in the development of Liberia’s first National Agricultural Extension and Advisory Services Policy ● Coached the team in policy development; wrote various drafts for team review and discussion; obtained input from district and county-level MOA staff; designed a Stakeholders Workshop to acquire additional input to the policy; and finalized the Policy for submission to the Legislature 2012 Agriculture Project Design Consultant, USAID Africa Bureau Surge Team/University of Missouri. USAID/Mali and Home-Based ● Contributed expertise, experience, and advice to the team process of writing a Project Appraisal Document (PAD) for a large-scale Cereals Value Chain Project for USAID/Mali. In Mali: reviewed and analyzed documents and data, consulted with USAID staff and other stakeholders, and drafted components of the PAD, including those related to background, extension, results framework, and indicators ADDITIONAL SELECTED ROLES: ● Agriculture and Enterprise Development Advisor, USAID /Liberia, Liberia (2009-2011) ● Community Development Team Leader/Sr. Food Security Specialist, Agriculture and Rural Development, Inc. (TetraTech/ARD)/USAID. Local Governance & Community Development Project. Afghanistan (2008- 2009) ● Evaluation Specialist, Asian Development Bank, Afghanistan (2006-2007) ● Chief of Party, Center for International Education/USAID, Afghanistan (2005-2006) EDUCATION Ph.D. International Agricultural and Extension Education, University of Illinois, IL, 1984 M.S. International Agricultural and Extension Education, University of Illinois, IL, 1980 B.A. Community Development, University of Massachusetts LANGUAGE English – Fluent, Bahasa Indonesian – Working Proficiency, Spanish – Working Proficiency 110 Raymond Waldron - International Expert Mr. Raymond Waldron is a monitoring and evaluation professional with more than 30 years of experience in international development. Mr. Waldron’s roles have involved an increasing level of responsibility, and he has served in leadership roles such as Chief of Party, Senior Advisor, or Senior Evaluator. While much of his experience has been concentrated in Latin America and the Caribbean (LAC), Mr. Waldron also has significant experience in Africa, Asia, and the Middle East. Mr. Waldron has extensive experience working with USAID in a variety of capacities and on projects and evaluations in a variety of sectors, especially those with a focus on food security and agriculture. Mr. Waldron holds an M.Sc. degree in Resource Economics /Statistics from Oregon State University in Corvallis, OR. PROFESSIONAL EXPERIENCE 2016- Agricultural and Evaluation Consultant, The International Solutions Group, Evaluation of Title II LAUNCH Program, ACDI/VOCA (Liberia) ● Assisted with document review and writing of inception report with research tools ● Participated in implementation of field-based data collection ● Worked on data analysis and report preparation 2014-2015 Senior Advisor/Coordinator, Mission Director’s Office, USAID/Guatemala, Various Implementers (Guatemala) ● Provided strategic, policy and technical leadership advice to the Mission Director, Technical Office Directors, AORs/CORs and COPs of over 20 implementing partners and indirectly 50 subs to improve implementation of the Mission’s $260 million Western Highlands Integrated Program (WHIP). ● Advised Office Directors and their AOR/CORs on mid-term performance and impact baseline evaluations for WHIP, the Title II Food for Peace mid-term evaluation, and the Health communications Capacity Collaborative Activity and various other regionally funded activities ● Worked tasked with data collection across economic growth, environment, health, education, and democracy under an M&E program ● Revived and guided an executive committee of IPs to improve communications, coordination, and integration of activities and lasting results. ● Facilitated conference calls on program integration ● Consulted with GoE of Guatemala officials, other donors, and Heads of USG Agencies who were stakeholders in reducing poverty and malnutrition in the Western Highlands. 2013- Chief of Party, DevTech Systems, Inc., Colombia (2013) ● Directed the Monitoring and Evaluation Program (MEP) that supported USAID/Colombia management of four Development Objectives; a portfolio with annual obligations over $150 million implemented through 24 programs/projects including over 2,500 activities carried out by 20 Implementing Partners (IPs). ● Organized a Data Quality Standards workshop for USAID/Colombia ● Oversaw meetings with and trainings of stakeholders in M&E, field visits to verify reported indicator and outcome results, and verification of compliance with Environmental Management Plans. ● Coordinated with Mission’s Program and Technical offices to harmonize PMP and M&E Plans and encourage high quality reports for evidence-based decision-making 2012 Senior Evaluator, The QED Group, LLC. Armenia ● Led an external evaluation team for USDA’s Caucasus Agricultural Development Initiative in Armenia, approximately $5 million per year ● Reviewed available baseline data ● Organized focus groups of beneficiaries and conducted key informant interviews ● Analyzed results in terms of benefit/cost, cost recovery, sustainability and environmental effects ● Presented findings and recommendations to US Ambassador in Armenia, USAID/Armenia Director, and 111 Senior USDA Officials in Washington, D.C. 2011 Food Security and Natural Resources Consultant, Guatemalan Cooperative Federations and NGO Consortiums, USAID Feed the Future/USDA Food for Progress Proposals ● Led large NGO staff teams in design of Feed the Future (FTF) and Food for Peace proposals in Guatemala, providing insights for effective monitoring and evaluation that respond to USG’s Data Quality Standards ● Wrote Environmental Compliance, EMMP sections for proposals in Africa ● Proposed various methods of measurement and evaluation to capture rural household productivity, consumption/nutrition and health data 2010-2011 Chief of Party, Agro-dealers Strengthening Programs, CNFA, Zimbabwe ● Led startup and initial implementation of $4 million FTF program with 7 local staff, 10 Farmer To Farmer Volunteers, to train and assist 250 village Agro-dealers introduce improved farming methods and inputs ● Mentored M&E, extension, and capacity building staff in work-plan revisions, best practices for value chain development and timely reporting against the PMP ● Recruited/mentored M&E Specialist in baseline, BDS, training, and design of demonstration plots for implementation/evaluation ● Presented preliminary results from data analyses that compared progress against baseline levels to USAID and wider donor community concerned with food security ● Supervised and supported with FtF Volunteers baseline and quantitative-data reports ● Conducted 1st annual reviews of Agency’s performance monitoring indicators and data quality assessment (DQA). Involved staff in USAID PERSUAP certification training ADDITIONAL SELECTED ROLES: ● Chief of Party, Agro-dealers Strengthening Programs, CNFA, Mali (2010) ● Chief of Party, MarChE, CNFA, Haiti (2009–2010) ● Lead Agribusiness/Food Security Consultant, FINTRAC in Guatemala, Nepal, and PADF in Haiti (2007–2009) ● Assistant Mission Director, USAID/Ecuador (2005-2007) ● Office Director, Democracy, Governance & Conflict Prevention, USAID/Egypt (1997-1999) ● Agricultural Development Officer, USAID/Egypt (1994-1997) ● Agricultural Development Office Chief, USAID/Guatemala & ROCAP/Costa Rica (1991-1994) ● Agricultural Development Division Chief, USAID/Peru (1983-1988) ● Agricultural Economist and Food for Peace Officer, USAID/Niger, USAID/Sudan and USAID/Washington D.C. (1977-1983) EDUCATION M.Sc. Resource Economics/Statistics, Oregon State University, Corvallis, OR, 1976 B.A. Biology/Spanish/ SUNY, Brockport, NY, 1969 Cert. Foreign Affairs Counter Threat -Dipl. Security and Foreign Service Institute 2016 Cert. Exec. Leadership - Federal Executive Institute, 2003 Cert. Agribusiness - Harvard Business School, Cambridge, MA, 2001 Cert. Economic Development - American University, Washington, DC, 1990 LANGUAGE English – Fluent, Spanish – Professional Proficiency. 112 Raya Abagodu - Senior Local M&E Specialist Mr. Raya Abagodu is an independent consultant with more than 30 years of experience working in the public sector, civil society, and donor community in Ethiopia and other countries in East Africa. Mr. Abagodu has worked in a variety of sectors in international development, with much of his work concentrated in agriculture, food security and livelihoods, and nutrition￾based projects. Mr. Abagodu has experience as a Team Leader, Evaluation Specialist, and other senior consulting roles on performance evaluations, qualitative assessments, and impact evaluations. Mr. Abagodu holds an M.Sc. degree in Agricultural Economics from Alemaya University, and is fluent in Amharic, English, and Oromoiffa. PROFESSIONAL EXPERIENCE Jun-Aug 2016 Evaluation Expert, IFC/WBG with Economist Associates, Mid-Term Review of the Ethiopia Investment Climate Program, Phase I (Ethiopia) ● Desk research and qualitative methods (KII, FGD and observations) to review and validate the status of implementation of the Program through an assessment of the extent of achieving the program outcomes and, whenever feasible, impacts; ● Formulated mid-course corrective measures and recommend actions for an effective implementation and goal achievement. May-Aug 2016 Senior Consultant M&E, Review of the UK’s Climate High-Level Investment Program (CHIP), (Ethiopia) ● Supported the Climate Resilient Green Economy (CRGE) of Ethiopia. Qualitative assessment through the collection of more in-depth feedback from regional, Woreda and kebele level GoE staff ● Undertook data collection to allow for in-depth analysis of the CRGE plan and its implementation that enable third party verification of some of the results reported by the CRGE Facility, etc. Dec 2015-Jun 2016 Team Member &Evaluation Expert, UNICEF-UNFPA, Ethiopia ● Inception phase of the Roadmap to realize the London Girl’s Summit Commitment to end Child marriage and FGM/C by 2025-Ethiopia. ● Design the National MEL Framework and Evaluation Plan of the inception through extensive desk research of documents including GoE of Ethiopia policy, legal framework and the national policy on HTP, ● Assisted with the qualitative study and interviews of stakeholders. Sep-Nov 2015 Team Leader, USAID/DFATD, Comparative Study on GRAD/FSF. Social Impact, Ethiopia ● Led the qualitative assessment of the approaches and strategies of the projects in helping target beneficiaries achieve food security and graduate from the PSNP program Jun-Jul 2015Evaluation Consultant, Country Strategy and Program Evaluation, AfDB, Ethiopia ● Assessed the development results of AfDB’s key interventions in Ethiopia and, in particular, to what extent interventions have made a difference and why or why not; and to suggest lessons and potential improvements, to feed into the development of the new Country Strategy Paper (CSP). ADDITIONAL SELECTED ROLES: ● Senior Evaluation Expert, Process Evaluation of the Promoting Basic Services Phase 3 Program (Jun-Sept 2015) ● Evaluation Consultant, Horn of Africa Leadership and Learning for Action (HOLLA) (Jan 2015-Present) ● Team Leader, Food and Water Security Strategy, World Vision/ICRAF (Aug-Sept 2014) ● Evaluation Consultant and Team Member, Evaluation of the Dutch Multi-Sector Support and Food Security Program, ECORYS & MMA Consultancy (Jan-Feb 2014) ● Consultant/Coordinator of Impact Evaluation, Impact Evaluation of the USAID-GCC Peace Center Climate and Social Resilient Project, (Dec 2012- Present) ● Assessor and Design Expert of DRM & PSNP program "Assessment of Ethiopia's Productive Safety Net Program (PSNP) and the Household Asset Building Program (HABP): Climate Smart Initiative Building Food 113 Security through Climate Resilience.", World Bank, (Oct 2012- Feb 2013) ● Evaluator, Evaluation of the Humanitarian Emergency Response to the Horn of Africa Crisis, Save the Children USA, (Jul-Aug 2012) ● Evaluator, Final Evaluation of ACT Alliance Emergency Response to Horn of Africa Crisis, ACT Alliance, Mar￾Apr 2012 ● Principal Evaluation consultant, Disaster Risk Management Program Final Evaluation, ACF-UK, Dec 2011- Jan 2012 ● Senior Reviewer/Evaluator, Mid-term Review of the Pastoral Community Development Project, World Bank, Mar-Jun 2011 EDUCATION M.Sc. Agricultural Economics - Alemaya University, Ethiopia, 1991 B.Sc. Agricultural Economics - Addis Ababa University, Addis Ababa, Ethiopia, 1982 LANGUAGE Amharic – Fluent, Oromoiffa- Fluent, English- Fluent. 114 Kedir Nuri - Senior Local Expert Mr. Kedir Nuri is an agricultural science professional with more than 30 years of experience in agriculture and international development. Mr. Nuri has extensive experience as a consultant for international development agencies in Ethiopia, serving as Team Leader, M&E specialist, and Trainor. Additionally, Mr. Nuri is a recognized Ethiopian expert in private sector development, as well as gender program evaluation, research, and training. Mr. Nuri holds a Master’s degree in Development Economics from University of Queensland, Australia and is fluent in Amharic, Guaragigna, and English. PROFESSIONAL EXPERIENCE 2015 Team Leader, End-line Impact Evaluation, SNV-Ethiopia Led a team of experts for an Endline Impact Evaluation of Gender and Environment Responsive Beekeeping Project in three Districts in Oromia and Gambella Regions. 2015 Lead Training Facilitator, PACT-Ethiopia ● Developed manual on Extractive Industries Transparency Initiative, EITI, and facilitated three-day training programs for each of two batches of senior experts drawn CSOs experts and MSG members. Sponsored by the World Bank through Pact Ethiopia 2015 National Expert, EU/EDF-Ethiopia ● Supported the Ministry of Women, Children and Youth Affairs and Regional Authorities in Ethiopia to start up activities in the enhancement of women’s economic status in Ethiopia. ● Conducted a trend analysis on skills gaps and market opportunities for six towns in the four target regions (Oromiya, Benishangul Gumuz, Gambella and SNNP Regions). 2014 Team Leader, Sustainable Agriculture and Food Security Enhancement through Integrated Recovery Support Mechanisms project, Vita/RTI (Ethiopia) ● Conducted a Baseline Survey for the Sustainable Agriculture and Food Security Enhancement through Integrated Recovery Support Mechanisms project implemented in five Districts in SNNPR. It is a three-year initiative funded by European Union. ● Led a survey team and successfully documented benchmark situation in the five target districts.2012. 2014Team Leader, Baseline Impact Evaluation, SNV-Ethiopia ● Led a team of experts for a Baseline Impact Evaluation of Gender and Environment Responsive Beekeeping Project in three Districts in Oromia and Gambella Regions. 2013 External Consultant, Norwegian Church Aid-Ethiopia ● Conducted an Organizational Capacity Assessment of Inter-Religious Council of Ethiopia, jointly sponsored by NCA and Life and Peace Institute: ● Provided service as an advisor on partner assessment of the IRCE; Wrote the final assessment report. 2013 Team Leader, Final/Endline Evaluation of the Pact Metebaber - Project, PACT-Ethiopia ● Led the Final/End-line Evaluation of the Pact Metebaber Project - Improving the Wellbeing of Women and Girls in Ethiopia. The terminal evaluation covered six-selected sample districts drawn from Afar, Amhara, Oromia and SNNP Regions and Addis Ababa City Administration. 2012National Consultant, Support for Rural Women Entrepreneurs, World Bank-Ethiopia ● Prepared a Concept note for grant program based on viable SMEs identified by rural women entrepreneurs 115 in Oromia, Amhara. SNNP and Tigray Regions. The needs assessment covered several districts selected from the four regions of Ethiopia. Sponsored by World Bank. ADDITIONAL SELECTED ROLES: ● Team Leader, Impact Evaluation of Community Based Tourism, Irish Aid Ethiopia (2012) ● Evaluation and Survey Expert, Hifab International, Ethiopia (2011) ● Team Leader, Save the Children, UK, Ethiopia (2010) ● Team Leader, World Learning Ethiopia (2010) EDUCATION M.A. Development Economics, University of Queensland, Australia, 1991 B.A. Economics, Addis Ababa University, Ethiopia 1984 LANGUAGE English – Fluent, Amharic, Fluent, Guragigna, Mother Tongue 116 Mazengia Abera Birra - Logistician Mr. Mazengia Birra is a public health professional with more than 5 years of experience in international development. Mr. Birra has provided logistical and research support to evaluation and assessment teams for clients such as USAID, DFID, and UNICEF. Mr. Birra specializes in heath and gender projects, with additional experience in agriculture and livelihood evaluation. Mr. Birra holds a Master’s degree in Public Health (MPH) from Addis Continental Institute of Public Health, as well as a MA in Social Psychology from Addis Ababa University and is fluent in Amharic, Afan Oromo, and English. PROFESSIONAL EXPERIENCE 2013- Independent Research and M&E Consultant ● Worked on over 25 international development program or evaluation teams, including: ● Served as Data Collector for USAID’s Respectful Maternity Care project, under Jhepigo-Ethiopia ● Coordinated a formative research study and KAP survey for UNICEF’s De-worming & Adolescent Girls' Nutrition, under PRIN International Consultancy & Research ● Served as Qualitative Researcher for USAID’s Economic Strengthening as an Integral Part of HIV Prevention, Care and Treatment: Case of Highly Vulnerable Children &Households, under FHI 360 ● Supervised the monitoring and survey work for SNV’s Apiculture Scaling-up Program for Income and Rural Employment (ASPIRE), under PRIN International Research & Consultancy Service ● Provided trainings in life skills for USAID’s University Success Program-for Ethiopian Young Women project, under FHI360-Addis Ababa University. 2011-2013 Research Associate, Capacity Development Project on Strengthening Multi-Sectoral Planning, Budgeting, M&E, and Information Management, JICA (Ethiopia) ● Organized and conducted capacity development workshop/trainings on project proposal writing, project cycle management, plan and project marketing, prioritization, M&E, information management, strategic planning, participatory planning, budgeting, public investment program ● Organize experience sharing workshops and identify best practices for sharing together with Districts and Zones. ● Plan for and arrange various travels: assessment, monitoring, review meetings, etc. 2008-2009 Early Childhood Care and Development (ECCD) Project Coordinator, Save the Children USA (Ethiopia) ● Established and sustained good communication with stakeholders (Zone/District GoE Offices, Kebeles, CBOs, cooperatives, etc.) ● Mobilized community to participate and have a feeling off ownership in the project ● Organized and provided trainings for CMC on management of ECCD centers 2008 Project Coordinator, Ethiopian Social Accountability Program (ESAP-1), Jerusalem Children and Community Development Organization (Ethiopia) ● Provided all administrative and management oversight for the Social Accountability Program, including conducting and coordinating meetings with stakeholders. ADDITIONAL SELECTED ROLES: ● Coordinator, Children and Youth Program, the Ethiopian Gemini Trust (2007-2008) ● Social Worker, the Ethiopian Gemini Trust (2005-2007). EDUCATION MPH Masters of Public Health, Addis Continental Institute of Public Health (2016) M.A. Social Psychology, Addis Ababa University, Ethiopia (2011) B.A. Educational Psychology, Addis Ababa University, Ethiopia (2003) LANGUAGE English – Fluent, Amharic – Fluent, Afan Oromo – Fluent. 117 United States Agency for International Development Entoto Street Addis Ababa, Ethiopia