ADVENTIST DEVELOPMENT & RELIEF AGENCY (ADRA) MULTI-YEAR ASSISTANCE PROGRAM (MYAP) FINAL EVALUATION REPORT No. FFP-A-00-08-00084-00 AUGUST 2008 – December 2013 EVALUATION TEAM: DARELL MCINTYRE – Team Leader & Marketing PAUL FREEMAN – Nutrition & Health CESAR TIGUE -- Agriculture ALICE WILLARD – Gender & Resilience JANUARY 2014 Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique ii ACKNOWLEDGEMENT The entire Evaluation Team wishes to thank ADRA for the opportunity to contribute to their activity in Mozambique through this final evaluation report. The collaboration provided by Mr. Dawit Habtemariam, ADRA/International Acting Director for Evaluation greatly facilitated our work, and was highly appreciated. We would also like to thank staff in the Maputo and Mocuba offices of ADRA who ensured that we were able to obtain the information needed for this report. Without the particular assistance of those in the ADRA Mocuba field office who supported us, including traveling with us and translating among English and Portuguese, this report would not have been possible. We also want to thank the local leaders who translated between Portuguese and the several local dialects, without whom communication would not have been feasible. Finally, we want to thank those members of the communities visited and GOM officials for their time, patience, and understanding to meet with us and provide the invaluable information and insight into the program and food insecurity challenges facing Mozambique. Although this report benefited from a wide range of sources that provided the input, the Evaluation Team alone is responsible for the direction, level, and content of the analysis. The conclusions and recommendations contained in this report are our own, and do not reflect those of ADRA or any person or institution cited or mentioned therein. The Evaluation Team Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique iii TABLE OF CONTENTS ACKNOWLEDGEMENTS . . . . . . . . . iii TABLE OF CONTENTS . . . . . . . . . iii LIST OF TABLES . . . . . . . . . . v LIST OF ACRONYMS . . . . . . . . . vi EXECUTIVE SUMMARY . . . . . . . . . vii 1 INTRODUCTION . . . . . . . . . 1 1.2 BACKGROUND TO OSANZAYA PROJECT . . . . . 1 1.3 EVALUATION METHODOLOGY . . . . . . . 2 1.3.1 Evaluation Purpose . . . . . . . . . 2 1.3.2 Evaluation Objectives . . . . . . . 2 1.3.3 Evaluation Methods . . . . . . . . . 3 1.3.4 Evaluation Limitations. . . . . . . . . 4 2 PROJECT DESIGN and RESULTS ACHIEVED . . . . 5 2.1 Project Design . . . . . . . . . 5 2.2 Changes Over Tim . . . . . . . . . 7 2.3 Goal: Reduce Food Insecurity in Targeted Five Districts in Zambézia Province, Mozambique . . . . . . . . 8 2.3.1 Strategic Objective: (SO1) Improved Income Growth of 37,500 Rural Beneficiaries. . . . . . . . . . 10 2.3.2 Strategic Objective: (SO2) Improved Health and Nutrition Status for 40,000 Beneficiaries 2.4 EFFECTIVENESS OF THE PROJECT INTERVENTIONS . . . 11 2.4.1 Evaluation Question No. 1: To what degree did the project activities meet the needs of the project beneficiaries and is aligned with the country’s agriculture and development strategy and USAID/Mozambique development goals, objectives, and strategies? . . . . . 11 2.4.2 Evaluation Question No. 2: Has the project achieved its objectives? To what extent did the interventions contribute to the expected results or objectives? . . . . . . . . . 14 2.4.3 Evaluation Question No. 3: To what extent did the project resources (inputs) lead to results? Could the same results been achieved with fewer resources, or whether alternative approaches could have been adopted to achieve the same results? . . . . . . . 19 2.4.4 Evaluation Question No. 4: What are the medium and long term effects, both intended and unintended, of a project intervention? Were effects due to the project intervention and no other factors? . . . . . 22 2.4.5 Evaluation Question No. 5: What is the likelihood that the benefits of the project will endure over time after the completion of the project? Has the project planned for the continuation of project activities, developed local ownership for the project, and developed sustainable partnerships? . . 26 2.4.6 Evaluation Question No. 6: What are the lessons learnt, success stories, areas of improvement, and recommendations for similar programs? . . 29 Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique iv 3 CROSS-CUTTING ISSUES . . . . . . . 32 3.1 GENDER . . . . . . . . . . 32 3.2 RESILIENCE. . . . . . . . . . 39 4 MONITORING & EVALUATION . . . . . . . 41 5 SUMMARY CONCLUSIONS . . . . . . . 42 5.1 Agricultural Productivity . . . . . . . . 42 5.2 Commercialization . . . . . . . . . 43 5.3 Health & Nutrition . . . . . . . . . 43 5.4 Cross-Cutting Issues . . . . . . . . 44 5.5 Monitoring & Evaluation . . . . . . . . 44 ANNEXES Annex I EVALUATION SCOPE OF WORK . . . . . . II Annex II EVALUATION SCHEDULE OF VISITS . . . . . VII Annex III LIST OF PERSONS INTERVIEWED . . . . . VIII Annex IV QUESTIONAIRES . . . . . . . . VIII Annex V MAP OF INTERVENTION DISTRICTS IN ZAMBÉZIA PROVINCE . XIV Annex VI REFERENCES . . . . . . . . XVIII Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique v LIST OF TABLES Table 2.1: Eligibility Criteria to Select Individual Beneficiaries . . . 6 Table 2.3.1a: Direct Beneficiary Adoption . . . . . . 9 Table 2.4.2.1: Farming Techniques in Project Area . . . . . 15 Table 2.4.2.2a: Volume & Value of Crops FY 2010 . . . . . 15 Table 2.4.2.2b: Volume & Value of Crops FY 2011 . . . . . 15 Table 2.4.2.2c: Volume & Value of Crops FY 2012 . . . . . 16 Table 2.4.2.2d: Volume & Value of Crops FY 2013 . . . . . 16 Table 2.4.2.2e: Prices Paid for Selected Commodities . . . . 17 Table 3.1.2a: Nutrition Achievements . . . . . . . 34 Table 3.1.2b: Crops Cultivated By Gender . . . . . . 35 Table 3.1.4: Decision-Making Changes . . . . . . . 36 Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique vi LIST OF ACRONYMS ADPP People to People Development Assistance (Ajuda de Desenvolvimento de Povo para Povo) ADRA Adventist Development and Relief Agency BOM Mozambican Opportunity Bank (Banco Oportunidade de Moçambique) CDR Results Demonstration Center CLC Community Leadership Council DPAZ Provincial Department of Agriculture of Zambézia FDD District Development Fund FGD Focus Group Discussion HIV Human Immuno-deficiency Virus I.M. Monitoring Indicator IPEME Institute for the Promotion of Small and Medium Enterprises IPPT Indicator Performance Tracking Table I.R. Intermediate Results INCAJU National Cashew Institute (Instituto Nacional do Caju) LOA Length of Activity MT Metric Tons MYAP Multi-Year Assistance Program NGO Non-Governmental Organization NUIT Tax identification Number OSANZAYA Make Zambézia Happy (Definition in Chuabo Language) PITTSS Innovative Program for Technology Transfer SCIP Strengthening Communities through Integrated Programming SDAE District Economic Activity Services SIMA Mozambican Agricultural Market Price Information System SIMAPZ Zambézia Provincial Agricultural Marketing System T&V Training & Visit Approach to Extension Services USAID United States Agency for International Development WFP World Food Programme Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique vii EXECUTIVE SUMMARY Statistically, Mozambique remains one of the world’s poorest countries, ranking 185th out of 186 countries. Its history has included centuries of colonial neglect, implementation of Marxist economic theories, and 30 years of intense guerrilla warfare. Annual per capita income is about $424, although in rural areas (such as Zambézia, the zone of intervention) it is closer to $100. Major causes of poverty include low investment in education, high adult illiteracy, low agricultural productivity, limited economic opportunities, high underemployment, and poor infrastructure. Agriculture is central to economic life in Mozambique, and remains the key to economic and social development at the national and household levels. ADRA has worked in Mozambique since 1987, transitioning from relief to development￾oriented activities as Mozambique began recovering from a prolonged civil war and a major national drought. ADRA has implemented three Title II-financed programs over the past 15 years. The most recent has targeted five districts (Mocuba, Maganja da Costa, Ile, Pebane and Lugela) in Zambézia Province. The MYAP strategy has been to reduce food insecurity and increase rural incomes in a sustainable way that combines commercialization with increased productivity and strengthened value chains of select agriculture products and improve health, nutrition, water and sanitation in a mutually supportive and integrated approach. Additional initiatives, literacy, WASH and disaster preparedness, were implemented by a local NGO (ADPP) and an international NGO, Samaritan’s Purse. The Evaluation Team met in ADRA/Mozambique offices for initial discussions and planning the field visits for the evaluation. Team members had received copies of the scope of work (SOW) and relevant program documents prior to commencing the evaluation, and developed the lists of interview questions used during the group and key informant interviews. The Evaluation Team traveled as a single group to each community, but divided into specific thematic areas for discussions with community members. ADRA initially identified ten communities from among the potential universe of communities for interviews and discussions – two from each of the five MYAP districts -- ensuring a representative selection of communities with varying geographical and socio-cultural backgrounds. ADRA’s presence in each community varied from five to fifteen years, depending on the number of previous Title II programs. The major evaluation limitations were first the exensive geographical coverage of the MYAP versus the time available for field visits, and second, the Baseline and Final surveys restricted our ability to clearly assign causality to the MYAP’s impact. The lack of a control group, by which we could make actual comparisons, also precluded a more definative cause-and-effect relationship between interventions and results. However, we were able to mitigate this through our interviews, where anecdotal evidence was given by the beneficiaries as to the positive results of the project. The third was the relevance of the numerical MYAP benchmark indicators, which remained generally constant over the LOA rendering them more indicative than actual numerical targets. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique viii ADRA had originally envisioned a five-year MYAP, in line with its two previous DAPs. Upon later guidance from USAID, the MYAP was designed, approved, and funded for three years, beginning at the end of Fiscal Year 2008. In FY 2011, ADRA requested and was autorized an additional year’s extension of the MYAP through FY 2012 with increase in financial resources. A second, and final, extension was granted to lengthen *the MYAP through FY 2013 – again with increase above the original funding level. The Goal, remained unchanged during the LOA. However, Stategic Objectives, Intermediate Results, and other MYAP indicators were modified. The initial MYAP included SO 3: Increased Community Resiliency to Mitigate Against Shocks. Samaritan’ Purse, a local NGO, was to implement the key elements of disaster preparedness; together with the water/sanitation construction and rehabilitation activities, but could never get agreement with the Government of Mozambique on the technical details of this component. This led to dropping the component from the project in the third year. The two single year extensions have meant that a mid-term review and a redirect towards the middle of the third year (which would have been the case for a five-year program) did not occur. Instead, the project has had the unenviable task of pushing forward targets to out years that were not part of the original design, and revising its approach during year three to incorporate guidance on more fully integrating its activities. Exacerbating implementation from changing the MYAP’s LOA, was the collapse of the monetization program. This resulted in an immediate shutdown of the other MYAPs, although ADRA was able to continue in FY 2013 with its residual funding. In addition to the difficulty in planning and implementing a MYAP with a constantly changing LOA, personnel turnover occurred as each end date approached and staff sought, and found, alternative employment. Reestablishing beneficiary trust is another important negative consequence of staff turnover. The MYAP’s goal is to: Reduce Food Insecurity in Targeted Five Districts in Zambézia Province. The strategy to achieve SO1: Improved Income Growth of 37,500 Rural Beneficiaries is to produce sustainable growth in rural income in five districts of Zambézia Province by integrating marketing, increased productivity, and strengthened value chains of select agriculture products. The strategy to achieveSO2: Improved Health and Nutrition Status for 40,000 Beneficiaries is through improved health and nutrition status of children under age five, and improved hygiene behaviors, access to sanitation solutions, and adequate clean water. The overall conclusion of the Evaluation Team was that the project achieved most of its objectives. However, the team was not able to directly attribute an objective cause-and￾effect relationship between interventions and results because of the baseline and final survey populations, and the lack of a control group. Nonetheless, interviews did establish a connection that is arguably relevant. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique ix The main conclusions and lessons learned in Agricultural Productivity were:  communities are not homogeneous entities and generally conform to three groups, i.e., innovators or explorers (who are those most disposed to accept and apply new technologies or approaches), followers (those who are willing to accept new technologies or approaches once they can see the results), and resisters (those for which any change is unacceptable).  confine the number of “new” technologies or crops to the minimum number needed to demonstrate the effect, and using crops familiar to the beneficiaries.  introducing mechanization is the next required intervention to raise production and productivity.  improving beneficiary “ownership” of the new technologies requires a more participatory approach to extension  community or group production (demo plots or association areas) was a strategy used by the project to transfer knowledge to more farmers than would be possible in a one-to-one approach. The Evaluation Team concluded that farmers often prefer individual productive enterprises and cooperative marketing, and the evidence was that producers gave priority to individual production when faced with labor or climatic constraints. Better options are available that avoid group production. The main conclusions and lessons learned in Commercialization were:  the importance of literacy and small business training as both lead to more productive and profitable production and marketing enterprises.  a market-led approach that viewed agriculture as a private-sector for-profit enterprise was essential, and Associations, Unions, and Cooperatives are the means for furthering economic gains.  reducing transaction costs for buyers through aggregating production and quality standards were necessary for access to wider and more profitable markets.  the continuing need is to emphasize post-harvest technologies, particularly storage and quality preservation. The main conclusions and lessons learned in Health & Nutrition were:  sustaining the successful improvements in nutrition and health and community ownership will be influenced by the MOH and the logistic challenge of providing reasonable supervision and continuing education of community health workers.  Community Leadership Councils that bring together community level leaders and representatives of all groups of community health workers and health organizations within the community were an important part of the project’s success.  the quality of what is communicated is important for success across many behaviors.  literacy and numeracy training were clearly important in enabling the community level organizations and community health workers to perform well and should be a part of future MYAPs. The main conclusions and lessons learned in Cross-Cutting Issues were:  empowering women was due both to literacy and participation in village associations. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique x  being able to use their own funds has improved womens’ ability to upgrade household nutrition.  funds from the village associations have enabled women make more decisions, and more informed decisions, unilaterally or with their husbands -- funds from the village associations have enabled this.  higher household income allowed the women to purchase additional household assets and improve household nutrition, two key elements in improved resilience.  because the project did not include disaster preparedness/planning, a more consolidated plan for community resilience never emerged to help tie these different elements together in a more comprehensive fashion. The main conclusions and lessons learned in Monitoring & Evaluation were:  ADRA should consider expanding the M&E resources (including staff) so that independent field visits in a randomized, but routine, manner are possible.  standardizing databases from earlier projects so that longitudinal studies or other types of analysis might have helped inform both trend data and provide more robustness to explanations on over and under-performance on key variables.  Data from the communities needs to find a way back to the communities, so they understand how their opinions, work, and community outreach affect project performance results. This can help communities feel ownership.  the variety of local languages makes conducting surveys extremely time-consuming, as data collection tends to operate via at least one filter. Identifying, training, and periodically using community members with a secondary school education (even if no longer resident in the target communities) may help with survey data collection. 1 INTRODUCTION Statistically, Mozambique remains one of the world’s poorest countries, ranking 185th out of 186 countries.1 This low ranking is indicative of its economic, health, nutrition, education, and other social indicator levels. Its history has included centuries of colonial neglect, implementation of Marxist economic theories after independence from Portugal, and 30 years of intense guerrilla warfare. Annual per capita income is about $424, although in rural areas (such as Zambézia, the zone of intervention) it is closer to $100.2 An estimated 54% of the country’s population lives at or below the Mozambican poverty level of $1 per day (2003 baseline survey). Major causes of poverty include low investment in education, high adult illiteracy, low agricultural productivity, limited economic opportunities, high underemployment, and poor infrastructure. Although the Mozambican economy has shown steady improvement, it is coming from a very low base. Agriculture is central to economic life in Mozambique, and remains the key to economic and social development at the national and household levels. 1.2 BACKGROUND TO THE OSANZAYA PROJECT ADRA has worked in Mozambique since 1987, transitioning from relief to development￾oriented activities as Mozambique began recovering from a prolonged civil war and a major national drought. ADRA conducted a needs assessment in 2003, and involved key stakeholders both in the design of survey instruments as well as through interviews and secondary data collection to ensure both its relevance and to promote local ownership. ADRA has implemented three Title II-financed programs over the past 15 years. The most recent has targeted five districts (Mocuba, Maganja da Costa, Ile, Pebane and Lugela) in Zambézia Provence. The MYAP strategy has been to reduce food insecurity and increase rural incomes in a sustainable way that combines commercialization with increased productivity and strengthened value chains of select agriculture products, principally maize, pigeon peas, groundnuts (peanuts), and cashew. The MYAP design was to improve health, nutrition, water and sanitation in a mutually supportive and integrated approach. Two additional initiatives, literacy, WASH and disaster preparedness, were included, and were implemented by a local NGO (ADPP) and an international NGO, Samaritan’s Purse. Mocuba, Lugela, Ile, Maganja da Costa, and Pebane Districts lie at the junction of three agro-ecological zones: the Low and the Middle altitude zones of Zambézia, and the southern extreme of the Northern coastal zone. Rain fed small scale crops based on local varieties that are cultivated for consumption predominate, i.e., maize, cassava, beans (including pigeon pea), groundnuts, sweet potatoes, sorghum, millet, and rice. Other crops are cultivated for trade purposes, with emphasis on cotton, beans, sesame, sunflower, tobacco, tea, and cashew nut. The main economic activities and the basic source of livelihood for the population in the selected districts are agriculture and animal breeding; providing products for 1 UNDP Human Development Index (2013). 2 IMF projection (2005), cited in USAID/Mozambique Strategy Statement (May 24, 2006). Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 2 consumption and trade. Family groups practice these activities where natural conditions are favorable, such as fertile soils, reasonable rains, and a good river network. While Ile, Mocuba and parts of Lugela contain very productive areas, with very good soils and rainfall that exceeds 1,200 mm annually, Maganja da Costa and Pebane are drier with principally sandy soils. However, the low land areas have good soils, and these areas have high agricultural potential for producers in Zambézia province. 1.3 EVALUATION METHODOLOGY 1.3.1 Evaluation Purpose The final qualitative evaluation allows ADRA to assess the outcomes of the MYAP and identify successful food security strategies upon which ADRA can build future interventions. The evaluation analyses of the data and beneficiary interviews answer the important question of the “why” behind the outcomes, and identifies specific recommendations to enhance future ADRA food-security programming. The primary issues for the final evaluation were the project’s strengths, areas for improvement, lessons learned, and recommendations for similar projects in the future. Another key dynamic was the degree of impact and sustainability achieved. Any recommendations for changes or additions to sustainability strategies are included in the evaluation report. The final evaluation also assessed the target population’s capacity and prospects in terms of continuing sustainable and effective food security activities on its own after the MYAP ends. 1.3.2 Evaluation Objectives The purpose of this evaluation was to answer the following six questions as they pertain to agricultural productivity, commercialization, health and nutrition. These same questions are equally relevant to the cross cutting themes of gender and resilience. 1. To what degree did the project activities meet the needs of the project beneficiaries and are aligned with the country’s agriculture and/or development strategy and USAID/Mozambique development goals, objectives, and strategies? This includes the extent to which the project was designed taking into account the economic, cultural and political context and existing relevant project activities. 2. Has the project achieved its objectives? To what extent did the interventions contribute to the expected results or objectives? 3. To what extent did the project resources (inputs) have led to results? Could the same results have been achieved with fewer resources or whether alternative approaches could have been adopted to achieve the same results? 4. What are the medium and long-term effects, both intended and unintended, of a project intervention. Were effects due to the project intervention and no other factors? 5. What is the likelihood that the benefits of the project will endure over time after the completion of the project? Has the project planned for the continuation of project Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 3 activities, developed local ownership for the project, and developed sustainable partnerships? 6. What are the lessons learnt, success stories, areas of improvements and recommendations for similar programs? 1.3.3 Evaluation Methods The evaluation team met in ADRA/Mozambique offices for initial discussions and planning the field visits for the evaluation. The Team also met with Ms. Anne Fisker, ADPP Project Director for the literacy and basic business training element of the program. The Evaluation Team held a brief introductory visit at USAID/Mozambique. The methodology for the evaluation included the following activities: Literature Review: Team members had received copies of the scope of work (SOW) and relevant program documents prior to commencing the evaluation. The SOW (Annex I) provided a general overview of the MYAP evaluation, as well as suggested methodology and reporting format. Other documents reviewed included the FY 2008- FY2011 Development Activity Program Proposal, annual reports, and the data from the final survey report. Interview Questions: The evaluation team developed the lists of suggested interview questions used during the group and key informant interviews during the first day at the ADRA field office in Mocuba. Annex IV details the final list of questions. Team: The evaluation team traveled as a single group to each community, but divided into specific thematic areas for discussions with community members. The team traveled with several ADRA staff members who could translate the questions and discussions between English and Portuguese. The ADRA team helped the evaluators identify community members who could translate between Portuguese and the local languages from among the people participating in the focus group discussions. In some cases, most often in the agricultural and commercialization focus groups, they were community leaders or the more senior members of the marketing groups. The team usually conducted two interviews each day in two distinct communities that usually lasted about two hours each. On some occasions, members of the Evaluation Team conducted additional interviews with the relevant GOM officials subject to their availability. Sampling: ADRA initially identified ten communities from among the potential universe of communities for interviews and discussions – two from each of the five MYAP districts. ADRA’s presence in each community varied from five to fifteen years, depending on the number of previous Title II programs. This approach ensured a representative selection of communities with varying geographical and socio-cultural backgrounds. Focus Group Discussions: The Evaluation Team conducted focus group discussions in nine of the ten selected communities in Ile, Lugela, Maganja da Costa, Mocuba, and Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 4 Pebane districts in Zambézia province from December 3 to December 13, 2013. However, the Evaluation Team was unable to conduct focus group interviews in one community because of a misunderstanding regarding scheduling. Annex II details the specific locations visited. Attendance was quite broad and involved community leaders, men and women project participants, and sometimes included non-project community residents. These indirect beneficiaries provided information on their perception of the program, and provided insight into the diffusion and adoption of the introduced technologies and practices among this group. Key Informants and Relevant Stakeholders Interviews: The Evaluation Team interviewed Project Management, Supervisory, and Technical staff during this same period. The Team had additional interviews with key GOM officials and obtained further input during the debriefing session held in Mocuba on December 13. 1.3.4 Evaluation Limitations The major evaluation limitations were first the extensive geographical coverage of the MYAP versus the time available for field visits, and second, the Baseline and Final surveys did not sample appropriate populations. The third was the relevance of the numerical MYAP benchmark indicators, which remained generally constant over the LOA even though the LOA was extended twice. This rendered them more indicative rather than actual numerical targets. The evaluators were fortunate in being able to visit each of the five districts where the MYAP had been implemented. While the current delay in the start of the rainy season was a real constraint to agricultural production, it did permit travel to locations that otherwise might have been inaccessible. The results of the baseline and final surveys were not entirely informative. The initial baseline surveyed the broader population that included potential beneficiaries, while the final survey also surveyed this broader population. The latter included both beneficiaries (direct and indirect) and non-participants. This restricted our ability to clearly assign causality to the MYAP’s impact. Moreover, it diluted any results of project performance, since the beneficiaries had greater opportunities for economic and health￾related improvements than the general population. Agricultural production, marketing, and pricing are subject to a number of important variables that are outside the control of the MYAP. The lack of a control group, by which we could make actual comparisons, also precluded a more definative cause-and￾effect relationship between interventions and results. However, we were able to mitigate this through our interviews, where anecdotal evidence was given by the beneficiaries as to the positive results of the project. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 5 2 PROJECT DESIGN and RESULTS ACHIEVED 2.1 Project Design The strategic framework links directly to USAID Mozambique’s Strategic Plan, through “Strategic Objective 6: Rapid rural income growth sustained in targeted areas”. SO1 and SO2 contribute to Mission’s “IR 6.1: Increased agricultural productivity of key crops in targeted areas” and “IR 6.2: Increased sales by agricultural producers in targeted areas”. The program includes USAID FFP’s Strategic Framework -- prioritizing activities to: enhance and protect human capabilities (health, nutrition, water and sanitation, hygiene activities); enhance and protect livelihood capacities (value￾chain development, agriculture productivity); enhance and protect community resiliency (risk mitigation; and increase community capacity to influence factors affecting food security (market information, business skills and literacy training, nutrition activities). The MYAP reduces food insecurity and increases rural incomes in a sustainable way that integrates commercialization with increased productivity and strengthened value chains of select agriculture products (SO1). It improves health, nutrition, water and sanitation (SO2). It is also compatible with strategies of the Government of Mozambique at national, provincial, and district levels. The population faces many challenges in the selected districts, including low productivity, low access to inputs (improved seeds and fertilizers), sensitivity to climate, few livelihood options, health issues, distance from markets, poor road infrastructure. The farther communities are from markets translates as fewer options to diversify their household income earning opportunities. To increase rural incomes, ADRA assisted smallholder farmers to produce a set of products that have a high potential for profit based on market demand. ADRA’s market study identified specific products with market growth potential that are suited to agronomic conditions of the targeted program areas, i.e., peanuts, maize, pigeon peas, and cashew. ADRA and its partners then offered packages of technical services and marketing/business skills training to develop the production and marketing chains of the select products. This market-demand approach to agriculture productivity and commercialization resulted in improved household and community economic, health, and nutrition well-being. This, in turn, strengthened, in theory at least, their capacity to withstand and recover more rapidly from external shocks. Families in the project area have an average of two plots of land used for agricultural activities. According to the ADRA 2013 field survey, the average size of land holdings per family was 1.58 hectare. Of the total respondents, 95.1% said they own land for farming, compared to 91% at baseline. The remaining 4.9% said their land was either leased, share cropped, or borrowed during the last crop season. One of the plots is generally located close to the house and the other an average of 200-400 meters from the house. Only six out of ten communities interviewed, in particular Ile, Maganja Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 6 da Costa, and Pebane, claimed to have access to wetland areas for cultivation. No communities directly influenced by the project in Lugela have access to low land areas. ADRA worked in a new district (Pebane) as well as in new communities in previous program districts (Mocuba, Maganga da Costa, Ile, and Lugela). Moreover, they used a commercialization-led approach in the MYAP to improve income generation. Past ADRA programs had implemented a production-led approach. ADRA included new activities (e.g., business skills, marketing, planning, etc.) with selected farmers from the previous and current programs. They used Hearth methodology across all communities with a high percentage of malnourished children, which they did only on a pilot basis in the past. Table 2.1: Eligibility Criteria to Select Individual Beneficiaries Individual Eligibility Criteria Ag Market H&N Wat/San Is a resident in target community X X X X Has access to a minimum of two hectares of land X X Willingness to work with at least one pre-selected crop X Becomes a member of one of the Farmers Groups (FGs) X X Willing to implement technical recommendations for production and post-harvest handling X X Willing to train other farmers X X Person is respected among community, has credibility to be association leader X X Willing to belong to association, which will select individuals who will market their product X Must complete improved production training with ADRA (includes literacy training) X X Willing to participate in a village savings scheme, if applicable X X Committed to work in a group and try new things X X Child under age 5 X Pregnant or lactating woman X Participate in growth of monitoring their children X Willingness to participate in training ( ind. or instit.) X X Limited or no access to sanitary facilities ( ind. or instit.) X Limited or no access to protected water sources X House whose primary water source is unprotected surface water X SOURCE: ADRA MYAP (revised July 2009) The MYAP included in its design a gender strategy in which the analysis noted four key points:  women’s decision-making participation was limited  women comprised the bulk of farm labor  female literacy levels were low (18% in some areas)  money earned by women (in petty trade) was used to feed the family, while money earned by men went towards social engagements and family responsibilities. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 7 The proposal therefore emphasized an inclusive approach for women and men towards participation in all project activities, reinforced in the different training initiatives (literacy, business training, etc.), community outreach (mothers and fathers clubs), and participation on village water committees (with a specific intention of female parity in management). The M&E plan incorporates indicators required by FFP, USAID/Mozambique, and those required for managing program implementation. Baseline and final surveys, and the final evaluation provide evidence of the overall impact of the program. Monetizing wheat helped meet consumer demand and conserved scarce foreign exchange. 2.2 Changes Over Time ADRA had originally envisioned a five-year MYAP, in line with its two previous DAPs. Upon later guidance from USAID, the MYAP was designed, approved, and funded for three years, beginning at the end of Fiscal Year 2008. In FY 2011, ADRA requested and was autorized an additional year’s extension of the MYAP through FY 2012 with increase in financial resources. A second, and final, extension was granted to lengthen the MYAP through FY 2013 – again with increase above the original funding level. The Goal, remained unchanged during the LOA. However, Stategic Objectives, Intermediate Results, and other MYAP indicators were modified. The initial MYAP included SO 3: Increased Community Resiliency to Mitigate Against Shocks IR 3.1: Community resiliency protected and enhanced The program will target a 60% increase in communities with disaster early warning and response systems in place over the life of the program Coordinate/strengthen capacity of sector￾level INGC; establish and train community risk management committees; provide disaster mitigation kits to committees SOURCE: ADRA MYAP (revised July 2009) Samaritan’ Purse, a local NGO, was to implement the key elements of disaster preparedness; together with the water/sanitation construction and rehabilitation activities. Unfortunately, Samaritan’s Purse could never get agreement with the Government of Mozambique on the technical details of this component. This situation led to dropping the component from the project in the third year. Consequently, none of the focus groups could identify a coherent plan for the community beyond individual common sense and previous experience. In addition, the project design was prior to the launch of USAID’s Resilience Policy Guidance (2012)3 . 3 The policy guidance defines resilience as “the ability of people, households, communities, countries and systems to mitigate, adapt to, and recover from shocks and stresses in a manner that reduces chronic vulnerability and facilitate inclusive growth” (USAID Resilience Policy Guidance, 2012). Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 8 The two single year extensions have meant that a mid-term review and a redirect towards the middle of the third year (which would have been the case for a five-year program) did not occur. Instead, the project has had the unenviable task of pushing forward targets to out years that were not part of the original design, and revising its approach during year three to incorporate guidance on more fully integrating its activities. Both of these actions have had a negative impact on executing a coherent M&E plan. Exacerbating an already fluid implementing environment resulting from the changing of the MYAP’s LOA, was the collapse of the monetization program. In FY 2013 the Monetization entity established by four of Title II NGOs accepted a US$5.5 million dollar personal check from one buyer that subsequently was returned for insufficient funds. This resulted in an immediate shutdown of the other MYAPs, although ADRA was able to continue in FY 2013 with its residual funding. ADRA had contemplated requesting an additional one-year extension had these funds been available. In addition to the difficulty in planning and implementing a MYAP with a constantly changing LOA, personnel turnover occurred as each end date approached and staff sought, and found, alternative employment. ADRA had to identify, train, and deploy replacement staff. This meant that valuable time, effort, and continuity were lost that would otherwise been working to achieve project results. Reestablishing beneficiary trust is another important negative consequence of staff turnover. 2.3 Goal: Reduce Food Insecurity in Targeted Five Districts in Zambézia Province The program strategy is to produce sustainable growth in rural incomes in the target region that integrates marketing, increases productivity, and strengthens value chains of select agriculture products in five districts of Zambézia Province. 2.3.1 Strategic Objective: (SO1) Improved Income Growth of 37,500 Rural Beneficiaries IR 1.1: Increased sales by agricultural producers in targeted areas The program will target a 15% increase in tonnage of agriculture produce marketed over the life of the program Establish and strengthen farmer groups and associations; promote select marketable crops; enhance value-chain linkages for producers; improve market information dissemination and systems; train producers on timing of produce marketing to secure higher sales prices; convene agriculture fairs; train producers in business skills and literacy; institute village banking/savings scheme; improve post-harvest handling, processing and storage. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 9 IR 1.2: Increased agricultural productivity of selected crops The program will target a 50% over the baseline in annual yield rates for targeted crops over the life of the program Establish and strengthen farmer groups and associations; promote select marketable crops; promote use of high quality seeds; provide technical assistance through extension services; train producers in production techniques. SOURCE: Mozambique ADRA International MYAP (revised 2009) Impact indicator 1: % of farmers who adopted at least three technologies disseminated by the end of the project. Proportion of farmers in target area who practice at least three of the improved agricultural technologies promoted by the project. Impact Indicator 1 emphasizes the importance of the rates of technological adoption. The MYAP established an increased adoption rate of three or more technologies per farmer to above 60% of the target population. The MYAP’s M&E system showed the direct beneficiary adoption rate of more than three agriculture techniques reached 85.3% in 2013. This was clear during the discussions with farmers, where most of them were able to identify different techniques recommended by ADRA OSANZAYA’S extension services, i.e., planting in rows, improved seeds, weeding twice, and increasing organic matter into the soil, as practices adopted in their own fields. Moreover, they were able to draw spacing between crops and rows, even for different types of crops and intercropping when asked. Based on discussions with beneficiaries, introducing improved seeds, using adequate spacing, and weeding were the most often-used technologies. Establishing adoption rates for indirect beneficiaries is problematic. The project was expected to have a spillover effect by transferring technologies from direct beneficiaries to the “padrinhos” (5 per farmer), Data from the final survey (ADRA 2013), in which farmers were randomly selected, show that 75.3% of them used between one-to-two techniques, but only 24.7% adopted more than three techniques. Usually adoption rates for indirect beneficiaries occur when farmers procure information and inputs for agricultural production from many different sources, including the wider family, community, farmer groups and associations, and external agencies such as public extension, input suppliers, commercial farmers, radio programmes, and occasionally posters and leaflets. Table 2.3.1a: Direct Beneficiary Adoption District Direct Benef. Adopt > 3 technol. Mocuba 1,076 83.9% Lugela 1,987 84.5% Ile 1,138 84.7% Maganja da Costa 1,575 88.1% Pebane 1,498 84.6% Total 7,274 85.3% Source: ADRA M&E system 2013 Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 10 The reason farmers do not adopt new practices are many and varied. Some do not see the value in changing what they have always done. Others feel that the new practice is risky in terms of time and money and the returns do not adequately compensate, and some want to see their neighbors succeeding with a new practice before they adopt. For example, OSANZAYA’s extension services promoted conservation agriculture (an approach that brings together proven good agricultural practices for dry land agriculture. Indirect beneficiaries resisted adopting this as a package of techniques due to the additional labor required. Impact indicator 2: % increase in the yields of target crops. Percentage increase in yields of target crops as compared with the baseline. Due to using improved agriculture techniques, farmers were able to improve their yields. However, yields for rain fed agricultural production vary per agro-ecological environment (soil type, moisture and crop requirement). ADRA’s measurements show that the current average yield from the start of the project to the end has changed. For example, maize increased from 0.9 to 2 MT/ha (133%), ground nuts from 0.7 to 1 MT/ha (43%), and pigeon pea from 0.9 to 1.4 MT/ha (56%). 2.3.2 Strategic Objective: (SO2) Improved Health and Nutrition Status for 40,000 Beneficiaries IR 2.1: Improved health and nutrition status of children under age 5 The program will target an improvement 6% decrease in underweight children, and 5% reduction in stunting over the life of the program. Establish/strengthen Community Health Councils; recruit and train Community Health Volunteers); apply Positive Deviance/HEARTH model to identify model mothers and establish mother groups; through above structures disseminate and encourage positive health messages and practices in growth monitoring, hygiene, diet diversification, food preparation, breast￾feeding and natal, disease prevention, health-seeking and care-giving behaviors. IR 2.3: Improved hygiene behaviors, access to sanitation solutions, and adequate clean water The program will target an improvement by 40% over baseline of caregivers using appropriate hand washing over the life of the program. Conduct hygiene promotion using PHAST model; construct bio-sand water filters; rehabilitate hand￾pumps and improve hand-dug wells; train community representatives in pump/well/borehole maintenance; construct pit latrines for individual households and for institutions. SOURCE: ADRA MYAP (revised July 2009) The Evaluation Team concluded that the project achieved its objectives in nutrition, growth-monitoring coverage, decreasing severe malnutrition, and in exclusive breastfeeding. Similarly, the project achieved good household level coverage of beneficiaries for behavior change communication. These achievements are worthwhile since there is good evidence that all the key behavior changes promoted by the project Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 11 in these areas and the key methodologies that the project used to address them -- monthly growth monitoring, monthly visits to households and, mothers’ groups -- reduce childhood malnutrition, morbidity, and mortality.4 , 5 Focus Group discussions and key informant interviews demonstrated that networks of CLCs, Community Health Volunteers, IMCI volunteers, and mothers and fathers groups are well established and functioning. All these community health workers demonstrated their functionality, and it would not have been able to achieve the same results without them. CLCs are important for organizing community health workers to work and plan together to solve their own health problems in a sustainable learning organization. Without this organization, local commitment to and sustainability of health activities would soon falter. Common vision and goals mutually support and motivate group members. Mothers need inclusion at the household level at least once a month to achieve reduced infant and child mortality. Therefore, a large number of Community Health Volunteers are needed. Mothers groups and community health volunteers are efficient means to improve child nutrition6 . 2.4 EFFECTIVENESS OF THE PROJECT INTERVENTIONS 2.4.1 Evaluation Question No. 1: To what degree did the project activities meet the needs of the project beneficiaries and is aligned with the country’s agriculture and development strategy and USAID/Mozambique development goals, objectives, and strategies? 2.4.1.1 Agricultural Productivity The project is consistent with Food Action Plan (2008) and the National Agriculture Strategy (PEDSA) (2009), whose policy is to develop the agricultural sector in the medium and long term with “a prosperous, competitive, equitable and sustainable agricultural sector” whose main objective is “... to contribute to food security, income and profitability of agricultural producers and to a rapid, competitive and sustainable increase in market-oriented agricultural production”. The policy base is three priorities, i.e., food and nutritional security, competitiveness of domestic production and higher income levels of producers, and sustainable natural resource use and environmental conservation. The vision has four strategic pillars: (i) Agricultural Productivity and Nutrition -- the increase in productivity, production and competitiveness in agriculture, particularly in 4 P.Freeman, H.B.Perry, S.K.Gupta, B.Rassekh Accelerating Progress in Achieving the Millennium Development Goal for Children through Community-Based Approaches Global Public Health Vol 7 No 4 April 2012 P 400-419. 5 USAID MCHIP Building on the Current Evidence to Strengthen Community- Based Service Delivery Strategies for Promoting Child Survival 2011. 6 Davis TP, Wetzel C, Hernandez Avilan E, de Mendoza Lopes C, Chase RP, Winch PJ, et al. Reducing Child Global Undernutrition at Scale in Sofala Province, Mozambique, using Care Group Volunteers to Communicate Health Messages to Mothers. Global Health: Science and Practice 1(1):35 - 51. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 12 nutritious food value chains that contribute to a proper diet; (ii) Market Access – improving services and infrastructure for better market access and making the guiding framework of the agricultural sector conducive to agricultural investment; (iii) Natural Resources -- sustainable use and the integral exploitation of land and, water resources; and (iv) Institutions -- by strengthening agricultural organizations and institutions. The project’s relevance is evident in Zambézia Province where its activities increased availability of and access to food among targeted households and communities. The resulting increase in produce available for self-consumption and marketing originated from selecting appropriate crops. Based on a matrix of potential crops versus agro￾ecological zones and districts, and a comprehensive marketing study, the project selected seven specific products: maize, groundnuts, pigeon pea, cassava, sesame seed, cashew nuts and soybeans. The priority crops were maize, groundnuts, pigeon pea, and cashew nuts. The selected crops and varieties were appropriate since they are part of the food security and marketing commodities familiar in the selected districts. The project introduced improved varieties with high productivity potential. Farmers make crop and varietal selections based on several criteria, including input availability, labour (both hired and household), experience, prices, and environmental factors such as climate, soil, and available water. One of the main interventions of the project was to transfer appropriate technologies that improve soil productivity while also improving soil conservation. These technologies included improved seeds, use of adequate spacing, thinning, incorporating organic matter, intercropping, crop rotation, and mulching. OSANZAYA was very successful implementing this component because it was able to disseminate appropriate and affordable agriculture conservation techniques. Knowledge transfer through learning by doing at Results Demonstration Centers (CDR), training, and literacy was the best approach adjusted to local conditions the literacy level of the beneficiaries. 2.4.1.2 Commercialization As far as can be determined through reviewing the literature and beneficiary interviews, these five districts in Zambézia Province had never benefited from a truly marketing-led assistance program. Heretofore, the assumption was simply that the market for additional produce existed, and the need was for producers to increase production to satisfy existing demand shortfalls. Consequently, previous assistance projects had a production-led strategy with marketing an afterthought. This was confirmed in interviews with beneficiaries who were unable to describe, in any detail, the marketing system beyond the initial point of sale -- usually at the farmgate -- that was operating prior to the project’s initiation. ADRA’s OSANZAYA project broke from that historical approach and designed and implemented a project that was demand oriented. Basic production technologies, such Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 13 as not burning plant residue before planting, planting in rows, proper plant population, weed control, and improved seed use were incorporated, but these were not the driving force of the project. ADRA staff correctly concluded that volumes of produce must be increased at each sales point, and that quality standards must be introduced and observed for there to be a significant improvement in sales. Prior marketing had consisted of each producer marketing individuallly that portion of his or her production not set aside for household consumption, or for seed stock for the next year. Traditional buyers would arrive at harvest and go from producer to producer purchasing whatever few sacks of product was available at each stop. The need to increase volume was the constraint that kept producers from moving to a different level of commercial agriculture with its consequent increase in product value. The project’s approach of getting producers to form Associations with approximately 25 members each targeted this directly, and the requirements to bring secondary beneficiaries into the process ensured that the necessary volumes could be achieved. Although these indirect beneficiaries were not formally members of the Associations, they were able to see and capture the benefits of combined sales. The next step was to group Associations into Unions that further increased volume per sales point. This meant that the range of buyers could be expanded to include large purchasers, e.g., World Food Programme (WFP), MADAL, EXPORT MARKETING, OLAM, and wholesalers, etc. The final step on the ladder was to have several Unions join together to form a legally constituted Cooperative. ADRA also held several agricultural fairs that increased awareness of the Associations and Unions, and the types and qualitity of their commodities. This is a recognized and useful activity for raising awareness among potential buyers. Along with these larger purchases came the requirement for quality standards. Traditionally each producer simply filled his sacks with whatever he wanted to sell, regardless of quality. Producers, in the majority, adopted these quality standards, and noted in interviews how they were able to negotiate a higher price for their commodities because of its higher quality. A few Associations and Unions had to learn the quality lesson after their commodities were rejected by the larger buyers, and thusly they had to forego the higher prices paid to other Associations or Unions. Market discipline is commonly quite a stern taskmaster. It was obvious from the interviews, hearing of the extra household income derived from these sales and how it was used, that the project did appropriately target the needs of the beneficiaries, as well as the Government of Mozambique’s and USAID’s development goals of increasing rural incomes and reducing food insecurity. That Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 14 achievement was the result of appropriate analysis and design, understanding the local environment, and highly qualified local ADRA staff. 2.4.1.3 Health and Nutrition The project designed activities to meet the health and nutrition needs of the poor rural target population. When the project began, health conditions were poor, as indicated by the high incidence of stunting, and high rates of diarrhea and malnutrition in the surveyed households. In part, the poor health conditions were due to inadequate access to health services, and in part due to lack of knowledge about causes of common illnesses, ways to avoid contracting the illnesses, and effective cures. A high proportion of mothers did not follow recommended child feeding practices. Evidence for this comes from health focus group discussions and the Zambézia Needs Assessment Survey7 2.4.2 Evaluation Question No. 2: Has the project achieved its objectives? To what extent did the interventions contribute to the expected results or objectives? 2.4.2.1 Agricultural Productivity According to the ADRA OSANZAYA’s 2013 ARR, 14,870 ha (123.9% of the targeted 12,000 ha) were cultivated using three or more improved agricultural technologies. These technologies included improved seeds, adequate spacing, thinning, incorporating organic matter, intercropping, crop rotation, and mulching. The Evaluation Team confirmed that most farmers responded, and that due to the introduction of these improved agriculture practices, farmers improved production and expanded their cultivated areas. In some instances, the expansion in cultivated areas was due to ADRA OSANZAYA’S requirements and the need to increase produce to boost their marketing shares. The positive results experienced by farmers using improved techniques in past seasons encouraged them to expand the areas using improved technologies. Other farmers also adopted these interventions after observing the good results through exchange visits between associations. Results from the final survey conducted by ADRA (2013) show that mixed cropping is the most frequently used technology, followed by crop rotation, weed control, and row planting, respectively (Table 2.4.2.1). The randomly conducted final survey involving direct, indirect, and other community members showed that farmers have already incorporated these technologies within their farming practices. Direct beneficiaries indicated they use most of the techniques introduced by the project, i.e., adequate spacing in rows, improved seeds, and weeding. ADRA was successful by improving existing techniques, i.e., enhancing mixed cropping by recommending a combination of 7 Mark Langworthy Zambezia Needs Assessment Survey Report of Findings Tango International October, 2007 Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 15 a cereal (maize) and legume (pigeon pea) and groundnuts, combined with cassava, and using appropriate spacing to increase nitrogen content in the soil. Indirect beneficiaries had difficulty accessing improved seeds that ADRA introduced through Results Demonstration Centers (CDR) where participation was limited to only farmer￾association members. The project did not create seed banks, or appropriate seed supply systems in partnership with the SDAE and the private sector, which limited the spread effect. 2.4.2.2 Commercialization The answer to this question has two facets, i.e., quantitative and qualitative. Quantitatvely the annual sales data contained in the annual reports and the final survey do indicate that the project achieved its objective of significant increased rural household incomes through improved marketing interventions. Tables below show an increasing trend of greater production and sales receipts over the course of the MYAP. Table 2.4.2.2a: Volume & Value of Crops FY 2010 Source: ADRA ARR FY 2010 (rounded) Table 2.4.2.2b: Volume & Value of Crops FY 2011 DISTRICT MAIZE GROUNDNUT PIGEON PEA CASHEW NUTS TOTAL MT $US MT $US MT $US MT. $US MT $US Mocuba 572.53 94,648 74.16 17,522 41.62 18,126 19.80 8,228 708.12 138,524 Pebane 502.97 115,223 25.96 12,775 528.93 127,998 Ile 100.40 12,037 174.85 72,588 264.01 136,598 35.65 17,784 574.92 239,008 Lugela 318.60 50,697 29.20 12,293 347.80 62,990 Maganja da Costa 14.00 2,311 246.88 65,380 44.98 20,457 305.87 88,147 TOTAL 1,005.53 159,693 998.88 270,713 334.83 167,017 126.34 59,244 2,465.64 656,667 DISTRICT MAIZE GROUNDNUT PIGEON PEA CASHEW NUTS TOTAL MT $US MT $US MT $US MT. $US MT $US Mocuba 1,017.68 158,222 116.57 54,777 231.45 81,927 49.97 42,526 1,415.67 337,452 Pebane 861.02 334,078 598.87 469.373 1,459.89 803,451 Ile 108.71 14,933 555.96 547,870 34.00 23,810 698.67 586,612 Lugela 510.66 54,479 133.69 44,628 11.50 9,274 655.85 108,381 Table 2.4.2.1: Farming Techniques in Project Area Farm Techniques # Inter-view % of use Crop Rotation 309 46.3 Mix Cropping 481 72.1 Incorporating organic fertilizer 11 1.6 Seed preparation 33 4.9 Improved seed use 10 1.5 Row planting 131 19.6 Water management 2 0.3 Weed controlling 210 31.5 Improved soil preparation 49 7.3 Biological pest control 11 1.6 Source: ADRA Final Survey 2013 Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 16 Source: ADRA ARR FY 2011 (rounded) Table 2.4.2.2c: Volume & Value of Crops FY 2012 Source: ADRA ARR FY 2012 (rounded) Table 2.4.2.2d: Volume & Value of Crops FY 2013 Source: ADRA ARR FY 2013 (rounded) The final survey reports that the dollar value of gross sales was $46.11 per household. This is almost a doubling of the $23.39 per farming household for selected crops reported in the baseline. Although this may be due, in part at least, to increased land under cultivation (Average size of land cultivated = 1.2 ha under baseline and 1.92 ha reported under final survey). Attributing the increases directly to the project interventions are difficult to verify. First, the two data sets, baseline and final, do not reflect surveying the same populations each time. The more indicative data sets in this case are the annual reports, which show general increases over the Life of the Activity (LOA). The second major variable is the weather during the growing season, which has been established in the Annual Results Reports (ARRs) and the interviews has highly variable. Third, and most important from a marketing perspective, pricing is highly responsive to the relative supply and demand environment of each commodity at the time of sale. One crop, cashew, is also subject to world market fluctuations. Small-scale producers are usually “price takers” in the marketplace. Their volume is insufficient to determine prices. The only exceptions are producers targeting very special niche markets, where there are few suppliers for this very selective commerce. Maganja da Costa 145.00 57,500 105.00 83,569 250.00 141,069 TOTAL 1,637.05 227,634 1,678.55 994,225 365.14 126,555 799.34 628,552 4,480.08 1,976965 DISTRICT MAIZE GROUNDNUT PIGEON PEA CASHEW NUTS TOTAL MT $US MT $US MT $US MT. $US MT $US Mocuba 183 38,765 36 28,012 581 150,350 801 217,128 Pebane 446 343,498 36 7,972 171 104,433 652 455,903 Ile 335 57,327 313 274,041 521 248,316 64 66,312 1,233 645,996 Lugela 663 126,107 266 82,179 929 208,286 Maganja da Costa 35 15,000 45 15,071 65 30,179 145 61,250 TOTAL 1,181 222,200 830 660,551 1,449 504,889 300 200,924 3,760 1,588,563 DISTRICT MAIZE GROUNDNUT PIGEON PEA CASHEW NUTS TOTAL MT $US MT $US MT $US MT. $US MT $US Mocuba 545 131,376 74 30,310 439 149,061 1,058 310,747 Pebane 545 269,530 76 23,361 117 31,581 738 324,472 Ile 167 40,438 118 95,978 117 41,008 9 2,638 411 180,062 Lugela 774 136,062 25 8,586 799 144,468 Maganja da Costa 362 193,631 161 55,094 104 27,954 627 276,679 TOTAL 1,486 307,877 1,099 589,488 818 277,111 230 62,173 3,633 1,236,608 Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 17 The FY 2013 ARR did report that the prices for commodities sold by the Unions were above average market clearing prices (see Table 2.4.2.2e below). Interviews confirmed this where we were told that the Unions could negotiate a higher price based upon the quality of the product being delivered. Table 2.4.2.2e: Prices Paid for Selected Commodities Product Negotiated prices by Union Farmers (US$/Kg) Average market price (US$/Kg) % difference between negotiated and average prices Shelled groundnuts 0.40 – 0.44 0.33 +27% Unshelled groundnuts 0.74 – 0.77 0.67 +12.6% Maize 0.20 – 0.30 0.10 – 0.16 +92.3% Source: ADRA ARR FY 2013 As the quality selection is an integral part of the commercialization activities of the MYAP, we may conclude that there is a tentative direct cause-and-effect relationship between at least some of the program interventions and increased income from sales. Without control groups we cannot positively confirm further direct linkages. 2.4.2.3 Health and Nutrition The project achieved its nutrition objectives in relationship to growth monitoring coverage, decrease in severe malnutrition, and exclusive breastfeeding. Similarly, the project achieved good household level coverage of beneficiaries for behavior change communication in many areas. The new crops produced through the agricultural component that increased the food groups available to children enhanced the integration with the health and nutrition component. Achieving 56.7% in the final survey for target population access to improved water source, 33.4% over the baseline, is an outstanding achievement. The project achieved the indicator targets for hand washing, however health focus groups discussions revealed that soap is a luxury item that most households do not have for most of the year. While natural products such as ash and gee are used instead, there is only published evidence that soap prevents diarrhea.8 Project achievements in relationship to several important activities were not satisfactory. While the proportion of children receiving solid/semi-solid food the minimum number of times per day increased from the baseline of 65% to 73% in 2011, it decreased thereafter to 53.2% in 2012 and 42.5% in 2013, which is 53.1 % of project target of 80%. The final survey also indicated similar low levels of achievement. 8 Luby, S.P., Agboatwalla, M., Feikin, D.R., Painter, J., Billhimer, W., Altaf, A., and Hoekstra,R.M., 2005. Effect of hand washing on child health: a randomized controlled trial. Lancet, 366, 225_233. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 18 Similarly, project-monitoring data indicate that while the proportion of children receiving the minimum number of food groups per day increased from a baseline of 20%, this level stayed around the 26% level for the next three years before rising to 34.5% only in 2013. Poor environmental conditions and the quality of behavioral change messages may have contributed to this result. Effective behavioral change requires both good coverage and message quality. Most health focus group discussions demonstrated that community members had generally learned well about the four food groups and the need to give a mixture of them regularly as they did through “enriched porridge”. While different focus groups described different foods in their “enriched porridge, the crops promoted by the project were usually included. However, giving these foods occurred only once a week in some communities in times of drought. Similarly, the group members described how good harvest enabled them to use some of the profits to buy such foods as eggs and chicken that they did not produce and soap. Some participants in the Community Leadership Council focus groups suggested including small-scale income generating activities in addition to agriculture in more vulnerable areas subject to drought. In addition, while the project monitoring system indicates good coverage of behavior change communication and good results in relationship to the Impact Indicators concerning hand washing and use of latrines, this is not so for indicators concerning continuing breastfeeding for children aged 6 to 24 months, and knowledge about malaria and HIV prevention. ADRA MYAP Project Director noted that he found changing behavior in relationship to some behaviors was a slow process, and that the Ministry of Health was looking for better behavior changing approaches. Even if people know about taking preventive behaviors, this knowledge alone will not necessarily motivate taking this action. To bring about this action we need to base our behavioral change communication on locally relevant research that identifies priority groups for change and their determinants of change9 . Educational and behavioral change communication materials used by the project were adapted from previously developed materials, and not based on behavior change research in relationship to the current target population. Consequently, they may not be as effective as they could be. In a Sofala, Mozambique study, also using one community health worker per 12 households, achieving large and statistically significant behavioral and nutritional changes resulted from baseline basic behavioral change research10 . This baseline behavioral change research was credited with contributing to the large magnitude and statistically significant behavioral and nutritional changes achieved 9 Technical and Operational Performance Support Program . USAID Designing for Behavior Change For Agriculture, Natural Resource Management, Health and Nutrition. 2013 10 Davis TP, Wetzel C, Hernandez Avilan E, de Mendoza Lopes C, Chase RP, Winch PJ, et al. Ibid. Glob Health Sci Pract. 2013;1(1):35-51. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 19 2.4.3 Evaluation Question No. 3: To what extent did the project resources (inputs) lead to results? Could the same results been achieved with fewer resources, or whether alternative approaches could have been adopted to achieve the same results? 2.4.3.1 Agricultural Productivity The activities to increase agricultural productivity of selected crops were successful primarily because the project was able to station 27 extension workers at the districts and localities using the Training and Visit (T&V) approach. The project also developed operational guidelines with clear criteria for selecting communities, creating Associations, and Unions. This allowed the project to introduce a uniform approach across districts. Introducing the T&V approach to extension with the Unified Extension System combined frontline workers in different agricultural sub-sectors that included crops and natural resources conservation into one system. ADRA subsequently modified the T&V system from the individual contact farmer to the contact group approach. One extension officer worked with 10-12 groups of 20-30 smallholders, amounting to roughly 250 households. The project achieved almost all indicators except for legalizing farmers Associations. The difficulties to legalizing Associations resulted from farmers needing to pay for the legalization fees, obtaining their identification documents, and writing a constitution. ADRA extension workers often took the lead in explaining to members the advantage of having their Association legalized. The unintended consequence of this activism was that some members referred to their Association as the “ADRA Association”. Therefore, they were expecting ADRA to pay for the legalization. The most deleterious impact of constantly changing the LOA of the MYAP was eliminating the ability to modify the T&V approach with a more participatory and discovery-based extension approaches during a “normal” mid-course review. This would have resulted in a more participatory approach based on principles such as value chain development and market orientation emphasizing demand-driven services, downward accountability, community extension management, and farmer-to-farmer extension approaches using innovative farmers. With this approach, extension staffs require additional and different kinds of knowledge and skills, such as on developing business plans and the capacity to interact with different actors in the value chain. The concentration shifts from not only transferring technology, but also to facilitating innovation. There is little adoption and hence even less innovation without markets or without input and services suppliers. The extension worker becomes a specialist who knows how to bring different sources of knowledge together (farmers’ knowledge, research knowledge, market knowledge, etc.) for the benefit of the farmer. Unions are using the CDR as a way for increasing surplus, demanding that farmers’ Associations increase their CDRs in size and numbers to boost production. Farmers tend to see this approach as returning to collective farming -- the system adopted by the Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 20 Mozambican Government after independence. When asked if they prefer to work in CDR’s rather than their individual farms, farmers were unanimous that they prefer to work on their individual farms. CDR’s demand additional labour from farmer Associations, so whenever there is a delay, in rainfall for example, they give priority to their own fields. An alternative for CDR is the Innovative Program for Technology Transfer (PITTSA) approach based on the traditional concept of transferring knowledge through extension to farmers by establishing one-hectare farmers. The associations demonstrate a full technology package in this CDR as recommended by the extension agents, but cultivated by the extension worker. This strategy assumes that new technologies can drive innovation and hence increase production, marketing and competitiveness. The current experience is that the market is the main driver of agricultural innovation, and requires technological and institutional change. The PITTSA extension model is useful in bringing research and extension together, and is generally attractive to extension workers who are already have a “machamba” and see this as an incentive. The most critical concerns are the effects on female extension workers and the complete absence of a market or business orientation. Some farmers have reached the limit of their production capabilities once they increased to two-three hectares. They have no capacity to increase their production areas using hoes as the main technology. For these graduated farmers, the project should have introduced other production technologies, e.g., animal traction, etc. Although, the numbers of animals11 have been increasing in the area, there is no tradition and knowledge of using animal traction. The Evaluation Team was concerned with the continuity and availability of improved seed varieties after the closure of the project. An effective way to improve seed availability is by supporting small-dealers supplying seeds in small quantities. Building networks of these agro-dealers enables the seed supply network to become more accessible to farmers. Focused training courses can build the capacity of the agro￾dealers to provide advice on cultivating the seed varieties they supply. Training could also support demand analysis and inventory control to avoid excess unsold inventory. 2.4.3.2 Commercialization The project design focusing on commercial agriculture, as well as the ADRA staff qualified to implement this type of project were critical assets for this project to succeed. The MYAP encompassed an extensive geographical area, consequently adequately 11 Animal breeding also represents an important activity for the economy of Mocuba, Lugela, Maganja da Costa and Pebane districts; there is a visible increase in animal production within the districts that contributes to job creation and production of animal protein. The family and enterprise sectors practice animal breeding, especially in the breeding of goats, pigs, chicken and cattle. In Maganja da Costa, there are about 30,000 people involved in animal breeding activities at an average of six head of cattle in the area of direct influence, most of them on a family basis. Lugela has much less cattle with approximately 4,000 producer families. In Mocuba, there are 24,000 people involved in animal breeding activities, most of them on a family basis. (MAE, 2012). Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 21 reaching the targeted communities with enough frequency to affect the changes desired required mobilization resources such as vehicles, fuel, maintenance, etc. Training programs were critical, as an entirely new and unfamiliar (to the beneficiaries at least) form of marketing was being proposed. Farmers in this area are not accustomed to joint marketing, and the effort needed to overcome long-held fears was only possible through a training program. Demonstration plots are an essential part of any agricultural program. Producers who live in a precarious agricultural environment are particularly astute at “seeing before believing”. Without ADRA facilitating contact between the Unions and potential commercial clients, the increased marketing volumes and gross profits would not have occurred. We have ample proof of that in previous agricultural programs in Mozambique, and elsewhere, where production-driven projects simply fell off or disappeared completely once project inputs ended. The literacy and basic business training program implemented by ADPP was instrumental in establishing the base-line knowledge and ability required for any economic enterprise. It also had an important impact on women, as the training empowered them within the household. Calculating cost-per-beneficiary can be a sometime useful tool to benchmark benefit￾cost analyses. However, confining projects to large urban settings where the cost of accessibility to the beneficiary population is quite low would result if it were the primary consideration. This MYAP covers a significant portion of Zambézia Province. Therefore, it is logical that implementing costs will be higher. We could find no evidence of extraordinary expenditures from visually surveying ADRA’s physical facilities, and ADRA has the reputation among US NGOs as having among the most modest pay scales. Their marketing approach was devoid of extraneous or inappropriate initiatives that would have led to dilution of focus on the central achievement of increasing income. In this regard, extreme caution is required to “integrate” a commercial activity with one emphasizing health, nutrition, etc. What may appear as a “good idea” or “mutually supportive” activities can easily undermine the economic initiative. Alternative approaches are always possible to achieving any desired goal. From the data we have been able to review, and the discussions we have had with beneficiaries and other key person interviews, it is evident that the commercialization approach chosen by ADRA fit well within the cultural context. It permitted a relatively short time￾frame for the communities to actually see increased incomes. The requirement that each association member mentor another community producer gave the basis for a positive spread effect among the communities, as well as assuring sufficient supply of commodities for sale. This also reduces the MYAP’s cost-per-beneficiary to the extent that it occurs. 2.4.3.3 Health and Nutrition The MYAP’s achievements are important because there is good evidence that all the key behavior changes promoted by the project and the key methodologies that the project used to deal with them -- monthly growth monitoring, monthly visits to Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 22 households, and mothers’ groups to control malnutrition -- are effective in reducing childhood malnutrition, morbidity and mortality.12 , 13 Interventions, including behavior change communication to improve child health in poor rural populations miles from the nearest health facilities in developing countries, must reach mothers at the household level at least once a month to be most effective. Child Survival interventions delivered only at the health center level have not have been proven effective in doing this. Similarly, outreach patrols from health centers and other health facilities, while effective for immunization campaigns, have not proven effective for the behavior change that is necessary to deal with severe malnutrition, malaria, and diarrheal diseases. There is also good evidence that community level mothers’ groups, such as those using the Hearth approach, used initially and the broader approach used in the latter years of this project do decrease childhood malnutrition in an ongoing manner. Better sanitation requires providing at least concrete slabs for latrines. As the project results show, this combined with good behavior change communication at the household levels leads to better community level sanitation. How close the local potable water supply is to the household inversely correlates with the incidence of diarrheal disease. Therefore, there is no effective way of improvement other than supplying good potable water through bores and wells, as close to the community as possible. 2.4.4 Evaluation Question No. 4: What are the medium and long term effects, both intended and unintended, of a project intervention? Were effects due to the project intervention and no other factors? 2.4.4.1 Agricultural Productivity Project interventions were able to reduce household hunger periods from initially six months to four months, and increase frequency of meals per day to two. Results from the field survey (ADRA 2013) also show the average number of months households have enough to feed themselves is 7.8. The period from December to March is the most difficult time of the year in terms of providing food to the household. June to September is a period in which families have enough food available. Hypothesizing that farmers will continue to produce using improved technologies, will continue to use improved seed varieties, and will continue to have access to markets, the hunger period should be reduced over the medium term. 12 P.Freeman, H.B.Perry, S.K.Gupta, B.Rassekh Accelerating Progress in Achieving the Millennium Development Goal for Children through Community-Based Approaches Global Public Health Vol 7 No 4 April 2012 P 400-419. 13 USAID MCHIP Building on the Current Evidence to Strengthen Community- Based Service Delivery Strategies for Promoting Child Survival 2011. . Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 23 Production is a way for increasing household consumption and for increasing household income through marketing. Historically, many of the poorest households sold immediately after harvest when prices were at their lowest. Access to markets was a major constraint for many households, especially for the more remote communities. The market price information provided by radio or SMS assisted farmers understand the value of their crop and allowed them to negotiate fair prices when they sold. ADRA prioritized group formation to improve supply of commodities as well as link them up with broader markets. The majority of farmers use traditional systems of covered raised platforms made with tree bark for on–farm grain storage that provides only rudimentary protection from pest and disease. OSANZAYA promoted an improved grain storage design called the Gorongosa storage system. This met with some resistance because it requires cement, which is beyond the means of many farmers. However, with improved marketing leading to higher prices, farmers were willing to adopt them. This significantly reduces the post-harvest losses in the medium and long-term. The design is very similar to traditional storage systems widely used across the Sahel, but the storage vessel is made of fired clay. A useful and practical experience would be to see how the improved design can be adapted to utilize local construction techniques. The MYAP introduced improved farming practices based, in part, on the local production system, which improves the chances that they may continue. Transferring technologies using existing production systems are important for increasing the adoption levels. Technical changes are typically slow. Time is required to both test the effectiveness and efficiency of a new technology and to build awareness and confidence sufficient for adoption. OSANZAYA’S farmers adopted early because they observed significant improvements in combining improved productivity and market access. Farmers implemented incremental changes that are relatively easy shifts from existing practices, rather than technical leaps that require major changes to the way they work. Success with simple changes can open the door to further, more complex ones. It is important that agriculture extension services transform and facilitate the demand-led services that empower communities to take control of their livelihoods. That transformation requires continuing producer linkages to the best ideas for achieving more secure livelihoods. The emergence of “innovative” farmers'' has led to some farmers experimenting independently. The fieldwork during this evaluation specifically sought out farmers who were innovating by using a new technique with other crops, changing their cultivation practices, or adopting a new crop or copping pattern. It was evident that some farmers are experimenting with modifications to their farming systems or to techniques – and are doing so successfully. Some of these were not radically new techniques, but modifications and refinements of existing ones. Often it is the more affluent who can afford to take the heightened risks involved. Not all were successful, e.g., one farmer tried to use mulching with cassava production, but it failed because the mulching attracted insects that ate the cassava. The impact of these farmers goes far beyond their limited numbers. These innovators are well known and their experiments and Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 24 improved practices provide a demonstration of what is possible in their communities. Field extension staffs, because of their interactions with the community, know of these innovative farmers. The degree to which this knowledge is disseminated by the farmers themselves or by OSANZAYA’s extension services, happens in a structured and planned manner through the CDRs. This structured approach, testing new techniques under local field conditions, provides an additional “laboratory” wherein the experiences gained can then be extrapolated to other crops. Farmers are increasing their production areas and diversifying their livelihood strategies due to increased money circulating in some areas Considerable enterprise diversification is occurring, e.g., combining rain fed crops, increasing production areas and hiring additional labor, increasing livestock production, increasing number of small retail stores “barracas”, and using forest products to make mats, bags, etc. The exact nature of the enterprise mix varies depending on the individual’s household location and economic wealth. Rainfall (delays and excess) have affected agricultural production during the MYAPs implementation. Delayed rainfall affects the farmers’ capacity to have a successful season, particularly for maize and pigeon pea production. This is more prevalent in the coastal districts of Maganja da Costa and Pebane. For example, a Union located in Maganja da Costa had lost 50% of its pigeon pea production during the 2010-2011 agricultural season due to inadequate rainfall. 2.4.4.2 Commercialization The intended medium and long term effects are the vast majority of the current number of Associations and Unions will remain actively marketing their agricultural production. Moreover, there will be an ever-increasing number of Associations, Unions, and Cooperatives forming throughout Zambézia Province in response to more widespread household income distribution. Unintended consequences could be an elevated level of corruption within these organizations that leads to their downfall. The MYAP has tried to prevent this by requiring rotating membership within each Directorate. Nevertheless, there are amounts of money here that are very significant among the beneficiaries, and transparency in all business transactions is necessary. Any loss of trust among the groups will be disastrous. A second unintended consequence will be if the groups lose their “business first” orientation, and try to incorporate non-economic services. From the number of failed donor-funded projects around the world that tried to promote cooperatives, it is all too evident that any deviation is usually fatal. Governments often promote cooperatives as an easier mechanism to “control” disbursed rural populations. A third unintended consequence would be an attempt by the buyers to fragment the Associations, Unions, and Cooperatives so that the producers are forced to return to Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 25 individual sales. Because the Union had alternative buyers, one buyer’ attempt was thwarted. As noted earlier, without a control group it is difficult to assign direct attribution to the MYAP’s commercialization activities. Neither agricultural production nor marketing operate in an isolated manner because many forces both within and outside of their control affect them. Based upon the data and interviews we know there has been an increase in rural incomes among the direct and indirect beneficiaries. And considering that the approach of joint marketing has had a positive effect on marketing elsewhere in the world, it is logical to conclude that without the MYAP these improvements in rural household incomes would not have occurred. 2.4.4.3 Health and Nutrition The project achieved the generally good coverage and behavior change results through establishing community based organizations and health volunteers, a well-developed systematic program of community-based training, and supervision linked with a well￾developed community information system. Providing basic supplies, such as forms, education materials, flip charts, and scales, was also key. Project activities supplying materials for latrines and new and repaired water sources also contributed to the gains made in health and nutrition. These changes should last for the medium term. If maintaining the project activities for at least another five years occurs, lasting improvement should result in childhood nutrition and health, some community health related behavior, and community health organization. However, behavior change communication methodologies need to be strengthened further to cover all areas of health behavior as noted in 2.4.3.3 above. The CLC focus groups and key informant interviews of community health workers also indicated that community members developed skills in community organization and planning relevant across many other sectors. The members of CLCs included all cadres of community health workers and the members of the mothers and fathers groups. Ninety-eight of the 107 participants in the CLC focus groups were community health workers or members of mothers and fathers groups. Much of the training in the communities in health took place through the CLCs. Many people in the community received training in community mapping, planning, and reporting on their activities, as well as a wide variety of topics beyond their area of basic training. Quarterly reports listed over seventeen areas of training through the CLCs. There is good evidence that the adult literacy program, implemented as part of this project, would most likely also have been a contributor to the behavior change, training, and other results achieved14. However, it is not possible to say to what extent from the evidence available. Given that, one-half of all adults 19 and older in ZambesIa were 14 P Sandford, J Cassel, M Montenegro, R Sanchez The Impact of Women’s Literacy on Child Health and its Interaction with Access to Health Services. Population Studies Vol 49 (1) 1995 Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 26 illiterate, and that community health workers clearly need literacy and numeracy skills to fulfill their roles15 , literacy must be important. 2.4.5 Evaluation Question No. 5: What is the likelihood that the benefits of the project will endure over time after the completion of the project? Has the project planned for the continuation of project activities, developed local ownership for the project, and developed sustainable partnerships? 2.4.5.1 Agricultural Productivity Currently the main source of production credit is the District Development Fund (FDD) that allocates approximately seven million meticais (Mt) per District per year. Legalized farmer Associations and Unions can access this investment fund for long-term needs. However, the Evaluation Team heard concerns that the repayment term (usually three years) was too short for some investments, and that it is difficult for poorer farmers and women to access this fund. It was beyond the scope of this evaluation to examine the working of this fund, but the fund criteria merits review to establish the validity of these concerns. Broadening access to the FDD should be a priority for the government. Access to extension services for the direct beneficiaries was very good, particularly through the CDRs. However, it was very limited for individual farms of association members, and non-existent on indirect farms. Exacerbating this situation is the limited capacity of the public extension services to deliver their services. Although public extension services cover all five districts, employing 34 extension agents and technicians, they serve a mere 11% of all farming families.16 These government agents are responsible for disseminating information on technology, particularly in the household sector, which accounts for the bulk of food production. However, the extension services can only be effective in their mission if they are equipped, deployed to the localities, and have human and financial resources. The District SDAEs are responsible for all agricultural, livestock, forestry, wild life, and fishery activities in the districts. They are also responsible for local mining, tourism, marketing, and other local economic activities. The veterinary officers undertake extension work for livestock, while the forestry and wild life officers largely enforce regulations. ADRA’s exit strategy was establishing a partnership with SDAE through cost sharing with local extension infrastructure. However, the Evaluation Team found that ADRA covered most of the expenses. This partnership foresaw the SDAE incorporating OSANZAYA’s service delivery to all beneficiaries after the project’s closure. All interviews, including with the district directors of SDAE, found the expectation is that the SDAE will continue with activities, but with less capacity due to lack of human and financial resources. 15 . Mark Langworthy Zambezia Needs Assessment Survey Report of Findings Tango International October, 2007 16 It is estimated that the current coverage ratio is about 1 extension agent per 230 producers. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 27 Markets will be the driving force behind continuity of OSANZAYA’s agriculture activities. The distance from the main markets in Mocuba, Nampula, Quelimane, and Alto Molocue will affect the permanence of most associations to continue with improved productivity and production. There will be differences among the districts, e.g., some of the Associations may continue with recommended agricultural improvements that significantly influence their household food consumption rather than household income. Farmers that have improved productivity and increased food availability for household consumption may continue with these activities if they do not represent additional labor costs or needs, while communities with better access to markets will have greater opportunities for increasing household income. Mocuba is a dynamic center of agricultural marketing and transportation network, and is one of the six urban areas with municipal status in Zambézia Province. Municipalities and district centers are important as market hubs for services to rural areas and for marketing agricultural produce. A number of wholesale and retail traders operate in Mocuba. Their importance as potential rural growth poles with social infrastructure (transport, telecommunications, trading establishments, storage facilities, banks, administration, schools, health facilities, etc.) is partially attributable to the high and often diversified agricultural production in the surrounding rural areas. Recent opening of new access and tertiary roads has considerably increased trade from neighboring districts. As a municipality, Mocuba can successfully function as a rural growth pole and a center for local economic development because it has the requisite degree of administrative and financial autonomy. Therefore, Associations from districts located closest to Mocuba have the highest probability of continuing their activities. MYAP beneficiaries may continue activities on their individual farms, but use the Associations as selling mechanisms. CDRs are labor consumers and farmers tend to focus on their individual farms rather than producing in large fields. Farmers see this approach as the reintroduction of collectivization forms of production. For more remote communities, i.e., Lugela, Pebane and Magaja da Costa, with less access to sizeable markets, the impacts may be more on household food availability than on cash income. ADRA’s has produced booklets using picture cards/images to increase beneficiary participation and learning as refresher materials. Without project extension workers, farmers will rely on these instruments to maintain the standards, quality and innovation. The Evaluation Team was able to review booklets on technologies and negotiations, but unfortunately, they are available only now that the project has ended. Providing these during initial training and practices in the CDR would maximize their effect. ADRA must make it a priority to distribute these to all Association and Union beneficiaries. Activities to be undertaken by another project will continue, i.e., ADRA’s PRODEZA, where improved production is being promoted in a technical way that involves understanding and responding to farmer priorities. There remains insufficient member empowerment in Associations to build their capacity to continue with the activities, and a dependency culture exists amongst many. The PRODEZA program will outsource their extension services to promote practical ideas on improved cultivation, including Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 28 introducing improved crop varieties in Maganja da Costa and Mocuba Provinces. Existing Associations will also support many of the poorer households by sharing costs and benefits. Unions will continue to work on marketing issues, and therefore those linked to PRODEZA will continue to operate. 2.4.5.2 Commercialization The current best direct evidence we have that the benefits of the project will endure over time comes from the field interviews with the Association and Union membership. Their response to our direct question asking if they will continue was unanimously positive. Good intentions alone will not ensure survival, but the prospect of earning more money will. Not all of these groups interviewed were of equal strength or stage of development. Invariably, some will probably fail. Nevertheless, stronger groups may also emerge as the fundamental purpose of combined marketing disseminates over time through the normal process of social discourse. The best indirect evidence we have comes from the donor projects that have promoted cooperatives. The USAID (Alliance for Progress) experience is telling in this regard. The Alliance for Progress and its predecessor fostered two types of cooperatives throughout Latin America. To advance programs in literacy, health care, and some combined purchasing of inputs, Agricultural Cooperatives were established. The second were Savings & Loan Cooperatives, aka, Credit Unions. Today it is a practical impossibility to find the Agricultural Cooperatives sponsored by USAID’s predecessor agencies. In contrast, the S&L Cooperatives are working well. In fact, the second largest financial institution in Ecuador is an S&L that began out of the cooperative program. The clear difference between the success and failure was a business outlook and strategy from the outset. For the most part, ADRA has adopted a facilitating role in its commercialization approach. They have gone beyond that at times by supplying sacks to the producers (reported 30%), and provided transportation of produce to warehouses. In each case, it was early in the project, and there was a particular need. For example, the first sale to WFP was critical if this buyer was to continue purchases from the MYAP's producers, as were early sales by those groups who were more distant and isolated from markets. Both of these subsidies have ended, and Association and Union members interviewed have stated that they are able to purchase the sacks and hire transportation. ADRA has produced two guides, one for commercialization and one for grades and standards. ADRA distribute these, and they will serve as a reminder of the key ingredients needed for successful sales. Sufficient sales have occurred so that the Associations and Unions and the potential buyers know each other and have a history of completing negotiations and sales transactions. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 29 2.4.5.3 Health and Nutrition The MYAP’s benefits should endure in the short to medium term after the project ends. The Evaluation Team found key behavioral changes established in relation to nutrition, health, and sanitation, but not in association to all desired behaviors. Focus Group discussions and key informant interviews demonstrated local ownership of the networks of CLCs, Community Health Volunteers, IMCI volunteers, and mothers and fathers groups. However maintaining these changes for the longer term requires some supervision and continuing educational support. The CLCs and local community health workers have established sustainable partnerships in many areas. Nonetheless, there is some doubt that is the case in all areas. Given the large number of community health workers involved (over 2850 Community Health Volunteers and 4,222 Hygiene Promoters health center staff alone are inadequate to give adequate supervision and continuing training to more than a limited number of workers. In the CLC focus groups community health volunteers remarked that they forgot what they were meant to do when the ADRA technical staff did not visit regularly. However, managing this problem is possible if some experienced community health workers could take on this role at the community level. Health focus group discussions noted maintenance as a major problem in providing potable water sources. The Water Supply Bore Drilling Pump Repair Specialist recommends that users should contribute to a common fund that hires a trained fulltime local worker who maintains all local water sources. In Sofala, community health volunteers, one per 12 households educated their beneficiaries regularly about nutrition after being educated themselves in Care Groups by paid Health Promoters. Achieving highly significant reductions in severe malnutrition in children less than 23 months of age at low cost is possible. However, achieving this used the approaches to behavior change communication described above in 2.4.3.3. ADRA’s technical personnel fulfilled a similar role to the “Health Promoters” in the MYAP. 2.4.6 Evaluation Question No. 6: What are the lessons learnt, success stories, areas of improvement, and recommendations for similar programs? 2.4.6.1 Agricultural Productivity Training community members on specific topics and providing them with the necessary materials to implement the activities is critical to increase rates of adoption. This can also have a huge success if the training is conducted within local agricultural practices rather than bringing in a completely new set of activities. One reason for the success of the farmer Associations and their respective Unions is linking the Associations to Unions who can then target buyers and offer them attractive prices for selected crops. Increasing the interest of indirect beneficiaries (farmers who are not members of the farmer Associations) to participate in the joint sales of produce Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 30 permitted the entire community of farmers to benefit through increased family incomes, and not just the lucky sub-group who happen to be members of the local Associations. Communicating to farmers within the 276 farmer Associations on how to increase crop productivity, how to select for higher quantity and greater yields, how to improve storage, and how to meet market demands for specific products was effective. Literacy allowed producers to adopt the better the technologies faster. Extension support needs to move towards a more empowering strategy, rather than an expert-driven approach, to build greater self-resilience within communities. The process begins with CDR for two years, and then promotes the Farmer Field Schools used elsewhere in the country. This process also supports the incremental changes that carry farmers forward in a longer process of technology adoption and change. Locating the extension workers at the community level is a more effective way for transferring technologies since close proximity allows working closely on a permanent basis and arranging exchange visits among farmers. Rehabilitating the storage places at Associations and Unions and introducing 100 improved Gorongosa-type silos at the community level proved significant, as did linking community-built centralized warehouses storing hundreds of tons of commodity for bulking and large commercial sales. Short cycle projects (3-4 years) increase the pressure for more direct intervention rather than empowering local people, since the project has to meet its indicator targets. This limits community participation and ownership. 2.4.6.2 Commercialization The most important lesson learned is that agriculture is an economic enterprise and commercialization is to be taken seriously if rural household incomes are to rise based upon agriculture. The assumption that “markets exist” or that the challenge is to produce more and selling is inherent is false. Almost all rural dwellers are connected to and dependent upon markets. It is that connection that gets them the items they cannot produce, such as salt, oil, sugar, clothing, medicine, food, etc. Each of these essentials has a cost, and therefore income must be generated. A well-designed market-led program can best meet that need. The best success stories were provided in response to the question asking how the additional household income was being used. Responses included: buying a bicycle or a motorbike, purchasing corrogated metal roofing for the home, buying clothing, buying more food, etc. One particularly telling response was a father saying that he could now pay his childrens’ fees to attend secondary school. Three areas require attention moving forward. The first is the post-harvest technology of grain storage. Losses between harvest and sales can entirely negate a potential Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 31 sale. All grains come with insects. When these hatch, the larvae begin to eat the grain, destroying its economic value. Affordable grain storage silos are available, and ADRA has used both the Gorongoza and FAO metal silos in the MYAP. Interviews with producers revealed that many understand the economic benefit of storage and selling at a date more distant from the period of harvest. This potential economic benefit can be entirely lost simply due to pest infestation of the produce. This should be the number one priority as a next step – in particular at the Union and Cooperative levels. The second area is enterprise education. The training given in the project met the basic need very well. Without that initial training, including literacy, producers would have had an extremely difficult task of understanding and adopting the elements of a market-led approach. The recommendation is to continue to provide training that is usually found in other programs for small businesses, e.g., production planning, financial management, cost budgeting, etc. Reviewing the value of commodities sold demonstrates that agriculture is a business, and further training in small-business will further the sustainability of the Associations, Unions, and Cooperatives. The third need expressed in the interviews was for some form of mechanization for land preparation. Although this more properly belongs in the Agriculture section, it does have a strong bearing on marketing. Producers stated they had more land available for production, but were limited from expanding because they had reached the limits of the available manual labor. The ADRA ARR found that there is not a surplus of labor available to hire during the peak demand time of land preparation and harvest. Serious analysis and discussion need to occur with the purpose of relieving this constraint. 2.4.6.3 Health and Nutrition Overall most nutrition, health and sanitation related objectives were achieved. However, objectives were not achieved in some behavior change areas. There is good community ownership and support for sustainability of community based health activities post project but the extent to which these will be achieved will be influenced by the MOH and the logistic problem of providing reasonable supervision and continuing education of so many community health workers. Providing behavior change communication down to the household level through community workers, such as the community health and sanitation volunteers, leads to good coverage of beneficiaries and widespread improved knowledge and behavioral change. Better approaches to behavioral change communication are needed for more effective sustained behavioral change in the variety of nutrition used, continuing breastfeeding from six months to two years of age, and preventing HIV and malaria. Combining mothers and fathers in the same groups is an effective way to improve male knowledge about child nutrition and their active participation in improving the lives of their children. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 32 Combining agricultural diversification and crop improvement aspects with nutrition in the same program leads to diversification of the nutrition for children. Community Leadership Councils, or similar community level organizations, using the same broad approach to training, involving community level leaders and representatives of all groups of community health workers and organization within the community must be a part of future MYAP projects. Moreover, future MYAPs should train community health workers, selected based on their skills and community support, as basic local trainers and supervisors of other less skilled workers. In turn these basic community level trainers would work with their local CLCs and health center personnel to provide overall health-related supervision and education. Strengthening behavioral change communication by using current state of the art methodologies that identify priority subgroups for change and their determinants of change is another intervention that will strengthen MYAP outcomes. 3 CROSS-CUTTING ISSUES 3.1 GENDER As with the other sections of this evaluation, gender responds to the individual evaluation questions one by one, primarily using the data sources of the project documentation (especially the annual reports), the final survey, the field observations and focus group discussions. The project did not disaggregate any of the data sources (such as the reports or survey), except in one analysis on the different types of crops planted by men and women (discussed below)17 . The focus group discussions, by contrast, included only women in each of the nine communities (the consultant dropped one Focus Group Discussion (FGD) on gender/resilience due to insufficient attendance), including 88 women in all five districts. The evaluation methodology section of this report describes the process by which the project identified communities, and people chosen for the focus group discussions. There was some randomization in some communities (the FGD used 10-12 women, if more were present; they either went to the other FGDs or departed). The focus group discussion instrument (included in the evaluation’s Annex IV) emphasized the effects of the project on women’s lives: decision-making, water access, agricultural and health practices, association membership, as well as sustainability. Consequently, for gender, the basic relevant project metrics provided a broad picture of 17Note that the project disaggregated none of these indicators for the two surveys, nor on the indicator performance tracking table (IPTT). According to the ADRA M&E Director (acting), this type of disaggregation was not required at the start of the project (2008); both surveys had the analytical capacity to disaggregate results (respondents’ sex is a code on the survey form). In a meeting with FANTA and USAID/FFP, the stakeholders decided to replicate the baseline exactly and not present results in a disaggregated fashion. (ADRA M&E Director (acting), personal communication, 14 December 2013). Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 33 change under the project’s interventions, while the FGD provided the context and experiential change, rather than a comparison with context and more quantitative data. This analysis also used the data from the final survey report. 3.1.1 Evaluation Question No. 1: To what degree did the project activities meet the needs of the project beneficiaries and is aligned with the country’s agriculture and development strategy and USAID/Mozambique development goals, objectives, and strategies? USAID/Mozambique’s results framework (at the time, as the Mission is now revising its strategy and the draft was not yet available to evaluation team) did not explicitly include a section on gender. The Assistance Objective (Inclusive Growth of Targeted Economic Sectors) implies a focus on improving access to resources, and includes stunting as one of its key indicators. Most of the results framework elements included in Intermediate Result 1 (Agricultural Productivity Increased) emphasizes improving agribusinesses and increasing access to markets, agricultural technologies. ADRA’s MYAP works well within both that framework and the Food for Peace framework definition for food security as access, availability, utilization, and risk management. None of these elements need be exclusively one gender or another, and ADRA’s approach emphasized that inclusivity. In this respect, the project design was well within the USAID framework. There was no time during the evaluation to interview officials at the Mozambican Ministry of Women and Social Work, so how well the project fit into the national program is unknown. From one interview, this is a relatively weak ministry in Zambézia. However, the project management noted good relations with both the agricultural and health ministries – and particularly bad ones with the ministry tasked with disaster preparedness, so it seems likely that the project operated within national policy norms. Women’s participation in both the village Associations and the various village-level health activities (in particular, the men’s and women’s club) gave them the opportunity to use experiences gained from one activity to affect results in the other. For example, the profits from the joint sales gave women funds that they used to purchase household goods, and make improvements to the overall health of the family. Women purchased plates, pots, mattresses, as well as food items; most women now washed their clothes more frequently with the improved access to water. They built drying racks for their dishes, as well, and tried to ensure that each child had their own plate (rather than eating from a communal plate). Taking sick children to the hospital was often by the father on the bicycle that he had purchased with funds from the joint sales of produce. Access to Improved Water Source. Seven of the nine FGDs reported the presence of an improved well that generally reduced the time it took to get water -- often to under an hour. Access to water without the improved well took at least two hours, and, in one case, it still takes all day. The walk to and from the well takes time, and there is often a queue to get water, so even more time is spent waiting for a turn at the pump. The walk home from the well goes more slowly, as the filled buckets and jerry cans are heavy. The women also noted that, given the option of going to a traditional well or the improved one, they will go to the improved well because the water is cleaner and there Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 34 are fewer bugs. When the well is closer, women can go to the well more often, and do not feel the need to ration water as before. This allows them to practice the hand washing and other hygiene behaviors promoted by the project. 3.1.2 Evaluation Question No. 2: Has the project achieved its objectives? To what extent did the interventions contribute to the expected results or objectives? The project, based on its final IPTT, achieved at least 90% on all of its monitoring indicators except one (knowledge of malaria symptoms), and most met more than 100% of their targets. The impact indicators are much less positive for production and overall consumption patterns, but are closer to the life of activity (LOA) targets for the childhood nutrition variables. As there were few other projects operating in the project districts, and minimal extension activities by the government, the community outreach and technical assistance provided by the project are key factors in achieving those monitoring results, and contributing to the impact ones. With the emphasis on gender, however, it is interesting to focus more on the changes to stunting and underweight, as the FGDs provided frequently emphatic testimony to the changes in child health and nutrition that the women achieved with their progeny. The results in the table represent changes in the overall population. The women in the focus groups, by contrast, were all participants in the project activities. Women in all nine FGDs noted that their children were healthier. The enriched porridge contributed to children crawling and walking earlier, and definitely made a difference between the older and younger children’s development. Growth monitoring and the active community watchfulness (through the mother’s and father’s groups, especially) meant that families with children who were faltering were able to understand what was happening, and the health groups encouraged them to seek help. Children liked the taste of the enriched porridge. Mothers added peanuts, coconuts, and vegetables in various combinations, although the most common addition was peanuts. Training in the four food groups meant that mothers were at least aware of the effect of different classes of foods on their children, even though they were often unable to grow or purchase those throughout the entire year. Other contributing factors to children’s health were the focus on exclusive breastfeeding and improved hygiene and sanitation practices. Table 3.1.2a: Nutrition Achievements Variable Baseline Final LOA Target Stunting (<5) 41.5% 37.3% 35% Stunting (<2) 20.8% 17.4% 15% Underweight (<5) 31.2% 26.1% 25% Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 35 Agricultural achievements for the women, as displayed in the table, show that the project-sponsored crops (peanuts, pigeon peas, maize, cashew, sweet potatoes, rice, and cassava) all experienced somewhat disproportionate increases among women. Some are specifically crops that the Associations marketed; however, all are crops that the families consumed. Women noted (Putine, Lugela Districts) that the Association had helped teach them how to determine what part of the harvest they needed to keep, what they could sell, and what they should keep for storage. While production varied depending on rainfall and sudden-onset weather conditions (flooding, for example), certain agricultural techniques resonated with the women. They noted planting in rows explicitly, and the remaining FGD noted only that they would continue doing what they had learned. All of the cassava fields the evaluation team saw were in rows, and each had an individual hillock to retain soil moisture. Association membership was almost universal among the FGD participants. The members noted that the proceeds from the joint sales went towards purchases of household items, as well as school fees, hiring field labor, agricultural inputs, bicycles, and savings. Only one FGD reported having savings accounts. In another case, the FGD respondents noted that it would not be cost-effective to set up a bank account in Mocuba, since the cost of transportation to the town would be more than their savings. Association membership meant that women had access to their own funds. The literacy training undertaken by ADPP meant that Association members also knew how to read the sales prices, and understand how much money they would be able to make, given the volume of their own production. They understand that the Association gets a better price for the joint sales. In one community, however, there had been a challenge with getting the buyer to come on time, and the maize had started to spoil in the storage facility. In another community, the farmers had planted peanuts two years in a row, and then pigeon peas (with the intent of selling those crops). But there had been poor harvests for peanuts, and the pigeon pea crop had been flooded, so the sales proceeds were lost. Because the farmers knew what they might have gotten for their harvest, this was frustrating. The women noted they might try a different crop for sale during the next growing season. Table 3.1.2b: Crops Cultivated By Gender Crop Sex Male Female Total 261 311 572 38.4% 45.8% 84.2% Cashew % of Total 52 59 111 7.7% 8.7% 16.3% Ground Nuts % of Total 201 233 434 29.6% 34.3% 63.9% Pigeon Pea %0f Total 247 304 551 36.4% 44.8% 81.1% Sweet Potatoes 82 85 167 12.1% 12.5% 24.6% Rice % of Total 76 101 177 11.2% 14.9% 26.1% Cassava % of Total 279 316 595 41.1% 46.5% 87.6% Total %of Total 319 360 679 47.0% 53.0% 100.0% Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 36 3.1.3 Evaluation Question No. 3: To what extent did the project resources (inputs) lead to results? Could the same results been achieved with fewer resources, or whether alternative approaches could have been adopted to achieve the same results? Key informant interviews (with the Chief of Nampevo, Ile District, and the District Health Officer in Pebane) noted explicitly that the community outreach and follow-up were critical in sustaining training advances for both agriculture and health activities. In both cases, the officials noted that there were key health differences between the ADRA communities and others, and attributed that to the community outreach. Some of the health changes the Chief noted included: increased ante-natal care, fewer cases of (gestational) anemia, increased use of enriched porridge and health-care seeking behaviors for children. Some of the agricultural changes she had noticed in Ile included planting in rows and post-harvest storage. For Pebane, there is a close interaction between the community volunteers and the health center, including twice-monthly meetings at the health center. Referrals from ADRA communities for health care are higher, as are community volunteers actually helping families with transportation to the health center. This type of field supervision and follow-up are the foundation for a community-based development paradigm. Accessing the different communities routinely means transportation and labor, over difficult roads and seasonal driving conditions that further impede access. 3.1.4 Evaluation Question No. 4: What are the medium and long term effects, both intended and unintended, of a project intervention? Were effects due to the project intervention and no other factors? The project listed its gender strategy primarily as one of inclusivity (having women participate equally in project activities). From the broader USAID gender policy perspective, a strong gender empowerment effect pervaded all of the FGD findings. Table 3.1.4: Decision-Making Changes Topic Before18 Now Planting Men Both together (5/9); men with women’s input (2/9); men alone (2/9) Agricultural sales Men Both together (3/9); men alone (4/9); men with women’s input and veto (2/9) Seeking medical care Men Both or either (7/9); women (2/9) School attendance Men Men (5/9); both (4/9) Household purchases Men Both (6/9); women alone (2/9); men alone (1/9) 18 Single women or widows could make decisions by themselves, even before, and that persists. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 37 Attending social events Men; always both (1/819) Both (4/8); depends or other (3/8); men alone (1/8) Women having access to their own funds resulted in a clear shift in decision-making, as Association membership allowed each member an equal say in participation and an equal share in the proceeds. The literacy training allowed women to know what those proceeds would be, and to ensure that their husbands gave them the precise amount. Four FGDs noted that, before, the women did not know how much the harvest would bring, so they only knew what their husbands told them. Sometimes the husbands drank part of the proceeds, or used them to pay for debts about which the wives knew nothing. Now that each had their own source of funds, they could discuss how much they would be able to spend or save. Men still tended to pay for school fees (in one focus group, the listening children answered this question before the women could with a shout of ‘papa’). Either parent could unilaterally seek medical attention, although they would often both go to the health center. For example, the husband would take the sick child to the health center on the household bicycle, and the woman would follow on foot (or arrange other transport). The attention of the community health groups (the mothers and fathers clubs) also helped identify when children needed care that is more advanced. Of particular note was that now women felt that they could speak up about agricultural decisions, and could make decisions independently if they disagreed with what their husbands; this was especially true if the woman controlled her own fields. At the same time, one respondent noted that ‘if there was harmony in the marriage, you made decisions together’ (Dedere, Maganga District). There was no indication of challenge to the men in these changes in decision-making; instead, there is more respect for the women. The FGDs noted that this was due to the Association, which had resulted in higher prices for their agricultural produce (8/9). Association membership was valued (the women who were not members wanted to join), and directly tied with these increased funds. Having their own money meant that most women were able to purchase household goods directly (without asking permission), since the funds were considered their own. In the absence of the Associations, there would have been considerably less cash in the household, and thus, far less opportunity to improve domestic assets. Almost all of the women’s purchases concentrated around household goods or other types of family support (school fees or clothes); in only two cases did women note that they had purchased other items. In one case, a woman had already purchased corrugated metal sheets for roofing, and was saving her funds for the rest of the inputs necessary to build a new house. In the final case, the woman used her funds to procure additional agricultural labor. When the women talked about men using the sales proceeds to buy beer (prior to the Association), there was tremendous agreement among all the participants in each group. Now that women had their own money (and knew how much to expect when their husbands returned from the Association meeting), they also knew that their 19 One community skipped this question due to time. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 38 husbands had their own money. Husbands would also buy clothes or bicycles, and many would still drink beer – but not necessarily. Both spouses would sometimes buy presents for each other. 3.1.5 Evaluation Question No. 5: What is the likelihood that the benefits of the project will endure over time after the completion of the project? Has the project planned for the continuation of project activities, developed local ownership for the project, and developed sustainable partnerships? Both key informants noted that supervision was critical to the project’s results, and that their own resources would not be sufficient to maintain that level of supervision. For the chief, the functionality of the water committee was a primary challenge, as she had already had to intervene (with her own funds) three times to repair wells that the community had not been able to do. At the community level, there were cases where the FGD participants already noted some disparity in purchases (for example, one Association president had purchased a motorcycle – since that village Association knew what each member had contributed and what they had received, they wondered whether this was really simply from his share). Village Associations operate on transparency, and the relatively low levels of literacy and the visibility of goods out of the average both contribute to challenges to that transparency. ADRA is operating another health project in the same districts, and the messages and reinforcement that the community groups may need will continue beyond the life of the MYAP. The women, however, have a slightly different version of what will continue after the project ends, and a clear idea of why. Eight of the nine FGDs expected that the project’s practices would continue, because they have seen that these work. The women planned to teach their children, and have these practices continue, generation after generation. While not everyone is sanguine about the continuity of the Associations (which depend on both price and product), the groups noted the following as the major practices that they will continue: enriched porridge and planting in rows. Also noted were exclusive breastfeeding, hygiene/sanitation, and food storage. They noted that other people in the community wanted to join the Associations, and they wanted to continue the literacy training so that they would be able to attend school themselves (if it could be held at a time and place where they would be able to travel). For most of these communities, the project’s activities had stopped several months ago, yet there was still an obvious sense of pride in their new knowledge and the role that membership in the Association or participation in the health clubs had had on their own sense of accomplishment in the community. They knew that there was still help that they needed, but also that ADRA was leaving behind some practices that the communities could manage to maintain on their own. 3.1.6 Evaluation Question No. 6: What are the lessons learnt, success stories, areas of improvement, and recommendations for similar programs? Integrating Activities Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 39 The MYAP integrated its activities largely by proximity, rather than deliberately. Members of the Associations were also members of village water committees or community health groups, but the training and outreach events by ADRA staff occurred at different times. Originally, the project design was not to integrate these two, however, integration from the start might have increased the connection between the messages (for example, growing peanuts and using some in the enriched porridge for improved child nutrition). Because food security includes all of these components, the intention is for them to be mutually reinforcing; separating out agriculture from health/nutrition can help achieve the individual strategic objectives, but does not produce the interaction necessary to achieve the overall goal. It is true that this was not a requirement at the onset of the project, but it became one. And did not appear to be well represented in the focus groups by virtue of individual memberships overlapping in more than one track of the project’s activities. Community Outreach and Ownership Community outreach for all activities was critical to ensuring that the different messages (especially for health and sanitation) were disseminated and reinforced among the community members. This outreach meshed well with the existing health system, as it provided a first-step treatment for community management of some diseases, and community referral for other maladies. Membership in the village Association was a source of pride, as well as a source of increased income, and the role of ADRA’s TA in training for both health and agriculture/marketing was clear, mostly effective, and definitely pervasive. However, the communities do not necessarily yet demonstrate a sense of ownership in continuing their own outreach practices. The women in the FGD thought that the Associations would continue, although there was already concern about both transparency and utility; when the communities received funds, they thought the Association was important. When they did not receive funds, the utility of joint sales was not as convincing. 3.2 RESILIENCE 3.2.1 Evaluation Question No. 1: To what degree did the project activities meet the needs of the project beneficiaries and is aligned with the country’s agriculture and development strategy and USAID/Mozambique development goals, objectives, and strategies? USAID/Mozambique did not include a disaster preparedness/response element in their results framework germane to the MYAP development in 2008. Providing an outreach and response to communities to prepare for and respond to disasters was obviously part of the government strategy, and one of the three strategic objectives in the design of the project. However, the partner tasked with implementing this strategic objective was unable, despite more than a year’s negotiation, to respond to the concerns of the provincial representative. Even with the intersession of the national ministry, the project was unable to reach an agreement on the technical approach, and this required removing the entire strategic objective (and the contract with the partner) during year Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 40 three. While not all of the districts are equally vulnerable to cyclones or floods, all are somewhat vulnerable, and all are currently experiencing drought conditions. Deleting this component meant that the needs of those beneficiaries could not have been met. None of the communities, based on the FGDs, had disaster response plans. When pressed for their actions during a flood or cyclone, respondents stated that they took their families to high ground in the former instance, and took shelter in the mosque in the case of the latter. Moreover, the community would take the elderly or those who had limited mobility to safety as well. Resilience is more than disaster management, and it is the project’s other effects on food security and household income that reduce chronic vulnerability. The project’s inclusive approach, as discussed in the Gender section, led to greater access to resources by women. FGD information, responding to the questions in paragraphs below notes the relative vulnerability and the household’s ability to cope with shocks. 3.2.2 Evaluation Question No. 2: Has the project achieved its objectives? To what extent did the interventions contribute to the expected results or objectives? Because this element of the project was put aside, this question is no longer applicable. 3.2.3 Evaluation Question No. 3: To what extent did the project resources (inputs) lead to results? Could the same results been achieved with fewer resources, or whether alternative approaches could have been adopted to achieve the same results? While the communities have neither preparation for nor advance notice of disasters, they now have savings and other assets that they can use to recover from disasters and seasonal losses (such as drought or floods). Not all of these assets are evenly divided, and the coping strategies listed in the final survey persist in the responses from the FGDs. Respondents noted that they could now use bicycles to go to other communities, either to buy food or seek work. For shorter-term (seasonal) stresses, they would wait until the waters receded (for floods) or until the rains came (for drought). They would use their savings to purchase food for their family, shift to crops that were more drought-tolerant (cassava), and find wild foods. In the event of disasters, all but one community noted that they had received no assistance whatsoever. In the case of cyclones, the respondents in two (cyclone-prone) communities felt that it would be much more difficult to recover from these, as the cyclones sometimes destroyed houses. Even with savings from the Association’s higher prices for their crops, replacing a home requires the savings of many seasons (a corrugated tin roof can cost hundreds of metacais), and the rest of the building inputs are equally costly. 3.2.4 Evaluation Question No. 4: What are the medium and long term effects, both intended and unintended, of a project intervention? Were effects due to the project intervention and no other factors? Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 41 Six of the eight FGDs (one community skipped responding to these questions, since they had not suffered from disasters) noted that the Association would not help with recovering from the loss of crops. Two of the FGDs felt they were better prepared to withstand shocks because of the Association-related income and subsequent savings. 3.2.5 Evaluation Question No. 5: What is the likelihood that the benefits of the project will endure over time after the completion of the project? Has the project planned for the continuation of project activities, developed local ownership for the project, and developed sustainable partnerships? The final survey noted that household income had almost doubled during the life of the project. Participants in the FGDs indicated that they would continue with Association membership, as well as the numerous agricultural and health practices noted in earlier sections of the evaluation report. There is no explicit plan to continue the project activities, except the stated volition of FGDs; as in the earlier gender section. The performance of the Associations will likely be a major factor in determining whether or not the community members wish to continue participating. 3.2.6 Evaluation Question No. 6: What are the lessons learnt, success stories, areas of improvement, and recommendations for similar programs? Future programs will likely need to include resilience as an explicit objective, especially as this can help indicate a longer-term effect on households. And a progression from time-delimited food security interventions toward more sustained movement away from the most vulnerable. Programs such as ADRA, with its emphasis on market skills and integration, could fill an important niche in value chain analyses, moving communities from vulnerability through increasing degrees of viability. With the competition from Feed the Future, ADRA may want to consider developing an internal theory of change that includes both literacy and disaster preparedness into the more classic Title II food security paradigm. 4 MONITORING & EVALUATION ADRA operates under the Food for Peace Title II M&E protocols. In this case, it meant a population-based survey at baseline and final, with only a final evaluation (since it began as a three year-program). Indicators used at the impact level, and some monitoring indicators, are from the Generic Indicators list, with others drawn from earlier Title II programs and slightly modified to fit the results framework at the beginning of the MYAP. The project has conducted beneficiary-based surveys annually, in addition to the two population-based ones. The project developed detailed monitoring forms for all activities, including sign-in sheets for training activities and other outreach events, as well as ‘downstream’ forms to help the district coordinators sum up participation for monthly (and then quarterly reporting). The M&E team, based in Mocuba, maintained a library with hard copies of all of these community-based forms, and created a database by indicator to help with Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 42 quarterly and annual reporting. The database and subsequent reports are all kept in Excel, which makes it relatively easy to generate the tallies for reporting, as well as graphs (especially for the survey findings). The various implementers of the community outreach, that is, the extension agents for agriculture/marketing and health/nutrition, initially wrote all of these forms out. The M&E team conducted training for the extension agents, some of the community leaders, and district coordinators on how to complete the data collection forms, and does a periodic field visit on a random basis to crosscheck information, although the schedule is dependent on logistics (availability of vehicles and/or access to the communities due to the rainy season). The only indicator that relies entirely on farmer self-reporting is the amount of land under cultivation. The project has not had the resources to conduct individual measurements, but has trained farmers in measurement using the demonstration farms. With the project ending in three years, year three saw a significant staff turn-over, including the M&E staff, chief of party, and all of the technical coordinators; the annual year extensions have made it difficult to retain staff that might have otherwise stayed for the full five year duration of the original MYAP. The current staff of two has changed the reporting format somewhat, in that they enter data and aggregate only at the level of the Association (rather than individual members). The data still exist in the hard copies from the communities, but any type of disaggregation at the household or individual level would not be possible without considerable effort to enter information, and then validate to ensure that the households matched. 5 SUMMARY CONCLUSIONS AND RECOMMENDATIONS 5.1 Agricultural Productivity The major lesson learned by ADRA from other programs around the world is that communities are not homogeneous entities. They have found that community residents generally conform to three groups, i.e., innovators or explorers (who are those most disposed to accept and apply new technologies or approaches), followers (those who are willing to accept new technologies or approaches once they can see the results), and resisters (those for which any change is unacceptable). When a project implements a non-differentiating approach, it expends much time, energy, and resources on the latter two groups with little result. In this MYAP, ADRA identified sub-sets within the communities that formed the basis for the Association membership. They then limited the numbers of farmers in each Association that they could adequately support with each ADRA field agent. Requiring each Association member to reach out to others in the community provided a means for incorporating the second group (followers) to receive the benefits of the technological innovations. The second important intervention was to confine the number of “new” technologies or crops to the minimum number needed to demonstrate the effect, and using crops familiar to the beneficiaries. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 43 As noted earlier, introducing mechanization is the next required intervention to raise production and productivity. In addition, improving beneficiary “ownership” of the new technologies requires a more participatory approach to extension. Finally, community or group production (demo plots or association areas) was a strategy used by the project to transfer knowledge to more farmers than would be possible in a one-to-one approach. The Evaluation Team concluded that farmers often prefer individual productive enterprises and cooperative marketing, and the evidence was that producers gave priority to individual production when faced with labor or climatic constraints. Better options are available that avoid group production. The MYAP has amply demonstrated the importance of literacy training in successful Associations, Unions, and Cooperatives. This initiative needs to continue, as does small business training. Both lead to more productive and profitable production and marketing enterprises. 5.2 Commercialization The MYAP’s success is directly attributable to the market-led approach that viewed agriculture as a private-sector for-profit enterprise. Moreover, the rationale for Associations, Unions, and Cooperatives as means for furthering economic gains, and not as instruments for providing social services. Reducing transaction costs for buyers through aggregating production and quality standards were the necessary ingredients for access to wider and more profitable markets. The continuing need is to emphasize post-harvest technologies, particularly storage and quality preservation. Producers can only take advantage of price increases that occur somewhat distant from the harvest by maintaining the quality of the produce. 5.3 Health & Nutrition Overall, the MYAP project was successful in improving the nutrition and health of its beneficiaries. There is good community ownership and support for sustainability of community based health activities. Nevertheless, the extent these will be achieved will be influenced by the MOH and the logistic challenge of providing reasonable supervision and continuing education of community health workers. The large number of these workers is essential to providing satisfactory coverage down to the household level. Training suitable basic community level trainers who would work with their local CLCs and health center personnel could mitigate this problem. The project demonstrated that Community Leadership Councils, bringing together community level leaders and representatives of all groups of community health workers and health organizations within the community, were an important part of the project’s success. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 44 The behavioral change communication coverage was good. However, the quality of what is communicated is important for success across many behaviors. Strengthening the quality of this communication is possible using current state of the art basic methodologies that identify priority subgroups for change and their determinants of change. Literacy and numeracy training were clearly important in enabling the community level organizations and community health workers to perform well and should be a part of future MYAPs. 5.4 Cross-Cutting Issues Gender. Women are more empowered now than at the beginning of the project, due both to literacy and participation in village associations. Being able to use their own funds has improved their ability to upgrade household nutrition. Children are livelier from the enriched porridge, and women know better how to breastfeed younger children more appropriately (timing and placement). Women make more decisions, and more informed decisions, unilaterally or with their husbands -- funds from the village associations have enabled this. The communities may not be able to sustain these changes without the financial inputs from the village associations, although the women planned to continue some of the health/nutrition and agricultural practices, even if the association did not continue. Resilience. Inputs from the project’s activities allowed the women to purchase additional household assets and improve household nutrition, two key elements in improved resilience. However, because the project did not include disaster preparedness/planning, a more consolidated plan for community resilience never emerged to help tie these different elements together in a more comprehensive fashion. Given the likelihood of both short and longer-term environmental disasters in this province, as well as the increasing likelihood of recurrence of insurgency in some of the target districts, having plans of action would be useful to develop in whatever remaining months the project may be able to fund. 5.5 Monitoring & Evaluation  ADRA should consider expanding the M&E resources (including staff) so that independent field visits in a randomized, but routine, manner are possible. This would help provide a quality control for data and minimize mistakes through refresher training for community-based staff. It is simply not possible for two staff to cover five districts.  Standardizing databases from earlier projects as part of a country strategy so that longitudinal studies or other types of analysis might have helped inform both trend data and provide more robustness to explanations on over and under-performance on key variables. ADRA might want to consider this, even retrospectively, for the three Title II projects to provide additional insight for future USAID work in Zambézia. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique 45  Data from the communities needs to find a way back to the communities, so they understand how their opinions, work, and community outreach affect project performance results. This can help communities feel ownership. This permits assessing the community’s own performance against criteria developed jointly, and this helps determine subsequent year work plans for both the community and ADRA. the variety of local languages make conducting surveys extremely time-consuming, as data collection tends to operate via at least one filter. Identifying, training, and periodically using community members with a secondary school education (even if no longer resident in the target communities) may help with survey data collection. ANNEXES Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique II ANNEX I: EVALUATION SCOPE OF WORK I. INTRODUCTION This is a scope of work for the final evaluation of the ADRA Multi Year Assistance Program (MYAP) in Mozambique. In 2008, the Adventist Development and Relief Agency (ADRA) International and United States Agency for International Development (USAID) signed a cooperative agreement to fund a Multi-Year Assistance Program (MYAP) in Zambézia province, the districts of Mocuba, Ile, Maganja da Costa, Lugela and Pebane. The MYAP’s name is Osanzaya Zambézia, a local phrase, meaning “Make Zambézia Happy”. ADRA and its partners, ADPP and Samaritan’s Purse, have been implementing the Osanzaya project. ADRA International is organizing the final evaluation of the MYAP, and it will be conducted during October-December 2013. II. PURPOSE The final qualitative evaluation will cover the project implementation period from August 2008 to October 2013. It will allow ADRA to assess the outcomes of the MYAP and identify successful food security strategies upon which ADRA can build future interventions. The evaluation will document lessons learned during the life of the project and make specific recommendations so that corrective actions can be taken to enhance future ADRA food security programming. The primary issues to be addressed by the final evaluation are the project’s strengths, areas for improvement, lessons learned, and success stories and providing recommendations for similar projects in the future. Another key dynamic, which the evaluation team will look into, is the degree to which impact and sustainability were achieved. Any recommendations for changes or additions to sustainability strategies will be included in the evaluation report. The final evaluation will also assess the target population’s capacity and prospects in terms of continuing sustainable and effective food security activities on its own after the MYAP ends. ADRA intends to use the evaluation not only as a means to verify, whether the original goals and objectives of the project have been achieved, but also as a learning tool that will allow ADRA to enhance the impact of future food security projects. Ultimately, the final evaluation is expected to show what lessons, if any, positive and/or negative, intended and/or unintended, are learned. It is expected that such lessons will benefit ADRA’s own current and future food security projects around the world. Furthermore, if and when applicable, the knowledge gained will be disseminated to all other organizations that deal with food security projects. The evaluation intends to answer the following questions? 7. To what degree did the project activities meet the needs of the project beneficiaries and is aligned with the country’s agriculture and/or development strategy and USAID/Mozambique development goals, objectives, and strategies? This includes the extent to which the project was designed taking into account the economic, cultural and political context and existing relevant project activities. 8. Has the project achieved its objectives? To what extent did the interventions contribute to the expected results or objectives? 9. To what extent did the project resources (inputs) have led to results? Could the same results have been achieved with fewer resources or whether alternative approaches could have been adopted to achieve the same results? Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique III 10. What are the medium and long-term effects, both intended and unintended, of a project intervention. Were effects due to the project intervention and no other factors? 11. What is the likelihood that the benefits of the project will endure over time after the completion of the project? Has the project planned for the continuation of project activities, developed local ownership for the project, and developed sustainable partnerships? 12. What are the lessons learnt, success stories, areas of improvements and recommendations for similar programs? IV. APPROACH and METHODOLOGY It is helpful to remember that the process of evaluation is never far from its social setting. In view of this, the evaluating team may realize that no matter how objectively the data was gathered and analyzed, in the end, the final interpretation cannot totally be free from the social and political climate of the time and the personal biases of the evaluators. Therefore, the evaluating team is expected to be unduly astute with its written presentation as this involves the lives of many whose welfare could be affected either positively or negatively. The team may keep in mind that we are social beings and as such, every assessment should take place in a given cultural context. Consequently, there are ideas that do not make sense outside their social milieu. This evaluation takes place in the context of two cultures, that of the funder's culture and that of the beneficiary's culture. The evaluating team should keep in mind that it is undertaking a major responsibility in its attempt to make a cross-cultural analysis and interpretations. The evaluation process will focus on the guidelines designed by USAID for the Final Evaluations of Food for Peace Food Security Projects. The team leader will prepare lists of questions to be answered through interviews. A beneficial evaluation is a result of reliable data collection and analysis. Data collection methods may include: general observation, in-depth interviews, key informants and Focus Group Discussions with beneficiaries, staff, community, Ministry of Agriculture (MOA), Ministry of Health (MOH) and other stakeholders. Appointments for interviews, field trips as well as logistics support will be arranged by Osanzaya staff who will also facilitate the work of the evaluation team. The team will be supplied with relevant project documents, including baseline and end line reports, annual reports and other project documents required for the study. At the end of the field work, the team will provide a presentation of preliminary findings to the relevant technical team of the project in Mozambique in a workshop to be held at a site deemed most appropriate. Feedback from this workshop will be used by the evaluation team in preparing their draft final report. In the preparation of the final report, the evaluating team is requested to provide the reader with, as much as possible, accurate sources of its information and conclusions. All evaluation statements must be backed by existing data. When this is not the case, the team is required to state this fact and provide a rationale for its observations and conclusions. V. SCOPE The final evaluation is expected to use primary and secondary sources from the monitoring and reporting system. While it will be led by external team of evaluators. ADRA staff will actively participate in the review and planning of activities. The scope of the final evaluation will be limited to the following specific objectives: Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique IV • Assess the effectiveness the MYAP in terms achievement of the goal, SOs and IRs as per the approved performance indicators on the Indicator Performance Tracking Table (IPTT). • Compare planned targets with actual achievements and identify the reasons for any over or under achievement of those targets. • Determine the extent to which program achievements can be attributed to ADRA and/or its partners. • Assess the intended and unintended benefits, program effects on non-clients and prospects for the sustainability of program interventions and impacts. • Evaluate the contributions of the program to the achievement of the goals and objectives of the donor (USAID) in Mozambique. • Identify the critical problems encountered and analyze their underlying reasons. Assess how ADRA and/or its partners exerted efforts to deal with those problems. • Identify lessons learned, good and innovative practices and recommend strategies and activities to be adopted in similar programs in the future. VI. CITIZENS PRIVACY A General Use of Data ADRA considers it unethical for any member of the evaluation team to use information gathered from unsuspecting citizens during the evaluation assignment for anything other than the evaluation under study. Should a viable reason present itself for using the information obtained for other purposes, then, ADRA must be consulted and prior permission secured. This must be adhered to, especially when the material is of a controversial nature and exclusively involves the private lives of the target population. B. Distribution of Evaluation Report The ultimate responsibility for gathering and disseminating information from all of USAID-funded ADRA projects around the world lies within ADRA International. Therefore, ADRA expects the evaluation team, particularly hired consultants, to turnover to ADRA all the data and other information that were used as the basis of the team's final inferences. It is ADRA's position that no evaluation is final until it is: 1) presented to ADRA, 2) both the consultants and ADRA have discussed the contents in an open manner and 3) clear understandings of all conclusions and any differing views are reached between the consultant and ADRA as reflected in the final document. ADRA does not edit or change in any form or fashion the final report of the evaluation team without the Team’s consent. In the event the evaluation team and ADRA remain to have a difference of opinion regarding the final report of the evaluation, ADRA distributes the document intact but will attach a letter to the report stating its own position. VII. COMPOSITION OF EVALUATION TEAM The evaluation team will consist of three external consultants Darell McIntyre (team leader- expert in value chain/agricultural marketing, Cesar Tique agricultural expert, Paul Freeman Health/Nutrition expert, Alice Willard gender & resilience expert, and Dawit Habtemariam Acting Director, Evaluation Office-ADRA/I). The following may participate during the field work including ADRA Mozambique Program Director, MYAP Director, M&E Coordinator, technical Coordinators, a community representative, MOA and MOH representative and, a USAID Mission representative, as feasible. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique V The team leader will be responsible for writing the evaluation report and comply with reporting requirements under Section IX. The Acting Director of Evaluation will ensure that the evaluation is planned, organized and managed smoothly. VIII. BUDGET The budget for the Final Evaluation of project will be attached IX. TENTATIVE CALENDAR Arrival in Mozambique (US based evaluators) December 1, 2013 Documentation review December 1-3, 2013 Orientation of Team, meeting partners December 3, 2013 Field Work – Data Collection December 4-13, 2013 Preliminary briefing in Macuba and team departure December 15, 2013 Writing of the draft Report December 16-30, 2013 Review of the draft document by ADRA/Headquarters December 31, 2013 Revision/Final Evaluation Report and submission to HQ January 10, 2014 X. REPORTING REQUIREMENTS The evaluation team leader will prepare the following and submit them to the Evaluation Office: Work Plan: The team leader will prepare a detailed work plan which will include the methodology to be used, and submit it to ADRA Evaluation Office for review and feedback in accordance with the above schedule. Workshop: The evaluation team will present an oral summary of preliminary findings in a workshop format for key Project office staff and stakeholders. The clarification resulting from this discussion will be incorporated in the evaluation report. ADRI/International is solely responsible for dissemination of the final evaluation report. First Draft Report: The Lead Evaluator will submit to ADRA/International Evaluation Office and project office a copy in English of the first draft report, which will include key findings, recommendations and feedback from the workshop. ADRA will send back comments to the Lead Evaluator one week after receiving the report. The Lead Evaluator will address the comments in the final draft report. Final Report: The Lead Evaluator will send an electronic copy in English of the final written evaluation report to the Evaluation Office at ADRA/International no later than one week after he receives comments from ADRA on the first draft report. The final report will include modifications and justifications for variations from the original design (should there be any), goals and activities agreed upon with ADRA and project team. The report will include the following sections: Title Page. The title page will state the name and title of grantee, project name location (country and district(s)), cooperative agreement number, project beginning and ending dates, names and positions of evaluators involved in writing and editing the Final Evaluation Report, date of submission, and date and name of the document. List of Acronyms. Unusual or obscure acronyms should be identified at the beginning of the report. Executive Summary. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique VI The executive summary synthesis should be no more than two pages in length and will include: background of project, evaluation methodology, accomplishments and results of the project, concerns and recommendations. Table of Contents. The table of contents should outline each major topic section, appendices, figures, maps, tables, etc. Body of the evaluation. The body of the evaluation report will include the following in sequential order: Introduction and Background. The introduction and background will include at a minimum: justification for awarding grant, goals and objectives of the grant, implementation methods, and the purpose of the evaluation. Evaluation Methodology. The evaluation methodology will include at a minimum: description of data collection and evaluation sites selection processes. Discussion and Analysis. This is where the findings are clearly stated and discussed in detail. All the recommendations and the summary of the evaluation are based on this section of the document. Supplementary Issues and Questions. This section will address in sequence the supplementary issues and questions outlined in this Scope of Work. Conclusions and Recommendations. This section presents the main conclusions based on this final evaluation. It should outline the impact of the project. Results Highlight If appropriate, provide a one-page description of some element of the program, with supporting data, that would make a good stand-alone communication piece for ADRA or USAID to distribute or to post on the Office WebPage. Appendices. The appendices included will be at the discretion of the evaluation team. However, the appendices must include the scope of work, itinerary for the evaluation visit, list of individuals interviewed during the evaluation, interviewers’ questionnaires, references cited and maps. Additional appendices such as case studies, etc.. may be included as determined appropriate by the evaluation team. Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique VII Annex II: EVALUATION SCHEDULE OF VISITS 12/3/2013 09:00-10;00 Meeting with ADRA Country Director 10:00-1100 Meeting with USAID/Mozambique 13:00-15:00 Travel Maputo to Quelimane 15:30-17:00 Travel Quelimane to Mocuba 12/4/2013 08:30-13:00 Meetings with Project Director, Coordinators, M&E 13:00-14:00 Lunch 14:00-14:30 Meeting with Director, District Economic Activities 14:30-15:00 Meeting with Director, Women’s Health & Social Action 15:00-17:30 Meetings with Project Director, Coordinators, M&E 12/5/2013 08:00-09:00 Travel Mocuba to Alto Benfica 09:00-11:00 Meeting with Alto Benfica community 11:00-12:00 Return to Mocuba 12:00-13:30 Lunch 13:30-14:00 Travel Mocuba to Nadala 14:00-16:00 Meeting with Nadala community 16:00 Return to Mocuba 12/6/2013 07:00-08:00 Travel Mocuba to Lugela Sede 08:00-9:00 Meeting with SDAE of Lugela 09:00-10:00 Meeting with SDSMAS of Lugela 10:00-11:00 Travel Lugela Sede to Putine 11:00-12:30 Meeting with Putine community 12:30-14:30 Travel Putine to Mangala w/lunch on road 14:30-16:00 Meeting with Mangala community 16:00-17:00 Return to Mocuba 12/8/2013 07:00-08:30 Travel Mocuba to Ile Sede 08:30-09:30 Meeting with SDAE of Ile 09:30-10:30 Meeting with SDSMAS of Ile 10:30-12:00 Travel Ile Sede to Liguangua 12:00-13:30 Meeting with Liguangua community 13:30-14:30 Travel Liguangua to Nampevo w/lunch on road 14:30-16:00 Meeting with Nampevo community 16:00-17:00 Return to Mocuba 12/9/2013 07:00-08:00 Travel Mocuba to Ganga 08:00-09:30 Meeting with Ganga community 09:00-11:00 Travel Ganga to Dedere 11:00-12:00 Meeting with Dedere community 12:00-13:00 Travel Dedere to Maganja Sede w/lunch on road 13:00-14:00 Meeting with SDAE of Maganja da Costa 14:00-15:00 Meeting with SDSMAS of Maganja da Costa 15:00 Travel Maganja da Costa to Pebane 12/10/2013 08:00-08:30 Meeting with SDAE of Pebane 08:30-09:00 Meeting with SDSMAS of Pebane 09:00-10:30 Travel Pebane Sede to Mujaiane (Malema) 10:30-11:30 Meeting with Mujaiane community 11:30-13:00 Travel Mujaiane to Mitale (Impaca) w/lunch on road 13:00-14:00 Meeting with Mitale community 14:00 Return to Mocuba Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique VIII Annex III: LIST OF PERSONS INTERVIEWED ADRA: Lynn Boyd ADRA Country Director Armindo Salto Country Program Officer Farai Muchiguel MYAP Director fmuchiguel@adramozambique.org Florêncio Máquina Agriculture Coordinator fmaquina@adramozambique.org Anselmo Lisboa Health and Nutrition Coordinator alisboa@gmail.com Osvaldo Chura Commercialization Coordinator osvaldocchp@gmail.com Enoque Muabsa M&E Coordinator emuabsa@adramozambique.org João Saraiva M&E Assistant jsaraiva@adramozambique.org Miriam Chilundo M&E Officer for PRODEZA mchilundo@adramozambique.org Anonymous ADRA Mocuba Director Jose Popisco ADRA Pebane District Coordinator Luis Gigueira Driver ADPP: Anne Fisker, ADPP Literacy Director OTHER NGOs Garret Bernat, Water Supply Bore Drilling Pump Repair Specialist. Government of Mozambique: Ilidio Alfonso Jose Bande, Provincial Director, Provincial Directorate of Agriculture Abel Jaime Ernesto, District Director, Mocube District Economic Activities Service Anonymous, SDAE Lugela Extension Worker Anonymous, SDAE Lugela Extension Worker Natalino Fernando Moise, SDAE Maganja da Costa Director Zeferino Freitas, SDAE Pebane Extension Worker Judith Caetano, Public Health Director & Provincial Deputy of Health Zambesia Virgilio Sopinho, Pebane District Health Officer DSS Fariucha Antonio Leguachane health center Malema BENEFICIARIES: Antonio Fernando Lampiao, Cooperative President Time did not permit gathering the names of the many other beneficiaries, both direct and indirect, who provided input through group sessions. However, their contributions made this report possible. IMCI community volunteers interviewed in their village Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique IX Annex IV: QUESTIONAIRES Agriculture Producer Focus Group Interviews Project Design/Objectives: -Is the project responding to your needs? Why? Or why not? -What are major problems occurring in the agriculture production and productivity in your area? -Are the selected crops relevant to your food security, income and livelihoods? -Indicate major changes occurred in the area as result of the introduction of the project? 1.2 Adoption of Technologies: -What type of crops you are growing? Why? -When do you started to grow them? -What improvements occur in your farm since 2008 as a result of these technologies? -What is the most effective technology? Rank them and state why. -Which technology is the most labor-intensive? -How do these technologies change how much/the quality of what you produce? -What technologies do you think you will continue to use these technologies? -What would you do differently? Access to Inputs/Storage: -How do you get access to inputs? Who provides them? -What do you think are the best ways to increase access to inputs? -Do you know know where to get these inputs once the project ends? -Do you use an improved storage place? -How do you maintain it? -What changes as resulted in your life due to establishing a storage place? -What would be their main constraint? Project Extension Services: -Have you received any assistance from the extension services? -What did you learned in the CDR? Did you transfer the same to your farm? -What changes have you noticed in in your farm and livelihood as a result of the messages from the extension services? What were the three main lessons that you got with this project? Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique X Agricultural Marketing Focus Group Interviews with Farmer Associations -What is the point of sale for your produce? -How do you prefer to work? Individually or within the Association? Why? -How do you receive information on prices for products? From which markets? -What are the benefits of being a member of the Association? Economic? Social? -To what extent Farmer Business Organizations (FBO) increase farmers’ access to markets? What are the main reasons? -Did farmers really obtain more bargaining power in the market place by participating in FBOs? -What is the role that agricultural collecting centers (ACC) played for FBOs? -What are the difficulties with being a member of the Association? -Do you think that the Association promotes and maintains cohesiveness from the start to today? How does it resolve different points of view? -What is the participation of women in the Association? What can be done to improve women’s participation? -To what extent were the different levels of organization (Associations, Unions and cooperative of Unions) the project established appropriate to address the issues identified in problem analyses? Any redundancy? -What are the main factors of success for FBOs that can be considered successful? Could the same results have been obtained with by other means? -What is needed to keep your Association running in the future without ADRA’s help? Agricultural Marketing Key Person Interviews with Union delegates -What are the benefits of being a member of the Union? -What are the difficulties with being a member of the Union? -Do you think that the Union promotes and maintains cohesiveness from the start to today? How does it resolve different points of view? -What is the participation of women in the Union? What can be done to improve women participation? -What type of sales do use and has it changed since the project began? Clients￾What is needed to keep your Union running in the future without ADRA’s help? Agricultural Marketing Key Person Interviews with Government Officials -What are the Government of Mozambique policies that promote sales of the agricultural commodities produced by the MYAP beneficiaries? -What are the main constraints to agriculture in your District? In your Province? Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique XI Health and Nutrition Focus group questions for members of local Community Leadership Councils -What activities do you know that the Ozanzaya Zambesia project has been conducting in your area? -What improvements in the health of the community have you seen as a result of the project? -What do you do differently as a result of what you learned in the project that you did not do before it? -Which activities does the CLC do now? -Will you continue these activities after the project? -What does the CLC do with the local health center? -What training activities has your CLC received from the project? -What does your local Hygiene Promoter do to improve the health of the community? -What does your local Community Volunteer do to improve the health of the community? -What do women’s and fathers’ groups do? -What changes in feeding of your child have you adopted as a result of project activities? -What do you include in your enriched porridge? Tell us about where you get these food groups from? Source? Cost Availability throughout the year? -What foods are grown in your area as a result of the project? -Do you give these foods to your children? -How often do you give your children food from different food groups each week? -In your usual activities, when do you wash your hands? -Do you use soap when you wash your hands? -Do you have soap in your house now? Tell us about when you have it? -Do you have a latrine near your house? Do you and your family use it? -How far away from your house is your water supply? Does it work, explain? -Do you have a water supply committee in your area? Do people contribute to it? -Name as many things that you know that work to prevent malaria? -What are all the ways someone can get AIDS? -Can you get AIDS from mosquitoes? Can you get AIDS from sharing food? -Name as many things that you know that work to prevent spread of AIDS? -Do you have any questions to ask us? Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique XII Health & Nutrition Questions for Village Mothers with a Child Less Than Five Years of Age (Half of mothers should have child more than 6 months old) -Have you heard about the Ozanzaya Zambesia project operating in your village over the past 5 years? -What activities do you know that this project has been conducting in your area? -What do you do differently as a result of what you learned in the project that you did not do before it? -Has the health of your children changed since this project if so how? -How old was your child when you gave him/her any food or drink other than breastfeeding? -What food groups do you know? -What changes in feeding of your child have you adopted as a result of project activities? -What do you include in your enriched porridge? Tell us about where you get these food groups from? Source? Cost Availability throughout the year? -What foods are grown in your area as a result of the project? -Do you give these foods to your children? -How often do you give your children food from different food groups each week? -In your usual activities, when do you wash your hands? Do you use soap when you wash your hands? Do you have soap in your house now? -Do you have a latrine near your house? Do you have soap and water near your latrine? -How far away from your house is your water supply? -Name as many things that you know that work to prevent malaria? -Name as many things that you know that work to prevent spread of AIDS? Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique XIII GENDER QUESTIONS (for FGDs and KIIs) -Who in your family/household makes decisions about: What crops to plant? Who will buy the agricultural produce (what you harvest)? When to seek medical attention (especially for young children or women)? Whether a child continues going to school (or returns to school)? What you will buy for the household? Who attends social events? -Have these decision-making processes changed (from when the project started)? What changed? -What particular decisions changed, and how; do you think you will continue to operate this way? Why/why not? -What changes have you seen in your children’s health status (since the project started)? -Does it take you more/less time to get enough water for your household’s needs? How/when has this changed? What caused the change? -Do you belong to a village Association assisted by ADRA? If so: What have you used the funds for? What differences has this made for your family? -Have you seen changes in how the men in your household behave (positive or negative) because you now have a new source of income? -What activities do you plan to continue now that ADRA is ending? -Is there someone you know whose life has really changed (because of the project interventions) that we should talk with in more detail? RESILIENCE QUESTIONS -How does your family /household prepare for natural disasters (such as cyclones, etc.)? How has this changed in the last few years? -How quickly does your family/household recover from natural disasters? -What is the sequence towards recovery (i.e., what do you need first, how do you get it)? -What types of assistance have you received in the past? -What was most useful? Why? -What was least useful? Why not? -When there is a drought/flooding during the growing season, what does your family/household do to cope with the loss of crops (slower onset disaster)? -What effect does having a village Association or community projects make to how quickly you can recover from disasters or drought/flooding? -What do you think you still need to learn or do in order to recover more quickly from disasters/crop failures? Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique XIV Annex V: MAP OF PROJECT DISTRICTS IN ZAMBÉZIA PROVINCE Final Evaluation Report MYAP FFP-A-00-00084-00 - Mozambique XV ANNEX VI: REFERENCES ADRA International (2009): Fiscal Year 2009 Annual Result Report ADRA International (2010): Fiscal Year 2010 Annual Result Report ADRA International (2011): Fiscal Year 2011 Annual Result Report ADRA International (2012): Fiscal Year 2012 Annual Result Report ADRA International (2013): Fiscal Year 2013 Annual Result Report ADRA International (2013): Multi-Year Assistance Program (MYAP): Endline Survey Draft Dawit Habtemariam Acting Director, ADRA International Evaluation Office FAO (2004): Agriculture Conservation in Mozambique: the Case of Northern Mozambique. Hanlon, J. and T. Smart (2008) Do bicycles equal development in Mozambique? Woodbridge, UK: James Currey. Swanson, R. (15/04/2013): What Worked Best? Review of Title II Behavior Change Programs in Mozambique 2008-2013. MYAP Organization Reviewed. MYAP – Multi Year Assistance Program Osanzaya – Zambézia Quarterly Reports January– March FY 2009 to April – June 2013 ADRA Mozambique 2013 Osanzaya – Zambézia (Make Zambesia Happy) ADRAs Income Generation Program (IGP) Multi-Year Assistance Program (MYAP) Proposal Application P.L. 480 Title 11 Mozambique/ ADRA International 2009 Republic de Mocambique Ministerio da Saude Atencao Integrada As Doencas Da Infancia Para Agented Communitarios De Suade Guia Do Participante OMS USAID UNICEF Pacote Nutricional Basico Misau- Departmento de Nutricao 2007 (Revised 1-19-2012) AED USAID UNICEF