CRS Fararano Joint Midterm Review July 2017 2 Acknowledgements The members of the review team wish to thank the staff of Catholic Relief Services and partners Organe de Développement du Diocèse de Toamasina, Bureau du Développement de l’Ecar de Mananjary, Caritas Morombe, and Conseil Diocésain de Développement, as well as USAID Madagascar Mission staff and management, and Fararano Chief of Party James Hazen for all the support received during the review. We extend our gratitude to Bridget Ralyea, Agreement Officer’s Representative of the Fararano project, who played an instrumental role in facilitating this review and making it possible. Thanks to Adam Reinhart for reviewing and editing the sections related to agriculture and livelihoods. Special thanks go to the many Fararano field staff that did their utmost to facilitate the team’s schedule and other needs. This was invaluable in allowing us to carry out the review in as complete a manner as possible. Hopefully we've listened well, our observations are grounded in reality, and our suggestions are realistic and practical. We appreciate that many people have labored long and hard on the Fararano project to improve project performance. We thank Catholic Relief Services Country Office staff, Catholic Relief Services and ADRA management in headquarters, and all of the people who gave freely of their time to discuss project processes, challenges, ideas and suggestions, and accompany the team in the field. Thanks to Monica Mueller (TANGO International) for assembling and editing team contributions and finalizing the evaluation reports. A special thanks to Viviane Voahanginirina, who organized the logistics for USAID staff. To the more than 20 communities in 10 communes that took time from their long, busy days to participate in the midterm review, it is our hope that the recommendations provided by the review team are actively used to develop and improve the program and policies that enhance their food security and livelihoods. CRS Fararano Joint Midterm Review Team Core Team Members: Arif Rashid USAID Food for Peace, Senior M&E Advisor, Review Team Leader Chris Seremet Catholic Relief Services, Technical Advisor, WASH Eddy Rasoanaivo USAID Madagascar Mission, Food Security Specialist Jefferson Shriver Catholic Relief Services, Senior Technical Advisor, Value Chain and Markets Justin Mupeyiwa USAID Zimbabwe Mission, M&E Specialist Melanie Thurber USAID Food for Peace, Nutrition Advisor Natsayi Nembaware ADRA, Senior Technical Advisor for Nutrition Nicole Van Abel USAID Food for Peace, WASH Advisor Case Studies: Holly Tripp USAID Food for Peace, Program Assistant 3 Table of Contents Acronyms............................................................................................................................5 Executive Summary ..........................................................................................................7 Introduction to the Fararano Project ...........................................................................11 Project Overview...................................................................................................................................... 11 Project Purposes and Sub-purposes..................................................................................................... 12 Midterm Review Objectives ...........................................................................................13 Methodology.....................................................................................................................14 Limitations and Challenges...................................................................................................................... 16 Findings .............................................................................................................................17 Program Quality ........................................................................................................................................ 17 Program Quality: Recommendations.................................................................................................... 20 Purpose 1: Undernutrition is Prevented among Children under 2 ............................................... 21 Purpose 1: Recommendations................................................................................................................ 40 Purpose 2: Increased Household Incomes (Monetary and Non-monetary)................................ 45 Purpose 2: Recommendations................................................................................................................ 60 Purpose 3: Community capacity to manage shocks is improved ................................................... 63 Purpose 3: Recommendations................................................................................................................ 67 Program Monitoring ................................................................................................................................. 68 Program Monitoring: Recommendations............................................................................................. 71 Conclusion ........................................................................................................................72 Annex 1: JMTR Review Protocol................................................................................lxxiii Annex 2: Additional Case Studies................................................................................ cxli 4 Table of Tables Table 1: Area, yield and production, selected crops, 2016.................................................................. 51 Table 2: Sales and income data, selected crops, 2016........................................................................... 52 Table of Figures Figure 1: JMTR sample sites...............................................................Error! Bookmark not defined. Figure 2: Venn diagram showing integration of interventions at household level ..................Error! Bookmark not defined. Table of Case Studies Case study 1: MCHN.................................................................................................................................... 24 Case study 2: Community health and sanitation .................................................................................... 39 Case study 3: Lead farmer........................................................................................................................... 47 Case study 4: FFA reforestation................................................................................................................. 55 Case study 5: The landless..........................................................................................................................cxli Case study 6: Female head of household ...............................................................................................cxlii 5 Acronyms Ar Ariary (Malagasy currency) ARR Annual Results Report BDEM Bureau du Développement de l’Ecar de Mananjary BNGRC Bureau National de la Gestion de Risques et Catastrophes CCFLS Community Complementary Feeding and Learning Sessions CDD Conseil Diocésain de Développement CHV Community Health Volunteer CLGRC District Level Comité Local CLTS Community-Led Total Sanitation CLTN Community-Led Total Nutrition CMAM Community-based Management of Acute Malnutrition CNV Community Nutrition Volunteer CPO Collection Point Organization CRS Catholic Relief Services CSA Centre de Service Agricoles CSB+ Corn Soy Blend+ CU2 Children under two CU5 Children under five DFAP Development Food Assistance Program DiNER Diversification for Nutrition and Enhanced Resilience DRM Disaster risk management DRMC Disaster Risk Management Committee DRR Disaster risk reduction FFA Food for Assets FFP Food for Peace FMNR Farmer-Managed Natural Regeneration FTP File transfer protocol GMP Growth monitoring and promotion ICRAF World Agroforestry Center IMCI Integrated Management of Childhood Illness IPTT Indicator Performance Tracking Table JMTR Joint Midterm Review KFF Fokontany Development Committee M&E Monitoring and evaluation MCHN Maternal and Child Health and Nutrition MEAL Monitoring, Evaluation and Learning MoH Ministry of Health MVhousehold Most vulnerable household NCBA-CLUSA National Cooperative Business Association – Cooperative League of USA NRM Natural resources management ODDIT Organe de Développement du Diocèse de Toamasina ODF Open-defecation free ONN National Office of Nutrition 6 PIRS Performance Indicator Reference Sheet PiSP Private input service provider PLW Pregnant and lactating women PO Producer Organization PPP Public-Private Partnership PSP Private service provider RUA Road Users Association SAM Severe acute malnutrition SBCC Social and behavior change communication SECALINE Surveillance and Education for Schools and Communities on Food and General Nutrition SILC Savings and Internal Lending Community SMART Skills for Marketing and Rural Transformation SMILER Simple Measurement of Indicators for Learning and Evidence-based Reporting ToC Theory of change UPNNC Unité de Programme National de Nutrition Communautaire/ National Community￾based Nutrition Program Unit WASH Water, sanitation, and hygiene 7 Executive Summary This is the report of the midterm review of Fararano, a five-year, USAID Food for Peace Development Food Assistance Program whose goal is to reduce food insecurity in 48 communes in three of the most vulnerable regions in Madagascar: Atsinanana, Vatovavy Fitovinany and Atsimo Andrefana. Fararano provides an integrated package of nutrition, agriculture, disaster risk reduction, environment and gender-focused activities to reduce food insecurity in these target areas. It is implemented by Catholic Relief Services and four local implementing partners – Organe de Développement du Diocèse de Toamasina, Bureau du Développement de l’Ecar de Mananjary, Conseil Diocésain de Développement and Caritas Morombe – as well as several technical partners. This was a joint review, in that it covered Fararano and another USAID Food for Peace award, ASOTRY. Both reviews were conducted by a team of development professionals representing USAID Food for Peace, the USAID Mission in Madagascar, Catholic Relief Services and the Adventist Development and Relief Agency (ADRA). The lines of inquiry were guided by an evaluation protocol, including detailed objectives for the review. The main focus was on perceived program effectiveness, constraints inhibiting effectiveness and means of overcoming these constraints. The review followed primarily qualitative methodology and made reference to select quantitative data from program monitoring. Review preparations began in January 2017; the in-country mission, including a validation workshop and a USAID Food for Peace debriefing, which took place in April and May of 2017. The team returned to Madagascar in June for a workshop on recommendations. The report was finalized in November with the support of an outside consultant contracted by Technical, Operational and Performance Support (TOPS). Following are the main findings regarding overall quality, each of the three program purposes and program monitoring. The full report contains extensive recommendations that resulted from both the review exercise and the recommendations workshop. Program Quality While staff recruitment, staff training, formative research and barrier analysis are on schedule and on target, the review team noted issues with integration of interventions, geographic targeting, quality of implementation, theory of change and sustainability. Many households do not receive critical support from Fararano to increase agricultural production or off-farm income; increase access to health, nutrition, water and sanitation services; or improve behaviors around maternal and child health nutrition, water, sanitation and hygiene. Approximately 60 fokontany in Ifandiana, Mananajary, and Brickaville are extremely remote, and it is a challenge for Fararano staff to ensure the quality of interventions in these communities. Farm sizes are small, and yield and profitability are low; therefore, it is unlikely that Fararano will achieve Purpose 2 (agriculture) objectives without revising its strategy and intervention package to increase food access. While Fararano has been piloting various market-based approaches to improve sustainability of 8 various agriculture-related service provisioning, the activity has yet to implement strategies to improve sustainability of outcomes for maternal and child health and nutrition (MCHN) and disaster risk management practices. Private service providers, private input service providers, commodity aggregation and collective marketing have the potential to improve sustainability of production and crop-based income; however, these initiatives are at an early stage and the review team could not assess their performance. Maternal and Child Health and Nutrition (MCHN) (Purpose 1) Overall, Fararano communities and participants and regional/district authorities appreciated Fararano inputs and were seeing positive change in their communities. Lead mothers and community health volunteers were highly engaged and wanted to make a difference in their communities. Across implementing partners and geographic areas, participants spoke highly of community-led total nutrition, care groups and cooking demonstrations. Key Purpose 1 successes include the number of community volunteers trained in health and nutrition, an increased number of women visiting health facilities for antenatal care and child health visits and coordination with USAID/Mikolo and other Ministry of Health activities. Key Purpose 1 challenges include whether messages were communicated effectively enough between nutrition promoters and between lead mothers and neighborhood women, context￾specific targeting and related interventions and consistency of approaches across implementing partners and geographic zones. While coordination with UPNNC1 was evident, reinforcing this nutrition platform further is a potential opportunity for continued positive impacts of Fararano after the project ends. Overall, Fararano water, sanitation and hygiene (WASH) implementation is behind schedule and must be prioritized going forward. The number of WASH staff was minimal and should be increased so they can provide adequate oversight and guidance over WASH implementation in the last two-and-a-half years of the project. Moreover, much of the WASH implementation was focused at the household level, but the entire community must be sensitized to WASH messaging for results to be observed. Therefore, community-level WASH implementation must occur. Access to safe water is critical to achieving nutritional outcomes; therefore, the review team recommends prioritizing water infrastructure such as installing as many water systems as possible while ensuring appropriate engineering oversight, speed up installation and focus on sustainability (i.e., operations and maintenance). In addition, water quality in the target areas was uncertain, so promoting household water treatment must remain central to the Fararano campaign. Sanitation in the project area is still inadequate; thus, the community-led total sanitation campaign must be strengthened, including providing substantive follow-up. In addition, latrine construction should be implemented, as well as making latrines fly-proof. Finally, the handwashing social and behavioral change campaign must be more robust, systematic and 1 Unité de Programme National de Nutrition Communautaire/ National Community-based Nutrition Program Unit 9 comprehensive. Agriculture (Purpose 2) The Producer Associations (POs) and Collection Point Organizations (CPOs) developed by Fararano have the potential to help farmers get better prices for their produce. Fararano has been promoting a mix of staple and high-value cash crops (e.g., fruits, spices) for income generation, which lends itself more to achieving income gains. These are market-based approaches, and if farmers benefit from the POs and CPOs, they will likely continue even after the project ends; however, they would continue to need capacity-building support. Fararano can link them to capacity-support service providers. The review team found them at an early stage, so could not thoroughly assess their scope, potential and challenges. Similarly, the Private Input Service Provider (PiSP) and Private Service Provider (PSP) models that Fararano developed and implemented have the potential to promote sustainability. The PiSP and PSPs can emerge as private sector service providers; however, they do not seem to be financially viable entities, and Fararano should assess and make adjustments so these are financially viable business models. While Fararano has been using the farmer leader approach as the main vehicle to transfer improved production technologies and management practices, this approach has faced serious challenges. The idea is to have farmer leaders use demonstration plots to promote technologies and management practices and train participating farmers. But the quality of the demonstration plots is often sub-optimal (sometimes even poorer than adjacent plots). Farmer participation is inconsistent, irregular and limited to a few farmers, and training quality is less than ideal for effective learning. As a result, the adoption of project-promoted techniques is minimal, and farmers do not remember what they learned. Fararano staff do not analyze yield and yield potential with the farmers; therefore, participating farmers and project staff have little idea of what outcomes to expect from their involvement with Fararano. In addition, the farmer leader methodology is used only for program-based trainings with no organizational or farmer aggregation component that would give it a chance to continue beyond the life of the project. The Savings and Internal Lending Community (SILC) groups seemed to be very successful, but their reach is very limited. CRS has yet to explore the potential of the SILC group approach beyond savings and lending. The irrigation facilities and roads that Fararano rehabilitated/constructed seem to be need￾based and appropriate to achieve the project objective; however, the standard design and implementation procedures for construction and rehabilitation work were not always followed. Disaster Risk Management (DRM) (Purpose 3) The intent of Purpose 3 is to strengthen communities’ capacities to predict, mitigate and respond to shocks, and to improve natural resources management (NRM) by developing NRM, disaster risk reduction and emergency preparedness plans that focus on improved planning, communications, governance and safety net responses designed by and for communities. Generally, it appears the disaster risk management (DRM) and NRM committees are cohesive, as their willingness to complete detailed DRM/NRM plans is strong, however, they appear to be 10 hesitant to carry out plan activities without guidance from implementing partners. The DRM plans are detailed, but understanding their own content in those plans varies substantially. It does not appear that Fararano engages with committees regularly. Since the preferred means for field agent travel is by bicycle, and each agent is responsible for working with many fokontony, it is difficult to attend committees regularly. Engaging with the committees is important because it builds their ability to understand the activity content of their plan. Regular engagement also builds their confidence to implement their identified activities, particularly if the activities require external technical and financial support. The committees’ ability to conduct disaster simulation exercises is also weak due to poor engagement. Program Monitoring The Fararano monitoring and evaluation (M&E) system was well designed to track indicators with all necessary components, from data capture to results sharing. This allows the project team to review progress of selected indicators. The ZOHO platform greatly increased staff access to monitoring data. Fararano has technical staff among the M&E team capable of developing the database and data collection tools for mobile devices, which gives the project flexibility to adjust the tools and the system as new challenges are encountered or new indicators added. Fararano’s M&E system uses a centralized database where all information from partners is centrally stored. The system is linked to an analytical routine at the front end, which greatly increased access to data as staff can generate bi-variate frequency tables based on their needs and interests. Partner staff receive a copy of their dataset through the SHAREit application for further analysis. Major data quality issues were related to data collection, particularly the validity and reliability of data collected by community volunteers. The volunteers use forms to capture data from the field and sometimes lack standardized forms suited to evolving data needs. The community volunteers were therefore creating their own data collection forms. This has resulted in inconsistent data points across the volunteers, which created serious data reliability issues. The data reported by the field agents cannot be objectively verifiable. There were also validity issues mainly linked to the lack of measurement standards, such as defined training and developing standards. Reliability issues related to the system also include not monitoring the quality of various interventions, including social and behavior change sessions. Fararano has yet to develop a system to check data quality, which became far more complicated and challenging given that data are collected using tablets with no physical paper trail. There is no back-up data remaining on the iPad after the information is uploaded to the database at the end of each month. The M&E system does not capture the effect of some major investments including infrastructure, gender integration and the community mobilization work. Overall Assessment Fararano has been successful in setting up implementation processes, such as for care groups, Community-Led Complementary Feeding and Learning Sessions and cooking demonstrations, Community-Led Total Nutrition, Community-Led Total Sanitation, lead farmers, demonstration 11 plots, producer organizations and Disaster Risk Management Committees. Fararano developed most of the guidance materials, completed staff training and is on track to achieve many of the output targets. Major issues identified by the Joint Midterm Review (JMTR) team include high staff workload, implementing more interventions for which it has capacity, remote location of some fokontany, lack of tailored strategy to account for geographic variability and lack of a viable sustainability strategy. These issues have further been complicated by the absence of a strong feedback loop to identify challenges in implementation systematically and address them promptly and consistently across partners. If CRS and its Fararano partners take serious corrective measures to improve the quality of implementation, Fararano has the potential to achieve its intended goal. Based on the JMTR team recommendations, the program should develop a tool to ensure proper and transparent follow-up on actions made toward these recommendations. Introduction to the Fararano Project Project Overview Catholic Relief Services (CRS) and partners are currently implementing a five-year USAID/Food for Peace Development Food Assistance Program (DFAP) called Fararano (“harvest season”). The program goal is to reduce food insecurity in 48 communes in rural Madagascar. Fararano operates in three of the most vulnerable regions of Madagascar: Atsinanana, Vatovavy Fitovinany and Atsimo Andrefana. Fararano reaches over 70 percent of the population in seven districts, 48 rural communes and 464 fokontany with an integrated package of nutrition, agriculture, disaster risk reduction, environment and gender-focused activities to reduce food insecurity in these target areas. CRS works with four local implementing partners: Organe de Développement du Diocèse de Toamasina (ODDIT), Bureau du Développement de l’Ecar de Mananjary (BDEM), Conseil Diocésain de Développement (CDD) and Caritas Morombe in four different dioceses. In addition, CRS works with several technical partners including National Cooperative Business Association – Cooperative League of USA (NCBA-CLUSA) on value chains, World Agroforestry Center (ICRAF) on agroforestry, No Strings International on communication techniques using puppetry, Centre ValBio on reforestation and environmental education, Bio-D on bio-fuel systems and Harvard University on research on 12 nutrition and climate change. CRS and partners collaborate closely with 10 government ministries at national and decentralized levels to strengthen program quality and sustainability. The CRS Fararano team consists of a Program Management Team comprised of the chief of party; deputy chief of party; team leaders for nutrition, agriculture/livelihoods and community; cross-cutting specialists for gender, Monitoring, Evaluation and Learning (MEAL), social and behavior change communication (SBCC), Go Green and Nutrition Partnership; internal auditor; commodity manager and program and technical coordinators for each partner institution including ODDIT, BDEM, CDD, Caritas and NCBA-CLUSA. The Program Management Team helps to guide the strategy, share lessons learned and best practices and ensure that program quality meets standards set by each technical area. Each partner has technical coordinators and field staff, typically including two field agents (one each for agriculture and community) and one nutrition promoter in each of the 48 communes. 2 A supervisor for each partner covers a district (or five to eight communes) and supports the field staff to implement activities. CRS-based specialists help coordinate among partners, ensure consistency and quality and lead research, studies, exchanges and learning across partners to promote quality, integration and sustainability. A robust monitoring and review team has also been composed. Overall challenges in CRS Fararano areas have mainly been tied to two factors. First, climatic conditions linked to El Niño and late rainfall due to climate change result in later-than-normal harvests, particularly in the southwest. Second, a fragile political system leads to weak and under-resourced government structures, which prevents the program to fully transition its interventions over to the government – part of the sustainability strategy. In addition, with the emergency in the Deep South due to the drought, CRS mobilized staff and resources to respond to the urgent needs of the most vulnerable, thus pulling them away from the project. Project Purposes and Sub-purposes P1: Undernutrition is prevented among children under two 1.1 Women and children have improved consumption of diverse and nutritious foods 1.2 Women and children (especially during the 1,000 days) utilize preventive and curative maternal and child health and nutrition services 1.3 Households practice optimal water management, hygiene, and sanitation behaviors P2: Increased household incomes (monetary and non-monetary) 2.1 Increased diversified agriculture production 2.2 Increased on- and off-farm sales by households and producer organizations P3: Community capacity to manage shocks is improved 3.1 Community-based disaster mitigation systems meet national standards 3.2 Community-based disaster preparedness systems meet national standards 2 To respect the care group model, some communes may have more than one nutrition promoter to maintain the ratio of promoters to care groups, in line with the approach. 13 3.3 Community-based disaster response systems meet national standards 3.4 Community-based social safety net mechanisms strengthened Midterm Review Objectives The objectives of the midterm review for Fararano are:3 1) Assess the overall strategy of Fararano in terms of its relevance for addressing food insecurity with targeted impact groups, taking into account contextual changes that may have occurred since the award began implementation. This will entail reviewing the strategies that ensure that the target groups are reached by the award, reviewing the theory of change and assessing the hypotheses, risks, and assumptions made during the design of the program. 2) Assess the quality of inputs, implementation and outputs to identify factors that enhance or detract from the efficiency, quality, acceptability and effectiveness of the activities implemented, and the likelihood that they will contribute to sustained achievement of project goals. 3) Review the level and effectiveness of coordination and collaboration with external organizations that are critical to achieve goals and purposes. This includes actors that provide complementary services necessary to achieve outcomes, actors that will provide essential services to sustain the outcomes after the end of the two awards, actors that influence people’s access to goods and services and organizations that promote or impede an “enabling environment.” 4) Present, through quantitative data and qualitative information, evidence of changes (intended and unintended outcomes) associated with interventions and outputs, assess how well the observed changes support the theories of change and logic of the logframe, and identify factors (both internal and external) in the implementation or context that impede or promote the achievement of targeted results. 5) Related to collaborative learning and action –Review systems for capturing and documenting lessons learned and assess the extent to which they are used in implementation and refining program design, including feedback from the perspective of stakeholders and participants. Assess processes to use evidence, including baseline results and monitoring data, for adjusting program strategies. Assess how well the program is seeking out, testing and adapting new ideas and approaches to enhance effectiveness or efficiency. 6) Related to sustaining impact – Determine the extent to which outcomes, systems and services are designed and being implemented to continue after the award ends, and assess progress made on implementing sustainability strategies. What activities are being implemented to ensure that the service providers will have continuous access to required resources and capacity strengthening support? How has Fararano been creating demand and influencing the motivations of the beneficiaries and service providers? What has been 3 ADRA-CRS Madagascar Midterm Review Scope of Work, 9 December 2016. 14 done to establish and strengthen i) critical linkages necessary to sustain resources, and ii) capacities that may positively or negatively influence sustainability? Has the program identified the indicators and planned for a phased transfer of responsibilities yet? 7) Relative to the major cross-cutting themes in both awards – Determine the appropriateness and effectiveness of support for gender equity in terms of access to, participation in and benefits from interventions. Assess the extent to which interventions target youth, support greater capacities for local governance and address sources of environmental risk. Methodology Process Overview. The reviews of sister programs ASOTRY and Fararano each consisted of a preparation stage, a fieldwork stage and a reporting stage. Preparations began in January 2017 and included reviewing the draft statement of work, document review, protocol and tools development, site selection and logistics planning. The ASOTRY mission took place from April 11 to 26, followed by the Fararano mission from April 27 to May 12. Both missions included introductory meetings with USAID and awardees, fieldwork, a validation workshop, a debriefing and a recommendation and planning workshop. The validation workshops took place at the end of each mission (April 25 for ASOTRY and May 10 for Fararano) and involved staff from ASOTRY/Fararano, implementing partners for the respective award, USAID mission staff and staff from ADRA/CRS headquarters. The workshops aimed to validate the Joint Midterm Review (JMTR) team’s observations with ASOTRY and Fararano implementers’ staff. The JMTR team gave a debrief to USAID/Food for Peace (FFP) Madagascar before the end of the mission to review findings. The team returned to Madagascar in June to review and finalize recommendations with both projects. The recommendation workshop (June 20 to 22) aimed to provide Fararano and ASOTRY staff an opportunity to review the feasibility of the recommendations, analyze the challenges to implement them, develop alternative recommendations in cases for which the original recommendations were deemed particularly challenging to implement and develop an action plan to implement the recommendations. Methodology. The methodology for both ASOTRY and Fararano reviews was guided by a detailed protocol (Annex 1).4 The JMTR primarily used a qualitative approach. Both reviews began with a desk review of documentation such as the baseline study report, technical narrative, theory of change and logical framework, monitoring data and reports including annual and quarterly reports, monitoring and evaluation (M&E) plan, the Indicator Performance Tracking Table (IPTT) and implementation manuals as applicable. During the course of the reviews, the team continued to review, consult and analyze information from these and additional relevant sources. The team also analyzed the annual monitoring data. 4 The protocol contains all interview guides and further details about the review process. 15 During the mission, the JMTR team applied primary data collection methods, including semi￾structured in-depth-interviews and group discussions with program managers, technical staff, service providers (frontline staff and volunteers) and participants, as well as some non￾participants. The JMTR team observed learning/training sessions as available, and conducted in￾depth visits of infrastructure projects. Inquiry focused on perceived program effectiveness, constraints inhibiting effectiveness and suggested means of overcoming these constraints. Key informants included government staff, commune mayors, USAID/Mikolo program management staff, key management staff from the Surveillance and Education for Schools and Communities on Food and General Nutrition (SECALINE), awardee and implementing partners at country office and field levels, participants, indirect beneficiaries, non-beneficiary community members, community leaders and Government of Madagascar representatives at national and local levels. Fieldwork also included direct observationover project activities such as nutrition sessions, growth monitoring and promotion, knowledge and technology transfer sessions, demonstration plots and other household and community-based activities. The team also conducted in-depth visits of irrigation and road infrastructure supported by the project. Review Team. The JMTR was implemented by a team of development professionals representing FFP, the USAID Mission in Madagascar, CRS and ADRA led by the FFP Senior M&E Advisor. The team was comprised of “core team members” and “observers.” Core team members participated in the full review process for both the ASOTRY and Fararano awards, or as much of the process as possible, and led the investigations in assigned areas per their technical expertise. Observers provided ideas and input to the core team in their areas of expertise but did not partake in the analysis. See Annex 1 (protocol) for detail on the team members. Sample. The JMTR team selected a purposive sample of interview sites representing variation in agro-ecological zones, livelihood strategies, proximity to major infrastructure (markets, roads, towns) and access to resources and services, quality of service delivery and coverage/intensity of project services. Other considerations for sample selection included physical accessibility and the team’s ability to visit all sites in the time available for the review, which meant choosing communities requiring less travel time to reach. For example, a number of the communities in 16 the Southeast region targeted by BDEM are extremely remote. These communities can be accessed only by foot, and many require several days’ travel to reach. The JMTR team left these communities out of the sampling frame. In addition, some communities were identified as unsafe considering the security situation and were omitted from the sampling frames as well. The JMTR team visited 10 communes where ASOTRY is implemented (Ambondromisotra, Tsarasaotra, Mahazoarivo, Vohiposa, Mahatsinjony, Anjoma, Marosavoa, Soaseràna, Beroy and Maroarivo), and 10 communes where Fararano is implemented (Mahatrsara, Vohitravinona, Antaretra, Anosimparihy, Voreo, Antanimieva, Behompy, Tsianisiha, Belalanda, Miary and Ambohimahavelona). The team visited and interviewed one or two communities in each commune. Analysis. The JMTR team recorded field notes and convened daily to discuss and process emerging findings. The team thus began preliminary analysis in country during and after fieldwork, vetted initial findings through the validation workshops, and continued its analysis post-mission using qualitative analytical methods. The Baseline Status: As many as 365,000 people (77 percent) in the Fararano target area live in extreme poverty (below $1.90 a day). Over 30,000 children age 0 to 59 months (40 percent) are stunted, approximately 21,000 children age 6 to 23 months (95 percent) do not receive a minimum acceptable diet, and approximately 32,000 mothers (60 percent) do not practice exclusive breastfeeding. Approximately 26,000 children age 0 to 59 months (34 percent) reported suffering from diarrheal diseases in the two weeks prior to the baseline survey and more than 81,000 households (85 percent) do not use an improved water source. Members of 91,000 households (95 percent) do not wash hands using any cleansing agent, 94,000 households (98 percent) do not use an improved latrine, and approximately 68,000 households (71 percent) reported practicing open defecation, suggesting 26,000 households using some sort of latrine but not improved – an issue that can be tackled by SBCC. As many as 85,000 women of reproductive age do not consume a diet with minimum diversity. More than 105,000 farmers do not use at least two sustainable natural resource management practices, and 90,000 farmers do not use at least two sustainable livestock management practices. Limitations and Challenges The lack of consistent participation of all JMTR team members was one limitation of the review. All JMTR team members have full-time jobs and other responsibilities; therefore, in some cases, a team member was not available to participate fully in the review of his or her technical component in both awards. As a result, other colleagues with similar expertise reviewed the component. This posed challenges to interpreting observations because of the varied experience of these team members. To mitigate this challenge whenever possible, the JMTR members overlapped in the field or had meetings to share their observations and interpretations. In addition, the team leader accompanied the new members in the field to ensure consistency in observation and interpretation. 17 Related to the point above, securing technical specialists from CRS, ADRA and FFP who could commit two months to participate in the review was a challenge. It was planned for the FFP Gender Advisor to join the team, but ultimately this did not work out due to travel-related complications. While another team member was assigned to address gender aspects of the review, the gap in specialized gender expertise on the team meant that gender integration and other gender aspects could not be fully evaluated. Another limitation was that due to time and scheduling constraints, the JMTR team could not interview an adequate number of non-project participants. The travel policy of USAID, ADRA and CRS defined the time that the team could spend in the communities. Some of the communities selected were remote and it took significant travel time to reach them. This limited interview time in the communities. Since the JMTR team tried to interview a large number of direct participants, limited time was available to interview non-participants. For this reason, the JMTR team also could not interview a large number of government and non￾government stakeholders. Given the timing and the allotted time for fieldwork and the limited number of infrastructure (road and irrigation) projects in the target areas, the JMTR team visited only a small number of infrastructure projects, which was inadequate to gain an understanding of the quality and condition of the infrastructures developed by the two projects. The JMTR team therefore prioritized i) active interventions where the projects made the greatest resource investments, and ii) interventions considered to be making a relatively high contribution toward achieving strategic objectives. The team recognizes that the resulting trade-off was inadequate attention to the range of infrastructure interventions. While the Fararano M&E system produces data of reasonable quality, the JMTR team identified several issues related to data quality; therefore, the JMTR team is careful in making inferences. Although individual members drafted their own sections, assembling the draft reports has been a challenge, taking longer than anticipated. To address this issue, FFP requested TOPS to assist with assembling and editing the two reports. Findings Program Quality The JMTR team reviewed various aspects related to program quality; each is discussed below. Staffing and Training The project recruited all staff members by FY 2015. Fararano faced challenges to find qualified staff for the SBCC Specialist and the Livelihoods Team Leader for the CRS team; however, CRS eventually filled these positions. CRS and its partners formed partnerships with local higher education institutions for agriculture 18 and nutrition to identify recent graduates who were willing and able to live and work in these areas. These relationships with local institutions were a major success for finding qualified candidates for technical positions. Fararano coordinated with USAID/Mikolo in common areas, used training materials from USAID/Mikolo, and jointly conducted trainings for Community Health Volunteers (CHVs) on Integrated Management of Childhood Illnesses (IMCI) to harmonize activities. Interviews with Fararano staff and direct participants suggest that the staff are dedicated and highly motivated; however, the JMTR team identified various challenges to program quality in relation to program purposes integration, geographic coverage, access to land and challenges to sustainability. Formative Research Fararano completed formative research for eight key behaviors in FY 2016 and integrated SBCC components with Care Groups, miranjaka, disaster risk reduction (DRR), Community￾Led Total Nutrition (CLTN) and Go Green environmental components. Fararano staff produced short videos on key messages with community members and presented them to the community members. Integration of Program Purposes at the Household level Fararano was designed as a multi-sectoral food security project, and in fact, the project has been implementing multi-sectoral interventions at the project level. However, it has not been offering a multi-sectoral package of interventions to a large proportion of households. Many extremely poor households receive interventions under only one sector (purpose). The Fararano Theory of Change (ToC) and baseline study indicate that multi-sectoral interventions are needed to achieve food security objectives. For example, the baseline survey revealed that 77 percent of households in the Fararano target area are extremely poor ( 3,000 people), rather will work with the private sector and communities to expand or rehabilitate existing systems. ● Fararano plans to construct or rehabilitate 20 monoblocks. These are single structures designed for muti-purpose use by public that include toilets and showers for men and women, as well as laundry facilities. The construction and location of monoblocks will take into account the distance and access to these facilities to ensure that women’s and girls’ workloads, distances to travel and safety are prioritized. Monoblock construction will be cost shared with private entrepreneurs who will manage the service as part of the GFWSS. ● The project plans to installed Canzee wells. The project used a more conservative ratio (200 people per pump) in estimating the number of wells compared to what the Madagascar Government recommends. In installing deeper wells, it plans to use two government approved hand pumps. Canzee pump is a water lifting technology used in combination with manual drilling. Both pumps are manufactured locally in Madagascar and approved in the national procedures manual. Purpose 2: Households have increased and diversified agriculture production and sustainable economic well being ● Fararano implements a modified nutrition sensitive approach called the Diversity for Nutrition and Enhanced Resilience (DiNER) fair to allow vulnerable female and male farmers to access nutritious (fruit, vegetable, orange fleshed sweet potato) or staple (rice, corn, cassava) food seeds, saplings and cuttings; organic fertilizer and tools; and veterinary inputs (chicks). A total of 11,000 HH will receive one-time barcoded vouchers in Y2 (between $10-15/HH) and participate in a total of 44 DiNER fairs (1/Commune). LM’s, LM-G and LF will also receive vouchers for a one time start-up kit in Y2 to promote production activities. ● It promotes the five SMART skills through the SILC approach. The SILC approach builds skills on group management; knowledge on savings, loans, reimbursement, benefits; increased access to credit (through the internal lending process); but also it provided emergency funds in times of need, increased social cohesion and provided skills on numeracy, accountability and discipline. A new 3 month add-on to the existing SMART skill #2 modules will strengthen skills around financial education. ● Fararano works with households to select crops that may be best used and could provide the cii most food for household use and/or sale. To support this process, and in line with the Ministry’s interest in diversification and intensification, Fararano applies CRS’s “whole farm – whole family” approach to food and nutrition security. ● The project uses Lead Farmers (LF) approach. The LFs play an important role in creating linkages between the community and the CSA, which is the government entity at the commune level providing resources for farmers. To sustain the LF approach and ensure that new techniques are rolled out and that knowledge is transferred across community lines—and also aligned with the Fararano Exit Strategy— LFs receive project certification and are registered at the CSA when s/he demonstrates the capacity to continue supporting farmers based on 4 criteria: 1) adequate technical knowledge and skills on improved practices they are promoting; 2) sufficient access to resources required to continue promotion and adoption of new techniques; 3) motivation needed to continue promotion and adoption; and 4) partnerships to support continued promotion and adoption. ● Fararano uses two primary approaches to strengthen livestock, aquaculture and other production techniques: 1) introduction, demonstration and adoption of new techniques via LF if these specific production areas are prioritized, and 2) improved veterinary inputs and services via collaboration with other partners and a network of PiSP linked to the private sector. CRS and partners plan to train 44 community (1/Commune) PiSP (primarily boys and girls 18-24 years of age) on vaccinations and basic remedies for major animal diseases, and then will be linked to regionally certified veterinarians to obtain a supply of materials and/or services to be provided on fee-basis to farmers in the target areas. ● The project prioritizes three types of hydro-agricultural infrastructure depending on the region: 1) small scale irrigation systems diverting water from sources or rivers; 2) small dams to reduce flooding and/or hold water for irrigation or linked to aquaculture; and 3) drainage systems. Approximately 26,000 HH will be engaged in FFA activities during the dry season. FFA daily rations are in line with FAFSA II recommendations and best practices to reach the most vulnerable and contribute to dietary quality and diversity during the lean season. ● Fararano works with HH and buyers to extend the reach of value chains into rural communities. Any farmer with excess production and interest can participate. The value chain approach integrates a nutrition-sensitive lens to determine potential areas to improve the nutritional value of the product, the use of incomes to purchase more nutritious foods and to mitigate the potential negative effects such as overselling HH production to the detriment of women’s and children’s nutrition. Fararano supports a total of 360 Producer Organizations to identify new market opportunities, develop their own business plans and have strong relationships with private sector by the end of the project to facilitate sustainability and gradual phase out of project support. ciii ● It plans to provide SMART46 skills training to approximately 24,000 members via 1,317 (3 x 439 fokontany) groups. As each group places differently along the vulnerability spectrum (Recover￾Build-Grow), trainings are conducted only when members are ready. ● Fararano facilitates linkages with buyers. The lead farmers play an active role as partners, providing stable markets for producers, carrying out training as necessary, and facilitating access to value chain financing. The project also facilitates linkages with input providers. The project plans to build demand for inputs (seeds, tools, natural fertilizers and pesticides, storage and processing technologies) and services (tillage, veterinary) through CPOs, which have a larger production base to incentivize dealers. ● The project facilitates linkages with financial service providers. The project facilitates contact with Microfinance Institutions (MFI) for larger CPOs looking to access credit to expand their business. NCBA CLUSA works with existing MFIs to provide basic training in loan application directly to producers. Training on basic numeracy is integrated into SILC, and additional training on literacy or other basic skills are considered depending on group needs. Where relevant, training on mobile banking services will be conducted with considerations on male and female access and control issues expanded upon in Gender Analysis. ● Road rehabilitation activities using FFA target vulnerable HH (with members able to work) just prior to the on-set of the lean season to allow for families to have additional resources available and stocked away in preparation for this period. Maintenance and sustainability of roads will be led by IMA members with increased governance capacity. Purpose 3: Communities’ resilience to shocks is enhanced and natural resource degradation is reduced ● The National Bureau for Risk and Disaster Management (BNGRC) has established Risk and Disaster Management committees at district, commune, and fokontany levels (via VDC) whose mandate is to develop measures to prevent/mitigate, improve preparedness plans and respond to disasters.47 , Fararano plans to align its activities with the BNGRC. ● To strengthen community based VDC-led GRC capacities and strategies, Fararano facilitates integrating simplified image-based plans and conduct participatory simulations into training and coaching activities led by FA. Fararano coordinates activities with other partners at the national 46 Selecting products, analyzing markets, calculating costs, incomes and profits for a new agro-enterprise, working with business development services, building business plans, collective marketing; and reviewing actual costs, income and profits at the end of the season 47 BNGRC Activities on BNGRC website: http://www.bngrc.mg/mission/43-les-activites-du-bngrc.html civ (BNGRC and civil society platforms), regional (WFP and ACF in southwest, ECHO funded DIPECHO activities in Vatovavy Fitovinany) and commune levels through participation in cluster meetings, and regular communication. ● Fararano FAs train and support VDC to develop disaster risk reduction (DRR) and emergency preparedness plans that are aligned with national norms, validated by the CCGRC and in line with (or will inform) commune level plans. ● The project works with No Strings48 using puppets with children ages 6-14 to broadcast-quality puppet films – Tales of Disasters (short 10-15 minute films about cyclones and floods49) produced and translated in Malagasy. Puppets form a vehicle through which to communicate essential information, especially (though not limited to) children who will return home and share with other family members. ● Fararano establishes Early Warning System (EWS) in all 44 communes. The EWS is integrated directly into the Commune level development plan. Fararano works directly with Communal GRC Committees and with the VDC at the community (fokontany) level to develop these plans. A total of 44 gender-responsive GRC pictorial plans are developed at the Commune level that consider specific needs of men, women, girls and boys and the disabled. ● Among the NRM activities, Fararano supports: (1) watershed development and protection, (2) management of resources using Farmer Managed Natural Regeneration (FMNR) of trees and Evergreen Agriculture (EGA), and (3) deploy agroforestry practices to protect water recharge and sensitive areas. All activities are aligned with the GOM’s strategy to promote a sustainable green revolution as part of the MAP. ● Fararano plans to develop an action plan to work together with CCGRC and VDC to establish (if not in place), review/revise (if exists) their GRC plans, modify indicators to collect and responsibilities for each involved party (commune, fokontany, CHVs, LMs, CSB, etc.). Fararano plans to have 42 communal based plans in place by the end of Year 2 (40% in year 1 and 60% in year 2). The EWS will be led by female and male members of the VDC at the community level and community members participate on a voluntary basis. For data collection, already identified volunteers (LF, CHV, miranjaka, LM) will provide information based on finalized set of indicators (both standard and locally adapted as needed) to be collected. ● FA provides community members (VDC) and volunteer farmers (including LF) the necessary 48 No Strings was founded in 2003 by a former War Child and Goal humanitarian aid worker and two of the leading talents of the original Muppet Show and Fraggle Rock, after spontaneous use of an old puppet at a displacement camp in Sudan attracted hundreds of people at a time when important messages were proving difficult to disseminate clearly. Short clips of Tales of Disaster can be found http://www.bing.com/videos/search?q=Tales+of+Disaster+youtube&docid=4883104190497289&mid=D11317A329A9976D48D3D11317A329A 9976D48D3&view=detail&FORM=VIRE5#view=detail&mid=D11317A329A9976D48D3D11317A329A9976D48D3 49 It is important to note that activities focused on cyclones and floods will take place three months before the cyclone season. cv skills and tools to conduct assessments of the ecology, the most appropriate types of indigenous trees and agroforestry interventions (e.g. riparian buffer zones, hedgerows, wind breaks, woodlots, etc.) for reforestation activities and work closely with irrigation and water infrastructure activities to develop comprehensive micro watershed plans that consider water needs for agriculture (including animal husbandry), HH use and other uses by communities. ● Fararano will provide the SMART Skills training that include natural resource management principles and practices so that they can develop, implement and monitor their own NRM programs. Watershed planning techniques are incorporated into LF and SMART PSP training programs to ensure that more sustainable practices are used to increase soil fertility and reduce deforestation while labor and energy saving technologies for cooking (sustainable charcoal production, energy saving cookstoves from local materials) are integrated into CG, CCFLS and CBMGP activities. Fararano supports reforestation activities to strengthen watersheds and reduce soil run-off. ● ICRAF facilitates training of “Trainers” (FA, LF and SMART PSP) on FMNR starting in the southwest to focus on areas that have sustained prolonged droughts and suffered significant environmental degradation and will be scaled up to other regions where appropriate. ● ICRAF plans to facilitate ToT with Fararano technical staff and provide technical assistance to identify and develop Evergreen Agriculture (EGA) management strategies to enhance crop/pasture productivity, develop communication materials, and build capacity of Fararano partners and communities. In 44 communes, demonstration plots on farmland and rangeland currently under threat of land degradation will be established while EGA activities are integrated into trainings for 230 SMART PSP and approximately 600 LF (those focused on crop based agriculture). ● Fararano closely work with the Cantonment and CIREF to collaborate and provide support to VDC for the establishment of watershed management plans and reforestation activities. These plans are validated and approved by the Ministry to establish them as part of the government system and will allow for additional support beyond the life of the project. ● Fararano implements activities to raise community awareness, especially land-restricted and FHH, on the process of acquiring land and obtaining titles to secure land ownership. Through technical assistance provided by the J&P50, assessments of land use, land tenure and natural resources utilization sessions are conducted at the commune level, starting first in the eastern regions where access to land is more restricted and scaling up to the south-central and southwest. The J&P Commission conducts three fundamental trainings at the commune level with existing Civil Society Organizations (CSO) focused on 1) Good Governance principles; 2) 50 CRS Madagascar has worked with the J&P Commission since 1994. Since 2009, J&P Commission has worked with CRS and advocated for good governance of mining revenue and provided training and debates on land tenure issues. Through private support from CRS, J&P Commissions have provided advice on land tenure procedures. cvi Advocacy; and 3) Land Tenure and Natural Resource Rights and Responsibilities). ● The project works with the communes, fokontany, and private sector partners to establish and scale up Private Public Partnership (PPP) business models. Fararano business models aims to decentralize the input supply chain for infrastructure and maintenance services, as well as create small enterprise opportunities for men, women, and associations. ● Fararano staff invites mayors in each of the participating communes to learn practices on good governance – participatory budgeting processes, transparency in communication, advocacy to regional and national levels for community issues. The Fararano program plans to build capacity around good governance including developing local networks of civil society and government that allows for community conversations and accountability to local authorities to ensure good governance practices are upheld. ● The project staff work with community and religious leaders to target HHs with unmet basic needs and sensitizing them on possible revenue generating activities, including SILC groups and existing on and off-farm community activities. VDCs are sensitized on the importance of integrating and including the most vulnerable in existing off farm community groups (i.e. bee￾keeping, basket weaving) and the project provides basic training on selected both on- and off￾farm activities via LF. Fararano also works with the Ministry of Population and Social Protection to identify the most vulnerable to determine how government or other partner-sponsored programs can support these individuals. ● Fararano staff work with local authorities to find fallow land which can be exploited for agriculture activities including: land that belongs to the state or is owned but not utilized by certain members of the community. Table 1. Summary Overview of ADRA/ASOTRY and CRS/Fararano (LOA) CATEGORY CRS/Fararano ADRA/ASOTRY Total Value (Commodities + CASH) 39,177,000 USD ● 20,013,300 202e ● 3,892,900 ITSH • Total MT 20,820 MT • Regions • Atsinanana • Vatovavy Fitovinany • Atsimo Andrefana • Haute Matsiatra • Amoron’i Mania • Atsimo Andrefana cvii No. Districts 1. Brickaville 2. Toamasina II 3. Ifanadiana 4. Mananjary 5. Toliara II 6. Sakaraha 7. Morombe 10 No. Communes 49 32 (6 SALOHI) No. fokontany (including any that were also targeted under the former MYAP “SALOHI” 464 (22 SALOHI) 408 Population 539,480 491,169 Beneficiaries (individuals) 363,945 264,380 IV. EVALUATION TEAM The JMTR will be implemented by a team of development professionals, including representatives from FFP, USAID Mission, CRS, and ADRA. Some members of the team are designated as core team members while others are designated as observers, as indicated in the list below. Core Team Members are expected to participate in the full review process for both projects, or as much of the process as possible, and will have responsibility for leading investigations in assigned areas. These persons will develop data collection tools in their areas of responsibility and will also prepare presentations for Ground Truthing/ Verification Workshops and will analyze the findings and formulate the recommendations linked to the findings. For the three program components in each project (corresponding to Purposes), one of the Core Team Members will be designated as the lead for that component and will have the final say on the formulation of recommendations in the assigned areas and on the content and the recommendations. Observers will provide ideas and input to the Core Team Member in their areas of expertise, however, the core team may or may not incorporate their input into the analysis. A. Team Members Arif Rashid (JMTR Team Leader, Senior M&E Advisor, Core Team Member). Arif Rashid will be responsible for coordinating implementation of the JMTR, including facilitation of cviii meetings/workshops/ debriefings, working with FFP, Mission, CRS and ADRA to develop implementation schedules, facilitating the sharing of information between team members, providing support to the project implementers in developing recommendations, and putting together the analysis from team members. In addition, he will have Core Team Member responsibilities associated with Disaster Management, Resilience Capacities to manage shocks; and climate change and environmental compliance of both projects, Sustainability, Targeting, Management Systems, and Partnerships. He will also contribute to the review of monitoring and evaluation systems. Adam Reinhart (FFP Agriculture/Food Security Advisor, Core Team Member). Adam Reinhart will lead the review of Agriculture, Natural Resource Management, Livelihoods and Food security. He will review the agriculture, natural resource management, on and off farm income opportunities, and the processes used to achieve these outcomes. Chris Seremet (CRS Technical Advisor -WASH, Core Team member). Chris Seremet will be responsible for reviewing the quality of infrastructures, and contributing to the WASH component. He will closely work with Nicole Van Abel to review the WASH activities (both infrastructure and social and behavior change) and contributing to the analysis and formulation of recommendations. Jefferson Shriver (CRS Senior Technical Advisor, Value Chain and Markets, Core Team Member). A Value Chain Specialist, Jefferson will be responsible for reviewing the implementation of value chain and agri-business development activities and processes, including market opportunities and constraints. He will analyze the opportunities and challenges for targeted beneficiaries to participate in value chains and market. He will analyze current economies of scale and farmer organization models, value chain upgrading strategies, producer access to services (inputs, finance), government policies that help or hinder value chain agriculture development, and business linkages / alliances. He will review a range of value chain dynamics to understand how the projects are performing. Justin Mupeyiwa (USAID M&E Specialist, Core Team Member) Justin will be responsible for M&E. He will lead the review of the monitoring system for both of the projects. This would include the review of the M&E plan, efficiency and effectiveness of the monitoring system, data quality assurance mechanism, data use in general with a particular focus on the use of baseline data in refining project design. He will coordinate with other members to receive inputs. Melanie Thurber (FFP Nutrition Advisor, Core Team Member). Melanie will lead the review of the interventions and processes being used to promote maternal and child health, nutrition, infant and young child feeding practices under the 1000 days approach. She will coordinate with Natsayi to investigate the MCHN components in both projects. She will review the ration basket and size, and the quality and effectiveness of the social and behavioral change approaches. cix Natsayi Nembaware (ADRA Senior Technical Advisor for Nutrition, Core Team Member). Natsayi will team up with Melanie Thurber to review the MCHN activities and the processes used by both of the projects to promote MCHN. She will review the quality and effectiveness of the social and behavior change processes and contributing to the analysis of the observations and interviews and formulation of recommendations. Nicole Van Abel (FFP WASH Advisor, Core Team Member). Nicole Van Abel will be responsible for the WASH components. She will coordinate with Chris Seremet to review the WASH components in both projects and will be taking the lead in analyzing the observations and interviews and formulating recommendations. Bridget Ralyea, (FFP HQ, Observer). Bridget is the Geo Team Leader for Southern Africa Region and will be joining the JMTR from April 27 through May 11. She will provide input to Arif Rashid on the overall review process. Elizabeth Brown (FFP Nutrition Team Lead, Observer). Elizabeth will be joining the JMTR from April 10 through 26. She will team up with Carla Boussen to review gender and youth integration and targeting of the ASOTRY project and will provide input to Carla Boussen. Eddy Rasoanaivo, (Madagascar Mission, Observer). From April 27 through May 11. Martin Zafy, (Gender and Socio-organization coordinator of ASOTRY, Observer). He will participate from April 11 to 26. He will also liaise with the ASOTRY to take the team to the field and coordinate with the field staff. Beth Ceryak and Holly Tripp (FFP HQ) will participate in the review to document the discussions and analysis. They will also document case studies in the field. Beth will participate from April 10 through 26 while Holly will participate from April 27 through May 11. Tantely Randrianarisoa (Logistics Coordinator for the MTR team). Tantely will join the MTR team from April 10 through May 12. He will provide logistical support to manage the use of the six rented vehicles (3 from ADRA and 3 from CRS), and any other vehicles that may join the MTR. He will also manage hotels reservations or changes to these reservations, confirm appointments with project staff in field locations, and cater for the logistical needs of the MTR team members. cx B. Assignment of Responsibilities Table 2 indicates core team member responsibilities in the JMTR. Table 2. JMTR Core Team Member Responsibilities INVESTIGATION TOPIC CORE TEAM MEMBER ASOTRY FARARANO Overall Program Design Theory of Change, including risks and assumption Arif (With support from all team members) Targeting of Beneficiaries Arif (With support from all team members) Collective Impact at the Goal Level Arif (With support from all team members) Collective Impact at the Purpose level - P1 Health & Nutrition Melanie (Lead), Natsayi, Nicole, & Chris Collective Impact at the Purpose level - P2 On and Off farm Income Adam (Lead), & Jefferson Collective Impact at the Purpose level - P3 Disaster Management & Household Resilience Arif (Lead) Inputs, implementation, Outputs, Outcomes and Sustainability SP 1.1 Improved health and nutrition behaviors of caregivers and children under five. SP1.1 Women and children have improved consumption of diverse and nutritious foods Melanie & Natsayi SP1.2 Increased utilization of health and nutrition services for women of reproductive age and children 0-59 months SP1.2 Women and children (especially during the 1,000 days) utilize preventive and curative maternal and child health and nutrition services Melanie & Natsayi SP1.3 Reduced incidence of water- and hygiene-related illnesses for children under five SP1.3 1.3-Households practice optimal water management, hygiene, and sanitation behaviors Nicole & Chris SP2.1 Increased Agriculture SP 2.1-Increased diversified Agriculture Adam cxi Production Production SP 2.2 Increased Agriculture Sales SP 2.1 Increased on and off farm sales by HHs and POs Jefferson SP 2.3 Increased engagement of women and men in micro￾enterprises --- Jefferson and Carla SP 3.1 Community disaster mitigation Assets improved SP3.1 Community-based disaster mitigation systems meet national standards SP3.2 Community-based disaster preparedness systems meet national standards SP 3.3 Community-based disaster response systems meet national standards Arif, Nicole and Chris SP 3.2 Community response capacities improved SP 3.4 Community based social safety net mechanisms strengthened Arif and Adam --- Cross Cutting Areas Gender Equity Carla Governance Arif Environmental Management Arif Targeting of Youth Carla Natural Resource Management Adam Implementation Systems and Resource Management Monitoring and Evaluation Justin Management & Partnerships Arif cxii Collaborative Learning and Action Justin and Arif Integration & Complementarity Adam Targeting Arif, Adam and Melanie Sustainability Adam, Melanie, and Arif V. INFORMATION TO BE OBTAINED The following sections outline the information that will be obtained to achieve each of the seven objectives of the JMTR while meeting the purpose of the JMTR to generate recommendations for the remainder of the lives of the ASOTRY and Fararano projects. A. Objective 1: Relevance of Project Strategies To accomplish Objective 1, the JMTR will review the theories of change for the two projects. These have been developed around a vision and pathways of change for particular types of targeted impact groups, so the JMTR will review the targeting systems for the two projects to identify strengths and weaknesses in how the projects identify participants compatible with the Theories of Change that have been developed. The JMTR team will also examine the history of the projects, particularly how they have evolved since inception and the critical features of the operating environment that have affected, positively or negatively, project implementation and the impact that has been achieved. The JMTR will review the baseline study, formative research and all other studies to determine what changes may be needed to project activities and implementation mechanisms to ensure that the strategies remain valid for the local context. The analysis will inform the formulation of recommendations associated with the overall strategy of the program toward achieving the goals of the two projects. Information for understanding the history of the program and contextual factors affecting implementation will be obtained from project monitoring reports, as well as through interviews with project implementation staff and partner leadership representatives. B. Objectives 2 (Outputs) and 4 (Outcomes): Project Inputs, Approaches, Outputs and Outcomes The JMTR will examine the outputs produced under each Sub-Purpose to identify what the projects have accomplished on the ground. The team will review inputs and processes used to produce outputs as well as the quality of the outputs in terms of achieving change. The JMTR will use both qualitative information and available quantitative data to assess the effects and impact of the outputs produced by the project at different levels, including effects at the cxiii immediate outcome level, effects at the Sub-Purpose Level for clusters of outcomes, effects at the Purpose level for the set of Sub-Purposes and impact collectively at the project Goal level. JMTR team members will look for evidence on how members of target groups have changed their ideas, attitudes, and practices as a result of project activities and will identify potential reasons for why some beneficiaries have started applying project promoted practices while others have not. At every level, the evaluation will assess the sustainability and replicability of the changes that have been observed. The review team will consider staffing and activity resources, community participation, participant targeting, asset transfers, the extent to which activities and outputs demonstrate a commitment to the cross-cutting issues of gender, youth, governance and the environment, and sustainability strategies. The review will investigate the expected change as per the project documents, but will also be observant for new areas of unexpected effects and impact, both positive and negative, that are occurring as a result of project activities. Recommendations may be formulated to include these in the projects' monitoring and evaluation tools. The team will obtain information on processes, outputs and outcomes through observation of sessions and training, interviews and reviews of project monitoring reports, including quarterly and annual reports, reviews of key secondary data, interviews/focus group discussions with project participants, and interviews with project implementation staff. JMTR team members will analyze performance monitoring data collected by the projects, as well as review the performance management plans and IPTTs. JMTR team members will review technical guidance, including implementation manuals, on key processes and approaches used by the projects and observe learning/training sessions that are planned by the projects when the JMTR team is in the area. C. Objective 3: Coordination and Collaboration The JMTR will obtain information on external actors, i.e., other projects and service providers that are being implemented on the same space and relevant to the strategies of the ASOTRY and Fararano Projects. These include external actors who provide complementary services necessary to achieve and sustain project outcomes, external actors that influence people’s access to goods and services, and external actors that promote or impede an “enabling environment”. The JMTR team will work with project implementation staff to identify the most important of these external actors for each component of the project and will then analyze the quality of the coordination and collaboration with the relevant project. Particular attention will be given to interaction with other USG-funded activities as well as coordination with GoM services. The team will obtain information on the quality of coordination and collaboration with key external actors from project monitoring reports, interviews with project implementation staff, and interviews with representatives of these key external actors. [Objective 4 is combined with Objective 2] D. Objective 5: Collaborative Learning and Action Collaborative Learning and Action refers to (a) facilitating collaboration internally and with cxiv external stakeholders; (b) making adjustments to project’s theories of change and implementation strategies based on continuous learning; (c) how monitoring and evaluation systems provide information for decision-making and reporting, and (d) how best practices and lessons learned emerging from project implementation are documented and shared outside the project. The JMTR will obtain information to understand the systems being used for capturing, documenting and disseminating lessons learned and best practices. The M&E systems for both projects will be examined to understand how information generated from the systems is being used to enhance the effectiveness or efficiency of the project in achieving impact. Emphasis will be placed on the collection, analysis and management of data to enable iterative learning and evidence-based improvements to project design and implementation. The JMTR will identify changes that have already occurred in the project strategy or implementation as a result of new ideas or approaches brought into the project from outside sources and will formulate recommendations for how to seek out appropriate new ideas and approaches in the remaining life of the projects. The team will obtain information for collaborative learning and action from a review of the revised theories of change, projects' PMPs, monitoring reports and other reports, interviews with project management, interviews with project implementation and technical staff, interviews with the leadership of partner organizations, and interviews with knowledge management staff. E. Objective 6: Sustaining Project Outcomes and Impact The JMTR will review the sustainability strategies, interview staff and management, interview project participants, and interview private and public stakeholders to determine how likely project-generated outcomes and impact are to be sustained after the projects end. The review will identify the organizations, services and relationships that are being developed that are necessary to sustain the service delivery after the project ends to sustain each of the outcomes planned by the projects and will analyze the threats to these that could affect likely sustainability. Specific attention will be paid to the appropriateness, efficiency and efficacy of capacity building activities targeted to participants and local partners to enable them to sustain project impact. F. Objective 7: Cross-Cutting Themes (Gender, Governance, Targeting Youth, Climate Change and Environment) For each of the five major cross-cutting themes in each project, including gender, governance, targeting of youth, climate change and environment, the JMTR team will assess how well the projects are operationalizing the cross-cutting themes to achieve the proposed impact. The team will obtain the information on cross-cutting themes through interviews with project management, project implementation staff, leadership of partners and stakeholders, and project participants. The team will review project monitoring reports, and specific strategy documents related to each cross-cutting theme, including assessment reports and barrier analyses. VI. EVALUATION PROCESS The JMTR will be undertaken over a period of approximately six months from February through end of July of 2017 and will be implemented in four phases: cxv → Phase 1: Evaluation Preparation (February 1 through April 10) → Phase 2 (in Madagascar): Review of the ASOTRY Project (April 11-26) → Phase 3 (in Madagascar): Review of the Fararano Project (April 27 – May 12) → Phase 4: Evaluation Recommendations Finalization and Processing (May 15 – July 28) The current schedule for the JMTR is provided in Annex A. The sections which follow describe the major activities planned in each phase. A. JMTR Preparation (February 1 through April 10) During the JMTR preparation period, the composition of the review team was finalized (as indicated in section V.A. above), a draft protocol including a basic operational plan (this document) was drafted, the JMTR team will review the background documents (listed below), draft the key questions and data collection tools, and select the sites for field visits. The Core Members of the JMTR team will assemble in Antananarivo by April 9. On Monday, April 10, the team will hold an initial meeting to meet each other and to discuss the JMTR plan. Also on that day, an orientation meeting will be held with USAID/FFP Madagascar. The following sections describe these JMTR outputs and processes in more detail. 1. JMTR Protocol. This document is the protocol that guides implementation of the JMTR. The Zero Draft of the protocol will be released for review by organizers of the JMTR and members of JMTR team 7 March. Comments, questions or suggestions on the zero draft should be sent to Arif Rashid (arashid@usaid.gov). On or about March 20, a conference call will be organized to discuss the protocol toward finalizing it into a working draft that will be widely circulated by March 24, as the final planning guide going into the implementation phase of the JMTR. Further changes will be made once the JMTR arrives in-country and has a chance to discuss the projects with project implementers. 2. Background Document Review. There are many documents that could be reviewed to prepare for the JMTR. However, given time limitations for team members, the available documents have been divided into two sets. One set represents required reading for all team members in order to be able to understand the full strategy for each project as well as the range of activities being undertaken. The second set of documents should be reviewed by individual team members when possible. Required Reading. The following documents are required reading for all team members and must have been reviewed before the Core Members assemble in-country on April 10. ● Approved Project Technical Narratives ● Theories of Change and Logical Frameworks (from the Monitoring and Evaluation Plans) ● Indicator Performance Tracking Tables (from the Performance Monitoring Plans unless there has been a subsequent revision) ● Annual Results Reports for 2015 and 2016 ● All Quarterly Monitoring Reports ● Baseline Study Report Other Reference Documents. The following documents are also available and should be reviewed cxvi by team members when possible. ● Performance Monitoring Plans ● Gender Analysis Reports ● Barrier Analysis Reports ● Value Chain Analysis ● Site Activity Records and Training Summaries ● Project Implementation Manuals (or training curricula and training materials in lieu of a PIM) ● Participant Registration Data ● Environmental Mitigation and Monitoring Plan ● Data Quality Assessments Obtaining Documents. A directory has been established on the USG Google Platform for access by USG members of the JMTR https://drive.google.com/open?id=0B￾z6z4EtEFroWHl0UC1YM3BEbWs . For others who do not have access to the USG platform, the documents are available on the DropBox https://www.dropbox.com/sh/7rm2ugijfydxm66/AAA0C-2rPvgtQsxoAWKo6Oeza?dl=0. 3. Data Collection Tools. Section VII below provides guidance on the data collection tools, primarily topical outlines, that will be needed for the JMTR. Core Team Members with support from Observers will be responsible for developing the initial draft of the data collection tools. The deadline for completing these is March 28 so that team members will have time to review background documents and this protocol. These drafts will be sent to the JMTR team leader who will review them, provide feedback and then incorporate working versions of the tools in this protocol in Annex B. 4. Site Selection. Section VII provides guidance on how sites will be selected for field visits for the JMTR. During this preparation phase, CRS and ADRA should assemble the lists of sites that have been requested in Section VII. The JMTR Team Leader will arrive in-country by April 8 and will meet with ASOTRY and Fararano management teams shortly after arriving to finalize the preliminary selection of sites by April 11. 5. Orientation Meetings. Four meetings are planned for April as described below. Initial JMTR Team Meeting, April 10. The initial meeting of the JMTR Team will occur immediately after core team members have assembled in Antananarivo. In this meeting, team members will be introduced to each other, and the JMTR team leader will provide an overview of the evaluation process and answer any questions that team members may have. Orientation Meeting with USAID/FFP Madagascar, April 10. Following the initial team meeting, an orientation meeting will be held with representatives of USAID/FFP Madagascar at which the JMTR Team will be introduced, an overview of the review process will be provided by the team leader and the JMTR team will answer any questions that USAID/FFP may have about the review. In addition, the JMTR team will seek to understand the expectations of USAID/FFP and any specific interests that they would like to see covered by the review. cxvii Project Orientation Meetings, April 11 (Fararano) & April 12 (ASOTRY). In these half-day meetings, project management staff will present an orientation to the project for the JMTR Team. The key content of these presentations should include an overview of the strategy of the project, an overview of the resources (money, commodities and staff) available for the project, a map showing geographic locations, a description of the outputs and activities under each component, a description of the SBC/ service delivery mechanism, a description of the roles and responsibilities of partners, and a description of the key challenges affecting the project. The purpose of these meetings is to obtain clarity on the types of outputs produced by the project, the stakeholders that need to be interviewed to understand the impact of these outputs, and additional data sources for information to support the review. Discussions will also be held around the site selection for the qualitative interviews to determine what the sites represent in terms of outputs and quality. Attendees to this meeting include the project management and technical team, selected representatives from implementing partners, and anyone else from the project likely to be involved in implementing or supporting the JMTR. After the overview presentation, the JMTR technical specialists will meet with the technical specialists form the projects who will make a detailed presentation on processes, outputs, indications of outcomes, challenges, any recommendations that s/he may have. Meeting with the interpreters, April 13. The JMTR team will meet with the interpreters and will orient them about the key questions and the field work processes. B. Phases 2 & 3: Reviews of ASOTRY and Fararano On April 13, the JMTR Team will travel to Amoron’i Mania region. From April 14 through 22, the team will review the ASOTRY Project and from April 27 through May 10, the team will review the Fararano Project. A Debriefing meeting with the USAID will be held on May 11. The reviews of the two projects will follow the same process as described in the next section. Descriptions of the two debriefings to be held before the team departs the country are described at the end of this section. 1. Project Review Process for ASOTRY and Fararano. Over the 15 days with ASOTRY and the 14 days with Fararano, the JMTR Team will undertake the following process. Meetings with Project Stakeholders in Antananarivo, April 10 through 12. On these days, JMTR Team members will meet with the key stakeholders that are based in Antananarivo. The primary purpose of these meetings to explain the objectives and the processes of the midterm review. The team may also conduct interviews depending on the availability of the staff. Given the limited time, the JMTR will need to be strategic on the use of this time and will consult with the project management teams for each project to determine with whom meetings should be arranged and for which members of the JMTR Team. Field Data Collection, April 15-22 (ASOTRY) & April 28- May 6 (Fararano). Over this period, the JMTR team will conduct interviews, hold group discussions; observe SBC, care group, and farmer field school sessions as they take place in the field; and observe project activities to obtain qualitative data. The JMTR will plan to spend two days in each region covered by the two projects. The specific data collection process in each region is described in cxviii section VII.C below. A detailed itinerary for the field work will be developed in collaboration with project management staff as described above. Information Processing & Preparation for the Ground Truthing Workshop, April 24-26 (ASOTRY) & May 8-10 (Fararano). During the field work, JMTR team members will process data as it is obtained, and in this period the team will continue analyzing information to identify major findings and recommendations that will be presented for discussion at the Ground Truthing Workshop. Each JMTR team member will use this time to prepare the Powerpoint presentation(s) that she/he plans to present in the Ground Truthing Workshop. Ground Truthing Workshop, April 25 (ASOTRY) & May 9 (Fararano). During and following the field work, the evaluation team will start formulating preliminary observations on outputs produced, the outcome being achieved, the processes being observed, interventions being implemented to promote sustainability (resources, capacities, motivation, and linkages), and lessons learned captured. Given the time available for this workshop, however, only the major observations and recommendations resulting from the JMTR investigations will be shared in the Ground Truthing Workshop with project implementation staff. These will be discussed to ensure that they reflect reality and are described appropriately. Two major outputs are targeted for the workshop. These are (1) agreement on the validity of key observations assembled so far from the review and (2) preliminary agreement on major recommendations for the remaining life of each project. Participants in each workshop will be project implementation staff and the JMTR team. This is an in-house event, only for those participants who are fully engaged in the project. Post-Workshop Processing, April 24 (ASOTRY) & May 8 (Fararano). This one day period is required to ensure that results of the discussions in the Ground Truthing Workshop are captured before the JMTR team moves on from ASOTRY to Fararano and from Fararano to stakeholder debriefings. JMTR Team members who would present in the Ground Truthing Workshop will revise their presentations to incorporate changes as a result of discussions. Each Core Team Member will also prepare a condensed version of her/his presentation that can be consolidated with other team member presentations for the stakeholder and USAID/FFP debriefings. 2. Pre-Exit Debriefings. A debriefing will be conducted by the JMTR team before they leave Madagascar, as described below. USAID/FFP Debriefing, May 10. The JMTR team will debrief the review findings to USAID/FFP Madagascar. Participants in this meeting will be the representatives from USAID. The JMTR team leader will facilitate the presentation, and all JMTR core team members, if they are still in-country, will attend. C. Phase 4: Recommendation Finalization and Processing After departing the country, the JMTR will work remotely to produce a report documenting the analysis of the findings and finalizing the recommendations by June 5. The JMTR team will not produce a report rather it will formulate recommendations, supporting observations and cxix the analysis of the findings. Following this workshop, the analysis of the findings and the recommendations which will be incorporated in the PREP for IY 4 for each project. JMTR Recommendation Processing and Planning Workshop, June 12-14. Over this three day period, the recommendations from the JMTR will be reviewed and finalized. Since the project management staff will have seen most, if not all, of the recommendations, they will have already begun strategizing on how to respond to the recommendations. The JMTR team leader and other members of the JMTR team who can be available for the workshop will be present to facilitate the processing and planning. Should any of the projects find any of the recommendations un-implementable, the project will be asked to develop alternative recommendations that can be implemented during the project life. By the end of the workshop, each project will have developed provisional implementation plans to be incorporated in the coming PREP. Debriefing at FFP HQ including partners, TBD. The JMTR final recommendations and preliminary plans for operationalizing recommendations developed in the workshop will be presented to HQ-based stakeholders. VII. QUALITATIVE DATA COLLECTION Qualitative data will be collected primarily through interviews and group discussions with representatives of project implementation staff, project participants, non-participants, and project partners including consortium, technical and GoM partners. The JMTR team members will also observe SBCC sessions, care group meetings, and farmer field school sessions as they are being implemented. Based on the review objectives and review questions, each team member will develop key questions for each topical areas. These topical outlines or a set of key questions or tools will be used to guide interviews and group discussions. Annex B (to be completed by JMTR team members by April 13) contains the working draft tools that will be used to start the field work. Ideally, the JMTR will gather information from all of the partners and participants listed in Section III above describing the two projects. JMTR Team members will provide project management staff with a list of the meetings, interviews and Focus Group Discussions (FGDs) that they would like to have arranged, and the JMTR team will work with project management staff to develop schedules for meetings. A. Composition of Interviews and Group Discussions Key informant interviews are normally held with between 1 to 4 persons, ideally no more than two, and these interviews typically last no more than one hour. Using semi-structured tool, in￾depth interviews are normally held with individuals to gain an in-depth knowledge about a topic of interest. FGDs are organized with from five to no more than ten persons. They are called focus groups because all of the members of the group have a common feature, so FGD facilitators must ensure that participants meet the desired common criterion, i.e., members of a VSL or SILC group, farmers who received seed from the project, lead mothers, lead fathers, etc. FGDs should normally not extend beyond two hours. In both cases, interviews and discussions should be held in secluded locations so that bystanders or passersby cannot influence the discussions. cxx B. Topical Outlines or Checklists JMTR Team members will develop topical outlines or checklist, which include key thematic questions to be used in key informant interviews or focus group discussions. These tools will be drafted by the individual Core Team Member assigned to specific investigations and tailored to the types of participants and approaches being used by each project. The questions in a topical outline are fairly general and used to stimulate discussion. Reviewers will keep in mind at all times that the purpose of the information gathering is to understand what the project has done, what changes have occurred as a result, what has helped or hindered achievement of these changes, and how likely are the changes to be sustained after the project ends. The discussion facilitator will be free to explore in more depth any interesting topics that may come up during each discussion, related to these objectives. A suggested sequencing of questions in a topical outline, drawn from a Program Constraints Assessment (PCA) approach, would be as follows. 1. Are you familiar with the ASOTRY or Fararano Project? (Describe the project, if the name is not familiar.) 2. How would you describe what this project seeks to accomplish? 3. How have you or other members of your household participated in this project? For how long have you/they been involved? 4. Please describe how you or your family has benefitted from the project. 5. Please describe how you or your family has been negatively affected by the project. 6. Who, in your opinion, has benefitted most from the project? 7. Are there other people who should be benefitting from the project but are not? Please describe them for us. 8. What constraints do you believe inhibit the project from fully accomplishing its purposes? 9. What suggestions do you have for addressing these constraints or otherwise enabling the project to have greater impact? For each question, the interviewer/facilitator should have an idea on what kind of response to expect, based on a review of the background documents, but should avoid leading the respondent to make these responses. After a respondent has completed answering a question and an expected topic has not come up, the interviewer/facilitator can then ask…what about this?…. noting that the respondent did not spontaneously report on the topic. Before beginning an interview or discussion, an introduction and explanation of the purpose of the review will be provided with stress put on the importance of obtaining useful information that reflects reality. The evaluation team will seek consent from the participants, and no names will be recorded in interviews and group discussions. If portable recording devices are used, the device should be shown to respondents and not activated until after respondents have introduced themselves. cxxi C. Observation of Project Activities In addition to conducting interviews and group discussions, JMTR team members will observe implementation of project activities and physical sites where project investments have been made. These should be regularly scheduled project activities, not activities organized only for the JMTR. Section III, describing each of the projects, lists the different types of sites that the members of the JMTR team should visit. D. Region-Level Data Collection A two day process is envisioned for data collection in each region in which the ASOTRY and Fararano Projects are being implemented, as described below. Field Visits to Communities and Sites. In each region, the JMTR team will conduct interviews and group discussions with representatives of project participants, including both targeted beneficiaries and intermediaries. Over the two-day schedule, the full team will conduct information gathering in four villages. In addition to conducting interviews and group discussions, JMTR team members will also observe, where possible, implementation of project activities and physical sites where project investments have been made. Section III, describing each of the projects, lists the different sites and activities that should be targeted for site visits. Team Analysis and Sharing of Information Between JMTR Members. In order to be able to identify synergies and interaction between different interventions in each project and to share information that is relevant for investigations being conducted by other JMTR team, JMTR team members every evening. The purpose of these meetings is to share insights and observations. Translation. It is expected that some meetings with local staff and project partners will be carried out in English. Many meetings especially at the community-level, however, will require translation assistance. This assistance in translation will be arranged by CRS and ADRA and logistics plans for travel and lodging should include interpreters. Documentation. Team member notes will be taken by individual team members during the data collection process. Wherever possible, same sex interviewer and participant will be utilized. Team members will also use photography as a data collection tool. E. Sampling and Participant Selection cxxii Project Interest in Selecting Sites Organizers of mid-term reviews are often reluctant to rely too much on projects to select the sites for field visits in an evaluation. The perception is that project implementers will want the reviewers to produce the best view possible about the project. This, however, is not in the best interests of development, the project or organizations implementing projects. Just as all projects are achieving at least some good impact, all projects also have implementation challenges and problems. Mid-term reviews are opportunities to fix these problems and address the challenges. Moreover, if they do not get addressed by the time of the final evaluation, final evaluators will find them and criticize the project for not addressing them. The project that has used the Mid-term Review to really improve the effectiveness and efficiency, will ultimately have greater impact and be viewed more favorably in the final assessment. Since this is a Mid-Term Review, the sample of participants and sites for field visits will not be chosen randomly. These will be selected strategically, so that the review team can observe what is working and what is not working in each project, as well as any particularly innovative approaches. The recommendations to be generated by the JMTR will propose scaling up interventions that are contributing to achieve the project’s goal, modifying interventions to have greater impact, suspending interventions that do not contribute enough to achieve higher level outcomes relative to investment, piloting new interventions relevant for targeted impact groups, improving the effectiveness of implementation systems, or improving efficiency in use of resources. Reviewers need to see the problems in order to be able to propose recommendations to fix them. The JMTR will collect data from each of the regions being covered by each project. For each district, ASOTRY and Fararano will classify villages according to the following criteria. 1. Participated in the previous projects (SALOHI and / or FELANA) 2. Accessibility in terms of proximity to major infrastructure (markets, main roads, basic services) 3. Demonstrating good impact (e.g., early adopters) or demonstrating poor impact - This criterion is difficult to assess at this stage in the project life. However, front-line staff usually has a pretty good idea on villages which have been adopting the project-promoted messages more readily than other villages. The objective of the JMTR is to see examples of both. 4. Intensity of intervention, i.e., activities under only one purpose being implemented, activities for two purposes, or activities for all three purposes Once the lists have been compiled, the JMTR team will purposively select sites in collaboration with project management to ensure clarity on classification, representative balance on types of sites, a wide representation on interventions, and logistical considerations. For interviews and group discussions in each site to be visited, project staff, volunteers and village leaders will identify individuals for interviews and discussions to ensure that households selected represent the economic and geographic diversity of a village. F. Data Analysis The information gathered by the JMTR team will be analyzed at multiple points during the review process. As the reviewers are holding interviews and group discussions, they will probe and explore topics in more depth with respondents to ensure clear understanding. This represents the first level of analysis. A second level of analyses occurs at the JMTR team meeting scheduled for every evening, cxxiii through the sharing of notes on observations, interpreting the observations, cross checking with other team members and immediately before the Ground Truthing Workshop when the evaluation team is processing information individually and sharing information with other team members. The advantage of qualitative methods is to have the ability to conduct real time analysis of the information. As individual team members encounter information from other team members that they may not have had a chance to discuss with respondents, they will have the opportunity to do so when they go again to conduct interviews/discussions. A third level of analysis occurs when reviewers cross-reference responses from interviews and discussions with existing data bases. In addition, the evaluator will query project implementation staff as a third source of information. The result of this level of analysis is that each evaluator will formulate his/her own key observations relative to the assigned topics. A fourth level of analysis occurs in the Ground Truthing Workshop in which preliminary observations and findings are presented to project implementation staff. If project implementation staff seriously question a particular finding or the interpretation of an observation that has been presented by the evaluation team, the opportunity exists in the workshop to further discuss and analyze the finding to reach the truth. Finally, a fifth level of analysis will occur after the JMTR team disperse and begin assembling the content that will go into the draft report. Team members will have the opportunity to look at the information that has been gathered in more depth as they are writing. VIII. IMPLEMENTATION SCHEDULES The complete schedule for implementing the JMTR is provided in Annex A. The scheduled for site visits with ASOTRY and Fararano team will also be included in Annex A, but will be added to the protocol in late March after the JMTR Team Leader has worked with project management teams to develop preliminary plans that can be reviewed by core team members. ADRA and CRS RESPONSIBILITIES CRS and ADRA will ensure that the necessary information has been passed to the GoM to allow the team to implement the review in the communities that will be selected. ASOTRY and Fararano project staff (including ADRA, CRS and partner representatives) will serve as informants to the JMTR and support the review process by supplying lists of project sites, sharing project documents, advising about local protocols, making orientation presentations to the JMTR, arranging meetings with the stakeholders, and making logistical arrangements. In addition, ADRA and CRS will ensure that their project partners are informed of the process and that implementing partners will need to make staff available as possible for interviews and other consultations with the JMTR team. ADRA and CRS will also provide professional interpreters (as required) for team members and facilitate in-country travel and logistical arrangements for the JMTR team members including vehicles, printing/copying and access to data bases. Members from USAID will pay for their own accommodation and food. They will need assistance, however, with hotel bookings. JMTR SUMMARY REPORT cxxiv There will be one report for the two projects. The JMTR Summary Report, not to exceed 50 pages excluding annexes, will seek to meet the following criteria: ● Represent a thoughtful and well-organized effort to objectively review what has been working well and should be continued as is, what did not and why and how to modify? ● Address all review questions included in the protocol; ● Include the protocol as an annex; ● Explain the evaluation methodology in detail. All tools used in conducting the evaluation, such as questionnaires, and checklists will be included in an Annex in the final report; ● Disclose limitations to the evaluation, with attention to the limitations associated with the review methodology; ● Findings should be specific, concise, and supported by evidence from the analyses of the quantitative secondary data from annual monitoring and/or qualitative interviews and observations; ● Properly identify and list all sources of information in an annex; ● Include recommendations that are supported by evidence and will inform adjustments to the remaining period of the project life. An illustrative format for the report is provided in Annex C cxxv ANNEX A: JMTR Schedule # Step Responsible Person Date(s) Comments Participation of FFP/PVO Team Members Review Preparation 1 JMTR Team Meeting by Conference Call Call organized by FFP AoR January 30 (Monday) The AoR will introduce the team and the JMTR Team leader will facilitate the meeting to discuss the review process and development of the review protocol JMTR members and CoPs will participate in call 2 Provision of initial background documentation required to develop the protocol CRS & ADRA January 20 (Friday) Required background documents at this stage are the Approved Technical Narratives, Current Results Frameworks, Annual Results Reports and Project Staffing Structures CoPs will upload the documents into a dropbox 3 Completion of first draft of Review Protocol JMTR Team Leader January 31 (Tuesday) The draft protocol will include the description of methodology and operational plan for the review. Team leader 4 Provision of feedback on draft Review Protocol FFP, ADRA & CRS February 7 (Tuesday) Written feedback on the draft protocol sent to JMTR Team Leader Review and comment on draft cxxvi 5 Completion of Working Draft Review Protocol JMTR Team Leader February 14 (Tuesday) Draft protocol revised, incorporating feedback received Team leader 6 Second JMTR team meeting by Conference Call Call organized by FFP AoR February 23 (Thursday) The FFP AoR will facilitate a conference call with the JMTR Team, ADRA, CRS and Mission to discuss the review protocol and preparations that need to be undertaken Participate in call 7 Provision of remaining background documents (via DropBox) ADRA, CRS, & FFP March 2 (Thursday) These include the full Performance Monitoring Plan, Baseline Report, Gender Analysis, Reports on Other Formative Analyses, Quarterly Performance Reports, a list of available project databases, Environmental Mitigation and Monitoring Plan, Consultancy Reports, and Trip Reports. ASOTRY and Fararano CoPs, and PVO Country backstops 8 Categorization of participating communities by criteria specified in the Draft Review Protocol ADRA and CRS February 28 (Tuesday) Community lists sent to the JMTR team leader/ members CoPs of ASOTRY and Fararano 9 JMTR members develop Draft Data Collection Tools JMTR Team Members March 15 (Wednesday) Draft tools sent to the JMTR team leader to be reviewed and incorporated in the Review Protocol Review documentation and develop draft tools for technical area cxxvii 10 Preliminary selection of field work sites JMTR Team Leader April 3 (Monday) Field work sites will be selected by JMTR Team Leader as per the Review Protocol Team leader with the team members and ASOTRY and Farararno 11 JMTR Team Arrives in Madagascar JMTR Team April 8/9 (Saturday/Sunday ) Logistical arrangements by ADRA/CRS JMTR Working Group. Arrive in-country 12 In-briefing meeting with the Mission, ADRA and CRS Country Office Management JMTR Team April 10 (Monday) To clarify review process and begin obtaining information. Meeting attended by JMTR team Full participation ADRA – ASOTRY 13 Project Orientation ASOTRY ASOTRY CoP and Key Staff April 11 (Tuesday) Project orientation for JMTR team to further clarify project activities, stakeholders, best practices, and challenges. Individual interviews by JMTR team members with ASOTRY implementation staff, with Implementing Partners, Technical Partners, Government Stakeholders, Private Sector Stakeholders and others based in Tana JMTR team members will participate 14 Meeting with the interpreters, finalizing the JMTR Team with the April 12 (Wednesday) To finalize the field work schedule, orient the interpreters, finalized the ASOTRY team will facilitate cxxviii tools and finalizing the field schedule ASOTRY staff tools and continue interviews by JMTR team members with ASOTRY Implementing Partners, Technical Partners, Government Stakeholders, Private Sector Stakeholders and others based in Tana. 15 Travel to Amoron’i Mania region in morning and field work in afternoon JMTR Team Members April 13 (Thursday) Logistics by ADRA. Travel to ASOTRY Areas Hotel USAID Hotel (L’Artisan 0340464353) (Electricity normally stable-few cuts, Wifi available) 16 Field work JMTR Team Members April 14 (Friday) Logistics by ADRA. 17 Field work JMTR Team Members April 15 (Saturday) Logistics by ADRA 18 JMTR team travels to Haute Matsiatra region JMTR Team Members April 16 (Sunday) Logistics by ADRA Hotel Zomatel (USAID cxxix Hotel?) 0340725527 (Electricity normally stable-few cuts, Wifi available) 19 Easter Monday (rest day) JMTR Team Members April 17 (Monday) Logistics by ADRA 20 Field work in Haute Matsiatra JMTR Team Members April 18-19 (Tuesday and Wednesday) Logistics by ADRA Some communes may not be possible to include in the MTR due to time to travel. These include Ambinanindovoka and Mahazony in Ambalavao, and possibly Soavina, Ambondromisotra and Vohitrafeno north of FIanarantsoa. 21 JMTR team travels to Atsimo Andrefana region region (Betioky) JMTR Team Members April 20 (Thursday) Logistics by ADRA Hotels Hotel Tsaramandroso 0330941479, Hotel Mahasoa 0331216037, Hotel cxxx Odette (Electricity available only a few hours a day, no Wifi) 22 Field work JMTR Team Members April 21 -22 (Friday and Saturday) Logistics by ADRA Some communes may not be possible to include due to time to travel. These include Marosavoa, Lazarivo and Soaserana in Betioky, and Belafika Haut, Vohitany, Gogogogo, Ankilimivory in Ampanihy. To all other locations, travel time is at least 2 to 3 hours to get there and the same to come back. In this case the only option to increase coverage, is to add one day to the time in the region, and to have 1/3 of the team sleep in the town of cxxxi Ejeda, which would give then easier access to Belafika Haut, Gogogogo, and Vohitany. Hotel in Ejeda Guest house of Lutherian Hospital 033 08 987 32 (Electricity a few hours a night, No Wifi, maximum 8-10 people) 23 JMTR team travels to Tulear in the morning JMTR Team Members April 23 (Sunday) Logistics by ADRA Hotel USAID Hotel (Hippocampe) (Alternative if # of rooms limited: Amazone 0337921965) (Electricity normally stable-few cuts, Wifi available) cxxxii 24 Information processing JMTR Team Members April 24 (Monday) Logistics by ADRA 25 Verification (Ground Truthing) Workshop JMTR Team Members April 25 (Tuesday) Presentation of preliminary findings to ASOTRY implementers for verification and refinement 26 Post-Workshop Processing JMTR Team Members April 26 (Wednesday) Documenting the key points and analysis JMTR team members CRS – Fararano 27 Project Orientation Fararano Fararano CoP and Key Staff April 27 (Thurs) Project orientation for review team to further clarify project activities, stakeholders, best practices, and challenges. Individual interviews by JMTR team members with Fararano implementation staff, with Implementing Partners, Technical Partners, Government Stakeholders, Private Sector Stakeholders and others based in Tana Full participation 28 Meeting with the interpreters, finalizing the tools and finalizing the field schedule JMTR Team with the Fararano staff April 28 (Friday) To finalize the field work schedule, orient the interpreters, finalized the tools and continue interviews by JMTR team members with Fararano and ASOTRY Implementing Partners, Technical Partners, Government Fararano team will facilitate cxxxiii Stakeholders, Private Sector Stakeholders and others based in Tana. 29 In-Briefing with CRS team in Atsimo Andrefana JMTR Team Members April 29 (Saturday) Logistics by CRS Hotel Toliara: USAID Hotel (Hippocampe), Wifi and electricity 30 Field work JMTR Team Members April 30 (Sunday, Day Off), May 1- 2 (Mon-Tues) Logistics by CRS Hotels Toliara: USAID Hotel, Wifi and electricity TEAM 1 (Vatovavy Fitovinany) 31 Travel by car to Ranomafana JMTR Team Members May 3 (Wed) Logistics by CRS Hotel: Ranomafana: USAID Hotel (Setam or other), electricity and no wifi at hotel but phone service network does work 32 Field work continues JMTR Team May 4-5 Logistics by CRS Hotels: cxxxiv Members (Thurs-Fri) Ranomafana: USAID Hotel (Setam or other), electricity and no wifi at hotel but phone service network does work 33 Travel to Tana JMTR Team Members May 6 (Saturday) Logistics by CRS Travel TEAM 2 (Atsinanana) 34 Travel by Plane to Tana JMTR Members May 3 (Wed) Logistics by CRS Travel 35 Travel by Car to Toamasina JMTR Members May 4 (Thurs) Logistics by CRS Hotel Toamasina: USAID Hotel (Calypso), electricity and wifi 36 Field Work JMTR Members May 5-6 (Friday-Sat) Logistics by CRS Hotel Toamasina: USAID Hotel (Calypso), electricity and wifi 37 Travel to Tana by car or plane JMTR Members May 7 (Saturday) Logistics by CRS Travel cxxxv TEAMS BACK TOGETHER in TANA 38 Information Processing & Preparation for Verification Workshop JMTR Team Members May 8-9 (Mon-Tues) Information processing, and preparation for Verification Workshop Full participation 39 Verification (Ground￾Truthing) Workshop JMTR Team Members with Fararano Implementing Staff May 10 (Wednesday) Presentation of preliminary observations to Fararano implementers for verification and refinement. Full participation 40 Interview/meeting with Fararano Stakeholders JMTR team May 11 (Thursday) Interview/ meeting with selected Government stakeholders JMTR Members 41 Out-briefing meeting with USAID Madagascar JMTR Team May 12 (Friday) Presentation of preliminary findings and discuss next steps Full participation 42 JMTR team leaves Madagascar JMTR Team May 12 (Saturday) Recommendations Finalization and Processing 43 Individual team members will analyze findings and draft notes and develop recommendations by JMTR Team Members May 15 through June 5 JMTR team members continue analysis and draft their analysis to support recommendations Continue analysis remotely and draft content for report cxxxvi sector areas 45 In-Country JMTR Recommendation and Planning Workshop FFP AOR June 12 - 14 (Monday thru Wednesday) Based on the analysis, the JMTR team will present the recommendations. ASOTRY and Fararano will have an opportunity to develop alternative recommendations in case if any recommendation deemed un￾implementable. The workshop will help to agree on a set of recommendations. AoR and PVO Country Backstop participation is expected. 46 The Review Notes – Analysis linked to recommendations finalized JMTR Team Leader with the help of JMTR team members June 22 Based on revisions made in the workshop to recommendations, the recommendations report finalized JMTR Team members 47 De-briefing to DC-based Stakeholders FFP AOR and JMTR Team Leader June 19 - 20 Review final recommendations and preliminary plans for operationalizing recommendations presented to HQ￾based personnel --- 47 Action Plans drafted and submitted to FFP ADRA & CRS July 20 Draft Action Plans for operationalizing review recommendations submitted to FFP FFP AoR 48 Review/revise Draft Action Plan with ADRA FFP & ADRA July 27 Meeting in DC with Madagascar Staff joining remotely to finalize action plan FFP AoR 49 Review/revise Draft FFP & CRS July 28 Meeting in DC with Madagascar Staff FFP AoR cxxxvii Action Plan with CRS joining remotely to finalize action plan 50 Action Plans Finalized ADRA and CRS August 15 Final Action Plans, incorporating feedback from FFP, submitted to FFP FFP AoR cxxxviii ANNEX B: DATA COLLECTION TOOLS cxxxix ANNEX C: Madagascar Joint Mid-Term Review (JMTR) ILLUSTRATIVE JMTR SUMMARY REPORT FORMAT (The length of this report should not exceed 50 pages, excluding Annexes. Preliminary target page limits for each section are shown in parentheses.) I. EXECUTIVE SUMMARY (2 Pages) II. BACKGROUND (4 Pages) A. Overview of the Joint Mid-Term Review Purpose & Objectives B. JMTR Methodology III. REVIEW OF THE ASOTRY PROJECT (20 Pages) A. Overview of the Project (including project history, targeted impact groups, theory of change, logical framework, geographic coverage, and resources) B. Key Outputs Under Each Purpose (including an overview of the implementation process including quantity and quality of production, description of participants and beneficiaries, key observations, lessons learned and specific recommendations) C. Project Outcomes and Impact Under Each Purpose (including an assessment of the immediate life-of-project impact, likelihood of this impact being sustained after the project ends, lessons learned and recommendations for enhancing sustained impact.) D. Cross-Cutting Themes (Including a summary of specific actions taken by the project to operationalize the cross-cutting themes, an assessment of how these have contributed to achieving project impact, lessons learned, and recommendations for the remaining life of the project.) E. Overall Program Design (Including an analysis of the current relevance of the targeting systems, theory of change and logical framework with recommendations for the remaining life of the project) F. Project Implementation Systems (Sections on project management, partnership, knowledge management, and project integration, major observations from the review, lessons learned and recommendations for the remaining life of the project.) IV. REVIEW OF THE FARARANO PROJECT (20 Pages) A. Overview of the Project (including project history, targeted impact groups, theory of change, logical framework, geographic coverage, and resources) cxl B. Key Outputs Under Each Purpose (including an overview of the implementation process including quantity of production, quality of production, description of participants and beneficiaries, key observations, lessons learned and specific recommendations) C. Project Outcomes and Impact Under Each Purpose (including an assessment of the immediate life-of-project impact, likelihood of this impact being sustained after the project ends, lessons learned and recommendations for enhancing sustained impact.) D. Cross-Cutting Themes (Including a summary of specific actions taken by the project to operationalize the cross-cutting themes, an assessment of how these have contributed to achieving project impact, lessons learned, and recommendations for the remaining life of the project.) E. Overall Program Design (Including an analysis of the current relevance of the targeting systems, theory of change and logical framework with recommendations for the remaining life of the project) F. Project Implementation Systems (Sections on project management, partnership, knowledge management, and project integration, major observations from the review, lessons learned and recommendations for the remaining life of the project.) V. SPECIFIC TOPICS (3 Pages) A. B. C. D. V. CONCLUDING REMARKS (1 Page) cxli Annex 2: Additional Case Studies Case study 5: The landless Nafarriko Tsianisiha commune, Tsiafanoka fokontany Nafarriko, 75, is an elderly woman who has lived her entire life in the same village in the Tsianisiha commune in southwest Madagascar. She is not a direct beneficiary of the Fararano project but lives among those who are. By her own definition, she is among the poorest in the community that should – but does not – receive any help from the project since she does not have any young children and does not own any land. Nafarriko has no memory of her father; she remembers her childhood with her single mother and six siblings as marked by constant hunger. Her mother fed them maize when she could but it was never enough for the hunger to go away. Nafarriko was married at age 12 and had her first child at age 15. She now has 16 children; her youngest is 25. Her children have either married and live in separate households or moved away. However, five of her grandchildren remain in her household along with her husband and an elderly uncle she cares for. Her grandchildren are young but they are all over two and do not qualify for receiving rations. Nafarriko and her husband do not own any land or any other assets. They work the field by their home and as payment, are able to take half the cassava they farm. The other half goes to the owner of the land. She heard about the opportunity to participate in the Food for Assets reforestation activity but was not there when people were recruited or selected for the activity. She asked if she could participate for the rice ration but was told by whom she perceived to be in charge that she can participate but she will not receive any rations for her work. The only other interaction Nafarriko has had with Fararano activities or staff was when the community health volunteer told them they had to build a latrine, which they did. Nafarriko and her household use the latrine but still do not treat their water. Nafarriko insisted repeatedly that the most beneficial thing for them would be more Food for Asset opportunities to be able to work for rice. She says cassava is the only thing she and her household ever eat and their lives would be better off with more variety in their diet. Nafarriko with her husband and grandson cxlii Case study 6: Female head of household Baolineza Anushiparie commune, Maromanitra fokontany Baonileza, 49, is a female head of household in Maromanitra, a community in the Anushiparie commune. She is a farmer and owns land, but not enough to comfortably provide for her family. She was born and raised in this village along with her six siblings. Her parents were farmers and ensured Baonileza and all her siblings went to school. Baonileza has fond memories of her childhood and remembers food being plentiful. She does not remember how old she was when she was married, but thinks she was around 16. She moved to the nearby fokontany where her husband was from and had six children. When she was about 30, she gave birth to twins shortly before her husband passed away. She then moved back to Maromanitra with her children and starting farming on father’s land. Her mother had passed away while she was living in the different village with her husband, and her father died shortly after her return to her home village; she received her father’s land as an inheritance. The village Baonileza left was not the same one to which she returned. Due to the series of cyclones, locusts and droughts, the plentiful food she remembered was replaced by chronic food insecurity. She does not sell any of the crops she produces on her small plot because she barely has enough to feed her family. Many times, the only thing they have to eat is tavolo, which is Malagasy for arrowroot. Baonileza explains that tavolo powder is very bitter and a lot of work is required to make it edible. She used to own cows but they all died, along with the rest of the cows in the village, from an illness that quickly overtook them. Baonileza has participated in some Fararano activities and would like more opportunities for her to be more active. She was involved with reforestation, a Food for Assets activity, and enjoyed both the work and the rice she received. She is also part of the Disaster Risk Reduction committee for the fokontany. Her eldest daughter is a leader mother and helps the neighborhood women learn information that can benefit their health and their families. Baonileza also participates in a Savings and Internal Lending Community group but her involvement is sporadic since she does not always have the required cash to be included. Baonileza has received agriculture training from a different project but she does not remember the name of it. She knows there are Fararano lead farmers in this community but she does not know who they are or how to benefit from their knowledge. In Baonileza’s opinion, the biggest causes of food insecurity in her community are the seasons, alternating between droughts and excessive rain, and the lack of infrastructure. Baonileza also described the lack of health care providers and facilities. The nearest healthcare facility is a half-hour walk to the commune center and often the doctor is not there. There is a community health volunteer in the village but she mainly caters to children under five, leaving limited options for older children and adults. Baonileza would like to see the road fixed since it is the only way in and out of the village. She would also like more access to clean water. She currently gets water from the river, which is close by, but very dirty. She uses Sur Eau, a chemical water treatment, as much as she can, but it is expensive and often she cannot afford it. Additionally, Baonileza thinks more infrastructure projects would encourage camaraderie amongst the villagers. She feels adamant about the benefits of the community working collectively to improve their quality of life. Baonileza (far left) eating porridge made from Tavolo