Process Evaluation of the Maharo Resilience Food Security Activity in Madagascar September 2022 IMPEL | Implementer-Led Evaluation & Learning Associate Award ABOUT IMPEL The Implementer-Led Evaluation & Learning Associate Award (IMPEL) works to improve the design and implementation of Bureau for Humanitarian Assistance (BHA)-funded resilience food security activities (RFSAs) through implementer-led evaluations and knowledge sharing. Funded by the USAID Bureau for Humanitarian Assistance (BHA), IMPEL will gather information and knowledge in order to measure performance of RFSAs, strengthen accountability, and improve guidance and policy. This information will help the food security community of practice and USAID to design projects and modify existing projects in ways that bolster performance, efficiency, and effectiveness. IMPEL is a seven-year activity (2019–2026) implemented by Save the Children (lead), TANGO International, Tulane University, Causal Design, Innovations for Poverty Action, and International Food Policy Research Institute. RECOMMENDED CITATION IMPEL. (2022). Process Evaluation of the Maharo Resilience Food Security Activity. Washington, DC: The Implementer-Led Evaluation & Learning Associate Award. PHOTO CREDITS Charlie Forgham-Bailey / Save the Children DISCLAIMER This report is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of the Implementer-Led Evaluation & Learning (IMPEL) award and do not necessarily reflect the views of USAID or the United States Government. CONTACT INFORMATION IMPEL Associate Award c/o Save the Children 899 North Capitol Street NE, Suite #900 Washington, DC 20002 fsnnetwork.org/IMPEL IMPEL@savechildren.org PREPARED BY: Process Evaluation of the Maharo RFSA in Madagascar Acknowledgments i ACKNOWLEDGMENTS The Causal Design research team would like to recognize the assistance of key individuals instrumental in facilitating the data collection, research, and findings central to this report, including Cara Raboanarielina and Daniel Rooney with Catholic Relief Services, and Mohamoud Ahmed, Tom Spangler, and Corena Sharp with IMPEL. Special thanks also to Mara Mordini, Justin Mupeyiwa, and Adam Trowbridge with USAID for their technical support during the inception phase. We also are grateful to Eliane Ralison and the field team and their tremendous work with data collection under difficult circumstances. IMPEL | Implementer-Led Evaluation and Learning ii Table of Contents TABLE OF CONTENTS Acknowledgments...............................................................................................i List of Tables......................................................................................................iii List of Figures.....................................................................................................iii Acronyms...........................................................................................................iv Executive Summary ...........................................................................................vi Methods ................................................................................................................................................ vi Findings.................................................................................................................................................. vi Cross-Cutting Findings................................................................................................................... vi Intervention-Specific Findings..................................................................................................... viii 1. Introduction...................................................................................................1 2. Methodology .................................................................................................2 Intervention Selection....................................................................................................................2 Primary Data...................................................................................................................................2 Secondary Data ..............................................................................................................................3 Data Analysis..................................................................................................................................4 Challenges and Limitations ............................................................................................................4 3. Findings .........................................................................................................5 Cross-Cutting, Contextual Findings................................................................................................5 3.1.1. Kere/Drought ................................................................................................................5 3.1.2. COVID-19.......................................................................................................................6 3.1.3. Gender Inclusion ...........................................................................................................8 3.1.4. Youth Inclusion............................................................................................................10 3.1.5. People with Disabilities Inclusion................................................................................11 3.1.6. Links with Local Governance.......................................................................................13 3.1.7. Community Consultations...........................................................................................15 3.1.8. Collaborating, Learning, and Adapting........................................................................18 Purpose 1: Governance and Social Services.................................................................................19 3.2.1. Miranjaka Gender Champions ....................................................................................19 Purpose 2: Nutrition, Health, Water, Sanitation, and Hygiene....................................................22 3.3.1. Care Groups.................................................................................................................22 Purpose 3: Resilience and Livelihoods.........................................................................................24 3.4.1. Disaster Risk Reduction Committees..........................................................................24 3.4.2. Lead Farmers...............................................................................................................27 3.4.3. Savings and Internal Lending Communities................................................................29 3.4.4. Private Input Service Providers...................................................................................31 Process Evaluation of the Maharo RFSA in Madagascar Table of Contents iii 4. Conclusions and Recommendations.............................................................36 Annex A: Maharo Interventions........................................................................39 Annex B: Evaluation Questions and Methodology.............................................43 Annex C: Inclusion Findings by Intervention......................................................50 Annex D: Works Cited .......................................................................................53 Annex E: Data Collection Tools..........................................................................54 LIST OF TABLES Table 1. Number of groups sampled that have people with disabilities members/participants...............11 Table 2. Findings rubric (with illustrative findings reflective of cell location)............................................36 Table 3. CRS and implementing partner respondents interviewed............................................................44 Table 4. Fokontany and commune-level respondents interviewed ...........................................................45 Table 5. Validation exercises conducted per intervention .........................................................................46 Table 6. Inclusion findings by intervention.................................................................................................50 LIST OF FIGURES Figure 1. Reported voucher prices compared to market prices for PiSP commodities..............................33 Figure 2. Selected interventions and their Purposes..................................................................................44 IMPEL | Implementer-Led Evaluation and Learning iv Acronyms ACRONYMS AT Agent Terrain/Du Terre/Field Agent ADRA Adventist Development and Relief Agency BGNRC Bureau National de Gestion des Risques et Catastrophes/National Office of Risk and Disaster Management BHA Bureau of Humanitarian Assistance CAQDAS Computer-assisted qualitative data analysis software CC Community Consultation CCGRC Comité Communal de la Gestion des Risques et Catastrophes/Commune-level Disaster and Risk Management Committee CDD Conseil Diocésiain de Développement/Diocesan Council of Development – CRS’ implementing partner in Ampanihy CG Care Group CGRC Comité Local de la Gestion des Risques et Catastrophes/Local-level Disaster and Risk Management Committee CHW Community Health Worker CLA Collaborating, Learning and Adapting CM Community Mobilizer CRS Catholic Relief Services DP Demonstration Plot DRR Disaster Risk Reduction Committee ELS Equipe Local de Secour/Local Rescue Team FLC Farmer Learning Center HI Humanity Inclusion IDEAL Implementer-Led Design, Evidence, Analysis and Learning IGA Income Generating Activity IMPEL Implementer-Led Evaluation and Learning Associate Award IP Implementing Partner KFF Komity Fampandrosoana ny Fokontany/Village Development Committee KII Key Informant Interview LF Lead Farmer LM Lead Mother MEAL Monitoring, Evaluation, Accountability and Learning MGA Malagasy Ariary PE Process Evaluation Process Evaluation of the Maharo RFSA in Madagascar Acronyms v PiSP Private input Service Provider PSP Private Service Provider PWD People with Disabilities R&I Refine and Implement RFSA Resilience Food Security Activity RMM Reny Mpiara-Monina SILC Savings and Internal Lending Community USAID United States Agency for International Development USD United States Dollar VSL Village Savings and Loans IMPEL | Implementer-Led Evaluation and Learning vi Executive Summary EXECUTIVE SUMMARY This Process Evaluation (PE) of the Maharo Resilience Food Security Activity (RFSA) in southern Madagascar, implemented by Catholic Relief Service (CRS), examines and assesses the nature, quality, and process of the RFSA’s implementation. Its core overarching evaluation question is: How well has Maharo been effectively implementing core and critical pieces of its intervention in order to address the RFSA objectives of enhanced food security and resilience? The PE will assist the United States Agency for International Development’s Bureau for Humanitarian Assistance (USAID/BHA) and CRS gauge the state of implementation, contend with identified challenges, and facilitate adaptive management to maximize effectiveness. It will also provide contextual information to inform Maharo’s endline impact evaluation. Methods This report focuses on seven key interventions spanning Maharo’s three purposes: Community Consultations (CCs), Miranjaka Gender Champions, Care Groups (CGs), Disaster Risk Reduction (DRR) Committees, Lead Farmers (LFs), Savings and Internal Lending Communities (SILCs), and Private input Service Providers (PiSPs). It relies on three core sources of primary data: 1) Key informant interviews (KIIs) with implementing partners throughout the implementation chain; 2) Validation exercises based on observation and informal interviews with participants; and 3) Ground-truthing sessions with CRS technical staff, to garner feedback, questions, and additional contextual information. In addition, a joint Collaborating, Learning, and Adapting (CLA) workshop informed the findings and recommendations of this report. Qualitative data was coded and analyzed using ATLAS.ti computer-assisted qualitative data analysis software (CAQDAS). The PE best represents a case study methodology, in which a one-time data collection was utilized to examine the implementation process and assess how intervening variables, including local context and conditions, influenced implementation to date. Findings are specific to Maharo, though some observations may be relevant to RFSA implementation across contexts. Findings Cross-Cutting Findings Kere/Drought. The severe seasonal drought or kere presented considerable challenges across interventions, including reducing time availability to participate and implement Maharo interventions. It also led community workers to reschedule and shorten meetings. The kere also increased participants’ expectations of receiving some form of incentive due to the influx of emergency actors distributing aid and impeded participants from being able to implement trainings in agriculture and dietary diversity. COVID-19. Overall, COVID-19 had a limited impact on Maharo’s implementation on the ground. However, it did impact the quality of implementation by necessitating online training instead of in person, preventing technical and activity staff from traveling to implementation zones, and affecting communities’ motivation with the suspension of elements such as community celebrations. The inability to travel and meet in person also created additional workloads for some CRS staff and impaired inter- Process Evaluation of the Maharo RFSA in Madagascar Executive Summary vii sectoral collaboration between CRS and partners. In some cases, the distribution of key materials was also affected due to global supply chain kinks and regional border closures. Gender Inclusion. Local norms and customs that discount women’s leadership and participation are considerably entrenched. Maharo has made notable inroads by working with local stakeholders, particularly local leaders, to facilitate Miranjaka awareness-raising. Additionally, Maharo has had relatively broad participation by women in other interventions. Nonetheless, some gaps in community workers’ gender sensitivity and knowledge remain, suggesting a need for more thorough training on these topics. Youth Inclusion. Youth participation varied across interventions, with higher participation in DRR Committees, CGs, and as Miranjaka. Maharo expressly targeted and trained youth for DDR Committees, and the youth participating as Miranjaka appeared to be dynamic and energetic. Youth have less representation and are less involved in PiSP, SILC, and LF interventions due to resource constraints and lower interest in agriculture among youth. People with Disabilities (PWD) Inclusion. There is a clear commitment from Maharo to PWD inclusion, but physical and cultural barriers to meaningful participation complicate this commitment. Respondents across interventions expressed mixed reactions, or personal preferences, regarding PWD inclusion. There were also mixed efforts to actively recruit participants. There is some evidence that PWD remain largely reticent to participate in Maharo interventions because of the social stigma. However, there is limited to no data indicating active advocacy for PWDs or effective recruitment mechanisms that would directly address such stigma. Local Governance. Maharo is notable for its intentional and overall effective coordination with local leaders, particularly at the fokontany level (versus the mayorie level). These efforts have borne fruit, especially for Miranjaka and CGs, for whom fokontany chiefs prove to be critical allies. Fokontany chiefs have lent credibility to such interventions, secured spaces, recruited participants and intervention leaders, and, in certain locations where it remains contentious, provided essential cover and permission for gender equality activities. There are instances of conflict, at times, generated by fokontany chiefs when they perceived they were being sidelined in local affairs, further underscoring their essential role as local stakeholders. CLA. COVID-19 caused a significant disruption in the Refine and Implement year by delaying CCs, assessments, and studies, consequently leading to delays in integrating learnings and findings into implementation. CRS has now begun incorporating findings and demonstrating a high commitment to an adaptive management approach. However, building a learning culture and truly adopting adaptive management principles and CCs remain a work in progress. Recommendations: ● Consider providing additional training on inclusion for community workers, led by gender, youth, and PWD sensitivity experts, to enhance meaningful, inclusive participation and combat persisting stereotypes. Trainings should also include practical instruction on successful inclusive implementation. IMPEL | Implementer-Led Evaluation and Learning viii Executive Summary ● Continue to anticipate possible ramifications while working intentionally outside of local governance structures. Communicate reasoning and develop mitigation plans, though continue, first, to collaborate with local authorities, including informal actors, as best as possible. Intervention-Specific Findings Community Consultations. CCs, in particular, appear to represent the best in applied systems thinking, complexity awareness, knowledge transfer, and practicality. Although, there remain significant obstacles to their influence on Maharo interventions. CCs have generated credibility and have effectively garnered “local buy-in.” However, the complexity inherent in implementing emergency and resilience activities in an established social system undergoing stress is underscored by timing complications with activity rollout, delays in follow-up (partly due to COVID-19), and perceptions of unmet expectations among participant communities. Recommendations: ● Increase emphasis on community ownership and the role of CCs as facilitating a partnership of shared responsibility, rather than purely relaying community needs to implementing partners, to prevent feelings of frustration and better manage expectations. Miranjaka Gender Champions. Miranjaka have advanced through regular training and engagement with local communities. They appear to have effectively coordinated with Village Development Committees (Komity Fampandrosoana ny Fokontany, KFF) and fokontany chiefs to establish their credibility and effectively recruit participants. Some Miranjaka have expressed fatigue as they and participants contend with the negative effects of the kere/drought on their motivation, participation, and overall time availability. Limited training materials (which were reported across interventions) also complicated implementation. Care Groups. CG data reflects consistent reporting of effective training. Overall, Lead Mother (LM) respondents could consistently recall the key lessons and training objectives. Key complications were the availability of training materials, kere/drought-related time constraints (affecting both LMs and participants), and the variability in literacy among LMs. On the one hand, varying literacy facilitated participation and recruitment but at times created inconsistencies and self-awareness of capacity. Finally, at times community members perceived both Miranjaka and LMs to be privileged in some ways by CRS and paid, resulting in resentment. Recommendations (CGs & Miranjaka): ● Consider variability in literacy among Miranjaka and LMs, and effects on training. DRR Committees. DRR Committees have been robustly implemented. Maharo has distributed phones; DRR plans are mostly either drafted or in place; and messaging (e.g., for severe weather alerts or fall armyworm) is now underway. Youth inclusion was especially apparent in DRR committees. However, some of the underlying architecture of DRR committees appears unreliable because some respondents suggested that they get their information from friends in better-connected locations and then pass along the information locally. Others cited the contract date with Telma as a looming threat to sustainability. Process Evaluation of the Maharo RFSA in Madagascar Executive Summary ix Recommendations: ● Continue to monitor DRR operability, particularly around communication and early warning systems, to ensure movement towards accuracy, timeliness, and availability of information. ● Continue to monitor the finalization and dissemination of DRR plans (working towards a date of delivery) and usability. Lead Farmers. LFs are the most directly affected by the kere/drought, which limited the participation among farmers, the applicability of training in new techniques, and the viability of demonstration plots. LF interventions nevertheless advanced, with demonstration plots, new fencing, and goat sheds apparent throughout the sampled communities. Some LFs have expressed a commitment to their tasks and role, despite the headwinds. Recommendations: ● Despite current investments, greater sensitization is required to create acceptance of climate￾smart and soil-generating crops, especially for those that do not bear fruit. (This is heightened by the kere/drought and prevailing food insecurity). ● Demonstration plots could assist in this effort but will require greater oversight, monitoring, and would have to prove effective compared with traditional techniques. ● Review whether staffing is adequate to effectively monitor the 100+ Farmer Learning Centers, the proliferation of which stems from a shift to a cluster model. SILCs. The SILCs appear to be Maharo’s most successful and sustainable intervention because they are well utilized, particularly their solidarity funds, with demonstrable positive spillover effects. Though the kere/drought negatively impacted some SILC members’ ability to make contributions, overall, the SILCs were noted to be highly appropriate and relevant in the kere/drought context. Limited market opportunities nonetheless limit SILC funds from being invested in livelihoods. Recommendations: ● Consider providing additional support to the SILCs, which are struggling to establish savings due to a lack of interest accumulation or limited member contributions due to kere/drought. PiSPs. Though the PiSP intervention has solved the problem of obtaining seeds from long distances for both farmers and PiSPs, many farmers complained that voucher commodities cost significantly more than market commodities. The price difference contributed to a sense of dissatisfaction with the vouchers and some resentment. Some voiced a preference for cash to buy the types of seeds they preferred, for prices with which they agreed. Additional challenges include a lack of start-up cash for PiSPs to buy seeds, staggered deliveries of vouchers and commodities, requiring participants to travel multiple times, and seeds arriving too late or early for planting. Recommendations: ● Increase communication efforts to clarify reasons for the different prices, the added benefits, and the quality of PiSP seeds. ● Consider providing direct cash to respondents—ample literature spanning contexts suggests this is effective. IMPEL | Implementer-Led Evaluation and Learning x Executive Summary Cross-Intervention Recommendation: • Consider what incentive options are possible to motivate community workers and participants, given that motivation and time availability appear to be the greatest challenge to implementation quality and, consequently, impact. Also, increase communications among communities about the nature of community workers’ role and volunteer nature to avoid resentment or misperceptions. Process Evaluation of the Maharo RFSA in Madagascar Introduction 1 1. INTRODUCTION This Process Evaluation (PE) of the Maharo Resilience and Food Security Activity (RFSA) in southern Madagascar, implemented by Catholic Relief Services (CRS), is carried out under the Implementer-Led Evaluation and Learning (IMPEL) Associate Award platform for the United States Agency for International Development Bureau for Humanitarian Assistance (USAID/BHA). As a PE, this report specifically examines and assesses the nature, quality, and process of Maharo implementation up until April 2022. It is designed and was conducted to answer the overarching evaluation question: How well has Maharo been effectively implementing core and critical pieces of its intervention in order to address the RFSA objectives of enhanced food security and resilience?1 This PE is intended to assist USAID/BHA and CRS gauge the state of implementation, contend with identified challenges, and facilitate adaptive management as all parties maximize effectiveness. It also provides contextual information to inform Maharo’s endline impact evaluation. More precisely, this report gauges and assesses the systematic strengths and weaknesses of the implementation strategy and processes, as well as contextual factors that directly affect implementation. The most visible and relevant contextual factors that are examined intentionally include COVID-19 and the ongoing kere or drought. The evaluation also examines Maharo’s inclusivity approach regarding gender, youth, and people with disabilities (PWD), and, more granularly, how implementation interacted with local governance structures and stakeholders, like fokontany chiefs, elders, and other formal structures such as the Village Development Committees (Komity Fampandrosoana ny Fokontany (KFF)). This report primarily relies on qualitative data from key informant interviews (KIIs) with stakeholders throughout the implementation chain. It also includes findings from a validation component to gauge the state of implementation and identify areas of special concern regarding implementation follow￾through. This report also benefits from feedback gathered through ground-truthing sessions held with CRS and a Collaborating, Learning, and Adaptation (CLA) workshop attended by Causal Design, CRS, USAID/Madagascar Mission, and BHA staff. The report concludes by providing recommendations to improve Maharo’s implementation for the future. Causal Design acknowledges that the CRS team has persevered in the implementation of Maharo despite extreme setbacks, including a severe kere/drought in most locations, the cyclone Batsirai in others, and COVID-19 restrictions that limited travel and communication. We, nevertheless, hope that our observations and recommendations are taken seriously by the implementing team and relevant technical teams to augment learning and adaptive program management, as we believe the data collection and subsequent analysis yield important insights for all parties that can inform the next steps. 1 Sub-questions are located in Annex A and are referred to specifically in the relevant report section. IMPEL | Implementer-Led Evaluation and Learning 2 Methodology 2. METHODOLOGY The Maharo PE is based on the analysis of primary qualitative data (discussed more below) and predominantly constitutes a case study methodology, with findings specific to Maharo and its current operating context. Results are neither generalizable nor address causality or outcome attribution, though some lessons learned and recommendations may indeed be relevant for other RFSAs implemented in difficult environments. Data collection was conducted at one point in time (or, more precisely, over a month) to retrospectively examine implementation to date. Data collection instruments were designed to reveal the steps involved in implementation, identify moments and factors that intervened in the process, and adjustments or notable adaptations by implementers, if applicable. The sampling of diverse respondents allowed for a validation and robustness check on findings. Though a higher frequency of responses may indicate trends, patterns, and findings, in some instances, individuals’ responses are noted if they are particularly illustrative of, or uniquely articulate, ground￾level challenges. In the same vein, while acknowledging inherent limitations, this report leverages the advantages that extensive qualitative data present. This report does not present a static set of simplistic observations, but it references the complexity confronting implementing partners (IPs) operating in a crisis environment. Additionally, it presents the challenges that arise when implementing a mostly linear activity within a complex system constituted by diverse stakeholders, with variable incentives. It also assesses language and responses indicative of assumptions, sentiments, and even grievances that might arise from the presence of an outside intervention. Intervention Selection Causal Design consulted with BHA/Washington, the USAID/Madagascar Mission, and CRS to determine a select set of priority interventions—among the universe of overlapping interventions—to prioritize for examination under this PE. A summary of each of the selected interventions is presented in Annex A. The seven interventions, selected based on their representativeness and significance in the Maharo RFSA as a whole, are: ● Purpose 1: 1) Miranjaka Gender Champions ● Purpose 2: 2) Care Groups (CGs) ● Purpose 3: 3) Disaster Risk Reduction (DRR) Committees, 4) Lead Farmers (LFs), 5) Savings and Internal Lending Community (SILCs), 6) Private input Service Providers (PiSPs) ● Cross-cutting purposes: 7) Community Consultations (CCs) Primary Data PE findings are based on qualitative inquiry. As the PE focuses on the implementation process, it necessitates examining the links throughout the implementation “chain.” Examining each part of the process was achieved through KIIs (n = 82) and a validation exercise. In addition, ground-truthing sessions and a CLA workshop were held to garner additional context, clarify, and verify findings and recommendations. Each step is detailed below, including lists of respondents interviewed and geographic sampling, presented in Annex B. Data collection tools are presented in Annex E. Process Evaluation of the Maharo RFSA in Madagascar Methodology 3 Key Informant Interviews The main source of data for this PE is KIIs with stakeholders across the implementation chain, from national-level CRS and IPs to ground-level implementers, such as community workers. Geographic sampling of community-level actors and participants covered communes and fokontany in all three regions targeted by Maharo, purposively selected to include both interior and littoral areas and both rural and urban areas. Causal Design staff conducted interviews with CRS and IPs remotely over Zoom using simultaneous translation as necessary. As Causal Design staff were unable to travel to Madagascar due to the COVID-19-related travel restrictions, data collection at the fokontany and commune level was conducted by Madagascar-based Consultants Associates2 and supervised remotely. Validation Exercise At BHA’s request, Consultants Associates also conducted a validation exercise to provide additional verification that interventions are taking place according to Maharo’s intended cascading system. While narrow in scope, the validation exercise provided additional qualitative data that built on findings from the KIIs. Ground-Truthing Sessions Stakeholder feedback on findings and recommendations was solicited through ground-truthing sessions with CRS technical staff. Four remote ground-truthing sessions, divided by purpose and intervention, were held over Zoom using simultaneous translation between May 3 and May 6, 2022. Sessions were highly fruitful and consisted of Causal Design presenting findings and initial recommendations, then garnering CRS’s feedback, questions, and additional contextual information and details. Collaborating, Learning and Adapting Workshop At CRS’s request, Causal Design facilitated a 4-day joint Collaborating, Learning, and Adapting (CLA) workshop alongside the Adventist Development and Relief Agency (ADRA)—implementing the FIOVANA RFSA—after BHA and CRS had provided an initial review of the Maharo PE report. Causal Design, CRS, ADRA, USAID/Madagascar Mission, and BHA staff attended the workshop. Additional context and refinements to recommendations that emerged from the workshop have been incorporated into this PE report. Secondary Data CRS also provided secondary data, including activity documents, quarterly and annual reports, youth and gender strategies, technical documentation about the seven interventions, and studies conducted as part of the Refine and Implement (R&I) year. Reviewing this secondary data, in combination with follow￾up clarifications with Maharo technical staff, informed tool design and the construction of a preliminary codebook for analysis by identifying key themes anticipated to emerge in primary data. 2 Consultants Associates conducted the Maharo Baseline Study and, as such, were familiar with the Maharo RFSA and context. IMPEL | Implementer-Led Evaluation and Learning 4 Methodology Data Analysis Both primary data, in the form of English language transcripts, and secondary data were coded and analyzed using ATLAS.ti computer-assisted qualitative data analysis software (CAQDAS). Primary data was coded using both deductive and inductive approaches, whereby a codebook developed based on themes identified from secondary data was applied and continuously modified to incorporate emergent themes. To avoid problems of inter-coder reliability, all KII coding was conducted by one analyst, and all validation data was coded by another. Challenges and Limitations COVID-19-related travel restrictions prevented Causal Design staff from traveling to Madagascar for data collection, directly supervising field teams, and conducting coordinator training. In lieu, Causal Design conducted a remote training-of-trainers over 4 days and regularly communicated with the enumeration team throughout training and data collection. This allowed both parties to ask questions and clarify any points of confusion, including responses, to maximize data quality. Language barriers also posed a challenge, as they necessitated the use of translators for remote interviews, consequently increasing the risk that some clarity, nuance, and details may have been lost in translation. Finally, given the PE’s scope and intention, it must be emphasized that this PE does not assess the impact, effectiveness, or efficiency in any formal capacity, as it focuses entirely on the quality of implementation. Process Evaluation of the Maharo RFSA in Madagascar Findings 5 3. FINDINGS This section presents the PE’s main findings. Cross-cutting findings affecting multiple interventions are presented first. These relate mostly to contextual factors, particularly the kere/drought, COVID-19, and local governance in Maharo areas, as well as gender, youth, and PWD inclusivity in Maharo interventions. Then, findings by intervention and purpose are presented, including on the kere/drought, COVID-19, local governance, and inclusivity. A brief summary of each intervention, its goals, and implementation is presented in intervention-specific sections. Further details of each intervention are presented in Annex A. As a qualitative analysis, findings are relayed using subjective language. Findings are contextualized around the number of respondents reporting a certain view, with outliers noted. Quotations are sometimes provided, not in lieu of evidence and analysis but as a means to illustrate relevant phenomena or important observations. Quotations are attributed to respondents’ roles or organization, and community workers are identified by only their commune to preserve anonymity. Cross-Cutting, Contextual Findings 3.1.1. Kere/Drought The kere or drought, while not an unexpected or exceptional circumstance given its perception by local communities as a recurring seasonal state of continuous hunger, was nonetheless noted to be particularly severe and posed a challenge to implementation in all areas sampled. The LF intervention was the most directly affected, though respondents from all interventions noted the kere/drought’s effect on participant time availability and motivation, as energy was redirected towards finding food or income generation. Time Availability Across interventions, the most significant effect of the kere/drought was on community members’ time availability to participate in Maharo interventions. Community workers noted that community member participation and availability to attend meetings, trainings, and household visits decreased as the need to search for food and generate income increased. However, a significant majority of community workers noted that they were able to mitigate negative impacts on attendance by rescheduling and adjusting meetings to fit household schedules. As noted by one male Community Mobilizer (CM) from Tsihombe, Tsihombe, “When we carry out a CC, we ask for the communities’ opinion saying, ‘What is the most convenient time for you?’ Therefore, drought hasn’t changed people’s participation because they are the ones who set the time for the meeting.” One Lead Mother (LM) from Imongy, Tsihombe, also noted that she combined Reny Mpiara-Monina (RMM) meetings with Community Health Workers’ (CHW) visits to be more convenient for participants. Community workers also reported decreasing the duration of meetings to avoid taking up time that participants needed to search for food, and due to participants’ decreased attentiveness and concentration because of the kere/drought. For example, one LM from Tsihombe, Imongy, recalled IMPEL | Implementer-Led Evaluation and Learning 6 Findings reducing the length of trainings for RMMs from 5 to 2 hours. One female Miranjaka from Tsihombe, Tsihombe recalled: “I do not hold the visit for too long because people would not grasp much of the training if they were hungry.” Another Miranjaka, a male from Beloha, Beloha, alluded to the “embarrassment” of holding long household visits during times of hunger. Emergency Actors and Expectations of Remuneration Respondents across interventions frequently noted that the kere/drought posed a challenge by raising community members’ expectations of receiving some form of incentive due to the influx of emergency actors distributing aid, somewhat undermining Maharo’s development-focused methodology. In a few cases, communities might reject or refuse to participate in Maharo interventions if they did not receive food supplies, detailed further in Section 3.1.6 Links with Local Governance. One female CM from Beloha, Beloha, recalled, “We had encountered a problem when we wanted to conduct a Social Profiling in the fokontany of Barabay. The people did not accept the meeting until there was a session allowance, although we did not plan for this. The hunger is too high in the area, and people do not want to waste their time attending meetings.” Two respondents (a LM from Tsihombe, Tsihombe, and a male Miranjaka from Beloha, Beloha) noted that the act of recording people’s names as part of Maharo’s Monitoring, Evaluation, Accountability, and Learning (MEAL) requirements, in particular, could create expectations for payment. Behavior Uptake The kere/drought also posed challenges for respondents to take up certain behaviors promoted in sensitization. This was a particular challenge for LFs who could not implement some of the advised agriculture practices due to a lack of rain, such as planting and applying fertilizer. One male LF from Imongy, Tsihombe noted that everything they planted did not grow and ended up dying. The Kere/drought also affected livestock interventions, as explained by one male LF from Itampolo, Ampanihy: “In fact, it would be difficult to implement the techniques that CRS has given us because the soil here is dry; there are no vegetables we can get to feed these animals. In fact, many of our livestock die each year due to the lack of grass and water to drink, even though they are sent to roam around. And now that they are going to be kept in an enclosure during the famine, what shall we give them to eat?” The kere/drought also inhibited the uptake of dietary behaviors promoted by Miranjaka and LMs. As one LM from Beloha, Beloha recounted: “We persuade pregnant mothers to adopt a varied diet when they have money. But they respond that where on earth can they find money to adopt such a diet?” 3.1.2. COVID-19 Implementation Continuation CRS respondents were largely positive about their ability to adapt to the COVID-19 pandemic. For example, CRS staff spoke positively about moving communications online, which entailed training staff Process Evaluation of the Maharo RFSA in Madagascar Findings 7 in Teams technology, ensuring all staff had internet access and smartphones and tablets, and expanding the number of internet service providers due to problems with network coverage. CRS respondents were also positive about their ability to implement during COVID-19, given Maharo’s decentralized model, which relies on community workers present at the fokontany level. This was echoed by community workers (except for CCs, detailed more in Section 3.1.7 Community Consultations), who reported that COVID-19 had little impact on their ability to act among communities. Many noted that they adhered to COVID-19 precautions while conducting interventions, such as social distancing, mask-wearing, and conducting meetings outside. However, Causal Design was not able to validate this as the observation team was unable to observe community workers conducting interventions in the field. In many cases, there appeared to be little awareness of COVID-19 and its presence at the fokontany level, and even indications that not all community members owned or used facemasks. For example: “We did not have this disease at all in our region, and it did not impact my works. We have already been equipped, and we wear face masks when we do home visits.” (Male Miranjaka, Behabobo, Beloha) “We don't really think about that here… It's like we didn't think about the disease and we didn't respect the preventive gestures. As we are in a remote countryside, there was no awareness-raising on that.” (Male PiSP, Androka, Ampanihy) Implementation Quality CRS respondents acknowledge that the quality of some implementations may have decreased because of necessary COVID-19 precautions. Among COVID-19’s most significant impacts in this regard were delays on CCs, the R&I year, and subsequently, CLA, discussed further in Section 3.1.7 Community Consultations and Section 3.1.8 Collaboration, Learning, and Adapting. Some aspects of interventions had to be modified or eliminated under COVID-19, which some CRS respondents reflected may have affected community workers’ and participants’ motivation. For example, under normal circumstances, some interventions would have entailed community meals and celebrations to mark milestones, exchange visits, and travel to in-person trainings, with accompanying remuneration, none of which were possible under COVID-19. The absence of such activities may have contributed to some community workers’ and participants’ sense of dissatisfaction and complaints about a lack of motivation or incentives to implement or participate in Maharo. COVID-19-related border closures and lockdowns also posed a problem, as they prevented key technical and activity staff from traveling to the implementation zones, meaning that sub-office and field staff were forced to implement based on their prior experience or learnings from a week-long online training. Border closures disproportionately affected implementation in Beloha and Tsihombe districts. While CRS staff from CRS’s Tulear sub-office were still able to travel within Ampanihy, Tsihombe and Beloha were cut off. Border closures also affected CRS’ staffing, with CRS respondents IMPEL | Implementer-Led Evaluation and Learning 8 Findings recalling difficulties hiring suitable consultants to conduct R&I assessments and studies (detailed more in Section 3.1.8 Collaboration, Learning, and Adapting) and a consultant to conduct additional analysis on the qualitative CC data for which CRS was awarded a small Implementer-Led Design, Evidence, Analysis and Learning (IDEAL) grant. COVID-19 adaptations also created additional burdens for CRS staff. For example, the number of training sessions held was much higher than anticipated, given the need to reduce the number of people attending meetings. While this may have had positive impacts on trainees, it significantly added to the workload of CRS trainers and technical staff and delayed the completion of trainings. COVID-19 also entailed challenges with inter-sectoral cohesion and collaboration due to an inability to meet in person and strategize. As one CRS respondent expressed: “We don't have as strong cross-sectoral oriented integration as we could… It gets a little bit tough when you get into individual communities and talk to the ATs [Agents du Terre/Field Agents] and see the linkages between the MCs [CMs] and the ATs. And that's a function of that initial period where we had to start up but we, the key personnel, were blocked out in Tana and we had to do everything by Teams… we still don't have great integration with cross sectoral strategy. It’s like implementation proceeded at the expense of strategy and full integration.” Distribution COVID-19 also affected the availability and distribution of some materials. For example, border closures were cited for having delayed the distribution of LM recognition kits. Additionally, motorcycles and spare parts imported from abroad were delayed due to pandemic-related supply-chain problems. Though inevitable, community workers on the ground who did not appear to be aware that the reason for the non-receipt of materials was due to the pandemic felt these challenges acutely. 3.1.3. Gender Inclusion Overall, CRS respondents spoke very positively about Maharo’s gender sensitivity and achievements in gender outcomes. This was largely attributed to the significant efforts of the Miranjaka and the CRS gender inclusion team, who ensured that gender inclusion was taken into account across purposes and interventions. Gendered Participation Gender inclusion efforts, successes, and challenges varied by intervention, partially due to the nature of interventions, and are summarized in Annex C. Notable, and crosscutting themes include: ● Women are well-represented as members in mixed-gender groups, such as DRR committees, LF Demonstration Plot groups, and SILCs, in particular. Process Evaluation of the Maharo RFSA in Madagascar Findings 9 ● Though Miranjaka and LMs make efforts to include men, men’s participation and engagement with Miranjaka and LM interventions remains limited, as these sensitizations are generally considered to be women’s topics. Some men are not interested in gender equality discussions. ● Across interventions, there remains some reluctance for women to express themselves and make decisions in public forums or in front of men. ● There is strong evidence of deeply entrenched gendered divisions of labor, though communities do not view this as problematic. o SILC: Though men have higher incomes and are able to make larger contributions, fewer men than women are SILC members because women usually conduct the types of income-generating activities that typically require loans, such as sales. o DRR & LF: Men and women conduct very different types of physical labor as part of LF Demonstration Plot groups and DRR Committees, though communities perceive the division of labor to be complimentary. However, women face constraints in becoming LFs, as communities do not perceive them to have authority in agricultural matters. Women’s participation in livestock interventions is particularly limited due to persisting traditional beliefs, as explained by one male LF from Itampolo, Ampanihy: “Despite the equality of rights that is being promoted among Miranjaka, the participation of women in farming is relatively limited as it is difficult for local men to allow women to enter into the goats' enclosure because it is taboo, especially during a woman's menstrual period. It is not a woman's job to build and maintain the goat's pen; it is really a taboo. The only thing a woman can do is go there to bring animal feed such as black￾eyed peas branches or stalks or sweet potatoes when they are back from the garden. The rest don't seem to match with our customs.” Training on Gender Sensitivity Maharo relies on a cascade-training model for gender and other inclusivity topics, where Field Agents/ Agents du Terre (ATs) are trained, and are then responsible for training community workers on gender, youth, and PWD inclusion. However, community workers’ responses indicate that in some cases, there is a need for more effective gender training, directly from experts in gender inclusion and sensitivity. For example, many community workers voiced that there is “no discrimination” in their groups or interventions, suggesting a lack of full understanding of gender, PWD, and youth-sensitivity. Similarly, one male Miranjaka, from Behabobo, Beloha, said there is “no difference at all [between sensitizing boys and girls] because they have the same life.” Some expressed more explicit problematic attitudes towards gender roles, differences, and equality. Some illustrative quotes include: “The problem I personally find is that women really boast when given a platform and given a chance… When something happens, they boast a lot because they get a chance to express themselves.” (Male CGRC Member, Androka, Ampanihy) IMPEL | Implementer-Led Evaluation and Learning 10 Findings “Opinions from women can never be superior compared to men. [Moderator: What could be the reason for the low opinion of women compared to men?] That's God's organization.” (Male Equipe Local de Secour/Local Rescue Team (ELS) Member, Androka, Ampanihy) “[Moderator: What causes violence at home?] Maybe violence is already in us men. Or it's natural.” (Male Miranjaka, Anampaly, Tsihombe) “I don't fully agree with the fact that men and women are equal… Respect between spouses may no longer be established, considering the topics of Miranjaka. The father of the family may no longer have his authority … It is difficult to fully achieve gender equality because Androy's society is governed by certain traditions, such as: the sacrifices cannot be performed by women even if men and women have equal rights.” (Male CM, Itampolo, Ampanihy) 3.1.4. Youth Inclusion Overall, respondents had less to say about the inclusion of youth in interventions. As with gender, youth inclusion successes and challenges varied by intervention, and are presented by intervention in Annex C. Notable and crosscutting themes include: ● Across interventions, there remains some reluctance among youth to express themselves and voice disagreement with elders. One male CM, from Imongy, Tsihombe, added that youth from rural communes are even less likely to be vocal than youth from less remote areas due to having less education. ● Youth participate more in DRR Committees, CGs, and Miranjaka interventions as they are perceived to be more dynamic and energetic. o DRR Committees: Youth have good participation, with some groups having a majority youth membership. This is partially because youth ambassadors and youth leaders were trained specifically on DRR issues. Furthermore, youth are also perceived to be motivated and energetic, which is beneficial for DRR mobilization, sensitization, and tasks, such as construction. o CGs: Youth are involved as both RMM participants and LMs. Despite initial fears that young LMs would not be listened to, young LMs have proved to be influential and dynamic. o Miranjaka: Youth are targeted more for household visits and as model households, as they are more open to receiving and following the advice of Miranjaka messages. ● Youth have less representation and are less involved in PiSP, LF, and SILC interventions. o LF: Despite their physical strength, which is viewed as an asset for agricultural activities, fewer youths participate in Demonstration Plot groups and are less likely to become LFs as they are perceived to have less authority and experience with agricultural, livestock, and fishing matters. Furthermore, youth do not view agriculture as a fruitful Process Evaluation of the Maharo RFSA in Madagascar Findings 11 opportunity, given increasing population pressure, climate change, and limited amounts of arable land available. o PiSP: Youth are less well-represented among PiSPs due to a lack of funds to start sales businesses. o SILC: While youth may participate in youth-only SILCs, one male SILC respondent, from Imongy, Tsihombe, noted that while young women took loans to start small businesses, young men were less likely to take loans due to a fear of misspending funds. 3.1.5. People with Disabilities Inclusion CRS respondents hailed the commitment to include PWDs as a success in itself, noting that disability remained a strong taboo in Madagascar and development projects have historically rarely included PWDs. However, PWD inclusion under Maharo has not yet been as successful as gender inclusion, though this is unsurprising, given the contextual status quo of extreme marginalization of PWDs in Maharo target areas. Details of PWD inclusion per intervention are presented in Annex C. Notable takeaways include: ● The number of PWDs involved varied across interventions, as shown in Table 1. For interventions where group membership was voluntary, many had no PWD members, usually attributed to the fact that there were no PWDs in the fokontany or that no PWDs had chosen to sign up. In some cases, PWDs were represented indirectly, with other household members participating in their stead. For DRR Committees and CCs, PWDs had been purposefully selected to participate to ensure inclusion. Speaking speculatively, many respondents from groups with no PWD members said they would welcome PWDs into the group if any wanted to join. Table 1. Number of groups sampled that have people with disabilities members/participants CGs DDR Committees LFs SILCs CCs Number of groups with PWD members/ participants 4 / 9 5 / 9 ELS 5 / 7 CGRC 2 / 9 3 / 9 7 / 9 Note: Equipe Local de Secour/Local Rescue Team (ELS). Comité Communal de la Gestion des Risques et Catastrophes/ Commune-Level Disaster and Risk Management Committee (CGRC). ● Responses were mixed regarding the quality of PWD participation. Some success stories were noted, such as a visually-impaired community member who held leadership positions in both the fokontany’s Lead Fisher and livestock LF groups and was very vocal in discussions. However, it was frequently reported that PWDs were shy to speak up during group and community meetings. This was particularly the case for disabled women, who were likely to speak only when spoken to. ● PWDs faced significant physical barriers to agriculture and DRR-related interventions. o PiSPs: One female PiSP from Androimpano, Ampanihy, noted that PiSP agricultural equipment was not disability-friendly, and PWDs had difficulty traveling to PiSP shops to collect items. o DRR Committees: Physical disabilities inhibited PWDs from participating in DRR Committee activities, as they could not conduct manual labor tasks such as construction. IMPEL | Implementer-Led Evaluation and Learning 12 Findings o LFs: It was clear that physical barriers were considered a major obstacle that prevented PWDs from joining LF interventions, given their physical nature. PWDs could also face prejudice regarding their participation. Some LFs feared that PWDs would “bother” other participants, perceiving PWDs to be unable to contribute in Demonstration Plot groups. In the words of one male LF from Androka, Ampanihy, “How do we connect with people with disabilities? We can’t let them in because there’s nothing they can do; they just bother us at work.” Another male LF from Anjampaly, Tsihombe, said: “In general, their [PWDs’] position in the organization is that of ordinary members, while in the real sense that their participation is less obvious because as much as they are disabled, their potential is limited, even in terms of financial contribution. They are given special freedom to contribute, so when they want to contribute, they can, and if they do not contribute, they can do so. However, this does not exclude them from membership because they get opportunities like every member.” Integration of PWD Inclusion Strategy Humanity & Inclusion (HI) plays a key role in Maharo’s PWD inclusion by providing training to CRS staff on disability and raising awareness and recognition of PWD issues. However, some CRS respondents voiced difficulties integrating HI and PWDs into Maharo and its strategy. For example, one recalled difficulties applying the Washington Group Questionnaire into the Madagascar context. According to some CRS respondents, staff turnover at HI and COVID-19, which prevented in-person group meetings, also hamstrung the desired alignment of Maharo’s disability and food security goals. Types of Disability Across the board, most community worker respondents seemed to understand disability as meaning only physical disability, and responses regarding inclusion efforts and challenges seemed to focus on overcoming physical disabilities (e.g., transportation to meeting sites, having others conduct manual labor on their behalf, etc.). However, a few CRS and community worker respondents raised the question of mental versus physical disabilities. A few explicitly noted that they did not know how to deal with or adapt their strategies to those with mental disabilities and that the materials they had were not adapted to those with non-mobility-related disabilities (e.g., for blind or deaf people). Training Across interventions, there appeared to be gaps in training on how to include PWDs. Some community workers explicitly stated that they had not been trained on PWD inclusion. Others recalled that training had only consisted of the message that PWDs had full rights as other people but did not cover how to adapt sensitization methods for PWDs. As a result, many community workers remained unsure of how to include PWDs in their sensitizations and were hesitant in some cases. Further training on this could prove to be highly beneficial—one Miranjaka, a male from Behabobo, Beloha, who attended a PWD inclusion day and received a training from HI in Tranoroa, spoke positively about his experience and showed a much greater knowledge of how to engage PWDs and the importance of PWD inclusion. Process Evaluation of the Maharo RFSA in Madagascar Findings 13 Direct Support PWD inclusion efforts largely hinge on HI training CRS staff on disability. However, to truly bring benefits to PWDs under Maharo, there appears to be a need to integrate direct support for PWDs. This was noted by several community worker respondents across interventions and CRS respondents explicitly. CRS respondents acknowledged this limitation and noted that CRS is in discussions with HI to transition HI’s role away from purely training CRS staff and move towards more direct implementation. 3.1.6. Links with Local Governance Successes of Operating within Local Governance Systems Across interventions, respondents regularly emphasized the importance of working within local governance systems3 to gain permission to conduct interventions, identify key leaders of interventions, and establish the credibility of the intervention leaders among participants. Across interventions, working with fokontany chiefs, in particular, was regarded as part of the necessary formalities to conduct interventions in the fokontany. As one LM from Anjampaly, Tsihombe said, “Respect and good manners are always needed, so the president should always be informed of any event… Cooperation with the president is always necessary and effective because the existing structure in the company must be respected.” Beyond formalities, however, fokontany chiefs were cited as critical to raising awareness of interventions and mobilizing people to participate. Their authority gave community workers credibility and “validation,” and some added that it would not be possible to conduct interventions without involving local authorities. Typical sentiments included: “As a member of Miranjaka, I would not be recognized at all if I was not elected in front of the elders and the fokontany chief, in the presence of the community.” (Male Miranjaka, Itampolo, Ampanihy) “My collaboration with the fokontany chief is successful because without him, the place to hold the CC is not known. He also encourages people to come to the meeting and when the meeting is held without a fokontany chief, it is difficult to conduct it. If the fokontany chief is not standing there, the meeting is considered to be a nonsense meeting.” (Female CM, Tranovaho, Beloha) Working through local authorities strengthened the organizational capacity of volunteers, as again, the credibility of fokontany chiefs, in particular, was crucial to helping to find locations for, and communicating the dates and times of key events, such as meetings and PiSP distributions. Two LFs also noted that fokontany chiefs issued authorization for LF groups to use land as demonstration plots. 3 Local governance, in this report, is conceived of as the de facto and institutionalized process of local decision￾making that may include formal and informal structures (Klick 2016) or the “the interactive process of multi￾stakeholders (including government) in order to resolve common problems” (Saito 2008:6). IMPEL | Implementer-Led Evaluation and Learning 14 Findings In some cases, local leaders assumed an active role in conducting Maharo interventions themselves. Fokontany-level DRR Committee respondents, for instance, noted that they and KFFs worked together to implement DRR plans and activities, such as construction. Some Miranjaka also noted that KFF members and fokontany chiefs helped conduct sensitizations. Obtaining the support of local authorities was especially essential to the Miranjaka component, as its focus could be highly contentious and controversial among community members, including leaders. CRS staff noted that in order to be effective, Miranjaka needed the buy-in and involvement of fokontany chiefs and traditional authorities to be able to deliver gender-related messaging and even, in some cases, to ensure Miranjakas’ security. To support the Miranjaka intervention and forge links with local leaders, Maharo developed and has been instigating the “Advocacy for Traditional Leaders Approach,” which appears to be successful in garnering the support of local authorities and allowing Miranjaka to advance in their mission. Challenges Operating within Local Governance Systems Though the majority of community workers spoke positively about their relationships with local leaders, CRS respondents noted that in some cases, relationships had to be managed very carefully due to questions of personal interests and incentives. One CRS respondent, for example, reported that local officials regularly expected some form of payment. To overcome this, they permitted the fokontany chiefs to distribute LM recognition kits. As stated previously, one male member of a fokontany-level DRR Committee from Androka, Ampanihy, noted that, in order to obtain buy-in, it was necessary to give “a small gift which opens the way.” A particularly problematic area was the distribution of food aid. CRS respondents noted that Maharo’s food aid is targeted to vulnerable households that meet specific criteria based on its census. Because different organizations relied on (unverified) household lists provided by chiefs, some local officials publicly castigated CRS and its methods, which resulted in local discord. CRS acknowledged that its way of targeting, while more accurate, could, in the short term, result in limited access to certain localities where chiefs continued to protest. CRS also noted that given that distribution occurs at the cluster level, it was likely to be mayors rather than chiefs who are most likely to limit access and that chiefs may be parroting the arguments of mayors in interviews. The perception of chiefs influencing outcomes, particularly recruitment for Maharo community worker roles, also created consternation. CRS respondents recalled that, for those interventions that required tests to qualify for a specific position, such as phytosanitary officers and PiSPs, not all candidates would pass, which communities sometimes interpreted as meddling by chiefs to favor family members or friends. In some cases, chiefs, in contrast, were frustrated and protested when they felt as though they were purposefully sidelined in important local affairs, including their ability to influence these very same appointments. This is further discussed in Section 3.4.2. Lead Farmers. In some ways, even the above challenges cited in working through local governance structures underscore the centrality of governance stakeholders such as chiefs. As others have observed in other rural, agrarian settings, the mere presence of external development actors can disrupt local systems of governance and create tensions when traditional authority is upended by entities with resources and the power of distribution (McNamara 2019). CRS appears cognizant of this risk and cited their work with Process Evaluation of the Maharo RFSA in Madagascar Findings 15 KFFs (discussed further in Section 3.2.1 Miranjaka) and their gender programming sensitization with local leaders, as evidence of their focus on the “last mile.” However, the risks and rewards of working through chiefs and other informal actors are not always apparent, and obstruction by chiefs may reflect rational fears of a diminished local authority as much as it does corruption or favoritism. 3.1.7. Community Consultations Validation Findings Validation data illustrates that Community Consultations (CCs) occurred in all visited fokontany, following a very consistent structure. All fokontany saw participation from both youth and “adults,” and both men and women in the community. These groups are split up by age and gender to discuss pressing issues and opportunities for improvement within the community. Issues discussed included agriculture and pests, water access, availability of schools, hospitals, and veterinary care services, malnutrition, kere/drought, natural disasters, and infrastructure, such as road construction. Participants referred to using visual learning techniques in these sessions, such as drawing problems and maps of the village, which seemed to be a particularly accessible method to represent and discuss problems. In many cases, meetings were followed by a community lunch and sometimes a wider celebration, including dancing. In one instance, in Tranovaho, Beloha, the CC also entailed a hygiene-awareness raising. Implementation Successes Similar to the inclusion of PWDs, CRS views the inclusion of a CC component—as a relatively new concept in the context of southern Madagascar and designed to maximize inclusivity and ownership of the challenges and responses—is viewed as a success in and of itself. CRS respondents recalled that, historically, few (if any) genuinely participatory approaches had been implemented in Madagascar, with previous iterations of CC methodologies typically consisting of CMs serving as messengers between implementers and communities. By contrast, Maharo’s CC component trains CMs to become “social technicians” with developed expertise in community engagement in an effort to maintain an ongoing conversation and build trust between implementers and communities. In this way, CRS emphasizes that COMMUNITY CONSULTATIONS Community Consultations (CC) are meetings with community members in which community members discuss community problems and capital in order to prioritize, plan, monitor, and adapt Maharo interventions in their area. CCs aim to contribute to sustainability by developing community responsibility and ownership of Maharo goals and interventions. In order to be inclusive, CCs entail separate discussions based on age and gender, including PWDs and marginalized groups, and utilize practical, visually-oriented tools. CCs are conducted by CMs, who are salaried CRS staff hired from local communities. Typically, CCs take place over 3 days and culminate with a community meal (though this model is planned to be changed to be less time-intensive for future CCs). CRS has also won an IDEAL-funded grant to analyze the qualitative data obtained through CCs. IMPEL | Implementer-Led Evaluation and Learning 16 Findings CCs and CMs under Maharo reflect a substantively better community engagement with mutual accountability compared with the superficial participatory models of past projects. CRS respondents also hoped that the CM methodology would have wider-ranging effects on increasing recognition of the value of social technicians and investment in them to ensure successful implementation and impact. CRS respondents recalled that even within Maharo, CMs had to overcome the challenge of being perceived as low-level community volunteers or messengers rather than as technical, skilled staff. However, this was overcome by CMs’ significant training, with CMs gaining confidence in themselves and recognition from other technical staff. One CRS staff member suggested that the term “CM” could be a misnomer—given that they are tasked with far more than rounding up participants, as the name would suggest—and would, in retrospect, ask CMs to provide a name for themselves that they felt accurately portrayed their skillset. Capacity Building and Recruitment Training for CMs was effective, evident both from validation data, where respondents recalling the process of CCs indicated that CCs were being held according to the correct methodology and in interviews with CMs themselves, where they reflected positively on the training they had received and showed a strong awareness and belief in CC principles. For example, some CMs specifically spoke about the four core principles (participation, inclusion, sustainability, and intervention quality) and the importance of these. CMs interviewed also elaborated on the importance of CCs for the activity to be relevant and participatory. As one female CM from Beloha, Beloha, stated, “The most important [aspect of Maharo] is the CC because the real needs of the community come out from this… CC is a way to collect ideas and feedback from the community; it is not a place to receive complaints, but often their problems and aspirations are reflected in the ideas that come out.” A further success is that CMs greatly improved their professional skills and employability as part of Maharo. CRS respondents noted that there was a large appetite among CMs to grow and develop their professional skills; a key focus of CM training focused on developing CMs’ own career goals. This support included creating opportunities for CMs to gain university qualifications in sustainable development. One CRS respondent emphasized that CMs were filling a significant capacity gap in the south of Madagascar and thus had a contribution well beyond Maharo, both for CMs themselves and for other organizations working in the area. In their own words: “It's very important to me that these young, bright, committed people from the south of Madagascar have a future that wouldn't normally be available to them. In the humanitarian and development sector, they would normally be written off as people with no experience, no training, and no contribution. I hope, well, I know, that by the end of Maharo project, there will be 80 very marketable people on the job market, who we have trained up and given experience to.” Hiring Local A facilitator of success for the CCs and linked social and behavior change interventions was that CMs were hired from local communities. This familiarity and language ability contributed to creating trust Process Evaluation of the Maharo RFSA in Madagascar Findings 17 among community members and local governance, also the uptake of behavior change mechanisms. For example, one male CM from Androka, Ampanihy, noted, “First of all, I am a local person; that is to say, I am a child of the area, so it is easy for me to convince the chief of the fokontany and the community… The presence of our CM working here also facilitates Maharo's entry into the fokontany because the community sees that their children are working there.” However, one CRS cautioned that the influence of hiring locally could be overstated, noting that CMs are still viewed as “CRS people” on the ground. Implementation Challenges Community Meals Many CMs emphasized the critical role of providing community meals in mobilizing community members to participate in CCs. However, it was noted that the original three-day CC model had been changed to a more spread-out model, whereby groups of community members would meet on separate days, and community meals would not be included. The change was made because the three-day model was found to be too time consuming and onerous for participants. However, some CMs noted that when food was not provided, people were reluctant to attend meetings, which could lead them to refuse to participate or cause anger and frustration among communities. For example, one female CM from Tranovaho, Beloha, recounted, “In the fokontany, there are also small problems, for example: in the countryside, when food is not distributed, people are reluctant to gather and they are angry saying: you call us to meet for nothing but we are hungry, there is no water to drink and no food to cook.” Transport Several CMs reported that limited access to transport, partially linked to COVID-19 supply chain problems, created a burden for CMs and posed challenges for conducting interventions across all fokontany. One male CM from Itampolo, Ampanihy noted that the inability to travel decreased monitoring and hence decreased communities’ motivation to put into practice plans made as part of the CCs: “For example, the Community has already agreed with Maharo, to build a house to sell pesticides for pests… However, the problem is: people don’t do the work unless they see me moving around.” Local Ownership of the CC Follow-up In theory, CCs provide a stimulus for community-led development through identifying “small doable actions” that communities can implement themselves, and substantiating these with Maharo interventions. However, it appeared that on the ground, a full sense of local ownership had not yet taken hold, causing some frustration among community members. Validation data and interviews with CMs found that many communities are excited about the prospects of a Maharo partnership, but none have perceived further action taking place following the CCs. Communities largely have not seen any impact following CCs, nor does it appear that many see a clear path or next steps ahead. Rather, they see progress as stalled, leaving some disappointed and frustrated. As one female CM from Beloha, Beloha, recalled: “Because of the delay in response from CRS, people are complaining and getting tired of waiting. Besides, they have to work for a daily living while the results of CRS activities are not yet visible… they complain and are impatient because to date, CRS has not implemented the action plans obtained from the CC.” IMPEL | Implementer-Led Evaluation and Learning 18 Findings Delays CRS respondents acknowledged that, despite the appetite on the part of communities, CRS had not yet been able to fully deliver following the CCs, due to a number of factors, most notably, COVID-19. Unlike other community workers, CMs, as CRS staff, were obligated to strictly adhere to protective measures, which including postponing meetings which required large groups to gather. As such, these delays had a knock-on effect of delaying the realization of CC findings into action plans. CRS staff also noted that environmental compliance and procurement processes delayed implementation. For example, USAID’s Initial Environmental Examination review took 18 months, during which it was not possible to commence any infrastructure activities. 3.1.8. Collaborating, Learning, and Adapting At time of writing, few instances of meaningful changes in response to the collaborating, learning, and adapting (CLA) process and R&I year were identified. This was largely attributed to delays caused by COVID-19, which effectively derailed the R&I year. As previously discussed, delays in conducting CCs subsequently delayed CRS from integrating the findings of the CCs into action plans and implementation. CRS staff also recalled that border closures caused difficulties being able to hire qualified international consultants into Madagascar to conduct the planned R&I assessments and studies, leading to delays in their completion. One CRS respondent added that difficulties hiring external consultants also led to a reliance on national consultancies, and consequently a compromise in the quality of the studies conducted. The delays in conducting studies and assessments meant that implementation had to roll out ahead of the completion of the studies, effectively reversing the planned concept and purpose of the R&I year, and leading to what one CRS respondent called “technical slippage.” As one CRS staff member stated, “You hire these teams to implement, we didn't think it made sense to shift them to become data collectors and to try and undertake studies… When we couldn't get our regional technical advisors into country, and when we couldn't undertake those studies, our choices were to either wait and keep completing reports for USAID or to just get out there and start doing it.” Despite being around a year behind the planned schedule, CRS has now begun the process of incorporating findings, though CRS respondents acknowledged that this process of integration was still a work in progress. One example that illustrates a tangible success in Maharo’s CLA is the incorporation of CC findings on shocks—windstorms in particular—into other CRS projects. In this example, CCs revealed that tiomena (windstorms) was a significant threat to livelihoods and resilience. While not possible to incorporate this into Maharo, the finding about tiomena informed the design of another CRS project funded by the World Bank. However, a few CRS respondents noted that the R&I studies had largely been underwhelming and had not produced much new knowledge that drastically altered or informed Maharo’s implementation or Theory of Change. There are positive signs that Maharo is making strong headway and efforts to create CLA and adaptive management systems. CRS respondents reported holding regular strategic review meetings and monthly pause and reflect sessions. The CLA workshop and action plans developed during the workshop Process Evaluation of the Maharo RFSA in Madagascar Findings 19 also indicate a strong commitment towards CLA. CRS respondents also viewed the CC methodology itself as an effort towards adaptive management. For example, a few CRS respondents felt that CCs were helping to create a culture shift within the organization towards listening more to community needs and building a more participatory implementation model. As one CRS respondent noted, speaking about the sharing of CC findings with the wider CRS team: “The team really got, you know, understood, it's that we should be, we are in partnership. It's not an us and a them. It's a we, right? And I think within our team, it's slowly understanding that and now realizing that if we're going to really create purposeful behavior change, the motivations come from the participants, not us.” However, some CRS respondents nonetheless conceded that building a learning culture and truly adopting the principles of the CCs is still a work in progress. They felt that there remained a degree of fear of failure, and that some teams did not place CLA as a high priority, viewing it as an additional burden to their implementation work; consequently, CLA meetings did not always have full attendance and participation from all necessary staff. Some also felt that there remained a tendency to focus on Maharo-specific MEAL targets, rather than bigger-picture impacts and long-term effects more broadly. Purpose 1: Governance and Social Services Purpose 1: Formal and informal local governance systems deliver inclusive, equitable, and effective social services and safety nets to most vulnerable households and individuals. 3.2.1. Miranjaka Gender Champions Validation Findings The validation exercise revealed that, among a Miranjaka’s tasks, meeting and coordinating with KFFs was the most consistently done. Dad’s Schools and Safe Space discussions, in contrast, had only been held in Ampanihy, which CRS respondents in ground-truthing sessions attributed to COVID-19 border closures, which meant that Ampanihy remained accessible for CRS’ regional office in Tulear, allowing interventions in Ampanihy to proceed at a more advanced pace than Beloha and Tsihombe. Where Safe Space sessions were held, meetings were differentiated by gender, and held approximately one month apart, though it was intended that eventually men and women would meet together. Topics included community-based intervention and mobilization in the face of disasters and crises (e.g. repairing community assets, making financial contributions, etc.), sharing domestic responsibilities, MIRANJAKA GENDER CHAMPIONS The Miranjaka’s role is to promote gender equality and inclusion at community and household levels through a range of channels, including working with local governance as part of the KFF/VDC, conducting household visits, raising awareness at community meetings, and holding Safe Space discussions and Dad’s Schools. Miranjaka receive capacity building from CRS project staff on topics such as gender norms, women’s empowerment, positive masculinity, and on how to conduct Miranjaka tasks. IMPEL | Implementer-Led Evaluation and Learning 20 Findings domestic abuse, alcohol abuse, and human rights. The reporting of non-gender specific topics covered in Safe Space discussions may reflect a combination of factors, including: a) Individuals who play multiple roles within a community confusing their responsibilities or areas of focus (which CRS acknowledged as in alignment with their feedback of happenings on the ground); and b) the sensitive nature of training on gender in Tsihombe, in particular, where extra steps were taken to build support among local leaders through a specific advocacy for traditional leaders approach, previously mentioned above. Where Dad’s Schools were held, they occurred approximately every three months. Topics included sharing domestic responsibilities, financial decision-making, domestic abuse, and alcohol abuse. Some anecdotal evidence suggested that the training did resonate, though this is difficult to confirm. Implementation Successes In the most general sense, Miranjaka appear to be underway in good order. Miranjaka consistently reported regular and positive engagement with KFF, with very consistent reporting on the number of KFF meetings attended and household visits conducted per month (with the exception of Safe Spaces and Dad’s schools, as detailed in Section 3.2.1.1. Validation Findings). Validation exercise data indicate both intimate familiarity with Miranjaka objectives and tasks, as well as consistency across space (at least for KFF coordination and startup). For example, all but one respondent from the validation exercises cited meeting with KFF twice per month (the respondent from Androka reported meeting once per month). While responses around training are sometimes inconsistent, Miranjaka largely reported that they received adequate training, comprehend the training, and are committed to the Miranjaka objectives and their role within the community. Respondents also demonstrated creativity, and ability to adapt to circumstances, by altering the timing of their visits, the duration, and even how they approached the subject of hunger when pressed for material assistance in meetings. Some also relayed their experience practicing their emotional intelligence (or ‘soft skills’), by purposefully not getting angry during a conflict with a husband, for example. Finally, as discussed in Section 3.1.6. Links with Local Governance, Miranjaka are now operating in areas traditionally hostile to some of its messaging, with the support of local leadership. This, by itself, is a considerable inroad. Implementation Challenges Despite the overall strong organization and implementation of Miranjaka so far, several, mostly field￾level challenges have hampered the inception phase. These can be divided between programmatic or logistical issues and less measurable “ideational” issues, including perceptions of trainers, a woman’s ability to fully participate, and motivation of the Miranjaka, for example. These are difficult sentiments to confirm, and some of these observations reflect only one or two responses but are nevertheless important to report, given their potential to disrupt activities over the long term. Shortages of Training Materials Among the more technical complications were limited training materials, both during training for Miranjaka themselves and limited materials to leave with households to reinforce messages passed on during household visits. As one male Miranjaka from Behabobo, Beloha recalled, “When I do home visits, people usually say, ‘please leave your book with us for us to read [small laugh].’ It would be good to have Process Evaluation of the Maharo RFSA in Madagascar Findings 21 posters to stick on the wall, to help others memorize during the sensitization. We also need pamphlets and flyers to give to the household.” Literacy Several respondents, mostly male, acknowledged that illiteracy among Miranjaka themselves presented a challenge. One illiterate Miranjaka, a male from Imongy, Tsihombe, for example, would bring his wife to trainings and household visits to help him learn and conduct sensitization. Another literate, male Miranjaka, from Beloha, Beloha, recalled how varying literacy levels affected Miranjaka trainings: “80% of the Miranjaka are illiterate. It's very difficult for them to follow the training that was presented on slides, they are country people, and it's like they're just watching a movie or a video, and they don't even know what it is. It was better to do the training on debate than on those slides. That's the problem with delivering the training… Even the trainer was frustrated.” Lack of Compensation Miranjaka also reported flagging interest or motivation, citing their lack of payment. One male Miranjaka from Itampolo, Itampolo, explained: “Sometimes, I'm not enthusiastic because I don't get any money to motivate me. The 6,000 Ariary [c. 1.50 United States dollar (USD)] from the training period is the only thing I got; as a result, I get tired of doing it.” One CRS respondent echoed this view, noting that Miranjaka often work much longer than the anticipated eight hours per month and felt that it would be better to start paying Miranjaka a small stipend for the work they do. Persisting Gender Norms Evidence suggests that, in spite of the perceived successes of Miranjaka, women’s liberty to fully engage with lessons remains limited. Data from across respondents confirm that very traditional roles and norms persist in Maharo implementation areas and, generally, contrast with the baseline findings that suggest higher levels of women’s “empowerment.” This is, of course, the very target of Miranjaka lessons but also compounds the resistance to it, especially among men. Negative Perceptions of Miranjaka Data also indicate some negative perceptions of, or frustration with, the Miranjaka themselves, on a certain level, largely related to their inability to provide resources: As one male Miranjaka from Anjampaly, Tsihombe, noted, “If, for example, you are going to gather people now, we are free. But if it was me, who lives here who wants to gather people, they don't want me to gather them… We find that when it's among ourselves in the community, nobody wants to attend. People are reluctant because I have nothing to share.” IMPEL | Implementer-Led Evaluation and Learning 22 Findings Purpose 2: Nutrition, Health, Water, Sanitation, and Hygiene Purpose 2: Nutritional status (acute and chronic) of pregnant and lactating women, adolescent girls, and children under 5 improved and sustained. 3.3.1. Care Groups Validation Findings Based on the validation exercise, most RMMs meet regularly twice a week, usually under a tamarind tree in the fokontany. Full or near full participation by members is common, and members seem engaged with the trainings. Topics covered across fokontany are similar and include prenatal care, breastfeeding, vaccinations, hygiene, and mosquito net use. LMs have a book that they use to teach the sensitizations. While there are some issues around respondents recalling training content, in interviews, most LMs interviewed appear to remember and repeat their sensitization training. There also seems to be considerable uptake of knowledge learned in CGs. LMs, and participants themselves, for instance, note that they are trying to practice what they have learned, such as going to hospitals to get prenatal care. Others report confronting several barriers to putting lessons into practice due to a lack of resources. Participants from Beloha and Androimpano, for instance, explained that they were unable to wash their hands as often as recommended due to a lack of water, while participants from Beloha and Tranovaho reported being unable to follow advice about latrine use because their latrines were broken. Implementation Successes Much like the Miranjaka, the data reflects consistent reporting of effective training, with overall strong recall among CG and LM respondents of key lessons and training objectives. Several respondents discussed the importance of their outward appearance (with an emphasis on clean clothing) in order to build credibility and project the care and importance they ascribed to their roles and the lessons. Overall, LMs demonstrated a commitment and interest in their role and its objectives. CARE GROUPS CGs consist of 10–15 volunteer LMs who work to promote best practices in WASH, health, nutrition, early childhood development, and gender roles. CGs follow a cascade approach, whereby LMs are trained by Field Agents and CHWs through the CGs, and then share learnings to their communities by conducting household visits and meeting with RMM groups, which consist of 10–15 pregnant women or women with children aged under 2 from the LM’s neighborhood. CGs also work with other Maharo actors, such as SILCs, LFs, Miranjaka, and CHWs to share information. Process Evaluation of the Maharo RFSA in Madagascar Findings 23 Implementation Challenges As with the Miranjaka, the nature of challenges is again divided between some basic material and logistical (or sequencing) challenges that CRS might address, while others reflect perceptions and thus engagement. The latter may be more difficult to immediately address but are nevertheless important and reflect some of the interconnectedness of agents in a system and the effects of an outside intervention. Shortages of Training Materials The most basic complication reported by LMs was limited access to training materials, which resulted in more cumbersome training, having to share materials, and meaningful delays in implementation. As recalled by two LMs: “We all meet in one place, but 15 people in each CG. We form a circle, and we take turns using the picture box. [Moderator: What is the result?] No problem, but you have to wait your turn to use the picture box.” (LM, Anjampaly, Tsihombe) “The training was last November; we didn't work in December because there was no equipment yet. It was only in January that the equipment distribution staff arrived, and then the work began.” (LM, Imongy, Tsihombe) Notably, this material shortage also influences perceptions. For instance, one LM from Beloha, Beloha, noted: “The training was clear for me. However, I didn’t like it that, when being given the tools, some people got some while others didn’t at all. It induced jealousy for those who did not get. Also, the way they shared the tools was not fair, they gave to whom they kind of liked; that is not okay for us. That was the real dilemma during this training.” The lack of training materials itself, ground-truthing suggests, stems in part from a switch to a clustered approach, which resulted in an increase in CGs and LMs that had not been originally anticipated. Varying LM Capacity As with the Miranjaka, differentiating capacity was also reported among LMs. Some LMs hold multiple positions within the community, including as CHWs. Some LMs are literate, and their responses to interview questions are clear and articulate. Others, however, struggled to address the questions and meet the objectives of the LM role. CRS generally views the fact that literacy is not used as a criterion for the Miranjaka or LMs as a positive. While this report does not disagree, it recognizes, based on responses, that notably, even perceptions of inequities can threaten activity credibility and, thus, participation. Across activities, it was notable that the few best-educated individuals in communities tend to serve in multiple roles. Time Constraints Another factor affecting implementation includes the time constraints of participants and LMs themselves. Home visits are routinely cut short. Overall, trainings were fewer than planned and shorter (2 versus 5 hours, for example, and combined with CHW visits). All stakeholders are at times constrained IMPEL | Implementer-Led Evaluation and Learning 24 Findings by duties related to water collection, funerals (cited regularly), sickness at home and among children (also cited regularly, of note), and complications stemming from the kere/drought. Negative Perceptions of LMs The practical constraints affecting CG interventions noted above may exacerbate negative perceptions around interventions requiring time but without the provision of incentives or food transfers, especially in light of severe kere/drought. For example, one LM from Androka, Ampanihy, cited a suspicion, and resentment, among the participants: “When doing household visits, some accept and some don’t because they think that my sensitization is just a lie. [Moderator: What makes them say that the conversation is just a lie?] Some say you get paid every month, but we don’t.” While some of these factors, particularly the availability of training materials, are more clearly under the remit and control of CRS, questions of resentment, suspicion, or negative perceptions, including unequal treatment, should not be dismissed. These have resulted directly from the presence of Maharo, even if unintentionally, and reflect a disruption of a social system under strain because of a severe kere/drought. Implications are addressed more in Section 4. Conclusions and Recommendations. Purpose 3: Resilience and Livelihoods Purpose 3: Extremely poor and vulnerable households, communities, and institutions have improved absorptive and adaptive capacities to respond to, manage, and recover from disasters, climate shocks, and chronic vulnerability. 3.4.1. Disaster Risk Reduction Committees Validation Findings Based on the validation exercise, no ELS/CGRC visited has an updated DDR plan in place. While all fokontany have selected a committee, all are still in the initial phases of establishing an active committee. Half of the ELS/CGRCs are still waiting on training and equipment and all seem to be struggling with maintaining momentum, as well as keeping members active and engaged in the process. Androka, the most advanced fokontany and sole fokontany that has a draft DDR plan is faced with the DISASTER RISK REDUCTION CRS has facilitated the creation of fokontany-level DRR groups, called either CGRC or ELS, which share information about weather and climate with communities through methods such as radio and loudspeakers. Fokontany-level DRR committees are led by fokontany chiefs and typically have between 10–15 members representing a range of community members and interests. Fokontany-level DRR committees also collaborate with commune-level DRR committees CCGRCs and Bureau National de Gestion des Risques et Catastrophes (BGNRC) to communicate fokontany-level local knowledge and pass on information. Maharo supports both fokontany- and commune-level DRR Committees through capacity building and supporting the creation of DRR plans. Process Evaluation of the Maharo RFSA in Madagascar Findings 25 challenge of communicating the plan and making it accessible to their community. These findings are, in part, explained below. Implementation Successes and Challenges A key success, previously mentioned, of the DRR intervention is its strong inclusion and active engagement with youth. A few respondents spoke about tasks they had started conducting as part of DRR committees, such as sharing early warning information and construction and reinforcement for vulnerable people, though overall, many groups noted that they had conducted few tasks and were still at the early stages of establishing committees and DRR plans. A further success is that communication systems are being implemented, including sending messages about the recent cyclone and fall armyworms. CRS also distributed phones to community members as part of the communications effort. However, there remain serious questions about the sustainability of the communication system, discussed further below. Implementation Challenges Sustainability of Messaging Though a communications system has been established, it is reliant on a pay-per-use system from Telma, and it is unclear whether partnerships with mobile network operators will remain in place after the end of the RFSA. Ground-truthing confirmed that, as of now, messaging is not sustainable beyond October 2024. As one CRS respondent explained: “We’re trying to build something that expands beyond the life of the project. And if you hire out Telma, which we do now and that's what we use to blast out messages… That's great during the project… But what happens in October 1, 2024 when there's a problem? Until we actually have something that's workable for them beyond life of project, it's a little tough for [BNGRC] to get engaged and excited about it because [they’ve] seen this from actors ever since cell phones became popular, say 15 years ago… We actually have distributed phones to everyone. We actually do some messages in advance of a cyclone. We actually did send a message yesterday about fall armyworms. The gap is that we're doing it off a pay-per-use system from Telma, and we know that can’t extend beyond the life of the project.” Reliability of Communication Chains and Information Sources Separate from sustainability concerns, the DRR committee members interviewed depicted a system of emergency communication that is informal, ad-hoc, and unreliable. For instance, one male CGRC respondent from Anjampaly, Tsihombe, noted, “We get it [news/alerts] from out-of-town colleagues via phone calls… It is difficult to get information because even the radio does not pick up. As soon as we have news, we have to broadcast it.” Similarly, another male CGRC respondent from Androka, Ampanihy, noted, “In fact, we call friends or families living outside our areas such as in Tulear or Antananarivo. That's how we get information because no one has notified us so far. These authority structures have not IMPEL | Implementer-Led Evaluation and Learning 26 Findings yet informed us. Later, a CDD [Conseil Diocésiain de Développement, an IP in Ampanihy] team sends us an SMS to let all our friends know that there is a risk, and we send the same texts or message to all the members.” Responses from ground-truthing suggest that communication systems had been relatively untested at the time of this PE’s data collection, before cyclone Batsirai on February 16th, and that there have since been many SMS blasts, and it is likely that communications will continue to improve. DRR Plan Progress and Support Three out of four ELS/CGRC respondents who spoke about the status of DRR plans specifically allude to a specific plan. One was clear that there was no plan as of yet. Of the three who alluded to having a DRR plan, however, only one, a female ELS member from Itampolo, Ampanihy, could very specifically articulate the nature of the plan: “It's a plan, but we don't have a copy of it. The CDD has its copy. [Moderator: Who co-designed this plan?] It was a group work at that time: CDD/MAHARO, WFP, ADRA, and all the members of the CGRC.” During ground-truthing, CRS respondents confirmed that 265 fokontany have draft plans in place, but some remain unarticulated. Somewhat more circumspect, responses suggest that the plans are more about kere/drought and food insecurity than a climate shock or natural disaster event. Responses from ground-truthing argue that, in fact, the foci of plans are tailored to the most relevant shocks experienced per region. Arguably the most glaring challenge, based on the number and consistency of responses, is the perception that there is limited government and outside support (including from CRS). Some responses acknowledge the support but cannot address specifically what it consists of. Others acknowledge training, but not the material resources with which to follow through. Some illustrative quotes include: “But what about us at the grassroots level, because if there is a natural disaster, we go to the villages in the countryside. How do we get there? On foot, but the distance between us and some villages reaches up to 70 kilometers. How can we get there? … The government does not care about CGRC and the organized structure under the Committee on Risks and Natural Disasters.” (Male CGRC Respondent, Androka, Ampanihy) “[Moderator: What are the outcomes of this particular partnership?] By gaining knowledge through training, we know that the training is about risk and risk management. [Moderator: Can you explain that?] I can't explain; I just know the topic. We are still waiting for the next step.” (Male CGRC Respondent, Anjamply, Tsihombe) In response to these findings, during ground-truthing, it was noted that COVID-19 altered how training and activities were conducted—on the one hand by limiting access and follow-up (possibly leading to some of the frustration noted above), and on the other by creating more work. It was estimated, for example, that CRS conducted four times the anticipated number of trainings to reach fewer participants overall. This stretched resources and created inevitable delays. Process Evaluation of the Maharo RFSA in Madagascar Findings 27 3.4.2. Lead Farmers Implementation Overview and Successes On the one hand, the validation component suggests considerable, if somewhat variable, progress, especially in light of the intense kere/drought. There is observable agricultural activity underway in demonstration plots, new fences, and new goat sheds. (Of note, the demonstration plot in Androka, however, was reported to be too far for most farmers to reasonably access.) On the other hand, the kere/drought and resulting food insecurity (discussed more below) seriously impacted implementation by limiting both the motivation and availability of farmers, as well as negatively affecting the uptake of training given the lack of water to sow seeds or utilize fertilizer correctly. Positively, while farmers were often frustrated by the kere/drought, they also demonstrated creativity and resilience. Two male LFs from Imongy, Tsihombe, for example, described a concerted, collective effort among the community at large: “We put in practice the cultivation techniques we've learned, and we don't stop at the model farm, but also reach everyone's plantation. We made sure to encourage one another and help each other while practicing our activities at the field; all members gave a helping hand to one another, especially while preparing the land of cultivation and the seed planting.” Five separate LFs responded that they perceive themselves as role models in their community. Implementation Challenges Some respondents perceived training as either poorly timed or lacking in relevance to their situation, though this may be exacerbated by or are themselves a function of the kere/drought. Furthermore, more respondents approved of the training in interviews than not. Nevertheless, for illustration: “It would be difficult to implement the techniques that CRS has given us because the soil here is dry; there are no vegetables we can get to feed these animals. In fact, many of our livestock die each year due to the lack of grass and water to drink, even though they are sent to roam around. And now that they are going to be kept in an enclosure during the famine, what shall we give them to eat?” (Male LF, Itampolo, Ampanihy) LEAD FARMERS LFs aim to strengthen famers’ capacity through trainings on agriculture and the environment, including strengthening capacity in dryland, watershed, pest and soil fertility management, improving production, and increasing forest cover. The LF intervention follows a cascade approach, whereby LFs, selected from the community, take part in cluster-level trainings at a Farmer Learning Center (FLC); LFs then share knowledge with farmers in the community through Demonstration Plot groups, consisting of 10–20 community members. Maharo supports FLCs and Demonstration Plot groups with start-up kits and supplies. IMPEL | Implementer-Led Evaluation and Learning 28 Findings “The seeds and materials arrived late, the training was done too soon, so, when it started to rain, everything was already forgotten, so, we had a hard time applying them, among them was the half-moon technique which practice required many conditions.” (Male LF, Imongy, Tsihombe) There are several references to an emerging frustration and jealousy, similar to other interventions. For example: “A lot of members are not enthusiastic when being encouraged, and say: ‘You just make us waste our time by making us attend this meeting, while you get money from the training you got from CDD, you just mess with us, you make us waste the time we should be using for finding food to feed our children, you don't give us money, not even a tiny bit amount.’” (Male LF, Androimpano, Ampanihy) “For us leaders, we are not discouraged yet; we are waiting for CRS to come and distribute goats and equipment to us. But the people we lead are already a bit reluctant and looking to be discouraged and are starting to complain: the other communes have already received their shares and we are the only one that haven't received yet.” (Male LF, Itampolo, Ampanihy) Finally, and somewhat peculiarly, as it is in contrast to what is normally reported, one CRS respondent referenced conflict with the fokontany chief. On the one hand, this again underscores the fokontany chief’s pivotal role in facilitating Maharo, which is consistent with responses elsewhere. On the other hand, this stands in contrast to responses from other interventions that depict the chief as a key ally. “Sometimes the LFs and the chief of the fokontany are not getting on well with each other. And they have some kind of, it's not for all cases, but are some cases when the farmer is not getting on well with the chief fokontany, and he's demotivated because the chief fokontany does not consider him and there is no anchor with the local authority yet, that is what we need. And this is the fact that the head of the chief of the fokontany is not caring about the LFs because sometimes the LFs [think they mean the chief fokontany] want he's somebody he knows to be the LF and just yet the group does not accept that.” (CRS Respondent) Notably, the same respondent described a process of selecting LFs that discounts, even if unintentionally, the input of the fokontany chief, which could be the source of the conflict. Respecting traditional roles is critical, and doing otherwise, even if deemed necessary, is likely to upset local system dynamics that may further inhibit implementation (McNamara 2019). Process Evaluation of the Maharo RFSA in Madagascar Findings 29 In ground-truthing, CRS acknowledged this conflict and described it as somewhat inevitable given that chiefs will want to pick family members, though the LF must meet certain criteria and pass a test (described in Section 3.1.6. Links with Local Governance). One CRS respondent suggested that KFFs could play a mediating role in such conflicts and, likewise, that LFs need to take responsibility for ensuring good relations with chiefs. All of these observations are plausible but yet again underscore how central the chief is to effective activities and the need to better account for potential complications that ensue from purposefully circumventing chiefs, especially if based on assumptions of corruption, paternalism, or favoritism. 3.4.3. Savings and Internal Lending Communities Validation Findings Where it was possible to observe SILC documentation, it was evident that SILCs were diligent in documenting transactions and the group’s balance. The validation exercise also showed that the security of SILC funds is high, with three keys, each held by a different person, needed to get into the cash box. Implementation Successes Respondents across the implementation chain spoke very positively of SILCs, and findings indicate that, of all of Maharo’s interventions, SILCs are among the most successful and sustainable. Many community workers felt that the SILCs were Maharo’s most important component for mitigating the effects of the kere/drought, discussed further in Section 3.4.3.3. Implementation Challenges. A male CM, speaking of SILCs in his area, Tranovaho, Beloha, said, “It is really thriving, and it generates income for the community members in Soamanitra and Ambalanosy, for instance.” A success story recorded from a female SILC member in Itampolo, Ampanihy, was: “It was perfect because when I was not doing SILC, I always struggled in life, but when I did SILC, our lives changed. My children stopped studying in the past because of poverty, as I did nothing but selling firewood. But when I did SILC, it improved.” SILC respondents and validation data indicate that SILCs have active participation from members in contributing and taking loans. For example, documentation from one SILC visited as part of the validation exercise, which had 30 members and had been established eight months ago, showed that all members had taken out loans. The group had collected 8,775,000 Malagasy Ariary (MGA) (c. 2,167 USD) in total, and had accumulated 4,214,000 MGA (c. 1,040 USD) in interest. However, in some cases, SILC SAVINGS AND INTERNAL LENDING COMMUNITIES SILC is a method of Village Savings and Loans (VSL) that CRS has used in the past. Under Maharo, SILCs, typically consisting of 15–20 people, are set up and trained by Field Agents and private service providers. Groups are self-managed, and each develops its own rules relating to contributions, but all groups have a solidarity fund to be loaned or donated to community members in case of birth, illness, or funerals, and a general fund, to which regular rules and interest rates apply. Under Maharo, CRS is also experimenting with separate SILCs for specific groups, such as youth, men, and women. IMPEL | Implementer-Led Evaluation and Learning 30 Findings representatives noted that the kere/drought-affected people’s participation at SILC meetings, due to the competing need to go and find food, detailed more in Section 3.4.3.3. Implementation Challenges. Loan Types–Solidarity Fund Though it was not possible to conduct a full investigation into the types of loans and their uses, it appears that solidarity funds are more utilized than loans from the general fund. SILC members interviewed spoke in more detail about using the solidarity fund to support people dealing with deaths in the family or with newborns. One CRS respondent noted that solidarity fund loans gave additional social benefits, explaining: “For instance, if there is that kind of event where everyone, where the whole communities are gathered … that it's a shame for us if food was not there, because that kind of shame, you don't have enough money for the funeral events. So the support that they received from SILC is really important to them, because it's a kind of prestige, to be able to hold these kind of events thanks to the money from SILC.” Spillover Effects SILCs also showed notable spillover effects. One SILC respondent said that it was possible for non￾members to borrow from the SILC funds if they presented a letter to the group. A CRS respondent recalled that another spillover effect was that even communities not targeted by the SILC intervention were inspired to create SILCs, and wanted SILC interventions. Indeed, one of the SILCs interviewed for this PE, from Behabobo, Beloha, had not, in fact, been set up by CRS but had heard about the SILC methodology and set up their own. The group had grown from 12 to 24 members in its ten months of existence. Implementation Challenges Loan Types - General Fund A few SILC, PiSP, and CRS respondents noted that loans were used to support and finance businesses and fishing livelihoods, and buy livestock for distribution. As one female PiSP from Ampanihy, Androipano, said, “we couldn’t have done the PiSP without getting money from SILC.” However, one male SILC member from Imongy, Tsihombe, noted that a lack of market opportunities prevented members from taking loans to invest in businesses. In his area, people typically grew and produced castor oil, but people “do not dare” to take a loan to expand production as they were not sure of being able to sell it. He added that the lack of loans being taken out prevented the SILC funds from growing by accumulating interest. Though a lack of market opportunities constitutes a challenge, this finding does not represent a particular shortcoming of the SILC methodology overall, given the above-mentioned findings that SILC members benefit greatly from the SILC solidarity funds.4 Kere/Drought Impacts on SILC Of all Maharo interventions, SILC was noted by the most respondents to be the most relevant to the kere/drought context. Community-level respondents interviewed reported that the SILCs gave people the opportunity to access money immediately to solve their needs. This was especially important in the 4 These findings echo other research that has found that the impact of VSL groups is often their solidarity and social component, rather than their livelihood impact, e.g., https://www.fsnnetwork.org/event/village-savings￾and-loan-associations-vslas-emergencies-cares-pilot-taiz-yemen Process Evaluation of the Maharo RFSA in Madagascar Findings 31 kere/drought context as people were unable to earn income through agriculture. Some community workers, including the Miranjaka and LMs, reported that they encouraged community members to become part of SILCs to mitigate the kere/drought. However, SILC respondents interviewed noted that during the kere/drought, some members struggled to make contributions. A female SILC member from Behabobo, Beloha, similarly recalled, “The association has become fragile, and there is little money in the association because some members are no longer able to pay dues.” A few added that some members were leaving SILCs due to an inability to contribute. For example, one male SILC member from Anjampaly, Tsihombe, recalled that membership had reduced from 16 to 13, noting, “[The drought] has had an impact because that's why some members are leaving, those who don't have goats to sell or cassava to sell can't contribute. They say they will leave with nothing.” However, one female SILC respondent from Behabobo, Beloha, noted that their group had been lenient with membership contributions in order to accommodate the effects of the kere/drought on people’s incomes: “We had to make it easier for everyone to pay the membership fee, despite the rules that already exist. Sometimes insufficient membership fees are still accepted. Sometimes, there are those who do not even pay their membership fee for a week, but the association is tolerant of this because famine is known to all.” 3.4.4. Private Input Service Providers Validation Findings Validation confirmed that community members in all nine visited-fokontany had received and used vouchers. PiSPs interviewed understood the voucher process, correctly describing the voucher value amounts and divisions (70,000 MGA (c. 17 USD) for fishing equipment; 76,000 MGA (c. 18.45 USD) in total for farmers, divided into 30,000 MGA (c. 7.35 USD) each for creeping and upstanding crops, and 16,000 MGA (c. 4 USD) for materials), and also understood the amount of compensation. Among voucher customers informally interviewed, it appeared that while some knew which criteria CRS uses to distribute vouchers, others did not. All PiSPs visited had received seeds (or, in one case, fishing equipment). Seeds typically included corn, sorghum, millet, ambatry, and mokuna. Some shops still had undistributed stock - while the exact reasons for this are not clear, this could be due to additional seed stock being distributed via PiSPs as part of the Tabiry emergency program; limited land availability; or that some recipients had not used THE PRIVATE INPUT SERVICE PROVIDER The PiSP intervention consists of providing vouchers for farmers and fishermen for items such as seeds and fishing equipment. Vouchers can only be spent at approved PiSPs. In order to be part of the activity, potential PiSPs are first identified through a community vote. Identified PiSPs are then trained and tested to determine their business and financial readiness, with only those that pass these tests considered eligible to take part in the PiSP intervention. IMPEL | Implementer-Led Evaluation and Learning 32 Findings their vouchers, being unable to plant them due to the kere/drought. Only one PiSP visited had received tools (in Androka, Ampanihy). In Itampolo, Ampanihy, as equipment had not arrived yet, half of the allotted voucher value for equipment was being used to buy additional seeds. Implementation Successes Recruitment PiSPs interviewed reported that they had been elected to their position by the community, though fokontany chiefs were also involved in their selection. They noted that they were selected based on their previous experience with running a business, with a few having worked on previous CRS projects. After training, only PiSPs who passed an exam were permitted to start working. Pass rates did not appear to be particularly high, with two respondents reporting pass rates of only five of 12 (in Anjampaly, Tsihombe) and one in four (in Androipano, Ampanihy). This appears to echo one CRS respondent, who noted that the priority was to ensure that PiSPs were independent and, therefore, sustainable through capacity building on commerce and stock management. Distance Though LFs were not explicitly asked about their experiences with the PiSP intervention, a few nonetheless mentioned that it had solved problems of obtaining seeds from far distances. For example, one male LF from Imongy, Tsihombe, recounted, “Before, we had to walk for a long time in order to be able to plant some seeds, but, now, we can buy the seeds we want to plant without moving from here… We don't have to waste our time going back and forth, and we get to use safe seeds… We get rid of our struggles and worries thanks to the existence of PiSP. Our plants are growing well when there is rain.” A few PiSPs also noted that the seed distribution had eased their burden when trying to find supplies from far away markets, saving them both time and money. One male PiSP from Itampolo, Ampanihy noted, “The goods from CRS help us a lot, because without it, the job as a PiSP might not be possible.” These sentiments could, however, suggest problems for long-term sustainability, if PiSPs struggle to source seeds. Implementation Challenges Voucher Seed Prices The most commonly voiced complaint among voucher recipients was that voucher commodities cost a lot more than market commodities, and as a result, vouchers did not provide farmers with enough seeds. Figure 1 shows a comparison between reported market and PiSP prices, according to respondents. Not all LFs interviewed who spoke of the PiSP seeds were convinced of the additional value of the CRS-provided seeds to justify the price discrepancy, which consequently could cause some resentment among voucher customers. For example, one male LF from Androipano, Ampanihy, stated vehemently, “Seeds we get with 600 Ariary or 700 Ariary the cup at the market, from PiSP we get them with 1500 Ariary or 2000 Ariary the cup, so, imagine what we can get with 76 000 Ariary, how much can we get with that amount? Using vouchers is ok for us, but buying seeds with that price is not. In other words, our collaboration with PiSP is not a success.” Process Evaluation of the Maharo RFSA in Madagascar Findings 33 Figure 1. Reported voucher prices compared to market prices for PiSP commodities Another male PiSP from Itampolo, Ampanihy, noted that the voucher system was not fully understood by all voucher recipients, and this, combined with the elevated seed prices, could lead to some distrust between farmers and PiSPs: “People think that the distribution of seeds is in my interest but people don't think it is in their interest… People have trouble using the voucher because it is not a type of money they use daily… It creates a lot of mistrust between us, yet we are the ones who will have to work together for a long time… It seems to damage my fame.” Seed Sourcing, Timing, and Delivery Due to challenges obtaining seeds caused by the kere/drought, delays due to lengthy procurement processes, and limited collaboration between agencies in Madagascar, CRS established a system whereby PiSPs sourced their own seeds and were later reimbursed. However, some PiSPs struggled as they did not have the start-up cash required. One male PiSP from Anjampaly, Tsihombe, recalled that, because his money had been “stuck in that seed,” he had been unable to construct proper storage for the seeds, and was considering taking on a debt until he had enough cash. Critically, one male PiSP from Tranovaho, Beloha, noted that CRS had not yet paid PiSPs back for the vouchers, which had caused him to incur debt as he had taken a loan from the SILC in order to purchase seeds to sell. He recalled: “There is the delay in payment from CRS, which results in huge interest rates to be paid to Voamami [SILC] and the inability to take a wide range of equipment for sale… I ask CRS to speed up the process of paying the money because the delay causes deep inconvenience to the PiSPs … we PiSPs, are losing money instead of benefiting from sales.” Respondents also noted delays in receiving vouchers and seeds, and noted that seed and voucher deliveries are staggered. Some voucher recipients, therefore, had to travel to PiSPs multiple times, in some cases long distances, in order to reimburse their vouchers. Another problem voiced by voucher recipients was that seed distribution did not correspond to the planting season. Some from Beloha (Beloha), Itampolo (Ampanihy), and Imongy (Tsihombe) noted that the seeds were distributed late, in February, though the planting season is in November and December. By contrast, one male LF from Imongy, Tsihombe, recalled that “The time at which the seeds were distributed, it was done way long before it started to rain, and, as a result, they all died, none of them sprouted.” CRS and LF respondents alike acknowledged the risk of distributing seeds too early before planting, which may lead to seeds being consumed as food. However, CRS respondents at the CLA workshop noted that seasonal rainfall IMPEL | Implementer-Led Evaluation and Learning 34 Findings has been extremely unpredictable in recent years, making it difficult to time seed and voucher distributions appropriately. During ground-truthing and at the CLA workshop, CRS respondents noted that the above problems in seed sourcing and delivery were compounded by the additional rollout of the USAID-funded Tabiry emergency seed distribution, which was also delivered through PiSPs. The expansion from the initially planned 16,000 households under the Maharo PiSP intervention to 60,000 households after Tabiry, as well as delays in funding approval, posed a significant challenge for seed procurement and delivery. Tool Availability Across locations, many PiSPs had not received agricultural equipment and tools to distribute to voucher recipients. These shortages are explained by difficulties finding equipment providers in the Androy region, where commerce and markets are not well developed. While some PiSPs were able to source locally made materials, not all could make this adaptation. CRS acknowledged these challenges and that they were seeking out solutions, including trying to source raw materials, such as iron ore, and reaching out to artisans and youth groups in Abovombay to produce agricultural equipment and tools. Voucher System Some respondents also did not appreciate the prescriptive nature of vouchers, which were limited to seeds only, and also dictated the types of seeds to be bought. Across locations, voucher respondents voiced preferences for different types of seeds. Some seeds, notably mokuna, which was new for many and is used for soil improvement rather than consumption, were not appreciated or even bought. CRS respondents acknowledged that, despite LF trainings on climate-smart techniques such as mokuna, it might nonetheless take time for farmers to adapt these new techniques and crop types. Some respondents, both farmers and PiSPs, explicitly said that they would prefer cash to vouchers to buy their preferred seeds at prices they preferred and expressed some resentment at being forced to use the voucher system. Evidence also suggests that vouchers are not fully appropriate and do not address recipients' needs during the kere/drought. For example, the accessibility of seeds did not change the fact that people could not successfully plant due to the kere/drought. One male PiSP, who sold fishing equipment from Tranovaho, Beloha, recalled that buying equipment was a lower priority than buying food. Though no PiSPs interviewed claimed to condone voucher-selling, a few nonetheless noted that some voucher recipients exchanged vouchers (worth 76,000 MGA (c. 18.45 USD)) for cash (around 30,000 MGA (c. 7.35 USD) or 40,000 MGA (c. 9.80 USD)) in order to buy food. One female PiSP respondent from Androipano, Ampanihy, felt particularly strongly that vouchers were inappropriate during the kere/drought and even were a cause of “suffering,” given their limitations. Process Evaluation of the Maharo RFSA in Madagascar Findings 35 She noted that, in her community, people would prefer in-kind food aid: “CRS should provide food aid, and they should directly distribute the sacks from CRS straight to those vulnerable people, and it doesn't matter even if CRS doesn't establish PiSP here anymore… When you come with a voucher, you are poor. We are really suffering. We cannot buy coffee or tea with a voucher. We can only get seeds through the voucher. When people go home, they are hungry. Then, that person is sad. People asked us, “why are we given vouchers instead of cash?” Then we answered that we don’t know the reason, but we are just here to sell; we don’t know. We signed a contract with CRS and they are free to do what they like to us. We don't know how to explain about it to you but this is the money they gave. Then, accept what they gave. In fact, we want them to share something like WFP Falimandroso so we won't be in difficulty… WFP gave us a sack of rice, five liters of oil, 30 tins of beans. We are happy. We don't know what to do with CRS.” While CRS respondents in ground-truthing sessions felt strongly that vouchers were more appropriate than cash, data from this PE indicate that cash is preferable in the kere/drought context. Even if cash is not spent on seeds but is diverted to meet other pressing needs, fears of cash being misspent are likely unfounded, based on extensive literature demonstrating how recipients spend cash to address livelihood needs and not on “temptation goods” (Evans and Popova 2017). IMPEL | Implementer-Led Evaluation and Learning 36 Conclusions and Recommendations 4. CONCLUSIONS AND RECOMMENDATIONS There is no doubt that Maharo has persevered through kere/drought, COVID-19, and other climate shocks and variability, to lay the foundation for an effective RFSA. Validation and observation data broadly confirm the steady roll-out and good faith implementation of core interventions. Spatial variability in quality is, for the most part, explained by COVID-19 travel restrictions, which further altered training modalities and restricted touch points between CRS and participating communities. CCs, in particular, appear to represent the best in applied systems thinking and complexity awareness, as well as knowledge transfer and practicality, though there remain significant obstacles to its influence on activity interventions. COVID-19 again played a role in this regard, and ground-truthing suggests that disconnects between CC and activities are currently, and intentionally, being addressed. However, such disconnects are not just a function of timing but also embedded norms, expectations, and even perceptions among implementers of the community members with whom they are partnering. Indeed, the above findings—all of which directly affect implementation—can mostly be arranged as a 2x2 table, illustrated in Table 2, that distinguishes between these more logistical versus “ideational” implementation challenges. Table 2. Findings rubric (with illustrative findings reflective of cell location) Practical/“Mechanical” “Ideational” Internal to CRS • Supply chains • Household lists • Telma contracting • Travel restrictions • Voucher distribution • Comments disparaging certain stakeholders, • Use of words like “need to convince” (locals) • Internal perceptions of hierarchies among local staff Maharo Communities • Recruitment • Literacy • Training • Jealousy • Chiefs feeling sidelined • Frustration with time demands and lack of incentive • Preference for cash In fairness, CRS is doing more to tackle the “ideational” complications than is conventional (mostly through the CC intervention). However, evidence suggests that there remains a somewhat staggered, piecemeal approach in places, and complications, like with local officials, for example, are at times easily dismissed, and at times taken seriously. As noted throughout the report, these perceptions and negative feelings—whether among participants, CRS staff, or local leaders—are themselves critical to CRS’ credibility and its ability to effectively implement (and are in part the direct result of implementing interventions to date). Complexity awareness, therefore, requires acknowledging not only the logistical challenges involved in implementation but the complex set of interactions inevitable when a development intervention interfaces with a locally-established social (and social-ecological) system. Process Evaluation of the Maharo RFSA in Madagascar Conclusions and Recommendations 37 Recommendations, both general and intervention-specific, are presented below. General ● Consider what incentive options are possible to motivate community workers and participants. This emerged by far as the most pressing issue at the CLA workshop, where the appropriateness, and the types of incentives, was discussed at length. Across interventions, motivation and time availability for participating in Maharo appear to be the greatest challenge to implementation quality and, consequently, outcomes and impact, a situation exacerbated by the kere/drought emergency context. CRS should thus consider what forms of incentive or motivation can be feasibly rolled out. Additional consideration: o Communication and managing expectations. It is important for CRS to clearly communicate to the entire community the volunteer nature of community workers, as well as disclose potential incentives communities and workers may receive for transparency purposes and to avoid community resentment/misperceptions of favoritism or personal gain from CRS participation. o Type of incentive. Linked to the above, incentive options (e.g., community celebrations at key milestones, performance-based stipends/promotions for community workers, etc.) should be sensitive so as not to exacerbate perceptions of Maharo community workers as privileged by other community members. o “Many hats.” In reality, many community workers hold multiple roles within their community and within the Maharo RFSA itself. The combination of these time demands and requirements on community workers should be accounted for when establishing incentives. o Sustainability. Questions of the long-term sustainability of incentives or payments should be tempered by considering the potential benefits, increased outcomes, and increased momentum that may result from interventions if community workers and participants are well-motivated to participate during the course of Maharo. Expectations can also be managed by communicating clearly to participants what incentives will consist of, their purpose, and their duration. o External actors. It may be valuable or, arguably, necessary for CRS to coordinate with other actors in implementation areas to achieve some degree of alignment and reduce competition between actors for community members’ time and participation. Inclusion ● Consider providing additional training on inclusion for community workers, led by gender, youth, and PWD sensitivity experts, to enhance meaningful, inclusive participation and combat persisting stereotypes. Trainings should also include practical instruction on successful inclusive implementation. Local Governance ● Continue anticipating possible ramifications if working intentionally outside of local governance structures. Communicate reasoning and develop mitigation plans, though continue, first, to collaborate with local authorities, including informal actors, as best as possible. IMPEL | Implementer-Led Evaluation and Learning 38 Conclusions and Recommendations CCs ● Increase emphasis on community ownership and the role of CCs as facilitating a partnership of shared responsibility, rather than purely relaying community needs to implementing partners, to prevent feelings of frustration and better manage expectations. Miranjaka & CGs ● Consider variability in literacy among Miranjaka and LMs, and effects on training. DRR ● Continue to monitor DRR operability, particularly around communication and early warning systems, to ensure movement towards accuracy, timeliness, and availability of information. ● Continue to monitor the finalization and dissemination of DRR plans, working towards a date of delivery. Lead Farmers ● Despite current investments, greater sensitization is required to create acceptance of climate￾smart and soil-generating crops, especially for those that do not bear fruit. (This is heightened by the kere/drought and prevailing food insecurity.) o Demonstration plots could assist in this effort but will require greater oversight and monitoring. The plots would also have to prove effective compared with traditional techniques. ● Review whether staffing is adequate to effectively monitor the 100+ FLCs, the proliferation of which stems from a shift to a cluster model. SILC ● Consider providing financial support to SILCs, which are struggling to build up savings due to a lack of interest accumulation or member contributions due to kere/drought. PiSPs ● Increase communication efforts to clarify reasons for the different prices and added benefits and quality of PiSP seeds. ● Consider providing direct cash to respondents - ample literature spanning context suggests this is effective. Process Evaluation of the Maharo RFSA in Madagascar Annex A: Maharo Interventions 39 ANNEX A: MAHARO INTERVENTIONS Community Consultations CRS conducts Community Consultations (CC) to mobilize community members and ensure community engagement in Maharo interventions. Through CCs, participants themselves are able to select, prioritize, plan, monitor and adapt Maharo’s interventions in their area. This helps to build trust between the RFSA and participants, and ensure that interventions are as relevant and effective as possible (e.g., taking into consideration community members’ ability to attend). It also plays a role in sustainability by developing community responsibility and ownership of Maharo goals and interventions. CCs are undertaken by Community Mobilizers (CMs), who are selected from local communities and trained by Maharo’s Social Behavior Change team. CMs are responsible for carrying out CCs in several fokontany, based on geographic location. CCs strive to be inclusive and capture the thoughts of women, youth, PWDs, and other marginalized groups. CCs are carried out with participants using practical, visually-oriented tools to ensure participation. CRS won an IDEAL-funded grant to develop a participatory system for analyzing and sharing the qualitative data obtained through CCs. They started piloting this approach in 6 selected fokontany since October 2021. Miranjaka Miranjaka Gender Champions work to promote gender equality and inclusion through a range of channels, targeting household and community-level. Their key tasks include: ● Working with traditional leaders to discuss gender gaps and gain leadership support for gender equality tasks (e.g., men’s safe space): ● Serve as community volunteers to conduct household visits and coordinate gender messages ● Conduct home visits to spread messaging such as GBV prevention and referring people experiencing violence to protection services, joint decision-making, equitable roles and responsibilities, access to and control over household chores and assets ● Hold Safe Space discussions with targeted participants, such as youth groups and women (specifically CGs) ● Hold ‘Husbands/Dad’s School’ every quarter to promote a ‘papa model’ (promotes positive masculinity) ● Work with KFF (VDC) to ensure gender equality, inclusivity, and GBV prevention and protection through Komity Miaro Zo approach (human rights committee within KFF members) Miranjaka also receive capacity building from CRS staff to ensure the effectiveness of their intervention, such as gender norms, concepts of women’s empowerment and positive masculinity, and training on how to conduct tasks such as home visit and safe space discussion techniques. Miranjaka are also supervised by staff from the SDPPSPF (Service de District de Population, Protection Sociale, et Promotion des Femmes/District Population Service, Social Protection, and Women’s Empowerment). IMPEL | Implementer-Led Evaluation and Learning 40 Annex A: Maharo Interventions Care Groups Care Groups (CGs) is a community-led behavioral change approach primarily targeting pregnant women, breastfeeding women, and women with children under 2 years old. CGs consist of 10-15 volunteer Lead Mothers (LMs), who work to promote best practices in health, nutrition, water, sanitation and hygiene (WASH), early childhood development, and changing gender roles around childcare in their communities. LMs’ tasks include attending trainings through the Care Group, conducting household visits, and collaborating with other community actors, such as SILCs, Producer Learning Centers, and Miranjaka to share information and promote good health and nutrition. LMs are selected by a Reny Mpiara-Monina (RMM), a group of 10 to 15 pregnant and lactating women, women with children aged <2 years, and adolescent girls from their same neighborhood. LMs attend monthly Care Group meetings with other LMs to receive training on health and nutrition topics, which they then cascade to the RMMs. CGs are trained and mentored by Field Agents, each of whom supports 6 CGs. CGs also work with Community Health Workers (CHWs). CHWs assist to mobilize communities and women to participate in CGs and identify a LM. CHWs also attend monthly Care Group Meetings, serving as ‘local supervisors’ and a resource that LMs can approach with questions. Disaster Risk Reduction Committees Under Maharo, CRS has facilitated the creation of fokontany-level Disaster Risk Reduction (DRR) groups, called either CGRC (Comité de Gestion des Risques et des catastrophes) or ELS (Equipe Local de Secour). These committees help to share information about weather and climate with community members through methods such as radio and loudspeakers. They also meet monthly with commune-level DRR committees, known as CCGRCs (Comité Local de la Gestion des Risques et Catastrophes) to communicate fokontany-level local knowledge. CCGRCs are responsible for passing information to district and higher levels of the BGNRC (Bureau National de Gestion des Risques et Catastrophes / National Office of Risk and Disaster Management). CGRCs/ELS typically have between 10 and 15 members which may include: 1. Fokontany chief 2. Elders 3. Representative of Religious and Traditional Associations (ART) 4. Representative of the community health system (CA) 5. Representative of the community natural resources management system (e.g. VOI, VNA, KASTI, or other NRM committee forms) 6. Representative of the community infrastructure management (AUE, AUP) 7. Representative of the KFF committee at village level 8. Private sector representatives (e.g. traders, wholesalers, operators) 9. Representative of women's associations (Miranjaka) 10. Representative of the targeting and support committee for the most vulnerable 11. Youth representative Process Evaluation of the Maharo RFSA in Madagascar Annex A: Maharo Interventions 41 12. A member of the municipal council living in the fokontany 13. Representative from other services in the fokontany (e.g. teachers, health workers) 14. Representative of other local groups and citizens from the fokontany (e.g. farmers, fishers, SILCs) 15. Representative of people with disabilities (PWDs) Maharo supports both commune-level and fokontany-level DRR Committees through capacity building, and providing support to create and update DRR plans. Lead Farmers The aim of the Lead Farmers (LF) intervention is to strengthen the capacity of farmers with regard to agriculture and the environment, such as: strengthening capacity in dryland, watershed, pest, and soil fertility management; improving production; and increasing forest cover. LF facilitate Demonstration Plot (DP) groups. These are at the fokontany level and consist of 10-20 learners, ideally from the same village, who meet regularly during the farming season. These groups, led by the LF, identify and prioritize problems they are facing, brainstorm and analyze solutions and design action plans for these; and implement action plans, with the support of field officers. LF also take part in cluster-level Farmer Learning Centers (FLC), which consist of all LF from that cluster. LF are trained on the FLC and LF approach, technical innovations, group facilitation and leadership techniques. Maharo also supports FLCs and DP groups with start-up kits and supplies for the LF. Savings and Internal Lending Communities Savings and Internal Lending Communities (SILC) is a method of Village Savings and Loans (VSL) that CRS has used in the past. SILCs typically have between 15 and 30 members. SILCs are self-managed through a General Assembly and a Management Committee. Each SILC has their own rules regarding the frequency and quantity of contributions. SILCs receive training from Field Agents and private service providers (PSPs). Field Agents are, in turn, supervised by Field Supervisors, who aim to train and certify Field Agents to become PSPs. As PSPs, Field Agents will operate as independent service providers on a market-led basis, and will continue to offer their training and services to communities for a fee. There are no rules on how members should spend savings. However, SILCs should receive messaging and sensitization from other actors, such as CGs, LMs, and Producer Learning Centers, on aspects such as financial management, health, nutrition, WASH, and livelihoods. CRS is experimenting with separate SILCs for specific groups, such as youth, men, and women. Private input Service Provider The two main tasks under the PiSP are vouchers and promoting youth entrepreneurship. IMPEL | Implementer-Led Evaluation and Learning 42 Annex A: Maharo Interventions The voucher program aims to promote the private sector and increase the availability of agricultural inputs, especially seeds, for farmers. Vouchers are distributed to participant farmers/fishermen for items such as seeds and fishing equipment. These vouchers can be spent from approved PiSPs, who are selected based on CC, including a community vote to identify who is considered trustworthy to sell items on a voucher basis. One or two proximity PiSPs are targeted in each fokontany. The process of establishing PiSPs entails: ● Receiving information from communities ● Identifying PiSPs who sell seeds ● Contracting selected PiSPs through contracts or letters of commitment ● Collaborating with suppliers (CTAS, Agrivet) ● Training PiSPs in store management, use of vouchers, and tools to use – this training is done by community-based ● Ongoing monitoring and support of PiSPs The types of PiSPs currently operating are: ● PiSP Boutiques (selling seeds and agricultural inputs) ● PiSP Phyto (selling phytosanitary treatment items) ● PiSP Artisan (blacksmiths) ● PMS (seed producers) Under the youth component, youth are given capacity buildings regarding income generating activities and entrepreneurship via Youth Clubs. From October 2021, Youth First also started distributing small loans to youth entrepreneurs. Process Evaluation of the Maharo RFSA in Madagascar Annex B: Evaluation Questions and Methodology 43 ANNEX B: EVALUATION QUESTIONS AND METHODOLOGY This Annex provides additional information pertaining to the methodology of this PE, including: the full list of evaluation questions; further details of field data collected, including sampling selection and respondent lists; and details of how ground-truthing sessions and the joint CLA workshop were held. Evaluation Questions Main evaluation question: How well has Maharo been effectively implementing core and critical pieces of its intervention in order to address the RFSA objectives of enhanced food security and resilience? Sub-questions include: ● Are implementation interventions gender and context sensitive, and inclusive of key, local partners, leading to greater community engagement and adoption? ● What considerations within the implementation plan have been most effective at minimizing friction (or disagreement between stakeholders) and facilitating progress? ● Which factors, whether contextual or programmatic, interfere with more effective implementation? Kere/Drought-specific sub-questions include: ● To what extent has the kere/drought measurably altered programming, or diverted resources, in order to deliver emergency assistance? ● How has the kere/drought directly affected the participation and wellbeing of key local partners, and local decision-making? ● Which intervention components remain most viable, and/or most important in light of the crisis, separate from emergency food assistance? Sampling Intervention Sampling Since an exhaustive tracking of each activity under each purpose was not feasible for this PE, Causal Design determined a sample of seven activities to examine in communications with BHA/Washington, the USAID/Madagascar Mission, and IPs. Interventions were selected based on their representativeness and centrality to Maharo and its Theory of Change. Selected activities and their Purposes are shown in Figure 2. IMPEL | Implementer-Led Evaluation and Learning 44 Annex B: Evaluation Questions and Methodology Figure 2. Selected interventions and their Purposes Respondent Sampling Causal Design used a purposive sampling approach for KIIs and validation exercises. This entailed sampling respondents across the implementation chain from each activity intervention and interviewing respondents from CRS, implementing partners, community workers, and participants (through the validation exercise). Causal Design also sampled across geographies, using a maximum variance sampling approach. Sampling focused on district and commune levels based on where specific activities were carried out, and with the goal of including both littoral and interior fokontany. The goal of distributing responses by geography is to gain a comprehensive rather than idiosyncratic depiction of implementation quality. As a qualitative inquiry, this PE cannot assume a representative sample. However, the use of both purposive and maximum variance sampling ensured the relevance of selected respondents, and as much breadth as possible from the respondents and fokontany selected. Key Informant Interviews The tables below present the total numbers of KIIs interviewed for this PE. Table 3. CRS and implementing partner respondents interviewed Respondent Type Number CRS Staff 12 Implementing Partners 3 Total 15 Purpose 2: Nutritional status of PLW, adolescent girls, and children <5 improved and sustained Purpose 1: Formal and informal local governance systems deliver inclusive, equitable, and effective social services and safety nets to most vulnerable households and individuals Purpose 3: EP and VHH households, communities, and institutions have improved absorptive and adaptive capacities to respond to, manage, and recover from disasters, climate shocks, and chronic vulnerability Care Groups SILC PISP Miranjaka DRR Committee Building Community Consultations Lead Farmers Process Evaluation of the Maharo RFSA in Madagascar Annex B: Evaluation Questions and Methodology 45 Table 4. Fokontany and commune-level respondents interviewed Intervention Respondent Ampanihy Beloha Tsihombe TOTAL Androka Itampolo Androipano Beloha Tranovaho Behabobo Tsihombe Anjampaly Imongy Community Consultations Community Mobilizers 1 1 1 1 1 1 1 1 1 9 Care Groups Lead Mothers 1 1 1 1 1 1 1 1 8 Miranjaka Miranjaka 1 1 1 1 1 1 1 1 1 9 DRR Committees CCRG 1 1 1 1 1 1 1 7 ELS 1 1 1 1 1 1 1 1 1 9 Lead Farmers Lead Farmers 1 1 1 1 1 1 1 1 1 9 SILC SILC Members 1 1 1 1 1 1 1 1 1 9 PiSP Vouchers PiSPs 1 1 1 1 1 1 1 7 Total 8 8 7 6 8 7 7 8 8 67 IMPEL | Implementer-Led Evaluation and Learning 46 Annex B: Evaluation Questions and Methodology Validation Exercise At BHA’s request, the PE also included a validation exercise, conducted by Consultants Associates and supervised remotely. The aim of the validation was to provide additional verification that interventions are taking place according to Maharo’s intended cascading system. The validation exercise, while narrow in scope, nevertheless provided additional qualitative data that built on findings from the KIIs. A validation tool was developed, which entailed a series of observations or informal KIIs with participant respondents. Enumerators were instructed to conduct at least one exercise or interview activity for each intervention in a random fokontany from all nine communes sampled. This was not always possible due to challenges with availability of key respondents on the ground at the time the exercise was conducted. Table 5 shows the validation exercises conducted across all sampled communes. Table 5. Validation exercises conducted per intervention Key ✔ Validation exercise conducted in one fokontany ✔✔ Validation exercise conducted in two fokontany x Validation exercise not conducted n/a Validation exercise not conducted because the intervention does not occur in that commune * Validation exercise not conducted because the intervention has not been conducted ** This exercise was not on the original validation tool, but the enumerator had a chance to observe this in the field in some instances. Intervention Validation Exercises Ampanihy Beloha Tsihombe Androka Itampolo Androipano Beloha Tranovaho Behabobo Tsihombe Anjampaly Imongy SILC Observe: Any SILC documentation x ✔ ✔ x ✔ x x x ✔ Observe: SILC meeting/training x ✔ x x x x x x x PISP Speak to: Voucher customers ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Process Evaluation of the Maharo RFSA in Madagascar Annex B: Evaluation Questions and Methodology 47 Intervention Validation Exercises Ampanihy Beloha Tsihombe Androka Itampolo Androipano Beloha Tranovaho Behabobo Tsihombe Anjampaly Imongy Observe: Voucher transactions in PISP shop ✔ ✔ ✔ x x ✔ x x x Observe: PISP notebook with voucher and seed registered info** x x x x x ✔ x ✔ x Care Groups Speak to: RMM members ✔ x*** ✔ ✔ ✔ ✔ ✔ ✔ ✔ Observe: RMM training session x x x x x x ✔ x x Lead Farmers Speak to: Lead Fisher group members x x n/a n/a ✔ n/a n/a x n/a Speak to: Demonstration Plot members ✔✔ ✔ ✔ ✔✔ x ✔ ✔ ✔ ✔✔ Observe: Demonstration Plot tour ✔✔ ✔ ✔ x x ✔ ✔ ✔ ✔✔ Observe: Demonstration Plot group training session x x x x x x x x x Miranjaka Speak to: Safe Space discussion participants ✔ ✔ ✔ x* x* x* x* x* x* Observe: Safe Space discussion x x x x* x* x* x* x* x* IMPEL | Implementer-Led Evaluation and Learning 48 Annex B: Evaluation Questions and Methodology Intervention Validation Exercises Ampanihy Beloha Tsihombe Androka Itampolo Androipano Beloha Tranovaho Behabobo Tsihombe Anjampaly Imongy Speak to: Dad’s School participants ✔ ✔ ✔ x* x* x* x* x* x* Observe: Dad’s School x x x x* x* x* x* x* x* Speak to: KFF members ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Observe: KFF meeting x x x x x x x x x Community Consultations Speak to: Community consultation participants ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ Observe: Community Consultation ✔ x x x x x x x x DRR Committees Observe or confirm existence of updated DRR plan x* x* x* x* x* x* x* x* x* Process Evaluation of the Maharo RFSA in Madagascar Annex B: Evaluation Questions and Methodology 49 Ground-Truthing Sessions After all primary data had been collected and analysis was underway, Causal Design organized a series of ground-truthing sessions between May 3 to May 6, 2022 to share preliminary findings with CRS technical teams and key staff. The aim of sessions was to verify findings and obtain additional detail and context where necessary. As such, ground-truthing sessions served in part as a data collection exercise themselves for Causal Design, and proved highly fruitful. Sessions were held remotely on Zoom, with simultaneous translation, and lasted between 1 and 1.5 hours. Sessions were divided by Purpose and Intervention and purpose as follows: ● Session 1. Care Groups and Miranjaka activities (Purposes 1 & 2) ● Session 2. SILC, PiSP and Lead Farmer activities (Purposes 2 & 3) ● Session 3. DRR Committees (Purposes 1, 2 & 3) ● Session 4. Community Consultations (Purposes 1, 2 & 3) and Learning Collaborating, Learning, and Adapting Workshop Causal Design organized and facilitated a four-day joint Collaborating, Learning and Adapting (CLA) workshop, attended by technical teams from both CRS and ADRA, implementers of the FIOVANA RFSA. Each day included plenary presentations and discussion, as well as breakout sessions. The majority of discussion took place in French or Malagasy, with simultaneous translation for English-only speakers. The workshop was held between June 13 and June 16, 2022, after BHA and CRS had provided an initial review of the Maharo PE report. Central themes for each of the four days were as follows: ● Day 1. Reflecting on and unpacking PE findings and recommendations ● Day 2. Identifying next steps/actions for incorporating process monitoring insights into programming ● Day 3. Discussing cross-program learnings and adaptations ● Day 4. Chart the path forward and making action plans for adaptations and integration into other RFSA learning and M&E processes The CLA workshop, while primarily intended to facilitate the creation of action plans following from the PE, also provided additional context for findings and refinement for recommendations, which were integrated into this PE report. IMPEL | Implementer-Led Evaluation and Learning 50 Annex C: Inclusion Findings by Intervention ANNEX C: INCLUSION FINDINGS BY INTERVENTION Table 6. Inclusion findings by intervention Gender Youth PWDs Community Consultations • Differences in topics of interest: o Women: families, daily income generating activities, gender, SILC o Men: Livestock, agriculture, infrastructure, fishing • Still some reluctance for women to express themselves by some Beloha and Tsihombe respondents • Mixed data on youth’s participation: • Around half CMs reported that young people active and express their thoughts • One noted reluctance to talk in meetings • Youth from rural communes are less vocal, due to less education • Young women could be shy to speak in front of male CMs • No PWDs in 2 CCs (due to no PWDs in the area) • Call on PWDs to ensure they contribute and “encourage others not to treat them as outcasts” • Almost all CMs aware of awareness principles, but one (Androimpano) said he had had no training and had little to say • Reluctance to speak up or voice disagreements in meetings, especially women: “The difference is disabled women don't participate much because they are feeling down; I mean, they think that, on this planet, nothing is made for them. Disabled men, on the other hand, are enthusiastic about sharing their opinions.” (CM, Behabobo, Male) Miranjaka • Mixed results on men’s engagement/ participation—some enthusiastic, some not • Only Miranjaka from Ampanihy district had conducted Safe Space or Dad’s Schools • Women sometimes reluctant to speak up in meetings • Miranjaka target youth for HH visits and meetings • Youth more open to Miranjaka message • Mixed results on participation: • Youth don’t attend meetings because they are busy with livelihoods • Youth have more time to attend meetings as have less work to do Findings for Miranjaka and Care Groups • LMs & Miranjaka Spread messages about PWD inclusion and rights • PWDs sometimes reluctant to speak up in meetings • Need for more training on PWDs, going beyond equal rights to practical advice on how to incorporate PWDs, particularly Process Evaluation of the Maharo RFSA in Madagascar Annex C: Inclusion Findings by Intervention 51 Gender Youth PWDs Care Groups • Attempts to include men in sensitizations—no disagreement, but largely considered women’s topics • Men less time available due to agricultural work • Youth included in Care Groups • Some Lead Mothers are youth, and perceived to be dynamic and influential those with non-mobility-related disabilities • Need for more direct support for PWDs • Some Lead Mothers were PWDs • Some reticence to work with PWD expressed by LMs SILC • Many women in SILCs, some in leadership positions • More utilized by women than men • Men able to contribute more • Women sometimes reluctant to speak • Youth involved in youth-only SILCs • Reluctant to speak or disagree with elders • Male youth reluctant to take loans, due to fears of misspending funds Findings for SILC, PiSP and LF: • Little participation/membership from PWDs, though some exceptions • PWDs sometimes shy in meetings and don’t speak up • Need for more direct support for PWDs • Physical handicap barriers a particular problem: o PiSP equipment not suited for PWDs o Little recognition of PWDs contribution for LF interventions PiSP • More women than men were voucher recipients as queuing for vouchers considered women’s work • Few youth PiSPs, due to the need for start-up costs to buy seeds and set up businesses Lead Farmers • Both men and women involved in trainings and participated in work on the demo plot (though this followed gendered lines) • Women sometimes reluctant to speak • Taboos against women engaging in livestock • Few women Lead Farmers as they were perceived to not be influential enough • Some youth receive LF’s trainings, but generally more older people than youth • Physical strength an asset • Few youth Lead Farmers as they were perceived to not be influential enough IMPEL | Implementer-Led Evaluation and Learning 52 Annex C: Inclusion Findings by Intervention Gender Youth PWDs DRR Committees • Both fokontany and commune-level committees include women (often at least half) • Mixed results in women’s participation: o Women are active participants in meetings and taking action o One ELS from Androimpano noted Miranjaka’s successful at encouraging women’s participation • Women reluctant to make decisions • Gendered divisions of labor affected ability to participate • Some problematic attitudes among DRR members in Androka: “The problem I personally find is that women really boast when given a platform and given a chance… When something happens, they boast a lot because they get a chance to express themselves.” (CGRC, Androka, Male) “Opinions from women can never be superior compared to men. [Moderator: Why?] That's God's organization.” (ELS, Androka, Male) • Variation in number of youth in DRR Committees—some have many youth, some none at all • In one commune (Behabobo), no youth due to perceived immaturity and alcoholism, which would “harm” the association • Where youth present, noted to be active in committee decisions and tasks • Around half ELS, and a few CGRCs, sampled had no PWD members • In some cases represented indirectly by family members • Difficulties participating in tasks and attending meetings • Mixed results on active participation and reluctance to speak in meetings • PWDs in committees request direct support Process Evaluation of the Maharo RFSA in Madagascar Annex D: Works Cited 53 ANNEX D: WORKS CITED Evans, D. & Popova, A. (2017). Cash Transfers and Temptation Goods. Economic Development and Cultural Change, 65(2). https://doi.org/10.1086/689575 Klick, M. (2016). The Effect of State-Local Complementarity and Local Governance on Development: A Comparative Analysis from Post-War Guatemala. World Development, 82, 1–13. https://doi.org/10.1016/j.worlddev.2016.01.005 McNamara, T. (2019). The Limits of Malawian Headmen’s Agency in Co-Constructed Development Practice and Narratives. Journal of East African Studies, 13(3), 456–484. https://doi.org/10.1080/17531055.2019.1599196 Saito, F. (2008). Decentralization and local governance: Introduction and Overview. In F. Saito (Ed.), Foundations for Local Governance (pp. 1–24). Leipzig, Germany: Physica-Verlag. IMPEL | Implementer-Led Evaluation and Learning 54 Annex E: Data Collection Tools ANNEX E: DATA COLLECTION TOOLS Added as a PDF. CRS, Youth First and Humanity & Inclusion KII Tool Communication Protocol and Informed Consent Introduction and program information: Hello, my name is ________________. I work with Causal Design, a research organization conducting a process monitoring evaluation of CRS’ Maharo program. Specifically, we are examining the implementation of Maharo projects and factors that might be affecting implementation of these activities, either positively or negatively. In particular, we would like to talk to you about implementation of the SILC activities. We will be examining impact and effectiveness at a later time. We are specifically examining the following Maharo activities: ● Care Groups ● Disaster Risk Reduction Committees ● Lead Farmers ● Miranjaka (Gender Champions) ● Savings and Internal Lending Communities (SILC) ● Private Input Service Providers (PiSP) ● Community Consultation Goal of this interview: We have invited you to this interview as one of Maharo’s key implementing partners. We would like to hear your perspective on how the Maharo support has been delivered, and your experiences implementing activities under the Maharo program. We would like to hear about both facilitators and obstacles, with the aim of ascertaining how implementation could be improved in the future. We would also like to hear about the inclusion of women, youth, and people with disabilities; and effects of the drought/kere and COVID-19 on program/activity implementation and activities. We aim for the interview to take up around 1 hour of your time. Managing Expectations: We cannot guarantee that every suggestion raised during this interview will be acted upon, but all suggestions will be analyzed and those that occur frequently may be taken forward as recommendations to program leadership. The analysis results will be shared with CRS, and it will be up to their discretion to share them with you and others. Informed Consent: With your permission, I will be recording this interview. What you say during this interview will be shared with a team of analysts, but nobody else. We will remove your name and other personally identifying information from the recording, and your name will not appear in the analysis or any report. So please, speak openly to us – your voice is very important to help us learn about the successes and challenges of this program. For FGDs/interviews with more than one respondent: We would like to hear from all of you, so I encourage you all to speak openly, even if you may disagree with what another participant is saying. I ask that you please speak one by one, so we can record all of your answers accurately. We will try to make sure everyone has a chance to speak. Do you agree with being interviewed by us, and to our conversation being recorded? [CONFIRM CONSENT] Thank you very much for agreeing to take part in this interview. Do you have any questions before we begin? Interview Information Enumerator/s Interview Date Original Interview Language French Malagasy English Approx Interview Length Delete as appropriate <30 mins 30mins 45 mins 1hr 1hr+ Respondent Information Respondent Type Delete as appropriate ● HI – Humanity & Inclusion ● Youth First ● CRS CoP / CG / CC / Miranjaka / DRR / LF / SILC / PISP / Environmental Compliance Number of Respondents Interviewee Name/s Gender District Leave blank if not relevant Commune Leave blank if not relevant Fokontany Leave blank if not relevant Observations Please provide details about the interview setting, interviewee’s responsiveness, and any other relevant information Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 55 Interview Questions Background I’d like to start by hearing some background information about your involvement in the Maharo program. 1. Which components of Maharo are you particularly involved in? Note to Enumerators: Probe for our select 7 activities above and concentrate on these. 2. What is your particular role within Maharo and how long have you been involved? 3. From your understanding, what is the main purpose of your involvement and activities/[key activity] as part of the Maharo program? 3.1. What are your key roles and tasks? 3.2. What are your objectives as part of the Maharo program? 3.3. Do you think these tasks and objectives are reasonable given the context you/Maharo works in, and are you optimistic/pessimistic about realizing them? ■ Why?/Why not? Effectiveness of CRS Support Note to Enumerators: Skip this section for CRS Respondents 4. What support for implementation have you received from CRS so far? What did this consist of? (e.g. trainings, material or financial support) 4.1. What are your reflections, positive or negative, in the way this support was delivered? Could anything have been improved, and why? Please be as specific as possible. (e.g. format of trainings; any instances of delays) Effectiveness of Implementation Now I would like to hear about your experiences implementing activities as part of the Maharo program. Note to Enumerators: If respondents talk only about the effectiveness or impact of the activities, steer them back to the topic of how activities are implemented, or their role in and experience of carrying out these activities. 5. How did your organization (Youth First / HI / CRS) make the transition from planning Maharo activities (thinking about them ‘in theory’) to their actual implementation? What are the essential, concrete ‘next steps’ involved in this transition? 5.1. Which steps or actions during this stage do you think are fundamental to making implementation work best? IMPEL | Implementer-Led Evaluation and Learning 56 Annex E: Data Collection Tool 5.2. What aspects of the transition from planning to actual implementation worked well, and which were more complicated? Why? How could they be improved? 6. Are there any other organizations or actors (local, formal or informal, like chiefs or womens’ groups, government bodies, businesses or even other aid agencies) that are critical to you being able to effectively carry out your work? 6.1. Who/what are they, and how are they essential to effective implementation? 7. What are some of the key considerations, in particular with regards to the communities where activities take place, that you included in your preparations to ensure implementation runs smoothly? (Eg: gender dynamics, poverty, focus of economy (ie. agriculture, fishing)terrain, out migration, etc.) 7.1. How do you accommodate them/factor them into your planning, or implementation now? Can you be specific? 8. Thus far, what have been some of your priorities, or requirements, for start-up/laying the groundwork for Maharo implementation? (e.g. securing partnerships, recruitment, establishing lines of communication, procurement) 8.1. What aspects of start-up/laying the groundwork has worked well, and which have been more complicated? Why? How could they be improved? 9. Other than these, what are your other critical priorities or requirements for implementing or rolling out your key activities? (e.g. communication, logistics, coordinating with communities) 9.1. Which aspects of establishing these have worked well, and which were more complicated? Why? How could these be improved? Linkages 10. Above we asked about other organizations and actors with whom you partner in order to carry out your work. May you clarify: 10.1. How effective would you describe these links with key partners at the moment? 10.2. What are some effective elements of these partnerships? What do you consider important for ensuring you work together to implement activities effectively? 10.3. What specific outcomes have emerged out of these partnerships? What factors were responsible for these outcomes? Please be as specific as possible. 11. What limitations do you see in your partnerships at the moment? How could they be improved, or what would you change moving forward? (e.g. communications, type of support, expanding to other/different partners, geography or sectoral knowledge). Adaptations 12. How does your implementation in practice compare to how you expected or planned it would roll out? Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 57 12.1. In particular, how did COVID-19 affect your implementation? 12.2. What were the key outcomes of the refine-and-implementation (R&I) period? ■ What specific adaptations did you make based on your learnings/observations during this phase? ■ If you could not adapt, why not? What resources or other support were you missing? 12.3. Are there any other expected or unexpected local conditions or obstacles that have affected implementation thus far? Please describe them. How did you adjust? ■ If you did not/could not adapt, why not? What resources or other support were you missing? 13. Have you been unable to conduct any activities you wanted or that were planned? Why? 14. Other than the refine and implementation (R&I) period, what systems are currently in place to continue to adapt or ensure implementation? What do these look like? 14.1. Are there any examples of this in practice? Drought 15. How has the drought/kere affected implementation? 15.1. What changes or adaptations have you had to make to the way you implement because of the drought/kere? Have they been effective? 15.2. With households unable to support agriculture in the face of extreme drought, what are the effects on your Maharo activities/implementation? Probe for both implementation and outcomes. 15.3. Were there any aspects of implementation or activities that were completely derailed by the drought? Which ones, and how? 16. For activity-specific respondents only (everyone except COP, Country Representative, and Environmental Compliance): In your opinion, what were the most important aspects or contributions of your activities in the drought/kere context? 17. For all respondents: What were the most important aspects of the Maharo program as a whole in the drought/kere context? (Probe for: activities other than the emergency food assistance) Inclusivity Gender 18. The baseline findings (refer to this in more detail if that is not clear) suggest that women in this region possess a high degree of ‘agency’ (ie, influence over household decisions). In what areas of household decision-making do women have more or less influence or a say? IMPEL | Implementer-Led Evaluation and Learning 58 Annex E: Data Collection Tool ■ If more information needed: Causal Design was commissioned to conduct a baseline study for Maharo under the IMPEL/Implementer-Led Design, Evidence, Analysis and Learning (IDEAL) activity. Our baseline findings on gender suggested that women had more agency than CRS/ADRA’s previous gender analyses suggested. 19. How have you tailored/organized and conducted your activities and implementation to ensure equal access, benefits, and participation for both men and women? 19.1. What does men and women’s participation in activities currently look like? In what aspects do they participate, and in how do they participate? ■ How does participation differ between men and women? 19.2. What considerations are important to ensuring equal access for both men and women when implementing programs (not just designing them!)? 19.3. What limitations or gaps still exist in implementation efforts to ensure equal access? How might they be overcome? PWD 20. How have you tailored/organized and conducted your activities and implementation to ensure equal access, benefits, and participation for people with disabilities (PWDs)? 20.1. What does PWD participation in activities currently look like? In what aspects do they participate, and in how do they participate? 20.2. What considerations are important to ensuring equal access for PWDs? 20.3. What limitations or gaps still exist in implementation efforts to ensure equal access for PWDs? How might they be overcome? Youth 21. How have you tailored/organized and conducted your activities and implementation to ensure equal access, benefits, and participation for youth? Note to Enumerators: ‘Youth’ refers to adolescents and young people aged 10 to 29. 21.1. What does youths’ participation in activities currently look like? In what aspects do they participate, and in how do they participate? ■ How does participation differ for young men and young women? 21.2. What considerations are important to ensuring equal access for youth? 21.3. What limitations or gaps still exist in implementation efforts to ensure equal access for youth? How might they be overcome? Is there anything else you would like to say or ask before we conclude the interview? Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 59 Thank you so much for your time! We greatly appreciate your perspective on all of this! If you have any further questions or concerns, please do not hesitate to contact Sophie Turnbull (sophie.turnbull@causaldesign.com) or Matt Klick (matthew.klick@causaldesign.com). Approx Interview Length Delete as appropriate <30 mins / 30mins / 45 mins / 1hr / 1hr+ IMPEL | Implementer-Led Evaluation and Learning 60 Annex E: Data Collection Tool Community Mobilizer KII Tool Background Information for Enumerators Community Consultation CRS conducts Community Consultations (CC) to mobilize community members and ensure community engagement in Maharo activities. Through CCs, beneficiaries themselves are able to select, prioritize, plan, monitor and adapt Maharo’s activities in their area. This helps to build trust between the project and beneficiaries, and ensure that activities are as relevant and effective as possible (e.g. taking into consideration community members’ ability to attend activities). It also plays a role in sustainability by developing community responsibility and owernship of Maharo goals and activities. CCs are undertaken by Community Mobilizers, who are selected from local communities and trained by Maharo’s Social Behaviour Change (SBC) team. Community Mobilizers are responsible for carrying out CCs in several fokontanys, based on geographic location. CCs strive to be inclusive and capture the thoughts of women, youth, PWDs and other marginalized groups. CCs are carried out with beneficiaries using practical, visually-orientated tools ensure participation. CRS won an IDEAL-funded grant to develop a participatory system for analyzing and sharing the qualitative data obtained through CCs. They started piloting this approach in 6 selected fokontanys since October 2021. Communication Protocol and Informed Consent Introduction and program information: Hello, my name is ________________. I work with Consultants Associates, a Madagascar-based data collection firm, in partnership with Causal Design, a research organization. We are conducting a process monitoring evaluation of CRS’ Maharo program, which aims to enhance food security and resilience in three districts of Madagascar: Ampanihy, Beloha, and Tsihombe. The Maharo Program includes a range of activities in a range of sectors, including a Community Consultation component. We specifically are examining how Maharo projects are being implemented, and anything that might be affecting implementation of these activities, either positively or negatively. We will be examining impact and effectiveness at a later time. Goal of this interview: We have invited you to this interview to share your views on the Community Consultations and CRS’ support for these committees under the Maharo program. We would like to hear your perspective on how Maharo’s support has been delivered, and your experiences implementing activities as a Community Mobilizer. We would like to hear about both facilitators and obstacles, with the aim of ascertaining how implementation could be improved in the future. We would also like to hear about the inclusion of women, youth, and people with disabilities in Community Consultations; and effects of the drought/kere on program/activity implementation and activities. We aim for the interview to take up around 1 hour of your time. Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 61 Managing Expectations: We cannot guarantee that every suggestion raised during this interview will be acted upon, but all suggestions will be analyzed and those that occur frequently may be taken forward as recommendations to program leadership. The analysis results will be shared with CRS, and it will be up to their discretion to share them with you and others. Informed Consent: With your permission, I will be recording this interview. What you say during this interview will be shared with a team of analysts, but nobody else. We will remove your name and other personally identifying information from the recording, and your name will not appear in the analysis or any report. So please, speak openly to us – your voice is very important to help us learn about the successes and challenges of this program. For interviews with more than one respondent: We would like to hear from all of you, so I encourage you all to speak openly, even if you may disagree with what another participant is saying. I ask that you please speak one by one, so we can record all of your answers accurately. We will try to make sure everyone has a chance to speak. Do you agree with being interviewed by us, and to our conversation being recorded? [CONFIRM CONSENT] Thank you very much for agreeing to take part in this interview. Do you have any questions before we begin? Interview Information Enumerator/s Interview Date Original Interview Language French Malagasy English Approx Interview Length Delete as appropriate <30 mins 30mins 45 mins 1hr 1hr+ Respondent Information Respondent Type • CM – Community Mobilizer Number of Respondents Interviewee Name/s Gender Age Delete as appropriate 18 - 30 years old 30 - 60 years old 60+ years IMPEL | Implementer-Led Evaluation and Learning 62 Annex E: Data Collection Tool District Leave blank if not relevant Commune Leave blank if not relevant Fokontany Leave blank if not relevant Observations Please provide details about the interview setting, interviewee’s responsiveness, and any other relevant information Interview Questions Background Firstly, we would like to hear some background about Community Consultations: 1. From your understanding, what is the main purpose of Community Consultations? 2. What would you consider your main activities to be? 2.1. Which of these activities do you consider the most important? Why? (ie. Because they are most effective, most relevant/needed for the community, the ones conducted most often?) Effectiveness of CRS Support 3. What support for implementation have you received from CRS so far? What did this consist of? (e.g. trainings, material or financial support) 3.1. What are your reflections, positive or negative, in the way this support was delivered? Could anything have been improved, and why? (e.g. format of trainings; any instances of delays) Effectiveness of Implementation Now I would like to hear about your experiences implementing activities as a Community Mobilizer: Note to Enumerators: Ask for each of the activities mentioned in Q2 above; if more than 3 main activities were mentioned, ask about the 3 most important. Note to Enumerators: If respondents talk only about the effectiveness or impact of the activities, steer them back to the topic of how activities are implemented, or their role in and experience of carrying out these activities. 4. Can you briefly describe what implementation of your key activities entails/looks like? Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 63 5. What other actors (local, formal, informal, governance, business, NGOs/civil society groups, or NGOs) do you work with or are involved in carrying out your activities as Miranjaka? 5.1. Please describe how you work with these actors and why these links are important for effective implementation/your work? 6. What would a ‘successful’ implementation of each these activities look like? 6.1. Why would you consider it successful? What factors contribute to it being a success? (e.g. availability of time, contextual factors) 6.2. What role would you play or specific actions on your part would contribute to this success? 6.3. Note to Enumerators: Try to steer answers towards how activities are implemented, not just focusing on impact as a success. 7. What kind of challenges have you encountered while implementing your activities? (Probe specifically for COVID-19) 7.1. What aspects of the implementation do these affect? (e.g. planning, availability of time/materials, transport) 7.2. How do these challenges affect your ability to implement activities? 7.3. How have you adapted to these obstacles/challenges? 7.4. If you did not/could not able to adapt, why not? What resources or other support were you missing? 7.5. How might these challenges be overcome in the future? Linkages 8. Above we asked about other organizations and actors with whom you partner in order to carry out your work. May you clarify: 8.1. How effective would you describe these links with key partners at the moment? 8.2. What are some effective elements of these partnerships? What do you consider important for ensuring you work together to implement activities effectively? 8.3. What specific outcomes have emerged out of these partnerships? What factors were responsible for these outcomes? Please be as specific as possible. 8.4. What limitations do you see in your partnerships at the moment? How could they be improved, or what would you change moving forward? (e.g. communications, type of support, expanding to other/different partners, geography or sectoral knowledge). IMPEL | Implementer-Led Evaluation and Learning 64 Annex E: Data Collection Tool Drought 9. How has the drought/kere affected your ability to implement activities? 9.1. What changes or adaptations have you had to make to the way you implement because of the drought/kere? 10. How did the drought/kere affect the effectiveness, impact, and potential outcomes of Community Consultations? 10.1. Did beneficiaries’ or members’ participation with Community Consultations change because of the drought? How and why? 11. In your opinion, what were the most important aspects or contributions of the Community Consultations in the drought/kere context? 11.1. What were the most important aspects of the Maharo program as a whole? (Probe for: activities other than the emergency food assistance) Note to Enumerators: Be aware that not all respondents may be able to reflect on the Maharo program as a whole or have enough information to answer this question with lots of detail. It may be helpful to mention some of the other activities being examined in this Process Monitoring, but do not feel the need to spend too much time on this question if respondents are unsure how to answer. Inclusivity Gender 12. The baseline findings (refer to this in more detail if that is not clear) suggest that women in this region possess a high degree of ‘agency’ (ie, influence over household decisions). In what areas of household and community decision-making do women have more or less influence? ■ If more information needed: Causal Design was commissioned to conduct a baseline study for the IMPEL/Implementer-Led Design, Evidence, Analysis and Learning (IDEAL) activity. Our baseline findings on gender suggested that women had more agency than CRS/ADRA’s previous gender analyses suggested. 13. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for both men and women? 13.1. What does men and women’s participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ between men and women? 13.2. What considerations are important to ensuring equal access for both men and women when implementing programs (not just designing them)? 13.3. What limitations or gaps still exist in implementation efforts to ensure equal access? How might they be overcome? Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 65 PWD 14. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for people with disabilities (PWDs)? 14.1. What does PWD participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ for men and women with disabilities? 14.2. What considerations are important to ensuring equal access for PWDs? 14.3. What limitations or gaps still exist in implementation efforts to ensure equal access for PWDs? How might they be overcome? Youth 15. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for youth? Note to Enumerators: ‘Youth’ refers to adolescents and young people aged 10 to 29. 15.1. What does youths’ participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ for young men and young women? 15.2. What considerations are important to ensuring equal access for youth? 15.3. What limitations or gaps still exist in implementation efforts to ensure equal access for youth? How might they be overcome? Linkages 16. Above we asked about other organizations and actors with whom you partner in order to carry out your work. May you clarify: 16.1. How effective would you describe these links with key partners at the moment? 16.2. What are some effective elements of these partnerships? What do you consider important for ensuring you work together to implement activities effectively? 16.3. What specific outcomes have emerged out of these partnerships? What factors were responsible for these outcomes? Please be as specific as possible. 16.4. What limitations do you see in your partnerships at the moment? How could they be improved, or what would you change moving forward? (e.g. communications, type of support, expanding to other/different partners, geography or sectoral knowledge). Is there anything else you would like to say or ask before we conclude the interview? IMPEL | Implementer-Led Evaluation and Learning 66 Annex E: Data Collection Tool Thank you so much for your time! We greatly appreciate your perspective on all of this! If you have any further questions or concerns, please do not hesitate to contact Eliane Ralison (+261 331165119), Sophie Turnbull (sophie.turnbull@causaldesign.com) or Matt Klick (matthew.klick@causaldesign.com). Approx Interview Length Delete as appropriate <30 mins / 30mins / 45 mins / 1hr / 1hr+ Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 67 Care Groups KII Tool Background Information for Enumerators Care Groups Care Groups is a community-led behavioral change approach primarily targeting pregnant women, breastfeeding women, and women with children under 2 years old. Care Groups consist of 10-15 volunteer Lead Mothers, who work to promote best practices in health, nutrition, WASH, early childhood development, and changing gender roles around childcare in their communities. Lead Mothers’ activities include attending trainings through the Care Group, conducting household visits, and collaborating with other community actors, such as SILCs, Producer Learning Centers, and Miranjaka to share information and promote good health and nutrition. Lead Mothers are selected by a Reny Mpiara-Monina (RMM), a group of 10 to 15 pregnant and lactating women, women with children aged <2 years, and adolescent girls from their same neighborhood. Lead Mothers attend monthly Care Group meetings with other Lead Mothers to receive trainings on health and nutrition topics, which they then cascade to the RMMs. Care Groups are trained and mentored by Field Agents (also known as Agents du Terrain Nu & Wash). Each field agent supports 6 Care Groups. Care Groups also work with Community Health Workers (CHWs). CHWs assist to mobilize communities and women to participate in Care Groups and identify a Lead Mother. CHWs also attend monthly Care Group Meetings, serving as ‘local supervisors’ and a resource that Lead Mothers can approach with questions. Communication Protocol and Informed Consent Introduction and program information: Hello, my name is ________________. I work with Consultants Associates, a Madagascar-based data collection firm, in partnership with Causal Design, a research organization. We are conducting a process monitoring evaluation of CRS’ Maharo program, which aims to enhance food security and resilience in three districts of Madagascar: Ampanihy, Beloha, and Tsihombe. The Maharo Program includes a range of activities in a range of sectors, including Care Groups. We specifically are examining how Maharo projects are being implemented, and anything that might be affecting implementation of these activities, either positively or negatively. We will be examining impact and effectiveness at a later time. Goal of this interview: We have invited you to this interview to share your views on the work of the Care Groups and CRS’ support for Care Groups under the Maharo program. We would like to hear your perspective on how Maharo’s support has been delivered, and your experiences implementing activities as members of a Care Group. We would like to hear about both facilitators and obstacles, with the aim of ascertaining how implementation could be improved in the future. We would also like to hear about the inclusion of women, youth, and people with disabilities in Care Groups and their activities; and effects of IMPEL | Implementer-Led Evaluation and Learning 68 Annex E: Data Collection Tool the drought/kere on program/activity implementation and activities. We aim for the interview to take up around 1 hour of your time. Managing Expectations: We cannot guarantee that every suggestion raised during this interview will be acted upon, but all suggestions will be analyzed and those that occur frequently may be taken forward as recommendations to program leadership. The analysis results will be shared with CRS, and it will be up to their discretion to share them with you and others. Informed Consent: With your permission, I will be recording this interview. What you say during this interview will be shared with a team of analysts, but nobody else. We will remove your name and other personally identifying information from the recording, and your name will not appear in the analysis or any report. So please, speak openly to us – your voice is very important to help us learn about the successes and challenges of this program. For interviews with more than one respondent: We would like to hear from all of you, so I encourage you all to speak openly, even if you may disagree with what another participant is saying. I ask that you please speak one by one, so we can record all of your answers accurately. We will try to make sure everyone has a chance to speak. Do you agree with being interviewed by us, and to our conversation being recorded? [CONFIRM CONSENT] Thank you very much for agreeing to take part in this interview. Do you have any questions before we begin? Interview Information Enumerator/s Interview Date Original Interview Language French Malagasy English Approx Interview Length Delete as appropriate <30 mins 30mins 45 mins 1hr 1hr+ Respondent Information Respondent Type ● CGs (Lead Mothers) Number of Respondents Interviewee Name/s Gender Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 69 Age Delete as appropriate 18 - 30 years old 30 - 60 years old 60+ years District Commune Fokontany Observations Please provide details about the interview setting, interviewee’s responsiveness, and any other relevant information Interview Questions Background Firstly, we would like to hear some background about Care Groups: 1. From your understanding, what is the main purpose of this Care Group? 2. What would you consider your main activities to be? 2.1. Which of these activities do you consider the most important? Why? (ie. Because they are most effective, most relevant/needed for the community, the ones conducted most often?) Effectiveness of CRS Support 3. What support for implementation have you received from CRS so far? What did this consist of? (e.g. trainings, material or financial support) (Probe for: systems to support and mentor Lead Mothers) 3.1. What are your reflections, positive or negative, in the way this support was delivered? Could anything have been improved, and why? (e.g. format of trainings; any instances of delays) Effectiveness of Implementation Now I would like to hear about your experiences implementing activities through this Care Group: Note to Enumerators: Ask for each of the activities mentioned in Q2 above; if more than 3 main activities were mentioned, ask about the 3 most important. Note to Enumerators: If respondents talk only about the effectiveness or impact of the activities, steer them back to the topic of how activities are implemented, or their role in and experience of carrying out these activities. IMPEL | Implementer-Led Evaluation and Learning 70 Annex E: Data Collection Tool 4. Can you briefly describe what implementation of your key activities entails/looks like? 5. What other actors (local, formal, informal, governance, business, NGOs/civil society groups, or NGOs) do you work with or are involved in carrying out your activities as Miranjaka? 5.1. Please describe how you work with these actors and why these links are important for effective implementation/your work? 6. What would a ‘successful’ implementation of each these activities look like? 6.1. Why would you consider it successful? What factors contribute to it being a success? (e.g. availability of time, contextual factors) 6.2. What role would you play or specific actions on your part would contribute to this success? 6.3. Note to Enumerators: Try to steer answers towards how activities are implemented, not just focusing on impact as a success. 7. What kind of challenges have you encountered while implementing your activities? (Probe specifically for COVID-19) 7.1. What aspects of the implementation do these affect? (e.g. planning, availability of time/materials, transport) 7.2. How do these challenges affect your ability to implement activities? 7.3. How have you adapted to these obstacles/challenges? 7.4. If you did not/could not able to adapt, why not? What resources or other support were you missing? 7.5. How might these challenges be overcome in the future? Linkages 8. Above we asked about other organizations and actors with whom you partner in order to carry out your work. May you clarify: 8.1. How effective would you describe these links with key partners at the moment? 8.2. What are some effective elements of these partnerships? What do you consider important for ensuring you work together to implement activities effectively? 8.3. What specific outcomes have emerged out of these partnerships? What factors were responsible for these outcomes? Please be as specific as possible. 8.4. What limitations do you see in your partnerships at the moment? How could they be improved, or what would you change moving forward? (e.g. communications, type of support, expanding to other/different partners, geography or sectoral knowledge). Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 71 Drought 9. How has the drought/kere affected your ability to implement activities? 9.1. What changes or adaptations have you had to make to the way you implement because of the drought/kere? 10. How did the drought/kere affect the effectiveness, impact, and potential outcomes of this Care Group’s activities? 10.1. Did beneficiaries’ or members’ participation with this Care Group and their activities change because of the drought? How and why? 11. In your opinion, what were the most important aspects or contributions of this Care Group in the drought/kere context? 11.1. What were the most important aspects of the Maharo program as a whole? (Probe for: activities other than the emergency food assistance) Note to Enumerators: Be aware that not all respondents may be able to reflect on the Maharo program as a whole or have enough information to answer this question with lots of detail. It may be helpful to mention some of the other activities being examined in this Process Monitoring, but do not feel the need to spend too much time on this question if respondents are unsure how to answer. Inclusivity Gender 12. The baseline findings (refer to this in more detail if that is not clear) suggest that women in this region possess a high degree of ‘agency’ (ie, influence over household decisions). In what areas of household and community decision-making do women have more or less influence? ■ If more information needed: Causal Design was commissioned to conduct a baseline study for the IMPEL/Implementer-Led Design, Evidence, Analysis and Learning (IDEAL) activity. Our baseline findings on gender suggested that women had more agency than CRS/ADRA’s previous gender analyses suggested. 13. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for both men and women? 13.1. What does men and women’s participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ between men and women? 13.2. What considerations are important to ensuring equal access for both men and women when implementing programs (not just designing them)? 13.3. What limitations or gaps still exist in implementation efforts to ensure equal access? How might they be overcome? IMPEL | Implementer-Led Evaluation and Learning 72 Annex E: Data Collection Tool PWD 14. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for people with disabilities (PWDs)? 14.1. What does PWD participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ for men and women with disabilities? 14.2. What considerations are important to ensuring equal access for PWDs? 14.3. What limitations or gaps still exist in implementation efforts to ensure equal access for PWDs? How might they be overcome? Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 73 Youth 15. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for youth? Note to Enumerators: ‘Youth’ refers to adolescents and young people aged 10 to 29. 15.1. What does youths’ participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ for young men and young women? 15.2. What considerations are important to ensuring equal access for youth? 15.3. What limitations or gaps still exist in implementation efforts to ensure equal access for youth? How might they be overcome? Is there anything else you would like to ask or say before we conclude the interview? Thank you so much for your time! We greatly appreciate your perspective on all of this! If you have any further questions or concerns, please do not hesitate to contact Eliane Ralison (+261 331165119), Sophie Turnbull (sophie.turnbull@causaldesign.com) or Matt Klick (matthew.klick@causaldesign.com). Approx Interview Length Delete as appropriate <30 mins / 30mins / 45 mins / 1hr / 1hr+ IMPEL | Implementer-Led Evaluation and Learning 74 Annex E: Data Collection Tool Miranjaka KII Tool Background Information for Enumerators Miranjaka Miranjaka Gender Champions work to promote gender equality and inclusion through a range of channels, targeting household and community-level. Their key activities include: ● Working with traditional leaders to discuss gender gaps and gain leadership support for gender equality activities (e.g. men’s safe space): ● Serve as community volunteers to conduct household visits and coordinate gender messages ● Conduct home visits to spread messaging such as GBV prevention and referring people experiencing violence to protection services, joint decision-making, equitable roles and responsibilities, access to and control over household chores and assets, ● Hold Safe Space discussions with targeted participants, such as youth groups and women (specifically Care Groups) ● Hold ‘Husbands/Dad’s School’ every quarter to promote a ‘papa model’ (promotes positive masculinity) ● Work with KFF (VDC) to ensure gender equality, inclusivity and GBV prevention and protection through Komity Miaro Zo approach (human rights committee within KFF members) Miranjaka also receive capacity buildings from CRS project staff to ensure the effectiveness of their activities, such as gender norms, concepts of women’s empowerment and positive masculinity, and trainings on the how to conduct activities such as home visit and safe space discussion techniques. Miranjaka are also supervised by staff from the SDPPSPF (Service de District de Population, Protection Sociale, et Promotion des Femmes / District Population Service, Social Protection, and Women’s Empowerment). Communication Protocol and Informed Consent Introduction and program information: Hello, my name is ________________. I work with Consultants Associates, a Madagascar-based data collection firm, in partnership with Causal Design, a research organization. We are conducting a process monitoring evaluation of CRS’ Maharo program, which aims to enhance food security and resilience in three districts of Madagascar: Ampanihy, Beloha, and Tsihombe. The Maharo Program includes a range of activities in a range of sectors, including Miranjakas.We specifically are examining how Maharo projects are being implemented, and anything that might be affecting implementation of these activities, either positively or negatively. We will be examining impact and effectiveness at a later time. Goal of this interview: We have invited you to this interview to share your views on the work of the Miranjakas and CRS’ support for Miranjaka under the Maharo program. We would like to hear your perspective on how Maharo’s support has been delivered, and your experiences implementing activities as Miranjakas. We would like to hear about both facilitators and obstacles, with the aim of ascertaining how implementation could be improved in the future. We would also like to hear about the inclusion of women, youth, and people with disabilities in Miranjakas and their activities; and effects of the Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 75 drought/kere on program/activity implementation and activities. We aim for the interview to take up around 1 hour of your time. Managing Expectations: We cannot guarantee that every suggestion raised during this interview will be acted upon, but all suggestions will be analyzed and those that occur frequently may be taken forward as recommendations to program leadership. The analysis results will be shared with CRS, and it will be up to their discretion to share them with you and others. Informed Consent: With your permission, I will be recording this interview. What you say during this interview will be shared with a team of analysts, but nobody else. We will remove your name and other personally identifying information from the recording, and your name will not appear in the analysis or any report. So please, speak openly to us – your voice is very important to help us learn about the successes and challenges of this program. For FGDs/interviews with more than one respondent: We would like to hear from all of you, so I encourage you all to speak openly, even if you may disagree with what another participant is saying. I ask that you please speak one by one, so we can record all of your answers accurately. We will try to make sure everyone has a chance to speak. Do you agree with being interviewed by us, and to our conversation being recorded? [CONFIRM CONSENT] Thank you very much for agreeing to take part in this interview. Do you have any questions before we begin? Interview Information Enumerator/s Interview Date Original Interview Language French Malagasy English Approx Interview Length Delete as appropriate <30 mins 30mins 45 mins 1hr 1hr+ Respondent Information Respondent Type ● Miranjaka Number of Respondents Interviewee Name/s Gender IMPEL | Implementer-Led Evaluation and Learning 76 Annex E: Data Collection Tool Age Delete as appropriate 18 - 30 years old 30 - 60 years old 60+ years District Leave blank if not relevant Commune Leave blank if not relevant Fokontany Leave blank if not relevant Observations Please provide details about the interview setting, interviewee’s responsiveness, and any other relevant information Interview Questions Background Firstly, we would like to hear some background about Miranjakas: 1. From your understanding, what is the main purpose of Miranjakas? 2. What would you consider your main activities to be? 2.1. Which of these activities do you consider the most important? Why? (ie. Because they are most effective, most relevant/needed for the community, the ones conducted most often?) Effectiveness of CRS Support 3. What support for implementation have you received from CRS so far? What did this consist of? (e.g. trainings, material or financial support) 3.1. What are your reflections, positive or negative, in the way this support was delivered? Could anything have been improved, and why? (e.g. format of trainings; any instances of delays) Effectiveness of Implementation Now I would like to hear about your experiences implementing activities as Miranjakas: Note to Enumerators: Ask for each of the activities mentioned in Q2 above; if more than 3 main activities were mentioned, ask about the 3 most important. Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 77 Note to Enumerators: If respondents talk only about the effectiveness or impact of the activities, steer them back to the topic of how activities are implemented, or their role in and experience of carrying out these activities. 4. Can you briefly describe what implementation of your key activities entails/looks like? 5. What other actors (local, formal, informal, governance, business, NGOs/civil society groups, or NGOs) do you work with or are involved in carrying out your activities as Miranjaka? 5.1. Please describe how you work with these actors and why these links are important for effective implementation/your work? 6. What would a ‘successful’ implementation of each these activities look like? 6.1. Why would you consider it successful? What factors contribute to it being a success? (e.g. availability of time, contextual factors) 6.2. What role would you play or specific actions on your part would contribute to this success? 6.3. Note to Enumerators: Try to steer answers towards how activities are implemented, not just focusing on impact as a success. 7. What kind of challenges have you encountered while implementing your activities? (Probe specifically for COVID-19) 7.1. What aspects of the implementation do these affect? (e.g. planning, availability of time/materials, transport) 7.2. How do these challenges affect your ability to implement activities? 7.3. How have you adapted to these obstacles/challenges? 7.4. If you did not/were not able to adapt, why not? What resources or other support were you missing? 7.5. How might these challenges be overcome in the future? Linkages 8. Above we asked about other organizations and actors with whom you partner in order to carry out your work. May you clarify: 8.1. How effective would you describe these links with key partners at the moment? 8.2. What are some effective elements of these partnerships? What do you consider important for ensuring you work together to implement activities effectively? 8.3. What specific outcomes have emerged out of these partnerships? What factors were responsible for these outcomes? Please be as specific as possible. IMPEL | Implementer-Led Evaluation and Learning 78 Annex E: Data Collection Tool 8.4. What limitations do you see in your partnerships at the moment? How could they be improved, or what would you change moving forward? (e.g. communications, type of support, expanding to other/different partners, geography or sectoral knowledge). Drought 9. How has the drought/kere affected your ability to implement activities? 9.1. What changes or adaptations have you had to make to the way you implement because of the drought/kere? 10. How did the drought/kere affect the effectiveness, impact, and potential outcomes of Miranjakas’ activities? 10.1. Did beneficiaries’ or members’ participation with Miranjakas and their activities change because of the drought? How and why? 11. In your opinion, what were the most important aspects or contributions of Miranjakas in the drought/kere context? 11.1. What were the most important aspects of the Maharo program as a whole? (Probe for: activities other than the emergency food assistance) Note to Enumerators: Not all respondents may be able to reflect on the Maharo program as a whole or have enough information to answer this question with lots of detail. It may be helpful to mention some of the other activities being examined in this Process Monitoring, but do not feel the need to spend too much time on this question if respondents are unsure how to answer. Inclusivity Gender 12. The baseline findings (refer to this in more detail if that is not clear) suggest that women in this region possess a high degree of ‘agency’ (ie, influence over household decisions). In what areas of household and community decision-making do women have more or less influence? ■ If more information needed: Causal Design was commissioned to conduct a baseline study for Maharo under the IMPEL/Implementer-Led Design, Evidence, Analysis and Learning (IDEAL) activity. Our baseline findings on gender suggested that women had more agency than CRS/ADRA’s previous gender analyses suggested 13. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for both men and women? 13.1. What does men and women’s participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ between men and women? Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 79 13.2. What considerations are important to ensuring equal access for both men and women when implementing programs (not just designing them)? 13.3. What limitations or gaps still exist in implementation efforts to ensure equal access? How might they be overcome? PWD 14. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for people with disabilities (PWDs)? 14.1. What does PWD participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ for men and women with disabilities? 14.2. What considerations are important to ensuring equal access for PWDs? 14.3. What limitations or gaps still exist in implementation efforts to ensure equal access for PWDs? How might they be overcome? IMPEL | Implementer-Led Evaluation and Learning 80 Annex E: Data Collection Tool Youth 15. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for youth? Note to Enumerators: ‘Youth’ refers to adolescents and young people aged 10 to 29. 15.1. What does youths’ participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ for young men and young women? 15.2. What considerations are important to ensuring equal access for youth? 15.3. What limitations or gaps still exist in implementation efforts to ensure equal access for youth? How might they be overcome? Is there anything else you would like to ask or say before we conclude the interview? Thank you so much for your time! We greatly appreciate your perspective on all of this! If you have any further questions or concerns, please do not hesitate to contact Eliane Ralison (+261 331165119), Sophie Turnbull (sophie.turnbull@causaldesign.com) or Matt Klick (matthew.klick@causaldesign.com). Approx Interview Length Delete as appropriate <30 mins / 30mins / 45 mins / 1hr / 1hr+ Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 81 DRR Committees KII Tool Background Information for Enumerators Disaster Risk Reduction Committees Under Maharo, CRS has facilitated the creation of fokontany-level disaster risk reduction (DRR) groups, called called either CGRC (Comite de Gestion des Risques et des catastrophes) or ELS (Equipe Local de Secour). These committees help to share Meteo information about weather and climate with community members through methods such as radio and loudspeakers. They also meet monthly with commune-level DRR committees, known as CCGRCs (Comité Local de la Gestion des Risques et Catastrophes) to communicate fokongany-level local knowledge. CCGRCs are responsible for passing information to District and higher levels of the BGNRC (Bureau National de Gestion des Risques et Catastrophes / National Office of Risk and Disaster Management). CGRCs/ELS typically have between 10 and 15 members which may include: 1. Fokontany Chief 2. Elders 3. Representative of Religious and Traditional Associations (ART) 4. Representative of the community health system (CA) 5. Representative of the community natural resources management system (e.g. VOI, VNA, KASTI, or other NRM committee forms) 6. Representative of the community infrastructure management (AUE, AUP) 7. Representative of the KFF committee at village level 8. Private sector representatives (e.g. traders, wholesalers, operators) 9. Representative of women's associations (Miranjaka) 10. Representative of the targeting and support committee for the most vulnerable 11. Youth representative 12. A member of the municipal council living in the fokontany 13. Representative from other services in the fokontany (e.g. teachers, health workers) 14. Representative of other local groups and citizens from the fokontany (e.g. farmers, fishers, SILCs) 15. Representative of people with disabilities Maharo supports both commune-level and fokontany-level DRR Committees through capacity building, and providing support to create and update Disaster Risk Reduction Plans. Communication Protocol and Informed Consent Introduction and program information: Hello, my name is ________________. I work with Consultants Associates, a Madagascar-based data collection firm, in partnership with Causal Design, a research organization. We are conducting a process monitoring evaluation of CRS’ Maharo program, which aims to enhance food security and resilience in three districts of Madagascar: Ampanihy, Beloha, and Tsihombe. The Maharo Program includes a range of activities in a range of sectors, including Disaster Risk Reduction Committees. We specifically are examining how Maharo projects are being implemented, and anything that might be affecting implementation of these activities, either positively or negatively. We will be examining impact and effectiveness at a later time. Goal of this interview: We have invited you to this interview to share your views on the work of the DRR Committees and CRS’ support for these committees under the Maharo program. We would like to hear IMPEL | Implementer-Led Evaluation and Learning 82 Annex E: Data Collection Tool your perspective on how Maharo’s support has been delivered, and your experiences implementing activities as part of the DRR Committee. We would like to hear about both facilitators and obstacles, with the aim of ascertaining how implementation could be improved in the future. We would also like to hear about the inclusion of women, youth, and people with disabilities in DRR committees; and effects of the drought/kere on program/activity implementation and activities. We aim for the interview to take up around 1 hour of your time. Managing Expectations: We cannot guarantee that every suggestion raised during this interview will be acted upon, but all suggestions will be analyzed and those that occur frequently may be taken forward as recommendations to program leadership. The analysis results will be shared with CRS, and it will be up to their discretion to share them with you and others. Informed Consent: With your permission, I will be recording this interview. What you say during this interview will be shared with a team of analysts, but nobody else. We will remove your name and other personally identifying information from the recording, and your name will not appear in the analysis or any report. So please, speak openly to us – your voice is very important to help us learn about the successes and challenges of this program. For interviews with more than one respondent: We would like to hear from all of you, so I encourage you all to speak openly, even if you may disagree with what another participant is saying. I ask that you please speak one by one, so we can record all of your answers accurately. We will try to make sure everyone has a chance to speak. Do you agree with being interviewed by us, and to our conversation being recorded? [CONFIRM CONSENT] Thank you very much for agreeing to take part in this interview. Do you have any questions before we begin? Interview Information Enumerator/s Interview Date Original Interview Language French Malagasy English Approx Interview Length Delete as appropriate <30 mins 30mins 45 mins 1hr 1hr+ Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 83 Respondent Information Respondent Type Delete as appropriate ● CCGRC - Comité Local de la Gestion des Risques et Catastrophes (Local-level Disaster and Risk Management Committee) ● CGRC - Comite de Gestion des Risques et des catastrophes (Disaster and Risk Managment commitee at Fokontany Level) ● ELS - Equipe Local de Secour (Locar Risk Managment commitee at Fokontany Level) Number of Respondents Interviewee Name/s Gender Age Delete as appropriate 18 - 30 years old 30 - 60 years old 60+ years District Commune Fokontany Leave blank for commune-level respondents Observations Please provide details about the interview setting, interviewee’s responsiveness, and any other relevant information Interview Questions Background Firstly, we would like to hear some background about this DRR committee: 1. From your understanding, what is the main purpose of this DRR committee? 2. What would you consider your main activities to be? Probe for: formation of DRR plans; meetings with BNGRC and fokontany/commune-level committee counterparts; communication of Meteo information to communities) 2.1. Which of these activities do you consider the most important? Why? (ie. Because they are most effective, most relevant/needed for the community, the ones conducted most often?) 2.2. Have you developed any DRR plans? Who was involved in creating these plans, and what problems are the plans intended to address? IMPEL | Implementer-Led Evaluation and Learning 84 Annex E: Data Collection Tool Effectiveness of CRS Support 3. What support for implementation have you received from CRS so far? What did this consist of? (e.g. trainings, material or financial support) 3.1. What are your reflections, positive or negative, in the way this support was delivered? Could anything have been improved, and why? (e.g. format of trainings; any instances of delays) Effectiveness of Implementation Now I would like to hear about your experiences implementing activities as part of the DRR Committee: Note to Enumerators: Ask for each of the activities mentioned in Q2 above; if more than 3 main activities were mentioned, ask about the 3 most important. Note to Enumerators: If respondents talk only about the effectiveness or impact of the activities, steer them back to the topic of how activities are implemented, or their role in and experience of carrying out these activities. 4. Can you briefly describe what implementation of your key activities entails/looks like? 5. What other actors (local, formal, informal, governance, business, NGOs/civil society groups, or NGOs) do you work with or are involved in carrying out your activities as Miranjaka? 5.1. Please describe how you work with these actors and why these links are important for effective implementation/your work? 6. What would a ‘successful’ implementation of each these activities look like? 6.1. Why would you consider it successful? What factors contribute to it being a success? (e.g. availability of time, contextual factors) 6.2. What role would you play or specific actions on your part would contribute to this success? 6.3. Note to Enumerators: Try to steer answers towards how activities are implemented, not just focusing on impact as a success. 7. What kind of challenges have you encountered while implementing your activities? (Probe specifically for COVID-19) 7.1. What aspects of the implementation do these affect? (e.g. planning, availability of time/materials, transport) 7.2. How do these challenges affect your ability to implement activities? 7.3. How have you adapted to these obstacles/challenges? 7.4. If you did not/could not able to adapt, why not? What resources or other support were you missing? Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 85 7.5. How might these challenges be overcome in the future? Linkages 8. Above we asked about other organizations and actors with whom you partner in order to carry out your work. May you clarify: 8.1. How effective would you describe these links with key partners at the moment? 8.2. What are some effective elements of these partnerships? What do you consider important for ensuring you work together to implement activities effectively? 8.3. What specific outcomes have emerged out of these partnerships? What factors were responsible for these outcomes? Please be as specific as possible. 8.4. What limitations do you see in your partnerships at the moment? How could they be improved, or what would you change moving forward? (e.g. communications, type of support, expanding to other/different partners, geography or sectoral knowledge). It may be helpful to mention some of the other activities being examined in this Process Monitoring, but do not feel the need to spend too much time on this question if respondents are unsure how to answer. Drought 9. How has the drought/kere affected your ability to implement activities? 9.1. What changes or adaptations have you had to make to the way you implement because of the drought/kere? 10. How did the drought/kere affect the effectiveness, impact, and potential outcomes of the DRR Committees’ activities? 10.1. Did beneficiaries’ or members’ participation with DRR Committees change because of the drought? How and why? 11. In your opinion, what were the most important aspects or contributions of the DRR Committees in the drought/kere context? 11.1. What were the most important aspects of the Maharo program as a whole? (Probe for: activities other than the emergency food assistance) Note to Enumerators: Be aware that not all respondents may be able to reflect on the Maharo program as a whole or have enough information to answer this question with lots of detail. Inclusivity Gender 12. The baseline findings (refer to this in more detail if that is not clear) suggest that women in this region possess a high degree of ‘agency’ (ie, influence over household decisions). In what areas of household and community decision-making do women have more or less influence? IMPEL | Implementer-Led Evaluation and Learning 86 Annex E: Data Collection Tool ■ If more information needed: Causal Design was commissioned to conduct a baseline study for the IMPEL/Implementer-Led Design, Evidence, Analysis and Learning (IDEAL) activity. Our baseline findings on gender suggested that women had more agency than CRS/ADRA’s previous gender analyses suggested. 13. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for both men and women? 13.1. What does men and women’s participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ between men and women? 13.2. What considerations are important to ensuring equal access for both men and women when implementing programs (not just designing them)? 13.3. What limitations or gaps still exist in implementation efforts to ensure equal access? How might they be overcome? PWD 14. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for people with disabilities (PWDs)? 14.1. What does PWD participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ for men and women with disabilities? 14.2. What considerations are important to ensuring equal access for PWDs? 14.3. What limitations or gaps still exist in implementation efforts to ensure equal access for PWDs? How might they be overcome? Youth 15. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for youth? Note to Enumerators: ‘Youth’ refers to adolescents and young people aged 10 to 29. 15.1. What does youths’ participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ for young men and young women? 15.2. What considerations are important to ensuring equal access for youth? 15.3. What limitations or gaps still exist in implementation efforts to ensure equal access for youth? How might they be overcome? Is there anything else you would like to ask or say before we conclude the interview? Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 87 Thank you so much for your time! We greatly appreciate your perspective on all of this! If you have any further questions or concerns, please do not hesitate to contact Eliane Ralison (+261 331165119), Sophie Turnbull (sophie.turnbull@causaldesign.com) or Matt Klick (matthew.klick@causaldesign.com). Approx Interview Length Delete as appropriate <30 mins / 30mins / 45 mins / 1hr / 1hr+ IMPEL | Implementer-Led Evaluation and Learning 88 Annex E: Data Collection Tool Lead Farmers KII Tool Background Information for Enumerators Lead Farmers The aim of the Lead Farmers activity is to strengthen the capacity of farmers with regard to agriculture and the environment, such as: strengthening capacity in dryland, watershed, pest, and soil fertility management; improve production; and increase forest cover. Lead Farmers facilitate Demonstration Plot groups. These are at the fokontany level and consist of 10-20 learners, ideally from the same village, who meet regularly during the farming season. These groups, led by the Lead Farmers, identify and prioritize problems they are facing, brainstorm and analyze solutions and design action plans for these; and implement action plans, with the support of field officers. Lead farmers also take part in cluster-level Farmer Learning Centers (FLC), which consist of all Lead Farmers from that cluster. Lead Farmers are trained on the FLC and Lead Farmer approach, technical innovations, group facilitation and leadership techniques. Maharo also supports FLCs and Demonstration Plot groups with start-up kits and supplies for the Lead Farmers. Communication Protocol and Informed Consent Introduction and program information: Hello, my name is ________________. I work with Consultants Associates, a Madagascar-based data collection firm, in partnership with Causal Design, a research organization. We are conducting a process monitoring evaluation of CRS’ Maharo program, which aims to enhance food security and resilience in three districts of Madagascar: Ampanihy, Beloha, and Tsihombe. The Maharo Program includes a range of activities in a range of sectors, including Lead Farmers. We specifically are examining how Maharo projects are being implemented, and anything that might be affecting implementation of these activities, either positively or negatively. We will be examining impact and effectiveness at a later time. Goal of this interview: We have invited you to this interview to share your views on the work of the Lead Farmers and CRS’ support for Lead Farmers under the Maharo program. We would like to hear your perspective on how Maharo’s support has been delivered, and your experiences implementing activities as Lead Farmers. We would like to hear about both facilitators and obstacles, with the aim of ascertaining how implementation could be improved in the future. We would also like to hear about the inclusion of women, youth, and people with disabilities in Lead Farmer activities; and effects of the drought/kere on program/activity implementation and activities. We aim for the interview to take up around 1 hour of your time. Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 89 Managing Expectations: We cannot guarantee that every suggestion raised during this interview will be acted upon, but all suggestions will be analyzed and those that occur frequently may be taken forward as recommendations to program leadership. The analysis results will be shared with CRS, and it will be up to their discretion to share them with you and others. Informed Consent: With your permission, I will be recording this interview. What you say during this interview will be shared with a team of analysts, but nobody else. We will remove your name and other personally identifying information from the recording, and your name will not appear in the analysis or any report. So please, speak openly to us – your voice is very important to help us learn about the successes and challenges of this program. For interviews with more than one respondent: We would like to hear from all of you, so I encourage you all to speak openly, even if you may disagree with what another participant is saying. I ask that you please speak one by one, so we can record all of your answers accurately. We will try to make sure everyone has a chance to speak. Do you agree with being interviewed by us, and to our conversation being recorded? [CONFIRM CONSENT] Thank you very much for agreeing to take part in this interview. Do you have any questions before we begin? Interview Information Enumerator/s Interview Date Original Interview Language French Malagasy English Approx Interview Length Delete as appropriate <30 mins 30mins 45 mins 1hr 1hr+ Respondent Information Respondent Type ● Lead Farmers Number of Respondents Interviewee Name/s Gender Age Delete as appropriate 18 - 30 years old 30 - 60 years old IMPEL | Implementer-Led Evaluation and Learning 90 Annex E: Data Collection Tool 60+ years District Commune Fokontany Observations Please provide details about the interview setting, interviewee’s responsiveness, and any other relevant information Interview Questions Background Firstly, we would like to hear some background about Lead Farmers: 1. From your understanding, what is the main purpose of the Lead Farmers? 2. What would you consider your main activities to be? 2.1. Which of these activities do you consider the most important? Why? (ie. Because they are most effective, most relevant/needed for the community, the ones conducted most often?) Effectiveness of CRS Support 3. What support for implementation have you received from CRS so far? What did this consist of? (e.g. trainings, material or financial support) 3.1. What are your reflections, positive or negative, in the way this support was delivered? Could anything have been improved, and why? (e.g. format of trainings; any instances of delays) Effectiveness of Implementation Now I would like to hear about your experiences implementing activities as Lead Farmers: Note to Enumerators: Ask for each of the activities mentioned in Q2 above; if more than 3 main activities were mentioned, ask about the 3 most important. Note to Enumerators: If respondents talk only about the effectiveness or impact of the activities, steer them back to the topic of how activities are implemented, or their role in and experience of carrying out these activities. 4. Can you briefly describe what implementation of your key activities entails/looks like? Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 91 5. What other actors (local, formal, informal, governance, business, NGOs/civil society groups, or NGOs) do you work with or are involved in carrying out your activities as Miranjaka? 5.1. Please describe how you work with these actors and why these links are important for effective implementation/your work? 6. What would a ‘successful’ implementation of each these activities look like? 6.1. Why would you consider it successful? What factors contribute to it being a success? (e.g. availability of time, contextual factors) 6.2. What role would you play or specific actions on your part would contribute to this success? 6.3. Note to Enumerators: Try to steer answers towards how activities are implemented, not just focusing on impact as a success. 7. What kind of challenges have you encountered while implementing your activities? (Probe specifically for COVID-19) 7.1. What aspects of the implementation do these affect? (e.g. planning, availability of time/materials, transport) 7.2. How do these challenges affect your ability to implement activities? 7.3. How have you adapted to these obstacles/challenges? 7.4. If you did not/could not able to adapt, why not? What resources or other support were you missing? 7.5. How might these challenges be overcome in the future? Linkages 8. Above we asked about other organizations and actors with whom you partner in order to carry out your work. May you clarify: 8.1. How effective would you describe these links with key partners at the moment? 8.2. What are some effective elements of these partnerships? What do you consider important for ensuring you work together to implement activities effectively? 8.3. What specific outcomes have emerged out of these partnerships? What factors were responsible for these outcomes? Please be as specific as possible. 8.4. What limitations do you see in your partnerships at the moment? How could they be improved, or what would you change moving forward? (e.g. communications, type of support, expanding to other/different partners, geography or sectoral knowledge). IMPEL | Implementer-Led Evaluation and Learning 92 Annex E: Data Collection Tool Drought 9. How has the drought/kere affected your ability to implement activities? 9.1. What changes or adaptations have you had to make to the way you implement because of the drought/kere? 10. How did the drought/kere affect the effectiveness, impact, and potential outcomes of Lead Farmers’ activities? 10.1. Did beneficiaries’ or members’ participation with Lead Farmer activities change because of the drought? How and why? 11. In your opinion, what were the most important aspects or contributions of the Lead Farmers in the drought/kere context? 11.1. What were the most important aspects of the Maharo program as a whole? (Probe for: activities other than the emergency food assistance) Note to Enumerators: Be aware that not all respondents may be able to reflect on the Maharo program as a whole or have enough information to answer this question with lots of detail. It may be helpful to mention some of the other activities being examined in this Process Monitoring, but do not feel the need to spend too much time on this question if respondents are unsure how to answer. Inclusivity Gender 12. The baseline findings (refer to this in more detail if that is not clear) suggest that women in this region possess a high degree of ‘agency’ (ie, influence over household decisions). In what areas of household and community decision-making do women have more or less influence? ■ If more information needed: Causal Design was commissioned to conduct a baseline study for the IMPEL/Implementer-Led Design, Evidence, Analysis and Learning (IDEAL) activity. Our baseline findings on gender suggested that women had more agency than CRS/ADRA’s previous gender analyses suggested. 13. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for both men and women? 13.1. What does men and women’s participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ between men and women? 13.2. What considerations are important to ensuring equal access for both men and women when implementing programs (not just designing them)? 13.3. What limitations or gaps still exist in implementation efforts to ensure equal access? How might they be overcome? Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 93 PWD 14. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for people with disabilities (PWDs)? 14.1. What does PWD participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ for men and women with disabilities? 14.2. What considerations are important to ensuring equal access for PWDs? 14.3. What limitations or gaps still exist in implementation efforts to ensure equal access for PWDs? How might they be overcome? Youth 15. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for youth? Note to Enumerators: ‘Youth’ refers to adolescents and young people aged 10 to 29. 15.1. What does youths’ participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ for young men and young women? 15.2. What considerations are important to ensuring equal access for youth? 15.3. What limitations or gaps still exist in implementation efforts to ensure equal access for youth? How might they be overcome? Is there anything else you would like to ask or say before we conclude the interview? Thank you so much for your time! We greatly appreciate your perspective on all of this! If you have any further questions or concerns, please do not hesitate to contact Eliane Ralison (+261 331165119), Sophie Turnbull (sophie.turnbull@causaldesign.com) or Matt Klick (matthew.klick@causaldesign.com). Approx Interview Length Delete as appropriate <30 mins / 30mins / 45 mins / 1hr / 1hr+ IMPEL | Implementer-Led Evaluation and Learning 94 Annex E: Data Collection Tool Private Input Service Providers KII Tool Background Information for Enumerators Private Input Service Providers The two main activities under the PiSP are vouchers and promoting youth entrepreneurship. The voucher program aims to promote the private sector and increase availability of agricultural inputs, especially seeds, for farmers. Vouchers are distributed to beneficiary farmers/fishermen for items such as seeds and fishing equipment. These vouchers can be spent from approved PiSPs, who are selected based on community consultation, including a community vote to identify who is considered trustworthy to sell items on a voucher basis. One or two proximity PiSPs are targeted in each fokontany. The process of establishing PiSPs entails: • Receiving information from communities • Identifying PiSPs who sell seeds • Contracting selected PiSPs through contracts or letters of commitment • Collaborating with suppliers (CTAS, Agrivet) • Training PiSPs in store management, use of vouchers, and tools to use – this training is done by community-based • Ongoing monitoring and support of PiSPs The types of PiSPs currently operating are: • PiSP Boutiques (selling seeds and agricultural inputs) • PiSP Phyto (selling phytosanitary treatment items) • PiSP Artisan (blacksmiths) • PMS (seed producers) Under the youth component, youth are given capacity buildings regarding IGAs and entrepreneurship via Youth Clubs. From October 2021, Youth First also started distributing small loans to youth entrepreneurs. Communication Protocol and Informed Consent Introduction and program information: Hello, my name is ________________. I work with Consultants Associates, a Madagascar-based data collection firm, in partnership with Causal Design, a research organization. We are conducting a process monitoring evaluation of CRS’ Maharo program, which aims to enhance food security and resilience in three districts of Madagascar: Ampanihy, Beloha, and Tsihombe. The Maharo Program includes a range of activities in a range of sectors, including Private Input Service Providers (PISPs). We specifically are examining how Maharo projects are being implemented, and Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 95 anything that might be affecting implementation of these activities, either positively or negatively. We will be examining impact and effectiveness at a later time. Goal of this interview: We have invited you to this interview to share your views on the work of the PISPs and CRS’ support for PISPs under the Maharo program. We would like to hear your perspective on how Maharo’s support has been delivered, and your experiences implementing activities as PISPs. We would like to hear about both facilitators and obstacles, with the aim of ascertaining how implementation could be improved in the future. We would also like to hear about the inclusion of women, youth, and people with disabilities in PISPs and their activities; and effects of the drought/kere on program/activity implementation and activities. We aim for the interview to take up around 1 hour of your time. Managing Expectations: We cannot guarantee that every suggestion raised during this interview will be acted upon, but all suggestions will be analyzed and those that occur frequently may be taken forward as recommendations to program leadership. The analysis results will be shared with CRS, and it will be up to their discretion to share them with you and others. Informed Consent: With your permission, I will be recording this interview. What you say during this interview will be shared with a team of analysts, but nobody else. We will remove your name and other personally identifying information from the recording, and your name will not appear in the analysis or any report. So please, speak openly to us – your voice is very important to help us learn about the successes and challenges of this program. For interviews with more than one respondent: We would like to hear from all of you, so I encourage you all to speak openly, even if you may disagree with what another participant is saying. I ask that you please speak one by one, so we can record all of your answers accurately. We will try to make sure everyone has a chance to speak. Do you agree with being interviewed by us, and to our conversation being recorded? [CONFIRM CONSENT] Thank you very much for agreeing to take part in this interview. Do you have any questions before we begin? Interview Information Enumerator/s Interview Date Original Interview Language French Malagasy English Approx Interview Length Delete as appropriate <30 mins 30mins 45 mins 1hr 1hr+ IMPEL | Implementer-Led Evaluation and Learning 96 Annex E: Data Collection Tool Respondent Information Respondent Type ● PISP Number of Respondents Interviewee Name/s Gender Age Delete as appropriate 18 - 30 years old 30 - 60 years old 60+ years District Commune Fokontany Observations Please provide details about the interview setting, interviewee’s responsiveness, and any other relevant information Interview Questions Background Firstly, we would like to hear some background about PISPs: 1. From your understanding, what is the main purpose of PISPs? 2. What would you consider your main activities to be? 2.1. Which of these activities do you consider the most important? Why? (ie. Because they are most effective, most relevant/needed for the community, the ones conducted most often?) Effectiveness of CRS Support 3. What support for implementation have you received from CRS so far? What did this consist of? (e.g. trainings, material or financial support) 3.1. What are your reflections, positive or negative, in the way this support was delivered? Could anything have been improved, and why? (e.g. format of trainings; any instances of delays) Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 97 Effectiveness of Implementation Now I would like to hear about your experiences implementing activities as PISPs: Note to Enumerators: Ask for each of the activities mentioned in Q2 above; if more than 3 main activities were mentioned, ask about the 3 most important. Note to Enumerators: If respondents talk only about the effectiveness or impact of the activities, steer them back to the topic of how activities are implemented, or their role in and experience of carrying out these activities. 4. Can you briefly describe what implementation of your key activities entails/looks like? 5. What other actors (local, formal, informal, governance, business, NGOs/civil society groups, or NGOs) do you work with or are involved in carrying out your activities as Miranjaka? 5.1. Please describe how you work with these actors and why these links are important for effective implementation/your work? 6. What would a ‘successful’ implementation of each these activities look like? 6.1. Why would you consider it successful? What factors contribute to it being a success? (e.g. availability of time, contextual factors) 6.2. What role would you play or specific actions on your part would contribute to this success? 6.3. Note to Enumerators: Try to steer answers towards how activities are implemented, not just focusing on impact as a success. 7. What kind of challenges have you encountered while implementing your activities? (Probe specifically for COVID-19; availability of inputs and supplies; economic profitability) 7.1. What aspects of the implementation do these affect? (e.g. planning, availability of time/materials, transport) 7.2. How do these challenges affect your ability to implement activities? 7.3. How have you adapted to these obstacles/challenges? 7.4. If you did not/could not able to adapt, why not? What resources or other support were you missing? 7.5. How might these challenges be overcome in the future? Linkages 8. Above we asked about other organizations and actors with whom you partner in order to carry out your work. May you clarify: 8.1. How effective would you describe these links with key partners at the moment? IMPEL | Implementer-Led Evaluation and Learning 98 Annex E: Data Collection Tool 8.2. What are some effective elements of these partnerships? What do you consider important for ensuring you work together to implement activities effectively? 8.3. What specific outcomes have emerged out of these partnerships? What factors were responsible for these outcomes? Please be as specific as possible. 8.4. What limitations do you see in your partnerships at the moment? How could they be improved, or what would you change moving forward? (e.g. communications, type of support, expanding to other/different partners, geography or sectoral knowledge). Drought 9. How has the drought/kere affected your ability to implement activities? 9.1. What changes or adaptations have you had to make to the way you implement because of the drought/kere? 10. How did the drought/kere affect the effectiveness, impact, and potential outcomes of PISPs’ activities? 10.1. Did beneficiaries’ or members’ participation/involvement with PISPs and their activities change because of the drought? How and why? 11. In your opinion, what were the most important aspects or contributions of PISPs in the drought/kere context? 11.1. What were the most important aspects of the Maharo program as a whole? (Probe for: activities other than the emergency food assistance) Note to Enumerators: Be aware that not all respondents may be able to reflect on the Maharo program as a whole or have enough information to answer this question with lots of detail. It may be helpful to mention some of the other activities being examined in this Process Monitoring, but do not feel the need to spend too much time on this question if respondents are unsure how to answer. Inclusivity Gender 12. The baseline findings (refer to this in more detail if that is not clear) suggest that women in this region possess a high degree of ‘agency’ (ie, influence over household decisions). In what areas of household and community decision-making do women have more or less influence? ■ If more information needed: Causal Design was commissioned to conduct a baseline study for the IMPEL/Implementer-Led Design, Evidence, Analysis and Learning (IDEAL) activity. Our baseline findings on gender suggested that women had more agency than CRS/ADRA’s previous gender analyses suggested. 13. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for both men and women? Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 99 13.1. What does men and women’s participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ between men and women? 13.2. What considerations are important to ensuring equal access for both men and women when implementing programs (not just designing them)? 13.3. What limitations or gaps still exist in implementation efforts to ensure equal access? How might they be overcome? PWD 14. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for people with disabilities (PWDs)? 14.1. What does PWD participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ for men and women with disabilities? 14.2. What considerations are important to ensuring equal access for PWDs? 14.3. What limitations or gaps still exist in implementation efforts to ensure equal access for PWDs? How might they be overcome? Youth 15. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for youth? Note to Enumerators: ‘Youth’ refers to adolescents and young people aged 10 to 29. 15.1. What does youths’ participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ for young men and young women? 15.2. What considerations are important to ensuring equal access for youth? 15.3. What limitations or gaps still exist in implementation efforts to ensure equal access for youth? How might they be overcome? Is there anything else you would like to ask or say before we conclude the interview? Thank you so much for your time! We greatly appreciate your perspective on all of this! If you have any further questions or concerns, please do not hesitate to contact Eliane Ralison (+261 331165119), Sophie Turnbull (sophie.turnbull@causaldesign.com) or Matt Klick (matthew.klick@causaldesign.com). Approx Interview Length Delete as appropriate <30 mins / 30mins / 45 mins / 1hr / 1hr+ IMPEL | Implementer-Led Evaluation and Learning 100 Annex E: Data Collection Tool SILC KII Tool Background Information for Enumerators SILCs Savings and Internal Lending Communities (SILC) is a method of Village Savings and Loans (VSL) that CRS has used in the past. SILCs typically have between 15 and 30 members. SILCs are self￾managed through a General Assembly and a Management Committee. Each SILC has their own rules regarding the frequency and quantity of contributions. SILCs receive training from Field Agents and private service providers (PSPs). Field Agents are in turn supervised by Field Supervisors, who aim to train and certify Field Agents to become PSPs. As PSPs, Field Agents will operate as independent service providers on a market-led basis, and will continue to offer their training and services to communities for a fee. There are no rules on how members should spend savings. However, SILCs should receive messaging and sensitization from other actors, such as Care Groups, lead Mothers, and Producer Learning Centers, on aspects such as financial management, health, nutrition, WASH, and livelihoods. CRS is experimenting with separate SILCs for specific groups, such as youth, men, and women. Communication Protocol and Informed Consent Introduction and program information: Hello, my name is ________________. I work with Consultants Associates, a Madagascar-based data collection firm, in partnership with Causal Design, a research organization. We are conducting a process monitoring evaluation of CRS’ Maharo program, which aims to enhance food security and resilience in three districts of Madagascar: Ampanihy, Beloha, and Tsihombe. The Maharo Program includes a range of activities in a range of sectors, including SILCs. We specifically are examining how Maharo projects are being implemented, and anything that might be affecting implementation of these activities, either positively or negatively. We will be examining impact and effectiveness at a later time. Goal of this interview: We have invited you to this interview to share your views on the work of the SILCs and CRS’ support for SILCs under the Maharo program. We would like to hear your perspective on how Maharo’s support has been delivered, and your experiences implementing activities as members of a SILC. We would like to hear about both facilitators and obstacles, with the aim of ascertaining how implementation could be improved in the future. We would also like to hear about the inclusion of women, youth, and people with disabilities in SILCs and their activities; and effects of the drought/kere on program/activity implementation and activities. We aim for the interview to take up around 1 hour of your time. Managing Expectations: We cannot guarantee that every suggestion raised during this interview will be acted upon, but all suggestions will be analyzed and those that occur frequently may be taken forward as recommendations to program leadership. The analysis results will be shared with CRS, and it will be up to their discretion to share them with you and others. Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 101 Informed Consent: With your permission, I will be recording this interview. What you say during this interview will be shared with a team of analysts, but nobody else. We will remove your name and other personally identifying information from the recording, and your name will not appear in the analysis or any report. So please, speak openly to us – your voice is very important to help us learn about the successes and challenges of this program. For interviews with more than one respondent: We would like to hear from all of you, so I encourage you all to speak openly, even if you may disagree with what another participant is saying. I ask that you please speak one by one, so we can record all of your answers accurately. We will try to make sure everyone has a chance to speak. Do you agree with being interviewed by us, and to our conversation being recorded? [CONFIRM CONSENT] Thank you very much for agreeing to take part in this interview. Do you have any questions before we begin? Interview Information Enumerator/s Interview Date Original Interview Language French Malagasy English Approx Interview Length Delete as appropriate <30 mins 30mins 45 mins 1hr 1hr+ Respondent Information Respondent Type ● SILC Number of Respondents Interviewee Name/s Gender Age Delete as appropriate 18 - 30 years old 30 - 60 years old 60+ years District Commune IMPEL | Implementer-Led Evaluation and Learning 102 Annex E: Data Collection Tool Fokontany Observations Please provide details about the interview setting, interviewee’s responsiveness, and any other relevant information Interview Questions Background Firstly, we would like to hear some background about this SILC: 1. How long has this SILC been functional? 1.1. Who are its members and why? 1.2. What do members typically spend loans on and why? 2. From your understanding, what is the main purpose of this SILC? 3. What would you consider your main activities to be? 3.1. Which of these activities do you consider the most important? Why? (ie. Because they are most effective, most relevant/needed for the community, the ones conducted most often?) Effectiveness of CRS Support 4. What support for implementation have you received from CRS so far? What did this consist of? (e.g. SMART and other trainings, material or financial support) 4.1. What are your reflections, positive or negative, in the way this support was delivered? Could anything have been improved, and why? (e.g. format of trainings; any instances of delays) Effectiveness of Implementation Now I would like to hear about your experiences implementing activities as members of this SILC: Note to Enumerators: Ask for each of the activities mentioned in Q2 above; if more than 3 main activities were mentioned, ask about the 3 most important. Note to Enumerators: If respondents talk only about the effectiveness or impact of the activities, steer them back to the topic of how activities are implemented, or their role in and experience of carrying out these activities. 5. Can you briefly describe what implementation of your key activities entails/looks like? Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 103 6. What other actors (local, formal, informal, governance, business, NGOs/civil society groups, or NGOs) do you work with or are involved in carrying out your activities as Miranjaka? 6.1. Please describe how you work with these actors and why these links are important for effective implementation/your work? 7. What would a ‘successful’ implementation of each these activities look like? 7.1. Why would you consider it successful? What factors contribute to it being a success? (e.g. availability of time, contextual factors, governance, having separate SILCs for different groups) 7.2. What role would you play or specific actions on your part would contribute to this success? 7.3. Note to Enumerators: Try to steer answers towards how activities are implemented, not just focusing on impact as a success. 8. What kind of challenges have you encountered while implementing your activities? (Probe specifically for COVID-19, competition between other savings groups in the community) 8.1. What aspects of the implementation do these affect? (e.g. planning, availability of time/materials, transport, governance) 8.2. How do these challenges affect your ability to implement activities? 8.3. How have you adapted to these obstacles/challenges? 8.4. If you did not/could not able to adapt, why not? What resources or other support were you missing? 8.5. How might these challenges be overcome in the future? Linkages 9. Above we asked about other organizations and actors with whom you partner in order to carry out your work. May you clarify: 9.1. How effective would you describe these links with key partners at the moment? 9.2. What are some effective elements of these partnerships? What do you consider important for ensuring you work together to implement activities effectively? 9.3. What specific outcomes have emerged out of these partnerships? What factors were responsible for these outcomes? Please be as specific as possible. 9.4. What limitations do you see in your partnerships at the moment? How could they be improved, or what would you change moving forward? (e.g. communications, type of support, expanding to other/different partners, geography or sectoral knowledge). IMPEL | Implementer-Led Evaluation and Learning 104 Annex E: Data Collection Tool Drought 10. How has the drought/kere affected your ability to implement activities? 10.1. What changes or adaptations have you had to make to the way you implement because of the drought/kere? 11. How did the drought/kere affect the effectiveness, impact, and potential outcomes of this SILC’s activities? 11.1. Did beneficiaries’ or members’ participation with this SILC and its activities change because of the drought? How and why? 12. In your opinion, what were the most important aspects or contributions of this SILC in the drought/kere context? 12.1. What were the most important aspects of the Maharo program as a whole? (Probe for: activities other than the emergency food assistance) Note to Enumerators: Be aware that not all respondents may be able to reflect on the Maharo program as a whole or have enough information to answer this question with lots of detail. It may be helpful to mention some of the other activities being examined in this Process Monitoring, but do not feel the need to spend too much time on this question if respondents are unsure how to answer. Inclusivity Gender 13. The baseline findings (refer to this in more detail if that is not clear) suggest that women in this region possess a high degree of ‘agency’ (ie, influence over household decisions). In what areas of household and community decision-making do women have more or less influence? ■ If more information needed: Causal Design was commissioned to conduct a baseline study for the IMPEL/Implementer-Led Design, Evidence, Analysis and Learning (IDEAL) activity. Our baseline findings on gender suggested that women had more agency than CRS/ADRA’s previous gender analyses suggested. 14. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for both men and women? 14.1. What does men and women’s participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ between men and women? 14.2. What considerations are important to ensuring equal access for both men and women when implementing programs (not just designing them)? 14.3. What limitations or gaps still exist in implementation efforts to ensure equal access? How might they be overcome? Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 105 PWD 15. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for people with disabilities (PWDs)? 15.1. What does PWD participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ for men and women with disabilities? 15.2. What considerations are important to ensuring equal access for PWDs? 15.3. What limitations or gaps still exist in implementation efforts to ensure equal access for PWDs? How might they be overcome? IMPEL | Implementer-Led Evaluation and Learning 106 Annex E: Data Collection Tool Youth 16. How have you tailored/organized and conducted your activities to ensure equal access, benefits, and participation for youth? Note to Enumerators: ‘Youth’ refers to adolescents and young people aged 10 to 29. 16.1. What does youths’ participation in activities currently look like? In what aspects of the activity do they participate in? ■ How does participation differ for young men and young women? 16.2. What considerations are important to ensuring equal access for youth? 16.3. What limitations or gaps still exist in implementation efforts to ensure equal access for youth? How might they be overcome? Is there anything else you would like to ask or say before we conclude the interview? Thank you so much for your time! We greatly appreciate your perspective on all of this! If you have any further questions or concerns, please do not hesitate to contact Eliane Ralison (+261 331165119), Sophie Turnbull (sophie.turnbull@causaldesign.com) or Matt Klick (matthew.klick@causaldesign.com). Approx Interview Length Delete as appropriate <30 mins / 30mins / 45 mins / 1hr / 1hr+ Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 107 Validation Tool Cover Page – District-Level Checklist Enumerator Name District ACTIVITY Possible Validation Exercises Location (Note Commune and Fokontany, and note how close it is to where the activity takes place) Quick notes or observations (This could be around weather, trouble finding people, obstacles or positives) SILC  Observe: Any SILC documentation  Observe: SILC meeting/ training PISP  Speak to: Voucher customers  Observe: Voucher transactions in PISP shop Care Groups  RMM members  Observe: RMM training session . IMPEL | Implementer-Led Evaluation and Learning 108 Annex E: Data Collection Tool ACTIVITY Possible Validation Exercises Location (Note Commune and Fokontany, and note how close it is to where the activity takes place) Quick notes or observations (This could be around weather, trouble finding people, obstacles or positives) Lead Farmers  Speak to: Demonstration Plot members  Observe: Demonstration Plot tour (Priority)  Observe: Demonstration Plot group training session Miranjaka  Speak to: Safe Space discussion participants (Priority) .  Speak to: Dad’s School participants (Priority)  Speak to: KFF members  Observe: Safe Space discussion (Priority) .  Observe: Dad’s School (Priority)  Observe: KFF meeting Community Consultations  Speak to: Community consultation participants  Observe: Community Consultation Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 109 ACTIVITY Possible Validation Exercises Location (Note Commune and Fokontany, and note how close it is to where the activity takes place) Quick notes or observations (This could be around weather, trouble finding people, obstacles or positives) DRR Committees  Observe or confirm existence of updated DRR plan (Note: CD is meeting with ELS/CGRC as part of data collection already, so just an observation component for DRR) IMPEL | Implementer-Led Evaluation and Learning 110 Annex E: Data Collection Tool Observation Checklist SILC Enumerator Name Date District Commune Fokontany Observations Please provide details about the interview setting, participants’ responsiveness, and any other relevant information Informal Interviews / Observations Notes (If unable to complete, note this and reasons why e.g. lack of time, no respondents available, etc.) Observations SILC documentation Documentation could include a ledger with credits/debits, or meeting notes, etc. Note details on: - Size and number of transactions - Where documentation is kept and with who - Existence of formal records/documentation, whether a ledger, meeting minutes, or attendance sheet SILC meeting Observe: - Number, genders, and extent of participation among members - Presence/absence of members and leaders - Location of meetings - Management or leadership during the meetings - Process, rules and regulations of meetings - Content/topics of discussion Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 111 PISP Enumerator Name Date District Commune Fokontany Observations Please provide details about the interview setting, participants’ responsiveness, and any other relevant information Informal Interviews / Observations Notes (If unable to complete, note this and reasons why e.g. lack of time, no respondents available, etc.) Informal Interviews Voucher customers Ask about: - The process of receiving vouchers - What they think of the voucher program - What they think of the selected PISPs (voucher-receiving shops) - Assess their understanding/comfort of voucher system and any challenges - What are vouchers spent on - Confirm any processed payments Observations Voucher transactions in PISP shop Observe: - Interactions between shopkeeper and voucher recipients - Availability of supplies - Shopkeeper’s/owner’s understanding and comfort with vouchers as payment system - Gauge relevance/practicality of voucher system in this community or environment - What are vouchers spent on IMPEL | Implementer-Led Evaluation and Learning 112 Annex E: Data Collection Tool Care Groups Enumerator Name Date District Commune Fokontany Observations Please provide details about the interview setting, participants’ responsiveness, and any other relevant information Informal Interviews / Observations Notes (If unable to complete, note this and reasons why e.g. lack of time, no respondents available, etc.) Informal Interviews RMM Members Ask about: - Participation in Lead Mother/Care Group activities and frequency of these - Conduct of Lead Mothers - Information gleaned/effectiveness of meetings/trainings Observations RMM Training Observe: - Assess Lead Mother and participants’ interaction, engagement and demeanor - Location of training - Participants present and their characteristics - Assess content, its relevance and delivery Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 113 Lead Farmers Enumerator Name Date District Commune Fokontany Observations Please provide details about the interview setting, participants’ responsiveness, and any other relevant information Informal Interviews / Observations Notes (If unable to complete, note this and reasons why e.g. lack of time, no respondents available, etc.) Informal Interviews Demonstration Plot members Ask about: - Participation to date and frequency of meetings/trainings with Lead Farmer - Specific learnings/effectiveness of trainings - Relevance and ability to implement learnings Observations Demonstration Plot Observe: - Relative location of plot compared to community - State of plot (tilled, under cultivation, abandoned, fenced, etc.) - Inquire into nature and stage of demonstration(s) - Probe for: crop type, irrigation/seed/fertilizer focus, etc.? Demonstration Training Observe: - Assess Lead Farmer and participants’ interaction, engagement and demeanor - Location of training - Participants present and their characteristics - Assess content, its relevance and delivery IMPEL | Implementer-Led Evaluation and Learning 114 Annex E: Data Collection Tool Miranjaka Enumerator Name Date District Commune Fokontany Observations Please provide details about the interview setting, participants’ responsiveness, and any other relevant information Informal Interviews / Observations Notes (If unable to complete, note this and reasons why e.g. lack of time, no respondents available, etc.) Informal Interviews Safe Space Participants Ask about: - Frequencies of Safe Space discussions and reflections on their delivery - Relevance and understanding of content/curriculum - Any actions that resulted from discussions Dad’s School Participants Ask about: - Frequency of Dad’s School meetings and reflections on their delivery - Relevance and understanding of content/curriculum - Any actions that resulted from Dad’s School KFF Members Ask about: - Frequency of Miranjaka’s attendance at KFF meetings - Role and relevance of Miranjaka’s attendance - Relevance and understanding of content/curriculum - Any actions that resulted from Miranjaka’s attendance Observations Dad’s School Observe: - Miranjaka’s and participants’ interaction, engagement and demeanor - Location of Dad's School - Participants present and their characteristics - Assess content/curriculum, its relevance and delivery Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 115 Informal Interviews / Observations Notes (If unable to complete, note this and reasons why e.g. lack of time, no respondents available, etc.) KFF Meeting Observe: - Miranjaka’s and meeting participants’ interaction, engagement and demeanor - Location of meeting - Assess content, its relevance and delivery Community Consultations Enumerator Name Date District Commune Fokontany Observations Please provide details about the interview setting, participants’ responsiveness, and any other relevant information Informal Interviews / Observations Notes (If unable to complete, note this and reasons why e.g. lack of time, no respondents available, etc.) Informal Interviews Community Consultation Participants Ask about: - The nature of their participation - Key lessons from the consultations - Any observed changes in implementation that have been realized - Challenges or criticisms of process Observations Community Consultation Observe: - Participants present and their characteristics - Community Mobilizer’s and participants’ interaction, engagement and demeanor - Location of meeting IMPEL | Implementer-Led Evaluation and Learning 116 Annex E: Data Collection Tool Informal Interviews / Observations Notes (If unable to complete, note this and reasons why e.g. lack of time, no respondents available, etc.) - Assess content, its relevance and delivery - NOTE: If any physical assets are being/were built, try to see these in person and assess their existence, state (complete, under construction, etc.), and condition DRR Committees (CRGC/CCRGC/ELS) Enumerator Name Date District Commune Fokontany Observations Please provide details about the interview setting, participants’ responsiveness, and any other relevant information Informal Interviews / Observations Notes (If unable to complete, note this and reasons why e.g. lack of time, no respondents available, etc.) Observations DRR plans Observe: - The existence of any plans if possible - Publication date - Level of formality/detail - Where plans are kept and with who - NOTE: If any physical assets are being/were built, try to see these in person and assess their existence, state (complete, under construction, etc.), and condition Process Evaluation of the Maharo RFSA in Madagascar Annex E: Data Collection Tools 117