OUTCOME HARVESTING EVALUATION REPORT USAID Hay Tao Date: January, 2023 ii This document is made possible by the generous support of the American people through the United States Agency for International Development (USAID) and the U.S. President’s Emergency Plan for AIDS Relief under the terms of Award No. 72068718C00001. The contents are the responsibility of Pact and do not necessarily reflect the views of USAID or the United States Government. 1140 3rd St. NE, Suite 400 Washington, DC 20002 Tel: 202.466.5666 www.pactworld.org iii CONTENTS ABBREVIATIONS AND ACRONYMS ........................................................................................................................1 EXECUTIVE SUMMARY..................................................................................................................................................7 INTRODUCTION AND EVALUATION PURPOSE................................................................................................9 BACKGROUND: USAID HAY TAO.........................................................................................................................11 METHODOLOGY ..........................................................................................................................................................14 Selection of Outcome Harvesting 14 Outcome Harvesting Methodology 15 Step 1: Design the Harvest..............................................................................................................15 Step 2: Harvest Initial Outcomes...................................................................................................15 Step 3 & 4: Engage Stakeholders to Refine and Substantiate Outcome Statements.........16 Step 5: Analysis of Substantiated Outcomes...............................................................................18 Methodological Considerations and Limitations 23 FINDINGS.........................................................................................................................................................................25 Evaluation Question 1: What are the key outcomes to date of the USAID Hay Tao Activity? 25 Finding 1.1: Substantiators agreed that USAID Hay Tao outcomes were real, observable changes at the outcomes level, validating the existence of the outcomes...................27 Finding 1.2: The majority of outcomes were considered to be positive changes with a few turning points and no negative changes. ...............................................................................28 Finding 1.3: USAID Hay Tao Activity took a systemic approach to the change pathway, targeting outcomes using diverse strategies, as evidenced by the multiple types of change exhibited by each outcome........................................................................................29 Finding 1.4: Across the 33 outcomes, most exhibited a role change, and the least exhibited type of change was that of rules.............................................................................................31 Finding 1.5: The most exhibited type of change differed across SAs....................................31 Finding 1.6: All outcomes were a result of interventions focused on increasing technical capacity of stakeholders; providing new research, approaches, or technologies; and/or convening or mobilizing actors. ..............................................................................................32 Finding 1.7: The majority of outcomes (approximately 60%) demonstrate a practice and/or behavioral change, resulting from the application of new techniques,. .........................34 Finding 1.8: Approximately half of outcomes demonstrate improved natural resources management through new, improved, or enacted laws, regulations, frameworks, or standards. .....................................................................................................................................35 Finding 1.9: More than one third of outcomes demonstrate an increase in community involvement in natural resources management (community based natural resources management, CBNRM). ............................................................................................................36 Finding 1.10: More than 20% of outcomes are results that advance sustainable livelihoods. ........................................................................................................................................................38 Evaluation Question 2: To what extent do the outcomes align with the project’s theory of change? 39 Finding 2.1: Harvested outcomes appeared across the USAID Hay Tao results chains, affirming the Activity’s systems approach and multipronged strategy working at all levels and across a diverse range of stakeholders and sectors. ......................................39 Evaluation Question 3: To what extent are outcomes a result of the Activity’s contribution? 42 Finding 3.1: Substantiators, on average, highly agreed with the contributions analyses conducted by USAID Hay Tao................................................................................................43 iv Finding 3.2: Overall, USAID Hay Tao made a significant contribution to harvested outcomes......................................................................................................................................44 Evaluation Question 4: To what extent are the outcomes significant to people’s lives and sustainable NRM? 47 Finding 4.1: On average, USAID Hay Tao outcomes were deemed “highly significant to people’s lives and sustainable NRM” by substantiators. ...................................................47 Finding 4.2: Seven outcomes received a perfect significance score of 10 out of 10 from substantiators..............................................................................................................................48 Finding 4.3: The lowest significance score received from substantiators was that of 5, for two outcomes focused on multi-stakeholder advocacy, out of a possible score of 10. ........................................................................................................................................................48 Finding 4.4: The majority of harvested outcomes were beyond the “tipping point” stage of change, suggesting greater likelihood of endurance and continuation beyond the USAID Hay Tao Activity...........................................................................................................49 Finding 4.5: Substantiators determined that eight outcomes were at the enduring level of change, indicating that the outcomes have a high likelihood of continuing without further intervention...................................................................................................................50 Finding 4.6: USAID Hay Tao and substantiators disagree on the sustainability of outcomes related to SCE.............................................................................................................................52 Finding 4.7: There was no correlation between type of change (role, resource, rule, relationship, system) and the degree of sustainability; types of change were found at all degrees of sustainability, though outcomes with multiple types of change were internally consistent...................................................................................................................53 ATTACHMENTS ........................................................................................................................ 56 I. Summary of Harvested Outcomes & Substantiator Responses 56 II. Outcome Harvest Substantiation Form 66 III. SA Results Chains with Harvested Outcomes 70 1 ABBREVIATIONS AND ACRONYMS ABS Access and Benefit Sharing AD2M Appui au Développement du Menabe et du Melaky (Support for the Development of Menabe and Melaky) AJE Association des Journalistes Environnementaux (Association of Environmental Journalists) APJ Agent de Police Judiciaire (Judiciary Police Agent) APMA Aire Protégée Menabe Antimena (Menabe Antimena Protected Area) ARSIE Association du Réseau des Systèmes d'Information Environnementale (Association of the Network of Environmental Information Systems) AVG Alliance Voahary Gasy BIANCO Bureau Indépendant Anti-Corruption (Independent Anti-Corruption Bureau) CBNRM Community-Based Natural Resource Management CC Climate change CCP Conservation and Communities Project CESCE Conseil Économique, Sociale, Culturelle et Environnementale (Economic, Social, Cultural and Environmental Coucil) CITES Convention on International Trade in Endangered Species of Wild Fauna and Flora CJE Clinique Juridique Environnementale (Environmental Legal Clinic) CLA Collaborating, Learning, and Adapting CNFEREF Centre National de Formation, d'Etudes et de Recherche en Environnement et Foresterie (National Center for Training, Studies and Research in Environment and Forestry) CNFTF Centre National de Formation de Techniciens Forestiers (National Training Center for Forestry Technicians) CNKH Confédération Nationale Kolo Harena Sahavanona (Sahavanona National Kolo Harena Confederation) CNPE Coalition Nationale de Plaidoyer Environnemental (National Environmental Advocacy Coalition) COAP Code des Aires Protégées (Protected Area Code) CODE Communication pour le Développement du Menabe CRDA Commission de Réforme du Droit des Affaires (Business and Corporate Law Reform Commission) CRORFAD Comité Régional pour l’Opérationnalisation des Réserves Foncières et de l’Agriculture Durable (Regional Committee for the Operationalization of Land Reserves and Sustainable Agriculture) CSO Civil Society Organization 2 CSR Corporate Social Responsibility CSSM Commission de Supervision de la Stratégie Mangroves (Commission for the Supervision of the Mangrove Strategy) CTD Collectivité Territoriale Décentralisée (Decentralized Territorial Collectivity) CTM Confédération du Tourisme à Madagascar (Confederation of Tourism in Madagascar) DAPRNE Direction des Aires Protégées, des Ressources Naturelles renouvelables et des Ecosystèmes (Department of Protected Areas, Renewable Natural Resources and Ecosystems) DFC Droit Foncier Communautaire (community land rights) DPSE Direction de la Programmation et du Suivi-Evaluation (Department of Programming and Monitoring-Evaluation) DRAE Direction Régionale de l'Agriculture et de l'Elevage (Regional Department of Agriculture and Livestock) DREDD Direction Régionale de l'Environnement et du Développement Durable (Regional Department of Environment and Sustainable Development) DULCC Direction de l’Unité de Lutte Contre la Corruption (Directorate of the Anti￾corruption Unit) EDBM Economic Development Board of Madagascar EESS Evaluation Environnementale et Sociale Stratégique (Strategic Environmental and Social Assessment) EFTA École de Formation des Techniciens Agricoles (Training School for Agricultural Technicians) EIESS Evaluation d’impact environnemental social et stratégique (strategic environmental and social impact assessment) ENMG École Nationale de la Magistrature et des Greffes (National School of Magistracy and Court Registry) FAPBM Fondation pour les Aires Protégées et la Biodiversité de Madagascar (Madagascar Biodiversity Fund) FID Fonds d’Investissement pour le Développement (Development Investment Fund) FiTI Fisheries Transparency Initiative FIVE Force Instrumentale pour Valoriser Ensemble Menabe FIVMPAMA Fivondronan'ny Mpandraharaha Malagasy (Malagasy Entrepreneurs’ Association) FIVOI Fiharoan'ny Vondron'Olona Ifotony eto Menabe FNF Fonds National Foncier (National Land Tenure Fund) FOFIFA Foibem-pirenena momba ny Fikarohana ampiharina amin'ny Fampandrosoana eny Ambanivohitra (National Center for Applied Research to Rural Development) FORMAPROD Programme de Formation Professionnelle et d'Amélioration de la Productivité Agricole (Program for Vocational Training and Improvement of Agricultural Productivity) 3 FVTM Federasionan'ny Vehivavy Tantsaha Malagasy (Federation of Malagasy Women Farmers) FWLP Fisherwomen Leadership Program FY Fiscal Year GEF/SGP Global Environment Facility/Small Grants Program GELOSE Gestion Locale Sécurisée (secured local management) GGC Green Globe Certification GIS Geographic Information System GIZ Deutsche Gesellschaft für Internationale Zusammenarbeit GOM Government of Madagascar IHSM Institut Halieutique et des Sciences Marines (Fishery and Marine Sciences Institute) ITO International Tuna Organization IUCN International Union for Conservation of Nature KfW Kreditanstalt für Wiederaufbau KMD Kômity Mpanatanteraka ny Dina (Dina executive committee) LMMA Locally Managed Marine Area LOA Life Of Activity LPDE Lettre de Politique de la Décentralisation Emergente (Policy Letter on Emerging Decentralization) MATSF Ministère de l'Aménagement du Territoire et des Services Fonciers (Ministry of Land Use Planning and of Tenure Services) MECIE Mise en Compatibilité des Investissements avec l’Environnement (Decree on Compatibility of Investments with the Environment) MEDD Ministère de l'Environnement et du Développement Durable (Ministry of Environment and Sustainable Development) MEF Ministère de l'Économie et des Finances (Ministry of Economy and Finance) MEL Monitoring, Evaluation, and Learning MEN Ministère de l’Education Nationale (Ministry of National Education) MERL Monitoring, Evaluation, Research, and Learning METFP Ministère de l'Enseignement Technique et de la Formation Professionnelle (Ministry of Technical Education and Vocational Training) MGA Malagasy Ariary MICC Ministère de l’Industrialisation, du Commerce et de la Consommation (Ministry of Industrialization, Trade and Consumption) MID Ministère de l'Intérieur et de la Décentralisation (Ministry of Interior and Decentralization) MIHARI MItantana HArena an-dRanomasina avy eny Ifotony (Madagascar LMMA Network) 4 MINAE Ministère de l'Agriculture et de l'Élevage (Ministry of Agriculture and Livestock) MinJus Ministère de la Justice (Ministry of Justice) MinTour Ministère du Tourisme (Ministry of Tourism) MNP Madagascar National Parks MOU Memorandum Of Understanding MPA Marine Protected Area MPEB Ministère de la Pêche et de l'Économie Bleue (Ministry of Fisheries and of the Blue Economy) MSP Marine spatial planning N/A Not Applicable or Not Available NCV Natural Capital Valuation NGO Non-Governmental Organization NR Natural Resource NRM Natural Resource management NTER Non-Tax Environmental Revenue OH Outcome Harvesting ONE Office National pour l'Environnement (National Office for the Environment) OPJ Officier de Police Judiciaire (Judiciary Police Officer) OSPJ Officier Supérieur de Police Judiciaire (Superior Judiciary Police Officer) PA Protected Area PAG Plan d‘Aménagement et de Gestion (Development and Management Plan) PALoS Plan d'Aménagement Local Simplifié (Simplified Local Management Plan) PANEDD Plan d’Action National de l’Environnement et du Développement Durable (National Action Plan of the Environment and Sustainable Development) PAP Plan d'Aménagement des Pêcheries (Fisheries Management Plan) PEJAA Programme de Promotion de l'Entreprenariat des Jeunes dans l'Agriculture et l'Agro￾industrie (Program for the Promotion of Youth Entrepreneurship in Agriculture and Agribusiness) PEER Partnerships for Enhanced Engagement in Research PES payment for Ecosystem Services PHE Population, Health, and Environment PNDE Plan National de Décentralisation Émergente (National Plan on Emerging Decentralization) PPNT Propriété Privée Non Titrée (private property without land title) PPV Producteurs et Préparateurs de Vanille (vanilla producers and processor) Q Quarter 5 RDQA Raw Data Quality Assessment ReNiAla-CC Réseau National pour la lutte contre le Changement Climatique (National Network to Fight Against Climate Change) RNDTF Référentiel National du Diplôme des Techniciens Forestiers (National Standards for the Training and Diploma of Forestry Technicians) RNM Radio Nationale de Madagascar (National Radio of Madagascar) SA Strategic Approach SAC Schéma d'Aménagement Communal (Commune Management Plan) SAF/FJKM Sampan'Asa momba ny Fampandrosoana/Fiangonan'I Jesoa Kristy eto Madagasikara (Branch of Activity concerning the Development/Church of Jesus Christ in Madagascar) SAIC Schéma d'Aménagement Inter-Communal (Inter-Communal Development Plan) SEG Secrétariat d'État auprès du Ministère de la Défense Nationale chargé de la Gendarmerie (Secretary of State to the Ministry of National Defense in charge of the Gendarmerie) SIAGAP Système d'Information et d'Aide à la Gestion des Aires Protégées (Information and Support System for Protected Area Management) SIF Solidarité des Intervenants sur le Foncier SIP Système d’information Pêches (Fisheries Information System) SNRSE Stratégie nationale sur la responsabilité sociétale des entreprises (CSR national strategy) SMART Spatial Monitoring and Reporting Tool STD Services Techniques Déconcentrés (Devolved Technical Services) SVPN Mafatoky Sustainable Vanilla for People and Nature “Mafatoky” SWIOFISH South West Indian Ocean Fisheries Governance and Shared Growth Project TA Technical Assistance TAFO MIHAAVO Tambazotran' ny Fokonolona Mitantana Harena Voajanahary (Network of Grassroots Communities Managing Natural Resources) TBD To Be Determined TGRH Transfert de Gestion des Ressources Halieutiques (Management transfer of halieutic resources) TGRNR Transfert de Gestion des Ressources Naturelles Renouvelables (management transfer of renewable natural resources) TOC Theory Of Change TOR Terms Of Reference TSS Terrain à Statut Spécifique (Land with Specific Status) URI-CRC University of Rhode Island, Coastal Resources Center USAID United States Agency for International Development 6 USD United States dollar USG United States Government VOI Vondron'Olona Ifotony (community-based organization) WRI World Resources Institute WWF World Wide Fund for Nature 7 EXECUTIVE SUMMARY In 2021-2022, USAID Hay Tao undertook an internal evaluation that aimed to explore project outcomes and to assess the project’s contribution to those outcomes and their relevance to larger project goals. The main research questions guiding the evaluation were: (1) What are the key outcomes to date of the USAID Hay Tao Activity? (2) To what extent do the outcomes align with the Activity’s theory of change? (3) To what extent are outcomes a result of the Activity’s contribution? (4) To what extent are the outcomes significant to people’s lives and to sustainable natural resources management? USAID Hay Tao selected the Outcome Harvesting (OH) methodology for its participatory and complexity-responsive approach. In total, USAID Hay Tao “harvested” 33 outcomes, spread across the Activity’s five strategic approaches. Individuals external to the Activity who were knowledgeable served as independent sources who “substantiated” the outcomes, providing an objective assessment. Selection of Featured Outcomes 1.1 Improved vanilla cultivation techniques were applied by vanilla producers and processors. 1.3 65% of PA have renewed their PAG using the new PAG framework. 2.1 More effective and timely patrols to fight fires, deforestation, and respond to other PA infringements, with information gathered from innovative technologies, including Forest Watcher and SMART. 2.3 National Environment Office commits to take over the management of the Hay Natiora data portal and integrating the portal into its existing systems ensuring sustainability. 2.6 Investigative journalists utilized improved capacity to provide higher quality coverage of land tenure issues and to prepare more effective journalistic products. 3.1 Social and conservation entrepreneurship (SCE) advanced after the submission of the draft law and the implementation of new SCE business models. 3.5 The national Natural Capital (NC) network established to integrate NC valuation in governmental decision-making and investment/economic activities. 3.6 Eleven additional PAs across an additional 501,800 hectares financially supported as a result of $58 million+ addition to the Madagascar Protected Areas and Biodiversity Fund (FAPBM). 4.3 Improved ecological and socio-economic land management in 38 communes in Menabe and MaMaBay (Outcome 4.1); four new communal management plans (SACs) in APMA addressing environmental and social concerns and harmonized with PA management. 4.5 Dinan'i Menabe – affecting all the 5 districts of the region – improved and formalized to include sustainable environmental considerations and CBNRM (Outcome 4.4); update of the Antongil Bay Dinabe concerning the exploitation of fisheries resources. 4.7 45% of targeted OPJs and OSPJs applied capacity development and tools for environmental crime detection, arrest, prosecution, conviction, and penalty, resulting in strengthened law enforcement. 5.2 Increase in community reporting of environmental crimes through AVG's 512 hotline and in￾person visits to the environmental legal clinics (CJEs). 5.3 Since 2019, court decisions regarding environmental crimes have increasingly utilized evidence provided by communities and informants in Morondava and Maroantsetra. 5.5 Launch of the national CSR strategy (SNRSE) and mobilization around a shared vision for CSR environmental issues. 8 Selection of Featured Findings • USAID Hay Tao Activity took a systemic approach to the change pathway, targeting outcomes using diverse strategies. (Finding 1.3) • The majority of outcomes (approximately 60%) demonstrate a practice and/or behavioral change, resulting from the application of new techniques. (Finding 1.7) • Approximately half of outcomes demonstrate improved natural resources management through new, improved, or enacted laws, regulations, frameworks, or standards. (Finding 1.8) • More than one third of outcomes demonstrate an increase in community involvement in natural resources management (community based natural resources management, CBNRM). (Finding 1.9) • Overall, USAID Hay Tao made a significant contribution to harvested outcomes. (Finding 3.2) • On average, USAID Hay Tao outcomes were deemed “highly significant to people’s lives and sustainable NRM” by substantiators. (Finding 4.1) • The majority of harvested outcomes were beyond the “tipping point” stage of change, suggesting greater likelihood of endurance and continuation beyond the USAID Hay Tao Activity. (Finding 4.4) Conclusion and Recommendation The internal evaluation findings were overall very positive, surfaced unanticipated results, and confirmed expected results. Substantiators independently verified the outcomes. The analytic framework provided an assessment of outcome sustainability. Outcomes with higher levels of sustainability are more likely to endure without future intervention, or with light touch intervention approaches. The evaluation team recommends that outcomes with lower levels of sustainability be supported in future interventions, to leverage gains made by USAID Hay Tao. 9 INTRODUCTION AND EVALUATION PURPOSE USAID Hay Tao is a USAID-funded Activity in Madagascar, implemented by Pact in a consortium with the World Resources Institute (WRI), University of Rhode Island Coastal Resources Center (URI-CRC), Alliance Voahary Gasy (AVG), and Solidarité des Intervenants pour le Foncier (SIF). USAID Hay Tao aims broadly to improve the management of natural resources in Madagascar, and more specifically to enhance and reinforce local-level outcomes by creating an enabling and empowering environment for effective community-based management and protection of biodiversity resources in Madagascar. In 2021-2022, USAID Hay Tao undertook an internal evaluation that aimed to explore project outcomes and to assess the project’s contribution to those outcomes and their relevance to larger project goals. In this way, the evaluation aimed to complement the indicators in the Activity’s monitoring, evaluation, and learning (MEL) plan to provide additional findings regarding Activity results that were not captured through predetermined metrics. The primary intended audience for the evaluation is USAID Hay Tao staff and partners, who intend to use the findings as part of learning review processes. An additional audience is the funder, USAID, as well as peers in the larger sector working on climate change mitigation and natural resource management, especially in Madagascar. The main research questions guiding the evaluation were: 1. What are the key outcomes to date of the USAID Hay Tao Activity? 2. To what extent do the outcomes align with the Activity’s theory of change? 3. To what extent are outcomes a result of the Activity’s contribution? 4. To what extent are the outcomes significant to people’s lives and sustainable to natural resources management? Evaluation methodologies are designed to respond to research questions within resource constraints but are also informed by core principles. The main principles informing this evaluation included: 1. Participation of key stakeholders: Because USAID Hay Tao works at the systems level, stakeholder buy-in is critical for the operationalization of learnings and recommendations. It was important to USAID Hay Tao staff to achieve stakeholder buy-in for the evaluation, and the USAID Hay Tao team determined that a participatory approach was most conducive. The final methodology therefore required a participatory approach. 2. Complexity-responsiveness: USAID Hay Tao staff were interested in exploring outcomes rather than merely affirming expected results, because theories of change were adjusted over the life of activity and strategic shifts took place. The Activity also operated in a challenging context and at a complex system level, and it was therefore important to take a complexity-aware approach which is often more inductive and exploratory. These principles were best met, and questions best answered, using the Outcome Harvesting (OH) methodology. OH was selected because: Its methodology is flexible enough to enable a highly participatory approach and empower local stakeholders to identify and assess what outcomes are meaningful to them. Pact’s organizational guiding star is Engaged Communities – supporting communities to own and lead their own development – and therefore empowering stakeholders through evaluative processes is a critical approach the organization takes. In this case, the Activity team wanted to focus on the 10 primary outcomes identified as significant by local partners, enabling local partners to articulate their criteria of success. OH can be implemented in a manner that is highly participatory and was done so in this process. The methodology is appropriate for a complex project, defined as one that works in a dynamic context with uncertain causal pathways and unanticipated results. The USAID Hay Tao Activity had undergone programmatic shifts, including at the level of the objective, and implemented a significant diversity of activities, which were challenging to convey in a unified story to an external audience. An evaluation is an important opportunity to step back and assess the unanticipated and rethink how change is understood. In addition, USAID Hay Tao worked toward complex outcomes with multiple moving parts that cut across the project’s strategic approaches (i.e. objectives). OH’s emphasis on exploration – inductive – provided an opportunity to recode outcomes in a manner that was not limited by how the project was initially framed, such as through its reporting systems. The methodology requires substantiation and an assessment of the degree of relevance of the outcome to the problem at hand. USAID Hay Tao project staff wanted more rigorous evidence of outcomes, beyond anecdotes and reductive indicators. However, most of the Activity outcomes were so complex and required significant coordination and collaboration – for which USAID Hay Tao often played a convening or formative role – that many outcomes were hard to measure definitively and tangibly. For example, USAID Hay Tao works closely with the USAID Mikajy Activity, by developing resources implemented by the sister program. For this reason, it is often difficult to isolate USAID Hay Tao contributions to outcomes. The OH methodology enables this contribution assessment via external validation through substantiation with third party or expert observers. The internal evaluation was led by the USAID Hay Tao MEL Manager and her team, with methodological support from Pact’s DC-based global evaluation team and technical and analytic support from Pact’s global natural resources management team. 11 BACKGROUND: USAID HAY TAO USAID/Madagascar’s Conservation and Communities Project (CCP) helps protect Madagascar’s natural capital through improved conservation of the country’s unique biodiversity (Nature), promotion of resilient livelihoods to provide alternatives to unsustainable natural resource management (NRM) practices (Wealth), and concrete actions to secure effective local management and ownership of natural resources (NRs; Power). Within this framework and directly aligning with CCP’s goals, the USAID Hay Tao Activity works alongside the site-based USAID Mikajy Activity, also under CCP, to enhance and reinforce local￾level outcomes by creating an enabling and empowering environment for effective community-based management and protection of biodiversity resources in Madagascar. USAID Hay Tao’s biodiversity interests especially focus on reducing vulnerability to climate change (CC), reducing unsustainable NRs use, and combatting wildlife trafficking. Further, CCP and by extension USAID Hay Tao directly align with the third goal of the United States Government (USG) Mission to Madagascar’s Integrated Country Strategy (ICS) under Objective 3.1: Conserve biodiversity and secure NRs while promoting resilient livelihoods in Madagascar. As part of the consortium implementing the Activity, Pact provides technical leadership for all policy analysis, institutional development, and network strengthening aspects. World Resources Institute (WRI) develops the data portal, which enhances stakeholders’ data management and use to improve community￾based natural resource management (CBNRM) and protected area (PA) management by integrating the use of data and technology. University of Rhode Island Coastal Resources Center (URI-CRC) leads activities focused on coastal communities and marine PAs, engaging the Ministry of Fisheries and Blue Economy (MPEB), establishing competency-based certification programs for national- and community-level PA managers, and improving the policy framework and evidence base for integrated population, health, and environment (PHE) efforts. Alliance Voahary Gasy (AVG) serves as a platform for advocacy, networking, outreach, information dissemination, and civil society capacity development, supporting the development of training curricula and combatting NR crimes to improve NRM. Solidarité des Intervenants pour le Foncier (SIF) plays a key role in improving policy on NR and land tenure rights and in supporting policy reviews and analyses, technical assistance (TA), and advocacy targeting decision￾makers. The USAID Hay Tao Activity’s overall approach and strategy are based on the following theory of change (TOC): IF evidence-based, sound policy and legal frameworks related to land tenure, wildlife, and NRs are established and harmonized across sectors and governance levels ... and IF ... stakeholder (Government of Madagascar [GOM], civil society organizations [CSOs], community-based NR managers, NGOs, private sector) capacity is strengthened to meaningfully participate in developing and implementing policies and laws at all levels, ... THEN ... biodiversity will be conserved, and NRs secured while resilient livelihoods will be promoted. 12 USAID Hay Tao addresses this TOC through five Strategic Approaches (SAs): 1. Support conservation stakeholders on sustainable livelihoods, NRM, and environmental justice through capacity development 2. Improve and make available data and information for evidence-based decision-making on conservation and sustainable livelihoods 3. Identify and foster innovative financing options for community-based conservation 4. Strengthen public sector institutions in policy-making and enforcement of tenure rights and NRM 5. Support civil society and private sector coalitions to advocate for environmental and social governance through constructive engagement The Activity’s results framework, in the graphic below, is aligned to CCP’s objectives and illustrates the overall Activity logic and how its five SAs build up to achieve CCP’s objectives. 13 14 METHODOLOGY Selection of Outcome Harvesting Outcome Harvesting is an evaluation methodology that was originally developed by Ricardo Wilson-Grau and colleagues, and fielded starting in approximately 2004 before being codified in an externally available publication in 2012.1 Its contribution to the field is a methodology that does not rely on predetermined objectives to set criteria of success and measure progress towards that success, but rather looks backwards by identifying what has changed and the extent of the contribution of the project to that change. In this sense, it is more of an inductive, exploratory approach.2 The typical monitoring approach to measuring outcomes looks at predetermined metrics (often presented as indicators) and assesses change stemming from activities: The Outcome Harvesting approach to measuring outcomes looks backward: it explores what outcomes have been observed, and uncovers the project’s relative contribution to those outcomes: OH is an evaluation methodology that is particularly well suited to complex contexts, defined here using Michael Quinn Patton’s framing of complexity as a factor of the degree of agreement actors have over a problem and their certainty over its solution.3 In situations of high complexity, program strategies must adapt or pivot to remain contextually relevant, the theory of change may be formative or not well evidenced because cause and effect are not clear, and the project may anticipate there to be a number of unexpected, both positive and negative, outcomes. OH helps in these situations because it begins by identifying an observed outcome of interest, and then substantiates the contribution of a project to that outcome. While the main methodology is outlined in several publications, OH – like most complexity-responsive methodologies – is necessarily adaptable to the context in which it is applied. In the case of USAID Hay Tao, the Activity team tailored the level of stakeholder participation and which system stakeholders were engaged in substantiation (more discussed below under substantiation). In addition, the main methodology provides little to no specification of what analytic framework to utilize, which enables evaluations to adapt 1 Ricardo Wilson-Grau and Heather Britt, Outcome Harvesting. (Cairo: Ford Foundation: 2012). 2 For more information on OH, please see the original technical brief Outcome Harvesting, by Ricardo Wilson-Grau and Heather Britt, published in 2012 by the Ford Foundation; or the full-length book, Outcome Harvesting: Principles, Steps, and Evaluation Applications by Ricardo Wilson-Grau, published in 2018 by Information Age Publishing. 3 Michael Quinn Patton, Developmental evaluation: Applying complexity concepts to enhance innovation and use. (Guilford Press: 2011) (1) What activities did we do? (2) What were the outputs of the activities? (3) What were the outcomes of the activities? (1) What is the observed/ hypothesized outcome? (2) How did the change agent contribute? (3) Can it be substantiated? 15 analysis to the main lines of inquiry. In the case of USAID Hay Tao, the analytic framework prioritized the following: 1. Consideration of outcomes related to the various Strategic Approaches of the project to understand progress toward main objectives based on the theories of change 2. Understanding the degree of sustainability or relative permanence of the outcome through a systems-framework approach 3. Understanding the extent to which the outcomes are perceived by system stakeholders as important to larger natural resources management in Madagascar The remainder of this methodology section outlines the main steps taken by USAID Hay Tao in keeping with the OH methodology, and notes areas that were tailored to enable context responsiveness. Outcome Harvesting Methodology Step 1: Design the Harvest The OH approach taken by USAID Hay Tao required participation from USAID Hay Tao program stakeholders, specifically consortium partners: these are organizations that work with Pact to implement the USAID Hay Tao program, and lead in various technical aspects. For this reason, the evaluation began with internal workshops to introduce the methodology and build capacity to understand and implement it. These discussions began when the COVID-19 pandemic broke out globally. Over the course of several workshops in 2020 and 2021, the USAID Hay Tao MEL team, with support from Pact’s global evaluation team, introduced the main purpose of OH as well as the methodology. All workshops took place virtually. At this time, the primary research questions were established and agreed upon. Step 2: Harvest Initial Outcomes To draft initial outcomes, the team reviewed existing project documentation, and organized two internal virtual workshops with all consortium partners. The workshops were facilitated by Activity MEL staff and took place in May 2021. Participants brainstormed an extended list of outcomes and drafted short descriptions. After this workshop, in July 2021, ‘coaches’ – individuals tasked with being points of contacts and facilitators for their topical areas – were selected from among the consortium technical teams and prepared to work with the Activity team to formalize the outcomes. In a subsequent workshop, in October 2021, coaches led participants to narrow their list of outcomes to those that were most significant to Activity goals and draft complete outcome descriptions. (See Attachment III: Outcome Harvest Worksheet) Complete outcome statements built from existing documentation and observation, and included: outcome statement, problem statement, contribution statement, and significance statement. Is it important to clarify the definition of “outcome” used by USAID Hay Tao. The concept is defined in the Outcome Harvesting methodology as “a change in the behaviour, relationships, actions, activities, policies, or practices of an individual, group, community, organization, or institution” (Wilson-Grau and USAID defines ‘outcome’ broadly. As described in USAID’s Glossary of ADS Terms, “Outcomes are any result higher than an output to which a given output contributes, but for which it might not be solely responsible.” The Outcome Harves�ng methodology defines outcomes more specifically as changes observed in the targeted individual, group, community, organization, or institution. 16 Britt, 2013). This definition is in keeping with, but also more specific and narrowly focused, than the definition used by USAID (see text box). Outcomes using the OH definition can range from immediate outcomes, to intermediate outcomes, to long-term outcomes. This ‘degree’ of outcome is determined for the most part by how significant the outcome is and how distant it is from the original intervention along the results chain, rather than being determined by the time frame: it is possible that an intermediate result occurs relatively quickly, or that an immediate result takes time to manifest. The degree of outcome has more to do with the degree of control an intervention has over the outcome: outcomes that are closer to the intervention tend to be more immediate, whereas outcomes that have resulted from a series of subsequent actions taken by other stakeholders over time tend to be more long-term. This is visualized best in a graph included in a guidebook on the methodology Outcome Mapping4 that plots the degree of control an intervention has against a progression along the classic results chain. That control starts off high, with the intervention (represented by the bolded line) starting off with high control at the input level, and that control reduces over time, as other stakeholders or system actors (represented by the dotted line) exert more control over the outcomes. Similarly, external stakeholders start off with low control at the input level and exert high control at the impact level: This understanding of outcomes – that there is a spectrum that is determined by degree of control by the intervention or control by system actors – is why this evaluation considers the degree of sustainability of an outcome (see Step 5: Analysis). Step 3 & 4: Engage Stakeholders to Refine and Substantiate Outcome Statements The OH methodology as originally construed did not specify how substantiation should take place, for example how many outcomes should be substantiated and by how many substantiators, and research teams have taken different approaches. Substantiation entails engaging third-party stakeholders in reviewing the details of the outcomes and giving their assessment of the accuracy of those details. Some OH practitioners substantiate all, some only substantiate outcomes for which there is no existing evidence, some take a random sample, and some substantiate those what seem to vary from what is perceived based on existing information. The USAID Hay Tao team determined to substantiate all proposed outcomes, 4 The graphic was originally attributed to Terry Smutylo. See Figure 2 on p. 9 in Sarah Earl et al., Outcome Mapping: Building Learning and Reflection into Development Programs (International Development Research Center, 2001) 17 for two reasons: (1) the team wanted to triangulate third-party perspectives to increase rigor and reduce bias in the contribution analysis, and (2) given the systems-level nature of the USAID Hay Tao activity, the team also wanted third-party perspectives regarding the value of the outcome to natural resources management overall in Madagascar. A total of 89 individuals participated as substantiators across the 33 outcomes: 89 total substantiators 60 men 29 women 40 national (39) & international (1) substantiators 49 subnational substantiators Community (13), Commune (5), District (8), Regional (23) Substantiation took place over six months (March-September, 2022) through four national and regional (Maroantsetra and Morondava) workshops, as well as through individual in-person interviews and directly emailed substantiation worksheets for those unable to attend the in-person workshops or participate in in-person interviews. Number of in-person substantiation workshops Number of substantiators participating in in-person workshops Number of substantiators participating via emailed worksheets Number of substantiators participating via individual interviews 4 63 17 9 During in-person workshops, participants were organized into five groups, one for each of USAID Hay Tao’s strategic approaches (SAs), depending on the participants’ perspective or expertise. In each group, USAID Hay Tao OH coaches presented each harvested outcome by providing outcome descriptions and responding to questions from participants. Collectively, participants provided recommendations on adapting the harvested outcome language, and the outcomes language were subsequently adjusted. Final outcomes language reflects these suggestions. Then, individually, participants completed substantiation worksheets for each outcome for which they felt they had sufficient knowledge to corroborate. The forms essentially asked for them to confirm or deny the information collected about each outcome and to provide any additional perspective (See Attachment III: Outcome Harvest Substantiation Form). These forms were collected by OH coaches and used during analysis. After self-reflection, participants then engaged in a discussion facilitated by USAID Hay Tao coaches to share perspectives. Twenty-six substantiators were unable to attend the workshops for logistical or timing issues. Those that participated via interview were interviewed by a USAID Hay Tao team member familiar with the topic, who described the outcome and led the substantiator through an interview similar in fashion as the 18 workshop, recording responses in the substantiation worksheet. Those that participated via email were sent the substantiation worksheet that workshop participants utilized. Substantiators were selected for their perspectives and expertise. Because USAID Hay Tao works at the systems level, it is impossible to identify an individual who has knowledge about the outcome yet who does not have some relationship with it. Therefore, the evaluation team worked to triangulate each outcome with more than one substantiator wherever possible, and identified substantiators at various levels: national, regional, and community. USAID Hay Tao attained informed consent from all substantiators, and clearly explained that all perspectives – negative and positive – were equally valuable to the evaluation and would not impact the future of USAID Hay Tao. Range of Substantiators Per Outcome Mean Number of Substantiators Per Outcome 1-14 3.5 Step 5: Analysis of Substantiated Outcomes To reduce bias, data analysis was led by a quasi-external evaluator from Pact’s Washington, DC based global evaluation team, a member of the Learning, Evidence and Impact (LEAP) team, with knowledge about the USAID Hay Tao program (hereafter, “Analyst”), and reviewed by another quasi-external evaluator with no knowledge about the USAID Hay Tao program, also from the LEAP team. The USAID Hay Tao MEL team submitted all data in the form of substantiation worksheets, outcome descriptions, and substantiator lists. To respond to the research questions, USAID Hay Tao took the following approaches: Research Question Analytic Approach What are the key outcomes to date of the USAID Hay Tao Activity? Harvested outcomes were described qualitatively and quantitatively. Qualitative. The Analyst performed inductive, qualitative coding by first reading through the descriptions and drafting an initial list of descriptive codes. Then, the Analyst performed successively deeper analysis through multiple reads of the outcome descriptions to develop and refine pattern codes. The coded narratives were then reviewed per code, and findings developed based on the content. Coded narratives were also assessed across codes, and findings developed based on the content. The results of the inductive analysis are presented in the findings section, organized by finding. Quantitative. In addition to qualitative analysis, the Analyst analyzed the outcomes using descriptive statistics. We assessed the distribution of outcomes across SAs, the category of change (coded as: positive change, turning point, negative change), and the type of change. For the latter, USAID Hay Tao built from the USAID 5-R Systems framework5 which understands a 5 For more information on this framework, please see: https://usaidlearninglab.org/sites/default/files/resource/files/5rs_techncial_note_ver_2_1_final.pdf 19 Research Question Analytic Approach system along five dimensions: resource, role, rule, relationship, and result. USAID notes, “Collectively these 5Rs can serve as a lens for assessing local systems and a guide for identifying and monitoring interventions designed to strengthen them.” Figure 1 Visual of USAID 5R Framework USAID Hay Tao modified this framework slightly to (i) incorporate resources as a type of outcome rather than just an intervention input, and (ii) to replace the “R” of “result” (which generally describes an outcome), with a type of change (an outcome) that focuses on the nature of a system – of interconnected and mutually reinforcing parts – and not just components of those parts (this is in keeping with the USAID description of a result from a systems change). The “R” was therefore replaced with a “system” result. In short, the modification redescribes each of the 5-Rs as a type of change. USAID Hay Tao thereby used the following definitions: Role: Systems involve numerous actors who take on various defined roles: producer, consumer, funder, and advocate. A role change describes a new way of acting that exhibits a difference in how responsibility is understood and realized through changed behaviors. Relationship: Interactions between the actors in a system create particular forms of relationships. Some may be commercial, others more administrative and hierarchical. A relationship change describes a new way that individuals interact that exhibits a difference in their interaction, for example whether these demonstrate changed cooperation or coordination. 20 Research Question Analytic Approach Resource: Systems need resources to transform inputs into outputs. Resources can be financial (budget, donation), technical (guidance documents), technological (digital platforms), material infrastructure). A resource change describes a resource that is introduced to the system or an existing resource that is repurposed to greater effect, enabling system actors to pursue their actions. Rule: Rules govern systems, both formally and informally. Rules define or assign roles, determine the nature of relationships between actors and establish the terms of access to the resources on which the system depends. They can be in the form of policies, agreements, frameworks, or formal legislation. A rule change describes the introduction of a new rule or an existing rule that is adapted or operationalized in a more effective way. System: A system is composed of interconnected, mutually reinforcing parts. A change in one part of the system impacts other parts, creating a constant state of emergence that produces certain outcomes. A system change describes a new way that interdependent system parts work together to produce change. Each outcome was coded for the type of change it exhibited. To what extent do the outcomes align with the project’s theory of change? Qualitative. Harvested outcomes were analyzed and identified as belonging to one of five Strategic Approaches (SA) through deductive analysis: outcomes were coded as belonging to one of the SAs. The Theory of Change of USAID Hay Tao is available in graphical format via a results chain, and the relevance of each outcome was assessed relative to the results chain step. Outcomes were visually laid atop the results chain, and analysis was performed to identify whether the preponderance of outcomes lay in the initial, middle, or latter stages of the results chain. To what extent are outcomes a result of the Activity’s contribution? Quantitative. For each outcome, USAID Hay Tao described the Activity’s contribution. Each substantiator was first asked to rate their degree of agreement with the accuracy of the contribution statement on a 3-point Likert scale (Do Not Agree, Partially Agree, Totally Agree). Outcomes with contribution statements that were agreed on by substantiators proceeded to contribution analysis: the logic was that substantiators who did not agree with the contribution statement could not objectively assess the USAID Hay Tao degree of contribution. Outcomes with contributions statements that received partial or total agreement from substantiators proceeded to contribution analysis. The contribution of USAID Hay Tao to each outcome was assessed on a 10-point scale (1 being ‘no contribution’, 10 being ‘would not have happened without USAID Hay Tao’). Substantiators rated each outcome. This is a cost-effective and qualitative approach to providing a retrospective counterfactual. 21 Research Question Analytic Approach To what extent are the outcomes significant to people’s lives and sustainable? Quantitative. The significance of each outcome to Madagascar’s natural resources management and to people’s lives was assessed on a 10-point scale (1 being ‘not significant to people’s lives, 10 being ‘very significant to people’s lives’) by substantiators. The sustainability of each outcome was assessed building from the Initiative for Global Development’s (IGD) innovation maturity model and tipping point concept. The IGD innovation maturity model suggests that there are five stages an innovation progresses through before becoming fully established in a given system: ad-hoc stage, pilot stage, sticky stage, scale stage, or systemic stage6. It is important to note that the IGD model is concerned with the relative independence of an innovation: whether it lives on without need of intervention. This is why USAID Hay Tao selected the model to understand and measure the sustainability of the Activity’s outcomes. Sustainability is therefore in reference to the intervention, and therefore stages of the model are relative to the outcome in the intervention system. With this in mind, two terms in the model caused some confusion among evaluation stakeholders: “scale” and “systemic.” Some understood scale to mean widespread, “system-wide” (such as national) use. Others understood systemic to mean “impact.” After early feedback from USAID, the evaluation team was concerned that external evaluation consumers would misinterpret outcomes labeled as “scale” or “system” as being in reference to the entire system of natural resources management in Madagascar, rather than in relation to the intervention system more specifically. To avoid confusion, the evaluation team replaced the word “scale” with “replicating,” and the word “systemic” with “enduring.” USAID Hay Tao’s modified definitions and terms are as follows: Ad-Hoc: Preliminary activity that is experimental or uncoordinated, without a clear plan for expansion. 6 This degree of change framework was developed based on an adaptation of tipping point concepts used in previous Pact Outcome Harvesting methodologies, adapted from the Initiative for Global Development’s (IGD) Innovation and Systemic Change Maturity Model, in Adrienne Gifford et al., Pioneering New Operating Models and Measurement Techniques for Private-Sector Led Development (IGD: 2018). The original definitions – which were modified by USAID Hay Tao – were: Ad Hoc: Stage of preliminary activity occurring on an experimental or uncoordinated basis without a clear plan for program activities, stakeholders and participants, funding, and/or monitoring. Pilot: Stage of program activity when experimentation becomes controlled as plans begin to take shape around specific activities, actors come together and engage, funding is identified, and/or monitoring methods are identified and tested. Stickiness: Stage of activity when experimentation ceases and plan becomes clearly defined or “sticky,” leading to the buy-in, consistency, and strategy related to stakeholders, funding, and/or monitoring, which helps programs reach the “tipping point” required to take off and build to large numbers quickly. Scale: Stage of activity when widespread, internally driven involvement leads programming, and scope of the project expands significantly. Systemic Change: Stage of activity when new practices and ideas spread independently and sustainably without further need for intervention. 22 Research Question Analytic Approach Pilot: Expanded activity in which actors come together in a more coordinated fashion around specific objectives. Sticky: Defined activity with clear plans and buy-in across stakeholders with associated resources, demonstrating a “tipping point” at which more actors are able to engage. Replicating: Activity becomes more widespread and increases in scope. Enduring: Activity spreads independently, may be adapted by actors, without need of further intervention. For each outcome, substantiators were asked to identify what stage of maturation the outcome was in, from their perspective. In addition, USAID Hay Tao staff that developed the outcome statements also rated each outcome’s change type (role, responsibility, resource, relationship) by degree of sustainability. [To minimize confusion, substantiators were not asked to provide degrees of sustainability for type of change, just for the overall outcome.] The IGD framework puts the tipping point between Pilot and Sticky change, which corresponds to a shift from early adopters who pilot the change, to an early majority who help sustain (“stick”) the change. USAID Hay Tao updated the IGD terminology with the new terms noted previously, and with the following visual: The results of the analysis are presented in the findings section, organized under each research question. 23 Methodological Considerations and Limitations While OH provides several benefits and was uniquely suited to the needs of the USAID Hay Tao team based on the evaluation purpose and principles, like any methodology and research activity, there are limitations. First, only outcomes that informants were aware of and deemed important were captured: the evaluation cannot harvest unknown outcomes. In the case of this evaluation, outcomes were harvested by USAID Hay Tao and partners. USAID Hay Tao was advised to limit to a manageable number of outcomes, approximately 30. Similarly, while all outcomes were encouraged – positive and negative, anticipated and unanticipated – all of the final substantiated outcomes were positive; the team could not identify negative outcomes with enough detail to advance to substantiation. However, the team identified several positive outcomes that they termed “mitigated” – outcomes that experienced setbacks or challenges along the change pathway. Many of these outcomes, during substantiation, were subsequently categorized by substantiators as lower-level outcomes – those that had not progressed far along the maturity spectrum. These outcomes are identified as part of findings for Evaluation Question 1.3. Since the original design did not require a list of negative outcomes to answer the research questions, this limitation is noted but does not affect the value of the findings. In short, readers should be aware that no negative outcomes are included, which does not mean there were no negative outcomes. Similarly, while the evaluation team made every effort to consider all aspects of the project, outcomes in this evaluation should not be taken as a generalization across the USAID Hay Tao activity. The OH methodology is not designed to provide a generalized assessment of a project, but rather it provides a picture of outcomes free from predetermined metrics and reduces bias through substantiation. Second, by nature of the participatory design, the evaluation requires the participation of individuals who helped to influence the outcome in the selection and description of that outcome. Like any participatory methodology, this introduces a degree of bias. The team mitigated this bias by (i) making substantial efforts to include a significant number of external substantiators, (ii) including substantiators at various governing levels, and (iii) triangulating between USAID Hay Tao assessments of outcomes and substantiator assessments. We found a marginal difference between USAID Hay Tao and substantiatior responses regarding contribution and significance – with USAID Hay Tao being more conservative – but not enough to suggest a systematic difference. Nevertheless, findings are presented from both sources separately. Third, by nature of the systems-approach that USAID Hay Tao took, substantiators were all in some way members of the system rather than completely neutral observers. The USAID Hay Tao activity involved different system actors (for example, government actors, civil society organizations, private sector businesses large and small, media actors, judicial system actors), different geographies (for example, Protected Areas, national government, regional administrations, communities), and different sectors and sub systems (for example, the legal system, business sector, livelihoods). It was not possible to find an observer that was not in the system and yet knowledgeable enough to provide analysis. This is an unavoidable limitation of a systems-level project using the OH methodology. The original methodology does not require that substantiators be wholly separate, just that they do not benefit from the evaluation or do not work directly on the outcome, which reduces bias. The evaluation mitigated this by seeking substantiation across various levels and from different categories of sources (for example, civil society and government for the same outcome); and, no substantiator was a member of the USAID Hay Tao consortium. Fourth, given that the evaluation was launched during the start of the COVID-19 pandemic, there were unavoidable delays between evaluation steps and many of the initial workshops took place virtually (63 of 24 the 89 substantiations were via in-person workshops; the remainder were via email or remote interviews). As a result, harvested outcomes are limited to those identified by October 2021, and late-stage project outcomes were not included. It is highly likely that there are other outcomes that emerged in the last year of the project that are not included here. In addition, because substantiation took place in the final year of the project, and up to 12 months after outcomes were selected, it is possible that the substantiators may have limited memory of the USAID Hay Tao activities that contributed to the outcomes, which affected their responses. Fifth, substantiators were invited to substantiate aspects of each outcome for which they felt most comfortable: they were not required to comment on all aspects. For this reason, in any given outcome, there may be different numbers of substantiators providing responses to different substantiation questions. This does not affect findings, though this report will note where there are different numbers of substantiators. In addition to these limitations, the following considerations or challenges should be taken into account. First, the OH methodology was new to the USAID Hay Tao evaluation team and the evaluation team members were required to learn and then manage the larger Activity team to undertake the participatory approach. There was an element of praxis – learning and application – though overall the evaluation team was advised by Pact’s global evaluation unit which has substantial experience with OH. Second, it is challenging for many intervention teams to think about results at the outcome level, beyond outputs and activities. A different set of participants may well have identified different outcomes with different prompts. Relatedly, and third, members of the USAID Hay Tao have noted in the past that it can be culturally challenging to identify outcomes: program staff preferred to remain modest and not “claim successes,” particularly given the systems-level at which the project operated, working as they did with many stakeholders. For this reason, outcomes identified for substantiation in many ways are the ones that were the most obvious and already accepted by external stakeholders as USAID Hay Tao outcomes, whereas other OH evaluations also tend to include outcomes that are more exploratory. This is reflected in the finding that the median score from substantiators on degree of change across the outcomes is 3 and the mean 3.2, which is “sustainable change.” Other OH evaluations have a greater mix of outcomes with lower degrees of change. 25 FINDINGS The following section presents the main findings of the evaluation organized by evaluation question. Evaluation Question 1: What are the key outcomes to date of the USAID Hay Tao Activity? Overall, 33 outcomes were identified, selected, described, and substantiated. Outcomes were relatively evenly distributed across SAs, with the most in SA1 (8 outcomes) and the least with SA5 (5 outcomes). Note that this is not a generalization across the entire project: this does not imply that SA 1 had more outcomes, just that the SA 1 technical team identified, proposed, and determined to validate this number of outcomes. A summary of outcomes is provided below. Complete outcome statements and associated scores are included in Attachment I. Outcome Statement SA 1: Support conservation stakeholders on sustainable livelihoods, NRM, and environmental justice through capacity development 1.1 Improved vanilla cultivation techniques applied by vanilla producers and processors. 1.2 Updated management tools facilitated the critical renewal of improved TGRNR contracts. 1.3 65% of PA have renewed their PAG using the new PAG framework. 1.4 New capacity development approach and standards incorporated into community-based management of marine and coastal resources by national civil society, MIHARI. 1.5 Women have increased leadership in the fisheries value chain. 1.6 Madagascar introduced globally applicable sustainable development standards in PA management (IUCN Green List). 1.7 Improved regional planning via participatory process led to more realistic national reforestation target of 75,000 ha per year. 1.8 47 organizations cascaded their technical knowledge to grassroots CSOs and improved the impact of their activities in Menabe and MaMaBay ecoregions. SA2: Improve and make available data 2.1 More effective and timely patrols to fight fires, deforestation, and other PA management, with information gathered from innovative technologies, including Forest Watcher and SMART. 5 7 7 6 8 0 2 4 6 8 10 SA5 SA4 SA3 SA2 SA1 Quantity of Outcomes Strategic Approaches Number of Outcomes Harvested by Strategic Approach (SA) 26 Outcome Statement and information for evidence￾based decision￾making on conservation and sustainable livelihoods 2.2 Improved use of evidence and technology for decision-making to reduce pressure on the Menabe Antimena PA and improve living conditions of PA migrants. 2.3 National Environment Office commits to take over the management of the Hay Natiora data portal and integrating the portal into its existing systems ensuring sustainability. 2.4 GoM reasserted its zero-tolerance policy for environmental crimes, which lapsed during COVID-19. 2.5 Investigative journalists with improved research and reporting capacity contribute as informants regarding environmental crimes. 2.6 Investigative journalists utilized improved capacity to provide higher quality coverage of land tenure issues and to prepare more effective journalistic products. SA3: Identify and foster innovative financing options for community￾based conservation 3.1 Social and conservation entrepreneurship (SCE) advanced after the submission of the draft law and the implementation of new SCE business models. 3.2 Three hotels made progress in meeting “Green Globe” certification, including financial investments. 3.3 Two associations of women craftworkers, FIVEMA and FITARATRA, in MaMaBay established nurseries and restored natural resources to achieve women-led sustainable development. 3.4 In 2020, a national SCE platform was set up, the first multi-stakeholder platform of its kinds in Madagascar to ensure coordination and exchange for promoting SCE in Madagascar 3.5 The national Natural Capital (NC) network in Madagascar was established by GOM and relevant stakeholders in March 2021 with the objective of integrating NC valuation in governmental decision-making and investment/economic activities. 3.6 Eleven additional PAs across an additional 501,800 hectares financially supported as a result of $58 million+ addition to the Madagascar Protected Areas and Biodiversity Fund (FAPBM). 3.7 Advanced fund management capacity by the Tany Mena Foundation enabled improved livelihoods and biodiversity conservation in the Southern ecoregion. SA 4: Strengthen public sector institutions in policy-making and enforcement of tenure rights and NRM 4.1 Improved ecological and socio-economic land management in 38 communes in Menabe and MaMaBay. 4.2 Improved management of coastal and marine tenure rights. 4.3 Four new communal management plans (SACs) in APMA completed and validated, addressing environmental and social concerns and harmonized with PA management. 4.4 Dinan'i Menabe – affecting all the 5 districts of the region – was improved and formalized to include sustainable environmental considerations and CBNRM. 27 Outcome Statement 4.5 Improved legal capacity and the development of a shared vision among participating stakeholders resulted in the update of the Antongil Bay Dinabe concerning the exploitation of fisheries resources. 4.6 Improved legal attention to natural resources governance in the court system, particularly land tenure cases. 4.7 45% of targeted OPJs and OSPJs applied capacity development and tools for environmental crime detection, arrest, prosecution, conviction, and penalty, resulting in strengthened law enforcement. SA5: Support civil society and private sector coalitions to advocate for environmental and social governance through constructive engagement 5.1 Increased collaboration and a shared vision by diverse national and regional stakeholders in support of more sustainable NRM. 5.2 Increase in community reporting of environmental crimes through AVG's 512 hotline and in-person visits to the environmental legal clinics (CJEs). 5.3 Since 2019, court decisions regarding environmental crimes have increasingly utilized evidence provided by communities and informants in Morondava and Maroantsetra. 5.4 Private sector actor DHL increased corporate social responsibility investments to strengthen the adoption of eco-gestures and conservation agriculture. 5.5 Launch of the national CSR strategy (SNRSE) and mobilization around a shared vision for CSR environmental issues. Finding 1.1: Substantiators agreed that USAID Hay Tao outcomes were real, observable changes at the outcomes level, validating the existence of the outcomes. Outcome Harvesting aims to identify and validate outcomes that are often challenging to observe in real life (i.e., they are intangible) or for which it is difficult to obtain rigorous evidence because evidence generation was not planned from the start (i.e., with a baseline) which is the case for unexpected outcomes or when there have been programmatic shifts. For this reason, an important indicator is whether substantiators agree that outcomes identified by project stakeholders truly exist. Positively, a majority of substantiators indicated that they “totally agree” with outcome statements, a few indicated they “partially agree,” and none indicated that they “do not agree.” Substantiators indicated that they “totally agree” with 82% of the outcome statements (27 out of 33), “partially agree” with 18% of the outcome statements (6 out of 33), and none indicated that they “do not agree.” On a Likert scale in which 1 was “do not agree,” 2 was “partially agree,” and 3 was “totally agree,” the mean score was 2.8 and the median score was 3. 28 Finding 1.2: The majority of outcomes were considered to be positive changes with a few turning points and no negative changes. USAID Hay Tao staff coded each outcome as a negative change, a positive change, or a turning point (defined as an important milestone though not a full-fledged negative or positive change). No negative outcomes were identified by the project team for substantiation. This does not mean that no negative changes occurred during the life of the Activity: indeed, change is friction, and any positive change undoubtedly experienced negative repercussions. Nevertheless, the project team did not prioritize negative outcomes for substantiation (please see Limitations section for more discussion). However, it is important to note that because outcomes are complex, most of the outcomes noted as “positive changes” are a result of nonlinear change pathways. While the overall result is positive and at the outcome level, it is often the result of a journey that experienced setbacks along the way. Some of these are reflected in the final sustainability score for a given outcome: an outcome that was challenged along the way likely did not progress as far along the innovation maturity model as others. The USAID Hay Tao team described these internally as “mitigated successes.” [Such “mitigated successes” are captured under Evaluation Question 3 focused on sustainability, in which the evaluation team assessed the degree of change of each outcome.] 0 6 27 0 5 10 15 20 25 30 Disagree Partially Agree Totally Agree Quantity of Outcomes Level of Agreement Average Substantiator Agreement with Outcome Statement ǿAs the Malagasy adage says, ǻSitraka enti-matory, ho valiana raha mahatsiaroǼ (A blessing received at bedtime, it will be answered at dawn). Your intervention encourages us to do what we must do for the country.· Anonym ous OH Subst ant iat or 29 Finding 1.3: USAID Hay Tao Activity took a systemic approach to the change pathway, targeting outcomes using diverse strategies, as evidenced by the multiple types of change exhibited by each outcome. USAID Hay Tao staff categorized each outcome by type of change using a modified version of the USAID 5-R Systems framework: role, relationship, resource, rule, system. An outcome can exhibit multiple types of change, ranging from one to all five, demonstrating the complex interplay of changes that need to happen to produce an outcome. [Note: The configuration of types of change are not associated with whether an outcome was coded as a positive change, turning point, or negative change.] 0 5 28 0 5 10 15 20 25 30 Negative Change Turning point Positive Change Quantity of Outcomes Type of Outcome Change Classification of Harvested Outcomes Instances of Changes Type of Change 24 A ROLE change describes a new way of acting that exhibits a difference in how responsibility is understood and realized through changed practices and/or behaviors. 16 A RELATIONSHIP change describes a new way that individuals interact that exhibits a difference in their interaction, for example whether these demonstrate changed cooperation or coordination. 21 A RESOURCE change describes a resource that is introduced to the system or an existing resource that is repurposed to greater effect, enabling system actors to pursue their actions. 13 A RULE change describes the introduction of a new rule or an existing rule that is adapted or operationalized in a more effective way. 14 A SYSTEM change describes a new way that interdependent system parts work together to produce change. 30 Overall, outcomes exhibited 88 instances of change. The median number of types of change per outcome across all 33 outcomes was 3. SA1, SA2, and SA3 all had median scores of 3 for instances of change, while SA4 and SA5 had median scores of 2. This analysis shows that outcomes were predominantly a result of multiple types of change, confirming that the USAID Hay Tao intervention targeted the outcome from various approaches. In Depth: Multiple types of change in a single outcome Outcome 5.5 exhibited the most types of change: all five. This outcome related to the commitment by MEDD and EDBM to promote CSR to help the private sector invest more in environmental activities in the development of their businesses, by creating enabling framework conditions such as the national CSR strategy (SNRSE). The national strategy, launched in October 2020 by MEDD and EDBM, demonstrated a change in rule and resources, which in turn demonstrated a change in the expectations of individual roles and responsibilities, as well as relationships between individuals on this topic as they collaborated on a shared vision. Figure 1: Regional consultation in Fianarantsoa (Matsiatra Ambony region) as part of the elaboration of the national CSR strategy (March 2022) Only three outcomes exhibited a single type of change. Two of these exhibited a role change, and one a resource change. We hypothesized that outcomes that exhibit few types of change are those that require less complex change processes or that are less sustainable. In point of fact, all three of these outcomes with only single types of change were rated by USAID Hay Tao staff as less sustainable, and specifically as “pilot changes” (meaning a stage of program activity when experimentation becomes more intentional but has not expanded beyond the immediate participants; see Methodology for full definitions of tipping point changes) (substantiators rated two of these outcomes higher than did the USAID Hay Tao team). 31 For example, one outcome related to the strengthening of law enforcement, the sentencing decisions of judges, and increased collaboration between OPJs and OSPJs. These were coded only as a role change: actors understand their roles and responsibilities in a new way and then adjust their practice and/or behavior accordingly. This was rated as a pilot by USAID Hay Tao, because while actors came together in a more coordinated way around specific objectives, the result had not passed beyond the “tipping point” when more actors engage (see Methodology section and findings under Evaluation Question 4 for more discussion about tipping points). Finding 1.4: Across the 33 outcomes, most exhibited a role change, and the least exhibited type of change was that of rules. Overall, the change category exhibited in most outcomes was that of role change (24 out of 33 outcomes), followed in successive order by resource change (21 out of 33), relationship change (16 out of 33), system change (14 out of 33), and rule change (13 out of 33). It is no surprise that the most prevalent type of change exhibited across the USAID Hay Tao harvested outcomes was that of roles. This is an expected result given the focus of the USAID Hay Tao Activity on changing how system actors perceive their roles and responsibilities, and how those system actors act on that changed perception through new practices and/or behaviors. A role change is a foundational requirement of the behavioral changes that in turn result in resource changes, relationship changes, rules changes, and overall systems changes. As such, the prevalence of roles changes is not unexpected and is an indication of foundational progress along the USAID Hay Tao overall theory of change. The limited number of system changes is also expected. System change describes a new way that interdependent system parts work together to produce change, which is no mean feat. It requires political commitment and transformation, which in the case of Madagascar with a general election bisecting the USAID Hay Tao tenure, has been a limiting factor. The pace of planned laws and policies supported by USAID Hay Tao has thus been adversely affected, and it is these rules that underpin systemic change. What USAID Hay Tao has been successful in doing, and the data above validates, is positively affecting resource allocation, making information more accessible for future decision- and policy-making, and developing the capacity of actors in the environment space to strengthen their roles. USAID Hay Tao has created collaborations that help to align mindsets and goals and enable shared learning and spark new ways of managing natural resources in a more effective and sustainable way. It has set the foundation for systemic change. Finding 1.5: While each SA exhibited all changes, for most, one type of change was dominant. Figure 2 shows the quantity of different types of change across SAs. For example, SA1 had 4 role changes, 1 relationship change, 8 resource changes, 3 rules change, and 7 system changes. “USAID Hay Tao's involvement has made us more aware of the impact of our daily actions on the environment and has encouraged us to respect it…. Although the positive impacts are not immediate, this program will contribute highly to the protection of the environment…” Anonymous Substan�ator “The use of technology ensures transparency in conservation efforts, especially through the sharing of patrol activities results…. The knowledge imparted to us allowed us to share the results, anticipate fires with the location of cleared areas, and encourage law enforcement in all the PAs where we work.” Anonymous Substan�ator 32 These findings are relatively surprising. For example, USAID Hay Tao expected the most change in resources under SA3, which deals with identifying and fostering innovative financing options for community-based conservation. However, it should be noted that resources are more than funds or budgets and can take non-monetary form: capacity resources, tools, and so on. This may explain the number of resources exhibited under SA1 which focuses on capacity development for sustainable livelihoods, NRM, and environmental justice for conservation stakeholders. USAID Hay Tao expected most systems changes under SA4, which focuses on policies. The fewer number of systems changes under SA4 is not surprising given the complexity of law and policy formulation and acceptance noted above. More system changes happened in SA1. This may be due to the supportive systems that provide and contribute to capacity development, such as the certification and training programs that were established and formalized. For example, USAID Hay Tao supported the standardization of PAG tools with MEDD, and since March 2021 SAPM used the new PAG framework, contributing to 12 PAs that completed their renewal. In addition, a new TGRNR monitoring and evaluation tool was developed to standardize the monitoring of VOIs and transfer areas by the MEDD and its branches at the national level. It is also a reference evaluation tool for decision-making on whether or not to renew TGRNR contracts. Finding 1.6: All outcomes were a result of interventions focused on increasing technical capacity of stakeholders; providing new research, approaches, or technologies; and/or convening or mobilizing actors. USAID Hay Tao took a multi-pronged approach to strengthening sustainable NRM, as exemplified in the five strategic approaches. The three specific interventions that were most widely represented in the harvested outcomes relate to (i) increasing technical capacity through capacity development efforts (ii) providing new information via new research or technologies, and (iii) convening or mobilizing actors. Many outcomes included a combination of the three intervention approaches: thirteen (13) outcomes were the result of two intervention approaches, and eight outcomes were the result of all three intervention approaches. Figure 2 Types of change exhibited across SAs 33 In Depth: Three intervention approaches together contributed to improved communal land use planning through the inclusion of environmental and social concerns Outcome 4.3 was a result of three intervention approaches, and exhibited four kinds of change: role change, relationship change, rule change, and overall intervention system change. To advance a common vision for improved NRM and sustainable land tenure in the Menabe landscape, USAID Hay Tao through SIF supported four core communes of APMA (Rural Commune (CR) Beroboka Avaratra, CR Delta, CR Marofandilia and CR Tsimafana) to develop their SACs (communal management plan). SACs are responsible for outlining commune plans for land management and spatial planning, and together with PAG for PAs, are the official spatial planning tools at the commune level of Madagascar and thereby important contributors to sustainable NRM. Before the Activity, none of the communes had SACs and stakeholders were aware of the APMA PAG that their SACs must map to. As a result of USAID Hay Tao support, the four communes now have SACs that were developed based on improved evidence, address environmental and social concerns affecting natural resources sustainability, biodiversity, and habitats in their communes, and are better aligned with the development and management plan of the entire PA via the APMA PAG and in a more sustainable manner. USAID Hay Tao first provided new evidence via research efforts. In FY 2019, the Activity carried out a parcel census at the Fokontany level in eleven communes affecting APMA, including the four core communes, to collect the necessary data for the development of the spatial planning tool. USAID Hay Tao also convened stakeholders, which led to changes in relationships. To launch the effort, USAID Hay Tao worked with key stakeholders to develop and sign a collaboration agreement including the mayors of the four communes, MATSF/DGAT, MEDD/DGGE, Fanamby Association and USAID Hay Tao (represented by Pact and SIF). USAID Hay Tao helped to set up a Communal Committee on Land Use Planning (CCAT) for each commune, implemented CCAT member consultation workshops and workshops with the members of the Regional Committee on Land Use Planning (CRAT). USAID Hay Tao also built technical capacity and awareness that influenced a role change through buy-in and consultation activities. USAID Hay Tao organized multiple consultation workshops for each commune in July 2021 with commune communities on development scenarios adapted to the sustainable harmonization of socio-economic land management and NRM. This consisted of validating the territorial data and the results of the diagnostic and prospective analyses noted earlier, and identifying relevant territorial issues. This step led to the formulation of shared commune visions and localized scenarios for the SAC. Then, the commune visions were verified for coherence with existing standards and policies through a consultation workshop with CRAT (Comité Régional de l’Aménagement du Territoire). Once the drafts were validated as compliant, four more workshops were organized by USAID Hay Tao to validate the spatial scenarios and set development guidelines with CCAT members for each commune in August and September 2021. This led to the validation of the final zonings, among other topics, and the formal submission of the SACs to MATSF. 34 Finding 1.7: The majority of outcomes (approximately 60%) demonstrate a practice and/or behavioral change, resulting from the application of new techniques thanks to technical capacity development or access to improved technical resources. Twenty (20) outcomes describe the application of new techniques, a result of technical capacity development or access to improved technical resources. This application demonstrates a change in how roles and responsibilities are understood, and resulted in a practice and behavioral change in favor of improved NRM. In Depth: From technical capacity development, to practice and behavioral changes and outcomes. Selection of outcomes exemplifying practice and behavioral change after USAID Hay Tao technical capacity development or support: Improved cultivation techniques applied by PPV in the Analanjirofo, Atsinanana, and SAVA regions as a result of the introduction of new standards with support from USAID Hay Tao (Outcome 1.1) Utilization of the new PAG framework by 65% of PAs, which was supported by USAID Hay Tao (Outcome 1.3) Implementation of new capacity development approach and standards to community-based management and governance of marine and coastal resources in strategy and workplan by MIHARI, as a result of USAID Hay Tao assistance (Outcome 1.4) PA managers utilized information gathered from innovative technologies, including Forest Watcher and SMART, improving evidence-based PA management, such as in more effective and timely patrols to fight fires and deforestation in PAs managed by MNP (including Kirindy Mite, Andranomena, Masoala, Ankarafantsika) (Outcome 2.1) Investigative journalists utilized their improved research and reporting capacity to contribute as informants regarding environmental crimes (Outcome 2.5) and to provide higher quality and more effective coverage of land tenure issues in the media and to prepare more effective journalistic products (Outcome 2.6) Three hotels in Menabe applied improved sustainable practices through efforts to reform hotel management to meet “Green Globe” certification (Outcome 3.2) 45% of USAID Hay Tao supported OPJs and OSPJs applied capacity development and tools for environmental crime detection, arrest, prosecution, conviction, and penalty, resulting in strengthened law enforcement, particularly with regard to sentencing decisions by judges, despite structural challenges such as their reassignment of location. (Outcome 4.7) 35 Finding 1.8: Approximately half of outcomes demonstrate improved natural resources management through new, improved, or enacted laws, regulations, frameworks, or standards. Institutionalization and/or formalization of sustainable NRM practices is necessary to educate, enable, and empower actors who are advocates of improved NRM, and to hold others to account. Institutionalization and formalization are necessary though not sufficient conditions for improved NRM. USAID Hay Tao worked significantly to build formal institutional commitment to improved NRM across the spectrum: by working to draft laws put to Parliament or regulations adopted by GoM, by developing certification programs and standards operationalized by Ministries and by other NRM stakeholders, and more. 16 outcomes feature stories of new, improved, or enacted laws, regulations, frameworks, or standards. In Depth: Selection of new, improved, or enacted laws, regulations, frameworks, or standards supported by USAID Hay Tao Tools and standards developed and implemented with technical support from USAID Hay Tao: • USAID Hay Tao collaborated with the Directorate of Training and Support to Partnerships (DFAP) of the Ministry of Agriculture and Livestock (MinAE), to develop and roll-out a national reference system regulating the profession of vanilla producers and processors (PPV) which was formalized in 2022. The system identifies the key competencies in sustainable and resilient vanilla cultivation and processing. (Outcome 1.1) • MEDD institutionalized improved TGRNR monitoring and evaluation tools, including a cost￾effective assessment of social and economic impacts, which was cascaded down by DREDD officers in at least 28 TGRNR sites, as a result of technical and convening support from USAID Hay Tao. (Outcome 1.2) • Improved regional planning led to a more realistic national reforestation target of 75,000 ha per year, as a result of a participatory process convened by MEDD with support from USAID Hay Tao involving key stakeholders including regional STDs, private sector, CSOs and grassroot communities, and the provision of technical and practical reforestation tools. (Outcome 1.7) Improved utilization of existing rules and standards with support from USAID Hay Tao: • Since 2021, SAPM has used a new PAG framework which integrates improved considerations including adaptation to climate change, sustainable development, and community well-being through PHE. The PAG framework was developed with technical and mobilization support from USAID Hay Tao. To date, 65% of PAs have renewed their PAG using the new framework. (Outcome 1.3) • Madagascar for the first time participated in a certification program for PAs that aims to mitigate biodiversity loss from poor management. USAID Hay Tao provided financial and technical support to the Expert Assessors Group (EAGL) to adapt IUCN Green List indicators to the context of PA management in Madagascar, introducing globally applicable sustainable development standards to the country, and supported capacity development of PA managers and DREDD staff. (Outcome 1.6) • As a result of the utilization by PA managers of information gathered from innovative technologies introduced or tailored to Madagascar by USAID Hay Tao, including Forest Watcher and SMART, there has been improved evidence-based PA management, such as more effective and timely patrols 36 to fight fires and deforestation in PAs managed by MNP (including Kirindy Mite, Andranomena, Masoala, Ankarafantsika) (Outcome 2.1). Formalization of practices or regulations with support from USAID Hay Tao • A draft law on social and conservation entrepreneurship (SCE) was developed and submitted to GoM and Parliament by MICC (formerly MICA) with technical support from USAID Hay Tao (Outcome 3.1). The realization of the law was the result of mobilization support from USAID Hay Tao to convene and set up a national SCE platform bringing together the GOM, the private sector, CSOs and technical and financial partners, the first multi-stakeholder platform of its kinds in Madagascar to ensure coordination and exchange for promoting SCE in Madagascar (Outcome 3.4). • Land using planning in APMA was improved to include environmental and social concerns affecting natural resources sustainability, biodiversity, and habitats. This was achieved through more responsive communal management plans (SACs) in the four core communes as a result of the harmonization of land management at the commune level with the development and management plan of the entire PA. USAID Hay Tao conducted a parcel census at the Fokontany level to collect the necessary data for the development of the spatial planning tool, raised community awareness, conducted participatory mapping, organized community consultations to establish local vision for SACs, and supported the technical integration of the plans. (Outcome 4.3) • Dinan'i Menabe and Dinabe of Antongil Bay were improved and formalized with technical, awareness-raising, and shared vision mobilization support from USAID Hay Tao (Outcomes 4.4, 4.5) • The launch of the national CSR strategy (SNRSE) in 2020 by MEDD and the Economic Development Board of Madagascar (EDBM) took place with support from USAID Hay Tao. The Activity played a critical convening role, successfully securing a MOU between the EDBM, MEDD and USAID Hay Tao to encourage private sector involvement in biodiversity conservation. This grouping, with financial and technical support from USAID Hay Tao, developed the SNRSE. (Outcome 5.5) Finding 1.9: More than one third of outcomes demonstrate an increase in community involvement in natural resources management (community based natural resources management, CBNRM). Community involvement in NRM (community based natural resources management, CBNRM) is necessary for sustainability: it engenders buy-in, it recognizes that those at critical sites are most able to identify and report on infractions, and that those at sites are most able to understand what policies and approaches are conducive to sustainable livelihoods that are attractive for that community. Eleven (11) of the harvested outcomes featured CBNRM stories, which is in keeping with USAID Hay Tao focus on engaging communities. [Note: One of the USAID Hay Tao purposes was to advance CBNRM in Madagascar. The Activity’s performance reports provide additional findings related to CBNRM. This finding is limited to outcomes harvested through this internal evaluation.] 37 In-Depth: USAID Hay Tao contributions to strengthened CBNRM Of the eleven outcomes featuring CBNRM, three fell under SA 1, Support conservation stakeholders on sustainable livelihoods, NRM, and environmental justice through capacity development. Outcome 1.2 describes the improvement of TGRNR monitoring and evaluation tools and related skills transfer to local agents with the help of USAID Hay Tao that was then institutionalized by MEDD. The improved tools enable the renewal of TGRNR (management transfer of renewable natural resources) contracts to CBOs (community-based organization, VOI) based on more effective evaluation that takes into account social and economic impacts rather than just environmental impacts. Thus, it advances the involvement of local communities. The model was cascaded down by DREDD officers in at least 28 TGRNR sites. Outcome 1.4 describes how MIHARI implemented a new capacity development approach and standardized community-based management and governance of marine and coastal resources in its strategy and workplan, and Outcome 1.5 describes MIHARI’s women’s leadership program that aimed to increase the participation of women – who are responsible for 50% of the small-scale fishing sector – in decision-making in regional resource governance structures, such as in the Ambanja fisherfolk's federation. One CBNRM outcome fell under SA3, which is about identifying and fostering innovate financing options for community-based conservation. Outcome 3.7 focused on improving the technical and financial management of VOIs via improved capacity in the Tany Mena Foundation, which funds VOIs in support of communities involved in NRM and sustainable development in the southern ecoregion. Three of the outcomes featured under SA4, which focuses on strengthening public sector institutions in policy-making and enforcement of tenure rights and NRM, contributed to CBNRM. Outcomes 4.4 and 4.5 describe the improvements to customary laws through participatory processes. Dinan'i Menabe – affecting all the 5 districts of the region – was revised to include sustainable environmental considerations after a participatory community-centered approach, enabling community contributions to and oversight over the reform of customary law in favor of improved NRM. And the Dinabe of Antongil Bay was revised with the participation of CSOs (namely PCDDBA, FPBA and AVG) and PA managers (WCS) and other stakeholders at the local level as a result of improved legal capacity and the development of a shared vision among participating stakeholders. Four of the five outcomes of SA.5, which focuses on supporting civil society and private sector coalitions to advocate for environmental and social governance through constructive engagement, featured CBNRM results. For example, Outcome 5.2 describes an increase in community reporting of environmental crimes through AVG's 512 hotline and in-person visits to the environmental legal clinics (CJEs). This demonstrated communities’ changed awareness and attitude toward the importance of sustainable NRM to their livelihoods. Outcome 5.3 describes how court decisions regarding environmental crimes have increasingly utilized community and informant reporting in Morondava and Maroantsetra since 2019. This is a result of an improvement in enabling mechanisms available to communities and informants: denunciations are now facilitated through a formal process linking communities through CSOs to OPJs for investigation, with CSOs supporting with evidence generation to ensure community reports are of high quality. 38 Finding 1.10: More than 20% of outcomes are results that advance sustainable livelihoods. A critical component of USAID Hay Tao was its integrated and holistic approach, recognizing that sustainable NRM requires the buy in from those who live on or near protected areas through their sustainable livelihoods. More than 20% of the outcomes explicitly referenced the contribution to sustainable livelihoods. For example, improved cultivation techniques applied by vanilla producers and processors (PPV) in the Analanjirofo, Atsinanana, and SAVA regions increased productivity from 500g-1kg of vanilla from a single plant, to 1kg-5kg, which was also correlated with a 25% reduction in forest fires between 2018 and 2023 (Outcome 1.1) (for more information, see In-Depth: The contribution of USAID Hay Tao to more sustainable and resilient vanilla cultivation under Finding 3.3). In Depth: Contributions of innovative financing options for community-based conservation and sustainable livelihoods SA3, which focuses on innovative financing options for community-based conservation, contributed the most to sustainable livelihoods with four outcomes referencing the topic in contrast to all other SAs. For example, Outcomes 3.1 and 3.4 describe how interest in new social and conservation entrepreneurship (SCE) led to the development of a national, coordinating platform with the GoM, the private sector, CSOs and technical and financial partners, the first multi-stakeholder platform of its kinds in Madagascar to ensure coordination and exchange for promoting SCE. This led to a draft law being produced and submitted to Parliament and GoM, which explicitly supports the contribution of new business models to support local community welfare in tandem with biodiversity conservation. Outcome 3.2 focuses on sustainable private hotel management: efforts to reform hotel practices to make them more environmentally sustainable have resulted in advances in meeting “Green Globe” certification by three engaged hotels in Menabe, in addition to their additional financial investments in local conservation. And Outcome 3.3. describes the efforts of two associations of women craftworkers, FIVEMA and FITARATRA, in MaMaBay that established nurseries and restored penjy (a reed that grows in water, and is used in wickerwork such as hats, mats, baskets, sacks, and packing bags) and raphia resources, to enable the sustainable harvesting of these natural resources used in the craft industry. This demonstrated advances in women-led sustainable development. 39 Evaluation Question 2: To what extent do the outcomes align with the project’s theory of change? Finding 2.1: Harvested outcomes appeared across the USAID Hay Tao results chains, affirming the Activity’s systems approach and multipronged strategy working at all levels and across a diverse range of stakeholders and sectors. Harvested outcomes appeared across USAID Hay Tao’s theory of change (ToC). The USAID Hay Tao ToC is visually mapped in SA-specific results chains. The results chain maps linearly display change from left to right, with expected results or steps in blue, and strategic approaches and activities in yellow. Links to larger USAID biodiversity goals and threat reductions are in purple, green, and brown boxes. Aspects of the results chains are included below; larger scale versions are included in the appendix of this document. Analysts identified where the harvested outcomes appeared in each SA results chain and found that they appeared spread across the maps. The harvested outcomes are identified in orange hexagons in the maps included below, showing how not only were the outcomes aligned with the ToC but that these occurred across the development/change pathway. Figure 3 Example of entire SA results chain with outcomes overlaid (see Attachment III for full size images) 40 Figure 4 Section of SA1 results chain with USAID Hay Tao outcomes identified (orange circles) Figure 5 Section of SA2 results chain with USAID Hay Tao outcomes identified (orange circles) 41 Figure 6 Section of SA3 results chain with USAID Hay Tao outcomes identified (orange circles) Figure 7 Section of SA4 results chain with USAID Hay Tao outcomes identified (orange circles) 42 Figure 8 Section of SA5 results chain with USAID Hay Tao outcomes identified (orange circles) 43 Evaluation Question 3: To what extent are outcomes a result of the Activity’s contribution? USAID Hay Tao contribution to each outcome was assessed on a 10-point scale. Substantiators were asked to identify where on the scale they thought contribution fell for each outcome, from 1 being “happened without USAID Hay Tao” to 10 “happened because of USAID Hay Tao.” A physical line was added between score 5 and 6 to visually demarcate less or more USAID Hay Tao contribution. A score of 6 and higher suggests that USAID Hay Tao had an outsized influence over the outcome, whereas a score of 5 or below suggests that USAID Hay Tao had a more marginal influence over the outcome. This approach provides a retrospective counterfactual. ← 1: Happened without USAID Hay Tao Happened because of USAID Hay Tao: 10 → 1 2 3 4 5 6 7 8 9 10 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ Before analyzing substantiators’ assessments of USAID Hay Tao contributions for each outcome, we first analyzed substantiator agreement with the contribution statement. This was necessary to ensure that contributions assessments were made on outcomes with which substantiators had confidence in. High outcome agreement means that substantiators are assessing the USAID Hay Tao contribution of a statement they are confident in, whereas low agreement means that substantiators are assessing the contributions of a statement they are not confident in. A contribution statement with low agreement should not proceed to contributions analysis because the contribution analysis would be biased: substantiators would likely provide an assessment of USAID Hay Tao contribution for a contribution statement they already had low confidence in. As no contribution statement received low agreement, all were able to proceed to substantiation. Finding 3.1: Substantiators, on average, highly agreed with the contributions analyses conducted by USAID Hay Tao. The majority of substantiators indicated they “totally agree” with the contribution statement as described by USAID Hay Tao. On average, substantiators indicated that they “totally agree” with 91% of the contribution statements (29 out of 32), “partially agree” with 9% of the contribution statements (3 out of 32), and none indicated that they “do not agree.” (One contribution statement was not assessed on degree “Without the initiative, influence and contribution of USAID Hay Tao, neither the bill nor the business model defining an SCE will happen. Platform members have already tried on their own, but with the unifying skill of USAID Hay Tao, the initiative has gained momentum. Without USAID Hay Tao's initiative, the promotion of SCE would not have had this magnitude.” Anonymous Substan�ator 44 of agreement; the substantiators noted that they preferred to allow more time for the outcome to solidify before commenting). On a Likert scale in which 1 was “do not agree,” 2 was “partially agree,” and 3 was “totally agree,” the mean score was 2.8 and the median score was 3. As the majority of contribution statements received positive agreement, the result of this analysis suggests that contributions analysis could proceed with all outcome statements. In Depth: Understanding “partially agree” contribution analysis responses One of the outcomes for which substantiators partially agreed with the contribution statement was Outcome 5.1, which focuses on the increased collaboration of national and regional multi-stakeholder decision-makers and actors for more sustainable NRM. This resulted in advocacy actions promoting good governance of natural resources and land under the leadership of the Alliance Voahary Gasy (AVG) and Sehatra Iombonana ho an'ny Fananantany (SIF). The USAID Hay Tao contribution statement discussed the advocacy mobilization of AVG and SIF and other CSOs such as the ROHY movement and regional CSOs such as FIVE Menabe, Code Menabe and Lampogno. Together, they developed press releases that were adopted by actors as varied as Parliament, the independent commission for human rights, and the President of the Republic; technical documents; and national resolutions, such as the Dinan'i Menabe update. AVG and SIF mobilized regional CSOs and helped define a shared NRM vision for each landscape. In partially agreeing with the contribution statement, one substantiator noted that the visibility of the CSOs in the outcome was less apparent, though the outcomes were tangible: “Given the regime’s political circumstances and problems, participation of CSOs is not visible. But if we refer to the number of workshops and trainings conducted, CSOs break the record. Yes, for table action but not for actions on the ground.” Finding 3.2: Overall, USAID Hay Tao made a significant contribution to harvested outcomes. Substantiators assessed USAID Hay Tao contributions to each outcome on a scale of 1 to 10, where 1 was described as “no contribution” from USAID Hay Tao, and 10 was described as “would not have happened without USAID Hay Tao.” The chart below presents the average contribution scores from substantiators for each individual outcome. The median and mean across all outcomes rounded to “8,” which suggests USAID Hay Tao had a relatively high contribution to outcomes. 0% 9% 91% 0 10 20 30 40 Do Not Agree Somewhat Agree Totally Agree Quantity of Outcomes Degree of Agreement Average Substantiator Agreement with Contribution Statement 45 Three outcomes received a perfect contribution score of 10 from substantiators, indicating that substantiators think these outcomes could not have happened with USAID Hay Tao activity intervention. These outcomes related to USAID Hay Tao’s role in: • Enabling vanilla producers and processors (PPV) in the Analanjirofo, Atsinanana, and SAVA regions to improve cultivation techniques, which has contributed to reduced forest degradation and climate change impacts on the value chain since June 2021 (Outcome 1.1) • Encouraging the Government of Madagascar to reassert its zero￾tolerance policy for environmental crimes, which had lapsed during COVID-19 when lockdowns and border closure measures pushed up unemployment and increased pressures on natural habitats and consequently on biodiversity as communities engaged in unsustainable use of natural resources to meet livelihoods needs. USAID Hay Tao led communication campaigns and encouraged watchdog functions to hold judicial bodies accountable (Outcome 2.4) • Helping improve the management of coastal and marine tenure rights after corrections were made to identified gaps — including the development of the Menabe region Marine Spatial Plan (MSP) and fisheries — related Dina and Dinabe databases which are now integrated into the Hay Natiora portal to improve fisheries management. USAID Hay Tao originally identified the gaps in tenure assessments (MAEP/DGPA, now MPEB/DGPA, MEDD, and MATSF) starting in 2020 (Outcome 4.2). “USAID Hay Tao activities trigger our commitment to our environment and society.” Anonymous Substan�ator 46 In-Depth: The contribution of USAID Hay Tao to more sustainable and resilient vanilla cultivation Madagascar produces approximately 80 percent of the world's vanilla, and the trade of vanilla provides the main source of foreign currency for the country. Yet, most vanilla farmers live in poverty, and vanilla cultivation contributes to forest degradation and is extremely vulnerable to climate change. A strategic environmental assessment carried out by the MEDD in 2017 highlighted how the vanilla sector constitutes a threat to the forests of the North-East, due to the conversion of forest plots into vanilla fields. USAID Hay Tao worked to integrate the idea of sustainability and resilience to the vanilla industry, via the development of national PPV capacity building tools and the training of PPV technical trainers. The Activity’s approach enables farmers to remain at their current plot rather than convert new forest plots into fields, by providing farmers with more efficient semi-intensive and climate-smart techniques. The approach also enabled buy-in from key stakeholders from the start to stimulate ownership and contribute to sustainability. USAID Hay Tao provided technical assistance and organized regional consultations with critical vanilla stakeholders to identify and confirm the key competencies required for sustainable and resilient vanilla cultivation and processing, those competencies that should be certified, the requirements of a training program, and the design of the national standards for sustainable and resilient vanilla. Together with MinAE, USAID Hay Tao organized a national workshop in which the National Standards for Sustainable and Resilient Vanilla Cultivation and Processing were validated with participants including MinAE, MICA, METFP, representatives of the National Engineering Team of Agricultural and Rural Training (ENIFAR), agricultural training centers, and national and regional vanilla platforms. Since 2020, USAID Hay Tao and MAEP, now MinAE, with the support of the Antsinanana branch of the National Center for Applied Research to Rural Development (FOFIFA), trained regional technicians from the MAEP, regional training centers, support NGOs, and technicians from farmers' organizations and cooperatives to disseminate and support the use of the PPV Standard and associated tools in 2020 and 2021 for Analanjirofo, Antsinanana and SAVA. A total of 62 technical trainers in the Analanjirofo, SAVA and Antsinanana regions received theoretical and practical trainings. These 62 trainers have started to coach PPVs in the application of the improved technique of vanilla cultivation and preparation which integrates environmental issues and climate change impacts on the value chain since June 2021. USAID Hay Tao also designed and made available reference training modules to facilitate the application of the national sustainable and resilient vanilla framework. As one substantiator noted, “USAID Hay Tao built institutional support for the implementation of the Ministry's activities.” Before the training, a farmer could yield 500g to 1kg of vanilla from a single plant. After the trainings, and after applying the improved cultivation technique, a farmer could yield between 1kg and 5kg from a single plant. In addition, while correlated with and not necessarily caused by the change in vanilla cultivation, between 2018 and 2023 there was a 25% decrease in active fires as reported in the Hay Natiora portal in vanilla producing regions of Analanjirofo, SAVA and Sofia. Buy-in created Regional consultations conducted National standards developed and validated PPV train-the￾trainers PPV improved 25% reduced forest conversion Vanilla yields increase from 500g-1kg to 1- 5kg 47 Evaluation Question 4: To what extent are the outcomes significant to people’s lives and sustainable NRM? This research question asks two distinct questions: significance of the outcomes and their degree of sustainability. Accordingly, the analytic framework considers degree of significance and degree of sustainability. For degree of significance, substantiators were asked to rate each outcome on a 10-point scale with 1 being “not very significant to people’s lives and sustainable NRM”, and 10 being “highly significant to people’s lives and sustainable NRM”. Like the contribution analysis, a line was drawn between 5 and 6 to visually demarcate high and low significance. ← 1: Not very significant to people’s lives and sustainable NRM Highly significant to people’s lives and sustainable NRM: 10 → 1 2 3 4 5 6 7 8 9 10 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ Finding 4.1: On average, USAID Hay Tao outcomes were deemed “highly significant to people’s lives and sustainable NRM” by substantiators. Substantiators assessed the significance of the outcomes on a scale of 1 to 10, where 1 was described as “not significant to people’s lives and sustainable NRM” and 10 was described as “very significant to people’s lives and sustainable NRM.” The chart below shows the distribution of scores. (Data below reflect responses from 32 out of 33 outcomes. One outcome received no substantiator score for significance, as the substantiators noted that they wanted to wait for the project to be completed before accurately assessing significance). 48 Finding 4.2: Seven outcomes received a perfect significance score of 10 out of 10 from substantiators. Of the 33 outcomes, 7 received the highest significance score of 10 out of 10, which means that substantiators think that those outcomes were “very significant to people’s lives and sustainable NRM.” One was in SA1 (Outcome 1.4), which related to the fact that the executive secretariat of MIHARI included services and capacity development to improve CBNRM. Improved CBNRM resulted in improved fish catches, for example after the closure of the fishing grounds managed by LMMAs. Three outcomes were from SA2 (Outcomes 2.1, 2.4, 2.5), including one related the fact that the National Environment Office took over management of the Hay Natiora portal to ensure sustainability of the database, indicating the importance of data-backed decision making; and the inclusion of journalists as environmental crime informants, indicating a change in perspective regarding the importance of environmental crime and the role of citizens in combatting it. Data￾backed decisions from the Hay Natiora platform were significant in sustainable NRM especially for PA managers and conservationists, and contributed to reducing deforestation and fires. Such reduced threats enabled the continuation of the ecosystem services provided by PAs, including for example freshwater and water for irrigation. One outcome was in SA4 (Outcome 4.1) which relates to improved land tenure security and administration resulting from USAID Hay Tao support. Two outcomes were from SA5 (Outcomes 5.2, 5.3), related to the contributions of communities to the increase in reporting of environmental crimes by communities through hotlines and environmental legal clinics and the completion of environmental crimes cases following CSO legal follow up. Community reporting reinforces the stewardship role of VOIs on their TGRNR which include a no-take zone, and a sustainable usage zone, which they use for their livelihoods, for example beekeeping and firewood. Finding 4.3: The lowest significance score received from substantiators was that of 5, for two outcomes focused on multi-stakeholder advocacy, out of a possible score of 10. The lowest significance score received was 5, which is midway along the spectrum and slightly closer to the description that the outcome was “not very significant to people’s lives and sustainable NRM” than “highly significant to people’s lives and sustainable NRM.” These relate to Outcome 4.7, regarding strengthened law enforcement due to improved sentencing decision by judges and enhanced collaboration between OPJs and OSPJs; and Outcome 5.1, regarding advocacy actions by multi-stakeholder actors to promote good governance of natural resources. “Thank you to USAID Hay Tao because it is the only one that has the courage to get involved in this sector which is one of the most problematic sectors in Madagascar.” Anonymous Substan�ator “USAID Hay Tao program has broadened the scope of involvement from grassroots communities to local, communal, and regional decision-making bodies, as well as other sectors such as land tenure.” Anonymous Substan�ator 49 In depth: Low significance for outcomes related to multi-stakeholder advocacy Outcome 5.1 relates to multi-stakeholder advocacy to promote good governance of natural resources. It is in many ways one of the least tangible outcomes, given that it describes an intervention that requires multiple steps along the causal pathway before resulting in changes felt by Malagasy people. Yet, such practices and behaviors are a necessary precondition for sustained NRM. This outcome demonstrates a shared vision and independent actions of critical actors, as civil society stakeholders were engaged in advocacy activities such as related to land reform, Dina laws, PAs, untitled private property (PPNT), and financial resources for NRM, among others. The aim of the intervention was to hold formal decision makers accountable and thereby to encourage them to change their practices to support sustainable NRM more proactively. Substantiators felt that the advocacy was less tangible for final, downstream impactees. One noted regarding this outcome specifically: “The project remains theoretical; the benefit of the project does not reach the target population.” Another noted, also specifically regarding this outcome, “NRM is still weak because sustainable livelihoods are not tangible.” It is possible that lack of confidence in the judicial system is what is driving this perspective, given the more significant scores in other outcomes of communities engaged in environmental crimes reporting. In addition, it can be challenging to understand the important role collaboration and multi-stakeholder advocacy play in the long game toward significant, tangible change. One recommendation would be to raise more awareness regarding the important role that such work plays in laying a solid foundation of sustainable change. Finding 4.4: The majority of harvested outcomes were beyond the “tipping point” stage of change, suggesting greater likelihood of endurance and continuation beyond the USAID Hay Tao Activity. Substantiators were asked to assess the degree of change – ad hoc, pilot, sticking, replicating, enduring – at the outcome level. Out of the 33 outcomes, substantiators concluded that sticking change was the most prevalent (10 outcomes), followed by replicating and enduring change (8 outcomes each) and lastly by pilot change (7 outcomes). No outcomes were identified as ad hoc change. The IGD framework puts the tipping point between Pilot and Sustainable (“Sticky”) change, which corresponds to a shift from early adopters who pilot the change, to an early majority who help sustain (“stick”) the change (see adapted figure 9 above). The chart below shows USAID Hay Tao’s outcomes by degree of change using the adapted IGD maturity model tipping point. Based on that model, the majority – 26 out of 33 – of harvested outcomes are beyond the tipping point, suggesting that USAID Hay Tao outcomes are more sustainable and are more likely to survive the close of the Activity. Figure 9 The IGD maturity model framework with tipping point identification 50 It is important to note that the IGD model as used here is concerned with the relative independence of an innovation: whether it lives on without need of intervention. This is why USAID Hay Tao selected the model to understand and measure the sustainability of the Activity’s outcomes. Sustainability is therefore in reference to the intervention, and therefore stages of the model are relative to the outcome in the intervention system. [See Methodology section for a larger discussion] Finding 4.5: Substantiators determined that eight outcomes were at the enduring level of change, indicating that the outcomes have a high likelihood of continuing without further intervention. Eight outcomes under SA1, SA2 and SA3 were identified by substantiators as “enduring” changes. This kind of change is one where the behavior spreads independently, may be adapted by actors, without need of further intervention. 51 In depth: From Pilot Change to Sticking Change, to Replicating Change, to Enduring Change Outcome 4.6 describes a pilot change, describing improved legal attention to natural resources governance in the court system, particularly land title cases, resulting from increased prioritization of environmental cases among magistrates and improved application of laws concerning land tenure linked to other challenges such as reconciliation of the environmental, urban and land planning laws. With more time, it is expected that this will result in sticking change when practices have entered into regular patterns and expanded into behaviors. That shift would reflect a “tipping point.” This pilot change is in contrast to Outcome 2.6, which describes a sticking change. Investigative journalists utilized their improved capacity to provide higher quality and more effective coverage of land tenure issues in the media and to prepare more effective media products on a regular basis, including for example the “Paradisa karakaina” video which won the third prize of the Grand Prix Malina 2022. The outcome also demonstrates improved relationships between journalists and governing institutions. One substantiator noted, “The change can be seen in the relationship between journalists and the sources of information at the level of institutions, who have become more accessible and more open.” The depth and quality of the changed behavior brings this beyond the tipping point and is expected to “stick” among the targeted change agents, yet it has not moved to be replicated beyond the targeted group of journalists. Outcome 4.4 is an example of replicating change. Dinan'i Menabe— affecting all 5 districts— were improved and formalized to include sustainable environmental considerations, paving the way for improved CBNRM, and an indicator of changed attitudes and perceptions among Menabe communities now willing to actively collaborate in the preservation of their natural resources together with various stakeholders – GoM, civil society, judiciary. It is a replicating change because of how the intervention encouraged the involvement of actors beyond the targeted group, and the ripple effect throughout the system of Dina. One substantiator noted, “In addition to the environmental actors who initiated the project to update the Menabe Dina, USAID Hay Tao has broadened the scope of involvement from grassroots communities to local, communal, and regional decision-making bodies, as well as other sectors such as land tenure.” The last stage of the maturity model is the enduring stage, which is when a change is fully established and is independently adopted by actors without intervention. Outcome 1.3 describes the critical importance of effective PA management. UASID Hay Tao helped revise the PAG framework to include community well-being and sustainable development, thereby improving the viability of conservation targets by considering the impacts of climate change. Since March 2021, SAPM has been using the new PAG framework which integrates improved considerations including climate adaptation and sustainable development, through the PHE approach. To date, 65% of PAs have renewed their PAG using the new framework. According to substantiator scores, this change is fully established and is independently adopted by actors without further intervention. 52 Finding 4.6: USAID Hay Tao and substantiators disagree on the sustainability of outcomes related to SCE. While substantiators identified 8 outcomes as enduring, it must be noted that USAID Hay Tao Activity staff only identified one outcome as at the enduring level of change. The graph below shows the difference between substantiator (blue) and USAID Hay Tao (red) determinations regarding sustainability. As can be seen in the graph, USAID Hay Tao Activity staff were more conservative in identifying degrees of sustainability than substantiators. The “enduring” outcome that USAID Hay Tao Activity staff and substantiators agreed on was Outcome 1.3 described above, regarding the operationalization of the PAG framework and PAG renewals by 65% of PAs. Five of the outcomes identified as “enduring changes” by substantiators were rated as “scale changes” by USAID Hay Tao staff, which is not so far off: it is a one￾degree difference. However, two outcomes had a more significant degree of difference between substantiators and USAID Hay Tao: one was identified by Hay Tao staff as “sticking” and one as “pilot.” Both of these related to social and conservation entrepreneurship (SCE). In depth: Contrasting perspectives between USAID Hay Tao and Substantiators on SCE outcomes USAID Hay Tao and substantiators disagreed most widely on two outcomes related to social and conservation entrepreneurship (SCE). USAID Hay Tao identified Outcome 3.4 as a “sticking” change, rather than the “enduring” change selected by substantiators. This outcome is about the coordination and exchange in the promotion of SCE in Madagascar. In 2020, a national SCE platform was set up, bringing together the GoM, the private sector, CSOs and technical and financial partners, the first multi-stakeholder platform of its kinds in Madagascar 53 to ensure coordination and exchange for promoting SCE in Madagascar, paving the way for draft SCE legislation which will enable more sustainable business models and reduce pressure on natural resources. Of the outcome, one substantiator noted, “The scope concerns several actors at the local and national level. Indeed, the bill and the SCE concept concern the majority of economic actors in Madagascar.” Another noted, “It is a perennial and sustainable innovation on the concept of SCE. The bill is a beacon for all SCE actors.” However, USAID Hay Tao cautioned that while the SCE platform was a critical, necessary condition, it is not sufficient, and much more needs to be done to ensure a continuation of SCE practices to incentivize the private sector to support sustainable NRM. The widest variation between USAID Hay Tao staff members and substantiators was regarding Outcome 3.1. USAID Hay Tao staff members identified it as a “pilot” change, rather than the “enduring” change selected by substantiators. Outcome 3.1 described improved leadership interest and engagement in the topic of SCE as a result of MICC (formerly MICA) successes in advancing SCE at the national level, such as by submitting the draft law to GoM and Parliament and supporting the establishment and implementation of SCE business models, which will invest profits into environmental protection, benefiting local community welfare and biodiversity conservation. The implementation of 10 to-date business models will enable local structures to allocate between 5% to 20% of their profits to biodiversity conservation, to reforest 60 hectares of land, to improve the well-being of 221 members of local communities and of 41 households. On this topic, USAID Hay Tao also cautioned that SCE is fragile, and noted that the SCE concept “is fairly new.” The outcome does not describe an expansion or independent replication. Finding 4.7: There was no correlation between type of change (role, resource, rule, relationship, system) and the degree of outcome sustainability; types of change were found at all degrees of sustainability. Analysts also considered the difference in degrees of sustainability between the types of changes. For example, to what extent are “role” changes expected to be more sustainable than “resource” changes? We hypothesized that there would be more sustainability in role changes because those are the most straightforward types of change and the precursor for longer term outcomes and behavioral change. To answer this question, analysis was conducted using USAID Hay Tao staff perspectives on sustainability rather than that of substantiators for the following reasons: (i) Substantiators were asked to provide an overall assessment of degree of change for each outcome and not an assessment of degree of change disaggregated by type of change – this would have been overly complex; whereas USAID Hay Tao staff went a step further and coded each outcome by degree of change (ad hoc, pilot, sticking, replicating, enduring) for each type of identified change (roles, relationships, resources, rules, systemic). Therefore, a single outcome had multiple types of change at different degrees. (ii) As demonstrated earlier, USAID Hay Tao staff were more conservative in their perspectives of the likelihood of sustainability than substantiators, and therefore this analysis provides a more conservative analysis of the sustainability of various types of change. The chart below shows the 88 types of changes exhibited across the 33 outcomes, with their degrees of sustainability, as identified by USAID Hay Tao staff. Overall, as noted earlier, on average each outcome exhibited three types of changes (Finding 1.3). All outcomes had relative consistency in the degrees of change for each type, with at most a one-degree difference: for example, a pilot role change may be matched with a pilot resource change or sticking rule change. No outcome, for example, had a pilot type of change and an enduring type of change. Only one outcome had a variation beyond one degree: Outcome 54 3.6 exhibited a role and relationship change at the sticking level, and a resource change at the enduring level. Overall, from the perspective of types of change, system changes were evenly distributed across pilot, sticking, and enduring degrees, with four outcomes each. Role changes took place most at the enduring level (5 outcomes were role changes at the enduring level). Resource changes took place most at the pilot (6 outcomes) and enduring (also 6 outcomes) levels. Relationship changes took place most at the sticking level (6 outcomes). And role changes took place most at the sticking (8 outcomes) and enduring (7 outcomes) levels. Figure 10 Degree and type of change according to USAID Hay Tao across the outcomes From the perspective of degrees of change, pilot changes took place most among outcomes with resource changes (6 outcomes were resource changes at the pilot level). Sticking changes took place most among outcomes with a role change (8 outcomes were role changes at the sticking level). Replicating change took place fairly evenly across roles, relationships, and resources (5 outcomes, 4 outcomes, and 5 outcomes respectively). And enduring change took place most among outcomes with a role change (7 outcomes), resource (6 outcomes) and rule (5 outcomes) changes. 55 In Depth: A combination of “sticking” and “enduring” types of change Outcome 3.6 exhibited a role and relationship change at the sticking level, and a resource change at the enduring level. This outcome was the only one to exhibit a two-degree difference in degree of change between its multiple types of change. “Sticking” role change “Sticking” relationship change “Enduring” resource change This outcome relates to a change in resource commitment to FAPBM. As of December 2021, FAPBM’s capital amounted to $139 million thanks to capital contributions of $8 million from Agence Française de Développement and $4.5 million from Conservation International in 2020, and $45.7 million from KfW in 2021. As a result, the number of PAs funded by FAPBM has increased from 31 in 2018 to 42 in 2022, contributing to improved NRM over an area of 501,800 ha, including two marine PAs. With the funds, it expects to fund a total of 63 PAs in 2023, essential to keeping PAs open and functional by enabling PA management, including patrolling, protection, and affording professional enforcement agents. This outcome clearly demonstrates a resource change that enables the continued operation of PAs, which do not receive government funding for management purposes. It also is a result of an intensive communications and fundraising campaign that changed how people perceive their roles and responsibilities, and how they interacted – coordinated, cooperated, communicated – with other actors and sectors on the topic of improved NRM. For example, the communication campaign included factsheets, flyers, white papers, annual reports, and were used to sensitize prospective and current donors and financial partners on the importance of PA and biodiversity conservation and acknowledge FAPBM role, activities, results and responsibilities in PA management. That increased awareness changed the perception of the role of funders in ensuring continuity of management of PAs. It also changed how stakeholders worked together; through the fundraising and communication campaign efforts, a partnership developed to ensure harmonization of support to FAPBM to avoid redundancy and competing strategies. USAID Hay Tao played an important convening role. The partnership evolved into a more formal consortium of partners composed of AHT Consulting Group, USAID Hay Tao and KfW to support FAPBM in efforts to increase its capital. The consortium meets regularly to discuss the implementation of the fundraising campaign and the cost-benefit analysis of PA management to support and justify the need to invest in PAs. For these reasons, the resource change of this outcome also exhibited role and relationships changes. While the new capital to FAPBM is enduring, role and relationship changes are still at the sticking level for this outcome. When the partnership and the fundraising expand and begin replicating beyond this core group, the degree of change may increase. 56 ATTACHMENTS I. Summary of Harvested Outcomes & Substantiator Responses Outcome Statement Significance to People’s Lives Degree of USAID Hay Tao Contribution Sustainability Types of Change Substan￾tiator USAID Hay Tao Substan￾tiator USAID Hay Tao Substantiator SA 1: Support conservation stakeholders on sustainable livelihoods, NRM, and environmental justice through capacity development Outcome 1.1: Vanilla producers and processors (PPV) in the Analanjirofo, Atsinanana, and SAVA regions applied improved cultivation techniques that have reduced forest degradation and climate change impacts on the vanilla value chain since June 2021. 8 8 10 10 Enduring Change Role Resource System Outcome 1.2: Improved TGRNR management tools institutionalized by MEDD, including cost-effective assessments of social and economic impacts, facilitates the critical renewal and improvement of TGRNR contracts, the improved credibility of VOIs (community-based organizations) as stewards of their TGRNR sites, and advances the involvement of local communities (CBNRM); the model was cascaded down by DREDD officers in at least 28 TGRNR sites. 9 9 9 9 Enduring Change Role, Resource, Rule, System Outcome 1.3: Effective PA management is crucial. Since March 2021, SAPM has been using the new PAG framework which integrates improved considerations (including adaptation to climate change and sustainable development and community well-being) through PHE. To-date, 65% of PA have renewed their PAG using the new framework. 8 9 7 10 Enduring Change Resource, Rule, System Outcome 1.4: A national-level coalition of civil society in coastal and marine protection, MIHARI, implemented a new capacity development approach and standards into community-based management and governance of marine and coastal resources in its strategy and workplan. 10 8 8 8 Pilot Change Resource, Rule, System Outcome 1.5: Women -- who are responsible for 50% of the small￾scale fishing sector – have increased leadership in the fisheries value 8 7 7 7 Pilot Change Resource, System 57 Outcome Statement Significance to People’s Lives Degree of USAID Hay Tao Contribution Sustainability Types of Change Substan￾tiator USAID Hay Tao Substan￾tiator USAID Hay Tao Substantiator chain, specifically in decision-making in regional resource governance structures such as in the Ambanja fisherfolk's federation, as are result of a new women’s leadership program implemented by MIHARI. Outcome 1.6: Madagascar for the first time participated in a certification program for PAs that aims to mitigate biodiversity loss from poor management and adapted key indicators from the IUCN Green List to PA management, introducing globally applicable sustainable development standards. 7 8 8 6 Sticking Change Resource, System Outcome 1.7: Improved regional planning led to more realistic national reforestation target of 75,000 ha per year, as a result of a participatory process involving key stakeholders including regional STDs, private sector, CSOs and grassroot communities, and provision of technical and practical reforestation tools. 8 10 8 8 Replicating Change Role, Resource, System Outcome 1.8: 47 organizations cascaded their technical knowledge to grassroots CSOs and improved the impact of their activities at local and regional levels in Menabe and MaMaBay ecoregions, after improving their technical capacity in topics ranging from advocacy, land tenure, environmental law enforcement, whistleblowing, vanilla, innovative technologies, data management, to environmental justice, through capacity action plans and training programs. 9 8 9 9 Sticking Change Role, Relationship, Resource SA2: Improve and make available data and information for evidence-based decision-making on conservation and sustainable livelihoods Outcome 2.1: Improved evidence-based PA management, such as in more effective and timely patrols to fight fires and deforestation in PAs managed by MNP (including Kirindy Mite, Andranomena, Masoala, Ankarafantsika), as a result of the utilization by PAs managers of information gathered from innovative technologies, including Forest Watcher and SMART. 10 10 9 8 Enduring Change Role, Resource, Rule, System Outcome 2.2: Improved use of evidence and technology (including the Hay Natiora portal) for decision-making to reduce pressure on the Menabe Antimena PA and improve living conditions of PA migrants resulting from a collaborative approach, the establishment of the CRORFAD Committee tasked with the operationalization of NA (substanti ators noted that it is “too 10 8 10 Pilot Change Role, Resource, System 58 Outcome Statement Significance to People’s Lives Degree of USAID Hay Tao Contribution Sustainability Types of Change Substan￾tiator USAID Hay Tao Substan￾tiator USAID Hay Tao Substantiator the land reserve and sustainable agriculture in the Menabe Region, and the application of new technology to identify viable sites. early to say”) Outcome 2.3: Institutionalized commitment by the government of Madagascar for evidence-based decision making, through the decision by the National Environment Office to take over the management of the Hay Natiora data portal and integrating the portal into its existing systems ensuring sustainability. 9 10 10 10 Replicating Change Role, Relationship, Resource, System Outcome 2.4: Reassertion of GoM’s priority regarding environmental protection through the enforcement of its zero￾tolerance policy for environmental crimes, which lapsed during COVID-19 when lockdowns and border closure measures pushed up unemployment and increased pressures on natural habitats and consequently on biodiversity as communities engaged in unsustainable use of natural resources to meet livelihoods needs, after communication campaigns and watchdog functions. 10 8 10 10 Replicating Change Relationship, Rule Outcome 2.5: Investigative journalists with improved research and reporting capacity contribute as informants regarding environmental crimes, including hatsake (land clearing and slash-and-burn agriculture), illegal cultivation of maize and peanuts inside PAs by mafia networks and corrupt local elected officials, and exploitation of climate migrants, which demonstrates an increased appreciation for the critical importance of the environment and the need for collective efforts. 10 10 9 10 Enduring Change Role, Relationship Outcome 2.6: Investigative journalists utilized improved capacity to provide higher quality and more effective coverage of land tenure issues in the media and to prepare more effective journalistic products on a regular basis, such as a video report entitled “Paradisa karakaina” which won the third prize of the Grand Prix Malina 2022 awarded annually by Transparency International - Madagascar Initiative, also a result of improved relationships between journalists and governing institutions. 9 9 8 8 Sticking Change Role, Relationship, System SA3: Identify and foster innovative financing options for community-based conservation 59 Outcome Statement Significance to People’s Lives Degree of USAID Hay Tao Contribution Sustainability Types of Change Substan￾tiator USAID Hay Tao Substan￾tiator USAID Hay Tao Substantiator Outcome 3.1: Leadership interest in and engagement in the topic of social and conservation entrepreneurship (SCE) has significantly advanced as a result of MICC (formerly MICA) successes in advancing SCE at the national level, such as by submitting the draft law to GoM and Parliament and supporting the establishment and implementation of SCE business models, which will invest profits into environmental protection, benefiting local community welfare and biodiversity conservation. 7 10 9 10 Enduring Change Role, Resource, Rule Outcome 3.2: Advances in local private sector contributions to sustainable practice through efforts to reform hotel management to meet “Green Globe” certification, resulting in three engaged hotels in Menabe and their additional financial investments. 9 8 7 8 Pilot Change Role, Relationship, Resource Outcome 3.3: Two associations of women craftworkers, FIVEMA and FITARATRA, in MaMaBay established nurseries and restored penjy (a reed that grows in water, and is used in wickerwork such as hats, mats, baskets, sacks, and packing bags) and raphia resources to achieve sustainable development of these natural resources used in the craft industry, demonstrating a changed awareness, attitude and practice toward the sustainable management of resources utilized in private entrepreneurship and livelihoods and advances in women￾led sustainable development. 9 10 8 9 Sticking Change Role, Resource Outcome 3.4: In 2020, a national SCE platform was set up, bringing together the GOM, the private sector, CSOs and technical and financial partners, the first multi-stakeholder platform of its kinds in Madagascar to ensure coordination and exchange for promoting SCE in Madagascar, paving the way for draft SCE legislation which will enable more sustainable business models and reduce pressure on natural resources used for economic activities. 8 8 8 9 Enduring Change Role, Relationship, Rule Outcome 3.5: The national Natural Capital (NC) network in Madagascar, composed of the GOM, the private sector, CSOs and technical and financial partners, was established by GOM and relevant stakeholders in March 2021 with the objective of integrating NC valuation in governmental decision-making and investment/economic activities, demonstrating a change in attitude 8 10 7 5 Sticking Change Role, Relationship, Rule, System 60 Outcome Statement Significance to People’s Lives Degree of USAID Hay Tao Contribution Sustainability Types of Change Substan￾tiator USAID Hay Tao Substan￾tiator USAID Hay Tao Substantiator and knowledge regarding the importance of NC to sustainable NRM, and the institutionalization of this change in attitude and knowledge as exemplified by a formal network led by GOM and stakeholders. Outcome 3.6: Increased continuity of conservation and environmental protection via financial sustainability of PA management through increased resources and capacity of the Madagascar Protected Areas and Biodiversity Fund (FAPBM), thanks to improved communications and fundraising resulting in over $58 million addition to the Fund in 2021, enabling FAPBM to support 11 additional PAs across an additional 501,800 hectares. 7 9 7 7 Enduring Change Role, Relationship, Resource Outcome 3.7: Improved technical and financial management by community-based organizations (VOIs) in the southern ecoregion resulting in improved livelihoods and biodiversity conservation, resulting from advanced fund management capacity by the Tany Mena Foundation in support of communities involved in NRM and sustainable development. 8 7 8 7 Replicating Change Role, Resource SA4: Strengthen public sector institutions in policy-making and enforcement of tenure rights and NRM Outcome 4.1: Improved ecological and socio-economic land management in 11 communes in Menabe and 27 communes in MaMaBay resulting from the introduction of participatory land surveys, enabling the harmonization of sustainable natural resources management with land tenure management and spatial planning. 10 9 9 10 Sticking Change Role, Resource Outcome 4.2: Improved management of coastal and marine tenure rights as a result of corrections to identified gaps – including the development of the Menabe region Marine Spatial Plan (MSP) and fisheries -related Dina and Dinabe databases which are now integrated into the Hay Natiora portal to improve fisheries management– originally identified in the tenure assessment (MAEP/DGPA, now MPEB/DGPA, MEDD, and MATSF) since 2020. 8 10 10 10 Replicating Change Resource Outcome 4.3: Land using planning in APMA was improved to include environmental and social concerns affecting natural resources sustainability, biodiversity, and habitats through more 8 9 8 10 Sticking Change Role, Relationship, Rule, System 61 Outcome Statement Significance to People’s Lives Degree of USAID Hay Tao Contribution Sustainability Types of Change Substan￾tiator USAID Hay Tao Substan￾tiator USAID Hay Tao Substantiator responsive communal management plans (SACs) in four communes which will contribute to better NRM and sustainable land tenure in the Menabe landscape as a result of the harmonization of land management at the commune level with the development and management plan of the entire PA in a more sustainable manner. Outcome 4.4: Dinan'i Menabe – affecting all the 5 districts of the region – was improved and formalized to include sustainable environmental considerations after a participatory community￾centered approach, paving the way for improved, community-based NRM. 9 10 9 8 Replicating Change Relationship, Resource, Rule Outcome 4.5: The Dinabe concerning the exploitation of fisheries resources in MaMaBay was updated in 2021, correcting articles that were rejected by the courts in 2016, with the participation of CSOs (PCDDBA, FPBA and AVG) and PA managers (WCS) and other stakeholders at the local level as a result of improved legal capacity and the development of a shared vision among participating stakeholders. 8 10 8 8 Sticking Change Relationship, Rule Outcome 4.6: Improved legal attention to natural resources governance in the court system, particular land title cases, resulting from increased prioritization of environmental cases among magistrates and improving the application of laws concerning land tenure which are linked to other challenges such as the reconciliation of the environmental, land, urban and land planning laws. 8 8 9 9 Pilot Change Role Outcome 4.7: 45% of targeted OPJs and OSPJs applied capacity development and tools for environmental crime prosecution, resulting in strengthened law enforcement, particularly with regard to sentencing decisions by judges, despite structural challenges such as the reassignment of location. 5 5 9 8 Sticking Change Role SA5: Support civil society and private sector coalitions to advocate for environmental and social governance through constructive engagement Outcome 5.1: Diverse national and regional stakeholders have increased collaboration in support of more sustainable NRM, 5 7 6 7 Sticking Change Role, Relationship 62 Outcome Statement Significance to People’s Lives Degree of USAID Hay Tao Contribution Sustainability Types of Change Substan￾tiator USAID Hay Tao Substan￾tiator USAID Hay Tao Substantiator demonstrating a shared vision and independent actions, and are engaged in advocacy activities to promote good governance of natural resources, such as related to land reform, Dina laws, protected areas (PAs), untitled private property (PPNT), and financial resources for NRM, among others. Outcome 5.2: Communities have a changed awareness and attitude toward the importance of sustainable NRM to their livelihoods, as evidenced by an increase in community reporting of environmental crimes through AVG's 512 hotline and in-person visits to the environmental legal clinics (CJEs). 10 10 9 8 Replicating Change Role, Relationship Outcome 5.3: Since 2019, court decisions regarding environmental crimes have increasingly utilized community and informant reporting in Morondava and Maroantsetra, demonstrating an increase in community engagement in the improvement of environmental governance, and an improvement in enabling mechanisms available to communities and informants: denunciations are facilitated through a formal process linking communities through CSOs to Judicial Police Officers (OPJs) for investigation, with CSOs supporting with evidence generation to ensure community reports are of high quality. 10 10 9 8 Replicating Change Role, Rule Outcome 5.4: Responsible business development is critical to sustainability, and private sector actor DHL demonstrated increased commitment to strengthen the adoption of eco-gestures, conservation agriculture, and the environmental protection awareness of actors and the population in Analamanga, Menabe, Vakinankaratra, Antsinanana and Analanjirofo through increases to its corporate social responsibility investments. 8 8 7 8 Pilot Change Relationship, Resource Outcome 5.5: A more enabling framework for the role of CSR in environmental sustainability and investment opportunities therein has emerged in Madagascar as a result of (i) the launch of the national CSR strategy (SNRSE), which was launched in October 2020 by MEDD and EDBM, exemplifying increased awareness by GOM of the important role that private entities play in supporting and advancing NRM, and (ii) mobilization around CSR 8 8 8 8 Pilot Change Role, Relationship, Resource, Rule, System 63 Outcome Statement Significance to People’s Lives Degree of USAID Hay Tao Contribution Sustainability Types of Change Substan￾tiator USAID Hay Tao Substan￾tiator USAID Hay Tao Substantiator environmental issues, with SNRSE demonstrating collaboration across stakeholders with a shared vision, which is critical to sustainability and the ultimate success of the endeavor. 64 Outcome Harvest Worksheet OUTCOME SHORT DESCRIPTION Describe the details of the selected outcome based on available information (WHO? WHAT? WHEN? WHERE?) OUTCOME LONG DESCRIPTION Describe WHO did WHAT, WHEN and WHERE. TYPE OF CHANGE What type of change did this outcome include? Select all that apply and identify the degree of change: Ad Hoc Change Pilot Change Sticking Change Replicating Change Enduring Change o Role change o Relationship change o Resource/tools change o Rules change o System change OUTCOME IMPORTANCE Clarify the relevance/importance of the outcome (WHY?); describe the critical value of the outcome. 65 USAID HAY TAO CONTRIBUTION Provide tangible evidence linking the direct contribution of the program team to this outcome (HOW?). Explain where the evidence can be found and how to access it. Please be as specific as possible. CONTACT WHO is accountable for this result? (Name & contact information of the organization / actor) SIGNIFICANCE Why does this outcome deserve attention in the context, and/or for USAID Hay Tao objectives. Please be as specific as possible. DESCRIPTION OF THE PROBLEM What specific challenge was addressed by the outcome? SUBSTANTIATION Who would you recommend to substantiate this result? The substantiator must know the result but can not have been part of the USAID Hay Tao program (e.g. he/she has not benefited from USAID Hay Tao resources). Name, title, contact, role with USAID Hay Tao. 66 II. Outcome Harvest Substantiation Form SECTION A Name: 1. Are you aware of the USAID Hay Tao (or insert partner or relevant description) program? ☐ No, I do not know anything about it ☐ Yes, I know about it If no, skip to Section B. If yes, please describe your interaction with USAID Hay Tao: 2. What has been your interaction with USAID Hay Tao? Select all that apply: ☐ I participated in USAID Hay Tao activities ☐ I received support from USAID Hay Tao ☐ I have seen USAID Hay Tao materials ☐ Other, please describe: SECTION B Please read this outcome description: (insert relevant outcome description) Now, please answer the following questions: 3. To what extent do you agree or disagree with this outcome statement? ☐ Fully agree 67 ☐ Partially agree ☐ Disagree Optional comments: If you disagree, please skip to Section C. If you agree, please answer the following question: 4. What degree of change do you think this outcome corresponds to? Please check one box. 1 2 3 4 5 1.Ad Hoc Change (one-off, temporary) Pilot Change (basic change has happened, but fragile) Sticking Change (change has happened, strong among the actors or institutions) Replicating Change (change is getting larger, more actors or locations) Enduring Change (change is throughout the system, and built in) ☐ ☐ ☐ ☐ ☐ Optional comments: 68 SECTION C Please read this outcome contribution description: (insert relevant outcome description) Now, please answer the following questions: 5. Based on the contribution description, to what extent do you agree or disagree that USAID Hay Tao had a role in the outcome? ☐ Fully agree ☐ Partially agree ☐ Disagree Optional comments: 6. On a scale of 1-10, where 1 is “Happened without USAID Hay Tao” and 10 is “Only happened because of USAID Hay Tao,” please select where you think this outcome falls (select only one number by checking the box): ← 1: Happened without USAID Hay Tao Happened because of USAID Hay Tao: 10 → 1 2 3 4 5 6 7 8 9 10 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ Optional comments: 69 7. On a scale of 1 to 10, where 1 is “not very significant to people’s lives” and 10 being “highly significant to people’s lives”, please select where you believe this outcome falls (select only one number by checking the box): ← 1: Not very significant to people’s lives Highly significant to people’s lives: 10 → 1 2 3 4 5 6 7 8 9 10 ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ ☐ Optional comments: THANK YOU! III. SA Results Chains with Harvested Outcomes