i | INRM MEL PLAN USAID Integrated Natural Resource Management (INRM) HEARTH Monitoring and Evaluation Toolkit APRIL 2022 HEARTH Monitoring and Evaluation Toolkit ii Activity Title: Integrated Natural Resource Management Sponsoring USAID Office: Center for Environment, Energy, and Infrastructure Contract Number: 7200AA20F00010 Task Order Number: STARR II – 7200AA18D00020 Period of Performance: July 30, 2020 – July 19, 2025 Contracting Officer: Stella Alexander-Sergeeff COR: Ioana Bouvier Ceiling price: $34,976,131 Contractor: DAI and partners: American Museum of Natural History Center for Biodiversity Conservation, Conservation Strategy Fund, New America, Resource Equity, Social Impact, Stantec, and University of Rhode Island Coastal Resource Center Date of Publication: April 2022 Authors: Samantha Cheng, Mike Duthie, Daniel Evans, Aaron Ferguson, Andres Gomez, Scott Miller, Christina Seybolt, and Meredith Wiggins Front Cover photo: Collecting field data using GPS receivers, Democratic Republic of the Congo. Photo by OSFAC. Submitted by: Robin Martino, Chief of Party Integrated Natural Resource Management DAI 7600 Wisconsin Ave, Suite 200 Bethesda, MD, 20814, USA Telephone: 301.771.7600 Fax: 301.771.7777 www.dai.com This document was produced for review by the United States Agency for International Development. It was prepared with support from the Integrated Natural Resource Management Task Order, under the Strengthening Tenure and Resource Rights II (STARR II) IDIQ. The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. HEARTH Monitoring and Evaluation Toolkit iii Contents Contents .................................................................................................................................... iii Acronyms .................................................................................................................................. vi Overview .................................................................................................................................... 1 Background............................................................................................................................................................................ 1 Developing a cross-sectoral package of Monitoring, Evaluation, Research, and Learning tools...................... 1 How can Missions and implementing partners use this toolkit to measure environmental and development outcomes?..................................................................................................................................................... 2 Alignment with the Triple Bottom Line and Sustainable Development Goals..................................................... 2 Toolkit Development and Key Outcomes of Interest................................................................................................ 3 Indicator Guidance and Core Household Questionnaire.......................................................................................... 3 How To Use This Toolkit................................................................................................................................................10 Household Sampling and Data Collection Guidance .................................................... 11 Participant-Based Survey and Population-Based Survey Sampling Approaches..................................................11 Data Collection Administration and Frequency.........................................................................................................12 Data Management..............................................................................................................................................................12 Privacy, Confidentiality, and Ethics................................................................................................................................12 Toolkit Modules 14 Household Roster & Respondent Identification ............................................................ 14 Household Rosters............................................................................................................................................................14 Respondent Identification ................................................................................................................................................15 Food Security and Nutrition ............................................................................................... 16 Pathways to Change ..........................................................................................................................................................16 Recommended Outcomes and Indicators...................................................................................................................16 Performance Indicator Reference Sheets.....................................................................................................................20 Gender Equality & Social Inclusion (GESI) ...................................................................... 35 Pathways to Change ..........................................................................................................................................................35 Recommended Gender Outcomes and Indicators....................................................................................................36 Performance Indicator Reference Sheets.....................................................................................................................39 Health ....................................................................................................................................... 51 Pathways to Change ..........................................................................................................................................................51 Recommended Outcomes and Indicators...................................................................................................................52 Performance Indicator Reference Sheets.....................................................................................................................59 HEARTH Monitoring and Evaluation Toolkit iv Biophysical ............................................................................................................................... 86 Overview..............................................................................................................................................................................86 Framework for Biophysical Outcomes.........................................................................................................................86 Data Standardization, Structure, And Management ..................................................................................................87 Recommended Outcomes and Indicators...................................................................................................................88 Biophysical Measurement Guidance..............................................................................................................................94 Performance Indicator Reference Sheets.................................................................................................................. 119 Conservation Knowledge, Attitudes & Practices (KAP) ............................................ 147 Pathways To Change...................................................................................................................................................... 147 Recommended Outcomes and Indicators................................................................................................................ 147 Performance Indicator Reference Sheets.................................................................................................................. 150 Agriculture & Land .............................................................................................................. 157 Pathways To Change...................................................................................................................................................... 157 Recommended Outcomes and Indicators ................................................................................................................ 157 Performance Indicator Reference Sheets.................................................................................................................. 162 Conservation Enterprise (CE) Benefits .......................................................................... 180 Pathways To Change...................................................................................................................................................... 180 Recommended Outcomes and Indicators................................................................................................................ 180 Performance Indicator Reference Sheets.................................................................................................................. 183 Collecting Information from Implementing Partners and The Private Sector................................................. 190 Resilience ............................................................................................................................... 192 Pathways To Change...................................................................................................................................................... 192 Recommended Outcomes and Indicators................................................................................................................ 192 Performance Indicator Reference Sheets.................................................................................................................. 195 Socio-economic Well-being .............................................................................................. 202 Pathways To Change...................................................................................................................................................... 202 Recommended Outcomes and Indicators................................................................................................................ 202 Performance Indicator Reference Sheets.................................................................................................................. 205 Governance .......................................................................................................................... 211 Pathways To Change...................................................................................................................................................... 211 Recommended Outcomes and Indicators................................................................................................................ 211 Performance Indicator Reference Sheets.................................................................................................................. 213 Private Sector Engagement ............................................................................................... 216 Pathways To Change...................................................................................................................................................... 216 Recommended Outcomes and Indicators................................................................................................................ 216 Performance Indicator Reference Sheets.................................................................................................................. 218 HEARTH Monitoring and Evaluation Toolkit v Annexes 225 Annex 1. Time Use ............................................................................................................. 225 Annex 2. Guidance For Using Global Forest Watch Data ...................................... 228 Annex 3. Conservation Valuation ................................................................................... 229 Annex 4. Land Measurement ............................................................................................ 232 Annex 5. Data Sharing Agreements ................................................................................ 234 Annex 6. Socioeconomic Status ...................................................................................... 236 Tables and Figures Table 1: Overview of Outcomes and Recommended Indicators by Sector..................................................... 4 Table 2: Item Weights Used in the Equations for Estimating Individual Scores for each Domain of the SWPER Index....................................................................................................................................................................41 Table 3: Mean and Standard Deviation for the Standardization of the SWPER Scores...............................42 Table 4: Cut-offs Used to Categorize the SWPER Domains into Low, Medium, and High Empowerment Levels..................................................................................................................................................................................42 Table 5: Modified EBV Framework of Variables as a Guide for Indicator Development.............................87 Table 6: Brief Summary of Types of Tools and Methods Used to Monitor Species.....................................96 Table 7: Types of Indicator Species and Definitions.............................................................................................99 Table 8: Common Indicators for Water Pollution............................................................................................. 113 Figure 1: HEARTH MERL Framework: People, Planet, Prosperity, Peace, and Partnerships....................... 3 Figure 2: Before and After Comparison of Simulated Forest Extent Data within HEARTH Activity Sites ........................................................................................................................................................................................... 105 Figure 3: Simulated Percent Forest Loss Data in HEARTH Activity Sites and a Group of Comparison Sites.................................................................................................................................................................................. 105 HEARTH Monitoring and Evaluation Toolkit vi Acronyms AFOLU Agriculture, Forestry, and other Land Use ARSSI Ability to Recover from Shocks and Stresses Index ATR Ability to Recover A-WEAI Abbreviated - Women’s Empowerment in Agriculture AWI Absolute Wealth Index CCP Conservation and Communities Project CE Conservation Enterprise CHW Community Health Worker CIFOR Center for International Forestry Research CPI Consumer Price Index CWI Comparative Wealth Index DDL Development Data Library DHS Demographic and Health Surveys DNA Deoxyribonucleic Acid EBV Essential Biodiversity Variables ENRM Environmental and Natural Resource Management EPA Environmental Protection Agency FAO Food and Agriculture Organization FFP Food for Peace FIES Food Insecurity Experience Scale FTF Feed the Future GCC Global Climate Change GESI Gender Equity and Social Inclusion GFW Global Forest Watch GhG Greenhouse Gas GIS Geographic Information System GLAD Global Land Analysis and Discovery GPS Global Positioning System HAZ Height for Age Z Score HEARTH Health, Ecosystems, and Agriculture for Resilient Thriving Societies IIED International Institute of Environment and Development IM Instrument Measurement INRM Integrated Natural Resource Management IP Implementing Partner IRB Institutional Review Board IWI International Wealth Index KAP Knowledge, Attitude and Practices LandPKS Land Potential Knowledge System LCC Land Capability Classification LCU Local Currency Units HEARTH Monitoring and Evaluation Toolkit vii LMIC Low-to-Middle-Income-Countries LSMS Living Standards Measurement Studies MCV Measles Containing Vaccination MDD-W Minimum Dietary Diversity of Women MERL Monitoring, Evaluation, Research, and Learning MII Method Information Index NGO Non-Government Organization NTFP Non-Timber Forest Products OFDA Office of Foreign Disaster Assistance ORS Oral Rehydration Solution PBSs Population-Based Survey PEN Poverty Environment Network PII Personally Identifiable Information PIRS Performance Indicator Reference Sheet PMA Performance Monitoring for Action PPI Poverty Probability Index PPP Purchasing Power Parity PSE Private Sector Engagement RAND Research and Development TP Total Production SAGE Site-Level Assessment of Governance and Equity SDG Sustainable Development Goals SEI Shock Exposure Index SES Socioeconomic Status SWPER Survey-Based Women’s Empowerment Index UP Units of Production USAID United States Agency for International Development USD United States Dollar USG United States Government HEARTH Monitoring and Evaluation Toolkit 1 Overview Background The United States Agency for International Development (USAID) has long recognized the importance of cross-sectoral programming to advance sustainable development. Integration is key to both the Biodiversity Policy and the Environmental and Natural Resource Management (ENRM) Framework, and to operationalizing a One Health approach to development. Yet, monitoring, evaluation, research, and learning (MERL) can be a challenge for integrated activities because practitioners need to assess performance across multiple sectors and technical elements. USAID’s Health, Ecosystems, and Agriculture for Resilient Thriving Societies (HEARTH) program is working with the Integrated Natural Resource Management (INRM)1 team to help USAID Missions meet this challenge. Developing a cross-sectoral package of Monitoring, Evaluation, Research, and Learning tools The HEARTH program fosters public-private partnerships to advance integrated investments in conservation, agriculture, health, governance, and other sectors. HEARTH aims to promote the conservation of threatened ecosystems and the well-being and prosperity of communities that depend on them. While USAID Missions co-design, fund, and manage integrated activities in the field, a cross￾bureau HEARTH team in Washington supports the design process and MERL across the program. Together, HEARTH and INRM have created a suite of indicators and guidance that will help USAID Missions monitor HEARTH progress and aggregate common metrics from across HEARTH sites to build the evidence base around the effectiveness of integrated strategic approaches. Using common indicators to monitor metrics common to multiple HEARTH sites will facilitate comparison across similar components of different HEARTH activities. This toolkit packages these tools as customizable modules for different sectors—including biodiversity and climate change, food security, health, governance, and economic growth—so that individual HEARTH activities can apply the modules most relevant to their work. The modules draw on global best practices in each sector to allow robust and consistent measurement of a broad range of outcomes for integrated programs. 1 INRM is a five-year Task Order that provides on-demand support services and technical assistance for USAID Missions, Bureaus, and Independent Offices, managed by the Center for Environment, Energy, and Infrastructure (EEI) in the Bureau for Development, Democracy, and Innovation (DDI). The activity is designed to help USAID Operating Units (OUs) achieve higher impact environment programming and to support the uptake of principles and approaches outlined in the Agency’s ENRM Framework. HEARTH Monitoring and Evaluation Toolkit 2 How can Missions and implementing partners use this toolkit to measure environmental and development outcomes? This toolkit was designed as a “one stop shop” for key indicators across the multiple sectors represented in HEARTH. The toolkit indicators were selected based on an extensive technical review by sector experts at INRM and USAID/Washington, with a focus on robust, credible metrics that are widely used in each sector. Not all indicators will be relevant to each HEARTH activity, nor do they replace adaptive management indicators that measure progress toward intermediate results. Rather, the toolkit indicators should compliment activity-specific context and adaptive management indicators that are rooted in the local context. HEARTHs can select the modules and indicators relevant to their theory of change and include them in data collection tools, which may include household surveys and approaches to biophysical measurement, as relevant. Alignment with the Triple Bottom Line and Sustainable Development Goals The goals of the cross-sectoral HEARTH MERL system align with the “Triple Bottom Line” framework focusing on the “3 Ps” – People, Planet, and Profit/Prosperity. This framework aims to measure the social, environmental, and financial impacts of activities, and is at the core of HEARTH’s model of private sector engagement (PSE). “People” and “Planet” are at the center of the human well-being and biophysical indicators respectively, and align with goals of improving health, education, food security, and ensuring healthy environments, while protecting the planet from degradation and other threats. “Prosperity” includes poverty reduction, conservation enterprise development, and profitability for the private sector partners engaged in HEARTH activities. The Sustainable Development Goals expand this framework to include Peace and Partnerships. “Peace” is integrated across the HEARTH activities, many of which aim to improve governance of natural resources and thereby reduce conflict and displacement. “Partnerships” between the public and private sector are at the base of the HEARTH model, and many HEARTH activities additionally focus on increased participation and inclusion of marginalized groups. The HEARTH MERL indicator modules are organized using this People, Planet, Prosperity, Peace, and Partnerships framework, which highlights the interconnected and cross-sectoral nature of HEARTH’s intended outcomes (see Figure 1 below). HEARTH Monitoring and Evaluation Toolkit 3 Figure 1: HEARTH MERL Framework: People, Planet, Prosperity, Peace, and Partnerships Toolkit Development and Key Outcomes of Interest INRM reviewed situation models, results chains, and technical approach documents for a set of HEARTH activities to identify a common set of outcomes and indicators across the HEARTH portfolio. Based on a preliminary review, a set of outcome sectors were identified that most closely linked to the HEARTH activities’ theories of change in line with the 5-Ps Framework. Outcomes shared among three or more activities were prioritized to help ensure that findings can be aggregated across HEARTH activities. For the full list of key outcomes of interest and the HEARTH portfolio indicators by sector (see Table 1 below). Indicator Guidance and Core Household Questionnaire This document contains guidance for defining and collecting data for each of these indicators for Missions and implementing partners (IPs), including Performance Indicator Reference Sheets throughout. This guidance draws heavily on established best practices, such as the Demographic and Health Surveys HEARTH Monitoring and Evaluation Toolkit 4 and Feed the Future programs for “People” indicators. In addition to this guidance, INRM developed a core questionnaire to provide a basis for household surveys to facilitate ease of take-up. It should be emphasized that it is important for Missions and IPs to adapt the questionnaire to their local country context – which might include adding/removing answer choice options, updating question text or translations, etc. Areas where edits for local context are typically required are identified in the tool and following guidance. While most indicators will be collected through household surveys, some rely on other data collection approaches – in particular, the Site-Level Assessment of Governance and Equity (SAGE) tool for measuring aspects of governance, administrative data from private sector partners on employment or agricultural prices, and biophysical measurements from geospatial and other data sources. This guidance includes discussions on sampling for household data collection as well as other data collection approaches, as relevant. Table 1: Overview of Outcomes and Recommended Indicators by Sector Food Security & Nutrition Link to 5 Ps Outcomes HEARTH Portfolio Indicators Increased dietary diversity ● Percent of women of reproductive age consuming a diet of minimum diversity (MDD￾W) People Improved household food security ● Percent of households experiencing moderate and severe food insecurity, based on the Food Insecurity Experience Scale (FIES) Improved children’s dietary intake ● Percent of children 6-23 months receiving a minimum acceptable diet (MAD) Reduction of potential exposure to zoonotic diseases ● Percent of households consuming high-risk wild meat in the past year Gender Equality & Social Inclusion Link to 5 Ps Outcomes HEARTH Portfolio Indicators Increased women’s role in decision-making ● Percent of women achieving high empowerment on the survey-based women’s empowerment index (SWPER) People Reduced acceptance of gender-based violence Change in women’s time use ● Percent of women spending 11 or more hours per day on non-paid work HEARTH Monitoring and Evaluation Toolkit 5 Link to 5 Ps Outcomes HEARTH Portfolio Indicators Greater awareness of the importance of gender equality and women’s empowerment in men and boys ● Percent of households with gender parity on decision-making ● Percent of men that do not justify violence against women Health Link to 5 Ps Outcomes HEARTH Portfolio Indicators Increased use of family planning services ● Percent of women/men of reproductive age who are using a contraceptive method ● Percent of women given information on contraception methods (Method Information Index) Increased access to maternal health services ● Percent of pregnant women who have attended at least two comprehensive antenatal clinics ● Percent of pregnant women who deliver assisted by a skilled attendant at birth ● Percent of pregnant women who deliver at a health facility People Increased health expenditures to seek treatment for sick family members ● Average amount of health expenditures per sick family member Increased access to healthcare services ● Average number of outpatient consultations per sick family member Increased access to treatment for common childhood illnesses ● Average number of visits for community-based treatment per sick child five and under ● Percent of children 0-35 months of age who received their first dose of measles-containing vaccine by 12-months of age Improved access, reliability, and affordability of safe household drinking water ● Number of people gaining access to basic drinking water services ● Number of people gaining access to safely managed drinking water services ● Number of people receiving improved service quality from an existing basic or safely managed drinking water service ● Percent of children under five with diarrhea in the past two weeks HEARTH Monitoring and Evaluation Toolkit 6 Link to 5 Ps Outcomes HEARTH Portfolio Indicators ● Percent of children under five with diarrhea in the past two weeks treated with ORS Biophysical Link to 5 Ps Outcomes HEARTH Portfolio Indicators Population status of target species maintained or improved ● Change in presence/absence of target species(s) across target area over a set time interval ● Change in abundance of target species(s) across target area over a set time interval Planet Improved or maintained condition of natural ecosystems ● Change in presence/absence of target species(s) across target area over a set time interval* ● Change in abundance of target species(s) across target area over a set time interval* ● Change in total area of one or more classes of vegetation, e.g., forest class, across the defined spatial unit(s) ● Change in Land Capability Classification (LCC) * Target areas selected here should reflect areas that indicate change to ecosystem condition, e.g., areas important for wildlife movement. Reduce loss of habitats for biodiversity ● Change in total area of one or more classes of vegetation, e.g., forest class, across the defined spatial unit(s) Pollution of water sources reduced ● Turbidity of natural aquatic environments (surface, freshwater sources) near agricultural activity sites ● pH of natural aquatic environments (surface, freshwater sources) near agricultural activity sites ● Presence / absence of E. coli bacteria in drinking water sources (freshwater, surface) around agricultural activities sites ● Change in concentration of nitrites and nitrates in natural aquatic environments (surface, HEARTH Monitoring and Evaluation Toolkit 7 Link to 5 Ps Outcomes HEARTH Portfolio Indicators freshwater sources) near agricultural activity sites Reduced greenhouse gas (GhG) emissions from focal enterprise activities ● GhG emissions, estimated in metric tons of CO2 equivalent, reduce sequestered, or avoided in full or in part by USG assistance ● Number of hectares under improved management expected to reduce greenhouse gas emissions as a result of USG assistance Conservation Knowledge, Attitudes, and Practices Link to 5 Ps Outcomes HEARTH Portfolio Indicators Planet Improved knowledge and attitudes towards conservation and natural resource management (NRM) ● Average score measuring the perceived importance of protecting nature and the environment Reduced unsustainable use of resources ● Percent of households who engaged in unsustainable use of ecosystem resources in the past year ● Percent of households that cleared land for cultivation in the past year Agriculture and Land Link to 5 Ps Outcomes HEARTH Portfolio Indicators Increased agricultural productivity ● Average crop production, by targeted crop ● Average crop yield, by targeted crop ● Percent of households participating in farmer groups Prosperity Increased use of sustainable/regenerative practices ● Percent of households using HEARTH promoted technologies/practices ● Number of hectares under improved management practices or technologies Increased land tenure and security ● Percent of households with legally recognized land tenure/rights ● Percent of households with perceived tenure security HEARTH Monitoring and Evaluation Toolkit 8 Conservation Enterprise Benefits Link to 5 Ps Outcomes HEARTH Portfolio Indicators Prosperity Increased benefits from conservation enterprises ● Average household income from environment products ● Percent of households that used or benefitted from any community services provided by the project ● Number of full-time equivalent jobs created Resilience Link to 5 Ps Outcomes HEARTH Portfolio Indicators Increased household resilience ● Average score on the ability to recover from shocks and stresses index (ARSSI) Prosperity Use of natural resources to reduce effects of shocks and stresses ● Average score measuring the extent that households rely on natural resources during times of stress Increased use of renewable and clean energy sources ● Percent of households using renewable fuel sources or grid-connected electricity Socio-Economic Well-being Link to 5 Ps Outcomes HEARTH Portfolio Indicators Prosperity Increased socio-economic well-being ● Percent of households below the comparative threshold for the poorest quintile of the Asset￾Based Comparative Wealth Index (CWI) Increased financial inclusion ● Percent of households participating in micro￾finance, lending programs and/or banking Governance Link to 5 Ps Outcomes HEARTH Portfolio Indicators Peace ● Increased community participation in resource governance ● Increased rights and/or security ● Average score across Site-Level Assessment of Governance and Equity (SAGE) outcome areas HEARTH Monitoring and Evaluation Toolkit 9 Link to 5 Ps Outcomes HEARTH Portfolio Indicators ● Strengthened resolution mechanisms ● Improved monitoring and enforcement Private Sector Engagement Link to 5 Ps Outcomes HEARTH Portfolio Indicators Partnerships Increased private sector engagement ● Number of USG engagements jointly undertaken with the private sector ● Number of private sector enterprises that engaged with the USG ● Investment leveraged from private sector engagements with the USG to support U.S. Foreign Assistance objectives (in United States Dollar (USD)) HEARTH Monitoring and Evaluation Toolkit 10 How To Use This Toolkit This toolkit presents a menu of options for outcomes and recommended indicators across the HEARTH activities. Before using this toolkit, activities should have developed a robust theory of change – through first drafting their situation model and results chains during the co-design workshops, many of which have been completed already, and then validating and refining those results chains during start-up workshops. Based on the activity theory of change, HEARTHs should develop their Activity MERL Plan, which should draw directly from the toolkit. It is not expected that all outcomes or indicators will be relevant for all activities, but that activities should select those in line with their results chains and activity theory of change. Additionally, there might be activity-specific outcomes not included in this toolkit because they were not generally applicable across the HEARTH portfolio, and Missions and IPs should therefore include additional indicators in their MERL plans, as relevant. When developing activity MERL plans, the indicators in this toolkit are intended to be used both to standardize reporting for monitoring data, as well as a basis for evaluation data collection. While monitoring trends in these indicators over time may be important for some activities, USAID anticipates that Missions and IPs will also identify important questions about the causal impact of their activities during the start-up activities, best answered using evaluation approaches. Which indicators will be part of monitoring systems, and which will be used to answer evaluation questions, will affect how the toolkit is operationalized. In addition, it is expected that MERL plans will likely include qualitative data sources, important to further explaining monitoring and evaluation results and exploring learning questions in more depth, in addition to the quantitative data collected using the approaches from the toolkit. HEARTH Monitoring and Evaluation Toolkit 11 Household Sampling and Data Collection Guidance While most indicators will be collected through household surveys, some rely on other data collection approaches. In particular, the SAGE tool for measuring aspects of governance, administrative data from private sector partners on employment or agricultural prices, and biophysical measurements from geospatial and other data sources. For indicators measured using surveys, HEARTHs should consider from whom to collect data. It is typically not possible or necessary to collect data from all targeted beneficiaries, so most surveys utilize a sample. This section discusses sampling for household data collection, and sampling for other indicators that rely on non-household survey approaches will be discussed in their relevant Performance Indicator Reference Sheet (PIRS). Participant-Based Survey and Population-Based Survey Sampling Approaches2 Participant-Based Surveys (PaBSs) are conducted among a sample of the population that participates in a project’s interventions. PaBSs are typically used in the context of project monitoring and are recommended in cases where the target population is expected to be different from the overall population. This is typically the case on USAID projects, including when individuals, farmers, households (or other groups) are targeted for specific characteristics they share or when they self-select or choose to participate. PaBSs can be used for performance or impact evaluations using baseline, interim, and endline surveys, especially when project outcomes and impacts are not expected to accrue at the population-level, but rather be more concentrated among direct participants. One possible challenge with PaBSs is that sometimes activities do not have a confident identification of participants at the time of a baseline, particularly for new activities or those with rolling implementation. An alternative approach is population-based surveys (PBSs), which are conducted among a sample of the entire population living within a project’s area of coverage. Typically, PBSs are recommended when program effects are expected broadly across the full population within a geographic area. Depending on the HEARTH activity, data to measure indicators can be collected either through a project’s routine monitoring systems or through periodic PaBSs. PaBSs are generally recommended over PBSs for HEARTH, largely because HEARTHs are anticipated to have specifically targeted beneficiaries. For more information comparing routine monitoring and PaBSs (when appropriate), sampling frame guidance, and data collection approaches, please see the Participant-Based Survey Sampling Guide for Feed 2 This section is summarized from the Purpose and Background section of the Participant-Based Survey Sampling Guide for Feed the Future Annual Monitoring Indicators: Stukel, Diana Maria. “Participant-Based Survey Sampling Guide for Feed the Future Annual Monitoring Indicators.” Feed The Future. USAID, September 2018. https://pdf.usaid.gov/pdf_docs/PA00TBMK.pdf. HEARTH Monitoring and Evaluation Toolkit 12 the Future Annual Monitoring Indicators3 which is broadly applicable for the HEARTH context. Depending on whether data are collected through monitoring systems or PaBSs, data may be reported through routine reporting systems, or annual (or baseline/interim/endline) survey reports, respectively. Data Collection Administration and Frequency While implementing partners will likely be collecting data for many monitoring indicators on a routine basis and often through their administrative or management systems, data on outcome indicators, such as those covered in this guidance, is typically collected less frequently – often at just a few points during the project and occasionally just at baseline and endline. These indicators tend to change much more slowly and are usually more expensive to collect so more frequent measurement is typically not necessary. In addition, decisions will need to be made regarding whether to collect cross-sectional or panel data. A cross-sectional approach involves surveying different activity participants at each point in time, whereas a panel approach involves surveying the same participants at each point in time. Panel approaches have the benefit of being able to measure whether outcomes for a specific household have changed over time, while cross-sectional surveys reflect general changes for the sample population over time. In general, panel surveys allow for more precise measurement of change, but they are more challenging and costly to implement, as there are additional costs to track the same households over time. HEARTH activities should consider the benefits and drawbacks of each approach, and determine which best fits given their data needs and resource constraints. Data Management All HEARTH activities should include data management plans as part of their Activity MERL Plans, which should cover the entire lifecycle of both primary and secondary data. Primary data are any data that is collected/generated directly by the activity or evaluation team, using methods such as surveys or interviews. Secondary data, which has been collected by someone else but made available for use to others, such as administrative data from private sector partners, satellite data, etc., should also be included in data management plans. Data management plans should be sure to cover both quantitative as well as any qualitative data used for monitoring and/or evaluation purposes. Data management plans should cover data access, privacy, security, and general management across the lifetime of an activity: data collection, transfer, storage, analysis, and dissemination/sharing (including USAID Development Data Library [DDL] submission, as relevant). Privacy, Confidentiality, and Ethics Data privacy governs how data are collected, shared, and used. This is of particular concern when data contains personally identifying information (PII). Teams should start thinking about privacy prior to data 3 ibid HEARTH Monitoring and Evaluation Toolkit 13 collection, and whether it is even necessary to collect PII (names, addresses, Global Positioning System [GPS] coordinates, etc.). Privacy and confidentiality should also be of paramount concern if activities are collecting any information which might result in risk for respondents, including data about illegal activities such as deforestation or poaching. It is important that prior to conducting any data collection, all respondents consent to participate in the survey or interview. This should include a consent script or text, which should outline for respondents the potential risks and benefits to participating, as well as how their information will be used. For example, if activities plan on publishing de-identified data to the DDL, that must be made clear to respondents in the consent script. In addition, data collection protocols may require review by an institutional review board (IRB). IRBs are independent committees established to review and approve research involving human subjects, to ensure that all human subject research be conducted in accordance with all federal, institutional, and ethical guidelines.4 While data collection for project monitoring or performance evaluations typically do not meet the definition of “research” to require an IRB review, it is best practice to err on the side of caution for large scale data collection, particularly whenever potentially sensitive topics are discussed. Moreover, impact evaluations most often do meet the criteria for “research” as they typically seek to develop generalizable knowledge, and therefore typically require IRB review. Additionally, some countries have country-specific ethical review requirements that should be fully explored well in advance of data collection. Some IRB or ethical review committees do not meet regularly, so approvals can sometimes take months, and teams should plan accordingly. The remainder of this document includes sections for each outcome sector, with a high-level summary of pathways through which HEARTH activities might impact each outcome, the recommended indicators, and detailed PIRS for each. 4 This is in line with U.S. federal “Common Rule” regulations governing ethical research and composition/activities of IRBs. Research is defined as a systematic investigation, including research development, testing, and evaluation, designed to develop or contribute to generalizable knowledge. Human subjects are defined as a living individual about whom an investigator conducting research: (i) obtains information or biospecimens through intervention or interaction with the individual, and uses studies, or analyzes the information or biospecimens; or (ii) obtains, uses, studies, analyzes, or generates identifiable private information or identifiable biospecimens. For more details, please see 22 CFR 225 Research falling under Foreign Relations regs: Agency for International Development. Protection of Human Subjects 56. FR 28012ed. Vol. 56. 42 U.S.C. 300v-1(b). gov.info, April 2012. https://www.govinfo.gov/app/details/CFR-2012-title22-vol1/CFR-2012-title22-vol1-part225/summary HEARTH Monitoring and Evaluation Toolkit 14 Toolkit Modules Household Roster & Respondent Identification Household Rosters A household is defined as a person or group of persons that usually live and eat together.5 Household rosters provide a variety of information important for both constructing indicators and disaggregations. In addition to calculating household size (which is important for any per capita indicators) household rosters provide important information on (1) sex, which is integral for sex-disaggregation for many indicators, and (2) age, which the construction of many food security/nutrition and health indicators relies on (e.g., percent of stunted (Height for Age Z Score (HAZ) < -2) children under five (0-59 months); percent of women/men of reproductive age who are using a contraceptive method). Rosters can also include information on education – including literacy, highest-level of education, and enrollment for school-aged household members – which activities might both want to track changes in trends over time, as well as analyze program impacts for households with different education levels. Consideration should be given whether to collect names of household members. Benefits of doing so include facilitating identifying household members for subsequent rounds of data collection (if activities have chosen a panel design, whereby the same household is interviewing at multiple time periods), and in helping with the flow of the survey (e.g., by referencing children by name in later health modules, instead of “Child 1”). The core questionnaire includes household member names for this reason. However, if there are concerns over maintaining confidentiality of respondents, or if subsequent rounds of data collection are not planned for the same households, then collecting names might not be necessary. The core questionnaire, based on the Feed the Future household survey, includes example questions on the following per household member, which should be adapted by activities as relevant based on their monitoring and evaluation needs and local context: ● Relationship to primary household decision-maker ● Sex and age 5 This definition is taken from the FTF Indicator Handbook: Feed the Future. “Feed the Future Indicator Handbook.” US Government's Global Hunger and Food Security Imitative, September 2019. https://fr.fsnnetwork.org/sites/default/files/ftf_agriculture_guide_0.pdf. HEARTH Monitoring and Evaluation Toolkit 15 ● Residence – Usual household member, spent last night at the house, time since spent the night in the house ● Education – Ever attended school, currently attending school, highest level of school attended/completed Respondent Identification Depending on how Missions and IPs choose to implement their surveys, it might be helpful to include a few questions after the household roster to identify which household member is the intended respondent for each subsequent module, to help facilitate data collection by not jumping between different respondents. While for many modules, the intended respondent will be the primary household decision-maker, others are targeted to the primary female household decision-maker (who may or may not be the same person), or other qualifier. For example, if activities include the Food Consumption Score indicator, the intended respondent for this set of questions is the primary adult decision-maker responsible for meal planning and/or food preparation from households, regardless of gender. Each subsequent PIRS includes a section on identifying the intended respondent to help facilitate. HEARTH Monitoring and Evaluation Toolkit 16 Food Security and Nutrition Pathways to Change There are several pathways through which HEARTH activities might impact food security, diets, and nutrition. To achieve impact, HEARTH activities must be intentional in designing strategies and interventions and in measuring outcomes to improve diets and nutrition. For example, nutrient adequacy and caloric availability might be increased due to increased incomes used to buy more healthy food varieties, as well as increased agricultural productivity, used to grow a greater variety of healthy foods. Changes in food sources (i.e., cultivated, bought, or wild caught/gathered) may also impact the varieties of food consumed and overall access to food. It is recommended that both food security and changes in diets are measured as both types of information are necessary to understand impact. While collecting dietary data can be expensive and time-consuming, USAID is supportive of methods such as the Diet Quality Questionnaire that reduce the burdens of data collection.6 Access to food, as well as increases in variety and quality of food consumed are expected to ultimately reduce women and children’s exposure to inadequate diet and poor health/malnutrition in both the short and long term. Ideally, HEARTH activities should measure both food security and diet outcomes. Recommended Outcomes and Indicators Outcome Description Recommended Indicator & Duration Increased dietary diversity The minimum dietary diversity of women (MDD-W) is a validated proxy indicator for the quality of the diet for women of reproductive age (15 - 49 years). Dietary diversity is a key characteristic of a high-quality diet with adequate micronutrient content and is thus important to ensuring the health and nutrition of both women and their children. Research has validated that women of reproductive age that consume foods from five or more of the 10 food groups in the MDD-W indicator are more likely to consume a diet higher in micronutrient adequacy than women consuming foods from fewer than five of these food groups.7 Indicator: Percent of women of reproductive age consuming a diet of minimum diversity (MDD-W) Source: FTF Indicator EG.3.3-10 [IM-level] Percent of female participants of USG nutrition-sensitive agriculture activities 6 Herforth, A., Martínez-Steele, E., Calixto, G., Sattamini, I., Olarte, D., Ballard, T., and Monteiro, C. (2019). Development of a diet quality questionnaire for improved measurement of dietary diversity and other diet quality indicators (P13-018-19). Current Developments in Nutrition, 3(Supplement_1). https://doi.org/10.1093/cdn/nzz036.p13-018-19 7 “Introducing the Minimum Dietary Diversity – Women (MDD-W) Global Dietary Diversity Indicator for Women.” fao.org. Food and Agriculture Organization of the United Nations, July 2014. http://www.fao.org/nutrition/assessment/tools/minimum￾dietary-diversity-women/zh/. HEARTH Monitoring and Evaluation Toolkit 17 Outcome Description Recommended Indicator & Duration Women’s dietary diversity (based on the response of one woman of reproductive age in the household) is recommended rather than household dietary diversity (1) to maintain consistency with FTF data collection, (2) to increase accuracy (by having one woman report on the food she ate, rather than asking a respondent about household members generally, or extrapolating from one individual to the household), and (3) because research indicates that MDD-W is more appropriate than household dietary diversity for measuring nutrient adequacy.8 consuming a diet of minimum diversity; HL 9.1-d [Zone of Influence (ZOI)-level] Percent of women of reproductive age consuming a diet of minimum diversity9 Duration: 5-10 minutes Improved household food security The Food Insecurity Experience Scale (FIES)10 was developed by the Food and Agriculture Organization of the United Nations (FAO) and estimates the probability that a household is either moderately or severely food insecure. It captures lack of access due to money and other resources. Most existing food insecurity indicators focus on potential consequences of food insecurity (e.g., nutrition outcomes), adequacy of diet (food consumption scores, dietary diversity), or physical experience and behavior (e.g., household hunger scale). The food insecurity prevalence based on FIES measures the access dimension of food security based on households’ psychological and behavioral experience with accessing food in the desired quantity, quality, and continuity. The FIES was developed to complement existing food and nutrition indicators; hence, when used in combination with other existing indicators, it will offer a more comprehensive understanding of causes and consequences of food insecurity. The analytic treatment of the data through the Rasch model based on Indicator: Percent of households experiencing moderate and severe food insecurity, based on the FIES Source: FTF Indicator EG-e [ZOI-level] Prevalence of moderate and severe food insecurity in the household based on the FIES11 Duration: 3 minutes 8 Tufts University, Boston, MA. “Household Dietary Diversity Score (HDDS).” Data4Diets: Building Blocks for Diet-related Food Security Analysis | INDDEX Project. International Dietary Data Expansion Project, 2018. https://inddex.nutrition.tufts.edu/data4diets/indicator/household-dietary-diversity-score-hdds. 9 MacCartee, Julie, and Katie West. “Feed the Future Indicator Handbook.” Agrilinks. Feed the Future, March 23, 2018. https://agrilinks.org/post/feed-future-indicator-handbook. 10 Food and Agriculture Organization of the United Nations (FAO). “Food Insecurity Experience Scale (FIES)”. Policy Support and Governance Gateway. 2018. https://www.fao.org/policy-support/tools-and-publications/resources-details/en/c/1236494/. 11 MacCartee, Julie, and Katie West. “Feed the Future Indicator Handbook.” Agrilinks. Feed the Future, March 23, 2018. https://agrilinks.org/post/feed-future-indicator-handbook. HEARTH Monitoring and Evaluation Toolkit 18 Outcome Description Recommended Indicator & Duration sound statistical methods allows for testing the quality of the data with respect to their validity and reliability and ensures that the indicator estimates are comparable across cultural and socio-economic contexts. Although this guidance provides detailed instructions and additional resources to simplify calculation of FIES, because of the statistical methods used, it does require a slightly higher level of experience or training to calculate relative to other food security indicators. Improved children’s dietary intake WHO guiding principles on feeding the breastfed child and the nonbreastfed child recommend that children aged 6–23 months be fed meals at an appropriate frequency and in a sufficient variety to ensure, respectively, that energy and nutrient needs are met.12 This indicator combines information on minimum dietary diversity and minimum meal frequency, with the extra requirement that non￾breastfed children should have received milk at least twice on the previous day. Thus, it provides a useful way to track progress at simultaneously improving the key quality and quantity dimensions of children’s diets. Indicator: Percent of children 6 - 23 months receiving a minimum acceptable diet (MAD) Source: FTF Indicator HL.91-a [ZOI-level] Percent of children 6-23 months receiving a minimum acceptable diet 13 Duration: 5 minutes per child under two years Reduction of potential exposure to zoonotic diseases Given the COVID-19 global zoonotic pandemic, USAID is interested in measuring whether HEARTH programs reduce pressure on, and consumption of, endangered or high-risk wildlife, especially wildlife that could harbor zoonotic diseases. Respondents will be asked if anyone in their household has eaten wild meat from a select list of species over the past year, with follow-up questions on the frequency of consumption and source. The list of animals would be adapted for each country's context, and a subset Indicator: Percent of households consuming high-risk wild meat in the past year week Source: N/A Duration: 1 minute 12 World Health Organization and the United Nations Children’s Fund (UNICEF). “Indicators for assessing infant and young child feeding practices: definitions and measurement methods.” 2021. https://data.unicef.org/resources/indicators-for-assessing￾infant-and-young-child-feeding-practices/. 13 MacCartee, Julie, and Katie West. “Feed the Future Indicator Handbook.” Agrilinks. Feed the Future, March 23, 2018. https://agrilinks.org/post/feed-future-indicator-handbook. HEARTH Monitoring and Evaluation Toolkit 19 Outcome Description Recommended Indicator & Duration of wild animals would be identified as high-risk by the HEARTH activity. HEARTH Monitoring and Evaluation Toolkit 20 Performance Indicator Reference Sheets INDICATOR TITLE: Percent of women of reproductive age consuming a diet of MDD-W APPLICABILITY: This indicator is applicable for HEARTH activities that have explicit consumption, diet quality, or other nutrition-related objectives and/or outcomes. Use of this indicator is encouraged for activities that are inherently nutrition-sensitive (e.g., resulting in improved women's empowerment, control over income, etc.) but that do not necessarily have explicit objectives related to consumption. The MDD-W is a prevalence indicator, which reflects the percent of a population of interest that is above or below a defined threshold (in this case, women who are consuming a diet of minimum diversity). Prevalence indicators are intuitive and understandable to a broad audience of stakeholders, and MDD-W will be useful for reporting and describing progress toward improved nutrition for women. DEFINITION: A woman of reproductive age is defined as a woman 15 - 49 years of age, consistent with FTF and FAO guidance. However, the age range of responding women can be broadened if a HEARTH intervention targets a different age group. This indicator captures the percent of women of reproductive age who are consuming a diet of minimum diversity. A woman of reproductive age is considered to consume a diet of minimum diversity if she consumed at least five of 10 specific food groups during the previous day and night.14 The 10 food groups included in the MDD-W indicator are: 1. Grains, white roots and tubers, and plantains 2. Pulses (beans, peas, and lentils) 3. Nuts and seeds15 (including groundnuts) 4. Dairy 5. Meat, poultry, and fish 6. Eggs 14 For additional detail on collecting and analyzing the minimum dietary diversity indicator, please see the 2021 update to FAO’s Minimum Dietary Diversity for Women guide. (FAO. “Minimum Dietary Diversity for Women: An Updated Guide to Measurement from Collection to Action.” 2021. Rome. https://doi.org/10.4060/cb3434en.) 15 “Seeds” in the botanical sense includes a very broad range of items, including grains and pulses. However, seeds are used here in a culinary sense to refer to a limited number of seeds, excluding grains or pulses, which are typically high in fat content and are consumed as a substantial ingredient in local dishes or eaten as a substantial snack or side dish. Examples include squash/melon/gourd seeds used as a main ingredient in West African stews and sesame seed paste (tahini) in some dishes in Middle Eastern cuisines. HEARTH Monitoring and Evaluation Toolkit 21 INDICATOR TITLE: Percent of women of reproductive age consuming a diet of MDD-W 7. Dark leafy green vegetables 8. Other vitamin A-rich fruits and vegetables 9. Other vegetables 10. Other fruits It is a food group diversity indicator that reflects one key dimension of diet quality – micronutrient adequacy – summarized across 11 micronutrients: vitamin A, thiamine, riboflavin, niacin, vitamin B-6, folate, vitamin B-12, vitamin C, calcium, iron and zinc. Assuming that data for this indicator are collected through a participant-based sample survey, the numerator is the sample-weighted number of women 15 - 49 years of age who consumed 5 out of 10 food groups during the previous day and night. The denominator is the sample-weighted number of women 15 - 49 years of age with food group data. Note: Using the data collected for this indicator, activities may wish to create an additional indicator measuring the average number of food groups consumed by women of reproductive age. This will allow managers to better understand progress made under this indicator and would be especially useful in situations where dietary diversity is very low at baseline. DATA COLLECTION: Data on women’s dietary diversity should be collected by asking the respondent to recall all foods and drinks that she consumed yesterday (during the day and/or night), whether she consumed these items at home or anywhere else. All foods and drinks, snacks, or small meals, should be included as well as main meals. HEARTH recommends an open-recall method, whereby the respondent should be prompted to think about what she ate/drank when she first woke up, later in the morning, mid-day, during the afternoon, in the evening, and before going to bed or during the night. As the respondent recalls foods, the enumerator should select the food groups, as relevant. A list of 23 food groups is provided in the core questionnaire, along with space to write any other foods eaten (to be classified later). If the respondent mentions mixed dishes like a porridge, sauce, or stew, she should be probed for the ingredients. If foods are used in small amounts for seasoning or as a condiment, they should be included under the condiment food group. For any food groups not mentioned, the enumerator should probe and confirm that no food from that food group was consumed yesterday. Data should be collected annually at the same time of year since the indicator will likely display considerable seasonal variability. If possible, data should be collected at the time of year when diversity is likely to be the lowest to best capture improvements in year-round consumption of a diverse diet. However, HEARTH recognizes that data for this indicator may be collected in the postharvest/sale period when data for other indicators, such as crop yields, are collected. In this HEARTH Monitoring and Evaluation Toolkit 22 INDICATOR TITLE: Percent of women of reproductive age consuming a diet of MDD-W case, the indicator value may reflect a best-case scenario in terms of access to a quality and diverse diet by female participants. In addition to the standard MDD-W indicator questions, follow-up questions have been added regarding the food source for food groups likely to be wild caught or gathered; fish, leafy greens, fruit, roots/tubers, or grubs/insects. Note that wild animal meat/organs are already measured separately from domestic animal meat/organs, so no follow-up questions related to food source are required. These questions will help determine the extent to which wild caught or gathered foods contribute to MDD-W. Although the standard MDD-W module is recommended, HEARTHs may consider using the Diet Quality Questionnaire (DQ-Q)16 as an alternative data collection approach when dietary diversity is a less central outcome or there is not enough survey time to complete the full MDD-W module. The DQ-Q is a list-based survey module that can be used to rapidly collect dietary data. The tool includes binary yes/no questions about consumption of 29 unique food groups, including both healthy foods and less healthy foods. The DQ-Q takes about five minutes to administer and does not require implementers to have nutrition expertise or specialized training. Population-level data obtained from the DQ-Q tool can be used to calculate numerous diet quality indicators, including MDD-W.17 ADAPTATION: Ensure country-specific food items are added to the existing food groups. The food groups themselves should not be edited, but the specific items within each food group should be adapted to the local conditions. The FAO MDD-W Guide to Measurement18 has details for adapting the food groups, and HEARTH activities may seek input from a nutrition specialist as needed to properly allocate country-specific food items to their respective food groups. If activities would like to collect data about target foods, these items may be disaggregated and asked about in a new question that is independent from, but adjacent to, the food group it would otherwise belong to. For example: QEx1: Foods made from soy or soy products; QEx2: Foods made from other kinds of beans, peas, or lentils [add any local names]? 16 Herforth et al., 2019. “Development of a Diet Quality Questionnaire for Improved Measurement of Dietary Diversity and Other Diet Quality Indicators.” Current Developments in Nutrition, Volume 3, Issue Supplement_1. 17 Vogliano, Chris. “Measuring Healthy Diets to Advance Nutrition Globally Using the Diet Quality Questionnaire.” AgriLinks. Feed the Future, Nov 02, 2021. https://agrilinks.org/post/measuring-healthy-diets-advance-nutrition-globally-using-diet-quality￾questionnaire. 18 FAO. “Minimum Dietary Diversity for Women: An Updated Guide to Measurement from Collection to Action.” 2021. Rome. https://doi.org/10.4060/cb3434en. HEARTH Monitoring and Evaluation Toolkit 23 INDICATOR TITLE: Percent of women of reproductive age consuming a diet of MDD-W Additional food groups that might be of specific interest to HEARTH activities include insects and small protein foods, and wild foods and neglected and underutilized species (for a full list, please see the FAO MDD-W Guide to Measurement). UNIT: Percent TYPE: Outcome DIRECTION OF CHANGE: Higher is better MEASUREMENT NOTES INTENDED RESPONDENT: Women of reproductive age from sample households. This should ideally be the primary adult female decision-maker in the household (to streamline data collection), but if this person is not of reproductive age, another adult female in the household of reproductive age may be used for reporting. In that case, ideally the respondent would be randomly selected among eligible women in the household. REPORTING NOTES In addition to reporting the percent value, the number of participant households of the nutrition￾sensitive activity must be reported, to allow a weighted average percent to be calculated across HEARTH activities for reporting. Additionally, activities should report on the total sample size (including any disaggregation for participant households vs. comparison/control households if an evaluation is being conducted). HEARTH Monitoring and Evaluation Toolkit 24 INDICATOR TITLE: Percent of households experiencing moderate and severe food insecurity, based on the Food Insecurity Experience Scale (FIES) DEFINITION: The indicator measures the percentage of households that experienced food insecurity at moderate and severe levels during the 12 months prior to data collection. The severity of the experience of food insecurity is defined as a measurable latent trait (a characteristic that is not directly observable, but can be measured indirectly, for example by taking into account behavioral and psychological experiences, in this case around food insecurity). It is measured through the FIES, a measurement scale established by FAO. The indicator is based on an estimation of the probability that each household belongs to a specific category of food insecurity severity (moderate and severe), as determined by the household’s position on the scale.19 The inability to access food results in a series of experiences and conditions that are common across cultures and socio-economic contexts. These experiences range from being concerned about the possibility of obtaining enough food, to the need to compromise on the quality or the diversity of food consumed, to being forced to reduce the intake of food by reducing portion sizes or skipping meals, to the extreme condition of feeling hungry and not having the means (money or other resources) to access food. The new FIES global indicator for measuring food insecurity (access) is calculated from answers to a set of eight questions that covers a range of severity of food insecurity.20 The questions refer to difficulty in accessing food due to lack of money or other resources and reflect the food-related behavior and experiences of the household. The questions are as follows: 1. During the past 12 months, was there a time when you or others in your household were worried you would not have enough food to eat because of a lack of money or other resources? 2. During the past 12 months, was there a time when you or others in your household were unable to eat healthy and nutritious food because of a lack of money or other resources? 3. During the past 12 months, was there a time when you or others in your household ate only a few kinds of foods because of lack of money or other resources? 4. During the past 12 months, was there a time when you or others in your household had to skip a meal because there was not enough money or other resources to get food? 5. During the past 12 months, was there a time when you or others in your household ate less than you thought you should because of a lack of money or other resources? 19 Technical resources, including the datasets and the FIES statistical program, are available at the FAO’s Voices of the Hungry website (http://www.fao.org/in-action/voices-of-the-hungry/fies/en/). An e-learning course that provides guidance on the collection and analysis of data, and on how the information provided by the FIES can be used to inform and guide policy, is also available: “SDG Indicator 2.1.2 - Using the Food Insecurity Experience Scale (FIES).” FAO Elearning Academy. Food and Agriculture Organization of The United States, 2020. https://elearning.fao.org/course/view.php?id=360. 20 For detailed definition and background, refer to FAO’s Voices of the Hungry paper, Methods for Estimating Comparable Prevalence Rates of Food Insecurity Experienced by Adults throughout the World: Rep. Voices of the Hungry Technical Report. Food and Agriculture Organization of The United States, August 2016. https://www.fao.org/3/i4830e/i4830e.pdf. HEARTH Monitoring and Evaluation Toolkit 25 INDICATOR TITLE: Percent of households experiencing moderate and severe food insecurity, based on the Food Insecurity Experience Scale (FIES) 6. During the past 12 months, was there a time when your household did not have food because of a lack of money or other resources? 7. During the past 12 months, was there a time when you or others in your household were hungry but did not eat because there was not enough money or other resources for food? 8. During the past 12 months, was there a time when you or others in your household went without eating for a whole day because of a lack of money or other resources? The response categories for each of the questions include ‘Yes (1),’ ‘No (0),’ and ‘Refused.’ Cases with ‘Refused’ are excluded from the analysis. The prevalence of food insecurity is calculated using the one-parameter logistic model, also known as the Rasch model, which is the simplest formulation for an Item Response Theory-based model.21 The Rasch model assumes that households’ responses to each of the eight binary questions (0/1) are conditionally independent (meaning that the only statistical link between them is the fact that all of them contribute to measure only one and the same food insecurity latent trait), and that each question has the same discrimination power with respect to food insecurity severity. Based on these assumptions, the model uses conditional maximum likelihood procedures to generate estimates of both the questions’ and households’ severity parameters.22 Provided the data are consistent with the Rasch model assumption, the estimated household severity parameters are defined on a continuous, interval-level scale of the severity of food insecurity (latent trait). An interval scale is one where the difference between points on the scale is measurable and consistent. Households are assigned to categories of severity after statistically determining appropriate thresholds that define the categories. Based on the application of the FIES in more than 140 countries in 2014 - 2016, FAO has suggested cross-nationally comparable thresholds that correspond to the severity level of the 5th question “Ate less than should'' (to separate “mild” from “moderate” levels of severity) and of the 8th question “Did not eat for a whole day” (to separate “moderate” from “severe” levels) for global monitoring purposes. Adopting these thresholds (after adjusting the country’s metric to make the country-specific scale’s severity parameters comparable to the global standard scale and thus to other Feed the Future target countries), households with a sample-weighted sum of the probabilities of being between the severity level of the 5th item on the FIES global reference scale (adjusted on the country’s metric) and the 7th item, are assigned to the “moderate” category of food insecurity. Households with a 21 For details about item response theory in the context of food security measurement, refer to Introduction to Item Response Theory Applied to Food Security Measurement: “Introduction to Item Response Theory Applied to Food Security Measurement - Basic Concepts, Parameters and Statistics.” Voices of the Hungry. Food and Agriculture Organization of The United States, 2014. https://www.fao.org/3/i3946e/i3946e.pdf. 22 ibid HEARTH Monitoring and Evaluation Toolkit 26 INDICATOR TITLE: Percent of households experiencing moderate and severe food insecurity, based on the Food Insecurity Experience Scale (FIES) sample-weighted sum of the probabilities of being greater than or equal to the severity level of the 8th item on the FIES global reference scale (adjusted on the country’s metric) are assigned to the “severe” food insecurity category.23 Note: The documentation referenced here provides detailed instructions and templates for calculation of the FIES. However, the calculation does require at least a moderate degree of familiarity with statistical data analysis and at least a basic familiarity with the R statistical programming language (or more advanced understanding of other software, although the guidance and templates are provided in R). ADAPTATION: N/A UNIT: Percent DISAGGREGATE BY: Level of Severity: Moderate, Severe TYPE: Impact DIRECTION OF CHANGE: Lower is better MEASUREMENT NOTES INTENDED RESPONDENT: Primary adult decision-maker responsible for meal planning and/or food preparation from sample households. REPORTING NOTES In addition to reporting the percent value, the number of participant households of the nutrition￾sensitive activity must be reported, to allow a weighted average percent to be calculated across HEARTH activities for reporting. Additionally, activities should report on the total sample size (including any disaggregation for participant households vs. comparison/control households if an evaluation is being conducted). 23 The 5th item refers to the question, “In the past 12 months, did you eat less than you thought you should?”, and the 8th item refers to the question “In the past 12 months, did you go a whole day without eating?” on the global reference scale developed by FAO’s Voices of the Hungry project. Note: The severity threshold for moderate to severe food insecurity has been recently updated from the 4th to the 5th item by FAO. The key resource document from the FAO, titled “The Food Insecurity Experience Scale-Development of a Global Standard for Monitoring Hunger Worldwide”, has not been revised yet. HEARTH Monitoring and Evaluation Toolkit 27 HEARTH Monitoring and Evaluation Toolkit 28 INDICATOR TITLE: Percent of children 6-23 months receiving a minimum acceptable diet (MAD) DEFINITION: This indicator measures the percent of children 6-23 months of age who receive a minimum acceptable diet (MAD). The “minimum acceptable diet” indicator measures both the minimum feeding frequency and minimum dietary diversity, as appropriate for various age groups. If children meet the minimum feeding frequency and minimum dietary diversity for their respective age group and breastfeeding status (i.e., there is an extra requirement that non-breastfed children should have received milk at least twice on the previous day) then they are considered to receive a minimum acceptable diet. Tabulation of the indicator requires that data on breastfeeding, dietary diversity, number of semi￾solid/solid feeds, and number of milk feeds be collected for children 6-23 months the day and night preceding the survey. The indicator is calculated as follows: Numerator: children 6–23 months of age who consumed a minimum acceptable diet during the previous day. The minimum acceptable diet is defined as: • Breastfed children: receiving at least the minimum dietary diversity and minimum meal frequency for their age during the previous day; • Non-breastfed children: receiving at least the minimum dietary diversity and minimum meal frequency for their age during the previous day as well as at least two milk feeds. Denominator: children 6–23 months of age. Minimum dietary diversity for children 6-23 months is defined as five or more food groups out of the following 8 food groups (refer to the WHO IYCF operational guidance document cited below): 1) Breast milk 2) Grains, roots, tubers, and plantains 3) Pulses (beans, peas, lentils), nuts, and seeds 4) Dairy products (milk, infant formula, yogurt, cheese) 5) Flesh foods (meat, fish, poultry, and liver/organ meats) 6) Eggs 7) Vitamin-A rich fruits and vegetables 8) Other fruits and vegetables Minimum meal frequency for breastfed children is defined as two or more feedings of solid, semi-solid, or soft food for children 6-8 months and three or more feedings of solid, semi￾solid or soft food for children 9-23 months. HEARTH Monitoring and Evaluation Toolkit 29 INDICATOR TITLE: Percent of children 6-23 months receiving a minimum acceptable diet (MAD) Minimum meal frequency for non-breastfed children is defined as four or more feedings of solid, semi-solid, soft food, or milk feeds for children 6-23 months. For non-breastfed children to receive a minimum adequate diet, at least one of the four feeds must be a solid, semi-solid, or soft feed. For more detailed guidance on how to collect and tabulate this indicator, refer to the WHO document: Indicators for assessing infant and young child feeding practices.24 DATA COLLECTION: For all children under two, a question is first asked whether the child was breastfed yesterday during the day or at night (including a follow-up question if at first the answer is no, to clarify other ways that babies might be fed breast milk, including by spoon or bottle, or by another woman). Babies are counted as being breastfed if the answer to either question is yes. Additional questions include asking about consuming infant formula, other milk (e.g., tinned, powdered, or fresh animal milk), yogurt, or thin porridge yesterday during the day or at night. Finally, caregivers are asked to recall all other foods and drinks that each child consumed yesterday during the day or night. This should include all foods and drinks, any snacks, or small meals, as well as any main meals. The following table from the WHO guidance25 has the categorization of each type of food into the main food groups used to construct the dietary diversity component of the indicator: Food Group Variables Breast milk Q4: Was [NAME] breastfed yesterday during the day or at night? (Asked separately, not as part of open recall or list-based recall) Grains, roots, and tubers Q7B: Porridge, bread, rice, noodles, pasta or [insert other commonly consumed grains, including foods made from grains like rice dishes, noodle dishes, etc.] 24 World Health Organization and the United Nations Children’s Fund (UNICEF). “Indicators for assessing infant and young child feeding practices: definitions and measurement methods.” 2021. https://data.unicef.org/resources/indicators-for-assessing￾infant-and-young-child-feeding-practices/. 25 ibid HEARTH Monitoring and Evaluation Toolkit 30 INDICATOR TITLE: Percent of children 6-23 months receiving a minimum acceptable diet (MAD) Q7D: Plantains, white potatoes, white yams, manioc, cassava, or [insert other commonly consumed starchy tubers or starchy tuberous roots that are white or pale inside] Legumes and nuts Q7N: Beans, peas, lentils, nuts or [insert commonly consumed foods made from beans, peas, lentils, nuts, or seeds] Dairy products Q6B: Infant formula such as[insert local names of common formula] Q6C: Milk from animals such as fresh, tinned or powdered milk Q6D: Yogurt drinks such as [insert local names of common types of yogurt drinks] Q7A: Yogurt, other than yogurt drinks Q7O: Hard or soft cheese such as [insert commonly consumed types of cheese] Flesh foods Q7I: Liver, kidney, heart or [insert other commonly consumed organ meats] Q7J: Sausages, hot dogs, ham, bacon, salami, canned meat or [insert other commonly consumed processed meats] Q7K: Any other meat such as beef, pork, lamb, goat, chicken, duck or [insert other commonly consumed meat] Q7M: Fresh or dried fish or shellfish Eggs Q7L: Eggs Vitamin A-rich fruits and vegetables Q7C: Pumpkin, carrots, sweet red peppers, squash or sweet potatoes that are yellow or orange inside? [any additions to this list should meet “Criteria for defining foods and liquids as ‘sources’ of vitamin A”] Q7E: Dark green leafy vegetables such as [insert commonly consumed vitamin A-rich dark green leafy vegetables] Q7G: Ripe mangoes, ripe papayas or [insert other commonly consumed vitamin A-rich fruits] HEARTH Monitoring and Evaluation Toolkit 31 INDICATOR TITLE: Percent of children 6-23 months receiving a minimum acceptable diet (MAD) Other fruits and vegetables Q7H: Any other fruits such as [insert commonly consumed fruits] Q7F: Any other vegetables such as [insert commonly consumed vegetables] To calculate meal frequency, first the number of total milk feeds must be calculated (which is the sum of the number of times yesterday that the child consumed any formula, milk, or yogurt). Then, to calculate the total number of feeds (milk + food), the number of soft, solid, or semi-solid food feeds yesterday should be added. For more details on this indicator construction and data collection, please see the WHO guidelines.26 ADAPTATION: Country-specific adaptation may be relevant for the food groups as part of the minimum dietary diversity component. UNIT: Percent DISAGGREGATE BY: Sex: Male, Female Age: 6-8 months; 9-23 months TYPE: Outcome DIRECTION OF CHANGE: Higher is better MEASUREMENT NOTES INTENDED RESPONDENT: Dietary information should be collected from the primary caregiver of each child under two from sample households. 26 ibid HEARTH Monitoring and Evaluation Toolkit 32 INDICATOR TITLE: Percent of children 6-23 months receiving a minimum acceptable diet (MAD) REPORTING NOTES In addition to reporting the percent value, the total number of children under two from participant households and the total number of participant households of the nutrition-sensitive activity must be reported, to allow a weighted average percent to be calculated across HEARTH activities for reporting. Additionally, activities should report on the total sample size (including any disaggregation for participant households vs. comparison/control households if an evaluation is being conducted). HEARTH Monitoring and Evaluation Toolkit 33 INDICATOR TITLE: Percent of households consuming high-risk wild meat in the past year APPLICABILITY: Consumption of wild meat has been an increasing focus in development programming because of its important role in food security and nutrition, but also because of its intersection with both conservation and zoonosis. For these reasons, we recommend this indicator be tracked by all HEARTHs, not just those that specifically target wild meat consumption. DEFINITION: Wild meat is defined by the FAO as terrestrial animal wildlife used for food. This excludes both marine and freshwater animals (although these can be included by HEARTHs if specific marine or freshwater species are identified as important) and livestock. It includes wild meat that is purchased, received as a gift, or directly collected (hunted, trapped, etc.). DATA COLLECTION: Respondents will be asked if anyone in their household has eaten wild meat from a list of species over the past year (Q1). Species included should be those at high-risk for transmission of zoonotic disease, which may include bats, primates, rodents, and/or ungulates as appropriate for the local context. In addition, HEARTH activities may choose to include species that are of particular interest for conservation and/or illegal poaching. Households will be counted in this indicator if they report consuming any of the targeted high-risk wild meats over the last year (consumption of species of conservation importance can be reported separately). For each species that respondents select, follow-up questions will be asked regarding how frequently they consume wild meat from that species (Q2), and from which sources (Q3). Frequency of consumption should include the following answer choices: daily; weekly; every other week; monthly; every other month; every 3-4 months; every 5-6 months; yearly. Sources should be adapted to the local context but should include the following: purchased – cooked/cured; purchased – raw; wild caught; traded goods/services/barter; borrowed; gift. Differentiating between meat purchased already cooked/cured compared to raw will help determine additional exposure related to handling (rather than just consumption). In some contexts, wild meat consumption may be illegal or considered taboo, so respondents may be likely to under-report. If this is a particular concern for a HEARTH activity, and especially if that activity has a focus on wild meat, we recommend considering one of a handful of approaches that seek to generate more accurate estimates of illicit behavior through anonymization techniques. These include randomized response and unmatched count/lists. In these approaches, the response of the individual is masked/hidden, but the prevalence of the population can be estimated. There is a growing literature on these approaches, including on when they are most HEARTH Monitoring and Evaluation Toolkit 34 INDICATOR TITLE: Percent of households consuming high-risk wild meat in the past year useful. A good starting point is Harriet Ibbett, Julia P.G. Jones, and Freya A.V. St John “Asking sensitive questions in conservation using Randomised Response Techniques” (2021).27 ADAPTATION: HEARTHs should identify the list of important species to ask about. While this should include species of importance from either a conservation or a zoonosis perspective, only species with zoonotic importance should be counted in this indicator as high-risk (consumption of species of conservation importance can be reported in a separate indicator). In addition, HEARTHs should adapt the list of wild food sources as appropriate for their local context. Finally, HEARTHs should determine if more detailed questions on wild meat consumption are required. UNIT: Percent DISAGGREGATE BY: N/A TYPE: Impact DIRECTION OF CHANGE: Lower is better MEASUREMENT NOTES INTENDED RESPONDENT: Primary adult decision-maker responsible for meal planning and/or food preparation from sample households. REPORTING NOTES In addition to reporting the percent value, the number of participant households of the HEARTH activity must be reported, to allow a weighted average percent to be calculated across HEARTH activities for reporting. Additionally, activities should report on the total sample size (including any disaggregation for participant households vs. comparison/control households if an evaluation is being conducted). 27 Ibbett, H., Jones, J. P. G., and St John, F. A. V. (2021). Asking sensitive questions in conservation using randomised response techniques. Biological Conservation, 260, 109191. https://doi.org/10.1016/j.biocon.2021.109191 HEARTH Monitoring and Evaluation Toolkit 35 Gender Equality & Social Inclusion (GESI) Pathways to Change Through inclusion as participants in HEARTH activities, it is expected that women will have increased roles in household-level (agriculture, borrowing, productive assets) and community-level decision￾making. Specifically, for conservation enterprise activities targeted for women participants (e.g., traditional crafts, agriculture, etc.), it is expected that women will change the allocation of time between productive/income-generating activities, unpaid household work, and leisure time. Additionally, activities targeted towards changing gender norms may reduce the acceptance of gender￾based violence. As an indirect result of the HEARTH activities increasing women’s role in decision￾making, it is also expected that men may be more aware of the importance of gender equality and women’s empowerment. However, it is important to note that changes in household decision-making or income between spouses can sometimes have adverse effects and may result in increases in gender￾based violence – underscoring the importance of measuring changes in this indicator. In addition to gender equality, HEARTH activities should also have a focus on social inclusion for other marginalized populations (such as youth, LGBTQIA+, Indigenous Peoples, people with disabilities, etc.). HEARTH implementing partners are encouraged to collect disaggregated data and information for the relevant groups to inform their participation in the activities (see USAID’s Inclusive Development Analysis).28 The inclusion of these groups should also be considered in the sampling strategy (e.g., activities may choose to oversample households from these groups to ensure representation) and evaluation design (e.g., specific evaluation questions related to the program impacts for these groups may be included). HEARTH implementing partners and USAID Operating Units are encouraged to reach out to the Bureau of Development, Democracy, and Innovation’s Inclusive Development Hub for support in engaging with these marginalized populations. 28 Cotton, Anthony, Aline Magnoni, Derek Simon, and Brett Tolman. “Suggested Approaches for Integrating Inclusive Development Across the Program Cycle and in Mission Operations.” (2018). https://usaidlearninglab.org/sites/default/files/resource/files/additional_help_for_ads_201_inclusive_development_180726_final_r .pdf. HEARTH Monitoring and Evaluation Toolkit 36 Recommended Gender Outcomes and Indicators General Note: The recommended options imply data collection from both an adult male and adult female, which increases the complexity of the survey, but will be required to cover all of these outcomes. If questions can only be asked of an adult male or adult female, then some of the indicators will not be able to be reported. Outcome Description Recommended Indicator & Duration Women’s increased role in decision-making The Survey-Based Women’s Empowerment Index (SWPER) was developed by analyzing responses to DHS questions among partnered women in 34 African countries (Ewerling et al., 2017),29 and was more recently adapted into a version designed to be applicable in all low- and middle-income countries (LMIC)s (Ewerling et al., 2020).30 SWPER includes 14 questions that represent three dimensions of empowerment: attitudes toward violence, social independence, and decision-making. The questions were chosen because of their strong correlation with gender gaps in health and education, which are hypothesized to be caused or affected by women’s agency. The premise behind the measure is that women’s agency narrows these gender gaps, or when these gaps narrow, women acquire more agency. With the recommended DHS questions, this index can be Indicator: Percent of women achieving high empowerment on the SWPER Source: DHS Women’s Module32 Duration: 8 minutes Reduction in acceptance of gender-based violence33 29 Ewerling, Fernanda, John W Lynch, Cesar G Victora, Anouka van Eerdewijk, Marcelo Tyszler, and Aluisio J Barros. “The SWPER Index for Women's Empowerment in Africa: Development and Validation of an Index Based on Survey Data.” The Lancet Global Health 5, no. 9 (2017). https://doi.org/10.1016/s2214-109x(17)30292-9. 30 Ewerling, Fernanda, Anita Raj, Cesar G Victora, Franciele Hellwig, Carolina VN Coll, and Aluisio JD Barros. “SWPER Global: A Survey-Based Women's Empowerment Index Expanded from Africa to All Low- and Middle-Income Countries.” Journal of Global Health 10, no. 2 (2020). https://doi.org/10.7189/jogh.10.020434. 32 “Demographic and Health Survey Module Woman's Questionnaire.” Demographic and Health Survey. United States Agency for International Development, June 19, 2020. https://www.dhsprogram.com/pubs/pdf/DHSQ8/DHS8_Womans_QRE_EN_19Jun2020_DHSQ8.pdf. 33 It is not recommended to ask about experience of violence, given that this is not expected to be a direct outcome of HEARTH activities, and because asking about experience of violence increases potential for psychological trauma from interviewing. HEARTH Monitoring and Evaluation Toolkit 37 Outcome Description Recommended Indicator & Duration constructed to compare across HEARTHs in addition to comparing responses to individual questions.31 Change in women’s time use Detailed time use surveys (e.g., asking for primary and secondary activities broken down into 15-minute increments) are commonly seen as the most rigorous approach to measuring time use. However, these approaches are time consuming to implement. It is therefore recommended to ask a set of stylized questions about time spent on a limited set of tasks to measure this outcome. For more detailed discussion on measuring time use, please see Annex 1. Alternatives such as A-WEAI34 or the World Bank Living Standards Measurement Survey,35 which include diary and recall approaches, may be considered for activities for which changes in women’s time use are a primary outcome of interest. Indicator: Percent of women spending 11 or more hours per day on non-paid work Source: N/A Duration: 5 minutes Greater awareness of the importance of gender equality and women’s empowerment in men and boys It is recommended to ask the same set of questions for the decision-making and attitudes towards violence dimensions of SWPER to both women and men, allowing for comparison. To construct the indicator for decision-making parity, each respondent (male and female) should be categorized as adequate if they make any decisions Indicator: Percent of households with gender parity on decision-making Source: DHS Men’s Module36 Duration: 4 minutes 31 It should be noted that this DHS question set is similar to A-WEAI Module 6.2: “Feed the Future Zone of Influence Survey Methods - Questionnaire.” Feed the Future, 2020. https://docs.google.com/spreadsheets/d/18drihQ1qe39L1Qj9qXSA0M3Yf7E4MXrR/edit#gid=1928718979. 34 Abbreviated Women’s Empowerment in Agriculture Index (A-WEAI) – Section 6.6a Time Allocation (Hourly Diary): “Feed the Future Zone of Influence Survey Methods - Questionnaire.” Feed the Future, 2020. https://docs.google.com/spreadsheets/d/18drihQ1qe39L1Qj9qXSA0M3Yf7E4MXrR/edit#gid=1928718979. 35 Living Standards Measurement Study (LSMS) example: “Socioeconomic Survey 2018-2019.” Ethiopia - Socioeconomic Survey 2018-2019. World Bank, February 24, 2021. https://microdata.worldbank.org/index.php/catalog/3823/related-materials. 36 “Demographic and Health Surveys Man's Module Questionnaire.” Demographic Health Surveys, May 17, 2020. https://dhsprogram.com/pubs/pdf/DHSQ8/DHS8_Mans_QRE_EN_11Jun2020_DHSQ8.pdf. HEARTH Monitoring and Evaluation Toolkit 38 Outcome Description Recommended Indicator & Duration jointly or alone. Then, household gender parity is scored as one if the adequacy score for the male and female are equal, otherwise zero. To construct the indicator for attitudes towards violence, a score is calculated with an overall negative value indicating that violence is more accepted, and an overall positive value indicating that violence is less accepted. Other options which were considered to measure this outcome tend to be much longer, broader, and less commonly used (e.g., Horizons and Promundo, Gender-Equitable Men (GEM) Scale; Promundo, International Men and Gender Equality Survey (IMAGES)). Indicator: Percent of men that do not justify violence against women Source: DHS Men’s Module37 Duration: 2 minutes 37 “Demographic and Health Surveys Man's Module Questionnaire.” Demographic Health Surveys, May 17, 2020. https://dhsprogram.com/pubs/pdf/DHSQ8/DHS8_Mans_QRE_EN_11Jun2020_DHSQ8.pdf. HEARTH Monitoring and Evaluation Toolkit 39 Performance Indicator Reference Sheets INDICATOR TITLE: Percent of women achieving high empowerment on the SWPER DEFINITION: The SWPER Global is a suitable common measure of women’s empowerment for LMICs, addressing the need for a single consistent survey-based indicator of women's empowerment that allows for tracking of progress over time and across countries at the individual and country levels (Ewerling et al., 2020).38 SWPER includes 14 questions that represent three dimensions of empowerment: attitudes toward violence, social independence, and decision-making. The following table includes the 14 questions used to construct each dimension of the index: Question Answer Choice Code Attitude to violence 1. Beating justified if wife goes out without telling husband Yes = -1 No = 1 Don’t Know = 0 2. Beating justified if wife neglects the children Same as above 3. Beating justified if wife argues with the husband Same as above 4. Beating justified if wife refuses to have sex with the husband Same as above 5. Beating justified if wife burns the food Same as above Social Independence 6. Frequency of reading newspaper or magazine Not at all = 0