THEMATIC EVALUATION OF BENEFICIARY VERIFICATION AND TARGETING APPROACHES IN YEMEN EVALUATION REPORT This publication was produced for review by the United States Agency for International Development. It was prepared by International Business & Technical Consultants, Inc. (IBTCI). December 15, 2020 ABSTRACT Commissioned by USAID BHA, this thematic evaluation examines beneficiary targeting, beneficiary verification, and complaint feedback mechanisms (CFM) among six implementing partners in Yemen with interventions involving cash programming, food vouchers and management of malnutrition. The evaluation uses an explanatory case study approach to answer four evaluation questions: 1) What are the current policies and practices for beneficiary targeting; How compliant are IPs with their stated policies; 2) What are the current policies and practices for beneficiary verification; How compliant are IPs with their stated policies; 3) How are IPs handling complaints about targeting; and 4) What are the most successful and least successful practices for beneficiary targeting, verification and associated complaint mechanisms in Yemen? A two-person evaluation team conducted the evaluation remotely, interviewing 71 IP and USAID staff and experts, using data from four purposefully sampled community survey, and reviewing IP program and internal documents. Partners used a range of operational procedures, guidance adapted from home office, and internal documents to guide activities. Whereas CFM procedures are well documented across partners, procedures for beneficiary verification are poorly documented. Partners generally adhere to the documentation presented to the team. Variation in practices among IPs centers mostly on the use of community committees to nominate and register beneficiaries. While the evaluation team cannot correlate this to reaching the most vulnerable, the team concludes that this had the downstream effect on partners conducting verification of all registered households. In addition to strong and moderate practices among partners, the report provides recommendations for strengthening policies and procedures. THEMATIC EVALUATION OF BENEFICIARY VERIFICATION AND TARGETING APPROACHES IN YEMEN EVALUATION REPORT December 15, 2020 International Business & Technical Consultants, Inc. (IBTCI) USA: 8618 Westwood Center Drive Suite 400 Vienna, VA 22182 USA Jordan 3rd floor, 46 Al-Aiyan Building Princess Taghreed bint Mohammad St. Sweifieh, Amman, Jordan DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of International Business and Technical Consultants, Inc. (IBTCI) and do not necessarily reflect the views of USAID or the United States Government. i TABLE OF CONTENTS Acronyms List ii Executive Summary 1 Findings and Conclusions 14 Introduction – Policy Guidance to Implementation 16 Beneficiary Targeting 16 Beneficiary Verification 25 Community-Based Management of Malnutrition 29 Community Feedback Mechanisms 33 Differences Between the North and the South 38 Strong and Moderate Practices 39 Conclusion 44 Recommendations 45 List of Tables TABLE 1: OFDA IP PROGRAM SUMMARY 9 TABLE 2: FFP IP PROGRAM SUMMARY 9 TABLE 3: SAMPLE BY SECTOR, PARTNER AND REGION 11 TABLE 4: KEY INFORMANT INTERVIEWS 11 TABLE 5: KEY INFORMANT INTERVIEWS 12 Annexes ANNEX 1: Scope of Work 49 ANNEX 2: Evaluation Design 53 ANNEX 3: List of Key Informant Interviews (Redacted) 69 ANNEX 4: Data Collection Tools (Key Informant Interviews) 68 ANNEX 5: Community Survey Tools 72 ANNEX 6: Signed Conflict of Interest Forms 111 ANNEX 7: Bibliography (Redacted) 114 ii ACRONYMS LIST AAP Accountability to Affected Populations AFS Agriculture and food security BHA USAID Bureau for Humanitarian Assistance CaLP Cash Learning Partnership CFM Community feedback mechanisms CFW Cash for Work CHS Core Humanitarian Standard on Quality and Accountability CHV Community health volunteer CMAM Community management of acute malnutrition CTP Cash transfer programs COVID-19 Coronavirus disease 2019 CU5 Children under five DHIS District Health Information Software DHO District Health Office ERMS Economic Recovery and Market Systems FFP Food for Peace FGD Focus Group Discussion FSAC Food Security and Agriculture Cluster FSL Food Security and Livelihoods GHO Governorate Health Office HH Households IASC Inter-Agency Standing Committee IBTCI International Business & Technical Consultants, Inc. IDP Internally Displaced Person IEC Information, Education and Communication IM Information Management IOM International Organization for Migration IP Implementing Partner IPC Integrated Food Security Phase Classification MAM Moderate Acute Malnutrition MEAL Monitoring, Evaluation, Accountability and Learning MEL Monitoring, Evaluation and Learning MOH Ministry of Health MOPIC Ministry of Planning and International Cooperation MPCA Multipurpose Cash Assistance MUAC Mid-Upper Arm Circumference NGO Nongovernmental Organization NFI Non-food items iii OCHA United Nations Office for the Coordination of Humanitarian Affairs OFDA Office of U.S. Foreign Disaster Assistance OTP Outpatient Therapeutic Program PDM Post-Distribution Monitoring PLW Pregnant and Lactating Woman ROYG Republic of Yemen Government RRM Rapid Response Mechanism RUTF Ready-to-use Therapeutic Food SEA Sexual exploitation and abuse S&S Shelter and Settlements SAM Severe Acute Malnutrition SCMCHA Supreme Council for Management and Coordination of Humanitarian Affairs SOP Standard Operating Procedures STC Southern Transitional Council TFC Therapeutic Feeding Center TOR Terms of Reference TSFP Targeted Supplementary Feeding Programs WFH Weight for Height USAID United States Agency for International Development WASH Water, Sanitation and Hygiene WFP World Food Programme YMELP Yemen Monitoring, Evaluation and Learning Project 1 EXECUTIVE SUMMARY The USAID Bureau for Humanitarian Assistance (BHA) commissioned International Business and Technical Consultants, Inc. (IBTCI) to conduct a thematic evaluation of a select group of implementing partners (IPs) in Yemen to assess three main processes associated with selecting individuals and households (HH) to receive humanitarian assistance: beneficiary targeting, beneficiary verification and complaint and feedback mechanisms (CFM). The objectives of this evaluation are: 1) assess IP beneficiary targeting and verification practices; 2) document partners’ policies and practices for handling complaints about beneficiary targeting; and, 3) identify practices partners should consider utilizing when the operating environment in Northern Yemen becomes less restrictive, and practices that should be avoided. The key evaluation questions answered by this evaluation are: 1. What are the current policies and practices for beneficiary targeting?1 How compliant are IPs with their stated policies? 2. What are the current policies and practices for beneficiary verification? How compliant are IPs with their stated policies? 3. How are IPs handling complaints about targeting? 4. What are the most successful and least successful practices for beneficiary targeting, verification, and associated complaint mechanisms in Yemen? The evaluation is a mixed-method with an embedded design; qualitative data are supported by quantitative data. The evaluation centers on six implementing partners who received one to two￾year awards between January 2018 and October 2019. Partners represent a range of strong to moderate practices. Within these projects, the evaluation sampled seven sectors/subsectors. The evaluation team relied on three data collection methods: a review of IP external and internal documents, 71 semi-structured interviews conducted remotely, and four community surveys with 77 respondents. Almost all the data collected for the evaluation are based on perceptions. The team attempted to reduce selection and response bias through the triangulation of other sources of data. However, this was limited due to a) the small sampling of staff interviewed per IP and b) reporting to USAID was insufficient to be conclusive or to corroborate the interview data. A case study approach was used for analysis, with each IP representing one case. The evaluation team mapped out processes for beneficiary targeting, beneficiary verification and complaints mechanisms, compared policies with practices, and explored similarities, differences, and gaps across subsectors within a case. Across cases, the evaluation team compared findings and conclusions to explore patterns and trends to answer the evaluation questions. Community-based management of malnutrition (CMAM) uses a different approach to beneficiary targeting and beneficiary verification and is thus analyzed separately. Analysis stemming from the cross-case comparison and supplemental data from stakeholders was used to identify successful and 1 For the purpose of the evaluation, targeting is the use of a set of criteria to select beneficiaries for assistance. Beneficiary is defined as an individual or household receiving assistance who was selected for assistance using a set of criteria. 2 unsuccessful approaches and inform recommendations. Because the evaluation is thematic and does not focus on individual partner performance, it does not mention IPs by name. The following is a summary of the main findings and conclusions by question, followed by recommendations. CMAM, complementarity between programs, targeting the most vulnerable, and differences between North and South Yemen are also addressed. Q1. What are the current policies and practices for beneficiary targeting? How compliant are IPs with their stated policies? Findings Five of six partners’ policies reviewed by the team vary in their level of detailed guidance. The majority of IPs had documented procedures in program team operational guidelines for cash programming. Partners engage local authorities and community influencers, conduct community￾wide meetings and form committees. Broad-based representation of communities, mitigating the influence of local leaders and engagement of women in committees is less clear among partners. Criteria are discussed with committees and two partners reported communities changing the criteria, adding income and marginalized communities. While the evaluation team does not have disaggregated data for aid recipients, documents indicate that partners largely modeled their criteria on the criteria issued by Food Security and Agriculture Cluster (FSAC). Partners applied different approaches to beneficiary targeting. Most partners’ policies and guidance were ambiguous about applying the criteria, nominating beneficiaries and registering beneficiaries. All partners had committees nominate beneficiaries and two partners also had them conduct registration. In addition to the beneficiary lists developed by the committees, partners also incorporated internally displaced person (IDP) lists and names from other IP interventions. Although partners reported being pressured by the government, none of them reported giving in to these requests. Following nomination, some partners post lists or present the information to communities as a form of endorsement to help identify inclusion and exclusion errors. While committees may have been told to apply more than one criterion, there is no evidence that they consistently did so. Only one partner ranks beneficiaries according to weighted criteria. Conclusion Based on an analysis of documents and interviews, partners largely followed their policies. It is inconclusive whether having clear operational guidance led to better targeting as there was not clear guidance for beneficiary nominations and registrations. The criteria used by partners were modeled on FSAC criteria although some expanded the criteria to include children under five (CU5) and pregnant and lactating women (PLW) suffering from malnutrition and HHs with elderly, disabled or chronically ill members. Partners either had committees nominate beneficiaries as an initial step in beneficiary targeting or had committees conduct nominations and registration at the same time. When committees nominated beneficiaries and partners registered eligible HHs, IPs adhered to their sampling strategies used for verification. Partners shift to 100 percent verification when inaccuracies between registration and verification data 3 exceed thresholds established by the organization. Question 2 provides a more detailed discussion on verification. Q2. What are the current policies and practices for beneficiary verification? How compliant are IPs with their stated policies? Findings Verification procedures are less well documented than beneficiary targeting practices. Monitoring, Evaluation, Accountability and Learning (MEAL) teams typically lead efforts to verify eligible beneficiaries. Partners conducted verification based on three approaches. Where the partners’ staff or external enumerators registered beneficiaries, a random selection of approximately 20 percent was verified to ensure that beneficiaries met the criteria and that beneficiary information was correct. One of these partners confirmed that verification is based on a random sample of HHs. Four partners said that they shifted to 100 percent verification. When committees registered beneficiaries, two partners verified all eligible HHs. The remaining two partners conducted registration and verification simultaneously, meaning that they didn’t conduct verification of the registration data. Of the four partners who said they shifted to 100 percent verification, three provided food vouchers. All partners seek to avoid duplication, although they found this to be challenging. While the various UN clusters coordinate activities on the ground, the information is not always up-to-date and smaller organizations do not always keep the cluster informed of developments. If the World Food Programme (WFP) is operating in the area, partners will compare their lists with those from WFP. They also check with local and international non-governmental organizations (NGOs) with limited success. Inquiries about current and previous assistance are included on verification forms, and four partners also rely on reviews by committees and the community to mitigate duplication. Conclusions Except for describing the percentage of beneficiaries verified and thresholds for errors, none of the partners adequately describe implementation, data protocols, methodologies for sampling and randomization or data entry. Partners followed their guidelines for the sample size used for verification, yet four of six said that they moved to 100 percent verification. IDP lists, a lack of documentation in combination with sequencing of nominations and registrations, staffing, and weak data collection forms most likely contributed to partners exceeding threshold errors and moving to verification of entire registration lists. Community-based Management of Malnutrition USAID identified management of malnutrition as one subsector, implemented by two partners, for the evaluation. Partners presented Yemen CMAM Guidelines (2013) as their overarching policy. For both partners, Community Health Volunteers (CHVs) act as the conduit for conducting community outreach to CU5 and PLW for moderate and severe acute malnutrition (MAM/SAM). Using the standards for Yemen, cases are referred to the nearest Partner D site based on mid-arm circumference measurements (MUAC) and observation of bilateral pitting 4 oedema. CHVs are assigned to communities by either the IP or health facility staff and are responsible for a catchment area of 50 HHs that are visited two to four times per month. Both partners provide training, support, and supervision to CHVs. Health facility staff re-examine all cases referred to Partner D facilities to confirm diagnosis and provide treatment. MAM/SAM cases with complications are admitted to a health facility for close monitoring. Although partners support different types of outpatient facilities and/or programs, the only discernable difference between MAM and SAM cases was the type of treatment prescribed and the follow-up schedule. Health facilities assign CHVs to follow-up on MAM and SAM cases. Complementarity between Programs Complementarity or the link between the Office of U.S. Foreign Disaster Assistance (OFDA) and Food for Peace (FFP) programs was not sufficiently observed, except in the case of one IP which screened HHs for MAM/SAM during registration and referred diagnosed cases to nearby treatment facilities. While IPs may wish to include registered cases of MAM/SAM as part of FFP, programs did not always align with caseloads and timing. OFDA partners cross-pollinated activities, when feasible. For example, MPCA recipients served as a pool for kitchen gardening and HH sanitation. Names of MAM/SAM HHs from a health program constituted 80 percent of beneficiaries for MPCA. All eligible HHs needed to meet the criteria to be included in the assistance. Targeting the Most Vulnerable All partners have written criteria for identifying the most vulnerable households for aid. There is not a great deal of evidence to support the claim that partners targeted the most vulnerable beneficiaries, except in the case of one partner that reported ranking beneficiaries following nomination of the most vulnerable by committees. When asked if they are reaching the most vulnerable, partners said community committees are able to do so by applying the criteria. However, without any kind of ranking prioritizing beneficiaries beyond the caseload, it is impossible to determine if this is the case. Line staff in organizations expressed concerns about ranking since it raised expectations among those who were considered not vulnerable enough, creating conflict and overwhelming complaint and feedback mechanisms. Differences between North and South Policies were universally applied across Yemen for BHA-funded programs. Partners working in the North faced travel delays and government interference and IPs reported that this did significantly impact beneficiary targeting, verification and CFM. IPs improvised by combining registration and verification and relying on field or NGO staff. IPs reported that they did not accept government lists. Although partners in the North were more likely to conduct verification of all beneficiaries, the reasons provided - Rapid Response Mechanism (RRM) lists and errant committees - were the same, regardless of region. Q3. How are IPs handling complaints about targeting? Findings 5 CFM are universal requirements for OFDA and FFP funding, and most partners use the same system for different donors. Of the three processes examined in this evaluation, CFM policies and procedures are the most documented. All the partners have policies and procedures stemming from their overarching Accountability to Affected Populations (AAP) policies and those adapted from global guidance. Complaints are categorized based on severity and follow specific procedures for documenting, managing, and responding to complaints, especially for those most serious. CFM is usually overseen by MEAL teams. Some partners possess more elaborate staffing schemes than others, and one partner has at least one to two dedicated CFM staff in each field office. The most popular feedback channels are hotlines, toll-free numbers, and suggestion boxes. Face-to-face is also a major channel for feedback, although IPs handle complaints about beneficiary targeting and verification differently. Partners were less sure how to reach those who may not be able to access feedback channels because of poor reception or lack of access to a phone. Although several staff said that the post-distribution monitoring (PDM) surveys show that the majority prefer hotlines, one partners’ accountability assessment survey found that the majority preferred using face-to-face or suggestion boxes. Partners are working to expand their reach through field monitors and CHVs, regular travel to the field, holding meetings and distributing information, education, and communication (IEC) materials. Conclusion Partners have solid CFM systems with clear responsibilities, particularly around serious complaints. Partners uses multiple channels. The most centralized channels – phones, suggestion boxes, SMS and WhatsApp are also the ones that are usually recorded as part of CFM. While community members seem to prefer face-to-face meetings with IPs, most field level staff address these types of complaints directly and bypass the CFM system entirely. One partner that had procedures and forms for field level reporting also had ten times the number of complaints as the other IPs. Data from the CFM systems help verify that targeting is going as planned. If only a portion of complaints and feedback are aggregated and analyzed, it is difficult for managers to discern if issues are because of lack of policies, issues around implementation approaches or localized problems. Q4. What are the most successful and least successful practices for beneficiary targeting, verification and associated complaint mechanisms in Yemen? Findings Even though partners did not consistently document their policies and procedures, partners use a number of sound practices for beneficiary targeting. Partners provided good examples of outreach to communities, local stakeholder engagement and work with committees. The evaluation team has less clarity around how partners engaged women, membership selection and committee composition, indicating that overall, they were weaker in these areas. Committees working with four partners signed terms of reference (TORs) outlining roles, responsibilities and in one case, clear guidance surrounding ethics. While IPs discuss criteria with committees, only two reported that they modified their criteria as a result. 6 Partners used three approaches to beneficiary nomination, registration and verification. When committees nominated beneficiaries and partners conducted registration, partners adhered to their sampling policies, requiring additional resources. Partners reporting community endorsement of lists also reported using complaints and feedback to improve lists. Forms used by partners and a lack of documented procedures most likely pushed verification rates above thresholds, thus triggering increased verification efforts. FSAC criteria were not uniformly applied across partners. The evaluation team found that some partners’ staff were not always in agreement on the criteria. A lack of clarity around the term “income” and parameters for timeframes around CU5 and PLWs with MAM/SAM, for example, meant that these were applied differently by staff during registration and verification. Good practices include asset-based criteria and taking MUAC measurements during registration. Four of six partners do not rank vulnerabilities when determining the final beneficiary lists. Except for one partner, the remaining partners’ guidance did not explicitly address caseload quotas for beneficiary nomination or registration by committees. Even those who gave community members assurances that assistance wasn’t guaranteed faced community backlash and strains on CFM systems when caseloads were exceeded. Limiting beneficiary nomination and identification to five percent above caseloads is a good practice. Partners possess similar overarching procedures for CFM and most use multiple channels, in addition to face-to-face meetings and help desks in the field. The degree to which organizational set-up or different channels were successful is challenging to determine as the team does not have complete sets of detailed complaint/feedback data. While partners do ask beneficiaries during post-distribution monitoring about preferred CFM channels, this practice is limited. One partner conducted an accountability assessment which examined people’s preferences for communicating and receiving information. This is a good practice since it provided more nuanced information about face-to-face feedback and provided recommendations, which the partner adapted. Because some partners’ CFM procedures extending to the field level are not robust, complaints and feedback from face-to-face meetings and community committees are not being recorded. The lack of reporting hinders management from understanding the scope and scale of issues arising from targeting. Conclusion In conclusion, there are policies and practices worth adopting for beneficiary targeting, beneficiary verification. To standardize operations, all partners need to improve their SOPs, especially around data collection efforts. Although the evaluation team is unable to conclude which combination of nomination, registration and verification efforts were more successful in targeting beneficiaries, partners who had committees nominate beneficiaries and staff conduct registration were less likely to increase their verification from a sample to a census. The following recommendations are based on the findings and conclusions. Recommendations for IP 7 • Because not all partners have clear operational guidelines for beneficiary targeting, and none have adequate guidance for beneficiary verification, partners should update their policies and procedural documents to fill in gaps and standardize processes across program teams. • Define criteria and create definitions to avoid misinterpretation by staff, community committees and enumerators. • Notify communities of beneficiary nomination lists, whether posting or through announcements, along with extensive CFM outreach to help identify inclusion and exclusion errors early. • Registration and verification should follow the same protocols used for HH surveys, which should improve data collection and data entry and reduce errors stemming from the comparison between registration and verification. Partners should refine and document sampling procedures. This may reduce the prevalence of verifying all registered beneficiaries. • Implement registration and verification separately so that data sets can be compared for reliability and accuracy. • Incorporate MUAC measurements into registration when malnutrition among CU5 and PLWs is a criterion. Besides being verifiable, MUAC measurements add value because women and children suffering from malnutrition are referred to treatment. • Expand the reach of CFM by expanding opportunities for face-to-face interaction and increased field presence through dedicated female sub-office staff, volunteers and/or CHVs for outreach to reduce exclusion errors. Recommendations for USAID • Work with the clusters to provide timely and up-to-date information to minimize duplication of assistance through better geographical targeting, greater coordination, and more accurate lists. • Explore ways to promote data sharing of beneficiary lists among partners. • If the Cash Consortium is a model USAID is interested in pursuing, design and implement an evaluation at the outset to capture key data to help understand whether scaling-up is warranted or feasible. 8 INTRODUCTION International Business & Technical Consultants, Inc. (IBTCI) implements the Yemen Monitoring Evaluation and Learning Project (YMELP) for the United States Agency for International Development (USAID). The purpose of YMELP is to monitor, verify and analyze foreign trends in humanitarian programming in Yemen, funded by the USAID Bureau for Humanitarian Assistance (BHA). The purpose of this evaluation is to improve implementing partners’ delivery of humanitarian assistance in Yemen through better beneficiary targeting, beneficiary verification and complaint and feedback mechanisms (CFM). Comparing the processes, procedures and practices of a select group of USAID partners addresses USAID’s three primary objectives: 1) assess nongovernmental organization (NGO) partner beneficiary targeting and verification practices; 2) document partners policies and practices for handling complaints about beneficiary targeting; and, 3) identify practices partners should consider utilizing when the operating environment in northern Yemen becomes less restrictive, and practices to be avoided. The primary audiences for the evaluation are BHA’s Yemen teams based in Washington, D.C., Amman, and Budapest. Additional audiences for the report are their implementing partners providing humanitarian assistance in Yemen and the offices’ respective monitoring and evaluation teams. PROGRAM OVERVIEW For over a decade, Yemen has been roiled in armed conflict and militant insurgencies. The conflict between the Republic of Yemen Government (ROYG) and the northern-based Ansar Allah movement took center stage in 2015 when Houthi militants took control over the capital Sana’a. With the ROYG receiving support from a Saudi-led coalition and the Houthis backed by Iran, the war has turned into a proxy battle for the region. Further compounding the devastation is the recent split between the ROYG and the Southern Transitional Councils (STC), a separatist group backed by the United Arab Emirates. More than 100,000 Yemenis have died because of the conflict; 23,000 fatalities were reported in 2019 alone2 . Almost 80 percent of the population is food insecure, and almost one-third is at risk of famine.3 The operating environment in Yemen is deteriorating. According to the United Nations Office for the Coordination of Human Affairs (OCHA), two hundred of the country’s 333 districts are classified as hard-to-reach due to the conflict, regulatory restrictions, interference by the Houthi government and increased violence against humanitarian partners.4 In March 2020, USAID suspended humanitarian assistance in Houthi-controlled areas, except for activities mitigating the risk of famine and delivering life-saving service. At the same time, the COVID-19 pandemic has further exacerbated the ability of partners to deliver aid as authorities have announced that international NGOs should work from home, cancelled or rejected permits that involve crossing 2 BBC, “Yemen Crisis: Why is there War?” February 10, 2020, https://www.bbc.com/news/world-middle-east-29319423. Accessed June 15, 2020. 3 Human Rights Watch, “Yemen: Events of 2019”. Retrieved from https://www.hrw.org/world-report/2020/country￾chapters/yemen. Accessed June 15, 2020. 4 OCHA, Yemen Humanitarian Response Plan June – December 2020, Page 10. 9 governorate lines, and cancelled or rejected permits for trainings or workshops involving more than 10 individuals. USAID Humanitarian Assistance In 2019 Office of U.S. Foreign Disaster Assistance (OFDA) and Food For Peace (FFP) committed approximately US$746 million for funding for non-food and food humanitarian assistance in Yemen.5 OFDA funding supports agriculture and food security (AFS), economic recovery and market systems (ERMS), humanitarian coordination and information management, health, logistics support and relief commodities, Multipurpose Cash Assistance (MPCA), nutrition, protection, shelter and settlements (S&S), and Water, Sanitation and Hygiene (WASH). FFP provides funding for cash-for-food, food vouchers, procurement and food distribution, nutrition, and complementary services. USAID identified six implementing partners (IPs) 6 for inclusion in the evaluation, representing a range of strong, fair, and weak beneficiary targeting and beneficiary verification practices. Three of six partners implemented both OFDA and FFP programs. Tables 1 and 2 provide a summary of IP programs, arranged by USAID office. TABLE 1: OFDA IP PROGRAM SUMMARY Partner Award Period Amount No. People Targeted Sectors Partner A 1/1/19 – 11/30/20 US$29.6 M 327,000 AFS, ERMS, Health, Nutrition, WASH Partner B 1/1/18 – 9/30/20 US$5.5 M 79,100 MPCA, Logistic Support, Protection, S&S, WASH Partner C 10/1/18 - 3/31/20 US$11.2 M 81,080 AFS, ERMS, WASH Partner D 3/1/19 - 9/30/20 US$16.7 M 247,688 AFS, ERMS, Health, MPCA, Protection, WASH Partner E 12/1/18 – 6/30/20 US$18.5 M 996,025 Health, Nutrition, Protection, WASH Partner F 8/1/18 – 4/30/20 US$3 M 71,910 AFS, ERMS, Health, MPCA, Nutrition, WASH TABLE 2: FFP IP PROGRAM SUMMARY Partner Award Period Amount No. HHs Targeted Sectors Partner A 12/1/18 - 9/30/20 US$32 M 15,000 HH Food distribution, food vouchers, complementary activities Partner C 10/1/19 - 6/30/20 US$7 M 6,560 HH Food distribution, food vouchers 5 Source: https://www.usaid.gov/sites/default/files/documents/1866/yemen_ce_fs11_09-30-2019.pdf. Of the total funds allocated, BHA allocated US $53.2 million for non-food assistance and $59 million for food assistance to implementing partners. 6 USAID identified seven implementing partners, but dropped one after approval of the evaluation’s Inception Report. 10 Partner E 10/1/18 – 12/31/20 US$39 M 25,487 HH Food distribution, food vouchers EVALUATION QUESTIONS The evaluation seeks to answer four questions. Although the terms of reference (TOR) proposed five questions, the approved inception report reworked these questions by embedding regional differences into each question.7 These revisions have no impact on the evaluation’s purpose or objectives. The top-line questions are: 1. What are the current policies and practices for beneficiary targeting?8 How compliant are IPs with their stated policies? 2. What are the current policies and practices for beneficiary verification? How compliant are IPs with their stated policies? 3. How are IPs handling complaints about targeting? 4. What are the most successful and least successful practices for beneficiary targeting, verification, and associated complaint mechanisms in Yemen? EVALUATION APPROACH, METHODOLOGY AND LIMITATIONS This is a thematic evaluation9 of partner beneficiary targeting and verification approaches in Yemen. It is holistic and not an evaluation of individual partners or a risk assessment. The evaluation design is an explanatory cross-case study, with each IP serving as a case. The evaluation team used a mixed method approach to the evaluation, relying on document review, key informant interviews and community surveys. Analysis of qualitative data is supported by quantitative data. The evaluation design matrix linking the evaluation questions to data collection and analysis methods is in Annex 2. The evaluation was conducted over six months by a team of two expatriates; one based in Washington D.C. and the other in Kampala, Uganda. The team leader is seasoned monitoring and evaluation specialist with over 15 years of USAID experience, including two years as a technical director for a third-party monitoring project funded by OFDA. The subject matter expert is a monitoring and evaluation specialist with twelve years of experience mostly with humanitarian organizations, including two years in Yemen. The evaluation team was supported by a Yemeni field researcher. Following USAID’s approval of the inception report and questionnaires on July 23, 2020 and a virtual evaluation planning workshop held between August 17 and August 21, 2020, the team collected most qualitative data over a five-week period, followed by survey implementation, analysis and report writing over the remaining six weeks. 7 Annex 2 contains an explanation of the evaluation questions from the inception report. 8 For the purpose of the evaluation, targeting is the use of a set of criteria to select beneficiaries for assistance. Beneficiary is defined as an individual or household receiving assistance who was selected for assistance using a set of criteria. 9 According to the Department of State and USAID Glossary of Terms (2009), a thematic evaluation is an evaluation that focuses on a specific cross-cutting area. 11 Evaluation Sample The sample for the evaluation is purposeful and designed to create variation among sectors while allowing for patterns to emerge across subsector activities and operating environments. The evaluation focuses on specific sectors and subsectors of interest to USAID: AFS, cash for work (CFW), moderate acute malnutrition (MAM) 10, food vouchers, MPCA and WASH. Without any awareness surrounding USAID’s perception of the partners’ approaches to beneficiary targeting and beneficiary verification, the evaluation team identified eight sectors/subsectors across the six partners that represented a balance between interventions implemented in North and South Yemen. Table 3 presents the sample used for qualitative data collection by sector, IP and region. TABLE 3: SAMPLE BY SECTOR, PARTNER AND REGION Sector Subsector No. Partners North South Nutrition MAM 2 Food vouchers n/a 2 MPCA n/a 2 ERMS Temporary Employment 2 AFS Improving Agriculture Production 1 AFS Livestock 1 WASH Sanitation 1 Data Collection Methods and Sources The team relied on three types of data for the evaluation: documents, key informant interviews and survey responses of beneficiaries and stakeholders. All data collection was done remotely. The evaluation team reviewed a large number of program documents (award documents and IP deliverables) provided by USAID, internal policies, standard operating procedures (SOPs), guidelines and forms used by the IP to conduct beneficiary targeting, beneficiary verification and implementation CFM (Annex 7). The second source of information is key informant interviews with staff from IPs, USAID and cluster leads (Annex 3 and 4). The evaluation team interviewed a range of country office and field staff identified by IP senior management to discuss policies, practices and any obstacles faced during implementation. In addition to senior management, the team met with people from the program and MEAL teams. The team also held discussions with USAID staff and representatives of relevant UN clusters and experts on good practices and obstacles faced in Yemen. Seventy-one semi-structured interviews were conducted (Annex 5). Table 4 lists the number of interviews by IP or category. TABLE 4: KEY INFORMANT INTERVIEWS 10 Based on an earlier request by USAID, the evaluation team will use term community-based management of acute malnutrition (CMAM) instead of MAM. 12 IP / Category No. of Interviews Partner A 12 Partner B 8 Partner C 11 Partner D 9 Partner E 15 Partner F 7 USAID 4 Experts 5 TOTAL 71 The evaluation team recorded the majority of interviews with the permission of IP staff and reviewed the recordings or transcriptions against the interviewer’s notes. The evaluation team also commissioned surveys with community committee members, community leaders, community health volunteers (CHVs), and health facility staff.11 Apex, an independent survey firm based in Sana’a, conducted four structured telephone surveys with non￾representative samples of beneficiaries and stakeholders provided by the IPs. Details about the number of respondents is presented in Table 5. TABLE 5: KEY INFORMANT INTERVIEWS Survey No. Respondents12 Community Volunteer Committee Members 36 Community Leaders 18 Community Health Volunteers 14 Health Facility Staff 11 TOTAL 77 Data Analysis Techniques To understand approaches used by each IP and across IPs, a case study approach was used to produce two sets of analyses to answer the main evaluation questions. For each partner or case, the evaluation mapped out processes for beneficiary targeting, beneficiary verification and complaints mechanisms, compared policies with practices, and explored similarities, differences, and gaps. Across cases, the evaluation team compared findings and conclusions from each case study to explore patterns and trends to answer the evaluation questions. Because CMAM uses an entirely different approach to beneficiary targeting and beneficiary verification, it is treated separately. Analysis stemming from the cross-case comparison and supplemental data from 11 In addition to the key informant instruments, Annex 5 includes the survey questionnaires. 12 Five partners provided overall contact information for 153 people. Once duplicate numbers were removed, there was a pool of 96 potential respondents and 23 replacements. Seventy-seven interviews were conducted, and 24 people had numbers that were wrong or no longer in service for overall response rate of 81.1 percent. 13 stakeholders was used to identify successful and unsuccessful approaches and inform recommendations for future programs. During the inception phase, the four evaluation questions were linked to key sub-questions, including themes of keen interest to USAID, and both were linked to the data collection instruments. The evaluation team coded and reviewed program documents to create the questionnaires. As data collection progressed, the team developed a more elaborate set of inductive codes for each process. The evaluation team updated codes as themes emerged from new data and analysis. Throughout the evaluation, the evaluation team met frequently to discuss emerging themes and patterns and update the coding structure as needed. The evaluation team’s second round of analysis coded qualitative data supported by the survey data across implementing partners. Based on convergent findings from the analysis, the team extrapolated conclusions. Recommendations are based on conclusions. Because the evaluation is thematic and not evaluations of partners’ performance against its award, none of the partners are identified by name. Because the purpose of the evaluation is to elucidate and improve upon beneficiary targeting, beneficiary verification and CFM, anonymity should increase utilization by shifting the focus from individual partners to overall policies and processes. Limitations and Mitigation Strategies Most data collected for the evaluation are based on the perception of the respondents and IP reports to USAID. Perceptions can differ from one person to another and can be difficult to interpret without an understanding of the underlying context for the response. Selection bias is evident. The evaluation team interviewed an average of ten key informants per partner. IP senior management or points of contact provided names and contact information for all respondents. Although the evaluation team attempted to interview as many staff as possible, without an in-country presence, organizational charts or staff lists, the evaluation team had less control over identifying key informants. Many staff who worked in the North had left their organizations by the start of data collection, which meant that the evaluation team had to rely on field staff in the South for their perceptions of the North. In addition, senior MEAL staff of three partners were new and had not yet travelled to Yemen. Response bias is a limitation. Because IPs’ grants had ended or were scheduled to close this year, recall may have influenced responses. By the time the team was able to line up interviews, five months had lapsed since the cessation of activities in the North. Language differences may have influenced the interpretation of questions and responses of both respondents and interviewers. Lastly, respondents may have been reluctant to be candid because they did not want to jeopardize future reinstatement of funding in the North. Another limitation is that the team had an incomplete set of partner program documents and other correspondence to USAID, leading to potential gaps in the team’s understanding of how projects implemented beneficiary targeting, verification, and complaint mechanisms over time. The team’s ability to distinguish and articulate best practices was also constrained as there is little detailed information on the composition of community committees, percentages of eligible 14 beneficiaries disqualified as a result of registration or verification, breakdowns of targeted beneficiaries by vulnerabilities, and disaggregated complaint data. To the extent possible, the evaluation team mitigated these limitations by triangulating data among the key informants and with external and internal documents. Interviews were recorded and reviewed to pick up details that the interviewer may have missed during the interview. In cases where responses were not clear, the interviewer scheduled another interview or followed up using email. Terminology The report uses standardized terminology. All cash and food voucher programming falls under an umbrella office called food security and livelihoods (FSL), even if food vouchers and OFDA programming sit in two different offices. MEAL (monitoring, evaluation, accountability, and learning) is used instead of monitoring and evaluation. All interventions related to cash programming (i.e., MPCA and CFW) are collectively known as cash transfer programs (CTP) and this includes food vouchers unless noted in the report. All committees, whether community volunteer, relief or community development are referred to as community committees. Beneficiary nomination is the initial list of eligible beneficiaries produced by community committees. Registration is the first collection of biodata and vulnerability information, whether collected by a committee or an IP. Checking the accuracy of registration data is verification. CFM encompasses all feedback mechanisms, including beneficiary feedback mechanism, client responsibility and accountability and complaints recording mechanism, and the avenues for beneficiaries to make complaints or provide feedback are called channels. Common terminology facilitated comparisons among partners during analysis. FINDINGS AND CONCLUSIONS USAID’s humanitarian programming aims to reach the most vulnerable. Community-based targeting, in conjunction with geographic targeting, are processes used by partners to identify those most in need. Community-based targeting relies on community committees to identify households (HHs) by applying a set of vulnerability criteria. Once these beneficiaries are identified, verification validates the beneficiary lists generated by the communities. Key steps for selecting beneficiaries are: 15 The chart is an amalgamate of processes used by partners and provides a reference for understanding and comparing partners’ policies and practices. Variations among partners centers on the nomination and the registration of beneficiaries, which affected verification efforts. The Findings and Conclusion section discusses this in more detail. CFM are the systems employed by implementing partners to obtain feedback from the populations they serve. Complaints and feedback are another form of validation of the beneficiary targeting process. They are important sources of information that help partners identify weaknesses in the system, remove HHs that do not meet the criteria, and provide assistance to those who are eligible. The following section presents overall findings for policies and processes for beneficiary targeting, beneficiary verification and CFM across IPs. Following each section, a conclusion sub-section will focus on whether partners’ practices were in alignment with their policies. CMAM is addressed separately since targeting and verification differ from the other sectors examined in this study. Five partners, as mentioned in the program description, worked in both North and South Yemen. Because programs in the North closed or shifted to the South following the suspension of United States Government assistance, findings may amalgamate experiences. Any differentiations in beneficiary targeting and verification that the team that was able to elicit from IPs is discussed in the section Differences Between North and South Yemen. Intervention Area Selected Meeting with Local Officials Meeting with Local Influencers Data Collection: Nomination Community Committees Community Mobilization Data Cleaning / Prioritization Data Collection: Registration Data Cleaning Data Collection: Verification Validation of Registration Lists Final Lists of Beneficiaries Lists BENEFICIARY SELECTION PROCESS: TARGETING AND VERIFICATION 16 Introduction – Policy Guidance to Implementation All the partners evaluated are guided by their Accountability to Affected Populations (AAP) policy,13 which is based on the four commitments by the Inter-Agency Standing Committee (2017) and the Core Humanitarian Standard on Quality and Accountability (CHS). OFDA’s Application Guidelines (2019) requires partners to describe how they will implement AAP throughout the program cycle. OFDA also requires partners to submit an AAP/framework. Application Guidelines issued by USAID’s newly created Bureau of Humanitarian Assistance requires partners to submit their AAP plan/framework as part of their proposal. Both documents reference Inter-Agency Standing Committee’s (IASC) five commitments associated with. CHS standards and guidance interpret the AAP policy and are designed to “make it simpler and easier for aid workers to implement standards.”14 AAP is a people-centered framework for humanitarian actors and designed to enhance accountability and improve program quality. Two CHS commitments are important since the accompanying guidance is often incorporated into partners’ policies. Commitment four is “communities and people affected by crisis know their rights and entitlements, have access to information and participate in decisions that affect them.” Commitment 5 is “communities and people affected by crisis have access to safe and responsive mechanisms to handle complaints.” Beneficiary Targeting Targeting is a process for reaching those most in need of assistance while maximizing the use and impact of limited resources. The Cash Learning Partnership (CaLP) defines four approaches to identifying beneficiaries, whether individuals or HHs, for assistance: administrative, geographic, self, and community-based targeting. All of USAID’s partners conduct geographic targeting by working in discreet locations based on assessments conducted by the partners and other agencies, the Integrated Food Security Phase Classification (IPC) ratings for districts in Yemen, recommendations by the humanitarian cluster for each sector, and agreement with the Ministry of Planning and International Cooperation (MOPIC) in the South, the Supreme Council for Management and Coordination of Humanitarian Affairs (SCMCHA) in the North, and authorities at the governorate and district level. The Food Security and Agriculture Cluster (FSAC) is the overarching coordination mechanism for all organizations working in the sectors covered by the cluster, including CFW, food vouchers, MPCA, agricultural production and livestock. FSAC also sets the recommended criteria for community-based targeting. Q1. What are current policies and practices for beneficiary targeting? How compliant are IPs with their stated policies? All the IPs in this study use community-based targeting to target beneficiaries15 for assistance. According to CaLP (2015), communities are better adept at identifying those most in need, resulting in community ownership of results, community empowerment and reduced costs to organizations. Targeting is a process that leads to a final set of HHs slated to receive assistance. Key steps for targeting addressed in this section are: engaging communities, working with 13 USAID’s newly created Bureau of Humanitarian Assistance requires partners to submit their AAP plan/framework as part of their proposals for fiscal year (FY) 2021 and 2022. 14 Source: https://corehumanitarianstandard.org/the-standard/history, accessed October 15, 2020. 15 Beneficiaries and households are used interchangeably throughout the report. 17 community committees, identifying beneficiaries based on a set list of criteria, collecting detailed biodata and vulnerability information, and narrowing down the list of eligible beneficiaries. Whereas the first two steps and the criteria are straightforward, the last three steps – nomination, registration, and selection – are intertwined by some partners in both their policies and practices. Even though this delineation of beneficiary targeting may seem artificial to some partners, it allows for comparisons across cases. Staffing Program teams are responsible for beneficiary targeting, beginning with identification of sub￾districts and villages to finalizing the list of beneficiaries. Depending on the size of the population targeted for assistance and the size of the program team, the MEAL team (including CFM staff) and volunteers, mobilizers or enumerators may provide support during beneficiary nomination and registration. For two partners, the MEAL teams and the program teams shared responsibilities as registering beneficiaries is done simultaneously with verification. MEAL and information management (IM) staff may also assist with data entry and data cleaning following beneficiary nomination and registration. Policies Five out of six partners provided overarching CTP SOPs for FSL programming, which encompasses food vouchers. All six partners also provided some documentation or supplemental guidance for at least one or more processes involved in beneficiary targeting. Policy documents range from a two-page overview to very detailed procedures across multiple documents. None of the partners have a complete set of detailed operational guidance from the point where villages targeted for assistance are agreed upon with local authorities, to finalizing lists of eligible beneficiaries. Targeting criteria are either identified in the SOPs or provided in a separate document for all subsectors, except WASH. Community Notification and Initial Engagement IPs engage with local stakeholders to facilitate the selection of beneficiaries. Four of six partners’ policy documents address this engagement. During these meetings with local authorities at the sub-district level, IPs explain their objectives, the activities, the criteria, and the total caseload. The purpose of these meetings is to confirm support for interventions, ensure access and security, and receive explicit or implicit agreement for their work. In the case of CFW, the IP reaches agreement on the infrastructure priorities. Similar meetings are held with key stakeholders such as sheikhs, tribal elders, and other village leaders such as imams, school principals and health facility heads to mobilize participation in community-wide meetings. All partners hold these meetings, although some respondents provide more detail than others. IPs also advertise the meetings by putting up posters in public areas or using loudspeakers, and one uses community mobilizers to facilitate attendance. Staff from three IPs mentioned holding meetings in neutral spaces such as schools and health centers. Community meetings are addressed in more detail in four of five partners’ documents provided to the evaluation team. During the community meetings, IPs provide an overview of the organization, the project’s or intervention’s objectives, activities, and selection criteria. Staff from one IP reported that they also shared funding levels, caseloads, and timelines. In addition to 18 providing an overview, partners also present targeting criteria and form community committees. Almost all community leaders (17 of 18) said that the community was informed of who would receive assistance. In addition to program staff, three partners’ MEAL and/or CFM staff attended these meetings to discuss CFM, channels, and feedback processes. One partner handed out flyers describing proposed interventions and CFM. This partner also mentioned posting the targeting criteria in a public place. Mobilizing the community to attend these meetings is important since it is during these meetings that community committees are formed. Committees represent villagers and are used to help identify beneficiaries. Even though three IPs had minimum attendance requirements, interviews did not reveal the extent to which they met these requirements. Women are encouraged to participate, and one partner stresses in its SOPs that staff engage with local stakeholders to facilitate their participation. Despite efforts, field staff lamented that it was difficult for women in conservative areas to participate in these meetings and that men served as their proxies. To encourage women’s participation, three partners held separate meetings for women, run by female staff and mobilizers. Community Committees Committees are an important part of community-based targeting. Four partners have SOPs and two of these are very specific about the formation of community committees16 and their role. These two partners also have very clear guidance on committee member qualifications, the number of members, and committee composition. None of the SOPs address how community committees are organized and all are vague about how members are nominated and selected. Procedures for validating the elections are included in two of the three partners’ SOPs. Two of six partners address committee training in their SOPs. A committee may represent one village, a cluster of related villages or a section of town. The evaluation team is unable to discern the total number of committees and areas covered from partner reports. Prior to forming committees, staff from two partners said that they present qualifications for members, such as being honest, trustworthy, respected and literate, prior to nominations. Candidates, according to IPs, are nominated during the community meeting. However, staff and documents also indicate that in some cases sheikhs and village leaders provide names beforehand. Four IPs said that committees are elected17 by a show of hands although this was not clear as some respondents also referred to the process as “selection.” Committee members surveyed18 said that 69 percent report that they were elected; 12 percent were appointed by local authorities or sheikhs; and 11 percent were appointed by IPs. In areas where prior committees had once existed, those present at the meetings affirmed existing members. When partners lack confidence in a committee, it is dissolved, and a new committee is formed or a parallel committee (referred to as “branch” by one IP) created. Two partners had results of the election signed by the local sheikh and/or district officials. 16 During the community meetings, new committees are formed through a selection process or existing committees are validated. 17 In cases where multiple interventions were implemented by an IP the same committee would be responsible for targeting beneficiaries for both interventions. 18 N=35 for all community survey responses. 19 Committees typically consist of eight to ten members, according to the IPs, although the survey results indicate that they tended to be smaller, with 69 percent having seven or less members. Interviews and information provided by IPs suggest local leaders such as sheikhs and village elders often served on these committees and were assigned the role of chairman. Whereas this most likely facilitated identification of beneficiaries, access and security, local leaders may also serve as sub￾district or district officials because they are educated. All IPs emphasize women’s participation in the committee. Policies referring to women’s participation vary. Two IPs have minimum requirements for women to encourage participation. A third partner’s guidance recommends forming two committees if one is unattainable. None of the SOPs provide direction about what to do in cases when women do not attend these meetings or women are not chosen to serve on the committee. Respondents spoke of difficulties engaging women in conservative areas, especially in Abyan and Hajjah governorates. Cultural norms in these areas prohibit men and women from mixing together. Partners also require committee members to be literate, which is another obstacle for women’s participation. Most partners in their reports to USAID did not break down the gender composition clearly enough to ascertain the extent of women’s participation. At least two partners formed separate committees for women, according to respondents. Even when IPs worked with female committees, men often served in leadership roles. Once the committees are finalized, all partners said that they provide an orientation on roles and responsibilities, the criteria, and its application for targeting, and completing the beneficiary nomination and/or registration forms. Three partners train on protection principles and at least two partners discuss CFM with committees because they are viewed as an integral conduit for receiving and addressing complaints. This was confirmed by 80 percent (28) of committee members surveyed. Four partners had committee members sign codes of conduct or Terms of Reference (TOR). 19 All IPs provided the committees with the criteria in Arabic and the majority of nomination and registration forms received from partners were in Arabic or Arabic and English. Criteria All partners presented criteria in their proposals for the selection of beneficiaries in the sectors examined. Criteria for food vouchers, CFW and MPCA were variations of FSAC criteria20 19 The TOR provided by one of the IPs is comprehensive, addressing both the responsibilities of the relief committees and the IP. It also highlights the voluntary nature of the position and emphasizes money or favors will not be exchanged with community members, their families or IP staff. 20 FSAC criteria is presented as first-line and second-line response for interventions involving distribution of food, cash, vouchers, CFW, and different forms of livelihoods. First-line is associated with IPC 4 and 5 and second-line with IPC 3. Most criteria do not change from first-line to second-line, although there are added requirements for activities other than food, cash and voucher distributions. FSAC CRITERIA (2020) • HHs with PLW / CU5 admitted into SAM or MAM programs • HHs headed by women/widows who live independently with their children • HHs headed by children • HHs headed by an elderly person • HHs headed by a chronically ill or physically challenged person • IDPs • Vulnerable families hosting IDPs • HHs from socially or economically marginalized communities 20 OFDA interventions for AFS and WASH usually listed additional criteria or requirements that aligned with the intervention (e.g., land or structural requirements or prior experience). Partners proposals and program documents suggested that targeting criteria would be prioritized by the communities. While criteria were reviewed or confirmed with the community committees, only two partners said that they adjusted their criteria as a result of these discussions. One partner said that the criteria are not tailored to individual communities. All committee members surveyed said that they used criteria for selecting beneficiaries and 29 percent (10 of 25) said that they were involved in developing or changing the criteria. The evaluation team compared targeting criteria from the proposal, and available policy documents, registration forms, verification forms and program reports. These did not always align. Partners were not always consistent with FSAC criteria. Not all partners registered cases or follow-up cards as criteria for MAM/SAM among CU5 and PLWs. Two did not require cards. All partners targeted female-headed HHs, child-headed HHs, IDPs, vulnerable families hosting IDPs and HHs from marginalized communities. Four partners either identified little or no income as a criterion and one of them also included the lack of assets as part of the criteria. Four partners were more expansive since they included elderly, chronically ill, disabled members opposed to requiring them to be the heads of HHs, two included PLWs and CU5. Partners implementing MPCA and CFW applied slightly more inclusive criteria by including HHs with elderly and disabled members. (One of these partners recently updated their criteria to align with FSAC.) Both partners’ CFW criteria also required HHs to have at least one able￾bodied family member over the age of 18. Criteria for AFS programs were similar to their FSL programs but also included requirements such as land and water and the ability and/or experience to undertake the livelihood activity. HHs selected for latrines had to have either a water source or the benefit of a water intervention and be located in a designated cholera area. Nominating Beneficiaries Committees are charged with the process of applying criteria to identify eligible HHs. Partners’ guidance included varying amounts of detail about the beneficiary nomination process, and the majority were ambiguous. Summaries of the IPs’ guidance are: ● Detailed selection criteria are reviewed based on the project objective in discussion with the local authorities and representatives of the targeted community. Based on the agreed beneficiary selection criteria (with a focus on HHs with malnourished children and PLWs), committees submit their beneficiary nomination list, plus an additional 5 percent for a reserve list of possible replacements. Committees use standardized forms and registration questionnaires. ● The nomination process is conducted through a community committee. A nomination form based on the intervention’s criteria is distributed to all committees. The process is expected to take two to three weeks. During beneficiary nomination, the IP’s “community facilities and M&E assistance closely follow the beneficiary process.” ● FSL staff liaise with committees to make beneficiary lists based on the predetermined vulnerability criteria. When referral lists are provided by the nutrition team and other sectors integrated with FSL they are shared, and it is explained to the communities why they were selected. 21 ● Referrals come from different sources, including community committees, NGOs, other departments in the organization, the UN, relevant clusters, and stakeholders on the ground. ● Committees are given a list that they can fill out with the names of families nominated by them that match the criteria for selecting / targeting families. These lists are completed by the community and also approved by the director general of the district concerned with targeting21 . ● The FSL coordinator is responsible for forming community committees to register potential beneficiaries based on the selection criteria and for developing an interim beneficiary list from the community committee register and sharing this list with the MEAL department for verification. Community committees are responsible for identifying eligible HHs by applying vulnerability criteria. Only one partner includes in its SOPs how to approach the number of caseloads: five percent more than the total number nominated/registered. Committees, according to IPs, go house-to-house to nominate beneficiaries and are supervised by program staff. All community members surveyed said that they visit HHs, however they also stated that they receive referrals from other programs, community meetings and recommendations from local authorities. Whereas four partners have committees nominate beneficiaries by gathering basic HH data and vulnerabilities, two IPs have committees register eligible beneficiaries by collecting more detailed information about the HHs, including recording ID information, at the same time. Two partners make a point of having community members sign off on beneficiary lists. One partner pointed out that they have women on the committee sign each page. Lists There are multiple lists of beneficiaries incorporated into the targeting process. These include those associated with Rapid Response Mechanism (RRM) and nutrition and other programs implemented by an organization. OCHA, IOM and the clusters provide IDP lists. Except for one partner that was implementing MPCA in conjunction with protection programming, IPs said that they do not accept lists from government authorities, despite being pressured to do so. The FSAC Cluster Coordinator said that local authorities provide lists of IDPs, including those in centers and those staying with host families. The confusion is most likely because of the misunderstanding of the respondents or the evaluation team about the term “list”. Other names of beneficiaries may come from other interventions. One partner’s nutrition team provides the FSL team with a list of HHs with children and PLWs with MAM/SAM, which accounted for 70 to 80 percent of their MPCA beneficiaries. Another partner implementing MPCA had lists identified by women’s groups, referrals from other programs and local authorities. Initial registration lists for MPCA are often used to cull beneficiaries for livelihood or WASH activities. Just over half of the community members surveyed (18 of 35) reported receiving lists of beneficiaries mostly from the IPs, although a few respondents mentioned getting lists from other committee members and health facilities within their areas of jurisdiction. Community committees usually review these lists prior to registration for duplicates and HHs that are no longer in the area. 21 For its USAID interventions, the IP worked only in the south. 22 Registering Beneficiaries All partners require biodata and vulnerability information for HHs, regardless if beneficiaries are targeted by committees or if beneficiaries were identified through other means. The sequencing of how this information is obtained differs among the partners:22 1) the committees are responsible for completing the registration forms; 2) the committees nominate beneficiaries and the IP conducts registration; or 3) the committees are responsible for an initial list of nominated HHs and the IP conducts registration and verification as a single exercise. During the registration process, partners collect detailed information about the beneficiaries including names, ID type and numbers, and HH vulnerabilities. Registration “verifies” that HHs meet the criteria and collects more detailed information. Once registration is complete, the list of beneficiaries is verified by the IP. Led by the program teams, two partners implementing MPCA register and verify beneficiaries separately. One partner collects data jointly with committee members and the other uses only program staff. Both use detailed forms covering multiple interventions. Once registration is complete, the data are entered into databases. Community committees, under the supervision of the program teams, register beneficiaries for two partners. Both implement CFW, with one partner also using this approach for food vouchers. Using excel forms to collect biodata and vulnerability information, members go house-to-house to nominate and register beneficiaries. One IP uses different forms for each OFDA-funded activity while the other partner uses one form for multiple interventions, including FFP. Data are entered into a database and reviewed to ensure biodata information are correct and duplicates are removed prior to verification. Two organizations simultaneously register and verify beneficiaries using external enumerators. Both implement food vouchers in addition to other cash programming. “Field teams” train community nutrition volunteers to conduct house-to-house surveys, including MUAC measurements of CU5 and PLWs. The second IP hires external enumerators using a formal vetting process. Enumerators, trained and managed by the MEAL team, also travel house-to￾house to collect data. During this process, the MEAL team addresses feedback and concerns raised by community members during data collection. Enumerators are not from the communities receiving assistance and are directed by committee members to the nominated HHs. Of the four partners that conduct registration and verification separately, only one IP’s CTP SOP has any substantive guidance about registering eligible beneficiaries. This partners’ SOP also includes a section on common challenges and proposed solutions, including what happens if more people than anticipated meet the targeting criteria in the areas covered by the intervention, and what to do in the case of a divorced female-headed HH living with a well-off family. The other IPs provided much less detailed documents. When registration and verification are conducted together, policies are discussed in the verification section.23 Registration of HHs can take between two and five weeks, depending on the number of beneficiary HHs and the use of mobile data collection platforms. The number of HHs registered, 22 A distinction between beneficiary nominations and registration is made to permit comparisons with practices. 23 One IP doesn’t refer to registration in its SOP and the other identifies registration and verification as a single task. 23 according to the IPs, ranged between 10 and 20 HHs per day, although the upper limit seems unrealistic in rural areas. Except for one partner’s guidance which listed the steps and staff responsible for finalizing registration lists, there is little guidance on how to process the data once beneficiaries were nominated or registered. Of the SOPs reviewed, none of the policies distinguished between registration using mobile devices and paper-based surveys during the registration process. Beneficiary Selection According to interviews and program documents, most of the beneficiaries for MPCA, food vouchers, one of the AFS activities and WASH interventions were IDPs, HHs with children and PLWs with MAM/SAM or female-headed HHs.24 Most community members surveyed (83%) thought that all respondents met the selection criteria and 17 percent thought that most of them met the criteria. Reasons why recipients did not meet criteria were favoritism or being too well￾off to qualify for assistance. The majority of the HHs selected met at least one criterion. Although three partners indicated that they ranked or scored beneficiaries, only one used formulas to prioritize eligible FFP beneficiaries in the South following beneficiary nomination. Nominated HHs are scored on a scale of 0 to 100 prior to registration using a weighted vulnerability criteria (e.g., 25 points for IDPs, 5 points for elderly-headed HHs). Once scores are tabulated for each HH, the top-ranking HHs are registered and verified with the remaining HHs put on a waiting list. The remaining two partners provided inconsistent responses and documentation and clarification emails determine that neither uses this method for beneficiary selection. One partner is planning to implement the system for CTP, and the other, according to one senior manager, no longer uses it as, “80-90 percent of the households are in need and meet the selection criteria” for food voucher assistance. Following registration and/or verification, partners try to reduce inclusion errors by having multiple committee members signing off on registration lists. One partner posts beneficiary lists when it is safe to do so. Inclusion and exclusion errors, according to staff, are raised through complaints and feedback channels, mostly through calls to hotlines and toll-free numbers. Large numbers of complaints prompted two IPs to further inspect their lists and conduct additional verification efforts, which validated these complaints. Complementarity As much as partners would like to integrate food vouchers with CMAM or other OFDA-funded programming, the relationship was unidirectional for one partner. This partner used beneficiary registration as an opportunity to assess HHs for cases of malnutrition (a criteria) and refer those with low MUAC scores to neighboring health facilities. The other partner interviewed said that the receipt of food vouchers would be considered for other OFDA assistance, if they met the criteria. Providing food vouchers to HHs with CU5 and PLWs with MAM/SAM proved more challenging. The diagnosis and treatment of MAM/SAM cases occurs over the life of the OFDA award, whereas targeting for food vouchers has little flexibility because caseloads are determined 24 Only one partner defines household in its SOP. None of the IPs define female-headed household. 24 quite early in the process and once the transfer of assistance begins, there is usually no opportunity to accommodate additional beneficiaries. Within OFDA programming, interventions were designed to be integrated and nutrition-focused. All of the partners had MAM/SAM as part of their criteria, however, only one partner interviewed targeted 70 to 80 percent of their MPCA assistance to HHs that were treated to MAM/SAM at the facilities the IP supported. The same IP targeted MPCA HHs for additional AFS interventions. One partner provided protection beneficiaries with MPCA assistance in one area, using this as an entry point for community acceptance of protection activities in another area. Reaching the Most Vulnerable Reaching the most vulnerable is a priority for every organization implementing humanitarian assistance in Yemen. Eighty percent (24.3 million) of Yemen’s population is in need of assistance, leaving a situation where the needs are much greater than the available assistance. Partners report that they reach the most vulnerable through geographical targeting because where they work (IPC 4 and 5 areas), there are few or no organizations supplying aid. The neediest areas are further refined through assessments and negotiations with district officials. One partner added that it specifically targets remote areas as a way to reach the most vulnerable. Community￾based targeting is another way that implementing partners can reach the most vulnerable because committee members are able to identify HHs with the greatest needs. Because Yemeni culture emphasizes taking Partner D of the weakest members of society, according to respondents, the committee members know where the most vulnerable live. Additionally, committee members representing the most vulnerable (e.g., women, the disabled or professions such as a teacher or health facility worker) help identify these families. Discerning who is the most vulnerable is not always clear. FSAC criteria are mutually exclusive when it comes to the head of the HH; the only variations are whether the HH has registered cases of MAM/SAM, are IDPs and/or from marginalized communities. Nonetheless, disagreements arose around whether a HH with MAM/SAM cases should still receive assistance if the child had been stabilized or the HH had assets. Some CTP programs did not consider income a vulnerability criterion. Among those that did, at times the targeting did not consider large families or non-payment of government wages. Lack of or use of different definitions for HH, status of MAM/SAM, chronic illness, disabled, or little income were not defined, most likely adding to the confusion. Because partners do not report data according to vulnerability, it remains unclear if they are able to reach the most vulnerable. Determining whether certain beneficiaries are more vulnerable than others is a form of ranking as those HHs meeting the most criteria rise to the top of the list. In a country that has been experiencing conflict in the last six years, with over 100,000 IDPs in 2020, and with conservative tribal communities where gender norms prevent women from traveling alone or girls from going to school, and with the overwhelming majority in need of assistance, it is difficult to discern if committees are able to reach the most vulnerable. The majority of IPs told beneficiaries to nominate and/or register the same number of HHs as caseloads, which implies that even they are unable to confirm if there were HHs more in need than those on the lists provided. 25 The ranking of beneficiaries also requires partners to trade-off between determining who meets the most criteria and exacerbating tensions. When registered HHs exceeded caseloads, tensions rose between the community and IP staff, overloading committees and hotlines with feedback and complaints. Even in cases where the IP ranked HHs prior to registration, senior management stated that a number of those who were not selected lined up at the IPs offices to complain. Conclusion Partners with policies seemed to follow them, when they were available, and to the extent that they were detailed enough for operationalization. Two partners had extremely good policies and SOPs for beneficiary targeting, although one seemed to be more tailored to Yemen than the other. Except for two partners with procedure documents on beneficiary targeting, selection and verification, partners’ guidance was less detailed and part of more expansive CTP SOPs covering everything from start-up to close-out. In addition, guidance was focused on different aspects of beneficiary targeting. For example, one partner who is in the midst of adapting and drafting SOPs had extremely operational guidance on communicating with communities but only listed high-level policies on committee formation. A different partner with policies lacking substance, focused on the mechanics of registration. Although one partner identified scenarios for issues faced during beneficiary targeting and registration, there was a lack of policies and procedures to guide committees for applying the criteria when nominating or registering beneficiaries. How criteria were applied by committees was not clear in most cases and one partner who provided guidance and scoring sheets did not use them, according to a follow-up email. Definitions of vulnerability criteria were lacking in most cases and only one partner explicitly defined HHs. None of the partners had operational guidance for notifying committees of cases when nominated or registered cases are rejected or how these should be handled from a policy perspective. Policies set the tone, but SOPs and supplemental procedural documentation are required for implementation so that the same policy is not misinterpreted by staff and applied unevenly within or across interventions. Beneficiary Verification Q2. What are the current policies and practices for beneficiary verification? How compliant are IPs with their stated policies? Verification ensures that targeted populations meet the established criteria for relief assistance. By design, through random HH sampling and error thresholds, verification reduces exclusion and inclusion errors and minimizes fraud. IP staff also consider complaints and feedback, technical scoping and oversight of infrastructure and some forms of AFS assistance, and post-distribution monitoring (PDM) for CTP as forms of verification. This section looks specifically at verification of HHs registered and the procedures followed by partners. Staffing Whereas program teams are responsible for beneficiary targeting, MEAL teams are in charge of beneficiary verification. The extent to which program teams are involved depends on the number of MEAL staff able to participate in field activities, use of external enumerators, workload, and ability to obtain travel permits. For large efforts such as verifying food voucher beneficiaries or MPCA, four of the partners interviewed used external enumerators or volunteers to verify HHs 26 for CTP and food vouchers. Two partners relied on internal staff for the sectors examined. Partners generally have between two and seven MEAL staff (plus one person dedicated to CFM) and where feasible, other MEAL staff travel to site to provide support. Oversight is provided by either director of the sub-office and/or the MEAL coordinator. Policies Although there was guidance and procedural documents for verification, much of it lacked specificity. While the MEAL team leads most verification efforts, procedures and policies were usually found in program team SOPs. MEAL plans provided by two partners also provided some detail about verification. Although documents provided by IPs outlined sample sizes and thresholds for a second verification, lines of authority and/or internal reporting, partner policies and guides lacked important detail around training, instrument protocols, approaches to sampling beyond “randomization,” including approaches for selecting villages and HHs. There is little written information on processing beneficiary nomination and registration data in preparation for verification, and only one IP’s documents addressed operations and logistics. Two partners had guidelines for processing registration/verification data in their CTP SOPs, and one of them had additional instructions for protecting data in their FFP MEAL plan. Of the five partners queried, three partners provided data protection policies, one provided USAID’s Security Guidance, and the last organization is currently updating their policy. Five of six IP’s documents mention protocols for approving lists following verification; these vary in their specifics. IP’s protocols and staffing for verification are being updated. Recently, two partners added MEAL coordinators and staff, and another created a program quality unit encompassing MEAL. Three IPs have new MEAL coordinators. The team received draft verification protocols from one partner and a different partner sent multiple versions of the same verification form. Seemingly, IPs are revising and updating procedures for new agreements in the wake of the USG closure of programs and as new senior staff come onboard. Sampling Five partners provided documents identifying the sample size for verifying AFS, CTP and food baskets nomination and/or registration lists. Except for AFS, these documents identified error thresholds for triggering additional verification. In the case of CTP and food baskets, the proposed sample size is between 10 and 30 percent25 as captured in their proposals, MEAL plans or SOPs. The percentage of beneficiaries, according to policy documents, varied depending on the number of beneficiaries and the use of external lists. One partner had conflicting thresholds, and another stressed the importance of doing 100 percent verification for food basket distributions. Although thresholds for a second round of verification is in four partners documents, none of the IPs really explain what redoing the sample or conducting 100 percent verification (i.e., of the village or the entire list) means. The documents and the interviews did not shed full light on the percentage of beneficiaries verified as a matter of practice and policy. According to staff interviews, four partners expanded their sample from 10 to 30 percent to 100 percent for CFW and food vouchers and from 50 to 25 CaLP recommends verifying at least 10 percent of the beneficiary list, source: https://www.calpnetwork.org/wp￾content/uploads/2020/01/CaLP_Urban_Toolkit_web.pdf 27 100 percent for AFS and the WASH interventions not linked to previous MPCA interventions. All three FFP partners said they shifted to verifying the entire registration list by the third cohort, including two partners who both register and verify beneficiaries at the same time. IPs, whose staff registered beneficiaries for MPCA, increased but did not substantially change the size of their sample (20%) for verification, except in the North where one partner faced difficulties in getting permission for staff to travel and verified all the registered beneficiary HHs by phone. One MPCA implementer, however, had to conduct 100 percent verification in some villages due to problems with the data and complaints received from HHs not selected for assistance. In addition to exceeding error threshold rates, additional reasons for increasing sample sizes to 100 percent provided to the evaluation team are: more available resources, suspicions of lists provided by the community committees and/or OCHA and International Organization of Migration (IOM), and large numbers of complaints. It is unclear if these changes are permanent. Verification As stated earlier in this section, partner documents lack specificity around verification procedures. Guidelines for the two IPs conducting joint registration and verification provide a broad overview of the process and are in line with other IP’s SOPs for registering beneficiaries. The two partners registering and verifying beneficiaries separately have no information except for sample size and error thresholds, data entry and finalizing lists after verification. In the case where IPs verify committee registered data, one partner’s policies identify staff and outline procedures for approvals and coordination, while the other IP’s CTP SOP refers to a missing attachment. Similar to registration, partners seem to be evenly divided in terms of how they have approached verification. Two partners conduct registration and verification simultaneously and refer to it as “100 percent verification” although neither verifies the data collected by external enumerators. Two partners, who also happen to implement MPCA, verify a sample of beneficiaries registered by the program team in alignment with their policies. One IP stratifies the sample by location and verifies a small sample of verified beneficiaries in each village. Both partners use the original registration forms during verification, making corrections to the form or in a separate template. One of the partners verified its entire list of MPCA beneficiaries in the North by phone because staff were unable to travel to the area. MEAL staff led verification efforts using a separate form when committees registered beneficiaries. Both initially started verifying a sample of beneficiaries although during implementation, the percentage was increased to 100 percent. Both partners implemented CFW and one of these also provided food vouchers. The three partners implementing AFS and WASH HH interventions applied different registration and verification procedures in line with those described for CTP. Program staff who worked on these two subsectors reported that they conduct another form of verification as they re-check vulnerabilities when conducting site visits, to assess requirements or conduct technical inspections. Integration of Gender into Data Collection Efforts Staff from all partners said that female enumerators are used to collect data from women, while field staff from two IPs stated that men can interview female headed-HHs. Staff from two partners said that they employ external female volunteers to aid in verification efforts (20% and 28 60%, respectively). Female committee members and female staff will also help with registration and verification as needed. Data Processing All of the partners limit access to the data sets associated with registration and verification. Partners mostly use excel spreadsheets to process, update and clean lists used for registration, verification, and final beneficiary lists. Except for the two partners that verify a sample of registered beneficiaries, the MEAL teams are responsible for managing the lists until they are finalized. Finalizing the Beneficiary Lists Except for one partner, the MEAL team produces a report on verification for the FSL team. Staff from four IPs said that information on HHs being dropped from the lists and the reasons why are included in these reports. One IP noted that the MEAL and program teams discuss dropped HHs, and if there is any disagreement, staff from the two teams go together to investigate the contentious HHs. Since partners often use community committees as a channel for CFM, three partners mentioned that they inform them of the dropped HHs and the reasons for their removal from the list. Avoiding Duplication The humanitarian crisis in Yemen has resulted in a rapid increase in the number of humanitarian actors operating in different locations around the country. According to the 2020 Humanitarian Response Plan, there are 208 partners in Yemen. Duplication of assistance to HHs is mitigated through coordination of efforts, partners reviewing lists and by registering and/or verifying HHs. Because the clusters coordinate aid efforts, five partners said that they avoid overlap because they are mostly working in areas where there are no other organizations providing similar or any assistance. One IP added that local authorities also coordinate assistance. Partners providing food vouchers check their beneficiary lists with the World Food Programme (WFP) or other partners in the area. Three partners said that they check with other international and local organizations in the area. One partner checks with other departments in the organization. Two partners ask more detailed questions during registration about the type of assistance, the name of the organization and when the assistance was received, and the other partner asks HHs if they are receiving assistance from another organization. Four partners consider the community committees and/or the community as another means to mitigate the overlapping of assistance. Avoiding duplication is a challenge for partners. Partners complained that international organizations do not want to share lists because of data privacy issues and the effort involved. Registration forms, and resulting data tables are different, which makes comparison difficult. Moreover, beneficiaries often have the same name or use multiple versions of identification to obtain assistance. Conclusion Both registration and verification are surveys but none of the documents provided indicate that it is treated as such. Of the four partners that conducted registration and verification as separate exercises, the MEAL team was in charge of the exercise; however, procedures are found in 29 program-issued documents such as CTP SOPs. Although these documents outlined sample sizes and thresholds for a second verification, lines of authority and/or internal reporting, none of the documents fully described the process. All partners policies and guides lacked important detail around training, instrument protocols, approaches to sampling beyond “randomization,” including how to select villages and HHs. There was little written information about the processing of beneficiary nomination and registration data in preparation for verification, and only one addressed operations and logistics. Protocols and staffing for verification are evolving. In the last year or two, MEAL coordinators and staff were added by two partners. Procedures were recently or are in the process of being updated. The team received draft SOPs from one partner and multiple versions of the same verification form from a different partner. IPs are revising and updating procedures for new agreements in the wake of the USG closure of programs and as new senior staff, such as MEAL coordinators, come onboard. Community-Based Management of Malnutrition CMAM is one of the subsectors chosen for the evaluation. Beneficiary targeting and verification processes are different than the other sectors explored and for this reason, the findings and conclusions are kept separate. For CMAM, the evaluation team considers beneficiary targeting as the process by which partners, and by extension the CHVs, determine which areas and HHs they visit for community outreach and conduct active case finding and referrals at the HH level. Beneficiary verification encompasses diagnosis of cases of MAM/SAM in CU5 and PLWs by staff at the health facilities and follow-up by the CHVs. Policies Beneficiary targeting and verification for CMAM were assessed for two purposively selected partners. Both partners are currently using the Yemen CMAM guidelines (2013) as their overarching reference document for their respective CMAM interventions. No additional CMAM related policies or SOPs were shared with the evaluation team. The guidelines emphasize the importance of ensuring children with SAM and MAM are identified before they develop severe medical complications and all children under 5 are routinely screened for SAM since it is one of the most prevalent causes of childhood mortality. Widespread identification of children with SAM and MAM at the community level is achieved through timely screening using the MUAC measurement and assessment for the presence of bilateral pitting oedema. The guidelines are required to be used by all partners and to guide nutritional program managers, supervisors, nutrition rehabilitation staff, CHVs, health institutes and universities, and NGOs providing nutrition services. Coordination National guidelines emphasize that CMAM activities should be planned, monitored, and managed in consultation with government, local authorities, and community committees if they exist. It is important to clarify the respective roles and responsibilities of various stakeholders. Both partners confirmed working closely with the Ministry of Public Health and Population, health offices at governorate and district levels and the nutrition cluster in order to harmonize 30 approaches and avoid duplication in CMAM programming. For both partners, selection of health facilities to support was done in coordination with the nutrition cluster, governorate, and district health offices. This was done to prevent situations where more than one partner was providing CMAM support to the same facilities. Selection Process for Community Health Volunteers Both partners adopted similar CHV selection processes. CHVs were selected from catchment areas of the health facilities they were attached to. For the most part, District Health Offices (DHOs) nominated CHVs. For one partner, CHVs were selected by a special committee. According to a Nutrition Specialist, CHVs were selected by a committee which included the DHO, governorate health offices (GHO) and local authorities in the community. He further emphasized that CHVs must meet certain requirements including being educated to a level where they can read and write, must possess community engagement skills or the ability to work in the communities, among others. The majority of recruited CHVs were female and in some cases were inherited from previous projects. One partner reported that the GHO in Sa’ada governorate requested that all CHVs from the previous project be dismissed and that a new set of volunteers be selected. However, it remained unclear why the GHO took such drastic action. Role of Community Health Workers/Volunteers According to Yemen CMAM guidelines, the role of community outreach workers is to identify CU5 and PLWs with acute malnutrition and refer them for treatment. They conduct early case￾finding in the community based on the presence of bilateral pitting oedema and low MUAC measurements, refer cases to the nearest outpatient or inpatient Partner D sites, visit absentees or defaulters at home and encourage them to return to outpatient/inpatient Partner D. They follow￾up with children who are not responding to treatment at home; investigate issues and offer advice. They record home visits and report to the health facilities that they are attached to in a timely manner. They give monthly feedback to community leaders or health committees on issues such as the number of malnutrition cases in each location, cure, default, and death rates. They conduct community sensitization on health and nutrition messages. Both implementing partners and the CHVs surveyed confirmed that CHVs conduct community outreach and mobilization, screening of children and PLWs, referral of identified SAM and MAM cases, and follow-up. CHVs surveyed also said that they implement community-based nutrition and health education (11 of 15) and provide individual counseling on health and/or nutrition (10). Village and Household Identification The CMAM guidelines do not mention how villages and HHs are identified. However, blanket targeting is recommended for contexts like Yemen where malnutrition rates are 10-14 percent or 5-9 percent with aggravating factors i.e. a crude mortality rate of more than 1 per 10,000 per day and epidemics of measles or whooping cough and high incidence of diarrheal diseases. Blanket supplementary feeding is normally implemented year-round, targeted at groups with the highest risk of being acutely malnourished. According to one IP, CHVs are assigned to geographic areas in one of three health facility zones (zone 1, 2 or 3) to avoid duplication. These zones are based on the distance from the heath facility. One partner, noted, that CHVs sometimes have difficulty reaching HHs in zones two and three because of the distance and the terrain. CHVs are typically allocated 50 HHs and visit them two to four times a month. The HHs visited are largely 31 determined by health staff, CHV supervisors and/or the IP. One health and nutrition officer added that CHVs also contact local authorities and community leaders to ensure that they are reaching everyone. In the case of one partner implementing food vouchers, CHVs also follow-up on cases identified during verification visits. CHVs visit approximately 10 HHs per day, based on discussions with local authorities. Both partners do not have written procedures for village and HH identification. Beneficiary Selection Criteria/Screening Yemen CMAM guidelines recommend the use of MUAC tapes as primary criterion for SAM and MAM among children and PLWs as it makes detection and treatment of malnutrition in community and health facility settings simple and more effective. For children aged 6-59 months, MUAC scores of less than 11.5 centimeters (cm) indicate SAM whereas scores of between 11.5 cm and 12.5 cm indicate MAM. Weight for height (WFH) measurements are also recommended. A WFH standard deviation below -2 z-score of the median (WFH < -2 z-score) indicates wasting. Moderate wasting is indicated by a WFH ≥ -3 and < -2 z-score and severe wasting is indicated by a WFH < -3 z-score. Infants under 6 months with bilateral pitting oedema and/or visible wasting are not measured but referred to the health facility where they are further investigated. PLWs with MUAC scores of < 23 cm are considered acutely malnourished. Both partners reported using MUAC as primary criteria for SAM/MAM screening for children 6-59 months and PLWs in line with the CMAM guidelines. CHV Training and Support According to Yemen CMAM guidelines, CHVs/CHWs should be trained on community mobilization, screening, health, and nutrition education. Health Partner D providers should also be oriented on the same topics, since they are responsible for supervision and coordination of CHV/CHW work. This is followed by periodic training and support including refresher training, mentoring and feedback meetings. Both partners confirmed conducting CHV initial training, refresher training and continuous support supervision. One partner indicated that they periodically check on CHVs to assess if MUAC measurements are being done correctly and that this informs capacity building efforts, including on-the-job mentorship. A nutrition manager said that they conduct bi-monthly meetings with CHVs to assess work done and address challenges faced. Another partner added that CHVs are also trained in reporting and follow-up of cases. As part of the CHV onboarding, they are oriented on nutrition activities, their roles and responsibilities, and how to work with the communities. Two respondents stressed the need for good communication skills with beneficiaries. Both partners also supervise CHVs as part of overall continuous quality improvement. Eleven of 15 CHVs surveyed said that they have enough training and information to do their jobs. Referrals CHVs refer children and PLWs found to have SAM or MAM to the nearest health facilities for further management, according to CMAM guidelines. Interviews with selected partner staff confirmed that CHVs refer children and PLWs diagnosed with malnutrition to health facilities in their proximity. They provide referral cards to each identified SAM and MAM case. One partner noted that MAM/SAM in zone 3 are normally treated from their homes since they are at a distance from health facilities. All CHVs surveyed said that they refer cases to health facilities 32 and clinics; most conduct follow-up (11 of 15) and just over half (8 of 15) also provide counseling. A Nutrition Officer from one partner said that as part of their programming, they give travel vouchers to support families’ access to stabilization centers. Management of Malnutrition Cases Referred to Health Facilities According to CMAM guidelines, a qualified health Partner D provider carries out a nutrition and medical assessment to determine if a child 6-59 months with SAM has a good appetite (passed the appetite test) and no medical complications, and can thus be treated in outpatient Partner D. The child will receive medication according to the treatment protocol, and a home ration of ready to use therapeutic food (RUTF) equivalent to about 200 kilocalories per kilogram of bodyweight per day (kcal/kg body weight/day) to last until the next health visit (usually weekly). For inpatient Partner D, anorexia or poor appetite, severity of illness and presence of medical complications are the major determinants for providing inpatient Partner D to children 6-59 months with SAM. One partner supports both targeted supplementary feeding programs (TSFP) at 77 health facilities, along with outpatient therapeutic programs (OTP) at 84 health facilities. According to a health and nutrition officer, health workers at supported sites check MUAC measurements written on referral cards and re-do the measurements to confirm that no mistakes were made. He also reported that these facilities no longer take z-scores. For MAM cases, children (6-59 months) are given Plumpy’Sup and Partner D givers are requested to bring children back in two weeks’ time to check for improvement, which is one week less than the CMAM guidelines. For SAM cases, children are given Plumpy’Nut. It is not clear if they are checked every two weeks or monthly. Besides therapeutic food, children are treated for worms, and given vitamin A and micronutrient powders. What is important to note is that none of the interviews indicated a difference between OTP and TSFP units within the health facility. It seemed that the only discernible difference between MAM and SAM is the type of therapeutic treatment (e.g., Plumpy’Nut or Plumpy’Sup) and when they were told to return. Interviews with staff from the other partner also confirmed that all referred cases are rescreened by health workers to confirm MUAC measurements and determine z-scores. MAM and SAM cases with complications are admitted into the health facility for close monitoring and proper management. According to one partner, MAM and SAM with complications, including failure to eat and oedema are referred to TFCs in a nearby hospital or to Doctors Without Borders or similar organizations. Once a child is discharged from a TFC, he/she will enter either the TSFP or OTP program. Children will be discharged when they meet the discharge criteria in the CMAM protocol. Follow Up of Malnutrition Cases According to CMAM guidelines, patients that miss an appointment are considered absentees and must be followed by the outreach workers and reasons for not returning for review must be established. These patients should be encouraged to return to the program. Staff from both partners and survey respondents say that CHVs follow-up on MAM and SAM cases. According to one IP, community volunteers conduct follow-up visits for children who do not show up at follow-up visits for OTP or TSFP. Health workers track cases and provide lists to the CHVs for follow-up. 33 Record Keeping for CHVs For both partners, CHVs are provided with registers to record information on the number of HHs visited, children and PLWs screened, MAM/SAM cases discovered, those referred, defaulters etc. CHVs provide records to both the partners and the health facilities that they are attached to. According to one IP’s nutrition specialist for one of the partners, an additional copy is created and submitted to the DHO. Both partners enter CHV data into their respective data management systems. One partner added that they also share copies of CHV reports with the nutrition cluster and the MOH. CMAM guidelines stipulate that CHVs are expected to keep records of HHs visited, children and PLWs screened, those referred, those followed up among others. This information is shared with the health facility that they are attached to. Record Keeping for Health Facilities Both partners confirmed that the supported health facilities use registers and reporting forms to capture and report client data. In the past, IPs have been picking data directly from health facilities, but this changed recently. According to the Senior Nutrition Manager for one of the partners, the SCMCHA MOH instructed IPs to desist from gathering data from health facilities and to instead collect it directly from the DHOs. One of the partners has a health information system called IMPACT (which is similar to the District Health Information Software (DHIS) where it enters, stores and visualizes data for nutrition services provided at supported health facilities and catchment areas. Conclusion For the most part, IPs are following CMAM guidelines as laid out. There are no differences in programming approaches used in the North and South except that one partner faced major challenges including travel restrictions and interference from local authorities in the North. In the absence of partner specific SOPs, it is difficult to confirm whether the IPs are implementing national guidelines as stipulated. Community Feedback Mechanisms Q3. How are IPs handling complaints about targeting? This evaluation looks at the mechanisms used to collect and respond to complaints around beneficiary targeting and verification. Serious complaints (e.g., fraud, sexual abuse, and exploitation) are part of CFM and how these complaints are classified and handled is mentioned briefly. Staffing CFM are under the purview of MEAL for all partners. One IP has a dedicated CFM team with one to two officers located in sub-offices and three partners have at least one CFM officer. The other two partners did not refer to specific CFM staff but MEAL staff more generally. Five partners had one CFM system for all activities, usually with one manager. One of the five partners said that it creates separate systems if the donor requires and the sixth partner had one CFM per award. Processes for CFM were the same for North and South Yemen. 34 Policies Among the partners is a wide range of documents setting policy and describing procedures for complaints and feedback. Three partners provided copies of global policies or frameworks (i.e., protection, client responsiveness and integrity). Five partners adapted global policies and procedures for Yemen; three are up to date. The evaluation team received extremely detailed SOPs from four organizations. All partners have documentation, but some partners are more detailed about the operational aspects of the CFM than others. Community Notification IPs use a variety of methods to inform community stakeholders about their CFMs. All of the partners inform communities about the CFM at the outset of their activities, usually at the time they first engage communities to form CVCs. Two partners orientate community committees on CFM so that the committees can assist with feedback. Two partners conduct sessions about CFMs prior to distributions, and the two IPs. One partner further explained that during these sessions, they not only talk about the process, but also the confidentiality around complaints and feedback, and the information complainants should expect to receive. Partners also advertise CFM on food vouchers. Two IPs’ staff also deliver presentations on CFM while in the field; one mentioned staff explaining CFM and handing out cards during beneficiary verification. Three partners’ CTP SOPs emphasize that CFM is discussed at the outset of community meetings and one partners’ CFM SOPs only makes mention of it. All the partners use different information, education, and communication (IEC) materials such as posters (five partners), leaflets and cards (four partners), banners (two partners), stickers (one partner) and awareness booklets (one partner). In addition to listing phone numbers and/or Whatsapp numbers, two partners added that they also use pictures to describe how to file a complaint on some of the materials. One partner mentioned using loudspeakers to inform communities about CFM and another sends “bulk” SMS messages. Mechanisms Policies cite a variety of channels to collect beneficiary and nonbeneficiary feedback and complaints, including hotlines, toll-free numbers, face-to-face, chat applications, suggestion 0 5 10 15 20 25 Phone/Hotline Suggestion Box Committees Help Desk Speak with IP WhatsApp Email SMS NUMBER OF RESPONSES Committees: Ways Community Members Register Complaints and Feedback 35 boxes and community committees. All IPs use hotlines/landlines (four have toll-free numbers) and face-to-face with staff during field visits for CFM. Three partners use SMS or WhatsApp. Four partners spoke of using community committees as a vehicle for receiving feedback. Health promotors and health facility staff are also considered channels because they receive feedback or complaints. The main channels mentioned by committee members surveyed were hotlines, suggestion boxes and committees. Five of six IPs’ policies identify that CRM or MEAL staff are responsible for responding to complaints. The guidance of five partners has scripts or key points that MEAL staff can use when speaking or responding to complaints and feedback. Partners’ policies recommend providing them with FAQs to address requests for information. Recording Complaints and Feedback All policies state that the recipient is required to record the complaint. Some organizations have standardized forms for staff to register complaints or for use by the MEAL/CFM team when they are in the field. Complaints recorded on paper are given to the person responsible for CFM, for that office to record and process. Three partners possess a code of conduct for employees in relation to CFM. Depending on the channel, different staff receive complaints. Partners have dedicated CRM or MEAL staff responsible for answering hotlines, toll-free numbers and SMS/WhatsApp if the hotline is a mobile phone number. Suggestion boxes are managed by MEAL staff, although the frequency to collect complaints depends on staff workload and location. Sometimes retrieval of complaints could take up to a month or program staff instead of MEAL staff would gather the complaints. Staff who receive complaints directly or through community committees are supposed to record complaints and pass them to MEAL to register. However, interviews and anonymized records indicate that program staff rarely record complaints. In the case of two partners, field staff or committees receive complaints and if they are able to resolve them, these are not recorded. CFM/MEAL staff are responsible for ensuring that complaints get registered in larger databases so that they can be addressed and processed for reporting. Four partners use online systems to record complaints, while two partners use Excel. Two of the partners use formatted tables for offices without strong internet connections. Both online and excel databases are password protected with limited access. One partner’s online system is managed from headquarters and sub-databases can be created for a country office, which can be aggregated nationally and globally. One partner has a separate log sheet for suggestion boxes and the complaint forms are kept in a locked filing cabinet. Partners classify complaints into anywhere from 3 to 8 categories. A typical system, for example, might use five categories: (1) request for information, (2) request for assistance, (3) dissatisfaction with assistance, (4) staff or partner misconduct, fraud, sexual abuse and One Partners Categorization and Response to Beneficiary Targeting: Positive feedback and suggestions are categorized as zero and one , respectively. Requests for assistance (category 3) states the MEAL staff should be able to handle most, but when the same request is received multiple times, the program team should provide a FAQ. MEAL staff are instructed to send emails to specific staff so that they can aggregated requests to specific areas. Allegations of fraud (category 5) include payments for favors, manipulation of processes for family benefit are to be reported confidentially within 24 hours to the internal auditor and CD. 36 exploitation, and (5) sexual exploitation and abuse (SEA) by actors external to the organization. One organization first divides activities into requests for information and assistance, discontentment with activities and processes, and complaints of a sensitive nature. Two partners assign a second code – low risk, medium risk (non-sensitive and sensitive) and high risk by one partner, and critical, high, medium, and low by the other. These classifications determine who in the organization leads efforts to formulate a response and the steps that need to be taken to address the complaint. In the case of two partners, these categories also determine timelines for responding to the complainant. Five IPs reported using a classification system for complaints. Serious cases around fraud and SEA are sent to senior management or the country-director, and strict processes are followed. Issues around targeting are typically sent to the program teams unless they can be addressed by the MEAL team. All the partners identify what should be done according to category or type of complaint, and four have graphics showing “decision trees” to help guide staff. All staff have clear policies on how complaints are recorded, and privacy ensured in their SOPs and/or data protection policies. Confidentiality and anonymity of complainants are addressed more clearly in some partners’ guidance than others. Staff from all IPs recognize the importance of anonymous complaints, but also stated that this made follow-up difficult. Managing and Responding to Complaints All partners have SOPs and policies outlining how responses are managed. Four SOPs outline who is to be informed of these complaints when they are assigned to program staff. Appeals related to complaints are addressed in two partners’ policies. Either the supervisor or a person in charge of the hotline manages complaints. The MEAL staff of five partners respond to complaints and feedback. For one partner without dedicated CRM staff, a senior supervisor manages the system. Less serious complaints are handled over the telephone or WhatsApp, while serious complaints, such as requests for information, are recorded and resolved by MEAL staff. Two partners provide staff handling the hotlines with FAQs to assist with their responses. When the MEAL team receives complaints that they cannot address, they send them to the program managers. This includes most complaints about beneficiary targeting, except when there is suspected malfeasance on the part of the committees or staff. If the inquiry is about why a HH did not receive assistance, program staff may provide an explanation based on targeting and verification records. If it requires a more thorough investigation, program staff and/or MEAL staff may lead the effort, depending on staffing and resources. Following a response from the program team, MEAL staff review feedback and respond. Not all partners follow this rule, formally or informally, except for one partner that has elaborate procedures and processes for recording and following up on complaints using multiple channels, including field staff addressing complaints obtained during face-to-face meetings, at help desks or relayed from committees. Three partners provide committees with support to address complaints, while one partner makes it clear to communities that all complaints are directed to staff. Seventy-seven percent of committee members and 83 percent of community leaders 37 surveyed said that they received complaints about not receiving aid or about the beneficiary selection criteria. Most committee members (21 of 27) solved the complaint themselves. Program and CFM/MEAL staff shared some frustrations. Program staff for two IPs felt that they were being watched by CFM/MEAL and that many complaints meant that they were not doing their jobs well. CFM staff said that program staff were not providing feedback to them to record. MEAL and CFM staff said that they were overburdened because of responding to complaints. Reporting and Analysis All the partners collect and report complaints to USAID. All FFP partners report complaints as part of their monthly reporting. OFDA partners are less regular in their reporting of complaints. The three partners whose documents the evaluation team reviewed, the reports focused on breakdown by sector and type. All partners that identified fraud noted in their reports that investigations were being conducted and separate reports filed with USAID. Of the two partners that identified cases where committee members were accused of misconduct, only one discussed clear follow-on steps. Except for one IP who produces weekly complaint and feedback reports, IP’s MEAL staff produce monthly complaint reports for senior management, and at least three produce quarterly reports; the latter two are in line with IP reporting to USAID. Internally, partners typically focus on efficiency metrics; the number of complaints, location, type of complaint, sector, and percentage of closed complaints. Whatever analysis is done is not reaching the field staff, according to interviews. One IP’s MEAL staff said they do quarterly presentations for staff. Two partners incorporate CFM data into lessons learned workshops at the end of their grants. Based on documents and interviews, issues around beneficiary targeting are most likely being discussed by program team staff rather than being reported and analyzed by CFM staff. Only one partners’ database included detailed staff responses that would permit such analysis. Four of six IP’s policies do not address internal reporting and dissemination. There is some evidence that complaints and feedback are being used to improve targeting. Partners spoke of complaints and feedback coming from certain areas that led to investigations that uncovered inclusion errors. Three IPs reported that significant numbers of complaints led to expanded verification efforts. Complaints and feedback also led to further investigation by partners, and one IP reported replacing committee members and conducting vigilant follow-up in those locations. Knowing CFM works The evaluation team asked respondents how they knew that CFM was working. Of the five partners that answered, all said that if they receive complaints, the system is working. Staff from three IPs said that in the PDM surveys, they ask beneficiaries if they are aware of a CFM and the types of channels they prefer. One partner conducted a separate mixed method accountability assessment in 2019. 38 Reach of Complaint Systems Access to complaint channels is not always assured. Targeted areas can be spread out over mountainous terrain and mobile reception can be poor. Illiteracy is more prevalent among women, a target group, and not all have access to mobile phones. When the evaluation team asked how partners ensure the most vulnerable voices are heard, a common refrain during interviews was “they can ask a relative or neighbor to call” as PDM surveys show that the majority of those who had received aid, prefer hotlines. It was shared that suggestion boxes were another way to reach those who may have access challenges, although one partner’s staff said that in remote areas, these complaints may be collected monthly. This is not to say that IPs do not recognize a gap. Partners said that people can reach out to community committees, IP field monitors and CHVs to voice their concerns. All partners orientate committees on CFM and one partner had CHVs distribute 10,000 booklets with pictures on how to file a complaint. Two partners said that they made a point of holding community meetings specifically for complaints, and all noted that staff regularly travel to the field, e.g., during implementation, verification and PDM, which gives those in remote areas, opportunities to reach out. For some partners without volunteers in the field, lean staffing of MEAL and protection at sub-offices, and where travel can be difficult, expensive, and at times dangerous, means that those in the most remote areas may not be reached. Conclusion All of the partners possess elaborate CFM policies and procedures. For the most part, partners are adhering to them. Field staff from at least three partners said that they handle complaints when they are approached by people, either during field visits, visits to the office, or when community committees approach them directly. The number and type of complaints serve as a barometer for targeting; two partners provided instances when investigated complaints pointed to inclusion errors. A review of four anonymized complaint logs suggest that two partners are relying only on hotline calls, suggestion boxes and SMS/WhatsApp messages. Face-to-face meetings and help desks were cited as two channels for complaints about beneficiary targeting. If this kind of feedback serves as another source of data for pinpointing inclusion or exclusion errors, then it should be ensured that this kind of feedback is addressed. Differences Between the North and the South IPs that operated in the North and the South stated that for the most part, their policies and processes for beneficiary targeting, beneficiary verification and complaint mechanisms were the same in the North and the South. Senior managers may have been reluctant to share their experiences about the challenges they faced in the North if they had the potential to jeopardize any future reinstatement of USG assistance. Delays and refusal by local authorities and SCMCHA make operating in the North challenging. Interference by government authorities, restrictions on travel, and increasing insecurity and attacks on aid workers made implementation very difficult. One partner that worked in a governorate traversed by the front lines faced challenges coordinating between the GHOs representing the North and the South. GHO interference in community activities in another governorate in the North, including the appointment of CHVs, resulted in the suspension of the program in the governorate. 39 While beneficiary targeting practices remained the same according to partners, SCMCHA and district authorities pressured partners to accept lists for targeting. They also requested final lists once selection was complete. While all IPs said that they were able to deflect government-issued lists, responses were inconclusive surrounding the question of partners providing lists to government authorities. Travel restrictions and refusals affected registration and verification efforts. Although respondents denied that processes were different in the North and the South, the environment had an impact on data collection efforts around registration and verification. One partner combined registration and verification to meet deadlines for distribution. Field MEAL teams had to rely on colleagues from the program team to support verification efforts. Partners hired enumerators from NGOs to collect data and one partner conducted verification by phone, but even these workarounds could be blocked or slowed down by SCMCHA. Partners were also more likely to conduct verification of all eligible HHs because of problems with committee and RRM lists. Likewise, travel restrictions, according to two partners, prevented staff from going to the field, thus limiting efforts to receive complaints in person or from suggestion boxes. Strong and Moderate Practices Q4. What are the most successful and least successful practices for beneficiary targeting, verification, and associated complaint mechanisms in Yemen? This section highlights strong and moderate practices among partners based on the interviews and an assessment of internal policy documents. Because of the lack of reported data on beneficiary targeting, beneficiary verification and complaint mechanisms and biases articulated in the limitations section, the evaluation team is unable to confirm whether practices identified as successful resulted in better targeting of humanitarian assistance, fewer complaints, or less inclusion or exclusion errors. In addition, some of the nuances around CTP and food vouchers proved elusive, given the difference in caseloads and use of RRM and other lists. Regional differences also remain opaque since partners, for the most part, were not as forthcoming as the evaluation team had hoped, regarding challenges faced in the North during implementation. Community notification and initial engagement Although partners’ documents described at various lengths the formation of community committees, one partner’s procedures for committee formation stands out as it addresses key steps in the formation process, including details about preparing for and conducting community meetings, the composition of the committees and the selection criteria. Having these procedures in place ensures consistent practices, especially in light of field staff turnover. Two partners’ SOPs for beneficiary selection detail social, political, and logistical considerations before undertaking community-based targeting while another suggests checking with people in the area to triangulate whether areas selected by local officials are truly in need. Providing detailed guidance is beneficial for staff new to their jobs and a good reminder for everyone else. Community notification takes different formats, including posting information about the meetings in public areas. The materials distributed by one partner implementing food vouchers describes the project and the complaint mechanisms and includes cartoons describing different forms of corruption and pictures of food items redeemed. To reach more houses, one partner 40 mentioned using loudspeakers to advertise meetings and another mentioned hanging advertisements in health clinics. Two partners suggested using CHVs, however, it is not clear if any of the partners do this to notify HHs about community-wide meetings. Meetings held in schools seem more likely to attract female participants than those held in mosques. All of the partners engage local authorities and stakeholders and view this support as essential for implementation. One good practice is making local authorities aware of future data monitoring and evaluation collection efforts. Sheikhs and village elders play a major role in community mobilization, but most partners were unable to articulate what their role is in nominating beneficiaries and serving on committees and how their participation might influence beneficiary selection. Three partners discussed inclusion and exclusion errors stemming from lists provided by committees. Acceptance and access are extremely important and required, but may come at the risk of favoritism, especially as the humanitarian crises worsens and more people need assistance. One partner vets committee members to ensure that they have no relationship with any extremist organization, which may be a useful practice in contested areas. Minimum requirements for participation remain a worthwhile goal but only one partner was clear about what representation means (i.e., one family member per HH). None described the election process in detail although attendees usually show hands when selecting members, which may be the only feasible way to elect members among illiterate populations. Requiring committee members to be from different groups, and including women and youth are a good practice. Requiring meeting participants to sign the results of the election with a fingerprint or having district officials approve the election, emphasizes the seriousness of the election. Women’s participation in committees is hard to discern since the figures reported by most partners are convoluted or absent. Only one partner consistently reported on committees and the proportion of female attendees. It is possible that partners are doing this internally, but failing to break down this information weakens the argument that women are more likely to identify more vulnerable HHs. It also makes it difficult to determine whether women’s participation or lack thereof is a result of culture and/or implementation. Partners create parallel committees; however, one senior manager noted that women’s role in selecting beneficiaries is marginalized. To address the lack of participation by women, one partner is conducting focus groups and including questions around women’s engagement in their baseline surveys to gain a better understanding and develop strategies for increasing women’s participation. The presentation of CFM by the MEAL staff during the community-wide meetings not only makes the community aware of the channels for complaints and feedback, but also underscores CFMs independence from the program team. CFM staff tend to be women, and in addition to female field staff, can foster the importance of women engaging in community committees. Committees The use of codes of conduct among not only committee members but also staff, represents a good practice. Only one partner provided a detailed TOR, which describes roles and responsibilities of both committee members and staff. It also describes unethical practices and obligates the members to follow the rules set out in the TOR. Having committee members sign these documents and the notes of the meetings emphasizes the seriousness of their role. Although training of committee members was not discussed in detail, two practices worthy of 41 consideration are training committee members in the partners’ field office and conducting mock nominations/registrations as part of the training. In cases where training protocols are not developed, partners should develop these to ensure that processes are standardized. Beneficiary Nomination and Registration Of the six partners included in the study, two IPs who led registration efforts verified a sample of all eligible beneficiaries, in accordance with their policies. Committee members helped register beneficiaries, but efforts were supervised closely by IP staff. The staff used forms rather than Excel spreadsheets and had two different teams leading each process. Female staff and committee members also played a role in registration. Since both of these partners spoke about MPCA, they had smaller caseloads than would be typical for food vouchers. The collection of registration data by committees and verification by the IP was not entirely successful as partners ended up verifying all registered HHs. Committees used Excel tables printed on large paper which most likely led to errors in recording data and data entry, which in turn impacted upon verification efforts. Although both partners were working in the Hajjah and/or Hodeida governorates, large discrepancy rates between registration and verification data may be representative of flawed processes used in both regions. All partners stated that they used the same beneficiary targeting and beneficiary verification procedures in both regions. Respondents noted that the potential beneficiaries identified by the committees are becoming increasingly adept at seeking to demonstrate to enumerators their need for assistance, using such practices as emptying kitchens or switching houses. Unannounced registration visits (if nominations and verifications are done separately) reduces the time that HHs have to “prepare” and increases the likelihood of identifying inclusion errors. Partners tried different ways to endorse beneficiary lists. One partner had committee members, including female members, sign off on each page of names. Another partner posted lists in places where security is not an issue, and staff from the latter organization felt that this reduced inclusion and exclusion errors. Similarly, partners also mentioned reading registration lists during public meetings. According to these partners, feedback and complaints resulting from public notification of beneficiary lists helped identify committees that included non-qualifying HHs and excluded beneficiaries meeting the criteria, thus improving targeting. Verification The best strategy used for sampling is randomizing HH selection stratified by location. In the case of one partner who uses a sample to verify beneficiaries, the sample is stratified so that a minimum number of HHs (e.g., 5) are randomly selected per location for verification. Not all locations were randomly selected because of time and resource constraints according to staff, but enough locations were visited to provide the IP with confidence that the majority of data were accurate. When issues arose, verification was increased to 100 percent but only in those villages where errors exceeded thresholds. Although more extensive and effective sampling strategies seemed to require more resources, less resources and time may have been required as efforts did not cover the entire eligibility list. 42 Partners who conduct registration and verification simultaneously are not verifying the data collected by enumerators, even if it is under the supervision of MEAL staff. Poor data collection by enumerators, whether intentional or not, cannot be detected without checking the data set. Using enumerators and volunteers, as long as partners ensure training and proper supervision, is good practice. This allows partners to conduct larger data collection efforts in a shorter period of time. One partner has strict hiring procedures in place, whereas other partners have long-term relationships with the volunteers as they perform other tasks during implementation. External data collectors from the same areas facilitate access to areas. However, as one partner pointed out, it is critical that they are not from the same communities receiving assistance. Forms Forms used by partners and/or committees for registration have different strengths and weaknesses. Two partners used forms to collect detailed information about HHs, including the types of assistance previously or currently being received and by whom. The best instrument has numbered sections and items, clear response choices and spaces to enter information. Of the three partners that collect data on income, one has clear codes for assets which is much more reliable than relying upon enumerator judgement of “low income or no income.” None of the forms are uniform across partners. Some forms provided are ambiguous, which most likely is contributing to verification error rates. Poorly worded questions make it unclear whether the enumerator is asking a question of the respondent or if the enumerator is being asked to provide an answer. Items which require a yes/no response to “no income and no means of accessing food’ or “not receiving adequate assistance” or “household with child with malnutrition” are based on judgement and judgement is hard to verify. “Total number of family members by age” can present problems if the respondent counts him or herself in the total number, whether “family” versus “household” is understood the same way, or the respondent knows the age of people classified as elderly. Only one partner requires information such as name, age, relationship to each other, and status (e.g., disabled, widowed, registered SAM/MAM) for each HH member, which is easy to verify but requires more time for the interview. Forms used for multiple interventions promote standardized data entry. Poorly designed forms make data collection and data entry difficult. One major oversight is that almost all the forms lack instructions, scripts, or questions about informed consent, except one verification questionnaire used for telephone interviews verifying registration data. Criteria Partners did not uniformly apply criteria and there wasn’t always agreement on the criteria within an organization. Although all partners target PLWs and children suffering from malnutrition, the lack of clear parameters for determining these cases caused confusion among many IP staff during registration and verification. General criteria such as CU5 with malnutrition, which appeared in some instruments relies on enumerator judgement. Even when enumerators required follow-up cards, they didn’t have guidance for an acceptable length of time since treatment. Program and MEAL teams from one IP experienced disagreements about whether healthy looking children with cards were still vulnerable. Two partners spoke of instances where health facility staff forged follow-up cards (neither case happened with USG 43 funding). Best practice is the case where a partner had enumerators take MUAC measurements during registration, referring HHs with suspected cases of MAM/SAM to nearby health facilities. While the partner performed verification and registration simultaneously, the measurements are verifiable. Vulnerability criteria around the lack of income also caused consternation among staff since there was no clear-cut division around what constituted “poor”. Staff expressed frustration that family size was not taken into account or that government employees, such as teachers, who had not been paid in months, were still disqualified. If staff could not justify this distinction, one wonders if the committees could as well. Caseload Quotas for Registration One partner identified in its SOPs the number of HHs registered above the number of caseloads (five percent) and another partners’ guidance provided suggestions for what to do when need exceeded demand. Even though partners said they told beneficiaries that nominations and/or registration did not guarantee assistance, respondents said that CFM systems became overwhelmed and field teams (including MEAL staff) faced hostility from rejected HHs in situations where the number of beneficiaries exceeded caseloads. When demand is greater than supply, one partner conducted beneficiary targeting on a rolling basis, fulfilling the needs in one location before moving to the next. Another partner sought to quell hostility between IDPs and host communities by assigning the same number of caseloads to each group. Limiting beneficiary nomination and identification to five percent above caseloads is a good practice. CMAM Challenges faced by partners regardless of where they worked include insufficient supplies, reaching certain populations and CHV turnover. Shortage of nutrition supplies meant that at some health facilities, beneficiaries would receive a smaller quantity of Plumpy’Sup and Plumpy’Nut than required or would go without. IDPs are on the move and it is difficult to screen and follow-up. Difficult terrain and long distances between HHs, particularly in the second and third zones, make community mobilization and HH visits difficult. Both partners found that it was very difficult to retain CHVs because they are typically young and leave their positions once they get married or find better jobs. Although partners followed CMAM guidance for Yemen, they should supplement this with their own operation guidelines for CHVs. Community Feedback Mechanisms All of the partners used feedback channels. The types used and how they were advertised may inform best practice. Without a full examination of complaint records or even the number of complaints for all partners included in the evaluation, the team is unable to determine which methods are more successful than others. The majority of partners assign CRM staff to respond to complaints and feedback, and this is a good practice as long as there are clear feedback loops, timelines, and enough staff to handle complaints. Since a large number of complaints are associated with requests for information and requests for assistance, providing FAQs to staff fielding complaints may reduce some of the burden on program staff. Two partners provided scripts for staff and one made clear that serious complaints required solid grounds of evidence and/or direct experience. Partners with CRM staff in the field not only were responsible for 44 receiving and recording complaints, but also had a role in investigating complaints either with or separate from the team, depending on the issue. Some partners felt it was essential to hire women in these roles, where possible, since women will typically not speak to a male about issues, particularly around protection. Staff who were in charge of complaints and performed traditional MEAL duties reported feeling overwhelmed. One partner conducted an accountability assessment for all its programs in Yemen, and its results contradicted much of what was reported by other partners during the evaluation. Although exact figures were not provided, the survey respondents preferred receiving information face-to-face, through community mobilization or volunteers. Less than 15 percent preferred mobile phones. Recommendations included holding frequent community sensitization meetings, creating posters and other IEC materials, and strengthening internal systems for analysis and reporting. This partner also had separate logbooks for recording complaints in the field and a tracker for suggestion boxes. The majority of partners used online systems to log complaints although the evaluation team suspects that a number of complaints received by program staff in the field are not making it into these systems. Two partners provided output from these systems that exceptional systems were in place: detailed information about the complainant, drop down fields were used to standardize data entry, information included recorded dates, staff member and office that recorded the complaint, office assigned to respond to the complaint, description of the response, staff member who responded, and the status and date of resolution. Of the four anonymized lists provided, two seemed to suggest the partner was capturing data from the field. The strongest systems are the ones that collect all feedback data, respond to it, and use it as an alert system for beneficiary targeting. CONCLUSION The overall processes for implementing beneficiary targeting, beneficiary verification and CFM are implemented similarly by partners. Partners engage local stakeholders and communities to help identify the most vulnerable using criteria, detailed data are collected and checked for accuracy, and complaint systems provide feedback on how well the first two processes are working. Without hard evidence in terms of which HHs were selected vis-à-vis the criteria and error rates, it is difficult to determine whether a certain approach to forming and engaging committees is more successful at identifying the most vulnerable than another. Extrapolating verification rates provides some clues in terms of how well committees chose beneficiaries, but it is not definitive. Significant time has passed for FSL and WASH interventions, the use of external lists is unknown, and comparing complaint records with locations is not feasible. Comparisons of documents, or lack thereof, identified strong and weak policies and guidance. Supplemental documents and interviews around practices, the evaluation team was able to ascertain good and moderate approaches used by partners. The lack of policies and guidance did not necessarily mean that IPs did not apply certain practices similar to those who did. Yet, one partner that had extensive and thoughtful guidance and SOPs, particularly around beneficiary targeting and CFM, may have been able to implement these processes more successfully because they expended less effort verifying beneficiaries, and investigated and adjusted their final lists because of complaints. They also concentrated their efforts by focusing 80 percent of their 45 assistance on HHs with registered cases of MAM/SAM. Even then, the partner was short on policies around verification, forms most likely hampered verification efforts and staff disagreed about the criteria. Another partner had solid procedures around committee formation and beneficiary targeting and CFM, but in the end did not conduct verification of registration lists. Other partners who were short on guidance employed similar strategies in terms of establishing community committees, focusing on MAM/SAM, and using a sample to verify beneficiaries. Major differences in practices may lie in the size of the target population, the speed required to deliver interventions or region as is the case of food vouchers. However, because guidance was generally applied for all CTP programming and partners noted no regional differences, the evaluation team assumes that this was the case for all programs involving community-based targeting. RECOMMENDATIONS Implementing Partners: Improve policies and guidance. While many procedures are documented, it would be beneficial for partners to review their guidance and fill in gaps, creating roadmaps for field operations. In emergency settings, turnover is high among senior management as well as line staff. When staff depart, so does their valuable knowledge and experience. Providing SOPs for engaging government authorities, particularly in the North, creates uniformity favorable to bureaucracies and prevents breaches in protocol that can have a lasting impact on relations. Documenting procedures for working with local stakeholders, community engagement, CVC formation, nominating processes, elections, training and working with CVCs to affirm or revise criteria and identify beneficiaries would act as a strong support to staff. Templates, checklists and even scenario planning should be included. Further, all partners need to develop clear protocols and data collection plans for verification, including standards for sampling good practices for HH surveys in Yemen or countries with similar terrain that can be adapted for beneficiary verification. MEAL staff should lead these efforts and include detailed procedures in their manuals. To engage women, you need women. Women’s participation in community committees is a worthy goal but may not be achievable in some locations. Partners should engage female staff and/or volunteers for facilitating women’s engagement in community meetings or lead separate meetings for women. One partner used a TOR stressing the importance of engaging women, youth and other target populations and had members sign this. Partners may want to conduct further investigations including focus groups, after-action reviews or even meetings with women after the delivery of aid has begun, in an effort to elucidate additional opportunities to have women participate in elections and serve as a voice for the populations being targeted. Female staff, volunteers and enumerators play an important role in data collection around beneficiary targeting, verification, and CFM. If female staff are unable to travel, partners could consider hiring field enumerators and their husbands to work side by side with staff to collect data or use CHVs, if possible, to help advertise complaint mechanisms and collect feedback. 46 Know your committee members. Illiteracy prohibits women and other representatives of target groups such as the elderly and disabled to participate in community committees. If the local sheikh is a government official and is leading the nomination process or serving as committee chairmen, then there may be little difference between a list developed by the government and a beneficiary list. Informal and formal leaders are essential to implementing these interventions under difficult conditions. However, without a complete understanding of the power dynamics with and across communities, some groups may be favored more than others. Improving guidance surrounding procedures would standardize processes for staff and increase staff awareness of the potential vulnerabilities that can arise when working with local stakeholders. Committee members should be vetted by district officials, staff or by other neutral parties. Definitions matter. A lack of specificity around the vulnerability criteria leaves room for interpretation by whomever is applying the criteria. Vulnerable female-headed HHs with little income, for example, can mean different things to different people. Parameters around “vulnerable”, a common term used in partners’ criteria, can be applied by looking at the exterior of a home, counting the number of family members or determining income. HHs have at times been referred to as families, which implies definitions are interchangeable. Partners also define HHs as a single unit, sharing a kitchen, a meal, or a table. If staff are unclear about terminology, this may also be an issue for committees and data collectors. Removing ambiguity around criteria definitions and their application should improve beneficiary targeting and verification. Definitions should be precise and confirmed among staff before engaging with local authorities. Any changes to terminology or definitions resulting from committee meetings should be documented, translated into Arabic and given to committee members. The term “household” needs to be defined and documented. Transparent beneficiary nomination lists and unannounced registration teams reduce inclusion and exclusion errors, where feasible. Although this may not be feasible in some areas, partners who posted lists or announced names and conducted unannounced registration were more likely to identify inclusion and exclusion errors. Registration and verification should follow the same protocols used for household surveys. Poorly worded instruments and the lack of standards around data collection increases the likelihood of large second-round verifications as data discrepancies exceed thresholds. Registration is the primary data collection effort with everything else, including comparisons with WFP lists, cash, and food voucher distributions, requiring data sets with as few mistakes as possible. Program teams should manage registration and ideally should have MEAL staff (or home office MEAL staff) lead or have input into instrument design. For both registration and verification, practices common to HH surveys such as pre-testing instruments, training enumerators with mock field visits, supervisors leading teams in the field and reviewing forms at the end of the day, creating tally sheets of problems encountered in the field, conducting backchecks (in person or by phone), and reviewing data entry for accuracy are advised. Investing time in structures to conduct registration surveys and verification surveys will be worth the effort if 100 percent of registered beneficiaries no longer need to be verified. Keep registration and verification as separate data collection efforts. Verification checks the accuracy of the data and soundness of data collection. Field verification of registration data should be employed for each cohort of beneficiaries identified to receive assistance in the South. 47 If the North opens and travel remains difficult, partners should identify teams of enumerators to conduct surveys, hire focal points to supervise surveys and use MEAL staff who are unable to travel to conduct back-checks by phone. Sampling design is critical for verification efforts to be successful. Samples should be stratified by village or unit, and HHs chosen at random. Because of the terrain, insecurity, and time, this might not be feasible in certain cases. While 10 percent verification is a figure often quoted for cash programming, accurate verification rates in countries facing humanitarian crisis are difficult to obtain. If registration is done well and CRM systems are working, there may be no need to sample more than 20 percent of registered beneficiaries, as two partners are doing. Because of inaccuracies in RRM and other lists provided by parties other than the committees and the IPs, partners may still have to verify all registered beneficiaries. USAID and partners need to come to an agreement as to what is an acceptable trade-off between accuracy and speed. MUAC measurements and referrals add value to those being targeted as well as those doing the targeting. There is a push among some partners to train mothers to take MUAC measurements of their children, and likewise enumerators or staff conducting registration can be trained to use the MUAC tape. While not all partners are implementing CMAM programs, screening for SAM/MAM while collecting data around vulnerabilities should be considered since it is an unambiguous, verifiable and valuable service. The presence of SAM/MAM is the main requirement for almost all the subsector interventions included in the evaluation. Additional screening may reach those in HHs who have not been diagnosed or may not know where to seek treatment. Since there were no timelines provided for when CU5 and PLWs who had been treated for MAM/SAM were no longer at risk, this became an issue during verification. In addition, some respondents expressed apprehension that tying assistance to active malnutrition cases may cause perverse incentives among HHs. However, the risk of this could be reduced if registration is done quickly after beneficiary nominations. Another option to reduce this risk is to include CU5 and PLWs at risk of malnutrition through MUAC measurements (e.g., 12.5 cm – 13.5 cm for CU5). If eligible beneficiaries are going to be chosen based on weighted scores, partners should do this following nomination. Scoring or assigning points to overlapping criteria will prioritize beneficiaries by vulnerability but can raise expectations among HHs and create tensions in communities. If partners are going to weight scores, it should be done following beneficiary nominations instead of following registration data collection since the latter requires more in￾depth data collection, creating greater expectations than basic lists. In addition, committees should be instructed to collect a small, fixed amount of nominated HHs over caseloads, to minimize expectations. Far reaching CFM systems can reduce exclusion errors. CFM systems should be designed to capture complaints from those being targeted: HHs with children and PLWs with MAM/SAM, female headed HHs, IDPs, etc. CFM staff should be located in field offices, and should ideally be female as women are more likely to provide feedback to women. CFM and MEAL staff should conduct outreach through: hosting regular community meetings to sensitize communities and receive feedback, posting signs in health clinics, designing posters and leaflets for illiterate populations, using loudspeakers and other forms of oral communication, and working with CHVs. Another option is to employ female volunteers as conduits for face-to-face complaints. While it is not advisable that each partner conducts its own assessment, USAID may want to use 48 a third-party monitoring and evaluation platform to conduct an assessment in a few districts. Another option is to include questions about the preferred method of receiving information and cell-phone usage during baseline assessments. USAID: Engage the clusters. Engage with FSAC and the other clusters in order to provide timely and up-to-date information, minimize duplication of assistance through better geographical targeting, improve coordination, and generate more accurate lists. Promote data sharing. USAID should facilitate discussions with partners and in Washington, DC to facilitate the sharing of beneficiary data among partners to help partners avoid duplicating assistance. Pending further investigation, guidance may be needed to distinguish the differences between research subjects, requirements for informed consent, and the level of data protection and procedures for program participants. USAID can facilitate the creation of a common registration form for use across partners. Take a closer look at the Cash Consortium pilot. Using a single system for beneficiary registration and verification was promoted by the heads of FSAC and the Cash Working Group as a means to standardize beneficiary targeting, create efficiencies and reduce costs. Drawbacks discussed with partners include: criticism that the systems created by the UN are only as good as the person in charge and that partners would lose some degree of autonomy if complaints were centralized. If USAID is interested in pursuing this model beyond a pilot, USAID should commission an evaluation so that indicators for measuring effectiveness, efficiency, sustainability, and feasibility, and data collection systems are established early in the process to assess whether common registration and verification systems for CTP should be scaled-up. 49 ANNEX 1: SCOPE OF WORK Yemen Evaluation and Learning Project 7200AA19M00002 Task Two: Evaluation Terms of Reference Evaluation Goal ● To conduct a thematic evaluation of partner beneficiary targeting and verification approaches in Yemen to inform USAID programming. Evaluation Objectives ● Assess NGO partner beneficiary targeting and verification policies and practices in the Yemen context ● Document partner policies and practices for handling complaints about beneficiary targeting in Yemen ● Analyze best practices in North and South Yemen, practices partners should consider utilizing when the operating environment become less restrictive, and practices which partners should avoid Note: This objective of this evaluation is to provide a holistic assessment of policies and practices, rather than a risk assessment or evaluation of individual partners. Overview of Programs to be Evaluated ● USAID has selected eight NGO partners for IBTCI to evaluate. Award information for all of these partners has been previously provided to IBTCI during OY1 startup: ● Partner C (OFDA) ● Partner F (OFDA and FFP) ● Partner B (OFDA and FFP) ● Partner A (OFDA and FFP) ● Partner G (OFDA) ● Partner D (OFDA) ● Partner E (OFDA) ● IBTCI should evaluate the following sectors for beneficiary targeting and verification: ● Food Security ● Nutrition ● Agriculture ● Economic Recovery and Market Systems ● WASH ● MPCA 50 Evaluation Questions ● IBTCI should answer the following questions as part of the evaluation: o What are the current policies and practices for beneficiary targeting? How compliant are IPs with their stated policies? o What are the current policies and practices for beneficiary verification? How compliant are IPs with their stated policies? o How are IPs handling complaints about targeting? o What are the differences between targeting approaches in Northern and Southern Yemen? o What are the best practices in Northern and Southern Yemen? What are practices IPs should avoid? What are policies IPs could enact if/when the operating context becomes more permissive? o Additionally, IBTCI should focus on the role of Community Volunteer Committees in all of these questions. Proposed Methods26 Both qualitative and quantitative methods can be used for this evaluation. This list includes methods USAID believes are appropriate, however, IBTCI should revise and propose additional methods as part of the Evaluation Inception Report. ● Document review and verification: IBTCI should perform a document review of IP stated policies related to community and beneficiary selection, followed by review of supporting documents to see how that policy is being implemented. ● Key Informant Interviews with IP staff, staff of coordinating bodies (such as clusters or working groups), and community leaders, and any other KIs IBTCI feels is relevant o IBTCI has several related questions within their standard site visit tools, which can be used as part of this evaluation, however, IBTCI should also conduct separate interviews to augment this. Interviews can be held by phone or in person. ● Direct observation: Where IBTCI has physical access and consent from IPs, monitors should observe beneficiary verification visits. This should be an observation only role. Monitors should not ask questions at this time. ● Focus Group Discussions: IBTCI should hold FGDs with beneficiaries regarding the selection process and what they understand selection criteria to be. Data Sources ● IP Community Targeting and Beneficiary Selection general policies (if they have written policy) ● Cluster and working group targeting and beneficiary selection general policies and guidance (if written or recorded in meeting minutes) ● IP proposals and work plans ● Key document verification 26 USAID is aware that some of these methods may be delayed due to the current COVID‐19 pandemic and encourages IBTCI to stay in active communication about program changes and timelines. 51 ● Interviews, FGDs, Observation Deliverables ● Per directions of the contract, once IBTCI receives notice from the Contract Officer that Task 2‐ Evaluation has been exercised, IBTCI should produce the following. IBTCI should refer to the Contract for details: o Evaluation Work Plan (within 14 days). This must include: ▪ Timeline ▪ Personnel and Staffing ▪ Budget o Evaluation Inception Report (within 30 days). This will become an annex to the final Evaluation report. The Contractor must provide detailed information related to the following: ▪ Program Overview ▪ Evaluation Framework ▪ Evaluation Design Matrix ▪ Methods Overview ▪ Data Verification and Data Quality ▪ Qualitative and Quantitative Protocol ▪ Qualitative and Quantitative Design ▪ Limitations ▪ Considerations ▪ Staffing ▪ Communications Plan ▪ Data Tools o Conference Call in‐briefs if/when requested by COR o Final Evaluation Report ▪ The report should not exceed 40 pages (excluding annexes and attachments) and should answer the objectives laid out above. Please see the contract for format details. o Final Presentation of Evaluation Report ▪ This may be combined with the IP Workshop in Amman or Yemen, and Final OY1 Report in DC Personnel ● The core evaluation design team needs to be stood up within five business days. This should include the team responsible for the Work Plan and Inception Report. IBTCI may 52 hire additional STTA consultants within available Task 2 budget, however, this does not need to be completed within the five-day design team window. Timeline ● The draft evaluation report will be due to USAID three weeks prior to the IP workshop and Learning Event (NLT November 30, 2020), noting USAID and IBTCI have not yet settled on final dates for those events. ● The draft final presentation for the evaluation will be due to USAID two weeks prior to the IP workshop and Learning Event. 53 ANNEX 2: EVALUATION DESIGN INTRODUCTION International Business & Technical Consultants, Inc. (IBTCI) implements the Yemen Monitoring Evaluation and Learning Project (YMELP) for the United States Agency for International Development (USAID). The purpose of YMELP is to monitor, verify and analyze foreign trends in humanitarian programming in Yemen funded by the Office of Foreign Disaster Assistance (OFDA) and the Office for Food for Peace (FFP). USAID has asked IBTCI to conduct a thematic evaluation27 of partner beneficiary targeting and verification approaches in Yemen. The objectives are described below. This Inception Report is the second deliverable.28 It presents the evaluation framework tied to the evaluation questions, outlines the evaluation methodology and approach, addresses the study’s limitations, and describes communication protocols. It also updates the staffing described in the approved Evaluation Workplan. PROGRAM OVERVIEW For over a decade, Yemen has been roiled in armed conflict and militant insurgencies. The conflict between the Republic of Yemen Government (ROYG) and the northern-based Ansarullah movement took center stage in 2015 when Houthi militants took control over the capital Sana’a. With the ROYG receiving support from a Saudi-led coalition and the Houthis backed by Iran, the war has turned into a proxy battle for the region. Further compounding the devastation has been the recent split between the ROYG and the Southern Transitional Council, a separatist group backed by the United Arab Emirates. More than 100,000 Yemenis have died as a result of the conflict; 23,000 fatalities were reported in 2019 alone.29 Almost 80 percent of the population is food insecure, and almost one-third is at risk of famine.30 In FY 2019, OFDA and FFP committed nearly $100 million to implementing partners (IPs) for the provision of humanitarian aid in Yemen.31 OFDA-funding supports agriculture and food security (AFS), economic recovery and market systems (ERMS), humanitarian coordination and information management, health, logistics support and relief commodities, multipurpose cash assistance (MPCA), nutrition, protection, shelter and settlements (S&S), and water, sanitation 27 According to the Department of State and USAID Glossary of Terms (2009), a thematic evaluation is an evaluation that focuses on a specific cross-cutting area. 28 Task 2 of IBTCI’s contract includes an optional evaluation with four deliverables: Evaluation Work Plan (F.5.1), Evaluation Inception Report (F.5.2), Draft Evaluation Report (F.5.3), and Final Evaluation Report (F.5.4). USAID activated the Task 2 Evaluation Option on May 21, 2020. 29 BBC, “Yemen Crisis: Why is there War?” February 10, 2020, https://www.bbc.com/news/world-middle-east-29319423. Accessed June 15, 2020. 30 Human Rights Watch, “Yemen: Events of 2019”. Retrieved from https://www.hrw.org/world-report/2020/country￾chapters/yemen. Accessed June 15, 2020. 31 According to USAID’s Complex Emergency Fact Sheet, USAID partially suspended $50 million in humanitarian assistance in Al Houthi-controlled areas due to restrictions and bureaucratic restrictions imposed by the government. USAID continues to fund more than $13 million in NGO activities aimed at mitigating the risk of famine and delivering life-saving services. Source: https://www.usaid.gov/sites/default/files/documents/1866/ Yemen_Complex_Emergency_Fact_Sheet_8_06-05-2020.pdf 54 and hygiene (WASH). Food for Peace (FFP) provides funding for cash for food, food vouchers, procurement and food distribution, nutrition and complementary services. The operating environment in Yemen is deteriorating. According to the United Nations Office for the Coordination of Human Affairs (OCHA), 200 of the country’s 333 districts are classified as “hard-to-reach” due to the conflict, regulatory restrictions, interference by the Houthi government, and increased violence against humanitarian partners.32 In March 2020, USAID suspended humanitarian assistance in all Houthi-controlled areas, except for activities mitigating the risk of famine and delivering life-saving service. At the same time, the appearance of COVID-19 cases further exacerbates the ability of partners to deliver aid, as authorities have announced that 1) international NGOs should work from home, 2) crossing governorate borders is prohibited, or 3) trainings/workshops with more than 10 individuals should be cancelled. USAID asked IBTCI to include seven partners in the evaluation which provide humanitarian assistance funded by OFDA and/or FFP: PARTNERS A-G. Tables 1 and 2 provide a summary of their programs by USAID office. TABLE 1: OFDA IP PROGRAM SUMMARY Partner Award Period Amount No. People Targeted Sectors PARTNER A 1/1/19 - 9/30/20 $29.6 M 327,000 AFS, ERMS, Health, Nutrition, WASH PARTNER D 1/2/19 - 6/30/20* $12.5 M 82,000 AFS, ERMS, Protection, WASH PARTNER E 1/1/18 - 9/30/20* $5.5 M 79,100 MPCA, Logistic Support, Protection, S&S, WASH PARTNER B 10/1/18 - 6/30/20* $11.2 M 81,080 AFS, ERMS, WASH PARTNER C 3/1/19 - 9/30/20* $16.7 M 247,688 AFS, ERMS, Health, MPCA, Protection, WASH PARTNER F 12/1/18 - 3/14/20* $18.5 M 996,025 Health, Nutrition, Protection, WASH PARTNER G 8/1/18 - 4/30/20* $3 M 71,910 AFS, ERMS, Health, MPCA, Nutrition, WASH *Awards under review TABLE 2: FFP IP PROGRAM SUMMARY Partner Award Period Amount No. HHs Targeted Sectors PARTNER A 12/1/18 - 9/30/20* $32 M 15,000 HH Food distribution, food vouchers, complementary activities PARTNER B 10/1/19 - 6/30/20* $7 M 6,560 HH Food distribution, food vouchers PARTNER F 10/1/18 - 6/30/20* $39 M 25,487 HH Food distribution, food vouchers *Award under review. 32 OCHA, Yemen Humanitarian Response Plan June – December 2020, Page 10. 55 EVALUATION PURPOSE The purpose of this evaluation is to provide evidence to improve the targeting of humanitarian assistance and verification of beneficiaries selected to receive assistance in Yemen. The evaluation will examine processes, procedures, and practices of a select group of USAID partners in order to inform programming more broadly. It is neither an evaluation of the respective IP’s activities nor a risk assessment. The objectives of the evaluation are to: 1) assess NGO partner beneficiary targeting and verification practices; 2) document partners policies and practices for handling complaints about beneficiary targeting; and 3) identify practices partners should consider utilizing when the operating environment in northern Yemen becomes less restrictive, and practices that should be avoided. The primary audiences for the evaluation are the OFDA and FFP Yemen teams, based in Washington, D.C., Amman, and Budapest. Additional audiences for the report are their partners providing humanitarian assistance in Yemen and the offices’ respective monitoring and evaluation teams. EVALUATION QUESTIONS The terms of references identified five questions. The evaluation team proposes a slight modification to the original set of questions: fold the differences between northern and southern into questions one and two, and reword the fifth question focused on best practices. Although the ordering and wording of these questions vary from the terms of reference, the evaluation approach and methodology used to answer these questions will achieve the evaluation’s purpose and objectives. The revised questions addressed to be addressed study are: 1) What are the current policies and practices for beneficiary33 targeting? How compliant are IPs with their stated policies? Explanation: This question will examine IP policies and practices from community notification of targeted assistance through completion of the final list of beneficiaries. The evaluation will document IPs’ policies, criteria and guidelines for beneficiary targeting based on IP internal documents, award documents and reports to USAID. The second part of this question, based primarily on interviews, will focus on how communities are informed of assistance and their knowledge of selection criteria; the role of community members in beneficiary identification; the use of selection criteria and processes for identifying people who meet those criteria; and how beneficiaries are prioritized and chosen for assistance. The question will also include an examination of the different approaches for northern and southern Yemen. The analysis will address the extent IPs overall were compliant with their policies and criteria. 33 For the purpose of this evaluation, a beneficiary is defined as an individual or household receiving assistance who was selected using a set of criteria established by the IP. 56 2) What are the current policies and practices for beneficiary verification? How compliant are IPs with their stated policies? Explanation: This question will examine the policies, procedures, and systems developed by IPs and the processes used for verifying beneficiaries meet targeting criteria. More specifically, the study will explore the roles and responsibilities within the IP and external entities, if applicable, in verifying beneficiaries; how IPs undertake beneficiary verification based on the criteria; the different approaches to beneficiary verification in male-headed and female-headed houses; and how individuals or households are informed that they do not qualify for assistance. This question will also explore any difference in beneficiary verification practices for northern and southern Yemen. The analysis will assess the extent to which IPs overall were compliant with their policies and criteria. 3) How are IPs handling complaints about targeting? Explanation: All IPs have complaint mechanisms. This question will document the formal and informal mechanisms used specifically for community complaints about beneficiary targeting and selection. It will also examine how communities are notified of complaint mechanisms, who receives and responds to complaints and how information is conveyed within an IP, including documentation on responses to beneficiary complaints. 4) What are the most successful and least successful practices for beneficiary targeting, verification and associated complaint mechanisms in Yemen? Explanation: Based on the findings and conclusions from questions 1 through 3, the question will identify successful and unsuccessful approaches across IPs and explore the reasons why. The analysis will pay particular attention to what works and does not work in southern Yemen and how good practices can be applied to different regions should areas of the country become less restrictive. Based on the answers to the evaluation questions, the evaluation will include recommendations to inform USAID humanitarian assistance programming. EVALUATION APPROACH AND METHODS IBTCI will use a mixed methods approach to the evaluation, relying on document review, key informant interviews, community surveys, and YMELP monitoring and verification data. The analysis will rely largely on qualitative data but will incorporate quantitative data to support findings. See below for the evaluation design matrix (page 65) linking the evaluation questions to data collection and analysis methods. Data Collection Methods Document Review In addition to the program documents (award documents and IP deliverables), IBTCI will collect and review written policies, criteria and guidance developed and protocols and forms used for beneficiary targeting, selection and verification. In addition, IBTCI will review any supporting documents pertaining to implementation. It will also collect procedures and an example of 57 records kept for affiliated complaint mechanisms. Based on the documents reviewed, IBTCI will create a matrix outlining the procedures and criteria used by IPs for different activities for the three focal areas of the evaluation. Key Informant Interviews Key informants for the evaluation will include IP staff from the country office and field offices including: the Country Office Director/Chief of Party (COP), program staff, and monitoring, evaluation, accountability and learning (MEAL)34 staff. Given that organizational structures and staffing varies among the IPs, additional interviews may include staff at a regional or governorate office, in addition to staff outside the program and MEAL teams. Other key informants include members of the OFDA and FFP Yemen team and leaders from the Yemen sector clusters. To elicit frank and honest answers, all interviews will be held in confidence. Because of restrictions in northern Yemen and due to the current COVID-crisis, all key informant interviews will be conducted by phone in English or Arabic. The evaluation team will incorporate quality assurance procedures into qualitative data collection. The team will conduct a set of “round-robin” interviews whereby at least two members alternatively interview key informants at the outset to improve consistency among the interviewers. In addition, semi-structured questionnaires and probes will be reviewed based on an initial set of interviews to confirm responses are robust. The team will seek permission and record all semi-structured interviews using the video conferencing software Zoom. A sample of these recordings will be compared with written notes for thoroughness and Arabic notes will be reviewed alongside the English versions for accuracy. This will be done at the outset of data collection to ensure high quality interview notes are produced during the evaluation. Summaries of interviews and impressions will accompany the interview notes. The Team Leader and/or the Subject Matter Expert (SME) will review them prior to finalization. Authorized recordings of interviews will be stored for reference and deleted following completion of the interview. Telephone Surveys The evaluation will include telephone surveys of community members. The purpose of these surveys is to confirm IP practices and to gain a better understanding of their role in beneficiary target, verification and complaints. The evaluation team has identified four groups of community members: community volunteer committee members, community leaders, community health workers and health facility staff. One of IBTCI’s subcontractors will assemble a team to conduct short confidential interviews in Arabic using a pre-coded form. Following finalization and translation/back translation of the instrument into Arabic, the evaluation team will work with YMELP and subcontractor staff to train the enumerators virtually using Zoom. Training will include a review of the purpose of the survey, interview protocols, scripts and questions, and mock interviews. A survey supervisor will oversee the enumerators. Enumerators will call respondents three times before identifying a replacement respondent. Call￾backs and response rates will be kept daily by the supervisor and provided to IBTCI weekly. One 34MEAL is used as a catch-all phrase for those involved in monitoring, evaluation, accountability and learning functions. The name for these teams will vary among the IPs. 58 of the evaluation team’s monitoring, evaluation and learning (MEL) specialists will back-check 15 percent of the telephone surveys to confirm respondents were interviewed and the interviews were done in an acceptable manner. Any operational issues raised by the subcontractor will be addressed by YMELP’s Deputy Chief of Party (DCOP). Surveys will be paper-based and responses entered into survey monkey or similar online software which will be used for aggregating responses. The survey supervisor will check the surveys for completeness before hand-off to a data entry specialist. A translator will translate Arabic versions of any open-ended questions into English by hand directly into the survey form. One of the evaluation team’s MEL specialist will review the first 20 cases of each survey for accuracy, including any translation of open-ended questions. YMELP Data YMELP collects data pertinent to the evaluation as part of its monthly monitoring and verification. The team will mine qualitative data for two specific lines of inquiry: beneficiary selection and accountability mechanisms. Data sets, in the form of excel tables, will focus on those activities corresponding IP subsectors identified for the evaluation. Considerations A number of considerations shape data collection. First, the study most likely will be done remotely due to COVID-19. The team may shift to in-person interviews should the contracting officer representative (COR) lift remote site visit protocols and ample time permits the shift from remote to field interviews. The proposed evaluation team consists of complementary skills in evaluation, qualitative research, and humanitarian assistance. Team members will be located in Yemen, the Middle East, and the United States to accommodate work hours and work days. The second is language. Three of four team members will be proficient in Arabic and English so that IP field staff can be interviewed in their native language. The third consideration is the security risks to community members. The evaluation team will used mixed gender teams for interviewing beneficiaries and will follow do no harm principles. Sampling Plan The proposed sample for the evaluation is purposeful and designed to create variation among sectors while allowing for patterns to emerge across subsector activities and operating environments. In the terms of reference, USAID identified seven IPs with a range of good, moderate, and weak systems for beneficiary targeting and verification. Based on these parameters, a review of grant documents and input from YMELP, the evaluation team has identified 7 sectors and 13 subsectors for inclusion in the study. Table 3 lists the sector, subsectors and location of activities by IP. 59 TABLE 3: SELECTED SECTORS AND SUB-SECTORS BY PARTNER Partner Sector Subsector Governorates35 North South PARTNER A Nutrition Food vouchers Management of Acute Malnutrition (MAM) N/A Abyan, Al Dhale’e and Lahj PARTNER D ERMS WASH Temporary Employment Sanitation Amran and Hajjah Al Mahwit PARTNER E MPCA N/A Hodeida and Taiz Lahj and Shabwah PARTNER B AFS ERMS Livestock Temporary Employment Dharmar, Ibb, Sana’a, and Taiz PARTNER C AFS MPCA Improving Ag Production N/A Al Dhale’e PARTNER F Nutrition Food vouchers MAM N/A Amran, Hajjah, Hodeida, Sa’ada and Taiz PARTNER G ERMS WASH Temporary Employment Sanitation Sana’a Al Mahwit For the majority of subsectors, the evaluation team will select one governorate, two districts, and four communities for the study. In the case of IPs that are operating in both northern and southern Yemen, the team will select one governorate in each region for the temporary employment and MPCA sub-sectors. Table 4 presents a stratification schematic for identifying districts, communities and community entities involved in for beneficiary targeting, and if applicable, beneficiary verification and complaints. For example, this may be a community volunteer committee (CVC) and community leader, or in the case of nutrition, a community health worker (CHW) and health facility. TABLE 4: SITE SELECTION PER GOVERNORATE Governorate District Community CVCs Governorate 1 District 1 District 2 Community 1 and 2 Community 3 and 4 4 CVCs Based on this plan, the evaluation proposes to conduct key informant interviews with field staff responsible for 32 districts and telephone surveys 64 communities.36 Timing of activities will be an important factor for determining the final set of districts and communities since recall for specific activities will diminish over time. To the greatest extent possible, the evaluation will focus on communities where beneficiary selection occurred after October 2019. The YMELP 35 The governorates identified are those where IPs are currently working. 36 These numbers are an estimate and may vary since IPs’ organizational structures, beneficiary selection and beneficiary verification processes are not uniform. 60 coordinator will work with IPs to obtain a most up-to-date list of sub-sector activities and their location so the team can create the sample. For each district, communities will be oversampled in case replacements are needed. Table 5 details the number of key informant interviews by position. These will be conducted with IP staff,37 cluster sector coordinators, OFDA and FFP staff. Interviews with staff at multiple levels within an organization provide different vantage points in relation to policies and practices. This number of interviews may change depending on the IP’s organizational structure and staffing for the grant(s). TABLE 5: KEY INFORMANT INTERVIEWS # Organization Type of Respondent No. Interviews 1 IP country office • Country Director (7) • Program Staff (13) • MEAL Staff (14) 34 2 IP sub-office • Program Staff (16) • MEAL Staff (16) 32 3 IP field office • Field Staff (32 districts) 32 4 Cluster Sector • Cluster Sector Coordinators 6 5 USAID • OFDA Yemen Team • FFP Yemen Team 4 TOTAL 108 Table 6 lists the total number of community surveys with CVC members and community leaders. Except for nutrition, CVC members are an integral part of beneficiary selection for most of the selected sub-sectors. Because a number of activities are targeted at female beneficiaries and most CVCs are comprised of both men and women, the evaluation team will survey one male and one female member per committee. Interviews with community leaders who are not members of the CVCs will provide an “outsider” perspective in terms of beneficiary selection and verification. TABLE 6: COMMUNITY SURVEYS # Type of Community Respondent Total No. Interviews 1 • Male CVC member • Female CVC member 112 2 • Community leader 56 3 • CHW 8 37 The evaluation team assumes that staff involved in beneficiary targeting, verification and complaint mechanisms at located at different offices within the organization: the country office, sub-regional offices at the governorate level, and field office at the district level. The total number may vary from IP to IP and will be determined following consultations with IPs’ country offices. 61 # Type of Community Respondent Total No. Interviews 4 • OTP site representative38 8 Total 184 Once communities are identified, IBTCI will work with IP field staff to obtain contact information for these respondents. Ideally, we will interview multiple respondents from the same community. If this proves unfeasible, surveys will be conducted until a number is reached for triangulation at the district level. Data Analysis The study will apply within case and cross case designs, using qualitative data from IP documents, key informant interviews, YMELP monitoring and evaluation data, and quantitative data from the community surveys. Within case is chosen for exploring variations in policies and practices and investigating potential reasons for non-compliance. Cross case analysis will be used to compare findings from the IP analysis to identify similar and dissimilar trends and patterns to answer the evaluation questions; this will be done for the entire universe of IPs, and by-subsector and region. Analysis stemming from the cross-case comparison and supplemental data from stakeholders will point to successful and unsuccessful approaches, and inform recommendations for future programs. Because the study is using multiple methods, the evaluation also includes an embedded design: quantitative information from the community surveys will be used to supplement the qualitative analysis. The evaluation will employ rigorous methods to analyze qualitative and quantitative data. IBTCI will create table shells for analysis based on the breakdown of the evaluation questions, data collection methods, sources and proposed analysis in the evaluation matrix (see below evaluation matrix on page 65). All of the evaluation questions will be linked to the data collection instruments. IBTCI will inductively develop a coding structure for qualitative data aligned with the table shells. Early in data collection, the team will jointly review a sample of interview notes to develop an initial coding structure. Evaluation team members will conduct a second review of coded notes to inductively identify additional sub-codes. These codes will be reviewed jointly and agreed upon revisions will be made to the coding list. The evaluation Team Leader and SME will code the remaining notes and meet intermittently to discuss emerging themes and patterns and update the coding structure as needed. Analysis from the coded data, extrapolated data from the document review and descriptive statistics from the quantitative data will be transferred to the table shells for further analysis. Based on convergent findings from the analysis, the team will extrapolate conclusions. Recommendations will be based on conclusions. IBCTI will employ quality assurance procedures for data analysis. On a regular basis, the team will meet virtually throughout the analysis phase to create codes for qualitative analysis, extrapolate findings from the data, review conclusions, and contribute to recommendations. The team, including YMELP staff, will conduct an internal findings-conclusions-recommendations 38 OTP is outpatient therapeutic program. OTPs may be delivered in health facilities or via mobile units. Communities will be identified in consultation with IPs. 62 workshop to review logical consistency and confirm the evaluations questions are answered satisfactorily. Work Planning Meeting and Coordination Once the inception report is approved, the evaluation team will hold a joint review and planning session to review the evaluation approach, design matrix, draft questionnaires and roles and responsibilities. As part of this meeting, the evaluation team will meet with YMELP staff to discuss the operating environment, coordination with IPs, logistics and timeline. Following this meeting, the team will meet at the outset at least once a week to discuss progress and solve issues arising from data collection and analysis. The Team Leader will also meet weekly with the YMELP DCOP to discuss implementation of the evaluation. All documents will be stored in a Google folder on IBTCI’s network. Access will be limited to the evaluation team and YMELP senior management. This will permit team members to access and edit documents simultaneously, and notify the owner of the file of comments and edits. Study Limitations IBTCI has outlined the limitations and risks to the evaluation and proposed mitigation strategies: • IP staff may not have available documentation of processes and procedures. Should there be no documentation, the team will use the criteria identified in the proposal and work with the IP to reconstruct policies and procedures for beneficiary targeting and verification. • For the majority of IPs, their grants are scheduled to end this month and staff affiliated with the grant may no longer be employed. The evaluation team will prioritize governorates where staff are still employed and active or reach out to former staff critical to the assignment. • Because a lot of data collection rests on the timely cooperation of the IPs, the evaluation timeline was built to accommodate delays. • A full-set of interviews for a given IP or any one sector or may not be achieved because of connectivity issues. Interviewers will make up to three attempts for key interviews using the respondents preferred method of communication. Short of an interview by telephone or over the internet, the team will resort to emailing the questions and requesting written responses. • Similarly, the evaluation team may not be able to reach community members for telephone surveys due to connectivity issues. The evaluation team will oversample community members to reach a minimum number of respondents. • IP staff may not be able to transmit documents or photographs of documents over the internet due to accessibility or bandwidth. Pending work from home orders being lifted, the team will continue work with IP senior staff to obtain relevant documents during data collection. 63 • Since IPs are providing the evaluation team with contacts for communities, there may be selection bias. The evaluation team will mitigate this limitation by requesting a larger universe of districts and communities within a governorate from the IP selecting sites independent from them and the YMELP team. • Response bias can occur due to variations in recollections, understanding of the questions posted, fear of repercussions, or desire for future assistance, or other bias that may affect self-reported responses. In addition, community members self-select for interviews, and those who agree to interviews may be different from those who choose not to participate. The sample size for each identified population, assurances of confidentiality, and triangulation of responses should minimize bias. In addition, triangulation of responses will increase the validity of findings. • Researcher bias associated with data collection will be mitigated through training of evaluation team members and enumerators to use the instruments. Communal coding and reviews of analysis by multiple team members, including YMELP staff, should reduce researcher bias. STAFFING Evaluation Team The primary evaluation team will consist of two international experts and a Jordanian/Yemeni monitoring and evaluation specialist. Evaluation Team Leader: Jill Tirnauer is responsible for leading the evaluation, drafting instruments, interviewing key informants, reviewing secondary source data, leading analysis, and drafting the evaluation report. Ms. Tirnauer will also develop the presentation to USAID and IPs, lead the presentation, and incorporate all feedback into the final report. Subject Matter Expert (SME): TBD. The SME will be an integral part of the evaluation team. The SME will contribute to finalizing instruments, conducting key informant interviews, reviewing secondary source data, contributing to analysis and providing input into the evaluation report. S/he will also participate in discussions with USAID an IPs. The SME will have advanced proficiency in English and Arabic, and have humanitarian and evaluation experience. Monitoring, Evaluation and Learning (MEL) Specialist: YMELP will hire one local MEL specialist based in Jordan or Yemen to support the evaluation, while also relying on Huda Al￾Naggar and Mosab Al-Mossabi already working for IBTCI in Sana’a as MEL Specialists to support as needed. They will provide input into the questionnaires, and conduct interviews with IP field staff. The M&E staff will also translate interview guides from English to Arabic, transcribe interviews and translate data from Arabic to English. They will be responsible for back-checking surveys and conducting quality checks. The consultants will also support data analysis and participate in the development of findings, conclusions and recommendations. The MEL Specialists will be proficient in both English and Arabic, and have MEL experience. Involving existing staff is beneficial because they know the program and the IPs, and supplements the additional consultant to ensure we can complete all deliverables. 64 Field and Home Office Support YMELP COP: Richard Columbia will provide overall management, guidance, and direction to the evaluation. YMELP DCOP: Abrahaim Asdodi will liaise closely with the Evaluation Team Leader in DC and subcontractor and YMELP staff in Yemen to facilitate communication with IPs, subcontractor and YMELP field staff and help mitigate challenges that may rise during data collection. YMELP Project Coordinator: Dalia Al-Sorouri. Based on instructions by the Evaluation Team Leader and DCOP, the Project Coordinator with liaise with IPs for information about subsector activities and respondent contact information. YMELP Database Officer: Aiman Noman will assist in compiling YMELP data and preparing data for analysis, including relevant charts and graphics. Home Office (HO) Project Director (PD) Robert Grossman will ensure that the evaluation is of high quality, meets the terms of the SOW, and is delivered on time and within budget. HO Manager Amber James assists with data analysis, compilation of findings, and supports review and copy-edit of deliverables as needed. COMMUNICATIONS The evaluation team will follow the same four-corner approach and protocols detailed in the Option Year 1 Implementation Plan. IBTCI’s home office or the office in Amman will coordinate with OFDA and FFP. In addition to the final presentation, the team will conduct mid￾term evaluation debrief as proposed in the workplan. Following notification to the IPs of the evaluation by OFDA and FFP, YMELP will introduce the evaluation team to the IPs and outline the evaluation. The Evaluation Team Leader and SME will schedule the first interview with the IP’s country director. Following this meeting, evaluation team members will be responsible for contacting IP staff for interviews. Once the subsector activities and locations are finalized, the YMELP coordinator will work with their IP field counterparts to obtain contact information for the community surveys. The evaluation team is aware of the additional burden the evaluation will place on IPs and will set up protocols for information requests and interview scheduling. The YMELP COP and DCOP will be cc’d on all communication with IPs. 65 EVALUATION MATRIX Evaluation Question Evaluation Sub￾Questions Data Collection Method Sources Analysis What are current IP policies and procedures used for beneficiary targeting? (Q1A) What beneficiary identification/selection criteria were used for subsector activities? Document review Key informant interviews IP award documents IP reports IP internal documents IP staff (HQ) Categorization and comparison analysis of themes and patterns using explanatory data within a case and across cases. What are the processes and procedures for beneficiary targeting and initial selection for subsector activities? What are the roles of IP staff and CVCs in beneficiary targeting? How were criteria and processes disseminated among staff and stakeholders? Key informant interviews IP staff (HQ) IP staff (sub￾regional) Categorization and comparison analysis of themes and patterns using exploratory data within a case and across-cases. What changes and why did IPs make to beneficiary targeting over the course of this grant? What are IP practices for beneficiary targeting? (Q1B) What criteria were used to select beneficiaries? Key informant interviews Telephone surveys YMELP data IP staff (sub￾regional) IP staff (field) CVC members/CHWs Local leaders/health facility YMELP perception data on beneficiary selection Triangulation and categorization of data based on themes and patterns within a case using explanatory data. How were beneficiaries targeted and selected? Inductive approach to identifying themes and patterns using exploratory data; integration of quantitative data based on coding structure; comparison analysis of mixed method data within a case and across cases. What were the roles of IP staff in identifying and selecting beneficiaries? Key informant interviews IP staff (sub￾regional) IP staff (field) Triangulation and categorization of data based on themes and patterns within a case using explanatory data. What was the role of CVCs in identifying and selecting beneficiaries? Key informant interviews Telephone surveys IP staff (field) CVC members/CHWs Local leaders/health facilities Inductive approach to identifying themes and patterns and comparison analysis of explanatory data within a case and across cases. In cases where there is more demand for aid than the available amount, how did IPs (and CVCs) prioritize the selection of beneficiaries? IP staff (sub￾regional) P staff (field) CVC members/CHWs Local leaders/health facilities 66 Evaluation Question Evaluation Sub￾Questions Data Collection Method Sources Analysis How knowledgeable were staff and community stakeholders of beneficiary criteria and procedures developed by the IP? Key informant interviews Telephone surveys YMELP data IP staff (field) CVC members/CHWs Local leaders/health facilities YMELP perception data on beneficiary selection Inductive approach to identifying themes and patterns using exploratory data; integration of quantitative data based on coding structure; comparison analysis of mixed method data within a case. How were beneficiaries notified of selection? Key informant interviews Telephone surveys IP staff (field) CVC members/CHWs Local leaders/health facilities Triangulation and categorization of data based on themes and patterns within a case using explanatory data. To what extent do IP targeting practices differ between northern and southern Yemen? n/a n/a Disaggregation of analysis for sub-questions and cross-case analysis. What are current IP policies and procedures for beneficiary verification? (Q2A) What are the policies and criteria used for beneficiary verification? Document review Key informant interviews IP award documents IP reports IP internal documents IP staff (HQ) Categorization and comparison analysis of themes and patterns using explanatory data within a case and across cases. What procedures (e.g., written protocols, data collection forms, quality assurance) do IPs use to verify beneficiary data? What are the roles of IP staff and/or external parties in beneficiary verification? What are IPs approaches for sampling beneficiary verification? To what extent did IPs verify men and women differently? IP program documents IP staff (HQ) How are data on beneficiary verification processed and stored? What changes and why did IPs make changes improve beneficiary verification during the course of this grant? Why? Key informant interviews IP (HQ) Categorization and comparison analysis of themes and patterns using exploratory data within a case and across-cases. What are IP practices for verifying beneficiaries? (Q2B) How were beneficiaries verified for subsector activities? Key informant interviews YMELP data IP staff (sub￾regional) IP staff (site) Inductive approach to identifying themes and patterns using exploratory data; integration of quantitative data based on coding structure; comparison analysis of mixed method data within a case and across cases. What were the roles of IP staff and external Key informant interviews IP staff (sub￾regional) IP staff (field) Categorization and comparison analysis of themes and patterns using 67 Evaluation Question Evaluation Sub￾Questions Data Collection Method Sources Analysis stakeholders involved in beneficiary verification? Telephone surveys CVC members/CHWs exploratory data within a case and across-cases. What protocols and/or forms were used for verifying beneficiaries? Key informant interviews Document review IP staff (sub￾regional) IP staff (field) IP internal documents Triangulation and categorization of data using explanatory data within a case. How were data stored following beneficiary verification? How familiar were staff and/or external stakeholders with IP procedures? Key informant interviews Telephone surveys IP field staff (sub￾regional) IP staff (field) CVC members Local leaders Inductive approach to identifying themes and patterns and comparison analysis of explanatory data within a case. How were unverified/not qualified individuals/households informed they would not receive assistance? To what extent do IP beneficiary verification practices differ from northern Yemen? n/a Data collected by the evaluation team. Disaggregation of analysis for sub-questions and cross case analysis. How compliant are IPs with their stated policies for beneficiary targeting/beneficiary verification? (Q1C/Q2C) How closely are IPs adhering to beneficiary selection criteria? n/a Analysis from Q1 Cross case comparison of summative analysis for Q1A and Q1B. How closely are IPs adhering to stated policies for beneficiary targeting/beneficiary verification? n/a Analysis from Q2 Cross-case comparison of summative analysis for Q2A and Q2B To what extent did compliance differ between northern and southern Yemen? n/a YMELP data Analysis from Q1 and Q2 YMELP data (AMI2-IP-3, AMI2-IP-4) Cross-case comparison of summative analysis by region. What factors may have played a role in the gap between policies/procedures and practices? n/a Analysis from Q1 and Q2 Trend analysis specifically focusing on regions (north/south), type of activity, and role of community committees. How are IPs handling complaints about beneficiary targeting? (Q3) What mechanisms are available for individuals and/or households to provide feedback on targeting or appeal decisions? Document review Key informant interviews YMELP data IP award documents IP reports IP internal documents IP staff (HQ) IP staff (sub￾regional) IP staff (field) Health facility staff YMELP perception data on accountability mechanisms Triangulation and categorization of qualitative and quantitative explanatory data within a case and across cases. 68 Evaluation Question Evaluation Sub￾Questions Data Collection Method Sources Analysis Who is responsible for responding to individuals/households complaining about targeting or appealing decisions about targeting? Key informant interviews IP staff (HQ) IP staff (sub￾regional) IP staff (field) Health facility staff Triangulation and categorization of explanatory data within a case and across cases. How do IP field staff and/or CVCs handle complaints? Key informant interviews Telephone surveys YMELP data IP records IP staff (sub￾regional) IP staff (field) CVC members/CHWs Local leaders/health facilities YMELP perception data on accountability mechanisms Inductive approach to identifying themes and patterns using exploratory data; integration of quantitative data based on coding structure; comparison analysis of mixed method data within a case and across cases. If complaints are made to IP field staff or CVCs, how are others within the organization informed? Key informant interviews IP staff (HQ) IP staff (sub￾regional) IP staff (field) Triangulation and categorization of explanatory data within a case and across cases. What are the most successful practices and least successful practices for beneficiary targeting, verification and associated complaint mechanisms? (Q4) What are the most successful practices for beneficiary targeting/verification/ complaint mechanisms in northern and southern Yemen? Key informant interviews n/a Sector cluster leaders OFDA/FFP staff Summative analysis for Q1, Q2, and Q3 Inductive approach to identifying themes and patterns and comparison analysis of exploratory data; integration of summative analysis for Q1, Q2, and Q3 disaggregated by region What are the least successful practices for beneficiary targeting/verification/ complaint mechanisms in northern and southern Yemen? Key informant interviews n/a Sector cluster leaders OFDA/FFP staff Summative analysis for Q1, Q2, and Q3 Inductive approach to identifying themes and patterns and comparison analysis of exploratory data; integration of analysis with Q1, Q2, and Q3 disaggregated by region. 69 ANNEX 3: LIST OF KEY INFORMANT INTERVIEWS (REDACTED) PARTNER A Country Director Director of USG Programs Sector Award Project Manager Sector Project Manager Sector Field Officer Sector Field Officer Sector Specialist Sector Specialist Sector Officer Sector Officer Program Quality Manager MEAL Officer PARTNER B Country Director Area Manager Program Manager Sector Officer Sector Officer MEAL Manager MEAL Specialist MEAL Reporting Officer Community Mobilizer Community Mobilizer Community Mobilizer PARTNER C Country Director Coordinator Sector Manager Sector Officer MEAL Coordinator MEAL Officer MEAL Assistant CFM Manager CFM Officer PARTNER E Country Director Former Sector Director Quality Assurance Manager Former Sector Team Member 70 MEAL Team Leader CFM Officer MEAL Officer IM Team Leader Partner F Deputy Country Program Manager Sector Advisor Sector Project Director Sector Specialist Sector Manager Sector Advisor Roving Sector Program Manger Roving Sector Program Manager Sector Program Manager Director of Program Development & Quality MEAL Lead MEAL Coordinator CFM Officer CFM Assistant CFM Assistant Partner G MEAL Coordinator MEAL Officer MEAL Officer Manager of Program Quality Program Manager Sector Lead Sector Specialist Senior Sector Officer Sector Technician Sector Specialist Sector Field Officer USAID C. Fisher, Humanitarian Affairs Officer J. Miller, Risk Management Specialist A. Savage, Field Planning Coordinator J. Balba, Field Planning Coordinator Expert Interviews W. Darwish, Head of Cash Consortium G. Dudi, FSAC Coordinator M. Al -Falahi, Roving Nutrition Cluster Coordinator S. Rabeea Ahmed, Senior Cash Advisor 71 E. Tuck, WASH Cluster Coordinator 72 ANNEX 4: DATA COLLECTION TOOLS (KEY INFORMANT INTERVIEWS) IP COUNTRY DIRECTOR – SENIOR MANAGERS Interview Date: Interviewer Initials: Respondent Name: Respondent Title: Organization: Contact Information: Sub-sector(s): Region(s): My name is [Insert Name(s)], I am part of a team from IBTCI conducting a thematic evaluation of approaches used for beneficiary targeting, beneficiary verification and complaint mechanisms for USAID humanitarian programs in Yemen pre-COVID 19. As part of our evaluation design, we will be looking holistically across 7 partners and 13 subsectors. In the case of [name of organization], we will be focusing specifically on [name of sub-sector] and [name of sub-sector]. The purpose of this meeting is to answer any questions you may have about the evaluation, get an overview of BT/BV and CMs, get a better understanding of your office structure and staff involved, and challenges your organization may have faced in carrying out these functions since the award of the grant. The interview should take no more than 50 minutes. All interviews are confidential and anything you tell us will not be attributed to you. All information you provide will not be shared beyond the team and all members have signed non-disclosure agreements. We expect to have a draft report to USAID by the third week of November. The results of this evaluation will be shared with partners in November. We will be recording this interview for our records only. The only people who will have access is the evaluation team. Do we have permission to record this interview? [Circle one: Yes / No] CD-Q1. Please describe the processes and procedures for beneficiary targeting following selection of the site through beneficiary verification? Are these the same for all OFDA and/or FFP activities? [If not] How are they different and what are the reasons for the different approaches? Are they the same across other donor programs funding similar activities? CD-Q2. [If applicable] Are these processes and procedures you described the same for programs in the north that target beneficiaries? If different, how are they different and what is the reason for these different approaches? CD-Q3. [Senior staff only] Who decides on what the criteria are for selecting beneficiaries? How do you prioritize beneficiaries? CD-Q4. How are complaints about beneficiary targeting or appeals from community members addressed? Is this information collated across activities and subsectors? CD-Q5. Which country offices, governorate and field offices are involved in beneficiary targeting, beneficiary verification and complaint mechanisms? Which staff are involved? [Confirm all offices and positions] CD-Q6. Have you faced any difficulties in targeting beneficiaries has [name of IP] faced during implementation of the OFDA and/or FFP grant? [Probe: security, authority interference, local pressure]. What did you do to overcome these obstacles? CD-Q7. Have you faced any difficulties in verifying beneficiaries during implementation of the OFDA and/or FFP grant? [Probe: security, access, number of households being disqualified for not meeting criteria] CD-Q8. Have you made any explicit changes to processes and procedures for beneficiary targeting or beneficiary verification since the award of the grant? If yes, what were these changes? CD-Q9. Are these processes and procedures similar to those used by [name of IP] in other countries? [If yes] Are these universal policies? [If no] Why do you think that is the case? 73 CD-Q10. Do you have any questions about the evaluation? Who else should we speak to about beneficiary targeting, beneficiary verification or complaint/feedback mechanisms? Do you have SOP manuals and/or policies from HQ for beneficiary targeting that you can send us? End of Interview: Thank you for your time. MEAL DIRECTOR (COUNTRY OFFICE) / STAFF INVOLVED IN BV AND CMS A. Beneficiary Targeting MEAL-QA1. Please describe the processes and procedures for beneficiary targeting and selection up until the list of eligible beneficiaries for [name of subsector activity]? [Excluding nutrition programs] Are these the same or different for [name of subsector activity 1] [and name of second subsector activity 2]? [If different] How they are different? B. Beneficiary Verification MEAL-QB1. Please describe the processes and procedures for beneficiary verification for [name of subsector activity] through finalization of the beneficiary list? Are these processes documented? Are these the same or different for different OFDA and/or FFP activities? [If different] How are they different? [Nutrition ONLY] Does the MEAL team verify beneficiaries receiving treatment for malnutrition for any reason? Please explain. MEAL-QB2. [If applicable] Are these procedures the same for northern and southern Yemen for [name of subsector activity] and [name of second subsector activity]? [If different] How are they different? MEAL-QB3. Are beneficiary verification processes different for male-headed households than they are for women￾headed households? [If different] How are they different? MEAL-QB4. What is your role in the beneficiary verification? [MEAL DIRECTOR ONLY] What is the role of MEAL staff in beneficiary verification and final selection? [Probe: who and where they are located] MEAL-QB5. What percentage of beneficiaries do you sample? Are these the same or different for [name of subsector activity] and [name of second subsector activity]? [If different] How are they different? MEAL-QB5A. What happens when you exceed the threshold of non-verified beneficiaries? Are there situations in the past where you have conducted 100% verification of beneficiaries? Why? MEAL-QB6. How do you verify whether or not a beneficiary is receiving humanitarian assistance through a different program? MEAL-QB7. What happens if your team finds that a beneficiary does not meet the criteria? When is the beneficiary informed that they didn’t meet the criteria? MEAL-QB8. Did you encounter difficulties verifying beneficiaries during the grant period? If yes, please explain. MEAL-QB9. Where is beneficiary verification information stored? What protections are in place to secure the data? MEAL-QB10. Have beneficiary verification processes and procedures changed since the award of the grant? If yes, what were these changes and why were these changes made? MEAL-QB11. What can be done to improve beneficiary verification? 74 Complaint Mechanisms MEAL-QC1. Who is responsible for receiving, recording and responding to complaints about not being targeted or selected for assistance? [NUTRITION ONLY] Is this the same in the case of complaints of not being visited or diagnosed with malnutrition? [If not] Why not? MEAL-QC2. Are the complaint mechanisms used for community feedback for beneficiary selection the same as those used for other complaints? Are complaint mechanisms the same across grant subsectors? Are the same across donors? MEAL-QC3. How do community voice their concerns about beneficiary selection? [Probe: formal and informal mechanisms, e.g., hotline, complaint box, FGDs, walk-ins to IP office, staff member, CVCs, community leaders] [If applicable] Are complaint mechanisms and the response different for northern Yemen and southern Yemen? [If different] How are they different? MEAL-Q3A. How are complaints about not being received by the local committees or local leaders handled? Are they equipped to handle these complaints? MEAL-QC4. How are beneficiaries able to file complaints if they live in areas too remote, don’t have a cell phone or able to physically a3cess established complaint mechanisms? [Probe: women, children/elderly/disabled] How does [Name of IP] solicit their concerns? MEAL-QC5. What system is in place to record, analyze and respond to beneficiary complaints around targeting? What is the flow of information regarding complaints from the field to the country office? MEAL-QC6. How do you know that a complaint has been addressed? Does the MEAL team or anyone else in the organization verify [Name of IP] response to beneficiaries’ complaints? If so, is the quality of the response considered? MEAL-QC7. How do you know the complaint mechanisms are working as planned? MEAL-QC8. Are complaint records or registry used for any type of analysis or reporting? Please explain. MEAL-QC9. What can be done to improve how it addresses complaints? End of Interview: Thank you for your time. If we have any questions, may we contact you again? PROGRAM SECTOR AND SUBSECTOR STAFF (COUNTRY OFFICE) – EXCLUDING NUTRITION A. Beneficiary Targeting PS-QA1. Please describe the processes and procedures for beneficiary targeting and registration for the [name of subsector activity] Is this process the same across activities where there are stated criteria to receive assistance? [If applicable] Is it the same for northern and southern Yemen? [If not] How are they different? PS-QA2. What is the role of your program staff in beneficiary targeting and registration? [Probe: who and where they are located] PS-QA3. What was the basis for the beneficiary criteria? Did the overall criteria used to identify beneficiaries change from the criteria in the proposed? [If yes] Why? PS-QA3A. Would it be possible to obtain a copy of the criteria used for [name of subsector activity]? PS-QA4. How are communities informed of the selection criteria for [name of subsector activity]? PS-QA5. How do you get community input into identifying households that meet the selection criteria? 75 PS-QA5A. How are local committee members typically chosen? Were new committees created for the grant? PS-QA5B. How are local committee members informed of beneficiary selection criteria and procedures? [If provided written criteria] Were the criteria provided in English or Arabic? PS-QA6. Do communities or the local committees have input into the criteria for their community? [If yes] does it apply to all communities receiving assistance or only to the communities where the adjustment was made? PS-QA7. [If beneficiaries are registered before verification] How are the registration records recorded and stored? What protections are in place? PS-QA8. If there are more beneficiaries than available assistance, how do you prioritize who receives assistance? PS-QA9. How are targeted beneficiaries told they will not receive assistance. If they are not told, why not? PS-QA10. Have there been changes to how beneficiaries are targeted or the registration process during the grant period? Please explain. PS-QA11. What can [name of IP] do to improve beneficiary targeting and selection? B. Beneficiary Verification PS-QB1. What role does the program team (i.e., technical staff) play a role in beneficiary verification? PS-QB2. Who determines what percentage of beneficiaries are verified? [If the program team is involved] Did it change during implementation? C. Complaint Mechanisms PS-QC1. How do community voice their concerns or complain about beneficiary selection? [Probe: formal and informal mechanisms, e.g., hotline, complaint box, community meetings, walk-ins to the IP office, CVCs, community leaders] How do most beneficiaries complain about beneficiary targeting? PS-QC2. How are beneficiaries able to file complaints if they live in areas too remote, don’t have a cell phone or able to physically access established complaint mechanisms? [Probe: women, children/elderly/disabled] What does [Name of IP] do to solicit their feedback? PS-QC3. Do program staff respond to complaints about community targeting? [Probe: who and where they are located based on the mechanism] PS-QC4. How are you informed of these complaints and the action taken? PS-QC5. Does anyone follow-up on staff’s response to concerns? PS-QC6. How are complaint statistics or analysis used by the program team? PS-QC7. What can [name of IP] do to improve how it addresses beneficiary complaints or concerns? End of Interview: Thank you for your time. If we have any questions, may we contact you again? 76 PROGRAM SUBSECTOR STAFF (COUNTRY OFFICE): NUTRITION PSN-Q1. Please describe a brief overview of your community management of acute malnutrition (CMAM) program. PSN-Q2. Please describe the role of [CHWs/community volunteers] in determining whether a person is suffering from malnutrition. What is the process s/he goes through from selecting a household, making a diagnosis and making a referral? PSN-Q3. What happens once a person is diagnosed in the case of malnutrition ? [Probe: name of facility, location and treatment] What happens once a person is diagnosed in the case of SAM? [Probe: name of facility, location and treatment] PSN-Q4. How do [CHWs/community volunteers] determine which communities to visit to screen for nutrition? How many communities are they responsible for? How often do they visit a community? PSN-Q5. When visiting a community, do [CHWs/community volunteers] visit all households or specific households? PSN-Q5A. [If they visit specific households] How do they identify specific households? [Probe: pre-determined criteria or priority households] PSN-Q5B. [If there are criteria for selecting households] How were they informed of these criteria? [Probe: training, policies/procedures] PSN-Q6. Are there written process and procedures for [CHWs/community volunteers] selecting communities and households? [If applicable] Would it be possible to obtain [a copy/copies]? PSN-Q7. Do [CHWs/community volunteers] keep written records of the households they visit, and the people they refer to treatment? [If yes] Who gets copies of these records? PSN-Q8. How does [name of IP] ensure [CHWs/CVCs] are correctly diagnosing cases of MAM and SAM? PSN-Q9. How does [name of organization] know that if a referral seeks treatment? Is there follow-up? [If yes] Is this recorded? [If yes] Who has these records? PSN-Q10. Who in your organization works directly with [CHWs/community volunteers]? [Probe: Who and where they are located] PSN-Q11. How do communities or households voice their concerns about [CHWs/community volunteers] visits (e.g., frequency, not showing up?) or their concerns about not receiving proper diagnosis or treatment for malnutrition? [Probe: formal and informal mechanisms, e.g., complaint box, hotline, surveys, staff member, health clinics, community leaders] PSN-Q12. How are beneficiaries able to file complaints if they live in areas too remote, don’t have a cell phone or able to physically access established complaint mechanisms? [Probe: women, children/elderly/disabled] What does [Name of IP] do to solicit their feedback? PSN-Q13. Are you informed of these complaints? If so, how are you notified? Are there records of these complaints? PSN-Q14. Who responds to these complaints? Are there records of responses to complaints? PSN-Q15. How do you know that a complaint has been addressed? PSN-Q16. Did the [CHWs/community volunteers] encounter any difficulties identifying households suffering from malnutrition? [If yes] please explain. 77 PSN-Q17. Have processes around selecting community and identifying households changed since the award of the grant? [If yes] How? PSN-Q18. What could be done to improve identifying households for nutritional screening? End of Interview: Thank you for your time. If we have any questions, may we contact you again? IP STAFF (FIELD LEVEL) – EXCLUDING NUTRITION FS-Q1. In the case of [name of subsector activity], when did the beneficiary selection process take place? Is the activity still being implemented? If not, when did it end? FS-Q2. Please describe the process for targeting and selecting individuals or households for [name of subsector activity] from the point a community is identified to creation of the list of beneficiaries for verification. What was your role in this process? [Probe: who is involved] FS-Q3. What is the role of [community groups/CVCs] in targeting and selecting beneficiaries for [name of subsector activity]? FS-Q4. How were members of these [community groups / CVCs] selected? FS-Q5. How were [community groups / CVCs] informed of the selection criteria for [name of subsector activity]? [Probe: conversation, written criteria] [If written] Were these provided in English or Arabic? FS-Q6. How does [name of IP] ensure [community groups/CVCs] are selecting beneficiaries that meet the criteria? FS-Q7. Were there any challenges to identifying and selecting beneficiaries for assistance? If yes, what were they? FS-Q8. Do you think the assistance provided for [name of subsector activity] is reaching everyone who meets the criteria? If not, why? FS-Q8A. [If it doesn’t reach those in need] How are beneficiaries prioritized if there are more people with needs than there is assistance being provided? FS-Q9. Were beneficiaries verified for the communities you manage? If yes, how were they verified and by whom? FS-Q10. [If applicable] Were there any difficulties faced during beneficiary verification? If yes, please explain. [Probe: male/ female-headed households] FS-Q11. To what extent were beneficiary verification procedures for [name of subsector activity] adequate? FS-Q12. How do communities voice their concerns about beneficiary selection? [Probe: formal/informal; direct feedback/hotlines/complaint boxes/other] FS-QC13. Are you aware of concerns or complaints about beneficiary selection? FS-QC13A. [If yes] How are concerns or complaints recorded? FS-QC13B. [If yes] Who responds to complaint about beneficiary selection? [Probe: who and where they are located based on the mechanism] FS-QC13C. [If someone other than the respondent responds] Are you informed of the response? If yes, how are you notified? 78 FS-QC13D. [If respondent responds to complaints] Do you record the response? If yes, what happens to this information? FS-Q14. Do you have any suggestions for improving beneficiary selection, verification and/or complaints can be improved? End of Interview: Thank you for your time. If we have any questions, may we contact you again? IP STAFF (FIELD LEVEL) – NUTRITION FSN-Q1. What is your role in the community-based management of malnutrition activity? FSN-Q2. Please describe the role of [CHWs/community volunteers] in determining whether a person is suffering from malnutrition. What is the process s/he went through from selecting a household, making a diagnosis and making a referral? FSN-Q3. What happens once a person is diagnosed in the case of MAM? [Probe: name of facility, location and treatment] What happens once a person is diagnosed in the case of SAM? [Probe: name of facility, location and treatment] FSN-Q4. How do [CHWs/community volunteers] determine which communities to visit to screen for nutrition? How many communities are they responsible for? How often do they visit a community? FSN-Q5. When visiting a community, did [CHWs/community volunteers] visit all households or specific households? FSN-Q5A. [If they visit specific households] How do they identify specific households? [Probe: pre-determined criteria or priority households] FSN-Q5B. [If there are criteria for selecting households] How were they informed of these criteria? [Probe: training, policies/procedures] [If written] were they provided the criteria in English or Arabic? FSN-Q6. Do you think the nutrition screening conducted by [CHWs/community volunteers] reaches everyone in their communities who meet the criteria and are potentially suffering from malnutrition? FSN-Q7. Do [CHWs/community volunteers] keep written records of the households they visit, and the people they refer to treatment? [If yes] Are you provided copies of these records? FSN-Q8. How does [name of organization] know that if a referral seeks treatment? Is there follow-up? [If yes] Who follows up? [If yes] Are there records of these visits? FSN-Q9. How accurate are the [CHWs/community volunteers] in diagnosing MAM and SAM? FSN-Q10. How do communities or households voice their concerns about [CHWs/community volunteers] visits (e.g., frequency, not showing up?) or their concerns about not receiving proper diagnosis or treatment for malnutrition? [Probe: formal and informal mechanisms, e.g., complaint box, hotline, surveys, staff member, health clinics, community leaders] FSN-Q11. How are beneficiaries able to file complaints if they live in areas too remote, don’t have a cell phone or able to physically access established complaint mechanisms? [Probe: women, children/elderly/disabled] How do you collect their feedback? FSN-Q12. Were you or someone in your organization informed of these complaints? If so, how were you notified? Are there records of these complaints? 79 FSN-Q13. Who responded to these complaints? Are there records of responses to complaints? FSN-Q14. Are households diagnosed with MAM or SAM eligible for another type of humanitarian assistance delivered by [name of IP]? If yes, which programs? FSN-Q14A. [If applicable] How are beneficiaries verified? Who is involved? FSN-Q14B. [If applicable] Has [name of IP] encountered any difficulties verifying these beneficiaries? If yes, please explain. FSN-Q15. Did the [CHWs/community volunteers] encounter any difficulties identifying households for nutrition screening or making referrals? [If yes] please explain. FSN-Q16. Have you changed your approach to community-based management of malnutrition since the award of this grant? How? FSN-Q17. What could be done to improve identifying households for nutritional screening? End of Interview: Thank you for your time. If we have any questions, may we contact you again? CLUSTER LEADERS / OFDA / FFP STAFF E-Q1. How long have you been working in the humanitarian sector in Yemen? In humanitarian assistance in general? E-Q2. How do different INGOs set beneficiary targeting criteria? E-Q3. Is there much variation among the INGOs for targeting beneficiaries in [name of sector/all sectors in general]? If yes, why? E-Q4. From your experience, what are the most successful practices used by INGOs for targeting beneficiaries? Do these practices apply for both [southern Yemen and northern Yemen / permissive and on-permissive environments]? If not, how would these best practices differ for these two [regions/situations]? E-Q5. From your experience, what are the least successful practices used by INGOs for targeting beneficiaries? To what extent do these practices apply to southern and northern Yemen? E-Q6. From your experience, what are the most successful practices used by INGOs for verifying beneficiaries? Do these practices apply for both [southern Yemen and northern Yemen / permissive and on-permissive environments]? If not, how would these best practices differ for these two [regions/situations]? E-Q7. From your experience, what practices should INGOs avoid when verifying beneficiaries? To what extent do these practices apply to southern and northern Yemen? E-Q8. If a beneficiary had a complaint about targeting, what processes and systems should an INGO have in place to address them? E-Q9. Is there anything else you would like to add about improving these processes in Yemen? End of Interview: Thank you for your time. 80 ANNEX 5: COMMUNITY SURVEY TOOLS (REDACTED) COMMUNITY COMMITTEE MEMBER Question Answer Members of community committees Researcher's name (REQUIRED) Interviewer_name Researcher's name – other (REQUIRED) Interviewer_name_Other Name of the implementing partner: (REQUIRED) Interview date: (REQUIRED) Name of the person interviewed: (REQUIRED) Job title of the person interviewed: (REQUIRED) Telephone number: (REQUIRED) Name of the health facility: (REQUIRED) Subsector: (REQUIRED) Region: (REQUIRED) Directorate (REQUIRED) Isolation (REQUIRED) Name of the interviewer: (REQUIRED) My name (name of person conducting the interview), from an independent ________ (organization) that is assessing how communities are involved in identifying beneficiaries of USAID-funded humanitarian programs and how complaints about donor assistance are handled. We got your name from (Name of Implementing Partner) and we were informed from (Name of Implementing Partner) that you were a member of a community committee that was determining beneficiaries to receive (Name of Intervention). 81 Question Answer Q1: Are you a member of the community committee? (Refer to one of the answers below) [Mark one response below.] 1 Yes 2 No 3 I don't know 4 No response This interview will take no more than 20 minutes. There are no right or wrong answers. All interviews we conduct are confidential and nothing you tell us will be attributed to you. All interview data are protected. The results of the interview will not affect whether or not your community will receive assistance in the future. We will use the information we have obtained from you and other members to improve the way aid is delivered. Do you agree to do this interview? Q2. Did the community committee attend a meeting together in the past months? 1 Yes 2 No 3 I don't know 4 No response Q3. How many members of the local committee? (Fill the void) Q4. How did I become a member of the local committee? (Do not read the answers. Refer only to one answer) 1 Appointed by a local authority 2 Appointed by a Sheikh 3 Appointed by the [name of IP] 4 Elected by the community 5 I asked to join the committee 6 As a local leader, I am part of the committee. 7 Other: [Fill in space provided] 8 I don't know / don't remember 9 No response 82 Question Answer Q5. Going back to the last time the local committee met to define the criteria on the basis of which the beneficiaries will be targeted to receive assistance. What type of assistance was provided? (Do not read the answers, just indicate what has been mentioned) 1 Food vouchers 2 Food basket 3 Cash assistance 4 Work for pay / Cash for work 5 Livelihoods such as seeds, tools, livestock, for example 6 Support for small business 7 Shelter 8 Shelter NFIs 9 Household latrines 10 Household water storage containers and other WASH NFIs 11 Hygiene NFIs 12 Other (please write in 13 I don't know 14 No response Q 6. To whom was this assistance provided? (I read the options below, except I don't know / No answer) Please indicate all that was mentioned 1 Local community 2 IDPs 3 Marginalized groups 4 I don't know / don't remember 5 No response 83 Question Answer Q 7. Going back to the last time the local committee met to discuss the criteria for determining the beneficiaries of assistance, when was that? (Read the answers except I don't know / No answer) Mark everything that is mentioned 1 Less than 3 months ago 2 Between 3 and 6 months ago 3 Within the last six months 4 Between six months and one year 5 Longer than one year 6 I don't know / don't 'remember 7 No response Q8. Going back to the last time the local committee identified the beneficiaries, were you provided with a list of names? (Indicate only one answer) 1 Yes 2 No 3 I don't know 4 No response Q 8 a. Who provided you with this list? Do not read the answers. Refer to everything that is mentioned 1 [Name of IP] 2 Government / local authorities 3 Sheikh 4 Another member of the committee 5 Other (Please write in) 6 I don't know / don't remember 7 No response Q 9. Going back to the last time the community committee identified beneficiaries, were there criteria for selecting the beneficiaries? (Indicate only one answer) 1 Yes 2 No 3 I don't know 4 No response 84 Question Answer Q 9 A. Who informed you about these criteria? (Read the answers except I don't know / No answer) Mark everything that is mentioned 1 Name of IP 2 Government / local authorities 3 Sheikh 4 Another member of the committee 5 Other (Please write in) 6 I don't know / don't remember 7 No response Q 9 B. Was the community involved in developing or changing the criteria for who is eligible to receive assistance? (Tick for one answer only) 1 Yes 2 No 3 I don't know 4 No response Q 9 C. Were you trained by the [name of IP] on how to identify beneficiaries? [Mark one response below.] 1 Yes 2 No 3 I don’t know 4 No response 85 Question Answer Q 10. Thinking back to the last time the community committee identified beneficiaries for assistance, how were they identified? [Do not read response categories. Mark all that apply.] 1 Through household visits 2 Through referral from other programs 3 Community meetings 4 Recommendations from local authorities 5 Recommendations from Sheikh Al 6 Recommendations from other local committee members 7 Recommendations from members in the community 8 Other (Please write in) 9 No response 10 Don’t know / don’t remember Q 11. Thinking back to the last time the community committee met to identify beneficiaries, did committee members visit households to get information about beneficiaries identified for assistance? [Mark one response below.] 1 Yes 2 No 3 I don’t know 4 No response Q 12. Did you know which households would receive assistance before it was given? [Mark one response below.] 1 Yes 2 No 3 I don’t know 4 No response Q 13. Were you told which households wouldn’t receive assistance? [Mark one response below.] 1 Yes 2 No 3 I don’t know 4 No response 86 Question Answer Q 13A. Were you told why these households wouldn’t receive assistance? [Mark one response below.] 1 Yes 2 No 3 I don’t know 4 No response Q 14. Thinking back to the last time you identified beneficiaries and assistance was given, to what extent did the beneficiaries who received assistance met the criteria? [Read response categories except don’t know/no response. Mark one response below.] 1 All of them met the criteria 2 Most of them met the criteria 3 Some of them met the criteria 4 None of them met the criteria 5 Don’t know / don’t remember 6 No response Q 14A. If they did not meet the criteria, why? Q 15. Thinking back to the last time you identified beneficiaries and assistance was given, were you informed by the [name of IP]/donor what to do if someone complained to you? [Mark one response below.] 1 Yes 2 No 3 I don’t know 4 No response Q 16. Thinking back to the last time assistance was given, did the [name of IP]/donor explain to the community how to make complaints or provide feedback? [Mark one response below.] 1 Yes 2 No 3 I don’t know 4 No response 87 Question Answer Q 17. What are the ways or channels for a community to register their complaints or provide feedback about the assistance given by the donor? [Do not read response categories. Mark all that apply.] 1 Hotline / telephone number 2 WhatsApp 3 SMS 4 Email 5 Suggestion box 6 Help desk at the place assistance was given 7 Go to the community committee 8 Speak with the organization’s employee 9 Go to a meeting 10 Other (please fill in) 11 Don’t know / don’t remember 12 No response Q 18. Has anyone complained to you about nor receiving assistance or how beneficiaries were chosen? [Mark one response below.] 1 Yes 2 No 3 I don’t know 4 No response Q 18A. Please explain. Q 19. Did you do anything of the following to resolve the complaint? [Read response category. Mark all that apply] 1 I solved the issue myself 2 I told someone else 3 Other (please fill in) 4 Don’t know / don’t remember 5 No response 88 Question Answer Q 19A. Who did you tell the complaint to? [Read response categories except don’t know/no response. Mark all that apply.] 1 Name of IP] staff 2 Sheik 3 Local authorities 4 Other (please fill in) 5 Don’t know / don’t remember 6 No response Q 20. Do you have any recommendations to improve how to identify beneficiaries for assistance? Q 21. Do you have any recommendations to improve how beneficiaries voice their complaints, especially for those who are unable to use a phone or talk to an employee of the organization? CONVENIENT TIME FOR BACK-CHECKING: (Use this section to write any comment or note by the answerer) Comments Do not forget to thank the respondent before leaving 89 COMMUNITY LEADER Question Answer Interviewer_name (REQUIRED) Interviewer_name_other (REQUIRED) Community Leaders Interview Date: Interviewer Initials: (REQUIRED) Respondent Name: (REQUIRED) Respondent Title: (REQUIRED) Organization Name: (REQUIRED) Contact Information: (REQUIRED) Sub-sector: (REQUIRED) Location/Region: (REQUIRED) My name is [Insert Name(s)], I am part of an independent organization conducting an assessment of approaches for engaging communities to identify beneficiaries for USAID humanitarian programs and how complaints about assistance are addressed related to donor assistance. The purpose of this interview is to understand the role of communities when targeting and selecting beneficiaries for assistance, if community members had complaints and how they were able to voice these complaints, how these complaints were addressed and any suggestions you may have for improving how humanitarian aid is delivered. We received your name from the organization [[Organization_Name]]. We understand from [[Organization_Name]] that there was a community committee in your locality responsible for selecting beneficiaries for some type of humanitarian aid. The interview should take no more than 20 minutes. There are no right or wrong answers. All interviews are confidential and anything you tell us will not be attributed to you. All data from this interview be protected. Q1. Are you familiar with the work of this community committee? [Check one box.] 1 Yes 2 No 3 Don’t know / don’t remember 90 Question Answer 4 No response Q2. Thinking back to the last time the community committee met to identify beneficiaries for assistance, what type of assistance was provided? [Do not read responses. Check one box.] 1 Food vouchers 2 Food basket 3 Cash assistance 4 Work for pay / Cash for work 5 Support for small business 6 Livelihoods such as seeds, tools, livestock, etc. 7 Household latrines 8 Household water storage containers and other WASH NFIs 9 Hygiene NFIs 10 Shelter 11 Shelter NFIs 12 Other (please write in) 13 I don’t know 14 No response Other (Please specify): Q3. Who was this assistance for? [Read response categories except don’t know/no response. Check all that apply.] 1 Local community 2 IDPs only 3 Marginalized community 4 I don’t know / don’t remember 5 No response Q4. Thinking back to the last time the community committee identified beneficiaries for assistance, when was this? [Read response categories except don’t know/no response. Check one response below.] 1 Less than 3 months ago 2 Between 3 and 6 months ago 91 Question Answer 3 Between six months and one year 4 Longer than one year 5 I don’t know / don’t’ remember 6 No response Q5. Were you involved in identifying beneficiaries for this assistance? [Read response categories except don’t know/no response. Mark one box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q5-A. How were you involved in identifying beneficiaries? [Read response categories except don’t know/no response. Mark all that apply.] 1 Member of community committee 2 Volunteer 3 Community mobilizer 4 I don’t know / don’t remember 5 No response Q6. Thinking back to the last time the donor provided assistance, was the community informed who would be eligible for assistance? [Mark one response below.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q7. Did you provide names to the committee of households you thought should receive assistance [Mark one response below.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q8. Thinking back to the last time the community committee met to identify beneficiaries, was there criteria for choosing beneficiaries? [Mark one response below.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response 92 Question Answer Q8-A. Who told you about the criteria? [Read response categories except don’t know/no response. Mark all that apply.] 1 [Name of IP][...] 2 Other community committee members 3 Sheikh 4 Local authorities 5 Other (Please write in) 6 Don’t know / don’t remember 7 No response Other (Please specify): Q8-B. Was the community involved in developing or changing the criteria for who was eligible for assistance? [Mark one response below.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q9. Thinking back to the last time the community committee met to identify beneficiaries, did committee members visit households? [Mark one response below.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q10. Did you know which households would receive assistance before it was given? [Mark one response below.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q11. Were you told which households would not receive assistance? [Mark one response below.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q11-A. Were you told why these households wouldn’t receive assistance? [Mark one response below.] Question relevant when: ${Q1} = 1 and ${Q11} = 1 1 Yes 2 No 93 Question Answer 3 Don’t know / don’t remember 4 No response Q12. Thinking back to the last time you identified beneficiaries and assistance was given, to what extent did the beneficiaries who received assistance met the criteria? [Read response categories except don’t know/no response. Mark one response below.] 1 All of them met the criteria 2 Most of them met the criteria 3 Some of them met the criteria 4 None of them met the criteria 5 Don’t know / don’t remember 6 No response Q12-A. If they did not meet the criteria, why? Q13. Thinking back to the last time beneficiaries were identified and given assistance, were you informed by [name of IP] / donor what to do if someone complained to you? [Mark one response below.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q14. Thinking back to the last time assistance was given, did the [name of IP] explain to the community how to make complaints or provide feedback? [Mark one response below.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q15. Has anyone complained to you about not receiving assistance or how beneficiaries were chosen? [Mark one response below.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q15-A. Did you do anything to solve this problem? [Read response categories except don’t know/no response. Mark all that apply.] 1 I solved the issue myself 2 I told someone else 3 Other (please fill in) 4 Don’t know / don’t remember 5 No response Other (Please specify): 94 Question Answer Q15-B. Who did you tell the complaint to? [Read response categories except don’t know/no response. Mark all that apply.] 1 Donor / [...] staff 2 Community committee 3 Neighborhood sheikh 4 Local authorities 5 Other (please fill in) 6 Don’t know / don’t remember 7 No response Other (Please specify): Q16. Do you have any recommendations to improve how to select beneficiaries for assistance? Q19. Do you have any recommendations to improve how beneficiaries voice their complaints, especially those who are unable to use a phone, meet with you or reach the committee? Thank you for your time. I assure you of the confidentiality of the information you have provided and as indicated earlier, these honest answers will help the project improve programming. (Use this section to write any comment or note by the answerer) Comments Do not forget to thank the respondent before leaving 95 COMMUNITY HEALTH WORKER (CHW) Question Answer Community Health Worker / Community Volunteers Interviewer_name (REQUIRED) Interviewer_name_other REQUIRED) IP: (REQUIRED) Interview Date: Respondent Name: (REQUIRED) Respondent Title: (REQUIRED) Contact Information: (REQUIRED) Health Facility Name: (REQUIRED) Sub-sector: (REQUIRED) Location/Region: (REQUIRED) INTRODUCTION: My name is [Insert Name(s)], I am part of an independent organization conducting an assessment of community-based management of malnutrition. The purpose of this interview is to understand how communities are identified and households are selected to be screened for malnutrition, your involvement once a person or household is diagnosed with malnutrition, the records you keep, and any suggestions you may have for improving how community health workers and community health volunteers are able to do their job. We received your name from [[IP]]. The interview should take no more than 30 minutes. All interviews are confidential and anything you tell us will not be attributed to you. All data we record will be protected. The results of our interview have no bearing on whether you or your community receives future assistance. We will use the information we obtain from you and similar [health workers / volunteers] to improve the delivery of community-based malnutrition programs. Do you agree to this interview? Q1. Are you presently a [community health worker / community volunteer]? [Check one box.] 1 Yes 2 No 3 Don’t know / don’t remember 96 Question Answer 4 No response Q1-A1. How long have you been a [community health worker / community volunteer]? [Read response categories except don’t know/no response. Check one box.] 1 Less than six months 2 Six months to one year 3 One to two years 4 More than two years 5 Don’t know / don’t remember 6 No response Q1-A2. How long did you work as a [community health worker / community volunteer]? [Read response categories except don’t know/no response. Check one box.] 1 Less than six months 2 Six months to one year 3 One to two years 4 More than two years 5 Don’t know / don’t remember 6 No response Q2. As a [community health worker / volunteer], are you responsible for…? [Read each line; check one box in each line.] Community outreach and mobilization 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Community based nutrition and health education 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Household screening and referral 1 Yes 2 No 3 Don’t know / don’t remember 4 No response 97 Question Answer Individual counseling on health and/or nutrition 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Follow-up home visits for problem cases 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Training volunteers 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q3. Do you work with IDPs, host-communities or marginalized groups? [Read response categories except don’t know/no response. Check all that apply.] 1 Host community 2 IDPs 3 Marginalized groups 4 Don’t know / don’t remember 5 No response Q4. How many households were in your target area? [Write the number in the space provided] Q5. How many times a month [do / did you] typically visit a village or area of town for screening malnutrition? [Read response categories except don’t know/no response. Check one box.] 1 One 2 Two to four times 3 Five to seven times 4 Eight or more times? 5 Don’t know / don’t remember 6 No response Q6. Who determined which villages or areas of a town you visit to screen for malnutrition. [Read each line; check one box in each line.] 98 Question Answer Self Q6_1 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Implementing partner Q6_2 1 Yes 2 No 3 Don’t know / don’t remember 4 No response CHW supervisor Q6_3 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Health facility director Q6_4 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Ministry of Health Q6_5 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Local authorities Q6_6 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Other (Please specify): Q6_7 99 Question Answer Q7. Thinking about the last village or area you visited to screen for malnutrition, how many households did you visit in a single day? Q8. When you visited this village or area, how did you decide which households to visit? [Write response verbatim in the space provided.] Q9. Were you trained how to select which household to visit for nutrition screening? [Check on box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q10. Did you use a protocol to screen household members for malnutrition? [Check on box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q11. Did you take anthropomorphic measurements such as MUAC and Z score? [Check on box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q12. If you screened a household and someone was malnourished, what did you do before you left the household? [Do not read response; Check all that apply.] 1 I referred them to a health facility or clinic 2 I counseled them 3 I gave them plumpy nut or another ready to eat therapeutic food (RUTF) 4 I visited the household again at a later date 5 Other (please specify) 6 Don’t know / don’t remember 7 No response 100 Question Answer Other (Please specify): Q13. What types of facilities did you refer them to? [Read response categories except don’t know/no response. Check one box.] 1 Health unit 2 Health clinic 3 Hospital 4 Mobile clinic 5 Outpatient Therapy Unit 6 Don’t know / don’t remember 7 No response Q14. Did you give them a piece of paper telling them where to go? [Check on box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q15. Did you keep records of your visits? [Check one box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q15-A. Did you give these records to anyone? [Check one box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q15-B. Who did you give these records to? [Read response categories except don’t know/no response. Check one box.] 1 Health facility director 2 Nutrition supervisor 3 Community health worker / volunteer supervisor 4 IP staff 5 Other (Please specify) 6 Don’t know / don’t remember 101 Question Answer 7 No response Q16. [Do / did] you follow-up with households where there someone was diagnosed with malnutrition? [Check one box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q17. If someone was diagnosed with malnutrition receive, were they more likely to receive other donor assistance [Check one box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q18. Do you think the training you received and information given to screen for malnutrition is adequate enough for meet the needs of the households you target? 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q19. Why not? [Write response verbatim in the space provided.] Q20. What can be done to improve community management of malnutrition? [Write response verbatim in the space provided.] على الباحث أنيخبرالمجيب أنالمشرفقديتصلبه/بهاللتأكد منالبياناتالتيتمالحصولعليها .وعلى الباحث /الباحثة أنيسألالمجيب عن أنسب وقت��تصالبه Thank you for your time. I assure you of the confidentiality of the information you have provided and as indicated earlier, these honest answers will help the project improve programming. End (Use this section to write any comment or note by the answerer) Comments Do not forget to thank the respondent before leaving final 102 HEALTH FACILITY STAFF Question Answer Interviewer_name (REQUIRED) Interviewer_name Interviewer_name_other (REQUIRED) Interviewer_name_Other Health Facility Staff IP: (REQUIRED) Interview Date: Respondent Name: (REQUIRED) Respondent Title: (REQUIRED) Contact Information: (REQUIRED) Health Facility Name: (REQUIRED) Sub-sector: (REQUIRED) Location/Region: (REQUIRED) My name is ------------------, from an independent organization conducting an assessment of community-based management of malnutrition. The purpose of this interview is to understand how communities are identified and households are selected to receive visits from community health workers or community volunteers, record keeping, any knowledge of complaints about who was targeted or screened, and any suggestions you may have for improving how community health workers and community health volunteers are able to do their job. We received your name from [[IP]] [name of IP].The interview should take no more than 30 minutes. All interviews are confidential and anything you tell us will not be attributed to you. All data we record will be protected. The results of our interview have no bearing on whether this facility or your community receives future assistance. We will use the information we obtain from you to improve the targeting of community-based malnutrition programs. Do you agree to this interview? Q1. What is your position at this health facility? 103 Question Answer Q2. How long have you been at this health facility]? [Read response categories except don’t know/no response. Mark one box.] 1 Less than six months 2 Six months to one year 3 One to two years 4 More than two years 5 Don’t know / don’t remember 6 No response Q3. What is your involvement with community health workers / volunteers? [Read each line; mark one box in each line.] Supervise their work 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Provide training 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Provide mentoring / coaching 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Provide nutrition supplies for CHWs to distribute 1 Yes 2 No 3 Don’t know / don’t remember 4 No response 104 Question Answer Provide information, education and communication materials for CHWs use 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Provide MUAC tape 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Assign CHWs households to visit 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Review CHW records to ensure that data is captured 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Other (please specify): Q4. How many community health workers or volunteers are affiliated with this facility? [Write the number in the space provided] Q5. How do community health workers or volunteers determine which villages or areas to visit to screen for malnutrition? [Read response categories except don’t know/no response. Mark all that apply.] 1 Review existing data on malnutrition to identify areas to visit 2 Get guidance from the health facility 3 Get guidance from the CHW supervisor 105 Question Answer 4 Get guidance from implementing partners 5 Get guidance from the Ministry of Health 6 Get guidance from local authorities 7 Other (please specify) 8 Don’t know / don’t remember 9 No response Q6. How do community health workers or volunteers decide which households to visit to screen for malnutrition? [Write response verbatim in the space provided.] Q7. Do community health workers or volunteers associated with this facility take anthropomorphic measurements such as MUAC and Z score? [Mark on box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q8. Do community health workers or volunteers keep records of who they visit? [Mark on box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q9. Does the health facility receive a copy of the records they keep? [Mark on box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q9-A. In general, what is the quality of these records? [Read response categories except don’t know/no response. Mark one box.] 1 Very good 2 Good 3 Fair 4 Poor 106 Question Answer 5 Don’t know / don’t remember 6 No response Q10. How does the facility know a person who is seeking treatment for malnutrition was referred by a community health worker or volunteer? [Read response categories except don’t know/no response. Mark all that apply.] 1 He/she presents a referral note 2 He/she verbally reports being referred by a community health worker / volunteer 3 Other (Please specify): 4 Don’t know / don’t remember 5 No response Q11. In general, how accurate are the community health worker or volunteer screenings for malnutrition? [Read response categories except don’t know/no response. Mark all that apply.] 1 Accurate all of the time 2 Accurate most of the time 3 Accurate some of the time 4 Not accurate 5 Don’t know / don’t remember 6 No response Q12. Once a person is referred by a community health worker or volunteer to this clinic for malnutrition, what happens to the patient? [Read each line; mark one box in each line.] Conduct medical assessment using MUC and Z score 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Provide nutritional rehabilitation with ready to use therapeutic ready to use food to eat at home 1 Yes 2 No 3 Don’t know / don’t remember 4 No response 107 Question Answer Provide medical treatment 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Recommend the patient returns to the clinic for additional monitoring 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Recommend the community health worker or volunteer follows up with the household 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Admit cases of severe acute malnutrition with complications 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Refer cases of severe acute malnutrition to another facility 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q13. Does this facility keep patient records documenting the diagnosis and treatment? [Mark one box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q14. How can households in the community voice their concerns about community health Partner D workers or volunteers visits such as the frequency of the visit, not receiving proper diagnosis or referral for treatment of malnutrition? [Do not read response; mark all that apply] 1 Tell someone at the clinic 2 Put a note in the suggestion box 108 Question Answer 3 Call a toll-free hotline number provided by the implementing partner 4 Communicate by SMS or WhatsApp with the implementing partner 5 Tell a local leader 6 Tell a local committee member 7 Visit the implementing partner 8 Tell IP field staff 9 Take a survey 10 Participate in a focus group discussion 11 Other (Please specify): Q15. Do you keep written records of these complaints? [Mark one box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q16. Is the implementing partner informed of these complaints? [Mark one box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q16-A. How is the implementing partner informed of these complaints? [Read each line; mark one box in each line.] 109 Question Answer Phone calls 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Periodic meetings 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Regular interactions with IP staff 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Written reports 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Other (please specify): Q17. Do community health workers or volunteers follow-up with patients who have been diagnosed with moderate or severe acute malnutrition? [Mark one box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q18. Does the implementing partner follow-up with patients who have been diagnosed with moderate or severe acute malnutrition? [Mark one box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response 110 Question Answer Q19. Do you think the nutrition screenings conducted by the community health workers or volunteers reaches all households that may have a family member suffering from malnutrition [Mark one box.] 1 Yes 2 No 3 Don’t know / don’t remember 4 No response Q19-A. Who does not reach it? [Do not read response; mark all that apply] 1 Distant communities 2 Communities near the front line 3 Communities in areas with rugged terrain 4 Internally displaced persons / communities 5 Marginalized groups 6 Other (Please specify): 7 Don’t know / don’t remember 8 No response Q20. What can be done to improve community management of malnutrition? [Write response verbatim in the space provided.] Thank you for your time. I assure you of the confidentiality of the information you have provided and as indicated earlier, these honest answers will help the project improve programming. End (Use this section to write any comment or note by the answerer) Comments Do not forget to thank the respondent before leaving final 111 ANNEX 6: SIGNED CONFLICT OF INTEREST FORMS 112 113 114 ANNEX 7: BIBLIOGRAPHY (REDACTED) Program Documents PARTNER A. Accountability to Affected Populations Plan OFDA in Yemen. PARTNER A. PARTNER A Protection Policy: Policy Document and Guidance. (Internal Document). 2018. PARTNER A. Beneficiary Verification Form. (Internal Document.) PARTNER A. Checklist for Forming Community Committees and Selecting Beneficiaries. (Internal Document) PARTNER A. Complaint and Feedback Mechanism. (Internal Document.) PARTNER A. Criteria and Selection Process of Beneficiaries and Operational Procedures for Verification Process of Beneficiaries for USAID Projects. (Internal Document.) PARTNER A. FFP Awareness Booklet. (Date Unknown.) PARTNER A. FFP: Monitoring and Evaluation Plan Narrative (Revised). 2018. PARTNER A. FFP: Quarterly Report April-June 2020. PARTNER A. FFP: Quarterly Report January-March 2019. PARTNER A. FFP: Quarterly Report January-March 2020. PARTNER A. FFP: Quarterly Report October-December 2019. PARTNER A. FFP: Quarterly Report April 2020-June 2020. PARTNER A. OFDA: Semi-Annual Report January-March 2019. PARTNER A. OFDA: Semi-Annual Report October-March 2020. PARTNER A. USAID Award Documents. 720FDA18GR00XXX. PARTNER A. USAID Award Documents. 72DFFP18GR00XXX. PARTNER B. Beneficiary Feedback Flowchart. (Internal Document) PARTNER B. Beneficiary Feedback Mechanism Standard Operating Procedure (Internal Document.). January 2018. PARTNER B. Beneficiary Verification SOP. (Internal Document.) January 2019. PARTNER B. BFM Database MPCA, extracted 2020 (Internal Document.) PARTNER B. Distribution Checklist. (Internal Document.) January 2019. PARTNER B. Eligibility Criteria (Internal Document.) PARTNER B. Eligibility Formula Steps (Internal Document.) PARTNER B. Household Information Form. (Internal Document.) PARTNER B. MPCA Registration Form V4. (Internal Document.) 115 PARTNER B. Registration Form (Internal Document.) PARTNER B. Semi-Annual Narrative Report April 2019-September 2019. PARTNER B. Semi-Annual Narrative Report October 2019-March 2020. PARTNER B. Semi-Annual Narrative Report September 2019-March 2019. PARTNER B. Standard Operating Procedures: Cash Transfer Programming. (Internal Document.) January 2019. PARTNER B. USAID Award Documents. 720OFDA18GR00XXX. PARTNER B. Verification Process Report Template. (Internal Document.) PARTNER C. Beneficiary Feedback Mechanisms Response Flowchart. (Internal Document.) PARTNER C. Beneficiary Feedback Mechanisms. (Internal Document.) PARTNER C. Beneficiary Selection and Targeting Form. (Internal Document.) PARTNER C. Beneficiary Selection Criteria. (Internal Document.) PARTNER C. Community Feedback, extracted 2020 (Internal Document.) PARTNER C. Guideline for Beneficiary Nomination, Registration and Verification. (Internal Document.) PARTNER C. OFDA Semi-Annual Report October 2018-March 2019. PARTNER C. Procedures for Changing Direct Beneficiaries and Next of Kin. (Internal Document.) PARTNER C. Procedures for Community Volunteer Committees Formation. (Internal Document.) PARTNER C. USAID Award Documents. 720FDA18GR00XXX. PARTNER C. USAID Award Documents. 72DFFP18GR00XXX. PARTNER C. FFP Beneficiary Registration and Verification Form. (Internal Document.) PARTNER C. FFP Beneficiary Selection Criteria. (Internal Document.) PARTNER C. FFP Final Program Report October 2018-June 2020. PARTNER C. FFP Fiscal Year 2019 Annual Results Report. November 2019. PARTNER C. FFP Quarterly Report April 2019-June 2019. PARTNER C. FFP Quarterly Report January 2019-March 2019. PARTNER C. FFP Quarterly Report January 2020-March 2020. PARTNER C. FFP Quarterly Report July 2019-September 2019. PARTNER C. FFP Quarterly Report October 2018-December 2018. PARTNER C. FFP Quarterly Report October 2019-December 2019. PARTNER C. OFDA Semi-Annual Report April 2019-September 2019. PARTNER D. Beneficiary Targeting Form. (Internal Document.) PARTNER D. Cash Relief Operating Procedures. (Internal Document.) 116 PARTNER D. Client Feedback. Extracted 2020. (Internal Document.) PARTNER D. Client Responsive Programming Framework, December 2019. (Internal Document.) PARTNER D. First Semi-Annual Report March 2019-September 2019. PARTNER D. Guidelines for Cash Programming. (Internal Document.) PARTNER D. PARTNER D Internal Data Privacy Policy. Internal Document.) PARTNER D. Monitoring, Evaluation, Accountability, and Learning Strategy V.1 January 2019. (Internal Document.) PARTNER D. Relief Committees Terms of Reference. (Internal Document.) PARTNER D. Semi-Annual Report October 2019-March 2020. PARTNER D. Sub-Sector Guidelines. (Internal Document.) PARTNER D. USAID Award Documents. 720FDA18GR00XXX. PARTNER D. Verification Questionnaire. (Internal Document.) PARTNER E. Accountability Assessment Report. 2019. PARTNER E. Beneficiary Registration Form. (Internal Document.) PARTNER E. Beneficiary Selection and Targeting: Food Sector and Livelihoods Sector Specific Guidelines. (Internal Document.) PARTNER E. Beneficiary Targeting and Verification Guide for FFP Emergency Food Security Activities. (Internal Document.) PARTNER E. Complaint and Feedback Categorization. (Internal Document) PARTNER E. CRM Reports. Extracted 2020. (Internal Document.) PARTNER E. Face-to-Face CRM Recording Book. Photo, 2020. PARTNER E. Monitoring, Evaluation, Accountability, and Learning Plan and Protocols. December 2019. PARTNER E. Putting Accountability into Practice: Accountability Manual. (Internal Document.) PARTNER E. Registration Form. (Internal Document.) PARTNER E. PARTNER E Policy: Data Protection Policy. October 2019. (Internal Document.) PARTNER E. SOP. November 2018. (Internal Document.) PARTNER E. Standard Operating Procedures for Cash Transfer Programming in Yemen. (Internal Document.) PARTNER E. Suggestion Boxes Tracking Table. Extracted 2020. (Internal Document.) PARTNER E. Verification Form for FFP Hajjah. (Internal Document.) PARTNER E. USAID Award Documents. 720FDA18GR00XXX. PARTNER E. OFDAs: Final Report, December 14, 2018-March 15, 2020. PARTNER E. OFDAs: Semi-Annual Report, December 15 – March 31, 2019. 117 PARTNER E. OFDAs: Semi-Annual Report, April 1, 2019 – September 30, 2019. PARTNER E. FFP: Quarterly Report, October 2018-December 2018. PARTNER E. FFP: Annual Report, October 2018-September 2019. PARTNER E. FFP: Quarterly Report, April 2019-June 2019. PARTNER E. FFP: Quarterly Report, October 2019-December 2019. PARTNER E. FFP: Quarterly Report, January 2020-March 2020. PARTNER E. FFP: Quarterly Report, April 2020-June 2020. PARTNER F. Accountability to Affected Populations PARTNER F Integrity Framework. (Internal Document.) PARTNER F. Anonymized Feedback Tracking Table. Extracted 2020. (Internal Document.) PARTNER F. CFW Registration Form. (Internal Document.) PARTNER F. OFDA: Final Report, September 2018-April 2020 PARTNER F. OFDA: Semi-Annual Report, April 2019-September 2019 PARTNER F. OFDA: Semi-Annual Report, October 2018-March2019. PARTNER F. Registration Form. (Internal Document.) PARTNER F. Report on Establishing Community Committee in Al Dhale’e Governorate. June 2020. (Internal Document.) PARTNER F. Standard Operating Procedures: Cash Transfer Programming. (Internal Document.) PARTNER F. Subsector Beneficiary Registration Form. (Internal Document.) PARTNER F. USAID Award Documents. 720FDA18GR00XXX. Other Documents CHS Alliance. Core Humanitarian Standard on Quality and Accountability. Source: https://corehumanitarianstandard.org/files/files/CHS_Plain_Language_English.pdf, accessed July 7, 2020. CHS Alliance. CHS Guidance Note and Indicators. Source: https://corehumanitarianstandard.org/files/files/CHS-Guidance-Notes-and-Indicators.pdf, accessed July 7, 2020. Cross, Tiare and Andrew Johnston. Cash Transfer Programming in Urban Emergencies: A Toolkit for Practitioners. Oxford, UK: Cash Learning Partnership, 2011. IFRC. Cash in Emergencies Toolkit. Source: http://rcmcash.org/, accessed July 7, 2020. Inter-Agency Standing Committee. Revised AAP Commitments Guidance Notes for Principals and Senior Managers, 2018. Source: https://interagencystandinPartner Bommittee.org/system/files/annex_1_and_2_to_the_revised_caap_march_2018_1.pdf, accessed July 10, 2020 Inter-Agency Standing Committee. Revised Accountability to Affected People and Protection from Sexual Exploitation and Abuse endorsed 2017. Source: https://interagencystandinPartner 118 Bommittee.org/accountability-affected-populations-including-protection-sexual-exploitation-and￾abuse/documents-61, accessed July 7, 2020. Inter-Agency Standing Committee. Best Practice Guide: Inter-Agency Community-Based Complaint Mechanisms. Geneva: International Organization for Migration, 2016. Price, Roz. Improving Beneficiary Feedback Mechanisms for Refugees. K4D Help Desk Report. Brighton, UK: Institute for Development Studies, 2018 Schwartz, Tim. A Model for Humanitarian Aid Beneficiary Targeting. April 7, 2018. Source: https://timothyschwartzhaiti.com/a-model-for-humanitarian-aid-beneficiary-targeting/, accessed July 10, 2020 U.S. Congressional Research Service. Yemen: Civil War and Regional Intervention (R43960, April 23, 2020), by Jeremy M. Sharp. Source: https://fas.org/sgp/crs/mideast/R43960.pdf, accessed June 10, 2020 20. UNHCA and World Food Programme. Joint Guidance: Targeting of Assistance to Meet Basic Needs. Geneva, Switzerland: 2018. United Nations High Commissioner for Refugees, Cash Learning Partnership, et al. Operational Guidance for Multi-Purpose Cash Grants. Geneva, Switzerland: 2015. USAID/OFDA. Application Guidelines. Washington, DC: October 18, 2019. Yemen Food and Agriculture Security Cluster. Accountability Programme Design and Standards Tool. Date unknown. Yemen Food and Agriculture Security Cluster. Draft Technical Guidelines for Livelihoods Conditional Transfer Programmes in Yemen. Date unknown. Yemen Ministry of Public Health and Primary Partner D Sector. Interim Guidelines for Community￾Based Management of Severe and Moderate Acute Malnutrition, Version 1.0. Sana’a, Yemen: 2013 119 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523