COVID-19 Evaluation iv Final Performance Report COVID-19 Evaluation Final Report March 6, 2023 [Insert Report Title] v SUBMITTED BY AWARD Cooperative Agreement No.: 720BHA21LA00007 AA Name under the Implementer-Led Design, Evidence, Analysis and Learning (IDEAL) Leader with Associates Activity (72DFFP19LA00001) Geographic Area – Worldwide October 1, 2021 – November 15, 2022 SUBMITTED TO Eunice Wavomba Agreement Officer’s Representative United States Agency for International Development Bureau for Humanitarian Assistance (BHA) 555 12th Street, NW, 8th Floor Washington, DC 20004 PHOTO CREDITS ICSA Agency per Save the Children CONTACT: Kimberly Cook Director, Collaborating, Learning and Adapting c/o Save the Children, Department of Hunger and Livelihoods 899 North Capitol St NE, Suite 900 Washington, DC 20002 Mobile: 1-202-274-1548 kcook@savechildren.org COVID19Evaluation@savechildren.org www.fsnnetwork.org DISCLAIMER This report is made possible by the generous support of the American people through the United States Agency for International Development (USAID). The contents are the responsibility of the COVID-19 Evaluation award and do not necessarily reflect the views of USAID or the United States Government. COVID-19 Evaluation 1 Table of Contents Acronyms............................................................................................................................................. 2 Overall Performance............................................................................................................................ 3 Overall Assessment ........................................................................................................................ 4 Key Findings.................................................................................................................................... 4 Changes and Amendments................................................................................................................... 5 Measuring Results ............................................................................................................................... 6 Methods......................................................................................................................................... 6 Inception Phase: November-February 2022 .................................................................................. 6 Data Collection and Analysis Phase: March-July 2022................................................................... 7 Reporting Phase ............................................................................................................................. 7 Dissemination Phase ...................................................................................................................... 7 Participation of and Accountability to Affected Populations ................................................................ 8 Risk Management................................................................................................................................ 9 Coordination........................................................................................................................................ 9 Lessons Learned................................................................................................................................. 10 Transition or Exit Strategy.................................................................................................................. 10 Documents Uploaded in BHA ART Module of AAMP .......................................................................... 10 Semi-Annual Report 2 Acronyms AA Associate Award AAMP Application and Award Management Portal AOR Agreement Officer’s Representative ART Award Reporting Tool BHA Bureau for Humanitarian Assistance CARES Act Coronavirus Aid, Relief, and Economic Security Act CDC Centers for Disease Control CVA Cash and Voucher Assistance COVID-19 Novel Coronavirus/Coronavirus Disease 2019 ET Evaluation Team FFP Office of Food for Peace FGD Focus Group Discussions HBMO Humanitarian Business and Management Operations HCIM Humanitarian Coordination and Information Management IDA International Disaster Assistance IDEAL Implementer-Led Design, Evidence, Analysis and Learning IP Implementing Partner ITT Indicator Tracking Table KII Key Information Interviews LWA Leader with Associates Award M&E Monitoring & Evaluation OFDA Office of US Foreign Disaster Assistance TANGO Technical Assistance to Non-Government Organizations International TPQ Technical Program Quality USAID US Agency for International Development WHO World Health Organization [Insert Report Title] 3 Overall Performance In 2022, under the US Agency for International Development (USAID), Bureau for Humanitarian Assistance (BHA) Implementer-Led Design, Evidence, Analysis and Learning (IDEAL) Leader with Associate Award (LWA) Activity, BHA issued the COVID-19 Evaluation Associate Award (AA). Technical Assistance to Non-Government Organizations (TANGO) International – in collaboration with Save the Children – conducted a performance evaluation of BHA’s response efforts to the Novel Coronavirus/Coronavirus Disease 2019 (COVID-19) pandemic in humanitarian contexts. This evaluation covered awards funded in FY 2020 and distributed from March through August 2020 as noted below. The World Health Organization (WHO) officially declared COVID-19 a global pandemic on March 11, 2020. In response, the US Congress passed the Coronavirus Preparedness and Response Supplemental Appropriations Act 2020, which provided $300 million in international disaster assistance (IDA), followed by the Coronavirus Aid, Relief, and Economic Security (CARES) Act, which provided an additional $258 million in IDA. These funds were distributed from March to August 2020 from the legacy Office of US Foreign Disaster Assistance (L-OFDA) and legacy Office of Food for Peace (L-FFP) as Pillar 3 of a four￾pillared strategy developed by the Department of State and USAID to prepare for, prevent, and respond to the pandemic. The three award goals set by BHA in the Monitoring, Evaluation and Learning (MEL) plan are paraphrased below: • 1: Mitigate the transmission of COVID-19 and pandemic-related Protection risks. • 2: Mitigate the secondary impacts of COVID-19 pandemic and control measures. • : Support the continuation of health and humanitarian programming. These supplemental funds – totaling $558 million – allowed 178 awards to be dispersed across 42 countries (including West Bank Gaza and the Pacific Islands, with 12 being global in scope) to 62 IPs. The largest percentage of funding (45%) was distributed to countries within BHA’s Office of Africa. The COVID-19 Evaluation assessed the performance of BHA’s global response to COVID-19, with specific focus on the supplemental funding, documenting successes, lessons learned, and unintended consequences of response activities. The evaluation objectives were addressed in four distinct evaluation domains: effectiveness, relevance, efficiency/timeliness, and coordination. The evaluation concluded with recommendations to BHA for improving their response efforts in future outbreaks of novel diseases or pandemic emergencies. The audience for this evaluation included USAID/BHA senior management, technical and geographic teams and IPs, among others. The evaluation incorporated the perspectives of and provides insights for a broad range of USAID/BHA internal and external stakeholders, including the awarded IPs and presented an opportunity for broad learning across the humanitarian sector. Key operational findings and recommendations were disseminated to BHA, IPs and the broader humanitarian community in support of their learning around COVID-19 and potential future outbreak responses. It provided stakeholders with information for future global responses to infectious disease and epidemics, including future responses to COVID-19. Additionally, the evaluation is intended to help inform future thematic studies or evaluations, which in turn will improve the ability and capacity for humanitarian actors and Semi-Annual Report 4 agencies to provide quality programming. All these audiences were considered and addressed in the program dissemination plan. The Evaluation team hosted inception webinars with BHA stakeholders and IPs who received COVID-19 supplemental funding during FY20. The Evaluation team presented the evaluation methods and plans, as described in the Inception Report, during the BHA Lunch and Learn on February 24, 2022 with 40-50 stakeholders present from the Technical and Program Quality (TPQ) division, among others. The team hosted a similar presentation for IPs on March 10, 2022, with additional opportunities to provide inputs through Zoom polls, comments, a Padlet board, and breakout rooms. The attendance rate and engagement by IPs were exceptional: 108 attendees from approximately 38 organizations across 30 countries were present. The COVID-19 Evaluation proceeded through the Inception, Data Collection and Analysis, Reporting, and Dissemination Phases working closely with BHA and IPs to adjust and adapt is needed. A no-cost extension was provided in September 2022 to allow approximately two additional months for the project close-out. The technical team, led by TANGO International, closed out all work on the contract on October 6, 2022. Overall Assessment The results of the COVID-19 Evaluation highlighted BHA’s and IPs’ capacities to rapidly respond and adapt to the various challenges created by the pandemic. IPs leveraged the COVID-19 supplemental assistance with other donor assistance and adapted approaches according to community feedback when possible. BHA provided direct communication on guidance and rapid approval of modifications. COVID-19 funded Awards were ultimately effective in building awareness of COVID-19 prevention; often by leveraging existing local partnerships. Challenges included inconsistent quality data and reporting, as well as the short timeframe of the funding, which limited both the extent to which the funding objectives could be addressed and implementation in new areas. The evidence also indicates the need for multi-pronged humanitarian response and coordination mechanisms, as well as shock-responsive safety nets, as the pandemic had far-reaching impacts beyond those directly related to health. Key Findings As hindsight is always 2020, it is important to look at the evaluation findings in the realities of the early weeks and months of the pandemic. Amidst many unknowns regarding both the first global emergency in USAID’s modern memory and the novel coronavirus, BHA had to make critical decisions about this funding. This, also in the context of massive global response needs, near-immediate global supply chain challenges, the domestic context including some of the administration and leadership priorities and policies that required negotiation, and the formation of BHA. BHA worked hard to prioritize sectors and geographies based on knowledge of humanitarian needs from the field but faced constraints in being able to adapt based upon an evolving understanding of the nature of the novel pandemic threat as well as a better understanding of the evolving direct and indirect (due to pandemic response) effects. BHA made the right decision to rely on trusted partners in pandemic situations, to support coordination, and to shift funding focus to food assistance-related [Insert Report Title] 5 supports which made a great difference in the relevance, effectiveness, and efficiency/timeliness of the response in crisis contexts. Despite some initial delays due to these challenges, BHA worked swiftly and collaboratively with partners to support timely implementation and modifications as needed and IPs delivered in the face of extraordinary challenges, often made possible by dedicated staff and community support. While just supplemental in nature, IPs used the funding to strengthen and expand upon existing programming from USG and other donors. In all, BHA largely met its funding targets and achieved significant contributions toward its objectives. Drawing on the remaining findings, the following recommendations and considerations aim to help BHA improve its response to future outbreaks and global emergencies. • Review how FY2020 learning informed the next years of BHA pandemic funding strategies. • Consider larger design questions: With the large geographic and sectoral scope for this limited funding, reflections should consider if/or how this spread limited results; if the decision to allocate funding as stand-alone awards instead of amendments was the right one; and to consider what the impact of the tranche system was - its advantages and disadvantages. • Conduct a learning exercise around how emergency funding can support a more sustainable capacity building approach, with national partners including some government actors where appropriate for the context. BHA may partner agency-wide to identify the institutions that can be invested in now for future pandemic response. • Capture lessons learned and good practices around digital/remote programming approaches to facilitate further development of these pandemic innovations. • Develop strategies to prepare to support populations that will be newly vulnerable due to large scale shocks, this includes populations in middle income countries, the urban poor, and rural populations that have not been BHA participants prior to the pandemic. • As BHA alone would not be able to address the full spectrum of health sector needs related to infection prevention and mitigation in humanitarian contexts, there are many learning opportunities around BHA’s contribution to health system supports in health emergencies. BHA should consider coordinating more closely with global health entities at USAID, the CDC, or WHO to develop better national level approaches in pandemic situations. Changes and Amendments The COVID-19 Evaluation proceeded through the Inception, Data Collection and Analysis, Reporting, and Dissemination Phases working closely with BHA and IPs to adjust and adapt is needed. The technical team, led by TANGO International, closed out all work on the contract on October 6, 2022. A no-cost extension was provided in September 2022 to allow approximately two additional months for the project close-out. In total, three modifications were issued throughout the award period, and they are as follows: 1. Modification P001 (1): the purpose of this modification was to increase the fringe benefits percentage from 21.60% to 27.58%. 2. Modification P002 (2): the purpose of this modification was to extend the performance period of the award from June 30, 2022, to September 30, 2022. 3. Modification P003: The purpose of this modification was to extend the performance period of the award from September 30, 2022, to November 30, 2022, remove paragraph (h), Special Semi-Annual Report 6 Award Requirement Relating to the Prohibition on Certain Telecommunications and Video Surveillance Services or Equipment (November 2020), update Section 1. 14 of the agreement by inserting Standard Provision No. 31, and update attachment 3-Standard Provisions by inserting Stand Provision No. 31. Measuring Results Methods The evaluation used mixed methods and multi-level data collection, drawing data from multiple sources – both primary and secondary, quantitative, and qualitative – to assess the evaluation questions. Results and evidence were triangulated to ensure each finding statement was supported by at least three of the five data sources listed below: 1. Country Case Studies: In-depth examination of all awards and field-level experiences in Nigeria, Iraq, and Colombia, representing each BHA Office (178 key informant interviews with IP staff, government, and community stakeholders and 146 focus group discussions (FGDs) with project participants representing 724 females/564 males). 2. IP E-survey: Qualitative and quantitative data for both general and sector-specific programming (74 award responses). 3. Deep Dive: Reviewed all documents for 30 purposively sampled awards, followed up by interviews with IPs and BHA activity managers/representatives group (535 documents reviewed, with follow-up interviews conducted with IP and BHA activity managers). 4. Shallow Dive: outcome level themes and missing indicator values extracted from remaining 81 awards’ final reports. 5. Award Data Analysis: Data for priority indicators compiled from monitoring data across reporting awards (933 data points for 45 priority BHA indicators). In addition, the evaluation conducted interviews with BHA stakeholders and reviewed various internal funding design and guidance documents. Inception Phase: November-February 2022 The Inception Phase started immediately at award. The first draft of the Inception Report was submitted to BHA on December 22, 2021, and the final version of the report was approved on February 16, 2022. The Inception Report included the following activities: preliminary gap analysis based on award monitoring data received from October 28, 2021 through December 13, 2021; consultation meetings with BHA stakeholders on key themes of interest; confirmed case study countries; deep dive award sample selection; and overall evaluation approach. The evaluation then hosted inception webinars with BHA stakeholders and the COVID-19 supplemental funding IPs. The evaluation methods and plans, as described in the Inception Report, were presented by the evaluation team during the BHA Lunch and Learn on February 24, 2022 with 40-50 stakeholders present from the Technical and Program Quality (TPQ) divisions, among others. A similar presentation was provided to the IPs on March 10, 2022, with additional opportunities for providing inputs through Zoom polls, comments, Padlet board, and breakout rooms. The attendance rate and engagement by IPs were exceptional: 108 attendees from approximately 38 organizations were represented across 30 countries. [Insert Report Title] 7 Data Collection and Analysis Phase: March-July 2022 The evaluation included five main method areas. The summary table below provides an overview of the data collected through each method followed by a description of each method. Data Collection by Method E-survey (# of awards) Deep Dives (# of Docs) Deep Dives (# of KIIs) Shallow Dives (# of docs) Indicator Analysis (# of values) Case Studies (# of KIIs) Case Studies (# of FGD participants) 74 535 26 81 933 178 724 female/564 male • IP E-survey: The survey provided qualitative and quantitative data for both general and sector￾specific programming. The survey was launched on June 16, 2022 and closed on July 14, 2022, and 74 IPs responded to the e-survey. • Deep Dive: The ET reviewed all documents for 30 purposively sampled awards (535 documents total), followed up by interviews with IPs (26 of 30) and a BHA activity managers/representatives group. Follow up Key Informant Interviews (KIIs) were conducted in June and July 2022. • Shallow Dive: The evaluation reviewed the remaining 81 awards that were not deep dives or case studies, for which there were final reports, for outcome level themes and missing indicator values between May and July 2022. • Award Data Analysis: Throughout the evaluation the ET analyzed 45 priority indicators (including 933 values total) and participant data compiled from BHA award monitoring data. As requested by BHA, the evaluation developed an in-depth sub-report outlining Data Gaps, provided as an appendix to the full evaluation report. • Country Case Studies: The case studies provided in-depth examination of all awards and field￾level experiences in Nigeria, Iraq, and Colombia, representing each BHA region. Each case study included an Evaluation Plan, with feedback from BHA at evaluation and country levels, as well as a Debrief presentation with overall preliminary findings to BHA focal points. The Colombia fieldwork commenced in March, Nigeria fieldwork in April, and Iraq in May. Reporting Phase The Evaluation team submitted the first draft of the Evaluation Report to BHA on August 8, 2022. BHA compiled comments from key stakeholders and the Evaluation team made revisions and responses according to those comments. BHA accepted and approved the final report and supplementary annexes on September 26, 2022. Dissemination activities, described next, began to take place during the reporting and revisions, including presentations of findings to various BHA stakeholders to serve the dual purpose of dissemination and gathering draft report feedback. Dissemination Phase The Evaluation team developed and agreed with BHA a Dissemination/Utilization Plan in June-July 2022. This plan outlined activities for the focus areas of providing strategic findings and recommendations, providing technical and sector-specific findings, and debriefing the evaluation process with respective audiences. In total, the Evaluation team reached approximately 80 BHA stakeholders and 120 Semi-Annual Report 8 humanitarian partners through the various finding’s presentations, with countless more to be reached through the sharing of written materials. The dissemination activities included the following: Evaluation Report: finalized, described above, to be shared internally by BHA October 21, 2022 BHA initial findings presentation and discussion: with the Evaluation Working Group, BHA leadership, and Response Management Team staff who were most involved in COVID-19 programming and with the evaluation. August 15, 2022 BHA strategic findings presentation and discussion: with a broad audience of BHA stakeholders, held two times at different time zones to include more participation. August 31 and September 1, 2022 M&E key findings and recommendations: specific discussion with the M&E team on the Data Gaps appendix and related recommendations. September 13, 2022 SMT presentation and discussion: recommendations were presented to senior management team members of BHA, and a high-level 4-page recommendations brief developed for this ongoing discussion with the Front Office. September 22, 2022 (External) global findings webinar: held for an external audience of IPs and the humanitarian community, with interpretation provided in Spanish, French, and Arabic. The early morning time slot for the webinar captured participants from all three BHA Offices. September 29, 2022 (External) findings brief: 10-page brief provided for all stakeholders and published publicly, disseminated in English, Spanish, French, and Arabic September 29, 2022 Evaluation debrief: an after-action review note on the evaluation process was provided by the evaluation team and discussed during a debrief session with BHA. September 29, 2022 Participation of and Accountability to Affected Populations As noted, this award focused on determining the effectiveness, relevance, efficiency/timeliness and coordination/coherence of IP activities, rather than direct engagement with program beneficiaries. As such, typical measurement of beneficiary/affected persons was not included in the Evaluation Work Plan or Monitoring & Evaluation (M&E) Plan. [Insert Report Title] 9 IPs were included in all steps of the evaluation methods and processes as described above, providing invaluable feedback on BHA, other Donor, IP and host government activities and outcomes. Risk Management As noted, this award focused on analysis of existing data and interaction with IPs, rather than with direct beneficiary programs, so there is little to describe regarding program risks and mitigation. However, ass part of the evaluation, the ET did conduct several country case studies including Nigeria, Iraq and Columbia. Each country had specific security concerns which were mitigated by following Save the Children’s and Tango’s internal risk analysis and mitigation measures as described in the Risk Management and Mitigation plan. This focused primarily on following travel safety and risk mitigation rules, engaging with local partners to support data collection in hard-to-reach locations and ensuring communications with local, regional and host governments to ensure a full understanding of the risk profile and adapting plans as needed. Program losses and Food commodity safety and quality assurance reporting are not relevant for this award. As noted above the award did not interact with direct beneficiaries and there were no physical commodities or other modalities used that would have implicated safety, quality or losses. Coordination This evaluation required significant coordination with BHA and it’s the COVID-19 funded IPs’. This began with coordination with BHA to determine the availability of data and quality and content requirements, with weekly meetings/coordination calls from the inception phase through the end of the program. • The ET continued to work with BHA throughout the evaluation phase to review, update, clean and validate data used as the evidence base for the evaluation program to ensure findings and conclusions were based in quality information to ensure program quality. • The ET worked closely with BHA to determine evaluation parameters and final outputs, from inception phase through final dissemination activities. • The ET also worked closely with BHA counterparts in decisions on the types and forms of dissemination products for various BHA stakeholders, which ranged from regular and ad hoc program progress meetings, findings presentations, to the final full Evaluation report. As discussed above there was significant engagement with the IPs during the evaluation. This started with an inception webinar using interactive tools to gather IP inputs and feedback on the proposed evaluation questions and methods. Close coordination was also necessary for the country case studies, IP e-survey, and the deep and shallow dives. For the case studies, the ET worked directly with country research teams to ensure collection of high￾quality data and that analyses aligned with broader Evaluation needs. This fieldwork was conducted in close coordination with both the BHA representatives in/working with that country context and the relevant IP field offices hosting the research in their operational areas. Each case study presented a debrief of key findings at the country level to these stakeholders. Semi-Annual Report 10 Lessons Learned Lesson learned during this Evaluation focused on a few key areas. • Availability and quality of data. As noted above, there was extensive coordination required around access to and quality and completeness of the award data. While we began discussion on the data very early on in the award, a final, complete dataset wasn’t available for full analysis until 3 to 4 months into the award. It was agreed with the BHA that this learning around BHA’s data monitoring system was important to summarize, and it was included as a complete and separate annex with recommendations in the Evaluation Report. These key data issues and recommendations were also presented to the BHA M&E Team. • Access to IP program staff. As the evaluation was conducted after these awards had finished, it was a challenge to identify and contact the original program staff, many of which had moved on to other programs. Recall was also an issue for some IP staff. • Security considerations. While the ET was able to conduct all the planned case studies, managing security considerations for the studies impacted their timing, scope and personnel. For example, the ET originally planned to have ET team staff (join local consultants) participate in the Iraq case study, but they were unable to participate in the field work due to COVID-19 travel restrictions and security consideration. • Understanding audiences. Conducting an audience mapping exercise early in the program helped the ET determine the target audiences for results and the appropriate communications products for the audience. • Award Management. This award and cooperative agreement as a collaboration between a humanitarian organization/partner and technical team worked well in terms of management, expertise, and cost-efficiency. Transition or Exit Strategy As with other reporting requirements, the Transition or Exit Strategy is not relevant for this award as there are no ongoing activities to transition. Additional evaluations on other COVID-19 funding sources and activities are being conducted under other award mechanisms. Documents Uploaded in BHA ART Module of AAMP Documents uploaded (or to be uploaded to ART/AAMP include the following. Other items typically uploaded to AAMP, such as humanitarian stories, unique beneficiaries, sector level breakdowns, emergency indicators, modalities, etc. are/were not relevant for this award. • FY 2020 COVID-19 Evaluation Final Performance Report (to be uploaded once approved by the AOR). • FY 2020 COVID-19 Evaluation Final Financial Reports have been uploaded.