USAID/HAITI SANTÉ ACTIVITY MID-TERM EVALUATION REPORT Haiti Evaluation and Survey Services (ESS) APRIL 2020 This publication was prepared independently by Social Impact, Inc. at the request of the United States Agency for International Development. PHOTO CREDIT: CARIS FOUNDATION USAID/HAITI SANTE ACTIVITY MID-TERM EVALUATION FINAL REPORT Date: April 2020 Submitted To: USAID/Haiti Health Office Evaluation Mechanism Number: AID-521-A-15-00009 Haiti Evaluation and Survey Services for USAID/Haiti This publication was prepared independently by Social Impact, Inc. at the request of the United States Agency for International Development. It was prepared independently by Nanette Barkey (Team Leader), Francois Willy Staco (Assistant Team Leader), and Louise Estavien (Sector Specialist) on behalf of Social Impact, Inc. Contact: Jennifer Mandel, Chief of Party 2300 Clarendon Blvd, Suite 1000 Arlington, VA 22201, USA DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV ABSTRACT Social Impact’s Haiti Evaluation and Survey Services project conducted an independent mid-term evaluation of the United States Agency for International Development (USAID)/Haiti’s Health Service Delivery (HSD) Santé Activity implemented by Caris Foundation International and its partners. Santé began in December 2017 and is scheduled to end in November 2021, with a total estimated ceiling of $98.5 million. The evaluation assesses Santé’s achievement of key results, relevance of its design, ability to respond to requests from key stakeholders, and plans to transition staff to the Ministry of Health. Santé was designed to improve utilization and quality of the health services for a catchment area of more than 4 million people, or 47 percent of Haiti’s population. To do so, Santé worked at the facility and community levels to promote a service-oriented culture, enhance local accountability, and strengthen management and operational capacity. In addition, the Activity aimed to build local organizations’ capacity and transfer implementation responsibility to the Haitian Government. The evaluation team found that Santé has expanded access to and use of essential health services. Improvements were also noted at the community level, particularly due to community health workers’ efforts. Santé has proven adaptable even though the original program assumptions did not hold true. It responded to numerous emerging demands, most notably from the President’s Emergency Program for AIDS Relief (PEPFAR) program, and political, economic and security instability throughout Haiti. However, Santé still faces challenges, particularly in achieving its Health Systems Strengthening Objective, potentially undermining the Activity’s long-term sustainability. ii | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV TABLE OF CONTENTS ABSTRACT I LIST OF TABLES AND FIGURES III ABBREVIATIONS AND ACRONYMS IV EXECUTIVE SUMMARY VI INTRODUCTION 1 EVALUATION METHODOLOGY 7 FINDINGS AND CONCLUSIONS 16 RECOMMENDATIONS 44 ANNEXES 47 ANNEX A: EVALUATION SCOPE OF WORK 47 ANNEX B: DATA COLLECTION TOOLS 57 ANNEX C: INFORMATION SOURCES 74 ANNEX D: EVALUATION MATRIX 77 ANNEX E: DETAILED SAMPLING PLAN 79 ANNEX F: DISCLOSURES OF CONFLICT OF INTEREST 83 ANNEX G: EVALUATION TEAM MEMBERS 86 ANNEX H: STATEMENT OF DIFFERENCE 87 ANNEX I: EVALUATION RESPONSE DOCUMENT 94 iii | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV LIST OF TABLES AND FIGURES Table 1: Activity Summary Information........................................................................................................................1 Table 2: Santé Results Framework................................................................................................................................4 Table 3: Key Informant and Group Interviews by Category and Sex...................................................................8 Table 4: FGDs by Type, Age, and Sex ..........................................................................................................................9 Table 5: Design Weights................................................................................................................................................13 Table 6: Choice of Facility .............................................................................................................................................26 Table 7: Transportation to Facility ..............................................................................................................................26 Table 8: Quality of Reception.......................................................................................................................................27 Table 9: Cleanliness of Facility......................................................................................................................................27 Table 10: Perception of Availability of Medications, Vaccines, and FP Methods at Santé Health Facilities.....................................................................................................................................................29 Table 11: Santé Clients’ Assessment of Completeness of Care ..........................................................................31 Table 12: Antenatal Care Visit Services Received ...................................................................................................33 Table 13: Recommendations for Increasing Health Facility Childbirth (n=669)...............................................33 Table 14: CHWs Provide Information on Oral Serum for Diarrhea..................................................................34 Table 15: HIV Tests and HIV Positive from Eight PEPFAR Site Registers in Q1 2018 and 2019.................35 Table 16: Know Where to Get an HIV Test............................................................................................................36 Table 17: Have Been Offered an HIV Test................................................................................................................36 Table 18: CHWs Assist in Appoint Scheduling ........................................................................................................39 Table 19: Referral System and Transportation at Santé Health Facilities..........................................................39 Figure 1: Activity Overview.............................................................................................................................................2 Figure 2: Santé Theory of Change .................................................................................................................................3 Figure 3: CAS Sampling ..................................................................................................................................................11 Figure 4: % of Santé Respondents Noting the Addition or Removal of Services in the Previous Year............................................................................................................................................................28 Figure 5: Prenatal and Childcare Services at Santé Health Facilities...................................................................29 Figure 6: Handwashing Station at Santé Health Facilities in the Catchment Areas.........................................37 iv | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV ABBREVIATIONS AND ACRONYMS AIDS Acquired Immunodeficiency Syndrome ATL Assistant Team Leader ART Antiretroviral Therapy CAS Catchment Area Survey CHW Community Health Worker CMMB Catholic Medical Mission Board DDG Diagnostic and Development Group DDS Direction Départementale Sanitaire DHS Demographic and Health Survey DO Development Objective DRR Disaster Risk Reduction DSAG Departmental-Level Santé Advisory Group EQ Evaluation Question ESS Evaluation and Survey Services ET Evaluation Team FGD Focus Group Discussion FP Family Planning FY Fiscal Year GBV Gender-Based Violence GI Group Interview GOH Government of Haiti HHF Haitian Health Foundation HIV Human Immunodeficiency Virus HLP Health Leadership Project HSD Health Service Delivery HSS Health Systems Strengthening IHE Institut Haïtien de l'Enfance IP Implementing Partner KII Key Informant Interview LTFU Lost to Follow Up M&E Monitoring and Evaluation MEL Monitoring, Evaluation, and Learning MNCH Maternal, Newborn, and Child Health MOH Ministry of Health MTE Mid-Term Evaluation NGO Non-Governmental Organization OCA Organizational Capacity Assessment OIG Office of Inspector General PEPFAR President’s Emergency Program for AIDS Relief PES Package of Essential Services/Paquet Essentiel des Services PLHIV People Living with HIV PY Project Year SA Sub-Awardee SAG Santé Advisory Group SDE Section d'Énumération (enumeration area) SI Social Impact, Inc. SOW Scope of Work SS Sector Specialist SSQH Services de Santé de Qualité pour Haïti v | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV TA Technical Assistance TL Team Leader USAID United States Agency for International Development USD United States Dollars WASH Water, Sanitation, and Hygiene vi | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV EXECUTIVE SUMMARY The United States Agency for International Development (USAID)/Haiti requested that Social Impact Inc.’s (SI) Evaluation and Survey Services (ESS) activity design and conduct an independent mid-term evaluation (MTE) of the Santé activity, which is implemented by Caris Foundation International and its sub-awardees. BACKGROUND Santé builds on 20 years of USAID investments for health service delivery (HSD) in Haiti and has two key objectives: 1) utilization of quality, essential health services at 164 facilities and in surrounding communities across Haiti, and 2) strengthening local management and operational capacities to deliver health services. Santé technical teams provide coaching and mentoring to facility- and community-based health providers via regular site visits in coordination with Haitian Department Health Offices (Direction Départementale Sanitaire [DDS]). The Ministry of Health’s (MOH) Package of Essential Services (PES) including primary care at facilities and community level interventions across multiple health areas (e.g., Human Immunodeficiency Virus (HIV), maternal, neonatal, and child health, family planning, nutrition, etc.) guided the technical assistance (TA) Santé delivered. The purpose of this report is to present this MTE’s findings, conclusions, and recommendations. Its primary audience is USAID, the President’s Emergency Plan for AIDS Relief (PEPFAR), Caris Foundation and its partners, and the Government of Haiti (GOH), particularly the MOH. The MTE addresses four evaluation questions (EQs): 1. To what extent and in what ways are the original Activity design and assumptions still relevant? In addressing this question, special attention should be given to the relevance or functionality of the National and Regional advisory boards. 2. To what extent and in what ways is Santé effectively balancing achieving its original objectives and addressing current concerns and requests of key stakeholders, especially USAID/Washington, USAID/Haiti, and the Government of Haiti? 3. To what extent and in what ways have Santé’s interventions been effective in achieving key results, especially increasing access to health services? The evaluation should consider client and key stakeholder satisfaction as well as other measures of access to and successful delivery of health services. 4. To what extent and in what ways is Santé planning for the transition of staff paid by Santé to the Ministry of Health? In addressing this question, the Evaluation Team should consider the Activity’s plans for the transfer of staff. EVALUATION METHODOLOGY The ET designed a mixed method MTE, consisting of document review, key informant interviews (KIIs), focus group discussions (FGDs), a catchment area survey (CAS), and HIV register reviews. During two weeks of in-country data collection, the ET conducted 42 KIIs with staff from the implementing organizations, the MOH at central, departmental, and site levels, and the USAID Health Team, as well as reviewed HIV testing and treatment registers at eight Santé sites. The ET hired a data collection firm to hold 30 FGDs and survey 1,900 adults living in catchment areas of Santé-supported facilities. vii | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1 Santé’s original design included core elements related to HSD – continuum of care, scaling-up, technical leads and technical teams – along with Health System Strengthening (HSS) for sustainability through capacity development and Santé Advisory Groups (SAGs). While underlying assumptions are not clearly outlined in Activity documentation, through KIIs with the implementing partner (IP) and USAID, the ET identified four foundational assumptions and aligned them with facts presented throughout various Santé documents. These assumptions included: 1) a comprehensive and multi-level transition from the previous USAID HSD Activity, 2) a team of equal implementation partners, 3) the ability to conduct regular site visits to build technical capacity, and 4) the need to establish good collaboration with the MOH from the start. Several assumptions were challenged during the first two years of implementation in the following ways: 1. Although challenges were anticipated, the relationship with the GOH was not strong initially, with a delayed Activity launch, resulting in a lack of a comprehensive and multi-level transition between USAID’s previous HSD activity and Santé. 2. Likewise, Santé spent much of its time in the first two years working to build relationships and trust with the MOH. 3. While their overall vision is the same, the MOH and USAID/Santé have different implementation priorities that had to be balanced both for relationship and implementation management. 4. Santé launched just before Haiti slipped into a period of sustained political and economic instability. The security situation hindered implementation, including reducing the number and frequency of site visits and onsite support to health facilities. Santé adapted well to many of the challenges to its underlying assumptions, particularly shifting USAID and PEPFAR priorities. While adaptability is a core element of Santé’s design, these challenges resulted in several changes affecting implementation. The IP has devoted less attention to certain design elements, notably HSS activities. Capacity building for sub-awardees, collaboration with the DDS, and community engagement via the SAGs are three other design elements toward sustainability that have not been executed as planned. EVALUATION QUESTION 2 Santé responded to requests from its key stakeholders (PEPFAR, USAID/Washington, USAID/Haiti, MOH) by incorporating them into existing activities or shifting resources as required, with USAID’s agreement. Most key informants stated that these demands, the most far reaching of which were from PEPFAR and MOH, were within Santé’s original scope of work, and that their inclusion did not distract from the overall objective of improving health service delivery. The biggest changes affecting Santé’s implementation have come in response to new PEPFAR requirements, under which Santé is partially funded. PEPFAR required Santé to change its approaches to HIV testing, treatment, treatment adherence, viral load testing, and indicator reporting. KIIs and FGDs with community health workers (CHWs) indicated that Santé responded with more frequent visits to PEPFAR sites, cross-training all technical staff on HIV, shifting resources to ensure the new HIV drug was properly introduced, recruiting CHWs to improve treatment adherence, and hiring extra staff for viii | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV data reporting. While PEPFAR’s requirements necessitated that Santé shift much of its focus to HIV, there have been positive secondary effects. For example, while cross-training technical leads started with non-HIV specialists, eventually all technical leads received training on the other health areas, and CHWs’ expanded role in managing HIV clients facilitated greater local treatment and tracking. The desk review indicated and KIIs concurred that Santé was also affected by MOH requests, the most important of which was MOH’s insistence on creating a Cellule de Coordination (coordination unit) and meetings held to agree on several concerns before implementation could begin. This resulted in Santé’s slow start. The MOH also asked for assistance transporting vaccines and the provision of patient registers to health facilities. Santé easily managed vaccine transport during routine site visits, while monies needed to procure the registers required USAID approval. The MOH, USAID/Washington, and USAID/Haiti requests were generally seen as mutually beneficial despite requiring programmatic and contractual changes. PEPFAR’s requests and requirements, in contrast, which many KIIs ultimately found valuable, resulted in more significant adjustments and reprioritization in the Activity itself. EVALUATION QUESTION 3 Most KII and FGD respondents and the CAS indicated that Santé improved access to and quality of health services at its facilities and the community level. Santé has leveraged physical improvements, supplies, and equipment at the facilities, to attract community interest, while Santé technical teams’ training and mentoring helped providers improve their clinical skills. Clients perceived that health care providers were more available, with newly acquired skills/knowledge, and providing client-centered care. Another positive change are the mobile clinics that link community members with more services locally and strengthen collaboration between CHWs and facility-based health providers. CHWs have directly provided more services, facilitated referrals more frequently, and mobilized community members to attend mobile clinics. This is due partly to Santé’s training, supervision, and mentoring of CHWs, and partly due to their expanded portfolio of community-level management of illness and preventive care. Clients reported improved satisfaction with their facilities as a result. However, according to key informants, some Santé facilities still have shortages of supplies, medications, services, and vaccines – and even providers. In addition, transportation for CHWs and access, especially for pregnant women to complete their antenatal care visits and deliver at health facilities, remain a challenge. While some of these challenges are within Santé’s purview (availability and quality of services), others are beyond its control (staffing, vaccines, medications). In addition, communication remains a problem for some survey participants who expressed that 1) some clients feel they are getting partial information about their illnesses, 2) some clients do not feel their provider is willing to talk about family planning, and 3) a couple of key elements, including birth planning, are not always included in antenatal visits. Given PEPFAR’s role in the Activity, the ET looked in detail at HIV services. Santé has improved HIV services at the facility and community levels. Santé effectively rolled out the new mandated HIV treatment, as well as the new HIV testing strategy, which has proven more efficient in identifying HIV positive clients. The ET notes that slow processing of viral load tests at the national laboratory causes delays in obtaining results and makes it more difficult for providers and people living with HIV (PLHIV) to know whether HIV suppression is effective. CHWs have played an important role in improving HIV ix | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV patients’ care, as they supported efforts to decrease numbers of patients lost to follow up (LTFU) and encourage patients to adhere to treatment. Despite many gains, challenges to quality health care remain formidable, including the ongoing economic and socio-political crises, security issues, and geographical constraints. Haiti’s mountainous landscape also poses challenges for patients who may live within a short distance of a health facility but, due to the terrain and/or lack of transport options, take much longer to reach it. EVALUATION QUESTION 4 The desk review and KIIs found that Santé has developed a basic road map for the transfer of the contractual staff salaries to the MOH in coordination with the Health Leadership Program (HLP), a USAID-funded Activity to build the capacity and finance of the GOH health sector. Santé is actively taking steps to prepare for this transfer by working to align the salaries, assignments, titles, and certifications for these staff. However, a number of challenges remain. Key informants note that for the plans to be effective, they need comprehensive USAID and MOH support. While HLP and MOH have developed a broad transition plan that they do agree on, it is not as detailed as informants believe the final roadmap needs to be. The final roadmap needs to be comprehensive and politically sensitive, taking into consideration the convoluted nomination process to government employment, as well as the legal requirements and responsibilities the government assumes for its staff. In addition, none of the KIIs or FGD respondents identified a funding source or management approach that would allow the MOH to take on an additional 1,600 workers’ salaries, currently funded by Santé. RECOMMENDATIONS RECOMMENDATIONS FOR SANTÉ The ET designed the following recommendations for Santé leadership to adjust programming so that it can effectively reach its results by the Activity’s end (please see report body for more details). 1. Continue to improve health service delivery with even greater attention to quality of services. Consider starting or revitalizing quality assurance committees. Just as Santé was able to decease LTFU by prioritizing and monitoring progress, Santé should use this approach to identify and monitor other key gaps in non-HIV areas. 2. Maintain emphasis on community health workers as a key component of health system strengthening. Ensure all CHWs are certified, continue to receive frequent and quality supervision, and advocate for improved working conditions. 3. Continue strategies to improve dialogue and relationship with MOH. Use the coordination committee (Cellule de Coordination) and other platforms for dialogue with MOH counterparts at the central and departmental levels. Align Santé site visits with DDS work plans and support their staff to participate in/lead coaching and mentoring. 4. Renew emphasis on the sub-awardees’ (SA) systematic capacity building. Conduct proposed Organizational Capacity Assessments (OCAs) and use results to create tailored capacity development plans for each SA. 5. Continue paying contractual staff and coordinate with the MOH and HLP as they finalize the broad plan. Santé should maintain communication with DDS, contractual staff, x | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV and the Cellule de Coordination while the MOH to lead the process of staff transfer. RECOMMENDATIONS FOR USAID The ET designed the following recommendations for USAID to support Santé to achieve its intended results through the duration of the Activity. 1. Coordinate the process led by HLP for the transfer of staff currently paid by Santé to the MOH. Using the roadmap and Santé’s support, USAID should collaborate with HLP in high-level discussions with the MOH and other donors about the absorption of these staff and funding to cover their salaries and share developments in the process with Santé. 2. Support Santé in improving relationships with MOH and expanding on health systems strengthening activities. Ways to do this may include maintaining open communication with the MOH via the Cellule de Coordination, encouraging co-planning with DDS, mitigating PEPFAR pressure, and/or facilitating capacity building for strong sub-awardees. 3. Conduct a Learning Event on the use of Advisory Boards. While the advisory boards were in the original design, they are practically difficult to implement and, while planned each year, have not taken off. USAID should take a learning approach to discuss the pros and cons of this approach as well as the associated trade-offs to understand if a shift in Activity design makes sense. FOR FUTURE ACTIVITY DESIGN AND IMPLEMENTATION The ET designed the following recommendations for implementers and policymakers establishing future health programming in Haiti. 1. Allow time in activity design for relationship building with government counterparts. Activity designs should not assume strong relationships with the MOH from the start. Implementers should create a relationship-building strategy, adjusting their approach as trust grows. They should incorporate trust-building activities into their workplans and consider how factors such as co-location and site visits factor into that plan. 2. Ensure culture of adaptive management such that revisions to policies and requirements from PEPFAR, GOH, or USAID/Washington can be incorporated as opportunities. Ask the IP to assess the overall risks of those Activity changes as a whole and provide the opportunity to suggest and make workplan adjustments to compensate for any changes. Activities could include relationship remediation, workload adjustments, and/or target adjustments. 1 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV INTRODUCTION The United States Agency for International Development (USAID) requested that Social Impact, Inc. (SI)’s Haiti Evaluation and Survey Services (ESS) design and implement an independent mid-term evaluation (MTE) of its Health Services Delivery (HSD) Activity, Santé (see Annex A: Evaluation Scope of Work). On December 1, 2017, USAID made a four-year Health Service Delivery Cooperative Agreement Award to the Caris Foundation. With a ceiling of US$98.5 million, Santé operates in Haiti’s ten departments (administrative divisions) to support 164 health facilities and their respective communities across the country to deliver the Haitian Ministry of Health (MOH)’s Package of Essential Services (PES). Table 1: Activity Summary Information This report’s purpose is to present the main findings, conclusions, and recommendations of this mid￾term evaluation. The primary stakeholders for this evaluation include USAID/Haiti, the President’s Emergency Plan for AIDS Relief (PEPFAR), the Caris Foundation and its partners,1 and the Government of Haiti (GOH), particularly the MOH. BACKGROUND OF LOCAL CONTEXT In Haiti, some health indicators have shown recent improvement, such as reduction in the HIV prevalence rate among adults and reduction in the total fertility rate. However, rates of maternal, neonatal, and infant mortality have remained stubbornly high, and immunization coverage has actually decreased since 2012.2 The Haiti Demographic and Health Survey (DHS) found that treatment seeking for children with diarrhea and acute respiratory infection symptoms decreased from 2012 to 2017 (52.9 percent to 39.3 percent and 39.4 percent to 32.7 percent, respectively).3 One in five children continue to be stunted.4 Additionally, just 41.6 percent of women gave birth attended by skilled health personnel according to the 2017 DHS, a slight increase from 37.3 percent recorded in 2012, but still low by global standards.5 Given these worrying trends, USAID/Haiti prioritized health as one of its four focus areas in its 2018 strategy. Santé represents the Mission’s most prominent investment in primary health care service 1 Partners initially included the Haitian Health Foundation and Catholic Medical Mission Board. 2 USAID Country Profile. January 2020. https://www.usaid.gov/sites/default/files/documents/1862/USAID_Haiti_Health_Fact_Sheet_-_January_2020.pdf 3 Stat Compiler: Haiti. The DHS Program, USAID. Accessed: April 16, 2020. https://www.statcompiler.com/en/# 4 Ibid. 5 Ibid. ACTIVITY SUMMARY INFORMATION Activity Name: Santé Implementer: Caris Foundation Agreement/Contract #: 72052118CA00001 Total Estimated Ceiling of the Evaluated Activity: $98,500,000.00 Life of Activity: December 2017 – November 2021 Active Geographic Regions: 10 geographic departments Development Objective(s) DO3: Health outcomes increased USAID Office: Health 2 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV improvement, complemented by investments in health systems strengthening (HSS) and water, sanitation, and hygiene (WASH). USAID/Haiti launched Santé in 2018, just before Haiti slipped into a period of sustained political and economic instability. In July 2018, the government removed fuel subsidies, which led to widespread and violent protests. Much of Santé’s implementation occurred in a context of frequent demonstrations, sporadic violence, and chronic fuel shortages, which led to profound dissatisfaction with the government and the cancellation of parliamentary elections.6 The unrest resulted in several extended periods ranging from a few days to up to two months of “paye lok,” during which the entire country was effectively paralyzed. This has subsequently affected Haiti’s health system and the many actors involved in it, including Santé. Haiti also saw a number of natural disasters, such as flooding and a sizeable earthquake, over the first two years of the Activity. ACTIVITY OVERVIEW Santé builds on more than 20 years of USAID investments for health service delivery in Haiti, including the previous HSD project (Services de Santé de Qualité pour Haïti [SSQH]). Santé is a complex Activity that is implementing multiple interventions simultaneously across numerous areas, including Essential Services, Child and Maternal Health, WASH, Tuberculous, Family Planning, and HIV/AIDS (see Figure 1), working in the same health facilities assisted by the SSQH Activity. Santé receives a portion of its funding from PEPFAR, and about one-quarter of the health facilities it supports are designated as PEPFAR sites focused on facilitating access to high quality, stigma-free, and comprehensive HIV/AIDS prevention, testing, and treatment.7 The Activity’s primary beneficiaries are women, children, youth, and people living with HIV (PLHIV).8 Figure 1: Activity Overview Strategies Areas Beneficiaries Increase Access and Utilization Child and Maternal Health Family Planning Nutrition Women, Children, Youth Strengthen Local Capacity WASH HIV & AIDS Tuberculous People living with HIV THEORY OF CHANGE The Activity’s theory of change posits that by 1) reducing barriers to quality health care, while also increasing the utilization and quality of essential health services, and 2) strengthening local management 6 Congressional Research Service 2020. 7 Santé PY3 Work Plan, pages 14. 8 Santé PY3 Work Plan, pages 13 and 80. 3 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV and operational capacities to deliver health services, Santé will reduce morbidity and mortality, improve quality of life, and build resilient communities (Figure 2). Figure 2: Santé Theory of Change RESULTS FRAMEWORK Santé aims to improve health service delivery (HSD) by providing technical support to health workers and Community Health Workers (CHWs) through supportive supervision, monitoring, and training, certification, and provision of supplies, materials, and equipment. Santé also aims to improve health facilities’ laboratory and medical waste interventions, the availability of health providers, and business planning for results-based financing (RBF) at designated health facilities, as well as other HSS activities.9 At the national level, Santé is part of a coordination unit with the MOH and USAID. The ten Department Health Offices (Direction Départementale Sanitaire [DDS]) are core partners in both improving the quality of HSD offered and HSS. Santé provides the DDS with a monthly stipend and technical assistance on planning and budgeting. Santé’s Project Year (PY) 1 included transitioning from SSQH, conducting assessments, intensive hiring and training, setting up or reinforcing supply chains, and strengthening access to the MOH’s integrated 9 Santé PY2 Annual Report, page 14. 4 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV PES at the primary care and community levels.10 PY2 began with a comprehensive review of interventions, accomplishments, and challenges, and saw a renewed focus on HIV to meet PEPFAR guidelines and targets.11 At the end of PY1 and throughout PY2, Santé had to adapt to a difficult operating environment due to natural disasters and increased insecurity throughout Haiti, including in the implementation zone, which hampered its interventions.12 As noted in the Results Framework (Table 2), Santé’s first objective is to increase access to and use of quality health services at the health facility and the community level. At the health facility level, HSD interventions focus on, but are not limited to, the health areas of Maternal and Child Health, Reproductive Health and Family Planning, HIV/AIDS and TB, Nutrition, and WASH. At the community level, it supports linkages between health facilities and communities through CHWs, advisory groups, referral networks, transportation, and mobile clinics that bring facility staff and CHWs together to deliver services where clients live. Objective Two focuses on HSS to ensure the sustainability of improvements in HSD, and includes a host of interventions dedicated to capacity building, accountability, and financing, as well as data systems and emergency preparedness. Transitioning the management of HSD to the Haitian MOH is a key aspect of Objective Two, including transferring the salary payments for 1,600 contracted health workers who work at MOH sites and whose salaries are currently paid by Santé to the MOH. The Activity’s Results Framework in Table 2 provides more information about Santé’s two objectives. Table 2: Santé Results Framework STRATEGIC OBJECTIVE RESULT 1. INCREASED ACCESS TO AND UTILIZATION OF QUALITY HEALTH SERVICES STRATEGIC OBJECTIVE RESULT 2: STRENGTHENED LOCAL MANAGEMENT AND OPERATIONAL CAPACITIES TO DELIVER HEALTH SERVICES Santé health service facilities strengthened by increased and fully integrated MSPP PES service delivery. Sub-Result 1.1: Maternal and Child Health (MCH) 1.1.1. Santé site health providers provide high-impact, life￾saving MNCH interventions 1.1.2. Distance learning system fully implemented and operational to provide RMNCH information and training 1.1.3. GOH health policies incorporate updated evidence￾based RMNCH policies and guidelines Sub-R 1.2. Family Planning and Reproductive Health (FPRH) 1.2.1. FPRH compliance and coverage ensured across Santé health facilities 1.2.2. FPRH priority sites providing long-term and permanent FP methods Sub-R 1.3. HIV/AIDS & TB 1.3.1. Comprehensive HIV/TB service delivery in PEPFAR sites strengthened Sub-R 1.4. Nutrition 1.4.1. Comprehensive nutrition training at the community level Sub-Result 2.1. Increased financial, performance, and political/democratic accountability 2.1.1. National Advisory Group highly functional 2.1.2. Strengthened DDS financial management capacity 2.1.3. DDS authority and capacity to manage, supervise, and monitor service delivery strengthened 2.1.4. Data management systems and processes operationalized at the DDS and facility levels 2.1.5. MSPP health facilities staff salaries supported 2.1.6. Data-driven decision-making at DDS level Sub-R 2.2. Creation and early implementation of a roadmap for transitioning of financing for Santé Sub-R 2.3. Facilitated Direct Agreements between USAID and five NGOs Sub-R 2.4. Expanded Results-Based Financing program for health facilities, NGOs 2.4.1. Santé sites have successfully implemented RBF strategies Sub-R 2.5. Increased local emergency preparedness 2.5.1. Emergency Response Systems in place 10 Santé PY1 Annual Report, page 3. 11 Santé PY2 Annual Report, page 7. 12 Santé PY2 Annual Report, page 1. 5 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV STRATEGIC OBJECTIVE RESULT 1. INCREASED ACCESS TO AND UTILIZATION OF QUALITY HEALTH SERVICES STRATEGIC OBJECTIVE RESULT 2: STRENGTHENED LOCAL MANAGEMENT AND OPERATIONAL CAPACITIES TO DELIVER HEALTH SERVICES 1.4.2. Strengthened ability to treat MAM and SAM in the community and at the site 1.4.3 Improved support for maternal nutrition Sub-R 1.5. WASH 1.5.1. Health facilities observing optimal hygiene practices 1.5.2. Communities strengthened in prevention of waterborne illness and treatment-seeking behavior Linkages between services and communities are ensured. Sub-R 1.6. Santé Advisory Groups at national, departmental, and site levels have achieved gender equity in participation and decision-making Sub-R 1.7. ASCPs successfully promote the MSPP PES Sub-R 1.8. Functionality of health referral networks improved Sub-R 1.9. Emergency transportation and communication protocols successfully implemented in referral networks EVALUATION PURPOSE & QUESTIONS This MTE’s purpose is to generate substantial information to guide the ongoing implementation, provide a benchmark that will be used to measure programmatic results, and uncover potential weaknesses in program implementation. It will serve as a mid-point reference to assess USAID’s investments in Santé since its launch in December 2017. This MTE will also alert stakeholders to changes in underlying assumptions and provide a firm foundation for future monitoring and evaluation. This MTE addresses four evaluation questions (EQs). Table 3: Evaluation Questions EQ 1 To what extent and in what ways are the original Activity design and assumptions still relevant? In addressing this question, special attention should be given to the relevance or functionality of the National and Regional advisory boards. EQ 2 To what extent and in what ways is Santé effectively balancing achieving its original objectives and addressing current concerns and requests of key stakeholders, especially USAID/Washington, USAID/Haiti and the Government of Haiti? EQ 3 To what extent and in what ways have Santé’s interventions been effective in achieving key results, especially increasing access to health services? The evaluation should consider client and key stakeholder satisfaction as well as other measures of access to and successful delivery of health services. 6 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV EQ 4 To what extent and in what ways is Santé planning for the transition of staff paid by Santé to the Ministry of Health? In addressing this question, the Evaluation Team should consider the Activity’s plans for the transfer of staff. Note that EQs 1 and 2 are inter-related in that they both address how Santé has evolved since its original conception, in response to the Haitian operating context and requests from key stakeholders. The MTE addressed these as separate questions while recognizing that Santé has adapted its design and implementation in response to multiple factors and demands. EQ 4 has been updated from the evaluation SOW. EQ 4 originally asked “To what extent and in what ways is Santé planning for the transition of its Activity to the Ministry of Health? In addressing this question, the Evaluation Team should consider the Activity’s capacity-building and hand over plan.” During introductory meetings, USAID/Haiti clarified that the MTE should specifically focus on the transfer of paid staff, which the MTE expanded upon in the findings section below. In addition, the evaluation team (ET) views all four EQs through the lens of the political, economic, and security issues that have beleaguered Haiti in the two years since the Activity began. While not directly a focus of this evaluation, the ET recognizes the impact of the current instability on Santé’s design assumptions (EQ 1), balancing of its original objectives (EQ 2), achievement of key results (EQ 3), and even the transfer of contractual staff (EQ 4). Where appropriate, the ET included findings related to this context below. 7 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV EVALUATION METHODOLOGY The Santé MTE used a mixed-methods design to answer the EQs. The ET began with a desk review of Santé and MOH documents to produce an Inception and Evaluation Design Report. The team then conducted primary data collection consisting of key informant interviews (KIIs), focus group discussions (FGDs), a catchment area survey (CAS), and reviews of HIV testing and treatment registers. The ET developed data collection tools (see Annex B: Data Collection Tools) for fieldwork in English, which were later translated to French or Haitian Creole, as appropriate. The ET designed these tools to address the EQs, building on the information from the document review, initial discussions with USAID, and the ET’s evaluation design. ESS contracted a local data collection firm, Diagnostic and Development Group (DDG), to conduct the FGDs and CAS. The ET collaborated with DDG to pilot and further refine the FGDs and CAS after development, and to supervise the data collection team training and implementation. The ET conducted all KIIs and the review of HIV registers. DATA COLLECTION DOCUMENT REVIEW The ET reviewed 21 Santé and MOH documents to understand the Activity design and implementation, extract findings relevant to the EQs, and inform data collection tool development so that the information obtained through the instruments would appropriately complement background document information. Categories of documents reviewed are summarized below and detailed in Annex C: Information Sources: • Santé Annual and Quarterly Reports • Santé Annual Work Plans • Santé Monitoring and Evaluation (M&E) Plans • Activity description, modifications, and mitigation plans • The MOH’s PES • The MOH’s Organization of Community Health Services Guide • Context documentation • Journals and publications KEY INFORMANT INTERVIEWS KIIs were designed to fill in gaps from the document review on Activity progress and solicit information about changes to the design and assumptions, requests from stakeholders, achievements toward key results, and lessons learned to date. The ET chose key informants to represent the perspectives of various stakeholder groups involved in implementing Santé, including staff from the IP (Caris) and sub-awardee (SA) organizations, the MOH at the central, departmental, and site levels, and the USAID Health team. The ET used three separate guides, one for each major stakeholder group, USAID, implementors (IP and SA), and MOH staff, with questions tailored for each group or sub-group (see Annex B: Data Collection Tools for guides). Questions also overlapped to ensure triangulation of information within and across stakeholder groups. 8 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Findings from KIIs/group interviews (GIs) contributed to the ET’s responses to all EQs (see Annex D: Evaluation Matrix for more information). The ET conducted the interviews face-to-face in French or English with key informants across Haiti, or, when necessary, via Skype (n=2). At the beginning of each interview, the team asked respondents for consent to be tape recorded, and all recordings were destroyed after the team finalized comprehensive notes detailing each interview. In total, the ET conducted 42 key informant individual or group interviews with 68 people (33 women and 35 men) from various stakeholder categories. These included 34 individual and eight group interviews (at the original key informant’s request). Six of the 11 SA interviews chose to invite more than one person to the interview, usually combining medical and administrative staff. One SA convened 17 people for the initial discussion, which the ET followed up with six interviews with one or two people per health sector. Table 3 shows the distribution of interviews by key stakeholder group and sex. Table 3: Key Informant and Group Interviews by Category and Sex KEY INFORMANT GROUP # OF INTERVIEWS TOTAL RESPONDENTS WOMEN MEN Implementers IP Staff 10 10 4 6 Sub-Awardee Staff 11 34 16 18 Ministry of Health Department Level 6 8 4 4 Site Level 6 7 4 3 Central Level 4 4 1 3 Donor Donor 5 5 4 1 Total 42 68 33 35 FOCUS GROUP DISCUSSIONS The evaluation design included focus group discussions with Santé health facility clients and CHWs involved in Santé’s implementation. The ET designed FGDs with clients to elicit their perceptions of Santé interventions at the community and health facility levels, as well as their use of and satisfaction with health services in their communities. In FGDs with CHWs, the ET asked about their interactions with Santé and changes they have observed in their health service delivery and community mobilization interventions. While data from the CHW FGDs addresses all the EQs, data from the client FGDs specifically addresses EQ 3. FGD facilitators and note-takers from DDG underwent training on FGD methodology and techniques, Santé interventions, the MTE questions and methods, and research ethics. FGD facilitators and note￾takers met with the ET to review, pilot, and refine the Creole versions of the FGD guides. The ET captured audio recordings of all FGDs with respondent permission. DDG then combined these audio recordings and notes into a comprehensive set of digitalized notes (one per FGD). Per the evaluation design, DDG conducted a total of 30 FGDs in all ten of Haiti’s departments; three FGDs per department, one with CHWs, and one each with male and female clients of Santé-supported 9 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV sites. The ET randomly selected two Santé-supported sites per department and shared them with DDG, who then recruited male clients from one site and female clients from the other site from a list of clients provided by the health facility. Table 4 depicts the number of participants, age (mean and range), and sex distribution of FGDs by category. Table 4: FGDs by Type, Age, and Sex The FGDs reflect good gender balance among both clients and CHWs, and the age range allows for a variety of perspectives and potential health needs (for clients). In addition to age and gender, the ET asked CHWs how long they have worked as a CHW, with an average service length of 11.2 years and a range of one to 31 years, reflecting a wide range of potential experience both before and during the Activity. CATCHMENT AREA SURVEY Neither the Santé Activity nor the previous SSQH Activity conducted a baseline study. The ET designed a catchment area survey that was intended to represent adult residents within the communes where Santé health facilities are located. By asking respondents what facilities they visit for care, the ET was able to capture Santé clients’ perceptions and utilization of services at Santé facilities, as well as reasons for residents’ choice of facility. The information captured through the survey addresses EQ 3. Given Santé’s emphasis on child and maternal health, the ET oversampled breastfeeding and pregnant women and caregivers of children under five as part of the survey. The final sampling plan included 800 men and 1,000 women, 400 of whom were to be pregnant, breastfeeding, or caregivers of children under five. The ET calculated the sample to yield statistics disaggregated by sex with a margin of error of +/- 4 percent and a 95 percent confidence level. For more details on the sampling approach, please see Annex E. The cross-sectional catchment area survey used a multi-stage stratified cluster sampling design. 1. Site Selection: Health Facility: The ET randomly selected 40 non-PEPFAR Santé-supported health facilities proportionally from three geographic areas – North, Center, and South. In addition, the ET randomly selected 20 of the 46 Santé-supported PEPFAR sites leading to four total strata: PEPFAR all regions, and non-PEPFAR North, Center, and South. Four of the selected sites were deemed a security risk and replaced by randomly selecting an alternative from their respective geographic area (Center region) and type prior to data collection (non￾PEPFAR or PEPFAR). Finally, the ET selected four additional sites (two PEPFAR, two non￾PEPFAR) for each of the three regions for a total of 12 potential replacement sites during data collection. The ET could not reach one Section d'Énumération (enumeration area) (SDE) in the TYPE OF PARTICIPANT MEAN AGE (RANGE OF AGE) NUMBER (BY SEX) Community Health Workers 49.9 (29-65 years) 82 (40 men/42 women) Female Clients 36.2 (19-71 years) 111 (all women) Male Clients 36.3 (18-77 years) 112 (all men) 10 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV South region due to weather conditions (heavy winds prevented the teams from reaching an island by boat), and replaced it with another SDE in the South region from the list. In the end, the CAS took place at a total of 57 health facilities with the ET dropping three PEPFAR sites due to issues with accessibility. 2. Enumeration Area Selection: After site selection, the ET then mapped the catchment areas, including all census enumeration areas – called SDEs in Haiti – within one kilometer of rural health facilities and a half kilometer of urban health facilities. The ET then randomly chose one SDE for each of the 60 health facilities for the enumerators to survey. Once surveying was underway, instead of replacing the three PEPFAR health facilities that were dropped under Step 1, the data collection firm visited a second SDE for a health facility already on their list to bring the total back to 60 SDEs. An additional SDE was added in one location where the selected SDE did not meet the 30-household minimum. Therefore, the ET conducted surveys in a total of 61 (rather than 60) SDEs for 57 health facilities. 3. Household Selection: During data collection, survey teams randomly selected 30 households within each SDE through a “random walk” methodology. Upon arrival, the survey team started in the center of the SDE, spun a pencil to determine the direction in which to walk, and walked until they encountered the first house. After that point, the team skipped N# houses before seeking their next respondent. N was determined by taking the number of households in the area and dividing it by 30; if the SDE contained 90 households the survey team visited every third household until they had completed 30 surveys. One SDE did not have the 30 households required, so the team used an adjoining SDE within the same catchment area of the original health facility, as noted under Step 2 above. 4. Respondent Selection: The final stage of sampling consisted of household member selection. Enumerators listed names all adult (18+ years) household members in a roster, along with their sex and breastfeeding/pregnancy/caregiver status. The ET instructed teams to prioritize selection of pregnant and breastfeeding women, and caregivers of children under five, until they reached a target of 400. After achieving that target, they were to return to random selection in which the data collection device (tablet) had been preprogrammed to randomly select a respondent form the roster. However, in practice, enumerators did not return to random selection after the targeted quota was met, leading to a greater oversampling of women than planned, and a resulting proportionally low percentage of men (see Figure 3 below). The final sample of Santé clients (55 percent of total respondents) was 85 percent female and 15 percent male. 11 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Figure 3: CAS Sampling The ET trained a total of 28 enumerators and six supervisors from the data collection firm, and 26 were hired for the CAS. The ET piloted and refined the questionnaire both during training and after the pilot. In total, the DDG conducted surveys with 1,900 respondents, 1,043 of whom were Santé clients. HIV REGISTER REVIEW AND EXTRACTION Because the ET did not design the survey to be an in-depth exploration of HIV testing and treatment given the prevalence rate of 2 percent in the country,13 PEPFAR’s emphasis on testing and treating people living with AIDS and achieving suppression of their viral load meant that these interventions required a deeper exploration at Santé-supported PEPFAR sites. ET members visited eight PEPFAR sites during the primary data collection and reviewed their HIV Testing and Antiretroviral Therapy registers for two specific time periods (October-December 2018 and October-December 2019) using a standardized tool to collect data on the number of people tested, number found to be positive, number started on treatment, and number with a viral load test completed. Quarter 1 of FY2019 and Quarter 1 of FY2020 were chosen to compare the same quarter over two years of the Activity in order `to capture changes in the HIV indicators in response to changes in PEPFAR programming. The ET compared these results against the data from the Santé database for the corresponding indicators during the same time period, and used findings to address EQ 3. DATA QUALITY ASSURANCE ET members and note-takers took detailed notes of KIIs/GIs and cleaned and shared detailed electronic summaries on a rolling basis throughout fieldwork. The ET also conducted internal debriefs regularly throughout fieldwork to discuss progress, make any needed adjustments in the evaluation schedule, and confer with the data collection firm about progress on the FGDs and CAS. The FGD facilitators and note takers submitted detailed summary notes and audio recordings to DDG for quality review prior to DDG’s sharing them with the ET and ESS. DDG and the ET anonymized the detailed KII and FGD notes and carefully reviewed them before sharing for coding. 13 Country Profile: Haiti. UNAIDS. Accessed: April 16, 2020. https://www.unaids.org/en/regionscountries/countries/haiti 1. Site Selection 2. SDE Selection 3. Household Selection 4. Respondent Selection 12 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV The ET checked the survey data for completeness and accuracy as they were collected. The ET ensured data quality for the CAS during fieldwork using three techniques: 1) supervisors accompanied enumerators to a household during the survey administration (n=98); 2) supervisors and enumerators both recorded responses (co-enumerated) to some questionnaires on separate devices, and recordings were compared (n=49); and 3) supervisors completed backchecks (audits) of completed questionnaires (n=180). In addition, the ET did a preliminary check on the first dataset of 100 surveys for completeness and accuracy during data collection. Enumerators uploaded the CAS data in real time from the electronic devices on which they were captured, connectivity permitting, and were then reviewed by the DDG quality control team before being shared with the ET. The ET checked, cleaned, and analyzed the full survey data set using the methods described below. DATA ANALYSIS The ET employed several data analysis methods to identify key findings from both the document review and collected data, as well as to draw conclusions and make recommendations. The type of analyses depended on the specific data being assessed (e.g., content analysis for qualitative KII and FGD data and descriptive statistics for the quantitative CAS and HIV register data). Analysis methods for the qualitative and quantitative data are discussed separately below. In addition, throughout data collection and analysis, the Team Leader captured preliminary findings, conclusions, and recommendations (FCR) by EQ in an FCR matrix. The matrix: 1) ensures the ET prepares a systematic and thorough response to each EQ; 2) verifies that preliminary analysis accounts for gender and social dimensions; 3) identifies any gaps where additional clarification or analysis may be necessary; and 4) serves as the basis for developing the evaluation report. To ensure the reliability and validity of the team’s findings, they also used the three types of triangulation relevant to this evaluation: methodological (triangulation across data collection methods – FGDs, KII/GIs, CAS, document review [DR]); data (triangulation across stakeholder groups, geographic areas, and by sex); and researcher (across the evaluation team members).14 QUALITATIVE ANALYSIS The Team Leader oversaw and managed a systematic, structured, and rigorous analysis of the qualitative data. This began with an identification of the main themes and sub-themes that emerged from each KII and FGD as relevant to the four EQs. Based on these themes, the team developed a coding framework in a tally sheet (Excel matrix), in which the first column contains the codes/themes (generally organized by evaluation question) and each subsequent column reflects whether the ET identified those themes/codes in each KII and FGD. To verify the identification of themes/codes present in the qualitative data, a member of the ET and one note-taker independently coded each set of notes, then compared their coding and reconciled any differences before adding to the broader tally sheet. This approach improves replicability and allows clear identification of the main findings as well as any patterns in the data. The ET identified patterns through triangulation and comparison within and across 14 Denzin, N. (2017). Sociological Methods: A Sourcebook. Routledge. ISBN-13: 978-0202308401. https://www.betterevaluation.org/en/evaluation-options/triangulation. 13 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV stakeholder groups, by type of health facility, and sex. Using the tally sheet as the framework, the ET applied several analysis methods to provide evidence for the MTE’s findings and conclusions. 1. Content Analysis – Content analysis entailed the ET’s intensive review of the tally sheet to identify and highlight the most notable themes that addressed each evaluation question. 2. Comparative Analysis – The ET compared KII and FGD data within and across stakeholder groups, by health facility type, and sex to assess the extent to which perspectives converged and/or diverged. This method enabled the ET to look for trends within and across sub-groups. 3. Gender Analysis – The ET disaggregated all data collected through KIIs and FGDs by sex and analyzed for effects on both male and female beneficiaries to identify any significant differences. QUANTITATIVE ANALYSIS Table 5: Design Weights The ET conducted quantitative analysis on the CAS and HIV registers. For the CAS, given the targeted sampling approach, the ET’s analysis used design and post-stratification (post-data collection) weights to calibrate estimates such that they were reflective of the overall population. First, the ET constructed design weights to consider the probability of selection across the stages of sampling stratification (see Catchment Area Survey above), including d1) Probability of health facility selection, d2) Probability of SDE selection, d3) Probability of household selection, and d4) Probability of individual selected ( ). The ET calculated weights and applied both at the individual and household level. In other words, design weights considered the probability of a household being selected (1/d1*d2*d3), and then the WEIGHT CALCULATION d1 Probability of health facility selection within given strata (PEPFAR (all), Non-PEPFAR North, Center, South) Within stratum: # of health facilities selected/total # of health facilities d2 Probability of SDE selection with health facility “catchment” area of 0.5 (urban) or 1 (rural) km from facility Within catchment: # SDE selected/total # of SDE d3 Probability of household selection within SDE Within SDE: # of households surveyed/total # of household d4 Probability of individual of a specific type 1/total # of eligible types of individuals within the household WEIGHT CALCULATION d1 Probability of health facility selection within given strata (PEPFAR (all), Non-PEPFAR North, Center, South) Within stratum: # of health facilities selected/total # of health facilities d2 Probability of SDE selection with health facility “catchment” area of 0.5 (urban) or 1 (rural) km from facility Within catchment: # SDE selected/total # of SDE d3 Probability of household selection within SDE Within SDE: # of households surveyed/total # of household d4 Probability of individual of a specific type 1/total # of eligible types of individuals within the household 14 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV probability of an individual within the household being selected based on respondent type including A) all eligible adults, B) adult caregiver of child under five, or C) pregnant/breastfeeding woman or caregiver to child under five (1/d1*d2*d3*d4_type). In addition to design weights, the oversampling of specific populations required the ET to apply post￾stratification weights to estimate indicators for the whole population or for women as a whole. In other words, they adjust for the disproportionate sampling approach that took place in the field. The known population totals for male versus female population in Haiti are: male 49.6 percent, and female 50.6 percent. 15 The proportion of males and females in the final Santé client sample was 15 percent male and 85 percent female. However, since the sampling proportions may differ significantly between SDEs, the ET generated an SDE-specific post-stratification weight for each SDE. The ET then constructed the final weights for the survey by multiplying the design weights by the post-stratification weight at the household and individual level. In this report, we present all results as weighted estimates of the target population. Once weights were applied, the ET ran summary statistics including frequencies and confidence intervals on CAS data to describe health care access, knowledge, use, and perceptions of quality. Due to limited information on non-Santé clients and the health facilities they accessed, data included herein is for Santé clients only. The ET calculated variables that asked about general information, such as knowledge of services provided, over the entire Santé sample. Visit-specific information, such as requesting family planning information, completeness of information received during visit, etc., was limited to respondents who had visited in the past year to improve recall and ensure the visit took place under the Santé Activity. The ET used tables and graphs to illustrate key points, estimates of the target population, and triangulated differences and results with qualitative data in answering EQ 3. For HIV register data, the ET calculated average rates of testing and infection across visited health facilities. These results are also presented in EQ 3. EVALUATION LIMITATIONS BIAS The ET was aware of several potential bias risks in data collection and analysis and developed mitigation strategies to overcome these: 1. Recall Bias: Recall bias is a common challenge in evaluations, especially for respondents who do not remember an experience accurately or omit details. This would be especially an issue for MOH or SA staff who work with multiple projects or donors and may incorrectly attribute experiences or outcomes to Santé. To mitigate this, the ET compared (triangulated) responses across respondent groups (categories of key informants, CHWs and client FGDs, and the survey), and with the results of the desk review to improve accuracy. The ET also probed during the KIIs and used their knowledge from Santé reports and other KIIs to facilitate accurate attribution. 15 Haiti Population. Country Meters. Accessed: March 2020. https://countrymeters.info/en/Haiti 15 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV 2. Response Bias: Response bias is the risk that key informants may have been motivated to provide responses that would be considered socially desirable or influential in obtaining donor support. Sub-awardees and the IP are especially prone to this bias, with clients and CHWs also affected, to increase the likelihood of future funding. The ET mitigated this risk by explaining at the start of the KIIs and FGDs that the purpose was to learn about both successful and unsuccessful strategies, achievements and shortfalls, and highlighted that the interviews were confidential. All respondents provided informed consent guaranteeing them anonymity. 3. Selection Bias: Selection bias is an inherent risk when implementers help to facilitate contact with Activity counterparts to ensure positive views of its implementation. To mitigate the risk of selection bias, the ET identified the individuals for KIIs and invited them for the interview. While the MOH and clinics provided lists of CHWs and clients for FGDs respectively, the ET contacted respondents directly via the data collection firm through random identification. 4. Gender Bias: Gender bias is a risk because most individuals have a subconscious sense of appropriate roles and behavior for women and men, which may present itself both in the responses given and the ET’s understanding and interpretation of results. The ET reviewed gender-sensitive approaches during the inception period by taking and discussing USAID’s Gender 101 training, focusing discussion on what possible gender preconceptions might come into play during this evaluation, and reviewing how to minimize these during data collection and analysis. In addition, the ET had high female representation and DDG teams conducting FGDs and CAS were gender balanced, to the extent feasible. Finally, the ET also actively searched for and found female respondents for the KIIs, FGDs, and CAS to ensure their voices were represented. OTHER LIMITATIONS AND MITIGATION STRATEGIES Under-sampling of men in the CAS: The initial evaluation design called for the survey data collection firm to oversample women who were pregnant, breast-feeding, and/or caregivers of children under age five, to ensure sufficient respondents for survey modules on those topics. Enumerators were to sample women until the target number (400) was reached, and then revert to sampling men and women randomly. However, with multiple teams collecting data simultaneously, and without defined protocols for tracking quotas, the data collection firm continued to prioritize women beyond the target number, resulting in fewer men than desired. The data analysis process included weighting at the individual or household level to adjust for the different probabilities of selection. However, as a result of the lower number of men surveyed, statistics for men are less precise than anticipated, while statistics for women and sub-categories of women are more precise than anticipated. No previous community-based household survey quantitative data for comparison: Since there is no community-based household survey data from the previous HSD Activity (SSQH), or from the start of this Activity, the MTE could not assess changes over time in access to, use, or perceptions of quality of health services for Santé clients. In absence of this data, the ET designed the catchment area survey to capture and compare Santé and non-Santé clients’ knowledge, use, and perceptions of services as a measure of how Santé was achieving its HSD objective. However, given the lack of information regarding the differences in health facility choice and clients, the ET determined that presenting Santé 16 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV versus non-Santé performance would be misleading, and non-Santé respondents were excluded from analysis. Some documents missing: Throughout the evaluation, the ET requested additional documents from USAID and/or the IP to complete the desk review and fully address the EQs. The ET received the majority of these, although a few key documents were not furnished. The ET did not receive the Y2 indicator table (not included in the Y2 Annual Report), the Y3 Monitoring, Evaluation, and Learning (MEL) plan, or the indicator data for Y3 Q1 report. The Annual Reports were found to include select, but not all, indicators, and therefore were not sufficient to allow the team to examine project achievements. These missing documents would have aided in responding to EQ 3. The plan (or road map) that outlines the goals and processes for transferring staff paid by Santé to the MOH (EQ 4) was provided during the writing of this report (April 10, 2020) and therefore not validated or discussed during data collection. The IP informed the ET on April 10, 2020 that one OCA was complete, and another was in progress. FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1 To what extent and in what ways are the original Activity design and assumptions still relevant? In addressing this question, special attention should be given to the relevance or functionality of the National and Regional advisory boards. To address this question, the ET first had to understand Santé’s original design and assumptions, then how they have changed in the first two years of implementation. Through the desk review, the ET identified key elements of the original design and assumptions and noted changes to them. To ensure a comprehensive answer to this question, the team then collected primary data through key informant/group interviews with the USAID Health Team, the Santé IP, Santé sub-awardees, and MOH officials at the national, departmental, and site levels, as well as through FGDs with CHWs. The relevance and functionality of the SAGs at the national, departmental, and site levels are discussed as a sub-question of EQ1. FINDINGS CHALLENGED ASSUMPTIONS The ET confirmed the foundational assumptions identified in the Activity proposal through discussion with the IP and USAID during KIIs. Key informants from USAID and the IP told the ET that the assumptions guiding Santé are: 1) a comprehensive and multi-level transition from the previous HSD Activity (SSQH); 2) a team of equal partners would implement Santé; 3) the ability to conduct regular site visits by technical teams would build the capacity of all health providers (sub-awardees and MOH); and 4) good collaboration with the MOH, particularly at the central or departmental level. Santé experienced challenges to each of these assumptions during the first two years of implementation, leading to changes in the design or implementation delays. These assumptions align with ideas the ET found throughout various Santé documents. 17 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Comprehensive and Multi-Level Transition from the Previous HSD Activity (SSQH): Both the Santé proposal16 and the PY1 Quarter 2 report17 indicated that the Activity assumed that the transition from SSQH would allow them to build on the previous Activity’s experience. The proposal describes a series of handovers at the site, department, and national levels with SSQH staff passing along lessons learned and site-specific strengths and weaknesses in month one.18 A transfer from SSQH did not take place at all levels, though there were many meetings at the national level and SSQH transferred notes and key documents. KIIs with IPs, SAs, and USAID staff suggested that the transition from SSQH was not as comprehensive or multi-leveled as anticipated and desired. Team of Equal Partners Would Implement Santé: Although USAID entered in the cooperative agreement with Caris as the prime partner, Caris proposed a team of equal partners, indicating that all three members of the consortium would share leadership and be core partners in implementation. The proposal included a steering committee for Santé comprised of the Caris, Catholic Medical Mission Board (CMMB), and Haitian Health Foundation (HHF) country directors. 19 In addition, the COP and DCOP positions would be held by Caris and HHF respectively, and the Technical Director would come from CMMB.20 Caris described the proposal as a joint effort – “The design of this project was developed by all three members of the consortium”21 – with a management structure designed for all three consortium members to hold key personnel positions to promote “close working relationships, a constant flow of communication, and high levels oftrust between the Prime (CFI) and the Partners(HHF and CMMB).” 22 Key informants (IP, SA, Donor) and the document review indicated that while the steering committee was formed and is functioning, Caris staff now hold all key personnel positions. HHF left the consortium after PY2, and SA and USAID KIIs explained that CMMB is considered a sub-recipient, similar to the other 25 SAs. Building Health Providers Capacity for HSD through Regular Technical Team Site Visits: The ET heard from KIIs (across all stakeholder groups) and FGDs with CHWs that Santé technical teams are building the capacity of health facility staff and CHWs. Slightly less than half the KIIs (evenly distributed across stakeholder groups) said that technical teams with specialized skills in community and clinical health care are coaching and mentoring health workers, and just more than a quarter of KIIs reported that this was improving health provider performance. However, key informants also identified obstacles to health providers’ widespread capacity building that include factors both outside of and within Santé’s control. More than half of KIIs with the IP, SA, and MOH cited the current socio-political situation in Haiti as an impediment to technical teams’ ability to reach sites. In contrast, however, KIIs with the MOH and SAs noted that poor relationships with some DDS hampered the efficacy of the technical team model. Other key informants from the MOH and SAs reported that Santé may have underestimated the resource gaps it found at MOH sites or the difficulty in reaching some sites. On the other hand, the ET heard from MOH and SA KIIs that Santé was able to overcome difficulties encountered at MOH health facilities. One sub-awardee key informant summed 16 Santé Proposal 2017, pages 27-28. 17 Santé PY1 Q2 Report, pages 24-25. 18 Santé Proposal 2017, page 24. 19 Santé Proposal 2017, page 22. 20 Santé Proposal 2017, page 37. 21 Santé Proposal 2017, page 13. 22 Santé Proposal 2017, page 22. 18 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV this up by explaining that at some MOH sites, the technical teams did “create a collegial relationship and shared the resources with the staff and they saw a real change in their performance.” Collaboration with the MOH: Finally, Santé assumed that they would collaborate closely with the MOH at the central and departmental levels. The ET found in the desk review and KIIs with USAID, SAs, and the MOH, that two issues with the central MOH were the most often cited in describing their working relationship. The first is the central MOH’s demand for a meeting (in April 2018) before Santé could begin implementation, which is discussed above. The second is the requirement that Santé create a coordination committee (Cellule de Coordination) with representatives from the MOH, USAID, and Santé, which is discussed in detail in EQ 2. The four-month delay in implementation led to strained relations between Santé and the MOH. KIIs with USAID, the IP, SAs, and MOH described Santé’s start￾up as “rocky,” “tough,” “difficult,” and “ugly,” but many of the same KIIs agreed that Santé and the MOH are currently working much better together, and attribute the improved collaboration to the creation of the Cellule (see EQ 2). CHANGED DESIGN Santé’s original design included core elements related to HSD: continuum of care, scaling-up, technical leads, and technical teams, along with HSS for sustainability through capacity development and SAGs. Continuum of Care and Scaling-Up: Santé’s original design focused on two principal approaches: Continuum of Care Across the Life Cycle and Scaling-Up. The proposal defines the first as “a holistic approach which assesses and responds to factors impacting health and development across the lifespan,” and the second means scaling-up as “strategies that have proven records of success.”23 Although virtually every report and work plan mentions continuum of care across the life cycle and scaling-up, the PY3 Work Plan provides the best description. However, continuum of care across the life cycle only refers to women and children, 24 while scaling-up is used in the context of an activity for newborns.25 When the ET asked key informants (USAID, IP, SA) about Santé’s design, none mentioned scaling-up, and only one (KII IP) referred to continuum of care across the life cycle, saying, “That one was well designed, but since we started implementation and with what we are doing now, you don’t see continuum of care.” Capacity Development: Santé proposed country (or local) ownership as an approach to sustainability through HSS.26 The principal components of local ownership are SAGs and capacity development for local partners (health facilities, DDS, and SAs).27 The SAGs are discussed separately below to respond to the EQ 1 sub-question. Santé envisioned capacity development for nearly 200 entities, 164 health facilities, ten DDS, and 25 SAs, using the organizational capacity assessment (OCA) tool to establish baseline levels and create tailored capacity building plans. For example, each site would do an OCA “to assess the level of missing/completeness of PES services and the necessary level of technical assistance needed."28 23 Santé proposal 2017, page 12. 24 Santé PY3 Work Plan, Annex B, page 95. 25 Santé PY3 Work Plan, page 19. 26 Santé Proposal 2017, page 2.0 27 Santé Proposal 2017, page 22. 28 Santé Proposal 2017, page 20. “Santé has done capacity building only at the site level and haven’t done anything for us as an organization. Nothing is given to us as the team that keeps the NGO running, even though we are the ones who remain after the project is over.” - Key Informant SA 19 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV However, Santé did not conduct the planned OCAs for the DDS and SAs in the first two years, 29 despite including them in each annual work plan.30 The ET asked SAs and DDS whether they had a capacity development plan under Santé, and all said they did not. In April 2020, Santé informed the ET that they had completed one OCA for a sub-awardee, another one was underway, and a third was planned. Location and Regularity of Site Visits: Nearly half the KIIs (across all stakeholder groups) described the Santé design as site-level coaching and mentoring of health providers through regular visits by Santé technical teams (IP=5, SA=4, DDS=3 Donor=2, MOH site=2, MOH central=2). Through triangulation with the desk review, the ET found references to the technical teams in the proposal with each team supporting six to eight health facilities.31 The 27 technical teams are each made up of one community health coordinator, one clinical coordinator, and an M&E coordinator (DR, IP, and MOH KIIs). Technical Leads in the Santé focus health areas (e.g., Maternal Health, Child Health, HIV/TB, Family Planning) are based in the three Santé regional offices, and support technical teams to develop the “most customized plan per site to ensure that the PES is conducted in a coordinated, standardized manner across the Santé project.”32 Thus, Santé’s design is to provide tailored technical support directly to sites to improve their HSD capacity. In several KIIs (DDS, SA), respondents compared Santé to SSQH, which embedded technical advisors at the DDS office and supported their outreach to health facilities and CHWs. One DDS explained, “the previous project had staff posted at the DDS office and they came to all our meetings, knew everything that was going on, and worked closely with us. Now the project staff are at the Santé offices.” Several KIIs (MOH, SA) observed that the technical teams often visit sites without their DDS counterparts. DDS Collaboration: In the desk review, the ET found that the DDS are meant to be important partners in the TA delivery at the health facility and community level.33 The Activity’s work plans and reports describe close collaboration between the ten DDS and Santé staff. According to USAID, Santé gives each DDS a monthly stipend (approximately $10,000 USD) to handle administrative and logistical support, and fund technical activities required to ensure results. The stipend also allows DDS staff to accompany the Santé technical teams on site visits and to reinforce this TA during their own site visits.34 The ET learned from KIIs with MOH and SA staff that DDS staff involvement in technical teams’ work with health staff varies from department to department, depending on the motivation of each DDS and technical team to coordinate their activities. Six KIIs (IP, USAID, MOH) mentioned that DDS staff do accompany technical teams. Other KIIs described some challenges to closer collaboration. Challenges mentioned by IP staff include DDS lack of resources, while a few MOH key informants suggested that they desired more timely updates or participation in supervision. ADVISORY BOARDS Santé initially planned to work through three levels of SAGs, designed to be implemented at the national (NSAGs), departmental (DSAGs), and site (SSAGs) levels. Santé envisioned the 175 SAGs as a 29 Santé PY1 Annual Report and Santé PY2 Annual Report. 30 Santé PY1 Work Plan and Santé PY2 Work Plan. 31 Santé Proposal 2017, page 36. 32 Santé Proposal 2017, page 36. 33 Santé PY3 Work plan, pages 8-9. 34 Santé PY3 Annual Work Plan, pages 9 and 5. 20 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV mechanism to give voice and ownership to community members, and for health providers to reach community members with important health messages. The proposal described the national level SAG as “the guiding force of Santé,” 35 and the SAGs at all three levels as “the bedrock of Santé and the key to successful implementation.” 36 More specifically, these bodies would make recommendations, provide key contextual information, serve as communication channels, and ensure that collaboration remains focused on patient and community needs.37 Despite describing SAGs as foundational, the desk review revealed that the SAGs were not implemented in the first two years, even though the PY1 Work Plan declares them a priority activity.38 The MEL plan for PY1 described the SAGs as an important component of community accountability and mobilization. Although the PY1 Annual Report acknowledged that Santé did not execute this intervention and stated that it was a priority for PY2, 39 the planned interventions for PY2 Quarter 1 did not include the advisory groups. 40 However, the approved PY2 Work Plan again emphasized the advisory groups. Santé also planned to use organizational capacity assessments to assess SAGs’ strengths and weaknesses at baseline.41 Unfortunately, the quarterly and annual reports did not include information indicating that Santé established any of the SAGs. The PY3 Work Plan again details how the SAGs will be implemented, beginning at the departmental level. The ET asked all KIIs and CHW FGD respondents about their knowledge of SAGs and perspectives on their relevance. Almost all KIIs across all stakeholder groups and CHW FDGs agreed that SAGs are not yet functional. While two key informants (IP) stated that Santé was implementing the SAGs – specifically, DSAGs in a few departments, where they are being piloted – all the others stated that they had not yet begun. While Santé did not establish a national level SAG, it did set up a coordination mechanism at the national level not initially in its design. The Cellule de Coordination (coordinating committee), which as discussed earlier, consists of representatives from Santé, USAID, and the MOH. This group, which meets every two weeks, depending on the security situation, to synchronize actions across Santé’s main actors, does not appear to replace the NSAG because it does not include members beyond donor, IP, and MOH staff. KIIs with USAID, MOH, SAs, plus CHW FGDs were ambivalent about the SAGs. Many respondents expressed optimism that they could be a useful tool for local ownership and a mechanism for two-way communication to obtain input into how Santé is implemented. However, they also expressed concerns that Santé or SAG members could misuse them if not executed carefully. Their concerns ranged from fear that SAG members would try to have family members hired, micromanage the health facility, or invade patient privacy, to worries that the Santé IP had not fully thought through how it would respond to SAG inputs, or that it did not have the flexibility to incorporate local voices into Santé interventions. Seven of the ten CHW FGDs indicated that SAGs could be a positive intervention, and five FGDs stated that service quality would improve if clients/community members were given a voice through the SAGs. 35 Santé Proposal 2017, page 12. 36 Santé Proposal 2017, page 20. 37 Santé PY1 Work plan, page 7. 38 Ibid. 39 Santé PY1 Annual Report, page 83. 40 Santé PY1 Annual Report, page 88-91. 41 Santé PY2 Work Plan. 21 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Respondents across KIIs and the CHW FGDs stated that if done poorly, there are risks for SAG members, the MOH/health system, and Santé. CROSS-CUTTING DESIGN ELEMENTS Gender equity/female empowerment and environmental compliance are two cross-cutting issues that the Santé design also addresses. The former focuses on equal access to health care, gender-based violence (GBV) prevention and treatment, and gender equity in staffing. The latter includes medical waste disposal, and environmental monitoring and mitigation plan (EMMP) development and implementation. Santé’s reports and work plans document several GBV-related activities, including the development of a GBV brochure for use in Mothers’ Clubs and training for health staff on how to treat GBV cases. Santé reports mention collaboration with the Women’s Ministry as well. The ET asked about gender issues in KIIs with USAID and IP staff, who confirmed the activities around GBV implemented with the Women’s Ministry, and gender equity among Santé staff. Clients in FGDs and the survey reported that health workers and CHWs provide health care equally to women and men, and treat both groups respectfully. Activity documents indicate that most Santé-supported health facilities have the basic items required for safe and ecological medical waste management, but that the safe disposal of medical waste remains a challenge. The majority of key informants in each of three key stakeholder groups asked about medical waste management (USAID, IP, MOH) said they were not knowledgeable about these activities. The few KIIs who could respond described a complicated system of transferring waste from one site to another, or to the DDS for burial or incineration, and advocated for Santé’s continued attention on environmental issues. CONCLUSIONS Assumptions: Assumptions are the base of an Activity design, and some original assumptions upon which the Santé Activity was designed, though not explicit in Activity documents, did not hold through implementation. Three overall factors led to the assumptions not holding true: 1. The security situation prevented the number and frequency of site visits and onsite support to health facilities. 2. The relationship with the government was not strong initially, and Santé spent much time in the first two years working to build relationships and trust with the MOH. 3. While their overall vision is the same, the MOH and USAID/Santé have different implementation priorities that required balancing both for relationship and implementation management. Advisory Boards: Santé has consistently stated the SAGs are a priority for implementation, despite not being implemented. 1. While the SAGs are a strong idea, the actual implementation of this Activity component is likely more difficult to implement successfully than anticipated in the original design. In fact, if implemented incorrectly, the component could pose some risks to Santé, SAs, and the MOH. EVALUATION QUESTION 2 22 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV To what extent and in what ways is Santé effectively balancing achieving its original objectives and addressing current concerns and requests of key stakeholders, especially USAID/Washington, USAID/Haiti, and the Government of Haiti? To answer EQ 2, the ET examined Santé documents (proposal, reports, work plans, and modifications) to determine the changes requested by stakeholders in the first two years, and the accommodations the Santé team made in response. In the KIIs with USAID and IP staff, the ET asked about the requests made of Santé and the implications of those changes for the Activity’s implementation. To determine how effectively Santé is balancing their original objectives with emerging requests, the ET asked key informants how they managed new requests and how their original plans changed as a result. The ET triangulated the results of the desk review with those from the KIIs. FINDINGS Changes Requested by PEPFAR and USAID The biggest changes Santé has had to make to its implementation have come in response to new requirements of the PEPFAR program, issued not just to Santé, but to all recipients of PEPFAR funds in Haiti. As explained in Haiti’s PEPFAR Country Operational Plan for 2019, progress toward epidemic control has stalled in the last three years, and the country is not on track to meeting its UNAIDS fast track targets, largely due to high numbers of HIV patients who are lost to follow up (LTFU).42 Every year, PEPFAR reviews the data coming from programs around the world and holds regional meetings with PEPFAR implementers funded by the CDC or USAID and MOH representatives to agree on Country Operating Plans. In the first two years of Santé implementation, new PEPFAR policies in the Haiti Country Operating Plan required Santé to introduce a new medication for HIV treatment, adopt new HIV and viral load testing strategies, achieve specified reductions in HIV patients LTFU, and comply with increased reporting protocols. KIIs with USAID, the IP, SAs, and MOH all confirmed that Santé made these modifications to HIV programming in response to the new PEPFAR policies. PEPFAR has been transitioning to a new drug, Dolutegravir, for HIV patients because side effects from other drug regimens have been reported as a cause of patient default. Beginning in PY1, Santé staff were trained on the new treatment protocol and it was incorporated at PEPFAR sites in PY2 according to Santé reports, work plans, and KIIs with USAID, the IP, SAs, and MOH. At the same time, health providers and CHWs were trained to maintain closer contact with clients as another means to improve treatment adherence, checking on those who were LTFU after 30 days rather than waiting 90 days. To track viral suppression in PLHIV, PEPFAR began requiring Santé to monitor patients more closely with viral load testing every three months, as opposed to every six months, according to Santé work plans, reports, and KIIs with USAID, IP, MOH, and SAs. PEPFAR required Santé to change its HIV testing 42 PEPFAR 2019. Haiti Country Operational Plan. https://www.state.gov/wp-content/uploads/2019/09/Haiti_COP19-Strategic￾Directional-Summary_public.pdf “We are not neglecting the other technical domains, but our focus is on reaching the targets for HIV care. We had to pull resources away to meet the targets/new goals set by PEPFAR.” - IP Key Informant 23 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV strategies as well, focusing on higher risk individuals, such as partners of HIV clients, according to Santé work plans, reports, and KIIs with USAID, the IP, MOH, and SAs. PEPFAR wanted Santé to perform fewer HIV tests, but elicit higher rates of HIV positive results, according to Santé work plans, reports, and KIIs with USAID, IP, SAs, and MOH. Finally, PEPFAR requested that Santé report on core PEPFAR indicators weekly, referred to as High Frequency Reporting, according to Santé reports and work plans. The ET confirmed this in the evaluation kickoff meeting with USAID, and through KIIs with USAID, IP, MOH, and SAs. Some changes to Santé emanated from changes in policy and funding at USAID/Washington. Changes respondents mentioned included the implementation of the Mexico City “gag rule” about abortion services (also referred to as Protecting Life in Global Health Assistance), new funding to combat Zika, and minor changes to how (non-PEPFAR) indicators are defined and reported. Changes Requested by MOH The MOH made several requests of Santé in its first two years, most of which have been implemented and a few that are still being discussed. At Santé’s start, the MOH insisted that a Cellule de Coordination (coordinating committee) be established, consisting of members from USAID, MOH, and the IP. In addition, MOH shared new policies and protocols, some of which Santé was involved in developing. Additionally, the MOH asked Santé for assistance transporting vaccines to health facilities and DDS offices and providing patient registers to MOH sites. MOH, via the Cellule de Coordination, asked for increased funding for DDS activities, and for the DDS and Santé teams to better coordinate their site visit scheduling. The ministry also requested that Santé regional directors be more involved in the DDS planning and budgeting process to ensure closer intervention synchronization. This was mirrored at the central level with the Cellule de Coordination providing guidance on how to align Santé and DDS work plans. Finally, the MOH has suggested replacing some of the sites included in Santé with other MOH facilities. Santé has not implemented this site relocation request because it requires high￾level consultations between USAID and MOH. Santé’s Response to PEPFAR, USAID, and MOH Requests As noted above, PEPFAR made by far the greatest number and most complex requests affecting Santé’s implementation. To respond to PEPFAR’s requirements, Santé gave more attention to HIV interventions to ensure that changes in programming are properly introduced and integrated. The consequences for not meeting these requirements include funding cuts; thus, attending to PEPFAR has, understandably, been a high priority. To implement the changes, technical teams have made more frequent visits to PEPFAR sites (weekly) compared to non-PEPFAR sites. Key informants (IP, MOH, USAID) explained “There should be a prioritization of the sites where the project works because there have been health projects from USAID at those sites for more than 15 years. The ones that are showing improved performance should ‘graduate’ and the sites that are not putting into action the coaching and TA they are getting should be dropped.” - Key Informant 24 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV that Santé trained all technical team members on key aspects of the new HIV interventions so they could incorporate them into site visits. The transition to the new HIV medicine took Santé, SAs, and MOH staff time and attention, requiring them to share the new protocol (how to give the drug, the doses, the transition from the old drug to the new one, and what side effects to look out for) with health providers and HIV patients. This was primarily done in the context of technical teams’ routine site visits, but Santé arranged additional meetings and trainings as well. Despite the initial investment of time and attention, the new HIV medication has fewer side effects, so clients are more likely to stay on treatment, which is expected to improve patient outcomes and reduce virus transmission, and in turn is expected to require less time and attention in the Activity’s remaining two years. To implement new PEPFAR policies, Santé decided to give CHWs more responsibilities, including managing HIV clients who are on treatment, tracking down those who are LTFU, and identifying people for targeted testing. Santé intended that shifting tasks to CHWs would reduce the burden on health staff and facilities, as well as on HIV patients, who could receive more services in their community. However, CHWs reported that the additional tasks contributed to their already heavy workload. Key informants (IP, SA, MOH) told the ET that in a few cases where CHWs were too busy to take on the additional HIV tasks, Santé shifted tasks to another cadre of community-level workers who specialize in HIV. Only one PEPFAR request required substantive change, which was for the high frequency reporting. This change required that USAID modify the Santé Cooperative Agreement. Key informants (IP, SA, MOH, USAID) explained that designated PEPFAR sites now submit data each Friday to the Santé M&E team to compile and send to PEPFAR every Monday. According to stakeholder KIIs, USAID/Washington and USAID/Haiti requests were relatively simple to address, and some were even beneficial. For example, Santé used the new Zika funding to buy 20 ultrasound machines for Santé-supported sites and fund training on how to use and maintain them. While the ultrasound machines were originally meant to detect microcephaly caused by Zika, they are also used routinely during antenatal visits, and have proven to be an added incentive for pregnant women to seek care. Santé was also able to respond to MOH requested changes. The technical teams were able to transport the vaccines to the sites and DDS offices. Several KIIs explained the need for fewer formal trainings to be organized, saving time and money because technical teams cascaded information directly to the health facilities. For requests that required additional financial resources, Santé had to consult USAID first. For example, with USAID’s approval, Santé reallocated funding to enable the printing and distribution of patient registers for MOH sites. Several KIIs (MOH, IP, SA) observed that the MOH-initiated changes were often mutually beneficial. For example, as one KII explained, although the patient registers were not originally in the Santé budget, “this is a basic need for health facilities and Santé needs for the facilities to have them, so their data are recorded properly.” 25 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Key informants had somewhat conflicting views on the extent to which Santé was able to effectively balance all the new requests with original Activity objectives. The ET heard from some KIIs with IP and SA staff that Santé’s technical team design and the extra training for those teams on HIV, in particular, allowed for the relatively easy integration of most PEPFAR and non-PEPFAR requests. However, not all interviewees reported that Santé has been able to seamlessly balance the various requests. Other key informants from the IP, SAs, and MOH reported that the focus on HIV has been a distraction from other health areas and activities. One observed, “the added pressure from USAID/PEPFAR has caused us to do things more quickly and that has led to poorer relationships with the sites.” Several others cited activities that they perceived as being neglected by Santé, such as capacity building for SAs, the SAGs (see EQ 1), coordination with the DDS, strengthening the referral system, and promoting health facility births. CONCLUSIONS From its beginning, Santé needed to stay flexible to respond to multiple stakeholders’ requests. With that flexibility, Santé was responsive to requests from PEPFAR, USAID/Washington and Haiti, and the MOH. 1. Requests from the MOH, USAID/Washington, and USAID/Haiti were generally mutually beneficial, but some did require significant programmatic changes. 2. PEPFAR’s requests and requirements resulted in more significant adjustments to the Activity itself. While Santé did incorporate the changes, it affected the speed of service and relationships with sites. Two specific challenges were the fact that the PEPFAR requirements added new responsibilities to already burdened community health workers, and that the PEPFAR priorities were not necessarily aligned with MOH priorities, making the collaborative approach to programming in the Santé design more difficult to implement. EVALUATION QUESTION 3 To what extent and in what ways have Santé’s interventions been effective in achieving key results, especially increasing access to health services? The evaluation should consider client and key stakeholder satisfaction as well as other measures of access to and successful delivery of health services. The ET used data from KIIs (all categories), FGDs (with clients and CHWs), the CAS, and desk review, including available activity monitoring data, to determine to what extent and in what ways Santé’s interventions have been effective in achieving key results. The ET focused on access to health services at both health facility and community levels and client and key stakeholder satisfaction. The ET collected primary data (CAS and FGDs) on client perceptions of Santé health facilities and services available. The CAS and FGDs also asked clients about their experiences consulting CHWs, and remaining barriers to accessing quality care. Through KIIs, the ET asked stakeholders about their perceptions of HSD and HSS “We have to keep responding to the requests, and until now all of them have been productive or helpful. However, if USAID or PEPFAR asks for something that is detrimental, we should say so and push back.” - Key Informant 26 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV and triangulated all these findings across data sources (FGDs, KIIs, CAS, and desk review), as described in the methodology. FINDINGS Santé Health Service Facilities Strengthened by Increased and Integrated MSPP PES Service Delivery Survey data addresses health facility usage in the catchment area. When survey respondents were asked about the last time they visited a health facility, 68 percent said they had visited a health facility in the last six months, 15 percent in the previous seven to 12 months, and 17 percent reported not visiting a health facility in the last year. Among survey respondents who said they go to a Santé facility, 76 percent reported visiting in the past six months. Female clients were more likely to have visited recently, with 81 percent visiting in the past six months, compared to 70 percent of men. Survey respondents who reported visiting Santé facilities were asked why they chose that facility over other options (Table 6). Not surprisingly, given that the ET conducted the survey in areas around Santé facilities, 91 percent of respondents said that distance was a reason they chose to visit the Santé facility. However, 20 percent also cited the quality of the staff as a motivation, and 14 percent reported that price was a factor. Almost 70 percent of the catchment area who reported going to Santé facilities reported walking as their transport mode, with an additional 25 percent reporting using motos (Table 7). Table 6: Choice of Facility WHY DO YOU GO TO THIS HEALTH FACILITY AS OPPOSED TO ANOTHER FACILITY? % 95% CI Distance 91 [88,95] Quality of the staff 20 [14,27] Price 14 [9,18] Equipment and supplies 4 [2,6] N/A or Don’t know 0 [0,1] Survey Variable: A8. Analytical sample size: 1,043. Individual-level weights applied (representative of all Santé clients in catchment areas). Table 7: Transportation to Facility WHAT KIND OF TRANSPORTATION DO YOU USE TO GET TO THE HOSPITAL / HEALTH CENTER IN GENERAL? % 95% CI Walk 70 [60,78] Four Wheel Vehicle 6 [3,12] Two Wheel Vehicle (Moto) 25 [18,33] Other 0 [0,0] N/A or Don’t know 0 [0,0] Survey Variable: C12. Analytical sample size: 1,043. Individual-level weights applied (representative of all Santé clients in catchment areas). A health facility’s clean appearance and a welcoming reception are two elements respondents perceive as ensuring a safe environment and providing quality care, which clients can readily appreciate. Santé 27 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV technical teams mentor health staff on soft skills like patient-provider interaction and communication with counterparts, as well as clinical techniques.43 To assist facilities to comply with the PES clean clinic standards, Santé has provided training and the necessary equipment and plans to monitor sites for continued compliance in PY3. 44 More than two-thirds of the client FGDs mentioned that health workers treat patients with more respect, and nearly all of them said that everyone is treated equally, regardless of wealth or education. The CAS asked Santé clients who visited in the past year to rate the reception they received and the level of cleanliness they experienced at the health facilities. The CAS also asked about the availability of hand washing stations. The results are in Table 8 Table 9, and Figure 6. Table 8: Quality of Reception HOW WOULD YOU RATE THE WELCOME YOU RECEIVE FROM STAFF WHEN YOU GO TO THE HOSPITAL / HEALTH CENTER? % 95% CI Poor 11 [7,17] Good 75 [69,81] Excellent 14 [10,19] Not applicable/Don’t know 0 [0,1] Survey Variable: C1. Analytical sample size: 926. Individual-level weights applied (representative of all Santé clients in the catchment area who visited a health facility in the past year). Table 9: Cleanliness of Facility Survey Variable: C2. Analytical sample size: 926. Individual-level weights applied (representative of all Santé clients in the catchment area who visited a health facility in the past year). As shown in the tables, 89 percent of Santé clients said that their reception by health workers was good or excellent, and 97 percent perceived that the health facilities were clean or very clean. Eleven percent of respondents rated hospitality as poor, and only 3 percent of Santé clients rated the health facilities as dirty. Santé clients in FGDs confirmed the survey findings, saying that the facilities are clean, and the staff is welcoming. Even though other stakeholders (KIIs and CHWs) did not emphasize these observations, clients’ perceptions of the facility and staff could contribute to their decision to use a Santé health facility. Clients in FGDs emphasized several changes in the Santé facilities’ physical infrastructure (clinics are well-maintained, more hygienic, have improved security, more space, chairs, and other furniture) in the past two years. 43 Santé PY2 Work Plan, page 21. 44 Santé PY3 Work Plan, page 23. WHAT IS THE STATE OF CLEANLINESS OF THE HOSPITAL / HEALTH CENTER (ROOMS, WAITING ROOMS, ETC.)? % 95% CI Dirty 3 [1,7] Clean 76 [66,84] Very clean 21 [14,31] 28 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Service Availability and Utilization Key informants from the IP and SAs told the ET that Santé has introduced new services to sites. Both male and female clients in FGDs confirmed that new services were available at facilities and that laboratories offer a wider variety of tests and give faster results. Clients in FGDs cited more family planning and maternal, newborn, and child health (MNCH) services, which corresponds with the CAS findings. Among Santé clients surveyed in the catchment area, 25 percent reported that more types of services were available at their facility in the past year, while 18 percent reported a decrease and 48 percent perceived the number of services to be the same as the previous year. Those who noted the addition or removal of services were then asked what services had changed. Over 15 percent of Santé clients noted the addition of the following services: family planning, MNCH, and laboratory services (Figure 4). Almost equal numbers reported the addition and removal of nutrition and TB services at their facility. Figure 4: % of Santé Respondents Noting the Addition or Removal of Services in the Previous Year Analytical sample size: 1,043. Individual-level weights applied (representative of all Santé clients in the catchment area). Most Santé clients recognized that sites offer antenatal and immunization services. In the catchment area, more than 70 percent of Santé clients confirmed that the sites always offer antenatal care and 79 percent confirmed that sites always offer immunization services. Twenty-one percent of the Santé clients reported that the sites do not offer the laboratory tests they need (Figure 5). 29 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Figure 5: Prenatal and Childcare Services at Santé Health Facilities Survey Variable: A3/A4/A7/A9. Analytical sample size: 1,043. Individual-level weights applied (representative of all adults in catchment areas). Clients and CHWs in FGDs expressed a desire for even more services to be offered at facilities and by mobile clinics, such as childbirth and cesarean delivery options at facilities that do not currently provide them, and screening for cervical cancer, hypertension, diabetes, and other conditions. Participants in KIIs and FGDs lamented health facilities’ limited hours, particularly those that do not offer evening or weekend hours. While the hours or services available at particular health facilities are not within Santé’s purview, the observation is made to suggest remaining barriers to care. A common barrier to care is the expectation that facilities may not carry the medication or commodities that patients need. ET asked survey respondents how often medications, vaccines, and family planning commodities of their choice were available. The results are in Table 10Error! Reference source not found.. Table 10: Perception of Availability of Medications, Vaccines, and FP Methods at Santé Health Facilities IS THE FOLLOWING AVAILABLE… PRESCRIBED MEDICATIONS1 VACCINES2 PREFERRED FP METHOD AVAILABLE3 % 95% CI % 95% CI % 95% CI No 6 [3,11] 1 [0,3] 8 [3,19] Sometimes 76 [68,83] 35 [29,43] 21 [14,30] Always/Yes 18 [12,26] 62 [55,69] 69 [58,79] Not applicable/I don't know 0 [0,2] 1 [1,3] 2 [1,5] 1 Analytical sample size: 1,043. Individual-level weights applied (representative of all Santé clients in catchment areas). 2 Question only asked to Santé clients who were caregivers of children <5. Analytical sample size: 596. Individual-level weights applied (representative of all Santé client caregivers of children <5 in catchment areas). 3 Question only asked to Santé clients who noted they currently use a family planning method. Analytical sample size: 446. Individual-level weights applied (representative of all Santé clients in catchment areas currently using FP). 30 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV While some of the obstacles raised are within Santé’s purview (availability and quality of services), other obstacles related to access to services like the supply of vaccines and medications are beyond its scope. Key informants across the board and CHW FGDs noted the lack of vaccines and vaccination cards in Haiti, which contrasts with the perspective of clients in the CAS and FGDs, who reported fewer challenges getting vaccines. Of Santé clients in the catchment area, only 18 percent reported that prescribed medications were always available, while 76 percent reported that prescribed medications were only sometimes available, and 6 percent reported that medications were not available. Reported availability of vaccines and family planning commodities was higher, with 62 percent of Santé clients who are caregivers to children under five noting that vaccines were always available, and only 1 percent noting that they were never available. For Santé clients currently using family planning methods, 69 percent indicated that their preferred family planning method was always available, while 8 percent noted it was never available. These survey findings were corroborated by both male and female client FGDs participants, who complained about the lack of drugs specifically, and a lack of materials more generally. KII and FGD respondents across stakeholder groups consistently reported remaining challenges in the availability of supplies, medications, services, and vaccines. 31 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Patient-Centered Services at Health Facilities During FGDs with clients who visited Santé facilities, the ET probed on timeliness, confidentiality, communication, and fairness. These clients offered their opinions that providers treat everyone equally and with respect; in fact, all male client FGDs offered the opinion that health workers treat clients with more respect, and four of ten female client FGDs shared the same observation. During FGDs, clients also noted that providers are more available than before, and they attributed greater availability to more staff or lower rates of staff absenteeism. Male clients in the FGDs emphasized a broadly positive impression of health facilities, while female clients were more likely to cite specific improvements, like patient education and the quality of providers. On questions of care, information, and confidentiality in the CAS, Santé clients reported mixed opinions. When asked if staff gave them all the treatment, prognosis, and follow-up care needed for their illness, 47 percent who visited in the past year felt the care provided was complete, 49 percent felt it was partially complete, and only 4 percent said they did not receive explanation. Similarly, 46 percent of clients reported that they received full information regarding their illness, while 47 percent believed they had received partial information and 6 percent reported that they had not received complete information (see Table 11). Santé clients who visited their clinic in the past year, 75 percent reported that nurses or CHWs spoke with them about HIV either sometimes or always, and just 12 percent never did (another 13 percent noted they were unsure, or the question did not apply). Regarding family planning, those who visited in the past year reported that 66 percent of their health care providers were always or sometimes willing to talk about family planning, while 28 percent of the clients reported that health providers were not open to talking about it (7 percent were unsure or noted the question did not apply). Table 11: Santé Clients’ Assessment of Completeness of Care DO YOU ROUTINELY GET… C4. COMPLETE INFORMATION ABOUT YOUR ILLNESS? C5. COMPLETE CARE INCLUDING TREATMENT, PROGNOSIS, AND FOLLOW-UP? % 95% CI % 95% CI No 6 [3,13] 4 [2,8] Partial 47 [37,56] 49 [40,58] Complete 46 [37,56] 47 [38,55] Not applicable/I don't know 0 [0,0] 0 [0,1] Survey Variable: C4/C5. Analytical sample size: 926. Individual-level weights applied (representative of all Santé clients in catchment areas). Regarding confidentiality of care, 44 percent of Santé clients who visited in the past year believed that staff always maintain their privacy when speaking to other staff or clients, while 20 percent believed they “Pafwa gen pasyan ki t konn mouri paske mis yo gade aparans avan yo pran swen yo e lè yo deside ede pasyan sa li gentan trò ta pou li. Sa kòmanse change nou ta renmen pa janm gen sa ankò menm.” (Sometimes patients would die because they looked at their appearance before they provided care and when they decided to help the patient it was too late. That has started to change, and we would never like to see that again.)” - Client FGD 32 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV sometimes did, and 10 percent noted they did not (26 percent were unsure). The survey posed a related question to Santé clients who said they used family planning, asking if the provider had treated them with respect and privacy. Approximately half of these Santé clients from the past year noted they had received respect and privacy, while 90 percent reported receiving one, but not both. Challenges to Quality Health Services The economic and socio-political crises in Haiti have created great difficulties for both clients and health providers/CHWs (FGDs) in their shared quest to maintain or improve the health of the population. Clients and providers face issues with transportation and insecurity. Clients said that the devaluation of the currency and lost wages made it more difficult for them to afford transport to health facilities and health-related costs. Striking differences were reported by KIIs (SA, MOH, IP) between the SA-supported and MOH facilities in terms of staffing and the availability of supplies and equipment, with the former perceived as having more of both. This supports the desk review finding from Santé’s initial rapid assessments, which showed the variation across sites.45 The desk review found, and key informants confirmed, that not all clinics are visited equally by Santé technical teams for a variety of reasons, including clinical priorities and services offered. Two reasons respondents highlighted particularly were topographical challenges and the focus on PEPFAR sites described in EQ 2. Transportation and geography also impeded client access to some health facilities, as reported in KIIs and FGDs. Sub-Result 1.1: Maternal, Newborn and Child Health (MNCH) Santé reported mixed success in achieving MNCH targets by the end of PY2, reaching 40,376 pregnant women with at least four antenatal visits (90 percent of target), ensuring 24,419 births were attended by skilled birth attendants (77 percent of target), and treating 16,372 children for diarrhea (109 percent of target). The number of children fully vaccinated dropped 6 percent from the previous year (97 percent), leading to 91 percent achievement of the target.46 The frequent stock outs of vaccinations (supply of vaccines is outside the control of the project) were the major contributing factor to the decrease, according to the IP. Santé aims to increase the utilization of antenatal services. While Santé met its target for PY1, in PY2 the number of pregnant women with four antenatal visits dropped 3 percent.47 The ET asked women who were pregnant or caregivers of children under five about their antenatal care during their last (or current) pregnancy. Given the age range of children, it is possible that some women’s pregnancies occurred prior to the Santé program, so the ET restricted analysis to those with children two years or younger. Using this parameter, the availability of services women reported can be illustrative and serve as a benchmark for future reference. Overall, female Santé clients in the catchment areas with children under two reported receipt of a wide range of antenatal services. Of the 318 women in this category, 94 percent reported having had their 45 Santé PY1 Q3 Report. 46 Santé PY2 Annual Report, page 12. 47 Santé MEL 2019, Annex B; Santé PY2 Annual Report. 33 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV blood pressure checked, and 95 percent reported receiving fetal heartbeat measures. More than 80 percent reported being provided with a tetanus shot and iron tablets, discussing breastfeeding with their provider, and being asked about vaginal bleeding. Fewer women discussed zika prevention (62 percent) and birth planning (72 percent) with their provider. Antenatal Care visit completeness is presented in Table 12. Table 12: Antenatal Care Visit Services Received DURING YOUR ANTENATAL VISITS DID THE HEALTH CARE PROFESSIONAL… % 95% CI Measure blood pressure 94 [88,97] Take fetal heartbeat 95 [90,98] Talk you to about breastfeeding 85 [76,92] Ask about vaginal bleeding 82 [74,89] Give you a tetanus shot 86 [79,91] Give you iron tablets 88 [79,94] Talk about zika prevention 62 [47,75] Ask where you planned to give birth 72 [61,82] Survey Variable: F4. Analytical sample size: 318. Individual-level weights applied (representative of Santé female clients that are pregnant or breastfeeding and caregivers of children <2 in catchment areas). One of the targets Santé has had trouble meeting is deliveries performed by skilled birth attendants. The ET asked survey respondents who were pregnant or the caregiver of a child under five about the biggest obstacles for women in their community to give birth in a health facility. The most cited reason was cost (46 percent), followed by distance (11 percent), and treatment by staff (7 percent). The CAS also asked an open-response question about what could be done to encourage more facility￾based deliveries; results are shown in Table 13. Over half the suggestions focused on expanding the availability of services by offering childbirth at facilities that do not currently offer it. Other suggestions focused on improving existing services (supplies/equipment or reducing the cost), and education about the advantages of facility delivery. Table 13: Recommendations for Increasing Health Facility Childbirth (n=669) WHAT COULD BE DONE TO MOTIVATE MORE WOMEN TO GIVE BIRTH IN A HEALTH FACILITY? % Offer maternity services at more facilities 27 Hire more qualified staff/train staff 26 Furnish facilities with more supplies/equipment 20 Reduce the price of delivery 17 Promote facility deliveries 10 Table 14 shows that more than 80 percent of Santé clients in catchment areas report that CHWs have provided counseling services about the use of oral rehydration solution (ORS), a treatment for 34 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV childhood diarrhea. This indicator lagged in PY1 due to nationwide ORS stockouts, 48 but Santé exceeded targets the following year with more than 16,000 treated cases.49 Table 14: CHWs Provide Information on Oral Serum for Diarrhea D5. DOES A NURSE OR ASCP TELL YOU ABOUT ORAL SERUM FOR A CHILD WITH DIARRHEA? % 95% CI No 10 [6,15] Maybe 7 [4,13] Yes 81 [75,87] Not applicable/don't know 1 [0,4] Survey Variable: D5. Analytical sample size: 596. Individual-level weights applied (representative of all Santé caregivers of children <5 in catchment areas). Sub-R 1.2. Family Planning and Reproductive Health (FPRH) The CHWs’ work in family planning tracks the steady gains Santé reported in key indicators in the area of family planning. While no targets can be set for family planning, in PY2 Santé recorded 319,268 women of reproductive age using modern methods, a 12 percent gain over the previous year, and 634,246 couple-years protection, a 15 percent gain over the previous year. In more than half of the male client FGDs, participants spoke of CHWs providing family planning information and products at the community level, with three groups suggesting that family planning services have improved; family planning was the most recognized service for male clients. Six female client groups also suggested that family planning services had improved, and five offered that CHWs taught women how to limit the number of children they have. Sub-R 1.3. HIV/AIDS and TB PEPFAR sites, which provide much more than just HIV services, have received more frequent technical team supervision and other investments compared to other Santé sites, given the emphasis on HIV described in EQ 2. The ET asked respondents about knowledge, testing, and treatment in the CAS, KIIs, and FGDs and collected register data at eight PEPFAR sites. The ET also reviewed available Santé indicator data to understand progress against targets. Regarding treatment, Santé met the target for number of adults and children enrolled in antiretroviral therapy (ART), and came very close to achieving the target for the overall number of individuals on ART. Santé reported re-enrolling 4,219 patients who had been categorized as LTFU. Across KII stakeholder categories, respondents said that the new HIV treatment causes fewer side effects and increases adherence, with a few noting that it has the potential to improve viral load suppression. 48 Santé MEL 2019, Annex B. 49 Santé PY2 Annual Report, page 12. 35 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV In seven of the ten FGDs with CHWs, participants volunteered that HIV is less stigmatized now than in the past. The CHWs also spoke about their work with PLHIV support groups and coordination with facility-based providers to be able to manage PLHIV at the community level. Santé’s HIV testing strategy changed in PY2 such that it aimed to reduce the numbers of people tested, but improve targeting to yield a greater percentage of positive results. The intervention tested more individuals than planned, but did succeed in identifying 53 percent more HIV positive cases than anticipated. The review of HIV testing and treatment registers showed the impact of the shift to targeted testing. While the same time periods were extracted in eight sites for two years (October-December of 2018 and 2019), the average number of people tested for HIV declined almost by half (from 7,110 to 3,915) because of the change to testing contacts of PLHIV (index testing) or others at high risk. Even though the number of HIV tests decreased by half, the number of people who were HIV+ only decreased by about 30 percent (from 45 to 32, see Table 15). The ratio of positive tests went from 1:35 (or 2.8 percent positive case) in 2018 to 1:25 (or 3.9 percent positive cases) in 2019 with the new strategy, minimizing the number of tests required to detect HIV+. The ET compared these data to the same sites in the Santé database and found them to be the same or within a small margin of error. Table 15: HIV Tests and HIV Positive from Eight PEPFAR Site Registers in Q1 2018 and 2019 Q1 FY18 Q1 FY19 COMMENTS TEST HIV+ TEST HIV+ Total 7,110 204 3,915 146 All eight sites combined Mean 1,580 45 820 32 Number per sites Ratio 1:35 1:25 Efficiency of the strategy Percentage 2.8 3.9 HIV+/ Tests The register review for viral load testing of PLHIV and the results of viral load tests showed discrepancies between the number of tests done and results revived. For example, the registers could show 50 viral load tests done in a month, but might show 20 (or 80) results received for the same month. Key informants and data managers at the sites explained that they routinely collect samples from PLHIV and send them to the national laboratory for viral load analysis, and the results can take several months to be returned to the site. The delays were particularly long in 2019 due to demonstrations and other disruptions that limited transport of tests and results, and access to reagents for testing. The survey found that 68 percent Santé clients in the catchment area reported having ever been tested for HIV, with about 10 percent more women being tested than men, likely explained by routine testing during pregnancy. Provider-initiated discussions of HIV were reported by 73 percent Santé clients. Santé clients knew about modes of HIV transmission, but about 60 percent said they had not heard about a medication to control HIV. Despite mobilization through CHWs, and facility providers, 14 percent of Santé clients in the CAS reported that they do not know where to get tested for HIV (Table 16). A high percentage of Santé clients (65 percent) confirmed being offered and taking an HIV test in the previous year, with about a third (33 percent) saying they had not been offered a test (Table 17). 36 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Table 16: Know Where to Get an HIV Test G3. DO YOU KNOW WHERE PEOPLE CAN GET HIV TESTS? % 95% CI No 14 [9,21] Yes 83 [75,89] Maybe 2 [1,6] Not sure 1 [0,1] Survey Variable: G3. Analytical sample size: 1,043. Individual-level weights applied (representative of all Santé clients in catchment areas). Table 17: Have Been Offered an HIV Test A11. HAVE YOU BEEN OFFERED HIV TESTING AT THIS HOSPITAL / CENTER FOR THE PAST YEAR? DID YOU ACCEPT THE TEST? % 95% CI No 33 [23,44] Yes, and I accepted the test 61 [51,71] Yes, but I didn’t accept the test 4 [3,7] Not applicable/Don't know 2 [1,4] Survey Variable: A11. Analytical sample size: 926. Individual-level weights applied (representative of all Santé clients in catchment areas who visited a health facility in the past year). Sub-R 1.4. Nutrition CHWs were responsible for many of the achievements in nutrition. In PY2, Santé started the mobile nutrition outreach, especially malnutrition screening, which is reflected in its results.50 Santé met or exceeded all but one of its PY2 targets for nutrition indicators. While facilities also conducted nutrition screening and treatment, CHWs performed regular nutrition surveillance and promoted infant and young child feeding practices in the community, especially as part of home visits to mothers with newborns. The Activity reported reaching 111,434 children under two through community-level nutrition interventions, a 61 percent increase from the previous year, and reaching 251,592 children under five with Vitamin A supplements, an 8 percent increase from the previous year. CHW FGDs most often noted the increase in their nutrition-focused work with children and infants, both screening for malnutrition and suggesting complementary feeding. In eight FGD groups, CHWs suggested that there are fewer malnourished children in their communities as a result. Five of ten FGDs with CHWs discussed their role in teaching mothers about the importance of breastfeeding and how to care for newborns, and similarly, five groups spoke about CHWs providing advice on caring for children, especially nutrition. Sub-R 1.5. WASH Key informants from the IP and SAs reported that Santé facilities have more supplies and equipment, and that they have a handwashing station and follow the “clean clinic” standards. This is supported by survey results and Santé indicator data. As indicated in Figure 6, 71 percent of Santé clients who had 50 Santé PY2 Annual Report, page 60. 37 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV visited a facility in the past year said that it always had a handwashing station. The ET reviewed Santé indicator data on handwashing stations51 and found that 97 percent of facilities had one in PY1 and 100 percent had one in PY2. Figure 6: Handwashing Stations at Santé Health Facilities in the Catchment Areas Survey Variable: C14. Analytical sample size: 926. Individual-level weights applied (representative of all Santé clients in the catchment area who visited a health facility in the past year). Sub-R 1.7. CHWs Successfully Promote the MSPP PES Santé works closely with CHWs to support activities that make health services more accessible to the communities. Specifically, Santé provides training or refresher courses for CHWs and helps them to earn MOH certification. Refresher training includes modules relevant to CHWs’ work in communities including WASH, nutrition and breastfeeding, maternal health, HIV/TB, and family planning.52 The community health coordinator on each technical team works with CHWs’ supervisors to optimize service delivery and data collection and strengthen the referral system from community to health institution, which then cascades information to the CHWs. Participants in client FGDs were particularly satisfied with CHWs’ performance in their communities; half of the FGDs discussions included observations that training is improving the quality of the CHWs’ work. According to KIIs with SAs and the MOH, CHWs have an expanded role as a result of Santé’s support and are now: • Offering more health services within communities, reducing the need to go to facilities; • Detecting and managing child malnutrition; • Collaborating with facility-based providers on management for HIV patients; • Offering a wider range of health education lessons; • Discussing gender-based violence (GBV) and disaster risk reduction (DRR); and • Mobilizing community members for mobile clinic visits. 51 Santé PY1 Annual Report; Santé PY2 Annual Report. 52 Santé PY1 Annual Report, page 57. 38 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV During client and CHW FGDs, both groups offered their insights on changes observed in community￾level care, especially the work of CHWs. Both groups reported that CHWs have more materials, better training, and are offering a higher quality of services compared to the period before Santé. In eight of the ten FGDs, CHWs suggested that the training they received from Santé has improved the quality of their services and, for some, has increased their motivation, although in six groups CHWs expressed challenges of motivation due to their low salary. CHWs’ salary scale is set by the MSPP and not within Santé’s control. CHWs talked about their work in helping to track patients in the community, including patients with HIV, pregnant women, and malnourished children. CHWs noted, as did some clients, that they make home visits to women with newborns as part of making sure that they receive proper postpartum care. CHWs spoke of their role in ensuring that HIV patients can receive treatment at home and noted that there is less stigmatization of HIV in the community now. CHWs in FGDs also unanimously mentioned the work they do to help communities prepare for natural disasters and disease outbreak. Clients credited CHWs with mobilizing communities, and expressed that the CHWs are respected sources of health education. The CHWs’ skills in family planning, maternal and child health, and nutrition were most highlighted in the client FGDs. Male clients in FGDs spoke about the CHWs’ role in providing health education on different topics, with family planning most often mentioned, and their work to mobilize people for mobile clinics. Female client participants in FGDs provided many more details of the services CHWs provided, especially about family planning, nutrition, and vaccination. Eight of ten female client FGDs discussed CHWs’ helpful role in explaining and promoting vaccination. Malaria was another topic female FGD participants brought up, indicating that CHWs described the role of mosquito nets in malaria prevention, signs of malaria in children, and what to do if children might have such symptoms. Other examples of health messages discussed include danger signs in pregnancy, the importance of vaccination, and measures to prepare for disasters and disease outbreaks. Client FGD participants were overall quite positive about the role of CHWs, with a few women suggesting they are even better than nurses in some respects. Sub-R 1.8. Functionality of Health Referral Networks Improved All groups participating in FGDs and KIIs mentioned an increase of referrals as a result of Santé; that is, patients were more frequently referred to a facility, or from one facility to a higher-level facility, to receive necessary services or supplies. Clients in FGDs also said that the referral system is improving. Survey respondents were less aware of a formal referral system, with just 12 percent reporting that their health facility had a referral system. However, among Santé clients who responded to the survey, 75 percent said that CHWs sometimes or always support them to make and attend appointments at health facilities (Table 18). In all FGDs with CHWs, participants mentioned that they have assisted in making more referrals since the beginning of Santé. CHWs said that they try to accompany patients to the health facility when they need to refer them, and often end up paying out of their own pocket for transport to ensure that the person arrives. Santé notes that referral forms are available at all sites to be used by health professionals, and in PY3 intends to put in place a clear referral protocol for CHWs with a 24-hour phone contact for clinical support.53 53 Santé PY3 Work Plan, page 16. 39 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Table 18: CHWs Assist in Appoint Scheduling DO COMMUNITY HEALTH WORKERS SUPPORT YOU TO MAKE AND GO TO AN APPOINTMENT AT THE HEALTH FACILITY? % 95% CI No 23 [17,32] Sometimes 43 [37,51] Always 31 [23,41] Not applicable/I don't know 1 [1,3] Survey Variable: G9. Analytical sample size: 1,043. Individual-level weights applied (representative of all Santé clients in catchment areas). Santé-supported health facility staff conducted mobile clinics to bring services closer to people, and CHWs support the effort by mobilizing community members to attend on the day of the clinic. Key informants and both CHWs and client FGD participants affirmed CHW’s important role in mobile clinics and the wide range of health education topics they share with community members during them. The CAS confirmed that clients of Santé facilities widely recognized services provided by mobile clinics, such as family planning (34 percent), preventive services (vaccinations, prenatal, well childcare) (48 percent), and curative services (34 percent). Sub-R 1.9. Emergency Transportation and Communication Protocols Successfully Implemented in Referral Networks The CAS suggests that about a third of Santé clients in the catchment area are aware of ambulance or transport services (Table 19). For those who stated their clinic had a transport system, 28 percent indicated that they had used it. Table 19: Referral System and Transportation at Santé Health Facilities TRANSPORT SERVICES % YES 95% CI C10. Does this hospital / health facility have transportation / ambulance service 35 [25,46] C10.1 Do you use (or need to use) this hospital / health facility transportation (only asked of those who indicated one exists) 28 [21,36] Survey Variable: C10/C11. Analytical sample size: 1,043. Individual-level weights applied (representative of all Santé clients in catchment areas). Survey Variable: C10_1. Analytical sample size: 352. Individual-level weights applied (representative of all Santé clients in catchment areas who are aware of transportation services). CONCLUSIONS Santé has successfully increased access to and quality of health services. It has almost consistently met its targets for implementation, and many clients reference improved services. 1. Santé has leveraged physical improvements, supplies, and equipment at the facilities to attract community interest, while Santé technical teams’ training and mentoring has helped providers to improve their clinical skills. Clients perceive that health care providers were more available, with newly acquired skills/knowledge, and they are using their new skills in their work, with new services offered at facilities. 2. Santé health facilities are increasingly providing client-centered services, and clients report better treatment at health facilities, likely a result of Santé training on patient retention and patient-provider communication. Positive changes have also been noted for the mobile clinics, which link community members with more services and strengthen collaboration between 40 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV CHWs and facility-based health providers. As a result of these improvements, clients report improved satisfaction with their facilities. 3. Some Santé facilities still have shortages of supplies, medications, services, and vaccines, and even providers in some cases. Staffing and the supply of vaccines and medications are beyond the control of Santé. In addition, communication remains a problem at some health facilities in that 1) some clients are getting partial information about their illnesses, 2) some clients do not feel their provider is willing to talk about family planning, and 3) a couple key elements are not always included in ANC visits. 4. Santé has improved HIV services at the facility and community levels. A new HIV treatment was effectively rolled out, as was the new HIV testing strategy, which has proven to be more efficient in identifying HIV positive clients. The ET notes that slow processing of viral load tests at the national laboratory causes delays in obtaining the results and makes it more difficult for providers and PLHIV to know whether HIV suppression is effective. CHWs have played an important role in improving care for HIV patients, as they have supported efforts to decrease LTFU and encourage patients to adhere to treatment. Santé’s work with CHWs has been responsible for increased linkages between health care facilities and services at the community level. 1. CHWs have directly provided more services, facilitated referrals to facilities more frequently, and mobilized community members to attend mobile clinics. This is due partly to Santé’s training, supervision, and mentoring of CHWs, and partly due to their expanded portfolio of community-level management of illness and preventive activities. CHWs activities are very appreciated, bringing health care and education to communities and facilitating their access to health services. 2. Some challenges remain to reach more of the population living in the catchment areas, such as transport for CHWs and for pregnant women to complete ANC visits and deliver at a health facility. Despite these gains, challenges to quality health care remain formidable, including the ongoing economic and socio-politic crises, security issues, and geographical constraints. Haiti’s mountainous landscape poses challenges for patients who may live within the same distance of a health facility but, due to the terrain and/or lack of transport options, take much longer to reach it. EVALUATION QUESTION 4 To what extent and in what ways is Santé planning for the transition of staff paid by Santé to the Ministry of Health? In addressing this question, the Evaluation Team should consider the Activity’s plans for the transfer of staff. To answer EQ 4, the ET examined available Santé documents to determine the extent to which plans had been formalized. Key among these documents was the Santé road map for transition. In KIIs with USAID, MOH, IP, and SA staff and FGDs with CHWs, the ET asked whether transition plans were known and asked respondents to identify any key obstacles and recommendations for transition. The ET triangulated the results of the desk review with those from the KIIs and FGDs. 41 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV FINDINGS Current Context Santé financially supports 1,600 contractual MOH workers who, while paid by the Activity, are not considered Santé staff. The responsibility for hiring, firing, and supervising these staff remains with the MOH.54 In the Activity’s first year, Santé reported that the total payment for MOH contractual staff was approximately $400,000 USD per month.55 A 2016 audit of SSQH found that the large number of health workers being paid by that Activity was a threat to its progress and sustainability. The audit report recommended that USAID/Haiti develop a plan with the GOH to transfer “the responsibility of paying the facilities health workers from USAID to another source.” 56 The 2019 PEPFAR Country Operating Plan also describes the lack of a plan to transition donor paid staff to the MOH as a risk to the sustainability of its investments in Haiti, and indicates that PEPFAR is working with the MOH Human Resources Office on such a plan. 57 Thus, the issue of transitioning contractual staff to the MOH pre-dates Santé, and was re-enforced as an important consideration in the second year of Santé implementation. For Santé, the transition process is intended to be facilitated in cooperation with another USAID Activity, Health Leadership Program (HLP). HLP is a five-year USAID-funded Activity supporting “the Government of Haiti to build its capacity to lead and finance the health sector and improve its planning and oversight of the health workforce.”58 During preliminary discussions with USAID, the ET was informed that HLP had drafted a plan for the transfer of health staff to the MOH, and this plan was under review by the MOH Human Resources Office. The HLP transition plan could not be shared with the Santé evaluation team at the time of the evaluation. In addition to the HLP plan, Santé is required by USAID to produce a road map that clearly communicates transition goals and processes (Sub-R 2.2. Creation and early implementation of a roadmap for transitioning of financing for Santé supported-MOH sites from USAID support to GOH).59 The Santé road map notes that the first stage of the transition – preparation – is underway. Since Activity inception, Santé has been working with the MOH to ensure that the list of contractors is updated and accurate, highlighting “issues of staff not being present; having inconsistent job titles from the approved [MOH] list, having different salaries from the approved [MOH] pay scale and unclear status within the [MOH].”60 In PY1 of the Activity, this included requesting the MOH write formal contracts, provide clear job descriptions, and implement timesheets.61 These steps continued in PY2 under Santé’s lead, in addition to actions taken to ensure workers, particularly CHWs, held proper MOH certification.62 54 Santé PY2 Work Plan, page 32. 55 Santé PY1 Annual Report, page 85. 56 USAID OIG Audit Report July 2016, page 3. 57 PEPFAR 2019 Country Operating Plan for Haiti, page 15. 58 Management Sciences for Health. “Health Leadership Program.” MSH. Accessed: April 15, 2020. https://www.msh.org/our￾work/projects/health-leadership-program 59 Santé PY3 Work Plan, page 33. 60 Santé PY2 Work Plan, page 32. 61 Santé PY1 Annual Report, page 11. 62 Santé PY2 Work Plan, page 29. 42 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV The second stage of the roadmap – engagement – includes meetings with USAID, MOH, and other involved parties, and the development and approval of a formal transition plan. Engagement steps, according to the report, have been delayed due to interruptions caused by COVID-19.63 The third and final stage – implementation – includes ensuring DDS are aware of the transition plan, a local plan is developed for each DDS, a rapid nomination process is put in place (see below) and staff are successfully transitioned.64 As in the second stage, these were also noted in the roadmap as having been delayed. Fieldwork confirmed that the transition of contractual staff should not be led by Santé. Stakeholder KIIs across all stakeholder groups and three CHW FGDs emphasized the need for engagement at the highest levels of the Haitian government and within USAID. A key informant from the IP suggested “a high-level meeting across MOH and the Ministry of Finance to come up with a plan,” while another thought that the Prime Minister and the head of USAID should be involved. Obstacles to the Staff Transition The desk review, KIIs with the IP, SA, MOH, and USAID, and FGDs with CHWs identified several obstacles regarding the staff transition. Funding: Every KII from the IP to the MOH to USAID noted that the MOH does not have the funding to pay these staff, and half the KIIs highlighted that the MOH budget is decreasing and well below the level recommended by the World Health Organization. This echoes the PEPFAR 2019 plan for Haiti, which applauded a proposed increase in the MOH’s budget, but acknowledged that the budget might not be approved and that 90 percent of the budget is currently spent on salaries.65 A 2017 World Bank report noted that health care spending per capita in Haiti is $13, below the average of $15 in low-income countries, and much lower than neighboring Dominican Republic ($180) and Cuba ($781).66 Both the World Bank report and the Santé roadmap note the high historical dependence on external funding, including years of significant USAID support.67 A short-term solution key informants from SAs, MOH, and IP suggested would be for donors to continue to provide payment for MOH workers even as they transitioned from contractual to official MOH staff. However, as many key informants from the IP, SAs, and MOH explained, once a person is hired as a government employee, the ministry that hired them has a legal obligation to that person. Therefore, this solution may not be practical if the MOH cannot guarantee funding/employment beyond short-term donor support. Nomination Process: Considering the legal requirements/securities provided to salaried government employees, the roadmap notes and many KIIs (IP, SA, MOH) described a complex nomination process 63 Santé Road Map, page 4-5. 64 Santé Road Map, page 5-6. 65 PEPFAR Country Operating Plan 2019, page 12. 66 “Haiti: New World Bank Report Calls for Increased Health Budget and Better Spending to Save Lives.” The World Bank. https://www.worldbank.org/en/news/press-release/2017/06/26/haiti-new-world-bank-report-calls-for-increased-health-budget￾and-better-spending-to-save-lives 67 Santé Road Map, page 2. “The reality is that the MOH does not have the means to match its own policy, as the budget is only about 4 percent of the national budget.” - IP Key Informant 43 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV requiring each position be approved by high-ranking government officials, both within and outside the MOH. The roadmap suggests that the large-scale staff transfer from the Activity to government payroll requires multiple departments and agencies to expediate the nomination process.68 There are also the lingering issues of seniority, benefits, qualification, performance, and certification that need to be resolved before transition can be done successfully. Key informants from the IP, SAs, and MOH told the ET that many of these staff have been working as contractors for many years, but their seniority may not carry over to the MOH, and they have not been accruing retirement or other benefits (health, education, etc.) during this time. A sub-awardee said, “The current set up could be in violation of the labor laws of Haiti, for example if their pension contributions have not been made.” KIIs with MOH and SA staff said it is unclear who should be doing performance assessments for promotion/firing, and not all contractual staff have the correct certification for the jobs they currently occupy. External Factors: In addition to internal constraints, the roadmap raises a number of potential external challenges likely to inhibit not just the staffing transition, but Santé interventions at-large. These include political instability instigating the closure or “lock down” of some or all of the country, affecting Santé’s intervention implementation, as well as the introduction of COVID-19, which has the potential to shift resources and focus away from the transition plan to address current crises.69 Risks to Transition Potential Reduction in HSD: The KIIs and FGDs with CHWs uncovered a range of concerns about the proposed staff transfer to the MOH, with 32 of the 42 KIIs, across all stakeholder groups (USAID, IP, SA, MOH) and half the CHW FGDs flagging risks to HSD. As noted above, the process of naming a person to become a civil servant is slow, and respondents expressed fear that if the MOH does not quickly and transparently absorb these staff, they may find employment outside the MOH system. Respondents pointed to current MOH health facilities that lack sufficient staff and/or CHWs, and prior issues with absenteeism. If the MOH is not able to retain current staffing numbers, respondents expressed concern about clients’ ability to access services, which could potentially reverse some of the improvements Santé has made. CHWs comprise the largest group of staff to be transitioned to the MOH. They play an important role in the expansion of preventive services and health management in their communities as described elsewhere in this report (EQ 1 and EQ 3). Many KIIs (MOH and SAs) and CHW FGD participants emphasized that challenges with the transfer could jeopardize the gains made by Santé in community health such as community management of HIV and malnutrition. 68 Ibid. 69 Ibid. “Start with the sites that have plenty of staff already (and a good ratio of current government employees to contractors) that could afford to lose some people and still be able to provide adequate patient care.” - SA Key Informant 44 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Advantages of Transition Job Security: While recognizing the obstacles and potential risks of transition, there are also potential advantages. Half the KIIs (MOH, IP, SA) and two CHW FGDs noted that working for the MOH offers long-term job stability to those currently reliant on donor funding and short-term contracts. All ten CHW FGDs had at least one respondent express an interest in becoming a government employee. Three of these FGDs, however, also had at least one CHW who stated an interest in remaining on a donor-funded payroll, with three groups noting concerns that salary would be reduced under the MOH. In addition to job security, as CHWs are already perceived by the populations they serve to be MOH employees, the transition would formalize their positions within their communities. CONCLUSIONS Santé has developed plans for the transition of staff to the MOH. However, the plans the ET observed do not directly address the most significant issues related to the transfer—specifically in the transition of staff to MOH. 1. Santé is creating and implementing the road map, including coordination with HLP, USAID, and the MOH at the central and departmental levels. However, most of the responsibility for transition lies with USAID and, specifically, the HLP Activity. 2. The transfer of staff to MOH payroll is likely to be comprehensive and politically sensitive due to lack of internal MOH funding, the nomination process, and external factors including political instability and COVID-19. Santé is collaborating with USAID and HLP, while allowing them to take the lead on planning and executing the process. While HLP is working directly with the MOH on a broad transition plan, the IP roadmap reflects the current actions Santé is taking as they await the next step. 3. Key informants note that for the plans to be effective, they need comprehensive USAID and HLP buy-in. While HLP and MOH have developed a broad transition plan that they do agree on, it is not as detailed as informants believe the final roadmap needs to be. The final roadmap will need to be comprehensive and politically sensitive. 4. None of the KIIs or FGD respondents could identify a funding source or management approach that would allow for the payment of the workers as consultants. Even more difficult would be securing the financial backing for adding 1,600 long-term staff as workers to the government payroll. RECOMMENDATIONS RECOMMENDATIONS FOR SANTÉ The ET designed the following recommendations for Santé leadership to adjust programming to effectively reach its results by the end of the Activity. 1. Continue to improve health service delivery with even greater attention to quality of services. Santé needs to sustain its achievements in increasing access to health care by strengthening the system in which they are provided. Just as Santé was able to decease LTFU by prioritizing and monitoring progress, the Activity should use this approach to identify and monitor other key gaps in non-HIV areas. 2. Maintain emphasis on community health workers as a key component of health system 45 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV strengthening. The role of CHWs in providing community-level health education, services, mobilization, and case management is central to a sustainable primary health care system. Ensure that all CHWs are certified, continue to receive frequent and quality supervision, and advocate for improved working conditions. 3. Continue strategies to improve dialogue and relationship with MOH. Use the coordination committee (Cellule de Coordination) and other platforms for dialogue with MOH counterparts at the central and departmental levels. Keep the Cellule updated on activities at the sites and communications with the DDS. Align Santé site visits with the DDS work plans and support their staff to participate in/lead the coaching and mentoring. If possible, consider opportunities for DDS based co-location. 4. Renew emphasis on the sub-awardees’ systematic capacity building. Conduct the proposed OCAs of all sub-awardees and use the results to create tailored capacity development plans for each SA. Focus attention on five to seven high scoring SAs who can be ready to receive direct USAID funding in Year Four. 5. Continue paying contractual staff and coordinate with the MOH and HLP as they finalize the broad plan. Santé should maintain communication with the DDS,i the contractual staff and with the Cellule de Coordination while allowing MOH to lead the process of staff transfer. RECOMMENDATIONS FOR USAID The ET designed the following recommendations for USAID to support Santé to achieve its intended results through the duration of the Activity. 1. Coordinate the process led by HLP for the transfer of staff currently paid by Santé to the MOH. Using the roadmap and support from Santé, USAID should collaborate with the HLP in high-level discussions with the MOH and other donors about the absorption of these staff and funding to cover their salaries and share the developments in the process with Santé. 2. Support Santé in improving relationships with MOH and expanding on health systems strengthening activities. Ways to do this may include facilitating open communication with the MOH via the Cellule de Coordination, requiring co-planning with DDS, mitigating PEPFAR pressure, and/or facilitating capacity building for strong sub-awardees. 3. Conduct a Learning Event on the use of Advisory Boards. While the advisory boards were in the original design, they are difficult to implement and, while planned each year, have not taken off. USAID should take a learning approach to discuss the pros and cons of this approach as well as the trade-offs associated with it. Ultimately, this iterative, learning-focused approach can help USAID better understand if it is open to a shift in Activity design that either focuses more on the advisory boards or focuses on other programmatic components. FOR FUTURE ACTIVITY DESIGN AND IMPLEMENTATION The ET designed the following recommendations for implementers and policymakers establishing future health programming in Haiti. 1. Allow time in activity design for relationship building with government counterparts. Activity designs should not assume strong relationships with the MOH from the start. Implementers should create a relationship-building strategy, adjusting their approach in 46 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV accordance with the growth in trust over time. They should incorporate trust-building activities into their work plans, and consider how factors such as co-location and site visits factor into those plans. 2. Ensure a culture of adaptive management such that revisions to policies and requirements from PEPFAR, GOH, or USAID/Washington can be incorporated as opportunities. Ask the IP to assess the overall risks of those changes to the activity as a whole and provide the opportunity to suggest and make workplan adjustments to compensate for any changes. Activities could include relationship remediation, workload adjustments, and target adjustments. 47 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV ANNEXES ANNEX A: EVALUATION SCOPE OF WORK STATEMENT OF WORK Performance Evaluation of Santé PURPOSE OF THE EVALUATION The purpose of this Statement of Work (SOW) is to set up a framework to carry out a midterm evaluation which will serve as a mid-point reference to assess the impact of the USG investments through Project Santé since its launch in December 2017. In this regard, this midterm evaluation is intended to provide substantial information that will guide the ongoing implementation of the four-year project as it will provide a benchmark that will be used to measure programmatic impact, and subsequently help uncover potential weaknesses in program implementation and lead to improvements. Additionally, this evaluation will alert stakeholders to changes in underlying assumptions, and provide a firm foundation for monitoring and evaluation. Key stakeholders implicated include: USAID/Haiti, the State Department, CARIS Foundation, its partners, USG contractors and grantees, the Government of Haiti (GOH), and other donors. SUMMARY INFORMATION Strategy/Project/Activity Name Project Santé Implementer CARIS Foundation Cooperative Agreement/Contract # 72052118CA00001 Total Estimated Ceiling of the Evaluated Project/Activity(TEC) $98,500,000.00 Life of Strategy, Project, or Activity December 2017 – November 2021 Active Geographic Regions 10 geographic departments Development Objective(s) (DOs) DO3: Health outcomes increased USAID Office Health 48 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV BACKGROUND Description of the Problem, Development Hypothesis (es), and Theory of Change Santé aims at achieving these following goals: Increasing utilization of quality, essential health services (MSPP PES) at facilities and the community by establishing a sustainable quality of service culture that reinforces local decision-making, accountability, local consumer/community engagement and health provider engagement; Strengthening local management and operational capacities to deliver health services through systems planning, coordination and standard setting; facilitating certain sites’ readiness to become direct USAID recipients; In order to strengthen the local management and operational capacities to deliver health services, Santé will continue to assist the sites to provide a comprehensive range of PES services that meet the needs of the population through accountable and efficient management. Santé will continue to provide support to the DDS to strengthen basic equipment and supplies needed based on budgets and work plans provided by each DDS and approved by the MSPP. Strengthening the existing systems for the implementation of a results-based financing (RBF) strategy that assists sustainable changes throughout the health system; Supporting increased transfer responsibility for financing the system to MSPP/GOH. Build the capacities of local entities Santé is one of the most important projects in Haiti in terms of service delivery approach. The project is supporting a catchment population that is estimated to 4,344,584 which is accounted for 47 percent of Haiti’s population. Santé’s theory of change reflects two significant changes at various levels which must occur to achieve sustained well-being for all women, children, youth and families in communities served by Santé, supported by investments from USAID and other donors. Santé continues to implement evidence-based approaches, developed with monitoring and evaluation data and related information, they will continue to refine the project’s theory of change, thus informing their results framework, MEL, strategy and communications. All of these will be achieved through these two integrated strategic approaches: 1) Effective and accountable health systems, and, 2) Increased community mobilization. Summary Strategy/Project/Activity/Intervention to be evaluated Below are a summary of activities Santé is delivering at facility and community levels and the improvement of skills of health care providers as stated in the agreement’s technical approach: the complete list will be shared upon request. 1. Increased quality and utilization of services - Facility • Strengthen linkages between community and facility through site advisory groups. • Assess the quality/time utilization for PES, specifically RMNCH, such as delivery procedures and protocols for each component to identify opportunities for offering a high-quality PE and a functioning referral system. 49 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV • Ensure FP services, compliance with U.S. Regulations and coverage through mobile clinics – including USAID regulations • Identify quality improvement and clinical risk management in the role of RMNCH and youth health. • Enhance the RMNCH continuum of care approach by using mobile technology. • Site data collection systems to be linked to existing national data collection systems to allow targeting of resources and services. • Emergency coverage with vehicles, national and regional ambulance services for the network – 24 hour plan. • Emergency phone lines will be made available. • Clear protocols for referring emergency patients. • Continuous enhancement of the ASCPs’ capacity to maximize network of community providers and health promotion. • Utilization of data to drive interventions in the community, with integrated case management. • Establish systems to track lost patients. • Implement continuous quality improvement approaches to align site readiness to national standards for RBF. • Strengthen management of NGO-run sites. 2. INCREASED COMMUNITY ENGAGEMENT AND POSITIVE HEALTH SEEKING BEHAVIORS – COMMUNITY • Create site/community advisory group as advocacy ‘champions’ to engage strong and influential partners within the MOH and other relevant ministries and sectors -women’s affairs, finance, education, planning and development; key NGOs; international development partners; medical professionals; and civil society representatives • Ensure active community and partner’s participation by engaging them in the planning, implementation, monitoring, evaluation and feedback process. • Define sensitization and mobilization activities with community members. Consider group discussions to address stigma, gender relations and best methods of communications. • Community mobilization activities must ensure that its members are motivated and influenced to take action or support initiatives that are beneficial for themselves – through village gatherings, village health days, popular theatre, youth/women’s/fathers’ groups, and print and electronic media. • Working with local communities to establish health service priorities and to build awareness and ownership of any new directions in health service provision. • Promoting self-directed problem identification and early service seeking behaviors. • Reducing HIV infection in Adolescent Girls and Young Women community activities education in schools, community groups • One task for SSAG will be to develop a community-based emergency transport strategy considering local public and private sector assets and capacity. • Support/scale up existing community-based FP distribution and linkages to facilities. • Promote community-level and site-based prevention and treatment education to ensure that STIs/RTIs awareness is raised including the role of untreated STIs/RTIs in infertility, pregnancy complications and HIV infection. 3. IMPROVED SKILLS OF HEALTH CARE PROVIDERS – WORKFORCE • The three national training sites will be strengthened to provide continuous RMNCH health care workforce development, including providing high-impact, life-saving interventions. • Strengthen provider’s ability in providing LARCs at facility levels. 50 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV • ASCP regular training on recognition of complicated labor/ health emergencies. • ASCP training in promoting PES and Zika-related WASH community-level training. • Education on labor complications; STI/GBV/ women’s health services • Expand methods for accessing knowledge such as distance learning tools, online library in Creole and French. Designated professional development time, Departmental QI conferences— share knowledge, best practices. • Enhance the skills of health providers to provide clinically safe woman-centered maternity care • Develop and support a rural/remote maternity workforce. • Facilitate a culture of interdisciplinary collaboration in RMNCH services. Summary of the Project/Activity Monitoring, Evaluation, and Learning (MEL) Plan Santé has defined a set of performance indicators containing a mix of outcome and output indicators that would help to capture project’s progress. Output indicators track the immediate products of project activities and provide feedback to managers on project performance to identify areas where implementation strategies may need to be adjusted. Outcome indicators measure the effects, or results, of project activities, at the higher levels of the project results framework. Indicators were selected based on the overall strategic approach of the project and closely reflect the work plan to capture the project’s ultimate goal. Indicators for the M&E system were selected in coordination with the Ministry of Health based on the overall strategic approach to the project and closely capture the work plan, link the main activities of the project with its stated goal. Santé performance Monitoring Matrix/MEL plan includes 44 indicators of which 17 USAID standard indicators. On top of the MEL plan, the Performance Indicator Reference Sheet (PIRS) provides the name of the indicator, unit of measure, disaggregation, numerator, denominator and data source. EVALUATION QUESTIONS 1. To what extent and in what ways are the original Activity design and assumptions still relevant? In addressing this question, special attention should be given to the relevance or functionality of the National and Regional advisory boards. 2. To what extent and in what ways is Santé effectively balancing achieving its original objectives and addressing current concerns and requests of key stakeholders, especially USAID/Washington, USAID/Haiti and the Government of Haiti? 3. To what extent and in what ways have Santé’s interventions been effective in achieving key results, especially increasing access to health services? The evaluation should consider client and key stakeholder satisfaction as well as other measures of access to and successful delivery of health services. 4. To what extent and in what ways is Santé planning for the transition of staff paid by Santé to the Ministry of Health? In addressing this question, the Evaluation Team should consider the project’s plans for the transfer of staff. 51 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV EVALUATION DESIGN AND METHODOLOGY The methodological approach shall include a combination of qualitative and quantitative methods, with data collection from both primary and secondary sources. As part of the evaluation design document, SI shall propose for USAID’s review, a detailed methodological approach to be used to address evaluation questions. This methodology will specify the research design, as well as methods and procedures for data collection and analysis. Efforts should be made to use multiple data collection methods and data sources, complementing literature review, interviews, and discussions to allow for triangulation of data and cross-validation of results. The evaluation matrix below provides a summary of the suggested data collection methods by evaluation questions. QUESTIONS SUGGESTED DATA SOURCES (*) SUGGESTED DATA COLLECTION METHODS DATA ANALYSIS METHODS 1. To what extent and in what ways are the original Activity design and assumptions still relevant? In addressing this question, special attention should be given to the relevance or functionality of the National and Regional advisory boards. Documents including performance monitoring data, expert knowledge, Project staff… Key informant interviews, desk review, focus group discussions…… To be determined by evaluation team 2. To what extent and in what ways is Santé effectively balancing achieving its original objectives and addressing current concerns and requests of key stakeholders, especially USAID/Washington, USAID/Haiti and the Government of Haiti? Documents including performance monitoring data, expert knowledge, beneficiaries, Project staff… Key informant interviews, focus group discussions, desk review…… To be determined by evaluation team 3. To what extent and in what ways have Santé’s interventions been effective in achieving key results, especially increasing access to health services? The evaluation should consider client and key stakeholder satisfaction as well as other measures of access to and successful delivery of health services. Documents including performance monitoring data, beneficiaries’ perceptions, Project staff… Key informant interviews, focus group discussions, survey of beneficiaries…… To be determined by evaluation team 4. To what extent and in what ways is Santé planning for the transition of its activity to the Ministry of Health? In addressing this question, the Evaluation Team should consider the project capacity building and hand over plan? Documents including performance monitoring data, Project staff, Key stakeholders Key informant interviews, focus group discussions, … To be determined by evaluation team 52 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV DELIVERABLES AND REPORTING REQUIREMENTS 1. Evaluation Work plan: Upon receipt of this Activity Request, Social Impact (SI) shall submit within two weeks a draft work plan to the Contracting Officer’s Representative (COR). The work plan will include: (1) the anticipated schedule and logistical arrangements; (2) a list of the members of the evaluation team, delineated by roles and responsibilities with their level of effort; (3) the identification of other required personnel and relevant local subcontractors, their LOE, roles and responsibilities and qualifications; and (4) the deliverable schedule. 2. Evaluation Design: Within two weeks of approval of the work plan, SI must submit to Contracting Officer’s Representative (COR) an evaluation design (which will become an annex to the Evaluation report). The evaluation design will include: (1) a detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; (2) draft data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited; (4) known limitations to the evaluation design; and (5) a dissemination plan. USAID/Haiti will take up to 10 business days to review and consolidate comments through the COR. Once the evaluation team receives the consolidated comments on the initial evaluation design and work plan, they are expected to return with a revised evaluation design and work plan within 5 business days. 3. In-briefing / inception report: Prior undertaking field work, the evaluation team will have an in-briefing with the USAID/Health Team and the Evaluation and Survey Services (ESS) COR to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, and to clarify any questions or logistic needs. 4. Evaluation Briefing/Presentation: The evaluation team is expected to hold a final presentation in person to discuss the summary of findings and recommendations to USAID within 20 business days after the conclusion of fieldwork. 5. Draft Evaluation Report: The draft evaluation report should be consistent with the guidance provided in Section IX: Final Report Format. The report will address each of the questions identified in the SOW and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, the Health Office will have 10 working business days in which to review and comment on the initial draft, after which point the ESS COR will submit the consolidated comments to the evaluation team. The evaluation team will then be asked to submit revised final draft report 5 business days hence, and again the Health Office will review and send comments on this final draft report within 5 business days of its submission. 6. Final Evaluation Report: The evaluation team will be asked to take no more than 15 business days to respond/incorporate the final comments from the Health Office. 53 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV The evaluation team leader will then submit the final report to the COR. All project data and records (FGD and KII summary reports) shall be submitted in full and should be in electronic form in easily readable format, organized, and documented for use by those not fully familiar with the intervention or evaluation, and owned by USAID. EVALUATION TEAM COMPOSITION The Evaluation Team shall be comprised of two Key Personnel positions: (i) a Team Leader, and (ii) an Assistant Team Leader. SI is strongly encouraged to sub-partner with a local Haitian firm for data collection purposes. The selected Haitian firms should demonstrate proven capacity in collecting data in the relevant fields (quantitative and qualitative data collection) and should have data analysis capabilities to perform all related data management functions. This approach is encouraged to build the local firm capacity and will also provide a Haitian perspective for the data collection and analysis. The Team Leader (TL) is ultimately responsible for the overall management of the evaluation team, coordinating the implementation of the evaluation, assigning evaluation responsibilities and tasks, and authoring the final evaluation report in conformity with this Statement of Work. The TL must be an experienced evaluation expert, with a documented track record of 10 years of experience in the field of evaluation. S/he should be fluent in French and English. S/he should have at least a Degree in Economics, Public Administration, International Relation or a related field. The Assistant Team Leader (ATL) helps the TL in the overall management of the evaluation team and the final products, in conformity with this Statement of Work. The ATL should be familiar with the Health sector. S/he must possess excellent writing and interpersonal skills and must be familiar with USAID programs, objectives, and reporting requirements. S/he should have experience in designing and implementing evaluations in the health sector and in conducting FGDs. Fluency in French is required. English and Haitian Creole are highly desirable, as is significant prior work experience in Haiti. A Master Degree in a health relevant field is required to ensure that all areas of technical expertise required for the evaluation are effectively covered. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. The evaluation team shall demonstrate familiarity with USAID’s evaluation policies and guidance included in the USAID Automated Directive System (ADS) in Chapter 200. EVALUATION SCHEDULE AND ESTIMATED LOE a) Schedule TIMING (ANTICIPATED WEEKS OR DURATION) SCHEDULED ACTIVITIES 4 Weeks Preparation of the work plan and evaluation design 3 Weeks USAID Approval of the Work Plan and Inception Report 1 week Within 5 business days of the inception report 54 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV TIMING (ANTICIPATED WEEKS OR DURATION) SCHEDULED ACTIVITIES 4 Weeks Data collection 2 Weeks Data analysis and Evaluation Briefing with USAID 2 Weeks Draft Report writing 2 Weeks USAID review of Draft Report 1 Week Incorporate USAID comments and prepare Final Report b) Estimated LOE in days by activity for a team of two TASK LOE FOR TEAM LEAD (TL) LOE FOR ASSISTANT TEAM LEAD (ATL) TOTAL LOE IN DAYS Document review/desk review/work planning and evaluation design drafting 20 20 40 In-brief, evaluation design finalization (including meetings with USAID) 1 1 2 Data collection 15 15 30 Data analysis 10 10 20 Out-Brief 1 1 2 Draft report 10 10 20 Final report 5 5 10 Totals 62 62 124 FINAL REPORT FORMAT The evaluation final report should include an abstract; executive summary; background of the local context and the strategies/projects/activities being evaluated; the evaluation purpose and main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations. For more detail, see “How-To Note: Preparing Evaluation Reports” and ADS 201mah, USAID Evaluation Report Requirements. An optional evaluation report template is available in the Evaluation Toolkit. 55 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV The executive summary should be 2–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (e.g., selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: • The Evaluation SOW; • Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; • All data collection and analysis tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides; • All sources of information, properly identified and listed; and • Signed disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of. • Summary information about evaluation team members, including qualifications, experience, and role on the team. In accordance with ADS 201, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within three months of the evaluation’s conclusion. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per ADS 201maa, Criteria to Ensure the Quality of the Evaluation Report, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. 70 Evaluation reports should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate the strategy, project, or activity. Evaluation reports should be readily understood and should identify key points clearly, distinctly, and succinctly. The Executive Summary of an evaluation report should present a concise and accurate statement of the most critical elements of the report. Evaluation reports should adequately address all evaluation questions included in the SOW, or the evaluation questions subsequently revised and documented in consultation and agreement with USAID. Evaluation methodology should be explained in detail and sources of information properly identified. 70 See ADS 201mah, USAID Evaluation Report Requirements and the Evaluation Report Review Checklist from the Evaluation Toolkit for additional guidance. 56 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Limitations to the evaluation should be adequately disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. Findings and conclusions should be specific, concise, and supported by strong quantitative or qualitative evidence. If evaluation findings assess person-level outcomes or impact, they should also be separately assessed for both males and females. If recommendations are included, they should be supported by a specific set of findings and should be action-oriented, practical, and specific. 57 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV ANNEX B: DATA COLLECTION TOOLS Researcher Name: Notetaker Name: Interview Date: Start Time: End Time: CONSENT FORM FOR KIIS Hello, I am _______________ and I work with Social Impact, an American company conducting research for the United States Agency for International Development (USAID) Haiti Office. We are conducting a midterm evaluation of the Santé Activity. We want to understand how successful Santé has been in meeting its objectives and how its experience over the past two years can inform future USAID/Haiti work. We intend to interview approximately 30 key informants like yourself, people who can provide a unique perspective on Santé. We have chosen you because of your unique knowledge and experience, yet we also want you to know that your participation in this interview is completely voluntary and you are free to decline to respond to any question for any reason and to end the interview at any time. You will not be provided with any payment for your participation in this interview. Your honest responses will help USAID inform and improve its future programming in Haiti. Your name will not be connected to any information you provide in this interview. The data from this interview (excluding your personally identifiable information) will be used by the Social Impact team for analysis purposes only and your answers will only be reported to USAID as part of an aggregated data set. If we use a direct quote in the report, it will not include your name or position, only that it came from a key informant interview. To ensure all your responses are captured accurately and can be reviewed during data analysis and report writing, we would like to record the interview. The recordings will be kept in a secure data storage and will be destroyed after submission of the report. This interview will last approximately one hour. If you have any follow-up questions, please contact Louisena Louis at xxx Do you understand that your participation in this interview is voluntary, unpaid, and that you can end the interview at any point? Yes___ (check) No___ (check) Are you willing to participate? Yes___ (check) No___ (check) Will you allow voice recording of the interview? Yes___ (check) No___ (check) 58 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV KEY INFORMANT INTERVIEW GUIDES FOR Santé KII Guide for MOH staff (central, departmental and site levels) Introduction: 1. Please tell me about your involvement in the Santé project – how long have you been associated with it and what is your role? Collaboration between Santé and the MOH: 2. Can you give me an example of how Santé and the MOH have worked well together? And one where the collaboration has not worked well? 3. How has the relationship between Santé and the MOH affected your work? How do you think it has affected the Santé project? 4. What are the strategies you recommend for a cooperative partnership between the MOH and Santé for the next two years? 5. Are the DDS work plans and budgets aligned with Santé’s interventions in each department? Is Santé building the capacity of the DDS? Clarity of roles: 6. Have the activities of Santé duplicated or interfered with the (responsibilities/authority) mission of the DDS? 7. Follow-up question for the Central level MOH: Have you advised the DDS about this issue? 8. Follow-up question for DDS and Sites: Do the DDS staff in this department understand the roles of Santé compared to the DDS, or is there any confusion? Modifications: 9. Does the Santé project need to be redesigned or should it continue as designed? Is the PES at the center of the project? 10. For the central level MOH: Do you know about any policy or funding changes happening at the MOH that would/will affect Santé in the next two years? 11. The original plan for Santé envisioned promoting participation and ownership through advisory groups at the central, departmental, and site levels. Do you think these would have the intended effect? Should they still be done? Why or why not? Transition: 12. Have you seen the roadmap for transitioning some of the contractual staff paid by Santé to the MOH? If so, what do you think of it? If not, what are the key elements that should be included? 13. What is the plan for maintaining the equipment procured by and infrastructure renovations done by Santé? What needs to be done to transfer their support of the supply chain to the MOH? 59 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV 14. For central level MOH staff: Have you been involved in the partnerships that Santé has established with other USAID-funded projects (e.g. EQUIP, PSM, etc.)? Do you expect the MOH to take over the direct cooperation with these projects after Santé ends? 15. What recommendations do you have for the Santé team and for USAID about Santé and their next health service delivery project? KII guide for Santé implementers (Caris and sub-awardees) Introduction: 1. How long have you been involved in the implementation of Santé, and what is your role? 2. What are the three most important accomplishments of the project and the three areas where Santé needs to improve its impact? 3. What are the primary assumptions (pre-conceptions?) that are guiding or influencing how you are implementing Santé at this point in time? 4. What are the factors beyond the project’s control that will influence its success? Fidelity of implementation: 5. How has the project changed its focus and activities? What caused these changes? Do you think that revisions to the design have been positive or negative? In what ways? 6. The original plan included advisory groups at the central, departmental, and site levels to promote participation and ownership. These were not implemented in the first two years – do you think they should be added now? Do you think they would improve local participation? 7. How have changes requested by the donor and the MOH affected Santé’s ability to accomplish its original goals? What can Santé do to minimize the disruption of these requests in the coming years? 8. How has the relationship between Santé and the MOH affected the project? Has it changed its design? 9. Should Santé be redesigned? If yes, should it be redesigned completely, or should small adjustments be made? Capacity building for sub-awardees: 10. How has your partnership with Santé developed the capacity of your organization/your staff? What should they do in the next two years to continue to maximize your abilities and impact? Have you done a formal capacity assessment and used the results to make a capacity-building plan? 11. Are the DDS plans and budgets (for six months and three years) available and do they influence how Santé is implemented in each department? If they are not being used, why not? Do the DDS’s that you work with have the capacity they need to plan and budget? Cross-cutting themes: 12. How can Santé be more successful in promoting gender equity across its activities? Would more collaboration with the Ministry of Women’s Affairs be helpful? 60 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV 13. With the many activities of Santé, has it been able to focus on the environmental impacts of medical waste and the “clean clinic” aspects of the PES? What ideas do you have about how to best implement the environmental activities? Who should implement them? Transfer to the MOH: 14. What are the obstacles to transferring contractual staff paid by Santé to the MOH? 15. Does Santé have a roadmap for this transition to the MOH? KII Guide for USAID Introduction: 1. How long have you been involved with the Santé activity and what is your role? 2. Broadly speaking, what do you see as the positives and negatives of Santé in its first two years? Design: 3. Do you consider the PES to be the central document/model guiding how Santé is designed? If yes, in what ways is the PES embedded into Santé? If no, what else is guiding Santé’s design? 4. The original design for Santé included an emphasis on capacity building – does Santé need to continue to build the capacity of the DDS, CHWs, and other health staff in the remaining time? 5. What changes would you recommend to the design of Santé? Requests from USAID/Haiti and USAID/Washington: 6. What are the main changes requested of Santé by USAID/Haiti? What do you anticipate will be asked of Santé in the next two years? 7. What are the main changes requested of Santé by USAID/Washington and PEPFAR in the first two years? What modifications do you think will be asked by them in the remaining time? Country ownership and the advisory groups: 8. How would you describe the relationship between Santé and the central MOH? 9. What changes has the MOH requested of Santé in the first two years? 10. The Cellule de Coordination was created to improve communication and collaboration between the MOH, USAID, and Santé. Have you participated in the Cellule? Is it effective? 11. The original proposal for Santé included an emphasis on advisory groups at the national, regional, and site levels, yet these haven’t been established. How has this affected country and community ownership of the project? Do you think the advisory groups should be set up at this stage? Transition to the MOH: 12. Have you seen the roadmap for transferring the contractual staff paid by Santé to the MOH? 13. What is your vision for the future payment of these staff who are currently paid by Santé? 14. What is the role of RBF in helping the MOH fund health services after Santé? 61 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV 15. Santé is supporting the supply chain for the facilities it supports as well as equipment maintenance at these sites. In two years’ time, would this be transitioned to the MOH? 16. Santé is collaborating with several other USAID-funded activities and other projects. Do you see a role for USAID in facilitating the transfer of those relationships to the MOH? Cross-cutting themes: 17. How has Santé integrated gender and female empowerment into its activities? What are its notable successes and failures in this area? 18. The safe destruction of medical waste and environmental protections have been core activities of Santé. Would these activities to be transferred to other IPs or to the MOH after Santé? Recommendations: 19. Where does Santé need to focus its efforts and investments in the remaining two years? 20. What are the resources and personnel required for Santé to achieve its objectives? 62 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV CONSENT FORM FOR FGDS Hello, I am _______________ and I work with Social Impact, an American company conducting research for the United States Agency for International Development (USAID) Haiti Office. We are conducting a midterm evaluation of the Santé Activity. We want to learn more about how Santé has been doing its work over the past two years and use that information to design future USAID/Haiti work. We are going to conduct 30 focus group discussions with men, women, and ASCPs in the areas where Santé is working. We want to hear from people who use health services (clients) and also from health providers at the community level. Your participation in the focus group is completely voluntary and your decision to take part in it will not affect your ability to get health services/your employment as an ASCP. You are free to decline to respond to any question for any reason and to end the interview at any time. You will not be provided with any payment for your participation in this interview. Your honest responses will help USAID inform and improve its future programming in Haiti. Your name will not be connected to any information you provide and the answers your give will be combined with the other focus groups we are doing and only the summary results will be reported to USAID. If we use a quote that you say in the report, we will not say that you said it, only that a person in a focus group in (name of location) said it. To ensure all your responses are captured accurately, we will be taking notes and recording the group discussion. The recordings will be kept in a secure data storage and will be destroyed after submission of the report. The focus group will last approximately one hour. If you have any follow-up questions, please contact Louisena Louis at xxx. Do you understand that your participation is voluntary, unpaid, and that you stop at any point? Yes___ (check) No___ (check) Are you willing to participate? Yes___ (check) No___ (check) Date _________________ Signature of Witness ___________________________________ 63 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV FOCUS GROUP DISCUSSION GUIDES FOR SANTÉ FGD Guide for Clients Care from health facilities (HFs): 1. How often have you visited the HF this year (in 2019)? 2. What prevents you from coming to the HF for care when needed? 3. Describe what you like about the facility and what you think needs to be improved? 4. What changes have you observed in your facility since Santé started (two years ago)? 5. Does your facility have the equipment and supplies that it needs? 6. Do different types of people (men/women, rich/poor) get better care/different treatment at the health facility? 7. Are there services that you prefer to get from the health facility and other services that you prefer to get from the community health worker (ASCP)? Care from ASCPs: 8. Can you tell us something you learned from the ASCP in your area? 9. What do you think about range/types of services that your ASCP provides? Should s/he do more or less? 10. What is your ASCP good at doing/not good at doing? What do you think in general about quality of the ASCP’s health skills and knowledge? 11. Are you aware of the ASCP being trained to provide new services/get better at his/her job in recent years? 12. Do you get care at mobile clinics? What kinds of care do you get from them? Recommendations for Santé/MOH: 13. What could Santé do differently in the next two years? What changes could they make to how they work with the health workers and with the community around the health facilities? 14. Have you heard about community-based health facility advisory groups? (If they have not, explain “They are made up of people like you who use health services who hold meetings and make recommendations to the Santé project and to health workers.”) What do you think of these advisory groups? FGD Guide for Community Health Workers (CHWs or ASCPs in Haiti) Introduction: 1. How long have you been an ASCP? 2. Are you paid by the Santé project or by someone else? 3. How often do you work/interact with someone from the Santé project? Revisions to ASCP responsibilities and certification: 4. What do you think about the MOH’s new certification requirements for ASCPs? 5. What are the new job responsibilities for ASCPs and how are you adjusting to them? 64 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV 6. How are ASCPs coping with the workload, transportation issues, and other issues they face? Service provision by ASCP under Santé: 7. Let’s talk about the new HIV/TB and PEPFAR systems/tools that Santé has introduced. What are the main changes and how have you integrated them into your work? 8. How have you changed the nutrition and child health services that you offer now? 9. Do you collaborate with the mobile clinics that provide family planning and reproductive health? 10. What are the obstacles faced by pregnant women that keep them from coming to the CPN four times, making delivery plans to come to the health facility, and get PNC for themselves and their newborn? What is the role of the ASCP in helping them overcome these barriers? Santé Investments and Activities: 11. What do you think about the Santé project’s support for ASCP trainings and certification? Did any of you benefit from that support? If so, please tell us about how it went. 12. How often do you refer patients to a health facility for services? Do you accompany them, or do they go on their own? Do you have suggestions about how to improve the transfer of patients to health facilities? 13. What is the role of the ASCP in emergency preparedness and after an emergency begins? 14. Santé has talked about setting up community advisory groups made up of a broad range of people, including ASCPs, to collaborate with them on how the project is implemented. What do you think? Transition to the MOH: 15. What have you heard about plans to transition staff paid by Santé to the MOH? 16. What recommendations do you have for Santé in its next two years? 65 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV CONSENT FORM FOR CATCHMENT AREA SURVEY Hello, I am _______________ and I work with (data collection firm) and we are conducting a survey for the United States Agency for International Development (USAID) Haiti Office about its Santé Activity. We want to understand how successful Santé has been in meeting its objectives and how its work over the past two years can inform future USAID work in Haiti. We are surveying 1,800 people who use health services in this area and can tell us about those services. Your participation in the survey is completely voluntary and will not affect your ability to use health services. You are free to decline to respond to any question for any reason and stop answering at any time. You will not be provided with any payment for your participation in this interview. Your honest responses will help USAID inform and improve its future programming in Haiti. Your personal information (e.g. name, address, phone number) will not be connected to any information you provide in this interview. Your answers will be combined with those from all the other people in the survey and only those total results will be reported to USAID and shared on their website. We will be recording your answers on this electronic device and then putting them together with what the other people say. The information that you share with us will be kept confidential to the fullest extent under Haitian law and U.S. Government policy. We are asking for about 20 to 30 minutes of your time today to answer questions about your and your family’s health and how you use health services. If you have any follow-up questions, please contact Louisena Louis at xxx Do you understand that your participation in this survey is voluntary, unpaid, and that you can stop at any time? Yes___ (check) No___ (check) Are you willing to participate? Yes___ (check) No___ (check) CATCHMENT AREA SURVEY FOR Santé Department: _____________________________ Commune_________________________________ Date: ____________________________________ Enumerator name: __________________________________________ Household Roster – please complete for all members aged 18 or older FIRST NAME AGE* SEX IF FEMALE, IS SHE PREGNANT OR BREASTFEEDING? IS THIS PERSON A PARENT OR CAREGIVER OF A CHILD AGE 0- 5? Yes No DK Yes No Yes No DK Yes No Yes No DK Yes No 66 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV FIRST NAME AGE* SEX IF FEMALE, IS SHE PREGNANT OR BREASTFEEDING? IS THIS PERSON A PARENT OR CAREGIVER OF A CHILD AGE 0- 5? Yes No DK Yes No Yes No DK Yes No Yes No DK Yes No *A person must be at least 18 years old to be included in the roster. Once the person has been selected for the survey, read the informed consent form and ask if s/he is willing to respond to the survey. Then, use the questions below to confirm the information provided in the roster for segmentation. Screening questions Sex: Age: If female, is she pregnant or breastfeeding? Y / N Is the respondent a parent or caregiver of a child under age 5? Y / N If the person selected is not currently now, can the survey team return today for an interview? Y / N* Has the person selected agreed to be interviewed after hearing the informed consent script? Y / N *If no, select another person from the HH for the survey What is the name of the health facility you usually/often go to? _____________________________ (The data collection device checks whether this response is the Santé-supported facility or not. If not, the respondent will be asked the questions in Module B). When is the last time you visited this facility? _________________________________ (If longer than one year, then ask if they have been to another facility in the past year.) 67 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Module A Client access – (for respondents naming a Santé-Supported health facility above) QUESTION ANSWER OPTIONS NOT APPLICABLE DON’T KNOW Does your entire family go to facility x when they are sick? Only I go there Some of my family All of my family Who in your family goes to this Health Facility (HF) for preventive care? Only I go there Some of my family All of my family Does this facility offer immunizations for children? No Sometimes Always Does this facility offer pre-natal care (CPN) for pregnant women? No Sometimes Always Have you gone to this facility when you were (or someone in your family was) sick in the last three months/six months? No Past three months Past six months Have you been to this facility when you were (or a family member was) well for preventive services in the past three months/six months? No Past three months Past six months Can pregnant women deliver their babies at this health facility? No Sometimes Always Why do you go to this health facility, as opposed to another facility? [Do not read answers – check all that are given.] Distance Quality of the staff Cost Equipment and supplies Does this facility offer the lab tests you need? No Sometimes Always Have you asked to be tested for malaria at this facility in the past year? No Yes, and I got the test Yes, but I did not get the test Have you been offered an HIV test at this facility in the past year? Did you accept the test? No Yes, and I accepted the test Yes and I didn’t do the test Have you asked for family planning at this facility within the last year? No Yes, and I got information Yes, but I got no FP information Has this facility changed the number of health services offered in the last year? No (go to Module C) Fewer More What services have they begun to offer or have been removed? Lab testing HIV testing and treatment Family planning Well child care 68 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Module B: (respondents who did not list a selected Santé-supported health facility above) QUESTION ANSWERS NOT APPLICABLE DON’T KNOW Does your entire family go to facility x when they are sick? Only I go there Some of my family All of my family Who in your family goes to this HF for preventive care? Only I go there Some of my family All of my family Why do you choose to go to this facility as opposed to another facility? Distance Quality of the staff Cost Equipment and supplies Have you heard about the Santé project? No (skip to Module C) Yes Have you participated in a Santé health service or educational session in the last year? No (skip to Module C) Yes, a health service Yes, an education session What kind of health services did you or a family member receive from a Santé event? Family planning Immunization HIV testing Pre-natal care 69 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Module C: Quality/Satisfaction (ask of all respondents) QUESTION SCORE NOT APPLICABLE DON’T KNOW How would you rate the reception you receive from the staff when you visit the health facility (HF)? Poor Good Excellent What is the state of cleanliness of the HF (rooms, waiting area, etc.)? Dirty Clean Very clean How would you rate the amount of time you have to wait at the HF? Too long Average Short How long do you usually wait to be seen when go to the HF when you are sick? Minutes/ hours Do you routinely get complete information about the illnesses for which you seek treatment? No Partial Complete Does the staff fully explain to you the treatment, prognosis, and follow-up care required for your illness(es)? No Partial Complete How would you rate the testing services at your HF (lab, ultrasound, x-ray, etc.? Poor Average Excellent Are the medications that are prescribed for you usually available at the HF? No Sometimes Always Do the staff maintain the privacy about your illness with other staff and clients? No Sometimes Always Do community health staff (ASCP, accompagnateur) support you to make and attend appointments at the HF? No Sometimes Always Have you ever used (or needed to use) this HF’s transport or referral system? No, I did not need to use it No, my HF does not offer transport or referral Yes If yes, transport or referral or both How do you usually travel to the HF? Walk Vehicle Moto Other How long does it usually take you to get there? [Record the amount of time by mode] Walk Vehicle Moto Other Does this HF have a place for the staff and patients to wash their hands (with soap)? Never Sometimes Always What types of care can be had through a mobile clinic? [Do not read the answers] Family planning Nutrition Vaccinations Pre-natal care Child health Sick care (when ill) Don’t know 70 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Module D: Child Health (for caregivers of children under 5) I want to confirm that you are a caregiver of a child under 5 and ask you the name(s) age(s) of the child(ren). Y / N (if no, skip this module) QUESTION NOT APPLICABLE DON’T KNOW When was the last time you took a child for a vaccination? (*skip next 3 questions) Previous year Within the last 2 years More than 2 years* Don’t remember* When you take your child(ren) for vaccinations, do they usually have the vaccines and supplies needed? Never Sometimes Always What do you think about the length of time you have to wait when you go for vaccinations? Too long A while Not long On average, how long do you have to wait to get your child vaccinated? Minutes/hours Has a nurse or ASCP ever told you about oral rehydration solution for when a child has diarrhea? No Maybe Yes Do you know how to assess whether a child is malnourished? No Maybe Yes If yes, did you learn about malnutrition assessment from a nurse, ASCP, or somewhere else? ASCP Nurse Other Can an ASCP manage most of the common illnesses in children? No Sometimes All the time What are the main problems you face when your child is ill and you want to take him/her to the HF? Cost Distance Other When you take a sick child to the HF near here, how do the nurses there treat you? Rudely Reasonably well Respectfully Do the nurses have the knowledge and supplies to treat sick children? Neither Knowledge Supplies 71 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Module E: Family Planning (ask of all respondents) QUESTION NOT APPLICABLE DON’T KNOW Have you ever heard about ways to prevent pregnancy that are permanent or temporary? No Temporary or Permanent Temporary and Permanent Are the medical providers (nurses, ASCPs, etc.) you know open to talking to adults about how to avoid getting pregnant? No Sometimes Always Have you ever talked with a health provider about using family planning (FP)? No (skip to next section) Yes Did the provider give you useful information about a range of FP methods and ask you questions about what you wanted to use? No Gave information Gave information and asked questions Did the provider give you complete information about the side effects of the FP methods? No Partial Complete Are you or your partner currently doing something or using any method to delay or avoid getting pregnant? No (skip to xx) Yes (next question) Is the FP method that you chose always available at the HF? No Sometimes Always Does the provider treat you with respect and protect your privacy? No Respect or privacy Respect and privacy 72 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Module F: Reproductive Health (for women who have a child under age 5 or are pregnant) Did you see anyone for antenatal care during your last pregnancy? Y / N (if yes, next question; if no, skip to questions about post-birth check-up) Did you go for antenatal care (CPN) at a health facility or a rally post? HF / rally post As part of your antenatal care during your last pregnancy, did a healthcare provider do any of the following at least once? YES NO DON’T KNOW Measure your blood pressure Listen to the baby’s heartbeat Talk with you about breastfeeding Ask if you have vaginal bleeding Give you an injection for Tetanus Give you iron tablets Talk to you about how to prevent Zika Ask where you plan to deliver the baby QUESTION For non-pregnant women: after you delivered the baby, did you have a check on your and the baby’s health with a provider? Yes No = skip next two questions Don’t remember = skip next two questions If Yes, who did you see for this visit? Nurse ASCP Other/DK If Yes, how long after the birth was the visit? # of days Don’t remember What is the most important obstacle that prevents women in your community from giving birth in a health facility? Cost, distance, partner is against facility delivery, responsibilities at home, other (specify) What are two things that could be done to motivate more women to give birth at a health facility? 73 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Module G: HIV Knowledge (for all respondents) What is the most common mode of transmission of HIV in Haiti? Sex Mother to Child Syringes Other Do nurses and ASCPs talk about HIV with you when you see them for other problems/diseases? No Sometimes All the time Don’t know Do you know where people can get tested for HIV? No Yes Maybe Not sure When people go for an HIV test, do they always get counseling along with the test? Never Sometimes Always Don’t know Have you ever been tested for HIV and received the results? I don’t want to know your results, only if you have taken the test and gotten the results. No Yes, but not gotten results Yes, gotten the results Don’t know If you have been tested, how was the pre- /post-test counseling you got? Poor Helpful Very good Don’t recall People who have HIV can take medicine to control the disease. Have you heard about anti-retroviral treatment (ART)? No Yes, but not sure what it means Yes, and know about ART Don’t know What are the obstacles that people with HIV who are on ART have to deal with to take the drugs regularly? Stigma - from family, friends, or others Lack of drugs or staff at the HF Transport to the HF Don’t know Are people who take the ART drugs regularly and go for tests still able to give HIV to another person? No Maybe Yes Don’t know Do you know where people can get the ART drugs and other care for HIV? No Maybe Yes Don’t know 74 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV ANNEX C: INFORMATION SOURCES Annual Reports, Santé Background Documents USAID/Haiti. Santé Program Year I Annual Progress Report. N.p.: USAID, 2018. Print. Reference #6 USAID/Haiti. Santé Program Year 2 Annual Progress Report. N.p.: USAID, 2019. Print. Reference #15 Quarterly Reports, Santé Background Documents USAID/Haiti. Santé Year 1 Quarter 2 Progress Report, N.p: USAID, 2018, Print. Reference #4 USAID/Haiti. Santé Year 1 Quarter 3 Progress Report, N.p: USAID, 2018, Print. Reference #5 USAID/Haiti. Santé Year 2 Quarter 1 Progress Report, N.p: USAID, 2019, Reference #10 USAID/Haiti. Santé Year 2 Quarter 2 Progress Report, N.p: USAID, 2019, Reference #11 USAID/Haiti. Santé Year 2 Quarter 3 Progress Report, N.p: USAID, 2019, Reference #12 Annual Work Plans, Santé Background Documents USAID/Haiti. Santé Year 1 Annual Work Plan, N.p: USAID, 2018, Print. Reference #1 USAID/Haiti. Santé Year 2 Annual Work Plan, N.p: USAID, 2019, Print. Reference #7 USAID/Haiti. Santé Year 3 Annual Work Plan, N.p: USAID, 2020, Print. Reference #16 Monitoring & Evaluation Plans, Santé Background Documents USAID/Haiti. Santé Monitoring, Evaluation and Learning Plan Year 1, USAID, 2018, Print. Reference # 2 USAID/Haiti. Santé Monitoring, Evaluation and Learning Plan Year 2, USAID, 2019, Print. Reference # 8 USAID/Haiti. Santé Monitoring, Evaluation and Learning Plan Year 1 Annex B, USAID, 2018, Print. Reference #3 USAID/Haiti. Santé Monitoring, Evaluation and Learning Plan Year 2, Annex B, USAID, 2019, Print. Reference # 9 Project Description & Modifications, Santé Background Documents USAID/Haiti. AID-OAA-C-14-00067 Program Description (from proposal). N.p.: USAID, Reference #13 USAID/Haiti. AID-OAA-C-14-00067 Modification 1-3 to the Cooperative Agreement (HFR). N.p.: USAID, Reference #14 USAID/Haiti. AID-72052118-CA-00001 Section C - Statement of Work. N.p.: USAID, n.d. Print. 75 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV USAID/Haiti. AID-72052118-CA-0000 J Modification 6 (PEPFAR reporting). N.p.: USAID, n.d. Print. USAID/Haiti. AID-72052118-CA-000 I Mitigation plan (HHF Departure). N.p.: USAID, 2019, Print. Reference #17 USAID/Haiti. AID-OAA-C-14-00067 Haitian MOH Package of Essential Services (PES) N.p.: USAID, Reference #18 USAID/Haiti. AID-OAA-C-14-00067 SSQH Mid-Term Evaluation Report N.p.: USAID, Reference #19 USAID/Haiti. AID-OAA-C-14-00067 SSQH Haitian MOH Organization of Community Health Services Guide N.p.: USAID, Reference #20 USAID/Haiti. AID-OAA-C-14-00067 SSQH SDSH Evaluation Report N.p.: USAID, 2016 Reference #21 USAID/Haiti. AID-OAA-C-14-00067 Community Health Systems Catalog Country Profile: Haiti N.p.: USAID, Print. Reference #22 Caris Foundation Road map for transition of contractual staff to the MOH. N.p.: Caris 2020 Print Context Documents Reviewed USAID/Haiti 2018 Strategic Framework 2018-2020. Retrieved from: https://www.usaid.gov/sites/default/files/documents/1862/USAID-Haiti-Strategic-Framework-2018- 2020.pdf USAID/Haiti 2020 Country Profile. Retrieved from: https://www.usaid.gov/sites/default/files/documents/1862/USAID_Haiti_Country_Profile_January_2020.p df USAID/Haiti 2020 Health Fact Sheet. Retrieved from: https://www.usaid.gov/sites/default/files/documents/1862/USAID_Haiti_Health_Fact_Sheet_- _January_2020.pdf Institut Haïtien de l’Enfance (IHE) et ICF. 2018. Enquête Mortalité, Morbidité et Utilisation des Services en Haïti 2016-2017: Rapport de synthèse. Rockville, Maryland, USA: IHE et ICF https://dhsprogram.com/publications/publication-SR249-Summary-Reports-Key-Findings.cfm Congressional Research Service R45034 Haiti’s Political and Economic Conditions March 2020. Retrieved from: https://fas.org/sgp/crs/row/R45034.pdf USAID OIG Audit Report NO. 1-521-16-006-P July 6, 2016. Retrieved from: oig/usaid.gov USAID/Haiti. AID-OAA-C-14-00067 Services de Santé de Qualité pour Haiti (SSQH) Evaluation Report. USAID 2016 Print USAID/Haiti. GHS-1-00-05-00005-00 Santé pour le Developpement et la Stabilité d’Haiti (SDSH) Evaluation Report No. 11-01-584 USAID. November 2011 76 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Journals and Publications Congressional Research Service R45034 Haiti’s Political and Economic Conditions 6 March 2020. Retrieved from: https://fas.org/sgp/crs/row/R45034.pdf https://crsreports.congress.gov Institut Haïtien de l’Enfance (IHE) et ICF. 2018. Enquête Mortalité, Morbidité et Utilisation des Services en Haïti 2016-2017: Rapport de synthèse. Rockville, Maryland, USA: IHE et ICF. Retrieved from: https://dhsprogram.com/publications/publication-SR249-Summary-Reports-Key-Findings.cfm Ministere de la Santé Publique et de la Population, Organisation des Soins de Santé Communautaire, Guide pour la mise en place de l’Équipe Santé de la Famille, July 2017. Retrieved from: https://mspp.gouv.ht Ministere de la Santé Publique et de la Population, Le Paquet Essential des Services, October 2015. Retrieved from: https://mspp.gouv.ht PEPFAR Country Operational Plan Haiti 2019 Strategic Direction Summary June 11, 2019 Documents requested but not shared Draft plan (road map) outlining the goals and processes for the transfer of staff to the MOH (EQ 4) – REVISED: SHARED APRIL 10, 2020 USAID/Haiti. Santé Year 2 Indicator results (not included in the Year 2 Annual Report) USAID/Haiti. Santé Year 3 Monitoring Evaluation and Learning plan USAID/Haiti. Santé Year 3 Quarter 1 indicator results (not included in the Q1 report shared with the ET) Contingency plan for local stakeholder sustainability / Organizational Capacity Assessment scores 77 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV ANNEX D: EVALUATION MATRIX DATA SOURCES ILLUSTRATIVE SUB-QUESTIONS ANALYSIS PLAN DATA COLLECTION METHOD EQ • Content analysis • Gap analysis • Comparative analysis • Gender analysis • Has Santé established advisory groups at the national, departmental, and/or site levels? If yes, when? If no, why not, and will they still be set up? • What is the advisory group’s expected role and what is the plan to build their capacity? • What were the original assumptions of the Santé team and how have they changed? • Is the design of Santé still focused on the PES? If yes, should this continue? If no, should the PES be the focus? • What changes, if any, should be made to revise the activity’s design? • Should future health programming include advisory boards to promote participation and ownership? • USAID health personnel • Caris and sub-awardees • MOH staff (central, departmental, and site) • Activity documentation • Organizational Capacity Assessment scores (for advisory boards, if available) • Desk review • KIIs • FGDs • EQ1: To what extent and in what ways are the original Activity design and assumptions still relevant? In addressing this question, special attention should be given to the relevance or functionality of the National and Regional advisory boards. • Content analysis • Gap analysis • Comparative analysis • What requests have been made of Santé by the MOH, USAID/Washington, and USAID/Haiti? • Have changes in PEPFAR funding and programming guidelines affected Santé’s activities? • How are the implementors adapting to the requests made of them by the donor and MOH? • Is Caris using an adaptive management approach effectively? • What are the anticipated changes that will come from the MOH and USAID in the remaining two years of programming? • USAID health personnel • PEPFAR staff • MOH staff (especially any on the Cellule de Coordination) • Santé management and senior technical staff • Desk review • KIIs • EQ2: To what extent and in what ways is Santé effectively balancing achieving its original objectives and addressing current concerns and requests of key stakeholders, especially USAID/Washington, USAID/Haiti and the Government of Haiti? • Trend analysis • Content analysis • Gap analysis • Comparative analysis • Gender analysis • Has the activity achieved the targets set for its core indicators? • What trends do the quarterly data show for selected indicators of MCH and HIV/AIDS/TB? • Are people choosing not to access services at Santé￾supported facilities because of quality issues (staff, supplies, equipment, infrastructure)? • Project monitoring data • Catchment area survey • USAID health personnel • Caris and sub-awardee personnel • GOH officials – MOH • Desk review • KIIs • FGDs • Santé Health Management Information • EQ3: To what extent and in what ways have Santé’s interventions been effective in achieving key results, especially increasing access to health services? The 78 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV DATA SOURCES ILLUSTRATIVE SUB-QUESTIONS ANALYSIS PLAN DATA COLLECTION METHOD EQ • What differences have clients observed in the availability • Quantitative analyses and access of health services since Santé began? • Are there differences between men and women in terms of access to and use of health services? (central, departmental and site) • Activity documentation • Relevant research and technical reports System (HMIS) • Catchment Area Survey • HIV testing and ART register reviews evaluation should consider client and key stakeholder satisfaction as well as other measures of access to and successful delivery of health services. • Content analysis • Gap analysis • Comparative analysis • Gender analysis • Which aspects of Santé does USAID expect to transfer to the MOH at the end of the activity? • Is Santé building the capacity of the MOH (at different levels) to be able to receive activities at the end of the Activity? • How else is Santé preparing to transition select activities to the MOH? • Are the DDS work plans and budgets guiding the implementation of Santé in each department? • Is the MOH able to pay for their staff who are currently paid by Santé? What will happen to the others? • USAID health personnel • Caris personnel • GOH officials – MOH (central, and departmental) • Activity documentation • (need the Santé capacity￾building and handover plan) • Relevant research and technical reports • Desk Review • KIIs • FGDs • EQ4: To what extent and in what ways is Santé planning for the transition of its activity to the Ministry of Health? In addressing this question, the Evaluation Team should consider the project capacity￾building and hand over plan. 79 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV ANNEX E: DETAILED SAMPLING PLAN PURPOSE OF THIS DOCUMENT This document summarizes the sampling approach as conducted for the USAID/Haiti Santé performance evaluation. This document was produced by members of the analytical/technical support team brought on to assist the evaluation team to conduct survey data analysis. The description of the approach below reflects what was actually done, in accordance with the approved design (even if described slightly differently) and appropriate technical names reflecting the actual methods and approaches followed. The purpose of this document is to ensure common understanding of the approaches followed for sampling, corrections/adjustments/weighting needed as a result of those approaches, and the calculations and procedures followed to carry them out. BACKGROUND The Santé performance evaluation team conducted a catchment area survey to measure access to, perception of, and satisfaction with health care services. The survey sample was intended to represent adult residents living in the Santé health facilities’ catchment areas. Multi-stage stratified cluster sampling was carried out, necessitating the application of design weights to obtain representative estimates for the population of interest. In addition, Santé’s target sub-populations were intentionally oversampled, including a) pregnant or breastfeeding women and b) female caregivers of children under 5. Application of design weights in these scenarios is standard practice. Design weights are required because of the sampling design, specifically that it is not a simple random sample (SRS) from a well-defined sampling frame (not feasible in this situation). This means that not every unit had the same chance of being selected. For example, households in clusters with 100 households would have a 1/100 chance of being selected, while households in clusters with 10 households would have a 1/10 chance of being selected. Not applying design weights in this case would be incorrect, as treating such households equally does not reflect the underlying populations, which the survey was intended to represent. As documented in the approved design report, the stated intention of the oversampling was to prioritize the selection of eligible women in the relevant stage of sampling to ensure that at least 400 were included, and then return to random selection of eligible individuals within a systematically sampled household. However, although the sample size was originally planned to include 1,000 women, of which 400 were to be in the targeted sub-populations, and 800 men, enumerators erroneously continued to focus their sampling on the sub-populations of interest. This ultimately resulted in a proportionally low percentage of male respondents in the sample. Importantly, the planned and approved oversampling itself would have already required that post-stratification weights be applied to estimate indicators for the whole population or for women as a whole; the importance of applying this adjustment was all the more reinforced by the further issue of fewer men in the sample than planned. As described earlier, these methods would have been applied during the analysis regardless. This document summarizes how the weights were applied in general, and specifically to address the under sampling of men. 80 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV SANTÉ SAMPLING PROCEDURES To properly construct the design and post-stratification weights, a detailed understanding of the Santé sampling procedures is needed. Based on information in the Santé design report and correspondence with the evaluation team and ESS, the description of how sampling was carried out is as follows: 1. Decision made to sample 1,800 households based on sample size calculations. A sample size of 900 each, male and female, resulted from calculating the sample size needed to estimate a proportion of 75 percent with 95 percent confidence level, 4 percentage point margin of error, and an assumed design effect of 2. However, to prioritize women, the 1,800 total was intended to be split disproportionately to target 1,000 women and 800 men. 2. Since it was not practically feasible to sample all 164 health facilities, the team sampled 60 Santé health facilities. Since one-third of the 164 health facilities are PEPFAR sites and those received greater USAID support than non-PEPFAR ones, the team decided to divided the health facility sample accordingly with 20 PEPFAR-supported sites (one-third) and 40 non-PEPFAR (two-thirds) in the sample. 3. Sampling of the health facilities was thus stratified by PEPFAR support. Non-PEPFAR supported site selection was further sub-stratified by region. In practice, this leads to four strata: PEPFAR all regions, and non-PEPFAR North, Center, and South. Note that in actuality, 17 PEPFAR health facilities were included along with 40 non-PEPFAR facilities for a total of 57 instead of 60 facilities. Three PEPFAR facilities were dropped due to issues with accessibility. 4. Per the approved design, each of those 60 Santé supported health facilities were mapped and SDEs within a half kilometer of the urban health facilities and 1 kilometer of the rural health facilities were identified. The ET randomly chose one SDE per catchment area for data collection. 5. Thus, the approved design was to sample one cluster (SDE) per health facility catchment area, with 30 households each per site. Note that in actuality, more than one SDE was surveyed per cluster in a few cases because there were an insufficient number of households in the one originally selected. Therefore, in those cases an adjacent SDE was selected to reach the total of 30 households. 6. Per the approved design, households were selected for the survey through systematic sampling in the field (a central landmark was identified in each SDE and households were approached at a specified interval based on the estimated number of households in each sampled SDE). 7. For most SDEs: Within a sampled household, given the existence of any eligible woman in the household (pregnant or breastfeeding, and/or caretaker of child under five), one such woman was selected as the respondent. Female respondents were prioritized in general, and otherwise, if no eligible woman was present, an adult male was selected as the respondent. In the final four SDEs: after the issue of highly disproportionate female/male respondents was identified, ESS requested that the research firm stop prioritizing women and just apply the systematic sampling strategy as planned in the approved design. DESIGN WEIGHTS As described earlier, multi-stage stratified cluster sampling was carried out, necessitating the application of design weights to obtain representative estimates for the population of interest. The design weights fully take into account all of the sampling procedures described above. 81 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV VARIABLE CREATED PROBABILITY/WEIGHT SANTÉ STUDY CALCULATION p1 Probability of selection of health facilities within strata Health facilities selected from within each of the four strata: S1: PEPFAR > All regions S2: Non-PEPFAR > North S3: Non-PEPFAR > Central S4: Non-PEPFAR > South Within each stratum: p1 = # health facilities selected / # health facilities available p2 Probability of selection of cluster within stratum Clusters (SDEs) selected from within each of the health facility catchment areas identified through spatial maps For each health facility catchment: p2 = # SDEs selected/ # SDEs in the catchment area p3 Probability of selection of household within a selected cluster Households surveyed from within each cluster (SDE) For each SDE: p3= # HHs surveyed per SDE / # HHs total in each SDE [p4 group]: p_adult p_paran p_fem_ans_par Probability of selection of an individual of specific type from among eligible individuals within the household Three types of individual probabilities. 1) All adults (includes males and females) 2) Caregivers of children < 5 (includes males and females) 3) Women who are either pregnant/breastfeeding OR caregivers of children < 5 For each type of individual: [each of the p4 probabilities] = 1 / total # of such individuals in the household w_hh Sampling weight of household Inverse of the overall probability of selection of the household w_hh= 1/ (p1*p2*p3) w_adult w_paran w_fem_ans_par Sampling weight of individuals Inverse of the overall probability of selection of the individual (varies depending on type of individual) w = 1/ (p1*p2 * p3 * [one of p4 as applicable to indicator]) POST-STRATIFICATION WEIGHTS The purpose of the post-stratification weights is to calibrate estimates of any population-level indicators such that they are reflective of the overall population. In other words, they adjust for the disproportionate sampling approach (female/male) that took place in the field. The known population totals for male versus female population in Haiti are: Male 49.6 percent, and Female: 50.6 percent (source: link). The proportion of males and females in the final sample overall was 13.3 percent male and 86.7 percent female. Since the sampling proportions in practice differed at the cluster level, a cluster￾specific post-stratification weight is generated for each cluster, as shown below. In summary, the cluster￾specific post-stratification weight is calculated as the known population total of men or women, divided by the actual share of men or women in the cluster) 82 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV VARIABLE CREATED CLUSTER-SPECIFIC POST-STRATIFICATION WEIGHT postw Value of post-stratification weight for males: 0.496 / % M in cluster Value of post-stratification weight for females: 0.506 / % F in cluster FINAL WEIGHTS The final weights for the survey are then constructed by multiplying the design weight by the post￾stratification weight. The final weight is then used as the pweight in Stata using svyset and tabout commands to tabulate indicators for analysis. Application of household weights vs. individual weights were dictated by the type of variable being tabulated (e.g. whether household- or individual-level indicator). Therefore, there is no one single weight applicable for any observation, but rather there are different possible weights that are appropriate to apply depending on the indicator being tabulated. VARIABLE CREATED FINAL WEIGHTS w_hh_final Sampling weight of household (w_hh) * Post-stratification weight (postw) w_adult_final w_paran_final w_fem_ans_par_final Overall sampling weight of individuals (one of w_adult, w_paran, or w_fem_ans_par, as applicable for a given indicator) * Post-stratification weight (postw) 83 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV ANNEX F: DISCLOSURES OF CONFLICT OF INTEREST Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Nanette Barkey Title Team Leader Organization Individual Evaluation Position? ☒ Team Leader ☐ Team member Evaluation Award Number (contract or other instrument) 100078.000.0002.0019.0000.00US USAID Project(s) Evaluated {Include project name(s), implementer name(s) and award number(s), if applicable) Santé I have real or potential conflicts of interest to disclose. ☒ Yes ☐ No Ifyes answeredabove,Idisclosethe following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, oris significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct orsignificant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. #5 - I previously worked for Pact, who works in the health sector (until February 2018). I currently work part-time for PSI who also works in the health sector, although my position as Chair of PSl's research ethic's board is compartmentalized from the rest of the organization. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Date 10/16/2019 84 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Disclosure of Conflict of Interest for USAID Evaluation Team Members Name Louise Estavien Title Sector Specialist Organization Social Impact Inc. Evaluation Position? ☐ Team Leader ☒ Team member Evaluation Award Number (contract or other instrument) AID-521-C-17-00002 USAID Project(s) Evaluated {Include project name(s), implementer name(s) and award number(s), if applicable) Santé I have real or potential conflicts of interest to disclose. ☐ Yes ☐ No Ifyes answeredabove,Idisclosethe following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, oris significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct orsignificant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature Louise ESTAVIEN Date 11/27/2019 85 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV Disclosure of Conflict of Interest for USAID Evaluation Team Members Name FRANCOIS WILLY STACO Title ASSISTANT TEAM LEADER Organization SOCIAL IMPACT INC. Evaluation Position? Team Leader Team member Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer name(s) and award number(s), if applicable) USAID Haiti Evaluation Services I have real or potential conflicts of interest to disclose. Yes No If yes answered above, I disclose the following facts: Real or potential conflicts of interest mayinclude, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or issignificant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature francois staco Digitally signed by francois staco Date: 2020.04.21 21:05:41 - 04'00' Date 04/21/2020 86 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV ANNEX G: EVALUATION TEAM MEMBERS Team Leader: Dr. Nanette Barkey is an evaluation specialist with extensive experience in health, capacity building, social marketing, child protection, and economic empowerment (including specialized work on integrated community-led development). She has 35 years of experience designing, implementing, and evaluating programs that use integrated development approaches to improve health, with a focus on preventive medicine and health promotion. Dr. Barkey has worked in Haiti since 1983. In 2009, she led a CDC-funded study in Haiti to improve community participation in Lymphatic Filariasis (LF) elimination programming, and in 2010 conducted research on how the earthquake affected people infected with LF. In 2012-13, Dr. Barkey was the Monitoring and Evaluation (M&E) Technical Advisor for a child protection project in Haiti, guiding the baseline, developing the M&E plan, and mentoring the M&E team. She was part of a Save the Children initiative in 2018-19 on the role of health workers in disaster response, heading the Haiti study and contributing to the global recommendations. Dr. Barkey earned a Ph.D. at the University of Florida, a Master’s of Science in public health from the University of South Florida, and a bachelor’s degree from Case Western Reserve University. She is fluent in English, French, and Portuguese, partially fluent in Haitian Creole, Kikongo, Lingala, and Swahili, and has lived in Mozambique, Angola, Zaire (DRC), and Laos. Assistant Team Leader: Dr. Francois Staco is a medical doctor with an MD from the University of Notre Dame of Haiti, a Master’s in public health from American Public University (USA), and a certificate in Health Systems Management from the National Institute of Public Health in Japan. He has broad experience in coordinating, managing, and supervising health-related programs and research, including the areas of reproductive health, HIV/AIDS, results-based financing, and health information systems. Dr. Staco has more than 15 years of experience providing technical assistance to the Ministry of Health and the Haitian Government through various projects funded by USAID, CDC, the Global Fund, and other donors. His primary expertise is in health systems and primary health care, with a particular focus on district management, environmental health, and program design. He is especially adept at coordination, vision, and judgement, with excellent negotiation and team management skills. Dr. Staco is fluent in English, French, and Haitian Creole, and partially fluent in Spanish, with a talent for writing work in multicultural environments. Sector Specialist: Dr. Louise Estavien is a medical doctor with a specialization in pediatrics from the Haitian State University. She earned a Master’s Degree in the management and administration of health services, with an additional certificate in evaluation, from the University of Montreal. Dr. Estavien has more than 20 years of experience providing technical assistance for the coordination, management, and supervision of health-related activities, including those focused on TB, result-based financing, cholera, immunization, school health, health financing, and HIV/AIDS with emphasis on preventing mother-to-child transmission and pediatric HIV management. She has provided technical assistance to the Haitian Government through various projects funded by USAID, CDC, the Global Fund, and other donors. Dr. Estavien contributed to a therapeutic guide for HIV pediatric care and has worked on several HIV research programs. She has a great capacity for coordination, capacity building, design and program evaluation, working in multi-cultural settings, and speaks English, French, and some Spanish. 87 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV ANNEX H: STATEMENT OF DIFFERENCE MID-TERM PERFORMANCE EVALUATION REPORT SANTE I. INTRODUCTION The purpose of this statement of difference is to inform Social Impact that USAID/Haiti does not agree with the Mid-Term Performance Evaluation findings for the Santé project. II. SUMMARY INFORMATION Strategy/Project/Activity Name Project Santé Implementer CARIS Foundation Cooperative Agreement/Contract # 72052118CA00001 Total Estimated Ceiling of the Evaluated Project/Activity(TEC) $98,500,000.00 Life of Strategy, Project, or Activity December 2017 – November 2021 Active Geographic Regions 10 geographic departments Development Objective(s) (DOs) DO3: Health outcomes increased USAID Office Health III. BACKGROUND Santé builds on 20 years of USAID investments for health service delivery (HSD) in Haiti and has two key objectives: 1) utilization of quality, essential health services at 164 facilities and in surrounding communities across Haiti, and 2) strengthening local management and operational capacities to deliver health services. Santé technical teams provide coaching and mentoring to facility- and community-based health providers via regular site visits in coordination with Haitian Department Health Offices (Direction Départementale Sanitaire [DDS]). The Ministry of Public Health and Population MSPP) Package of Essential Services (PES) including primary care delivered at the facility- and community-level across multiple health areas (e.g., HIV, 88 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV maternal, neonatal and child health, family planning, nutrition) guided the technical assistance (TA) Santé delivered. Santé aims at achieving the following goals: 1) Increasing utilization of quality, essential health services (MSPP PES) at facility and the community levels by establishing a sustainable quality of service culture that reinforces local decision-making, accountability, local consumer/community engagement and health provider engagement. 2) Strengthening local management and operational capacities to deliver health services through systems planning, coordination and standard setting; facilitating certain sites’ readiness to become direct USAID recipients. In order to strengthen the local management and operational capacities to deliver health services, Santé will continue to assist the sites to provide a comprehensive range of PES services that meet the needs of the population through accountable and efficient management. Santé will continue to provide support to the DDS to strengthen basic equipment and supplies needed based on budgets and work plans provided by each DDS and approved by the MSPP. 3) Strengthening the existing systems for the implementation of a results-based financing (RBF) strategy that assists sustainable changes throughout the health system. 4) Supporting increased transfer responsibility for financing the system to MSPP/GOH. 5) Building the capacities of local entities. Santé is one of the most important projects in Haiti in terms of service delivery approach. The project is supporting a catchment population estimated to be 4,344,584, which accounts for 47% of Haiti’s population. Santé’s theory of change reflects two significant changes at various levels which must occur to achieve sustained well-being for all women, children, youth and families in communities served by Santé, and supported by investments from USAID and other donors. As Santé continues to implement evidence￾based approaches, developed with monitoring and evaluation data and related information, they will continue to refine the project’s theory of change, thus informing their results framework, MEL, implementation strategy and communications. All of these will be achieved through two integrated strategic approaches: 1) Effective and accountable health systems, and 2) Increased community mobilization. A. SUMMARY STRATEGY/PROJECT/ACTIVITY/INTERVENTION TO BE EVALUATED Below is a summary of the activities Santé is delivering at facility and community levels and the skills improvement of health care providers as stated in the agreement’s technical approach. A complete list will be shared upon request. 1) INCREASED QUALITY AND UTILIZATION OF SERVICES IN FACILITIES ● Strengthen linkages between community and facility through site advisory groups. 89 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV ● Assess the quality/time utilization for PES, specifically RMNCH, such as delivery procedures and protocols for each component to identify opportunities for offering a high-quality PES and a functioning referral system. ● Ensure family planning (FP) services, compliance with U.S. regulations and coverage through mobile clinics. ● Identify quality improvement and clinical risk management in the role of RMNCH and youth health. ● Enhance the RMNCH continuum of care approach by using mobile technology. ● Link site data collection systems to existing national data collection systems to allow targeting of resources and services. ● Ensure a 24-hour plan for emergency coverage with vehicles, national and regional ambulance services for the network – 24 hour plan. ● Make emergency phone lines available. ● Define clear protocols for referring emergency patients. ● Support the continuous enhancement of the ASCPs’ capacity to maximize a network of community providers and health promotion. ● Use data to drive interventions in the community, with integrated case management. ● Establish systems to track lost patients. ● Implement continuous quality improvement approaches to align site readiness to national standards for RBF. ● Strengthen management of the NGO-run sites. 2) INCREASED COMMUNITY ENGAGEMENT AND POSITIVE HEALTH SEEKING BEHAVIORS ● Create site/community advisory groups as advocacy ‘champions’ to engage strong and influential partners within the MSPP and other relevant ministries and sectors such as women’s affairs, finance, education, planning and development; key NGOs; international development partners; medical professionals; and civil society representatives ● Ensure active community and partner participation by engaging these stakeholders in the planning, implementation, monitoring, evaluation and feedback process. ● Define sensitization and mobilization activities with community members. Consider group discussions to address stigma, gender relations and best methods of communications. ● Community mobilization activities must ensure that its members are motivated and influenced to take action or support initiatives that are beneficial for themselves – through village gatherings, village health days, popular theatre, youth/women’s/fathers’ groups, and print and electronic media. ● Working with local communities to establish health service priorities and to build awareness and ownership of any new directions in health service provision. ● Promoting self-directed problem identification and early service seeking behaviors. ● Reducing HIV infection among adolescent girls and young women using community activities and education in schools and community groups. ● One task for SSAG will be to develop a community-based emergency transport strategy considering local public and private sector assets and capacity. ● Support/scale up existing community-based FP distribution and linkages to facilities. ● Promote community-level and site-based prevention and treatment education to ensure that STIs/RTIs awareness is raised, including the role of untreated STIs/RTIs in infertility, pregnancy complications and HIV infection. 90 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV 3) IMPROVED SKILLS OF HEALTH CARE PROVIDERS ● The three national training sites will be strengthened to provide continuous RMNCH health care workforce development, including providing high-impact, life-saving interventions. ● Strengthen health provider’s ability in providing LARCs at facility levels. ● ASCP regular training on recognition of complicated labor/ health emergencies. ● ASCP training in promoting PES and zika-related WASH community-level training. ● Education on labor complications; STI/GBV/ women’s health services ● Expand methods for accessing knowledge such as distance learning tools, and an online library in Creole and French. Designated professional development time, departmental QI conferences—share knowledge, best practices. ● Enhance the skills of health providers to provide clinically safe woman-centered maternity care ● Develop and support a rural/remote maternity workforce. ● Facilitate a culture of interdisciplinary collaboration in RMNCH services. B. SUMMARY OF THE PROJECT/ACTIVITY MONITORING, EVALUATION, AND LEARNING (MEL) PLAN Santé has defined a set of performance indicators containing a mix of outcome and output indicators that would help to capture project’s progress. Output indicators track the immediate products of project activities and provide feedback to managers on project performance to identify areas where implementation strategies may need to be adjusted. Outcome indicators measure the effects, or results, of project activities, at the higher levels of the project results framework. Indicators were selected based on the overall strategic approach of the project and closely reflect the work plan to capture the project’s ultimate goal. Indicators for the M&E system were selected in coordination with the Ministry of Health based on the overall strategic approach to the project and closely capture the work plan and link the main activities of the project with its stated goal. Santé performance Monitoring Matrix/MEL plan includes 44 indicators of which 17 are USAID standard indicators. In addition to the MEL plan, the Performance Indicator Reference Sheet (PIRS) also provides the name of the indicator, unit of measure, disaggregation, numerator, denominator and data source. IV. EVALUATION QUESTIONS 1. To what extent and in what ways are the original activity design and assumptions still relevant? In addressing this question, special attention should be given to the relevance or functionality of the national and regional advisory boards. 2. To what extent and in what ways is Santé effectively balancing achieving its original objectives and addressing current concerns and requests of key stakeholders, especially USAID/Washington, USAID/Haiti and the Government of Haiti? 3. To what extent and in what ways have Santé’s interventions been effective in achieving key results, especially increasing access to health services? The evaluation should consider client and key stakeholder satisfaction, as well as other measures of access to and successful delivery of health services. 4. To what extent and in what ways is Santé planning for the transition of its activity to the Ministry of Health? In addressing this question, the Evaluation Team should consider the project capacity building and hand over plan. 91 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV V. EVALUATION FINDINGS A. EVALUATION FINDINGS FOR QUESTION #1: Santé’s original design included core elements related to HSD – continuum of care, scaling-up, technical leads and technical teams – along with Health System Strengthening (HSS) for sustainability through capacity development and Santé Advisory Groups. While underlying assumptions are not clearly outlined in Activity documentation, through KIIs with the implementing partner (IP) and USAID, the ET identified five foundational assumptions and aligned them with ideas it found throughout various Santé documents. These included: 1) a smooth transition from the previous USAID HSD Activity, 2) a team of equal implementation partners, 3) regular site visits to build technical capacity, 4) the MSPP’s Package of Essential Services (PES) would guide implementation, and 5) good collaboration with the MSPP from the start. Each assumption has been challenged during the first two years of implementation due to: 1. The relationship with the government was not strong initially, with a delayed Activity launch, resulting in missed opportunities for a smooth transition between USAID’s previous HSD Activity and Santé. Likewise, Santé has spent much of its time in the first two years working to build relationships and trust with the MSPP. 2. The availability of equipment and supplies and the capacity of clinic staff and community health workers was lower than anticipated. 3. While their overall vision is the same, the MSPP and USAID/Santé have different implementation priorities that had to be balanced both for relationship and implementation management. 4. Santé launched just before Haiti slipped into a period of sustained political and economic instability. The security situation hindered implementation, including reducing the number and frequency of site visits and onsite support to health facilities. Santé adapted well to many of the challenges to its underlying assumptions, particularly shifting USAID and PEPFAR priorities. While adaptability is a core element of Santé’s design, these challenges resulted in several changes affecting implementation. The IP has devoted less attention to certain design elements, notably health system strengthening (HSS) activities. Capacity building for sub-awardees, collaboration with the DDS, and community engagement via the Santé Advisory Groups (SAGs) are three design elements toward sustainability that have not been executed as planned. USAID Response: Evaluation Question #1 was not addressed or answered in the evaluation report. Findings on EQ #1 are not credible as the evaluator failed to properly identify the project assumptions. The evaluation team failed to fully grasp the project rationale and did not develop an analysis framework that would have enabled them to address the agreed upon Evaluation Question #1 as validated in the design document. Also several attempts were made to have the contractor share an updated data table to back their assumptions; however, they were not able to do so. Therefore, findings are formulated as statements/opinions without citing empirical data to substantiate them. 92 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV B. EVALUATION FINDINGS FOR QUESTION #3 Most key informants (KI) and focus group discussions (FGD) respondents and the CAS indicated that Santé improved access to and quality of health services at facility and community levels. Santé has leveraged physical improvements, supplies and equipment at the facilities, to attract community interest, while training and mentoring from the Santé technical teams helped providers improve their clinical skills. Clients perceived that health care providers were more available, with newly acquired skills and knowledge, and providing client-centered care. Another positive change are the mobile clinics that link community members with more services locally and strengthen collaboration between CHWs and facility-based health providers. CHWs have been able to directly provide more services, facilitate referrals to facilities more frequently, and mobilize community members to attend mobile clinics. This is due partly to Santé’s training, supervision and mentoring of CHWs, and partly due to their expanded portfolio of community-level management of illness and preventive care. Clients reported improved satisfaction with their facilities as a result. Some Santé facilities still have shortages of supplies, medications, services, and vaccines, and even providers. Transportation for CHWs to perform their jobs and access for pregnant women to complete their antenatal care visits and deliver at health facilities, remain a challenge. While some of these challenges are within Santé’s purview (availability and quality of services), others are beyond its control (staffing, vaccines, medications). In addition, communication between health providers and clients remains a problem at some health facilities in that (1) some clients feel they are getting partial information about their illnesses, (2) some clients do not feel their provider is willing to talk about family planning, and (3) a couple of key elements, including birth planning, are not always included in antenatal visits. Given PEPFAR’s role in the activity, the ET looked in detail at HIV services. Santé has improved HIV services at facility and community levels. Santé effectively rolled out the newly mandated HIV treatment, as well as the new HIV testing strategy, which has proven more efficient in identifying HIV-positive clients. The ET notes that slow processing of viral load tests at the national laboratory causes delays in obtaining results and makes it more difficult for providers and PLHIV to know whether HIV suppression is effective. CHWs have played an important role in improving HIV patients’ care, as they supported efforts to decrease LTFU and encouraged patients to adhere to treatment. Despite many gains, challenges to quality health care remain formidable, including the ongoing economic and socio-political crises, security issues and geographical constraints. Haiti’s mountainous landscape also poses challenges for patients who may live within a short distance of a health facility, but due to the terrain and/or lack of transport options, can take much longer time to reach it. USAID Response: The ET was unable to adequately address and answer the EQ #3 because of their use of inappropriate and unapproved data collection methods and analysis. 1. The methodology section describes in detail the data collection and analysis methods. However, the quantitative method used to answer EQ #3 is not based on social science-based models of evaluation and assessment that would also reduce evaluator bias and subjective judgments. While the evaluator specifically mentions the use of multi-stage sampling design, this section is ambiguous and difficult to understand due to a lack of information: nothing is said about the sampling frame (number of clusters); no explanation is provided as to what is considered as a cluster: health clinics or SDEs; no details offered on how the 1,800 sample size was calculated nor the rationale to inform how the level of confidence (95%) 93 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV was chosen.; the notion of quota is introduced in the sampling strategy inconsistent with a two-stage cluster sampling using random or systematic sampling being a probability sampling. Additional confusing core elements include; the process used to determine a quota of 400 pregnant women; the lack of clarity regarding which respondents were targeted adult residents or pregnant women, care givers, etc.); the varying sampling approaches mentioned throughout the document, such as targeted sampling approach, multi-stage cluster sampling approach, etc. 2. At one point, the ET references the multi-stage stratified cluster sampling as the design to guide the data collection yet did not implement the key steps required for such a design: calculate the sampling size using a sampling size formula; stratifying the cluster by geographic areas using the sampling frame; selecting the cluster (SDE) using a form of probability sampling (random or systematic sampling); and selecting the households within each selected cluster. 3. The ET randomly selected 40 non-PEPFAR Santé supported health facilities proportionally from the North, Center and South geographic areas. In addition, the ET randomly selected 20 of the 46 Santé￾supported PEPFAR sites leading to four total strata: PEPFAR all regions, and non-PEPFAR North, Center and South. What is the reasoning behind this stratification? What factors were considered (level of accuracy required of estimates, time and resource constraints, confidence intervals, etc.) in determining the 40 and 20 samples sizes? 4. This design (four stage-cluster design with a mixed random and quota selection of respondents) is flawed from a statistical point of view. USAID regrets that such a design was implemented without written agency approval. This design is not a multi-stage cluster sampling as was expected from the USAID￾approved evaluation design document nor as was stated by the IP in the evaluation report. (See also the comment in line 4). 5. The ET modified the post-stratification weights post-data collection in an attempt to compensate for not adhering to the USAID-approved sampling framework. The two-stage cluster sampling should yield data that are representative of the sampling frame. 6. Many sections of the evaluation mentioned challenges faced by the activity but failed to mention mitigation measures that were put in place by the IP to cope with them. VI. CONCLUSION The Santé Evaluation did not meet USAID/Haiti evaluation quality standards. It did not properly implement the SOW and did not consider every aspect of the program implementation challenges and mitigation strategies. Many statements were made out of context or without sufficient or substantial data to support the claims. USAID requires evaluation findings that are based on facts, evidence and data. This precludes relying exclusively upon anecdotes, hearsay and unverified opinions. In addition, some of the sampling methods that were used were inappropriate. Evaluations are expected to use methods that generate the highest quality and most credible evidence, corresponding to the evaluation questions being asked. Hence, for all the above reasons, USAID/Haiti provides this Statement of Difference. 94 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV ANNEX I: EVALUATION RESPONSE DOCUMENT HAITI ESS SANTÉ MID-TERM EVALUATION RESPONSE DOCUMENT SI Response: We sincerely apologize for the challenges surrounding the finalization of this Evaluation Report and deeply appreciate USAID’s commitment throughout the process in working with us to improve the report and maximize its utility. We recognize that there are limitations to the methodology and minor issues during implementation, though we believe these are not particularly unique to this evaluation. We also recognize that those limitations could be better explained in the report to help readers understand how to interpret the credibility of results, and if USAID prefers, we are happy to edit the report to make that clearer. However, we disagree with many of the statements in the statement of differences (SOD), many of which we believe are incorrect or the result of misunderstanding, and the overall conclusion that the results are not credible. Below we have provided the original USAID SOD text in black, followed by our response in red. We would also emphasize that we see this as a valuable learning opportunity. This experience emphasizes the importance of stakeholder engagement during the design process, as we believe that most, if not all, of these issues or misunderstandings could have been prevented with more effective engagement between SI, USAID and the IP during design. USAID Response: Evaluation Question #1 was not addressed or answered in the evaluation report. SI Response: While we recognize that there are limitations inherent in the evaluation approach and that perhaps those could be better discussed in the evaluation report, we disagree with this conclusion, as there appear to be some inaccuracies or misunderstandings in the supporting statements, as described below. The misunderstandings stem from two main sources. First, much of the justification for the statement of difference on this EQ centers on the identification of project assumptions, yet the IP has confirmed that the assumptions in the report are correct, and no alternative set has been offered by USAID or the IP. Second, there seems to be a misalignment of expectations with regards to how qualitative data, particularly from FGDs and KIIs, can be analyzed and presented based on the approved design. The sampling and data collection approach for qualitative data, as outlined in the approved design report and as commonly used in evaluation, generate limitations in the ability to present results quantitatively. While it is important to give indications of the prevalence of various perspectives, a simple percentage of responses from qualitative data collection can be misrepresentative, since the sample was not selected to be representative, the sample size is small for many stakeholder groups, and not all respondents are able to speak on each topic (or are probed on the same topics). The aggregation of respondents within FGDs further complicates this sort of analysis. Accordingly, different approaches to qualitative data analysis and presentation, as discussed in the approved design report, are typically used. Findings on EQ #1 are not credible as the evaluator failed to properly identify the project assumptions. SI Response: This is inconsistent with our understanding and direct feedback from the Santé COP. On p16 of the report, we wrote, “The ET confirmed the foundational assumptions identified in the Activity proposal through discussion with the IP and USAID during KIIs.” In a comment on the final version of the report (May 21 version), the Santé COP confirmed, “The assumptions are stated clearly in the accepted proposal for award and are the ones that have been stated in this paragraph [of the report].” This seems to confirm that the evaluation team used the correct assumptions in the evaluation report and indeed, no other set of assumptions was provided by USAID or the IP. 95 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV The evaluation team failed to fully grasp the project rationale and did not develop an analysis framework that would have enabled them to address the agreed upon Evaluation Question #1 as validated in the design document. SI Response: The feedback on the report does not seem to justify the statement that the ET failed to grasp the project rationale. Indeed, as noted above, the assumptions included in the report align with the assumptions used by the IP. Moreover, the ET implemented the evaluation design for EQ1 as outlined in the approved design report. If there are areas where the report does not reflect an understanding of the project rationale or the implementation of the evaluation departs from the agreed design, we are open to discussing and potentially revising the report as appropriate. Also, several attempts were made to have the contractor share an updated data table to back their assumptions; however, they were not able to do so. SI Response: Upon approval of the final report, which is also USAID’s concurrence to post the report to the DEC and quantitative dataset to DDL, we will submit to USAID a closeout file with the de￾identified data (qualitative and quantitative) and the de-identified quantitative dataset to the DDL. This follows the scope of work’s directions. We are unclear on specifically what USAID is requesting apart from the de-identified data, specifically how it refers ‘back to our assumptions.’ Therefore, findings are formulated as statements/opinions without citing empirical data to substantiate them. SI Response: We wish to clarify that the statement/opinions of key informants and focus groups participants are indeed empirical data. The ET triangulated this data, along with the data from other sources cited, to derive the conclusions and recommendations in the report. As noted above, there are significant constraints in presenting, in quantitative terms such as percentages, qualitative data derived from KIIs and FGDs using semi-structured interviewing approaches and non-random sampling. Specifically, by design, the sampling for data collection was purposive or targeted and the semi-structured approach to interviewing means that probing may differ across respondents based on their knowledge and experience. Moreover, FGDs collect views of multiple respondents, but do not necessarily track individual responses. Accordingly, presenting or analyzing such data as a percentage of respondents can be highly misleading and as such, is typically not recommended. Instead, other approaches to data analysis including triangulation, better suited to this type of data collection, were proposed and approved in the design report and used for the analysis. That said, it is often helpful to have an indication of the relative prevalence of a particular perspective among relevant respondents. The ET attempted to provide this, while ensuring not to identify respondents or present a misrepresentative picture of the data, yet if there are areas where it can be improved, we are open to attempting to improve specificity given these limitations. Finally, we note that this limitation (and perhaps the benefits to provide justification for the approach) could be better explained in the methodology section of the report, and we are happy to edit to clarify if USAID prefers. This misunderstanding also reinforces the importance of better engagement of USAID with the ET at the design stage to ensure full understanding of limitations and agreement with approach prior to starting data collection, analysis, and presentation. Evaluation Question 3 USAID Response: The ET was unable to adequately address and answer the EQ #3 because of their use of inappropriate and unapproved data collection methods and analysis. 96 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV SI Response: Thank you for the opportunity to provide additional clarity on the approaches used for this EQ, particularly with respect to the catchment area survey (CAS). We believe that there is some misunderstanding or perhaps lack of clarity in the presentation of the methods in the baseline report, as many of the concerns noted below are incorrect and are directly addressed in the report, although the detail is mostly provided in Annex E of the report. There was indeed an issue in implementation of the survey sampling approach but only at the level of the selection of the respondent within the household. The selection of households was done according to social science best practices (described in more detail below). However, the selection of the respondent for each household was done incorrectly, with the data collection firm significantly oversampling respondents from Santé priority target groups. This was identified and corrected by SI during data collection, but the individual respondents at conclusion of data collection included significantly more women than men. However, it is very important to note that the vast majority of survey questions are asked about household (any member) access to services or maternal and child health, for which women would be the most likely knowledgeable respondents. For these questions, the oversampling of women should not lead to any bias, since they respond about the household or maternal/child health. Nevertheless, for the other questions, SI applied commonly used and accepted statistical techniques (post-stratification weights) to account for the imbalance of men versus women in the final sample. As noted in the baseline report, the original, approved design planned for oversampling (in order to have a higher level of confidence on indicators related to maternal/child health services), though to a smaller degree, and the use of post-stratification weights, just as was done in the baseline report. In summary, while there were some issues in data collection implementation, SI acted quickly to correct them and applied social science best practice to correct for the issue in analysis. Therefore, we are confident that these issues did not generate significant bias in the CAS. Further detail is provided below in response to each point. 7. The methodology section describes in detail the data collection and analysis methods. However, the quantitative method used to answer EQ #3 is not based on social science-based models of evaluation and assessment that would also reduce evaluator bias and subjective judgments. While the evaluator specifically mentions the use of multi-stage sampling design, this section is ambiguous and difficult to understand due to a lack of information: nothing is said about the sampling frame (number of clusters); no explanation is provided as to what is considered as a cluster: health clinics or SDEs; no details offered on how the 1,800 sample size was calculated nor the rationale to inform how the level of confidence (95%) was chosen.; the notion of quota is introduced in the sampling strategy inconsistent with a two-stage cluster sampling using random or systematic sampling being a probability sampling. Additional confusing core elements include: the process used to determine a quota of 400 pregnant women; the lack of clarity regarding which respondents were targeted adult residents or pregnant women, care givers, etc.); the varying sampling approaches mentioned throughout the document, such as targeted sampling approach, multi-stage cluster sampling approach, etc. SI Response: The Statement of Difference makes the following points to demonstrate the assertion that the sampling section is ambiguous and difficult to understand. We acknowledge that perhaps the narrative in the body of the report could be improved for clarity with much of the detail only included in the Annex. However, each of these points is addressed in the report, particularly in the Annex E on sampling: - nothing is said about the sampling frame (number of clusters): The report clarifies that the sampling frame consists of enumeration areas (SDE) in the catchment area of targeted health 97 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV facilities. While the number of SDE in each catchment area or in the overall sample frame is not included in the report, this level of information would not typically be included, although we can provide if desired. o “After site selection, the ET then mapped the catchment areas, including all census enumeration areas – called SDEs in Haiti – within one kilometer of rural health facilities and a half kilometer of urban health facilities. The ET then randomly chose one SDE for each of the 60 health facilities for the enumerators to survey.” (p10) o “Per the approved design, each of those 60 Santé supported health facilities were mapped and SDEs within a half kilometer of the urban health facilities and 1 kilometer of the rural health facilities were identified. The ET randomly chose one SDE per catchment area for data collection. Thus, the approved design was to sample one cluster (SDE) per health facility catchment area, with 30 households each per site.” (p79) - no explanation is provided as to what is considered as a cluster: health clinics or SDEs: Although the sampling approach is complex and multi-stage, the report notes that Santé supported health facilities are the primary sampling unit followed by SDEs, households, and individuals as subsequent sampling units. o Figure 3 (p9) shown below provides a visual representation of this process and the preceding discussion in the report (p9-10) includes 4 bullets, one for each of these stages (Site Selection, Enumeration Area Selection, Household Selection, Respondent Selection). o The following is excerpted from the table on p80 of the report: Health facilities selected from within each of the four strata: o S1: PEPFAR > All regions o S2: Non-PEPFAR > North o S3: Non-PEPFAR > Central o S4: Non-PEPFAR > South Clusters (SDEs) selected from within each of the health facility catchment areas identified through spatial maps Households surveyed from within each cluster (SDE) Three types of individual probabilities. o All adults (includes males and females) o Caregivers of children < 5 (includes males and females) 98 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV o Women who are either pregnant/breastfeeding OR caregivers of children < 5 - no details offered on how the 1,800 sample size was calculated nor the rationale to inform how the level of confidence (95%) was chosen: The report specifies the approach and parameters used for sample size calculation in Annex E. More detail is provided in the Design Report, including the formula used to calculate the required sample size on p26 of the approved Design Report, and can be added into the report, if desired. o “Decision made to sample 1,800 households based on sample size calculations. A sample size of 900 each, male and female, resulted from calculating the sample size needed to estimate a proportion of 75 percent with 95 percent confidence level, 4 percentage point margin of error, and an assumed design effect of 2. However, to prioritize women, the 1,800 total was intended to be split disproportionately to target 1,000 women and 800 men.” (p79) - the notion of quota is introduced in the sampling strategy inconsistent with a two￾stage cluster sampling using random or systematic sampling being a probability sampling: Neither oversampling a target population (even using a ‘quota’-based system) nor targeting certain respondents within a household are inconsistent with social science best practices. It is very common for different sampling approaches to be used at different stages in a multi-stage sampling design. Indeed, many surveys (including FTF and DHS) require certain respondent groups to respond to certain modules. While there were issues in the selection of individual respondents within the household (described more above), this does not affect the validity of the sampling approach used to select households, as described in the baseline report. 8. At one point, the ET references the multi-stage stratified cluster sampling as the design to guide the data collection yet did not implement the key steps required for such a design: calculate the sampling size using a sampling size formula; stratifying the cluster by geographic areas using the sampling frame; selecting the cluster (SDE) using a form of probability sampling (random or systematic sampling); and selecting the households within each selected cluster. SI Response: The Statement of Difference correctly identifies critical aspects of a strong sampling approach, but incorrectly asserts that the ET did not use appropriate sampling techniques. The ET did, indeed, follow each of these steps, and all of them are described under the “Santé Sampling Procedures” on page 79 in the baseline report, Annex E. Specifically: - calculate the sampling size using a sampling size formula: The report specifies the approach and parameters used for sample size calculation in Annex E. More detail is provided in the Design Report, including the formula used to calculate the required sample size on p26 of the approved Design Report, and can be added into the report, if desired. o “Decision made to sample 1,800 households based on sample size calculations. A sample size of 900 each, male and female, resulted from calculating the sample size needed to estimate a proportion of 75 percent with 95 percent confidence level, 4 percentage point margin of error, and an assumed design effect of 2. However, to prioritize women, the 1,800 total was intended to be split disproportionately to target 1,000 women and 800 men.” (p79) - stratifying the cluster by geographic areas using the sampling frame: The report specifies that the PSU (health facilities) were stratified by both PEPFAR-support and region (geographic area) prior to selection. 99 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV o “Since it was not practically feasible to sample all 164 health facilities, the team sampled 60 Santé health facilities. Since one-third of the 164 health facilities are PEPFAR sites and those received greater USAID support than non-PEPFAR ones, the team decided to divide the health facility sample accordingly with 20 PEPFAR-supported sites (one-third) and 40 non-PEPFAR (two￾thirds) in the sample.” (p79) o “Sampling of the health facilities was thus stratified by PEPFAR support. Non-PEPFAR supported site selection was further sub-stratified by region. In practice, this leads to four strata: PEPFAR all regions, and non-PEPFAR North, Center, and South.” (p79) - selecting the cluster (SDE) using a form of probability sampling (random or systematic sampling): The report specifies that after health facilities (PSU) were selected, all SDEs (secondary sampling unit or SSU) in the selected health facility catchment area were identified, and one SDE was randomly selected from each facility’s catchment area. o “Per the approved design, each of those 60 Santé supported health facilities were mapped and SDEs within a half kilometer of the urban health facilities and 1 kilometer of the rural health facilities were identified. The ET randomly chose one SDE per catchment area for data collection.” (p79) - selecting the households within each selected cluster: The report specifies that absent a full list of households or resources to conduct a household listing, a systematic approach to quasi￾random sampling was applied. o “During data collection, survey teams randomly selected 30 households within each SDE through a “random walk” methodology. Upon arrival, the survey team started in the center of the SDE, spun a pencil to determine the direction in which to walk, and walked until they encountered the first house. After that point, the team skipped N# houses before seeking their next respondent. N was determined by taking the number of households in the area and dividing it by 30; if the SDE contained 90 households the survey team visited every third household until they had completed 30 surveys.” (p10) 9. The ET randomly selected 40 non-PEPFAR Santé supported health facilities proportionally from the North, Center and South geographic areas. In addition, the ET randomly selected 20 of the 46 Santé￾supported PEPFAR sites leading to four total strata: PEPFAR all regions, and non-PEPFAR North, Center and South. What is the reasoning behind this stratification? What factors were considered (level of accuracy required of estimates, time and resource constraints, confidence intervals, etc.) in determining the 40 and 20 samples sizes? SI Response: The report clarifies that the number of PEPFAR versus non-PEPFAR Santé facilities sampled was determined based on the proportion of PEPFAR versus non-PEPFAR Santé facilities overall. The same proportion was used to maximize representativeness of the sample. - “Since it was not practically feasible to sample all 164 health facilities, the team sampled 60 Santé health facilities. Since one-third of the 164 health facilities are PEPFAR sites and those received greater USAID support than non-PEPFAR ones, the team decided to divide the health facility sample accordingly with 20 PEPFAR-supported sites (one-third) and 40 non-PEPFAR (two-thirds) in the sample.” (p79) 10. This design (four stage-cluster design with a mixed random and quota selection of respondents) is flawed from a statistical point of view. USAID regrets that such a design was implemented without written agency approval. This design is not a multi-stage cluster sampling as was expected from the USAID- 100 | USAID/HAITI SANTÉ MID-TERM EVALUATION REPORT USAID.GOV approved evaluation design document nor as was stated by the IP in the evaluation report. (See also the comment in line 4). SI Response: As described above and in the report, particularly Annex E, the approach was indeed a multi￾stage cluster sampling approach designed according to social science best practices. There was an issue with individual selection in implementation (and no issues in sampling at other stages down to the household level), but it was addressed using commonly accepted statistical approaches (post-stratification weights). Accordingly, this implementation issue does not introduce bias into estimates of household-level indicators or indicators relevant to targeted populations (the vast majority of indicators in the survey). Moreover, potential bias on indicators for men (who were undersampled) were properly mitigated according to accepted best practices. 11. The ET modified the post-stratification weights post-data collection in an attempt to compensate for not adhering to the USAID-approved sampling framework. The two-stage cluster sampling should yield data that are representative of the sampling frame. SI Response: As noted in the Baseline Report, the original, approved design included post-stratification weights to account for the planned oversampling of targeted groups. Moreover, it is unrealistic to expect any large household survey will achieve perfect implementation and response rates, so adjusting sampling weights after data collection is a common approach to increasing the representativeness of results. Indeed, adjustment of sampling weights post-data collection is part of USAID FTF guidance to account for non￾response (see, for example, USAID Feed the Future Population Based Survey Sampling Guide, p.79). - “Importantly, the planned and approved oversampling itself would have already required that post￾stratification weights be applied to estimate indicators for the whole population or for women as a whole; the importance of applying this adjustment was all the more reinforced by the further issue of fewer men in the sample than planned. As described earlier, these methods would have been applied during the analysis regardless.” (p78 baseline report) 12. Many sections of the evaluation mentioned challenges faced by the activity but failed to mention mitigation measures that were put in place by the IP to cope with them. SI Response: If there are important challenges where related mitigation measures were not mentioned in the report, we are happy to consider them for revision in the report, to the extent they are corroborated with our data.