IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS, IRAQ EVALUATION JULY 21, 2020 This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International (MSI), A Tetra Tech Company for the MENA MELS Activity. i | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS, IRAQ Evaluation Contracted under GS00F280GA, Task Order 7200AA19M00017 Middle East & North Africa Monitoring, Evaluation, & Learning Services Activity Report prepared by Dina Khaled, Heather Lorenzen, and Gwynne Zodrow (MSI) DISCLAIMER The authors’ views expressed in this report do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV CONTENTS ACRONYMS...........................................................................................................III EXECUTIVE SUMMARY ........................................................................................1 GRANT BACKGROUND...................................................................................................................................... 1 EVALUATION PURPOSE AND QUESTIONS................................................................................................. 1 METHODOLOGY ................................................................................................................................................... 2 FINDINGS AND CONCLUSIONS ..................................................................................................................... 2 EVALUATION QUESTION 1: CHANGES IN SERVICE DELIVERY AND ACCESS ..................... 2 EVALUATION QUESTION 2: UPTAKE OF HEALTH SERVICES (E.G. GBV)................................ 4 EVALUATION QUESTION 3: MHPSS SERVICES .................................................................................. 5 EVALUATION QUESTION 4: REFERRALS ............................................................................................. 7 EVALUATION QUESTION 5: PROJECT HANDOVER AND REVITALIZATION TO DOH ... 7 RECOMMENDATIONS.......................................................................................................................................... 9 INTRODUCTION..................................................................................................10 GRANT BACKGROUND.................................................................................................................................... 10 EVALUATION PURPOSE AND QUESTIONS...............................................................................................12 EVALUATION QUESTIONS...................................................................................................................... 12 METHODOLOGY..................................................................................................13 LIMITATIONS ......................................................................................................................................................... 14 FINDINGS AND CONCLUSIONS......................................................................14 EVALUATION QUESTION 1............................................................................................................................. 14 CAPACITY BUILDING................................................................................................................................ 15 HUMAN RESOURCES................................................................................................................................. 16 REHABILITATION AND SUPPLIES ......................................................................................................... 17 PRIMARY HEALTH CARE SERVICES ...................................................................................................... 18 MATERNAL AND CHILD HEALTH/FAMILY PLANNING ...............................................................19 COMMUNITY HEALTH SERVICES.......................................................................................................... 20 PATIENT SATISFACTION ......................................................................................................................... 22 UNINTENDED OUTCOMES (EQ 1B)....................................................................................................24 CONCLUSIONS (EQ 1).............................................................................................................................. 25 EVALUATION QUESTION 2............................................................................................................................. 26 KEY BARRIERS TO INCREASING UPTAKE OF HEALTH SERVICES (EQ 2A)...........................27 CONCLUSIONS (EQ 2).............................................................................................................................. 28 EVALUATION QUESTION 3............................................................................................................................. 28 CHALLENGES AND BARRIERS TO MHPSS SERVICE DELIVERY (EQ 3A) .................................29 MHPSS PROGRAM SUSTAINABILITY (EQ 3B)....................................................................................31 CONCLUSIONS (EQ 3).............................................................................................................................. 32 EVALUATION QUESTION 4............................................................................................................................. 33 CONCLUSIONS (EQ 4).............................................................................................................................. 34 EVALUATION QUESTION 5............................................................................................................................. 34 SERVICE DELIVERY AFTER END OF GRANT (EQ 5A)....................................................................35 STAFF SUSTAINABILITY (EQ 5B)............................................................................................................ 36 CHANGE IN COMMUNITY ENGAGEMENT LEVEL (EQ 5C)........................................................37 CONCLUSIONS (EQ 5).............................................................................................................................. 37 ii | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV RECOMMENDATIONS ........................................................................................38 ANNEX 1: EVALUATION STATEMENT OF WORK......................................39 APPROACH FOR HEALTH EVALUATION – NINEWA PLAINS HEALTH GRANT.........................39 INTRODUCTION........................................................................................................................................ 39 EVALUATION PURPOSE ........................................................................................................................... 39 GRANT OVERVIEW.................................................................................................................................... 39 EVALUATION QUESTIONS...................................................................................................................... 40 EVALUATION APPROACH AND DATA SOURCES.........................................................................41 DATA ANALYSIS.......................................................................................................................................... 42 EVALUATION DELIVERABLES ................................................................................................................. 42 ANNEX 2: IMC FINAL GRANT OUTCOMES DASHBOARD........................43 ANNEX 3: SOURCES OF INFORMATION.......................................................44 KEY INFORMANT INTERVIEWS CONDUCTED........................................................................................44 ANNEX 4: DATA COLLECTION TOOLS ........................................................46 ANNEX 5: EVALUATION TEAM .......................................................................52 TEAM COMPOSITION ........................................................................................................................................ 52 iii | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV ACRONYMS ANC Antenatal Care APC Advancing Partners and Communities CHW Community Health Worker DOH Department of Health EQ Evaluation Question EWARN Early Warning Alert and Response FHI 360 Family Health International GBV Gender-Based Violence IDP Internally Displaced Person IEC Information, Education, and Communication IMC International Medical Corps IPC Infection Prevention Control JSI JSI Research and Training Institute, Inc. MENA MELS Middle East and North Africa Monitoring, Evaluation, and Learning Services MHPSS Mental Health and Psychosocial Support MOH Ministry of Health MSI Management Systems International NGO Nongovernmental Organization PFA Psychological First Aid PHC Primary Health Care PHCC Primary Health Care Centers PNC Postnatal Care STI Sexually Transmitted Infection UNFPA United Nations Population Fund USAID United States Agency for International Development WASH Water, Sanitation, and Hygiene WHO World Health Organization 1 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV EXECUTIVE SUMMARY Under the Middle East and North Africa Monitoring, Evaluation, and Learning Services Activity (MENA MELS), Management Systems International (MSI) conducted an evaluation of the subgrant Improving Health Services and Outcomes in the Ninewa Plains, Iraq. The subgrant was implemented by International Medical Corps (IMC) under the Advancing Partners and Communities (APC) Grant implemented by JSI Research and Training Institute, Inc. (JSI) in collaboration with Family Health International (FHI 360). GRANT BACKGROUND The grant was implemented between July 1, 2018, and July 31, 2019, in the Ninewa Plains region. The grant focused on six primary health care centers (PHCCs)—Al-Mowafaqia, Al-Quba, Al-Qush, Al-Sada, Bashiqa, and Wana—and one secondary facility, Sinjar General Hospital (maternity unit). The key areas of support included primary health care, reproductive health care, mental health and psychosocial support (MHPSS) services, referrals and linkages, gender-based violence (GBV) prevention and response, and community health care. The grant’s main objectives and supporting activities were as follows: • Objective 1: Improve delivery of the essential package of primary health care services to minority populations in the Ninewa Plains by strengthen the capacity of PHCC; facilitate links to advanced levels of care (secondary care site); engage community health workers (CHW); support health facility managers to strengthen health facility management. • Objective 2: Strengthen and integrate MHPSS services, post-traumatic treatment for children, and sexual and gender-based violence (SGBV) services in selected sites. • Objective 3: Develop and implement a formal handover process for services provided under Objectives 1 and 2 by transitioning management and quality assurance of services in Ninewa to the Department of Health (DOH). EVALUATION PURPOSE AND QUESTIONS This evaluation aims to examine the Ninewa Plains grant’s successes and challenges to inform future programming in this region. The evaluation seeks to learn key lessons from implementation and explore the factors that contributed to or impeded achievement of both intended and unintended outcomes. This evaluation addresses the following Evaluation Questions (EQs): 1. How did project support components (e.g., training, equipment, management support) contribute to changes in the delivery and access of health services at the six PHCCs and Sinjar General Hospital? 2. What key factors contributed to the uptake of health services among vulnerable women (e.g., women facing GBV)? 3. To what extent did the project’s health care training/support contribute to improved quality and availability of MHPSS services? 4. To what extent did project support improve the continuum of care in terms of referrals to and treatment by secondary health facilities? 2 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV 5. What were the successes and challenges of project handover and revitalization to DOH for core PHCC services in terms of staffing, management, record keeping, and finances? METHODOLOGY The evaluation employed primarily secondary data analysis, including qualitative and quantitative data from project documents. The evaluation team also collected primary data from a small sample of key informants through qualitative interviews. Due to the significant travel restrictions imposed in response to the COVID-19 pandemic, all interviews were conducted over Skype. Interviews were conducted in May and June 2020 with four IMC staff and four health facilities staff (Bashiqa, Mawofaqia, Sinjar, and Wana). See Annex 3 for a full list of interviewees. FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1: CHANGES IN SERVICE DELIVERY AND ACCESS CAPACITY BUILDING The grant trained 1,045 individuals in primary health care, GBV services, and mental health care. Analysis of the pre- and post-tests from the GBV, CHW, and rational drug use training reports confirmed that all staff increased their knowledge through the training sessions. The senior medical officers regularly visited and monitored service delivery quality, ensuring that the health care workers adhered to guidelines for standards of care. Health coordinators also conducted random supervisory visits every three months. IMC key informants highlighted that during the supervisory visits, they were clearly able to observe the staff using the knowledge and skills they had developed through training sessions. HUMAN RESOURCES IMC provided incentive support (e.g., stipends) to key DOH staff across all seven health facilities for 123 staff (e.g. doctors, nurses, managers, pharmacists, laboratory, radiology technicians, cleaners, and guards). These incentives helped eliminate many of the staffing gaps identified during the assessment phase and increased the hours of operation across the facilities. However, staff recruitment was challenging. The destruction in Sinjar and Wana led to brain drain, which affected the ability to identify qualified local female clinicians to provide services at the maternity units. However, based on interviews with health staff, after the grant ended many of the specialized doctors and those brought in from the outside by IMC were not still at the facilities. In addition, staff is now back to working less hours than when they received stipends. REHABILITATION IMC rehabilitated the Wana PHCC and maternity unit, the Al-Quba PHCC, and the Sinjar maternity unit within the Sinjar General Hospital and refurbished and equipped safe spaces within the PHCCs for the provision of GBV case management services. The project also reactivated 25 psychosocial units that were not functional prior to the grant. Due to the financial challenges faced by the DOH in the Ninewa Plains, many of the service delivery components would not have been possible without the support provided by the grant, particularly in sites that required rehabilitation to be reactivated. 3 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV PRIMARY HEALTH CARE SERVICES Grant documents indicate a significant increase in primary health care consultations at the six PHCCs and the Sinjar General Hospital. IMC conducted 115,731 primary health care consultations during the project, 90,000 of which were considered new beneficiaries. This number greatly exceeded the project’s target of 64,000 beneficiaries. Although examination of the data does not indicate a significant increase in beneficiaries during each quarter, KII findings highlight the improvement in service type and quality. However, after the project ended, services were extremely basic and did not involve physical examinations and follow-up because the women were being treated by a male nurse, which is culturally not always accepted. Sixty-one percent of primary health consultations were for illnesses with high morbidities. The top five morbidities were (1) acute upper respiratory tract infections (37 percent), (2) genito-urinary tract infections (10 percent), (3) gastro-intestinal diseases (9 percent), (4) acute diarrhea (7 percent), and (5) skin diseases (7 percent). MATERNAL AND CHILD HEALTH/FAMILY PLANNING Under the grant, IMC integrated sexual and reproductive health services into routine primary health consultations in the Al-Mowafaqia, Al-Quba, Al-Qush, Al-Sada, Bashiqa, and Wana PHCCs. IMC also established maternity units in Bashiqa, Sinjar, and Wana, refurbishing the sites and providing medical equipment and supplies and staff training and incentives. During the grant, 20,604 reproductive health consultations; 576 family planning consultations; and 465 deliveries were conducted. The number of reported deliveries rose from 20 per month in October 2018 to 70 per month in May 2019. Services had not been provided at the maternity units before grant implementation; provision of these services accounts for the most significant change in service delivery and is viewed as the project’s largest success by the interviewed project staff. However, IMC faced many challenges achieving this outcome, especially in recruiting medical doctors and midwives, particularly female doctors. Interviewed IMC staff observed that conducting safe deliveries at the supported maternity units reduced maternal mortality rates significantly. At the time of the handover to DOH, IMC staff reported that maternal deaths had been eliminated. Before implementation, most women gave birth at home assisted by midwives. COMMUNITY HEALTH SERVICES Community health services were provided at all locations. A total of 82 CHWs were trained on community health topics, and 68 CHWs, were supported to provide family planning education, referrals, and services. CHWs conducted 21,484 household visits reaching 81,953 beneficiaries. A total of 529 community-based referrals were provided. They also conducted 138 awareness-raising group sessions in the community, reaching 3,286 beneficiaries with messages related to general health issues. Additionally, 125 community-based events were conducted, reaching 3,062 beneficiaries. Community members also reported that the events and sessions helped them become better informed on the health issues affecting their community and on how to adopt healthier lifestyles. The IMC staff reported high turnover among the CHWs, due partly to the transient nature of the population—some remained internally displaced because of the regional crisis. 4 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV KII findings indicate that although the CHWs’ work was very beneficial in informing the community of services available, the workers faced many challenges due to cultural barriers, particularly in women’s ability to access services on their own and in the uptake of GBV and mental health services. Informants also indicated that security issues and weather impeded the CHWs’ work. According to interviewed IMC and facility staff, the awareness CHWs raised among the community contributed significantly to increasing the number of consultations at the facilities as community members became more knowledgeable about the availability of services. Strong linkages between the community health team and the health care facilities also enhanced referrals as community members began referring one another to the health care facilities. PATIENT SATISFACTION IMC staff reported a high level of satisfaction among the targeted beneficiaries with the primary and reproductive health care services they received. Based on a sample of 400 patient exit interviews conducted during the grant’s fourth quarter, April/May 2019, no patients reporting being dissatisfied with any aspect of service delivery. Respondents reported the lowest levels of satisfaction in the availability of pharmaceuticals. CONCLUSIONS (EQ 1) In a one-year timeframe, the IMC grant substantially increased the number of services provided at the target facilities and the number of clients receiving services. The project enhanced, and in some cases established, essential health services in the community. Rehabilitating and reactivating sites by providing essential equipment and supplies and building staff capacity. For example, establishment of the maternity units in Bashiqa, Sinjar, and Wana enabled women who would previously have given birth at home to give birth in an equipped and controlled hospital environment, reducing maternal deaths among beneficiaries. CHWs filled a critical gap in raising community awareness on both community health issues and the availability of services. The enhanced awareness created through these efforts helped community members identify when to seek services. EVALUATION QUESTION 2: UPTAKE OF HEALTH SERVICES (E.G. GBV) IMC provided GBV prevention activities in Al-Qosh, Al-Mowafaqia, Al-Quba, and Wana. Sexual abuse, child marriage, physical abuse, denial of resources, and emotional abuse were discussed at household visits and GBV awareness sessions that reached 1,499 beneficiaries. To enhance case identification and referrals, the GBV services were linked to other health services, particularly reproductive health and MHPSS services. IMC also established linkages and referral pathways with other partners implementing GBV services in the area. IMC trained 367 staff on basic GBV concepts and case management. IMC reported facing significant challenges in recruiting qualified staff to support the GBV component, particularly those displaying attitudinal readiness to work with GBV survivors effectively. Social workers drawn from the community were trained and mentored using the interagency GBV case management guidelines. IMC established quality control through monthly structured case consultation/debriefing sessions, observation visits, and one-on-one coaching. IMC staff also observed improvement in the training participants’ attitudes toward GBV survivors, including a reduction in the tendency to blame GBV survivors and judge the decisions taken by at-risk women. KIIs with IMC staff reported challenges incorporating GBV case management into the PHCCs since the DOH did not have a strong component of social work and case management. The lack of time and DOH weakness eventually affected these services’ sustainability after the project closed, as social workers were not supported by the DOH after the end of the grant, this component was not sustained. 5 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV KEY BARRIERS TO INCREASING UPTAKE OF HEALTH SERVICES Cultural norms and traditional practices hindered women’s ability to access services, particularly GBV and sexual and reproductive health services. Women’s access was restricted by their husbands and family members—during the household visits, women indicated that they needed to access GBV services but that if they attempted to do so, they had to be accompanied by their spouse, their spouse’s parent, or relatives. According to project documents, community mobilizers also confirmed that the stringent traditional practices created a barrier for some GBV survivors who would have otherwise wanted to access GBV services. The vulnerable women’s own lack of trust in the services created an additional barrier to their uptake of services. IMC sensitized the community on the importance of accessing services not only through the CHWS but also through increasing awareness of services among community leaders and engaging the community through celebratory events such as International Women’s Day celebrations. IMC staff report that initial uptake of services was slow, however by the end of the project, IMC reported increased uptake of both MHPSS and GBV services among vulnerable populations. Unfortunately, the enhanced uptake was observed during the last phase of the project, thereby limiting the achievements under this component. CONCLUSIONS (EQ 2) The GBV component addressed a clear community need and increased uptake of GBV services by combining these services with other primary and maternal health services and by embedding them within the PHCCs. Due to GBV’s sensitive nature, significant time was required to garner the community’s trust and effect behavior change to increase access to services. The project’s short duration therefore limited service delivery under this component. This limitation was exacerbated by the challenges in recruiting qualified staff from the community. Building capacity of DOH PHCC staff and of social workers from the community helped support this intervention’s sustainability and facilitated case referral, however social workers are not supported by the DOH, and this component was not sustained after closeout. As cultural norms and community perceptions of GBV and mental health issues are deeply ingrained, however, significant, long-term efforts are required to bring about sustainable behavior change and reduce the barriers women face. These factors, combined with the project’s short duration and the recruitment challenges during startup, limited uptake of services. The lack of a community health strategy within the DOH further limited uptake and sustainability. EVALUATION QUESTION 3: MHPSS SERVICES Prior to IMC’s support, the supported PHCCs did not offer MHPSS services. IMC initiated and expanded psychosocial services and integrated mental health care into primary health care and protection services. IMC’s MHPSS program focused heavily on training to increase knowledge and build capacity of DOH and PHCC facility staff on mental health care. The main challenges and barriers the MHPSS program faced are: 1. Sites for MHPSS services. To increase MHPSS service availability, the project supported 25 psychosocial units. The support included minor rehabilitation, basic equipment and furniture, and staff training. In project locations without psychosocial units (i.e., Wana and Quba), the program faced significant challenges in establishing services due to the lack of private areas to conduct MHPSS services. In these areas, the project had to shift planning to find private locations within the health facility to provide services, which delayed program initiation. 6 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV 2. Human resources. Staffing presented a major challenge in providing MHPSS services primarily because of the displacement of qualified staff and a lack of MHPSS staff (i.e., case managers or psychologists) within the DOH (e.g. Wana and Quba). Further obstacles included difficulty finding qualified male case managers who would work closely with female case managers and difficulty finding female outreach workers. 3. Service delivery. Throughout the project period, MHPSS case management teams provided services to 2,107 individuals in Al-Qosh, Al-Mowafaqia, Al-Quba, and Wana. Top reported stressors were prolonged displacement due to armed conflict (55 percent), financial and economic concerns (25 percent), and family neglect issues (20 percent). The number of patients receiving services is only 47 percent of the project’s targeted 4,500 individuals. IMC project documents reveal that the target was not reached due to late start in service provision, lack of private space for case management, and difficulty hiring qualified staff, as described above. 4. Community outreach. IMC worked closely with community leaders on program implementation and hired outreach workers from the community. This helped to empower communities to support their members with psychosocial and mental health issues and reduce stigma. Although the findings reveal sensitization and awareness support, the project team had difficulty retaining outreach staff, especially as staff returned to their communities after having been displaced. 5. Security. The MHPSS teams encountered security challenges. For example, in Al-Quba, MHPSS services, including home visits, awareness sessions, and case management sessions were suspended for two weeks for security reasons. MHPSS PROGRAM SUSTAINABILITY Despite the successes and the DOH’s willingness to take over managing the PHCCs, the DOH was not prepared to absorb MHPSS services. Project staff also reported that they believed the only MHPSS services that were sustained after the project were identification and referral of cases to hospitals and that these services continued because of health facility staff training and capacity building during the project period. Based on the interviews with two of the facilities that received MHPSS support, they only offer limited MHPSS services due to inadequate funding and lack of specialized staff. CONCLUSIONS (EQ 3) Despite the successes, MHPSS services were not sustained. The IMC project revealed that MHPSS programming requires a commensurate increase in capacity within the health system to attain sustainability. Challenges with staffing, stigma, and locations for service provision added to the instability. This is especially true in countries such as Iraq, which do not have strong MHPSS policies or strategies. More time was needed to fully integrate mental health services into the PHCCs, increase health professionals’ knowledge and service provision, and reduce stigma surrounding mental health care. Iraq has experienced violent conflict, suffering, and social upheaval for many years. MHPSS services will continue to be a high-priority need. It remains to be seen whether the IMC-supported MHPSS services, along with the MHPSS strategy for Iraq, will be sustained, along with the desired outcomes of improved access and service quality and, ultimately, improved patient mental health. 7 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV EVALUATION QUESTION 4: REFERRALS IMC focused on a holistic referral system that ranged from the community to health facility levels (primary and secondary). During the project period, 439 cases were referred to secondary care, with the majority being for emergency cases. One IMC staff reported that most patients returned to the PHCC after being referred and if they did not, IMC had staff follow up with the referral agency. Additional challenges that disrupted the continuum of care were the lack of DOH-dedicated ambulances. One KII reported that IMC could support only four or five ambulance units, which interfered with the ability to refer and the ability of patients to benefit from the referral process. One KII reported that the most significant change in community referrals was CHW training on referrals and follow-ups. They attributed the increase in community referrals to the high level of awareness on health-related issues within the community. CONCLUSIONS (EQ 4) Overall, the referral system’s design appeared to be strong and aligned with the Iraq Conceptual Framework for Referral from Primary to Secondary Level. The project improved care by putting in place protocols and a method of communication within the referral system. The mapping of services in each area supported the referral pathways, which continued to be used as IMC’s support phased out. However, the project was unsuccessful in incorporating the design in all areas, primarily because of lack of DOH ambulances at many of the sites, which impeded the continuum of care as secondary health services were far from IMC service areas. EVALUATION QUESTION 5: PROJECT HANDOVER AND REVITALIZATION TO DOH IMC collaborated and co-implemented with DOH Ninewa and Dohuk from project initiation to facilitate a smooth handover. However, the DOH had difficulty adopting a fast-paced timeline for full handover, especially for the MHPSS and GBV programs, which were not strong before the crisis. Main factors that influenced the handover process were: 1. Partnership and management. To improve PHCC management, IMC built the capacity of the district DOH staff by providing professional development courses for 55 DOH staff. IMC also supported the program managers by developing and disseminating supervisory tools and conducting joint health facility assessment visits. Although IMC coordinated closely with the PHCC managers and district managers during the handover process, it might have been more effective to involve the MOH in the higher-level discussions to expedite decision making (e.g., on staff deployment). 2. Finances. Based on key informants, finances were one of the biggest barriers, especially now as “Iraq is in a sensitive place economically, which impacts revenue and therefore impacts health service delivery.” KIIs with IMC staff also revealed that the DOH did not have the capacity to continue to provide incentives to retain staff in insecure and remote locations, nor did they have the same financing for supplies and consumables. 3. Staffing. Hiring and retaining staff for insecure and remote locations presented and will continue to present a challenge for the DOH due to lack of continued incentives. Further, staff turnover may lead to gaps in training and orientation for new staff or refresher trainings when needed. It is hard to find qualified staff who want to work in certain practice areas (e.g., MHPSS). Even the project had to pay staff more just to work in these areas, an option the DOH does not have. 8 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV SERVICE DELIVERY AFTER END OF GRANT The following presents information on the status of each type of service delivery: • PHCC services. Health services were active and running at project end, however one year later with no incentives some staff were gone and longer working hours were not in place. In Al￾Mowafaqia PHCC, the final monitoring report revealed a decline in consultations from 80–100 a day to fewer than 30 a day after IMC handed over. One key informant reported that the decline was due to staffing. • Maternal units. Of the three maternal units initiated during the project period, only Bashiqa and Sinjar continued after the project ended. The Wana maternity unit could not continue due to lack of staff, including a reproductive health doctor to provide services. • MHPSS services. Some MHPSS services continued after the project’s end, but they were limited mostly due to lack of specialized staff. Lack of ownership of MHPSS interventions within the PHCCs also reduced sustainability. • GBV services. Reports revealed that attitudes toward GBV changed during the project period. This shift was reflected by the DOH requesting to have GBV activities at the health facilities and more sessions on the subject. Nevertheless, based on the facility interviews, GBV services were still only being provided in a limited way at one facility, while two of the other facilities reported that the services were no longer available a year later. STAFF SUSTAINABILITY Financial support, fragile or weak human resource governance, and difficulties in hiring health professionals in insecure and/or remote locations were key challenges in staff retention. In the July 2019 final monitoring report for Al-Mowafaqia PHCC, the manager reported that despite multiple requests to the DOH for staff, after IMC left there were no doctors in the PHCC. Further, the Wana maternity unit closed after the project ended due to lack of staffing. Key informants reported that after IMC left, financial support for CHWs and GBV and MHPSS staff did not exist and that the DOH was “not ready to provide them with any financial support or incentives.” CHANGE IN COMMUNITY ENGAGEMENT LEVEL The DOH system does not provide for CHWs and outreach workers. The level of community engagement diminished by the end of the grant, the awareness of available services at the PHCCs and knowledge gained during the community sessions will likely continue to benefit some community members. However based on interviews with health facility staff they have seen decreases in community members seeking services, especially GBV and MHPSS services, which they related to the lack of current outreach in the area. CONCLUSIONS (EQ 5) IMC not only built capacity but also provided support for planning, management, and monitoring among the various DOH management levels. However, this support’s impact was limited by the DOH and health system’s ability to absorb it, which was hampered mainly by a lack of money and the shortage of qualified staff in insecure and remote locations. These factors constricted services and their expansion, as evidenced by the Wana maternity unit’s closure due to lack of staff; the end of the community outreach, case management, and GBV programs; and the decline in PHCC visits as IMC support ended. Despite consensus 9 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV that IMC support increased services and capacity, both reports and interviews clearly indicate that additional time might have strengthened sustainability and the project handover, particularly the GBV and MHPSS components. RECOMMENDATIONS The evaluation team presents the following programming recommendations: • To deliver sustainable results, projects that support primary health care should be implemented for longer than one year to properly build DOH and facility staff capacity and help build sustainable systems. This is especially important when programming includes reestablishing or initiating services (e.g., GBV/MHPSS). • GBV programming should create community-based comprehensive care centers tailored to women’s and adolescent girls’ needs, combining GBV and MHPSS services with additional services. Community-based interventions should be implemented through women’s centers and safe spaces whenever possible to enhance the level of trust in the services. • GBV programming should facilitate cooperation among the various ministries involved to ensure adoption of GBV international standards of care. This should include involvement at all levels of the government to strengthen associated policies and guidance nationally. • MHPSS programming should include the procurement of psychiatrists and psychotropic medications to improve the mental health outcomes of both men and women. • MHPSS programming in this region should work with national-level stakeholders to influence political buy-in, policy, and leadership on MHPSS to continue to improve mental health outcomes. • Projects that focus on MHPSS and GBV services should work from the outset on gaining local and national political buy-in, budget, and human resources to increase these services’ sustainability and public awareness about them after the project ends. • Health programming in the Ninewa area should work closely with the DOH and other key stakeholders at project initiation to develop an achievable plan to retain staff in remote and insecure locations. USAID should ensure that the facility-level MHPSS and GBV service providers are DOH staff and not supported by the project. This will improve the handover process and service sustainability. • Health programming in the region should support the DOH and other stakeholders to build a local resource pool of experts who can conduct continuing medical education and provide technical assistance to health facilities, especially for MHPSS and GBV services. • Community-based health programming should explore ways to retain CHW staff, especially after project end. Efforts to retain trained CHWs can address the high turnover rates reported and enhance the intervention’s long-term sustainability. • Programming should seek to actively engage and partner with other organizations on the ground to support a more comprehensive approach to women’s and adolescent girls’ empowerment and well-being. These components could be included in community-based women’s health centers through a complementary package of health and continuing education services. 10 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV INTRODUCTION Under the Middle East and North Africa Monitoring, Evaluation, and Learning Services Activity (MENA MELS), Management Systems International (MSI) conducted an evaluation of the subgrant Improving Health Services and Outcomes in the Ninewa Plains, Iraq. The subgrant was implemented by International Medical Corps (IMC) under the Advancing Partners and Communities (APC) Grant implemented by JSI Research and Training Institute, Inc. (JSI) in collaboration with Family Health International (FHI 360). This report presents the project’s background; evaluation questions (EQs) and methods; and key findings, conclusions, and recommendations for future programming. GRANT BACKGROUND Implementation took place between July 1, 2018, and July 31, 2019, in the Ninewa Plains region, which includes Al-Quba, Sinjar, Telafar City, and Wana and has a target population of 64,000 residents. The grant focused on six primary health care centers (PHCCs)—Al-Mowafaqia, Al-Quba, Al-Qush, Al-Sada, Bashiqa, and Wana—and one secondary facility, Sinjar General Hospital (maternity unit) (Figure 1)1. FIGURE 1: MAP OF APC FACILITIES The key areas of support included primary health care, reproductive health care, mental health and psychosocial support (MHPSS) services, referrals and linkages, gender-based violence (GBV) prevention and response, and community health care. See Annex 2 for IMC Final Grant Outcomes Dashboard of Results. 1 IMC selected supported facilities through a needs assessment as well as those that served vulnerable religious and ethnic minority populations. 11 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV The grant’s main objectives and supporting activities were as follows: • Objective 1: Improve delivery of the essential package of primary health care services to minority populations in the Ninewa Plains by strengthen the capacity of PHCC; facilitate links to advanced levels of care (secondary care site); engage community health workers (CHW); support health facility managers to strengthen health facility management. – Activity 1.1: Support health facility staff to ensure adequate human resource (HR) coverage for immediate and longer-term operations. – Activity 1.2: Procure equipment and medical commodities needed to improve health facility functionality and quality of health services. – Activity 1.3: Support provision of health services to ensure targeted communities have consistent access to high-quality health services. – Activity 1.4: In collaboration with all partners, adapt or create a care plan that links and refers clients from project-supported PHCCs to secondary and tertiary care services. • Objective 2: Strengthen and integrate MHPSS services, post-traumatic treatment for children, and sexual and gender-based violence (SGBV) services in selected sites. – Activity 2.1: Identify training and support needs for PHCCs to implement GBV and mental health services. – Activity 2.2: Procure equipment and/or commodities needed to improve provision of SGBV and mental health services. – Activity 2.3: Provide comprehensive MHPSS services to women, men, girls, and boys to mitigate effects of conflict and displacement, linking them with basic services and psychosocial units for more specialized support. – Activity 2.4: Provide training on scalable, evidence-based psychological interventions adapted to the local context, such as group interpersonal therapy or problem management plus. – Activity 2.5: Organize a national conference on mental health to collectively identify next steps to address protection and MHPSS concerns, in partnership with the Department of Health (DOH) and the World Health Organization (WHO). • Objective 3: Develop and implement a formal handover process for services provided under Objectives 1 and 2 by transitioning management and quality assurance of services in Ninewa to the DOH. – Activity 3.1: In collaboration with the DOH, establish a time-bound handover and revitalization plan for the DOH to take over management, quality assurance, and services. This plan will cover the DOH’s activities and relative responsibilities. – Activity 3.2: Monitor transition together with DOH officials, including assessing the DOH’s readiness to take on activities. 12 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV EVALUATION PURPOSE AND QUESTIONS This evaluation aims to examine the Ninewa Plains grant’s successes and challenges to inform future programming in this region. The evaluation seeks to learn key lessons from implementation and explore the factors that contributed to or impeded achievement of both intended and unintended outcomes. The evaluation also seeks to understand what, if any, additional measures are needed to secure better health outcomes for the target population, building on the gains achieved through this activity. EVALUATION QUESTIONS This evaluation addresses the following EQs: 1. How did project support components (e.g., training, equipment, management support) contribute to changes in the delivery and access of health services at the six PHCCs and Sinjar General Hospital? a. Were there key elements of support not provided by the project that could have a greater impact on delivery and access to health services? (e.g., comparison of PHCCs and WHO mobile clinics) b. Were there any unintended outcomes due to project interventions? 2. What key factors contributed to the uptake of health services among vulnerable women (e.g., women facing GBV)? a. What key barriers did the project help overcome to increase uptake of health services? 3. To what extent did the project’s health care training/support contribute to improved quality and availability of MHPSS services? a. What were challenges and barriers to delivery of these services? b. Can the MHPSS be sustained as integrated PHCC services going forward? 4. To what extent did project support improve the continuum of care in terms of referrals to and treatment by secondary health facilities? 5. What were the successes and challenges of project handover and revitalization to DOH for core PHCC services in terms of staffing, management, record keeping, and finances? a. Since project end, what project-supported health services are still being delivered in the targeted health facilities? b. Since project end, what facility staff are still in place? c. Since project end, have there been any changes in the level of community engagement (e.g., CHWs)? Under each of these questions, the evaluation considered the key project components in relation to the grant’s successes and challenges. For EQ 1, the evaluation sought to learn more about the following support components: 13 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV • Training. Rational drug use, introduction training, advanced trauma life support and adult basic life support, medical equipment training, infection prevention control (IPC), clinical management of rape, neonatal resuscitation • Human resources. Incentives to health staff • Community health. CHWs • MHPSS. Case management, reactivated psychosocial units, training • GBV. Case management, training, space, prevention activities/events • Supplies. Equipment, medicines, medical supplies In addition, the evaluation examined the health services the grant supported. The EQs address the following key areas of support: • Primary health. Outpatient consultations, treatment of communicable and noncommunicable diseases, emergency referral, dispensation of essential medicines, disease surveillance through early warning alert and response (EWARN) • Maternal and child health/family planning. Antenatal care, postnatal care, family planning, and sexually transmitted infections (STIs) • MHPSS and GBV services. Case management METHODOLOGY The evaluation employed primarily secondary data analysis, including qualitative and quantitative data from project documents. The evaluation team also collected primary data from a small sample of key informants through qualitative interviews. The team collected these data to fill gaps and add more specific implementation context into the analysis. The evaluation design included in-person key informant interviews (KIIs) at relevant PHCCs and related sites in the Ninewa Plains region. However, because of the significant travel restrictions imposed in response to the COVID-19 pandemic, the evaluation team adjusted the approach to focus on KIIs with project staff and a few health facilities. In May 2019, the team conducted KIIs with four IMC staff who are still located in Erbil/Dohuk (the deputy country director, field site/program coordinator, grants coordinator, and program coordinator). In addition to these interviews, the evaluation team asked KII respondents for recommendations of other possible remote key informants that could be interviewed to collect additional perspectives on grant outcomes. Interviewees suggested several facility-based staff. After receiving USAID approval, in June 2020, the evaluation team conducted interviews with key health staff at four of the targeted facilities (Bashiqa, Mawofaqia, Sinjar, and Wana). See Annex 3 for a full list of interviewees. The evaluation team conducted the KIIs over Skype. Interviews lasted between 60 and 90 minutes. An interview guide (Annex 4) was used to collect information consistently. The team recorded all interviews and drafted detailed notes for interview analysis. The evaluation team examined qualitative data from KIIs for patterns. 14 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV The evaluation team also reviewed all the available project data and documents to better understand changes between the beginning and end of the grant implementation period. The team sought to identify and extract good practices, lessons learned, challenges/barriers, recommendations, and outcomes relevant to each EQ. LIMITATIONS Due to COVID-19 travel restrictions, the team was not able to visit any of the facilities to collect primary data or conduct any observations. The team was only able to conduct a limited number of interviews due to facilities limited availability while responding to COVID-19. However the team collected and reviewed all available grant documents and data as well as interviewed all grant staff that is currently still employed with IMC. To supplement the IMC staff’s perspectives, interviews were also conducted with four of the health staff from the supported facilities. Another limitation was due to that fact the grant had ended over a year ago and some of the grant data was hard to interpret. The team did ask IMC staff follow up questions to any data issues, however the available staff were not always able to provide the needed detail answers. FINDINGS AND CONCLUSIONS EVALUATION QUESTION 1 How did project support components (e.g., training, equipment, management support) contribute to changes in the delivery and access of health services at the six PHCCs and Sinjar General Hospital? The team examined changes in each key area of the grant from start of project implementation to the end, including capacity building, human resources, rehabilitation and supplies, primary health services, patient satisfaction, community health services, maternal and child health, and family planning. These findings are supplemented with current information from key informants. In July 2018, prior to project implementation, IMC conducted assessments of 13 health care facilities in the Ninewa Plains to identify those most in need of support and rehabilitation. Many of the health facilities in the region had been destroyed or severely damaged during the 2014 invasion by the Islamic State group and the subsequent liberation efforts in 2017. IMC selected facilities to receive support based on the assessment findings and selection criteria included in the grant, such as servicing vulnerable religious and ethnic minority populations. Based on the assessment results, activities were supported in the Al-Mowafaqia, Al-Quba, Al-Qush, Al￾Sada, Bashiqa, and Wana PHCCs; the Bashiqa, Sinjar, and Wana maternity units; and the Sinjar emergency room. The evaluation team used the assessment reports to establish a baseline on the facilities’ status before the interventions and compared this information with project final reports and information from KIIs. Assessment findings for all facilities highlighted limited health care service delivery prior to implementation. Common findings across all facilities included lack of suitable infrastructure; limited equipment and supplies, including laboratory equipment and pharmaceuticals; lack of qualified staff to deliver health services; and nonfunctioning ambulances, limiting the transport of patients referred to higher-level facilities. 15 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV For example, the hospital in Sinjar was nearly destroyed during the liberation efforts, with only a semi￾functional emergency room remaining. The maternity unit was destroyed, forcing most deliveries in the region to take place in homes with midwife support. Complicated deliveries were referred to a hospital more than an hour away that lacked a functional ambulance, thus necessitating travel in a private vehicle through multiple security checkpoints. Pharmaceutical stocks were not sufficient to support the catchment population, and laboratory facilities were limited. Sinjar experienced shortages, including a complete lack of obstetricians, gynecologists, and midwives to assist with deliveries at the hospital. Many of the other target facilities suffered from similar challenges and damage. KIIs indicated that one of the grant’s major successes was the package of services delivered and the comprehensiveness of the support components provided. The support components enabled the DOH to resume service delivery in locations that had sustained destruction and lacked sufficient financial resources. The three maternity units, which had not been providing services before the grant implementation, experienced the most significant change in service delivery. Sinjar emergency room services also would not have been possible without the grant’s support. Hence, all the changes in service delivery at these sites can be attributed to the IMC intervention. KIIs reported and project documents indicated that the grant exceeded planned targets for service delivery despite the project’s limited duration. CAPACITY BUILDING KIIs with the IMC staff confirmed that trainings were perceived as one of the grant’s most important support components. Staff capacity building improved service delivery quality during the grant. Through the grant, 1,0452 individuals received training in primary health care, GBV services, and mental health care. The grant trained 301 health care providers (e.g., doctors, nurses, CHWs, midwives, and traditional birth attendants), of which 82 were CHWs and 219 health staff. The project also trained 367 service providers in identifying, referring, and caring for GBV survivors and 176 IMC and DOH staff on GBV case management and support. In addition, 191 people were trained in psychosocial support and 56 trained in scalable, evidence-based psychological interventions, and 25 national-level DOH staff were trained in MHPSS. Table 1 provides a breakdown of individuals trained at the PHCCs and maternity units by subject area.3 (See specifics on MHPSS training under EQ 3). TABLE 1: HEALTH STAFF TRAINED AT PRIMARY HEALTH CARE CENTERS AND MATERNITY UNITS Training Type Males Females Total Introductory Training Package 28 17 45 Rational Drug Use 11 4 15 Advanced Trauma Life Support and Basic Life Support 23 3 26 Medical Equipment Training 3 5 8 Infection Prevention Control 8 15 23 Clinical Management of Rape 4 15 19 Neonatal Resuscitation 16 141 157 2 Improving Health Services and Outcomes in the Ninewa Plains, Iraq, updated report: July 2018 – July 2019, two-page infographic, https://www.advancingpartners.org/sites/default/files/sites/default/files/resources/iraq_dashboard_web_0.pdf. 3 International Medical Corps. “FINAL REPORT From 1st of June 2018 through 31st of July 2019,” September 2019, 1–43. 16 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV Analysis of the pre- and post-tests from the GBV, CHW, and rational drug use training reports confirmed that all staff increased their knowledge through the training sessions. IMC project staff also reported observing improved outcomes and enhanced service quality following the training sessions. IMC developed checklists for the various types of services delivered to facilitate the project’s supportive supervision component. The senior medical officers regularly visited and monitored service delivery quality using these checklists, ensuring that the health care workers adhered to guidelines for standards of care. When supervisors identified areas of improvement during visits, interventions such as training and mentorship were implemented to address the gaps. Health coordinators also conducted random supervisory visits every three months. IMC key informants highlighted that during the supervisory visits, they were clearly able to observe the staff using the knowledge and skills they had developed through training sessions. During the implementation phase, IMC observed limited capacities among nurses and midwives providing services within the facilities. Nurses working in the emergency room lacked sufficient knowledge on triage, and the midwives faced challenges conducting deliveries and neonatal resuscitation. As part of the project’s capacity-building efforts, nurses received training on how to effectively triage patients, and the midwives received training in basic emergency obstetric and neonatal care, including neonatal resuscitation, in accordance with national clinical guidelines. Training components included the administration of antibiotics, uterotonic and anticonvulsant drugs, manual removal of the placenta, and assisted vaginal delivery. KIIs indicated observations of related skills uptake during subsequent supervisory visits. HUMAN RESOURCES IMC provided incentive support (e.g., stipends) to key DOH staff across all seven health facilities for 123 staff. According to the project’s final report, these staff included 14 part-time doctors, three full-time doctors, eight part-time nurses, six full-time nurses, one full-time PHCC manager, four part-time pharmacist assistants, two full-time pharmacist assistants, two full-time laboratory assistants, two full￾time radiology technicians, eight full-time cleaners, seven full-time registrars, eight full-time guards, and one full-time ambulance driver. For reproductive health departments, the key personnel receiving incentive support included seven reproductive health doctors, 18 nurses/midwives, six pharmacy assistants, nine guards, nine cleaners, and six ambulance drivers, in addition to one PHCC manager and one hospital manager. Findings from the initial facility assessment showed significant staffing shortages. IMC worked with Ninewa DOH to identify the human resources required within each of the facilities to support key grant activities. The Ninewa DOH provided additional staff to support implementation. All facility staff supporting grant activities that were beyond their regular DOH duties received monetary incentives from IMC. These incentives helped eliminate many of the staffing gaps identified during the assessment phase and increased the hours of operation across the facilities. Staff recruitment was challenging. Whenever possible, staff were selected from within the community to enhance community acceptance. However, the destruction in Sinjar and Wana led to brain drain, which affected the ability to identify qualified local female clinicians to provide services at the maternity units. After delays arising from the recruitment challenges, IMC increased the monetary incentives for the female doctors in those facilities and succeeded in recruiting female physicians for maternal and reproductive health services. The availability of female physicians increased the demand for services at the Wana PHCC. IMC hired two additional part-time physicians in February 2019 to meet the growing demand. In addition to the incentives, facility-based staff received other types of IMC support, including regular supervisory visits, spot checks, team meetings, and quality of care visits by medical officers and other technical staff to ensure staff adherence to national guidelines and quality standards for service delivery. Continuing medical education sessions were conducted weekly for relevant staff. 17 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV Based on the interviews with the health facilities staff, incentives were not continued after the end of the grant. Health staff reported that incentives helped motivate staff and extend the hours of the facilities. After the grant ended, one facility report that most staff remained, however the other three facilities said that they lost specialized doctors and other key staff. It was also reported that staff now work less hours than when receiving the incentives through the grant. REHABILITATION AND SUPPLIES IMC rehabilitated the Wana PHCC and maternity unit, the Al-Quba PHCC, and the Sinjar maternity unit within the Sinjar General Hospital and refurbished and equipped safe spaces within the PHCCs for the provision of GBV case management services. The project also reactivated 25 psychosocial units that were not functional prior to the grant. Al-Quba PHCC initially lacked adequate space for storing pharmaceuticals. To address this need, IMC established a new medical storage space as part of the facility rehabilitation process, ensuring pharmaceutical storage in line with national, WHO, and IMC standards. IMC provided additional assistance to the Bashiqa PHCC, which had suffered major damage during the liberation efforts, as well as storage space for nonmedical consumables. Due to the financial challenges faced by the DOH in the Ninewa Plains, many of the service delivery components would not have been possible without the support provided by the grant, particularly in sites that required rehabilitation to be reactivated. The grant provided medical equipment such as autoclaves, electrical sucker devices for adults and children, and electrical nebulizers. During the implementation phase, feedback from the PHCCs indicated that staff were not skilled in the medical devices’ proper use. To overcome this issue, IMC trained staff members at all the facilities on proper use and maintenance of the equipment. PHCCs and maternity units received pharmaceuticals and other medical consumables based on population needs identified through previous morbidity data and drug consumption records. The WHO supported IMC initially in providing pharmaceuticals to fill supply gaps until the items procured under the grant were delivered. The project reported no stockouts of essential medications during the grant. PHCCs and maternity units experienced stockouts for contraceptive commodities from the United Nations Population Fund (UNFPA) due to procurement restrictions on these items. However, once the UNFPA supply chain was restored, reproductive health kits for the supported maternity units and reproductive health clinics were provided, enhancing the quality of family planning services. The grant equipped facilities with information technology (IT) equipment, including desktop computers, to ensure there could be an inventory system to track pharmaceutical stocks. IMC trained staff on rational drug use and pharmaceutical chain management to ensure the effective use of the provided commodities. Staff were also familiarized with the Iraqi drug guidelines, which were posted in all the consultation rooms for quick reference to guide doctors with prescriptions, thereby eliminating nonessential disbursement of medications. The grant provided information, education, and communication (IEC) materials to support community￾based awareness-raising efforts. Fuel for the generators that provided electricity at the sites was also provided on a cost-sharing basis, in addition to furniture, hygiene materials, and other necessary supplies. At the DOH’s request, the grant provided additional medical and nonmedical supplies to three of the PHCCs for various community-based activities, including a school health program. 18 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV PRIMARY HEALTH CARE SERVICES Grant documents indicate a significant increase in primary health care consultations at the six PHCCs and the Sinjar General Hospital. The services supported included outpatient consultations for adults and children, treatment of communicable and noncommunicable diseases, emergency and cold case referrals, dispensation of essential medicines, disease surveillance through EWARN, and health education. IMC supported secondary health care services in the Sinjar emergency department, including curative consultations, emergency resuscitation services, and minor surgeries using local anesthesia. IMC conducted 115,731 primary health care consultations during the project, 90,000 of which were considered new beneficiaries (Figure 2). This number greatly exceeded the project’s target of 64,000 beneficiaries. Reported data indicate that 43 percent of the population (based on a combined catchment population of 211,000) benefited from primary care services provided during the grant period. IMC data indicated that the number of beneficiaries served rose from 270 in August 2018 to more than 17,000 in May 2019. Figure 2 shows a breakdown of the number of consultations conducted during each quarter of the project. Note that first quarter data include only the month of September, given the project’s startup date. Although examination of the data does not indicate a significant increase in beneficiaries during each quarter, KII findings highlight the improvement in service type and quality. All key informants reported that although the number of consultations may not have substantially risen, service quality significantly improved. For example, during the project period, women in Wana were treated and received follow-up care by a specialized female physician. However, after the project ended, services were extremely basic and did not involve physical examinations and follow-up because the women were being treated by a male nurse, who was not qualified to offer the same level of services. Cultural factors inhibit women from being examined by male providers. Also, interview findings indicated that before the project, women needed to travel much farther to receive services, so they reported a significant increase in beneficiaries compared with the period before project implementation. In addition, challenges in starting up CHWs’ work may have contributed to a slower uptake of services, as many facilities relied on the CHWs’ outreach activities to inform the community about availability of services. FIGURE 2: PRIMARY HEALTH CARE CONSULTATIONS PER QUARTER 3,615 9,662 20,209 19,729 3,850 12,893 21,964 22,634 7,465 22,555 42,173 42,363 Q1 Q2 Q3 Q4 Male Female Total 19 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV Sixty-one percent of primary health consultations were for illnesses with high morbidities. The top five morbidities were (1) acute upper respiratory tract infections (37 percent), (2) genito-urinary tract infections (10 percent), (3) gastro-intestinal diseases (9 percent), (4) acute diarrhea (7 percent), and (5) skin diseases (7 percent).4 MATERNAL AND CHILD HEALTH/FAMILY PLANNING Under the grant, IMC integrated sexual and reproductive health services into routine primary health consultations in the Al-Mowafaqia, Al-Quba, Al-Qush, Al-Sada, Bashiqa, and Wana PHCCs. Services included antenatal and postnatal care, family planning, and syndromic management of STIs. IMC also established maternity units in Bashiqa, Sinjar, and Wana, refurbishing the sites and providing medical equipment and supplies and staff training and incentives. At these facilities, beneficiaries received 24/7 Basic Emergency Obstetric and Neonatal Care services, postnatal care, family planning counseling, and referrals for emergency obstetric cases. According to the final report, 20,604 reproductive health consultations; 576 family planning consultations, including intrauterine contraceptive device insertion; and 465 deliveries were conducted during the implementation period. In general, maternal and reproductive health consultations increased during the project period (Figure 3).5 The number of reported deliveries rose from 20 per month in October 2018 to 70 per month in May 2019. IMC conducted regular supervisory visits to ensure adherence to quality of service standards, and project staff observed significant improvement in service quality throughout the project. FIGURE 3: REPRODUCTIVE HEALTH CONSULTATIONS PER QUARTER Source: IMC quarterly reports ANC= antenatal care; FP= family planning; PNC= postnatal care; RH= reproductive health Note: When aggregated, the totals differ from those given in the final report. The final report lists 4,780 postnatal care visits and 8,544 antenatal care visits, whereas the totals from all four quarterly reports are 4,717 and 6,241, respectively. 4 International Medical Corps. “FINAL REPORT From 1st of June 2018 through 31st of July 2019,” September 2019, 1–43. 5 Figure 2 data was taken from IMC quarterly reports. When aggregated it differs from total numbers reported in final report. Final report listed 4,780 PNC visits, 8,544 ANC, when numbers from all four quarterly reports the total are 4,717 PNC visits and 6,241 ANC. 485 1,950 2,483 1,323 230 1,369 1,700 1,274 1,418 5,265 6,715 8,670 Q1 Q2 Q3 Q4 ANC PNC RH/FP 20 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV Services had not been provided at the maternity units before grant implementation; provision of these services accounts for the most significant change in service delivery and is viewed as the project’s largest success by the interviewed project staff. However, IMC faced many challenges achieving this outcome, especially in recruiting medical doctors and midwives, particularly female doctors. The project should recruit female doctors from within the community whenever possible to garner community trust. Although some of the PHCCs were providing reproductive health services before the grant, not all the facilities were staffed by doctors, which resulted in nurses providing services. This issue was overcome during the grant. Interviewed IMC staff observed that conducting safe deliveries at the supported maternity units reduced maternal mortality rates significantly. At the time of the handover to DOH, IMC staff reported that maternal deaths had been eliminated. Before implementation, most women gave birth at home assisted by midwives. Complicated cases required personal transport to larger cities, which involved traveling long distances and passing through multiple security checkpoints. The support IMC provided to reactivate the ambulances, which operated 24 hours a day, reduced this challenge. Another success reported under the grant was the integration of family planning and other reproductive health consultation services in the maternity units, such as intrauterine contraceptive device insertion and removal. This enabled women from the community to ensure safe spacing between pregnancies. In addition, awareness-raising activities were conducted among men to encourage them to accompany their wives to the family planning sessions. These efforts enabled couples to make more informed decisions on their reproductive health, thereby further supporting enhanced health outcomes among women and children. COMMUNITY HEALTH SERVICES Community health services were provided at all locations. A total of 82 CHWs (36 males and 46 females) were trained on community health topics, and 68 CHWs, including seven team leaders, were supported under the grant to provide family planning education, referrals, and services. All CHWs were hired from the local community to ensure community acceptance and to support the intervention’s long-term sustainability. However, the IMC staff reported high turnover among the CHWs, due partly to the transient nature of the population—some remained internally displaced because of the regional crisis. To ensure gender mainstreaming, an equal number of male and female CHWs were hired under the project. The community health officer provided daily supervision to ensure effective community health messaging and referrals. “Previously, maternal deaths were high. When we applied services until handover there were no maternal deaths. Complications decreased due to the availability of antenatal care and postnatal care services. We had trained staff and provided reproductive health medical doctors, where previously they only had midwives. For MCH services we started from zero.” – IMC senior medical officer An IMC doctor conducting an examination on a newborn baby (IMC) 21 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV According to the final report, CHWs conducted 21,4846 household visits reaching 81,953 beneficiaries (Figure 4). CHWs referred beneficiaries who needed medical care to the supported PHCCs. A total of 529 community-based referrals were provided, of which 478 were to the PHCCs, 42 to MHPSS, and 9 to other nongovernmental organizations (NGOs). The household visits provided 1,250 pregnant women with education on antenatal care, breastfeeding, and postnatal care. FIGURE 4: NEW BENEFICIARIES REACHED THROUGH HOUSEHOLD VISITS BY CHWS KII findings indicate that although the CHWs’ work was very beneficial in informing the community of services available, the workers faced many challenges due to cultural barriers, particularly in women’s ability to access services on their own and in the uptake of GBV and mental health services. Informants also indicated that security issues and weather impeded the CHWs’ work. According to the final report, the community health team also conducted 138 awareness-raising group sessions in the community, reaching 3,286 beneficiaries (1,420 males and 1,866 females) with messages related to general health issues (e.g., nutrition, vaccinations); chronic and communicable diseases; water, sanitation, and hygiene; and reproductive health (including antenatal care, family planning, and healthy timing and spacing between pregnancies). Additionally, 125 community-based events were conducted, reaching 3,062 beneficiaries (819 males and 2,243 females). Specific event topics included teenage health, women’s health, breastfeeding, diabetes, sports, and school events. The CHWs distributed 86,772 information, education, and communication (IEC) materials during the household visits and community-based events. 6 This number is taken from the final report. It is inconsistent with other reported numbers: “Reached 91,329 people through household visits with health and hygiene education through community health workers” (final PowerPoint) and “2,062 household visits” (final infographic two-pager). 4,077 16,294 11,453 6,486 38,310 4,106 18,088 13,546 7,903 43,643 8,183 34,382 24,999 14,389 81,953 Q1 Q2 Q3 Q4 Total Males Females Total Community health worker conducting a household visit (IMC) 22 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV Project documents note that the surrounding community shared positive feedback on the various activities, indicating that the events were informative and enabled community members to receive health education in a friendly and supportive environment. Community members also reported that the events and sessions helped them become better informed on the health issues affecting their community and on how to adopt healthier lifestyles. According to interviewed IMC staff, the awareness CHWs raised among the community contributed significantly to increasing the number of consultations at the facilities as community members became more knowledgeable about the availability of services. Strong linkages between the community health team and the health care facilities also enhanced referrals as community members began referring one another to the health care facilities. Their enhanced awareness also contributed to a decline in complications related to communicable diseases (such as scabies and chickenpox) and an increase in the number of women accessing services. PATIENT SATISFACTION IMC staff reported a high level of satisfaction among the targeted beneficiaries with the primary and reproductive health care services they received. This finding is confirmed by those from patient exit interviews conducted during the grant’s fourth quarter, April/May 2019. A sample of 400 patients (or guardians of patients under 18) were interviewed after receiving a health service. Of those interviewed, 318 had received primary health services and 82 had received reproductive health services. Respondents were asked to rate whether they were satisfied, neither satisfied nor dissatisfied, or not satisfied with the following issues: physical environment, overall service experience, time and attention paid to patients during the consultation with the doctor, access to the services, information and communication between patients and doctors, and treatment with respect and dignity by staff. IMC-trained enumerators7 carried out the surveys. Results indicate high levels of satisfaction among the beneficiaries, with no patients reporting being dissatisfied with any aspect of service delivery. Figure 5 presents the reported levels of satisfaction on 10 elements of service delivery. Respondents reported the lowest levels of satisfaction in the availability of pharmaceuticals. According to interviews with IMC staff, there was a lack of psychotropic medications for mental health issues, which beneficiaries with serious mental health issues commonly needed. 7 Although patient exit interviews documented high levels of satisfaction with primary and reproductive health services, methods used to conduct the exit interviews involved convenience-based sampling, which could reduce the likelihood of including more challenging and complicated cases within the sample. 23 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV FIGURE 5: RESULTS OF PATIENT SATISFACTION SURVEY GAPS IN PROJECT SUPPORT (EQ 1A) Despite these successes, project staff reported that the grant did not provide all the necessary support needed for facilities to be fully functional. All the PHCCs and hospitals included in the intervention requested laboratory support, including equipment and the necessary reagents, coupled with corresponding training for staff, which the grant did not provide. Furthermore, facility staff requested support to advance secondary care because of the significant shortages of equipment and specialized physicians. For example, the Sinjar hospital reported a lack of equipment and medical professionals to conduct even minor surgeries. Due to the lack of secondary care services at the implementation sites, patients requiring more specialized care needed to go to the larger cities, which poses a challenge for many individuals, particularly women. Staff reported a strong need to support the DOH in decentralizing specialized services, enabling them to be offered outside the larger cities and enhancing access for more rural communities. Project staff reported that certain community members felt that activities targeting women would have been more effective if the service sites had been based within the community at women’s centers and other safe spaces allocated for women, rather than embedding these services within the PHCCs. However, other women preferred accessing GBV and mental health services within the PHCCs as part of maternal and child health services. GBV and mental health services are often stigmatized among more traditional cultures such as those in the Ninewa Plains. Seeking care within the PHCCs enabled women to access services under the guise of receiving other, less stigmatizing care. 9% 18% 22% 17% 19% 21% 15% 24% 20% 50% 91% 82% 78% 83% 81% 79% 85% 76% 80% 50% Ease of access Operational hours Cleanliness and orderliness Waiting area comfort and capacity Staff conduct Waiting time Sufficiency of explanation by doctor Visibility of health education materials Privacy and confidentiality Availability of pharmaceuticals Not Satisfied Neutral Satisfied 24 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV As many individuals in the catchment population are returning internally displaced persons (IDPs) and much of the Ninewa Plains region is still recovering from the conflict, communities there face challenges related to livelihoods, housing, and education, further contributing to their psychological stressors. These factors disproportionately impact women in a negative way. During the interviews, project staff highlighted the need to empower women from the beneficiary populations to reduce the culturally ingrained power dynamics that render women dependent on their male counterparts and less likely to adhere to their own health choices. Suggestions included providing financial management skills and vocational training in skills such as IT, tailoring, knitting, and baking. Interviewees suggested activities could be offered through creating a women’s center that provided a comprehensive set of services. Women could be assisted to set up small-scale businesses to enhance their financial independence. Additional suggestions by the project staff included enhanced programming for adolescent girls, including outreach and life skills training, to empower them to make informed decisions on their own health and well-being. UNINTENDED OUTCOMES (EQ 1B) According to the KIIs and project documents, there were no reported unintended outcomes from the grant activities. However, based on review of the project documents and KII discussions, the lack of CHW training on Do No Harm considerations related to GBV and mental health may have affected GBV service quality. Many members of the catchment population had been directly impacted and internally displaced during the conflict caused by Islamic State group militants. Media reports document significant levels of SGBV against women and girls in Northern Iraq at the hands of Islamic State group militants, particularly among Yazidi women.8 Traditional and cultural factors contribute to stigmatizing beliefs toward mental health issues and barriers to open discussion of GBV. For instance, according to project documents, GBV survivors within the catchment population reported that they needed GBV services but could not access them due to cultural norms that require them to be accompanied by a spouse or relative. Further, improper facilitation of discussions on GBV when victims seek treatment can have an unintended triggering and traumatizing effect on the victims. According to KIIs and project reports, IMC trained 19 staff members on clinical management of rape, but they did not adopt the protocol for the clinical management of rape in health facilities or implement this activity due to challenges in gaining government approval of the protocols. Under Iraqi law, health care facilities are legally obligated to report rape cases to police regardless of the victim’s preference, which goes against GBV standards of care and raises many ethical concerns on privacy and confidentiality. Although the project encouraged women to seek services, women were afraid to come to the facility for these services knowing that staff would have to report any rape cases to the police. Furthermore, project staff reported facing difficulties in incorporating GBV activities within the PHCCs as PHCC staff did not view GBV as a priority area. 8 D. Barnett, “Women Who Are Captured by Isis and Kept as Slaves Endure More Than Just Sexual Violence” The Independent, November 2016. Retrieved from https://www.independent.co.uk/news/world/middle-east/isis-sex-slaves-lamiya-aji-bashar-nadia￾murad-sinjar-yazidi-genocide-sexual-violence-rape-sakharov-a7445151.html 25 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV CONCLUSIONS (EQ 1) In a one-year timeframe, the IMC grant substantially increased the number of services provided at the target facilities and the number of clients receiving services. The project enhanced, and in some cases established, essential health services in the community. Rehabilitating and reactivating sites by providing essential equipment and supplies and building staff capacity—reflecting the project’s comprehensive approach—yielded significant and sustainable improvement. For example, establishment of the maternity units in Bashiqa, Sinjar, and Wana enabled women who would previously have given birth at home under the supervision of a midwife to give birth in an equipped and controlled hospital environment, significantly reducing maternal deaths among the targeted beneficiaries. Given that the DOH does not have a community health strategy, the project’s support to CHWs filled a critical gap in raising community awareness on both community health issues and the availability of services. The enhanced awareness created through these efforts helped community members identify when they should seek services, thereby improving their health outcomes, as seen in the reduction of complications related to communicable diseases. Recruitment of staff and outreach workers from the community reduced cultural and ideological opposition to health services as these staff spoke the same dialect and could relate to communities’ cultural practices, thereby facilitating open discussions with community members. GAPS IN SERVICES PROVIDED Laboratory capabilities in support of primary health care and maternal and child health were lacking, affecting diagnosis and treatment of disease, particularly respiratory illnesses and STIs. Filling this gap would add significant benefit to the activities in the Ninewa Plains, where 37 percent of all primary health consultations were attributed to acute upper respiratory tract infections. Enhanced diagnostic techniques would improve the management of pharmaceuticals stocks by reducing the likelihood of disbursing ineffective treatments due to misdiagnosis. Despite the validity of the suggestions put forth on adding components related to empowering women and adolescent girls and the clear linkages between women’s empowerment and improved health outcomes, it may not be feasible to tackle such a wide range of issues under one intervention. But an increase in active engagement and partnering with other organizations on the ground could support a more comprehensive approach to this issue, with the potential to include these components in community-based women’s health centers as a complementary package of health and continuing education services. UNINTENDED OUTCOMES The humanitarian crisis in the Ninewa Plains caused excessive amounts of psychological stress among the conflict-affected population. Psychological stressors are a known contributor to higher levels of SGBV and mental health disorders. The inability of some women to access support services, particularly those related to GBV and mental health, coupled with the legal obligations to report rape and the limited awareness of Do No Harm considerations among the project staff, can be more detrimental to the victims than not having these discussions. The high levels of sexual violence at the hands of the Islamic State group, particularly against Yazidi women, indicate a clear need to enhance the availability of victim support services. Furthermore, given the extensive trauma endured during the occupation, it would not be surprising to see significant, continuing levels of physical and psychological abuse among these communities. 26 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV EVALUATION QUESTION 2 What key factors contributed to the uptake of health services among vulnerable women (e.g., women facing GBV)? IMC provided GBV prevention activities in Al-Qosh, Al-Mowafaqia, Al-Quba, and Wana. IMC project staff reported that GBV is prevalent among the communities served. Sexual abuse, child marriage, physical abuse, denial of resources, and emotional abuse were discussed at household visits and GBV awareness sessions that reached 1,499 beneficiaries (932 female, 567 male). The project provided four of the supported PHCCs with a comprehensive package of GBV services to address the challenges vulnerable women experience and enhance their uptake of services. IMC also established structured psychosocial support services to target vulnerable and at-risk women. To enhance case identification and referrals, the GBV services were linked to other health services, particularly reproductive health and MHPSS services. IMC also established linkages and referral pathways with other partners implementing GBV services in the area. Safe spaces within the PHCCs were labeled as wellness rooms to reduce stigma. IMC trained all staff at the four supported PHCCs on basic GBV concepts. IMC reported facing significant challenges in recruiting qualified staff from outside the PHCCs to support the GBV component, particularly those displaying attitudinal readiness to work with GBV survivors effectively. Although these challenges delayed the component’s startup, the project overcame the challenges at most implementation sites. At two sites, the delays in recruiting qualified staff prevented the establishment of case management services during the project. Social workers drawn from the community were trained and mentored using the interagency GBV case management guidelines. IMC established quality control through monthly structured case consultation/debriefing sessions, observation visits, and one-on-one coaching. In addition, the survivor-centered attitude scale was used to evaluate the social workers’ attitudinal readiness to work directly with GBV survivors and to reveal areas in which the staff members might require further education and training. However, as social workers were not supported by the DOH after the end of the grant, this component was not sustained after closeout. The project trained 367 staff members (220 males and 147 females) on basic GBV concepts, GBV case management, psychological first aid, psychosocial support, GBV screening, and referral pathways. Analysis of pre- and post-test results on training participants’ knowledge of GBV concepts demonstrated an 80 percent increase following the training. IMC staff also observed improvement in the training participants’ attitudes toward GBV survivors, including a reduction in the tendency to blame GBV survivors and judge the decisions taken by at-risk women. This improved perceptions on how to better support GBV survivors and a greater sense of accountability among service providers. IMC trained staff on the importance of ensuring beneficiaries’ privacy and confidentiality. IMC staff reported that recruiting staff from the community enhanced the uptake of services by vulnerable women, who felt more comfortable being treated by someone from a common ethnicity or religious background who spoke the same dialect as they did. Project staff noted during the KIIs that in Sinjar, having a Yazidi midwife increased the likelihood of pregnant women accessing care at the maternity unit rather than continuing the common practice of delivering at home. This is particularly significant for complicated deliveries. Women feared traveling to Mosul because of the multiple checkpoints along the route and would therefore have had to travel to Dohuk, which was almost two hours away. 27 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV KIIs with IMC staff reported challenges incorporating GBV case management into the PHCCs since the DOH did not have a strong component of social work and case management. The lack of time and DOH weakness eventually affected these services’ sustainability after the project closed. One interviewed facility reported that the GBV services were no longer available, while another facility still offers the services but they were receiving fewer clients since there was not outreach currently being conducted. KEY BARRIERS TO INCREASING UPTAKE OF HEALTH SERVICES (EQ 2A) Cultural norms and traditional practices hindered women’s ability to access services, particularly GBV and sexual and reproductive health services. Women’s access was restricted by their husbands and family members—during the household visits, women indicated that they needed to access GBV services but that if they attempted to do so, they had to be accompanied by their spouse, their spouse’s parent, or relatives. According to project documents, community mobilizers also confirmed that the stringent traditional practices created a barrier for some GBV survivors who would have otherwise wanted to access GBV services. The vulnerable women’s own lack of trust in the services created an additional barrier to their uptake of services. IMC sensitized the community on the importance of accessing services. Additionally, the IMC team brought community leaders to the service delivery sites to enhance their awareness of the available services, which enabled the community leaders to serve as ambassadors within the community, promoting messages in support of the intervention. The IMC team further engaged the community through celebratory events such as International Women’s Day celebrations. Outreach workers conducted house visits and group awareness-raising sessions to build vulnerable women’s trust in the services. Case management services were also provided to vulnerable women in their homes. IMC staff report that initial uptake of services was slow. However, the comprehensive community mobilization efforts overcame the community’s skepticism. By the end of the project, IMC reported increased uptake of both MHPSS and GBV services among vulnerable populations. Unfortunately, the enhanced uptake was observed during the last phase of the project, thereby limiting the achievements under this component. Initially, the project planned to integrate only GBV case management services within the PHCCs. However, when the project staff realized the challenges vulnerable women faced in accessing the services, they engaged the community mobilizers and started training the PHCC staff on case identification and referrals to overcome this barrier. Regular sessions were conducted with key community leaders and relevant stakeholders to highlight the protection concerns related to reducing GBV among the target population, paving the way for enhancing women’s freedom of movement to access services. IMC staff reported experiencing difficulties in incorporating GBV activities within the PHCCs as PHCC staff did not perceive these services as a priority. This barrier was overcome through advocacy efforts with the DOH. IMC negotiated the acquisition of safe spaces within the facilities to provide services to vulnerable women and refurbished the spaces to ensure confidentiality. IMC mental health room at Sada PHCC (IMC) “There was a general reluctance from the community in accessing these services, since this was something new and they doubted the services and did not feel comfortable accessing the services. But after we engaged the key community leaders and figures, you could see there was acceptance at the community level.” – IMC field coordinator 28 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV CONCLUSIONS (EQ 2) The GBV component addressed a clear community need and increased uptake of GBV services by combining these services with other primary and maternal health services and by embedding them within the PHCCs. However, although some women preferred to access these services within the PHCCs so that they could appear to be accessing other, less stigmatizing services, other women were reluctant to access services there because they perceived the PHCCs as being dominated by men. Due to GBV’s sensitive nature, significant time was required to garner the community’s trust and effect behavior change to increase access to services. The project’s short duration therefore limited service delivery under this component. This limitation was exacerbated by the challenges in recruiting qualified staff from the community. Building capacity of DOH PHCC staff and of social workers from the community helped support this intervention’s sustainability and facilitated case referral. However, as social workers are not supported by the DOH, this component was not sustained after closeout. For more information on DOH handover processes and outcomes, see EQ 5. IMC’s multifaceted approach clearly had a positive impact on vulnerable women’s uptake of services. As cultural norms and community perceptions of GBV and mental health issues are deeply ingrained, however, significant, long-term efforts are required to bring about sustainable behavior change and reduce the barriers women face. These factors, combined with the project’s short duration and the recruitment challenges during startup, limited uptake of services. The lack of a community health strategy within the DOH further limited uptake and sustainability. Interventions could have prioritized higher-level advocacy efforts within the Ministry of Health (MOH) to promote sustained buy-in at the district level. EVALUATION QUESTION 3 To what extent did the project’s health care training/support contribute to improved quality and availability of MHPSS services? Prior to IMC’s support, the supported PHCCs did not offer MHPSS services. Under this grant, IMC initiated and expanded psychosocial services and integrated mental health care into primary health care and protection services. IMC provided MHPSS structural components and resources that are fundamental to achieving quality care (e.g., psychosocial units, MHPSS staff, training and guidelines). IMC’s MHPSS program focused heavily on training to increase knowledge and build capacity of DOH and PHCC facility staff on mental health care and to increase practitioners’ skills in providing mental health services. The menu of training programs and capacity building for the DOH staff included the Mental Health Gap Action Program, Psychological First Aid (PFA), PFA Train the Trainer, Detection and Referral, Interagency Standing Committee Guidelines, and Brief Therapy and Systemic Approach training. In addition, trainings on evidence-based psychological interventions were delivered to DOH Ninewa staff in Duhok and at the national level in Baghdad. The goal was to train 60 people; however, only 56 were trained because some DOH participants could not attend the trainings. The effectiveness of the clinical training for practitioners is well documented in IMC reports, which show an increase in knowledge. According to pre- and post-test results for each of the trainings, all trainings effected some improvement (Figure 6). 29 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV FIGURE 6: MHPSS PRE- AND POST-TEST RESULTS Increasing staff knowledge on and provision of MHPSS services benefited the community through increase in access to these services. KIIs indicated that community feedback was very positive and that people were very accepting of the services. Key informant interviews and project documents did not provide sufficient detail as to whether quality services were delivered as intended. CHALLENGES AND BARRIERS TO MHPSS SERVICE DELIVERY (EQ 3A) The main challenges and barriers the MHPSS program faced can be grouped into five broad categories: (1) sites for MHPSS services, (2) human resources, (3) service delivery, (4) community outreach, and (5) security. 1) Sites for MHPSS services. To increase MHPSS service availability, the project identified and supported 25 psychosocial units in the Ninewa region. These units had provided MHPSS services before the conflict but were deactivated in 2014 due to lack of DOH resources. IMC aimed to strengthen the units and resources so that the DOH could continue activities after project closeout. The support included minor rehabilitation, basic equipment and furniture, and staff training. Negotiations to use the spaces and the time it took to rehabilitate the areas delayed project startup. One KII with IMC staff indicated that this delay led to teams conducting “case management services at the community level or a mobile approach, which is not as effective as being done in a confidential and private space that encourages people to come for follow-up.” In project locations without psychosocial units (i.e., Wana and Quba), the program faced significant challenges in establishing services due to the lack of private areas to conduct MHPSS services. In these areas, the project had to shift planning to find private locations within the health facility to provide services. This shift required more intensive efforts, which delayed program initiation. 2) Human resources. According to project documents and discussions with IMC technical staff, staffing presented a major challenge in providing MHPSS services primarily because of the displacement of qualified staff and a lack of MHPSS staff (i.e., case managers or psychologists) within the DOH, particularly in Wana and Quba. Further obstacles included difficulty finding qualified male case managers who would work closely with female case managers (Aqlush area) 79% 59% 67% 63% 66% 41% 62% 55% Inter Agency Standing Committee Training PFA Trainings PFA Tot Case Management Pre-test Post-Test 30 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV and difficulty finding female outreach workers, as many were not allowed to work with NGOs (Mwafaqya area). 3) Service delivery. Throughout the project period, MHPSS case management teams provided services (i.e., individual consultations, group support activities, and awareness-raising sessions) to 2,107 individuals in Al-Qosh, A;-Mowafaqia, Al-Quba, and Wana (Figure 7). Top reported stressors were prolonged displacement due to armed conflict (55 percent), financial and economic concerns (25 percent), and family neglect issues (20 percent). FIGURE 7: MHPSS CASE MANAGEMENT SERVICES DURING THE PROJECT PERIOD BY AGE AND GENDER (UNIQUE VISITS) The number of patients receiving services is only 47 percent of the project’s targeted 4,500 individuals. IMC project documents reveal that the target was not reached due to late start in service provision, lack of private space for case management, and difficulty hiring qualified staff, as described above.9 4) Community outreach. Stigmatization of MHPSS services was reported to be a challenge. IMC worked closely with community leaders on program implementation and decision making and hired and trained outreach workers from the community. This helped to empower communities to support their members with psychosocial and mental health issues and reduce stigma. By acting as a bridge between the community and mental health resources, MHPSS outreach helped reduce stigma and improve accessibility of services. The project reached 12,100 individuals through home visits (Figure 8) that provided information on psychosocial well-being, schizophrenia, autism in children, causes of and treatment methods for anxiety, depression, enuresis, grief and loss, psychological support for children affected by war and conflict, mental health and mental health concepts, and maternal well-being. One facility key informant reported that community outreach created important links between the community and the PHCC, especially for those who live far from any health facility. They felt these connections were key in educating and increasing the number of community members who accessed PHCC services. Although the findings reveal sensitization and awareness support, the project team had difficulty retaining outreach staff, especially due to population movements in the area. Since staff was 9 International Medical Corps. “FINAL REPORT From 1st of June 2018 through 31st of July 2019,” September 2019, 1–43. 14 434 741 89 1278 18 409 368 34 829 0-4 5-17 18-49 50 Total Female Male 31 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV recruited from these communities, at times they themselves returned home leaving the communities they had been recruited from.10 Outreach staff also faced security issues, and for periods of time they would not be able to visits homes or move around freely. FIGURE 8: COMMUNITY OUTREACH HOME VISITS 5) Security. The MHPSS teams encountered security challenges. For example, in Al-Quba, MHPSS services, including home visits, awareness sessions, and case management sessions were suspended for two weeks for security reasons. MHPSS PROGRAM SUSTAINABILITY (EQ 3B) The collaboration between the DOH and IMC appeared to be strong, according to the KIIs and project reports. The IMC handover strategy was well designed and closely followed up by both partners. The following factors were essential in supporting the MHPSS program sustainability: • IMC co-implemented activities with DOH Ninewa and Dohuk; government officials attended trainings, engaged in hiring staff, and worked together to develop a handover plan and discuss it throughout the project period. • IMC supported a national MHPSS conference in January 2019 and conducted regular follow-up meetings between participants, including the MOH and DOH, to discuss challenges and updates on MHPSS program implementation and sustainability. • IMC enhanced the technical competencies of a large cohort of service providers and built facilities’ capacity to deliver MHPSS services. • To support community-driven services, IMC actively engaged community members and leaders throughout the project phases and included them in project implementation and decision making. • IMC and the DOH mapped services in each area to facilitate referrals. 10 International Medical Corps, “FINAL REPORT From 1st of June 2018 through 31st of July 2019,” September 2019, 1–43. Male 5,205 Female 6,895 32 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV Despite these successes and the DOH’s willingness to take over managing the PHCCs, the DOH was not prepared to absorb MHPSS services. During a third-party monitoring visit by FHI 360 in early 2019, PHCC staff at various project sites voiced their concerns that MHPSS and GBV service sustainability was unlikely.11 Project staff reported in KIIs that “the DOH has no capacity to provide the full MHPSS package” and that it “showed…willingness to carry on these activities…but…could not because they don’t have the qualified staff and the resources to do so.” Project staff also reported that they believed the only MHPSS services that were sustained after the project were identification and referral of cases to hospitals and that these services continued because of health facility staff training and capacity building during the project period. They further reported that it was unlikely that case management activities would continue because they did not have the requisite staffing or ongoing mental health care training. Based on the interviews with two of the facilities that received MHPSS support, although they still offer some MHPSS services they are limit due to inadequate funding and lack of specialized staff. As one respondent explained, “The services are still offered, but they are limited compared to under the project, so we have less beneficiaries, especially from far away. If we don’t go out into the community and remind them of our services, they don’t come.” CONCLUSIONS (EQ 3) The MHPSS program’s strength was its programmatic focus not only at the facility level but also at the community and government levels, especially around strengthening the health workforce’s capacity and service delivery. The comprehensive nature of the services and the full cascade of services from awareness to case management and treatment ensured a systematic approach to achieving results. Further, the MHPSS meetings held at the national level with key stakeholders helped push the MHPSS agenda and conveyed the topic’s importance. In fact, one key informant stated, “What IMC was doing at the end of the project was becoming a part of the national strategy.” Despite the successes, MHPSS services were not sustained. The IMC project revealed that MHPSS programming requires a commensurate increase in capacity within the health system to attain sustainability. Challenges with staffing, stigma, and locations for service provision added to the instability. This is especially true in countries such as Iraq, which do not have strong MHPSS policies or strategies. More time was needed to fully integrate mental health services into the PHCCs, increase health professionals’ knowledge and service provision, and reduce stigma surrounding mental health care. Iraq has experienced violent conflict, suffering, and social upheaval for many years. MHPSS services will continue to be a high-priority need. It remains to be seen whether the IMC-supported MHPSS services, along with the MHPSS strategy for Iraq, will be sustained, along with the desired outcomes of improved access and service quality and, ultimately, improved patient mental health. 11 International Medical Corps, “Quarterly Report January 1, 2019 – March 31, 2019,” April 2019. Evaluation team did not receive a copy of the assessment report. 33 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV EVALUATION QUESTION 4 To what extent did project support improve the continuum of care in terms of referrals to and treatment by secondary health facilities? Effective referral systems are essential to ensure a continuum of care. IMC focused on a holistic referral system that ranged from the community to health facility levels (primary and secondary). Operational support (e.g., fuel and maintenance for ambulances, incentives for drivers) combined with staff training and the development of a referral and care pathway strengthened the primary hospital care continuum. During the project period, 439 cases were referred to secondary care, with the majority being for emergency cases (Figure 9). FIGURE 9: PHCC REFERRALS TO SECONDARY CARE To ensure a coordinated effort of medical care to improve client outcomes, IMC developed a two-way system from the referral source (e.g., community, PHCC, GBV) to the referral center and back. This system consisted of referral forms the patient would give to the referral center to fill out and return to the originator. One IMC staff reported that most patients returned to the PHCC after being referred to pick up medication prescribed by the referral source. If they did not, IMC had staff follow up with the referral agency to confirm the patient reached the facility and obtain outcomes and feedback. Documentation of patient follow-up and outcomes is missing in reports received from IMC; therefore, although the system was clearly in place, it is difficult to determine if it worked. Additional challenges that disrupted the continuum of care were the lack of DOH-dedicated ambulances in some of the project areas. One KII reported that IMC could support only four or five ambulance units, which interfered with the ability to refer and the ability of patients to benefit from the referral process. One KII reported that the most significant change in community referrals was CHW training on referrals and follow-ups. They attributed the increase in community referrals to the high level of awareness on health-related issues within the community. The CHWs conducted extensive door-to￾door household visits, as well as awareness-raising sessions through events and sessions. There was a strong linkage between the CHWs and the PHCCs. Cold Case 13% Emergency 87% 34 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV CONCLUSIONS (EQ 4) Overall, the referral system’s design appeared to be strong and aligned with the Iraq Conceptual Framework for Referral from Primary to Secondary Level.12 The project improved care by putting in place protocols and a method of communication within the referral system. The mapping of services in each area supported the referral pathways, which continued to be used as IMC’s support phased out. However, the project was unsuccessful in incorporating the design in all areas, primarily because of lack of DOH ambulances at many of the sites, which impeded the continuum of care as secondary health services were far from IMC service areas. EVALUATION QUESTION 5 What were the successes and challenges of project handover and revitalization to DOH for core PHCC services in terms of staffing, management, record keeping, and finances? IMC collaborated and co-implemented with DOH Ninewa and Dohuk from project initiation to facilitate a smooth handover. The project included key PHCC stakeholders such as facility managers and CHWs to encourage their proactive participation in project implementation to ensure an effective and sustainable transition of service provision. IMC also engaged community members and leaders in project implementation to facilitate acceptance and trust and to promote sustainability. Overall, the handover stayed on track along with the participants bought into the process. However, several issues limited the handover process, including “local governance and security, supply availability, and human resource retention.”13 Further, the DOH had difficulty adopting a fast-paced timeline for full handover, especially for the MHPSS and GBV programs, which were not strong before the crisis. Several factors influenced the handover process, including (1) partnership and management, (2) finances, (3) staffing, and (4) record keeping. 1. Partnership and management. KIIs and reports showed that the project was grounded in strong collaboration, not just with DOH but also with other NGOs and the various working groups. The DOH-IMC partnership facilitated handover because it was built on shared priorities that were developed at project initiation. To improve PHCC management, IMC built the capacity of the district DOH staff by providing professional development courses for 55 DOH staff (40 male and 15 female). Further, IMC supported the program managers by developing and disseminating supervisory tools and conducting joint health facility assessment, monitoring, and evaluation visits. Action plans were developed after the visits to improve the quality of health services and prepare for a handover. IMC also held midyear project review meetings between the DOH and PHCC managers to ensure the final handover in July 2019. Although IMC coordinated closely with the PHCC managers and district managers during the handover process, one interviewed IMC staff reported that it would have been more effective to involve the MOH in the higher-level discussions to expedite decision making (e.g., on staff deployment). 12 USAID. “The Referral System Revised. Primary Health Care Project in Iraq (PHCPI),” March 2012, https://www.humanitarianresponse.info/sites/www.humanitarianresponse.info/files/documents/files/the_referral_system_revised. pdf 13 APC. Improving Health Services and Outcomes In The Ninewa Plains, Iraq. https://publications.jsi.com/JSIInternet/Inc/Common/_download_pub.cfm?id=22647&lid=3 35 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV 2. Finances. It is difficult to know if the DOH has adequate funding to continue providing the level of services provided during the project period. However, key informants reported that finances were one of the biggest barriers, especially now as “Iraq is in a sensitive place economically, which impacts revenue and therefore impacts health service delivery.” KIIs with IMC staff also revealed that the DOH did not have the capacity to continue to provide incentives to retain staff in insecure and remote locations, nor did they have the same financing for supplies and consumables. 3. Staffing. As discussed above and in detail below, hiring and retaining staff for insecure and remote locations presented and will continue to present a challenge for the DOH due to lack of continued incentives. Further, staff turnover may lead to gaps in training and orientation for new staff or refresher trainings when needed. As mentioned above, staffing was a significant challenge for the project, as well as for the DOH. It is hard to find qualified staff who want to work in certain practice areas (e.g., MHPSS). Even the project had to pay staff more just to work in these areas, an option the DOH does not have. 4. Recordkeeping. Many of the key informants did not comment on or were unsure of recordkeeping challenges and successes during the handover. However, one informant believed that recordkeeping needed to be improved. He stated, “We tried to support it through trying to equip at least the facility managers with IT equipment and training them on maintaining records, but it is something that needs to be worked on since there is no centralized reporting system that feeds up to the national level.” SERVICE DELIVERY AFTER END OF GRANT (EQ 5A) This section relies on the most recent information: final PHCC monitoring visits (conducted April/May 2019), the final report, and interviews with IMC staff who are still working in the region but not directly with the targeted facilities, and health facility staff still working on four of the facilities. The following presents information on the status of each type of service delivery: • PHCC services. The final PHCC monitoring visit reports revealed that the health services were active and running at project end. KIIs also indicated that the PHCC services continued, “hopefully with the same level of services and quality.” However, there were some changes as IMC support concluded, including a decline in accessibility as IMC incentives for longer working days were no longer provided and a decline in client consultations due to staff reductions. In Al-Mowafaqia PHCC, the final monitoring report revealed a decline in consultations from 80–100 a day to fewer than 30 a day after IMC handed over. One of the facility key informants reported that the decline was due to staffing: “Now there are only nurses who can do simple investigations (e.g., pregnancy tests or urine dips) and provide vaccination. Anything more than that they have to travel to another health facility.” • Maternal units. Of the three maternal units initiated during the project period, only Bashiqa and Sinjar continued after the project ended. The Wana maternity unit could not continue due to lack of staff, including a reproductive health doctor to provide services. Multiple key informants reported that the maternal units were one of the key successes of the project and handover, stating that “once IMC took over these facilities and were able to provide medical equipment and staffing, then the DOH didn’t have a problem and were able to continue running the activity since everything was already there.” 36 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV • MHPSS services. As discussed above, key informants reported that if any of the MHPSS services continued after the project’s end, it was the identification and referral of cases to hospitals. One informant reported that some of the PHCC managers were “not keen on buying into integrating the MHPSS services into the PHCCs.” Lack of ownership of MHPSS interventions within the PHCCs reduced sustainability. To fully integrate MHPSS services into the health facilities, continued support would be needed at both the national and local levels to institutionalize MHPSS into the health care system. Of the two interviewed facilities that received MHPSS support, MHPSS services were still being offered a year later, but they were limited and they had less clients due to lack of outreach. • GBV services. Reports revealed that attitudes toward GBV changed during the project period. This shift was reflected by the DOH requesting to have GBV activities at the health facilities and more sessions on the subject. 14 Nevertheless, according to KIIs and project reports, PHCC stakeholders believed that the lack of technical capacity made full ownership of GBV and MHPSS services unlikely.15 Further, one key informant reported that since the GBV staff were not DOH staff, it was unlikely that the program would continue. Based on the facility interviews, one facility reported limited GBV services being provided, while two of the other facilities reported that the services were no longer available.16 STAFF SUSTAINABILITY (EQ 5B) As the project ended almost one year ago (July 2019), it is difficult to determine whether all the facility staff are still in place. Financial support, fragile or weak human resource governance, and difficulties in hiring health professionals in insecure and/or remote locations were key challenges in staff retention. One project coordinator stated, “Unless you give them [doctors] a better package and incentives, they would prefer to stay where they are, due to security and safety.” In the July 2019 final monitoring report for Al￾Mowafaqia PHCC, the PHCC manager reported that despite multiple requests to the DOH for staff, after IMC left there were no doctors in the PHCC. Further, as discussed above, the Wana maternity unit closed after the project ended due to lack of staffing. Nevertheless, IMC key informants believed that most of the PHCC staff continued across the project sites, with one person reporting that “90–95 percent” of the PHCC and reproductive health staff are still present in the facilities. IMC key informants believed it was unlikely that the MHPSS and GBV staff were still working at the project sites. They reported that after IMC left, financial support for CHWs and GBV and MHPSS staff did not exist and that the DOH was “not ready to provide them with any financial support or incentives.” One informant reported that one nurse trained on GBV was probably still in the PHCC providing services because she is a DOH staff member. This was confirmed by some of the health facility staff, of the two interviewed facilities that received MHPSS support, specialized staff had left and although services were still offered they are limited. 14 International Medical Corps. “FINAL REPORT From 1st of June 2018 through 31st of July 2019,” September 2019, 1–43. 15 International Medical Corps “Quarterly Report January 1, 2019 – March 31, 2019,” April 2019. 16 One facility key informant (Sinjar) was unfamiliar with GBV service status. 37 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV CHANGE IN COMMUNITY ENGAGEMENT LEVEL (EQ 5C) The DOH system does not provide for CHWs and outreach workers. Instead, the DOH has health educators who are similar to CHWs but work in the schools. 17 One key informant reported that education sessions are still being conducted “in the schools, and public spaces and some house visits if a person has many diseases.” The informant further reported that the educators “are DOH staff so get normal salaries” and that in each area there are three to five educators and they continue the same work and take their normal salary. This statement differs from that of another key informant, who reported that the CHWs recruited were not DOH staff but instead were recruited from within the community and the DOH did not continue supporting them at the end of the project “due to financial issues.” Either way, the level of community engagement had certainly diminished by the project’s end since CHWs did not continue after the grant ended. Although, the awareness of available services at the PHCCs and knowledge gained during the community sessions will likely continue to benefit some community members as they share information, based on interviews with facility staff, they had noticed declines in clients seeking services, especially MHPSS and GBV services, which they thought was due to limited outreach being conducted since grant ended. CONCLUSIONS (EQ 5) The project reports and KIIs clearly demonstrate that the handover of the IMC-supported project was mostly a success. IMC not only built capacity but also provided support for planning, management, and monitoring among the various DOH management levels. However, this support’s impact was limited by the DOH and health system’s ability to absorb it, which was hampered mainly by a lack of money and the shortage of qualified staff in insecure and remote locations. These factors constricted services and their expansion, as evidenced by the Wana maternity unit’s closure due to lack of staff; the end of the community outreach, case management, and GBV programs; and the decline in PHCC visits as IMC support ended. Despite consensus that IMC support increased services and capacity, both reports and interviews clearly indicate that additional time might have strengthened sustainability and the project handover, particularly the GBV and MHPSS components. As one key informant stated, “We have more than 200,000 people living in camps and are IDPs. If we can make the health services good, I think people will come back.” 17 Key informant interview (5/2020) 38 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV RECOMMENDATIONS The evaluation team presents the following programming recommendations: • To deliver sustainable results, projects that support primary health care should be implemented for longer than one year to properly build DOH and facility staff capacity and help build sustainable systems. This is especially important when programming includes reestablishing or initiating services (e.g., GBV/MHPSS). • GBV programming should create community-based comprehensive care centers tailored to women’s and adolescent girls’ needs, combining GBV and MHPSS services with additional services. Community-based interventions should be implemented through women’s centers and safe spaces whenever possible to enhance the level of trust in the services. • GBV programming should facilitate cooperation among the various ministries involved to ensure adoption of GBV international standards of care. This should include involvement at all levels of the government to strengthen associated policies and guidance nationally. For example, whereas the clinical management of rape falls under the MOH’s jurisdiction, case management falls under that of the Ministry of Labor and Social Affairs. Coordination and collaboration among the multiple ministries involved should be emphasized to enhance intervention sustainability and support more comprehensive approaches. • MHPSS programming should include the procurement of psychiatrists and psychotropic medications to improve the mental health outcomes of both men and women. • MHPSS programming in this region should work with national-level stakeholders to influence political buy-in, policy, and leadership on MHPSS to continue to improve mental health outcomes. • Projects that focus on MHPSS and GBV services should work from the outset on gaining local and national political buy-in, budget, and human resources to increase these services’ sustainability and public awareness about them after the project ends. • Health programming in the Ninewa area should work closely with the DOH and other key stakeholders at project initiation to develop an achievable plan to retain staff in remote and insecure locations. USAID should ensure that the facility-level MHPSS and GBV service providers are DOH staff and not supported by the project. This will improve the handover process and service sustainability. • Health programming in the region should support the DOH and other stakeholders to build a local resource pool of experts who can conduct continuing medical education and provide technical assistance to health facilities, especially for MHPSS and GBV services. • Community-based health programming should explore ways to retain CHW staff, especially after project end. Efforts to retain trained CHWs can address the high turnover rates reported and enhance the intervention’s long-term sustainability. • Programming should seek to actively engage and partner with other organizations on the ground to support a more comprehensive approach to women’s and adolescent girls’ empowerment and well-being. These components could be included in community-based women’s health centers through a complementary package of health and continuing education services. 39 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV ANNEX 1: EVALUATION STATEMENT OF WORK APPROACH FOR HEALTH EVALUATION – NINEWA PLAINS HEALTH GRANT INTRODUCTION This document outlines the adjusted approach to conducting the evaluation for the “Improving Health Services and Outcomes in the Ninewa Plains, Iraq” grant implemented by International Medical Corps (IMC) under the Advancing Partners & Communities (APC) implemented by JSI Research & Training Institute, Inc. (JSI) in collaboration with FHI 360. Given that travel to the region is not possible due to limited local access to the target facilities and COVID-19-related travel restrictions, this modified approach relies on available project data and remote key informant interviews (KII) with project staff and other key stakeholders to collect as much information as possible to answer the five evaluation questions listed below. This document outlines the purpose of the evaluation, provides general background information on the grant, lists the evaluations questions and explains the general approach to collecting and analyzing the data. In addition, the document lists the key deliverables and dues dates, and provides in Annex 1 the draft instrument to be used for the remote KIIs. EVALUATION PURPOSE The purpose of this evaluation is to learn more about the successes and challenges of the Ninewa Plains grant in achieving the its objectives in order to inform future programming in this region. The outcomes of the grant were impressive (see key grant outcomes below). Indicator data showed large numbers of key health services provided by the supported health facilities. This evaluation will seek to learn key lessons from implementation. In addition, this evaluation will explore the factors during the grant period that contributed to or impeded achievement of both intended and unintended outcomes. The evaluation will also seek to understand what, if any, additional measures are needed to secure better health outcomes for the target population, building on the gains achieved through this activity. GRANT OVERVIEW “Improving Health Services and Outcomes in the Ninewa Plains, Iraq” grant was implemented by IMC under the APC project. Implementation took place between July 1, 2018 – July 31, 2019 and was implemented in the Ninewa Plains (target population 64,000 residents) in Sinjar, Wana, Al Quba and Telafar City. The grant focused on 6 Primary health care centers (PHCC): Wana PHCC, Al-Quba PHCC, Bashiqa PHCC, Al Sada PHCC, Al Mowafaqia PHCC, Al Qush PHCC; and 1 secondary facility: Sinjar General Hospital (maternity unit). The key areas of support provided by the grant focused on primary health care (PHC), RH, mental health and psychosocial support (MHPSS), referrals and linkages, Gender-Based Violence (GBV) prevention and response and community health. The grant’s main objectives and the supporting activities are provided below: • Objective 1: Improve delivery of the Essential Package of Primary Health Care Services to minority populations in the Ninewa Plains. Strengthen the capacity of primary health care centers (PHCC); facilitate links to advanced levels of care (secondary care site); engage CHWs; support health facility managers to strengthen health facility management. 40 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV • Objective 2: Strengthen and integrate mental health/ psychosocial services, post-traumatic treatment for children, and sexual and GBV services in selected sites. • Objective 3: Develop and implement a formal handover process for services provided under Objectives #1 and #2; transition management and quality assurance of services in Ninewa to the Department of Health (DOH). EVALUATION QUESTIONS To learn more about the successes and challenges of the grant’s implementation, this evaluation will examine the following evaluation questions. 1. How did project support components (e.g. training, equipment, management support, etc.) contribute to changes in the delivery and access of health services at the six primary health care centers (PHCC) and Sinjar General Hospital? a. Were there key elements of support not provided by the project that could have a greater impact on delivery and access to health services? (e.g. comparison of PHCC vs. WHO mobile clinics) b. Were there any unintended outcomes due to project interventions? 2. What key factors contributed to the uptake of health services among vulnerable women (e.g. women facing GBV)? a. What key barriers did the project help overcome to increase uptake of health services? 3. To what extent did the project’s health care training/support contribute to improved quality and availability of MHPSS services? a. What were challenges and barriers to delivery of these services? b. Can the MHPSS be sustained as integrated PHCC services going forward? 4. To what extent did project support improve the continuum of care in terms of referrals to and treatment by secondary health facilities? 5. What were the successes and challenges of project handover and revitalization to DOH for core PHCC services in terms of staffing, management, record keeping and finances? a. Since project end, what project-supported health services are still being delivered in the targeted health facilities? b. Since project end, what facility staff are still in place? c. Since project end, have there been any changes in the level of community engagement (e.g. CHWs)? 41 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV This evaluation will look at the key project components in relation to the key successes and challenges of the grant. In evaluation question one, the team will seek to learn more about the following support components: • Training - rational drug use, introduction training, ATLS, BLS, medical equipment training, IPC, clinical management of rape, neonatal resuscitation • Human resources (HR) - incentives to health staff • Community health – CHW • MHPSS - case management, reactivated psychosocial (PS) units, training • GBV – case management, training, space, prevention activities/events • Supplies - equipment, medicines, medical supplies In addition, the evaluation will examine the different health services that the grant supported. The evaluation questions will address the following key areas of support. • Primary health: outpatient consultations, treatment of communicable and non-communicable diseases, emergency referral, dispensation of essential medicines, disease surveillance EWARN • Maternal and Child health/Family planning: ANC, PNC, family planning and sexually￾transmitted infections (STIs) • MHPSS and GBV: case management EVALUATION APPROACH AND DATA SOURCES The evaluation team will use secondary data from project reports in addition to primary data collected through remote KIIs to answer the five evaluation questions. The evaluation team will conduct KIIs with JSI/IMC staff in DC and will connect remotely with the 4 IMC staff in Erbil/Dohuk (i.e. Deputy Country Director, Field Site/program Coordinator, Grants Coordinator, Program Coordinator). In addition to these interviews, the evaluation team will ask KII respondents for additional recommendations of other possible remote KIIs that could be conducted to collect additional perspectives of the outcomes of the grant. Other remote KIIs could include: WHO mobile clinic project staff, UNFPA staff, health cluster members, or GBV cluster members. These interviews will be conducted over skype and are expected to last 45- 60 minutes. A draft interview guide (Annex 1) will be used to collect information to help answer the evaluation questions. All interviews will be recorded and detailed notes will be drafted for interview analysis. The evaluation team will also review all the available project data and documents to learn more about the changes between the beginning and end of the grant implementation period. 42 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV DATA ANALYSIS The evaluation team will examine qualitative data from KIIs for patterns and look at project documents for comparisons between the start and end of the project. The team will seek to identify and extract good practices, lessons learned, challenges/barriers, recommendations and outcomes relevant to each EQ to better understand the grant’s success. The team will analyze transcribed data from the KIIs using content analysis techniques, in which text will be coded according to key themes of interest across the interviewees. The team will summarize responses related to each theme and include quotations from respondents to illustrate key findings. This will include highlighting “outlier” responses and experiences, such that the range of responses will be captured in the summary write-up for each evaluation question. EVALUATION DELIVERABLES Based on the information gathered, the evaluation team will draft a report that answers all five of the evaluation questions and share recommendations for future programming in similar settings. The final report will present findings associated with each evaluation question, followed by conclusions and recommendations for future programming. The evaluation team will conduct interviews after approval of the data collection instrument by USAID. Contingent upon USAID approval, the team plans to conduct interviews between March 30 and April 10. If needed, based on additional respondents suggested by KIIs, this period could be expanded to include additional interviews. Following data collection and analysis, the evaluation team will aim to submit the draft evaluation on April 30. The estimated deliverable dates are as follows: 43 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV ANNEX 2: IMC FINAL GRANT OUTCOMES DASHBOARD 44 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV ANNEX 3: SOURCES OF INFORMATION The evaluation team has collected and reviewed the following project documents to support answering the evaluation questions: • Final project report September 2019 • Quarterly reports 1, 2, 3 and 4 • PowerPoint presentations (Aug/Sept 2019) • Patient satisfaction exit interview quarterly report (September 2019) • Success story – IPC • Success story – Maternity Unit • Project 2 pagers and infographics “dashboards” • Monthly activity updates- July and September 2018 • Assessment report Tal-Afar (April 2018) • Full Assessment Report (July 2018) • Assessment report Sinjar District (Wana, Manara, Shikhqa PHCC) (March 2018) • Assessment report Sinjar District (General Hospital and Sinjar PHCC) (Jan 2018) • PHC Training Data • Final monitoring summary report May - July 2019 o Wana PHCC, Al Mowafaqia PHCC, Bashiqa PHCC, Sinjar Hospital, Al Quba PHCC, Al Sada PHCC, and Al Qush PHCC KEY INFORMANT INTERVIEWS CONDUCTED The evaluation team held informal interviews with key Washington based support staff from IMC and JSI prior to the IMC field staff interviews. Those interviews informed the team on general programming practices and outcomes. All interviews were conducted in 2020 over skype in English except for the contacts from Bashiqa and Wana PHCC which were conducted in Arabic. All interviews were recorded and detailed notes were developed. The following interviews were conducted with the IMC field staff and health facility contacts: TABLE 1: IMC STAFF Name Position in Project Date Megan Thompson Grants Coordinator May 5th Abdulmalik Abdulateef Senior Medical Officer May 6th Dickson Barasa Field Site Coordinator May 7th Nancy Odesho Program Coordinator May 7th 45 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV TABLE 2: HEALTH FACILITY STAFF Name Facility Job Title Date Abdullah Mahmood Abdullah Wana PHCC PHCC Manager/Nurse June 15th Loqman Baqir Hussein Bashiqa PHCC PHCC Manager/Doctor June 16th Mohammed Akram Ghanim Al Mawofaqia PHCC Medical Doctor / Represent the PHCC manager June 12th Dilshad Ali Abdullah Sinjar Hospital PHCC Manager / Pharmacist June 9th 46 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV ANNEX 4: DATA COLLECTION TOOLS Key Informant Interview (KII) Interview Guide (Project Staff) Health Evaluation – Ninewa This interview guide is intended for IMC in-country project staff. All other respondent guides will be adapted from these questions and adjusted to fit the context of the other specific respondent types. OVERVIEW Purpose: The purpose of the key informant interviews (KIIs) is to examine grant results, successes, and challenges to support planning for future health programming. The IMC project, “Improving Health Services and Outcomes in the Ninewa Plains, Iraq” had three primary tasks: 1. Improve an essential package of primary health care services to a target population of 64,000 minorities in Iraq’s Ninewa Plains; 2. Strengthen and integrate MHPSS, PTSD treatment for children; and GBV services; 3. Develop and implement a process to handover health services provision to the Iraqi government Department of Health (DOH) at the end of the grant. INTRODUCTION Hello, my name is_______________. I am a consultant with MSI conducting an evaluation for USAID of IMC’s health response in the Ninewa Plains, Iraq from July 2018- July 2019. The purpose of this interview is to examine grant results, successes, and challenges that will inform planning for future health programming in country. The questions address circumstances during the timeframe of the grant, as opposed to current circumstances. As someone who has been directly involved or is familiar with IMC’s programming in the Ninewa Plains, we would like to ask your views on IMC’s activities, specifically what worked and didn’t work, and what improvements could be made to ensure effective and sustainable results. We would also appreciate your views on any external factors that influenced programming in the area. The interview will probably take between 45-60 minutes. We will digitally-record the session, but your responses will remain anonymous and no names will be mentioned in any reports. We hope that you will speak openly and honestly about your experiences and viewpoints. Please give us as much detail as you are willing and able to provide, and feel free to ask any questions, or tell us if you do not understand a question being asked. You are free to opt out of participating now or at any time during the discussion. You can also choose not to answer any of the questions. Do you consent to this interview? If NO, Stop and thank the person for their time. If YES, Continue to Section I. 47 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV SECTION I. PRELIMINARY INFORMATION P1 Today’s date (dd/mm/yyyy) P1 ____/____/_____ P2 Time interview started: P2 _____: _____ (00:00 – 24:00) P3 Time interview ended: P3 _____: _____ (00:00 – 24:00) P4 Consent for interview granted P4 1=Yes (Proceed) 2=No (STOP) If no, why: __________________________ P5 Location of the interview P5 P6 Respondent’s name P6 P7 Respondent’s organization P7 P8 Respondent’s position in organization P8 P9 Respondent’s no. of months in organization P9 P10 No. of months working physically in Iraq/ or externally on the Iraq project emergency? P10 Months___________ P11 Activity locations the respondent is familiar with P11 Wana Mowafaqia Quba Al Qosh Sada Bashiqa Sinjar P12 Program area the respondent is familiar with P12 PHC MCH/FP CHW MHPSS GBV Management/admin Other_________________ P13 Interviewer’s name P13 SECTION II. Evaluation Question 1: SUPPORT Question: Response EQ1.1: Overall, do you think the project was able to provide the necessary support to improve the delivery of primary health services in the targeted PHCC and hospitals? If so, why? If not, what key elements were missing? As a result of the grant: a. Was there evidence of change – positive or negative in the delivery and access of health services? If so, please describe any changes and explain what contributed to the changes. If not, why not? b. Were there any unintended changes/outcomes – positive or negative – in the lives of recipients and in their environment? What were they? EQ1.1 a. b. c. 48 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV c. Of the different types of support provided (e.g. training, staff incentives, CHW, supplies) by the project, was there any components that were more important than others? Why and how? EQ1.2: What is your overall assessment of the change in the quality of care at the IMC supported health centers and hospital by the end of the project? a. a. Can you tell me about anything you would have liked to see improved in the PHCCs or hospital that was not part of the project? If so, why would these changes be important? Probe for different health services types: primary care (i.e. outpatient, referrals, etc.), MCH/FP, MHPSS, GBV. EQ1.2 a. EQ1.3: What do you think was the most significant change that took place due to the CHW’s activities? Explain a. How do you think CHWs helped to increase access to services? b. Were there any challenges with the CHW component of the project? If so, what and how did they affect the project's outcomes? EQ1.3: a. b. EQ1.4: What were the changes in the types of Maternal Child Health services offered by health facilities related to IMC’s support? a. What were the changes in the quality of Maternal Child Health services offered by health facilities related to IMC’s support? b. What were the challenges/barriers and the successes of delivering MCH and/or family planning services? EQ1.4: a. b. EQ1.5: Did the allocated resources for staffing (incentives and management) adequately respond to the level of needs and the operating environment? Why/Why not? a. What challenges or successes did the project face in delivering the staffing components? b. Were the incentives provided thought to be (or were they) sustained by the DOH after the project ended? Please explain. c. To your knowledge, are staff that received incentives still at the supported PHCCs and hospitals? if no, do you know why? EQ1.5: a. b. c. EQ1.6: In your opinion did the IMC-supported PHCCs and hospital have access to the basic medicines and commodities they needed during the project? a. How was this intervention implemented and adapted to ensure constant supply of medicines and commodities? b. To what extent did the supply chain continue after IMC handed over to the DOH? EQ1.6: a. b. EQ1.7: Overall, what is your general opinion/view of the effectiveness of the trainings IMC provided? a. What were the main challenges/ issues that influenced/ affected the effectiveness of the trainings? b. What changes in staff practices did you notice after the trainings? c. To your knowledge, are staff that were trained still at the supported PHCCs and hospitals? if no, do you know why? Probe for: Quality (e.g. curriculum, facilitators/trainers, materials), Quantity/frequency (e.g. numbers of sessions, length, follow-up, refresher training), staff retention after training EQ1.7: a. b. c. 49 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV EQ1.8: How did supportive supervision and management impact service delivery and access? Please explain. EQ1.8: EQ1.9: Has IMC’s intervention been shown to be as effective as other alternative approaches in the same context (i.e. WHO mobile clinics)? Why or why not? a. Were there key elements of support that the project did not provide that could have led to greater impact in delivery and access to health services? Please explain. b. Did you feel there was duplication of services? If so, which services? Probe: Ask the respondent to explain why and what key elements contributed to IMC or other projects success EQ1.9 a. b. COMMENTS BY INTERVIEWER: SECTION III. Evaluation Question 2: UPTAKE AMONG VULNERABLE WOMEN Question Response EQ2.1: How well did the IMCs program succeed in engaging vulnerable women and adolescent girls in accessing different health services? What about vulnerable women from religious and ethnic minorities? a. What was the most significant activity or approach that engaged these populations? b. What were the main challenges with engaging with these populations? Probe: make sure the respondent discusses women and adolescent girls and religious and ethnic minorities. Probe for different health services types: primary care (i.e. outpatient, referrals, etc.), MCH/FP, MHPSS. EQ2.1 a. b. EQ2.2: a. In your opinion, how did the program increase uptake of GBV services and case management for women and adolescent girls? a. What barriers do you think still remain related to women and adolescent girls accessing GBV services? b. In your opinion, how did the program increase uptake of GBV services and case management for religious and ethnic minorities? c. What barriers do you think still remain related to religious and ethnic populations accessing GBV services? EQ2.2 a b c EQ2.3: Do you have any suggestions for improving uptake of services by vulnerable women and girls? Please think about both preparedness and response and include information on uptake of health and GBV services. a. Do you have any suggestions for improving uptake of services by ethnic and religious minorities and other vulnerable populations? EQ2.3 COMMENTS BY INTERVIEWER: SECTION IV. Evaluation Question 3: MHPSS SERVICES Question Response EQ3.1: What were the changes in the types of mental health and psychosocial support (MHPSS) services offered by health facilities related to IMC’s support? a. What were the changes in the quality of MHPSS services offered by health facilities related to IMC’s support? EQ3.1 a. 50 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV b. What project support component was the most helpful in improving the quality of MHPSS services? and why? c. What were the successes of delivering MHPSS services? Please explain. d. What were the challenges/barriers of delivering MHPSS services? Please explain b. c. d. EQ3.2: What is your overall assessment of the availability of MHPSS services provided? a. Do you feel the project adequately supported vulnerable populations such as religious and ethnic minorities, women and girls with MHPSS services? How? b. In your opinion, what contributed the most to the improved availability of MHPSS services? Or c. In your opinion, why was the availability poor, what would you like to see improved? Probe: training, support, meds/supplies EQ3.2 a. b. EQ3.3: Based on your knowledge, have the MHPSS services continued to be sustained as integrated PHCC services? Why or why not? a. Are there any challenges or barriers that have hindered MHPSS services from being integrated? b. Is there anything the project could have done better to help support these services being sustained? EQ3.3 a. b. EQ3.4: In the monitoring reports the PHCC managers commented that the MHPSS intervention was an IMC program and not PHCC? a. What did they mean by this? Was the MHPSS service not integrated into the PHCCs? b. Are the PS units still active? EQ3.4 a. b. COMMENTS BY INTERVIEWER: SECTION V. Evaluation Question 4: REFERRAL MECHANISM Question Response EQ4.1: Based on your knowledge, how did the project contribute to changes in the referral system? EQ4.1 EQ4.2: IMC supported a referral mechanism to secondary and tertiary care. IMC also provided survivors of GBV referrals if additional support not offered by IMC was needed. How often do you think that patients/clients followed the referrals? a. If someone did follow a referral, do you think there was a good system to let the PHCCs know what happened at the referral center? b. How do you think this tracking system affected the quality of care? EQ4.2 a. b. EQ4.3: What were the challenges/barriers and the successes of the referral mechanism? EQ4.3 COMMENTS BY INTERVIEWER: 51 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV SECTION VI: Evaluation Question 5: PROJECT HANDOVER Question Response EQ 5.1 In your view, how was the overall handover process to the DOH? a. What were the challenges and successes? EQ5.1 Challenges Successes EQ5.2: IMC supported the DOH in health system strengthening and capacity building. In your opinion, what was the most significant change for the DOH related to IMC’s support? a. What led to the change? b. What could have been done better? EQ5.2 a. b. EQ5.3: To what extent does the DOH have ownership, capacity and resources to maintain the activity results? a. What are the main challenges that the DOH faced in trying to maintain the project activities after it ended? Probe to ensure the following are discussed: finance, staffing, management, and record keeping/HIS, finance. EQ5.3 a. EQ5.4: Since project end, have there been any changes in the level of community engagement, i.e. are community health workers still conducting home visits at the same level they were during the project? If YES, probe what made it successful? If NO, why not? EQ5.4 EQ5.5: Since project end, what project-supported health services are still being delivered in the targeted health facilities? What services are no longer being offered? a. Please explain why. EQ5.5 EQ5.6: Since project end, what facility staff is still in place? a. If staff left after IMC’s support ended, What factors led to staff departure? EQ5.6 COMMENTS BY INTERVIEWER: SECTION VI: FINAL COMMENTS Question Response F1: Do you have any other suggestions for improving health program implementation targeting religious and ethnic minorities in this type of setting? Please think about both preparedness and response. FI F2: Do you have any suggestions and contact information for other people we could talk to remotely to learn more about the grant? For example: DOH staff, health cluster members, other NGOs, etc. F2 F2. And finally, we have talked about a lot of things today; but do you have any final comments or questions? F3 COMMENTS BY INTERVIEWER: Thank you very much for your time! 52 | IMPROVING HEALTH SERVICES AND OUTCOMES IN THE NINEWA PLAINS EVALUATION USAID.GOV ANNEX 5: EVALUATION TEAM TEAM COMPOSITION The evaluation team is composed of two health subject matter experts (SME) and one monitoring and evaluation (M&E) experts who will work together to collect, analyze and synthesis data to answer the six assessment questions. Short summaries of team members are provided below: Gwynne Zodrow (Team lead/M&E expert): Gwynne Zodrow is a technical manager who provides M&E support to multiple government and private sector clients in a variety of fields including health, agriculture and food security. As a technical lead she is involved in all stages of evaluations including design, data collection and analysis and reporting. Ms. Zodrow recently worked with USAID’s Ghana Evaluate for Health project providing technical M&E assistance to the local team supporting implementing partners, the Mission, and managing multiple health evaluations and a national facilities survey. Ms. Zodrow has a Masters of Global Public Health from George Washington University. Dina Khaled (SME): Dina Khaled has over 19 years of experience in international health and extensive experience working with ethnic and religious minorities. In recent years, Ms. Khaled has focused on the refugee crisis, responding to emergency medical needs in Greece and Lebanon wherein she supported various clinics by offering medical translation and coordinating relief organizations. During this time, Ms. Khaled also established a medical clinic for Syrian refugees and provided direct support to over a thousand Yazidis escaping genocide in Iraq. Ms. Khaled has worked with the UN, USAID and Amnesty International to conduct research and supervise research teams for evaluations, assessments and monitoring activities. Ms. Khaled has a Masters’ of Public Health, International Health Promotion from George Washington University and is verbally fluent in Arabic. Heather Lorenzen (SME): Heather Lorenzen is a certified nurse practitioner with nine years of experience in emergency medicine and humanitarian response. As an emergency medical coordinator, Ms. Lorenzen has coordinated medical care, conducted needs assessments, developed and implemented training programs and conducted remote monitoring in Tunisia, Libya, Lebanon, Syria, Iraq and Jordan. Ms. Lorenzen has also served as a consultant, conducting needs and quality assessments for health systems and most recently co-authoring the Sphere Standards for Assessment, Monitoring and Evaluation. MSI home office support was provided by Debby Orsini (Technical Director) and Carly Green (Project Associate).