YEMEN BASIC HEALTH SERVICES (BHS) PROJECT Mid-Term Evaluation September 2009 This publication was produced for review by the United States Agency for International Development. It was prepared by Joy Riggs-Perla, Nahed Matta, and Nevine Hassanein through the Global Health Technical Assistance Project. Front Cover Photo: Director General of Health for Amran Governorate Confers with Mobile Team YEMEN BASIC HEALTH SERVICES (BHS) PROJECT Mid-Term Evaluation DISCLAIMER The views of the authors expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 09-001-255) is available in print or online. Online documents can be located in the GH Tech web site library at www.ghtechproject.com/resources.aspx. Documents are also made available through the Development Experience Clearinghouse (www.dec.org). Additional information can be obtained from The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-00 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION i CONTENTS Acronyms ......................................................................................................................................... iii Executive Summary .......................................................................................................................... v Findings and Recommendations for BHS .................................................................................. v I. Introduction and Background .................................................................................................... 1 Summary of Health Situation in Yemen .................................................................................... 1 USAID Population/Health Program ........................................................................................... 2 II. Scope of Work and Methodology .............................................................................................. 3 Background and Rationale for Evaluation................................................................................. 3 Summary of Scope of Work ...................................................................................................... 3 Evaluation Methodology ............................................................................................................ 3 III. Summary Project Description ................................................................................................... 5 Project Goals, Objectives, and Components ............................................................................ 5 Performance Monitoring and Project Targets ........................................................................... 5 IV. Progress Toward Meeting Goals and Achieving Results .......................................................... 7 IR 5.1: Increased Access to Quality Health Service and Participation at the Community Level ...................................................................................................................... 7 IR 5.2: Increased Health Knowledge and Improved Behaviors at the Community Level ....... 17 V. Project Management Issues ................................................................................................... 21 Performance Monitoring Plan ................................................................................................. 21 Project Management and Staffing ........................................................................................... 22 USAID Staffing .......................................................................... Error! Bookmark not defined. VI. Lessons Learned ..................................................................................................................... 23 VII. Findings and Recommendations for BHS Extension .............................................................. 25 Positive Points ......................................................................................................................... 25 Recommendations .................................................................................................................. 26 ANNEXES Annex 1. Documents Reviewed .................................................................................................... 31 Annex 2. People Interviewed ......................................................................................................... 35 Annex 3. Scope of Work ................................................................................................................ 39 Annex 4. Work Plan ....................................................................................................................... 47 Annex 5. Progress on M&E Indicators .......................................................................................... 53 ii YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION iii ACRONYMS AMTSL Active management of third stage of labor ANC Antenatal care BHS Basic Health Services (project) CMG Community mobilization group CPR Contraceptive prevalence rate CTO Cognizant Technical Officer DFID Department for International Development DG Director General DHS Demographic and Health Survey ECG Electrocardiogram EMOC Essential maternal and obstetric care ESD Extending Service Delivery (project) FP Family planning GAVI Global Alliance for Vaccines and Immunizations GTZ German Technical Cooperation HIHS High Institute of Health Sciences HTSP Healthy timing and spacing of pregnancy HSS Health Systems Support IMCI Integrated management of childhood illness IMR Infant mortality rate IR Intermediate result KMC Kangaroo mother care KAP Knowledge, attitude, and practice M&E Monitoring and evaluation MCH Maternal and child health MDG Millennium Development Goal MMR Maternal mortality rate MNCH Maternal, neonatal, and child health MNH Maternal and neonatal health MOPHP Ministry of Public Health and Population MVA Manual vacuum aspiration NGO Nongovernmental organization OP Operational plan PHC Primary health care PMP Performance monitoring plan PNC Postnatal care PPM Private practice midwives RH Reproductive health iv YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION ROYG Republic of Yemen Government SFD Social Fund for Development TFR Total fertility rate TOT Training of trainers U5MR Under-5 mortality rate WHO World Health Organization YMA Yemeni Midwives Association YMU Yemeni Women’s Union YPHR Yemen Partnership for Health Reform (project) YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION v EXECUTIVE SUMMARY Background: Yemen, with roughly 40% of the population living below the poverty line, must accelerate the rate of demographic change; poor health is not only a consequence of poverty but also a cause of it. Supported by its development partners and guided by the Yemeni Government’s Poverty Reduction Plan of 2006–2010, Yemen must find ways to improve health conditions rapidly. Though health indicators are improving, mortality and fertility are still too high, and the people of Yemen should not have to wait another 20 years before they see tangible improvements in the quality of their lives. USAID’s Basic Health Services (BHS) Project began in early 2006, building on the work of a previous project. In July 2009 USAID/Yemen commissioned the Global Health Technical Assistance Project to conduct a mid-term evaluation of the BHS Project, for which a two-year extension with additional activities had recently been approved. The mid-term evaluation will be used to guide work for the next two years. USAID/Yemen also asked the evaluation team to make recommendations about future areas of work in the health sector. The recommendations may be useful input to the new strategy development process. FINDINGS AND RECOMMENDATIONS FOR BHS The BHS Project, operating in five challenging governorates, is well–managed and making solid progress in a difficult environment with a fragile health care system. Activities implemented in the focus governorates are highly valued by health staff and officials at the national, governorate, and facility levels. The renovations and refurbishing of health facilities provided visible and immediate improvements and catalyzed higher rates of utilization. However, the proposed shift of emphasis from renovation and refurbishing to more intensive work on improving quality and building health system capacity is appropriate. Community midwives are the most important avenue to provide family planning and maternal and child health services in Yemen. BHS needs to intensify its supervision, monitoring, and support to midwives and expand the private practice midwifery program. The expansion of mobile teams to remote and underserved areas has helped reach more people with reproductive and child health services and provided much-needed support to small and poorly staffed clinics. This program should be expanded even further, with close oversight by BHS. A successful Best Practices pilot program in Sana’a, now being scaled up in six hospitals in two of the target and four new governorates, introduces globally recognized, low-cost, high￾impact services for mothers and newborns that will help prevent deaths and improve health outcomes. It should be expanded into general hospitals in the other three focus governorates. Religious leaders and women-led nongovernmental organizations (NGOs), with support from BHS, are performing a critically important role in community health education. They lend credibility to the effort, help reduce cultural sensitivity, and increase the acceptability of reproductive health messages. Since 2006, 664,669 people have participated in BHS community education sessions. BHS should work closely with other donors and the Primary Health Care section in the Ministry of Public Health and Population (MOPHP) to ensure a common approach to training, supervision, and quality improvement for reproductive and child health services. vi YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION For the extension period BHS should develop a complete performance monitoring plan and set outcome indicators so that USAID and the MOPHP can determine the efficacy and impact of BHS interventions. YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 1 I. INTRODUCTION AND BACKGROUND SUMMARY OF HEALTH SITUATION IN YEMEN Yemen, which has a population of about 22 million, is among the least-developed countries in the world; 40% of the population lives below the poverty line. With an annual growth rate of 3%, almost half the population is under 15. Overall literacy for adults is 44% and for women only 24%. Health and demographic indicators are improving but not fast enough to achieve the Millennium Development Goals (MDGs) established by the Republic of Yemen Government (ROYG) (see Table 1). The health care system is not well developed and is estimated to reach just over half the population. A high fertility rate (6.2 children per woman), closely spaced pregnancies, and early age of marriage and first birth create health problems for both women and their children. The proportion of deliveries attended by trained personnel is only 22%, and maternal mortality is very high. With gradual improvements in services for children (e.g. immunization, nutrition, treatment of pneumonia and diarrhea), the mortality rate for children under 5 is improving, but at least 50% of all infant deaths are in the neonatal period (first month of life).1 To significantly reduce deaths of children in Yemen, it is critically important to reduce neonatal mortality by encouraging mothers to space their children by at least two years, reduce high-risk births, and utilize quality antenatal, delivery, newborn, and postnatal care services. Lower fertility will make mothers and babies healthier and lower the newborn mortality rate. A major challenge in Yemen is to accelerate the rate of improvement in health, because poor health is not only a consequence of poverty but also exacerbates it. Poor people waste money on low-quality services from unqualified providers and are not able to work due to illness. Multiple closely spaced 1 Family Health Survey, Ministry of Health and Population and Pan Arab Project for Family Health, 2003. TABLE 1. DEMOGRAPHIC INDICATORS AND MDG TARGETS Indicator Rate (2003) MDG Goal for 2015 Infant mortality (per 1,000 live births) 77 41 Under-5 mortality (per 1,000 live births) 102 52.5 Maternal mortality rate (per 100,000 live births) 365 212.5 Total fertility rate (births per woman) 6.2 No goal Contraceptive prevalence rate (modern methods) 13.4% No goal (Source: Yemen and Millennium Development Goals, World Bank Working Paper No. 31 dated March 2003) TABLE 2: DEMOGRAPHIC TRENDS Indicator DHS 1991 DHS 1997 PRB 2000 Factsheet Infant mortality rate (1,000 live births) 84.5 75.3 77 Under-5 mortality rate (1,000 live births) 121.1 104.8 101.9 Total fertility rate (births/woman) 7.7 6.5 6.3 Contraceptive prevalence rate (%) 6.1 9.8 13.4 2 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION pregnancies not only undermine the health of women and children, but large families also make it harder for the poor to manage within their meager earnings. A burgeoning cohort of youth and young adults resulting from the population growth rates confounds the unemployment situation and creates the preconditions for instability and social unrest. In short, Yemen, supported by its development partners and guided by the Yemeni Government’s Poverty Reduction Plan of 2006– 2010, must find ways to improve health conditions rapidly. Though health and demographic indicators are improving (see Table 2), mortality and fertility are still too high and the people of Yemen should not have to wait another 20 years before they see improvements in the quality of their lives. USAID POPULATION/HEALTH PROGRAM USAID’s health program in Yemen is aimed at increased use of reproductive, maternal, and child health services (Strategic Objective 5). The two main programs are the Basic Health Services (BHS) Project and the Yemen Partnership for Health Reform (YPHR) Project. Both work with the ROYG on its programs related to achieving MDGs 4 and 5, which seek to reduce infant and child mortality and improve maternal health. The BHS Project supports two of USAID’s intermediate results (IRs): IR 5.1: ―Increased access to quality health services and participation at the community level,‖ and IR 5.2: ―Increased health knowledge and improved behaviors at the community level.‖ Project activities are organized in terms of those IRs. YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 3 II. SCOPE OF WORK AND METHODOLOGY BACKGROUND AND RATIONALE FOR EVALUATION The BHS Project, which began in early 2006, is implemented through an associate award to Pathfinder International through the USAID Global Health Bureau’s Extending Services Delivery (ESD) Project. BHS builds on the Catalyst Project that operated in Yemen in 2004–2006. In July 2009 USAID/Yemen commissioned a mid-term evaluation of the project through the Global Health Technical Assistance Project after granting BHS a two-year extension with additional activities. Findings from the evaluation will be used to guide the work for the next two years. USAID/Yemen also asked the evaluation team to make recommendations about future areas of work for USAID in the health sector. The recommendations may be useful input into the new strategy development process. SUMMARY OF SCOPE OF WORK The purpose of the mid-term evaluation (see Annex 3) is three-fold: 1. Evaluate the effectiveness of the BHS Project so far in achieving program objectives, which contribute to USAID/Yemen’s Strategic Objective (SO) 5, Increased use of reproductive, maternal, and child health services. 2. Review planned BHS activities for the extension period and assess whether they represent the right approach to addressing Yemen’s maternal and child health (MCH), reproductive health (RH), and family planning (FP) needs. 3. Make recommendations based on the findings for any modification to BHS activities during the extension period and for future USAID MCH and FP programming. The scope of work required a team of three: a team leader with evaluation experience in health and population, an MCH expert from USAID’s Global Health Bureau/MCH Division with knowledge of the region, and an RH care specialist with experience in Yemen. EVALUATION METHODOLOGY The in-country portion of the evaluation was conducted June 29–July 15, 2009. The team met with a wide range of project stakeholders, including Ministry of Public Health and Population (MOPHP) officials at the national, governorate, and district levels; development partners; NGO partners working with BHS; USAID officials; BHS project staff in Yemen and coordinators from each of the focus governorates; service providers; and some clients. A list of questions was prepared to obtain comparable information from the interviews. Though the team’s ability to travel to the governorates was constrained by security issues, one team member was able to visit project sites in Amran Governorate, which provided important information from the service provider and client perspectives. Team members also visited sites in Sana’a, such as the Al Sabeen hospital, where pilot Best Practices activities were implemented. The team reviewed a large number of project documents, other donor and MOPHP program documents, and data from surveys, service statistics, and project-generated monitoring and evaluation (M&E) data. (See Annex 4 for information on the analytic framework, stakeholder questions, and the timetable for evaluation deliverables.) 4 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 5 III. SUMMARY PROJECT DESCRIPTION PROJECT GOALS, OBJECTIVES, AND COMPONENTS The cooperative agreement for the associate award with Pathfinder International was signed in November 2005, so the BHS Project has been in operation for about three and a half years. The objective of the project is to help the MOPHP to increase access to quality health services in reproductive, maternal, and child health. Its activities are organized to help ensure better access to and utilization of RH/MCH services on the one hand and improve health knowledge and behaviors on the other. BHS has been working in five governorates—Amran, Al Jawf, Marib, Sa’ada, and Shabwa—that have particularly poor health indicators and few donor-supported health activities. For the extension period, Sana’a City was added, as were hospitals in four new governorates—Aden, Taiz, Lahej, and Ibb—where the Best Practices activity is being scaled up. The project extension will also intensify work in the five original governorates and add new activities related to hygiene and safe water, including linkages with school programs. The project base at the national level is with the Population Section of the MOPHP, which has responsibility for RH and maternal and newborn health (MNH). It is projected that BHS health promotion activities will reach about 14% of the population of Yemen. The approach is to fund the activities of highly respected Yemeni organizations that are committed to improving RH/MCH conditions and related community mobilization and education programs. Among them are religious and other foundations, women’s organizations, and associations such as the Yemeni Safe Motherhood Alliance. BHS also funds health education and promotion through governorate Community Mobilization Groups (CMGs) with multisectoral representation. PERFORMANCE MONITORING AND PROJECT TARGETS Though BHS collects a large amount of data to manage its work, it reports to USAID on a much smaller set of indicators (see Annex 5) and on the Operational Plan (OP) indicators USAID/Yemen needs for reporting to USAID/Washington. Quarterly and annual reports, consisting largely of numerical information on achievements in governorates, are provided to the MOPHP as well as USAID. Reports are also shared with the director general of each governorate where the project works. 6 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 7 IV. PROGRESS TOWARD MEETING GOALS AND ACHIEVING RESULTS This analysis of progress toward achieving project results is organized into three main categories: activities related to IR 5.1 (increasing access to quality health services and community participation), those related to IR 5.2 (increasing health knowledge and improved behavior), and findings on overall project management. For each activity area, the report provides a very short description, assesses progress and accomplishments, identifies issues or constraints, and makes recommendations. The recommendations in each of the three areas are consolidated in Section VII. IR 5.1: INCREASED ACCESS TO QUALITY HEALTH SERVICE AND PARTICIPATION AT THE COMMUNITY LEVEL Renovation, Equipment, and Supplies Brief Description This activity was considered central to the BHS project in promoting increased access to basic RH/MCH services. USAID asked the project to produce fast and highly visible results by renovating and equipping health facilities, establishing and equipping mobile outreach teams, training health providers, providing equipment and supplies, and improving clinic management and the quality of services (BHS agreement dated December 15, 2005). This was done in the five focus governorates. Renovation, equipment, and supplies responded to the following priorities: FP/RH, antenatal care, emergency obstetric care, newborn and child care, postpartum care, and finally general areas, such as laboratory, reception, administration, and pharmacy. BHS worked with the MOPHP to select facilities based on a needs assessment and a commitment that facilities have permanent staff. Achievements and Progress During the BHS base period (2006–2008), BHS renovated 24 health units and health centers, established 11 basic and emergency obstetric units in health facilities, built and furnished 17 housing units for health providers, and equipped 104 health facilities. The targeted number of health facilities refurbished to MOHPH standards and facilities equipped was almost achieved, although the targets had to be adjusted in 2007 when the USAID budget was reduced. The activity achieved its target in terms of an increase in numbers of clients in health facilities assisted by BHS compared to the baseline (from 105,400 in 2006 to 397,378 in 2008). MOPHP considers the geographical focus appropriate because the focus governorates, constrained by tribal conflicts, were the least achieving and most underserved. Most of these governorates received no other donor support. Central, governorate, and local staff valued this activity because of the visible impact on utilization rates for RH/MCH services. Establishing housing for health providers helped stabilize services in several facilities that otherwise would have been closed or functioned only partially. 8 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION Constraints, Issues, and Recommendations Except for a few sites in Amran the evaluation team was not able to visit many facilities for security reasons, so it was not possible to visit renovated facilities, assess utilization rates, gauge client satisfaction, or verify availability of contraceptives and other commodities. BHS reports that maintenance of facilities and the equipment provided is a constant concern because facility budgets are inadequate. BHS coordinators are instructed to identify weaknesses in monthly reports. The evaluation team saw field monitoring reports from BHS staff and BHS coordinators assigned to the governorates that contained a mix of positive and negative observations. While there are reports reflecting on the situation of the facilities, there are no analyses or organized system for follow-up on actions to correct problems. The BHS coordinators should summarize key issues and weaknesses resulting from monitoring visits and attempt to address them with the Director General or central project staff. The subsequent monthly report should start with updates on any actions taken. In the long run the MOPHP supervision system, once functional, should provide such information and follow-up. To sustain the investment in upgraded facilities and equipment purchased since the project began (including equipment of the mobile teams), BHS staff should set up a system for carrying out regular checks on vehicles and equipment. The project is looking into finding a cost-effective solution for building maintenance capacity at the governorate level. BHS should ensure that FP units in all assisted facilities receive more comprehensive attention in terms of furnishing, equipment, and training to ensure that clients can access the highest quality services. BHS should also regularly monitor the number of facilities it has renovated that are fully staffed according to MOPHP standards, as anticipated in the BHS agreement. Pre-Service Community Midwifery Training Brief Description The project has been helping the MOPHP to increase the number of trained providers by supporting the pre-service training of community midwives at the High Institute of Health Sciences (HIHS). Students must complete middle school to be eligible for the two-year certificate program that qualifies them as community midwives. The HIHS training is conducted in Sana’a and four branches in BHS target governorates. This is the only MOPHP￾affiliated institute responsible for training nurses, midwives, and a variety of other health paramedical personnel. All partners and the Social Fund for Development (SFD) use the HSIS for training midwives. On occasion the training requests are beyond staff capacity to handle. Several donors are attempting to assist. An assessment by the World Health Organization (WHO) in 2008 highlighted several limitations on capacity to handle the training load and conduct supervised mentoring. UNICEF and the SFD plan to conduct an assessment of the HIHS facility. The World Bank will also Midwives in Amran Governorate receiving on￾the-job training YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 9 commission an evaluation of pre-service training. During the evaluation team’s visit to the HIHS in Sana’a, the vice dean expressed a need to revise the midwifery curricula and build staff teaching skills in branches in the governorates as well as in Sana’a. Achievements and Progress BHS has provided much-needed support for pre-service training for community midwives. It established strict selection criteria for trainees, such as requiring midwives to be selected from the communities where they live and will work after graduation, because this helps overcome deployment and staffing problems in the focus governorates. BHS also selected what it considered the best instructors among the HIHS staff in each branch and supported them to work intensively with BHS-funded midwifery trainees to help ensure better skills development. In 2006–2008, 217 midwives were trained; the target was 200. BHS also trained 105 midwives for three days before graduation on the five best practices. Challenges, Constraints, and Recommendations HIHS capacity issues are beyond the BHS mandate. Several donors are attempting to assist; USAID should stay engaged and consider support for improving pre-service training capacities along with other donors because this is a major opportunity to ensure the future supply of midwives. BHS should regularly monitor the employment status and performance of the midwives trained by BHS and Catalyst (a total of 340). Community midwives are highly important to improving MNH. More and better follow-up of all BHS-supported trainees is needed, including developing supervision systems, refresher training, and further skills development. BHS should take advantage of the good relationships built up during trainee selection and training at HIHS to monitor their subsequent performance. Skills related to RH topics (ANC, delivery care, PPC, newborn care, FP) and private business management should be reinforced through on-the-job training, not just counseling. In-Service Training for Health Providers Brief Description The BHS project produced numerous training materials and conducted in-service courses for health providers, primarily physicians and midwives but also nurses/nurse technicians and technical assistants, to update their clinical skills. Several courses oriented supervisors and managers on project activities. The training materials ranged from clinical management guidelines to training-of-trainer (TOT) courses. All training was conducted in Arabic using local resources from the universities, MOPHP, and a pool of providers and consultants. Every training activity was facilitated by the BHS training advisor/coordinator. Achievements and Progress The BHS project has so far conducted about 99 training courses. Materials for physicians included clinical skills training for both trainers and trainees, best practices, and planning for essential maternal and obstetric care (EMOC). Midwives were counselled in FP, RH, and MCH as well as getting practical training for newborn care and service quality improvement. A one-day course was added to enable physicians to operate new equipment. Anaesthesia training was added to ensure that obstetric cases needing surgery were properly managed. In the five governorates 1,173 providers were trained; total training days were 6,183. 10 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION The TOT training list included FP, MCH, and best practices courses as well the counseling skills. Most guest lecturers prepared their own presentations but these were reviewed by the project training coordinator for content and methodology. To emphasize a competency-based training methodology, all training courses adopted the participatory approach and used case studies in theoretical sessions followed by hands-on training, usually in hospitals. Training was deemed successful if providers achieve a score of 75% or more on the post-course test. Courses for community health educators (morshidat) went beyond health education to practical training to assist with medical services whenever that was necessary. Training curricula for morshidat covered newborn care, improving skills, and best practices. Constraints, Issues, and Recommendations The high turnover of providers due to reassignments or promotions places a financial burden on the project because those who have been trained are replaced by new staff who must be trained. The absence of an MOPHP supervision system makes it difficult to follow up on those who have been trained. There is a lack of standardized monitoring and supervision tools at the national level or tools to monitor governorate or district service provision and quality, although efforts are underway to develop supervision mechanisms. Even if they were not too busy with other tasks, BHS coordinators themselves do not have enough technical skills to monitor all types of training. The training materials the evaluation team reviewed were found to be out of date. Furthermore, the lack of standards for clinical practice guidelines and training curricula to support such standards means that the technical content conveyed to trainees is not uniform. BHS should consider using MOPHP-approved in-service training materials whenever feasible as long as the curricula are of adequate quality. Staff should review the three-week home￾based training for MNH, a UNICEF competency-based course for community midwives that has been approved by MOPHP, which will use it in its own MNH program (supported by the Netherlands and DFID). Since BHS covers child health and nutrition, BHS should consider collaborating with the MOPHP Primary Health Care (PHC) Section to conduct the 16-day course on integrated management of childhood illness (IMCI), nutrition, TB, malaria, and RH. This course is being scaled up in 64 districts by the PHC Section, funded by the Global Alliance for Vaccines and Immunizations/Health Systems Support (GAVI/HSS). BHS should assess whether these can be utilized. It might be well to link with UNICEF for TOT capacity development to ensure good trainers for the BHS program. Master trainers from the PHC section could also be considered for use in BHS programs. The midwives are a highly important avenue to improving MNC. More and better follow-up of all BHS-supported trainees, supervision systems, refresher training and skills development, reliable systems for supervision of midwives with referral linkages, and expanding the program are all critically important. YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 11 BHS should ensure that all eligible2 community midwives working in BHS facilities receive IUD training now that the procedure is supported by the MOPHP and incorporated in their job description. If feasible, BHS might consider helping the MOPHP scale up this training in the focus governorates in alignment with other active partners and donors. In governorates where no other donors are doing so, BHS should look into management and leadership training using curricula like the HIHS GTZ-supported short courses for governorate, district, and facility managers who have not already been trained. Doctors in focus governorates could be trained on manual vacuum aspiration (MVA) for post-abortion care and on implants, linking with partners such as Marie Stopes and using existing curricula. Since newly graduated community midwives received best practices training, it would be useful to similarly train the midwives deployed in mobile teams and BHS-supported facilities, if possible. Mobile Teams Brief Description Supported by positive feedback and requests from governorate and national health managers for more mobile teams per governorate, the BHS project built on lessons learned from the CATALYST project to expand and reinforce mobile teams. The objective was to expand essential health services to more deprived areas within the five governorates, such as areas with no health facility at all, and to build the capacity of staff in any fixed health unit. The mobile teams were tasked with building relationships between the community and local health providers, increasing service utilization rates where a facility existed, monitoring staff performance and facility needs, assessing community health requirements, and setting a frequency for mobile team visits to targeted villages. The mobile team comprises a trained physician, two trained midwives, and a driver. The project established a list of equipment for the vehicle that would enable the mobile team to provide basic and essential health services for women, including antenatal care (ANC), postnatal care (PNC), delivery, gynecologic exams, and general health services. Child health includes IMCI and vaccinations. Men are given general physical exams. Mobile teams also provide ultrasound examinations, electrocardiograms, first aid, and simple lab tests, such as Hb%, ABO-RH, urine tests for sugar ketones and proteins, and pregnancy tests. Training courses for mobile team members ensure that providers have sufficient clinical skills and include an orientation tour with a successful mobile team to share the experience before going into the field. 2 MOPHP guidelines state that a midwife must have 3-5 years experience to be eligible to receive IUD training. Amran Health Office mobile team van 12 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION Achievement and Progress The BHS project increased the number of mobile teams in the five governorates from 3 to 12. The team monitors service utilization by filling out a form on numbers and type of cases seen and services provided. A substantial increase in the numbers of clients served is reflected in BHS M&E data as the number of mobile teams has expanded. Clients requiring diagnostic work or treatment beyond the capacity of the team are referred to the nearest governorate hospital or district hospital, even though there is no formal health care referral system. The BHS coordinator and sometimes the directors general oversee the quality of services. The coordinator’s monthly activity report contains information on the performance of mobile teams and any activities undertaken by facility staff to prepare for mobile team visits. The BHS Sana’a-based staff occasionally makes unannounced visits to mobile team sites. The evaluation team noted that community health education appears to be an activity assigned to a mobile team midwife during each visit. The physician examines clients. Facility staff see patients when the mobile team is not visiting. The capacity of the mobile team staff is built through training of all midwives and health educators (morshedas) in RH areas. Local health staff based at the facility are included. Challenges, Constraints, and Recommendations Most activities that BHS originally proposed for the mobile teams are underway. However, the project originally planned to have a roving repair team to help maintain mobile team equipment and equipment purchased by BHS for health facilities, and this has not yet happened. Resolving repair and maintenance problems is a matter of high priority for BHS and governorate health offices. There are no official referral forms or a follow-up system to enable the mobile team physicians to formally refer cases. Devising a simple referral form might be a small but useful step forward. Absence of mobile network coverage in some areas has caused communication problems among the teams, the Director General’s office, and the BHS coordinator, when, for example, the team is delayed in reaching a site or returning from one. Lack of a supervision system in the MOPHP raises the question of whether BHS activities will be sustainable after the project phases out. Difficult access to deprived areas due to roads and distances is an obstacle to supervision. Both female and male doctors are sometimes reluctant to adhere to the difficult schedule of these mobile outreach teams even though they do earn extra money from per diems. Low MOPHP pay is a major disincentive to physicians in general. However, the evaluation team did observe in Amran some very dedicated mobile team staff making visits despite extremely difficult road conditions. The MOPHP should devise a reward system for those who show such dedication. Provision of assistance to districts where there are immigrant and refugee camps is an unmet need, and current mobile teams are already too fully occupied to take on additional responsibilities. Because the mobile teams seem to be filling a very necessary function, the number of teams should be expanded as the budget allows. Keeping track of costs and patient loads would also YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 13 help other partners and the MOPHP determine whether to support expansion into other governorates. Best Practices Brief Description This activity was added to the BHS project with a $50,000 grant from the Global ESD project to implement five best practices identified by the Yemen country team. That team has members from MOPHP, several other Yemeni partners, and USAID and in September 2007 participated in the Bangkok meeting, ―Scaling-up High Impact FP/MNCH Best Practices,‖ which was funded by USAID and organized by ESD in collaboration with other USAID partners. The grant was a follow-up activity to countries with good action plans for scaling up best practices. The best practices program was piloted by BHS in Al Sabeen Hospital in Sana’a. The five best practices selected are: 1. Essential newborn care/infection prevention 2. Kangaroo mother care (KMC) for low-birth-weight newborns 3. Encouraging immediate and exclusive breastfeeding 4. Counseling on postpartum FP, healthy timing and spacing of pregnancy (HTSP), and the lactational amenorrhea method (LAM) counseling 5. Vitamin A supplementation for women The pilot, planned as a one-year program, was completed in November 2008. ESD assessed the activity as successful and gave BHS another $75,000 to scale the experience up in six more general hospitals in four new governorates, adding a sixth best practice, newborn immunizations for BCG and polio. BHS was also asked to integrate the teaching of best practices into the curricula of midwifery schools and add a seventh best practice for midwifery curricula, active management of the third stage of labor (AMTSL) to prevent postpartum hemorrhage. Expansion of Best Practices, now underway, will apply the Improvement Collaborative model. Achievements and Progress The BHS team should be commended for moving ahead with this activity, with ESD support, and ESD should be commended for providing an associate award and technical support to BHS to implement this and other activities, such as religious leaders’ training and the Safe Age of Marriage initiative. This collaboration between a Global Health project and a USAID/ Mission project to pilot and implement innovative programs is a good example of collaboration to improve country programs and advance global learning. The best practices selected are evidenced-based, low-cost, high-impact interventions that are usually neglected by health providers. They are being promoted by the global community and recommended by USAID’s MCH and Population and Reproductive Health (PRH) divisions as essential maternal, FP, and newborn care interventions that should be scaled up. The BHS monitoring records reflect increases in most of the best practices by at least 50% in the pilot hospital, which encouraged ESD to support the scale-up. Al Sabeen Hospital established both infection prevention and patient education committees, in collaboration with the MOPHP Quality Department and Health Education Department, which provided follow up (though continuing this will require full MOPHP commitment). 14 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION MOPHP endorsed scaling up Best Practices in the 2009 BHS work plan, and added a working group on best practices to its RH Technical Group. BHS gave MOPHP the Best Practices curriculum to train providers from 67 districts as part of an MOPHP health systems strengthening project funded by GAVI. Constraints, Issues, and Recommendations On a site visit to Al Sabeen Hospital, the evaluation team noted that at this very busy hospital, implementing all components is a challenge. Most of the normal deliveries were counseled, given Vitamin A and BCG, and the data recorded. However, women discharged after cesarean sections or post-abortion care are probably not counseled. Nor were the data recorded, as the counseling room is not near where those procedures are done. Staff in the inpatient ward are too busy to handle these tasks. Hospital staff are loaded with additional counseling and recording tasks and logbooks, even though this is not considered part of the staff’s regular duty. Since project support ended at the pilot site, no one is gathering data and analyzing it; thus, some of the best practices might easily be neglected. Senior hospital staff, although committed in principle, are too busy to monitor the activity. Information on client services provided within the Best Practices project needs to be reflected in the service statistics registers used by the hospital. As suggested in a BHS report, MOPHP should integrate collection of this information into the registers when considering a national scaling-up. The capacity of the hospital to implement infection prevention and client counseling are basic requirements for the success of Best Practices. Having adequate numbers of staff on all shifts is key to consistent quality care. There is a need to address budget shortages to remedy lack of supplies for infection prevention as well as normal services and to address staff shortages. Lack of incentives to implement services on a 24-hour basis and to provide counseling to all patients is a constraint (BHS report). Capacity needs to be built in such hospital management areas as team building, strategic planning and vision, operational planning, problem-solving, quality control, and M&E (BHS report). It will be important to observe in the expansion phase whether the Improvement Collaborative approach overcomes some of these challenges. The hospitals proposed for the expansion period overlap with the MOPHP’s MNH project, which focuses on maternal and newborn departments. The evaluation team recommends full collaboration with the MNH program to ensure strategic consistency. The BHS has already started this collaboration and invited the MNH team to infection committee meetings. As Best Practices is scaled up, it should be promoted as a vital component of the essential obstetric care program that the MNH program is working on. Establishing the right context is important: Best Practices represents simple interventions that can easily be scaled up but do not substitute for or undermine other important interventions for MNCH and FP services. BHS should consider scaling up Best Practices in the five target governorates as well as the new ones and include AMTSL as one of the key interventions in hospitals as well as in clinics and midwife-assisted home deliveries. The added value will be that those hospitals are the site for pre- and in-service training of midwives, whether training is supported by BHS or others. AMTSL should be included as the seventh component of Best Practices. YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 15 In the expansion phase, data collection and indicators need to be streamlined to make it feasible for further-scale up, considering the limitations of very busy hospitals and their capacity to collect and use information. The scale-up experience should be documented, but it should not be treated as a research study. Realistic data collection and analysis expectations will be critical because facilities will not be able to dedicate full-time staff for this function. BHS should discuss indicators selected with ESD. The burden on the BHS team of overseeing the scale-up of the Best Practices program and satisfying ESD reporting requirements should be carefully analyzed by Mission and BHS staff so that this activity does not jeopardize other project activities. Yemeni Midwives Association and Private Practice Midwives Brief Description The National Association of Yemeni Midwives (YMA) was established on September 2, 2004. It has financial, administrative, and technical autonomy. The USAID Yemen Partnership for Health Reform (YPHR) project supported its establishment and provided office, equipment, and hired staff. The YMA has 600 members who have actually paid the YRl 1,200 annual registration fee. The YMA aims to promote and strengthen the midwifery profession in Yemen. It seeks to upgrade the RH/FP/MCH skills, professional status, and career opportunities of midwives. Both the Yemen Partnership for Health Reform (YPHR) and BHS have offered YMA members capacity-building opportunities to upgrade their business planning, hands-on training, and supervision skills. YMA works with the BHS on the Private Practice Midwife (PPM) program: The BHS, with support from ESD core funds and in collaboration with the USAID Private Sector Partnership Project, supported establishment of the PPM program in November 2007. BHS invited YMA to assist with this program and learn from experience in another country (Uganda) as a way to build member capacity. YMA members were trained to coach private midwives on mapping their catchment area and to supervise them on a monthly basis through field visits, using a comprehensive checklist. The PPM program supports and encourages midwives to establish a private practice in their own homes or in community-provided or private settings within deprived areas to increase access to MNH care. With support from BHS, midwives who qualified were required to construct a separate area (room and rest room), furnished by BHS with essential equipment and supplies, to function as a private clinic. A private institute contracted by BHS trained them in basic private business management skills. BHS conducted a six-day clinical skills training to ensure their clinical competence. Clinic of a private practice midwife 16 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION Achievement and Progress Private Practice Midwives (PPM) To date 12 PPM facilities have been constructed and equipped and are functioning. YMA has been conducting supervisory visits and reporting to BHS monthly. In the last quarter (the first for this program), the 12 PPMs received 974 clients from the five focus governorates. The PPM program appears to be a promising initiative; one midwife observed during a field visit recorded 40 deliveries in a month. The volume of clients in general appears to be reasonable. Some PPMs are facing objections from physicians in hospitals who are concerned about their clinical competence and the lack of supervision. Some of the midwives in the PPM have not received BHS clinical skills training. Marie Stopes International is considering including BHS midwives in their Yemeni midwifery franchise pilot activity. In consultation with the MOPHP, BHS should explore the benefits to the midwives themselves of such a collaboration in the future. Yemeni Midwives Association YMA has had in-house management problems that have slowed or stopped its activities for more than a year. The Board of Directors was asked to leave and a election of a new Board is planned. A board director and a deputy who is a full-time BHS staff member were assigned temporarily. As BHS is funding this organization, the situation has at least the appearance of a conflict of interest. The evaluation team heard about the YMA’s organizational problem from MOPHP and other partners; some feel that it would be advisable to wait to see if these management and leadership issues are resolved. Because USAID (through PHR+) was instrumental in establishing this organization, it would be well to assess the situation after the next election before deciding how much support to provide. The clinic established at the YMA headquarters with BHS support is not functioning well and assisted no deliveries. It is not a good model of business management skills. While BHS staff say that they have not provided any encouragement or support for clinical services by the YMA, the project exercises considerable influence on YMA at this juncture and may want to rethink this issue. Constraints, Issues, and Recommendations YMA BHS should limit utilization of YMA assistance to supervising and mentoring the 12 PPMs and postpone other capacity-building investments until organizational issues are resolved. All YMA members who undertake supervision tasks must be qualified as trainers and mentors. Only trained midwives and qualified members should assist with this task. BHS should monitor the fieldwork of both PPM and YMA periodically and through unannounced visits. Because a BHS staff member has a temporary (unpaid) management role with the YMA, it is not clear whether it was BHS or the YMA that actually accomplished the activities detailed in the last BHS quarterly report. YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 17 PPM The PPM activity is just beginning but it seems promising considering the first-quarter data and the potential to make more services available in the community. The presence of a nearby hospital where emergency care could be provided is an ideal set-up so that a PPM can refer complicated cases or emergencies. This will also increase rapport between the midwife and facility staff. So far the PPMs appear to be supervised adequately; and BHS should ensure that this supervision is sustained. To improve clinical competence, the MOPHP/UNICEF three-week home-based midwifery training course with its supervision model could be explored. At some point in 2009, PPM business performance, productivity, and success should be assessed and measured. IR 5.2: INCREASED HEALTH KNOWLEDGE AND IMPROVED BEHAVIORS AT THE COMMUNITY LEVEL Health Education and Community Mobilization Brief Description To increase health knowledge and improve behaviors at the community level, the program designed health education interventions using multiple approaches. Aspects of the BHS health education and community mobilization activity are being implemented at the governorate level, among them building the capacity of MOPHP staff that work closely with communities, and drafting educational and promotional materials that are used nationwide by the MOPHP and other partners, such as the development and dissemination of a Community Health Education Manual, training of religious leaders, and formulating Community Mobilization Groups. Community Health Education Manual: The health manual presents 42 health messages in support of project interventions (selected areas of RH/MCH as well as other health messages) to be used by health educators who conduct community health awareness interventions. The manual was put together in coordination with other donors (UNICEF, WHO, UNFPA, and GTZ) and MOPHP to harmonize messages and maximize use of existing materials and other resources. The health messages in the manual include four on HTSP, three of which are in the section on FP and one in the section on pregnancy and prenatal care. Messages reflect the latest changes in immunization protocols. Community Health Worker training program in Sana’a conducted by the Yemeni Women's Union 18 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION Community Mobilization Groups (CMGs): BHS formed multisector CMGs in each of the five target governorates as the nucleus for carrying out community activities and coordinating inputs. Members included representatives of non-health sectors, such as the ministries of Religious Affairs, Youth, Education, Human Rights, and Agriculture and such national organizations as the Yemeni Women’s Union and the National Women’s Committee. Contacts were initiated with governorate-level institutions to form these local working groups for community education. Selection and recruitment of CMG members followed a well-structured and transparent process that started with writing job descriptions and scopes of work before selecting candidates. Community activities were carried out in coordination with the MOPHP and BHS. The CMGs follow the BHS-developed activity protocol (two group meetings and six field activities a month in five districts within the target governorates). The BHS also involved local volunteers (90 per governorate with youth [boys and girls] and Religious Guidance Institute graduates) to increase the base of educators that help disseminate health messages and reach more of the community through schools, mosques, and NGOs. Other Health Education: RH/FP/MCH messages were drafted from evidence-based practices on priority health topics: four on breastfeeding, including LAM; two on prenatal care; two on delivery; two on postpartum care; three on birth spacing, based on HTSP messages; three on nutrition in MCH; and four on newborn and child care. The project printed 100,000 of two brochures (50,000 of each). Achievement and Progress Dissemination of the Community Health Education Manual (including media coverage by the Al￾Thawra newspaper) was launched in March 2009. Ownership of different donors as well as the MOPHP was reflected in the number of copies printed and used by different implementers, such as the SFD. To ensure sustainability, the BHS project implemented a training of trainers (TOT) course on use of the manual targeting 29 health education directors from 22 governorate health offices. Participants were trained to present and explain details of the 42 messages, especially the ones about HTSP. The training was comprehensive; it covered all the messages as well as communication and training methods. Having a cadre of governorate trainers helped build staff capacity for community outreach. Governorate CMGs implemented a variety of activities that had reached 664,669 people by the end of 2008, according to BHS records. For additional education materials, the BHS project printed each of 20 messages on 3,000 posters each (a total of 60,000 posters. Of these 5,000 were distributed to each target governorate; 10,000 to the Health Education Center (Sana'a) to be distributed in other governorates; 100 to the University of Dhamar; 500 to CMG members and religious leaders during training courses; 600 to the Health Education and Information Directorate in Sioun and Hadramout Governorates; and 500 at the population international day in Hodeidah governorate. The local volunteers documented positive community feedback with requests that they conduct more health-related seminars and gatherings. Parents also sought the help of local volunteers to solve family conflicts, such as disagreements about the marriage at an early age of their daughters and sons. Constraints, Issues, and Recommendations In addition to governorate level staff, ideally facility and district health education officers should be involved in training and working with the Community Health Education Manual. Opportunities should be sought to make sure that the flip chart is familiar to health personnel and if possible used by all RH partners. The manual and the flip chart should be available in all facilities where BHS provides support and given to the mobile teams. YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 19 CMGs have limited capacity. BHS collects data on their activities but there is no connection to MOPHP facilities. The impact of this program is not yet known. CMGs need to be encouraged to link to the nearest facility through the Morsheda. Indicators like numbers of referrals made or requests for educational sessions could provide evidence of their usefulness. Involving pharmacists, assistant pharmacists, and environmental Morsheda in community education activities may offer opportunities to expand and strengthen the impact of health education programs. Religious Leaders Brief Description This approach is not necessarily new, but the design and approach was innovative. Working with the Foundation for Social Guidance, an NGO, BHS prepared a reference book on religious opinions in support of RH. At the request of religious leaders, three brochures were created in consultation with the MOPHP Health Education Center targeting religious audiences, the first with information on breastfeeding, the second with information on FP methods, and the third with information on the risks of early marriage. TOT sessions equipped the religious leaders with training skills to strengthen their community role and enrich their knowledge and information; 146 males and females were trained. Achievement and Progress By the end of April 2009 religious leaders had implemented 4,437 activities that had reached 419,147 people in the five governorates. Those reached include people attending Friday prayers, which are well attended and where religious leaders incorporate health messages into their sermons and made available the three new brochures. The project addressed gender issues to ensure that girls had access to valuable health information by training 20 female religious leaders from Amran and 20 from Marib. As part of the roll-out strategy BHS conducted a second phase of training for religious leaders, and conducted 12 workshops for male religious leaders and one for female leaders to build M&E capacity. Constraints, Issues, and Recommendations The high level of illiteracy generally and especially among women make use of written health education material challenging. Incorporating health messages into religious sermons helps overcome this constraint. With support from BHS religious leaders are playing a critically important role in community health education. They lend credibility to the effort and help reduce cultural sensitivity and increase the acceptability of RH messages. This support should continue and expand. BHS should seek opportunities to share the project’s experience with religious leaders more widely because it has great potential for a broad-based impact on societal norms in health. The Safe Age of Marriage Project Brief Description This is a new activity started in August 2008 with ESD support. BHS selected the Yemeni Women’s Union (YWU) to implement the activity as a pilot in two districts of Amran governorate, with BHS providing technical support and oversight. High rates of early marriage in Yemen pose concerns about early childbirth, with all the associated risks that contribute to higher maternal mortality among these young women. Parliament has debated setting the legal age of marriage at 18 for girls but a proposed new law has not yet been ratified. Social, health, and psychological problems resulting from early marriage have been studied by Yemeni research centers, women’s associations, and committees. Other consequences of early marriage are 20 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION reported to include high rates of divorce and marital insecurity, malnutrition and growth problems for children, abortion and low-birth-weight babies, labor and postpartum complications, and school dropout and illiteracy. Achievement and Progress BHS helped the YWU draft develop training curricula for 40 educators. A knowledge, attitude, and practice (KAP) pre-intervention survey in two districts of Amran governorate (400 families) was conducted and the data analyzed. YWU then initiated community awareness activities based on the information generated. While the survey lacks scientific rigor, the results do give an indication of the problem. It showed that 8.3% of girls were married at 10 to 12 and another 10.5% at 13 to 14. Some changes in attitudes about early marriage were documented, but poverty, traditional values and beliefs, and fear of wrong behavior still result in too many early marriages for girls. Constraints, Issues, and Recommendations There are no issues identified with this activity. The evaluation team was impressed by the leadership and YMU’s Sana’a staff. The team observed a training session for community volunteers in Amran using the new education manual. BHS’s decision to use the YWU was wise. YMU is a strong national NGO with several local branches. It is active on a variety of women’s issues and has the capacity to conduct pre- and post-assessment surveys and implement the awareness program. If successful, this approach could be adopted in other communities. With such sensitive and deeply rooted practices, it will take time to change behavior; however, intensifying awareness using community educators could provide some hope if this pilot effort is scaled up. Besides monitoring the number of people attending educational sessions, BHS should include changes in KAPs about early marriage as a project indicator for IR 5.2. Baseline information is now available for at least a few communities where changes can be documented with a post-intervention survey (see section V). YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 21 V. PROJECT MANAGEMENT ISSUES PERFORMANCE MONITORING PLAN While the Cooperative Agreement with Pathfinder calls for the formal submission to, and approval by, USAID of a performance monitoring plan (PMP) and annual reports, this was not done. However, BHS does report to USAID on a quarterly basis on the indicators shown in Annex 5, including the operational plan (OP) indicators USAID requires. The lack of a formally approved PMP may be because USAID itself has yet to develop its own PMP, though it is planning to do so in connection with the planned USAID/Yemen strategy development process. BHS collects a large volume of data and uses it to manage its program. The indicators reported to USAID (Annex 5) have been input/output-level indicators, such as numbers of facilities renovated or number of people trained, rather than health-outcome indictors that show progress in areas like services utilized or health worker performance improved. Annex A of the BHS Associate Award lists some proposed outcome indicators, such as ―percentage of mothers delivering at facilities who were provided Vitamin A.‖ While the Vitamin A capsules are not provided by the project, the need for Vitamin A supplementation postpartum is part of the midwifery curriculum as well as the Best Practices expansion. Other indicators might include percentage of children aged 12–23 months who received the PENTA vaccine, percentage of mothers in selected communities who deliver with skilled birth attendants, or increases in contraceptive use among communities in targeted areas. Collecting and reporting this kind of information will also help MOHPH or governorate officials focus elements of the program that they control, such as ensuring that a dependable supply of commodities are available at facilities. The project can collect such indicators from the mobile teams and private midwives as well as fixed facilities. Given the current unreliability of MOPHP service statistics, BHS may need to collect this kind of information (perhaps annually) in selected sites through its own coordinators or small-scale surveys, working with governorate counterparts. It is important to ensure that data reported to USAID are accurate. Outcome indicators should be selected to reflect the most important components of the project. Data are already collected using checklists on clinical skills acquired through training, best practices checklists, mobile team monthly reporting forms, and so on. A reasonable set of outcome indicators can give the project, USAID, and the MOPHP a sense of whether project investments are achieving the desired outcomes. The proposed Facilities Management and Quality Improvement activities planned for the extension period and associated indicators include data collection such as ―compliance with MOPHP standards as per checklists filled by supervisors.‖ These are good examples of how the project can quantitatively measure progress on reinforcing supervision systems in the sites where they are working. Besides measuring improvements in service delivery and health worker performance, BHS also must begin to demonstrate that its community mobilization and educational efforts are yielding results that warrant continued funding, from USAID or other partners. The numbers currently reported give a sense of the magnitude of the work; many of the increases in numbers of people reached are truly impressive. The Yemeni Women’s Union plans to repeat its small KAP survey related to early marriage. BHS should consider a small number of indicators that could measure changes in knowledge and attitudes among populations targeted by NGOs like YWU and others. When the evaluation team met with the leaders of the Al Saleh Foundation, they seemed very interested in assessing the impact of their programs. The Yemeni Safe Motherhood Alliance is already doing so through their DFID-supported project on Safe Motherhood and delaying age of marriage. 22 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION BHS should finalize a PMP for the extension period as soon as possible and seek formal USAID approval. This will also be highly useful to USAID, not only for project oversight purposes but perhaps also as input into its own PMP. PROJECT MANAGEMENT AND STAFFING The BHS team works in a highly collaborative and supportive office environment where each has clear roles and responsibilities. The project is valued by MOPHP officials, at both national and governorate levels. This is probably because BHS leadership has ensured that the project is highly responsive to the needs of each governorate as well as the national Deputy Minister for the Population Sector. The team believes that BHS needs to liaison more with the Director General for PHC on child health issues and to ensure harmonization of efforts underway, supported by the GAVI Health Systems Support (HSS) program, which is attempting, for example, to develop an integrated supervision checklist for the country. At the same time, anchoring the project with the Population section seems entirely appropriate given its emphasis on RH/FP and maternal/newborn health. The evaluation team believes that BHS should expand its use of external consultants or Yemeni consultants with international experience (and even long-term staff if necessary) to help build technical and clinical capacity during the extension period. As the project engages more intensively on capacity development in its geographic areas, training and skills development approaches need to become more standardized and include approaches that reflect internationally recognized evidence-based best-practices for RH/MCH. With less emphasis on renovation and refurbishing facilities, funds should be freed up to engage more technical assistance for this work. YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 23 VI. LESSONS LEARNED Important lessons are emerging from the BHS experience. Those summarized here are just a sample of those identified by BHS staff and emerging from the evaluation. Community Midwife Selection and Training: An essential element in ensuring that midwives who are trained can be employed in the communities where they are needed is a purposeful selection process. Candidates having the required educational attainment, e.g., secondary school graduates for Community Midwifery (certificate) training, should be identified within the communities where they are expected to serve. Due to cultural factors, women’s trust and acceptance of midwives depends on their coming from the same community. Changing Health Norms: BHS has learned that the most effective efforts to change societal norms, especially those related to culturally sensitive subjects such as age of marriage or girls’ rights to education or health services, are those led by Yemeni organizations that are accepted by the community. There is significant resistance to change associated with ―foreign ideas‖ that are viewed as contrary to Yemeni traditions, culture, and religious values. While there is still rigorous debate about a legal minimum age of marriage, it is a debate underway within Yemeni institutions and among Yemeni individuals. BHS has sought to work with religious leaders who share the MOPHP’s views about reproductive and child health, and has provided a mechanism at the community and national level for their engagement in community health education. Advice and guidance coming from religious leaders especially seems to have a much better chance of being accepted. Accelerating Health Improvements in Yemen: The health system in Yemen is relatively underdeveloped and financed, very poorly, from local budgetary sources. In many developing countries, donors are reluctant to pay for recurrent program costs, expecting that the funds should come from the country’s own resources. In Yemen, to support improved access to quality health services, development partners will need to pay many of these costs until fundamental reforms make these resources available. USAID needs to allow projects like BHS to pay for travel and per diem costs, for instance, to help governorate health officials engage in supervision or training activities to improve service quality. BHS already pays many of the running costs of the mobile teams; the MOPHP covers personnel and drugs. While it is always preferable to get as much MOPHP budgetary commitment as possible, achieving important outcomes for the people and building health systems to support services cannot wait for local budgetary support that may not soon be forthcoming. As local Dr. Abdul Ghani Al Ghuzzi, Amran Governorate Health Director, and Dr. Nevine Hassanein, BHS evaluation team member 24 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION capacity develops and the impact on the lives of people becomes more evident, gradually the Yemeni government can assume the recurrent costs. While it is always important to think about program sustainability, that should not be the primary concern for critical health programs in Yemen for the foreseeable future. YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 25 VII. FINDINGS AND RECOMMENDATIONS FOR BHS EXTENSION POSITIVE POINTS BHS project design is appropriate for achieving project objectives and contributing to USAID/Yemen’s Strategic Objective of ―Increasing Use of Reproductive, Maternal and Child Health Services.‖ BHS’s proposed shift of emphasis during the extension period from renovation and refurbishing health facilities to more intensive capacity building is appropriate and in line with evaluation team recommendations. Fortunately, the current MOPHP leadership in the Population Section is strong, inclusive of partners’ support, transparent, and very committed to RH issues. Establishing the RH Technical Group to coordinate development partner inputs has been extremely useful for all. BHS is a full partner with the Population Section on technical and policy matters, and with its interventions at the governorate level. There is great appreciation by MOPHP at national and governorate levels for the refurbishment and renovation work BHS has done to improve health facilities. Mobile teams, and the project activities of upgrading and equipping health facilities, have helped improve accessibility to MCH/RH services and fill a gap in some governorates that are subject to tribal conflicts, have the worst health indicators, and are the most underserved. Governorates that receive no other donor support particularly value BHS activities. BHS has provided much needed support for pre-service training for community midwives. Selection criteria that ensure midwives are selected from the communities they will live and work in helps overcome deployment and staffing problems. Another important BHS contribution is the Integrated Community Health Education Manual (42-message flip chart) developed in partnership with several donors and the PHC and Population sections. The manual streamlines existing messages related to MCH, FP, and other important topics. That all partners felt ownership was evident. The Best Practices pilot at Al Sabeen hospital is now being expanded to six additional hospital sites in four new governorates and is emphasized in the community midwifery training; it represents the evidenced-based, low-cost, high-impact interventions that are being promoted by the global community and recommended as essential material by the MCH and PRH divisions at USAID/W as essential maternal, family planning, and newborn care interventions that should be scaled-up. This activity represents a good model of global projects (ESD) and country program support and collaboration. Improvement Collaboratives, as part of the Best Practices program, are potentially a highly effective way to expand the program. While it is still too early to judge their success, they should offer a major important incentive to continue the work, scale up the interventions, and build hospital capacity. BHS is supporting other promising new initiatives, with ESD collaboration, such as the private sector midwife initiative, work on preventing early marriage, and collaboration with religious leaders. 26 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION Religious leaders are extremely important to community health education. They not only help the health education volunteers but also engage in broader advocacy for reproductive and child health. They lend credibility to the effort, help reduce cultural sensitivity, and increase the acceptability of interventions. The BHS team works in a highly collaborative and supportive office environment where everyone’s views are respected. The staff feel supported by the leadership and understand their role in the organization. The BHS team was highly organized in making documents and reports available to the evaluation team, both electronically and as hard copies. RECOMMENDATIONS Result 5.1: Increased Access to Quality Health Services and Participation at the Community Level National, Governorate and District Levels BHS’s principal counterpart at the national level is the Population Section, which is appropriate. However, because BHS is an integrated RH/MCH project, it must improve liaison with the MOPHP PHC Sector to ensure that child health interventions are coordinated. (BHS should not to be limited by MOPHP bureaucratic conflict and lack of consensus.) Whenever BHS is working in the same governorate with other partners, an RHTG mechanism led by the Director General is needed to ensure synergy, avoid overlap, share experiences, and use of the same tools and strategic approaches. Capacity for RH supervision needs to be built in district RH focal points as well as at the governorate RH coordinator, using existing supervision tools, such as the supervision system of UNICEF related to the three-week course on safe home births, and eventually the MNH supervision manual and checklist. Training BHS should consider using MOPHP-approved in-service training materials whenever feasible where the curricula are of adequate quality. Staff should look at the three-week competency￾based home based MNH training course for community midwives developed by UNICEF and approved by MOPHP, which will also be utilized by the MNH project. Since the program covers child health and nutrition, BHS should consider collaborating with the PHC on presenting the 16-day integrated (IMCI, Nutrition, TB, malaria, and RH) course. The PHC Section is scaling the course up with the GAVI/HHS program in 64 districts. Using non-standardized training materials developed by BHS and university professors may not ensure uniform skills or even state-of-the-art clinical information. Linking with UNICEF for TOT capacity development would ensure good trainers for the BHS program. Master trainers from the PHC section could also be explored for presenting BHS programs. The community midwives are highly important to improving maternal and newborn care. More and better follow-up of all BHS-supported trainees, developing supervision systems, refresher training and skills development, and reliable systems for supervision of private sector midwives with referral linkages, and expanding the program are critically important. In governorates where no other donors are doing it, it would be useful for BHS to look into management capacity development and leadership training using curricula such as the HIHS YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 27 (GTZ-supported short courses) for managers at the governorate, district, and facility level who have not previously been trained. It appears that such plans were developed but not implemented. It is crucial to follow up on maintenance for health facilities refurbished by BHS, especially equipment repair. If the plan with the DOD works, it would be adequate, but it would also be important to check with and link to MNH plans and systems. Facility rehabilitation was a huge investment by USAID that needs to be supported by BHS. Best Practices As Best Practices is scaled up, it should be promoted as an essential component of the broader essential obstetric care program that the MNH program is organizing. It must be put in the right context, because certain simple interventions can easily be scaled up but it should not substitute for or undermine other interventions for MNCH and FP services. Scaling up Best Practices in the five core governorates as well as the new ones should be considered, as should including AMTSL as an intervention in hospitals as well as in clinics and midwife-assisted home deliveries. Hospitals are the site for pre- and in-service training of midwives (with or without BHS support). AMTSL should be added as the seventh Best Practice. In the expansion phase of Best Practices data collection and indicators need to be streamlined to make further scale-up feasible considering the limitations of very busy hospitals and their capacity to collect and use information. ESD should reconsider requiring quarterly reports from BHS and data collection sheets for individual women in facilities. The scale-up experience should or course be documented but if it is treated as a research study it cannot be replicated in Yemen. Mobile Teams The planned increase in the number of mobile teams is a good idea given the increase in client rates and the contribution the teams are making to MCH in Yemen. Training for mobile team midwives needs to be reinforced with needed list and MVA and implants for team doctors. The drug list should also be expanded to include, e.g., oxytocin and magnesium sulfate. The costs associated with mobile clinics relative to patient loads need to be analyzed in order to give the population sector information that will be important for future sustainability. Other Activities BHS should focus on hand-washing and hygiene as a best practice for newborns and in its community education activities; it fits nicely with current activities. But the evaluation team recommends that BHS not expand as planned into a new activity on water quality and safety (filters, etc.). While that is important for Yemen, its complexity and the management effort required must be examined carefully. In general the team believes that the BHS project needs to intensify its efforts on capacity building and addressing such elements of the health system as supervision and quality of services to ensure maximum impact of the investment in current governorates and current with initiatives rather than take on completely new activities. 28 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION However, BHS should link with other current or planned USAID-supported activities in education, employment generation, the water sector, etc. Sharing its health education materials, strategies, and approaches would improve synergy and consistency within the USAID program. Result 5.2: Increased Health Knowledge and Improved Behaviors at the Community Level The evaluation ream recommends that BHS and USAID: Continue and intensify support for current organizations as planned (NSMA, YWU, Al Saleh, religious NGOs); where possible, shape activities to show specific outcomes rather than just numbers of people trained. Involve facility and district health education officers in training and utilization of the 42- message flip chart and look for opportunities to make sure that the flip chart book is familiar to health personnel and used where possible by all RH partners. The flip chart manual should be available to mobile teams and in all facilities where BHS provides support. CMGs have little capacity. BHS collects data on their activities but they have no connection to MOPHP facilities. Thus, their impact is unknown. Groups need to be encouraged to link to the nearest facility through its health education officer. Indicators on numbers of referrals made or requests for educational sessions could provide evidence of their usefulness. Project Management Issues It is recommended that BHS: Prepare annual reports containing data from the PMP and an analysis of progress, problems, and lessons learned, as the cooperative agreement requires. The quarterly reports are of reasonable quality but an aggregate view is difficult. Develop a clear M&E plan for the extension period and obtain formal approval from USAID, as the cooperative agreement requires. The OP indicators should not substitute for indicators specified included in agreement. The revised PMP should include outcome indicators, such as coverage rates for interventions for target facilities and best practices. It should also include more indicators for KAP changes based on surveys that NGO partners are planning or have conducted for selected geographic areas or population groups. The project needs to involve more short-term consultants with clinical experience and expertise in MNH, which is currently lacking among the Yemeni staff. Relying on the judgment of university professors who may not have state-of-the-art knowledge and have variable training skills is not sufficient. Experienced consultants, especially those fluent in Arabic, can help build clinical technical capacity among project staff and MOPHP counterparts. YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 29 30 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 31 ANNEX 1. DOCUMENTS REVIEWED PROJECT DOCUMENTS ―A Study of Increasing Awareness of Both Sexes of Youth and Families of the Safe Age of Marriage,‖ Yemeni Women’s Union, Amran Governorate, 2009 Associate Award for Yemen Basic Health Services (BHS) Project, dated December 15, 2005 BHS Quarterly Reports (2006–2008 and January–March 2009) BHS Work Plans for 2006, 2007, 2008 and 2009 and activity reports Community Health Education Manual and Flip Chart Technical Proposal for Yemen Basic Health Services Project, Pathfinder International, September 30, 2005 Yemen Basic Health Services Project, Statement of Work, Proposed Extension, 2009–2010, dated October 18, 2008 Weekly Updates (2006–2009) TRAINING MANUALS AND PACKAGES 1. Technical course for midwives: ―Reproductive Services and Maternal and Newborn Health‖ 2. Training on Reproductive Health for Religious Leaders 3. TOT on Reproductive Health for Religious Leaders 4. Clean Delivery for Home-based Births 5. RH-specific Counseling 6. TOT Curriculum on Generic and Method-Specific Counseling 7. TOT materials on Generic and Method-Specific Counseling 8. Best Practices for Midwives at end of pre-service 9. Best Practices in MNCH/FP for Midwives 10. Best Practices Training for Health Facilities Staff 11. Training of Community Educators on Safe Age of Marriage 12. Mapping Skills for Midwives 13. Business Course for Midwives 14. Mobile Teams Training 15. Training in Health Education and Nutrition 16. Training of Midwives in Counseling for RH/FP Pre- and Postnatal Care 17. Referral Guidelines and Training Manual 18. Training Manual for the Community Health Education Package 32 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 19. Improvement Collaborative Training, January 2009 20. Clinical Skills (3 parts) Program / Technical Binders 1. Best Practices Project 2. Equipment Distributed to Date and 2009 Distribution Plan 3. Media Coverage and Press Releases ( 2006–Present) 4. Work Plans (2006–2008) 5. BHS Annual Reports ( 2006–2009) 6. Quarterly Reports (2006–2008) 7. Weekly Updates (2006–Present) 8. Midwives in Private Practice Project (2008– 2009) 9. Religious Leaders Project: Monitoring, Reporting and Educational Materials (2006E2009) 10. Religious Leaders TOT and Study Tour – Activity Report 11. Religious Leaders Activities Reports (1 summary binder + 15 for details) 12. Mobile Teams Guidebook (Draft–A&E) 13. Mobile Teams Needs and Facilities Assessment and Training 14. Training activities related to dissemination of CHE manual (2) 15. Development and production of the Community Health Education Manual 16. Private Midwives Clinics Project 17. National Safe Motherhood Alliance 18. Training database 19. Practical neonatal training Management Binders 1. Cooperative Agreement, MAARD, Financial Reports, Others (2006–2008) 2. Staff and consultant resumes Monitoring and Evaluation 1. Reports from Governorates (sample copies) 2. Matrix of BHS planned and achieved activities, 2006–2009 3. M&E tools (checklists – report formats) and health services statistics YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 33 Other Health facilities atlas for each of the five governorates Samples of BHS educational materials OTHER KEY DOCUMENTS GAVI Alliance, Health Systems Strengthening (HSS) Application, Government of Yemen, May 2007 Maternal and Newborn Health (MNH), Towards a National Programme (October 2008– December 2012), Ministry of Public Health and Population, 20 August 2008 and 2007 Programme Summary National Guidelines for Constructing and Equipping PHC Facilities in Yemen, 2006 Population Reference Bureau, 20008 World Population Fact Sheet for Yemen National Reproductive Health Strategy 2006–2010, Republic of Yemen, Ministry of Public Health and Population, Population Sector, October 2006 ―Yemen and the Millennium Development Goals,‖ World Bank Working Paper Series No. 31, March 2003 Yemen Demographic and Health Surveys (1991 and 1997), Macro International Yemen Family Health Survey, Pan-Arab Project for Family Health, 2003 ―Yemen: Monitoring the Situation of Children and Women, Multiple Indicator Cluster Survey, 2006,‖ Ministry of Health and Population, Pan-Arab Project for Family Health, UNICEF USAID Strategic Objective 5: Strategic Framework and Indicators WRA Baseline findings from MHPS tool (Maternal Health Policy Score)—A Tool for Providing a Quick Assessment of the Policy Environment). 34 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 35 ANNEX 2. PEOPLE INTERVIEWED MINISTRY OF PUBLIC HEALTH AND POPULATION Dr. Jamila Saleh Al-Raiby, Deputy Minister for Population Dr. Ali A. Al-Mudhwahi, Director General of Family Health GOVERNORATE HEALTH OFFICES Amran: Dr. Abdul Ghani Al Ghuzzi, Director General Ms. Huda Jahlan, Reproductive Health Officer Saada: Dr. Hanboonsh Husein: Director General Ms. Nojood Sayfan, Reproductive Health Officer Al Jawf: Dr. Hussein Al Ghanimy, Director General Marib: Not able to attend Shabwa: Dr. Saleh Fedaak, Director General Dr. Naif Ali, Reproductive Health Officer Sana’a Health Office Al-Amanah Capital: Dr. Mohumad A. Baalaui, General Director Ms. Nazek El Azazi, Reproductive Health Officer Mr. Ahmed El Oudhary, Reproductive Health Assistant Officer AL SABEEN HOSPITAL Dr. Gamila El-Moayed, Hospital Director Dr. Nabila Shaif, Deputy Director Dr. Tawfik El-Bossaly, Chief of Ob/Gyn Department Ms. Fatma Saleh, Deputy Chief for Nursing Ms. Sayeda Aly Helal, Counseling Room Trained Midwife Ms. Fawzia El-Sheikh, Counseling Room Trained Midwife Ms. Salima Abdel Galil, Chief Midwife, FP Outpatient Unit HIGH INSTITUTE OF HEALTH SCIENCES Dr. Taha Yahya Al-Mahbashi, Deputy Dean of Academic Affairs Ms. Mona Al Hajri, Public Health Officer FOUNDATION FOR SOCIAL GUIDANCE Sheikh Yahia Najjar, Director Ms. Zal ElHam Mazkour, Religious Morsheda Sheikh Abdallah El-Khadmy, Senior Religious Leader Mr. Ahmed Bdel Rahman, Institute Secretary Mr. Mostafa Lardis, Institute Administrator 36 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION NATIONAL SAFE MOTHERHOOD ALLIANCE Ms. Nafisa H., Health of National Safe Motherhood Alliance and General Secretary, Higher Council for Motherhood and Childhood Ms. Jamila Gh. Al-Sharie, General Secretary Ms. Najwa Abd. Aladhi, Public Relations Officer Ms. Rashida Ali Hamoud Al Nusiri, Legal Reponsible AL SALEH SOCIAL FOUNDATION FOR DEVELOPMENT Mr. Ali Abdulrahman Al-Akwa, General Executive Manager YEMENI WOMEN’S UNION Ambassador Ramzia Abbas Aleryani, President YEMENI MIDWIVES ASSOCIATION Ms. Fatuna Nour El Deeen, Acting Chair of the Board Ms. Suad Saleh, Acting Deputy Chair (BHS) DEVELOPMENT PARTNERS Ms. Fran Roots, Country Representative, Marie Stopes International Dr. Regina Meyer, Programme Coodinator, GTZ Dr. Fawzia Hamed Jaffer, Credit Administrator, World Bank Dr. Jef Heuberger, Coordinator, Maternal and Newborn Health (MNH) Program, PSU Dr. Ali Abdelmegeid, Deputy Coordinator, MNH Program, PSU Dr. Ben Abdullah Kamel, Chief, Young Child Survival and Development Section, UNICEF BHS STAFF Dr. Hamouda Hanafi, Country Director and Chief of Party Dr. Yehya Babily, Chief of Party Dr. Hadi Alhamzi, Facility Management and Quality Improvement Specialist Dr. Ahmed Kamal, Monitoring and Evaluation Ms. Jamila Al Shareie, Community Mobilization Advisor Dr. Ahmed A. Assalaby, Technical Specialist Ms. Fawzia Yousef, Midwifery Specialist Ms. Najwa Al Rabí, BHS Coordinator – Amran Ms. Attaf Ankar, BHS Coordinator – Saada Mr. Mohamed Al Ansi, BHS Coordinator – Al Jawf Ms. Intissar Al Jeweed, BHS Coordinator – Marib BHS Coordinator – Shabwa: not able to attend Ms. Zaafaran Al Haddy, BHS Coordinator – Sana’a YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 37 YEMEN PARTNERSHIP FOR HEALTH REFORM Dr. Abdel Kadr Aly, Country Director CONTACTS FROM AMRAN FIELD VISIT Amran Governorate Health Office Dr. Abdel Ghany El-Ghozy, Director General Dr. Abdallah Korhosh, Amran PHC Office Director Ms. Nagwa Saleh El-Rahby, BHS Amran Governorate Coordinator Amran Health Institute Dr. Yehya Hazaa El-Haety, Director Mr. Mohamed El-Zobeiry, Deputy Director for Educational Issues Ms. Hoda Gahlan, Midwife, Tutor Shams, Assia, Fatheya and Hanan, Midwives, Tutors Amran Governorate Hospital (General Hospital) Dr. Abdel Ghany Faress, Amran Governorate Hospital Director Ms. Fawzia Saleh, Neonatal Nurse (trained midwife) Dr. Ahmed El-Sohy, Neonatologist (trained pediatrician) Dr. Abu Bakr, Neonatologist (trained physician) Dr. Afrah El-Oreify, Ob/Gyn & FP (trained GP) Dr. Zeinab El-Maakhedy, Ob/Gyn & FP clinic Yemeni Women’s Union (NGO hosting Amran Community Mobilization Team) Amran Governorate Awkaf Deputy General Director for Religious Guidance Amran Governorate Health Education Office Director Amran Governorate Media Office Representative Amran Governorate Youth & Sports Office Representative Amran Governorate National Committee Office Representative Amran Governorate Agriculture Office Representative Amran Governorate Social Development Funding Office Representative Member of YWU 3 Religious Morshedas, (Lady Volunteers) Mobile Team on field visit for Beny El-Zobeir Health Unit at Eyal Sareeh District: Dr. Abdel Malek El-Hedy, District Health Office Director Ms. Hayla Mohsen Makram El-Zobeir, Health Unit Midwife (25 years experience) Ms. Amina Hussein El-Zobeir, Health Unit Midwife (newly recruited) Mr. Mohamed Amer El-Zobeir, Health Unit Nurse Ms. Fouad Yehya Saleh El-Zobeir, Health Unit Morsheda Mr. Abdel Ghany El Zobeir, Health Unit Paramedic Dr. Mariam Abd-Raboh Doghany, Mobile Team Physician (highly experienced) 38 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION Ms. Sabah El Zafry, Mobile Team Midwife Ms. Saeeda El-Samawy, Mobile Team Midwife 2 Students (Midwives) from Amran Health Institute Thula District Hospital (Thula Governorate) Mr. Yehya El-Zohery, Thula District Health Office Director Dr. Abdel Malek El-Fahd, Thula District Hospital Director Dr. Waleed Bakhadr, GP, Thula District Hospital Dr. Abdo El-Garashy, GP, Thula District Hospital Mr. Ali El-Ezaby, Lab Asisstant, at Thula District Hospital Mr. Ali Hussein Youssef El-Ward, Administrator, Thula District Hospital Midwife Private Practice Ms. Assia Hamoud Mohamed El-Akwaa (delivered 40 cases in June 2009) YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 39 ANNEX 3. SCOPE OF WORK Global Health Technical Assistance Project GH Tech Contract No. GHS-I-00-05-00005-00 SCOPE OF WORK Yemen Evaluation June 8, 2009 I. IDENTIFICATION OF TASK USAID/Yemen, the Office of Health and Population, requests the assistance of the Global Health Technical Assistance Project (GH Tech) under Task Order No. 01 to conduct an evaluation of the Basic Health Services (BHS) project implemented by Pathfinder International. BHS is an Associate Award to the USAID/W Extended Service Delivery (ESD) project. The Mission is requesting the evaluation to be initiated during the last week of June 2009 and completed by the end of July 2009. II. BACKGROUND Yemen is one of the poorest, least developed countries in the Arab world and ranked 149 out of 177 countries in the 2004 UN Human Development Report. Yemen’s health indicators are among the worst in the world. The fertility rate is 6.2 births per woman, the population growth rate is 3.2%, contraceptive use rate for modern methods is 13.4%, the unmet need of family planning is 51%. These data were reported in the 2008 data sheet of the Population Reference Bureau (PRB). PRB also reported that 25% of women give birth by age 18. The rapid population growth places tremendous strain on the existing systems that provide education, job opportunities, water, and other resources to Yemenis. Though this problem affects many sectors, it directly impacts the health of Yemenis, specifically the health of its women and children, as clearly shown by the high maternal mortality rate of 365/100,000 and the under-5 mortality rate of 102/1,000 live births. The USAID response to health in Yemen is framed in Strategic Objective 5.0: “Increased Use of Reproductive, Maternal and Child Health Services.” The SO 5.0 result will be achieved through two Intermediate Results: IR 5.1 Increased Access to Quality Health Services and Participation at the Community Level, and IR 5.2 Increased Health Knowledge and Improved Behaviors at the Community Level. USAID/Yemen assistance strategy aims to support the Republic of Yemen Government (ROYG) to implement its goals in the health sector and in particular to implement its National Reproductive Health Strategy and National Population Plan, with a particular emphasis on improving Reproductive Health/Family Planning (RH/FP) and Maternal and Child Health (MCH) activities to achieve the MDG goals #4 and #5 through: Supporting prenatal care, birth preparedness, and safe motherhood services and practices, including family planning for healthy timing and spacing of pregnancy; Improving access to emergency obstetric care (EmOC) at district (basic EmOC) and governorate (comprehensive EmOC) levels; 40 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION Prevention and treatment of childhood illnesses; Reducing malnutrition among children (under 5) and improving the nutrition of pregnant and lactating mothers; Strengthening service delivery within the national immunization system; and Improving water quality, sanitation, and hygiene in schools and in homes. Basic Health Services Project (2006-2008) Through the BHS project, the Mission requested Pathfinder International under an associate award to the ESD project to support USAID efforts to assist the ROYG in improving its MCH and RH services in selected governorates. BHS was requested to follow a methodology similar to the one used by CATALYST project in 2004 and to work in close collaboration with Ministry of Public Health and Population (MOPHP) at all levels. Both CATALYST and ESD projects conducted needs assessment visits targeting all districts of all five focus governorates before a selection of priority districts was made. Needs assessments looked at facilities, personnel, social context, institutions, and projects working in these districts and gathered all information relevant to the Basic Health Services program objectives. The selection of priority districts for the BHS took into consideration prior USAID assistance results under CATALYST (for Amran, Marib and Shabwa governorates) and the BHSE /ADRA project (for Saada and Al Jawf governorates). The available data from the health surveys carried out by USAID PHR Plus was utilized as well as recent figures from the national census. USAID support during 2006–2008 through the BHS included upgrading services of health facilities; strengthening pre-service training for midwives; launching community-based medical mobile teams; and developing community education programs that include a strong involvement of religious leaders. Those activities were implemented in five northern and eastern governorates of the Republic of Yemen: Amran, Al Jawf, Marib, Sa’ada, and Shabwa. Planned BHS Activities for 2009-2010 USAID agreed to the request from the MOPHP to expand into new geographical areas; especially Sana’a capital, as well as continue to provide support to maintain the existing programs in the five target governorates served until now. Proposed BHS activities will continue to focus essentially on the delivery of integrated and improved MCH/FP services, including added attention to child health and safe water and hygiene as other essential components of health care and prevention of disease. Pathfinder is proposing to reach approximately 14% of the population of Yemen with culturally appropriate and gender-sensitive BCC messages to promote the use of maternal and child health services and the adoption of healthier behaviors. Pathfinder will facilitate the use of effective community mobilization and empowerment methodologies to involve community members in health care issues and behavior change activities. Activities planned for the current focus governorates of Amran, Al Jawf, Marib, Sa’ada, and Shabwa will include an assessment of facilities that were improved in the last three years in order to identify and address gaps in quality of service. It will also identify needs and gaps with regard to implement new activities needed to improve nutrition, water safety, and hygiene. Currently active community-based mobilization groups and religious leaders in these governorates will also be visited to identify possible new directions and to expand their activities and contribution to cover more services (such as water safety and hygiene) and new geographical areas (more districts). This review will also enable the BHS team to identify youth and women’s groups YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 41 capable of contributing to community education efforts and explore integration opportunities with non-health sectors, such as agriculture, education, and local administration (democracy and governance projects). In the four new governorates (Aden, Taiz, Lahej, and Ibb) where BHS will assist with the scaling￾up of Best Practices, the activities will reflect what has already been agreed by the Yemen Country Team for scaling up Best Practices and the focus will be on one major hospital in each governorate. Activities will be limited to training and minor equipment support for these hospitals. The impact of this activity is expected to benefit the whole governorate, as these hospitals are the main referral hospitals for RH/FP needs and many are also teaching hospitals that serve adjoining governorates. The BHS project is also utilizing financial assistance received from ESD project core funds to implement related initiatives, such as scaling up best practices in MNCH/FP and assisting community midwives set up their private practice in their communities. These activities started in Sana’a in 2007–2008 and will be scaled up in six major hospitals, two of which are located in the governorates of Amran and Saada, which are already among the areas BHS covered until now; and four are in other governorates: Aden, Taiz, Lahej, and Ibb. This is consistent with USAID instructions to expand to new areas in response to demands from the host country while continuing to support and monitor follow-up activities in the governorates where USAID has already invested substantially. In the capital city of Sana’a the BHS will emphasize improving access to EmOC services in primary health care facilities and assisting HC providers to set up and implement a working referral system to serve more people in basic EmOC facilities, including postpartum care and counseling services, and relieve the pressure on major hospitals that provide comprehensive EmOC. The specific activities will be decided after a careful assessment to identify available resources and key strengths and weaknesses related to improving selected PHC health facilities and relevant community-based programs and activities. Other Donor Support in Health The Netherlands and the European Commission support increased accessibility to reproductive health and family planning services. The UNFPA provides family planning assistance and commodities. UNICEF supports immunization activities and will also support maternal and child nutrition starting in 2009 through establishing a community health volunteer system. GTZ supports quality assurance and health sector reform work. III. PURPOSE OF THE ASSIGNMENT 1. Evaluate the effectiveness of the BHS project to date in achieving program objectives and deliverables and contributing to USAID/Yemen Strategic Objective 5.0 of ―Increased Use of Reproductive, Maternal and Child Health Services.‖ 2. Review planned BHS activities for the extension period and confirm that they represents the right approach to addressing the MCH and RH/FP programmatic needs of Yemen. Conduct a gap analysis for the FP and MCH programs of USAID/Yemen. 3. Make recommendations based on findings and gap analysis for any modifications to BHS activities during the extension period and for future USAID programming in MCH and FP. 42 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION IV. SCOPE OF WORK 1. Determine whether the BHS Project is achieving intended goals and results given program design, operating assumptions, and management structure. a. Is the program design of the BHS project appropriate to achieve the Mission’s Strategic Objective 5 – ―Increased Use of Reproductive, Maternal and Child Health Services”? b. Has the project achieved its yearly targets? To what extent has the BHS project met its goals and objectives? c. Assess the quality of project monitoring and evaluation activities; are data-gathering methods reasonable for monitoring progress and indicators? d. Were there specific project management policies, structure, or practices that contributed to either the success or failure of intervention implementation? 2. Assess the effectiveness of key technical components, capacity building, sustainability, effective partnerships, and approaches of the BHS Project. a. Has the project impacted on national policy related to FP/MCH? or it was only focused on the five target governorates? b. Has the BHS project strengthened the technical or management capacity of MOHP staff at various levels? c. Are the activities supported by BHS representing the right mix to address MCH/RH technical assistance gaps in Yemen? d. Are the MCH and RH activities presented in a sound integrated package? Is it in line with the MOHP strategy for integration? Has the project been successful in achieving impact in both sectors (MCH & FP)? e. Assess the quality of the training materials and courses offered to various health providers for FP, MH, and child health. f. Assess the quality of MCH/FP services at the upgraded health facilities (primary health clinics or hospitals) supported by BHS. g. What are the clinical standards used to upgrade facilities and train staff? h. Is the quality of services provided at the mobile clinics good? Are coverage and utilization high? Address reach, sustainability, and cost-effectiveness if possible. i. Assess the contents of BHS behavior change communication messages, its reach, and its relevance to the program components and target audience. j. Assess the quality and impact of community activities targeting religious leaders. k. Has the BHS project facilitated synergy, coordination, and information-sharing among other donors, and the Government of Yemen at all levels? l. Has the BHS coordinated with other sectors such as education and governance and addressed governance and gender issues in the health field? YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 43 3. Assess the BHS workplan for the extended period (2009-2010). a. Examine whether the new statement of work is built on lessons learned and best practices. b. Are the proposed assessments for districts and facilities following a sound methodology to decide on project sites? c. Are the activities proposed in the extension period appropriate and do they address key technical priorities and USAID/W strategic direction for MCH and FP. (Reference: USAID Report to Congress, 2008) d. Review and assess the Best Practices package implemented by the ESD Grant in Sana’a hospital and determine usefulness for replication. Should the package include additional interventions? e. Meet with UN and bilateral donors active in Yemen who are supporting MCH/FP activities. Identify any possible areas of overlap and opportunities for collaboration and synergy. 4. Identify lessons learned and recommendations for any modifications to the BHS program and future USAID programming related to MCH and FP. a. Based on current project experience and lessons learned, what are the essential activities that should be continued, expanded, or dropped over the life of the project? b. Provide recommendations for possible modifications to project approaches and activities. Recommendations should be based on global technical learning, Yemen programmatic needs, donors’ contributions, USAID/W technical priorities, and the newly developed strategy of USAID/Yemen. c. What policy and program gaps does the Mission need to address related to high fertility and child and maternal mortality that should be considered in the current program and for future USAID investment? V. METHODOLOGY Review relevant documents including policies, guidelines, strategic documents, relevant program reviews, evaluations, national surveys, annual operational plans of national programs, governorates and operational districts, and reports, etc. Review project documents: proposals, implementation plans, monitoring and evaluation plans, progress reports, review/evaluation reports, training curricula, etc. The team will conduct a two-day team planning meeting (TPM) upon arrival in Yemen and before starting the in-country portion of the evaluation. The TPM will review and clarify any questions on the evaluation SOW, draft an initial work plan, develop a data collection plan, finalize the evaluation questions, develop the evaluation report table of contents, clarify team members’ roles, and assign drafting responsibilities for the evaluation report. The TPM outcomes will be shared with USAID/Yemen, and the health team will participate in sections of the TPM. Field operations observations. Interview key informants, including USAID Mission management/staff and key Ministry of Health partners (MOH, national programs, Governorate, District and local health teams); key 44 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION health partners (Netherlands, GTZ, WHO, UNFPA, UNICEF, JICA); and relevant NGO partners. VI. TEAM COMPOSITION The assessment team will be comprised of two individuals, including one team leader, who are independent consultants with the following mixture of expertise and experiences: A team leader with experience leading an evaluation team; Strong technical knowledge on maternal, reproductive health/family planning, and child survival; At least 10 years experience of program planning, implementation, evaluation, and design in MCH and FP; and Experience managing and/or working with USAID-funded projects, with significant understanding of USAID contracting systems. Familiarity with the international donor environment is desirable. Members from the MCH and FP/PRH divisions of the Bureau for Global Health will join the team. VII. LEVEL OF EFFORT (LOE) USAID/Yemen anticipates that the period of performance of this evaluation will be approximately one month. This will include preparation days, in-country work in Yemen and the governorates, and report writing and finalization. The evaluation should begin on or about June 28, 2009. The team leader will be responsible for the overall planning, design, and implementation of the evaluation and work coordination among team members. It will be the team leader’s responsibility to submit a satisfactory report to USAID within the agreed timelines. The team leader is responsible for report writing and organization of the debriefing presentations. The final report should be made available within two weeks of the receipt of comments on the draft report from USAID and partners. Program schedules for field visits shall be provided before or on arrival in Yemen. Task/Deliverable Team Leader Technical Specialists (TCN) 1. Review background documents and offshore preparation work 3 days 3 days 2. Travel to Yemen 2 days 1 days 3. Team planning meeting and meeting with USAID/Yemen 2 days 2 days 4. Information and data collection. Includes interviews with key informants (stakeholders and USAID staff) and site visits 7 days 7 days 5. Discussion, analysis, and draft evaluation report in country 2 days 2 days 6. Debrief meetings with USAID and key stakeholders (preliminary draft report due to USAID) 1 day 1 day YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 45 Task/Deliverable Team Leader Technical Specialists (TCN) 7. Team leader meeting with USAID to synthesize findings/discussion 4 days 8. Depart Yemen/travel to U.S./other 2 days 1 day 9. USAID and partners provide comments on draft report (out of country) – 10 working days 10. Team revises draft report and submits final to USAID (out of country) 3 days 3 days 11. USAID completes final review 12. GH Tech edits/formats report (one month) Total Estimated LOE 26 days 20 days VIII. LOGISTICS USAID/Yemen will provide overall direction to the evaluation team, identify key documents, and assist in facilitating assignments. USAID/Yemen will assist in arranging meetings with key stakeholders as identified by USAID prior to initiation of field work. The evaluation team is responsible for arranging other meetings as identified during the course of this evaluation and advising USAID/Yemen prior to each of those meetings. Field visits to at least two of the target governorates will be scheduled. The Mission will process RSO clearances for the team. If for any reasons the RSO will not approve the field visits, the Mission will arrange for governorate, district, and clinic level staff to conduct interviews in Sana’a. A six-day work week is authorized while the team is in Yemen. GH-Tech will be responsible for all off-shore and in-country logistical support. This includes arranging and scheduling meetings, international and local travel, hotel bookings, working/office spaces, computers, printing, and photocopying. IX. DELIVERABLES AND PRODUCTS Work Plan: During the team planning meeting, the team will prepare a detailed work plan, which will include the methodologies to be used in the evaluation. The work plan will be submitted to the CTO at USAID/Yemen for approval no later than the sixth day of work. Methodology Plan: A written methodology plan (evaluation design/operational work plan) will be prepared during the team planning meeting and discussed with USAID prior to implementation. Discussion of Preliminary Draft Evaluation Report: The team will submit a rough draft of the report to the USAID CTO and PH team, who will provide preliminary comments prior to final Mission debriefing. This will facilitate preparation of a more final draft report that will be left with the Mission upon the evaluation team’s departure. Debriefing with USAID: The team will present the major findings of the evaluation to USAID/Yemen through a PowerPoint presentation after submission of the draft report and before the team’s departure from country. The debrief will include a discussion of achievements and issues as well as any recommendations the team has for possible modifications to project 46 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION approaches, results, or activities. The team will consider USAID comments and revise the draft report accordingly, as appropriate. Debriefing with Partners: The team will present the major findings of the evaluation to USAID partners (as appropriate and as defined by USAID) through a PowerPoint presentation prior to the team’s departure from country. The team will consider partner comments and revise the draft report accordingly, as appropriate. Draft Evaluation Report: A draft report of the findings and recommendations should be submitted to the USAID CTO prior to the team leader’s departure from Yemen. The written report should clearly describe findings, conclusions, and recommendations. USAID will provide comment on the draft report within two weeks of submission. Final Report: The team will submit a final report that incorporates team responses to Mission comments and suggestions no later than five days after USAID/Yemen provides written comments on the team’s draft evaluation report (see above). This report should not exceed 30 pages in length (not including appendices, lists of contacts, etc.). The format will include an executive summary, table of contents, methodology, findings, and recommendations for future investment in the health sector, including MCH/RH. The report will be submitted in English, electronically. The report will be disseminated within USAID. A second version of this report excluding any potentially procurement-sensitive information will be submitted (also electronically, in English) for dissemination among implementing partners and stakeholders. The final report will be edited/formatted by GH Tech and provided to USAID/Yemen approximately one month after the Mission has reviewed the content and approved the final revised version of the report. This final revised version of the report can be used as a working document while final report editing/formatting is in process by GH Tech. X. MISSION AND/OR WASHINGTON CONTACT PEOPLE/PERSON Dr. Iman Awad, Health Team Leader, Health and Population Office, USAID Yemen Dr. Nahed Matta, Yemen Country Coordinator, GH/HIDN XI. COST ESTIMATE (ATTACHED) XII. REFERENCES USAID Report to Congress on FY 08 MCH appropriation USAID Strategy for Yemen BHS Proposal BHS Statement of Work (2009–2010) BHS Annual Workplan 2007, 2008, and 2009 BHS Quarterly and Annual Reports Others TBD YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 47 ANNEX 4. WORK PLAN Yemen Basic Health Services (BHS) Project Mid-Term Evaluation Work Plan ANALYTIC FRAMEWORK Framework Reference Documents Pathfinder Cooperative Agreement 06-09 Annual Work Plans Summary of Program Inputs, Baseline and Targets (2008 update) Evaluation Tasks 1. Evaluate the effectiveness of the BHS project to date in achieving program objectives and contributing to USAID/Yemen’s Strategic Objective 5 of ―Increased Use of Reproductive, Maternal, and Child Health Services.‖ 2. Review planned BHS activities for the extension period and determine whether they represents the right approach to addressing the MCH and RH/FP programmatic needs of Yemen, taking into consideration support from other donors in this field and USAID assistance goals and objectives in the health field. 3. Make recommendations based on the evaluation findings about any modifications needed in the BHS project during the extension period and for future USAID programming in MCH and FP. Sources of Information Needs assessments and available data (KAP, etc) Program reports and project documents Training and educational materials Work plans and quarterly reports Interviews with MOPHP managers Interviews with service providers Interviews with governorate health officials Interviews with BHS staff Interviews with development partners Interviews with clients (where possible) Observations during field visits 48 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION Methodology Analysis of available data and documents Interviews using question lists with project implementers, MOPHP officials, donors, USAID, NGO implementation partners, project staff, and other stakeholders Observations (where feasible) of service provision, facilities (fixed and mobile), and communities on site visits Attend Reproductive Health Technical Group meeting (policy and program discussions). QUESTIONS FOR BHS STAKEHOLDERS AND PARTNERS MOPHP Central Staff How has the BHS project helped most in terms of assistance to the central MOPHP? What do you believe has been the most useful contribution to the Governorates? Is the project supporting anything you feel is of questionable value? As an integrated project supporting the MOHPH, has BHS contributed enough to child health in areas, such as Vitamin A supplementation, for example? How about the EPI program? How do they help? How well does BHS work with other development partners? How harmonized are the approaches of the development partners? Are there areas of difference? How are you involved with the BHS in terms of planning activities or approving work plans? Are you satisfied with BHS reporting on their activities? Are there any improvements you would like to see? What capacity-development activities has the BHS project undertaken at the national level? Do you meet with USAID independently to discuss BHS progress or problems? How do you value the clinic renovations and housing? What do you think about the mobile team effort and how can that be institutionalized? How do you feel about the private midwives initiative with the Yemen Midwives Association? Can that be expanded? Do you think it is better for BHS to focus on existing technical areas and work more intensively on those efforts or expand areas as proposed (water and nutrition)? Are there elements of the project that if introduced/piloted by BHS can be expanded by the MOPHP with help from other partners? Given the current weakness of the health information system, how do you monitor progress toward achieving MDG goals in FP and MCH? Do you believe that another Family Health Survey would be useful at this juncture? Have plans been made to move forward with one? YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 49 When BHS ends, what kinds of activities do you think the MOHPH will be able to continue with? How can they be sustained? What steps have been taken to establish quality assurance systems in Yemen? What kinds of supervision systems are in place and for what programs? Does the BHS project help in these areas? As special priorities, initiatives, or emergencies come up, do they detract from routine programs given the lack of manpower? How are these issues dealt with? Governorate Health Staff (Directors General and Reproductive Health Directors) How specifically has the BHS been supporting your work at the governorate level? How involved have you been in making decisions about the priority activities of BHS in your governorate? Who develops the needed equipment lists or decides which facilities need to be renovated? How useful has the support been? What is the most useful aspect of the project? How has it made a difference to your work? Are there other development partners working in the governorate and is there any overlap in activities? How important is the role of the BHS Coordinator, and if it is beneficial, why? In what ways does BHS build the capacity of staff at the governorate and district levels? Do you report to the central MOPHP on a quarterly basis on key indicators related to the Millennium Development Goals? Do you also report on the six reproductive health indicators? Do you report on the child health indicators? How do you think the BHS is helping increase utilization of health services? How do they do that? How do you know that there is increased utilization? Do you have supervision systems in place and how do they operate? For what programs? What is the role of the Governor in health services? Does he support your activities in some way? Do you work with Local Councils in some way? How important are they in your governorate? How does communication happen between districts and between the district and governorate levels in your area? Is there any sharing of experience and if so, how? How is BHS mobilizing communities and improving health knowledge and behavior among the population in your governorate? How is that useful to your programs? How useful is the mobile team work? How do people accept it? Do you think that you will be able to pay for the gasoline and vehicle maintenance in the future when the project finishes? Is the renovation work useful? Are the renovated buildings being maintained and is there a system to maintain them? How do you deal with the money collected from the facilities? How is it currently used? Is anyone helping your staff with management/leadership/financial training? 50 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION What do you think of the new multisectoral Governorate Community Group for conducting health education? Should it expand to other areas? DEVELOPMENT PARTNERS (WHO, UNICEF, UNFPA, MNH [Netherlands], GTZ, World Bank, Marie Stopes International, Yemeni HS 20/20 [USAID Project]) What are your focus areas for assistance in health? Do you see any overlap with the BHS or other donors? Do you have any presence in any of the governorates? Are they the same as the BHS project? What kind of relationship do you have with the BHS other than the central monthly meeting and donor breakfast? What activity of the BHS do you find the most valuable? Why? Should it continue or change in any way? Do you believe that another Family Health Survey is needed? Are you planning on providing support? We understand that there are many challenges and constraints in the health care system in Yemen because some basic components are missing, such as supervision systems, referral, quality improvement, etc. How do you manage with these constraints in your own program? How do you work to increase the capacity of the systems given these constraints? What do you think are future areas of investment for USAID in strengthening the national health care system? USAID (CTO and Mission Director) How do you monitor the BHS project? How has your field monitoring been affected by travel restrictions? What are the strengths and the weaknesses of the project? How stable is USAID funding for Yemen? What can you tell us about future operating year budget levels? Do you have any specific questions about BHS or overall guidance that we should know about? What do you think are the priorities for the future in the health sector for USAID investment? How does this mesh with MOPHP priorities? Why has the renovation work had to be curtailed due to budget problems from USAID? How do you feel about the proposed expansion of technical components within the extension phase of BHS? How do you feel about the proposed Best Practices activity being implemented in facilities outside of the original five governorates? YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 51 BHS Staff (Central and Governorate) What are the most important accomplishments to date by the BHS project? Why have they have been successful? What are the biggest challenges you face? You are working in the most disadvantaged areas of the country. What are the constraints and how are you addressing them? Have you had travel restrictions and how has that affected your work? Have USAID’s travel restrictions for their own staff adversely impacted on the BHS in any way? Are you comfortable with the project’s policies and systems? Is there sufficient communication among various staff about the work? Do you share in solving problems or discussing issues? Is BHS trying to accomplish too much given the challenges of working with a poorly developed health system? What changes, if any, do you recommend? How has the poor health information system and data collection for the country affected your work? Are you doing something to improve it? How do you know that your project interventions are achieving the expected outcomes? How do you disseminate this information? Do you feel that your work is appreciated by the people at the governorate level? National level? District or community level? BHS NGO Partners (Yemen Midwifes Association, National Safe Motherhood Alliance, Al Saleh Foundation, Yemeni Women’s Alliance, Foundation for Social Guidance [religious NGO]) What are your main activities in the health area? What is your relationship to the BHS project? What has been the most important outcome of your relationship with BHS? How has the BHS relationship helped you achieve your activities? Are you supported by other organizations or donors? What kinds of support do you need from the BHS in the future that you are not already getting? Has capacity been developed in your staff from the association with the BHS? Please illustrate specifically? Do you believe your activities with BHS are valued by the MOPHP? Do you have a direct relationship with the MOPHP or do you work through the BHS staff? Has the BHS helped you link with other NGOs in ways that have been helpful? 52 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION Clients at BHS Facilities (Hospitals/Centers/Units) or at Home What kind of service did you receive today? Did you receive it when you went to the health facility? Did you come here before? If yes, when and what changes do you feel happened? If no, why not before and what encouraged you to come today? Do you receive any health advice at the facility? If yes, who gave it to you? What kinds of health information do you receive from radio, TV, or magazines? Does the mobile team come to your area? If yes, how do you know of the visiting day? Do you feel they help you and your community? Do you know how many times they come to your area? Do all family members benefit from that visit? How far is the nearest health facility when you need medical assistance? Are you happy with what is offered through the health facility? If no, what else do you need? What do you think of the midwife being able to provide other services than delivery? Have you received any service from a midwife other than assistance with childbirth? Do you know about the disposable delivery kit (box)? Service Delivery Staff Trained by the BHS Project In what ways have you been involved with BHS project activities? How much have you benefited from them? What areas do you feel the BHS project should help you with beyond training? Do you know of other interventions by other organizations that are similar to BHS activities? Do you think the BHS interventions done at your facility were essential and comprehensive? How do you feel about the quality of services in your facility, and why? Did the BHS improve the standard of services, and how? Are you involved with facility funds? What do you know in relation to this issue? What do you know of the health education activity of the BHS project? Do you feel any changes in behavior of people or their demands? Evaluation Deliverables Product Date Work plan/Methodology July 4, 2009 PowerPoint presentation July 11, 2009 First full draft report July 15, 2009 Corrected draft report (Within 2 weeks of USAID comments) Final edited report (Within 1 month of USAID approval of final version) YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 53 ANNEX 5. PROGRESS ON M&E INDICATORS INDICATORS Baseline (Dec 05) (Catalyst) Achieved (Dec 08) Notes IR 5.1: Increased Access to Quality Health Service and Participation at the Community Level Sub-IR 5.1.1 Health facility refurbishments completed in all five governorates. Number of project health facilities renovated or constructed 21 45 24 sites completed during BHS 2006-08 Includes new 14 extensions for normal delivery and 1 comprehensive EmOC centers Number of project health facilities equipped 25 129 104 completed during BHS 2006-08 Medical equipment varies from complete operation theater to basic equipment for small health units Sub-IR 5.1.3 Increase in number of staff from project health facilities trained in delivery of quality FP/MCH services. Number of project health facility participants trained 531 1,458 927 trained during BHS 2006-08 Short-term training for doctors and midwives Number of facilities that have staff housing attached 3 20 17 housing units built by BHS; each has 2 apartments 23 = Number of existing housing units that were furnished by BHS IR 5.1: Increased Access to Percent of clinical facilities in target areas with adequate supply of contraceptives as per MOPHP standards N/A N/A Sub-IR 5.1.6 Mobile teams established and delivering services. 54 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION Number of clients served by mobile teams 22,576 206,302 183,726 clients served during BHS 2006-08 Number of mobile teams activated 3 10 7 mobile teams activated during BHS 2006-08 Number of health facilities served by mobile teams 60 200 140 health facilities served during BHS 2006- 08 Number of mobile team members trained / retrained 20 59 39 mobile teams members trained during BHS 2006-08 Sub-IR 5.1.7 Trained community midwives (CMs) implementing rural outreach activities. IR 5.1, cont’d Number of midwives trained (short-term) by BHS in essential RH/FP/MCH skills 383 799 416 midwives trained (short-term) during BHS 2006-08 Number of midwives in pre-service training 123 340 217 midwives trained in pre-service training during BHS 2006-08 IR 5.2: Increased Health Knowledge and Improved Behaviors at the Community Level Sub-IR 5.2.1 Health education messages developed and disseminated in support of project interventions. Number of health education messages developed, produced, and distributed 7 27 42 2 1 7 messages printed on posters; total of 14,000 copies during Catalyst 2004-05 20 Messages printed in posters for total of 60,000 copies during BHS 06-08 Health manual includes 42 health messages; 2,000 copies printed during BHS 2006-08. 2 brochures targeting the audiences of religious leaders; 50,000 copies of each printed during BHS 2006-08. 5,000 copies printed of mothers’ educational booklet on Best Practices during BHS 2006- 08. Sub-IR 5.2.2 Systematic counseling services in RH/MCH strengthened. Number of staff trained in counseling 120 194 74 staff trained during BHS 2006-08 Sub-IR 5.2.3 Community outreach workers, local council members, teachers, and agriculture extension workers mobilized and delivering health messages. Number of persons trained from non-health sector (religious leaders, educators, etc.) 0 202 146 male and female religious leaders and 56 community mobilization volunteers YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION 55 USAID OPERATIONAL PLAN (OP) INDICATORS (2008 ACCOMPLISHMENT AND 2009 TARGETS) Indicator (in USG-Supported Programs) 2008 Achieved 2009 Target Number of health facilities rehabilitated 53 12 Number of people trained in child health and nutrition 480 150 Number of people trained in maternal/newborn health 527 450 Number of children <1 year receiving PENTA vaccine 3rd dose 73,356 75,000 Number of households of children 0–23 months who received a water filter 0 (new) 1200 Number of counseling visits for FP/RH 32,791 35,000 Number of service delivery points providing FP counseling 209 220 Number of couple years of protection achieved at FP delivery points 17,403 20,000 56 YEMEN BASIC HEALTH SERVICES (BHS) PROJECT: MID-TERM EVALUATION For more information, please visit http://www.ghtechproject.com/resources.aspx Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 www.ghtechproject.com