MID-TERM EVALUATION OF THE USAID/PAKISTAN MATERNAL, NEWBORN AND CHILD HEALTH PROGRAM SEPTEMBER 2008 This publication was produced for review by the United States Agency for International Development. It was prepared by Pinar Senlet, Susan Rae Ross, and Jennifer Peters through the Global Health Technical Assistance Project. MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 08-001-087) is available in printed or online versions. Online documents can be located in the GH Tech web site library at www.ghtechproject.com/resources/. Documents are also made available through the Development Experience Clearing House (www.dec.org). Additional information can be obtained from The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-00. MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT i ACKNOWLEDGEMENTS The consultants would like to acknowledge the support they received from the staff of USAID/Pakistan, the PAIMAN Project, and Ministry of Health officials in conducting this evaluation. The team expresses special appreciation to Bill Conn, USAID, for his guidance throughout the evaluation process, and Nabeela Ali, PAIMAN Project Chief of Party, for making herself, her staff, and project information so readily available to the team. Our special thanks go to our partner local team, Eycon, for their outstanding assistance in organizing our in-country schedules and conducting interviews in districts where we could not travel. The team also greatly appreciates the contributions from representatives of other international organizations who generously gave their time to talk with the evaluation team. ii MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT iii ACRONYMS AMTSL Active management of the third stage of labor ANC Antenatal care BCC Behavior change and communication BEmONC Basic emergency maternal and newborn care CAM Communication, advocacy, and mobilization CCB Citizen Community Board CDK Clean delivery kits CHW Community health worker CIDA Canadian International Development Agency CMW Community midwife CEmONC Comprehensive emergency maternal and newborn care COP Chief of Party DFID United Kingdom Department for International Development DHIS District Health Information System DHMT District health management team DHQ District headquarters hospital EDO Executive District Officer EPI-MIS Expanded Program on Immunization Management Information System EMNC Essential maternal/newborn care ENC Essential newborn care FATA Federally Administered Tribal Areas FBC Facility-based committees FGD Focus group discussion FOM Field Operations Manager GOP Government of Pakistan HCP Health care provider HMIS Health Management Information System ICHP Improved Child Health Project IMCI Integrated Management of Childhood Illness IPC Interpersonal communications JHU/CCP John Hopkins University Center for Population Programs JSI John Snow, Inc. JICA Japanese International Cooperation Agency LHV Lady health visitor LHW Lady health worker LHW-MIS Lady Health Worker Management Information System LQAS Lot quality assurance sampling technique MAP Midwifery Association of Pakistan iv MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT M&E Monitoring and evaluation MIS Management information system MNCH Maternal, newborn, and child health MNH Maternal and newborn health MOH Ministry of Health MOPW Ministry of Population and Welfare Norad Norwegian Aid PAIMAN Pakistan Initiative for Mothers and Newborns PDHS Pakistan Demographic and Health Survey PIMS Pakistan Institute for Medical Science PNC Postnatal care QIT Quality improvement team RHC Rural Health Center RMOI Routine monitoring of output indicators SBA Skilled birth attendant SO Strategic objective TBA Traditional birth attendant THQ Tehsil headquarters hospital TT Tetanus toxoid VHC Village health committee UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund USAID United States Agency for International Development WHO World Health Organization WMO Woman medical officer MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT v CONTENTS MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT ACKNOWLEDGEMENTS .............................................................................................................................. i ACRONYMS .................................................................................................................................................iii EXECUTIVE SUMMARY ............................................................................................................................. vii MAJOR FINDINGS AND KEY RESULTS ............................................................................................. vii CONCLUSIONS .....................................................................................................................................ix SHORT-TERM RECOMMENDATIONS FOR THE PAIMAN PROJECT ...............................................ix LONG-TERM RECOMMENDATIONS FOR USAID ............................................................................... x I. INTRODUCTION ............................................................................................................................... 1 PURPOSE OF THE EVALUATION ........................................................................................................ 1 EVALUATION METHODOLOGY ........................................................................................................... 1 II. BACKGROUND ................................................................................................................................ 3 MATERNAL AND NEWBORN HEALTH IN PAKISTAN ......................................................................... 3 USAID/PAKISTAN HEALTH SECTOR ASSISTANCE ........................................................................... 3 ASSISTANCE FROM OTHER DONORS ON MATERNAL AND NEWBORN HEALTH ........................ 4 III. OVERVIEW OF THE PAIMAN PROJECT .................................................................................... 7 PROGRAM DESIGN AND IMPLEMENTATION STRATEGIES ............................................................ 7 MONITORING AND EVALUATION ........................................................................................................ 7 RESEARCH ACTIVITIES ....................................................................................................................... 9 MANAGEMENT AND ORGANIZATIONAL STRUCTURE ..................................................................... 9 FINANCIAL MANAGEMENT ................................................................................................................ 10 GRANTS MANAGEMENT .................................................................................................................... 10 RELATIONSHIPS, COORDINATION, AND COLLABORATION ......................................................... 11 IV. TECHNICAL COMPONENTS: FINDINGS, RESULTS, AND LESSONS LEARNED ..................... 13 SO1. INCREASING AWARENESS AND PROMOTING POSITIVE MATERNAL AND NEONATAL HEALTH BEHAVIORS ......................................................................................................................... 13 SO 2. INCREASING ACCESS TO MATERNAL AND NEWBORN HEALTH SERVICES ................... 16 SO 3. INCREASING QUALITY OF MATERNAL AND NEWBORN CARE SERVICES ....................... 18 SO 4. INCREASING CAPACITY OF MATERNAL AND NEWBORN HEALTH CARE PROVIDERS .. 21 SO5. STRENGTHENING DISTRICT HEALTH SYSTEMS .................................................................. 23 FUNCTIONAL INTEGRATION OF SERVICES .................................................................................... 24 V. CONCLUSIONS .............................................................................................................................. 27 VI. RECOMMENDATIONS AND FUTURE DIRECTIONS ................................................................... 31 SHORT-TERM RECOMMENDATIONS FOR THE PAIMAN PROJECT ............................................. 31 LONG-TERM RECOMMENDATIONS FOR USAID ............................................................................. 33 vi MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT MID-TERM EVALUATION OF THE USAID/PAKISTAN FATA PROJECT ACKNOWLEDGEMENTS ........................................................................................................................... 37 ACRONYMS ............................................................................................................................................... 39 EXECUTIVE SUMMARY ............................................................................................................................ 43 MAJOR FINDINGS AND CONCLUSIONS ........................................................................................... 43 SHORT-TERM RECOMMENDATIONS ............................................................................................... 44 LONG-TERM RECOMMENDATIONS FOR USAID ............................................................................. 44 I. INTRODUCTION ............................................................................................................................. 45 PURPOSE OF THE EVALUATION ...................................................................................................... 45 EVALUATION METHODOLOGY ......................................................................................................... 45 BACKGROUND .................................................................................................................................... 46 II. OVERVIEW OF IMPROVED CHILD HEALTH PROJECT ............................................................. 49 PROJECT DESIGN AND IMPLEMENTATION APPROACHES .......................................................... 49 MONITORING AND EVALUATION PLAN ........................................................................................... 49 ORGANIZATIONAL STRUCTURE AND MANAGEMENT ................................................................... 50 RELATIONSIPS, COORDINATION, AND COLLABORATION ............................................................ 50 III. FINDINGS AND LESSONS LEARNED ...................................................................................... 51 IR1: INCREASE ACCESS TO AND AVAILABILITY OF CHILD HEALTH SERVICES AT FACILITY AND COMMUNITY LEVELS ................................................................................................................ 51 IR2: IMPROVED QUALITY OF CHILD HEALTH SERVICES AT FACILITY AND COMMUNITY LEVELS ................................................................................................................................................ 52 IR 3: INCREASE COMMUNITY KNOWLEDGE AND ACCEPTANCE OF SERVICES AND BEHAVIORS ......................................................................................................................................... 53 IV. CONCLUSIONS AND RECOMMENDATIONS FOR THE FUTURE .............................................. 55 GENERAL CONCLUSIONS ................................................................................................................. 55 SHORT-TERM RECOMMENDATIONS FOR ICHP AND USAID ........................................................ 55 LONG-TERM RECOMMENDATIONS FOR USAID ............................................................................. 56 ANNEXES See Volume II. MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT vii EXECUTIVE SUMMARY The purpose of this evaluation was to conduct a midterm review of the PAIMAN Project. The evaluation, conducted in June-July 2008 by a team of three consultants through the Global Health Technical Assistance Project, assessed whether the project is achieving intended results, reviewed its organizational structures and technical approaches, and formulated recommendations for future directions. During field visits to selected project sites the team interviewed a broad range of stakeholders, including health managers, service providers, trainees, and community members. MAJOR FINDINGS AND KEY RESULTS Project Design and Management The goal of the PAIMAN project is to improve maternal and neonatal health in selected districts in Pakistan. Its framework outlines five strategic objectives to increase awareness of and access to maternal and newborn health services, to improve the quality of services, and to increase the capacity of health care managers and providers. The strategy and approaches of PAIMAN are appropriate and sound. Implementation has been challenging due to the high number of activities, multiple consortium partners, and the geographic spread and diversity of project districts. This complexity has been a challenge for the project’s performance and impact. Management of a consortium with a large number of partners and subcontractors necessitated several coordination mechanisms. A key issue is that consortium partner staff are not seconded to the project but remain housed at partner home offices. Though the project has three levels of indicators (outcome, output, and activity level), it primarily uses activity data for monitoring progress and reporting. John Snow, Inc. (JSI) has a thorough and standardized financing and accounting system in place. The current monthly burn rate is $1.1 million; the project anticipates expending all funding by September 2009. Project Implementation and Key Results The project has a clear and evidence-based communication strategy that targets nine groups, with multiple messages for each group. Of the numerous interventions underway, advocacy and mass media interventions seem strong. However, most interventions have not been implemented at a frequency or scale that would reach the majority of the target populations. For example, theater shows, a potentially effective means of outreach, have reached only a small percentage of the total population. Similarly, all events, local and larger-scale, have reached only about two percent of the population. This is not sufficient to effect population-level change. PAIMAN has successfully created support groups and trained lady health workers (LHWs) to be behavior change agents. The creation of support groups is likely to be taken to scale in coming years. This appears to be a sustainable and high-impact intervention. PAIMAN recently created three mass media interventions: a music video about male responsibility, a TV drama tackling issues of family planning and breastfeeding, and television advertisements addressing maternal and newborn issues. If these interventions prove successful, they seem likely to have impact. viii MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT Through its NGO subgrantees PAIMAN has helped establish nine birthing centers, which have been successful in meeting community needs. In one, PAIMAN successfully assisted the community to receive Civil Community Board funding to sustain it. PAIMAN has been successful in supporting the development of preservice training for community midwives (CMWs), a relatively new cadre of providers. The first batch of trained CMWs will be in the field in September 2008. There were no consistent responses about how the CMWs will be established and supervised in the community. PAIMAN has had discussions about accessing microfinance schemes to support the birthing rooms, but plans are not finalized. PAIMAN is also preparing to test different supervision models once CMWs are in the field, but many issues still need to be resolved. These plans need to be finalized and better communicated to all levels. PAIMAN has successfully increased the number of Ministry of Health (MOH) facilities that can provide basic maternal and newborn care. Overall, the number of births in upgraded facilities has increased, though it varies greatly by district and facility. PAIMAN upgraded 13 rural health centers (RHCs) to provide basic emergency maternal and newborn care (BEmONC) and 18 district headquarters hospitals and tehsil headquarters hospitals (DHQ/THQs) to provide comprehensive emergency maternal and newborn care (CEmONC). While some facilities are still waiting for instruments and staff, most are now equipped and staffed. Since most have only recently become functional, it is too early to expect major changes or assess the quality of services. PAIMAN has been advocating for these inputs, but some of the inefficiencies are beyond the control of the project. PAIMAN trained many providers who had never had any refresher training, which is a major accomplishment. The strategy of working through the provincial and district Health Development Centers has been important for developing a sustainable system. The project trained 1,600 providers in essential maternal and newborn care (EMNC). A major weakness of the training is that it does not include a clinical practicum or use of the partograph. Initially, the curriculum included both, but PAIMAN removed these two subjects so that male providers and pediatricians who do not conduct deliveries did not have to learn them (see Section IV, SO 4, for a detailed discussion). The team found that there is confusion about the difference between minor pregnancy and newborn ailments, such as urinary tract infections, anemia, and vomiting, and major complications that lead to maternal and newborn deaths. The behavior change communication (BCC) messages and training curriculum are not specific enough to make these distinctions (see Section IV, SO 3, and Annex M, Increasing Quality of Services, for a detailed discussion). In general, the team found referral systems to be inadequate. While the referral system is evolving as providers and health facilities become functional, the team did not find a clear referral system that can be communicated to women to ensure that they have access to facilities that meet their needs. PAIMAN successfully helped establish and reinforce district health management teams (DHMTs), provided management training to a broad range of health managers, and worked to improve the quality and utilization of health information systems. As of June 2008 the project had achieved five of the six indicator targets set for 2007. Between 2005 and 2008, in 10 project districts: – Births assisted by traditional birth attendants (TBAs) increased from 36 percent to 38 percent. Women who received three or more antenatal visits during pregnancy increased from 27 percent to 35 percent. MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT ix – Pregnant women who received at least two doses of tetanus toxoid (TT) during the most recent pregnancy increased from 40 percent to 43 percent. – Women who had a postpartum visit within 24 hours of giving birth increased from 34 percent to 39 percent. – Facilities upgraded and meeting safe birth and newborn care quality standards increased from 0 to 26. – On average, district health budgets increased by 52 percent. These results are impressive. However, there are vast differences in performance between districts, requiring district-level analysis for future directions. Project targets set for 2009 are significantly higher than the 2007 targets. At the current implementation rate, it is unlikely that the project will achieve its final targets for most of the indicators. It should be understood that it is too early to expect population-level impact in terms of significant behavior change and increased service utilization. CONCLUSIONS PAIMAN has been successful in putting key building blocks in place through training, communication and outreach, and facility renovations to enhance maternal and newborn care. The foundation is in place for increased utilization of services. The project staff is well respected by all stakeholders, and the project has established close alliances with partners at all levels. MOH contacts view PAIMAN interventions as important contributions to their work. However, continuous efforts are needed for the MOH to internalize and embed project activities into its national maternal and newborn health (MNH) program. The team believes that too much emphasis has been put on monitoring activities without sufficient analysis of output and outcome data to strategically guide the program. The functional integration of service interventions in Rawalpindi is innovative and could yield high impact if proven successful. The team believes that the intervention most likely will achieve results in Rawalpindi; many of the preparatory elements are already in place, and senior staff of both departments are highly committed and cooperative. However, this intervention has only been on the ground for a year. It is too early to assess whether service utilization has increased. PAIMAN has made great strides in developing some sustainable systems that can be improved. However, many other interventions are less sustainable; careful exit strategies are needed for them (see Annex F for a list of interventions and the team assessment of their sustainability). SHORT-TERM RECOMMENDATIONS FOR THE PAIMAN PROJECT The evaluation team believes that it is vital for the project to focus on sustaining gains achieved so far and improving upon areas that are still weak. The project in 2008 expanded to Swat District and two agencies in the Federally Administered Tribal Areas (FATA) but has only 14 more months remaining in the life of the project. Thus the team believes it should concentrate on consolidating and institutionalizing current activities to reach its objectives before scaling up to additional districts. If the project receives additional funding, we recommend that to increase population coverage it expand within the districts where it is already operating. For example, it should renovate additional hospitals and clinics and train their staff in strategically selected locations where useful BCC has already taken place. x MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT Due to the political climate, with additional funding the project might be urged to expand into other districts. If so, the team suggests that to improve interventions this should be done strategically, based on lessons learned so far from the PAIMAN experience. PAIMAN should shift its focus to analysis and use of output and outcome data to assist in strategic decisions, rather than tracking activities. There is also a need for more in-depth analysis of district data for future planning, given the vast differences between districts. PAIMAN has complete district data available for analysis. The evaluation team recommends that the PAIMAN consortium focus on the following priority interventions in the remaining 14 months of the project: – Simplify and clarify the number of BCC messages and target groups. – Help create new Citizen Community Boards (CCBs) wherever viable. – Finalize plans for establishing CMWs in the communities. – Assess the quality of care provided by private providers. – Incorporate a clinical practicum into EMNC training. – Strengthen supervision and referral systems. – Undertake a final evaluation of the functional integration of services intervention to assess results, make final modifications, and prepare recommendations for expansion. LONG-TERM RECOMMENDATIONS FOR USAID PAIMAN has achieved a great deal in the last four years, even though actual implementation has been going on for only two years. It is not reasonable to expect MOH and the other local partners to take full responsibility for current activities in the coming year. The evaluation team recommends that USAID continue technical assistance to the PAIMAN districts, tailoring the assistance to the specific needs of each district. The evaluation team suggests phased graduation of PAIMAN-supported districts over the next five years. High-performing districts could be graduated early because they provide a certain level of sustainable quality service before USAID assistance phases out and initiates similar interventions in new, preferably neighboring, districts. The ―graduate‖ districts could be encouraged to serve as role models to the new districts. The PAIMAN experience so far provides a wealth of insights on what works, as we have tried to capture in this report. Rather than trying out new approaches, a follow-on project should be encouraged to build on the lessons learned and sustain the gains of the project while giving special attention to areas that are still weak. Specifically, a follow-on project should take into account the following: – Behavior change messages need to be refined before any expansion. – The project should have an obstetrician/woman medical officer on staff to oversee the technical quality of training and service delivery. – Referral systems should be outlined at the outset of the intervention. – Selection of health facilities to be upgraded should be strategically linked to community interventions. MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT xi – Because teaching hospitals will be crucial, they need to be upgraded as training sites. – The training strategy should focus on staff in upgraded facilities; staff in other facilities should not be trained until they can return to an environment where they can practice their new skills under supervision. – Strengthening district health systems is crucial and should be part of any new project design. – A follow-on project should have fewer consortium partners. – A high design priority should be closer collaboration with the national MNH program, i.e., coordination on standardization, embedding project activities into the national program, and promoting MOH ownership. The progress on functional integration of services in Rawalpindi is promising. USAID should consider expanding the initiative to other districts in the new FALAH project once a final evaluation of the intervention is conducted. Finally, USAID should take a more active role in coordinating the MNH activities of international agencies and organizations. There is a need for leadership in the donor community to ensure closer and more structured collaboration. xii MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 1 I. INTRODUCTION PURPOSE OF THE EVALUATION The purpose of this evaluation is to provide the United States Agency for International Development’s Mission to Pakistan (USAID/Pakistan) with an independent mid-term evaluation of its maternal, newborn, and child health (MNCH) programs. The MNCH programs, managed by the Office of Health, are implemented through two projects: PAIMAN is a maternal and newborn project led by John Snow Inc. (JSI) and its partners, and the Improved Child Health Project (ICHP) is implemented in the Federally Administered Tribal Areas (FATA) by Save the Children. The objectives of this evaluation are to Assess whether MNCH program partners are achieving intended goals and results and meeting activity benchmarks in cooperative agreements and work plans. Evaluate the effectiveness of the management structure, administrative support, cost and partnerships, and collaborative plans. Evaluate the effectiveness of key technical components and approaches of the MNCH program. Establish whether demand for maternal and child health services is growing in project districts as a result, direct or indirect, of these projects. Document lessons learned and provide management, administrative, and technical recommendations for improving the efficiency and effectiveness of MNCH programs. Provide suggestions and options for future directions for the program and analyze the potential for program expansion at various levels of additional funding. The Evaluation Scope of Work can be found in Annex A. This report is confined to evaluation of the PAIMAN Project. The ICHP is evaluated in a separate report. EVALUATION METHODOLOGY The evaluation was conducted in June-July 2008 through the Global Health Technical Assistance Project; the in-country work took five weeks. The team was composed of Pinar Senlet, Team Leader; Susan Ross, Maternal Health Specialist; and Jennifer Peters, BCC/Community Mobilization Expert. A local firm, EyCon, was recruited to conduct interviews in sites where the team could not travel and to coordinate and manage in-country logistics. The evaluation methodology consisted of Review of background documents (see Annex B); Analysis of program and financial data and survey results; In-depth discussions with USAID/Pakistan and PAIMAN staff; Structured interviews with contacts in the central, provincial, and district offices of the Ministry of Health (MOH) and Ministry of Population Welfare (MOPW), donor organizations, implementing partners, and other stakeholders (see Annex C); and Field trips to Rawalpindi, Lahore, Khanewal, Peshawar, Sukkur, Lasbella, and FATA.1 1 Sukkur, Lasbella, and FATA sites were visited by EyCon. 2 MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT The team used both qualitative and quantitative data to conduct the evaluation. Quantitative information was derived from data sets maintained and formative research conducted by the project. The team also used findings from the FALAH Project baseline survey conducted in eight of the ten PAIMAN districts. Qualitative information was generated through interviews and observations at health facilities. To ensure that comparable information was collected during field visits, the team drafted standard guides for interviewing health managers and providers that reflected the questions posed by the evaluation scope of work. The team also drafted focus group discussion (FGD) guides that were used to collect information from community members in PAIMAN districts. EyCon conducted all focus group discussions (see Annex D, Findings from FGDs). The international team made field visits in Khanewal and Rawalpindi to observe service delivery, training, and community level interventions and interview health managers, service providers, trainees and community members; EyCon conducted the same activities in Sukkur and Lasbella. In Rawalpindi, the team conducted interviews and observed functional integration of family planning services. The international team also met with provincial officials in Lahore and Peshawar. MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 3 II. BACKGROUND MATERNAL AND NEWBORN HEALTH IN PAKISTAN Pakistan has some of the highest levels of maternal and newborn mortality in Asia. The maternal mortality rate averages 276 deaths per 100,000 live births, with great variation throughout the country. The infant mortality rate is 78 per 1,000 live births, and the newborn rate is 54 per 1,000 live births. The mortality rate for children under 5 is 94 deaths per 1,000 live births. Unfortunately, these rates have not decreased in the past decade; in many parts of the country they have increased. Although Pakistan has an extensive network of public facilities, they only reach about a third of the country’s population: the rest is served by the private sector, primarily for curative services. Most women (65 percent) deliver their babies at home; about 8 percent are assisted by a skilled birth attendant (SBA). Overall, only 39 percent of births are delivered by SBAs, so it is not surprising that most maternal and newborn deaths occur at home. Consistent with global findings, the major cause of maternal mortality are postpartum hemorrhage, pre-eclampsia and eclampsia, sepsis, and complications of abortions. Women and newborns die because (1) people do not know when or where to go for health services; (2) accessing health services is often challenging, particularly the transport costs and provider fees; and (3) there are delays in access to services, many of which are of poor quality, so there is little confidence that these providers make a difference. In addition, traditional social values often discriminate against women, lowering their status and affecting their food intake and nutrition, education, decision-making, physical mobility, and healthcare. Husbands, in-laws, and religious and community leaders all play significant roles in these customs. Societal pressures on women to have many children in quick succession, preferably sons, also increase their risk of morbidity and mortality. Public facilities need considerable improvement in their physical infrastructure (e.g., water supply, electricity), access to female providers, supply of drugs, and equipment to provide quality services; many of them are open for only four to five hours a day. Most providers do not have much capacity to manage normal births, let alone complications. Similarly, management systems at the district level are weak, especially referral, supervisory, and health information systems and coordination between the public and private sectors. While most women who can afford it go to private providers, these services primarily provide curative services. Moreover, private services are unregulated, which raises major questions about the capacity and quality of the services they offer. USAID/PAKISTAN HEALTH SECTOR ASSISTANCE USAID’s health program in Pakistan supports 10 of the 12 health and population objectives outlined in the Government of Pakistan (GOP) Ten-Year Perspective Development Plan 2001–2011. In 2003 the Government of the United States and the GOP signed a five-year, $115.7 million grant agreement to help the MOH, the MOPW, provincial and district governments, and the private sector to implement program activities. In consultation with the GOP, USAID agreed as part of its larger health portfolio to support provincial government programs to improve maternal, neonatal, and child health outcomes. The health program, which began in 2003, supports activities to improve MNH services, promote family planning, prevent major infectious diseases, and increase access to clean drinking water. The program is national; it works in underserved rural and urban districts in Sindh, Balochistan, Punjab, the Northwest Frontier Provinces, and the FATA. Among current health programs are: Health Systems Strengthening: Supports the community midwifery (CMW) program; health information awareness and government accountability; addressing health system challenges through 4 MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT grant assistance; and improving essential drugs and contraceptive logistics management (ABT Associates). FALAH Project, Diversification of Family Planning Activities in Pakistan: Addresses the need to increase and improve family planning services (the Population Council). Maternal and Newborn Health: PAIMAN is USAID’s flagship project to reduce maternal and neonatal mortality (JSI). HIV/AIDS Program: Gives grants to seven local nongovernmental organizations (NGOs) to increase HIV/AIDS awareness and promote healthy behaviors in high-risk groups (Research Triangle Institute). Strengthening TB Control: Helps the GOP to consolidate and accelerate complete treatment of TB patients (World Health Organization [WHO]). Polio Eradication: Provides assistance to national polio immunization campaigns to eliminate polio from Pakistan (WHO and UNICEF). Demographic and Health Survey (DHS): Gives funding and technical assistance for the Pakistan DHS and Maternal Mortality Study (Macro International and the National Institute of Population Studies). Disease Surveillance and Response: Supports design of a National Integrated Disease Surveillance and Response Program and a Field Epidemiology and Laboratory Training Program (U.S. Centers for Disease Control). Child Health in the Federally Administered Tribal Areas (FATA) of Pakistan: Works to improve the availability and quality of and demand for child health services throughout the FATA (Save the Children, USA). Safe Drinking Water and Hygiene Promotion: Gives technical assistance in hygiene and sanitation promotion, community mobilization, and capacity building to complement the GOP’s installation of water treatment facilities nationwide (ABT Associates). ASSISTANCE FROM OTHER DONORS ON MATERNAL AND NEWBORN HEALTH A number of other donor agencies and international organizations support the Pakistan MNCH program; most of them also support strengthening district health systems. In UNICEF’s maternal and newborn project, which is active in 17 districts, many elements are similar to the PAIMAN project. The broad child health project incorporates immunization campaigns (polio, measles, TT), child nutrition interventions, and training in integrated management of childhood illness (IMCI). UNICEF is also planning to initiate a health systems strengthening intervention to support district-level management of health services. UNFPA supports reproductive health and safe motherhood activities in 10 districts in all provinces. While the focus is on reproductive health, the organization supports selected maternal health initiatives. Both UNICEF and UNFPA work closely with the PAIMAN project. The GOP selects the UNICEF and UNFPA project districts. The United Kingdom Department for International Development (DFID) provides budgetary support and national policy-related technical support to the new National Maternal and Newborn Health Program. DFID is also planning to initiate a project to assist with health system strengthening, based on specific needs. WHO provides policy and technical assistance support for MCH as well as reproductive health, including family planning. MCH has been a long-term priority for WHO’s program in Pakistan. MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 5 The Japanese International Cooperation Agency (JICA) program emphasizes strengthening health information systems. JICA provided support for the development and pilot testing of a new district health management information system, DHIS. The system was pilot tested in four districts and the GOP is now preparing for national expansion. JICA also helps train hospital-based providers in safe motherhood. The German Technical Cooperation Program (GTZ) provides health systems strengthening initiatives in selected areas in AJK, NWFP and FATA. The Norwegian Embassy is planning to initiate support for a 5-year MNCH program in 10 districts in Sindh province. The program will be implemented through a consortium of UN agencies, including UNICEF, UNFPA, and WHO. The Canadian International Development Agency (CIDA) supports health initiatives in Pakistan, although health is not a priority for CIDA assistance to Pakistan. CIDA provides budgetary and technical assistance for health systems strengthening in two districts of Punjab and is also planning to support an MCH project in three districts of Balochistan through UNICEF. Almost all donor agencies the team met voiced the need for better coordination of donor MNH activities. The donors meet to discuss their activities and share information on an ad hoc basis, but there is a need for more structured coordination. A specific technical area that needs donor coordination is standardization of training curricula and BCC messages in MNH. (See Annex E for a summary of the activities of other donors in MNCH and health systems strengthening, including project timeframes, budgets, geographic focus, and plans for the future). 6 MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 7 III. OVERVIEW OF THE PAIMAN PROJECT PROGRAM DESIGN AND IMPLEMENTATION STRATEGIES PAIMAN lays out its goal, major objectives, and implementation strategy in the Strategic Framework drafted in 2005. The goal is to improve maternal and neonatal health in 10 districts in Pakistan. The strategy describes a ―Pathway to Care and Survival‖ continuum of care to respond to the needs of mothers and newborns. The project promotes access to skilled birth attendants (SBAs) as a long-term goal to reduce mortality and assists in positioning community midwives (CMW) as the focal point for obstetric care. The project works at all four levels (community, primary health care facilities, secondary [referral] facilities, and tertiary care facilities) to strengthen local capacity and improve access to quality services. The framework outlines five strategic objectives (SOs): SO1: Increase awareness and promote positive maternal and neonatal health behaviors. SO2: Increase access to and community involvement in maternal and child health services delivered through health and ancillary health services. SO3: Improve service quality in both the public and private sectors, particularly related to the management of obstetrical complications. SO4: Increase the capacity of MNCH managers and care providers. SO5: Improve management and integration of services at all levels. The only major revision to the original program design was the inclusion of a health systems strengthening component. The recent devolution of responsibility for health care services to the districts, where capacity is limited, necessitated additional interventions to strengthen the district health systems, which were incorporated into SO5. There were minor adjustments to the original scope as issues were identified during implementation. The team believes that the overall strategy and implementation approaches of PAIMAN are appropriate and sound, articulating how results will be achieved, including interventions at the facility and community levels and integration of various project activities. Implementation of the strategies, however, has been challenging: The project’s five-year work plan is overly complex, incorporating multiple interventions (for example, community BCC and demand creation activities) with little prioritization. The large number of consortium partners necessitates meticulous coordination, but some delays in implementation and duplications of effort were inevitable. If the project had focused on fewer priority activities implemented by a smaller number of partners, implementation could have been smoother and less complex. It might also have had more impact faster. (See Annex F for a list of PAIMAN interventions, priorities, and sustainability aspects.) Another design element that complicated implementation was the selection of project districts. The GOP made the selection; USAID/Pakistan and PAIMAN were not involved. The ten project districts in four provinces are spread over the entire country, and the districts have few commonalities. The diversity and geographic distribution is logistically complex and necessitates differing approaches in each district. Moreover, this design does not allow districts to learn from each other. MONITORING AND EVALUATION PAIMAN uses three types of indicators to track progress and evaluate interventions. Outcome indicators are listed in the project monitoring and evaluation (M&E) logframe. Among the outcome indicators, six are also included in USAID/Pakistan Strategic Objective 7 results framework. These are 8 MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT Percentage of births attended by SBAs; Percentage of women aged 15–44 who received three or more antenatal (ANC) visits during the last pregnancy; Percentage of pregnant women who report receiving at least two doses of TT during the last live birth; Percentage of women who report having a postpartum visit within 24 hours of giving birth; Number of district referral facilities upgraded and meeting safe birth and newborn care quality standards; and Increase in district health budgets of 50 percent or more over the life of project (from all sources other than USAID). Data for most of the outcome indicators are derived from a baseline household survey the project conducted in 2005; it is planning an end-line survey in 2009. Because a midterm survey was not planned, the project has not been reporting on the outcome indicators.2 Since it was not possible to obtain more frequent data on outcome indicators, the project developed a system to track output indicators to monitor progress. The Routine Monitoring of Output Indicators (RMOI) system has 17 indicators (see Annex G for a list). Partner organizations, depending on their technical roles, collect and report data to the project monthly. PAIMAN uses several government and project databases to collect RMOI data, including the Health Management Information System (HMIS), the District Health Information System (DHIS), the Lady Health Workers Management Information System (LHW-MIS), the Expanded Program on Immunization (EPI-MIS), health facility records and check lists, NGO grant reports, records of District Health Management Teams (DHMT) and private sector records. The third set of indicators is at the activity level. PAIMAN has established a meticulous system for tracking a vast number of activities against targets set in annual work plans. The team believes this is a good internal monitoring system, although the amount of information collected seems excessive. The team also observed that the focus has been on collecting activity information and comparing it to set targets rather than analyzing the information for project management. Using analysis of RMOI data for strategic decisions would be more useful than focusing on activities. These three sets of indicators are found in different documents and databases. They are not linked and are not part of overall project M&E. Activity data are compiled and reported to USAID, regional offices, districts, other partners, and other stakeholders quarterly and annually. In these reports, accomplishments against targets are listed by activity group (e.g., training), rather than by SO. RMOI data are not presented in the main body of the report; they are included in annexes under separate data sets that are not meaningfully aggregated or analyzed. This fragmentation of M&E makes it difficult, if not impossible, to understand whether accomplishing activities and meeting targets actually lead to better results. The team was informed that USAID/Pakistan had asked the project to report on activities rather than on RMOIs. The reports provide useful feedback to project staff at both headquarters and regional offices but are less useful for districts and other partners and stakeholders, probably due to the vast amount of activity information provided. 2 At the time of this evaluation, a household survey had just been completed to collect data for another USAID￾funded project, FALAH. The survey covered all PAIMAN districts and used the same baseline questionnaire. The evaluation team used this survey’s results as midline indicator values. MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 9 RESEARCH ACTIVITIES PAIMAN has conducted a wealth of surveys and research over the course of the project; the evaluation team found that the studies were relevant and were being used to improve project implementation. For example, baseline surveys were conducted in each district to evaluate the success of the project. Extensive qualitative formative research was done to refine and finalize the Communication, Advocacy, and Mobilization (CAM) strategy. A lot quality assurance sampling (LQAS) assessment of HMIS was conducted to train district managers on how to regularly monitor and assess HMIS performance in their own districts. These surveys and other research are referenced in Annex B. The findings and how they are used to improve project implementation are discussed below. MANAGEMENT AND ORGANIZATIONAL STRUCTURE PAIMAN is a consortium of seven partner organizations led by JSI. As the prime contractor, JSI is responsible for technical, administrative, and financial management of the Cooperative Agreement (see Annex H for a list of consortium partners and their responsibilities). The roles of the partners were clearly delineated at the outset; however, managing such a large number of organizations has been a challenge. Many contacts reported that coordination among partners was problematic, particularly early in the project. By now, the team found, the partnership has improved considerably due to the efforts of the partners and the leadership of the Chief of Party (COP). The project institutionalized several coordination mechanisms to implement the program effectively. Meetings of consortium partners are routine at the national, provincial, and district levels. To ensure better technical coordination, thematic groups were formed, with one consortium partner identified to lead each group based on its responsibilities and technical expertise. Though such mechanisms have certainly facilitated implementation, they demand considerable time and effort from both JSI and its partners. The PAIMAN main office in Islamabad is staffed by the COP, core JSI management, and the Johns Hopkins University Center for Communication Programs (JHU/CCP), which does not have its own office in Islamabad. Technical staff from partner organizations assigned to PAIMAN are not seconded to the project; they continue as staff of the partner organization and are housed in its central and provincial offices. The project maintains a field office in each province staffed by a JSI Field Operations Manager (FOM) and replicating the structure of the main office; staff of consortium partners are housed at the partner’s field offices and report to headquarters. JSI FOMs are responsible for coordinating the project activities of all partners. In all districts, PAIMAN partners share office space designated by the Executive District Officer (EDO) for routine monitoring of project activities there. This is not the optimal organizational structure in terms of programmatic and financial efficiency and effectiveness. The project has a centralized structure. The main office, with input and feedback from the provincial staff and partners, prepares the annual work plans. Once approved, the work plans are handed over to the provincial offices for implementation. The detailed lists of activities in the work plans have targets the field staff is charged with meeting. This structure allows for little flexibility and decision-making power in the field; however, the evaluation team found that it works reasonably well. The main office is highly responsive to the needs of the provincial offices and provides immediate feedback as the need arises. Another challenge to the efficiency of the consortium is that almost all partners are also active in other donor-funded projects. This has so stretched the capacity of several of the partners that PAIMAN experienced frequent turnover of key positions assigned to the project. Should an urgent need arise for another project, the partner would reassign staff there, leaving PAIMAN in search of a qualified replacement. 10 MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT FINANCIAL MANAGEMENT JSI has a thorough and standardized accounting system based on its experience working in other countries, including financial management of grants. The following is a brief budget summary for PAIMAN: Total ceiling for PAIMAN through September 2009: $49,943,8573 Obligations up to March 2008: $43,470,371 Total expenses through March 2008: $30,314,097 Budget remaining to be obligated as of March 2008: $ 6,473,486 The evaluation team examined the budget breakdown by management and technical costs for each partner organization (see Annex I, PAIMAN Budget Analysis). About 35 percent of the total budget is spent on management, including salaries, overhead, travel, equipment, etc., and the remaining 65 percent constitutes the program budget. Though consortium members have different overhead rates, the average is 14 percent. JSI has the largest budget share (49 percent); the remaining 51 percent is split across the seven consortium partners. This is a reasonable and balanced allocation of financial resources. An analysis of the budget by program reveals that BCC activities constitute the largest line item (31 percent) followed by costs of renovations (24 percent) and training activities (16 percent). The total budget for subgrants with NGOs is $6,000,000,4 18 percent of the total program budget. Given the scope and focus of the project, the distribution of funds across program areas seems appropriate and balanced. It does not appear, however, that PAIMAN is tracking cost per technical area, cost per partner, or the cost￾effectiveness of interventions. PAIMAN and USAID/Pakistan heavily emphasize tracking the burn rate of the project. The current monthly burn rate is $1.1 million, and the project anticipates expending all funding by September 2009. GRANTS MANAGEMENT PAIMAN designed an NGO grants mechanism to implement community-based interventions to promote MNH and to increase the capacities of local NGOs to provide such services. Financial and programmatic management of the NGO grants rests with JSI. The role of the local NGOs is to provide services in areas not covered by the government-supported LHW program. Since 2006 the project has provided grants to 37 provincial and district/community NGOs. The total budget for NGO grants is $6 million. Individual grants vary depending on the scope of work, but on average NGOs are awarded $40,000 a year. In addition to the local NGOs, as part of its NGO allocation PAIMAN provided a grant of $1,279,477 to a US-based NGO, Mercy Corps. PAIMAN instituted a flexible but thorough and transparent process for selecting NGOs for support. The request for proposals was widely publicized, and the project organized preproposal workshops for the NGOs to explain the application and selection processes. Committees comprised of representatives of JSI, local communities, and district and provincial health officials made the final selection. For the NGOs selected JSI organized training workshops on program and financial management and communication skills. Technical information about MNH was incorporated into the communications trainings conducted for LHWs and community health workers (CHWs). The NGOs have been instrumental in implementing such community-based activities as training of CHWs, organization of support groups and local events, medical camps with TT vaccinations, and coordination of Citizen Community Boards (CCBs) to establish birthing camps and purchase ambulances. 3 PAIMAN received additional funding of $6,300,000 for programming in FATA and Swat Districts in 2007 that is not included in this analysis. 4 The total amount for grants to NGOs includes substantial support for the Midwifery Association of Pakistan. MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 11 These activities are detailed below. PAIMAN is planning to phase out its support to NGOs by year-end 2008; the grants were intended to be for two years. RELATIONSHIPS, COORDINATION, AND COLLABORATION PAIMAN’s team of hard-working and committed staff has built productive partnerships with all stakeholders, including the GOP and other donors. The project leadership was praised and valued by all health sector partners interviewed by the team. Similarly, the project has formed satisfactory relationships with and been responsive to USAID as well as other projects funded by USAID/Pakistan. 12 MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 13 IV. TECHNICAL COMPONENTS: FINDINGS, RESULTS, AND LESSONS LEARNED SO1. INCREASING AWARENESS AND PROMOTING POSITIVE MATERNAL AND NEONATAL HEALTH BEHAVIORS Key activities to increase awareness of and demand for MNH services were (1) home visits and small group activities, such as LHW and CHW home visits and support groups, private sector interpersonal communications (IPC), and theater events; (2) events, both local and mega events, and health camps; (3) mass media initiatives (TV drama, video, advertisements); (4) a variety of community-based committees; and (5) advocacy to government officials at all levels, journalists, and religious leaders. Activities are implemented through four partner organizations: JHU, which provides technical assistance to development of mass media and innovative interventions; and Save the Children, PAVHNA, and Mercy Corps, which implement activities in selected districts. The CAM Strategy: PAIMAN’s evidence-based Communication, Advocacy, and Mobilization (CAM) Strategy is based on quantitative and qualitative research and a global and in-country literature review. This well-drafted strategy clearly spells out target audiences, key messages, and a national implementation roadmap. District roadmaps are drawn from the CAM strategy to ensure that activities are appropriate for the cultural context and are implemented on time. As a result of this extensive process, PAIMAN was asked to help the National MNH Program to develop its communications strategy, which is now awaiting MOH approval. PAIMAN has ambitious numbers of target groups to reach, messages to share, and activities to implement. This has allowed the project to test a number of innovative and some potentially very effective activities. On the other hand, the project is likely to have more impact in its final 14 months if it reduces the numbers of interventions, increases their repetition, and rotates messages at three-month intervals (see Annex G for a Sample District Communications Plan). While the project seems to be using effective means of outreach, all events together have reached only 2 percent of the population—not enough to produce population-level change. Focusing on the interventions with the most reach, or scaling one or two of them up significantly, would carry more impact. Some examples are: Focusing on LHWs and CHWs and events such as health camps in rural areas that bring services to those most in need; Focusing on making as many CCBs as possible functioning and sustainable, especially those located around birthing centers; Scaling up the most effective activities, such as the puppet theater; and Scaling up activities with the Ulamas (religious leaders), which seem to be a crucial way to reach men. 14 MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT Figure 1: Percentage of Beneficiaries Reached by Activity 33% 61% 2% 2%2% NGO beneficiaries LHW beneficiaries IPC beneficiaries Theater Events Target Groups and Messages: PAIMAN has identified nine target groups for its CAM activities. Two of these, traditional birth attendants (TBAs) and medical providers, are not true target audiences of CAM activities; their inclusion fits better under SO3, Quality of Care. The other seven target groups could be reduced to three: (1) women of reproductive age; (2) gatekeepers (husbands, family members, and communities); and (3) advocates (government officials, Ulamas, and journalists). The CAM strategy also incorporates a great number of messages (about 40 for women alone). Some messages, such as danger signs in pregnancy, need correction and simplification so that women and LHWs can better recognize when a situation is potentially life-threatening. The project might also consider including special messages for prima gravidas (see Annex K for a Key Messages and Target Groups Matrix.). There is also considerable confusion about who qualifies as an SBA among project beneficiaries, LHWs, and CHWs, all of whom refer to TBAs trained by PAIMAN as ―skilled.‖ Introducing CMWs into communities may further confuse the issue. Lady Health Workers: LHWs constitute a government-supported network of outreach workers; their coverage in PAIMAN districts is roughly 40 percent. PAIMAN has trained a total of 5,300 LHWs (62 percent of target); the remainder will be trained by September 2008. LHWs were trained for five days in BCC methodology and have very effectively introduced support groups into their routine work. Community support groups meet every two weeks to give women the opportunity to discuss problems, issues, and solutions to their own health needs. LHWs also conduct home visits, and provide family planning and iron folate and other simple medicines, giving women an opportunity to address concerns they may not want to share publicly. PAIMAN has succeeded in training an already functioning cadre of government field workers to be behavior change agents. The creation of support groups—a PAIMAN innovation—is likely to be taken to scale in coming years, and women already report having an easier time talking with their husbands and families about previously taboo subjects. Of all the beneficiaries reached by PAIMAN SO1 activities, an estimated 61 percent were reached by LHWs. This intervention appears to have the most impact, and to be the most sustainable. NGO Subgrants: PAIMAN provided subgrants to 37 local NGOs to access underserved deep rural communities not currently reached by LHWs. These NGOs implement MNH outreach activities in their communities, ranging from local events and community theater to creation of approximately 740 CHWs, whose function is the same as the LHWs. NGOs report reaching nearly 1.2 million individuals to date with MNH messages, greatly extending the reach of vital information into areas where it is much needed. MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 15 NGOs have also been instrumental in initiating CCBs to respond to community needs (NGO activities to increase access to services by the underserved are covered under SO2). Mass Media: PAIMAN has recently created three mass media interventions: a music video about male responsibility in pregnancy and childbirth (just launched); a 13-part TV drama tackling sensitive issues such as family planning, the three delays, and breastfeeding; and five 20-second television advertisements each addressing a key MNH area. One early success in mass media is that after contracts were negotiated for airing the video, the network decided to drop all charges when its staff viewed the video and realized the importance of its messages. It is too early to evaluate whether these interventions will ultimately prove successful, but PAIMAN’s evidence-based approach to their development, use of the most widely viewed local TV station, and innovative approach (video and drama) seems likely to have impact. Community-Based Committees: PAIMAN has piloted four different types of community-based committees to promote awareness of the needs of pregnant women and newborns: (1) village health committees (VHCs); (2) facility-based health committees (FBCs); (3) quality improvement teams (QITs); and (4) citizen community boards (CCBs). VHCs consist of men who agree to help LHWs to spread important MNH messages to other men in the community, particularly if there is a husband who is not supporting his pregnant wife. While VHCs seem to function best in areas where women can speak more openly with men about maternal health issues, where they do work they seem to be an asset. Both FBCs and QITs are aimed at improving links between communities and the nearest health facility; the goal is for them to become sustainable CCBs that assist communities to better access quality care. CCBs are government-recognized committees that receive funds from district governments to help them improve local conditions. CCBs receive 80 percent of the funding they require from the district, but must raise 20 percent within the community. Theater: Over the life of the project 351 theater performances were shown to 70,200 individuals. Theater activities comprise both community theater, implemented at events or health facilities, and Putlee Tamasha (puppet theater), a PAIMAN innovation incorporating MNH messages into a traditional form of community entertainment. Community theater, particularly puppetry, seems to be a great vehicle for communicating information. However, these activities account for only 2 percent of the beneficiaries reached, so it is difficult to say what their impact will be. Events: PAIMAN events include mass and local events, village fairs, and health fairs or medical camps. In total, 395 events reached some 82,500 individuals, 2 percent of total project beneficiaries. While these events help promote PAIMAN’s assistance to the MOH and the availability of improved MNH services in the area, their impact on uptake of messages has not been evaluated, and the reach is low compared to the work of LHWs and CHWs. However, health fairs in rural areas may improve access to needed services (e.g., TT vaccinations, antenatal checkups), and increase a community’s belief in government-run services generally, which may enhance the value of this type of event. Advocacy: The PAIMAN project reaches out to influential audiences, including government officials (national, provincial, and district), Ulamas, and journalists. At all levels a great deal of effort has been devoted to increase awareness of MNH issues and enhance commitment to improving the situation for mothers and newborns. Of all the advocacy interventions, working with religious leaders to target men with messages during Friday prayers about their responsibility to safeguard pregnant women seems likely to have great impact; it is particularly crucial given the difficulty in reaching this target audience through other activities. Private Sector Outreach: Greenstar has promoted its Good Life private sector clinics through a variety of mass media (TV spots, billboards) and IPC activities (37,548 meetings reaching 81,081 beneficiaries, 44 percent of the target). The project also ensures that providers conduct three Clinic Sahoolat days (free service days) to encourage uptake of services by disadvantaged women in their catchment area. The challenge still facing the project is to reach rural areas, where women have less access to quality services, 16 MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT to determine where they go for private sector care and insure that they understand what Good Life represents and where it is available. While only the end-line survey will be able to assess how much the program is increasing demand for MNH services, the SO1 component of this project seems set to make marked improvements in knowledge and awareness of MNH issues. Due both to the considerable budget allocation and the number of innovative activities, PAIMAN has already reached an estimated 2.5 million individuals through home visits, support groups, and events. Results of evaluation team efforts show good retention of key messages by community workers and increased knowledge among beneficiaries as well.5 Considering that there are still 18 months of activities to be implemented and the TV campaign will launch shortly, it is likely that many target project beneficiaries will have been exposed to lifesaving MNH messages multiple times through various channels, which increases the likelihood that the project will be a success. SO 2. INCREASING ACCESS TO MATERNAL AND NEWBORN HEALTH SERVICES Key activities to increase access to SBAs were the development of a cadre of community midwives and training of private providers by Greenstar. Key community-level activities to increase access to health services were (1) NGO activities, such as establishment of birthing centers and medical camps; (2) creation of emergency funds and transportation networks; (3) provision of 19 ambulances and training of staff in essential surgical skills and basic life support; and (4) training TBAs on clean delivery kits (CDKs). Community Midwives: The skills of midwifery tutors have been upgraded and the project has participated in curriculum development. Currently 900 CMWs are enrolled in programs. This has increased the number of students by 30–50 percent but has overextended many of the tutors. Therefore, providers in health facilities used for training sites also serve as preceptors for CMW students. These facilities and clinical staff have not received training or support from the project. The midwifery tutors stated that the three months time spent on theory is too short and should be extended to five or six months. CMWs are placed in DHQs and THQs for their practicum; they are supposed to conduct five deliveries there on their own and five home deliveries. The caseloads in some facilities are not sufficient for the CMWs to achieve these targets. The great variability in CMW training has major implications for the type of support they will need when they graduate. Of the CMWs completing their practicum that the team met, none had conducted a home delivery. However, at the THQ in Rawalpindi, most of the CMWs interviewed had completed 30 deliveries under supervision. When asked what they thought about the CMWs practicing independently, supervisors said they were not ready and it was not ―safe‖ for them to practice on their own. Asked how they would integrate back into their communities, CMWs students said they would ―tell women that TBAs are not skilled and no one should go to them. They should only go to the CMW because they are trained by senior doctors.‖ Most CMWs said they would have a birthing room in their house financed by PAIMAN. When CMWs were asked how they would help a woman with a complication, several said that PAIMAN was going to give them an ambulance. The first batch of trained CMWs was to be in the field in September 2008. There were no consistent responses about how they were to be established and supervised in the community. PAIMAN has had discussions about accessing microfinance schemes to support the birthing rooms, but plans are not finalized. PAIMAN has a project that will test different supervision models (e.g., LHV/LHS, Greenstar) once CMWs are in the field, but many issues still need to be resolved. The plans have not been communicated well to midwifery tutors or the CMWs. There does not appear to have been much work, 5 Results vary greatly by district, and use of the local team FGD findings would assist the project to know where best to intensify and focus its interventions. MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 17 particularly with the LHWs, TBAs, and other private providers, to prepare communities for the introduction of CMWs. In summary, PAIMAN has been successful in supporting the development of pre-service training for CMWs, but with the influx of this new cadre the capacity of midwifery tutors has been stretched. There are challenges in ensuring adequate caseloads for training, and some of the training facilities have not been updated. Although CMWs are required to conduct home deliveries during their training, this does not seem to be happening. Most important, plans for establishing and supervising them in the community need to be finalized and better communicated to all levels. This will necessitate a major emphasis at the community level to clarify roles, responsibilities, and working relationships of the various providers. Greenstar/Good Life Clinic: Studies indicate that the private sector provides 35–60 percent of maternal care services. The sector is quite complex; there are great variations in types of providers and levels of care and quality. Greenstar trained 550 female private providers, largely urban–based, on ENMC using PAIMAN’s trainers and curriculum. There was no clinical practicum, and partograph and ATMSL were not part of the training. Trainees have to provide three free consultation clinics, but it is unclear how many new clients attend and continue to go to these providers. Greenstar has a WMO who provides educational sessions after the training, but there is no direct observation of skills. It is unclear what the plans are for post-training follow-up (see Annex L). Forty-two private providers were trained on CEOMC at the PIMS. No providers have been trained in Upper Dir, Buner, Jhelum, Jaffarabad, and Lasbella. This was a special arrangement made by the COP because private providers cannot be trained in public institutions. One of the PIMS trainers has agreed to do some monitoring of these providers but has not yet done so. Data on the private provision of MNC services has been a major problem. In the first quarter of 2008, only 28 percent of Good Life providers reported on their services; the range was from 93 percent of providers in Buner to 0 percent in Upper Dir, Dadu, and DG Khan. It is therefore impossible to know if these providers are providing deliveries or managing complications; this is a major weakness of the program. Greenstar has just initiated a mechanism to collect data weekly via mobile phone SMS. To date about 250 of the 550 providers have been contacted and 159 have provided cell phone numbers. It is too early to know if this is a viable approach, especially since weekly reporting seems excessive. Birthing Centers and Emergency Transport: Through its NGO contracts (see SO1 for details), PAIMAN has helped establish nine birthing centers at BHUs and helped rural communities buy two ambulances for transporting women in obstetrical emergencies. The birthing centers were established through a joint effort between the DOH, a PAIMAN-supported NGO, and the communities. The centers are located in BHUs where there were previously no female providers to provide ANC, delivery, and PNC services. The DOH provides basic equipment, medications, and medical supplies. The NGOs pay the salary of the LHVs who provide MNC services. Because the LHVs are local, they are available 24/7; they conduct deliveries in the BHU as well as in homes. Neither of the LHVs interviewed had received PAIMAN training. The ambulances attached to two of the birthing centers offer the option of giving birth in a district hospital or being transported to one in case of emergency. The communities are responsible for gas and ambulance maintenance. In one case, PAIMAN was successful in helping a community acquire CCB funds (through which 80 percent of the costs for activities are funded). Setting up CCBs is a lengthy process, and some communities and districts seem to be better able than others to grasp this opportunity. The CCB concept is vital to the sustainability of rural access initiatives and should be lobbied for wherever possible. The communities interviewed around the two birthing centers visited were extremely positive about this activity, though they were also concerned about support from the project ending before they were viable enough to sustain themselves. The main concern is that the LHV will leave if the NGO stops paying her 18 MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT salary. Several of these communities are applying for CCB funds to help sustain these initiatives, but they will not be able to secure funding before the NGO contracts are scheduled to end in the next few months. Emergency Funds and Transport: Throughout all 10 districts, LHWs, CHWs, community committees, and NGOs have worked to establish revolving funds and taxi networks for emergency transportation. Access to emergency health care in underserved settings is often the most challenging aspect of community mobilization, which takes considerable time and effort. PAIMAN does seem to be having some success (e.g., preferential taxi fares for those in need of emergency transport, and community schemes to raise funds monthly from each family). Where CCBs are functioning, they in particular seem to offer a potentially sustainable solution to the issue of how to handle emergencies. Ambulances and Skilled Staff: Across the 10 districts PAIMAN has provided 19 of the 31 upgraded facilities with ambulances and staff trained to handle basic emergencies. Seven of the ten districts have committed to giving priority to obstetric emergencies, though it is unclear how this actually works. Community awareness of the ambulances seems low: they were not mentioned as a potential means of transport in community interviews. Often it is quicker to find local options to transport the woman because the time needed for an ambulance at the district or tehsil level to reach the community seriously delays access to care. Training of TBAs: Nearly 1,900 TBAs have been trained either by Greenstar or local NGOs on danger signs in pregnancy and childbirth, use of CDKs, and referral of emergencies. The majority of communities reported increased use of CDKs by TBAs during deliveries; this is an encouraging sign considering the large numbers of women in underserved communities who still deliver at home. The only downside to this is the reporting among LHWs and women that ―PAIMAN-trained‖ TBAs are SBAs. SO 3. INCREASING QUALITY OF MATERNAL AND NEWBORN CARE SERVICES PAIMAN activities to enhance the quality of MNC service include minor renovations, advocating for adequate staffing, and equipping 31 MOH facilities. In MOH facilities that have not been upgraded, PAIMAN provided ambu bags and EMNC protocols. Save the Children trained public providers in both upgraded and nonupgraded MOH facilities (SO4), and Greenstar trained private providers. Facilities for Normal MNC: The primary aim of upgrading health care facilities was to increase access to SBAs. In 2007, births in the upgraded MOH facilities increased, but this varies greatly by district and facility. We compared the average number of deliveries in 2007, as a target, to assess whether the increase continued in 2008. Most of the DHQ/THQ (12/18) met or exceeded the 2007 average, but only 11 of 13 RHCs were able to do so. Many of these facilities did not provide any delivery services before this project, so the percentage increases are substantial. Figure 2 shows the percentage of all births, by district, that took place in 31 upgraded MOH facilities and in Good Life clinics. MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 19 Figure 2: Percentage of Total Births in Upgraded MOH Facilities and Good Life Clinics, by District (2007 and January–March 2008) In 2007 the population coverage in upgraded MOH facilities averaged 4 percent of total births, which is not surprising at that point in the project. The figure is significantly higher when the trained private providers are included. For example, in Lasbella, the combined population coverage (Jan-Mar 2008) by PAIMAN-trained providers is about 19 percent, even though only 42 percent of private providers reported. Thus PAIMAN has successfully increased the number of facilities that can provide basic MNC. Overall the number of births has increased, but it is too early to know if these facilities can maintain the new levels. It is clear that sustained supported over many years will be necessary to achieve major changes in attitudes. Facilities to Manage Maternal and Newborn Complications: This intervention aimed to enhance access to emergency obstetric and newborn care. International standards recommend one CEmONC facility and four BEmONC facilities per 500,000 population. Generally there are enough BEmONC facilities, but most of these are private and urban. The capacity of both public and private facilities to manage complications is largely unknown but varies greatly. By these standards, Jhelum, Sukkur, Lasbella do not have enough CEmONC facilities (see Annex M). PAIMAN upgraded 13 RHC to provide BEmONC and 18 DHQ/THQ to provide CEmONC services. The project has been quite successful in advocating with the MOH to ensure that services are available 24/7 in most facilities. While some facilities are still waiting for instruments and other materials, most are now equipped and staffed to provide BEmONC services. Of the 18 CEmONC facilities, 13 are fully able to provide CEmONC. Sukkur and Lasbella suffer shortages in staff and blood supplies; PAIMAN has paid contract doctors to provide CEmONC services for two years in these districts. Because the providers are not resident at the facility, services are not available after 2 pm. It is unclear if the DOH will be able to fill these positions after the project; thus, there is a possibility that four of the 10 districts will not be able to provide CEmONC services. 20 MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT In discussions with providers many cited shortages of partograph forms, antibiotics, and injectable antihypertensives; this was worst in Lasbella and Sukkur. All the providers said that magnesium sulfate is not available. Most of the providers are more confident about identifying maternal complications after their training, but because many of them are still not comfortable managing complications, they refer these women to other providers, even though they should be able to manage the complications at their facility. Identification of obstetric complications has increased, but once again population coverage is low. In 2008, the facilities provided services to 6.58 percent of women who had complications; if private providers are included this increases to 17.2 percent. In summary, PAIMAN successfully supported establishment of 13 BEmOMC services. While some of the CEmONC facilities are not completely operational, largely due to lack of DOH staff, those that are operational have been a success. However, most of these facilities have only recently become functional, so it is too early to expect any major changes or assess quality of services. Stillbirths and Newborn Deaths: Intrauterine fetal deaths (stillbirths) are one of the RMOIs. This indicator is most meaningful when it is disaggregated by type (macerated or fresh). Fresh stillbirths can be an indicator of the quality of EmOC services, but it is often hard to track and is probably not necessary at this point in the project. Providers indicated that the majority of the stillbirths are macerated, but they are not clear about the causes. Several districts have high stillbirth rates—Rawalpindi (135), Dadu (135), and Sukkur (116), and private providers in Buner and Khanewal—that require further investigation. PAIMAN has focused on the newborn complications surrounding birth in the facility. All health care providers (HCP) interviewed commented that they feel more confident in doing neonatal resuscitation, although the project is not tracking the number of babies resuscitated or their outcomes. There has been little emphasis on managing babies with complications that are not born in the facility. The highest rates of newborn deaths are in Dadu, Sukkur, and Buner. Health Management Information System: PAIMAN provided registers to track obstetric complications and some training on complication definitions. The facilities visited had their data as bar graphs on the wall. The staff understood the key aspects of the data, but there was little understanding of how these figures related to basic population data (estimates of pregnant women in the area). Providers seemed to consider any reason a woman was admitted as a complication. Based on register reviews, the majority of complications occurred antenatally and 35–50 percent of these were incomplete abortions treated by dilatation and curettage (D&C). These limited findings suggest that few complications during childbirth or postpartum are coming to these facilities. In summary, there is great confusion about what is an obstetric complication; currently complications are reported as an aggregate, not by type. Providers will need significant support to clarify both diagnosis and reporting of complications. Although the RMOI also tracks case fatality rates, the numbers are so small that this is probably not a useful indicator for the project. Referral System: To ensure that women can receive prompt and appropriate health services, a functioning referral system is required. The referral system is evolving as providers and health facilities become functional, but the team did not find a clear referral mechanism that can be communicated to women to ensure that they do not waste time going to facilities that cannot meet their needs. For example, if a woman has prolonged labor and the closest facility cannot provide C-sections, the HCP and the woman should know which facilities, public or private, can provide these services. Referral to a higher￾level facility is not among the RMOI indicators. MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 21 SO 4. INCREASING CAPACITY OF MATERNAL AND NEWBORN HEALTH CARE PROVIDERS PAIMAN has worked to enhance the capacity of the Provincial (PHDC) and District Health Development Centers (DHDC) to conduct EMNC training; this is the first time the centers have conducted any technical training. Training on Normal Delivery: PAIMAN provided a one-month clinical training for 497 LHVs, nurses, and midwives on managing normal deliveries. Unfortunately, the training did not include the use of the partograph or active management of the third stage of labor (AMTSL), which are key aspects of normal delivery. In addition, most of the DHQ/THQs used as training sites did not routinely use these, so the LHVs did not learn these skills. However, the LHVs interviewed found the training very useful. On average 63 percent of providers who received the one-month training also received the EMNC training. Not surprisingly, this ranged from a high of 115 percent in Khanewal to less than 50 percent in Dadu, Jafferbad, Lasbella, Jhelum, and Sukkur (see Annex N). Training in Essential Maternal and Newborn Care (EMNC): Essential maternal care includes antenatal and postnatal care, management of normal deliveries, and management of nonsurgical maternal complications. The newborn component includes essential newborn care (ENC) and management of neonatal asphyxia, sepsis, jaundice, and low birth weight. This is a five-day training with lectures and practice on models, but no clinical practicum. The training does not include use of the partograph and has only brief sections on AMTSL, communication skills, and infection prevention. Training materials and protocols are in English; PAIMAN is currently translating the materials into Urdu. So far 1,599 providers have received EMNC training from Save the Children. They include managers of health facilities and service providers in both upgraded and other MOH facilities. On average 60 percent were female. In the Punjab districts, about 70 percent of the trainees were female, in Upper Dir only 33 percent, and in Dadu only 23 percent. To address the lack of female providers, in the NWFP the project is providing pre-service training for 107 LHVs. PAIMAN’s strategy is to provide EMNC training to providers in both the 31 upgraded and the nonupgraded health care facilities. The majority of providers (67 percent) that received EMNC training were from nonupgraded facilities. While training is a key input, training without adequate facility level inputs and trainee follow-up has had only limited success. Some of the upgraded facilities still lack basic inputs (e.g., magnesium sulfate), which limits their ability to use the information gained during training. Moreover, because data are only collected from the upgraded facilities, it was not possible for the team to understand what, if any, changes have resulted from the training of HCP personnel in nonupgraded facilities (see Annex N). Trainers and providers all liked the training because it was very practical, though they felt the time was very short for learning the new information and skills, and several thought there should also be clinical training. Health care providers could recite most of the danger signs, yet many found it difficult to link some of the danger signs with the associated complication. It was also challenging for many of them to express danger signs in terms that were easily understandable to pregnant women. There is very little understanding of the difference between pathologic and physiologic jaundice, for instance. Comprehensive EmONC: JSI trained 47 of a targeted 70 providers, among them obstetricians, pediatricians, and anesthetists, to deal with obstetric complications that require surgical interventions in addition to BEmONC services. The 10-day clinical training was conducted at the PIMS. PAIMAN was not able to reach the target because of government position vacancies. Training on Use of the Partograph and Active Management of the Third Stage of Labor: It does not appear that PAIMAN had identified use of the partograph and AMTSL as a weakness of the EMNC 22 MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT training, despite studies that show health facilities/staff, including teaching hospitals, do not regularly use these tools. It was fortunate for the project that the Midwifery Association of Pakistan (MAP) designed this training, under different funding. The MAP training is three days with a clinical practicum; PAIMAN training is two days without one. This training was added to PAIMAN in June 2007 for public sector providers only. PAIMAN and MAP signed a one-year agreement to train 400 female public sector providers on use of the partograph and AMTSL; this is 42 percent of female health providers trained on EMNC. To date MAP has trained 268 of them. The trainee lists are provided by the health ministries and are rarely compared with the PAIMAN training database. The current agreement with PAIMAN ends in July 2008; MAP has submitted a no-cost extension until December 2008 that is awaiting approval. Health care providers interviewed stated that none of them had used the partograph or AMSTL routinely before the training. Providers reported that the training was very useful, and many said that they are regularly practicing these skills. No training is planned after July 2008. Performance Assessment of Public Sector Providers: AKU conducted a performance assessment of providers trained in Punjab six months post-training. The pretest was repeated to assess knowledge retention, and providers had to demonstrate their skills in AMTSL and neonatal resuscitation. A sample of 38 MOs/WMOs and 38 LHVs, nurses, and midwives was tested. A passing score on the test is 80 percent. Both the MO (98 percent) and LHV (93 percent) groups retained their knowledge of essential maternal care. They also retained information about essential newborn care (MOs, 92%; LHVs, 86%). Knowledge of maternal and newborn complications was less strong. For MOs, knowledge of how to manage maternal complications fell significantly, from 97 percent on the post-test to 79 percent, which is almost at the 70 percent pretest level, and their knowledge of newborn complications decreased slightly, from 90 percent post-test to 83 percent. For the LHV group, knowledge of management of maternal complications dropped from 94 percent to 76 percent, and of newborn complications from 93 percent to 73 percent. It is particularly challenging to maintain new skills when many facilities have only recently started providing these services. The percentage of MOs who earned 80 percent or more on AMTSL skills fell from 97.3 percent on the post-test to 83.3 percent, and neonatal resuscitation skills dropped from 97.4 percent to 71.1 percent. In the LHV group, AMTSL skills dropped from 97.3 percent to 83.3 percent, and neonatal resuscitation skills from 100 percent to 76.3 percent. It is unclear how many of these providers had received the additional training on the use of the partograph and AMTSL. Infection prevention: PAIMAN has just started trained 96 of 120 providers on the WHO infection prevention model. It is too early to evaluate this activity. In conclusion, PAIMAN has trained many providers who had never had any refresher training, which is a major accomplishment. The strategy of working through the PHDC and DHDCs is important for making the system sustainable. A major weakness of the ENMC and MAP training is their lack of a clinical practicum, which is necessary to effectively change provider practices. International BEmONC training curricula have at least 10 days in the clinical rotation. UNICEF’s 15-day training on BEmONC in Pakistan includes a clinical practicum. Another weakness was omission of the partograph in the EMNC training. Lastly, training providers in facilities that are not equipped to provide services significantly reduces the effectiveness of the training and the likelihood that trainees will be able to use their new skills. Initially, the ENMC training curriculum designed by SAVE The Children included the partograph and AMTSL. PAIMAN removed these two aspects because male providers and pediatricians, who do not conduct deliveries, do not need training on deliveries. However, there is no reason why these providers need to be trained with MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 23 female providers who do conduct deliveries. It should also be noted that trainings conducted with assistance from UNICEF and JICA included partograph and AMTSL work, with the support of the MOH. Overall the EMNC training content is quite good, but several areas could be enhanced (see Annex K). Urdu training materials and protocols will particularly enhance LHV and CMW understanding of the materials. Providers’ ability to understand the difference between minor ailments and major complications could be better, as could their ability to provide such information to women is a way that is easily understandable. Based on the performance assessment, it appears that the training has been effective in enhancing providers’ knowledge and ability to manage normal deliveries. It has been harder for them to retain their knowledge and skills for managing maternal and newborn complications. SO5. STRENGTHENING DISTRICT HEALTH SYSTEMS For this objective, PAIMAN helped establish and reinforce district health management teams (DHMTs), provided management training to a broad range of health managers, and worked to improve the quality and utilization of health information systems. Devolving administrative and financial authority to the district governments has increased the role of DHMTs in improving health services. Although the concept of DHMTs is not new in Pakistan, their functionality and capacity vary greatly across the country. PAIMAN worked closely with the DHMTs and stakeholders in all districts to improve their performance. To help with this process, in 2005 an analysis of district-level decision space was conducted by Harvard School of Public Health and its partner Contech. The analysis assessed how much decision-making power devolved to the districts and the capacity of the districts to make several types of decisions. It was found that both decision-making power and the capacity to make decisions varied by district and function. Assistance included clarifying the composition of the DHMTs to allow appropriate representation, clarification of the roles and responsibilities of members, preparation of annual district health plans, and institutionalization of regular team meetings. DHMT performance was measured on such criteria as regularity of meetings, level of participation, chairmanship, decisions based on data review, and follow￾up of previous decisions. At the time of the evaluation, seven out of ten DHMTs were meeting all criteria. All districts had annual district health plans outlining activities, human resource development, district logistics and services, and budget estimates and a funds generation plan. PAIMAN also tracked the performance of DHMTs based on increases in district health budgets. From 2004–2005 to 2007–2008, district health budgets increased by 52 percent on average but there are significant district variations, with increases varying from 14 percent to 100 percent. PAIMAN conducted trainings for all health managers on strategic planning, logistics management, financial management, HR development, and supportive supervision. The training approach, based on experiential learning, incorporated presentations, brainstorming sessions, group exercises, and role-play. By the time of the evaluation, targets for management training had been achieved, but due to turnover of district staff additional managers now need to be trained. On field trips the evaluation team interviewed several district health managers. In general, they were satisfied with the quality and content of the trainings, but it is difficult to assess the impact of the trainings and whether there were actual improvements in health systems. Several systemic deficiencies, such as inadequate human resources, frequent transfers of key staff, and inadequate support from provincial health officers, undermine the effectiveness of training efforts. In particular, MOH supervision systems seem to be weak. The project emphasized improving the quality and use of health information systems. Currently, most districts in Pakistan are using HMIS, which is the general information system used to collect service data 24 MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT on a national basis. A new district level information system, DHIS, has been piloted in eight districts. The MOH has now decided to replace HMIS with DHIS nationwide. Only one PAIMAN district, Khanewal, is among the pilot districts currently using the DHIS. In other districts, PAIMAN trained managers in HMIS. In 2007 PAIMAN conducted an LQAS (lot quality assurance sampling technique) assessment of health information systems in some districts as a way to train district managers on how to monitor HMIS performance in their own district as well as to assess HMIS performance in project districts. The report provided useful insights into the quality and use of HMIS in the districts. HMIS data accuracy is highest among all systems—on average it scores 65 out of 100—but the degree of accuracy varies greatly by district. Levels of use and feedback from district and health managers score low for all districts, with an average of 11 out of 100. The survey results show that HMIS success in producing quality data and the use of the information for decision-making depend on factors far beyond technical training and capabilities. In sum, the evaluation team concludes that PAIMAN successfully implemented health systems strengthening interventions as they were set forth in work plans, and there is evidence that the efforts have enhanced the systems in place. Achieving more significant improvements in health systems requires holistic approaches and comprehensive support to address every function and deficiency of current structures. Such a comprehensive approach is beyond the scope and resources of an MNCH project. In retrospect, there are two areas where PAIMAN’s assistance could have been more effective: 1. It would have been more useful if the project had taken a flexible approach to strengthening health systems in the districts. Districts differed significantly in terms of management capacity and decision-making power, as the District-Level Decision Space Study found. The project applied a standard approach for all districts, without customizing it to the specific needs of each. 2. There was little effort to advocate at the provincial level for building up district health systems. Provincial health officers were informed of project activities but provided little support to the districts to enhance health systems interventions. Finally, it is difficult to assess the general viability of the gains achieved through health systems interventions. The evaluation team observed varying levels of district ownership of interventions. Those districts that experience concrete benefits from doing business differently are more likely to internalize changes and sustain gains. FUNCTIONAL INTEGRATION OF SERVICES In Pakistan, responsibility for providing maternal and child health and family planning services are split between the Health and Population Welfare Departments. The Health Department has primary responsibility for MNC and offers limited family planning services, such as counseling; the Population Welfare Department primarily provides family planning services and offers some MCH services. Ideally, both types of service should be managed by a single entity and both provided in the same facilities; that would greatly improve access to and availability of services. The PAIMAN Project undertook a pilot intervention, ―functional integration of services,‖ to devise mechanisms for facilitating client access to different service outlets. Functional integration refers to better coordination and cooperation between the two departments and building and reinforcing linkages between them, which either do not exist currently or do not function properly. After a preliminary study, PAIMAN began piloting the intervention in Rawalpindi District in 2007. Besides providing a situational analysis, the study used consultative processes involving all stakeholders to identify nine areas for functional integration, among them service elements, utilization of providers, contraceptive logistics, monitoring, and communications. The stakeholders agreed to work together on the MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 25 nine areas and on tasks assigned to them. A second study conducted in April 2008 to assess the process and achievements found mixed results: some tasks were carried out successfully, others were found not to be feasible. For example, facilitation of LHW referral of surgical contraceptive cases to surgical camps worked out well. On the other hand, establishing district coordination committees to supervise the existing district technical committees was found not to be feasible. It was too soon to assess the results of some other tasks. For example, the functions of male motivators were refined and they were placed in DOH service outlets only recently; it will take at least a year to evaluate their efficacy. The team thinks that functional integration of services is an innovative intervention and could have high impact if successful. In Rawalpindi, both departments are closely involved with and supportive of the intervention. The team believes it is likely to achieve results in Rawalpindi, where many of the preparatory elements are already in place and senior staff of both departments are highly committed and cooperative. It should be noted that this is a new intervention, having been on the ground for only a year. It is too early to assess results in terms of increased service utilization. While our observations are in line with the findings of PAIMAN’s assessment report, we believe that it is too soon to expand to other districts as the report recommended. It should also be recognized that Rawalpindi is very different from other districts—it is more developed, systems there work better, and the capacity of both departments there is higher than elsewhere in Pakistan. 26 MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 27 V. CONCLUSIONS The following summarizes the team’s conclusions: PAIMAN has been successful in putting in place the building blocks to enhance MNC through training, communication and outreach, and facility renovations. The foundation is now in place for increased utilization of services. However, it has taken the first half of the project to lay the groundwork for building the capacity of local organizations, health care providers, and managers, and much of this work is still ongoing, such as the training of CMWs, private providers, and LHWs. The project staff is well respected by all stakeholders and the project has been successful in building close alliances with partners at all levels. PAIMAN has worked tirelessly and made much progress despite many deficiencies in the health system, such as inadequate human resources at the MOH and frequent transfers of key staff. Project implementation was delayed in some areas because of unpredictable security and natural calamities. MOH contacts view PAIMAN interventions as important contributions to their work, but it will take continued efforts for the MOH to internalize and embed project activities into the National MNH program. For example, at the community level it is widely believed that PAIMAN will provide ambulances to CMWs and that it will continue to pay for LHVs in birth centers. While some of these perceptions may stem from overly ambitious expectations for the project, PAIMAN needs to ensure that it is promoting improved MOH services, not promoting itself. The abundance of partners and the complex division of labor among them has complicated program implementation. Since the staff of the partner organizations do not constitute a solid team (partner staff are not seconded to the project), gaps and redundancies are inevitable, and coordinating activities is very complex. For example, there are four partners implementing BCC interventions at different levels and locations; knowledge about other partners’ activities and coordination with them is poor. Similarly, four partners have roles in training different cadres of staff on a variety of topics. The project has established several coordination mechanisms, but they require considerable time and effort on the part of both JSI and its partners. At the request of USAID, PAIMAN focuses primarily on monitoring activities and has established a comprehensive system to track performance against targets. While this is a good internal monitoring mechanism, the level and numbers of activities tracked seem excessive. There is too much emphasis on monitoring activity and not enough RMOI analysis to strategically guide the program. Table 1 compares the findings of the project’s 2005 baselines to the 2008 FALAH baseline conducted in eight of the ten PAIMAN districts. These results are compared with project targets for 2007 and 2009. 28 MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT Table 1: Summary of Results of Outcome Indicators IR INDICATOR PAIMAN BASELINE 2005 TARGET 2007 FALAH FINDINGS 2008 TARGET 2009 1. % of births assisted by skilled birth attendants 36% 38% 38% 45% 2. % women who received 3 or more ANC visits during the last pregnancy 27% 35% 35% 50% 3. % pregnant women who received at least 2 doses of TT during the last live birth 40% 55% 43% 75% 4. % women who had a postpartum visit within 24 hours of giving birth 34% 38% 39% 45% 5. # facilities upgraded and meeting safe birth and newborn care quality standards 0 - 26 31 6. % increase in district health budgets - - 52% 50% As of June 2008 the project had achieved five of the six targets for 2007. It has been hard to reach the goal for ―% of pregnant women who received 2 doses of TT vaccinations‖ because TT vaccinations are primarily provided by the LHVs, who are few in number, and there are major cold chain issues. Although RMOI data indicates a slight upward trend in numbers of women receiving second TT shots, unless the project implements new approaches to increase supplies, it is unlikely that the project will be able to reach its 2009 target. What it is not reflected in Table 1 are the vast differences between districts for all indicators. For example, the percentage of women who deliver with a skilled birth attendant varies from 68 percent in Jhelum to just 22 percent in DG Khan district. Aggregated data thus have limited use for decision making and planning for future interventions. It should be noted that the final targets are significantly higher than the 2007 targets. The team is concerned that at the current implementation rate, it is unlikely that the project will achieve the final targets for indicators 1-4. For example, Indicator #1, ―% of births attended by SBA,‖ increased by only 2 percent between 2005 and 2008. It is not realistic to expect that it will improve by 7 percent within the next year. It should also be understood that even if final targets are achieved, it is too early to expect significant population impact in many districts. This program is still in its early years; it will require sustained continued support to reach critical mass and see changes at the population level. PAIMAN upgraded 18 hospitals to provide CEmONC services and 13 RHCs to provide BEmONC. These facilities vary in their ability to manage normal births as well as maternal and newborn complications. Six of the 18 hospitals are not able to provide comprehensive EmONC due to lack of staff. PAIMAN has been advocating for these inputs, but some of the inefficiencies are beyond its control. The project documents show confusion about the difference between minor pregnancy and newborn ailments (e.g., vomiting, jaundice) and major complications that lead to death. This began with the way the baseline findings are reported, leading to the development of the BCC messages on danger signs, the health care provider training curriculum, and record keeping. For example, the Buner MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 29 baseline states that 49.7 percent of women experienced a complication, yet it is well known that only 15 percent of all pregnant women have an obstetric complication. The BCC messages and training curriculum are not specific enough to make the necessary distinctions. Greenstar has trained female private health care providers, largely in urban areas. They used the PAIMAN training curriculum, which unfortunately does not include a clinical practicum or training on use of the partograph. While Greenstar provides educational sessions after the training, they do not include observation of deliveries or complications. There does not appear to be a system for EMNC post-trainee follow-up to assess knowledge and skills. By working through existing infrastructures (EDOs and DHMTs, the LHW program, pre-service training of LHV, and CMWs) and with current private providers, PAIMAN has made great strides toward developing sustainable systems that can be improved. However, there are many other interventions (e.g., paying LHV and contract doctor salaries) that are less sustainable; careful exit strategies are needed for these activities. The evaluation team also identified systemic efficiencies and gaps on site, such as inadequate supervision and referral schemes, that have not been addressed by (and are beyond the scope of) the PAIMAN project. Until these building blocks are in place, it is not realistic to expect that the gains of this project will be truly sustainable. The functional integration of services intervention in Rawalpindi is innovative and if successful could have high impact. The team believes that it will most likely achieve results there because many of the preparatory elements are already in place and senior staff of both GOP departments are highly committed and cooperative. However, because this intervention has only been in place for a year, it is too early to assess whether it has increased service utilization. 30 MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 31 VI. RECOMMENDATIONS AND FUTURE DIRECTIONS Our specific recommendations for USAID/Pakistan’s continued support for MNCH are categorized as either short-term or medium-term. Short-term recommendations, which are envisioned for the next 14 months, relate to programming in the remainder of the PAIMAN project. Medium-term recommendations for the five years through 2014 are provided to help USAID/Pakistan design future MNH programs. SHORT-TERM RECOMMENDATIONS FOR THE PAIMAN PROJECT In the past four years PAIMAN has put in place many of the building blocks for improved maternal and newborn services. However, sustained focus on priority interventions through the end of project is required, particularly because the project is already expanding into a new district and two FATA agencies. The project—already overstretched—will be challenged to reach outcome targets. The evaluation team believes it will take an additional 1–2 years for the project to reach its objectives. It is vital that the project focus on sustaining gains achieved so far and improving on areas that are still weak. This recommendation is based on the limitations of the absorptive capacity of local systems rather than the availability of USAID resources. If PAIMAN receives additional funding, some of it should be invested in increasing population coverage within the districts where the project is already operating. For example, where useful BCC has already taken place, it should renovate additional hospitals and clinics in strategic locations and train their staff, because demand will presumably increase in a short time. For political reasons the project might be urged to use additional funding to expand into other districts. In that case the team suggests that this be done strategically, based on lessons learned from the PAIMAN experience, in order to improve the interventions. The evaluation team did not expect to see major changes at a population level at this point in the project. However, now that many of the basic elements are in place, PAIMAN needs to track quality and increases in use of services against changes needed to begin to see population-level impact. It should collect information from the public sector, upgraded and nonupgraded facilities, and Good Life providers to understand district and tehsil coverage. Besides tracking activities, PAIMAN and USAID should analyze and use RMOI data to assist in strategic decisions. There is also a need for more careful analysis of district data given the vast differences between districts. The evaluation team recommends that PAIMAN no longer track fatality rates for maternal or newborn complications or stillbirths because the numbers are small and not meaningful. PAIMAN consortium partners, led by JSI, should convene as a unified team to clarify their responsibilities related to the following; the evaluation team recommends that the consortium strategically focus on the following priority interventions in the remaining 14 months of the project: – PAIMAN should work with the National MNH and LHW programs to simplify and consolidate the number of BCC messages and target groups. It should also clarify messages on major maternal and neonatal danger signs and the definition of SBA. – To effectively increase knowledge and change behavior, PAIMAN should ensure that messages are reinforced through careful selection of activities. PAIMAN should consider segmenting messages by theme (e.g., antenatal care) and conduct intensive campaigns for three months on one subject, so that the same messages are reaching target groups via numerous channels, such as interpersonal outreach, theater, and television. 32 MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT – LHWs and CHWs reach the largest numbers of beneficiaries and their work should continue. PAIMAN should conduct a mini-evaluation of the other BCC activities (events and theater) to determine the two or three most effective strategies. These interventions should be scaled up and the rest dropped. Where possible, events should only be conducted if they provide service in inaccessible areas. – PAIMAN should create new CCBs wherever viable and support existing CCBs to sustain key NGO activities (e.g., birthing centers, emergency transport schemes). The NGOs will require at least one more year of funding to make CCBs operational. – It is vital to ensure that graduating CMWs can effectively function in their communities. While there are many options for supporting and supervising CMWs, there are no final plans. PAIMAN needs to draft a clear plan for establishing and supporting them in the communities, one that clarifies their roles and working relationships with LHWs and TBAs. – A review of Good Life providers should be conducted, including (1) mapping of their locations; (2) assessment of post-training knowledge and skills; (3) clinical review of the quality of care; and (4) review of services data to learn what services are provided and what types of clients are receiving them. – PAIMAN should conduct a referral mapping exercise by district to identify the capacity of both public and private health facilities. Once this is completed, a communication plan should be developed to inform all health care providers and other influentials about facilities in their area. – More attention and support should be given to tightening supervision of both public and private services. – Health care providers in both sectors require further on-the-job training and support to accurately diagnose, analyze, intervene, and record clinical detail for obstetric and newborn complications. – To ensure that the 31 upgraded facilities are providing quality EmONC services, PAIMAN should allocate enough human resources to regularly monitor quality of care and provide on-the￾job training for health care providers trained on EMNC. A similar system should be established for private providers. – EMNC training, for both public and private providers, needs to incorporate clinical training that is consistent with international standards. It should include use of the partograph, AMTSL, infection prevention, and counseling skills, rather than bringing health care providers back for separate trainings. This will require a major review of the curriculum, which will offer an excellent opportunity to update technical content as well. – PAIMAN should retrain district managers in DHIS. Apparently DHIS has many advantages over the HMIS now that health services are being devolved to the districts. Because MOH seems committed to rolling out DHIS nationwide, it makes sense for PAIMAN to help districts adopt it effectively. – PAIMAN should continue to provide regular on-the-job mentoring for DHMTs. – PAIMAN should undertake a final evaluation of its functional integration of services intervention in Rawalpindi to assess results and make final modifications and recommendations for future expansion. MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 33 LONG-TERM RECOMMENDATIONS FOR USAID The Mission has informed the team that the project will be receiving additional health funding. Without knowing the amount and the geographic area of expansion, the team can provide only general principles. The guiding principle is to be strategic: keep the focus on fewer but potentially higher-impact MNH interventions. PAIMAN has achieved a lot in four years, especially considering actual implementation has been going on for only two. It is not reasonable to expect MOH and the other local partners to take full responsibility for current activities in the next year. The evaluation team recommends that USAID continue technical assistance to the PAIMAN districts, tailoring it to the specific needs of each district. The evaluation team suggests phased graduation of PAIMAN-supported districts over the next five years. High-performing districts could be graduated as they prove a certain level of sustainable quality service before USAID assistance phases out and initiates similar interventions in new, preferably neighboring, districts. The graduated districts then could be encouraged to serve as role models to facilitate implementation in the new districts. The PAIMAN experience so far provides a wealth of insights into what works, as the team has tried to capture in this report. A follow-on project should be encouraged to build on the lessons learned and sustain the gains of the PAIMAN project while giving attention to areas that are still weak, rather than trying out new approaches. Specifically, a follow-on project should be designed in light of the following: – Behavior change messages need to be refined before expansion occurs. – The project should have an obstetrician/woman medical officer to oversee the technical quality of training and service delivery interventions. – Referral systems should be specified when an intervention is introduced. – Selection of health facilities to be upgraded should be strategic and linked with community interventions. – Because teaching hospitals will be crucial, they need to be upgraded as training sites. – The training strategy should focus on staff in upgraded facilities; staff in other facilities should not be trained until they can operate to an environment where they can practice their new skills and be supervised. – Strengthening district health systems is crucial. – A follow-on project should have fewer consortium partners. – Closer collaboration with the national MNH program—coordination on standardization, embedding project activities into the national program, and promoting MOH’s ownership— should be a high priority. For example, all activities and materials supported by the project should include the MOH logo first, before USAID’s. Progress on the functional integration of services in Rawalpindi is promising. USAID should consider expanding the initiative to other districts as part of the new FALAH project once the intervention has been thoroughly evaluated. 34 MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT Finally, the evaluation team recommends that USAID take a more active role in coordinating the MNH activities of international agencies and organizations. There is a need for firm leadership in the donor community for closer and more structured collaboration. MID-TERM EVALUATION OF THE USAID/PAKISTAN IMPROVED CHILD HEALTH PROJECT IN FATA DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 37 ACKNOWLEDGEMENTS This report represents a collective effort, incorporating the opinions and observations of several individuals who generously gave their time to talk with the evaluation team. The team especially would like to thank William Conn for his guidance throughout the evaluation process and Dr. Amanullah Khan, Deputy Director Health of Save the Children’s Pakistan Country Office, for making his staff and project information so readily available. Special thanks go to our local partner team, Eycon, for conducting field trips and interviews in FATA. Among the most valuable contributions to this evaluation effort were the observations and insights offered by members of the FATA Health Department and FATA Secretariat, who graciously gave the team a better understanding of the realities and the child health needs in FATA. To all who are committed to improving child health in FATA, Pakistan, we offer these observations and recommendations in the confidence that further progress will be realized through continued collective efforts. 38 MID-TERM EVALUATION OF THE USAID/PAKISTAN FATA PROJECT MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 39 ACRONYMS ARI Acute respiratory infections AHQH Agency headquarters hospital AHMT Agency health management team ANC Antenatal care BCC Behavior change communication BHU Basic health unit CCM Community case management CD Civil dispensary CDD Control of diarrheal diseases CH Civil hospital CHC Community health center CHD Child Health Days CHW Community health worker CMS Community mobilization strategy ENC Essential newborn care FATA Federally Administered Tribal Areas FMT Female medical technician FR Frontier regions HCP Health care provider HF Health facility HMIS Health management information system ICHP Improved Child Health Project IMNCI Integrated management of newborn and childhood illnesses IMR Infant mortality rate IR Intermediate result LHV Lady health visitor LHW Lady health worker M&E Monitoring and evaluation MCHC Maternal and Child Health Center MNCH Maternal, neonatal, and child health MO Medical officer NGO Nongovernmental organization NMR Neonatal mortality rate ORS Oral rehydration salts QIT Quality improvement team RHC Rural health center SC Save the Children TBA Traditional birth attendant TT Tetanus toxoid 40 MID-TERM EVALUATION OF THE USAID/PAKISTAN FATA PROJECT USAID United States Agency for International Development WHO World Health Organization WRA Women of reproductive age MID-TERM EVALUATION OF THE USAID/PAKISTAN FATA PROJECT 41 CONTENTS MID-TERM EVALUATION OF THE USAID/PAKISTAN FATA PROJECT ACKNOWLEDGEMENTS ............................................................................................................................37 ACRONYMS ................................................................................................................................................39 EXECUTIVE SUMMARY .............................................................................................................................43 MAJOR FINDINGS AND CONCLUSIONS ...........................................................................................43 SHORT-TERM RECOMMENDATIONS ................................................................................................44 LONG-TERM RECOMMENDATIONS FOR USAID .............................................................................44 I. INTRODUCTION .............................................................................................................................45 PURPOSE OF THE EVALUATION ......................................................................................................45 EVALUATION METHODOLOGY ..........................................................................................................45 BACKGROUND ....................................................................................................................................46 II. OVERVIEW OF IMPROVED CHILD HEALTH PROJECT ..............................................................49 PROJECT DESIGN AND IMPLEMENTATION APPROACHES ...........................................................49 MONITORING AND EVALUATION PLAN ............................................................................................49 ORGANIZATIONAL STRUCTURE AND MANAGEMENT ...................................................................50 RELATIONSIPS, COORDINATION, AND COLLABORATION ............................................................50 III. FINDINGS AND LESSONS LEARNED .......................................................................................51 IR1: INCREASE ACCESS TO AND AVAILABILITY OF CHILD HEALTH SERVICES AT FACILITY AND COMMUNITY LEVELS ................................................................................................51 IR2: IMPROVED QUALITY OF CHILD HEALTH SERVICES AT FACILITY AND COMMUNITY LEVELS .........................................................................................................................52 IR 3: INCREASE COMMUNITY KNOWLEDGE AND ACCEPTANCE OF SERVICES AND BEHAVIORS .........................................................................................................................................53 IV. CONCLUSIONS AND RECOMMENDATIONS FOR THE FUTURE ..............................................55 GENERAL CONCLUSIONS .................................................................................................................55 SHORT-TERM RECOMMENDATIONS FOR ICHP AND USAID .........................................................55 LONG-TERM RECOMMENDATIONS FOR USAID .............................................................................56 42 MID-TERM EVALUATION OF THE USAID/PAKISTAN FATA PROJECT MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 43 EXECUTIVE SUMMARY The purpose of this evaluation was to conduct a midterm review of the Improved Child Health Project (ICHP) implemented by Save the Children in the Federally Administered Tribal Areas (FATA) of Pakistan. The evaluation assessed whether the project is achieving intended results, reviewed its organizational structures and technical approaches, and provided recommendations for future directions. A team of three consultants conducted the evaluation in June-July 2008 through the Global Health Technical Assistance Project. The team interviewed a range of stakeholders, including health managers, service providers, trainees, and community members met during field visits to project sites. MAJOR FINDINGS AND CONCLUSIONS The strategic objective of the project is ―to increase use of key child health services and behaviors‖ in FATA. ICHP aims to increase access to and availability of quality child health services and to increase knowledge and acceptance of child health services and behaviors at the community level. Working in FATA is increasingly challenging as security worsens. Although ICHP is behind on its set targets, it has still made significant progress. Effective relationships have been established with FATA officials, health care providers, and community leaders, which has greatly facilitated implementation in the last six months. A major obstacle to project implementation has been shortages of MOH health staff, especially female staff, worsened by staff absenteeism due to the fluid security situation. To support program implementation and facilitate reinforcement of health system management, ICHP helped to form an agency health management team (AHMT) for each agency in FATA. To increase access to child health services, ICHP has started major and minor renovations of health facilities, upgraded medical stores, and facilitated Child Health Days (CHDs). CHDs aim to ensure that health care providers, medicines, and vaccines are available on the same day to increase community access to needed services. The CHD approach seems to be successful, although inadequate supplies of essential medicines continues to be a major issue. The project has asked USAID to provide a waiver so it can purchase medicines. To improve the knowledge and skills of health care providers, the project conducts trainings in essential newborn care (ENC) and in integrated management of newborn and childhood illnesses (IMNCI). It is too early to assess the results of these trainings. The project formulated a community mobilization strategy to identify areas for intervention and to help bridge the gap between service providers and communities. Key activities to increase knowledge and promote healthy behaviors for child health are increasing community awareness through lady health workers (LHWs) and community health workers (CHW) managed through subgrants with local NGOs; community sensitization events, such as formation of local committees; and advocacy at agency, tehsil, village, and facility levels. The evaluation team concludes that the project can make a substantial contribution to increasing access to and the availability of quality child health services and increasing knowledge, awareness, and healthy behaviors related to child health issues, if the security situation allows them to implement the interventions. 44 MID-TERM EVALUATION OF THE USAID/PAKISTAN FATA PROJECT For security reasons the project, through its ―no-visibility‖ policy, has been very careful to work in the name of local agencies and agency health services. This approach is likely to improve the sustainability of interventions. CHDs seem to be a successful strategy for encouraging the community to access child health services. The major challenges for CHDs are ensuring adequate staffing and availability of essential medicines. SHORT-TERM RECOMMENDATIONS Since project momentum is just building, we believe that the project should be extended for two years, through 2011, so it can achieve the targets initially set. ICHP needs to carefully review its expansion plan for CHDs, mainly to ensure that health care providers are not pulled to this activity at the expense of routine services in their health facilities. USAID should expedite the medication waiver to ensure supplies of essential child health medicines. Meanwhile, ICHP should continue its efforts to secure medicines through a variety of mechanisms and explore the possibility of piloting commercial sales of medicines. To further build relationships with teaching hospitals, ICHP should provide some support for the teaching facilities in Peshawar. ICHP should be commended for securing WHO trainers and arranging to conduct training in teaching hospitals, which ensures adequate training caseloads to ensure quality training. For improved uptake of messages the project should keep behavior change communication (BCC) messages simple and the language used consistent for different groups. LONG-TERM RECOMMENDATIONS FOR USAID Since FATA is culturally, politically, and structurally different from the rest of Pakistan, USAID should design an integrated maternal, neonatal, and child health (MNCH) project with a water and sanitation component specifically for FATA. Depending on the funding available, the project could be expanded to cover other areas of primary health care. USAID needs to continue to advocate with FATA officials to enhance resources for MNCH. USAID and the project also need to engage traditional tribal leaders (Jirga elders) in areas where the project is working for greater support and buy-in from community leaders. It would be useful to provide pre-service training for women from FATA to become lady health visitors (LHVs) to increase the number of female health care providers who reside and work in these areas. MID-TERM EVALUATION OF THE USAID/PAKISTAN PAIMAN PROJECT 45 I. INTRODUCTION PURPOSE OF THE EVALUATION The purpose of this evaluation is to provide the United States Agency for International Development’s Mission to Pakistan (USAID/Pakistan) with an independent mid-term evaluation of its maternal, newborn, and child health (MNCH) programs. These programs, which are managed by the Office of Health, are implemented through two projects. PAIMAN is a maternal and newborn project led by John Snow Inc.; Save the Children implements the Improved Child Health Project (ICHP) in the Federally Administered Tribal Areas (FATA). The objectives of this evaluation are to: Assess whether MNCH program partners are achieving their goals and results and meeting benchmarks in the cooperative agreements and work plans. Evaluate the effectiveness of the management structure, administrative support, cost and partnerships, and collaborative plans. Evaluate the effectiveness of key technical components and approaches of the MNCH program. Establish whether demand for MNCH services is growing in project districts as a result, direct or indirect, of these projects. Document lessons learned and provide management, administrative, and technical recommendations for improving the efficiency and effectiveness of MNCH programs. Provide suggestions and options for future directions for the program, with the potential for program expansion at various levels of additional funding. The Evaluation Scope of Work is found in Annex A. This report is confined to evaluation of the Improved Child Health Project in FATA; the PAIMAN project is evaluated in a separate report. EVALUATION METHODOLOGY The evaluation methodology consisted of Review of background documents (Annex B); Analysis of program and financial data and survey results; In-depth discussions with USAID/Pakistan and Save the Children (SC) staff; Structured interviews with contacts in the Directorate of Health and Population Welfare Department of FATA, the FATA Secretariat, donor organizations, implementing partners, and other stakeholders (Annex C); and Field trips to Peshawar and the Orakzai and Khyber Agencies in FATA to visit the project office and observe service delivery, training, and community interventions, during which the team interviewed health managers, service providers, trainees, and community members (Annex C). The team used both qualitative and quantitative methods during the evaluation. Quantitative information was derived from data collected by the projects. Qualitative information was generated through interviews and observations at health facilities. A local firm, EyCon, was 46 MID-TERM EVALUATION OF THE USAID/PAKISTAN FATA PROJECT recruited to conduct interviews and field visits in Orakzai and Khyber Agencies where the international team could not travel. To help ensure that comparable information was collected during field visits, the team drafted standard interview guides. These data collection tools were designed to reflect the questions posed by the scope of work and were used to interview various health managers and providers. The evaluation was conducted in June-July 2008 through the Global Health Technical Assistance Project; the in-country work took five weeks. The team was composed of Pinar Senlet, Team Leader; Susan Ross, Maternal Health Specialist; and Jennifer Peters, Behavior Change and Communications (BCC) / Community Mobilization Expert. BACKGROUND FATA is a belt of seven semi-autonomous tribal agencies with a total population of 3.6 million people stretching north to south along the border between Pakistan and Afghanistan. More than 97 percent of the population lives in rural areas. The 460 or more health facilities in FATA are severely underequipped and regularly not functional due to staff absenteeism. Community maternal and child health coverage through lady health workers (LHW) is low, ranging from 13 percent in Bajaur Agency to 55 percent in Kurram. The 926 or so LHWs in FATA reach about one-third of the population. Child health indicators are generally poor throughout Pakistan. The under-5 mortality rate is high (94/1,000) and 82 percent of these deaths occur in the first year of life. Based on recent studies, the infant mortality rate (IMR) in Pakistan fell from 90.5 per 1,000 live births in 1990 to 78 in 2007. Unfortunately the neonatal mortality rate (NMR) has actually increased (from 51 to 54). Of even greater concern, the proportion of neonatal deaths has increased from 56 percent to 69 percent of the IMR, and 37 percent of children under 5 are chronically malnourished. The most common causes of childhood death are preventable childhood diseases, acute respiratory infections (ARI), and diarrheal diseases. Though immunization rates have increased from 21.8 percent in 1990 to 47 percent, they are still low. Six percent of children receive no vaccinations at all. Water and sanitation-related diseases are responsible for 60 percent of child deaths in Pakistan; it is estimated that diarrhea kills 230,000 children under 5 each year. In FATA the child health situation is even worse: under-5 mortality is 135/1,000 live births, and the IMR is 83/1,000 live births. Every year more than 100,000 FATA children under 5 suffer either diarrhea or ARI, which are easily treatable. A 2002 WHO bulletin indicated that neonatal tetanus alone was the cause of 22 percent of all infant mortality and 36 percent of neonatal deaths in FATA. Moreover, only 50 percent of women are vaccinated against tetanus. With less than 20 percent of deliveries conducted by a skilled provider, women in FATA face one of the highest maternal mortality rates in the world. The situation is exacerbated by conditions in FATA, where female literacy is the lowest in South Asia. A lack of information makes it harder to improve child health. For example, despite the proven benefits of providing newborns and infants with no other food but breast milk, only 16 percent of infants are exclusively breastfed during the first four months of life. FATA is a socially conservative society with very limited mobility for women. Patriarchal tribal traditions limit the role of women to household affairs and reproduction. Conservative values not only restrict female mobility but also hinder their access to services. Women’s lack of information about facilities and services, the nonacceptance of male service providers, lack of trust in modern medicine, misconceptions that vaccinations are birth control devices, and fatalism about morbidities and mortalities all contribute to declining health and social indicators. MID-TERM EVALUATION OF THE USAID/PAKISTAN FATA PROJECT 47 Socioeconomically, FATA is much poorer than Pakistan as a whole. Moreover, the security situation in FATA is not only dire but worsened over the several months preceding this evaluation. Many of the FATA agencies are designated as ―no go‖ areas for the staff of international organizations. Numerous criminals are now taking shelter in FATA, where provincial police are prohibited from entering. Tribal law, kidnappings, other criminal activities, and post-Afghanistan conflict factors are acute operating and security concerns for the project in most areas. SC conducted a household survey (November 2007–February 2008) to identify knowledge and attitudes about newborn, infant and child health care practices. Key findings were that Less than half the children are fully immunized (ranging from 53 percent in Bajour to 28 percent in Khyber Agency). 30 percent of women received no antenatal care (ANC) and 40 to 50 percent did not receive any tetanus toxoid (TT) vaccinations. 50 to 60 percent of women deliver at home with the help of traditional birth attendants (TBAs) or female relatives. 70 to 84 percent of children receive Vitamin A supplements. Poor perceptions about colostrum lead to delays in breastfeeding: over 80 percent of newborns receive green tea or ghutti before breastfeeding. Diarrhea is prevalent due to poor sanitary conditions, contaminated drinking water, and inadequate hand-washing practices. Knowledge of danger signs is fairly high, but understanding of appropriate treatments is not. Simultaneously, SC assessed 464 health facilities, and found that Less than 3 percent of health care providers have been trained on case management of diarrhea and ARI, and those who have are based at agency headquarters hospitals (AHQH). Shortages of essential medicines, particularly injectable antibiotics, are a major problem. On average, only 28 percent of all health facilities offer vaccinations; the range is from 85 percent of rural health centers (RHC) and 53 percent basic health units (BHU) to 14 percent of civil dispensaries (CD). Over 33 percent of health facilities need major repair, and another 24 percent need minor repair. 47 percent of health facilities lack adequate boundary walls, reducing the security of the services they offer. Though 70 percent of health facilities have electrical connections, 90 percent do not have a continuous electric supply, which requires generators. Reporting through the health management information system (HMIS) is almost nonexistent. There is a chronic shortage of health care providers and absenteeism is very high, particularly in RHCs and BHUs. 48 MID-TERM EVALUATION OF THE USAID/PAKISTAN FATA PROJECT MID-TERM EVALUATION OF THE USAID/PAKISTAN FATA PROJECT 49 II. OVERVIEW OF IMPROVED CHILD HEALTH PROJECT PROJECT DESIGN AND IMPLEMENTATION APPROACHES SC implements the ICHP Improved Child Health Project in partnership with the FATA Health Directorate and two local NGOs. The three-year project was launched in October 2006 and was expanded in March 2008 to cover six Frontier Regions (FRs) bordering FATA. The strategic objective of the project is ―to increase use of key health services and behaviors‖ so as to improve the health status of children in FATA. This objective contributes directly to USAID Strategic Objective 7, ―Improved Health in Vulnerable Populations.‖ Based on the findings of the household and facility surveys, the project specified three intermediate results (IRs): IR1: Increased access to and availability of health services IR2: Improved quality of health services IR3: Increased knowledge and acceptance of key services and behaviors at the community level. The project is centered on the global evidence for child health interventions that most effectively reduce under-5 mortality. Primary interventions are improving immunization coverage, preventing and treating ARI, controlling diarrheal diseases, improving essential newborn care, and growth monitoring and Vitamin A supplementation. The seven agencies contain approximately 317,000 children under 5. The project aims to reach all these children with preventive services and to provide access to curative services for the estimated 209,000 children who suffer from ARI and diarrhea. Some 1.5 million women of reproductive age (WRA) and men will benefit from community mobilization and other activities. Though the project design and approaches are sound and thorough, implementing them in FATA is challenging. The fluidity of security in the area is the main obstacle to improving accessibility and service coverage in most FATA agencies. Security concerns have led to difficulties in finding and maintaining project staff for critical positions and have impeded movement of staff within FATA. Moreover, within the MOH shortages of staff, especially female, and high absenteeism undermine project implementation. Thus there have been delays in initiating planned activities for reasons beyond the project’s control. Yet the project has still made significant progress, as is discussed in detail below. MONITORING AND EVALUATION PLAN ICHP’s monitoring and evaluation (M&E) plan is designed to track the progress of various activities and the effectiveness of their outcomes. The project uses a small number of indicators to collect process, output, outcome, and impact data. Impact data are collected only through baseline and final surveys. The project plans to collect data regularly on all other indicators. At the time of this evaluation, the project was mainly reporting on process and output; most data on outcome indicators were not yet available. ICHP reports to USAID quarterly and annually, but the reports are not shared with other stakeholders. However, the project keeps all stakeholders informed of project activities and progress by sharing other documents. 50 MID-TERM EVALUATION OF THE USAID/PAKISTAN FATA PROJECT ORGANIZATIONAL STRUCTURE AND MANAGEMENT ICHP is implemented by the project office in Peshawar. The office is headed by the project manager and staffed with technical, administrative, and financial personnel. There is a coordinator and an assistant coordinator placed in each agency and frontier region. The coordinators manage project implementation in their agency. The assistant coordinators have technical responsibility for implementing activities and are closely involved in training and services. All staff in FATA agencies and FRs are locally hired, and at the time of the evaluation most of the posts were filled. However, during the start-up period it has been a challenge to recruit and retain qualified local staff. The project manager in Peshawar reports to the deputy director for health in the SC Pakistan Country Office in Islamabad. The deputy director for health provides direction for and supervises the Peshawar office, and Pakistan Country Office staff provides technical and financial backstopping. The project has a decentralized structure with many of the technical and programmatic responsibilities delegated to the Peshawar office. Annual work plans and quarterly and annual reports are prepared in Peshawar and then fine-tuned and approved in Islamabad. The project is currently hiring a senior health advisor for the Peshawar office to make it possible to delegate more technical responsibility to the field. The Peshawar office manages the project’s finances, with oversight by the SC country office, and it also manages the subgrants with two local NGOs. Its financial/accounting systems and records are thorough. The total project budget for the period September 2006–August 2009 is $14.75 million, of which $11.5 million is for FATA and $3.25 million for the FRs. As of May 2008 the project had been allocated $8.4 million but had spent only $2.67 million, 18 percent of its total budget. RELATIONSIPS, COORDINATION, AND COLLABORATION ICHP has a team of hard-working and committed staff. It has built productive partnerships with all stakeholders. The project’s contributions are praised and valued by the FATA Department of Health and the FATA secretariat. The evaluation team was particularly impressed by the involvement in project activities and the ownership of the FATA secretariat. ICHP has been responsive to USAID and has formed satisfactory relationships with other USAID-funded projects active in FATA. It has been in daily interaction with the USAID CTO recently assigned to Peshawar. MID-TERM EVALUATION OF THE USAID/PAKISTAN FATA PROJECT 51 III. FINDINGS AND LESSONS LEARNED IR1: INCREASE ACCESS TO AND AVAILABILITY OF CHILD HEALTH SERVICES AT FACILITY AND COMMUNITY LEVELS Of the 406 health facilities identified as needing renovation, ICHP has begun renovations, both major and minor, in 111 of them, including six agency headquarters hospitals (AHQH). AHQH pediatric wards have been upgraded and child play and waiting areas established. All agency medical stores have received medicine shelves, refrigerators, and exhaust fans. ICHP plans to train staff in logistics management but has not yet done so. The project has established resource centers in all the agencies, providing furniture, whiteboards, multimedia equipment, and computers. These centers are used to conduct training for health care providers and health managers. ICHP has three main strategies to increase access to child health services: (1) facilitating of Child Health Days (CHDs) in FATA health facilities; (2) supporting MOH mobile outreach services in FATA; and (3) working with private providers on case management of ARI and diarrhea. Child Health Days: The key strategy to expand access to health services is through CHDs. Since health care providers are often not present at their assigned facility, access to services, particularly in rural areas, is very poor. CHDs are a way to ensure that providers, medicines, vaccines, and micronutrients are all available to the community on the same day. The LHWs mobilize the community, and community health workers (CHWs) inform families about services available at local health facilities on CHDs and encourage them to use them. CHDs are held at a variety of health care facilities, including CDs, BHUs, and RHCs. Some of the facilities are adequately staffed with medical officers (MO), lady health visitors (LHV), and vaccinators and can provide services without additional resources. However, because CDs have only a dispenser and BHUs have no medical officer, providers from other facilities must go to these facilities to provide a CHD. During field visits, the team found that at the CD and BHU level, LHVs and vaccinators were not always available on CHDs, which limited the services provided. The CHD approach seems to be very successful. To date, the 844 CHDs conducted have reached a total of 128,000 beneficiaries, an average of 151 people a day. About half the participants are children under 5 and 17 percent are WRA. It should be noted that currently only MOs have been trained in integrated management of newborn and childhood illnesses (IMNCI). The project plans to train LHVs and medical technicians (MTs) in IMNCI but has not yet done so. Thus, unless an MO is present at the CHD, no one has updated information on managing ARI or diarrheal disease. CHDs are provided on a rotating schedule at different health facilities throughout the agencies. This flexible approach allows health care providers to be moved to facilities where staffing is not adequate for a CHD. The frequency of CHDs ranges from once a week in some health facilities to once a month in others, depending on the location. The team’s field visit found that even CHDs once a week are often not frequent enough to cover the entire health facility catchment population. The project plans also to have CHDs in the communities in the future, working in close coordination with FATA officials, health care providers, and community leaders. A major issue for CHDs is access to essential medicines. During the team’s field visit, there were shortages of antibiotics and malaria medications in RHCs, BHUs, and CDs. Although when the concept of CHDs was first introduced FATA officials thought they would be able to provide medicines, this has not been possible. The project had not planned on procuring medicines, which is not normally allowed with USAID funding. SC has already spent its entire country discretionary budget supporting the MOH with medicines in the region. The project has applied to USAID for a waiver to buy medicines, but that is only a short-term solution. Both the project and FATA officials are keenly aware that the majority of community demand is for curative services. Thus, the success of the CHD approach will be limited if lack of medicines makes it impossible to provide some curative services. Mobile Outreach Services: The second strategy for expanding access is through the FATA Health Directorate’s mobile outreach services. Still relatively new, these services are not regularly scheduled. Mobile outreach services are designed more for curative than for primary health care interventions. The project plans to work with the Directorate of Health to add child health services to mobile outreach, but this has not yet been done. Expansion of Services Through the Private Sector: The third strategy to expand access to child health services is through the private sector. The project aims to enhance the ability of private providers to provide quality case management of ARI and diarrhea but this activity has not taken place. IR2: IMPROVED QUALITY OF CHILD HEALTH SERVICES AT FACILITY AND COMMUNITY LEVELS To improve the knowledge and skills of health care providers, the project conducts three main trainings: (1) training in essential newborn care (ENC); (2) training in IMNCI; and (3) training in community case management for private providers. Training in ENC: Currently 21 ENC master trainers train health care providers at the agency resource centers. Both public and private providers—MOs, LHVs, female MTs (FMTs), and nurses—receive the four-day ENC training. To date, 354 providers have been trained, 38 percent of the training target. Training in IMNCI: The project uses the WHO curriculum for IMNCI, which is an 11-day clinical training. There are eight WHO trainers for each group of 24 trainees. The project strategy is to train MOs first, to be followed by LHVs, nurses, and FMTs. Training is conducted in three teaching hospitals in Peshawar, which without any compensation have significantly revised their work to accommodate these students. For example, one of the senior pediatricians provided his office so that students would have a place to work. The two senior pediatricians visited were very supportive of the training, though it is sometimes difficult to coordinate the large numbers of students. IMNCI training began only in 2008; to date, 72 MOs have been trained. Of these, 12 have also been trained as master trainers by attending a five-day course and conducting one training with the WHO trainers. The intent is that agency master trainers will be able to train other health care providers. Whether this will happen at the agency level will depend on whether caseloads are adequate to ensure quality training. The project plans to conduct one training a month for the rest of the year, training at least another 144 MOs. Training of Private Providers: The project plans to train private providers on community-based case management of ARI and diarrhea, but this has not yet occurred. Strengthening Health System Management: To support program implementation and facilitate strengthening of health system management, ICHP helped form an agency health management team (AHMT) for each agency in FATA. Members of the AHMTs are political agents, MID-TERM EVALUATION OF THE USAID/PAKISTAN FATA PROJECT 53 community leaders, agency surgeons, medical superintendents, and ICHP coordinators and other staff. AHMTs meet monthly to discuss and facilitate implementation of project activities. So far 39 AHMTs have been formed in FATA. The project also established a project management committee that meets quarterly and provides general guidance. The project has worked to improve the HMIS in FATA. Its quality is very poor, and the information produced does not flow usefully into management decision making. In general, therefore, the system has minimal functional value. ICHP is working with the FATA Health Department to complement routine collection of essential information; it also trains health managers in FATA on maintaining and using the district health information system (DHIS), which is more appropriate for FATA agencies. IR 3: INCREASE COMMUNITY KNOWLEDGE AND ACCEPTANCE OF SERVICES AND BEHAVIORS Key activities to increase knowledge and promote healthy behaviors for child health are (1) increasing community awareness through LHWs and CHWs managed through subgrants with local NGOs; (2) community sensitization events, such as formation of local committees; (3) formation of quality improvement teams (QIT); (4) advocacy at the agency, tehsil, village, and facility levels; and (5) Child Health Days. Community Mobilization Strategy: ICHP’s community mobilization strategy (CMS) was developed early in the project to identify areas for intervention for behavior change and to increase knowledge about harmful practices and motivate families to adopt healthy practices to promote child health. It is intended to help bridge the gap between service providers and communities. The CMS is evidence-based, using a quantitative baseline study and informal interviews with providers to gather information on knowledge and practices across agencies. The strategy does not yet include key messages; the project is now hiring a consultant to help structure a message matrix and materials for the project. The four intervention areas targeted by the strategy (described below) seem to an appropriate number of interventions for a project of this size. Target Groups: The CMS defines 11 target groups. However, one, mass media/print and mass media/electronic, is not so much a target group as a channel for messages. Similarly, health care providers will not be direct targets of CM activities, though they may be part of advocacy or QIT activities. The project might consider simplifying the remaining nine groups to five: (1) women/primary caretakers; (2) families (secondary caregivers, especially fathers); (3) children; 4) community influentials (community leaders, religious leaders, school teachers, LHWs, NGOs); and (5) advocates (local administration, political agents). To improve uptake the project is urged to keep the messages and language used consistent for the different groups. Increasing Community Awareness: The project aims to train 1,520 LHWs in the FATA and another 58 in the Frontier Regions to increase community awareness of child health. So far 292 LHWs have been trained, 18.5 percent of the target. LHWs conduct support groups for women (a successful intervention adopted from the PAIMAN project) and make home visits to reach women with messages about child health. LHWs also use counseling cards during these sessions. The project promotes the messages in a phased manner, which is a good approach given the large number of messages and target groups to be covered by the initiative. The project has contracted with two local NGOs, PAIMAN Alumni and BeFARE, to work in non-LHW areas across the seven agencies. The NGOs are responsible for conducting outreach using CHWs, whose work mimics that of LHWs; mobilizing communities to attend CHDs; and forming QIT teams (see below for discussion of these interventions). Hundreds of CHWs have already been trained and 450 community awareness activities have been conducted, along with 23 advocacy events, 12 QIT teams being set up, and 26 CHDs. Community Sensitization Events: To date 83 sensitization events have been organized in the FATA. Sensitization events are organized by male and female community mobilization officers. These events may take place at facilities and through teachers and community leaders but are most often conducted by men targeting men with needed information. Reaching out through village elders and Ulamas and through local structures is crucial to reaching men with vital information. Quality Improvement Teams: To date, 25 QITs have been formed. These committees are comprised of influential community leaders and health facility staff. Based on field interviews, the roles of the QIT are to (1) promote awareness of health issues; (2) enhance understanding of and trust in health care providers; (3) raise community issues with health care providers; (3) develop joint solutions with health care providers to meet the needs of the community; and (4) advocate for additional resources. While QITs are fairly new, they already have some significant achievements. One QIT was able to stop the transfer of an effective LHV out of their area. Another was able to convince a mine rescue team to move out of the health care facility to make space for health services. In Ali Masjid, where malaria is endemic, the QIT for the first time received insecticide to prevent malaria in its village. Other QITs were able to resolve local water and electricity problems by generating funds for equipment repair. One challenge for the QITs is that because the teams have no formal identity, higher officials often do not give them the importance they deserve. Nevertheless, the QIT teams seem to be an effective community-facility mechanism for finding local solutions to enhance access to and quality of health services. Advocacy Events: To date, 25 advocacy events have been held across the seven agencies. Advocacy events are organized for decision makers at the agency, tehsil, village, and facility levels. Participants are given an orientation on improving child health and on project interventions. High-level advocacy has helped the project to ensure the safety of its workers across the region, which is vital to its work. Child Health Days: LHWs and CHWs promote the importance of CHDs and insure community participation. Besides providing services, they also raise awareness and promote healthy behaviors (see IR1 above for more information on CHDs). The project would like to add the Child-to-Child approach to its repertoire of community mobilization interventions. The Child-to-Child approach targets school-age children with information about their own health using school teachers; the aim is for the children to take the messages home to their parents and families. Given the security problems in the FATA region and the difficulty of organizing events and group gatherings, it is a good idea to test this approach to see if it is a viable means of spreading health information. MID-TERM EVALUATION OF THE USAID/PAKISTAN FATA PROJECT 55 IV. CONCLUSIONS AND RECOMMENDATIONS FOR THE FUTURE GENERAL CONCLUSIONS Despite the worsening security in the region, ICHP has already made significant progress. It has established effective relationships with FATA officials, health care providers, and community leaders that have greatly facilitated implementation over the last six months. Thus, the project is well positioned to achieve its objectives. The project can make a substantial contribution to increasing access to and the availability of quality child health services and increasing knowledge, awareness, and healthy behaviors related to child health if the security situation allows them to implement the interventions outlined in this report. The project, applying its no-visibility policy, has been very careful for security reasons to work in the name of local agencies and agency health services. This approach is likely to improve the sustainability of its interventions, since service delivery is provided through existing facilities and providers rather than an independent NGO. There is considerable interest, support, and involvement from the FATA Secretariat, although less from the FATA Health Directorate. Child Health Days seem to be a successful strategy under the circumstances to encourage the community to access child health services. The challenges are to ensure adequate staffing and the availability of essential medicines. The approach of having CHDs on a rotational basis allows flexibility to move staff, vaccines, and medicines throughout the agencies to ensure that all elements are in place for CHDs. SHORT-TERM RECOMMENDATIONS FOR ICHP AND USAID For the remaining life of the project, through August 2009, we offer the following recommendations for USAID/Pakistan and ICHP: Since the project is just now gaining momentum, it should be extended for two years so that it can achieve the targets set out in the initial proposal. Considering that only 18 percent of the total project budget has been spent so far, a no-cost extension should be possible and suitable. Given the severe shortages of staff and medicines, the CHDs should continue to be provided on a rotational schedule. ICHP needs to carefully review its expansion plan for CHDs to ensure that if the number is increased, health care providers are not pulled to this activity at the expense of routine services. USAID should expedite the medication waiver to ensure a supply of essential child health medicines. Meanwhile, ICHP should continue its efforts to secure medicines through a variety of mechanisms. The project should also explore options to sell medicines at the CHDs. While consumers are not used to paying for medication from the public sector, they do pay private providers. If medicines are provided more conveniently and cheaply at the CHDs, in the long run communities may be able and willing to pay for some medications. To further reinforce relationships with teaching hospitals, ICHP should provide some support for the teaching facilities (e.g., multimedia). ICHP should be commended for securing WHO trainers and arranging to conduct training in teaching hospitals, which ensures that caseloads are adequate to ensure quality training. For better uptake of messages, the project should keep BCC messages simple and the language used consistent for different groups. LONG-TERM RECOMMENDATIONS FOR USAID FATA and the Frontier Regions are culturally, politically, and structurally different from the rest of Pakistan. The team recommends that USAID fund an integrated MNCH project with water and sanitation components that is specifically designed for FATA and the FRs. Depending on the amount of funding, the project could also cover other areas of primary health care. An integrated primary health care project specifically tailored to meet needs in FATA and the FRs would be more effective than expanding nationwide projects into the region. This approach will also capitalize on the expertise of organizations that work in the area, promote cross-sectoral synergies, and promote sharing of lessons learned. USAID needs to continue to advocate with FATA officials to enhance resources for MNCH. USAID and the project also need to engage traditional tribal leaders (Jirga elders) wherever the project is working for greater support and buy-in from the community. Because there are few health care providers that are from the FATA area, providers from other areas need to be recruited to provide services there. We believe it would be useful to train women from FATA and the FRs to become LHVs to increase the number of providers who reside and work in these areas. For more information, please visit http://www.ghtechproject.com/resources/ Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 www.ghtechproject.com