TAKAMOL MID-TERM EVALUATION July 2009 This publication was produced for review by the United States Agency for International Development. It was prepared by Joy Riggs-Perla, Nevine Hassanein, Paula Hollerbach, and Nahed Matta through the Global Health Technical Assistance Project. Front cover photo: 6th October MCH Center Photo Credit: Joy Riggs-Perla, 2009. TAKAMOL MID-TERM EVALUATION DISCLAIMER The views of the authors expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 09-001-165) is available in printed or online versions. Online documents can be located in the GH Tech web site library at www.ghtechproject.com/resources.aspx. Documents are also made available through the Development Experience Clearinghouse (www.dec.org). Additional information can be obtained from The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-00 TAKAMOL MID-TERM EVALUATION i CONTENTS Acronyms ......................................................................................................................................................iii Executive Summary ...................................................................................................................................... v I. Introduction and Background................................................................................................................1 II. Scope of Work and Methodology..........................................................................................................5 III. Summary Project Description ...............................................................................................................7 IV. Analysis of Progress ...........................................................................................................................11 V. Effectiveness of the Management and Administrative System...........................................................33 VI. Strategic and Managerial Approaches of Takamol and CHL .............................................................37 VII. Prospects for Sustainability.................................................................................................................39 VIII. Best Practices and Lessons Learned .................................................................................................41 IX. Summary of Key Recommendations ..................................................................................................43 Annexes Annex 1. Documents Reviewed..................................................................................................................47 Annex 2. Individuals and Organizations Interviewed ..................................................................................49 Annex 3. Scope of Work and Evaluation Framework .................................................................................57 Annex 4. Analysis of the Takamol Performance Monitoring Plan (PMP) ...................................................75 Annex 5. The Monitoring and Evaluation Matrix: Relevant Output and Outcome Indicators from Takamol’s PMP During FY09/Q1................................................................................................83 ii TAKAMOL MID-TERM EVALUATION TAKAMOL MID-TERM EVALUATION iii ACRONYMS AEA Adult Education Agency AMTSL ANC Active management of the third stage of labor Antenatal care BCC Behavior change communication CDA Community Development Association CEDPA Centre for Development & Population Activities CEOC Comprehensive essential obstetric care CHL CPR Communication for Healthy Living Contraceptive prevalence rate CQIS Continuous Quality Improvement System CSI Clinical Services Improvement CSR DHS EOC EONC Corporate social responsibility Demographic and Health Survey Essential obstetric care Essential obstetric and neonatal care EWSO Egyptian Women Speak Out FHM Family Health Model FLE Family life education FP Family planning FY Fiscal year HCI Health Care International HM/HC Healthy Mother/Healthy Child HSR Health sector reform IL Implementation letter IMCI Integrated management of childhood illness IPC/C Interpersonal communication and counseling ISOP Standards of Practice for Integrated Maternal and Child Health and Reproductive Health Services JSI LAM John Snow International Lactation amenorrhea method M&E Monitoring and evaluation MCH Maternal and child health MMSS Maternal Mortality Surveillance System MOHP Ministry of Health and Population MOU Memorandum of Understanding MWRA Married women of reproductive age NCU Neonatal care units NGO Nongovernmental organization iv TAKAMOL MID-TERM EVALUATION NICU Neonatal intensive care unit NPC National Population Council OB/GYN Obstetrics and gynecology OJT On-the-job training PAC Postabortion care PHC Primary health care PMP Performance Monitoring Plan RCT Regional Center for Training RH Reproductive health RPC Regional Population Council RR Raedat Rifiat STS&P Sector of Technical Support and Projects SIF Service Improvement Fund SMC Safe Motherhood Committee SIS TA State Information System Technical assistance Tahseen TOT Previous USAID-funded FP/RH project Training of trainers UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund USAID United States Agency for International Development TAKAMOL MID-TERM EVALUATION v EXECUTIVE SUMMARY USAID/Egypt commissioned the Global Health Technical Assistance (GH Tech) Project to conduct a mid-term external evaluation of the Integrated Reproductive Health Services (―Takamol‖) Project being implemented by Pathfinder International. The evaluation took place February 8–March 7, 2009. Project sites in four governorates in Upper and Lower Egypt were visited and a wide variety of stakeholders interviewed. Highlights of the evaluation were presented to Takamol project staff, USAID, and Ministry of Health and Population (MOHP) officials before the team left Egypt. The Takamol project fits well with the MOHP Health Sector Reform program, which includes a national effort to upgrade and accredit primary health care (PHC) units using the Family Health Model (FHM). Takamol targets 200 PHC units in Upper and Lower Egypt and 25 hospitals in Lower Egypt for a program of renovation, equipment, and management and for clinical quality improvements designed to improve outcomes in maternal and child health (MCH), family planning (FP), and reproductive health (RH). Another 23 hospitals in Upper Egypt are targeted for a more limited set of quality improvement activities, building on a previous USAID project (Healthy Mother/Healthy Child). Takamol has a community mobilization and behavior change communication (BCC) component in communities near the facilities to increase demand for services and promote healthy behaviors. The Takamol project design emphasizes capacity building and sustainability because no USAID work in MCH/FP/RH in Egypt is envisioned beyond 2011. The key findings of the evaluation include the following: The Takamol project is a complex but well-managed activity that is making excellent progress toward its objectives. It is using innovative strategies to work on expanding and sustaining project interventions even while implementation is underway. The project is operating in an institutionally fluid environment in the MOHP because of the evolving nature of the FHM, integration of services, accreditation, and other elements of the policy reform program. Due to unexpectedly high inflation, Takamol will not be able to complete its full contractual scope of work and deliverables unless additional funds are added to the contract. Takamol senior staff are not as proactive and strategic as they could be in using their field experience and global state-of-the-art technical knowledge to influence MOHP policies and programs. The long term sustainability of Takamol’s program of building capacity, quality improvement, and supervision will depend in part on how well the tools developed by the project are institutionalized within the training and supervision systems of the MOHP. Continuation and expansion of the community mobilization activities initiated by Takamol will depend on finding a reliable institutional framework to stimulate these activities. Whether the impressive mobilization of private resources for health achieved by the Corporate Social Responsibility (CSR) program is sustainable will depend on identification of an institutional base within the corporate sector. vi TAKAMOL MID-TERM EVALUATION The following major recommendations emerged from the mid-term evaluation: USAID/Egypt, if feasible, should add enough funds to the Takamol contract to ensure that the scope of work can be completed in all 11 governorates. If funds are insufficient to complete the total number of facilities originally targeted, the mandate should still be to work in all 11 governorates, though with fewer facilities in the final two because of the strong potential for replication of project activities within governorates. The MOHP needs to identify local-currency or its own budgetary resources to fill the gap between September 2009, when financing from USAID ends, and July 2010, when the next budget cycle begins. The MOHP, USAID, and Takamol should work to ensure that facility quality assessment tools and on-the-job management and clinical training programs are incorporated into the supervision systems of the FHM and the facility accreditation program. Technical revisions should be made to ensure that the ISOP1 is updated to make it more appropriate for the PHC level. The Takamol project needs to work more collaboratively with other USAID partners and donors to better provide technical policy and strategic support to the MOHP. For example, working on issues such as reducing the high rates of FP discontinuation and improving the quality of postpartum FP services is important. Regional Population Councils (RPCs) offer a possible institutional base for sustaining the community programs initiated by Takamol beyond the life of the project, especially where governors are proactive about the goals of the 2008 National Population Conference. Takamol’s discussions with the MOHP on sustainability should focus on how governorate-level MOHP staff can encourage this role for the RPCs. The central MOHP should share successful experiences on community programs during meetings with governors and other local officials. A catalyst for maintaining and building new partnerships between corporations and health sector programs needs to be identified within a private entity like the American Chamber of Commerce. The more detailed list of recommendations is found in Section IX below. 1 Standards of Practice for Integrated Maternal and Child Health and Reproductive Health Services. TAKAMOL MID-TERM EVALUATION 1 I. INTRODUCTION AND BACKGROUND BACKGROUND Over the past 30 years Egypt has achieved impressive progress in reducing mortality and fertility. Contraceptive prevalence rates for modern methods, for instance, increased from 22.8% in 1980 to 57.6% in 2008. Child mortality was reduced from 191 per 1,000 births to 4 between 1975 and 2008. There have been no polio cases in the country since 2004, and the maternal mortality ratio plunged from 174 per 100,000 live births in 19932 to 53 in 2008.3 Few other countries can boast of such dramatic accomplishments. The health care system in Egypt is currently undergoing substantial reforms as the country moves toward universal health insurance. The Ministry of Health and Population (MOHP) is transforming its primary health care facilities to the Family Health Model (FHM), which covers maternal and child health (MCH), family planning (FP), and reproductive health (RH) services; basic emergency and curative care for the family as a whole; and dental health. Formerly vertical programs such as FP, immunization, and maternal/newborn health are being incorporated into fully integrated systems that incorporate training and supervision, which are also integrated. The changes in the health care system come at a time when fertility reduction in Egypt is at a plateau, particularly in urban areas (see Figure 1). FIGURE 1. FERTILITY TREND BY URBAN-RURAL RESIDENCE, EGYPT 1998–2008 With child mortality having declined steadily over the past 30 years, 87% of under-5 mortality in Egypt today occurs in the first year of life, and 66% of infant mortality is now in the neonatal period (see Table 1).4 2 Egypt National Maternal Mortality Study, 1992-93, MOHP. 3 Maternal Mortality Surveillance System, MOHP. 4 Egypt Demographic and Health Survey, 2008. 2 TAKAMOL MID-TERM EVALUATION TABLE 1. EARLY CHILDHOOD MORTALITY (MID 2003-MID 2008) Neonatal Mortality Infant Mortality Child Mortality Under-5 Mortality 16.3 24.5 3.9 28.3 Because child mortality is already low, further progress will require a focused effort to reduce the causes of death among newborns and young infants, as well as to promote maternal health, among the poor in both Upper and Lower Egypt. There are still dramatic disparities in health indicators based on income disparities (see Table 2). TABLE 2. HEALTH INDICATORS BY INCOME 5 Indicators Poorest 20% Richest 20% Total fertility rate 3.6 2.6 Contraceptive prevalence 53% 63% Infant mortality rate 59 23 Under-5 mortality rate 75 25 % of circumcised girls age 0-17 33.1% 15.3% Hepatitis C rates are among the highest in the world (15% of the adult population) and avian influenza continues to present a challenge with 52 human cases confirmed as of January 2009 and 23 fatalities. Between 2005 and 2008, malnutrition, as reflected in the proportion of children stunted and wasted, seems to have been increasing in rural Upper and Lower Egypt, which is alarming. USAID POPULATION/HEALTH PROGRAM USAID has been Egypt’s largest donor in the health sector, having invested about $1 billion since 1975 through technical assistance (TA) contracts, cooperative agreements, grants, and direct financing through implementation letters (ILs) to the MOHP and other public sector organizations. Funds are also made available through cash transfer programs tied to health policy reforms. The current USAID health program consists of the Integrated Reproductive Health Services (Takamol) project, the Communications for Healthy Living (CHL) project, Health Systems 20/20, substantial assistance to the avian influenza control program, support for infection control and hepatitis C and HIV/AIDS prevention and control, and the $150 million cash transfer program for health policy reform (Objective 5) within the broader cash transfer program signed in September 2007. In recent years the total budget available for all USAID programs in Egypt has declined; it is currently projected to be $200 million in FY 2009. USAID has signaled its intention to discontinue direct assistance to the Government of Egypt (GOE) via ILs by September 2009, except for avian influenza program funding, which comes directly from USAID/Washington. The Health Office’s agreement with the GOE ends in September 2011. USAID’s collaboration with the GOE in health is thus expected to end within two years. The Takamol Project, which began in March 2006, is due to end in February 2011. Implementation Letters, which have 5 USAID/Egypt, 2009: USAID/Egypt Population & Health Overview; History and Results; USAID/Egypt Health Program; Avian Influenza; Infection Control; Viral Hepatitis; HIV/AIDS; MCH-FP-RH; Health Sector Reform Program. PowerPoint demonstrations. TAKAMOL MID-TERM EVALUATION 3 been a direct source of support to the MOHP for training, supervision, and local consultants related to Takamol-supported work, will not continue beyond September 2009. A mid-term evaluation at this juncture provides a useful opportunity to assess issues affecting Takamol and to make recommendations for adjustments to enhance its final two years of work. It is especially critical to analyze issues and discuss prospects for sustaining the work beyond the period of USAID assistance. 4 TAKAMOL MID-TERM EVALUATION TAKAMOL MID-TERM EVALUATION 5 II. SCOPE OF WORK AND METHODOLOGY PURPOSE OF EVALUATION USAID-Egypt commissioned an external mid-term evaluation of the Takamol project through the Global Health Technical Assistance (GH Tech) project. The team consisted of a team leader, a maternal and newborn health advisor from USAID’s Bureau for Global Health, a demographer/FP specialist, and a maternal health specialist from Egypt. The evaluation had three purposes: (1) review, analyze, and evaluate the effectiveness and progress of the project so far in increasing use of FP and MCH services; (2) make recommendations about any adjustments needed to improve implementation and enhance sustainability; and (3) conduct a limited cross-program analysis of Takamol and the CHL/Johns Hopkins University projects that are running concurrently and were being evaluated at the same time. ANALYTIC FRAMEWORK The evaluation team used the statement of work and deliverables from the Pathfinder contract and performance monitoring plan (PMP) as its basic analytic framework. Project outcomes are evaluated against four major results to be achieved by the end of the project, which correspond to the major components of the project (see Sections III and IV below). A matrix was organized consisting of evaluation questions from the Mid-Term Evaluation Scope of Work, the source documents, and the assessment tools to be used (see Annex 3 for the full matrix). EVALUATION METHODOLOGY The evaluation methodology consisted of analyzing data and information gleaned from a variety of documents and project reports; the project’s PMP system and surveys; and extensive interviews with a variety of MOHP officials and managers at all levels (central, governorate, and district), MOHP service providers at primary health care (PHC) units and hospitals, clients at facilities, people active in community development associations (CDAs), staff of Takamol and other USAID projects, and representatives of firms involved in corporate social responsibility (CSR) activities, donors, and other stakeholders. The team used different interview questions for each category of stakeholder to ensure that similar kinds of questions were asked at each of the sites visited. Where possible, the team also observed services being provided to clients so that the two clinicians (ob/gyn and pediatrician) on the team could assess the quality of those services. Quality was assessed based on clinical protocols and guidelines used by the project (such as the integrated Standards of Practice, ISOP) and through interviews with service providers and examination of records. To complete the third objective, the team compared CHL and Takamol strategies, met with the CHL staff, visited a CHL site in Qena Governorate, and met with the CHL evaluation team several times to compare findings. Sites visited included clinics, hospitals, and CDAs in Luxor, 6th October, Qena, Ismailia, and Sharkia Governorates. The team visited facilities not assisted by the project or renovated by the MOHP, as well as facilities and CDAs that were either currently receiving Takamol assistance and had graduated. The team visited a replication site where Takamol approaches were being used without Takamol assistance. The team also asked to visit a facility that had received Takamol assistance but was still not performing well. Also, an unannounced visit was made to a Takamol-assisted clinic without notification to Takamol staff. The team met with three governors and district and governorate health teams in each site, attended two Regional Population Council (RPC) meetings, and observed a variety of CDA activities. 6 TAKAMOL MID-TERM EVALUATION TAKAMOL MID-TERM EVALUATION 7 III. SUMMARY PROJECT DESCRIPTION BRIEF SUMMARY OF THE PROJECT This report does not contain a full description of the Takamol Project because that information is available elsewhere and this report is more concerned with analyzing the program than describing it. The Takamol Project was awarded as a five-year cost-plus-fixed-fee contract to Pathfinder International (March 1, 2006 to February 29, 2011) authorized at $47.1 million. Subcontractors are John Snow International, Johns Hopkins University (JHU), Meridian Group International, and the American Manufacturers Export Group and its local partner, Health Care International (HCI). The purpose of the project is to help Egypt achieve ―sustainable reduced fertility and improved health outcomes [for] mothers and newborns.‖ Takamol and other USAID-supported projects all contribute to USAID’s Strategic Objective 20, to achieve healthier, planned families. The project objectives also include a substantial emphasis on building MOHP capacity, particularly at the governorate, district, and facility levels. Takamol activities are organized to achieve four main results: Result 1: Increased use of quality integrated MCH/FP/RH services at the PHC level Result 2: Increased use of quality integrated MCH/FP/RH services at the hospital level Result 3: Positive behavior change in communities Result 4: Improved capacity of MOHP and other partners to sustain performance of integrated MCH/FP/RH services The project works on strengthening the supply of services for Results 1 and 2 and on improving demand for services for Result 3. Result 4 is aimed at reinforcing the planning, management, and oversight capabilities of the MOHP, especially at the governorate and district levels; the explicit objective is to ensure sustainable continuation of major project activities. Takamol staff work closely with MOHP staff at all levels for each of the four objectives, supporting and guiding their work rather than undertaking activities directly. To the extent possible, training and supervision activities are always conducted with MOHP staff. Takamol contributes to the MOHP program for the FHM, facility accreditation, and ultimately the national insurance system as part of the comprehensive health sector reform program. USAID has committed through Takamol to renovate facilities as needed, mobilize communities, purchase equipment, and establish managerial and quality improvement systems in 200 clinics and 25 hospitals to complement similar activities that are financed in part by the GOE budget and by budget support provided by the EU through its Sector for Technical Support and Projects (STS&P). Because Takamol-assisted facilities are part of the MOHP master plan for conversion to the FHM, the project collaborates closely with the ministry in all areas, such as renovation, equipping, client flow, and training, that are compatible with FHM standards. The project is designed to assist districts with a series of activities that begin with mutual agreements on districts and facilities within each governorate to be targeted; renovation of facilities following the specifications for the new FHM; providing any needed equipment, especially for MCH, FP, or RH services; didactic and on-the-job training; and mentoring staff in technical and managerial topics related to MCH/FP/RH services. Governorate and district staff are trained to be trainers and their supportive supervision skills are developed. The project has introduced tools to improve both training and supervision (see Section IV for details). 8 TAKAMOL MID-TERM EVALUATION Community-level demand creation and behavior change activities are conducted in part through CDAs in communities where facilities are located. CDAs are given small one-year grants to catalyze educational activities aimed at RH, avoiding early marriage, discouraging the practice of female genital cutting, properly spacing pregnancies, increasing the use of services offered by PHC units and hospitals, increasing community ownership of PHC units, and providing education about avian influenza, personal hygiene, and other health-related topics. CDA activities typically involve groups of youth and religious leaders, adult literacy activities, microenterprise groups, and agricultural extension workers and other activities. The capacity development component (Result 4) includes helping establish or activate clinic and hospital management boards, activate service improvement funds, and improve the capacity of district MOHP staff to plan, train, supervise, and coach facility staff. One innovative Takamol strategy is to undertake the package of activities in each district and then ―graduate‖ the district by ending Takamol inputs. The project also encourages the MOHP and local government officials to replicate Takamol interventions using the same techniques and tools but without Takamol support. Sustainability subcommittees have been established within governorate RPCs chaired by the secretary general of the governorate, with representation from relevant ministries, using the 2008 National Population Conference and the president’s directives to focus on population and FP goals as a platform for engaging local government. Finally, gender concerns and public-private partnerships are viewed as cross-cutting issues that deserve special emphasis. Corporate Social Responsibility efforts have produced relationships with large corporations in Egypt, which have contributed both cash and in-kind contributions in support of the project. Gender issues are addressed mainly in CDA activities to empower girls and women; involve men through agricultural extension groups and religious leaders; help women obtain the identification cards necessary for voting and local financial transactions; improve literacy, and so on. Another important contribution of Takamol has been to encourage more female participation on facility boards. Table 3 summarizes the total anticipated geographic coverage of Takamol by the end of the project in February 2011. As of December 2008 it was working in or had completed work in nine governorates, 148 PHC centers, 21 hospitals in Lower Egypt, and, with more limited technical input, 23 hospitals in Upper Egypt. The catchment population of the hospitals and clinics where the project is working or has worked is about 12.3 million if hospital catchment areas are included in the calculation (i.e., the population of the entire district served by the hospital). By 2011, if the project completes all current deliverables it will cover a total of about 15.5 million. As a subset of this larger figure, the total population of clinic catchment areas is about 2.7 million. As the total population of Egypt in 2008 was 81.7 million, Takamol is working with facilities that serve 15% of the population. TABLE 3. PROJECT SCOPE Upper Egypt Lower Egypt Governorates 6 5 Urban Slums of Cairo, Giza, and Alexandria Hospitals TA as needed in intervention districts 25 Clinics 150 PHC facilities 50 PHC facilities TAKAMOL MID-TERM EVALUATION 9 RESULTS FRAMEWORK AND PROJECT TARGETS The project results framework, which is closely related to the USAID SO 20 Strategic Framework, is shown in Figure 2. FIGURE 2. USAID STRATEGIC OBJECTIVE 20 FRAMEWORK The project has a sound PMP that collects a rich variety of data, such as service statistics, pre- and post￾community-based knowledge, quality, and client satisfaction surveys in selected sites, and data related to training and CDA activities. Data are collected for 23 indicators organized by result area. A smaller number of other special indicators are reported to USAID for the Operational Plan and for Project Implementation Reviews. A management tracking system allows project leadership and staff to determine percentage of completion on all activities listed in the annual work plans. Quarterly and annual progress reports are submitted to USAID. Annual work plans also set PMP targets for the planning period. Annex 4 contains a detailed analysis of the Takamol PMP, responding to the four monitoring and evaluation (M&E) questions raised in the evaluation scope of work. 10 TAKAMOL MID-TERM EVALUATION TAKAMOL MID-TERM EVALUATION 11 IV. ANALYSIS OF PROGRESS RESULT 1: INCREASED USE OF QUALITY INTEGRATED MCH/FP/RH SERVICES AT THE PHC LEVEL At the national level there have been notable improvements in several MCH indicators, according to the preliminary report of the Egypt 2008 DHS Survey. Between 2005 and 2008, regular antenatal care (ANC) visits increased from 28% to 65% and medically assisted deliveries from 46% to 79%. The proportion of fully immunized children rose from 89% to 92%. However, the total fertility rate among women aged 15- 49 declined only slightly, from 3.1 to 3.0, and the contraceptive prevalence rate for modern methods increased, from 56.5% in 2005 to 57.6% in 2008. With such an increase in MCH coverage rates, the quality of services offered at the MOHP facilities must be improved to cope with increased demand and provide higher-quality services to satisfy clients. The 2008 DHS reported that 60% of women received FP services from a public sector facility, primarily at the PHC level. Furthermore, 62% of the women interviewed do not want another child. These recent data highlight the importance of the services that PHC units deliver, and also the unmet need to provide high￾quality FP services and reduce missed opportunities for FP counseling and services during ANC and immunization visits. Progress FIGURE 3. PROGRESS TO DATE The Result 1 component of the Takamol integrated model contributes to increased use of quality, integrated MCH/FP/RH services at the PHC level. To date (see Figure 3) 124 PHC units of the planned 200 have been renovated and equipped and are operational. Having completed the full cycle of activities, including six months of on-the-job training (OJT), Takamol has phased out training in 74 facilities, continues its work in 54, and has begun work in another 24. About 33 facilities have been accredited as FHM clinics. The MOHP and Takamol selected facilities using criteria based on needs, caseload, population size, staffing, and other MOHP renovation plans. Result 1 activities are scheduled to be implemented in 11 governorates in Lower Egypt (50 units) and Upper Egypt (150 units), and in urban slums in Cairo, Giza, and Alexandria. Key components are facility renovation, provision of basic equipment, and upgrading the technical and managerial skills of clinic staff. The Takamol model emphasizes the link between the facility and the community through mobilization activities that have 12 TAKAMOL MID-TERM EVALUATION 6 th October MCH Center — assessment team members chatting with clinic director and District Health Supervisor. Joy Riggs-Perla Joy resulted in higher caseloads. Activities and achievements of planned targets for Result 1 are on track (see Annex 5). Observations on Interventions Renovations: The PHC units the team visited had a unified floor plan matching the basic blueprints and finishing standards for the FHM. All areas of the clinics are renovated, not just the MCH and FP rooms, and thus the MOHP does not have to make any additional infrastructure investment, although it does supplement equipment in Takamol￾supported facilities when needed. Takamol collaborates closely with the EU’s STS&P to ensure that Takamol-supported facilities conform to FHM standards. Takamol facilities are now included in the MOHP accreditation master plan. About 33 facilities have so far been accredited, and all the rest are aiming for accreditation. The team questioned whether it was cost-effective for Takamol to hire engineers to oversee architecture and engineering of the construction, which is subcontracted competitively, rather than subcontracting this task also and having an engineering expert confirm that this model is cost-effective and time-saving. Facility upgrades are very visible to the community, and higher client rates were observed in most of these facilities. Training of facility staff: Training activities are scheduled throughout the renovation of the PHC units and completed by the time the unit is reopened. This minimizes service provision downtime during renovation. Takamol training is conducted largely by Health Care International (HCI) for most of the didactic and the OJT for PHC staff because the number of courses and people to be trained are huge; other local partners are used to train on community activities. Takamol teams help select HCI trainers, several of whom were trainers for previous USAID projects. Clinic staff all reported positive experiences with training. Takamol collects a considerable amount of data through pre- and post- training surveys of participants and can therefore gauge the effectiveness of both clinical and management courses. HCI reports that between the pre- and post-test scores for trainees for all courses increased 54%–80%. PHC facility staff are offered the following courses: Integrated MCH/FP/RH courses for physicians and nurses (9 days each) are didactic training using the ISOP manual; the nurse’s training includes practical work at a PHC unit. These are the first integrated courses supported by the MCH and FP sectors. Takamol revised the ISOP manual, building on work done during the Tahseen and Healthy Mother/Healthy Child (HM/HC) projects. The ISOP manual as approved by MOHP serves as a resource document with technical guidelines on all national MOHP services and programs covering MCH and FP as well as female genital cutting, the lactation amenorrhea method (LAM), integrated management of childhood illness (IMCI), and neonatal screening for congenital hyperthyroidism. About 3,000 copies of the ISOP were distributed TAKAMOL MID-TERM EVALUATION 13 to newly graduated physicians in their pre-service training; another 2,000 copies are being given to the MOHP. On–the-job training (OJT) is conducted at their PHC units for six to eight months after the integrated course for physicians and nurses by MOHP governorate and district staff, coached by Takamol-HCI staff. Clinic staff consider training of lab technicians on basic PHC lab tests to be an important complement to the upgraded PHC services. Outreach workers (RR: Raedat Rifiat) hired by the MOHP and by CDAs are trained on MCH and FP/FH messages; the training covers interpersonal communication to assist with health education tasks and postpartum visits. Sonar training is conducted for physicians to improve ANC services, although having a sonogram during pregnancy will not help predict most of the common maternal complications, such as postpartum hemorrhage, infection, pre-eclampsia or obstructed labor; poor interpretation is also a serious concern. However, Egyptian women demand this subsidized service for reassurance, and it may increase client flow and satisfaction. The MOHP TSO Technical/Clinical Advisor, and some hospital physicians, reported that sonar is used to reassure women after insertion of intrauterine devices (IUDs), a step that might help reduce the discontinuation rate. The Regional Center for Training (RCT) conducts five-day training courses in Cairo. Because providing sonar services is a matter of national policy in Egypt, and because it is likely to reassure women with IUD insertions, the team believes that continued support for this training is justified. Management training for clinic staff is done in three phases for the entire clinic staff and district supervisors. It emphasizes teamwork and supportive supervision. Staff at all levels appreciated the training, and the data collected by the project on its effectiveness demonstrates its value. Knowledge improvement scores increased with training in the majority of facilities. The project also measures concrete outcomes, such as whether facilities are actually following through on the training by making improvements in how Service Improvement Funds (SIFs) are used and whether membership of facility boards becomes more diverse. Most of the clinics improved their use of the SIFs, showed improvements among board members in knowledge of facility budgets and diversity, and found improvements (as reported by service providers) in the work climate at facilities. Referral system training: The Takamol team has helped the MOHP activate its referral system and developed a simple referral form. It brings PHC units, hospital, and district teams together to vitalize the system. The system is being implemented well in several PHC facilities but feedback from the hospital level has been minimal. Hospital staff identified this weakness, and are working to improve it. When the FHM is implemented at the hospital level, the referral system will be activated formally. The work of Takamol offers a solid foundation for this important component of the model. Safety and maintenance training: Takamol designed a course with practical training for selected facility staff on the repair and maintenance of basic equipment. Those trained were very pleased; they used the skills acquired to fix electrical and plumbing problems and felt empowered. Clinic directors thought this training was very helpful to maintain new equipment and enhance clinic operations. Supervision tools: Takamol developed, with the MCH and FP sectors, integrated performance checklists addressing quality issues that are used for self-assessment and district supervision and OJT. District and governorate personnel use a parallel assessment tool, the Decree 75 checklist, to evaluate standards of care and implement the Decree 75 incentive system. This tool lacks performance quality standards. The team observed differences in the application of the Takamol integrated tool. Because the Decree 75 checklist incorporates only outputs, clinic staff use the Takamol tool to help assess performance as a way 14 TAKAMOL MID-TERM EVALUATION to achieve a higher score on the Decree 75 checklist. The evaluation team also observed that the FP sector still applies the 100 indicators (Gold Star) tool to ensure quality. Clinics being accredited are evaluated using an accreditation checklist more closely related to the Decree 75 monitoring tool. Reactivation of facility boards and use of SIF: Although there are ministerial decrees governing facility boards and the use of SIF, clinic staff are not well informed about them and are fearful of using SIF funds. Takamol helps facilities to activate these decrees. Governors provide approval for expanded community participation in facility board membership, recommending the inclusion of five community members on the board (for a total of 11) and encouraging women’s participation. Boards are elected by the community, and the clinic director chairs the board. The facilities visited had active boards that encouraged donations and provided support in addressing clinic challenges. Budget transparency is encouraged, and donated amounts are displayed on facility bulletin boards. Clinic directors are trained in the use of SIFs; district teams were included to facilitate approvals. At the facilities visited, all trained doctors succeeded in using SIF funding to procure supplies, and one clinic used the funds to support transportation costs for nurses to conduct postpartum home visits. The difficulties one new clinic director who had not received such training had in using SIF funds reinforced for the evaluation team the importance of district supervision and OJT. Postpartum home visits: According to MOHP protocols, all registered births should be visited by a nurse three times in the first week and twice more before the 40th day, when the mother is expected to visit the facility for FP services and child vaccination. An RR joins the nurse and may make some visits alone, but only for health education. Visiting during the first week is crucial to check the woman and newborn and exclude complications as well as provide health educational messages. National coverage for this service has been low, in large part because of transportation issues. At the facilities visited, coverage rates were reported to be as high as more than 90%. The Takamol household survey also shows that reports of at least two postpartum home visits ranged from 68% to 100% and in four of the six communities rates were 80% or higher. The evaluation team was assured that the rates are accurate; some clinics use SIF funds to pay for transportation, and others assign nurses to visit homes near their own. More important, the Decree 75 checklist ties facility incentives to achieving this indicator. Community mobilization: All community mobilization activities include health education messages about MCH/FP/RH, postabortion care (PAC), and female genital cutting. All messages encourage behavior change and seeking appropriate services from the target facility. All materials used for that purpose were created by the USAID CHL project. These activities are detailed under Result 3 of this report. Constraints and Challenges The didactic ISOP training provides a good overview of the integrated MCH/FP services but is not sufficient to achieve adequate clinical skills in key technical areas—which underscores the importance of follow-up, hands-on practice. The course also covers topics that are not very relevant to PHC services, such as management of infertility, information on newborn resuscitation with an oxygen source, endotracheal intubation (ISOP, pp. 366–67), ruptured uterus, and septic shock. The ISOP would be more useful if updated to cover evidence-based interventions relevant to MCH/FP and newborn care offered at the PHC unit. The OJT appears to focus on management topics rather than technical competencies. Even trained district staff are not always experienced enough to offer good clinical OJT, a fact confirmed by interviews with program managers at the central level. The evaluation team learned that some of the new graduates who received the integrated training lacked knowledge of key interventions and clinical procedures. This observation reflects the importance of high-quality OJT to practice skills such as IUD insertion or managing normal deliveries. TAKAMOL MID-TERM EVALUATION 15 A study commissioned by UNFPA in 2008 indicated that the technical depth of FP and MCH is diluted in the FHM. Though the MOHP is committed to move ahead with the model, MOHP officials told the evaluation team that plans are underway to reinforce its FP/MCH aspects; Takamol staff could make important contributions. Recommendations The ISOP manual needs to be updated with technical information and new MOHP policies. The recently completed new child immunization schedule might offer an opportunity to review all sections and revise as appropriate. If the MOHP prefers to keep all topics in the current version of ISOP to serve as a reference book to doctors, sections that cover basic interventions should be simplified and incorporate best practices. The document should address management of the majority of cases presenting at the PHC level. Also, all references to EOC (essential obstetric care) should be changed to read EONC (essential obstetric and neonatal care), which is internationally accepted terminology. Before the ISOP is revised, the nine-day integrated training for doctors should be streamlined and center on subjects relevant to the PHC physician; minor changes to the course content might be required. Also, since the training of doctors is didactic, practical training and lab skills should be emphasized as much as possible. Training on normal delivery should emphasize the use of the partograph and active management of the third stage of labor (AMTSL) to prevent postpartum hemorrhage, not as an option but as the best practice (ISOP, p. 286). Oxytocin is the drug of choice (WHO, 2007). Routine newborn care (drying/wrapping, cord care, early breastfeeding) should be included in all courses, especially those for RRs. Clarity is needed about the policy on normal deliveries at the PHC unit. Some clinic staff have experience in deliveries and need to be encouraged; others need to be trained. A renovated, well￾equipped facility is certainly better than home deliveries. This observation was discussed with MOHP officials; Takamol should provide technical leadership to address the issue and ensure that oxytocin is on the essential drug list for PHC units. It is very important that PHC doctors and nurse/midwives be highly skilled, especially those in charge of renovated facilities where client rates and expectations are higher. Because midwives are a vital part of integrated MCH/FP services, Takamol should make a special effort to include them in PHC activities. Fortunately, the MOHP recognizes the importance of training PHC staff on practical skills and has initiated a continuous medical education program that assigns PHC doctors to emergency and ob/gyn departments once a week for a few months to observe and practice. Takamol can help the MOHP to systematize this training. Takamol OJT should be tailored to address the special needs of newly graduated physicians and use the post-test data of the integrated course to address training needs of those doctors as they move to their PHC units. Because neonatal morality now constitutes 57% of under-5 mortality, addressing it is the main challenge. Takamol should share this information with its training partners to reinforce all activities covering newborn care, particularly postpartum home visits, training nurses and RRs to identify danger signs during those visits, and referring patients to the clinic. Community health messages should emphasize basic newborn care (drying and wrapping, cord care, immediate and early breastfeeding, and kangaroo mother care). Practical basic suction (only bag and mask) should be incorporated into ISOP training for both doctors and nurses. The 2008 DHS highlighted issues related to FP coverage and preference in seeking FP services. With discontinuation rates in Egypt very high, improving FP counseling should be a priority for both 16 TAKAMOL MID-TERM EVALUATION nurses and doctors. Upgrading the ability of PHC doctors to counsel patients, deal with rumors, and deliver quality services, such as IUD insertion, is crucial. Takamol should place special attention on counseling skills in training programs, using the data it collects to identify areas of weakness (see the Result 3 section below). If time and resources permit, consider preparing an easy-reference booklet or pocket guide for doctors on key integrated MCH/FP services, such as ANC, child health, FP, normal delivery, and care of newborns. Considering all the quality assessment tools currently in use at the clinic level, there is a need to work with MOHP to streamline supervision and monitoring. RESULT 2: INTEGRATED HOSPITAL MCH/FP/RH SERVICES Takamol intervention at selected hospitals has been designed to increase use of quality maternal and neonatal health care services, especially in PAC and postpartum care, including FP and breastfeeding support. The MOHP is now adopting the integrated service concept as health policy in the FHM. The project built on the achievements and best practices of previous USAID projects such as HM/HC, which provided a solid foundation for hospital-based maternal and neonatal services in Upper Egypt. To achieve its target for increased use of quality integrated MCH/FP/RH services in hospitals, Takamol works in three areas: expanding the HM/HC model to 25 hospitals in Lower Egypt, firming up implementation in Upper Egypt, and introducing FP/RH/PAC/postpartum services and updating HM/HC clinical protocols. Progress The Results 2 targets and achievement are captured in six PMP indicators that relate directly to the influence of project interventions at the hospital level. Progress on PMP indicators 13–16 and 22–23 (Annex 5) shows the project is doing well and progress toward the planned targets is satisfactory. Expand HMHC model to 25 hospitals in Lower Egypt: Takamol and MOPH senior officials agreed on five Lower Egypt governorates and selected districts in each based on pre-established criteria. Takamol then screened the hospitals (district and general) to ensure adequate physical space, appropriate staffing, and committed management; 25 hospitals were selected. To meet its targets for Lower Egypt, Takamol designed an intervention implementation model (see Figure 4). A community awareness component in the catchment area was added to the model to increase use of hospital maternal and neonatal/FP/RH services. TAKAMOL MID-TERM EVALUATION 17 FIGURE 4. IMPLEMENTATION MODEL Select intervention hospitals Assess needs Conduct physical renovation Provide equipment/ furniture Provide upgraded services Activate In-class training Board/HSMC OJT Referral Community Awareness Implement CQIS As of December 2008 upgraded services are being provided in 16 hospitals and five more are being equipped. The model will be implemented in the remaining four hospitals in Gharbia Governorate in FY09 if additional funds are received. Formulation of hospital boards of directors and Safe Motherhood Committees (SMCs), both elected and trained, in 21 hospitals and didactic training for all intervention departments staff (obstetric and neonatal departments, labs, OR, and anesthesia related to obstetric practices) has been completed. Currently OJT, the continuous quality improvement system (CQIS), and referral system improvements are in progress in 16 hospitals. Strengthen implementation in Upper Egypt and introduce FP/RH/PAC/postpartum services: In Upper Egypt Takamol intervention follows a different approach, since HM/HC covered all governorates with a standard package of comprehensive essential obstetric and neonatal care (CEONC) services. Takamol has built on previous achievements through refresher training and added components dealing with quality postabortion and postpartum care, FP, and breastfeeding support. Intervention hospitals, general and district, were selected to coincide with the districts selected for PHC clinic improvements under Result 1. In Upper Egypt the CQIS training was the cornerstone of project activities and SMC members were key actors. Takamol started with CQIS refresher training for hospital SMC members, introducing a package of quality assessment tools to enable monitoring of performance on different service aspects (managerial and clinical) for departments where the project is implementing its intervention. FP indicators were also added to the list of MCH indicators discussed during monthly meetings. Following the training, an in￾house cofacilitated quality improvement system exercise is conducted to observe practice and identify areas needing improvement. At the governorate-level SMC, Maternal Mortality Surveillance System (MMSS) outputs are analyzed and discussed, as well as the clinical weaknesses identified. The department heads help identify substandard care and common causes of maternal and neonatal mortality, all of which shape future didactic training and OJT (Fig. 5). 18 TAKAMOL MID-TERM EVALUATION FIGURE 5. COMPONENTS OF TRAINING Refresher training on CQIS / activate the system Hospital needs assessment MMSS Refresher clinical training (including postpartum care/postabortion care/FP/RH) Governorate SMC Technical assistance as needed All Takamol interventions have been completed in 19 hospitals: Refresher CQIS training for SMC members Cofacilitated CQIS exercises to identify needs and assess clinical performance Meetings with the governorate SMC to analyze MMSS findings and performance weaknesses Refresher clinical training for obstetric and neonatal and support services to integrate postpartum/PAC/FP/RH and Governorate SMC recommendations OJT provided as needed. Update the HMHC clinical protocols: The CEONC Protocols update was designed to integrate RH and MCH services, keep the protocols consistent with related MOHP policies, reflect the most recent clinical evidence-based medicine practices, and standardize clinical management among practitioners. The following protocols were selected for the update: EOC/FP/RH for physicians and for nurses Separate neonatal protocols for physicians and for nurses Obstetric-related anesthesia protocols for anesthetists EOC/neonatal lab services for technicians IC for OR nurses. A coordinating committee has been formed that includes the Takamol consultant (a university staff member) plus 1–2 university consultants. The committee organizes the distribution and collection of different chapters from writers and reviewers, reviews the content, ensures consistency, and coordinates with MOHP counterparts, editors, and formatters. Takamol is planning a consensus-building workshop for July 2009 to approve the updated version for all stakeholders, including MOHP counterparts, university professors, and representatives of the private sector, relevant professional associations, and the Medical Syndicate. Takamol reports that most of the protocol chapters for both physicians and nurses have been distributed to reviewers and are now being finalized. Supportive services chapters (EOC-related anesthesia for TAKAMOL MID-TERM EVALUATION 19 anesthetists, EOC/neonatal lab services for physicians and technicians and IC for OR nurses) have already been reviewed. Observations on Interventions Hospital selection, needs assessment, and renovations: Hospital selection and needs assessments are conducted with the full participation of governorate health officials. Involvement of the MOHP at each step of the selection process emphasizes MOHP authority, which the evaluation team noted during meetings with the Directorate Health Undersecretaries in Upper and Lower Egypt. The hospital assessment tool, a ―quality assessment checklist,‖ covers inputs as well as clinical and managerial performance. The checklist is comprehensive and provides a scoring system that allows for tracking improvements. The evaluation team visited three hospitals supported by Takamol in Upper and Lower Egypt. Takamol has completed renovation of the maternity departments and neonatal centers in Lower Egypt hospitals only. For those hospitals, it provides equipment, furniture, and supplies based on the needs assessment and using MOHP standards. The maternal and neonatal centers were well maintained, designed for easy patient flow, and had available basic equipment and essential medications. Takamol helped the ob/gyn departments to set up an FP room to provide counseling and service; the rooms are equipped with complete sets of FP educational materials developed by CHL. The FP service in one ob/gyn department was open 24 hours a day. In addition, a recovery room was equipped to monitor the most critical postoperative period before the patient is transferred to a postpartum ward for regular follow-up; the equipment was added after the needs assessment revealed that the ward did not have what was necessary. Training of hospital staff: In Lower Egypt didactic sessions are strengthened by OJT, a competency￾based approach. University professors and assistant professors conduct both types of training. All didactic training is conducted by HCI using the Takamol Integrated CEOC/RH/FP/PAC/postpartum training package. Takamol, in collaboration with HCI, created training materials and designed courses for each category of providers. The courses are: Orientation An integrated CEOC/FP/RH package on ultrasonography An integrated CEOC/FP/RH package for nurses, including postabortion/postpartum/FP counseling and breastfeeding support Basic and advanced courses for neonatologists Incubator and newborn in Neonatal Intensive Care Unit at Ismailia General Hospital Joy Riggs-Perla Joy 20 TAKAMOL MID-TERM EVALUATION Neonatal care training for NCU nurses Essential obstetric anesthesia training for anesthesiologists Essential laboratory services for lab physicians and technicians Training of operating room nurses The evaluation team found that training packages were available in the units. The staff appreciated the professional competency of the trainers and especially the emphasis on OJT to improve their skills. The evaluation team observed an OJT session at a neonatal center in Menia El-Kamh hospital. The trainer, a university professor, examined all admitted cases, reviewed patient files, and discussed his observations with the staff. However, ob/gyn department staff demonstrated far less competence than neonatal staff. In the neonatal unit, for instance, staff were proud of reducing respiratory distress case fatality rates from 9% (2006) to 2% (2008). Training in Upper Egypt hospitals follows a needs-based approach based on SMC recommendations. Only refresher courses were conducted, in the form of a new integrated training package ensuring the addition of quality FP/postabortion/postpartum and breastfeeding support to CEOC skills. Four refresher courses were designed based on needs identified, with HCI conducting only the didactic training. HM/HC standard protocols for CEOC and newborn were being used not only by trained but also by new staff who had not yet been trained. Given high staff turnover in the MOHP, it is important to establish in-house training and mentoring capacity. Takamol is training two local trainers for each hospital to sustain OJT for new staff. The evaluation team met two such trainers but their role was not clear; they did not appear to have been assigned responsibility, for instance, to provide OJT to the PHC physicians sent by the MOHP to acquire skill competency for IUD insertion and normal delivery. Continuous Quality Improvement System (CQIS): Takamol, in collaboration with the curative care department at the central MOHP, added an FP component to the HM/HC CQIS, which is used to monitor quality of care and performance, managerial and clinical. The checklists are used to help shape the hospital improvement plan. The team observed a quality control officer (a member of the hospital SMC) in one of the Lower Egypt hospitals using the Takamol CQIS checklist in preparation for hospital accreditation, but its use is not yet part of the formal MOHP system. If not institutionalized, it may disappear when Takamol ends. Hospital boards: Takamol formulates and activates hospital boards in Lower Egypt, while in Upper Egypt the existing hospital boards were modified according to the Decree 239 modification to include five community members instead of two. The evaluation team met with board members of three hospitals and learned about donations and support activities of community members. Donations exceeded LE 2.2 million in Ismailia hospital, in cash and in kind. Didactic training orients the board members on leadership strategic planning and prioritization and on how best to utilize SIF and MOHP resources. Hospital budgets are shared, and plans and constraints discussed, in monthly board meetings (the evaluation team reviewed the minutes of the last Ismailia hospital board meeting). During OJT Takamol Project renovated and equipped recovery room for obstetrical patients at Sharkia District Hospital Joy Riggs-Perla Joy TAKAMOL MID-TERM EVALUATION 21 management coaches work to improve board member skills in developing improvement plans and addressing intervention priorities. Hospital Safe Motherhood Committees: In Upper Egypt SMCs were already active; in Lower Egypt Takamol helped the boards form hospital SMCs whose primary role is to investigate any maternal or neonatal death and monitor all quality issues. Hospital SMCs are supposed to meet monthly; they also meet with governorate SMCs quarterly, but apparently they only meet when there is a maternal death. Committee members receive managerial training as well as OJT so that they can monitor progress using the CQIS checklists. Community mobilization: This activity is designed to improve timely health care–seeking behavior of families in the hospital catchment area by working with nearby PHC units. Through a successful collaboration with CHL, Takamol staff provide technical content for the educational materials and ensure community distribution; CHL is responsible for producing information, education, and communication materials. To check the quality of services and the community’s reaction to better communications, Takamol monitors client satisfaction using an exit interview conducted by the nurse or social worker at the postpartum ward before discharge. (See Annex 4). Referral system: The Takamol project works on several levels of the referral system: 1. First is internal, interdepartmental, referral between the FP clinic and the ob/gyn department to ensure better postpartum and postabortion FP counseling and services. 2. Takamol is also updating cross-referrals between different neonatal intensive care unit (NICU) levels, clarifying the roles and responsibilities of referring, emergency care, and receiving NCUs, as strengthened by feedback and reporting. The MOHP has accepted the system, which will be activated once the necessary equipment for emergency care is in place. 3. Referrals between PHC units and the hospital require a feedback loop, which is still very poor. Takamol is tracking referrals from PHC units to hospitals, and they have improved in project sites. The project is also encouraging hospitals to fill out the forms indicating diagnosis and follow-up for the clinics. However, the district must complete the documentation cycle and ensure feedback to the clinics for patient follow-up. Currently the cycle is incomplete and needs work: district staff are not doing a good job of collecting feedback forms from hospitals and redistributing them to referring PHC units. (Improving referral systems is difficult in all countries.) CONSTRAINTS AND CHALLENGES Integrated maternal and neonatal protocols (EONC) need to be published but Takamol has printing budget constraints; these could be solved through CSR fundraising. There is a need, as noted earlier, to improve central- and governorate-level oversight of the quality of hospital services. Institutionalizing the CQIS checklist will enable MOHP authorities to monitor the quality of hospital services and to use the checklist as a self-assessment tool for accreditation. Postpartum FP is an area where missed opportunities abound. Such counseling has been strengthened at the hospital level, which is a good step forward, but there are opportunities for providing other services in addition to counseling. Immediate postpartum IUD insertion is an example. Another example is that not enough patients or doctors are communicating about tubal ligations for medical reasons, though that is a well-accepted practice in Egypt. Physicians interviewed by the team are willing, with the husband’s consent, to perform the procedure but do not seem to be offering counseling to women who would benefit from it. 22 TAKAMOL MID-TERM EVALUATION Referral systems can be reinforced with more involvement from district health staff. The turnover of MOH physicians and annual recruitment of new doctors is jeopardizing Takamol’s phase-out plans. RECOMMENDATIONS Strengthen the quality and emphasize OJT, given its value in ensuring clinical competence. Standardize the titles for training materials to reduce confusion. Work with central MOHP officials as appropriate to encourage institutionalization of the CQIS, linking it explicitly to successful accreditation. This also applies to PHC facilities. Consider adding the client satisfaction tool to the SMC’s monitoring role. Work collaboratively on a strategy with the MOHP, CHL, UNFPA, the Population Council, and other partners to strengthen postpartum FP, particularly immediate IUD insertion or implants. Takamol can help by obtaining expert views, examining WHO recommendations and feasibility issues in Egypt, and helping the MOHP reach an informed decision. CHL is already working on this. Share information with the central curative care staff about Takamol’s effort to establish in-house training capacity at hospitals and work with the Technical TSO office to support MOHP efforts in this area. Consider additional strategies to engage districts in improving the referral system, such as improving pickup of referral forms from hospitals and following up with PHC units. Help the MOHP to address the high discontinuation rates for FP. This should be a priority; both Takamol and MOHP staff recognize the importance of having a clear strategy. RESULT 3: COMMUNITY MOBILIZATION AND BEHAVIOR CHANGE COMMUNICATION Result 3 has three overarching goals: (1) mobilize communities to adopt healthier behaviors; (2) mobilize key community members and local businesspeople to support the PHC through donations and provision of needed supplies; and (3) mobilize health personnel to adopt interpersonal communication/counseling (IPC/C) skills. To accomplish these, Takamol staff members interview, screen, nominate, and orient one local CDA in each catchment area. The Ministry of Social Solidarity and the MOHP must approve the CDA nominated by Takamol. Prospective partners review the activity options, submit proposals, and sign grant agreements (not to exceed $10,000) to conduct community activities. Takamol builds on existing programs and encourages replication of community mobilization activities without Takamol funding. Project interventions are scheduled to ensure informed and enthusiastic support and substantial contributions from local communities as early in the process as possible. Local CDA staff raise community awareness of health issues, respond to the health needs of youth and adults, create demand for and use of MCH/FP/RH services, promote healthier behaviors, and support the work of the facility. Not all types of activities are necessary or relevant in all Youth drama group at Community Development Association (CDA) in Boayrat Village, Luxor Governorate Joy Riggs-Perla Joy TAKAMOL MID-TERM EVALUATION 23 intervention areas; those undertaken are decided upon largely through a survey and community discussion. Activities include programs in which youth receive health education and undertake community service, such as planting and painting (Shabab Takamol weeks, in which there have been 7,611 participants); participate in peer-to-peer activities; and receive Family Life Education in physical health, reproductive health, and gender. Activities directed toward adults include women’s empowerment (Egyptian Women Speak Out, in which 7,867 women have participated) and the leadership program; training of religious leaders; literacy education; programs to bring health messages to agricultural extension workers; and couple communication seminars. Informal family edutainment activities, such as puppet shows, skits, and plays, are also presented. Finally, there is a program to create a cadre of media health advocates. As of December 2008 Takamol had provided small grants to 131 local CDAs, for a total of LE 6,513,950. CDAs are encouraged to visit other villages and share their experiences, raise money, and implement activities with the donations beyond what the grants are funding. Total donations leveraged by CDAs now amount to LE 169,700. CDAs also conduct monitoring and evaluation (M&E) and submit monthly, quarterly, and final activity reports. They are asked to continue reporting on the replication of their activities and fundraising efforts for a year after phase-out. Takamol field staff members visit the CDAs regularly to monitor the quality of their activities during that period, which usually ends at about the same time as the PHC units graduate. In sites where CDA activities have been replicated without assistance from Takamol, the types of activities are reported to be similar, building on literacy, agricultural extension, and other programs already established. Takamol will compare the sustainability plans submitted by the CDAs before phase-out with the quarterly data CDAs provide for the year after to ensure the accuracy of data on replication of activities without Takamol support that can be shared with others. Community outreach workers and service providers, who are the primary channels for conveying health messages, receive IPC/C training. So far Takamol has conducted joint training for 1,889 MOHP and CDA Raedat Rifiat and RR district supervisors in order to create strong cooperative bonds. Home visits are the principal means through which RRs and nurses link and refer families to renovated PHC clinics. As of December 2008, 695,448 home visits had been made and 230,431 patients referred to PHCs. In addition, 1,863 health awareness seminars had been given. Progress Takamol is making steady progress in achieving its end-of-project targets for Result 3 (see Annex 5), as evidenced by the indicators related to CDA capacity building, dissemination of health knowledge, and increased service utilization at PHC clinics (see indicators 5a, 6a, 10, 11, 18a, and 18b). Various output and outcome indicators show that the average increase in knowledge among RRs is 50%, and 95 PHC units out of 98 units have reported an increase of more than 5% in total caseload (indicator 6a). Community participants’ knowledge of key MCH/FP/RH messages, measured before and after trainings and plays, increased by 71% for trainings and 192% for plays (indicator 18a). Household surveys showed improvement of 28%–126% in community members’ knowledge of key MCH/FP/RH messages (indicator 18b). Analysis of the indicator data collected on follow-up samples of PHC units in Batches 1 and 2 show that caseloads of PHC units continued to increase after Takamol phase-out. Of the 11 PHC units, 10 reported at least a 5% increase in total caseload, and most showed a more significant increase. Observations on Interventions CDAs are chosen very carefully. Forms have been developed to assess the institutional and financial capacity of nongovernmental organizations (NGOs). CDAs are selected on the basis of their capacity to mobilize communities. 24 TAKAMOL MID-TERM EVALUATION The accomplishments have been impressive. In three CDA meetings in Luxor and Sharkia, the evaluation team heard numerous testimonials to the success of these activities in improving literacy and the acquisition of income-earning skills, such as making liquid soap or jam; enhancing abilities to communicate and negotiate; acquisition of leadership and decision-making skills; enhanced self-respect and self-confidence; and increasing knowledge of a variety of health messages pertaining to such issues as nutrition, healthy timing and spacing of pregnancy, smoking, and female genital cutting. Many participants spoke of friends and family members with whom they had shared these messages, thereby diffusing what they had learned. There was a particularly strong commitment to self-sufficiency and replication in Luxor. Mutually beneficial cooperation between religious leaders and physicians in the delivery of health messages was often cited as a harmonious and convincing way in which to address these issues. Although data were not available with which to measure how effectively each activity generated greater use of MCH/FP services and greater contraceptive use in the catchment areas, research on Asia and the Middle East has repeatedly found certain interventions to be reliable predictors of FP/MCH use: provider communication of health messages, especially healthy timing and spacing messages; client-based FP counseling; acquisition of literacy skills; discussion of FP/MCH within mothers’ groups; and married couple communication. All these interventions are integrated into the Takamol model of community mobilization. The model also promotes community support for older age at marriage and reinforces the importance of antenatal and postnatal care, good nutrition, and personal hygiene, as well as the ill effects of smoking and female genital cutting. Outreach to gain male support of MCH/FP/RH is part of the program. On the basis of data collected in six communities, more married women of reproductive age (MWRA) participated in CDA activities after the intervention, resulting in an impressive improvement in knowledge of MCH/FP/RH messages and referrals to PHC services. (See Table 4.) TABLE 4. HEALTH MESSAGES AND REFERRALS Community MWRA participating in CDA activities who received MCH/FP/RH messages % of MWRA participating in CDA activities who were referred to the local PHC unit Pre-Takamol Post-Takamol Pre-Takamol Post-Takamol Mandoura 25% 74% 25% 77% Debig 20% 95% 20% 86% Boayrat 50% 95% 38% 76% Odaysat 60% 94% 0 83% Mansheyet Al-Abbasa 50% 88% 25% 31% Heish 25% 90% 0 42% Source: Household Surveys Report, 2009, Table 4, p. 22. Annex 4 contains specific suggestions for tabulating household survey data that should provide insight into the connection between participation in CDA activities and other outcome indicators, such as new attitudes and behavior both before and after the Takamol intervention. Upon request Takamol also provided tabulations of visits to postpartum women and receipt of postpartum and postabortion counseling. In all the communities surveyed (see Table 5), the proportion of postpartum women reporting that they had received two or more visits increased enormously. Similarly, the TAKAMOL MID-TERM EVALUATION 25 proportion who reported that they had received postpartum counseling increased in nearly all the communities. Postabortion counseling also increased in all communities among women who had had abortions or miscarriages in the year before the survey. In both Mandura and Boairat, there was an especially large increase in the proportions of postpartum women who had received two or more visits and who had received postpartum FP (from 0 to 100% in both communities). These two communities also experienced the greatest increase in modern contraceptive method prevalence rates (see indicator 1). TABLE 5. POSTPARTUM VISITS AND RECEIPT OF POSTPARTUM AND POSTABORTION COUNSELING Community Women Who Received Two Or More Postpartum Visits Women Who Received Postpartum Counseling Women Who Received Postabortion Counseling Pre￾Takamol Post￾Takamol Pre￾Takamol Post￾Takamol Pre￾Takamol Post￾Takamol Mandura 0 89% 0% 100% 20% 50% Debeg 4% 68% 100% 100% 0 80% Odaysat 0 100% 0 100% 17% 83% Boairat 0 83% 0 100% 9% 100% Heish 48% 78% 100% 100% 0 100% Monshaat El Abassa 45% 89% 100% 80% 0 44% Source: Takamol Household Surveys, requested tabulations, March 2009. There were no opportunities for the evaluation team to observe counseling sessions. However, many aspects of client-centered counseling were apparent. Takamol ensures that clients feel comfortable and information will be kept confidential by maintaining visual and auditory privacy during counseling and FP procedures, which are conducted in separate rooms with screens to shield examination tables from view. Counseling flipcharts and trays displaying the methods available were seen in all the PHC units and hospital FP counseling rooms. Memory aids, such as posters, flipcharts, illustrated booklets, and sample contraceptives, were available to help clients remember essential information and remind providers to discuss important points. Take-home booklets to disseminate accurate information to others were available, since clients often share the materials with partners, relatives, and friends. Improvement in provider counseling skills is an important part of Result 3 activities, given the need to improve the quality of FP/RH services and reduce the high FP discontinuation rates in Egypt. The project conducted a pre-intervention assessment of training needs to help formulate the strategy for building the counseling skills of doctors, nurses, and RRs. A one-day IPC/C training course was created and presented, with reinforcement provided through supervision visits. Takamol measures client satisfaction through exit interviews, which are gathered monthly; 50 clients from each unit are asked to rate their satisfaction with clinic cleanliness, waiting time, privacy, and interaction with service providers. Indicator 8 shows that 84 out of 99 PHC units have average client satisfaction scores of 80% or more (see Annex 5). However, follow-up data collected on 11 PHC units from Batches 1 and 2 show that only 4 achieved the same client satisfaction scores after phase-out. The primary sources of dissatisfaction reported by clients were long waiting times and limited interaction with service providers. What is striking is that the patient caseload in most of these PHC units also increased 26 TAKAMOL MID-TERM EVALUATION dramatically, which is undoubtedly responsible for much of the decline in client satisfaction (see indicator 6a). Constraints and Challenges While there are no apparent constraints on establishing CDA activities, there are problems related to their sustainability. In spite of training in bookkeeping, financial management, proposal writing, and networking, not all the CDAs are equally able to identify potential donors. Because the source of long-term institutional oversight and support of CDA activities is also not clear, the degree to which this component will continue after Takamol ends is also not certain. Regional Population Councils offer the best hope for sustainability, but they require committed governors and representatives from the ministries of Social Solidarity, Education, Agriculture, and Youth and Sports, who serve on RPC Sustainability Committees. Takamol staff members understand the need to share the lessons learned through their efforts and inform governors and RPCs in other governorates to ensure that they appreciate the benefits to be gained from the experience of the CDAs and are encouraged to replicate the model. Some RPCs have been exemplary, such as the one in Luxor. However, beyond the life of the Takamol Project, there must be a mechanism for catalyzing the RPCs to continue and expand this work. Cultural differences between Upper and Lower Egypt, as well as a difference in the history of donor funding, have made it more challenging for Takamol to work with CDAs in Lower Egypt. Upper Egypt is characterized by more tribalism, extended families, and more willingness to volunteer for community service. In Lower Egypt there have been fewer donor-supported community health projects and therefore less experience with collective activities to improve health. It is harder to organize public meetings or plays, and there is less enthusiasm for community activities. Certain programs appear more sustainable than others, primarily because they are already in place and supported by an institution. For instance, the literacy activities tap into the programs of the Adult Education Agency. AEA trains 40,000 people every year; Takamol takes advantage of those classes by helping AEA to use a health-related curriculum to teach literacy. The proportion of participants who are considered successfully literate among the enrollees is reportedly higher in Takamol classes (68%) than in non-Takamol classes (45.5%). AEA is willing to provide seed money to CDAs wishing to participate. Another clear challenge is the need to improve the IPC/C skills of health providers. Takamol has made excellent progress, but skill building needs to be intensified through more specific IPC/C￾oriented observations and on-the-job skills enhancement. Fortunately, because Takamol has collected data on the effectiveness of its training strategies, it knows which areas of IPC/C need more attention. Also, Takamol could do more to work more directly with midwives, as mentioned under Result 1. Women in Adult Literary Agency sponsored literacy class at El Nawafel CDA in Luxor Governorate Joy Riggs-Perla TAKAMOL MID-TERM EVALUATION 27 Recommendations The project should analyze the IPC/C skills that showed the least improvement between pre- and post-intervention surveys and identify solutions, such as more intensive training and on-the-job mentoring of health staff. Because the MOHP is interested in learning more about the long-term sustainability of CDA community mobilization activities and programs, Takamol should help the ministry learn from its experience about any factors that might raise the potential for replicating or sustaining community mobilization programs, specifically barriers and facilitators of sustainability. Takamol should draft very brief step-by-step guides in Arabic for various community mobilization activities to help governors and RPC members replicate them. Success stories should appear in the final section of each brief to illustrate the extent to which each activity has succeeded in improving knowledge and changing health behavior. Takamol staff members have produced an activity description manual to assist communities themselves, and the manuals are often shared between CDAs. Because most of the material already exists, the cost to the project should be low. Activity briefs should be provided to governors, RPCs, and the CSR Committee of the American Chamber of Commerce. The activity descriptions should highlight ways in which cultural differences have been addressed in activities selected by villages in Upper and Lower Egypt. Besides the tabulations already produced on the percentage of MWRA participating in CDA-run activities who received MCH/FP/RH messages or are referred to services, Takamol M&E staff should also analyze data from the pre- and post-intervention household surveys to assess the impact of participation in any CDA activities on attitudes and behavior, including contraceptive use and wives’ perceived support from husbands (see Annex 4 for specific suggestions). The data can be shared with the RPCs to document the impact of community mobilization not only on communication of messages but also on the promotion of positive health behaviors. One of the Takamol youth activities is a peer-to-peer activity for university students. Since these students do not necessarily reside in governorates where Takamol is working, this activity is an adjunct to the community mobilization activities and is not sustainable. The evaluation team agrees with Takamol’s decision not to continue this program. Working through its CSR committee, the American Chamber of Commerce conducted a meeting in March 2009 with its members, Egyptian NGOs working in MCH/FP/RH, and well-managed active CDAs seeking funding for future activities. If the meeting was successful, Takamol should encourage other such meetings before the end of the project. RESULT 4: MOHP CAPACITY TO SUSTAIN PROGRAM PERFORMANCE The unifying theme for activities under Result 4 is sustainability. As background it is important to understand that Takamol contributes to the MOHP’s program for introducing the FHM as part of health sector reform. Takamol’s support for training, systems development, and equipment focuses on MCH, FP, and RH within the FHM. For instance, Takamol equipment represents 75% of what a PHC unit requires; the rest is provided by the MOHP. Takamol support, especially its attention to quality improvement, is helping the MOHP facilities to move toward accreditation—so far 33 of the upgraded facilities have been accredited. Takamol’s work generally has a much larger component of community mobilization than is the case with facilities supported exclusively by the MOHP. The strategy for promoting sustainability is to increase MOHP capacity; increase stakeholder commitment, support, and replication of activities; link health with non-health partners; and involve private corporations through CSR. Strengthening MOHP capacity at the governorate, district, and facility level involves both PHC clinics and hospitals in activities like forming or reactivating facility boards, 28 TAKAMOL MID-TERM EVALUATION securing funds from SIFs, building district capacity for supervising and mentoring clinic staff, integrating district planning, developing clinical trainers at the hospital level, and reinforcing SMCs. Work to ensure that Takamol efforts complement and are institutionalized by the MOHP includes collaboration at the national level as well as within each governorate. Increasing stakeholder commitment and replication includes working with RPC sustainability subcommittees made up of officials from the ministries of Health, Finance, Education, Youth and Sports, and Social Solidarity. The committees meet once a month, chaired by the Secretary General of the Governor’s Office, to discuss how to sustain and expand facility and community activities initiated through Takamol. CSR, which is also considered a crosscutting theme, is aimed at helping corporations and other nongovernmental entities find mechanisms for contributing to health activities in ways that meet the needs of the private sector for accountability and management ease. Progress The project seems to moving toward in achieving targets under PMP Result 4. In-kind and cash contributions from corporations, NGOs, and individuals for project activities and facilities have reached US$3.37 million, 10% of total operational expenditures, against a life-of-project target of 11%. Verification of the assumptions used to calculate the in-kind contributions showed in most cases that the assumptions underestimated actual time and effort observed in the field (e.g., numbers of hours of time volunteered by community groups, etc). Targets such as ―percentage of MOHP staff at all levels who received capacity building training‖ are close to being met. Progress is also evident in the numbers of PHC units and hospitals with management boards that are meeting regularly and working in support of project objectives. Similarly the end-of-project target of 120 facilities that utilize at least 10% of the SIF revenues to pay for facility improvements is also slightly ahead of schedule, so that target is also likely to be achieved (see Annex 4). Through its sophisticated information system Takamol has been able to track other indicators, such as knowledge improvements from pre- and post- testing of facility staff in clinic management, management board formulation and gender balance, SIF revenue and expenditures, work climate improvement, and other factors. Most important, project staff actively review and use the data generated to identify underperforming facilities and set work priorities. Observation on Interventions Replication strategy: One of the most interesting and unusual elements of the Takamol work on sustainability is inherent in the design and phasing of interventions and the continuous push toward replication within districts and governorates even as Takamol is moving into new geographic areas. Takamol moves into batches of facilities, simultaneously doing renovations and training staff. The operational phase that follows involves OJT, coaching, community mobilization and behavior change work, and establishing the SIFs and management boards needed to facilitate local planning and management. After a full cycle of intensive activity with a facility and surrounding community, the project ―graduates‖ the facility and then monitors how well it is maintaining the program. District staff are from the outset partners in the effort to support facilities, with the understanding that after a facility graduates, the districts will provide the same support. Data are collected from 15% of graduated facilities to assess how well they continue to perform after project phase-out (see Annex 5). Informal oversight is also provided through periodic visits from Takamol staff; problems are addressed through the district offices. Concomitant with the graduation process, the Takamol staff are working with MOHP officials and RPCs in each governorate to encourage replication of activities beyond the facilities and districts the project targeted. While it is still too early to judge the success of this effort, the team observed signs that the strategy is making progress. The project reports replication and scale-up activities in five governorates. In TAKAMOL MID-TERM EVALUATION 29 Luxor, which has an extraordinarily proactive governor and RPC, the governor is pushing for full accreditation of all facilities by June 2009. He also believes that the governorate and district teams can take over the systems-strengthening work of Takamol and use the same techniques to improve quality in all the facilities. CDAs that had initial funding from Takamol are attempting to help neighboring CDAs initiate similar programs. In Luxor Takamol is working in a high proportion of the facilities and benefiting from the unusually strong support of the governor. Ismailia is another governorate where the RPC seems to have solid support from the governor, which facilitates plans for expansion. As the Takamol Project progresses into its final two years, staff need to be increasingly focused on working with governors and RPCs on replication. District capacity development: Takamol staff have devoted considerable time and energy to working with district health teams to enhance their capacity for supervision and OJT. The evaluation team met with district teams in all sites visited. Some are young, enthusiastic, and eager to learn. Others have worked in vertical programs for 20 years and are clearly having difficulty making the transition to a more supportive supervision style and working as a team to cover FP and MCH topics. Takamol strives to ensure that district supervisors master the skills needed to conduct OJT for PHC staff. The project has also attempted to improve the technical capacity of district teams by including them in the integrated clinical training of the PHC staff as both attendees and trainers. However, the results of this investment were not always evident. Reinforcing supervision is another critical component of district health office capacity development activities (the supervision tools were described earlier under Result 1). Fortunately, monetary incentives for completing supervision using the Decree 75 checklist are ensuring that supervisory visits are done, reportedly even more often than required. The MOHP funds this system from its own budget. While the Decree 75 checklist is an excellent start for measuring clinic performance, it does not provide as much emphasis or detail on the actual performance of clinical or management practice as the Takamol quality checklist (which was also developed with the MOHP). Many clinics and district supervisors appear to be using both; some reported that using the Takamol checklist helps facilities improve their scores on the Decree 75 checklist, but the Takamol checklist has yet to become a formal part of the MOHP supervisory system. Its use may not endure beyond the life of Takamol unless it is incorporated into the more official and therefore permanent system. Facility management improvements: At both hospitals and clinics, staff seem to be highly enthusiastic about revitalization or reconstitution of facility management boards and better use of the SIFs. Clinics were also benefiting from the basic repair and maintenance training. Including more community members on hospital and clinic boards is apparently paying dividends in greater transparency of facility budgets, more community ownership, and in the case of hospitals donations (e.g., equipment). Minutes of board meetings are taken, and members interviewed by the evaluation team readily recalled the last meeting and could generally remember the topics discussed. Training on better use of SIFs was also appreciated because, in addition to staff incentives, they are able to use funds to make small repairs and buy some drugs and supplies. One clinic was using SIFs to fund travel for nurses to make postpartum visits to households beyond walking distance from the clinic. District health staff conducted supervision visit at Al Saayda clinic in Luxor Governorate Joy Riggs-Perla 30 TAKAMOL MID-TERM EVALUATION Safe Motherhood Committees: The project contract describes Takamol’s role as different in Upper and in Lower Egypt. In Upper Egypt it works with SMCs to broaden their function to include integrated MCH/FP/RH, maintain links between the governorate SMC and the MMSS, and encourage dissemination of surveillance results throughout the governorate. In Lower Egypt, it works to (1) improve the capacity of MOHP officials at the governorate, district, and facility level to monitor maternal deaths and report to the MMSS (as is done in Upper Egypt); (2) build the capacity of the SMCs to analyze causes of maternal deaths and recommend corrective measures; and (3) broaden their function to monitor the integrated package of essential MCH/FP/RH. In Upper Egypt the MMSS seems to be functioning well at the governorate level in Luxor; staff trained under HM/HC are continuing the maternal death audits and tracking each death. SMCs meet regularly; the team saw mortality data and minutes of SMC meetings. Takamol participates in the governorate SMC meetings. SMCs seem to be active at the three hospitals visited. Takamol has supported hospital SMCs in Upper Egypt and revitalized them in Lower Egypt—it is unlikely that they were previously active. The PMP indicator for SMCs counts numbers of SMCs meeting only if the right members attend and an agenda discussing key issues is produced. Corporate Social Responsibility: The project has done a very good job of generating interest and funding from corporations. Its success is attributable in part to the fact that Takamol CSR staff are viewed as coming from the corporate culture and understanding the requirements of private business. Currently Takamol is urging the MOHP to establish a CSR committee or working group to ensure that the initiative will be sustainable beyond the life of the project. The evaluation team believes that even if a committee is constituted, it will be difficult for them to play the same role or have the same credibility with the corporate sector. Two important ingredients emerge from the team’s discussions with stakeholders from the corporate sector: (1) A catalyst and matchmakers are needed to identify a good fit between health sector programs and commercial firm areas of interest, and (2) a mechanism is needed for corporations to provide resources without much management oversight by their own staff, which requires financial and outcome accountability so that corporations feel they have made a wise CSR investment. Issues and Challenges District capacity: Some district staff lack mentoring and training skills. While building district capacity is important for sustainability, because district-level health systems have been weak for many years, Takamol has problems finding the desired caliber of district officers and dealing with staff turnover. It thus has to be selective and build training-of-trainers (TOT) capacity with those district and governorate staff who show more capacity and willingness to learn. The younger female physicians the evaluation team met in Luxor Governorate are good examples of Takamol investments that are likely to have the highest long-term benefits. The team did see evidence that integrated district planning has produced annual work plans, often posted on a wall, that show MCH/FP and other program goals. Lower Egypt SMCs: It is not clear how much Takamol has invested to strengthen the Lower Egypt MMSS and activate governorate SMCs. In both Ismailia and Sharkia they may not be meeting regularly, and Takamol input was not mentioned. Within Takamol both R4 and R2 teams manage SMC activities. The training is divided between HCI and United Group for different levels in the same governorate, which may dilute the effort. Recommendations Work with governorate and district health directors to target the district staff that have the most potential to become successful mentors and coaches. If they are not clinically competent, these staff can be used to supplement the team to ensure OJT at the clinic level. Because most OJT is managerial rather than clinical, these coached do not necessarily have to have long years of clinical experience. The MOHP program of sending new physicians for additional clinic experience to nearby TAKAMOL MID-TERM EVALUATION 31 hospitals once a month is a good way to address this problem. Takamol can help improve its management. Work with the MOHP at the central level to strengthen the Decree 75 checklists using the experience of the Takamol quality assessment tool to emphasize the actual performance of health workers. Ideally, at the end of the project there will no longer be a separate Takamol checklist because it will have been incorporated into the MOHP’s official supervision system tied to Decree 75. The national MMSS is a major achievement for Egypt, and a well-institutionalized and sustainable system that is worth strengthening in Lower Egypt. This is a legacy from an earlier USAID project that is worth protecting. Takamol should help Dr. Khaled to revitalize the national MMSS and SMC. Common issues related to maternal mortality could be discussed so that all MOHP sectors can assist: curative, private, blood bank, etc. Takamol should continue supporting the MMSS and SMCs to address avoidable causes of maternal mortality. For example, it could organize, with the central MOHP, regional meetings to discuss MMSS progress and issues. Governorates can benefit from each other, and those that are not performing well will take the matter seriously. Short-term local TA is an option. USAID might consider supporting an expert team to access the national MMSS for an independent view on how well it is currently functioning. Explore the feasibility of the American Chamber of Commerce (or a similar body) hiring a CSR point person with a job description covering activities similar to those undertaken by Takamol. This individual could act as the matchmaker between worthy health programs and interested corporations. The task must be expanded to include not only that kind of matchmaking but also to identify NGOs, donor-supported projects, or other vehicles for utilizing CSR funds in ways that are acceptable to corporations (a replacement for the role Takamol plays). The Egyptian Finance Executive Foundation fiduciary role should continue, if possible. 32 TAKAMOL MID-TERM EVALUATION TAKAMOL MID-TERM EVALUATION 33 V. EFFECTIVENESS OF THE MANAGEMENT AND ADMINISTRATIVE SYSTEM INTERNAL PROJECT MANAGEMENT The Takamol Project is a constellation of organizations with Pathfinder in the lead. Often such partnerships are difficult, with subcontractors jockeying for more control and recognition. In this case, all staff, including the local partner, HCI, identify themselves as Takamol staff, not by their formal employer. This suggests good strategic and managerial leadership. The evaluation team noted the benefits of having an all-Egyptian team in terms of costs (no expat support costs for housing, schooling, etc.) and close communication and understanding between the MOHP and the project. The Takamol team is technically and managerially solid, requests home office support only when needed, and presents a unified face to the stakeholders. Takamol is headed by a highly professional and experienced manager and by talented team leaders. The Chief of Party (COP) has demonstrated to the evaluation team and MOHP officials (including the Minister) his capability as a leader with a firm commitment to project outcomes and to supporting MOHP policy objectives. His management and interpersonal skills, as well as those of the senior management team, have overcome a number of the challenges of a health care system currently in transition. The project enjoys a good reputation among senior MOHP staff. All Results and Activity Team leaders are highly qualified and experienced in the areas they manage. In building on previous projects (Tahseen and HM/HC), Takamol recruited a large number of staff from those projects and from the MOHP who had experience and skills in technical and project management. Takamol, however, is not as strategic as it should be in identifying the most important technical issues and challenges facing implementation of the FHM and in using its extensive field experience to engage MOHP policymakers on necessary changes. The evaluation team identified a number of technical and programmatic issues that should have been identified, international best practice information compiled, and plans made to address those topics with MOHP officials using senior Takamol technical staff or consultants. Involving USAID in this kind of dialogue can work to the project’s advantage. Examples are (1) assisting the MOHP with more systematic follow-up to the UNFPA study6 that suggested that the FHM may be diluting attention to MCH and FP services; (2) relieving the policy confusion that no deliveries are allowed at PHC facilities and the related lack of oxytocin for AMTSL on the essential drug list for PHCs to use for either home-based or clinic deliveries; and (3) providing proactive leadership in identifying and promoting postpartum IUD insertion in hospitals that have trained and qualified ob/gyns to perform the procedure. While the Steering Committee is a good venue to raise these kinds of issues, unless a concerted program of follow-up is organized, they are likely to be forgotten in the press of other MOHP activities. The evaluation team does appreciate the difficulties Takamol staff have faced with changing leadership at the MOHP and the MOHP’s large number of competing priorities. The project appears to be appropriately staffed for the level of effort and work in 11 governorates. Five small regional offices help reduce the cost of travel, especially to Upper Egypt. (In the last year of the project a gradual decline in staffing might be expected, especially in the administrative area.) Delegation of authority between the COP, senior management, team leaders, and team members was clear, and team leaders and task managers were fully responsible for their own areas. 6 ―Study on Reproductive Health Impact of Family Health Model Pilots in Egypt,‖ El Zanaty & Associates, March 2008. 34 TAKAMOL MID-TERM EVALUATION COST AND FINANCIAL ISSUES The evaluation team did not analyze the budget in detail, but there are some broad issues that need attention. The project does not have pipeline problems, having spent 83% of the funds obligated and 61.4% of the total amount authorized. Takamol anticipates a $3.5 million shortfall in the budget for completing work in all 11 governorates for 200 clinics and 25 hospitals, due to inflation, particularly of the cost of materials for renovation but also overall inflation affecting, for instance, airfares. In addition to the shortfall, two other financial issues affect project implementation: (1) USAID has decided to discontinue IL financing of local costs for the MOHP and other ministries as of September 2009, and (2) the GOE has informed USAID that because of the decrease of the operating year budget from $400 million to $200 million, it wanted all FY 2009 funding to go toward budget support rather than TA. The evaluation team understands that discussions are underway but no final decision has yet been made. IL financing: The evaluation team believes that USAID is justified in withdrawing IL financing because the GOE is fully capable of incorporating these costs into its own budget. The current difficulty is temporary; funds for these purposes have not been included in the budget for the fiscal year that begins in July 2009. Therefore, when USAID IL funding ceases, the MOHP will need an alternate source of funding between September 2009 and July 2010, when these items can be budgeted. The elements of the two ILs that support MOHP activities related to the Takamol project are supervision and training on MCH and FP. The impact on supervision may not be as severe because Decree 75 incentive funds from the GOE are ensuring that governorate and district staff are motivated to make supervision visits. Central￾level supervision visits, which require funds for travel and per diem, will be negatively affected and will depend on what is available in the MOHP budget for supervision. The impact may be greater for travel and transport costs associated with training. These costs are estimated to be about $140,000, small compared with the size of the combined ILs. The MOHP is making a concerted effort to identify alternate sources of funding for most of the activities funded by the IL for the period after September 2009. The ministry is interested in exploring whether local currency could be made available to make up the deficit before July 2010 and would like more information from USAID on these funds to prepare for their discussions with MIC. Project funding in 2009 and beyond: If the GOE decides to accept FY 2009 funding for USAID projects, USAID has an opportunity to increase the Pathfinder contract and add funds beyond the current ceiling in order to complete the full set of 200 clinics and 25 hospitals as originally planned. The evaluation team recommends that this be done, if feasible. Because expansion of the FHM as currently planned is a combination of MOHP and USAID/Takamol–assisted facilities, completing the full number supported by Takamol will help achieve the targets set for the country. The team considered recommending that the project not be expanded into the last two governorates, Fayoum and Gharbia, and focus its activities in the nine governorates already engaged. The team, however, feels that this is a much less desirable option for the following reasons: (1) Even working with fewer facilities in Fayoum or Gharbia governorates, the project has an opportunity to introduce innovations and tools that can be used and expanded by the MOHP, RPCs, and governors; and (2) if this is the last USAID-supported MCH/FP/RH project in Egypt, the benefits are greater when as many governorates as possible have the opportunity to learn from, expand, and replicate the Takamol experience on their own. Fayoum and Gharbia could be replaced by two other governorates depending on the needs identified by the MOHP. If no further resources can be added to the project, inevitably Takamol will not be able to complete all the deliverables originally envisioned due to the unanticipated escalation in costs. TAKAMOL MID-TERM EVALUATION 35 CSI subcontract: Takamol has made a very useful contribution by helping CSI, now a registered NGO, to become more self-sufficient. CSI is now considering how to divest itself of unprofitable clinics and consolidate its management structure. USAID made a decision in mid-2008 not to continue funding CSI through the Takamol contract. For 10 years, both USAID and CSI should have been working systematically to help CSI become self-financing, particularly because USAID has been the sole source of support for this organization for the past 20 years. CSI is in discussions with the MOHP about whether MOHP wants to assume responsibility for any clinics or provide staffing. While the end of USAID funding is painful, the team does not believe that providing further assistance advances the goal of self￾sufficiency. CSI is likely to continue as a much smaller organization. Cost efficiency: The team requested information and looked at costs associated with certain activities, such as training, to get a sense of their magnitude, especially in relation to the likelihood of replication. The nine-day didactic training courses cost about $10,000 per clinic. OJT training for a six-month period is $4,000 per district. The team noted in the analysis of Result 1 that the didactic training, while necessary to introduce the entire package of clinical and management training, covered material not needed at the primary level. It may be worth condensing that course and giving more emphasis to OJT. At the clinic level the OJT is much more useful in building management skills than building clinical skills, which the MOHP is now working on by having less experienced doctors do hospital-based training in obstetrics and FP once a week. Eventually Takamol needs to fit its training materials into continuing MOHP training mechanisms. For hospital training the costs are $35,000 per hospital for didactic training and $23,000 for OJT. Didactic training days per hospital are 72 for the 15 courses, each with different target groups. For OJT total training days per hospital are 84 for all categories of staff (doctors, nurses, etc.). These costs are reasonable because as part of the effort Takamol is training two doctors in each hospital who will conduct most OJT in the future. If that effort is successful, training will be considerably less expensive for the MOHP. At the community level, estimating training costs is difficult because it is not clear what to consider ―training‖ and because there are so many cascade training activities within CDA-managed activities. The project simply divided the total amount spent to date on direct CDA support by the total number of beneficiaries for a figure of $3.72 per beneficiary. The costs appear to be reasonable and within the reach of CDAs that replicate activities, keeping in mind that not all Takamol-stimulated programs would necessarily be replicated by each CDA. USAID PROJECT OVERSIGHT The evaluation team did not find any problems with the relationship between USAID and the Takamol Project. The CTO is knowledgeable about the details of the project, meeting weekly with Takamol senior staff to review progress and discuss issues. Project Implementation Reviews conducted quarterly give the USAID Mission an opportunity to review progress and discuss problems in all program areas. Annual financial audits have revealed no serious problems. MANAGEMENT OF WORK WITH MOHP AND EXTERNAL PARTNERS Relationships with the MOHP appear to be excellent at all levels in the system. The evaluation team was informed by USAID partner projects like CHL and HS 20/20 that there is collaboration and regular consultation, and the interactions are productive. The Population Council was asked to review the ISOP, and Takamol used its updated material on LAM (with modifications), postpartum care, FP, and PAC. The evaluation team does have some suggestions for improving collaboration with other donors. Takamol was very open in sharing all training materials with UNICEF; however, little has been done to share its field experience in the application of the integrated MCH/FP model with organizations implementing similar programs in Egypt. Takamol should create an opportunity to exchange experiences 36 TAKAMOL MID-TERM EVALUATION and share the approaches and materials developed with organizations like UNICEF, UNFPA, and the Population Council that are engaged in similar activities. Takamol staff should consider, for instance, learning more about the UNICEF perinatal program in Minia, Assuit, Sohag, and Qena governorates, which focuses on maternal and newborn health. There are enough similarities between Takamol and UNICEF programs that collaboration and sharing experiences would benefit both. Takamol can learn, for instance, about UNICEF’s basic neonatal resuscitation activity. Sharing is especially important given that the results of the DHS 2008 reinforced the importance of placing high priority on reducing neonatal mortality in Egypt. UNFPA may also become more involved in maternal care issues. The team suggested that UNFPA staff contact Takamol to benefit from its program experience. RECOMMENDATIONS Takamol senior staff should move more strategically to bring its field experience and global state-of￾the-art technical knowledge to bear in influencing MOHP policies and programs. USAID should give Dr. Nasr El Sayed, MOHP, more information about the local currency account to facilitate MOHP discussions with MIC to identify ways to make up the shortfall related to discontinuation of IL funding. The Takamol Project should be fully funded up to the ceiling amount and, if feasible, the contract ceiling should be raised to add the funds needed to complete all facilities as planned. The salaries of staff should be increased as requested to prevent further staff losses. More formal information-sharing sessions with other donors working in MCH/FP should be organized to ensure that best practices and experience from Takamol are shared and that Takamol learns from the experience of others. TAKAMOL MID-TERM EVALUATION 37 VI. STRATEGIC AND MANAGERIAL APPROACHES OF TAKAMOL AND CHL The evaluation team was asked to assess the differences in approach to behavior change and demand generation (including community-based work, gender approach, developing best practices, etc.) of Takamol and CHL. The team met with the CHL evaluation team and project staff to discuss these differences and visited one CHL/SAVE-assisted CDA in Qena. The team also had access to the draft CHL evaluation report. However, because it was not possible to explore the CHL project in any detail, this report offers only a few observations. 1. Collaboration: CHL and Takamol have a good working relationship. All the educational and promotional material the evaluation team found in clinics and CDAs was produced by CHL except for a few older materials from previous projects. The brochures, posters, and flip charts were used by clinic providers, clients, and CDAs. The team saw that the CHL material (both older and new versions) was consistently available in all Takamol clinics and hospitals visited, usually in the ANC and FP counseling rooms. The JHU/CCP member of the Takamol project functions as a full-time liaison between the two projects. In this way, the team felt that CHL was fulfilling its mandate to provide communications and behavior change input to Takamol. 2. Contrast in mandates: The Takamol model has the advantages of being able to deal with both ―supply‖ and ―demand‖ improvements in support of integrated MCH/FP services at the PHC level. The work related to increasing demand for health services and utilization of PHC clinics is accompanied by very visible improvements in the infrastructure, equipment, and appearance of the clinics and the quality of care, which was cited frequently by clinic clients interviewed by the evaluation team. The behavior change and service utilization messages used by the CDAs are reinforced by clinic personnel during counseling sessions. The CDAs and clinics are linked in mutually reinforcing ways. CHL, on the other hand, has a different mandate, which is focused on communications and behavior change, not service delivery improvements. It functions as a crosscutting resource for other projects. 3. Different CDA models: The evaluation team did note some important differences in the Takamol and CHL/SAVE models for working with CDAs. CHL provides multiyear grants to CDAs with the amount declining each year to promote sustainability. The period of mentorship is longer and CHL￾assisted CDAs undertake a broader set of health and other activities based on the needs expressed by communities. Takamol only provides one year of funding but encourages CDAs to seek funding from other sources after that year is up (see the Result 3 discussion). It does not fund literacy programs or microenterprise but instead works with CDAs that are already engaged in such activities to build in the MCH/FP messages and link the CDA to the nearest clinic. The team visited CDAs in Luxor Governorate that had not been funded by Takamol but were replicating some of the same activities. CDAs visited in Takamol areas, especially in Upper Egypt, appeared to be highly energized, proud of their work, and confident of being able to continue in many areas. CDAs in Lower Egypt were noticeably less cohesive and less likely to be able to continue at the same level when Takamol’s inputs came to an end. Takamol staff believe this is due to both cultural differences between Upper and Lower Egypt and the history of donor funding. In the CHL/SAVE CDA visited by the evaluation team, there appeared to be much less confidence about continuing after CHL support ended. 4. Contrast in mechanisms: The team was asked to comment on any significant advantages or disadvantages of the project mechanisms, contract (Takamol) vs. cooperative agreement (CHL). The cooperative agreement mechanism may be more flexible in terms of changes to the statement of work or duration of the project than a contract, but the contract gives Takamol a measure of protection 38 TAKAMOL MID-TERM EVALUATION against those who want the project to make substantial changes or add new components. The team concluded that there are advantages and disadvantages of both, and that the nature of the activity should guide the choice of mechanism. TAKAMOL MID-TERM EVALUATION 39 VII. PROSPECTS FOR SUSTAINABILITY Many issues and recommendations related to sustainability have already been discussed. What follows is a summation of the evaluation team’s assessment of the prospects that elements of the Takamol program will continue beyond the life of the project. Service delivery improvements: From the beginning the Takamol Project design specifically cited encouraging sustainability by building the capacity of the MOHP. However, the most significant development for sustainability under Results 1 and 2 has come from Takamol being made a major implementing partner in the national program for expanding the FHM under the Health Sector Reform Program. Jointly developed Takamol-MOHP tools, such as the quality assessment checklists and training materials, have the best chance of being used beyond the life of the project if they become institutionalized in the training and supervision systems of the FHM and in accreditation standards. It is too early in the project for that to have happened yet, but it is not too early to work on ensuring that it does happen before 2011. The indications are generally positive; the ISOP has been adopted as an official MOHP guideline and distributed nationwide. District, hospital, and clinic staff are using the Takamol quality checklists to supplement the Decree 75 checklist, so they apparently recognize its value. Maintenance training, mobilizing SIFs, and strengthening clinic management are all activities that should improve a facility’s ability to continue those efforts beyond the life of Takamol. Building district capacity will not be completed by 2011, but tools and processes are now available from Takamol to continue that work. Aware, however, that unless some of these are institutionalized within the MOHP, they may not survive the project, Takamol staff are making efforts to ensure that is what happens. Community mobilization and education: The sustainability of Result 3 community work is less certain because the institutional base is more diverse and much depends on the culture and motivation of the communities themselves. Successful behavior change and increased utilization of MCH/FP/RH services is likely to continue because the people themselves recognize its value. The fact that Takamol does not create dependence by limiting CDA to small one-year grants and builds most of its CDA work on existing programs is also highly beneficial. The RPCs are the most promising mechanism to sustain the community mobilization programs. Some of the women’s empowerment, literacy promotion, and gender equality supported by the project also help sustain improvements in fertility and infant and child mortality. Women’s increased earning capacity in turn has a positive effect on child nutrition, and children—especially daughters—of educated mothers are more likely to be enrolled in school and to increase their educational attainment. A key approach is to encourage more egalitarian gender values, which among other benefits could encourage couple communication and female employment and weaken the preference for sons. Finally, educated women are more politically active and better informed about their legal rights and how to exercise them (e.g., obtaining ID and election cards). Improved enrollment rates for girls and adult literacy programs will help ensure the sustainability of future declines in mortality and fertility as the demographic transition in Egypt proceeds. The big picture: Without a doubt the MOHP in Egypt has the resources and capacity to continue the work initiated by Takamol at the facility, governorate, and district levels. If the MOHP can continue to work with the RPCs through activist governors, they can also increase the chances that both the facility and the community work will be carried on. Involving the corporate sector has been demonstrably successful, but the short-term initiative to carry forward the CSR program might need to come from the corporate sector itself. As long as they are shown ―win-win‖ opportunities, corporations are likely to remain interested. 40 TAKAMOL MID-TERM EVALUATION TAKAMOL MID-TERM EVALUATION 41 VIII. BEST PRACTICES AND LESSONS LEARNED Throughout this report, strategies, approaches, and interventions have been identified as successful or showing real potential for replication and sustainability beyond the life of the Takamol Project. The following points are worth reiterating: 1. Renovated facilities attract clients and help raise utilization rates. However, improving the quality of services is crucial to respond to the higher expectations of clients and to achieve better outcomes (proper management of cases and client satisfaction). Theoretical training for staff managing the renovated facilities will not necessarily lead to the desired upgrade in skills and outcomes; it must be complemented by well-structured OJT that happens on a regular basis, like continuing medical education. Takamol recognizes this fact in the structure of its programs. The MOHP has also recently begun to address the skill gap by sending younger doctors at clinics once a week for hands-on training at hospitals. Takamol staff can help systematize this kind of clinic training in the governorates and districts where it operates. 2. The involvement of religious and community leaders in health activities in Egypt is not new and is often cited as a reason why FP programs have enjoyed such broad-based support and acceptance. Takamol, however, has taken this involvement a step further by connecting the community religious and traditional leaders directly with health facility staff in a way that is mutually reinforcing, and hopefully enduring. Facility staff see an increase in utilization of clinic services and greater ownership by community leaders. They also are protected against criticism if they are viewed as providing a valuable service. Community and religious leaders feel empowered by their involvement in guiding youth or seeing members of their community achieving collective progress and in seeing tangible improvements in affordable health services. 3. The National Population Conference and the renewed emphasis on the role of National and Regional Population Councils offer excellent opportunities for the MOHP to link with other sectors to leverage resources. The MOHP has to be viewed as the core partner because it has the technical knowledge to reach the youth, education, agriculture, and social solidarity sectors with the right messages and is the only ministry that can offer quality FP services. The MOHP might offer supporting FP services in health facilities run by other sectors. 4. Takamol has made excellent use of data to identify problems or challenges early. Various audiences receive information through oral or written reports as they request. The results of household surveys have been communicated to CDAs to ensure that they fully appreciate the impact of their efforts. 5. Participation of community and health staff in PHC and hospital boards has ensured ownership and active collaboration. Keeping board members active, selection through election, and promoting the possibility of equal representation of males and females have been important factors in generating enthusiasm and sustaining community involvement. 42 TAKAMOL MID-TERM EVALUATION TAKAMOL MID-TERM EVALUATION 43 IX. SUMMARY OF KEY RECOMMENDATIONS RECOMMENDATIONS FOR TAKAMOL Recommendation Proposed Action Responsible Parties Proactive identification of policy and programmatic issues based on Takamol field experience and dialogue with MOHP, USAID, and other partners. Takamol senior team to systematically analyze and develop plans for addressing policy and strategic issues arising from Takamol‘s field experience with supporting the Family Health Model. Takamol with other stakeholders Develop simple ―how-to‖ guides for CDA activities for MOHP and RPCs. Practical and simple guidelines, ―cookbooks,‖ in Arabic are needed to ensure that key Takamol innovations are captured in a form the MOHP or RPCs can use to replicate activities. Takamol Help develop a coordinated strategy for improving postpartum FP as a follow-up to the 2008 National Population Conference. Help the MOHP to craft a coordinated strategy to address discontinuation rates and promote postpartum FP, especially immediate postpartum IUD insertions. Plan specific activities with the MOHP population unit to reverse the dilution of FP efforts within the Family Health Model. Help the AEA to create a curriculum addressing household and family impact of spacing children and limiting family size. Include maternal health. Takamol, CHL, UNFPA, Population Council with MOHP Takamol with MOHP Takamol with CHL and AEA Work more closely with donors like UNICEF and UNFPA to share Takamol experience and learn from their work in related areas. Organize periodic meetings to share program approaches, tools, training materials, etc. Takamol with USAID Revise the ISOP to ensure that guidelines are suitable for PHC facilities and emphasize high￾impact best practices such as AMTSL and basic newborn care. Work with MOHP on the ISOP revision now underway to make additional improvement as suggested in this evaluation. Takamol with MOHP Didactic physician and nurse training should be competency￾based and focus on PHC MCH/FP and newborn services. Maintain didactic training but integrate clinical training with the MOHP program of hands-on hospital-based training for new doctors. Ensure that nurses also receive sufficient clinical practice. Takamol Strengthen newborn care services in all activities because of the higher mortality rate compared to child mortality. Ensure newborn care is emphasized in provider training but also train RRs so that they can teach mothers, identify problems, and refer to clinics. Takamol with MOHP 44 TAKAMOL MID-TERM EVALUATION RECOMMENDATIONS FOR TAKAMOL Recommendation Proposed Action Responsible Parties Help MOHP to revitalize the National SMC and strengthen the MMSS and SMCs in Lower Egypt. Work with Dr. Khaled to ensure governorate SMC technical competence and help organize regional meetings to exchange experience. Takamol with MOHP Encourage CSR efforts to continue beyond the Takamol project. Help AmCham or others to establish CSR representative to broker partnerships with corporations, credible NGOs and projects, and MOHP facilities and programs. Takamol with corporate partners Recommendations for the MOHP Clarify policy to governorates and districts about PHC-based normal deliveries and put oxytocin on the essential drugs list for PHCs. Directives as appropriate MOHP Continue the excellent new program for strengthening clinical skills for young doctors with hospital-based training. Develop a continuing well-monitored program. MOHP Work with Takamol on ISOP updates to incorporate technical revisions suggested by the evaluation team. Review and agree on changes. MOHP with Takamol For the next two years, institutionalize Takamol tools and approaches in the MOHP supervision system, Decree 75, the Family Health Model, and accreditation schemes. Review the Takamol supervision tool with FP and MCH, then submit to Dr. Azza and Khaled Nasr for approval. When the Decree 75 supervision checklist is updated, strengthen performance monitoring. Submit the final version of the Decree 75 supervision tool to Dr. Nasr El-Sayed for official dissemination and institutionalization. MOHP with support from Takamol Work with Takamol to institutionalize the jointly developed tools to monitor provider performance and management improvements in PHC units, and institutionalize CQIS for hospitals and updated clinical protocols. Review Takamol tools and determine how and when they can be incorporated. MOHP with Takamol Accelerate the pace of accreditation of Takamol facilities. The MOH/TSO office should organize facility training schedules back-to-back with Takamol to complete requirements for accreditation. MOHP with Takamol TAKAMOL MID-TERM EVALUATION 45 RECOMMENDATIONS FOR TAKAMOL Recommendation Proposed Action Responsible Parties Recommendations for USAID If feasible, add sufficient funds to the Pathfinder contract to permit full implementation of the scope of work and contract deliverables in 11 governorates. Contract ceiling increase. If budget constraints require, maintain full set of facilities but seek non-project funding for some activities. USAID Brief the MOHP on procedures for requesting local currency from MIC to make up the temporary budget shortfall (especially for training activities) from the two ILs ending in September 2009. Arrange for briefing meeting. USAID Support Takamol efforts to improve networking with other donor partners. Participate in meetings as appropriate. USAID 46 TAKAMOL MID-TERM EVALUATION TAKAMOL MID-TERM EVALUATION 47 ANNEX 1. DOCUMENTS REVIEWED USAID DOCUMENTS Activity Approval Document: Support for Human Development and Economic Sector Development Cash Transfer Program. Audit Reports, Pathfinder International–Egypt Office for June 18, 2005 to June 16, 2006, June 16, 2006 to June 15, 2007 and June 16, 2007 to June 20, 2008. Cairo, Egypt, and National Population Conference 2008. Final Statement—Executive Summary and USAID Family Planning Conference Cable, June 9-10, 2008. Implementation Letters, Integrated MCH/RH/FP, January 15, 2009 Support for Human Development and Economic Sector Development Cash Transfer Program. No. 263-K￾650. USAID/Egypt Cash Transfer. Takamol/Pathfinder Contract and Pathfinder Proposal. USAID/Egypt. 2009. SO 20: Healthier, Planned Families (FY 2002–2011). USAID/Egypt Population and Health Performance Monitoring Plan. USAID/Egypt. 2009. USAID/Egypt Population & Health Overview; History and Results; USAID/Egypt Health Program; Avian Influenza; Infection Control; Viral Hepatitis; HIV/AIDS; MCH-FP-RH; Health Sector Reform Program. PowerPoint presentations. U.S. Economic Assistance Program in Egypt. 2006. USAID Assistance Agreement No. 263-0287. Sixth Amendment to Assistance Agreement between the Arab Republic of Egypt and the United States of America for Healthier, Planned Families, September 29, 2006. TAKAMOL PROJECT DOCUMENTS Achievements to Date. CQIS Checklist for Hospitals. HCI Presentation: Detailed Training Activities Supported by Related Training Documents. Integrated Reproductive Health Services Project ―Takamol.‖ Household Survey Report. PowerPoint Presentations on Results 1-4, Field Operations, Monitoring & Evaluation and Reporting, and Public Private Partnerships/Corporate Social Responsibility. Summary of Financial Pipeline. Takamol Project Annual Progress Report, October 1, 2006–September 30, 2007. Takamol Project Annual Progress Report, October 1, 2007–September 30, 2008. Takamol Project Performance Monitoring Plan. Takamol Project Annual Workplan, October 2008–September 2009. Takamol Project Quarterly Progress Report, October 1–December 31, 2008. Workshop Report of Governorates Family Medicine Coordinators, February 2009. 48 TAKAMOL MID-TERM EVALUATION OTHER DOCUMENTS CDA/CHL project: El-Gobeil Community Development Association, Danadara-Kena (Upper Egypt); Achievement Report–February 2009. CHL: Communication for Healthy Living, Johns Hopkins University Concept Paper, educational materials folder, and CD-ROM. CSI presentation (detailed establishment steps as NGO and current challenges). El-Zanaty, Fatma, and Ann A. Way. 2001. Egypt Demographic and Health Survey 2000. Calverton, Maryland: Ministry of Health and Population [Arab Republic of Egypt], National Population Council, [Arab Republic of Egypt], and ORC Macro. El-Zanaty, Fatma, and Ann A. Way. 2006. Egypt Demographic and Health Survey 2005. Calverton, Maryland: Ministry of Health and Population [Arab Republic of Egypt], National Population Council [Arab Republic of Egypt], El-Zanaty and Associates, and ORC Macro. El-Zanaty, Fatma. 2009. 2008 Egypt Demographic and Health Survey: Preliminary Findings and PowerPoint presentation. El-Zanaty, Fatma. 2008. Family Health Model Assessment Study, Funded by UNFPA. Health Systems 20/20/Egypt: Overview and Activities. Ministry of Health and Population Documents Accreditation Standards Assessment checklist (PHR); Summary Result Checklist for El-Saayda PHC unit at El-Zania district – Luxor (Upper Egypt). AEA Implementation Results of the Use of Health-based Literacy Curriculum in Five Governorates, February 2009. El-Gorna District Hospital (Luxor–Upper Egypt): Hospital Board of Directors. Completed CQIS checklist, December 2008. El-Gorna District Hospital (Luxor–Upper Egypt): SMC Formation and First Meeting Decisions. Final Evaluation of PHC Units (Decree 75 checklist) Monthly Form 2. Ismailia General Hospital (Sharkia–Lower Egypt) Neonatal Center: Comparable Annual Case Admissions and Case Fatalities between 2006 and 2008. Ismailia General Hospital (Sharkia–Lower Egypt) Board of Directors Donation Table (list of donors and amounts donated from Jan. 07 through Nov. 08). Ismailia Governorate Sustainability Activities (2009), developed by Social Solidarity, Health, and Education Undersecretaries (members of the Sustainability Committee – Ismailia Governorate RPC). Luxor MOHP District Office: Supervision Monitoring Checklist for PHC Units Related to Ministerial Decree 75. Luxor MOHP District Office: Supervision Monitoring checklist for PHC Units Developed by Takamol TA Office (includes clinical performance monitoring). Minia El-Kamh District hospital (Sharkia–Lower Egypt) Neonatal Center: Comparable Annual Case Admissions and Case Fatalities Between 2006 and 2008. Standards of Practice for Integrated MCH and RH Services, second edition, June 2008. TAKAMOL MID-TERM EVALUATION 49 ANNEX 2. INDIVIDUALS AND ORGANIZATIONS INTERVIEWED Location Name Title Organization MOHP Officials – Cairo Cairo Dr. Nasr Mohamed El-Sayed Minister‘s Assistant for PHC, Preventive Medicine and Family Planning MOHP Dr. Abdel Halim Population and Family Planning Sector MOHP Dr. Sahar El Sonbati Head, Population and Family Planning Sector MOHP Engineer Mohamed Abdel Rahman Supervisor, Central Directorate, TSO MOHP Dr. Azza El Desouky General Manager, Primary Health Care, TSO MOHP Dr. Khaled Nasr Undersecretary, PHC MCH Department MOHP Dr. Esmat Mansour Retired First Undersecretary for PHC MOHP Dr. Nagwa El-Hosseny General Manager, Quality Advisor, TSO MOHP Dr. Lamia Mohsen Prof. of Neonatology, Cairo University. Newborn advisor to the Minister of Health MOHP Government Officials – Governorates Luxor Dr. Samir Farag Head of Luxor City Supreme Council Luxor Governorate Regional Population Council Dr. Neamat Allah Undersecretary of Health MOHP Mr. Eid El Gowaly Undersecretary of Solidarity MOHP Mr. Nour El-Din Abdallah Undersecretary of Youth MOHP Mr. Fadl Aly Ahmed Undersecretary of Education MOHP Directorate Dr. Hanan Saleh PHC Directorate Manager MOHP Dr. Nabila Guirguis MCH Directorate Manager MOHP Dr. Nahed Moh. Ahmed FHM Directorate Coordinator MOHP District Dr. Obour Khairy District Manager MOHP Dr. Amel Farid MCH Manager MOHP Dr. Manar Gamal Moh FP Manager MOHP 50 TAKAMOL MID-TERM EVALUATION Location Name Title Organization Hagr Al-Aklta Clinic (Non-Takamol) Dr. Hussein El-Mohtady Clinic Director MOHP Al-Boayrat Clinic (Takamol phased out two months previous) Dr. Kirolos Megally Senior (First) Clinic Physician MOHP Dr. Hatem El-Wekeel Junior (Second) Clinic Physician MOHP Dr. Mohamed Salah Dentist MOHP Mr. Abdel Nasr El-Sayed Clinic SIF Manager MOHP Mr. Mohamed Enawy Member, Board of Directors (Teacher) MOHP Sheikh Hassan CDA (Boayrat Village) Mr. Ahmed Hassan Atteya CDA – Board Director Private Sector Ms. Amany El-Nouby CDA – Project Coordinator Private Sector Ms. Atteyat Abdo Teacher Private Sector Mr. Moh Abdel Hamid Azhar Institute Employer Private Sector Al-Gorna Hospital Dr. El-Khatib Hassan Aly Hospital Director MOHP Dr. Ahmed Abdel Moteleb Ob/Gyn Dept. Chief MOHP Dr. Maged Nasry Neonatal Dept. Chief MOHP Dr. Ayman Moh. Taha FP Dept. Chief MOHP Dr. Samy Hammad Al-Gorna District Director Al Kanak Al Gadeed Clinic (Operational) Dr. Fatma Ibrahim Clinic Director MOHP Mrs. Nadia Youssef Client (Exit Interview) MOHP El-Nawafel CDA (at Al-Karnak Al-Gadeed) Mr. Abou Elnaga Nofal CDA - Board Director Private Sector Ms. Fatma Faisal CDA – Project Coordinator Private Sector Mrs. Ragaa Awad CDA – Raedat Refeyate Private Sector Mrs. Sookeya Hagag CDA – Raedat Refeyate Private Sector El-Saayda Clinic (MOHP Replication Site) Dr. Shady Fawzy Senior (First) Clinic Physician MOHP Dr. Aly Ahmed Aly Junior (Second) Clinic Physician MOHP Dr. Viola Ezzat Zeneya District Director MOHP Qena El-Gabeel Clinic – Dandara Village (CHL Intervention Site) Dr. Adel Zekry Clinic Director MOHP TAKAMOL MID-TERM EVALUATION 51 Location Name Title Organization Dr. Abdel Moneim Danadara District Director MOHP Dr. Nasra Ameer Danadara District MCH Director MOHP El-Gabeel CDA - Dandara Village (CHL Intervention Site) Dr. Hazma Radwan CDA – Board Director Private Sector Mr. M. Abdel Fatah CDA – Project Coordinator Private Sector Six – October Dr. Fathy Saad Governor Six October Governorate Six-October Clinic (Slum Area Project Intervention Site) Dr. Naglaa Abou El-Wafa Clinic Director MOHP Dr. Nora Mohamed FP Specialist MOHP Dr. Osama Adbdel Gawad District MCH Director MOHP Dr. Atef Said District PHC Director MOHP El-Osra Elsaida CDA (Six October Governorate) Mrs. Abeer Hamdy CDA – Board Director Private Sector Ismailia (Sharkeya) Abdel Guelil El-Fakharany Governor Sharkeya Governorate Regional Population Council Dr. Sayed Abou El Kheir Undersecretary of Health MOHP Mr. Mohamed Abdallah Undersecretary of Solidarity MOHP Mr. Mohamed Maree Undersecretary of Education MOHP Dr. Fouad Abdel Baky Undersecretary of Youth MOHP Directorate Dr. Abdel Zaher Curative Manager MOHP Dr. Amal Ahmed Ismail PHC Manager MOHP Dr. Mervat Mansour MCH Manager MOHP District Dr. El-Sharkawy MCH Manager MOHP Dr. Ahmed Nabil Hospitals Manager MOHP Ismailia General Hospital Dr. Abou Zeid Mohamed Hospital Director MOHP 52 TAKAMOL MID-TERM EVALUATION Location Name Title Organization Dr. Ahmed Mira Quality Specialist MOHP Dr. Hussein Mostafa Ob/Gyn Dept. Chief MOHP Dr. Hanem Abdel Azeem Neonatal Dept. Chief MOHP Menia El-Kamh District Hospital Dr. Alaa Mekled Hospital Director MOHP Dr. Darwish Ahmed Quality Specialist MOHP Dr. Ezzat Mohamed Ob/Gyn Dept. Chief MOHP Dr. Soheir El-Alawy Neonatal Dept. Chief MOHP Shabanat Clinic and CDA Dr. Hala Shabana Clinic Director MOHP Mrs. Gehan Kamal Clinic Bookkeeper & Member of the Board of Directors MOHP Mr. Ahmed Abdel Gualil CDA – Board Director Private sector Mrs. Fatma Zeid Client – Exit Interview Beneficiary Bassateen El-Ismailia Clinic Dr. Mohamed Sayed Ahmed Clinic Director MOHP Dr. Hassan Zeyada PHC Clinic Specialist MOHP Showalia Clinic (Unannounced Clinic Visit) Dr. Mona Abdallah Clinic Director MOHP Belbis District Team Dr. Amal Abdel Rahman District Manager MOHP Dr. Yosry Mahmood Awad FP District Manager Assistant MOHP Dr. Maha Abu EL-Nour PHC District Manager Assistant MOHP USAID Officials Cairo Ms. Elizabeth Warfield Director, Office of Health and Education USAID/Egypt Ms. Lisa Childs CTO, Takamol Project USAID/Egypt Ms. Vikki Stein CTO, CHL Project, Acting Health Division Director USAID/Egypt Dr. George Sanad Health Management Specialist USAID/Egypt Ms. Shadia Attia M&E Officer USAID/Egypt Washington Ms. Holly Fluty Dempsey Chief, Health Division (away from post) USAID/Egypt TAKAMOL MID-TERM EVALUATION 53 Takamol Staff Cairo Eng. Mohamed Abu Nar Chief of Party Takamol Dr. Rania Moustafa Deputy Chief of Party Takamol Dr. Shahira Hussein Assistant Deputy Chief of Party Takamol Dr. Mohamed Afifi M&E/Reporting Team Leader Takamol Ms. Mona Hussein M&E Implementation Specialist Takamol Dr. Nagwa Samir PHC Services Team Leader (R1) Takamol Dr. Salwa Farag MCH Specialist (R1) Takamol Dr. Hala Youssef FP/RH Specialist (R1) Takamol Dr. Gasser Gad FP/RH Specialist (R1) Takamol Mr. Said El-Dib Hospital Services Team Leader (R2) Takamol Dr. Mohamed Mostafa Support Services & Obstetric Specialist (R2) Takamol Dr. Heba Sweed Quality Improvement Specialist (R2) Takamol Dr. Mohmed El-Hady Hospital Management Specialist (R2) Takamol Dr. Mohamed Farouk Neonatology Specialist (R2) Takamol Ms. Manal Eid BCC/IPC Team Leader (R3) Takamol Dr. Salah Eldin Elkhawaga Youth/Gender Specialist (R3) Takamol Dr. Alaa Bakr IPC-C Specialist (R3) Takamol Dr. Dina Bebawy NGO/Grants Manager (R3) Takamol Dr. Gamal ElKhatib Sustainability Team Leader (R4) Takamol Dr. Akram El-Zayat Capacity Building Specialist (R4) Takamol Mr. Mohamed Mansour Budgeting Procurement and Administration Team Leader Takamol Dr. Mohamed Ismail Sabry Integrated Services/Team Leader (R1- R2 Coordinator) Takamol Eng. Maged Youssef Field Operations Team Leader Takamol Dr. Marwan Abdel Fattah Field Operations Specialist Takamol Dr. Nader Nassif Private Sector/Corporate Responsibility Specialist Takamol Luxor & Qena Field Office Dr. Osama El-Krety Regional Field Office Manager Takamol Dr. Nagwa Attef Assistant Field Activities Manager Takamol Eng. Alaa Hamdy Senior Construction Field Manager Takamol 54 TAKAMOL MID-TERM EVALUATION Takamol Partners and Stakeholders Cairo / CHL Mr. Ron Hess Chief of Party, Communications for Healthy Living (CHL) JHU/CCP CHL Ms. Mohamed Farid Communication Advisor CHL Dr. Moshira Shafei FP & Communication Senior Advisor CHL Dr. Tawhida H. Khalil Senior NGO Communication Advisor CHL Cairo/HCI Dr. Samy Gadalla Chairman and Managing Director HCI Dr. Said Khalil Deputy Chairman HCI Dr. Alaa Afify Project Manager, Health Care International HCI Ms. Hanaa Ahmed Technical Support Officer, Technical Support Team HCI Dr. El Sayed Akl General Manager HCI Dr. Samir Aziz Senior Advisor HCI Dr. Mary Salama Senior Advisor (M&E) HCI Cairo/AEA Mr. Raafat Radwan Chairman, Adult Education Agency AEA Cairo/Save the Children – UK Mr. Steven Ashley Country Director Save the Children UK Dr. Alaa Sebeh Program and Advocacy Manager Save the Children UK Cairo/RCT Dr. Safa El Baz Regional Center for Training (RCT) Director RCT Cairo/EFEF Mr. Sherif Fathy Chairman – Egyptian Finance Executive Foundation (EFEF) EFEF Cairo/RWE Ms. T. Moussa GM Personal Assistant and Project Leader, Corporate Social Responsibility RWE, Egypt Cairo/SAIFPS (Specialized Association for Improved Family Planning, previously known as CSI) Dr. Mohamed Sweed Executive Director SAIFPS TAKAMOL MID-TERM EVALUATION 55 Dr. Ihab Fouad Kaldas Deputy Chairman SAIFPS Dr. Magdy Zein Technical Advisor SAIFPS Sahar Wahby Deputy Executive Director for Finance Administration and HR SAIFPS Cairo/Population Council Dr. Nahla Abdel-Tawab Associate, Reproductive Health Program, West Asia and North Africa Region Population Council Cairo/GH Team Leader Evaluation – CHL Project Mr. Mohamed Fathi Abdel Kader Assemblyman and CDA member Egypt People‘s Assembly Moncef M. Bouhafa Team Leader, CHL Evaluation Team GH Tech Cairo/Health Systems 20/20 Project Nadwa Rafeh Chief of Party, Health Systems 20/20 Project Abt Associates Cairo/UNICEF Dr. Magdy Sanady Health and Nutrition Officer UNFPA Cairo/ UNFPA Dr. Magdy Khaled Senior Maternal and Child Health Advisor UNFPA 56 TAKAMOL MID-TERM EVALUATION TAKAMOL MID-TERM EVALUATION 57 ANNEX 3. SCOPE OF WORK AND EVALUATION FRAMEWORK INTEGRATED REPRODUCTIVE HEALTH SERVICES PROJECT (“TAKAMOL”) MID-TERM EVALUATION (Revised: 11-26-08) PURPOSE OF THE EVALUATION The purpose of this evaluation, planned for February 2009, is to 1. Review, analyze, and evaluate the effectiveness of the Integrated Reproductive Health Services (―Takamol‖) project in achieving project objectives and deliverables and contributing to USAID/Egypt’s efforts to increase use of family planning and maternal/child health services. 2. Provide specific recommendations to ensure that the last 2½ years of the project are effective in achieving, to the extent possible, the intended goals of the project and of SO 20. This will include any recommendations on adjustments to the approach, minimum expected results, and/or activities, as well as necessary final/additional steps for sustainability, taking into consideration the nature of the procurement mechanism (contract) and available funding. 3. Implement a limited cross-program analysis in conjunction with the concurrent Communications for Healthy Living (CHL) mid-term evaluation. The resulting cross-program findings will be integrated into the final evaluation report of each project by the respective evaluation team leaders. BACKGROUND USAID has provided assistance to the Egyptian maternal/child health (MCH) and family planning (FP) programs since 1978. Over the last three decades support has been aimed at enabling the Government of Egypt (GOE) to increase demand for and utilization of MCH/FP services by Improving access to services and information; Improving the quality of clinical services; Ensuring adequate supplies of contraceptives and other commodities; Establishing effective policies and management and support systems. Support has been provided in many forms to a wide range of public, commercial, and NGO organizations and includes training to upgrade human resources; technical assistance to improve program coordination, management systems, research, and policy formulation; capital investment in renovation of clinical facilities; behavior change and community mobilization; and provision of vehicles, computers, furniture and equipment, and contraceptive commodities. In March 2006 USAID/Egypt began implementing its first integrated MCH/FP/RH project. The Integrated Reproductive Health Services project, known as Takamol (―integration‖ in Arabic), is a melding and enhancing of the previous Healthy Mother/Healthy Child project (1995–2005) and the Tahseen FP/RH project (2003-2006; before that, POP IV, 1997–2002). Takamol is implemented through a variety of international technical assistance and local implementing partners. The technical assistance component is implemented through a contract with Pathfinder International. Partners on this contract include John Snow Inc., Johns Hopkins University Center for Communication Programs, Meridian Group International Inc., AMEG (procurement agent), and local partner Health Care International. 58 TAKAMOL MID-TERM EVALUATION Takamol/Pathfinder The purpose of the Takamol/Pathfinder (―Takamol‖) contract is to assist the Ministry of Health and Population (MOHP) to achieve sustainable reduced fertility and improved health outcomes for mothers and newborns in its intervention areas throughout Egypt. The activities funded under this contract are expected to continue improvements in key indicators, including fertility and maternal, neonatal and post￾neonatal mortality. They are also expected to strengthen program planning, oversight, and implementation at different levels of the health system, so that continued progress can be achieved even after the end of USAID assistance to the health sector. End-of-project goals are for Takamol to work in 200 clinics in 11 Upper and Lower Egypt governorates, poor urban areas in Cairo, Giza, and Alexandria, and 25 district/general hospitals in Lower Egypt. The project focuses on (a) supporting the implementation of an integrated package of essential MCH/FP/RH services at both the primary health care (PHC) level and hospital-based care levels; (b) mobilizing communities, including community-level behavior change communication (BCC); and (c) strengthening MOHP capacity to sustain program performance at the national, governorate, district, and facility levels. Takamol has two cross-cutting themes central to its activities: gender and social responsibility/public￾private partnerships. Takamol works to improve the quality of care in both PHC facilities and relevant departments in hospitals by upgrading facilities, training service providers, and strengthening the referral system between PHC units and hospitals. As quality of care improves, Takamol begins to mobilize community leaders, religious leaders, youth, men, media, literacy facilitators, agriculture and irrigation extension workers, NGOs, outreach workers, theater groups, artists, and others to widely and consistently disseminate Takamol messages about healthier behaviors; begin the process of changing community norms to support behavior change; and mobilize community interest in both using and supporting clinics. To contribute to the long-term sustainability of this integrated model, Takamol reactivates dormant clinic boards, strengthening their capacities to utilize available funds such as service improvement funds. Takamol also works with the MOHP at all levels to build its capacity to replicate and sustain the model. Takamol has two cross-cutting themes that are central to all project activities: gender and mobilizing private companies and communities to partner with health initiatives. The project takes an all-inclusive approach to gender by working with both men and women to address gender inequities and enhance women’s knowledge and skills. With regard to the private sector, the project emphasizes the mobilization of private companies and advocates social responsibility/public-private partnerships as an initiative that uses a win-win philosophy to contribute to a healthier workforce and society. Because USAID/Egypt’s assistance to the MOHP will prematurely and unexpectedly phase out by 2009, this is a critical period in which successful approaches introduced during earlier phases of USAID/MOHP collaboration can be expanded and made more sustainable to further improve key indicators. Relationship to USAID/Egypt Strategic Objective 20 Through September 2011, USAID/Egypt will provide support to population and health activities of the GOE under the Strategic Objective ―Healthier, Planned Families‖ (SO 20). SO 20 was approved as part of the Mission Strategic Plan by USAID/Washington in January 2000 and was amended in 2004. ―Healthier, Planned Families‖ contributes to the USAID/Egypt Strategy ―A Globally Competitive Economy Benefiting Egyptians Equitably.‖ SO 20’s principal objective is to improve the quality, use, and sustainability of basic health and family planning services, supported by effective communication for behavior change, and health workforce development. Takamol’s objective of Sustainable Reduced Fertility and Improved Health Outcomes of Mothers and Newborns in its intervention areas contributes to the goals of SO 20. TAKAMOL MID-TERM EVALUATION 59 EVALUATION OBJECTIVES AND ILLUSTRATIVE EVALUATION QUESTIONS Final evaluation questions will be developed by the evaluation team in consultation with USAID/Egypt. The evaluation shall 1. Determine whether the Takamol Project is achieving intended goals and results given the program design and operating assumptions. a. Is the program design of the Takamol Project appropriate to achieve the Mission’s Strategic Objective 20 – ―Healthier, Planned Families‖? b. Has the project achieved yearly targets? To what extend has the Takamol Project met its goals and objectives? c. Assess the quality and effectiveness of project monitoring and evaluation activities; are data gathering methods reasonable for monitoring progress and indicators? d. Analyze changes to the organizational/strategic/contextual environment since the original design, which may impact project goals, objectives, or activities. Such changes would include any modifications in USAID goals as it phases out from the Population and Health sector or focuses on specific approaches or geographic areas (Sinai, poverty reduction), or mechanisms (PPPs/GDAs) USAID’s funding to the GOE via ILs, which will end prematurely and unexpectedly in 2009. MOHP/GOE approaches and/or organizational structures as it focuses on health sector reform and health insurance, and renewed efforts in population/family planning e. What are the key constraints or setbacks that may have hindered the achievement of results? f. To what extent are the Takamol Project achievements sustainable from financial, institutional, and programmatic perspectives? 2. Assess the effectiveness of key technical components and approaches of the Takamol Project. a. What has been the impact of integrating—for the first time in USAID/Egypt’s history—two historically vertical programs (MCH and FP)? Has the project been successful in achieving impact in both areas, and synergy among the two MOHP sectors? b. Do the project activities contain an appropriate mix and focus of technical interventions, particularly in light of the June 2008 National Population Conference and the GOE’s goal to revitalize efforts in population/family planning, as well as documented areas needing attention, such as wanted/unwanted fertility, gender preferences, method discontinuation, and side effects? c. Is the overall model as implemented in intervention communities and facilities appropriate and the most effective way of achieving the most significant and sustainable results? 3. Assess the effectiveness of management structures, administrative support, cost, and partnerships. a. Do Takamol’s existing administrative and management structures appropriately support the implementation of activities? b. Are Takamol’s staffing plans and organizational structures adequate and appropriate to meet objectives? c. Assess the costs of the various interventions implemented by the Takamol Project relative to the number of beneficiaries reached. d. Does it seem feasible to attempt nationwide scale-up of particular interventions that were successful? 60 TAKAMOL MID-TERM EVALUATION e. Has the Takamol Project facilitated synergy, coordination, and information-sharing among the USAID/Egypt team, NGOs, subpartners, other SO20 partners, other donors, and the GOE? 4. Document lessons learned and best practices and provide management/administrative and technical recommendations for improving efficiency and effectiveness. a. Based on current experience and lessons learned, what are the essential activities that should be continued or expanded over the life of the project? b. What changes, if any, should be considered by USAID/Egypt to make the Takamol Project more responsive and effective? In addition, there are a number of cross-program questions and issues that will be investigated by the Evaluation Team. USAID wishes to have this evaluation and the concurrent Communications for Healthy Living (CHL) mid-term evaluation jointly cover the following issues; in most cases, this information will already have been collected for the Takamol evaluation. It is expected that the team leaders for both evaluations will meet and synthesize findings on these issues during their last week in￾country. Review both project’s approaches to working with the community. CHL uses the Community Development Association (CDA) as an entry point into the community to effect behavior change in health, and health serves as an entry-point to engage civil society (the CDA) in introducing new forms of leadership (Arab Women Speak Out and Dawar) and community action (resource mobilization for expanded development). This in turn creates demand for services (at the health clinic or elsewhere) and ―pushes‖ on the clinic to provide quality services to the community. Takamol implements a holistic model that focuses on clinic improvements and building capacity of clinic staff so that they can provide quality services to the community. They simultaneously engage the community and civil society in behavior change activities, creating demand for services and accountability to the community for its health needs. Thus, it is a self-contained model inasmuch as it both creates demand and improves supply/quality. – What are the pros and cons of these different approaches? Are there any recommendations for modifications to either approach? Compare and contrast the different mechanisms (cooperative agreement vs. contract; acquisition vs. assistance). What have been the pros and con of each type of instrument and its ability to contribute to SO 20’s Results Framework (Strategic Objective and related Performance Indicators)? What is the ability of each instrument to respond to changes, opportunities, and needs of the MOHP and USAID? Describe both Pathfinder’s and CHL’s contributions to the public sector (MOHP and SIS in the case of CHL), the private sector, and the community. How does each project work with these different actors? How is each project working toward sustainability with these different actors? What expertise does each agency bring to its work? What is the level of its technical interventions, and its ability to carry out state-of-the-art work in its fields of expertise? How is it contributing to the general body of knowledge and state-of-the-art work in its fields? CHL was created to provide cross-cutting communication interventions to support SO 20 strategic priorities. Higher-level communication activities were not included in other projects. Did CHL meet its vision to provide cross-cutting communication services? Is it working? Does it make sense to have communication within a project or take it out (as in the case of CHL)? Has anything fallen through the cracks? What has been the impact and opportunity for each project to support and/or carry out health sector reform? TAKAMOL MID-TERM EVALUATION 61 Assess how each activity has worked with the MOHP (and SIS in the case of CHL). What has been effective in moving these public agencies forward? Compare and contrast the value of implementation letters, technical assistance, and long-term advisors. What is the value of these approaches in terms of sustainability? Compare and contrast each organization’s work in gender. In particular, how do they engage men? METHODOLOGY The evaluation team should consider a range of possible methods and approaches for collecting and analyzing the information that is required to assess the evaluation objectives. Data collection methodologies will be discussed with, and approved by, USAID at the start of the evaluation. The evaluation should take into consideration the current political/social/economic environment in Egypt; the demographic, family planning and maternal/child health situation; current GOE/MOHP strategies and directions; decentralization; program coverage; strategic partnerships, including with the private sector; community participation; and youth involvement. The evaluation team should use facilitative methods and activities that will enhance collaboration and dialogue among counterparts, particularly the MOHP and Takamol partners. The evaluation team will work in collaboration with the CTO for the Takamol Project. The CTO will organize all internal USAID meetings including linking the team with the Acting PH Team Leader. The evaluation team will propose and organize the evaluation process in collaboration with the CTO. The evaluation design and work plan will be presented to SO 20 team members for comments after the team planning meeting (TPM). The CTO, in collaboration with the Takamol Chief of Party, will arrange for an initial introductory meeting with appropriate stakeholders at the outset of the process. When appropriate, the CTO may participate in meetings with relevant stakeholders and partners. A general list of relevant stakeholders and key partners will be provided to the evaluation team by the CTO at the time of arrival but the evaluation team will be responsible for expanding this list as appropriate and arranging the meetings and appointments so as to develop a comprehensive understanding of the program and services offered through the Takamol contract. The evaluation team will also implement a cross-program analysis in conjunction with the concurrent Communications for Healthy Living (CHL) mid-term evaluation. The resulting cross-program findings will be integrated into the final evaluation report of each project by the respective evaluation team leaders. The final methodology and work plan will be developed as a product of the TPM and shared with the Mission prior to application. Document Review USAID/Egypt will provide the evaluation team with key project-related documents (see below) prior to the start of the in-country work. All team members will review these documents in preparation for the initial TPM. Prior to conducting field work, the evaluation team will review existing literature and data, including program strategies; quarterly reports; work plans; monitoring plans and reporting indicators/data; success stories; the contract and modifications (including the current pending modification); results from the USAID-Takamol Contract Review retreat held in February 2008; internal project financial records and reports as well as external audit reports; and any other reports and documents reflecting Takamol work in Egypt. Team Planning Meeting 62 TAKAMOL MID-TERM EVALUATION A two-day TPM will be held in Egypt before the evaluation begins. This meeting will allow USAID to present the team with the purpose, expectations, and agenda of the assignment. In addition, the team will – clarify team members’ roles and responsibilities; – establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion; – review and develop final evaluation questions; – review and finalize the assignment timeline and share with USAID; – develop data collection methods, instruments, tools and guidelines; – review and clarify any logistical and administrative procedures for the assignment; – develop a preliminary draft outline of the team’s report; and – assign drafting responsibilities for the final report. Internal USAID/Egypt Meetings Initial organizational/introductory meeting, at which the evaluation team will present an outline and explanation of the design of the evaluation (refer to the TPM noted above); Additional consultation meetings with USAID staff as needed to fully explain the project; A mid-evaluation meeting with USAID, following the field visits, to outline progress and implementation problems; A preliminary draft evaluation report review discussion during the final week; and A final evaluation debriefing: summary of the data, draft recommendations, and draft report. Field Visits/Key Informant Interviews Takamol currently intervenes in nine governorates (out of an eventual total of 11) as well as at a national level. After initial briefings by Takamol on project design, strategies, models, activities, and results, the evaluation team will arrange to visit selected sites in consultation with the CTO and the Takamol Chief of Party. Preliminary plans include at least one site visit to each of the following: a clinic in Upper Egypt; a hospital in Upper Egypt; a clinic in Lower Egypt; and a hospital in Lower Egypt (the exact sites will be selected before the TPM – at least 10 days before the team arrives – depending upon consultant time and resource availability and to ensure proper notification of GOE security; the CTO will send the tentative list of sites via email). The team may consider dividing into two groups for field visits to Upper and Lower Egypt. The evaluation team may be accompanied by a staff member from USAID/Egypt, as appropriate to facilitate meetings and field visits. The site visits will involve interviews (or focus group discussions as appropriate) with select governorate and district level MOHP staff, facility staff, clients, and community members. The evaluation team will also conduct interviews with central level MOHP staff, a donor organization (UNFPA), the Communications for Healthy Living project, the Specialized Association for Improving Family Planning Services (a local NGO that originated as a USAID project), the Regional Center for Training, a private sector partner, and any other key stakeholders identified during the TPM. TAKAMOL MID-TERM EVALUATION 63 Finally, during the last week in country, both CHL and Takamol evaluation team leaders will meet to compare and integrate cross-program findings and issues to add to their respective evaluation reports. Wrap-up and Debriefing At the conclusion of the field visits, there will be debriefing meetings with both USAID/Egypt and partners. The purpose of these meetings will be to share findings and get final inputs before preparing the report. DELIVERABLES Work Plan: During the TPM, the team will prepare a detailed work plan which will include the methodologies to be used in the evaluation. The work plan will be submitted to the CTO at USAID/Egypt for approval no later than the sixth day of work. Methodology Plan: A written methodology plan (evaluation design/operational work plan) will be prepared during the TPM and discussed with USAID prior to implementation. Discussion of Preliminary Draft Evaluation Report: The team will submit a rough draft of the report to the USAID CTO and PH Team, who will provide preliminary comments prior to final Mission debriefing. This will facilitate preparation of a more final draft report that will be left with the Mission upon the evaluation team’s departure. Debriefing with USAID: The team will present the major findings of the evaluation to USAID/Egypt through a PowerPoint presentation after submission of the draft report and before the team’s departure from country. The debriefing will include a discussion of achievements and issues as well as any recommendations the team has for possible modifications to project approaches, results, or activities. The team will consider USAID comments and revise the draft report accordingly, as appropriate. Debriefing with Partners: The team will present the major findings of the evaluation to USAID partners (as appropriate and as defined by USAID) through a PowerPoint presentation prior to the team’s departure from country. The debriefing will include a discussion of achievements and activities only, with no recommendations for possible modifications to project approaches, results, or activities. The team will consider partner comments and revise the draft report accordingly, as appropriate. Draft Evaluation Report: A draft report of the findings and recommendations should be submitted to the USAID CTO prior to the team leader’s departure from Egypt. The written report should clearly describe findings, conclusions, and recommendations. USAID will provide comment on the draft report within two weeks of submission. Final Report: The team will submit a final report that incorporates the team responses to Mission comments and suggestions no later than five days after USAID/Egypt provides written comments on the team’s draft evaluation report (see above). This report should not exceed 30 pages in length (not including appendices, lists of contacts, etc.). The format will include an executive summary, table of contents, methodology, findings, and recommendations. The report will be submitted in English, electronically. The report will be disseminated within USAID. A second version of this report excluding any potentially procurement-sensitive information will be submitted (also electronically, in English) for dissemination among implementing partners and stakeholders. The final report will be edited/formatted by GH Tech and provided to USAID/Egypt approximately one month after the Mission has reviewed the content and approved the final revised version of the report. This final revised version of the report can be used as a working document while final report editing/formatting is in process by GH Tech. 64 TAKAMOL MID-TERM EVALUATION TEAM COMPOSITION We would like to engage the services of a team of four local and international consultants to work on the evaluation team. The team should include specialists with the following areas of expertise: family planning/reproductive health, maternal/child health, sustainability, and behavior change communication/community mobilization. In addition, the team should include one local consultant who has an excellent understanding of the Egyptian public health system and USAID programs and is fluent in Arabic. At least one member should also have strong expertise in monitoring and evaluation. The Team Leader should be an independent consultant but one and/or two of the technical specialists could be USAID/Washington Global Health staff. The fourth team member will be the local technical consultant. Team Leader/Technical Specialist: Should be an independent consultant and have an MPH or related postgraduate degree in public health. S/he should have at least 10 years senior-level experience working in health systems programs in a developing country. S/he should have extensive experience in conducting qualitative evaluations/assessments. Excellent oral and written skills are required. The team leader should also have experience in leading evaluation teams and preparing high-quality documents. This specialist should have wide experience in implementation of USAID-funded MCH/FP/RH programs. S/he should also have a good understanding of project administration, financing, and management. The team leader will take specific responsibility for assessing and analyzing the project’s progress towards quantitative targets, factors for such performance, benefits/impact of the strategies, and compare with other possible options. S/he will also integrate the CHL-Takamol cross-program findings into the final evaluation report. S/he will provide leadership for the team, finalize the evaluation design, coordinate activities, arrange meetings, consolidate individual input from team members, and coordinate the process of assembling the final findings and recommendations. S/he will also lead the preparation and presentation of the key evaluation findings and recommendations to the USAID/Egypt team and key partners. Family Planning/Reproductive Health and Maternal/Child Health Specialists: The FP/RH and MCH specialists will have at least 7-10 years of experience in management of, or consulting on, FP/RH and MCH programs. S/he should have a proven background and experience in FP/RH and MCH and a strong understanding of the challenges facing FP/RH and MCH programs in Egypt or the Middle East. S/he should also have a good understanding of the relevant national programs in FP/RH and MCH, including the public and private sector. The FP/RH and MCH specialists will be responsible for assessing the ability of the project to achieve outcomes in FP/RH and MCH and provide technical leadership in FP/RH and MCH. The FP/RH specialist will also assess the technical quality of Takamol FP/RH and MCH interventions. S/he will document key lessons learned and provide recommendations for modifications in approach, results, or activities. These two specialists could be combined in one consultant if appropriately qualified. However this would not decrease the number of evaluation team members required. Behavior Change Communication/Community Mobilization Specialist: This specialist should have wide experience in implementation of behavior change communication and community mobilization programs in the areas of MCH/FP/RH. S/he should have a postgraduate degree in health promotion sciences or a related field with a minimum of 5-10 years experience working with USAID-supported behavior change and community mobilization programs in developing countries. S/he will analyze Takamol’s behavior change interventions in its intervention communities and assess the effectiveness and appropriateness of the approaches adopted by the project to improve MCH/FP/RH knowledge, health-seeking behavior, and health outcomes. S/he will also assess the technical foci of BCC activities, and whether they are the appropriate mix and topics for intervention communities. TAKAMOL MID-TERM EVALUATION 65 Sustainability Specialist: The sustainability specialist should have at least 5-10 years experience working in the area of systems/financial/institutional/management sustainability. S/he should have a good understanding of health systems in the Middle East, preferably in Egypt. The sustainability specialist will look at the sustainability of Takamol project approaches and activities as well as the ability of the project to leverage and influence MOHP programming, including adoption by the MOHP of Takamol innovations. TIMELINE AND ESTIMATED LEVEL OF EFFORT USAID/Egypt anticipates that the period of performance of this assessment will be approximately one month. This will include preparation days, in-country work in Cairo and the governorates, and report￾writing and finalization. The evaluation should begin on or about February 8, 2009. The team leader will be responsible for the overall planning, design, and implementation of the evaluation and work coordination among team members. It will be the team leader’s responsibility to submit a satisfactory report to USAID within the agreed timelines. The team leader is responsible for report writing and organization of the debriefing presentations. The final report should be made available within two weeks of the receipt of comments on the draft report from USAID and partners. Program schedules for field visits shall be provided before/on arrival in Egypt. Task/Deliverable Duration/LOE Team Leader ** Technical Specialists (x2) Local Consultant 1. Review background documents & offshore preparation work 4 days 3 days 3 days 2. Travel to Egypt 2 days 2 days 3. Team Planning Meeting and meeting with USAID/Egypt 2 days 2 days 2 days 4. Information and data collection. Includes interviews with key informants (stakeholders and USAID staff) and site visits.* 15 days 15 days 15 days 5. Discussion, analysis, and draft evaluation report in country 5 days 5 days 5 days 6. Debrief meetings with USAID and key stakeholders. (preliminary draft report due to USAID) 1 day 1 day 1 day 7. Team leader meets with CHL TL, & USAID to synthesize findings/discussion 2 days 8. Depart Egypt/Travel to US 2 days 2 days 9. USAID & partners provide comments on draft report (out of country) – 10 working days 10. Team revises draft report and submits final to USAID (out of country) 5 days 3 days 3 days 11. USAID completes final review 12. GH Tech edits/formats report (one month) Total Estimated LOE 38 days 33 days (2 people) 29 days * A six-day work week is authorized when working in country. ** Additional LOE may be required for the CHL and Takamol team leaders to meet to synthesize findings on the points identified in section 3 above during their last week in-country, depending on whether both evaluations are run concurrently or separately. 66 TAKAMOL MID-TERM EVALUATION LOGISTICS USAID/Egypt will provide overall direction to the evaluation team, identify key documents, and assist in facilitating a work plan. USAID/Egypt will assist in arranging meetings with key stakeholders as identified by USAID prior to the initiation of field work. The evaluation team is responsible for arranging other meetings as identified during the course of this evaluation and advising USAID/Egypt prior to each of those meetings. The evaluation team is also responsible for arranging vehicle rental and drivers as needed for site visits around Cairo but USAID/Egypt will facilitate travel to sites in the governorates (including air travel when/if necessary). USAID/Egypt can also assist with hotel arrangements if necessary but the evaluation team will be responsible for procuring its own work/office space, computers, Internet access, printing, and photocopying. Evaluation team members will be required to make their own payments. USAID/Egypt personnel will be available to the team for consultations regarding sources and technical issues, before and during the evaluation process. POINT OF CONTACT The evaluation team will work under the direction and guidance of USAID/Egypt MCH/FP Program Manager Lisa Childs and in collaboration with the SO 20 team. Lisa Childs MCH/FP Program Manager and Takamol/Pathfinder CTO Population and Health Division USAID/Egypt KEY DOCUMENTS: USAID/Egypt SO 20 Assistance Agreement, 2008 SO 20 AAD, 2007 SO 20 Operational Plan, 2007 SO 20 Performance Monitoring Plan Egypt Demographic and Health Survey, 2008 Preliminary Report Egypt Demographic and Health Survey, 2005 National Population Conference outcomes/report and USAID reporting cable, June 2008 Takamol/Pathfinder Contract (includes final proposal and USAID Statement of Work) Takamol/Pathfinder workplans Takamol/Pathfinder quarterly and annual reports Takamol/Pathfinder Performance Monitoring Plan Takamol/Pathfinder monitoring information and reports (baseline data, Household Survey results, monitoring database, etc.) Takamol/Pathfinder financial and audit reports Contract Modification currently in process (outcome of joint USAID-Pathfinder mid-term contract review, February 2008) TAKAMOL MID-TERM EVALUATION 67 Various other Takamol/Pathfinder project materials as appropriate Illustrative Evaluation Questions: Final Evaluation Questions will be developed by the evaluation team in consultation with USAID/Egypt. Assess overall progress made in implementing the project. Is the project/contract on track to achieve Life of Project goals and objectives? Assess the quality of and progress towards achieving minimum expected results and targets as outlined in the project Performance Monitoring Plan (PMP). What has been the impact of integrating—for the first time in USAID/Egypt’s history—two historically vertical programs (MCH and FP)? Has the project been successful in achieving impact in both areas and synergy between the two MOHP sectors? Assess the quality and effectiveness of project monitoring and evaluation activities: Are data￾gathering methods reasonable for monitoring progress and indicators? If not, explain and suggest possible modifications. Are PMP indicators appropriate and/or valid? If not, explain and suggest possible modifications. Has the project done an adequate job in documenting models, activities, challenges/weaknesses, and successes/achievements? Identify any strengths and weaknesses in project implementation, as well as challenges that have arisen that have impacted, or may impact, achievement of project results. Assess whether project activities contain an appropriate mix and focus of technical interventions, particularly in light of the June 2008 National Population Conference and the GOE’s goal to revitalize efforts in population/family planning; as well as documented areas needing attention such as wanted/unwanted fertility, gender preferences, method discontinuation, and side effects. Assess the effectiveness (technical and cost) of implementation approaches and models; this includes assessing each of the four results and two cross-cutting themes (gender and SR/PPPs), as well as the key project components and approaches (renovation, capacity building, community mobilization/behavior change, sustainability, etc.) to achieving project outcomes and results. Is the overall model as implemented in intervention communities and facilities appropriate and the most effective way of achieving the most significant and sustainable results? Assess the effectiveness and efficiency of the project’s management structure and systems (e.g., administering subgrants, providing technical assistance, liaising with the MOHP, etc.). Review staff composition and capacity, especially technical capacity of project personnel; project systems and management techniques; and the effectiveness of the relationship between Pathfinder and its subs. Does the technical assistance provided through Pathfinder International, John Snow, Inc., Johns Hopkins University, and Meridian Group International, Inc. meet the programmatic needs of the project and its stakeholders? How effective is the prime (Pathfinder International) in utilizing the skills and expertise of its subs? Are there any gaps in technical assistance? How effective has the project been in managing its work across its current intervention governorates and field offices? Assess whether the project has facilitated synergy, coordination/collaboration, and information￾sharing with USAID/Egypt, the Communications for Healthy Living project, other USAID projects (health and other sectors), other donors, and the MOHP, including MOHP sectors outside of FP/RH and MCH, such as Health Sector Reform. Analyze changes to the organizational/strategic/contextual environment since the original design that may impact project goals, objectives, or activities. Such changes would include any modifications in – USAID goals as it phases out from the Population and Health sector or focuses on specific approaches or geographic areas (Sinai, poverty reduction), or mechanisms (PPPs/GDAs) 68 TAKAMOL MID-TERM EVALUATION – USAID’s funding to the GOE via ILs, which will end prematurely and unexpectedly in 2009. Will this impact the type and extent of TA that Takamol provides to the MOHP? What changes, if any, will need to be made to the contract SOW? – MOHP/GOE approaches and/or organizational structures as it focuses on Health Sector Reform and health insurance, and renewed efforts in population/family planning Assess the effectiveness of the procurement mechanism (contract) in achieving project and USAID results. Has this mechanism enabled the project to adequately respond to technical/programmatic/strategic changes, needs, and opportunities? Assess progress toward sustainability particularly in the following areas: – Phased-out communities/facilities – Replication (adoption and scale-up) of project interventions in nonproject sites or spheres by the MOHP, other donors, etc. – Work with Community Development Associations (CDAs) – Project activities under Result 4: in particular, has the project built the capacity of the MOHP to sustain project results to the maximum extent possible? Is the project transferring, to the extent possible, key activities and capacity (e.g., in renovation, training, ability to update technical standards, etc.) to the MOHP? Is providing technical assistance to the MOHP via a parallel project the best approach to building the MOHP’s capacity? If not, explain. Recommend any adjustments in technical/programmatic approach, minimum expected results, activities, or management structure over the remaining 2½ years of the project, as well as any necessary final/additional steps for sustainability. In particular, make priority recommendations for final steps in the area of contraceptive security that are within the mandate and funding level of the contract. If changes to the contract statement of work are suggested, describe the approximate contractual/procurement implications of such recommendations. Highlight any Takamol models, programmatic interventions, lessons, or strategies that could or should be replicated if USAID funding were to become available after 2009 for MCH/FP/RH programs. TAKAMOL MID-TERM EVALUATION 69 EVALUATION FRAMEWORK Evaluation Questions Required Documents Assessment Methodology 1. Determine whether the Takamol Project is achieving intended goals and results given the program design and operating assumptions. 1.a. Is the program design of the Takamol Project appropriate to achieve the Mission‘s Strategic Objective 20 – ―Healthier, Planned Families‖? 1. SO 20 2. Project Statement of Work 3. Takamol Work Plan 4. Memorandum of Cooperation with various partners. 5. SOAG Program design assessment tool: to highlight project activities addressing goals of SO 20 with respect to Project Statement of Work; understanding program context (needs and challenges) 1.b. Has the project achieved yearly targets? To what extent has the Takamol Project met its goals and objectives? 1. Takamol annual work plans 2. Takamol annual achievements (annual reports) 3. Project PMP 4. Household survey results 5. Project quarterly reports Project achievements tool: to determine program objectives versus project achievements and planned targets 1.c. Assess the quality and effectiveness of project monitoring and evaluation activities; are data-gathering methods reasonable for monitoring progress and indicators? 1. M&E indicators (Performance Monitoring Plan) with its definitions 2. Other data collection instruments 3. Dissemination and use of data Verify M&E system Discuss with USAID staff the M&E project plan and details of each selected indicator Discuss with Takamol staff data-collection procedures, use of information, etc. Discuss with the MOH project information￾sharing, extent of use of data for decision making, and intention to promote sustainable application of innovations. 70 TAKAMOL MID-TERM EVALUATION 1.d. Analyze changes to the organizational/strategic/contextual environment since the original design that may impact project goals, objectives, or activities. Such changes would include any modifications in: 1.d.1. USAID goals as it phases out from the Population and Health sector or focuses on specific approaches or geographic areas (Sinai, poverty reduction) or mechanisms (PPPs/GDAs) 1.d.2. USAID‘s funding to the GOE via ILs, which will end prematurely and unexpectedly in 2009 1.d.3. MOHP/GOE approaches and/or organizational structures as it focuses on Health Sector Reform and health insurance, and renewed efforts in population/FP 1. Contract modifications 2. Interviews with USAID officials and MOHP (Studied against project goals and objectives) * List of activities added to meet MOHP newly adopted policies * Prioritized interventions and cancelled ones supported by justifications * List of excluded sites and those newly considered with criteria for selection * Updated program intervention model and/or intervention approaches Meeting series with MOH decision makers at different administrative levels to determine their understanding of project modifications and changes as well as project progress and problems. Meet with USAID on issues related to phase￾down of the program and its impact. Meet with HSR to determine extent of cooperation and recommended activities to strengthen the complementary role of the project toward MOH policy. 1.e. What are the key constraints or setbacks that may have hindered the achievement of results? (Already included in annual performance reports and shown in the M&E indicators) Meet with Takamol TA staff: with task￾responsible persons to determine and understand constraints and obstacles that hindered achievements. Meet with MOHP managers and USAID staff for their views. Explore HCI role. TAKAMOL MID-TERM EVALUATION 71 1.f. To what extent are the Takamol Project achievements sustainable from financial, institutional, and programmatic perspectives? Are they replicable and can MOHP take them to scale? 1. Sustainability plans for the different activities 2. Collaborate with GOE and other organizations targeting upholding of critical influential activities. 3. Dissemination of tools and products from the project Meet with Takamol TA staff: to understand from team leaders and project management team plans designed to sustain the positive to￾date achievements. Meet with central level directors to ensure realistic approaches for sustainability plans and their commitment to its implementation; discuss CSR initiatives related to long-term sustainability. Meet with district and governorate level managers. 2. Assess the effectiveness of key technical components and approaches of the Takamol Project. 2.a. What has been the impact of integrating—for the first time in USAID/Egypt‘s history—two historically vertical programs (MCH and FP)? Has the project been successful in achieving impact in both areas and synergy among the two MOHP sectors? 1. Household surveys and project data 2. Steering Committee meetings showing understanding and cooperation of both sectors 3.New activities reflecting the adoption of the integrated approach at MOHP facilities 4.Integrated training and counseling materials review 5. National guidelines (FP), Integrated SOP, Decree 75 supervision checklist, accreditation standards Meet the Undersecretaries of different governorates to get their feedback and opinions with respect to the integrated approach. Series of meetings with PHC, MCH directors at governorate, district, and facility levels (different governorates) to determine how successful the integrated approach proved to be and their recommendations. Meet with Population Council, PHC, training organization directors to get their feedback and views that could strengthen the new integrated approach. Meet with MOHP service providers and facility clients to view integrated services from that level. 72 TAKAMOL MID-TERM EVALUATION 2.b. Do the project activities contain an appropriate mix and focus of technical interventions, particularly in light of the June 2008 National Population Conference and the GOE‘s goal to revitalize efforts in population/family planning; as well as documented areas needing attention, such as wanted/unwanted fertility, gender preferences, method discontinuation, and side effects? 1. Recommendations of 2008 National Population Conference 2. Technical interventions and new approaches developed by Takamol TA office targeting 2008 NPC recommendations and GOE's goal. 3.Protocols, clinical standards 4.Evidence of how Takamol responded to conference recommendations 5.Evidence of how Takamol addresses gender, youth, discontinuation, side effects, etc. Meet with Takamol TA staff to identify the project approaches (existing and newly developed) that address the June 2008 National Population Conference recommendations and GOE's goals. Meet with MOHP managers to explore how project results and approaches are contributing to national population goals. 2.c. Is the overall model as implemented in intervention communities and facilities appropriate and the most effective way of achieving the most significant and sustainable results? 1. Takamol Community Intervention Model and example of implementation feedback and results 2. Different Takamol facilities‘ intervention models and example of implementation with its results Discuss with Takamol TA team leaders the results of implemented models designed for the community and facilities (basic and comprehensive) with respect to MOHP/Project goals and sustainability plans. Discuss with governorate level directors the appropriateness of different project interventions and their recommendations to attain better and sustained positive results. Discuss with health providers, clients, and community the interventions in Takamol. 3. Assess the effectiveness of management structures, administrative support, cost, and partnerships. 3.a. Do Takamol‘s existing administrative and management structures appropriately support the implementation of activities? 1. Takamol organizational structure and background of posted candidates Meet with Takamol TA staff (field level and implementers) to discuss administrative and managerial constraints that need to be addressed as well as proposed solutions Meet with Takamol COP to discuss role of contract partners. 3.b. Are Takamol‘s staffing plans and organizational structures adequate and appropriate to meet objectives? Project staffing plans, composition and organizational charts Meet with Takamol TA staff (team leaders and subordinates) to discuss implementation process difficulties (causes). Explore home office support from Pathfinder and partners. TAKAMOL MID-TERM EVALUATION 73 3.c. Assess the costs of the various interventions implemented by the Takamol Project relative to the number of beneficiaries reached. 1. Community intervention costs versus number of beneficiaries 2. Facility intervention cost versus number of beneficiaries Discuss with the MOHP beneficial interventions that could be sustained with respect to their cost-effectiveness. 3.d. Does it seem feasible to attempt nationwide scale-up of particular interventions that were successful? Any costing studies or information Discuss with Takamol TA staff successful interventions and the cost of each. Discuss with MOHP nationwide scale-up of Takamol proposed interventions with respect to cost. 3.e. Has the Takamol Project facilitated synergy, coordination, and information￾sharing among USAID/Egypt team, NGOs, subpartners, other SO20 partners, other donors, and the GOE? 1. Dissemination activities and supporting documents 2.Agreements and/or collaborative work and/or complementary activities with other parties and organizations 3. Specific assistance to CSI Meet with other organizations (CSI, CHL, USAID, RCT, UNFPA, Save the Children, Private sector [P&G], CDAs and others to determine extent of coordination. 4. Document lessons learned and best practices and provide management, administrative, and technical recommendations for improving efficiency and effectiveness. 4.a. Based on current experience and lessons learned, what are the essential activities that should be continued or expanded over the life of the project? Identify best practices (evidence-based). 1. Lessons learned and field experiences: a. List of positive field experience to facilitate selection of those to continue and/or expand b. List of negative activities to discuss discontinuation or modifications Select with Takamol TA staff activities that should be continued or expanded over the life of the project and partners based on lessons learned and field experience. Discuss with MOHP and selected partners activities to be continued or expanded over the life of the project. 4.b. What changes, if any, should be considered by USAID/Egypt to make the Takamol Project more responsive and effective? Meet with USAID, MOHP, and Takamol staff to explore whether any changes are warranted to the implementation of Takamol. 74 TAKAMOL MID-TERM EVALUATION 5. Assess the strategic relationship between Takamol and CHL and the various approaches employed by each project. 5.a How do the community approaches differ between the projects and the relative advantages/drawbacks of each? Project annual and quarterly reports, scopes of work, and field visits Meet with CHL evaluation team. Meet with CHL staff. Meet with MOHP staff. Meet with Takamol CHL coordinator. 5.b. What are the advantages and problems with the two mechanisms? How do they affect effectiveness and responsiveness of the projects? Meet with CHL and Takamol CTOs at USAID and project COPs, MOHP staff inputs. 5.c. How do the approaches and effectiveness of each project differ on gender, capacity building with public sector institutions, and health sector reform? Information gathered from CHL staff and field observations View CHL activities in field sites. Meet with CHL evaluation team. Meet with CHL staff. 5.d. How has CHL functioned as a cross￾cutting resource for communication activities? How has it served Takamol needs? Information from Takamol staff Meet with Takamol staff. 6. Assess the various aspects of long-term sustainability of the Takamol program interventions. 6.a. Phase-out of communities/facilities Project reports, quarterly and annual, field visits, and interviews Meet with MOHP managers and senior officials, health facility boards. 6.b. Replication (adoption and scale up) Safe Motherhood Committees, Regional Population Council Sustainability subcommittees Meet with MOHP managers and policy makers. 6.c. Work with Community Development Associations Any CDA documents or records Meet with MOHP managers, policy makers, and community leaders. 6.d. MOHP capacity (financial, managerial, etc) to carry on activities Any documents that quantify ‗capacity development‘ Meet with MOHP managers and policy makers. TAKAMOL MID-TERM EVALUATION 75 ANNEX 4. ANALYSIS OF THE TAKAMOL PERFORMANCE MONITORING PLAN (PMP) The Takamol Performance Monitoring Plan is comprised of 23 indicators, which cover three levels: the community level; the primary health care (PHC) unit, pertaining to both clinical indicators, such as measures of quality of care and percentage increase in caseload, and management-related indicators, such as board activation and use of available Service Improvement Funds (SIFs); and finally the hospital level, pertaining to both clinical indicators, such as quality of care and client satisfaction, and management indicators, such as board activation and use of available SIFs. A wide variety of data sources are used, which are collected at different intervals, such as Facility service statistics (monthly) Client exit interviews (monthly) at the PHC and hospital level SIF records (monthly) at the PHC and hospital level Community Development Association (CDA) activity progress reports (monthly) Quality assessments (monthly for PHCs, quarterly for hospitals) Facility management board meeting minutes (monthly) Pre- and post-event tests (event-based) Training reports (event-based) for all trainings Replication records (event-based) Corporate Social Responsibility documents/record for private-sector fundraising Household surveys, both pre-and post-intervention surveys The four issues addressed in this evaluation are: (1) Assess the quality and effectiveness of project monitoring and evaluation activities. (2) Are data-gathering methods reasonable for monitoring progress and indicators? If not, explain and suggest possible modifications. (3) Are PMP indicators appropriate and/or valid? If not, explain and suggest possible modifications. (4) Has the project done an adequate job in documenting models, activities, challenges/weaknesses, and successes/achievements? ASSESS THE QUALITY AND EFFECTIVENESS OF PROJECT MONITORING AND EVALUATION ACTIVITIES. The monitoring and evaluation (M&E) system developed by Takamol is well-conceived, systematic, and thorough. Important achievements of the Takamol project are measured largely through quantitative process, output, and outcome indicators rather than impact measures, and the use of data at all levels for feedback, problem-solving, and decision making has been ensured through frequent data analysis and distribution to Takamol staff, USAID, and other partners. On the basis of the data sources listed above, a variety of databases have been created that ensure efficient data entry and analysis. Although monthly reports are produced for the Result teams, quarterly data reports are even more useful and can be compared to the goals of the annual work plan. Written reports are shared internally and with USAID, and oral reports are provided to members of the Steering Committees and the Regional Population Councils. 76 TAKAMOL MID-TERM EVALUATION The quality of the data collected are assured through a variety of validation techniques, based primarily on internal review of the data for consistency and accuracy by field directors, who compare the data with previous reports, prior to forwarding the data to Cairo for data entry. In addition, at least one monthly visit per clinic is conducted by a Takamol technical person or a medical staff member in the field to assist in the replication of data collection on particular indicators, such as No. 16, the conduct of hospital-based client exit interviews that are used to assess satisfaction with the services received. They also validate supervisory skills to ensure that the forms are accurately completed. In one case cited, substantial differences existed in scoring between the assessment of the R1 technical staff person and a coach. Both the coach and his supervisor reviewed the form with the R1 technical staff person. The coach was new to the district team and did not understand the guidelines on completion of the form. One of the most innovative aspects of the M&E system is the continued collection of data in 15% of the PHC units in which Takamol has completed all of its activities, including the CDA activities. These units are purposively selected by Takamol technical staff members, who consider a variety of factors contributing to the sustainability of data collection, especially the anticipated stability of the team composition (rather than their performance), the understanding of data collection at all levels, and their willingness to provide data. Thus, Takamol staff members have an accurate picture of the extent to which their efforts are being sustained even after they phase out from PHC units and hospitals in order to assist new ones. (Please see Annex 5 for analysis of the data collected from the 15% of PHC units.) ARE DATA GATHERING METHODS REASONABLE FOR MONITORING PROGRESS AND INDICATORS? IF NOT, EXPLAIN AND SUGGEST POSSIBLE MODIFICATIONS. The indicators selected are specific in that they provide a clear description of what should be measured; some are neutral and objective, but others suggest reasonable targets for achievement; and they are measurable quantitatively (expressed in numbers or percentages). The indicators do not document everything in the program but only the primary activities. They are reasonable for monitoring progress and indicators. 1. For most of the indicators, the expected end-of-project goal is 90% of the total PHC anticipated units (e.g., 180 PHC units rather than 200; 22 hospitals rather than 25). There are two exceptions to this rule: With regard to activation of boards, 75% of the 200 are proposed to have elections, meet the regularly, and have actions plans (Indicator 22). With regard to the SIF, 60% of 200 units are projected to spend from the SIF, at least 10% of the fund (Indicator 23). This change was predicated on challenges involved in changing people’s behavior, which cannot be guaranteed by project staff. 2. The only target changed was in Indicator 1 (modern contraceptive methods prevalence rate). Originally the target selected was a percentage increase of 10%. However, Fatma El-Zanaty, the Egypt DHS survey director, who serves as Takamol’s technical advisor, advised Takamol to lower the target to > = 2–5%, which is a more reasonable target to achieve for increased modern contraceptive prevalence given the current mix of methods in Egypt. The mix is characterized by limited surgical sterilization for cultural and religious reasons (1% of current users among MWRA aged 15–49, according to the 2008 EDHS), limitations on the importation of Implanon implants, resulting in 0.5% current use, and limitations on counseling related to postpartum IUD use (Ministry of Health and Population and El-Zanaty and Associates, 2008). In addition, contraceptive discontinuation needs to be considered. Among MWRA whose ideal number of children is two but who have three living children, the main explanation given for the discrepancy (40%) is that their last pregnancy was a ―mistake‖ or an unplanned birth. Because the contraceptive failure rate in Egypt is only 3% per year (El-Zanaty and Way, 2001 and 2006), most of these ―mistakes‖ are the result of nonuse—or discontinuation—of family planning. One-fifth of the women explained the discrepancy between their ideal and actual number of children by their desire to have an extra child of a specific sex. Thus, a 10% increase is unlikely to be achieved over the life of the project. (See El-Zeini, 2008, for additional data on the characteristics of groups most likely to resist the notion of a two-child family.) TAKAMOL MID-TERM EVALUATION 77 3. Indicator 21 (Number of facilities/communities where MOHP/local authorities have replicated project interventions) has a very low LOP target. The denominator for Indicator 20 (percent of MOHP [FP/MCH] staff) is based on Takamol staff estimates of the number of MOHP staff members they feel they will be able to train, rather than official personnel records. 4. To date, 20 pre-intervention surveys and six post-intervention surveys have been completed. Among these, six communities have both pre- and post- intervention surveys (a set) and for 14 communities, only the pre-intervention survey has been completed. These surveys are conducted with a random sample of 200 households with an MWRA six months prior to the Takamol intervention and six months following. One challenge encountered has been the need to receive security clearance from the authorities to conduct the surveys. Refusal to permit a few surveys in Giza, Beni Suef, and Qena has led Takamol to increase the total number of surveys planned from 28 to 40. Disapproval was conveyed verbally, and no explanations were offered. Data on 64 items are collected in the survey, some of which are incorporated into Indicators 1, 3, 4, 10, 11, and 18b). ARE PMP INDICATORS APPROPRIATE AND/OR VALID? IF NOT, EXPLAIN AND SUGGEST POSSIBLE MODIFICATIONS. The data on which the indicators are calculated are exemplary in completeness and accuracy and are therefore valid and reliable. For instance, Indicator 8 (number of PHC units where the average client satisfaction score is > = 80%) is based on 50 interviews per month (150 per quarter) gathered by 2 RRs (1 interview per day for 25 working days per RR). The RRs report no difficulty in gathering the data or client refusal to participate in exit interviews. The data provided show the extent to which factors related to increased demand for services can actually produce more crowding, longer waiting time, inability to have all questions answered, or inability to obtain needed drugs during the visit, and hence a decline in client satisfaction, which can be brought to the attention of the PHC staff. Another particularly useful summary indicator is Indicator 18a (percent improvement in community participants’ knowledge of key MCH/FP/RH messages), which is based on comparisons of pre-/post￾tests for all community mobilization trainings and events, and Indicator 18b (percent improvement in community participants’ knowledge of key MCH/FP/RH messages), based on the household survey results, conducted so far in six communities (both pre- and post-intervention surveys). The collection of data from two sources allows for triangulation of data to ensure reliability of the data collected. Similarly, data collected in Indicator 3 (percent of mothers with children less than 12 months who reported making four or more ANC visits to the PHC during their last pregnancy) can be compared to the data reported for Indicator 6c (number of PHC units that reported 5% or more increase for new ANC users (aged < 30 with up to two children) to see if an increase in the number of new ANC users is related to the number of repeat visits made. Although the PMP indicators are appropriate, many of the definitions and formulas in the PMP are vague, lacking sufficient detail for a newcomer to be able to understand their correct calculation. The team needed to request clarification on the definition or calculation of the following indicators: Nos. 1, 2, 4, 5, 8, 12, 13, 15, 17, 20, 21, and 22. Questions were discussed in detail with the M&E specialist and need not be repeated. No printed manual exists with suggestions on appropriate individuals to assist in data collection,7 clear definitions, and instructions on calculation (including the formulas and the calculation of scores based on the data in the quality checklists). Rather, direct assistance is given by the field staff on how to complete the forms. However, in the hopes that some of these indicators will eventually be adopted by the MOHP, it is necessary to have an accurate and revised PMP document with clear 7 Individuals responsible for data collection at the PHC level include: service statistics (PHC head nurse), SIF records (district finance officer), client satisfaction exit interviews (RRs), and quality checklist (district supervisors); at the Hospital level: service statistics (hospital MIS), SIF record (finance officer), client satisfaction exit interviews (nurse or social worker, who belong to the hospital’s quality committee), and the quality checklist (hospital SMC). Takamol field office staff members collect the completed forms during their visits to the intervention facilities. 78 TAKAMOL MID-TERM EVALUATION definitions, information on the calculations, including the formulas, and the frequency of data collection vs. reporting. This is especially important since Takamol will be ending in two years, and some of the Takamol indicators may eventually be incorporated into the MOHP M&E system. 8 Here are a few suggestions on how to improve understanding of Indicator 2 (number of PHC units achieving 3% improvement in couple-years of protection). This is a particularly important indicator, since the recent decline in Egyptian fertility has been achieved through increasing contraceptive use, and reportedly policymakers do not understand the meaning of couple years of protection (CYP). A simple illustration needs to be provided on how the indicator has been calculated in order to ensure understanding of the indicator among various audiences. It is typically defined as the estimated protection provided by contraceptive methods for one couple during a one-year period, based on the volume of all contraceptives sold or distributed free of charge to clients during that period. The CYP is calculated by multiplying the quantity of each method distributed to clients by a conversion factor, which yields an estimate of the duration of contraceptive protection provided per unit of that method. The CYP for each method is then summed for all methods to obtain a total CYP figure. CYP conversion factors are based on how a method is used, failure rates, wastage, and how many units of the method are typically needed to provide one year of contraceptive protection for a couple. The calculation takes into account that some methods, like condoms and oral contraceptives, for example, may be used incorrectly and then discarded, or that IUDs and implants may be removed before their life span is realized. Since the CYP calculation reflects distribution, it is a way to estimate coverage and not actual use. It provides an immediate indication of the volume of program activity and can also allow programs to compare the contraceptive coverage provided by different family planning methods. The confusion on the part of those who examine the data may result in part because of the way in which the conversion factors are expressed in terms of fractions in the PMP. Below are the USAID conversion factors with the Egyptian conversion factors in parentheses. Oral contraceptives 15 cycles per CYP (Egyptian 1/13) Condoms 120 units per CYP (Egyptian 1/100) Vaginal foaming tablets 120 units per CYP (Egyptian 1/100) Depo-Provera® contraceptive injection Medroxyprogesterone acetate injectable suspension USP (distributed by Pfizer) 4 doses (ml) per CYP (Egyptian ¼ or 1/6 based on the brand) Noristerat injectable 6 doses per CYP Copper-T 380-A IUD 3.5 CYP per IUD inserted (Egyptian 3.2) Implanon implant 2.0 CYP per implant (Egyptian 2.0) 8 The MOHP has the technical capacity to collect data and calculate a wide range of indicators. The data collection system starts with logbooks that are manually completed at the PHC and hospital level. Monthly reports are submitted to the district-level information office, where the data are entered into an ACCESS database. Electronic reports are then submitted to the governorate-level information office, which produces summary reports that are then submitted to the MOHP central-level Health Information System. Although a current list of MOHP indicators was not available to the evaluation team, the data posted at the district-level office in Sharkia overlapped with some of the Takamol indicators. This similarity reflects the fact that Takamol reviewed the MOHP record data being collected that were relevant for calculation of its own indicators. The MOHP does not collect exit interviews, quality of service data, SIF, or board activation information; nor does it conduct pre- and post-intervention household surveys for the villages in which it is replicating the Takamol model. TAKAMOL MID-TERM EVALUATION 79 Since the data are provided by PHC units, no mention should be made in the calculation of vaginal foaming tablets, since tablets are only available through private sources, and only one conversion factor is needed for the one brand of injectable available in the PHC, that is, the Depo-Provera®. A second area of confusion relates to the time period to which the data refer. In the PMP it was noted that the data are collected twice (pre- and post- evaluations comparing six months pre- and six months post￾intervention). However, baseline data on the volume of all contraceptives sold or distributed by PHCs to clients are collected for one full year prior to the intervention. When the indicator is calculated after six months of operation, the value is compared to the same six-month period before the intervention to eliminate the effect of seasonality. The total CYPs are calculated for the entire six-month period by multiplying the contraceptive methods dispensed by their respective CYPs. Then the resulting value is compared to the value of CYPs for the same period pre-intervention. HAS THE PROJECT DONE AN ADEQUATE JOB IN DOCUMENTING MODELS, ACTIVITIES, CHALLENGES/WEAKNESSES, AND SUCCESSES/ACHIEVEMENTS? The Takamol PMP Plan provides accurate, reliable, and valid data at all levels of the program through the collection of a common set of measures and documents the program’s activities, development, implementation, and outcomes. In reviewing the data with the M&E specialist and his assistant, it seemed highly likely that all of the goals projected for the project will be achieved, except for two indicators: Indicator 2 (number of PHC units achieving 3% improvement in couple years of protection) and Indicator 5 (average number of postpartum care home visits per delivered woman). These two indicators will be challenging to achieve. Indicator 2 reflects the behavior change that the project seeks to create. Although increasing contraceptive prevalence has historically been the most important proximate determinant of fertility in Egypt, the fertility decline in Egypt has recently stalled. The total fertility rate was 3.3 in 1995–1997, 3.4 in 1996– 1998, and 3.5 in 1997–2000. It then resumed its decline to 3.2 in 2000–2003 based on the Interim DHS and has continued to decline to a low of 3.0 in 2005–2008 (based on the 2008 EDHS, Table 6) (Ministry of Health and Population and El-Zanaty and Associates, 2008). The decline is due to a drop in rural fertility; urban fertility has remained stable during the period. Upper Egypt is the only region in which the fertility rate decreased (El-Zanaty, 2009). This stall is attributable to a number of factors. The contraceptive mix available to Egyptian women is more limited, since sterilization for medical reasons and relatively few Implanon implants are in use, as previously discussed. Indicator 5 will also be more difficult to achieve in Lower Egypt. One of the major challenges encountered while working with PHC clinics is the proper activation of postpartum home visits. These visits can directly reduce maternal mortality rates through early identification of postpartum problems and education of new mothers and their families about important postpartum care issues. The challenge to the activation of postpartum visits is the result of a variety of factors: a shortage of trained nurses in many Upper Egypt PHC facilities; the burden of conducting home visits experienced by nurses, who already carry heavy workloads and cannot complete responsibilities at PHC clinics; and distance between homes and clinics and transportation costs, which nurses assume. However, progress has been made by providing feedback to FP and MCH directors at both the directorate and district levels; they previously agreed to encourage nurses to follow through on visits by stressing their importance. Solutions have been found, such as assigning nurses to visit homes close to their residences and using SIF funds to cover transportation costs. 1. Takamol staff members have used the Takamol indicators to a) Generate quarterly and annual reports on program activities and accomplishments for USAID and oral presentations for the MOHP. Quarterly and annual reports have highlighted the challenges/weaknesses demonstrated by the indicator data, which are discussed in a separate section. 80 TAKAMOL MID-TERM EVALUATION b) Inform MOHP decision makers on national program progress by presenting the national program data at steering committee meetings, highlighting achievements to date. c) Inform members of the Regional Population Council in Luxor on governorate-level progress by presenting the governorate-level data at meetings. d) Prepare written reports to those who request them. e) Prepare monthly reports for the results teams for internal review and discussion. f) Highlight the progress achieved to date, identify gaps, and communicate achievements and gaps to field directors, who in turn update program staff at the PHC and hospital level. Examples seen include reports on client satisfaction with PHC services. g) Highlight important program achievements. One notable example is the increase in the proportion of women who have received more than one service during a single visit. h) Communicate the results of the household surveys to the members of the Community Development Associations (CDAs), who in turn post the results in meeting places. i) Use the data to generate recommendations to improve future performance and then monitor the data to see if improvements are forthcoming. One notable example is the ability of communities in Lower Egypt to achieve the recommended number of PNC visits. j) Incorporate the data into communication and marketing strategies for partners, MOHP divisions, funders, and potential corporate donors. For instance, the staff responsible for Corporate Social Responsibility introduces the Takamol project in various forums to attract multiple partners predominantly from the NGO community and local business networks. They provide information to interested donors based on the Takamol menu of activities as they relate to their corporate areas of interest, relying largely on assessment results rather than detailed M&E data. They also have an update mechanism for donors to ensure that they are aware of the situation relating to their intervention, and sometimes donors request site visits. They are interested in qualitative rather than quantitative data. 2. Staff at the primary, hospital, district, and central level have used the Takamol indicators to a) Compare the expected program outputs to actual accomplishments; identify gaps; and generate recommendations to improve the program. b) Facilitate future program planning. c) Seek financial support for additional resources to improve program efficiency or expand the program’s scope. 3. At present, the USAID Mission plans on conducting the next Service Provision Assessment surveys in the catchment areas in which Takamol has focused its efforts in order to assess the impact of all of the project’s activities. ADDITIONAL TABULATIONS TO ASSESS THE IMPACT OF CDA ACTIVITIES ON ATTITUDES AND BEHAVIOR Outcomes can be considered in three time blocks, as short-term, intermediate, and long-term. Short-term outcomes are the immediate effects (over one to three years) of a program and focus on changes in knowledge, attitudes, beliefs, awareness, or skills gained by community residents. Intermediate outcomes are the effects that are achieved within three to five years of program initiation and often include changes in behavior, norms, or policies. Long-term outcomes are effects that are achieved within 4–6 years of program initiation and include changes in organization and systems or maintenance or a sustained improvement in desired behaviors, such as improved health outcomes. TAKAMOL MID-TERM EVALUATION 81 Takamol regularly provides data on 23 indicators, as well as four indicators on a sample of PHC units, which represent 21–22% of the PHC units in Batches 1 and 2. Given the breadth of information collected in the household surveys conducted before and following the Takamol intervention, a variety of short￾term outcomes based on the household survey data can also be analyzed through tabulations to provide stronger empirical evidence on the impact of community mobilization as the demand component of the Takamol model. The household surveys before and subsequent to the Takamol intervention query respondents on their previous participation in development programs (such as illiteracy eradication or literacy education, microenterprise loans, handicraft training, and awareness seminars). Given the small sample of women who are queried in each survey, it would be best to compare the attitudes and behavior of those who have participated in any of the activities, in comparison to those who have not at two points in time, before and following the intervention. Data are also gathered on whether MWRA age received information about MCH/FP/RH while participating in CDA activities or referrals for FP, ANC, or delivery during that period, which Takamol has already provided. However, if cell size permits, the pre- and post-intervention household survey data can be analyzed to note other changes at the community level, such as improvements in couple communication; support from the husband in health-seeking behavior; and changes in the women’s behavior related to use of contraception or visits to the PHC unit among those who have and have not participated in CDA activities, by educational attainment at the time of the pre-survey. 82 TAKAMOL MID-TERM EVALUATION TAKAMOL MID-TERM EVALUATION 83 ANNEX 5. THE MONITORING AND EVALUATION MATRIX: RELEVANT OUTPUT AND OUTCOME INDICATORS FROM TAKAMOL’S PMP DURING FY09/Q1 The data provided are based on tabulations provided by the M&E specialist for the evaluation team’s review, rather than the data in the Quarterly Report for FY09/Quarter 1 submitted to USAID. Indicator Progress to Date Target as of end of FY09 Life-of-project target 1. Modern contraceptive methods prevalence rate (CPR) See below 2. Number of PHC units achieving 3% increase in couple years of protection (CYP) 71 of 98 PHC units 117 PHC units 179 PHC units 3. Percent of mothers with children less than 12 months who reported making four or more ANC visits to the PHC unit during their last pregnancy See below 4. Percent of births attended by a skilled provider See below 5. Number of PHC units where the average number of postpartum home visits per woman is greater than or equal to 3 73 of 98 PHC units 117 PHC units 179 PHC units 6a. Number of PHC units that reported 5% or more increase in total caseload 95 of 98 PHC units 117 PHC units 179 PHC units 6b. Number of PHC units that reported 5% or more increase for new FP users (age <30 years with parity <= two children) 88 of 98 PHC units 117 PHC units 179 PHC units 6c. Number of PHC units that reported 5% or more increase for new ANC users (age < 30 and parity <= to two children) 75 of 98 PHC units 117 PHC units 179 PHC units 7. Number of PHC units with quality score of health services > = 80% 95 of 99 PHC units 118 PHC units 180 PHC units 8. Number of PHC units where the average client satisfaction score is > = 80% 84 of 99 PHC units 118 PHC units 180 PHC units 9. Number of PHC units where at least 40% of the clients received two or more MCH/FP/RH services during the same visit 98 of 99 PHC units 118 PHC units 180 PHC units 10. Percent of MWRA that were visited at home in the last six months by an outreach worker See below 11. Percent of MWRA who were referred to a PHC unit by an outreach worker See below 84 TAKAMOL MID-TERM EVALUATION Indicator Progress to Date Target as of end of FY09 Life-of-project target 12. Number of quarterly governorate SMC meetings conducted to review MMSS data and develop self￾improvement plans 18 meetings 25 meetings 40 meetings 13. Number of hospitals with an average quality score of EOC, neonatal care, postpartum/ postabortion care, and FP/RH services > = 80% 10 of 10 hospitals 19 hospitals 22 hospitals 14. Number of hospitals where at least 80% of the postpartum/postabortion clients received family planning counseling 10 of 10 hospitals 19 hospitals 22 hospitals 15. Number of hospitals where the percent of neonatal care unit (NCU) surviving cases increased by > = 1.2% 9 of 10 hospitals 19 hospitals 22 hospitals 16. Number of hospitals where the average client satisfaction score is > = 80% 10 of 10 hospitals 19 hospitals 22 hospitals 17. Number of PHC units that have an active referral system 76 of 91 PHC units 110 PHC units 169 PHC units 18a. Percent improvement in community participants‘ knowledge of key MCH/FP/RH messages 71% for community members trainings/ 92% for plays) > = 20% (pre- /post- tests for community mobilization training/ events) > = 20% (pre-/post- tests for community mobilization training/ events) 18b. Percent improvement in community participants‘ knowledge of key MCH/FP/RH messages: household survey results See below 19. Resources leveraged (cash and in￾kind) through the establishment of corporate/NGO/public sector and individual partnerships supporting health services, as a proportion of the operational expenses of the project (USD) US $3,371,088 (10% of total life-of-project operational expenses) 8% of total LOP operational expenditures 11% of total LOP operational expenditures 20. Percent of MOHP staff at all levels who received capacity building training by the project 74% 60% 80% 21. Number of facilities/ communities where MOHP/local authorities have replicated project interventions 56 (33 PHC units & 23 communities) & 4 hospitals 3 PHC units/ communities & 1 hospital 9 PHC units/ communities & 3 hospitals 22. Number of PHC unit and hospital management boards established/ reactivated that meet regularly and work in support of MCH/FP/RH quality 91 of 98 PHC units & 9 of 10 hospitals (at least one 98 PHC units and 19 hospital boards 150 PHC units and 22 hospital boards TAKAMOL MID-TERM EVALUATION 85 Indicator Progress to Date Target as of end of FY09 Life-of-project target and integration in project intervention areas meeting per quarter for 2 or more quarters) 23. Number of health facilities that utilize at least 10% of SIF revenues to pay for facility improvement 86 of 98 PHC units and 8 of 10 hospitals 78 PHC units and 19 hospitals 120 PHC units and 22 hospitals Indicator 1. Modern contraceptive methods prevalence rate (CPR) Community Pre- Post- Value Target El Mandura 59% 73% 24% % increase > = El Debej 59% 60% 2% to 2-5% El Odaysat 53% 61% 15% El Boairat 68% 84% 24% El Heish 74% 78% 5% Monshaat El Abassa 56% 58% 4% Indicator 3. Percent of mothers with children less than 12 months who reported making four or more ANC visits to the PHC unit during their last pregnancy Community Pre- Post- Value Target El Mandura 57% 81% 42% % increase El Debej 0% 90% N/A (Pre=0) > = 7% El Odaysat 0% 90% N/A (Pre=0) El Boairat 89% 100% 12% El Heish 24% 49% 104% Monshaat El Abassa 20% 41% 105% Indicator 4. Percent of births attended by a skilled provider Community Pre- Post- Value Target El Mandura 66% 95% 44% % increase >= 2.8% El Debej 10% 93% 833% El Odaysat 98% 100% 3% El Boairat 100% 100% 0% El Heish 100% 100% 0% Monshaat El Abassa 82% 100% 22% 86 TAKAMOL MID-TERM EVALUATION Indicator 10. Percent of MWRA that were visited at home in the last six months by an outreach worker Community Pre- Post- Target El Mandura 52% 51% Visit > = 70% of the MWRA in the six months prior to the survey El Debej 26% 93% El Odaysat 55% 100% El Boairat 59% 93% El Heish 56% 87% Monshaat El Abassa 55% 88% Indicator 11. Percent of MWRA who were referred to a PHC unit by an outreach worker Community Pre- Post- Value Target El Mandura 98% 78% -20% % increase > = 20% El Debej 92% 96% 4% El Odaysat 5% 79% 1480% El Boairat 39% 46% 18% El Heish 51% 94% 84% Monshaat El Abassa 68% 86% 26% Indicator 18b. Percent improvement in community participants’ knowledge of key MCH/FP/RH messages: Household survey results Community Pre- Post- Value Target El Mandura 44% 75% 70% % increase > = 10% El Debej 35% 79% 126% El Odaysat 40% 68% 70% El Boairat 61% 78% 28% El Heish 57% 80% 40% Monshaat El Abassa 46% 80% 74% ANALYSIS OF THE INDICATORS COLLECTED BY TAKAMOL ON A FOLLOW-UP SAMPLE OF PHC UNITS IN BATCH 1 AND 2 Review of the Indicator Data The tables below provide information on the definitions of the indicators that Takamol collects on selected PHC units in Batch 1 and Batch 2. The units in the sample were selected by different technical staff and represent facilities with average to good performance and relative stability in physicians and district teams. These two factors contribute greatly to the sustainability of program achievements initiated by the Takamol project. Accordingly, seven clinics were selected from Batch 1 (n = 32) and four from Batch 2 (n = 19). These PHC units actually represent 21–22% of the PHC units included in each batch. TAKAMOL MID-TERM EVALUATION 87 Since retention of physicians cannot be guaranteed over time, Takamol increased the size of the sample somewhat, above the 15% originally suggested, to get a clearer understanding of the situation. In the tables, PHC units that failed to achieve the target are highlighted in gray, and brief explanations for the failure are discussed. Data are collected on four indicators (Nos. 2, 6a, 7, and 8) and reported separately to USAID along with the 23 M&E indicators discussed above. Indicator 2. PHC units achieving 3% increase in couple years of protection (CYP) This is measured by multiplying the quantity of each FP method distributed to clients by a conversion factor. The factor reflects the protection period each unit of the method provides for the couple. Accordingly, the IUD has the highest couple years of protection. The CYPs for each method are then summed over all methods to obtain a total CYP figure. The percent increase is calculated by comparing the value of CYP during a specified period after phase-out to the value for the same time period before the intervention. Increase in CYP Among Batch 1 Clinics Increase in CYP Among Batch 2 Clinics PHC Unit (n = 7) Average Percent Increase in CYP over 1 Year (4 Quarters After Phase-out) Name of PHC unit (n = 4) Average Percent Increase in CYP over 6 Months (2 Quarters After Phase-out) Dabaiaa PHC 8% El Shabanat PHC 30% El Heish PHC 31% Basateen El Ismailia PHC –16% Kafret Nassar PHC –2% Dalas Health Group 127% Shenbab PHC 6% Kafr Nasser PHC 60% Meet Rahina Health Group 42% Monshaat El Hag PHC 55% Ebshna Health Group 67% Nearly all the PHC units showed a significant increase in the total caseload, especially Meet Rahina Health Group (257%) and Monshaat El Hag PHC unit (167%), both of which experienced an increase in number of outpatient clients compared to total caseload before the intervention. The only PHC unit that failed to achieve the target was the El Heish PHC. The physician on staff changed; families were not happy with his performance, and the clinic caseload was adversely affected. As expected, the percent increase in total caseload ranged from 3% to 257% among Batch 1 clinics and was somewhat less, 8% to 108% among Batch 2 clinics. 88 TAKAMOL MID-TERM EVALUATION Indicator 7. PHC units with quality score of health services > = 80% This is measured by calculating the total score for quality for all services provided, such as antenatal care, family planning, childcare, home visits, and infection control, in addition to the score of administrative performance. Quality Score of Health Services Provided Among Batch 1 Clinics Quality Score of Health Services Provided Among Batch 2 Clinics PHC Unit Name (n = 7) Average Quality Score over 1 Year (4 Quarters After Phase-out) PHC Unit Name (n = 4) Average Quality Score over 6 Months (2 Quarters After Phase-out) Dabaiaa PHC 91% El Shabanat PHC 95% El Heish PHC 97% Basateen El Ismailia PHC 98% Kafret Nassar PHC 90% Dalas Health Group 83% Shenbab PHC 92% Kafr Nasser PHC 90% Meet Rahina Health Group 91% Monshaat El Hag PHC 86% Ebshna Health Group 89% All of the PHC units achieved the Takamol target of 80% in their score on quality of all health services provided. The average score ranged from 86% to 97% among the Batch 1 clinics, and 83% to 98% among those in Batch 2, which shows that client perceptions of the quality of care they received remained high even after Takamol phased out its involvement with the PHC units. Indicator 8. PHC units with average client satisfaction score of > = 80% Measured through exit interviews where clients are asked about their satisfaction with clinic cleanliness, waiting time, privacy, and level of interaction with service providers. The final calculation is the total satisfaction scores for all interviewees divided by the number of clients interviewed in each PHC unit (50 per month). Client Satisfaction Score Among Batch 1 Clinics Client Satisfaction Score Among Batch 2 Clinics PHC Unit Name (n = 7) Average client Satisfaction Score over 1 Year (4 Quarters After Phase-out) PHC Unit Name (n = 4) Average client Satisfaction Score over 6 Months (2 Quarters After Phase-out) Dabaiaa PHC 76% El Shabanat PHC 72% El Heish PHC 88% Basateen El Ismailia PHC 91% Kafret Nassar PHC 71% Dalas Health Group 79% Shenbab PHC 78% Kafr Nasser PHC 95% Meet Rahina Health Group 74% Monshaat El Hag PHC 77% Ebshna Health Group 90% TAKAMOL MID-TERM EVALUATION 89 Only four of 11 PHC units achieved the target of greater than or equal to 80% in client satisfaction scores after phase-out. Only 29% of the seven clinics in Batch 1 achieved this target, compared to 50% among the four Batch 2 clinics. Although not shown in tabular form, the primary sources of dissatisfaction reported by patients in the seven units that failed to achieve the target were long waiting times and limited interaction with service providers. What is striking is that the patient caseload in most of these clinics also increased dramatically and is undoubtedly responsible for the decline in satisfaction voiced by clients, as shown in Indicator 6a. Only the El Heish PHC and the Dalas Health Group experienced very modest increases in caseload. El Heish surpassed the targeted client satisfaction score (88% average client satisfaction score), and Dalas Health Group was extremely close in achieving the target, with an average client satisfaction score of 79%. Conclusions Reviewing the data for all four indicators, it appears that the improvements introduced by the Takamol project are being maintained even after phase-out. The PHC units for the most part continue to reach the targeted goals for caseload, quality scores in service, and distribution of services. Those PHC units that have experienced the greatest percent increase in caseload, six months or one year following Takamol phase-out, experience declines in client satisfaction scores due to longer waiting times and less time spent with providers. Improved staffing levels or reduction in physician turnover would undoubtedly improve the level of satisfaction among PHC clients. 90 TAKAMOL MID-TERM EVALUATION For more information, please visit http://www.ghtechproject.com/resources.aspx Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 www.ghtechproject.com