SUDAN HEALTH TRANSFORMATION PROJECT PHASE II (SHTP-II) MID-TERM EVALUATION REPORT – PUBLIC DOCUMENT JANUARY 20, 2011 This publication was produced for review by Andrew Kantner, Deborah Armbruster, Mary Harvey, Geertruid Kortmann, and Anna McCrerey. It was prepared by Management Systems International. Sudan Health Transformation Project Phase II (SHTP-II) ii Mid-Term Evaluation Report – Public Document SUDAN HEALTH TRANSFORMATION PROJECT PHASE II (SHTP-II) MID-TERM EVALUATION REPORT – PUBLIC DOCUMENT Management Systems International Corporate Offices 600 Water Street, SW Washington, DC 20024 Contract No. DFD 1-00-05-00251-00, Task Order No. 2 Services Under Program and Project Offices for Results Tracking (SUPPORT) DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Sudan Health Transformation Project Phase II (SHTP-II) iii Mid-Term Evaluation Report – Public Document ACKNOWLEDGEMENTS The Evaluation Team would like to thank all individuals who participated in the SHTP-II Mid-Term Evaluation conducted between October 16 and November 19, 2010. We are especially indebted to Management Sciences for Health (MSH) project staff in Juba and subcontracting partner head office and field staff who were exceptionally generous with their time. Their patience in responding to our incessant questioning and imperfectly formed impressions of the evidence gathered was appreciated. Many of the findings and conclusions from this evaluation came from interviews with clinic-based service providers and support staff, Village Health Committees, and the occasional Home Health Promoter and Traditional Birth Attendant. We truly appreciate to their good natured and tireless efforts to brief us on their daily activities, operational constraints, and hopes for the future. The Team greatly benefited from discussions with Dr. Samson Baba, Director General for External Assistance and Coordination at the Ministry of Health. Our discussions with Dr. Baba occurred at the start, mid-point, and conclusion of the evaluation and helped immensely in sorting through the history and operational complexity of SHTP-I and II and gaining greater insight into the health conditions and implementation environments of Southern Sudan. Our Team also very much enjoyed our collaboration with Ms. Rebecca Alum William from the Ministry of Health. Ms. William participated in our field trip to Tonj South and offered useful insights into the functioning of Southern Sudan’s primary health care system and current communication and outreach strategies for working with local community organizations. The SHTP-II Mid-Term Evaluation was organized by the Office of Health at USAID/Sudan. The evaluation could not have been completed without the Mission’s substantive guidance and management support for our endeavor. The Team is especially grateful to Charles Lerman, Cliff Lubitz, Martin Swaka, Kawa Tong, and Anna Williams for their encouragement and support throughout the evaluation. The participation of many USAID staff in our field trips was especially welcome. These joint trips underscored the value of spending more time in the field observing firsthand what SHTP-II is accomplishing and how its efforts are judged by its potential beneficiaries. Finally, we would like to thank Ingrid Orvedal, Ami Henson, Emmanuel Taban, Salle Emmanuel Mike, Paul Temple, and Stefanie Frease from MSI SUPPORT for their logistical and technical support for the evaluation. Ingrid Orvedal should be singled out for her dedication to the evaluation effort and admirable attention to detail that kept everyone focused on what needed to be done at every waking hour. Emmanuel Taban should also be cited for his participation in two of the team’s field trips which afforded Kantner and Armbruster an opportunity to access his considerable knowledge of the SHTP-II project and his familiarity with subcontracting partner staff and field environments. Sudan Health Transformation Project Phase II (SHTP-II) iv Mid-Term Evaluation Report – Public Document TABLE OF CONTENTS Acronyms and Abbreviations........................................................................................6 Executive Summary.......................................................................................................9 I. Introduction and Project Description......................................................................18 II. Southern Sudan Health Sector Context ................................................................20 III. Evaluation Purpose and Methodology ...................................................................23 IV. Findings and Conclusions.......................................................................................25 V. Key Conclusions on Mid-Term Implementation....................................................40 VI. Observations on SHTP-II Project Design..............................................................45 VII. Recommendations ................................................................................................47 Annex 1: Scope of Work for the Mid-Term Evaluation of SHTP-II...........................52 Annex 2: Final Contractual Performance Indicators and Targets.............................58 Annex 3: Summary Matrix for Evaluation Team Field Observations........................64 Annex 4: Key Documents Reviewed ...........................................................................76 Annex 5: Contact List of Key Informants ...................................................................78 Annex 6: SHTP-II Evaluation Work Plan....................................................................80 Annex 7: Interview Guides for Evaluation ..................................................................81 Sudan Health Transformation Project Phase II (SHTP-II) 5 Mid-Term Evaluation Report – Public Document MAP OF SOUTHERN SUDAN (SOURCE: UNITED NATIONS OFFICE FOR THE COORDINATION OF HUMANITARIAN AFFAIRS) Sudan Health Transformation Project Phase II (SHTP-II) 6 Mid-Term Evaluation Report – Public Document ACRONYMS AND ABBREVIATIONS AAHI Action Africa Help International AB Abstinence and Be Faithful ABC Abstinence, Be Faithful, and Correct and Consistent Condom Use ACT Artemisinin Combination Therapy ADRA Adventist Development and Relief Association ANC Antenatal Care ARI Acute Respiratory Infections BASICS Basic Support for Institutionalizing Child Survival BCC Behavior Change Communication BPHS Basic Package of Health Services BRIDGE Building Responsibility for the Delivery of Government Services BSF Basic Services Fund CBO Community Based Organization CCM Comitato Collaborazione Medica CDC United States Centers for Disease Control and Prevention CHD County Health Department CO Contracting Officer COP Chief of Party COP Country Operational Plan CPA Comprehensive Peace Agreement CYP Couple Years of Protection DPA Darfur Peace Agreement DPT3 Diphtheria/Pertussis/Tetanus (three doses) EmOC Emergency Obstetric Care EPI Expanded Program on Immunization FANTA-2 Food and Nutrition Technical Assistance FAR Foreign Assistance Regulations FBO Faith-Based Organization FFSDP Fully Functional Service Delivery Point FP Family Planning FP/RH Family Planning and Reproductive Health GAVI The Global Alliance for Vaccines and Immunization GFATM Global Fund to Fight AIDS, Tuberculosis and Malaria GOSS Government of Southern Sudan HEAR Health, Education, and Reconciliation Creative Associates HHP Home Health Promoters HIV/AIDS Human Immunodeficiency Virus/Acquired Immune Deficiency Syndrome IEC Information-Education-Communication IMR Infant Mortality Rate IPT Intermittent Preventive Treatment IMC International Medical Corps IQC Indefinite Quantity Contract IRC International Relief Committee ITN Insecticide Treated Net JICA Japan International Cooperation Agency Sudan Health Transformation Project Phase II (SHTP-II) 7 Mid-Term Evaluation Report – Public Document JDF Joint Donor Fund JSI John Snow International KAP Knowledge, Attitude, and Practice LAM Lactation Amenorrhea Method LLITNs Long-Lasting Insecticide Treated Nets LMS Leadership, Management, and Sustainability M&E Monitoring and Evaluation MCH Maternal and Child Health MCHW Maternal and Child Health Worker MDTF Multi-Donor Trust Fund MIS Management Information System MMR Maternal Mortality Ratio MOH/GOSS Ministry of Health, Government of Southern Sudan MRDA Mundri Relief and Development Association MSI Management Systems International MSF Médecins Sans Frontières MTE Mid-Term Evaluation NGO Non-Governmental Organization NPA Norwegian People’s AID OFDA Office of Foreign Disaster Assistance ORS Oral Rehydration Solution PBC Performance-based Contracting PEPFAR President’s Emergency Plan for AIDS Relief PHC Primary Healthcare PHCC Primary Healthcare Center PHCU Primary Healthcare Unit PLWHA People Living with HIV/AIDS PMTCT Prevention of Mother-to-Child Transmission PSI Population Services International QA/QI Quality Assurance/Quality Improvement RFTOP Request for Task Order Proposal RH Reproductive Health RTC Regional Training Centers SAVE Save the Children SCP Subcontracting Partner SDP Service Delivery Point SDM Standard Days Method SHHS Sudan Household Health Survey SHTP-I Sudan Health Transformation Project Phase I SHTP-II Sudan Health Transformation Project Phase II SIDF Sudan Inland Development Fund SMOH State Ministries of Health SOW Scope of Work SPS Strengthening Pharmaceutical Systems STI Sexually Transmitted Infection SUPPORT Services Under Program and Project Offices for Results Tracking TA Technical Assistance Sudan Health Transformation Project Phase II (SHTP-II) 8 Mid-Term Evaluation Report – Public Document TASC3 Technical Assistance and Support Contract TBA Traditional Birth Attendant TB-CAP Tuberculosis Control Assistance Program TT Tetanus Toxoid UNDP United Nations Development Program UNFPA United Nations Population Fund UNICEF United Nations Children's Fund USAID United States Agency for International Development USG United States Government VCT Voluntary Counseling and Testing VHC Village Health Committee WASH Water, Sanitation, and Hygiene WFP World Food Program WHO World Health Organization Sudan Health Transformation Project Phase II (SHTP-II) 9 Mid-Term Evaluation Report – Public Document EXECUTIVE SUMMARY Introduction and Project Description The Sudan Health Transformation Project Phase II (SHTP-II) was developed by the United States Agency for International Development (USAID) and the Ministry of Health, Government of Southern Sudan (MOH/GOSS)) to strengthen the decentralization of primary healthcare services and improve the health status of the people of Southern Sudan. The prime contractor for the project is Management Sciences for Health (MSH) and it is implemented in collaboration with ten subcontracting partners (SCPs). The SHTP-II contract was signed between USAID and MSH on February 11, 2009, with the project scheduled to close on February 12, 2012. SHTP-II was designed to strengthen service delivery and community involvement in promoting health awareness and education and undertake initiatives to improve Southern Sudan’s health system. SHTP-II works in ten states and 14 counties (out of a total of 80 counties) across Southern Sudan. The key results established for SHTP-II were to: · Expand access and availability of high impact services; · Increase Southern Sudan's capability to deliver and manage services; and · Increase knowledge of and demand for services and healthy practices. These results were to be achieved by strengthening the provision of seven high impact services: · Child Health – Immunizations (through EPI services) and the diagnosis and treatment of diarrheal disease and acute respiratory infections; · Nutrition – Exclusive breastfeeding, the promotion of infant and young child complimentary feeding, and twice yearly Vitamin A supplementation; · Hygiene and Sanitation Practices – Household level water, sanitation, and hygiene; · Malaria – Prevention control interventions including the use of long-lasting insecticide treated nets (LLITN’s), intermittent preventive treatment (IPT), and the management of active cases using anti-malarial drugs; · Maternal Health – Antenatal, safe delivery, and postnatal care; · Family Planning – Child spacing and family planning information and services; and · Prevention of HIV/AIDS – PMTCT and behavior change to delay sexual debut and reduce multiple risk behaviors. In addition to these seven basic primary care interventions, SHTP II was given the responsibility of improving the management of Southern Sudan’s primary care system by working with County Health Departments (CHDs) to produce strategic plans, improve the supervision of primary care facilities, strengthen forecasting capacities for needed drug supplies and equipment, and develop county health budgets necessary for achieving project objectives. The strengthening of health system “governance” was also identified as an important objective for SHTP-II. These efforts were to be focused at the community level and include establishing and training Village Health Committees (VHCs); training, equipping, and fielding Home Health Promoters (HHPs); and involving community organizations (e.g., women’s, youth, and religious groups) in the cause of promoting improved health and the greater use of primary care facilities. Southern Sudan Health Sector Context Much of what is known about health conditions in Southern Sudan comes from the 2006 Sudan Household Health Survey (SHHS). This survey indicates that Southern Sudan has some of the worst health outcomes in the world (see Table 2). Infant, under 5 child and maternal mortality rates are at levels not seen in most developing countries for over 50 years. As of 2006, Southern Sudan had very low immunization coverage rates for all children aged 12-23 months (17.3%) and poor utilization rates for Vitamin A. Sudan Health Transformation Project Phase II (SHTP-II) 10 Mid-Term Evaluation Report – Public Document Currently, Southern Sudan is developing a health system centered on county-based primary healthcare. The Ministry of Health has developed a Basic Package of Health Services (BPHC) considered to be a cost-effective integrated approach for delivering primary care services. Basic services are provided at county-level Primary Healthcare Units (PHCUs) and Primary Healthcare Centers (PHCCs). Health services and community mobilization efforts are coordinated by County Health Departments (CHDs). There are significant challenges with infrastructure and limited staffing, as well as problems with timely and appropriate remuneration. Evaluation Purpose and Methodology By October, 2010 SHTP-II was slightly more than half-way through its 36-month implementation period. USAID/Sudan decided that a mid-project review was advisable in order to assess achievements and remaining challenges for the project and recommend how project objectives could best be achieved. The evaluation was also tasked with assessing whether the approach of the project was working and what implication this might have for future USAID health programming in Southern Sudan. The Evaluation Team reviewed a wide range of relevant project documents and background materials pertaining to the design and implementation of the project. The Team additionally assessed key quantitative performance indicators compiled by the project’s SCPs and sent quarterly to SHTP II. Qualitative assessments were undertaken through field visits to eight of the 14 counties in which SHTP-II support is being provided; namely Juba, Malakal, Mvolo, Mundri East, Mundri West, Wau, Terekeka, and Tonj South. Key Findings and Conclusions Quantitative Performance Indicators SHTP-II is currently reporting on 15 key indicators. In reviewing the project’s performance data, the most successful outcomes pertain to the percentage of children less than one year of age receiving DPT3, the provision of clean drinking water, and the percentage of children under five who received Vitamin A. However, the number of family planning counseling visits, the percentage of deliveries attended by a skilled birth attendant, and the percentage of all primary care facilities offering five of the seven high impact health services prescribed by BPHS are below target at the mid-point of the project. Serious questions remain about the validity of facility-based data collected by the project as well as the denominators used to estimate eligible populations within counties, so performance results based upon quantitative indicators should be viewed with caution. Implementation of Seven High Impact Services Child Health Child health services observed at the project’s affiliated PHCUs and PHCCs were operational, but still far from well￾established. EPI services were functioning in all counties visited by the Evaluation Team, although BCG and measles vaccines were temporarily stocked-out in most places. Cold chain problems were also encountered in several counties. Mothers observed by the Evaluation Team at PHCUs and PHCCs were bringing their children for routine postnatal care and treatment for various afflictions (mainly diarrhea, acute respiratory infections, and malaria). Oral rehydration salts (ORS) and zinc tablets for treating watery diarrheas were available in most primary care facilities. However, diagnostic and treatment protocols being followed for more serious bloody diarrheas were not always consistently explained or implemented across all sites. Cotrimoxazole for the treatment of pneumonia was found in the primary healthcare facilities visited in four counties, but was less reliably present in other counties visited by the Evaluation Team. Nutrition While quantitative performance indicators report that SHTP-II has achieved its target for Vitamin A distribution at the mid-point of the project, evidence from the field suggested that the supply of Vitamin A capsules was irregular. Exclusive breastfeeding was being actively promoted for up to six months in facilities where community midwives and MCHWs were working. Infant weighing scales for growth monitoring were seen in three counties, although they did not always appear to be in use at these facilities. Water and Sanitation The availability of clean water, the adoption of good personal hygiene and sanitation practices, and the adoption of effective waste management protocols remain significant problems at most facilities supported by SHTP-II. Many sites Sudan Health Transformation Project Phase II (SHTP-II) 11 Mid-Term Evaluation Report – Public Document did not have functional pit latrines with a clean source of water nearby. The disposal of waste (including medical waste) remains a major problem. Primary care facilities in most counties did not have the full package of basic WASH measures; namely latrines, hand washing facilities, access to safe water, and waste pits and incinerators for medical waste. Malaria Malaria is one of the most common health problems being treated at SHTP-II sites and remains one of the most deadly threats to children. Owing largely to high levels of utilization, anti-malarial drugs were often in short supply. The supply of insecticide-treated bednets has been irregular and, when available, they are often handed-out as part of ANC consultations and EPI events. Bednets appear to be widely used and much valued. There is currently little capacity to treat more advanced (complicated) malaria cases at county-level facilities. Maternal Health1 Maternal healthcare is underdeveloped at all project sites compared to child health and malaria control services. There is a severe shortage of well-trained community midwives and maternal and child health workers (MCHWs) capable of ensuring good quality antenatal care, safe delivery, and postnatal care. Safe delivery remains the major problem in addressing maternal health needs. Delivery instruments and supplies (e.g., blood pressure cuffs, delivery tables, infant cribs, and dressing materials) were often not available. Essential drugs for controlling bleeding during delivery (e.g., oxytocin and misoprostol) were totally absent. Most pregnant women in Southern Sudan still rely on traditional birth attendants (TBAs) for delivery care within their communities. Family Planning Family planning services are not adequately developed at most county-level facilities. There has been little systematic information, education, and communication work to promote family planning, which in Southern Sudan focuses on the maternal health benefits to be gained from spacing births through the use of reversible methods. Condoms were usually available at primary care facilities and are easily purchased in local pharmacies. The lactational amenorrhea method (LAM) is being advocated in some primary healthcare centers (as per the contract’s performance standards), but the Evaluation Team found evidence that incorrect information regarding the efficacy of this method (i.e. that it can be used for contraception for up to 24 months, when the accepted time period is six months) is occasionally provided to mothers. HIV Prevention SHTP-II is working to strengthen HIV/AIDS prevention services. The main focus of the project is to develop Prevention of Mother-to-Child Transmission (PMTCT) services in selected counties with counseling and testing services in place. The project is also responsible for supporting community-based activities that improve HIV/AIDS knowledge and behavioral change that reduces the risk of transmission, including messaging that promotes abstinence and being faithful (AB) to one partner.2 Most of the counties visited by the Evaluation Team did not have PMTCT services in place at primary care facilities since HIV testing capabilities were not present. Community-based HIV programs that include AB interventions were not observed in any county visited. Information, Education, and Communication (IEC) materials on HIV and AIDS were available at most primary care facilities, although these materials were usually not available in Arabic. There is currently no SHTP-II strategy to concentrate HIV services in regions with higher prevalence rates and address the needs of most￾at-risk sub-populations. Primary Healthcare System Strengthening Among the PHCUs and PHCCs visited by the Evaluation Team, facility infrastructure was not always completed to a high standard and some facilities have not been completely constructed or adequately renovated. The availability of clean water, hygiene and sanitation facilities, and waste disposal infrastructure were still problems at many sites. Additionally, SCPs were still supporting facilities that often do not contain the full complement of requisite staff. At the PHCU level, it was not uncommon to find only one community health worker (CHW) and no community midwife or maternal and child health worker (MCHW). At PHCCs, clinical medical officers, certified community midwives, and laboratory technicians were not always in place. 1 Maternal Health Services include Antenatal Care (ANC), Delivery Care, and Postnatal Care (PNC) 2 AB is the designation used in the SHTP-II contract signed between MSH and USAID. The more common ABC strategy (abstinence, be faithful, and condom use) was not employed. Sudan Health Transformation Project Phase II (SHTP-II) 12 Mid-Term Evaluation Report – Public Document The payment of salaries for facility-based service providers and support staff continues to be a major problem for Southern Sudan’s health system. It has proven to be a significant budgetary encumbrance for SHTP-II’s prime contractor and affiliated SCPs. A rapid assessment of the salary payment situation in selected SHTP-II counties (organized by USAID and conducted in June 2010) found that 66% of provider salaries were being paid by SCPs while the Government of Southern Sudan paid roughly 34% of all salaries. Government salary payments are sometimes late by as much as one or two months. The timely provision of drugs is still a challenge in Southern Sudan. Because the Ministry of Health still allocates Emergency Drug Kits with multiple drugs to PHCCs and PHCUs, some drugs are over-supplied while others are in chronically short supply. Anti-malarial drugs and some antibiotics tend to be depleted prior to the end of each quarter. Other supplies such as dressing materials and IV saline solutions are widely available and may even be over-supplied. No buffer stocks were being maintained in counties for civil emergencies or unanticipated disease outbreaks. The most constant complaint heard by members of the Evaluation Team concerned the delay in obtaining equipment promised to the SCPs by SHTP II. These supplies included furniture, blood pressure cuffs, delivery tables, drug storage shelving and “stocker” flow labels, IEC materials, testing kits, lab equipment, and vehicles. Much equipment was ordered a year ago, but as of August 2010 only a partial consignment of vehicles had been received. These delays have caused disruption to planned service delivery enhancements and have undermined trust that the SCPs were developing within their project staff and local communities. Patient referral continues to be a major challenge for Southern Sudan’s health system. Making referrals from PHCUs to PHCCs and from PHCCs to the nearest state hospital for Emergency Obstetric Care (EmOC) or the management of complicated malaria cases is often difficult owing to the lack of vehicles or boats, challenging road conditions, and the cost of traveling. However, some SCPs are pilot-testing the use of motorized carts and motorbikes (complete with patient side-beds) to move patients from remote settings to higher levels of care. SCPs are still facing serious problems in providing accurate and timely performance reports. SHTP II has assisted in the development of new clinic-based patient registers and reporting forms and has provided training in their use. Field validation procedures that spot-check the accuracy and completeness of clinic registers and verify the numerical accuracy of monthly and quarterly reporting have been developed. However, they have not yet been widely implemented. The IEC materials seen in the primary healthcare facilities supported by SHTP-II were not always well-designed or prominently displayed. Posters tended to be text heavy and were usually only available in English, a problem in largely Arabic-speaking northern counties. Many posters were about family planning and HIV even though they are among the least developed services at primary care facilities. Additionally, the Evaluation Team also did not see any flip charts or laminated one-page guidelines for use in counseling or IEC materials that could be given directly to patients. Village Health Committees (VHCs) have been formed at most primary care facilities supported by SHTP-II. They tend to concentrate their efforts on facilities (cleaning the grounds, building and maintaining pit latrines, checking bore wells, monitoring drug supplies, etc). Home Health Promoters (HHPs) are still not active in some of the counties visited by the Evaluation Team. The SCPs do not have clear training or activity plans for the HHPs. TBAs were working in all of the SHTP-II counties visited by the Evaluation Team, but they did not usually have much contact or support from SHTP￾II affiliated facilities in their areas. SHTP-II is also attempting to enhance the capacity of Southern Sudan’s health system in the focus counties where the project is engaged. These efforts include logistical and training support to County Health Departments (which recently included Leadership Management Training for County Health Directors), SCP supervisory support for primary care facilities, training of government services providers and facility-based management staff, and capacity strengthening of local SCPs. Management of SHTP-II The project has a centralized management system that concentrates staff in Juba and provides less decentralized support to the SCPs in the field than other donor projects such as the European supported Basic Services Fund (BSF). The management of the project has been problematic and there has been significant staff turnover, including the Chief of Party and several key technical staff. There have also been problems recruiting good local technical staff for the project, as the supply of well-qualified applicants is limited in Southern Sudan. While in the field, the Evaluation Team was told by several SCP partners that they would like to have more guidance from MSH on expectations for implementing the project. During the first year of the project, communication within Sudan Health Transformation Project Phase II (SHTP-II) 13 Mid-Term Evaluation Report – Public Document SHTP II was not considered sufficient for good program management. In particular, the SCPs would welcome more field visits and joint supervisory trips to the primary care facilities and community-based activities in their service areas. In many counties visited by the Evaluation Team, SHTP II staff had not made field visits since April/May 2010. And some counties had only been visited once since the start of the project. Meeting Contractual Performance Standards The SHTP-II contract identifies performance standards to be attained by the end of the project in February, 2012. Since the project is only at the mid-point of its operational life and there remain concerns about the validity of data being generated by the project’s M&E system in some focus counties, it is difficult to pass judgment on the extent to which SHTP-II is on-track to achieve its standards. The findings from this evaluation suggest that the project has made progress in delivering child health services in primary care facilities (EPI, ORS, and ARI) and the prevention and treatment of malaria (despite the many shortcomings that still plague these efforts). On the other hand, maternal health, family planning, and HIV prevention efforts are clearly lagging. It will be very difficult for SHTP-II to achieve most minimum performance standards for these services within the last year of the project. The work of the SCPs during the first year of project implementation has focused largely on strengthening the provision of services at primary care facilities. There has been less attention given to community outreach and participation initiatives that constitute a major component of the SHTP-II contract. Performance standards that involve the participation of VHCs, HHPs, and TBAs in community-level work seem unlikely to be met since not all VHCs have been trained and effectively deployed, many counties still do not have HHPs, and TBAs have not been effectively working with the project. Key Recommendations Key recommendations are summarized below (the complete list of recommendations is provided in the full report). 1. During the Remaining Year of the Project Implementation of Seven High Impact Services During the remaining year of the project SHTP-II needs to give greater emphasis to high impact services that are currently seriously deficient. · The greatest priority should be given to addressing the glaring deficiencies in safe delivery services. The SHTP￾II contract describes a “focused and intensive maternal program” and outlines a set of four activities and an appropriate description of how to support EmOC, including use of misoprostol. It is recommended that this SOW serve as the basis for SHTP II’s scale-up of maternity care activities in the upcoming year. Additionally, SHTP II should support a clinical refresher course for community midwives who have already received training and been assigned to SHTP-II supported facilities. This refresher course would ideally be designed from existing materials that have already been developed by MSH or other organizations working in maternity care. An initial assessment of the knowledge and skills of community midwives needs to be undertaken, so that the training curriculum addresses what skills the trainees need to acquire. These midwives should be mentored at a clinical site with competent midwives (possibly using five to six experienced expatriate midwives) to ensure they have the skills to manage normal births, recognize complications, and manage emergencies and referrals. · Family planning will also require greater attention in the remaining year of the project. Training on the provision of family planning services should continue. As more family planning commodities are introduced, SHTP II and USAID must conduct continuous compliance training and monitoring – and they must provide evidence of this compliance effort. It will also be important for SHTP-II staff to become more involved in the work of the newly revitalized MOH Reproductive Health Technical Work Group in order to better coordinate national family planning program efforts. · During the remaining year of the project, SHTP-II should work to strengthen the WASH component of the project. The supply of clean water, sanitation facilities, and the management of medical waste remain substantial problems in most project sites visited by the Evaluation Team. SHTP-II should also develop additional indicators to measure progress in establishing hygienic environments in and around primary care facilities and properly disposing of medical waste through secure incinerators. Sudan Health Transformation Project Phase II (SHTP-II) 14 Mid-Term Evaluation Report – Public Document · Efforts should be made to reconfigure the HIV component of the project away from clinical services and more toward a community-based prevention approach. It may not be feasible to establish PMTCT services in many additional SDPs, given the lack of HIV testing capacity in most SHTP-II counties. As there are many SCP staff who have been recently trained in HIV counseling (but are not working at full capacity due to delays in acquiring HIV testing kits), their skills would be best put to use as community prevention outreach workers. Their activities could include raising HIV awareness in local communities and reducing levels of high-risk behavior that encourage HIV transmission. SHTP-II’s HIV work should be concentrated in counties considered to have higher HIV prevalence. Primary Healthcare System Strengthening · PHCU and PHCC infrastructure is still deficient in many counties where SHTP-II is working. SHTP-II, in consultation with USAID, should attempt to identify opportunities for undertaking additional construction and renovation work during the remainder of the project (possibly in coordination with other USAID or donor projects). Consideration should be given to conducting a rapid field assessment of infrastructure needs, safe water and sanitation facilities, and environmental cleanliness at all service delivery points in the project’s 14 focus counties. This assessment would provide a clearer picture of current infrastructure deficiencies and suggest strategies for remedying deficiencies. However, the Evaluation Team acknowledges that remaining SHTP-II budgets are totally inadequate for addressing extant infrastructure needs. · USAID and SHTP II must assure the delivery of all equipment that has been ordered for the project. SHTP-II “hardware” is the most visible and tangible component of the project. Some SCPs were adamant that they would prefer to procure equipment and appear to have the capacity to do this. The SCPs have typically been waiting up to ten months for a process that likely could have been completed in one or two months if the SCPs had procured on their own. · Ensuring adequate staffing and the timely provision of salary payments continue to be enormous obstacles to the success of SHTP-II. USAID and SHTP II must play a greater advocacy role with MOH/GOSS and state governments in working to improve these systemic problems. SHTP-II is currently providing significant budgetary support for salaries to facility-based service providers and support staff that could be better deployed for other purposes. · Further efforts are needed to strengthen the capacity of County Health Departments. Steps should be taken to develop integrated county health plans in consultation with all stakeholders in order to better organize supervisory schedules and responsibilities. Where CHDs are strong, the SCPs should be able to provide grants to the CHD through a Memorandum of Understanding (MOU). CHDs should also be encouraged to develop annual workplans to be used to coordinate health activities, ensure greater harmonization of activities, and avoid duplication of effort. · SHTP II should intensify its reach beyond primary care facilities and strengthen community-based systems that have been put in place. Village Health Committees (VHCs), Home Health Promoters, and TBAs need more guidance on how to effectively promote healthy practices and raise the demand for services. They are critical for raising awareness within households and community organizations such as churches and schools. · Greater efforts are still required to ensure the timely provision of drugs through the MOH/GOSS commodity￾logistics system. Attention should be paid to reviewing essential drug management procedures from state to county health facilities. The essential drug kit should also be revisited in order to ensure that emergency obstetric drugs are being introduced into the distribution system. The kit should also contain sufficient quantities of anti-malarial drugs and antibiotics for treating bloody diarrhea and ARIs. It would also be advisable to advocate for the creation of drug depots at the county level to ensure a continuous supply of drugs in the case of emergencies. Sudan Health Transformation Project Phase II (SHTP-II) 15 Mid-Term Evaluation Report – Public Document Management of SHTP-II · If SHTP-II is limited to its three-year implementation period, it must stay focused on activities that will achieve the greatest impact. The value of the following activities are questionable for the remaining year of the project, as they have the potential to take project staff off focus and add to administrative burdens: o Starting Fully Functional Service Delivery Points;3 o Implementing Performance-based Contracting (PBC); o Proceeding with Micro-Grants for Community-based Activities; and o Continuing PMTCT and other HIV prevention activities as planned. · Performance-based Contracting (PBC) requires valid reporting of results and fully functional SHTP II and SCP supervisory support for M&E activities. The Evaluation Team questions the ability of SHTP II to implement an effective incentive-based PBC system in Southern Sudan’s current implementation environment. It is not recommended that an incentive-based PBC system be started if the project has only one year of additional implementation time remaining. Greater emphasis should instead be given to improving the quality and timely reporting of performance indicators. · Additional attention is needed to ensure the validity of performance data reported by the project’s M&E system. Field validation checks by SHTP II staff should occur once every quarter in all counties. SCP staff responsible for clinic recordkeeping and summary reporting should be visiting every PHCU and PHCC in their counties at least once per month (or more frequently for facilities with a history of reporting problems). · The Micro-Grants program has been slow to develop. If the project has only one additional year of implementation time, it is recommended that work on the micro-grants be discontinued. Instead, the SHTP II community mobilization expert should focus on working with the SCPs to improve and expand community￾level activities, including efforts of VHCs and HHPs to improve health awareness and behaviors in local communities. These activities are more likely to show results by the end of the project. · Communication, transparency, and the sharing of information are critical for the success of the project. The following activities are recommended to strengthen these important functions: o Share additional contractual, budgetary, and project performance information about the project with MOH/GOSS and attempt to meet and communicate on a more regular basis. o SHTP II and USAID (the COTR and Contracts Office) need to consult more frequently on a pre-set schedule, given the implementation delays that have occurred during the first 18 months of the project. SHTP II should work to improve its relationships with the SCPs by holding regular meetings and improving routine communication; supporting them by addressing high level issues affecting service delivery (salaries/incentives, construction/renovation, etc) and sharing feedback arising from field visits. · SHTP II and the SCPs should conduct quarterly supervisory visits in each SHTP-II county. GOSS staff from Juba should be invited to participate in these site visits. In addition, the SCPs should make efforts to involve the County Health Directors and their staff in these quarterly supervisory site visits. USAID should join at least one of these supervisory visits every quarter during the remaining year of the project. Efforts should be made to revise and standardize supervisory tools that currently exist based on MOH guidelines. All supervisory teams should be trained in their use to ensure consistency. Feedback to the counties should be undertaken prior to leaving the facility and a final report submitted within one week of the visit. · The SHTP-II funding split should be maintained as originally planned; namely 75% to the SCPs and 25% to the prime contractor (MSH). Funding levels for the SCPs should not be compromised during the last year of the project. In developing new NGO subcontracts, flexibility needs to be shown in allowing subcontracting 3 MSH has developed a Fully-Functional Service Delivery Point appraisal method to improve the quality of care at health facilities. The tool includes scoring protocols for assessing infrastructure, equipment, drug supplies, staffing, training capacity, community involvement and support, environmental quality, and management systems. These scores are used to develop site-specific remedial strategies for improving the range and quality of services. Sudan Health Transformation Project Phase II (SHTP-II) 16 Mid-Term Evaluation Report – Public Document partners to make reasonable and appropriate requests for equipment and supplies based on the particular constraints of their counties. · The Evaluation Team recognizes that SHTP-II is running short of funding and may not be able to finish the project over the remaining year of the contract. Therefore, the Evaluation Team recommends that USAID provide SHTP-II with more time and resources in order to focus on key areas with high potential for results. There should be carefully constructed guidelines for any project extension, agreed upon by the USAID Mission, MOH/GOSS, MSH, and the SCPs concerning priority activities, implementation plans, and regular monitoring activities that will help ensure the project can fulfill its promise. Sudan Health Transformation Project Phase II (SHTP-II) 17 Mid-Term Evaluation Report – Public Document I. INTRODUCTION AND PROJECT DESCRIPTION SHTP-II was developed by USAID in close collaboration with the MOH/GOSS to strengthen the provision of primary healthcare services in local communities, increase Southern Sudan’s capability to deliver and manage services, and improve the health status of the people of Southern Sudan. SHTP-II was originally tasked with serving 145 delivery points in nine focus counties and undertaking limited health system strengthening in three additional counties. The SHTP-II contract was signed by MSH and USAID on February 11, 2009, with the project’s estimated completion date set for February 10, 2012. The project was designed to be implemented in collaboration with non-governmental subcontracting partners (SCPs), many of which have worked in Southern Sudan for many years.4 The total three year budget for SHTP-II is $44,297,880. The original Sudan Health Transformation Project (SHTP-I) was implemented by John Snow International (JSI) between 2004 and 2009. The objectives of SHTP-I were to improve health facility infrastructure (including the rehabilitation of existing structures), provide basic equipment and supplies, strengthen the reliability of drug procurement and the commodity logistics system, introduce standardized training and clinical practice standards, and develop a reliable recordkeeping and reporting system for county-level health activities. When SHTP-I concluded its work in early 2009, it was decided that a second phase – SHTP-II – should build on the achievements of SHTP- I and accelerate progress in establishing a healthcare system based less on the exigencies of humanitarian relief and more on the requirements of developing sustainable high impact primary care services. It was anticipated that SHTP-II would focus on strengthening primary care services and raising community health awareness and behavior change. The key results established for SHTP-II were to: · Expand access and availability of high impact services practices; · Increase Southern Sudan's capability to deliver and manage services; and · Increase knowledge of and demand for services and healthy practices. These results were to be achieved by strengthening the provision of seven high impact services based on the Basic Package of Health Services (BPHS) model for Primary Care initiated by SHTP-I: · Child Health – Immunizations (through Expanded Programme on Immunization – EPI – services) and the diagnosis and treatment of diarrheal disease and acute respiratory infections; · Nutrition – Exclusive breast-feeding, the promotion of infant and young child complimentary feeding, and twice yearly Vitamin A supplementation; · Hygiene and Sanitation Practices – Household level water, sanitation, and hygiene (WASH); · Malaria – Control interventions including the use of long-lasting insecticide treated nets (LLITN’s), intermittent preventive treatment (IPT), and management of active cases using anti-malarial drugs; · Maternal Health – Antenatal, safe delivery, and postnatal services; · Family Planning – Child spacing and family planning information and services; and · Prevention of HIV/AIDS – PMTCT services established in selected primary care facilities and behavior change promoted to delay sexual debut and reduce multiple risk behaviors. In addition to these seven basic primary care interventions, SHTP II was given the responsibility of improving the management of Southern Sudan’s primary care system by working with County Health Departments (CHDs) to produce strategic plans, improve the supervision of primary care facilities, strengthen forecasting capacities for needed drug supplies and equipment, and develop county health budgets necessary for achieving project objectives. The strengthening of health system “governance” was also identified as an important objective for SHTP-II. These efforts were to be focused at the community level, and included establishing and training Village Health Committees 4 Since the signing of the SHTP-II contract, there have been no revisions to the contract that have affected the overall scope and design of project activities. However, the number of indicators to be tracked by the project was reduced and some targets lowered following the signing of the contract. Sudan Health Transformation Project Phase II (SHTP-II) 18 Mid-Term Evaluation Report – Public Document (VHCs); training, equipping, and fielding Home Health Promoters (HHPs); and involving community organizations (e.g., women’s, youth, and religious groups) in the cause of promoting improved health and the greater use of primary care facilities. A micro-grants program was proposed in the SHTP-II contract to fund community organizations to promote health awareness, knowledge, service utilization, and reductions in high risk health behavior. SHTP-II was also expected to address human resource capacity constraints facing Southern Sudan’s health system. This was to be undertaken through a “continuous engagement” human resource strategy that included formal training through seminars and workshops as well as on-the-job instruction and on-site mentoring approaches. SHTP II was also expected to be engaged at state and national levels to develop more standardized curricula and training materials that could be used to develop the skills of service providers and support staff employed at SHTP-II affiliated sites. If requested by MOH/GOSS, the project was also to provide technical assistance (TA) to the Regional Training Centers (RTC) that train most new health professionals in Southern Sudan. At the start of SHTP-II in February, 2009 there were five RTCs (in Hakim, Adol, Maridi, Yei, and Ganyiel) and also the prospect of five new RTCs being established during the three-year life of the project. An additional contractual obligation for SHTP-II entailed establishing goals for ensuring greater gender equity in the delivery of health services, improved household decision making affecting healthy behavior, and the accessibility and utilization of health services. The project was to make efforts to ensure equitable participation of men and women in CHDs, VHCs, and WASH activities and to increase male involvement in family planning and maternal health services. Additionally, the project was to provide disaggregation of performance indicators by gender to help determine if possible gender discrimination could be influencing the access and utilization of primary care services. SHTP-II was designed to work in ten states and 14 counties across Southern Sudan. The distribution of SHTP-II project sites by state, county, and subcontracting partner is shown below in Table 1. TABLE 1: SHTP-II STATES, COUNTIES, AND SUBCONTRACTING PARTNERS State County Subcontracting Partner 1 Central Equatoria Juba Adventist Development and Relief Association (ADRA) 2 Central Equatoria Terekeka ADRA 3 Eastern Equatoria Kapoeta North Save the Children 4 Lakes Wulu Save the Children 5 Western Equatoria Mvolo Save the Children 6 Western Equatoria Mundri East Mundri Relief and Development Association (MRDA) 7 Western Equatoria Mundri West Action Africa Help International (AAHI) 8 Western Equatoria Tambura International Medical Corps (IMC) 9 Upper Nile Malakal IMC 10 Jonglei Twic East CARE 11 Unity Panyijar International Relief Committee (IRC) 12 Northern Bahr el Ghazal Aweil South IRC 13 Western Bahr el Ghazal Wau John Snow International 14 Warrap Tonj South Comitato Collaborazione Medica (CCM) Sudan Health Transformation Project Phase II (SHTP-II) 19 Mid-Term Evaluation Report – Public Document II. SOUTHERN SUDAN HEALTH SECTOR CONTEXT 1. Current Health Status in Southern Sudan Southern Sudan emerged from 20 years of civil war with the north by signing the Comprehensive Peace Agreement (CPA) in 2005. Since that time, Southern Sudan has been administered by a semi-autonomous government from the capital city of Juba. A referendum was held on January 9, 2011, to decide whether Southern Sudan will become an independent state or remain unified with the Sudanese government in Khartoum. It is anticipated that Southern Sudan will opt for independence and soon become Africa’s newest country. The many years of civil war (largely concentrated in Southern Sudan) have left a legacy of severe underdevelopment. Only with the coming of peace and the signing of the CPA has it been possible to begin building Southern Sudan’s health system and bring quality health services to the long-neglected people of the south. Southern Sudan’s infrastructure remains weak, with few paved roads in the capital city Juba, and only dusty cratered dirt roads linking major population centers and towns (which can often be impassable during the rainy season). Southern Sudan’s education system is still rudimentary, with many rural communities having no schools; its commercial sector is plagued by the lack of modern banking and financial management facilities; and communication and transportation infrastructure are also problematic. The provision of health services was severely disrupted during the 20 years of civil war in Southern Sudan. Highly trained staff left the country; hospitals and clinics were destroyed; and those that managed to remain open stagnated in terms of improving the range and quality of services offered, the competencies of service providers, and the quality of patient care. The 2008 census reported that the total population of Southern Sudan was 8.2 million. However, this figure is generally considered to be an underestimate, in part owing to the substantial inflow of recent migrants of Southern Sudanese origin returning home from northern Sudan ahead of the 2011 referendum. The annual rate at which Southern Sudan’s population is growing cannot be reliably estimated at the present time. Much of what is known about health conditions in Southern Sudan comes from the 2006 Sudan Household Health Survey (SHHS). This survey indicates that Southern Sudan has some of the worst health outcomes in the world (see Table 2). Infant and under-five child mortality rates are at levels not seen in most developing countries for over 50 years. As of 2006, Southern Sudan had very low immunization coverage rates for all children aged 12-23 months (17.3%) and poor utilization rates for Vitamin A. Anthropometric measures of malnutrition among children were not exceedingly high by standards of other least developed countries, especially those in post-conflict environments. The maternal mortality ratio (number of maternal deaths per 100,000 live births) is estimated to be 2,053.9, an incomprehensibly high number in an era when most developing countries are making major strides in instituting safe motherhood programs and improved delivery services (including access to modern obstetric care). The 2006 SHHS reports that roughly three-quarters of all pregnant women in Southern Sudan do not receive any antenatal care and only 13.6% of all deliveries occur in a health facility; most deliveries occur in the home without the presence of a certified community midwife or adequately trained traditional birth attendant (TBA). Southern Sudan’s total fertility rate is essentially unknown since the 2006 SHHS did not ask questions on children ever born, children born in the last 12 months, or children currently living. A birth history capturing vital events over a three- to five-year period prior to interview (as is standard practice in demographic and health surveys) was also not attempted. When community midwives in the county of Wau were asked how many children women were having over their reproductive life spans, the Evaluation Team was informed that the average number was around eight children.5 Only 3.5% of currently married women were using contraception in 2006. Most of these users were employing the lactational amenorrhea method (LAM) that provides roughly six months of protection from conception if mothers exclusively breastfeed for six months following delivery. Even though the use of modern contraceptive methods is negligible in Southern Sudan, few women report an unmet need for family planning (1.2%), which implies that family planning awareness/knowledge and demand is low. The extent to which women are free to use contraception in Southern Sudan remains an issue. Anecdotal evidence obtained from several community midwives interviewed by the 5 In Southern Sudan no total fertility rate (TFR) estimates are available from any primary source (census, survey, or vital statistics registration). One report noted that the TFR was between 5.9 and 6.7, but no source for these estimates is given. They appear to have inferred from the contraceptive prevalence rate (see Michael, Janet, et al., Southern Sudan Maternal and Reproductive Health Rapid Assessment, 2007, p. 10). Many “proximate determinants” may affect the level of TFR in addition to the CPR – e.g., the proportion of women married, the use effectiveness of contraception, abortion, and lactational infecundity. Sudan Health Transformation Project Phase II (SHTP-II) 20 Mid-Term Evaluation Report – Public Document Evaluation Team suggested that many husbands are opposed to family planning, although nationally representative data from population-based surveys is not currently available on this issue. TABLE 2: DEMOGRAPHIC AND HEALTH MEASURES FOR SOUTHERN SUDAN6 Total Population (2008) 8.26 million (disputed) Annual Population Growth Rate Unknown Total Fertility Rate Unknown Infant Mortality Rate (2006) 102.4 (deaths per 1,000 live births) Under-Five Child Mortality Rate (2006) 135.3 (deaths per 1,000 live births) Contraceptive Prevalence Rate (CPR)(2006) 3.5% Unmet Need for Family Planning (2006) 1.2% Maternal Mortality Ratio (2006) 2,053.9 (deaths per 100,000 live births) Percentage of Pregnant Women with No ANC (2006) 73.8% Percentage of Deliveries within Health Facilities (2006) 13.6% Percentage of Children 12-23 Months Fully Immunized (2006) 17.3% Percentage of Children 0-59 Months with Fever who Receive Anti￾malarial Drugs within 24 Hours (2009) 12% Percentage of Children 6-59 Months with One or Two Dose(s) of Vitamin A in Last Six Months (2006) 13.6% Percentage of Children 6-59 Months with One or Two Dose(s) of Vitamin A in Last Six Months (2009) 48% Underweight Prevalence (2006) 14% Stunting Prevalence (2006) 18% Wasting Prevalence (2006) 7% Percentage of ANC Respondents Seropositive (2007) 3.7% Percentage of ANC Respondents Seropositive (2009) 3% Percentage of HIV-Infected People Accessing ART (2007) 12% Percentage of Women Aware of HIV (2006) 45.1% Percentage of Women Aware of PMTCT (2006) 31.7% Percentage of Women Knowing that Condoms can Block HIV Transmission (2006) 14.3% Percentage of Women 15-24 Literate (2006) 2.5% Life Expectancy Unknown There also appears to be considerable pro-natalist sentiment in Southern Sudan in the wake of the civil war that resulted in the loss of many able-bodied young men (only 40% of Southern Sudan’s population is male). However, there is currently little knowledge of alternatives to childbearing among reproductive-aged women in Southern Sudan, so it is unclear whether pro-natalist sentiment is strongly held. The age at marriage in Southern Sudan is very low – girls are often married by the age of 15. This not only increases the number of children women will have over their reproductive life spans, but raises the likelihood that young women will be exposed to the risk of conception and high-risk pregnancies associated with developmental immaturity. In 2007, an HIV Sentinel Surveillance Survey based on information collected from 10 antenatal facilities reported an HIV prevalence rate of 3.7% (a figure considerably below levels in several neighboring East African countries).7 A subsequent ANC Sentinel Surveillance Survey in 2009 based on 24 sites reported a total HIV prevalence rate of 3.0%.8 There was 6 Data for this table is taken from the following sources: (2006 Sudan Household Health Survey (SHHS), 2007 and 2009 ANC Sentinel Surveillance Survey, 2009 Health Management Information System (HMIS), and 2008 Sudan Population Census) 7 Southern Sudan HIV/AIDS Integrated Report (2006-2007), 2008, p. 6. 8 Southern Sudan Antenatal Care Clinics Sentinel Surveillance Report, September-December, 2009, 2010, pp. 23-24. Sudan Health Transformation Project Phase II (SHTP-II) 21 Mid-Term Evaluation Report – Public Document considerable regional variation in these results, ranging from site-specific estimates between 0.0% (in Aweil) and 15.5% (in Yambio). Given the growing presence of recently arrived migrants into Southern Sudan (both from the north and from other countries in East Africa), it is likely that HIV will become a larger problem. The 2006 SHHS reported that HIV awareness among Southern Sudanese women is low, and that less than 20% of women knew that condoms can be used to block the transmission of HIV. 2. Current Health System Structure in Southern Sudan Southern Sudan is currently building a decentralized community-based health system centered on the BPHS model, considered to be the most effective integrated approach for delivering primary care. At the community level, care is provided at Primary Healthcare Units (PHCUs) and Primary Healthcare Centers (PHCCs). Health services and community mobilization efforts are coordinated by the County Health Department (CHD). The PHCU is designed to provide basic services to local communities, including Expanded Program of Immunization (EPI) support, growth monitoring, the treatment of diarrhea and acute respiratory infections (ARI), antenatal care, the provision of anti￾malarial drugs and bednets, and information on disease prevention, personal hygiene, and nutrition (including Vitamin A). The PHCC is a higher-level facility that is usually the first referral point for PHCUs. The PHCC offers a broader array of services, including safe delivery care, family planning, and HIV prevention and testing services. According to MOH guidelines, one PHCU should be available for every 15,000 people and one PHCC for every 50,000 people. Planned staffing at PHCUs and PHCCs is shown in Table 3. These facility-based staff have typically completed primary schooling and have received basic practicum training in the services they provide. However, staff at these facilities are essentially auxiliary workers and do not always possess sufficient knowledge or skills for the tasks they are expected to perform, especially in such areas as maternity care (safe delivery) and the treatment of advanced cases of malaria, typhoid, and bloody diarrhea. County-level primary healthcare facilities are supported by state hospitals as well as national-level teaching hospitals and public health laboratories based in the capital city of Juba.9 The British-era Juba Teaching Hospital is the largest and best equipped hospital in Southern Sudan. At the present time, many services (including obstetric care, ARV treatment for HIV, and most surgical procedures) are only available at state and national-level hospitals. Moving patients to secondary and tertiary-level facilities from county-level primary care facilities will remain a major challenge until road and transportation infrastructure is significantly upgraded. TABLE 3: COMPLETE STAFFING ALLOCATIONS FOR PHCUS AND PHCCS AS OF 200910 PHCU Staff PHCC Staff CommunityHealth Workers (2) Clinical Officer/Medical Assistant (1) Community Midwife (1) CommunityHealth Workers (3) Support Staff (3) Certified Nurse Midwives (3) HIV/AIDS Counselor (1) Nurse (1) Laboratory Assistant (1) Pharmacy Technician (1) M&E/Bookkeeper (1) Public Health Technician (1) Support Staff(4) 9 There are currently fewer than 10 counties - from a total of 80 counties in Southern Sudan - with functional hospitals. 10 Ministry of Health, GOSS, Basic Package of Health and Nutrition Services for Southern Sudan, January 2009, pp. 37-38. Sudan Health Transformation Project Phase II (SHTP-II) 22 Mid-Term Evaluation Report – Public Document III. EVALUATION PURPOSE AND METHODOLOGY By October, 2010, SHTP-II was slightly more than half-way through its 36-month implementation period. USAID/Sudan decided that a mid-term evaluation (MTE) would be important in order to assess SHTP-II achievements and remaining challenges and recommend how the project could best meet its objectives. The evaluation was also assigned with the task of assessing whether the approach of the project was appropriate and what implications this might have for future USAID health programming in Southern Sudan. The key questions with respect to project design and implementation, as well as project management, are provided in Table 4.11 An evaluation team consisting of Andrew Kantner (Team Leader and independent consultant), Deborah Armbruster (USAID/W), Mary Harvey (USAID/W), Anna McCrerey (USAID/Liberia) and Geertruid Kortmann (an independent consultant working with the European-supported Basic Services Fund in Juba) was assembled by the Management Systems International (MSI) Services Under Program and Project Offices for Results Tracking (SUPPORT) project to undertake this MTE. The Team worked in Southern Sudan over the period from October 16 through November 18, 2010, but with most Team members either working in Southern Sudan for two weeks at the start of the evaluation (wave one) or during the last two weeks of the evaluation (wave two). TABLE 4: KEY QUESTIONS FOR THE MID-PROJECT SHTP-II EVALUATION12 Project Design and Implementation 1. Are there any issues with respect to project design and assumptions (documented or implied) that should be reconsidered based on experience to date? 2. Has the prime recipient met the terms and conditions of the contract? Did the subcontractors meet the terms and conditions of their subcontracts? 3. Is a cost-reimbursement, fixed fee contract a satisfactory contracting mechanism for the prime contractor? Are performance-based subcontracts reasonable for the subcontractors? 4. Do submitted reports meet contract requirements and program needs? 5. Does the project’s approach to subcontractor and government capacity building actually build their capacities in a meaningful way? 6. How successful has the project been in implementing activities set in their workplan? How closely do these activities align with the activities outlined in the project’s technical proposal and recommendations given in the SHTP-I Evaluation? Project Management 7. How effective are management structures (including HR structures, communication, and overall management of the contracts) between MSH HQ, MSH Juba, subcontractors, and USAID? 8. Did the program develop a staff development and mentoring plan consistent with building the sustainable capacity of the staff? 9. Was the project’s financial management adequate? 10. Has the program established a viable M&E system (including data validation)? 11. To the extent possible, the evaluation team should address the following question: Is the project worth the level of investment in terms of value for money? The MTE began by having the Team review a wide range of relevant documents and background materials pertaining to the design and implementation of the project. These documents included contract materials for the project, quarterly progress reports, and data generated by the project's monitoring and evaluation system. The Team gathered evidence through numerous sessions with staff from USAID's Office of Health in Juba (both informal and formal), as well as meetings with representatives from other donor organizations working in health; namely the European supported Basic Services Fund (BSF), the Multi Donor Trust Fund (MDTF) administered by the World Bank, bilateral organizations such as the United Kingdom’s Department for International Development (DFID), 11 See Annex 1 for the final approved SOW for the mid-term evaluation. 12 The questions listed here are the main questions from the SOW. See Annex 1 for the complete set of questions and sub￾questions. Sudan Health Transformation Project Phase II (SHTP-II) 23 Mid-Term Evaluation Report – Public Document Norwegian People’s Aid (NPA), and the Joint Donor Team,13 as well as United Nations development agencies working in Southern Sudan’s health sector.14 The evaluation also assessed the key quantitative performance indicators compiled by the project’s subcontracting partners and sent quarterly to SHTP II. Cumulative project performance results through the third quarter of 2010 were analyzed by the Evaluation Team. The quality of project recordkeeping was also assessed during field visits to project facilities and the CHD. Qualitative assessments were undertaken through extensive field visits to SHTP-II field sites. Visits were made to eight locations in eight counties in five states; namely Juba, Malakal, Mvolo, Mundri East, Mundri West, Wau, Terekeka, and Tonj South. The eight counties visited provided the Evaluation Team with a variety of implementation environments, ranging from the national capital (Juba), former garrison towns such as Malakal and Wau that have received more funding support from the national government in recent decades, rural counties with greater proximity to urban centers as in Terekeka, and more remote rural areas such as Mvolo, East and West Mundri, and Tonj South. The proposed field teams included one staff member from the Ministry of Health (MOH/GOSS), personnel from USAID's Juba office, a representative from the Basic Services Fund, and staff from Management Systems International (MSI).15 The travel schedule for the evaluation is presented in Annex 7. While in the field, visits were made to Primary Healthcare Units (PHCUs), Primary Healthcare Centers (PHCCs), and in a few areas, state hospitals. The field teams also met and traveled with representatives from the SCPs working in each county, County Health Officers, other staff from CHDs, and VHCs. These visits provided the team with an opportunity to meet with Clinical Health Officers (CHOs), Community Health Workers (CHWs), Maternal and Child Health Workers (MCHWs), community midwives, EPI providers, laboratory technicians, and pharmaceutical dispensers (among others). These discussions provided firsthand evidence on SHTP-II achievements and remaining challenges during the last year of the project. Methodological Limitations The methodology utilized for the MTE was able to generate a rich array of evidence (though not always highly consistent) within a very short period of time. One draw-back of the evaluation was that though the original selection of facilities was done randomly, the Evaluation Team was not always able to visit the counties it had initially hoped to, owing to a number of factors. For example, it was not possible to travel from the county of Mvolo to visit primary care facilities in Wulu owing to recent conflicts between tribal militias along the main road linking the two counties. Two additional counties were excluded due to security restrictions recommended by the USAID Regional Security Officer and as the evaluation took place toward the end of the rainy season, other locations were deemed inaccessible due to poor road conditions. The final choice of sites was made in consultation with the SCPs, taking into account the ease of accessibility and security conditions within each county. An additional limitation was the less than ideal circumstance that Team members were not able to travel together to the same facilities and counties. Each Team member encountered different implementation contexts and levels of achievement and there was no opportunity in the field to compare notes, “triangulate findings,” or agree on important issues requiring “reality checks” and follow-up among the entire group.16 This deployment pattern made it more challenging for the Team to reach consensus on main findings and recommendations since each Team member saw different things in different places. 13 The Joint Donor Team includes resources from the Netherlands, Canada, Sweden, United Kingdom, Norway and Denmark. 14 This includes the United Nations Children’s Fund (UNICEF), United Nations Population Fund (UNFPA) and the World Health Organization (WHO). 15 Unfortunately, due to other commitments only one MOH/GOSS staff member was able to participate in the field visits. 16 It should be noted that there was time for discussion after each site visit when the groups returned to Juba. Sudan Health Transformation Project Phase II (SHTP-II) 24 Mid-Term Evaluation Report – Public Document IV. FINDINGS AND CONCLUSIONS Findings reported here are based on assessments of (1) quantitative performance indicators tracked by the project; (2) evidence from background documents and field observations on the implementation of the project by SHTP II and its subcontracting partners; and (3) comparison of project performance with contractual expectations specified in the SHTP-II contract. 1. Findings from Performance Indicators SHTP-II is currently reporting on 15 contractual performance indicators, which is a reduction from the original list of indicators proposed for the project. Three of the original indicators are still under review and may be revised and retained by the project while additional indicators that were part of the original set have been dropped owing to their low priority and problems in assuring accurate reporting. A listing of all performance measures currently being tracked by SHTP-II with original proposed targets for the three years of the project are presented in Table 2.1 of Annex 2. Cumulative results covering the first four quarters of project activity (from September-December, 2009, through July￾September, 2010) and county-level breakdowns for the project’s 15 contractual performance indicators are shown in Tables 2.2 and 2.3 of Annex 3. Results for the first four quarters of project implementation summarized in Table 5 below suggest that there has been improvement in several key indicators. These results largely reflect clinic-based performance at facilities affiliated with SHTP-II, but for certain interventions such as polio, measles, and Vitamin A, distribution outcomes could also be influenced by special campaigns not attributable to SHTP-II activities. The most successful results pertain to improving the percentage of children under one year of age receiving DPT3, the provision of clean drinking water based on the sales and distribution of water purification tablets (WaterGuard) and sachets, and raising the percentage of children under five who received Vitamin A.17 TABLE 5: CUMULATIVE SHTP-II PERFORMANCE INDICATORS (OCTOBER-DECEMBER, 2009 THROUGH JULY-SEPTEMBER, 2010) Best Cumulative Performing Indicators by July-September, 2010 (Q4) 1. Percentage of Children under One who Received DPT3 29.2% 139% of Target 2. Liters of Drinking Water Disinfected with USG Point of Use Supported Products 189.6 Million Liters 135% of Target 3. Percentage of Children under Five who Received Vitamin A in Areas Assisted with USAID Funds 51.4% 114% of Target Worst Cumulative Performing Indicators by July-September, 2010 (Q4) 1. Number of Counseling Visits for FP/RH 3,983 Visits 26.6% of Target 2. Percentage of Deliveries with a Skilled Attendant at Birth in USG-Supported Facilities 22.9% 46.6% of Target 3. Percentage of all Health Facilities that Provide atlease Five of the Seven High Impact Services Using MOH-Approved Standards 77.3% 82.2% of Target Not all cumulative performance indicators tracked by SHTP-II were achieving satisfactory results. The number of family planning counseling visits, the percentage of deliveries attended by skilled birth attendants, and the percentage of all health facilities offering five of the seven high impact health services prescribed by the BPHS framework were below expectation at the mid-point of the project. While this latter indicator is below target, the result is actually surprisingly favorable given the conditions observed at many primary care facilities. It is worth pointing out that problems in measuring denominators for the indicators being tracked may be playing havoc with some results. A case in point is Mundri West, where it is claimed that 199% of all children under five have received Vitamin A doses and 229% of pregnant women had at least one ANC visit during their pregnancy. In Panyijar, 189% of 17 It should be noted that performance levels established for these indicators are not exceedingly ambitious. Having less than 30 % of children <1 year of age immunized for DPT 3 is not an impressive achievement by international standards. In addition, data on the distribution of water purification products conveys no information on their actual levels of use. Sudan Health Transformation Project Phase II (SHTP-II) 25 Mid-Term Evaluation Report – Public Document children are reported to have received DPT3 immunizations and 229% of all pregnant women have at least one ANC visit (see Table 2.3 in Annex 2). These results suggest that estimates of eligible county-level populations being used to compute certain indicators may be too low. However, it may also be the case that some facilities could be attracting clients from outside immediate county catchment areas, with the result that population-based performance indicators could be unexpectedly high. The recent influx of Southern Sudanese returning from northern Sudan ahead of the country’s referendum in 2011 could also be increasing patient loads at some facilities, especially those close to the border. The possibility of double counting when patients are first seen at PHCUs and then referred to PHCCs should also not be totally discounted. SHTP II has taken steps over the past year to improve the functionality of clinic-based reporting with the distribution of new registers based on MOH/GOSS guidelines, the training of facility-based staff in completing these forms, and the accuracy and timely reporting of results. Monthly performance reports are now routinely sent from PHCUs and PHCCs to SCP field offices. Quarterly reports are then transmitted from the SCPs to SHTP II in Juba. Reports are checked for internal consistency and numerical accuracy before being incorporated in quarterly progress reports that are submitted to the Office of Health at USAID/Juba. Despite these efforts to improve the accuracy and timely reporting of results, the Evaluation Team found evidence of poor recordkeeping in several counties. This raised serious concerns about the validity of aggregate performance data being reported by the project. (These concerns are discussed more extensively in the Monitoring and Evaluation sub-section below). 2. Field Observations of SHTP-II Activities18 A. Implementation of Seven High Impact Services19 Child Health Child health services observed at PHCUs and PHCCs were operational in counties receiving assistance from SHTP-II. The Evaluation Team observed mothers bringing their children to primary care facilities for routine postnatal care and the treatment of illnesses (mainly diarrhea, acute respiratory infections, and malaria). EPI services were being provided in all of the eight counties visited by the Evaluation Team. PHCUs were typically visited once a month by EPI outreach teams who spend the full day vaccinating children. However, not all children under one year of age were receiving the full number of the diphtheria/pertussis/tetanus (DPT) vaccine doses. Columns for DPT# on EPI registers were often left blank. Bacillus Calmette-Guérin (BCG) and measles vaccines were not available in most counties owing to temporary stock-outs. Child immunization cards for mothers to track immunization schedules were not being consistently used and child health cards that include both vaccination and growth monitoring were occasionally seen, but not always in use. Although there is significant immunization work going on in most counties, concerns remain with regard to the cold chain system and vaccine supplies. For example, in Terekeka, 60% of the refrigerators were out of order; recordkeeping of fridge temperatures was inadequate; vaccine supplies and stocks appeared to be poorly maintained; 18 Complete field notes prepared by individual members of the Team are presented in a companion volume to this report. A matrix of county-level results derived from the Evaluation Team’s field notes is presented in Annex 3. 19 The Evaluation Team made field visits to eight of the 14 counties in which SHTP-II support is being provided. It should be noted that there is considerable regional and county-level variation in achievements to date. These variations likely reflect differences in health infrastructure, access to reliable drug supplies, timely payment of salaries, variable environmental and security conditions, and the relative effectiveness of the project’s SCPs. Inconsistent results can confound agreement on what constitutes a broadly applicable finding. EPI CAMPAIGN – PANTHOU PHCC, NORTHERN BAHR EL GHAZAL Sudan Health Transformation Project Phase II (SHTP-II) 26 Mid-Term Evaluation Report – Public Document and personnel were not always competent in overseeing the cold chain. Continuous availability of all vaccines is not ensured in Terekeka since BCG and measles vaccines have been stocked out for several months. Despite continuing problems in providing reliable supplies of vaccines and challenges in maintaining cold chain systems, SHTP-II may be able to reach its contractual goal of having 50% of children under one year of age in project focus counties fully immunized against all major childhood diseases by the conclusion of the project. The SCPs working under SHTP-II have collaborated with CHDs and other partnering donor organizations (e.g., the GAVI Initiative, UNICEF, and WHO) in organizing clinic-based child immunization programs, community outreach services, and special EPI campaigns. VHCs have also been effective in some counties in mobilizing community organizations in EPI outreach and campaign initiatives. However, since not all counties have VHCs and HHPs trained and deployed, the project continues to face challenges in mounting a strong community-centered EPI program (there is further discussion of community mobilization activities later in the report). Oral rehydration salts (ORS) and zinc tablets for treating watery diarrhea were adequately supplied in most primary care facilities visited by the Evaluation Team and appeared to be commonly prescribed (at least according to the patient registers examined in Juba, Mvolo, Wau, and Malakal). However, diagnostic and treatment protocols being followed for more serious bloody diarrheas were not always consistently explained or implemented across all sites. Cotrimoxazole for the treatment of pneumonia was found in primary healthcare facilities in such counties as Juba, Mvolo, and Wau, but was more likely to be stocked-out in other counties visited by the Evaluation Team (e.g., Tonj South). Antibiotic syrups were commonly available through commercial pharmacies in larger towns such as Juba and Wau, but appeared to be less readily available in more rural market centers such as Mvolo. The Evaluation Team found evidence of progress in meeting contractual performance standards anticipated for diarrhea and ARI (pneumonia) and the provision of ORS and amoxicillin. However, there was little evidence that the project was making much headway in working with HHPs and TBAs in providing children with in-home care for diarrhea and ARI. The diagnosis and treatment of these conditions remained largely confined to clinic settings in the counties visited during the evaluation and attests to SHTP-II’s relative inattention to community outreach and in-home care at the mid￾point of the project. Nutrition Vitamin A is the primary nutritional intervention being provided at most primary care facilities in project focus counties. As noted previously, quantitative indicators imply that SHTP-II has already achieved its target for Vitamin A distribution at the mid-point of the project. However, field evidence suggested that the supply of Vitamin A supplements to primary care facilities was irregular. For example, Vitamin A in doses of 100,000 IUs appeared to be out of stock in Terekeka since early 2010. Although bottles of 200,000 IUs were available in each Terekeka facility visited (supplied in the essential drug kits), they were not being distributed. SCP staff in Terekeka claimed that 200,000 IUs were only appropriate for children above five years of age.20 Exclusive breastfeeding was being actively promoted for up to six months in facilities where community midwives and MCHWs were working. These health workers were aware of the importance of initiating breastfeeding immediately after birth and were encouraging this behavior for deliveries occurring at their primary care facilities. However, the Evaluation Team was told that in some communities, breastfeeding initiation was often delayed by one to two days and that these more traditional breastfeeding initiation practices were still quite common for births delivered at home by TBAs.21 Growth monitoring equipment (e.g., infant weighing scales) was seen at some sites, but it was unclear whether they were being widely used.22 Child malnutrition continues to be a significant problem in some regions of Southern Sudan; there have been recent reports of growing food shortages in Jonglei and several western counties. The Evaluation Team also noted that all 18 beds in the malnutrition ward of the Malakal Teaching Hospital were full and that four to five child deaths had occurred there in the previous month. According to the SHTP-II contract, the project was to undertake nutritional monitoring and complimentary feeding activities in the project’s focus counties. The Evaluation Team did not see any evidence of nutrition programs other 20 The Evaluation Team later confirmed what it had initially understood – one dose of 100,000 IUs is recommended for children six to 11 months and one dose of 200,000 IUs should be provided every six months for children 12 to 59 months. 21 Delaying the initiation of breastfeeding compromises the immunity enhancing effects of colostrum in the first days of life. 22 Upper arm circumference cuffs were not seen at SHTP-II supported sites (a common item at most primary healthcare facilities receiving support from the European-funded Basic Services Fund). These cuffs are mostly used in environments with severe child malnutrition or famine conditions. Sudan Health Transformation Project Phase II (SHTP-II) 27 Mid-Term Evaluation Report – Public Document than Vitamin A and breastfeeding counseling that could be attributed to the project. USAID does not provide funds for food and nutritional supplements through SHTP-II. Instead, it expects SHTP II and the SCPs to obtain these items from agencies that do provide food aid. The SHTP-II contract also anticipated that all nutritional interventions identified for the project (Vitamin A distribution, exclusive breastfeeding, maternal nutrition, the nutritional status of children under seven years of age, and complimentary feeding) would be provided through community outreach programs utilizing HHPs and TBAs. The Evaluation team saw no evidence of such activities at the mid-point of the project. Water, Sanitation, and Hygiene (WASH) The availability of clean water, hygiene and sanitation facilities, and waste disposal (including medical waste) were still problems at many service delivery points. Many sites did not have covered disposal sites, incinerators for destroying medical waste, or safety boxes for sharps. Pit latrines were often poorly constructed, were infested with flies, and did not appear to be frequently used or cleaned. Another problem is that clinics did not always have clear boundary demarcations (fences, walls, etc.). Facilities that are not fenced are exposed to surrounding animals that can drop excreta in facility compounds and allow local community members to freely wander into facility compounds and come into contact with hazardous waste material, thereby greatly compromising sanitation standards. While boreholes were installed and operational at most facilities visited by the Evaluation Team, they were not always located close to the facility. WaterGuard, a chlorine-based water purification tablet distributed by PSI, was available at many primary care facilities visited in Juba and Wau. Carter Center water filters for the prevention of guinea worm were also seen in several counties visited by the Team. Clean water jugs (with soap) were generally accessible and appeared to be well used at most sites. But this was not always the case. In Tonj South there were no hand washing facilities at any of the primary care facilities visited and in Terekeka no WASH activities were observed at sites supported by ADRA. ADRA has also reported problems in keeping water jugs on site as they are much coveted for household use. A more “theft-proof” system of water stations was under development for ADRA facilities in Terekeka. Unfortunately, SHTP-II does not have contractual performance indicators to measure progress in improving the cleanliness of facilities and the disposal of waste material, most notably medical waste. Additionally, no performance standards are proposed in the SHTP-II contract. It will therefore be difficult to monitor progress in cleaning up existing sites and ensuring that all SHTP-II service delivery points are meeting acceptable standards. An assessment of safe water supplies and the environmental cleanliness of clinic grounds could be undertaken as part of a recommended rapid assessment of infrastructure needs at service delivery points (SDPs) in all of the project’s 14 focus counties. The SHTP-II contract called for VHCs, HHPs, and facility-based staff to be well-trained in safe water, hygiene, and sanitation measures. The project was to provide outreach assistance to households and communities concerning the importance of clean water and sanitation, including the possible formation of local Water Management Committees to assist in maintaining borehole wells and pumping stations and increasing community ownership of these facilities. While boreholes were installed at all facilities visited by the Evaluation Team (much of this work completed under SHTP-I), community-centered efforts to encourage greater use of safe water, including WaterGuard purification tablets and improved personal hygiene, have not progressed as far as had been hoped at the mid-point of the project. VHCs have not been consistently trained and deployed to undertake WASH advocacy and behavior change initiatives and the lack of HHPs in some counties has constrained WASH community outreach efforts. Malaria Malaria is one of the most common health problems being treated at SHTP-II sites and continues to be a major threat to the survival of children. Owing largely to high utilization rates, the supply of anti-malarial drugs was often depleted before the scheduled arrival of new supplies every three months. Maintaining adequate supplies of Artemisinin Combination Therapy (ACT) for treating malaria in children was a notable challenge for the MOH/GOSS commodity/logistics system. Paracheck kits were also not seen in many facilities and laboratory facilities (where they existed) were often lacking equipment and reagents. For example, the supply of malaria paracheck kits has been unreliable in East and West Mundri, Mvolo, Terekeka, and Tonj South. Job aids and diagnosis and treatment algorithms for malaria were also not usually available. There is currently little capacity to treat more advanced INSECTICIDE-TREATED NET DISTRIBUTION – PANTHOU PHCC,NORTHERN BAHR EL GHAZAL Sudan Health Transformation Project Phase II (SHTP-II) 28 Mid-Term Evaluation Report – Public Document BEST PRACTICES – MATERNAL HEALTH The PHCU situated in the Canning Factory in Wau County was notable for having an exceptionally engaged community midwife who lived close to the facility, undertook daily postnatal visits to women in their homes following delivery (every day for the first week), and appeared to be achieving some success in having women come to the PHCU for ANC and delivery services (with this PHCU being able to offer referrals to the near-by State Hospital in Wau in case of obstetric emergencies). The Chief of the Village adjacent to the Canning Factory site told the Evaluation Team that most women in the community now want to have their deliveries at the PHCU rather than deliver at home and be attended by TBAs. malaria cases at county-level facilities. Some primary care facilities had quinine tablets, but injectable quinine was not seen at many PHCCs and never at any PHCUs. Anemia levels in more advanced cases (a major problem in children) were not being measured and blood transfusions could not be done at the project’s primary care facilities. Patient referral capacity was also inadequate for transporting advanced malaria cases to hospitals offering more advanced care. The supply of insecticide-treated bednets has been irregular. When available, they were often handed out as part of ANC consultations and EPI events. The Evaluation Team visited households in Terekeka and observed that bednets were widely used and much prized among the Mundari people. There was even considerable demand for used bednets and in Terekeka old bednets were seen protecting enclosures for goats and chickens and even covering plants. Performance standards set for malaria in the SHTP-II contract focus largely on service provider compliance with clinical standards for diagnosing and treating malaria (with first and second line malaria drugs). Efforts to increase the correct use of bednets, with special attention given to pregnant women and children under five, is a second priority activity. As no baseline information was collected at the start of the project for these two standards, it’s not possible to assess whether the project has made major strides in improving malaria care and prevention. The SHTP-II contract stipulated that 95% of all US-supported health facilities should be “fully compliant” with proper clinical standards by the end of the project, but no definition of what constitutes full compliance was proposed. Since full compliance would in all certainty include the ability to diagnose malaria, and since the Evaluation Team observed that few facilities had paracheck kits, the project would appear to have a long way to go before reaching the 95% compliance target. Maternal Health23 The Evaluation Team noted that tetanus toxoid and iron supplementation were being provided for pregnant women at some facilities, although these services were not tracked by SHTP-II as contractual performance indicators. In many counties, there continue to be shortages of well-trained community midwives and MCHW staff required to ensure quality antenatal care, safe delivery, and postnatal services. As a result, pregnant women still usually depend upon the services of TBAs. The Evaluation Team also noted that few women were coming to PHCUs when there were no female service providers on duty (as was the case at some PHCUs in Mvolo). Safe delivery remains the major problem in addressing maternal and newborn health needs. Equipment such as blood pressure cuffs, gloves, cord ties, delivery tables, infant cribs, scissors, and dressing materials were often not available. Drugs for controlling bleeding during delivery, an essential component of any safe delivery service, were not available in any facilities visited (even though oxytocin is currently on the MOH/GOSS approved drugs list). Misoprostol, which does not require refrigeration and has a longer shelf life than oxytocin, is neither on the approved drugs list or available anywhere in Southern Sudan at the present time. In most counties, the Evaluation Team found few recorded assisted deliveries at primary care facilities. For example, in Terekeka, ADRA reported that at least 80% of all deliveries take place at home. The Evaluation Team, however, also found evidence that the number of facility-based deliveries was increasing in some primary care facilities affiliated with SHTP-II. For example, the number of facility-based deliveries has been rising in the PHCCs in Mvolo County. Many PHCCs visited by the Evaluation Team had one or two community midwives in residence, but all too often did not have the necessary drugs and equipment to do their jobs. Currently, most community midwives can do little more than attend normal deliveries; advanced emergency obstetric care is beyond typical competency levels. The Evaluation Team also noted one disturbing case at a PHCC in Wau where a mother and newborn child (less than 24 hours old) in need of emergency care were not offered any assistance since the delivery had not occurred at the PHCC and emergency umbilical cord repair could not be performed at the site. Postnatal care appears haphazard and largely up to the discretion of the community midwife or TBA that attended the birth of the child. Some community midwives and MCHWs undertake proactive postnatal care (sometimes domiciliary-based, as at the Canning Factory site in Wau), while there appeared to be no systematic postnatal follow-up of mothers by maternity care staff at other sites. The highest level of mortality for mothers and newborns occurs 23 Maternal Healthcare include Antenatal Care (ANC), Delivery Care, and Postnatal Care (PNC) Sudan Health Transformation Project Phase II (SHTP-II) 29 Mid-Term Evaluation Report – Public Document BEST PRACTICES – FAMILY PLANNING In Tonj South, Comitato Collaborazione Medica (CCM) conducted a unique KAP survey before initiating family planning programming in the county. The survey revealed that there was a high level of resistance to family planning, yet unmet need exists as well. It reported that preferred methods were oral contraceptives and injectables. In addition, the survey revealed that family planning decision-making is often done by men. Based on these results, CCM developed a family planning approach in which maternal and child health were emphasized over family size reduction. during the first 24 hours, with higher levels continuing through the first week following delivery. While current WHO guidance is being updated, it is essential that women and neonates are closely monitored during this critical first week with follow-up at six weeks (unless problems are identified earlier). Overall, maternal healthcare was underdeveloped at all project sites compared to child health and malaria control services. The project was far from meeting minimum standards pertaining to the provision of basic prenatal, maternal and neonatal, and postpartum care at USG service delivery points; the proportion of pregnant women who deliver in primary care facilities with a skilled birth attendant; and the proportion of facilities offering emergency maternal and newborn care. There was no emergency obstetric care capability at any of the primary care facilities visited by the Evaluation Team. The ability to refer women with delivery problems to higher levels of care was very limited. Contractual performance standards to be achieved by SHTP-II at the end of its three-year implementation period will not be achieved unless program efforts are greatly intensified. Family Planning The Evaluation Team found that there has been little systematic information, education, and communication work to promote family planning under SHTP-II, which in Southern Sudan tends to focus on the maternal health benefits to be gained from spacing births through the use of reversible methods. Women occasionally do inquire about contraception when visiting primary care facilities and there were family planning IEC materials on the walls of many sites (although many looked recently affixed for the benefit of the Evaluation Team). However, demand for contraception is low owing in part to poor awareness and knowledge of family planning methods and strong pro-natalist sentiment, likely driven in part by high levels of infant and child mortality. Low family planning use also results from the inadequate supply of contraceptive methods and the fact that provider training in family planning and other reproductive health interventions has lagged behind training in child health. A Juba-based training program in family planning counseling and service provision for maternity care providers working in SHTP-II facilities was the first major FP/RH training event organized by SHTP II since the start of the project. Modern method choice remains quite restricted in Southern Sudan, which inhibits women’s ability to freely choose a method appropriate for their reproductive needs. Some SCP staff appeared to be aware of compliance requirements for family planning. For example, when queried about family planning compliance, CCM staff in Tonj South provided correct information about ensuring client consent and voluntary method choice. The Evaluation Team was not able to ascertain whether other SCPs were well-versed in compliance procedures and the urgency of adhering to these procedures once family planning services (and wider method choice) are better established. LAM is being advocated in some primary-care facilities as an effective method for delaying pregnancy, although oversold when mothers are told that LAM protects for up to 24 months. Several discussions with community midwives and MCHWs led the Team to believe that incorrect information was occasionally being provided to mothers about the efficacy of LAM as a means of forestalling conception. There have been quality assurance problems affecting the distribution of oral pills procured with USAID funds that have resulted in supplies not reaching the field (or as in East and West Mundri, stock having to be recalled from the field). However, some oral pills were available at PHCCs in Malakal and Wau, although the source of these supplies was not ascertained by the Evaluation Team. There were no injectable hormonal methods available (e.g. Depo-Provera). Several community midwives noted that injectables would likely become a popular method in Southern Sudan if supplies could be made available. Performance standards established for family planning in the SHTP-II contract stipulate that pills, condoms, the lactational amenorrhea method (LAM), and the standard days method (SDM) will be widely available in the project’s primary care facilities (with condoms available in all SDPs). Injectables were to be piloted in selected PHCCs situated in urban areas/towns or in counties with high numbers of returning migrants into Southern Sudan. Other contractual standards entail having more facility-based staff trained in family planning services and counseling and more primary care centers where approved clinical standards for family planning are available and practiced. Based upon the findings of this evaluation, SHTP-II will be seriously challenged to meet these standards within the remaining year of the project. Sudan Health Transformation Project Phase II (SHTP-II) 30 Mid-Term Evaluation Report – Public Document HIV Prevention Services The SHTP-II contract calls for Prevention of Mother to Child Transmission (PMTCT) services to be established in selected SDPs with HIV counseling and testing in place or in facilities within close proximity to these services. The contract also calls for HIV prevention initiatives to promote knowledge and behavior change (abstinence and being faithful to partners – AB). These community-based prevention efforts were to be implemented by VHCs and HHPs and engage local community organizations (i.e. in-school youth and church groups) to the fullest extent possible. These efforts were to complement HIV prevention efforts directed toward other HIV target groups, including military personnel and their families, truck drivers, and “all couples who do not know their HIV status.”24 These HIV prevention initiatives have been slow to develop. PMTCT services are currently only available at primary care facilities in Juba and Malakal. In Juba, ADRA currently supports two PMTCT sites that have ready access to the Juba Teaching Hospital. In Malakal, PHCCs did have HIV test kits available, PMTCT services were in place, and referrals to ARV treatment facilities at the Malakal Teaching Hospital were routine. In Wau, PMTCT services were available in the MOH/GOSS State Hospital, but were not being offered in any PHCC site affiliated with SHTP-II. Other counties visited by the Evaluation Team found no PMTCT services in place (e.g., more rural counties such as Mvolo and Terekeka). Additionally, many counties did have trained HIV counselors on staff at PHCCs, but they did not have much IEC material on HIV/AIDS (and usually none in Arabic), no HIV testing kits, and were usually not in close proximity to state-level facilities that did offer HIV testing and treatment services (which were conditions stated in the contract for developing PMTCT sites). Nevirapine, the principal drug used for treating pregnant women who are HIV positive, was not seen at any primary care site outside Juba and Malakal and was not being widely distributed below the state level by the MOH/GOSS commodity-logistics system. Community-based HIV prevention efforts (as indicated in the contract) have not been implemented at the midpoint of the project. No abstinence and being faithful (AB) interventions were noted in the counties visited by the Evaluation Team. There appears to be some stigma toward HIV patients in several locales. For example, in Yeri, the HIV counselor was given a one-room hut located far away from the main PHCC compound to perform his duties. He had few clients coming to avail of his counseling skills and little to offer in the way of a unified VCT service. He inquired several times about the delays in receiving HIV testing kits that had been promised for many months. The Evaluation Team also noted that prevention efforts were still planned in all PHCCs regardless of the different prevalence levels that characterize the spread of HIV in Southern Sudan. While HIV seroprevalance data were not available for most geographic sub-divisions, the Team did note that most counties were reporting few cases of HIV, although this finding may partly reflect the absence of widespread testing. There is currently no SHTP-II strategy to concentrate HIV services in counties thought to have higher prevalence rates. Additionally because no HIV test kits have arrived in most counties (Juba and Malakal being the two exceptions), the project’s PMTCT component is on hold at most facilities. However, nearly all SCPs have already allocated significant resources for training. While counselors were trained in prevention messaging, they were typically poised to deliver these messages in a facility-setting without plans to engage in community outreach. There was no assessment conducted to identify local cultural beliefs and attitudes surrounding HIV. As such, it is unclear whether project efforts were culturally attuned to local conditions. It is only fair to point out that few health projects in Southern Sudan have undertaken extensive community involvement and outreach initiatives. It is not always clear how to organize such efforts in the highly variable ethnic and tribal environments of Southern Sudan. B. Primary Healthcare System Strengthening In addition to providing the seven basic primary healthcare services, SHTP-II is working to strengthen Southern Sudan’s health system and infrastructure. The SHTP-II contract outlines performance standards for supporting MOH/GOSS operations at county and state levels. This assistance includes the provision of support to County Health Departments in developing strategic plans and budgets; improving the management of the commodity/logistics system; improving the monitoring and supervision of local health facilities; expediting the adoption of standardized health policies, clinical practices, and training curricula; and building the human resource capacity of health facilities in the project’s 14 focus counties. Another important element of health system strengthening is the provision of training and logistical support to VHCs, HHPs, and TBAs in working with local community organizations (e.g., youth, church, and women’s groups) to improve health awareness, the utilization of health services, and behavior change affecting health outcomes. This is a formidable agenda for a three year project. However, achieving progress in these areas is essential if local primary care facilities are to be well-utilized and made sustainable. 24 See Contract Number GHS-1-00-07-00006-00 signed by MSH and USAID on February 11, 2009, p. 24. Sudan Health Transformation Project Phase II (SHTP-II) 31 Mid-Term Evaluation Report – Public Document Infrastructure In several SHTP-II sites visited by the Evaluation Team, construction was completed to a high standard (for example, the PHCC facilities built by UNOPS with USAID funds in Mvolo). However, most counties had major issues with poor or incomplete infrastructure. Dirt floors were found in some PHCUs, when concrete floors were the anticipated standard, and doorways and windows were sometimes not fully constructed. Some PHCUs had not been finished and remained closed to the public. One room tukuls25 were still being used to house PHCUs in some counties and often consisted of little more than a single room for all services without the basic furniture (e.g., tables, examination bed, chairs, curtains for privacy) necessary. PHCCs typically have more adequate infrastructure with separate rooms or buildings for different services, including male and female patient wards. Due to the lack of housing in local communities, PHCU staff members were sometimes living in the health facilities to which they had been assigned, which also decreased the space available for clinic functions. It was also noted by the Evaluation Team that some PHCCs were situated in quarters more appropriate for PHCUs. For example, one PHCC in Wau was housed in a two room structure that was in need of repair. PHCC staff members were attempting to serve clients, store drugs and equipment, and maintain patient registers in these two rooms. The Wau County Health Department acknowledged that this facility had problems, but there seemed to be confusion regarding who was responsible for dealing with the sub-standard infrastructure at this PHCC. Staffing at Primary Healthcare Facilities SCPs were supporting facilities that often did not contain the full complement of requisite staff. At the PHCU level, it was not uncommon to find only one community health worker (CHW) and no community midwife or maternal and child worker (MCHW). At PHCCs, Clinical Officers/Medical Assistants, nurses, certified community midwives, and lab technicians were not always in place. There was much regional variability in staffing patterns, with some counties clearly deficient (e.g., PHCUs in Mvolo and Wau) while other counties had the proper number of staff for different facility configurations (e.g., in Malakal). It is important to note that SHTP-II is not directly responsible for the staffing of PHCUs and PHCCs. The recruitment of staff for these facilities is the responsibility of MOH/GOSS even though remuneration is currently a shared responsibility between MOH/GOSS and the SCPs (using funding supplied through the SHTP-II project). Salaries for Health Workers The payment of salaries for facility-based service providers continues to be a major problem for Southern Sudan’s health system. In many states and counties, the Ministry of Health does not have the resources to make timely salary payments to its staff. A rapid assessment of the salary situation in selected SHTP-II counties (conducted in June 2010) found that 66% of provider salaries were being paid by SCPs while the Government of Southern Sudan pays roughly 34% of all salaries.26 In former garrison towns such as Malakal and Wau (that traditionally received preferential budgetary consideration from GOSS), most salaries were paid by the government – although in Malakal, government salary support only consistently included Medical Assistants and nurses. Most primary care facilities operating in cooperation with SHTP-II were a mixture of government and SCP funded staff. In the counties visited by the Evaluation team, government salary payments were sometimes late, often by as much as one to two months. These delays impact staff morale and can lead to absenteeism and inattention to duties. One encouraging finding pertaining to salaries is that the Evaluation Team did not encounter instances in which SCPs were making top-off payments to staff receiving government salaries. There was a perplexing salary arrangement at primary care facilities in Mundri East and West, whereby one-third to one-half of all facility-based staff members were paid by the government, while the SCPs paid all remaining staff. In some cases, there were two nurses at a facility – one paid by the County Health Department and one paid by the SCP. There did not appear to be an immediate problem with this configuration as staff members were being paid on time. However, this arrangement has the potential to become a major issue if the government is unable to pay salaries at any point. Additionally, interviewed staff receiving government salaries could not give clear explanations as to why they were selected to be transferred from SCP to government payrolls. 25 Traditional mud huts with thatched roofs. 26 USAID asked MSH to do a full-scale health worker salary assessment study and produce recommendations. A useful deliverable was not provided. USAID had to recruit additional technical assistance to sort through the issue. Sudan Health Transformation Project Phase II (SHTP-II) 32 Mid-Term Evaluation Report – Public Document Procurement of Drugs The timely provision of drugs to primary care facilities is still a challenge for Southern Sudan. The current distribution strategy is a “push” system that sends a preset supply of stock to the field every quarter irrespective of actual demand and utilization levels in the field. As a result, some drugs are over-supplied while others are in chronically short supply. For example, anti-malarial drugs and some antibiotics tend to be depleted prior to the end of each quarter. Other supplies such as dressing materials and IV saline solutions are widely available and may even be over-supplied. In Malakal, a somewhat more proactive system has been developed whereby the County Health Department and IMC monitor drug flows and immediately go to the state to get additional drugs when stocks run low. Expired drugs were not found in most facilities, but when they were discovered the Evaluation Team was assured they were not being used and would soon be disposed. No drug buffer stocks were being maintained in counties for emergencies, unanticipated disease outbreaks, or surges in client demand that could be triggered by Southern Sudanese immigrants returning home from northern Sudan and other surrounding countries. Procurement of Equipment and Supplies SHTP-II has had major difficulties in procuring equipment and supplies for the project in a timely and cost-effective manner. This includes furniture, blood pressure cuffs, delivery tables, drug storage shelving and “stocker” flow labels, IEC materials, testing kits, and lab equipment. These delays have caused disruption to planned service delivery enhancements. There was evidence that delays in procuring equipment had undermined the trust that the SCPs were developing within their communities. Malakal was the possible exception since IMC had been exceptionally resourceful in procuring equipment through non-SHTP-II supply channels, which suggests that SCPs are likely capable of doing more of their own procurement locally. MSH believed that pooling SCP equipment budgets would be a more efficient approach in clearing purchases through the USAID Contracts Office. Unfortunately, the pooling approach has not proven to be efficient. There have been delays in submitting procurement requests from SHTP II and the equipment clearance process through the USAID Contracts Office has taken time. There have also been problems in dealing with the purchasing and shipping agencies that SHTP II has been working with in Kenya to get equipment ordered and delivered in a timely manner. Several firms have been utilized over the past year, apparently to mixed effect. An often-heard comment from many SCP representatives was that equipment purchases were much smoother under SHTP-I since they were able to do their own procurement and could often obtain supplies like furniture as well as shelving and “stockers” for drug storage on the local market. The Evaluation Team has considerable sympathy for this view. Patient Referral Mechanisms Patient referral continues to be a major challenge for Southern Sudan’s health system. Services such as emergency obstetric care, ARV and PMTCT treatment for HIV, post-abortion and fistula care, the treatment of advanced malaria cases, and the management of chronic disease are presently only available at state and national levels of the health system. Making referrals from PHCUs to PHCCs is often difficult owing to the lack of vehicles, difficult road conditions (often impassable in the rainy season), and the cost of traveling. County-level facilities usually have no ambulances or other 4x4 vehicles to transport patients. However, SCPs in Mvolo and Terekeka were pilot-testing the use of motorized carts and motorbikes (complete with patient side-beds) and boats to move patients from remote settings to higher levels of care. Monitoring and Evaluation (M&E) SHTP II has assisted in tool development with all SCPs and has distributed new patient and drug registers based on MOH/GOSS formatted guidelines. These new registers were found in all of the counties visited by the Evaluation Team. The SCPs have provided training in the use of these registers, although in counties such as East and West Mundri and Terekeka there appeared to be confusion about how to use the new registers. On the other hand, recordkeeping practices and the reporting of results appeared to be proceeding more smoothly in counties such as Mvolo and Wau. In East and West Mundri, clinic registers were not being accurately and completely filled out. In Mundri East and West it was noted that there were extensive data errors present at nearly all facilities visited. Problems included missing data, incorrect entries, and confusion about how to complete patient registers and other reporting forms. The Team noted that it was common for register pages to be filled with names, but with no information on illnesses or diseases appropriately checked. Health staff explained that when symptoms did not fit the categories listed, they simply left the check boxes blank. Additionally, there were many places where staff simply did not record the illnesses being diagnosed or treated because they had “too many patients.” In Malakal and Terekeka, there was confusion surrounding the use of the new MOH/GOSS registers and summary reporting forms supplied by SHTP II. Sudan Health Transformation Project Phase II (SHTP-II) 33 Mid-Term Evaluation Report – Public Document In Terekeka, inadequate on-the-job supervision by the SCP caused by lack of transportation, difficulties in accessing facilities in the rainy season, and insufficient numbers of health facility staff with requisite training in M&E procedures resulted in poor recordkeeping. In some PHCUs in Terekeka, the CHWs only entered data into the drug dispensing register, while other reporting forms (e.g., the out-patient department (OPD), children under five, ANC and delivery care, and family planning registers) were left blank. In some counties, health workers were trained in Arabic, but were expected to record information in English registers. For example, EPI registers were problematic in some counties in that the vaccinator could only write in Arabic and had to use homemade Arabic registers, while an assistant copied data into an English register. This dual recording approach generated inconsistencies that proved difficult to reconcile. The Evaluation Team also learned that the new government registers supplied by SHTP II were too detailed, included unnecessary items, and did not always have enough space. For example, at the PHCC in Yeri (Mvolo County), the Evaluation Team was told that SHTP-I reporting forms were easier to use than the latest MOH/GOSS forms introduced by SHTP-II. It was noted that the new forms did not have sufficient space for recording symptoms, no column for distinguishing between bloody and watery diarrhea, and not enough room to record drugs that had been prescribed. There was also a column to record the number of plague cases, a malady rarely seen in Southern Sudan and one that cannot be readily diagnosed given current laboratory capacities. New registers also did not have carbon paper backup. Additionally, it did not appear that results from quarterly performance reports were getting fed back from SHTP II to the field or discussed with the SCPs and facility-based M&E staff to any great extent. These results should be used to better identify achievements and shortfalls and adjust work plans and budget allocations to address unanticipated program needs in individual counties. As not all counties are performing equally or face the same logistical constraints, sharing best practices and challenges between SCPs could lead to improved performance. Field validation procedures that spot-check the accuracy and completeness of clinic registers and verify the numerical accuracy of monthly and quarterly reporting have been developed by SHTP II; however, they have not yet been widely implemented. Distribution and Utilization of IEC Materials The IEC materials that were encountered in primary care facilities were not always well-designed or prominently displayed. While they were readily seen in Malakal and Terekeka, other sites did not have many child health, maternity care, or hygiene/sanitation IEC materials. Many posters were targeting family planning and HIV prevention services; even though these were two of the least developed high impact services at most primary care facilities. In addition, the Evaluation Team did not see flip charts or laminated one-page guidelines for use in counseling, or any IEC materials that could be supplied directly to patients and taken home for further reference. It seemed apparent that the posters being used by the project were not developed to be culturally or locally specific and presumably were developed without much pre-testing. The Evaluation Team observed that nearly all posters were in English, even in Arabic-speaking areas, such as Malakal and Wau. Also, some posters were text-heavy and made little use of pictures or drawings conveying clear messages to illiterate clients. Perhaps of greatest concern was the lack of guidance on how to use these materials. Posters often seemed to be merely dropped off and appeared to serve only as wall decorations. Some SCPs were found to be taking extra efforts to strengthen IEC and behavior change activities. In Tonj South, CCM staff has been conducting training sessions using self-made or self-obtained training materials. They have also developed radio programs and short public service messages on health issues. CCM established health education radio programming and other BCC training in collaboration with the VHCs. Logistical and Training Support to County Health Departments SCPs are working to strengthen County Health Departments (CHDs). SHTP II and the SCPs have not undertaken systematic assessments that identified areas of needed system strengthening in each county. SHTP-II has also not developed standardized guidance for strengthening all elements of CHD operations. However, the SCPs have been working to improve recordkeeping and M&E practices within CHDs. They have also provided logistical and transportation support for drug procurement and distribution, and assisted in the monitoring of drug supplies and equipment shortages. SHTP II has also recently conducted a Leadership Management Training course for all County Health Directors in SHTP-II project areas. Sudan Health Transformation Project Phase II (SHTP-II) 34 Mid-Term Evaluation Report – Public Document The Evaluation Team found considerable variation in the extent to which SCPs were being effective in working to strengthen County Health Departments. Good collaboration with the CHD was noted in the counties of Malakal, Mvolo, Tonj South, and Wau where IMC, Save the Children, CCM, and JSI, respectively, have been providing support. In Terekeka, there did not appear to be effective joint strategic planning between the CHD, ADRA, and the other health partners in the county, although there was joint operational planning for some activities (e.g., for the running of vaccination campaigns). ADRA did help in collecting and distributing vaccines from Juba, released ADRA-funded vaccinators to support EPI campaigns, and provided monthly reports from each of its facilities to the CHD Surveillance Officer. ADRA has also provided training to some members of the CHD in recordkeeping and M&E management. The CHD in Terekeka has also received significant support from MDTF through NPA. In the counties of Mundri East and Mundri West, the CHDs did not appear to be making concerted efforts to deal with deficiencies in recordkeeping, drug procurement and storage, and the capacity of facility-based staff to provide services. SCP Supervisory Support for Primary Care Facilities In many counties, the SCPs were providing adequate supervisory support to the primary care facilities in their areas, although procurement delays have made their work less effective than it might have been. An example of good supervisory support was noted in the facilities supported by Save the Children in Mvolo and CCM in Tonj South. However, in some counties there was a noticeable lack of supervision and standardized supervisory checklists. In East and West Mundri, the SCPs typically spent no more than 60 to 90 minutes at any facility during a monthly “site visit” and were providing little facility-level TA. Their visits mostly consisted of arranging salary payments and dropping off drug supplies. In Terekeka, ADRA and the CHD did not conduct regular joint supervisory visits. In fact, ADRA’s supervision of clinical facilities appeared to be quite weak overall due to three constraints: 1) lack of access to transportation by ADRA staff; 2) inadequate number of staff to conduct regular supervision at all health facilities; and 3) insufficient training in conducting supervisory visits to SDPs. Since most County Health Departments do not have transportation available for field supervision, many SCPs have been helping to organize joint supervisory visits. These joint CHD-SCP inspection trips promoted improved coordination between the government and SHTP-II in addressing problems associated with infrastructure, staffing, salaries, and the delivery of care. Some SCPs were also providing financial management assistance to the CHDs and the facilities they support. Such collaboration helps build trust between the SCPs and county-level government health staff and should strengthen opportunities for improving the visibility of SHTP-II activities. SCP Service Provider and Management Training SCPs still do not have standardized curriculums or methodologies for providing service provider and management training. The Evaluation Team found little evidence that the SCPs had been provided with any customized training materials geared to improving the knowledge and skills of service providers or the managerial efficiency of support staff at primary healthcare centers. Despite this absence, SHTP II has reported that the training of health providers (both by MSH and the SCPs) has accelerated in recent months and will be intensified during the remaining year of the project. In the most recent reporting period (July to September, 2010) SHTP II conducted workshops on family planning (for training of trainers and health providers) and instruction in utilizing “Fully Functional Service Delivery Point” appraisals to assess the range and quality of service provision and the capacity of facilities to serve the needs of local communities.27 During the same quarter, SCPs provided facility-based training in the treatment of malaria; diarrhea and ARI management; HIV/AIDS community mobilization strategies; hygiene and sanitation; and M&E procedures. A concern expressed by one SCP County Director was that with so much training underway, service providers would too often be pulled away from their main task of serving clients, with the possibility that project results could falter in the short-run while provider competencies presumably grow. Several SCPs have also been providing extensive facility-based training to service providers and administrative staff over the past year. For example, in Terekeka, ADRA has conducted numerous training sessions since the start of their project in February 2010. Among the topics covered were childhood immunization, diarrhea and ARI management and treatment; malaria control; maternal health and family planning; M&E tools for health workers; HIV/AIDS training for youth, community, and religious leaders; and community-based management of primary healthcare for VHCs and HHPs. In Mvolo and Wulu, Save the Children has conducted training sessions to improve the knowledge and clinical 27The Fully Functional Service Delivery Point is defined by evidence-based protocols used for assessing the quality of facility infrastructure; the supply of essential equipment; the provision and resupply of drugs; the adequacy of service delivery and support staff; training capacity; community involvement and support, and the effectiveness of management systems. The FFSDP methodology is used to develop site-specific remedial strategies for improving the range and quality of services and linkages with local communities. Sudan Health Transformation Project Phase II (SHTP-II) 35 Mid-Term Evaluation Report – Public Document proficiency of service providers, the management and storage of drugs and supplies, and the use of new clinic registers and summary reporting forms. Capacity Strengthening of Local NGOs SHTP-II does not appear to have had much success in strengthening the capacity of local (non-international) NGOs. One notable example is the Sudan Inland Development Fund (SIDF) that operates in the counties of Mvolo and Wulu. The Evaluation Team visited the offices of SIDF and found little evidence of activity. Few staff were around (and they were mostly volunteers) and the SIDF Executive Director was away in Juba. SIDF had no functional vehicle and had not been making supervisory field visits for many months. What functionality that could be found at SIDF was largely due to the assistance they were receiving from the Save the Children field office in Mvolo. According to Save the Children, financial management remains a major problem for SIDF. Save the Children also observed that SIDF does not currently have the capacity to organize or fully appreciate the managerial implications of a shift to a performance-based contracting (PBC) system. Community Mobilization: Village Health Committees and Home Health Promoters Village Health Committees (VHCs) have been formed at nearly all facilities visited by the Evaluation Team. VHC members are volunteers and receive no remuneration. Many VHCs were created under SHTP-1 and have been functioning since 2007. The county of Wau was notable for the lack of VHCs at both PHCUs and PHCCs. Home Health Promoters (HHPs) were not active in most counties visited by the Evaluation Team. Exceptions were noted in Malakal and Terekeka where HHPs were conducting health awareness and education programs in their communities. VHCs typically meet twice a month and concentrate their activities at health facilities. Their efforts include holding staff accountable for attendance and performance, cleaning the grounds of the facility, building and maintaining pit latrines, checking borehole wells, monitoring drug supplies, and reporting on health problems identified at health facilities. Health promotion work at the community level appears more irregular, although some VHCs were doing health education work at local churches and schools. In Tonj South it was noted that VHC members knew their roles and responsibilities, were knowledgeable about common diseases, spent considerable time at their facilities, and worked to ensure the security of drug supplies. The training provided to VHCs by the SCPs typically occurred over a four-day period and covered the roles of the VHCs in health facility management and governance. Several VHCs that met with the Evaluation Team pointed out facility deficiencies such as unfinished structures, the lack of safe water points, inadequate drug supplies, and insufficient staff. They were appreciative of the efforts of the SCPs working in their counties, but expressed some impatience with facilities that were not finished and did not have the full range of services and requisite equipment promised. Traditional Birth Attendants (TBAs) were still active in all counties visited by the Evaluation Team, although they did not have much contact with SHTP-II supported facilities and have received no supplies or training from the project. The MOH has recently signaled that no new TBAs will be trained and that they will be allowed to gradually retire from the health system. The community midwife, with more education and training than most TBAs, appears set to become the frontline maternity care worker of the future. Home Health Promoters (HHPs) have also been trained in some counties to work with Village Health Committees in strengthening community-based health education and outreach services. HHPs are meant to be an important element of the BPHS model. However, HHPs were not active in several counties visited by the Evaluation Team (e.g., in Mvolo, East and West Mundri, and Wau). HHPs were found to be working in Malakal and were members of VHCs, but they had not received much training and had few supplies. In Terekeka, the HHPs had received training from ADRA and were working in local communities, but they were not VHC members. In short, the HHP cadre had not been consistently recruited, trained, or deployed at the time of this evaluation. It is important to note that the MOH/GOSS is still defining the future role for HHPs and this cadre will need more focused attention and consistent training throughout Southern Sudan in coming years. It is not the responsibility of SHTP-II (MSH or any of its SCP partners) to recruit and train HHPs. This task rests with the Government of Southern Sudan. However, SHTP-II could be more engaged in collaborating with the MOH in developing and refining training curriculums for HHPs and field-testing approaches for having HHPs productively engaged with local communities. Gender Equity SHTP II and the SCPs are well aware of the need to promote gender equity in all aspects of SHTP-II programming. This includes working to eliminate gender disparities in the utilization of health services, promoting greater male involvement in women’s health programs (especially with respect to reproductive health and maternity care), Sudan Health Transformation Project Phase II (SHTP-II) 36 Mid-Term Evaluation Report – Public Document transforming gender relations and household allocations within households that constrict accessibility to FP/RH and maternal care at primary care facilities, and combating the incidence of violence against women. While these and other gender issues are clearly articulated in the SHTP-II contract, there has not been measurable progress in promoting greater gender equity at the mid-point of SHTP-II. The progress on gender that the contract calls for entails transforming long-engrained social and cultural norms that cannot be expected to change appreciably within a three￾year time frame. The project also has no baseline measures or contractual performance indicators to assess progress on gender equity. So to the extent that women’s status may be improving in the counties where SHTP-II is deployed, it would be difficult to attribute such progress to any particular project initiative. The Evaluation Team did not encounter much field activity that was directly working to enhance gender equity. For example, the project has taken few steps as yet to encourage greater male involvement in family planning and maternal healthcare, in part since these services were not functional at most sites. Village Health Committees still tend to be dominated by men, although women were well-represented at some facilities and were vocal advocates for maternal and child health services. The project has not yet begun working with community-based youth and women’s groups to advance gender awareness and rights-based approaches to improve women’s health and social status. The Evaluation Team did not encounter any project activities dealing directly with violence against women or women’s education, employment, and income status. The Micro-Grants program under development by SHTP II may have a strong gender￾orientation and could offer new opportunities for involving women and girls in community outreach programs and access to income-generating activities. Many CHDs visited by the Evaluation Team were staffed largely by men, although this was not always the case. More than half of all CHD personnel in Wau were women, and they were involved in the full range of primary care services advocated by the BHPS model. Service providers at most primary care facilities were men. All of the Community Health Workers and Medical Assistants encountered by the Evaluation Team were men. Women only dominated in the provision of maternal healthcare. It is important to note that SHTP-II has little influence in determining the gender balance of facility-based staff. Recruitment practices are the responsibility of MOH/GOSS, although the project can certainly play a strong advocacy in ensuring that women are fairly represented in the ranks of service providers and support staff. Recordkeeping formats being utilized by the project allow for the disaggregation of client-provider contact data and indicator results by gender. However, this information is inconsistently being compiled and reported by SHTP II and its SCPs. Therefore, it is not clear whether more boys than girls are being immunized against major childhood diseases, or whether there are preferential treatment patterns by gender for diarrheal disease, ARI, malaria, and nutrition services. There is no reason why such information could not be more accurately generated by the project since reporting forms routinely record the gender of all patients seen at primary care facilities. 3. Management of SHTP-II SHTP-II Leadership and Staffing Overall the management of the project has been problematic and there has been high turnover of staff, including the Chief of Party and several key technical staff. The seeming inability of MSH to address the slow startup of SHTP-II and ensure strong leadership for the project led to delays in implementing many activities. There have been four Country Leads at MSH headquarters (in Cambridge, Massachusetts) assigned to backstop SHTP-II since its inception in February 2009. This high turnover has likely contributed to the lack of continuity in managing the project. Owing to implementation delays, the project has needed to increasingly rely on expensive short-term technical assistance from MSH headquarters, which has budgetary implications in terms of salary and overhead obligations for the project. The Evaluation Team was informed by SHTP II staff that the project has had problems recruiting and retaining local technical staff for the project as the supply of well-qualified applicants is limited in Southern Sudan. It is anticipated that more donors and external funds will flow into Southern Sudan after the referendum. This may cause additional human resource challenges and will likely warrant additional discussion between USAID and MSH on how to ensure the adequacy of SHTP-II personnel recruitment and retention policies. Project Communication and Coordination While in the field, the Evaluation Team was told by several SCPs that they would like to have more guidance from SHTP II on expectations for implementing the project. During the first year of project implementation by the SCPs in 2010, communication within SHTP II was not considered sufficient for good program management. In particular, the SCPs would have welcomed more field visits and joint supervisory trips to the primary care facilities and community￾based activities undertaken by the project. Many SCPs noted that SHTP II staff had not visited their counties since Sudan Health Transformation Project Phase II (SHTP-II) 37 Mid-Term Evaluation Report – Public Document April/May 2010. However, quarterly meetings between SHTP II and the project’s SCPs have been held in Juba during 2010 and have been essential for sharing field experience and discussing next steps for improving project performance. Effective coordination between the project and USAID has not always run smoothly over the first 18 months of the project. Meetings with USAID’s COTR have taken place, but the periodicity of these sessions has not been consistent and has tended to occur on an as needed basis rather than through regularly scheduled meetings. The USAID Contracts Office is in Khartoum, which has made contractual clearance procedures for personnel contracts, equipment, and program documentation (e.g., the Micro Grants manual) more challenging than in most other environments where USAID operates from a single country office. Additionally, there has been considerable turnover of USAID direct hire staff, as direct hire staff have been rotating into Southern Sudan on a yearly basis owing to its designation as a post-conflict, which has produced some discontinuity in managing SHTP-II. USAID personnel who were involved in the design of SHTP-II are no longer based in Juba and current USAID staff members have inherited project designs and contractual mechanisms that they did not develop and may not be comfortable with. Foreign Service National (FSN) staff members have not been consistently and extensively involved in SHTP-II management and oversight owing to other administrative and management duties in the Mission. Adequacy of SHTP-II Work Plans The SHTP-II Work Plan for FY 2009/10 went through several changes that resulted in confusion with respect to planned activities and deliverables for the first year of the project. Budget allocations and quarterly expenditure rates were also not clearly specified. The new FY 2010/11 Work Plan currently being negotiated with USAID will offer all parties an opportunity to agree on a more detailed implementation agenda and budgetary framework for the remainder of the project. Coordination with Other Projects Engaged in Primary Healthcare There did not appear to be effective coordination between various health projects operating at county levels. WASH Project activities to develop safe water and clean sanitation facilities were often not engaged in the 14 counties or facilities in which SHTP-II is working. Oxfam was seen installing and servicing borehole wells close-by some PHCUs and PHCCs in Mvolo and Mundri West, but the extent to which this activity is coordinated with the SCPs working in the same counties was not clear. However, it was noted that SHTP II is collaborating with UNICEF in the delivery of child health services (primarily EPI) and has recently signed a Memorandum of Understanding (MOU) with UNFPA that could lead to more regular supplies of contraceptives and safe delivery kits to the 14 counties in which SHTP-II is working. In East and West Mundri and Mvolo, the Evaluation Team discovered that Population Services International (PSI), a subcontracting partner to MSH under SHTP-II, was using PHCCs to store water purification tablets (WaterGuard). PSI was also supporting several community-based distributors in East and West Mundri tasked with treating cases of malaria and diarrhea in children under five. There were no formal arrangements between PSI and the SCPs for coordinating these activities. In fact, the only information the SCPs had about PSI activities was secondhand from facility-based staff. There could also be some confusion arising over remuneration for facility-based staff if multiple organizations are basing project activities at primary care facilities affiliated with SHTP-II. The Evaluation Team observed that there was little exchange of information and experience among projects tasked with building primary healthcare capacity in Southern Sudan. The European-funded Basic Services Fund (BSF) is working to build primary healthcare capacity in over 25 counties across Southern Sudan through a program that provides grant support to international and local NGOs. However, there did not appear to be much regular exchange of information on project activities between BSF and SHTP-II. It was also unclear whether the work of the US Office of Foreign Disaster Assistance (OFDA) that provides funding for constructing and equipping primary healthcare facilities in non￾SHTP-II counties was being coordinated with the project. The SCPs currently have little contact with state and national-level MOH officials or other donor initiatives working to strengthen infrastructure, human resources, commodity-logistics systems, and service delivery at higher levels of the healthcare system. For example, state-level MDTF initiatives supported by the World Bank and bilateral European and Canadian-funded projects coordinated by the Joint Donor Team do not usually provide direct support for primary care activities. Efforts to improve primary healthcare are ultimately dependent upon the capacity of state and national-level systems to provide support for primary health services at the county level. Sudan Health Transformation Project Phase II (SHTP-II) 38 Mid-Term Evaluation Report – Public Document 4. Other Views on SHTP-II Implementation Views from MOH/GOSS The MOH/GOSS is still very supportive of the primary healthcare model (i.e. the seven high impact services model for primary healthcare) being pursued by the project ((and other primary healthcare projects such as BSF) and have clear ownership over the design. In discussions with MOH/GOSS officials, the Evaluation Team was told that improvements could be made in the effectiveness of communication between SHTP-II and the MOH. MOH officials feel that they have not been adequately consulted about the awarding of subcontracts to SCPs, the financial management of the project, and progress in implementing field activities. At the present time, they feel there is not enough involvement of the MOH in monitoring project implementation and staying abreast of SCP efforts at the county level. They would like to be more engaged with SHTP-II monitoring activities. However, it must also be acknowledged that key MOH staff are extremely busy and may not also be able to participate in SHTP-II supervisory field visits and internal project meetings. The MOH noted that SHTP-II should not be implemented in isolation, but as part of a larger coordinated effort to build primary healthcare services for local communities. Additionally, the MOH believes that there have been too many delays in implementing SHTP-II. They are well aware of the personnel, procurement, and financial management problems SHTP II has had during the startup of the project and they are hoping for better results in the final year. The MOH believes that the best way to enhance performance would be to allow SCPs greater latitude in strengthening service delivery at county-level primary care facilities and improving the effectiveness of CHDs. The MOH also maintains that not enough attention has been given to ensuring that SCPs have the basic equipment and management systems in place to strengthen primary care service delivery and enhance community participation in health education and behavior change activities. Views of Other Stakeholders Other stakeholders interviewed by the Evaluation Team believed that the SHTP-II and BSF models for strengthening county-level primary healthcare offer the right approach for Southern Sudan at the present time. Concerns were also voiced that high quality primary healthcare services could not be developed and sustained unless capacities of the health system at state and national levels are greatly strengthened. It was noted that SHTP-II and BSF have different implementation modalities. BSF did not experience the contractual delays that afflicted SHTP-II. Most BSF grants to collaborating NGOs were awarded in two months as opposed to an average of nine to 12 months for SHTP-II’s more complicated performance-based contracting mechanisms. BSF also has a far smaller Juba-based headquarters staff than SHTP II and its Juba-based subcontracting partners. On the other hand, it was acknowledged that SHTP-II may have stronger project monitoring and oversight capacity than is typically found in BSF project sites. The Evaluation Team was told that donor coordination on health has improved. The MOH is providing greater direction to the donors than before and there is more consultation within the donor community. For example, USAID is now participating in monthly Health Sector Donor Group meetings with representatives from the World Bank, Joint Donor Team, DFID, CIDA, JICA, and the EC. SHTP II is now more engaged with other bilateral and multilateral donors (including UNICEF and UNFPA), as well as with the large number of NGOs working in Southern Sudan’s health sector. SCP coordination has also improved. The monthly NGO Health Forum is well-attended and information on health needs and programming are openly discussed. Personnel from SHTP II and Juba-based SCPs regularly attend these meetings. This opportunity to meet and provide updates on health conditions, field environments, and the status of project implementation is becoming more important as greater attention and resources are dedicated to Southern Sudan’s health sector. Sudan Health Transformation Project Phase II (SHTP-II) 39 Mid-Term Evaluation Report – Public Document V.KEY CONCLUSIONS ON MID-TERM IMPLEMENTATION 1. Quantitative Performance Indicators The final 15 contractual performance targets established for SHTP-II capture many of the principal activities for the project. These results imply that greater efforts should be made to bring the family planning and maternity care components of the project up to an acceptable standard. It should also be noted that there are currently no contractual performance indicators for measuring child health services other than EPI (namely ORS, ARI, and malaria treatment in children) and the environmental cleanliness and sanitary upkeep of PHCUs and PHCCs. The indicators currently being tracked by the project are valid and could be reliably compiled from administrative and clinic records if recordkeeping systems were accurate and well-maintained. However, this did not always appear to be the case in several counties visited by the Evaluation Team. There continue to be serious concerns in several counties about the validity of data being generated from the project’s reporting system. The estimation of denominators for some indicators also appears to be a problem, given the uncertain quality of county-level census data. This problem may partially account for the surprisingly high achievement levels reported for certain indicators (e.g., Vitamin A distribution). SHTP II has given considerable attention over the past year to improving the accuracy and timely submission of M&E records. However, it is still doubtful whether data quality has risen to a level that firmly establishes levels of achievement for the project. 2. Implementation of Seven High Impact Services SHTP-II has been working to strengthen the primary healthcare model proposed by the Basic Package of Health Services (BHPC). At the mid-point of SHTP-II, progress has been mixed; although it must be acknowledged that the SCPs have been implementing project activities for less than one year and have lacked essential equipment and timely guidance from SHTP II on how to proceed in implementing various components of the project. The SCPs have had some success in strengthening child health services, but progress has been uneven. EPI services are now operational in all counties, although BCG and measles vaccines were not available at the time of the evaluation and cold chain equipment was deficient in several counties. ORS for treating watery diarrhea and antibiotics for ARI were generally available, although the consistent and timely provision of ARI drugs has been a problem in some counties. Vitamin A supplies have not been consistent in all counties, although performance data reported by SHTP II suggests that Vitamin A distribution has been above established target thresholds. Despite numerous deficiencies noted by the Evaluation Team, the provision of child health services has progressed farther than most other high impact interventions being implemented by SHTP-II. Many primary care facilities affiliated with SHTP-II continue to have problems assuring adequate supplies of safe drinking water, functional sanitation facilities, and adequate waste disposal measures. Since there has been no systematic survey of SHTP-II facilities in all 14 counties, it is not clear how many facilities in different counties have sub-standard water, sanitation, and waste disposal facilities and what remedial steps may be required at different sites. Several high impact services, most notably safe delivery care, family planning, and HIV/AIDS have not shown any significant advance as yet under SHTP-II. While basic antenatal care was being offered in most facilities by community midwives and MCHWs, safe delivery and postnatal services were less well-established. The project is falling well short of expectations in developing safe delivery services, owing largely to the lack of trained midwifery staff, the availability of essential supplies and equipment, and (most critically) the non-availability of drugs for controlling hemorrhaging during delivery. Developing a family planning program in Southern Sudan that offers women voluntary choice of methods appropriate for different stages of the reproductive life span, provides good follow-up care (including the management of side WOMEN AND CHILDREN WAITING FOR SERVICES, LELO PHCU – MALAKAL, UPPER NILE Sudan Health Transformation Project Phase II (SHTP-II) 40 Mid-Term Evaluation Report – Public Document effects), and incorporates other essential components of a comprehensive reproductive health service (e.g., the diagnosis and treatment of sexually transmitted disease, safe abortion and post-abortion care, fistula repair, etc.) will take time. The HIV/AIDS component of the project is focused on prevention and counseling services. The original contractual expectations included establishing PMTCT services at selected PHCCs with counseling and testing facilities available (or those in close proximity to facilities offering these services). Owing to delays in supplying HIV testing equipment, adequate drug supplies and trained personnel needed to implement PMTCT, the project has so far fallen short in developing the HIV/AIDS services that were envisioned. 3. Primary Healthcare System Strengthening SHTP-II has made some headway in strengthening Southern Sudan’s primary healthcare system. Much of this activity has been centered on upgrading recordkeeping and data collection procedures and providing training in M&E methods. Over the past year, most SCPs affiliated with the project have been providing training to facility-based staff in the management and treatment of disease, the proper handling and use of medicines, hygiene and sanitation practices, the monitoring and storage of drug supplies, and recordkeeping. However, much of this training was initiated by the SCPs and appears to have been somewhat ad hoc. Training initiatives have not been well-coordinated across all counties nor geared to any uniform training curriculum developed by SHTP-II and/or approved by the MOH to ensure programmatic and national consistency. Southern Sudan’s primary healthcare system continues to be plagued by inadequate infrastructure, insufficient staffing of facilities (the lack of certified community midwives and MCHWs being a major constraint), and problems in providing effective patient referral to higher levels of healthcare. Difficulties in providing timely salary support for facility-based staff (service providers and support staff) continue to be the single largest constraint to the development of Southern Sudan’s primary healthcare system. Since the start of SHTP-II in February 2009, the project has repeatedly raised concerns with MOH/GOSS and USAID about the significant level of salary support SHTP-II was being asked to provide for facility-based primary healthcare providers. The SCPs have also raised these issues many times with County Health Departments. Long-term permanent solutions are largely outside the ability of any single donor project such as SHTP-II to do much about. Lasting resolution will require centralized system-wide approaches involving effective collaboration between the entire donor community and MOH/GOSS, it will also involve discussions and agreements between the Executive Branches of the GOSS and the states.. Most SCPs were providing extensive salary support in their counties using SHTP-II budget allocations. However, these payments are a sizable drain on the project’s resources and divert money away from other service delivery enhancements. The procurement of essential drugs through the MOH/GOSS commodity logistics system continues to present challenges to providing primary healthcare services in Southern Sudan. There are currently no procedures in place to supply counties with drugs that have stocked-out prior to the quarterly re-supply schedule of the government. There are also no back-up systems in place for when drugs are not re-supplied on schedule. While the shipment and delivery of drugs through the government’s commodity-logistics system has improved in recent years, it is still not capable of predictably supplying drugs for Southern Sudan’s primary healthcare system. Comprehensive assessments will also need to be undertaken to identify the type and quantity of emergency drug supplies that should be maintained by CHDs, the capacity of county-level storage facilities, and the ability of CHDs to transport needed supplies to primary care facilities when required. Major delays in procuring essential equipment and supplies have severely constrained the implementation of SHTP-II field interventions. This deficiency has impeded the ability of the SCPs to move forward on schedule in strengthening the range and quality of primary care services envisioned for the project. At the mid-point of the project, the inability of SHTP-II to deliver the equipment and supplies that were promised to the SCPs constitutes the largest single disappointment of the project. Under SHTP-I, SCPs could undertake local procurement, which so far appears to have worked better than the centralized procurement system adopted by SHTP II. IEC materials to be supplied by SHTP II for the project have been slow to get to the field. In several counties visited by the Evaluation Team, it appeared that wall posters had only arrived and been taped to walls within hours of the Team’s arrival. The IEC materials that were seen were not always well-designed (too text heavy without enough diagrams and pictures), not available in local languages, and often describing services such as family planning and HIV/AIDS prevention and treatment services that were not in place at the service delivery point. In several counties, SCPs have been working effectively with County Health Departments in identifying local-area health needs, assessing progress in providing local area populations with primary healthcare, addressing continuing deficiencies in the range and quality of services on offer, and developing coordinated county health plans that account for the Sudan Health Transformation Project Phase II (SHTP-II) 41 Mid-Term Evaluation Report – Public Document BEST PRACTICES-COUNTY HEALTH DEPARTMENT CAPACITY BUILDING In Malakal IMC has made significant progress towards many of the objectives of the project. It has established a good working relationship with the CHD and encouraged ownership of project objectives by the county; appropriate drugs have been made available in the right quantities with minimal to no stock-outs and stored properly with stock cards; facility services have been well organized; basic equipment was in place; vaccines were routinely available and cold chains maintained; water and soap were available in all facilities for hand washing; and sharps boxes were seen in all facilities. The factors in Malakal that appear to play an important role in these successes include: 1. An active and motivated CHD Director 2. A committed and motivated SCP with talented, innovative, and motivated staff 3. Recognition of the importance of relationship building with all stakeholders – CHD staff, facility personnel, and community leaders 4. Use of creative problem-solving to overcome challenges – e.g., the delays in procuring equipment through SHTP-II were partially overcome by obtaining materials through UNICEF and UNFPA These initiatives in Malakal might be made to work in other counties and implementation environments. On the other hand, unique advantages that may exist in a former garrison town like Malakal may not be readily transferable to more disadvantaged locales. activities of all government, SCP, and other NGO health initiatives. SCPs in such counties as Malakal, Mvolo, Wulu, and Tonj South have achieved success in developing good working relationships with their County Health Departments. However, in counties such as East and West Mundri and Terekeka the SCPs have had less success in engaging the CHDs. Reasons for these different outcomes should be systematically studied, but the Evaluation Team was able to conclude that some variability can be ascribed to the willingness of SCPs to proactively engage the CHDs; the effectiveness of County Health Directors and the overall quality of CHD staff; the proximity of SCP personnel to CHD offices; and opportunities to undertake joint monitoring and inspection visits. The SCPs have made good faith efforts to undertake regular supervisory visits of primary care facilities in their counties despite the late arrival of new vehicles purchased under SHTP-II. In several counties these visits were described as perfunctory; typically lasting only an hour or less and consisting of little more than the delivery of drugs and the collection of completed reporting forms. Primary care sites in other counties enjoyed more conscientious attention from the SCPs that entailed more frequent visitation, longer visits at facilities, more engagement with VHCs and HHPs (where present), and greater efforts to trouble-shoot implementation issues. However, SHTP II has not standardized expectations of the SCPs in providing training, nor has a standardized supervisory checklist for undertaking inspections of field sites yet been implemented. SHTP-II training activities have been slow to get underway, but they now appear to be gaining momentum. In the quarter from July to September 2010, the project greatly accelerated training activities, both through Juba-based workshops organized by SHTP II and facility-based training undertaken by the SCPs. SHTP-II still lacks standardized training curricula covering all seven BPHS primary care services. A new training program is currently being developed by SHTP II that holds the promise of bringing greater consistency to curricula and instructional standards for project staff during the remainder of the project. SHTP-II has not addressed broader human resource needs in Southern Sudan’s health system. No support has been requested by MOH or supplied by SHTP II to the regional training centers responsible for training doctors, nurses, community midwives, TBAs, and auxiliary service providers. The possible provision of support to Southern Sudan’s regional training centers was mentioned as a potential activity in the SHTP-II contract, but this activity was never clearly defined or programmed. Given that other donor projects, such as the MDTF, are working to address human resource issues in Southern Sudan and provide support to the regional training centers, it is unclear what additional role could be played by SHTP-II. The project might be best served by freedom from any expectation to support regional training centers and remaining more tightly focused on urgent needs directly related to core SHTP-II objectives. The Evaluation Team discovered that SHTP-II activities and levels of achievement varied considerably across counties. More comparative analysis and sharing of results among within SHTP II and the SCPs would be useful as a means of transferring knowledge on successful initiatives that could be emulated and scaled-up elsewhere. Such interchange of experience could also better illuminate unique circumstances faced by different SCPs in the highly varied implementation environments of Southern Sudan. Community involvement and mobilization initiatives are also necessary to improve the health status of local populations. The project is working with Village Health Committees to provide county-level health education and behavioral change messaging, encourage greater use of primary healthcare services, organize primary care outreach services for surrounding communities, and work to improve the accessibility and quality of service provision. VHC members are volunteers and do not receive remuneration for their time and efforts. This raises concerns about the long-term sustainability of the VHC mechanism and the ability of local communities to recruit dedicated and qualified Sudan Health Transformation Project Phase II (SHTP-II) 42 Mid-Term Evaluation Report – Public Document VHC members. Home Health Promoters were to be important cadres who would engage with local communities, but in many counties they have still not been recruited, trained, and sent to the field. At the mid-point of SHTP-II, the project has not made adequate progress in strengthening community mechanisms that promote greater involvement in primary healthcare. The SHTP II Micro-Grants program, if effectively focused on VHC and HHP efforts to mobilize communities in the cause of knowing about and promoting good health, could play an important role in furthering community support for Southern Sudan’s primary healthcare system. Insufficient attention has been given to promoting the cause of gender equity within the ranks of CHDs, service providers and support staff, VHCs, and community-based organizations involved in health education and outreach. However, it is worth remembering that most SCPs have been implementing field activities under SHTP-II for less than one year. 4. Management of SHTP-II Since October 2010, MSH has been providing extensive headquarters support to the project. This will likely improve the efficiency of the project, but it will also entail additional costs in salaries and overheads. During SHTP-II’s first 18 months, technical project staff were not always in place, the need to mentor local-hire staff was not given sufficient priority, financial management issues caused contention (and the resignation of some staff), and MSH may not have been sufficiently attentive to start-up problems being encountered by the project. Performance-based Contracting (PBC) requires valid reporting of results and fully functional SHTP II and SCP supervisory support for M&E activities. The Evaluation Team questions the ability of SHTP II to implement an effective incentive-based PBC system in Southern Sudan’s current implementation environment and in the context of the findings of this MTE. Instead greater emphasis should instead be given to improving the quality and timely reporting of performance indicators. USAID has been able to exercise tighter control over the project with a fixed-fee contract, but management loads have also risen substantially. A cooperative agreement would have reduced management burdens within USAID’s Sudan Mission and given direct hire staff more time to monitor the implementation of the project.28 However, a cooperative agreement was used for SHTP-I and did not prove to be a cure-all for contracting and procurement problems. The type of contract used for SHTP-II may not have been a major factor in accounting for delays in project implementation (primarily affecting staff recruitment and equipment procurement) that have occurred over the first 18 months of the project, but there seems no disputing the fact that equipment purchases were handled more efficiently under SHTP-I and that the SCPs were happier when they had control of their equipment budgets and could purchase supplies locally. Much can also be gained through effective interchange and coordination with other donor projects. The European￾funded Basic Services Fund (BSF) is supporting primary care services at the county level and yet BSF and SHTP-II do not interact frequently or on a regular basis. In addition, SHTP-II does not have extensive contact with World Bank staff responsible for coordinating MDTF activities or implementing partners such as NPA. It is a widely held view that the donor community in general could be doing a better job in coordinating project objectives and resources as a means of maximizing the impact of health programs in Southern Sudan. 5. Meeting Contractual Performance Standards The SHTP-II contract identifies performance standards to be attained by the end of the project in February 2012. Since the project is only at the mid-point of its operational life and their remain concerns about the validity of data being generated by the project’s M&E system in some focus counties, it is difficult to pass judgment on the extent to which SHTP-II is on-track to achieve its standards. The findings from this evaluation suggest that the project has made progress in delivering child health services in primary care facilities (EPI, ORS, and ARI) and the prevention and treatment of malaria (despite the many shortcomings that still plague these efforts). On the other hand, maternal health, family planning, and HIV prevention efforts are clearly lagging. It will be very difficult for SHTP-II to achieve most minimum performance standards for these services within the last year of the project. The work of the SCPs during the first year of project implementation has focused largely on strengthening the provision of services at primary care facilities. There has been less attention given to community outreach and participation initiatives that constitute a major component of the SHTP-II contract. Performance standards that involve the 28 It is only fair to say that there was some disagreement within the Evaluation Team over the importance of the contracting mechanism used for SHTP-II (the fixed fee contract) versus SHTP-I (a cooperative agreement contract). This issue was mentioned several times by staff from MSH and several SCPs as a factor they believe contributed to delays in implementation. The use of different contractual mechanisms cannot be disallowed as a relevant issue, but a more definitive judgment on the issue should be deferred to the final evaluation of SHTP-II. Sudan Health Transformation Project Phase II (SHTP-II) 43 Mid-Term Evaluation Report – Public Document participation of VHCs, HHPs, and TBAs in community-level work seem unlikely to be met since not all VHCs have been trained and effectively deployed, many counties still do not have HHPs, and TBAs have not been effectively working with the project. At the mid-point of the project, SHTP-II performance standards for strengthening primary healthcare systems in Southern Sudan appear far from being met. The Evaluation Team observed that progress has been uneven across the project’s focus counties in working to build capacity in CHDs, improving the supervision of primary care facilities, strengthening drug procurement and storage capacity, and providing standardized training curricula and new policy guidelines for improving service delivery – among the long list of initiatives identified in the SHTP-II contract. The Evaluation Team was not able to review any systematic information for measuring the extent to which system strengthening activities are meeting performance standards. Some of this information can likely be compiled from management information system (MIS) records maintained by the SCPs, but it had not been pulled together in time for this evaluation. If these measures cannot be readily obtained from the project’s administrative records or MIS recordkeeping system, it may be necessary to undertake a rapid field assessment of the project’s system strengthening efforts prior to the conclusion of the project. Sudan Health Transformation Project Phase II (SHTP-II) 44 Mid-Term Evaluation Report – Public Document VI. OBSERVATIONS ON SHTP-II PROJECT DESIGN Building community-level primary healthcare capacity is a high priority for Southern Sudan. Therefore, SHTP-II is deployed at an appropriate level of the health system and is addressing critical needs. Counties selected for the project were identified in collaboration with MOH/GOSS. The project’s 14 counties are widely scattered around Southern Sudan and have varying implementation environments. It can be debated whether SHTP-II might have achieved more success and enjoyed greater scale economies had it concentrated its work in more counties across fewer states and been working with fewer SCPs. Of greater concern is the fact that the SHTP-II project design was extremely ambitious, especially given the project’s short implementation period of three years. The project design basically calls on SHTP-II to undertake an extensive array of initiatives all at once, which is usually a prescription for not doing anything well. The SHTP-II design appears to have suffered from a lack of prioritization across the full range of project activities identified for the project. In addition, too much focus may have been placed on achieving quantitative targets rather than supplying the SCPs with the tools needed to provide services and develop the capacity of Southern Sudan’s primary care system. Many services being established by SHTP-II require enormous investments in service delivery capacity building and community-based behavior change and communication that take time to produce results. It is unrealistic to expect that a family planning service offering women wide method choice, adequate compliance safe-guards, and good follow-up support can be established within a weak primary healthcare system in three years. It has taken most developing countries decades of concerted program effort, political commitment, and funding support to build strong family planning services, a lesson USAID has learned well in other regions of the world. Safe delivery services (including emergency obstetric care) also require considerable investments in midwifery training, clinical infrastructure, and essential drugs and equipment that go well beyond the capacity of Southern Sudan’s current primary healthcare system. Attempting to setup PMTCT services in primary care facilities, when so many clinics lack HIV testing equipment or functional laboratories capable of measuring CD-4 counts, is also a questionable near-term undertaking. The SHTP-II project design and the contractual expectations embedded in the SHTP-II contract may not have given sufficient attention to the implementation environment awaiting SHTP II and the SCPs. Poor road infrastructure and transportation facilities, flooding during the rainy season, and civil disturbances owing to Southern Sudan’s unpredictable security environment all make for a challenging implementation environment. Previous experience with SHTP-I should have produced greater appreciation for the constraints that typify post-conflict environments. SHTP-II has attempted to setup seven high impact services in primary care facilities that are often not adequately constructed, fully staffed, or properly equipped. The project was launched on worthy principles and given the project’s limited time horizon heroic efforts have been made to strengthen Southern Sudan’s health system and mobilize greater community involvement in primary healthcare. It is inexplicable why SHTP-II was limited to just three years given the ambition of the project design. Major implementation constraints impeding the achievement of SHTP-II objectives were, to a considerable extent, beyond the control of the project and most likely its budget. The Evaluation Team noted the following questionable assumptions that appear to have influenced the design of a complex and ambitious project and may have subsequently compromised the achievement of results:29 · Strategic objectives of SHTP-II could be achieved in three years. · Sufficient numbers of qualified staff would be available to work with SHTP-II and the SCPs as well as primary care facilities (e.g., certified community midwives) and that HHPs had already been trained and placed in the field in most counties. · All commodities, including drugs and vaccines, would be consistently available through the MOH/GOSS commodity logistics system. · Equipment and supplies could be procured easily through a centralized budget mechanism administered by the Prime Contractor (MSH). 29 However, it is also true that there have been delays in implementing the project that cannot be ascribed to hindrances resulting from the design of the project. Sudan Health Transformation Project Phase II (SHTP-II) 45 Mid-Term Evaluation Report – Public Document · Facilities taken over by SHTP-II would only need minor repairs. The infrastructure budget could be safely held to a minimum. · County Health Departments had the means to assure that all service delivery points had clean water supplies and functioning sanitation infrastructure in place, including on-site hand washing facilities, latrines, and proper arrangements for disposing medical waste. · Salary support for facility-based service providers and support staff would be increasingly provided by MOH/GOSS. · Contract negotiations between SHTP II and the SCPs would go smoothly and there would be no major delays in signing subcontracts, hiring staff, and procuring equipment. · Initial performance targets for the project were realistic, even though there were no baseline estimates for numerous county-level measures. · All SCPs would have the necessary skills (including how to move from a development to a relief operation mode) to implement all project components in three years (actually two years on average, given the long delays in executing SCP subcontracts). · Performance-based Contracting could be effectively managed and the quality of data reported by the project is adequate for documenting valid comparisons of SCP results. · Micro-Grants for supporting community-based health initiatives could be developed, implemented, and evaluated easily and quickly. Sudan Health Transformation Project Phase II (SHTP-II) 46 Mid-Term Evaluation Report – Public Document VII. RECOMMENDATIONS Recommendations for the Remainder of the Project The Evaluation Team believes there is still sufficient time for SHTP-II to make meaningful contributions to the cause of primary healthcare in Southern Sudan; the following recommendations on implementing the seven high impact services for the project, primary healthcare system strengthening, and management of SHTP-II pertain to the remaining year of the project. These recommendations are intended to better focus, prioritize, and set realistic expectations for project activities. Implementation of Seven High Impact Services During the remaining year of the project SHTP-II needs to give greater emphasis to high impact services that are currently seriously deficient. 1. The greatest priority should be given to addressing the glaring deficiencies in safe delivery services. The SHTP-II contract describes a “focused and intensive maternal program,” outlines a set of four activities, and provides an appropriate description of how to support EmOC, including use of misoprostol. It is recommended that this SOW serve as the basis for SHTP II’s scale-up of maternity care activities in the upcoming year. Additionally, SHTP II should support a clinical refresher course for community midwives who have already received training and been assigned to SHTP-II supported facilities. This refresher course would ideally be designed from existing materials that have already been developed by MSH or other organizations working in maternity care. An initial assessment of the knowledge and skills of community midwives needs to be undertaken, so that the training curriculum addresses what skills the trainees need to acquire. These midwives should be mentored at a clinical site with competent midwives (possibly using five to six experienced expatriate midwives) to ensure they have the skills to manage normal births, recognize complications, and manage emergencies and referrals. 2. Family planning will also require greater attention in the remaining year of the project. Training on the provision of family planning services should continue. As more family planning commodities are introduced, SHTP II and USAID must conduct continuous compliance training and monitoring – and they must provide evidence of this compliance effort. It will also be important for SHTP-II staff to become more involved in the work of the newly revitalized MOH Reproductive Health Technical Work Group in order to better coordinate national family planning program efforts. 3. During the remaining year of the project, SHTP-II should work to strengthen the WASH component of the project. The supply of clean water, sanitation facilities, and the management of medical waste remain substantial problems in most project sites visited by the Evaluation Team. SHTP-II should also develop additional indicators to measure progress in establishing hygienic environments in and around primary care facilities and properly disposing of medical waste through secure incinerators. The Initial Environmental Examination (IEE) planned for the SHTP-II Project (with assistance from USAID’s Office of Health and the Mission Environmental Officer) may help to clarify future needs in the areas of safe water, sanitation, and environmental quality. 4. Efforts should be made to reconfigure the HIV component of the project away from clinical services and more toward a community-based prevention approach. It may not be feasible to establish PMTCT services in many additional SDPs given the lack of HIV testing capacity in most SHTP-II counties. As there are many SCP staff who have been recently trained in HIV counseling (but are not working at full capacity due to delays in acquiring HIV testing kits), their skills would be best put to use as community prevention outreach workers. Their activities could include raising HIV awareness in local communities and reducing levels of high-risk behavior that encourage HIV transmission. SHTP-II’s HIV work should be concentrated in counties considered to have higher HIV prevalence. 5. Steps should be taken to improve the use of patient cards in all health facilities and ensure that children under five have a child health card for growth monitoring and EPI. These cards can be given to mothers to take home in order to have a record of services provided and information on when next visits for immunizations and other services are scheduled. Primary Healthcare System Strengthening 6. PHCU and PHCC infrastructure is still deficient in many counties where SHTP-II is engaged. Considerable facility construction and renovation work was begun during the years of SHTP-I (2004-2009), but these efforts were not always completed to an acceptable standard. SHTP-II, in consultation with USAID, should attempt to identify opportunities for undertaking additional construction and renovation work during the remainder of the project Sudan Health Transformation Project Phase II (SHTP-II) 47 Mid-Term Evaluation Report – Public Document (possibly in coordination with other USAID or donor projects). Consideration should be given to conducting a rapid field assessment of infrastructure needs, safe water and sanitation facilities, and environmental cleanliness at all service delivery points in the project’s 14 focus counties. This assessment would provide a clearer picture of current infrastructure deficiencies and suggest strategies for remedying deficiencies. However, the Evaluation Team acknowledges that remaining SHTP-II budgets are totally inadequate for addressing extant infrastructure needs. 7. USAID and SHTP II must assure the delivery of all equipment that has been ordered for the project. SHTP-II “hardware” is the most visible and tangible component of the project. Some SCPs were adamant that they would prefer to procure equipment and appear to have the capacity to do this. The SCPs have typically been waiting up to ten months for a process that likely could have been completed in one or two months if the SCPs had procured on their own. 8. Ensuring adequate staffing and the timely provision of salary payments continue to be enormous obstacles to the success of SHTP-II. USAID and SHTP II must play a greater advocacy role with high-level officials of GOSS, MOH/GOSS and state governments in working to improve these systemic problems. SHTP-II is currently providing significant budgetary support to salaries for facility-based service providers and support staff that could be better deployed for other purposes. 9. Further efforts are needed to strengthen the capacity of County Health Departments. Steps should be taken to develop integrated county health plans in consultation with all stakeholders in order to better organize supervisory schedules and responsibilities. Where CHDs are strong, the SCPs should be able to provide grants through a Memorandum of Understanding. CHDs should also be encouraged to develop annual workplans to be used to coordinate health activities, ensure greater harmonization of activities, and avoid duplication of effort. 10. SHTP II and the SCPs should intensify their reach beyond primary care facilities and strengthen community-based systems that have been put in place. VHCs, HHPs, and TBAs need more guidance on how to effectively promote healthy practices and raise the demand for services. They are critical for raising awareness within households and community organizations such as churches and schools. 11. Training remains a top priority as a means of building health system capacity. The following activities are recommended to strengthen the project: · A training plan should be developed based on prioritized needs from the field. The plan should include training methodologies for use at facilities and in communities, as well as measures to assess training effectiveness. The SCP field office/project managers should be invited to work with SHTP II in developing training plans. · Many training materials have been developed and used in the field. Where appropriate, these should be shared and incorporated into the standardized training materials SHTP II is developing. · Creative and cost-effective ways to conduct training should be pursued and a variety of training methodologies used. · Follow-up and supportive supervision of facility staff, VHCs, and HHPs will be needed to determine competency levels and the use of new knowledge and skills. · Greater mentoring of SHTP II staff (including SCP staff) by expatriate consultants now coming to work on SHTP-II implementation should become a routine part of the project’s mentoring efforts. 12. The Evaluation Team judged the IEC component of SHTP-II to be a missed opportunity at the mid-point of the project. Additional IEC work will be necessary, including developing more counseling tools for services providers and practical take-home instructional and practicum materials for patients. Specific recommendations include: · The creation of more standardized sets of IEC materials · Greater focus on pictures (not text); where text is necessary, consider local languages · Pre-test IEC materials before distribution and edit appropriately Sudan Health Transformation Project Phase II (SHTP-II) 48 Mid-Term Evaluation Report – Public Document · Train health workers on how to appropriately use IEC materials (“train” includes on-the-job mentoring, modeling and role playing, community health messaging, and waiting room discussions, etc.). · Require clear use of USAID branding and marking materials on all SHTP-II IEC materials 13. Greater efforts are still required to ensure the timely provision of drugs through the MOH/GOSS commodity￾logistics system. Attention should be paid to reviewing essential drug management procedures from regional or state to county health facilities. The essential drug kit should also be revisited in order to ensure that recommended emergency obstetric drugs are included and being introduced into the distribution system. The kit should also contain sufficient quantities of anti-malarial drugs and antibiotics for treating bloody diarrhea and ARIs. It would also be advisable to advocate for the creation of drug depots at the county level to ensure a continuous supply of drugs in the case of emergencies. 14. The project should work more closely with UNFPA to obtain maternity and family planning equipment/commodities until procurement issues within SHTP-II are resolved. Discussions should be held with the newly revitalized MOH Reproductive Health Technical Work Group about supplying misoprostol across Southern Sudan as part of efforts to strengthen safe delivery services. Population Services International (PSI) could possibly facilitate the registration, licensing, and procurement of misoprostol for Southern Sudan. 15. The capacity of the Sudan Inland Development Fund (SIDF), a local NGO working with Save the Children, is currently not adequate for supervising primary care facilities in the counties of Mvolo and Wulu. SHTP II needs to work with SAVE to address the problems that have developed with SIDF and chart a course back to functionality. SIDF was the only local NGO that received an SCP subcontract under SHTP-II. Management of SHTP-II 16. If SHTP-II is limited to its three-year implementation period, it must stay focused on activities that will achieve greatest impact. The value of the following activities are questionable for the remaining year of the project as they have the potential to take project staff off focus and add to administrative burdens: · Starting Fully Functional Service Delivery Points · Implementing Performance-based Contracting (PBC) · Proceeding with Micro-Grants for Community-based Activities · Continuing PMTCT and other HIV prevention activities as planned. 17. Performance-based Contracting (PBC) requires valid reporting of results and fully functional SHTP II and SCP supervisory support for M&E activities. The Evaluation Team questions the ability of SHTP II to implement an effective incentive-based PBC system in Southern Sudan’s current implementation environment. It is not recommended that an incentive-based PBC system be started if the project has only one year of additional implementation time remaining. Greater emphasis should instead be given to improving the quality and timely reporting of performance indicators. 18. The Micro-Grants program has been slow to develop. If the project has only one additional year of implementation time, it is recommended that work on the micro-grants be discontinued. Instead, the SHTP II community mobilization expert should focus on working with the SCPs to improve and expand community-level activities, including efforts of VHCs and HHPs to improve health awareness and behaviors in local communities. These activities are more likely to show results by the end of the project. The SCPs have requested this assistance to increase the impact and success of their community-based activities. 19. Additional attention is needed to ensure the validity of performance data reported by the project’s M&E system. Field validation checks by SHTP II staff should occur once every quarter in all counties. SCP staff responsible for clinic recordkeeping and summary reporting need to be visiting every PHCU and PHCC in their counties at least once per month (or more frequently for facilities with a history of reporting problems). 20. As part of field validation efforts, greater attention should be paid to the quality of demographic estimates for catchment areas and/or counties that are used as denominators in the construction of population-based indicators (usually relying on percentages and rates of change). This problem is somewhat intractable given the paucity of local￾area demographic information in Southern Sudan. Whenever possible, reliance should be placed on census estimates of Sudan Health Transformation Project Phase II (SHTP-II) 49 Mid-Term Evaluation Report – Public Document county populations even though these are probably too low in many instances owing partly to the large influx of Southern Sudanese from the north and neighboring African countries in recent months. 21. Communication, transparency and the sharing of information are critical for the success of the project – particularly a complex project like SHTP-II. The following activities are recommended to strengthen these important functions: · Share additional contractual, budgetary, and project performance information about the project with MOH/GOSS and attempt to meet and communicate on a more regular basis. · SHTP II and USAID (the COTR and Contracts Office) need to consult more frequently on a pre-set schedule, given the implementation delays that have occurred during the first 18 months of the project. 22. USAID’s oversight of SHTP-II implementation (including project staffing, procurement of equipment, and contractual adherence) has been affected by the rapid turnover of direct hire staff, particularly in the Office of Health and in the Contracts Office. In order to ensure more effective managerial continuity for SHTP-II within the Mission, one or more FSN staff should be mentored to provide greater responsibility for working with SHTP II in monitoring progress and working to alleviate implementation problems that may be impeding the success of the project. The Evaluation Team also appreciates the heavy administrative workloads that all-too-often encumber USAID’s direct hire staff, but personnel from the Office of Health must get to the field more often to become better informed about SHTP-II achievements, implementation constraints, and remaining challenges for the project. 23. SHTP II should work to improve its relationships with the SCPs by holding regular meetings and improving routine communication; supporting them by addressing high level issues affecting service delivery (salaries/incentives, construction/renovation, etc.); and sharing feedback arising from field visits. 24. SHTP II and the SCPs should conduct quarterly supervision visits in each SHTP-II county. GOSS staff from Juba should be invited to participate in these site visits. In addition, the SCPs should make efforts to involve the County Health Directors and their staff in these quarterly supervisory site visits. USAID should join at least one of these supervisory visits every quarter during the remaining year of the project. Efforts should be made to revise and standardize supervisory tools that currently exist based on MOH guidelines. All supervisory teams should be trained in their use to ensure consistency. Feedback to the counties should be undertaken prior to leaving the facility and a final report submitted within one week of the visit. 25. The Evaluation Team noted that some counties were performing better than others. It would be useful for the SCPs to visit different counties and undertake comparative peer reviews of successful implementation. For example, the lessons learned and best practices found in Malakal should be reviewed and shared so that other counties and SCPs can benefit from this experience. This approach could help guide and motivate counties that are not performing up to expectation. 26. Within primary care facilities, the SCPs should organize quarterly exchange workshops for government and SCP facility-based staff on key best practices for enhancing service quality (including client satisfaction) and effectiveness. Different key interventions (e.g., malaria control and ARI diagnosis and treatment) could be given special attention at each of these workshops. 27. The SHTP-II funding split should be maintained as originally planned – 75% to the SCPs and 25% to the prime contractor (MSH). Funding levels for the SCPs cannot be compromised during the last year of the project. In developing new NGO subcontracts, flexibility needs to shown in allowing subcontracting partners to make reasonable and appropriate requests for equipment and supplies based on the particular constraints of their counties. 28. The Evaluation Team recognizes that SHTP-II is running short of funding and may not be able to finish the project over the remaining year of the contract. Therefore, the Evaluation Team recommends that USAID provide SHTP-II with more time and resources in order to focus on key areas with high potential for results. There should be carefully constructed guidelines for any project extension – agreed upon by the USAID Mission, MOH/GOSS, SHTP II, and the SCPs – concerning priority activities, implementation plans, and regular monitoring activities that will help ensure the project can fulfill its promise. Sudan Health Transformation Project Phase II (SHTP-II) 50 Mid-Term Evaluation Report – Public Document ANNEX 1: SCOPE OF WORK FOR THE MID-TERM EVALUATION OF SHTP-II Statement of Work (SOW) for Midterm Evaluation Sudan Health Transformation Project II PROJECT IDENTIFICATION DATA 1. Project Title: Sudan Health Transformation Project II (SHTP-II) 2. Project Number: GHS-I-00-07-00006-00 3. Project Dates: February 11, 2009 to February 10, 2012 4. Project Funding: $44,297,880 5. Implementing Organization: Management Sciences for Health 6. Contracting Officer’s Technical Representative (COTR): Charles Lerman I. General Specification of the Evaluation USAID/Sudan requests technical assistance from Management Systems International (MSI) to design and undertake a midterm evaluation of the Sudan Health Transformation Project II (SHTP-II). The main objective of SHTP-II is to enhance the decentralization of primary health service to improve the health status of the Southern Sudanese people. SHTP-II directly contributes to the USAID Investing in People Objective through the provision of the following seven high-impact services and practices:  Child Health  Nutrition  Malaria  Hygiene and Sanitation Practices  Maternal Health  Family Planning  Prevention of HIV/AIDS SHTP-II aims to achieve the following key results: Result 1: Expanded access/availability of high impact services and practices; Result 2: Increased Southern Sudanese capability to deliver and manage services; and Result 3: Increased knowledge of and demand for services and healthy practices To achieve these results, SHTP-II focuses on service delivery and community mobilization as well as health systems strengthening. The USAID/Sudan Office requests that the evaluation be completed by November 10, 2010 so that the findings, conclusions and recommendations can be used to inform the design of a follow-on project. Evaluation findings will also inform the development of the health section of a new Mission strategy and the design of a sector-wide health approach by all government and development assistance partners in the post-referendum environment. II. Background SHTP-II is managed by Management Sciences for Health. Its start date was February 11, 2009 and its end date is February 10, 2012. It partners with the International Rescue Committee (IRC) and subcontracts with nongovernmental Sudan Health Transformation Project Phase II (SHTP-II) 51 Mid-Term Evaluation Report – Public Document organizations (NGO) for delivering health services in 14 counties in all ten states of Southern Sudan. Table 1 shows SHTP-II’s subcontracting partners (SCP) and their geographic locations. TABLE 1: SHTP-II STATES, COUNTIES, AND SUBCONTRACTING PARTIES S/N State County Subcontracting Partner 1 Central Equatorial State Juba Adventist Development and Relief Ass. (ADRA) 2 Central Equatorial State Terekeka ADRA 3 Eastern Equatorial State Kapoeta North Save the Children 4 Lakes State Wulu Save the Children 5 Western Equatorial State Mvolo Save the Children 6 Western Equatorial State Mundri East Mundri Relief and Development Ass. (MRDA) 7 Western Equatorial State Mundri West Action Africa Help International (AAH-I) 8 Western Equatorial State Tambura International Medical Corps (IMC) 9 Upper Nile Malakal IMC 10 Jonglei Twic East CARE 11 Unity Panyijar International Relief Committee (IRC) 12 Northern Bahr El Ghazal Aweil South IRC 13 Western Bahr El Ghazal Wau John Snow Incorporated (JSI) 14 Warrap State Tonj South Comitato Collaborazione Medica (CCM) It seeks to establish fully functional service delivery points with coverage of 80% of the catchment population within the three years of the project life. Its target groups include children, women of reproductive age, pregnant women, and the general population. III. Purpose and Objectives The purpose of this activity is to implement a mid-term evaluation of SHTP-II in October and November 2010 shortly after the midpoint of its three-year agreement. Findings, conclusions, and recommendations from this evaluation will be used to inform a new program design. The objectives of this evaluation are to: 1. Assess program performance in meeting targets and accomplishing its three key objectives. 2. Determine how well the Prime recipient has met the terms and conditions of the contract – taking into consideration how the contact is aligned with GOSS/MOH priorities, 3. Assess how the program has supported the transition from relief to development, specifically the systems strengthening component and how effective the overall model seems to be. 4. Make recommendations to improve the current program and assist the design of the next program, including identifying lessons learned. IV. Review Methods The review will be carried out by a Team Leader who will be assigned a counterpart from the Ministry of Health. Other members of the team will consist of infectious disease (HIV/AIDS, malaria, and TB), maternal and child health, family planning, and health systems technical assistance (TA) providers. (One TA provider can specialize in more than one area.) Officers from the World Bank, the Basic Services Fund, and the Joint Donor Office might also join the evaluation teams. The team leader and team members should employ methodologies that collect quantitative and qualitative information. Multi-functional teams from the government, USAID, development assistance partners, and evaluation team members will conduct site visits to County Health Departments and facilities in areas operated by all nine subcontractors. The final design of the evaluation will take place during the preparation sessions leading up to the evaluation. The following key questions will be answered during the evaluation: Sudan Health Transformation Project Phase II (SHTP-II) 52 Mid-Term Evaluation Report – Public Document 1. Are there any issues with respect to project design and assumptions (documented or implied) that should be reconsidered based on experience to date? For example: A. Does SHTP-II have well-reasoned objectives? Are these objectives still valid or should there be a reformulation of them? B. Were the identification and definition of indicators satisfactory in the original contract? C. Were the number and location of geographic sites appropriate in the context of achieving results? D. Were appropriate targets set? E. MOH involvement and alignment with GOSS priorities? 2. Has the prime recipient met the terms and conditions of the contract? Did the subcontractors meet the terms and conditions of their subcontracts? (This question will be answered to the extent possible based on availability of subcontractor contract documents) 3. Is a cost-reimbursement, fixed fee contract a satisfactory contracting mechanism for the prime contractor? Are performance-based subcontracts reasonable for the subcontractors? 4. Do submitted reports meet contract requirements and program needs? For Example: A. Were workplans well-designed and do they correspond to the work that was actually undertaken? B. Were the budgets reasonable given pipelines and burn rates? 5. Does the project’s approach to subcontractor and government capacity building actually build their capacities in a meaningful way? For example: A. Has the program’s relationship with the MOH/GOSS been productive and engendered trust? B. Does the program serve on management and technical working groups? How does it interact with other major donor programs such as the Multi-Donor Trust Fund and the Basic Services Fund? C. Has the program been able to transfer lessons learned to other partners? Has the program been able to leverage resources and support from other programs? D. Has the program demonstrated success with its microgrants program? 6. How successful has the project been in implementing activities set in their work plan? How closely do these activities align with the activities outlined in the projects technical proposal and recommendations given in the SHTP-I Evaluation? Project Management 7. How effective are management structures (including HR structures, communication, and overall management of the contracts) between MSH HQ, MSH Juba, subcontractors, and USAID. For example: A. Has support from the prime recipient’s headquarters met the program’s needs (human resources, support from backstop, financial systems, contracts and grants management, communications, etc.)? For example: B. Did HQ empower the Chief of Party? C. Does the prime recipient have effective management practices within its Juba-based team, and in its relations with subcontractors? D. How well to USAID/Sudan staffing structures support this type of project? 8. Did the program develop a staff development and mentoring plan consistent with building the sustainable capacity of the staff? For example: A. Do both the expatriate staff and the indigenous staff have the skill sets necessary to successfully carry out the project? 9. Was the project’s financial management adequate? (This question will be answered to the extent possible based on budget information/documents available) For example: Sudan Health Transformation Project Phase II (SHTP-II) 53 Mid-Term Evaluation Report – Public Document A. Were program direct costs, such as labor, office space, non-expendable and expendable equipment and supplies, information technology, and banking and financial transactions satisfactory? B. Has procurement of supplies, equipment and vehicles for subcontractors been handled effectively and efficiently? C. Does the project have an effective micro-grants management plan? 10. Has the program established a viable M&E system (including data validation)? For example: A. Is the program providing feedback reports to the subcontractors and, if so, are they being used to improve the program? B. Is the M&E Plan being implemented and kept up to date? C. Is data disaggregated where appropriate? To the extent possible, the evaluation team should address the following question: 11. Is the project worth the level of investment, as far a value for money? V. Procedures Document Review USAID/Sudan will provide the entire team with key documents before the start of in-country work for their review. Team Planning Meetings The team members will have an initial meeting and map out a detailed implementation plan. They will decide activities, sub-activities, responsible parties, team composition, and milestones and deliverables for the exercise. Team members will then produce quantitative and qualitative interview instruments and schedule and organize the field visits. The team will keep the Health Team Leader informed about the progress of events on a daily basis. Interviews and Site Visits Key informant interviews will include but not limited to: · USAID Mission staff, including relevant members from the Front Office, Health/WASH Team, and the Program Office · Prime Recipient Management and Technical/Financial Officers · Subcontractor Management and Technical/Financial Officers in Juba and the field · Government of Southern Sudan Ministry of Health · County Health Departments · Village Health Committees · Micro-grant recipients · Counterpart Agencies and Projects (Basic Services Fund, Multi-Donor Trust Fund, UNICEF, PSI, etc.) · Beneficiaries (customer satisfaction survey) VI. Illustrative Schedule of Events Below is an illustrative list of the specific tasks to be accomplished by the team, with an estimated level of effort for each task. A six-day work week is authorized for Southern Sudan. VII. Team Composition The Midterm SHTP-II Evaluation core team will be composed of Andrew Kantner (Team Leader), Mary Harvey (Team Member), Deborah Armbruster (Team Member), Anna McCrerey (Team Member) as well as representatives from USAID/Sudan and the Government of Southern Sudan. A representative from the Basic Services Fund – Geertruid Kortmann will also join the team for a portion of the field work. Sudan Health Transformation Project Phase II (SHTP-II) 54 Mid-Term Evaluation Report – Public Document The team leader is expected to work with other team members to develop a plan for conducting the evaluation, including interview guides or other tools as necessary, and a schedule for its timely completion. The core team members are expected to develop the deliverables, including taking responsibility for writing the report. Qualifications for the external evaluators: 1. At least fifteen years of experience assessing or evaluating USAID-supported health projects 2. Previous experience serving as a Team Leader on a USAID-supported health project 3. Previous experience working in Africa 4. Experience in facilitation and providing leadership in collaborative and participatory evaluations with multiple stakeholders 5. Experience in arranging meetings, setting up travel schedules for field visits, and reporting on meeting outcomes (although significant logistical support will be provided by the SUPPORT team in Juba) 6. Extensive experience in leading stakeholder meetings 7. Excellent verbal and writing skills 8. Ability to produce preliminary and final reports on time VIII. Schedule and Logistics The in-country phase of the review will be conducted over a period of up to 30 days with a desired start date of October 14, 2010. The USAID/Sudan Health Team Leader, in conjunction with MSI, will arrange all of the interviews, meetings, site visits and debriefings in advance. MSI will be responsible for producing any PowerPoint presentations. It will also be responsible for the final report, including its dissemination. IX. Period of Performance Field work is to be carried out over a period of approximately 4 weeks, beginning on or about (o/a) Oct 14, 2010 and concluding o/a Nov 15, 2010. Project Level of Effort (LOE) Tasks (all External Evaluators unless otherwise noted) Work Days (6-day work week in Sudan, 5-day work week outside Sudan) Initial Preparation – Review background documents, etc. Travel to Juba 6 –prep; 2 -travel Team Planning Meeting – including a brief (receive feedback) from USAID, meet with subcontractors and stakeholders 6 Field Work – including briefings, site visits draft report preparation, and final debriefs/presentations. 24 Return Travel 2 Final Report Preparation in home country (only Team Leader) –incorporate feedback, complete final report and submit to MSI 5 Total days for Team Leader 45 (Team Leader) Sudan Health Transformation Project Phase II (SHTP-II) 55 Mid-Term Evaluation Report – Public Document X. Deliverables 1. Evaluation Materials (topic list, list of geographic areas and field sites, and list of respondents) 2. Juba Interview Notes 3. Stakeholder’s Meeting PowerPoint Presentation (two hard copies and one electronic copy) 4. Draft Report (two hard copies and one electronic copy) 5. Final Report (including recommendations) (one electronic copy) The draft report will be due one week after the departure of the evaluation team from Juba. The final report will be due one month of the evaluation team from Juba after receiving comments from USAID/Sudan and other stakeholders. Upon final approval of the content by USAID/Sudan, MSI will be responsible for editing and formatting the final report, which takes approximately 30 days. The final report in both hard (5 hard copies) and electronic format will be submitted to USAID/Sudan. XII. Mission Contact Person Charles Lerman, Health Team Leader, USAID/Sudan, clerman@usaid.gov XIII. Additional Objectives/Focus Areas for Consideration by Mid-Term Evaluation 1. Program performance in accomplishing its three key objectives: (1) expanded access and availability of health services and practices, (2) increased capacity of Southern Sudanese to deliver and manage services; and (3) increased knowledge of and demand for services and healthy practices among the population. 2. Prime contractor performance against (1) the terms and conditions of the prime contract and (2) the activities in the prime contractor’s workplan. Subcontractor performance in meeting the terms and conditions of their subcontracts. 3. Prime recipient success in managing the performance-based contracting and financing mechanism with its subcontractors. 4. Ability of the prime recipient to establish core indicators and use these indicators to measure results. 5. SHTP-II’s strengths, weaknesses, challenges to implementation, and development of creative, problem-solving solutions to those challenges. Challenges examined should include, at a minimum: health facility staff salaries, drug availability in the health facilities, infrastructure, provision of emergency transportation, and limited functionality of County Health Departments. 6. Timely development and implementation of a monitoring and evaluation plan by the prime recipient and among subcontractors to measure and achieve performance against targets. 7. Subcontractor performance against the terms and conditions of the subcontract. 8. Subcontractor performance in raising the skills of health clinic service providers. 9. Subcontractor performance in working with and building the capacity of County Health Departments. 10. Prime recipient success in working with and supporting the GOSS Ministry of Health programs, and State Ministries of Health 11. Prime recipient and subcontractor success in facilitating community mobilization and engagement in health awareness, behavior change and service use 12. Village Health Committee engagement and activities 13. Outcomes of prime recipients efforts to implement the gender components of the contract 14. Community and client satisfaction with the program. 15. Prime recipient effectiveness in creating linkages with and leveraging resources from other stakeholders, such as counterpart Implementing Partners, other donor-supported projects, and UN technical agencies. 16. Prime recipient’s organization and support functions (recruitment and staffing, subcontract management, microgrant management, procurements, and financial management) 17. Relationship between the prime recipient in Southern Sudan and its headquarters in the United States and the number and quality of its short-term technical assistance. Sudan Health Transformation Project Phase II (SHTP-II) 56 Mid-Term Evaluation Report – Public Document ANNEX 2: FINAL CONTRACTUAL PERFORMANCE INDICATORS AND TARGETS30 Table 2.1: Final Contractual Performance Indicators and Three Year Targets for SHTP-II 2009 Target 2010 Target 2011 Target Child Health Number of children less than 12 months of age who received DPT 3 from USG supported programs 16,500 20,750 25,913 Percentage of children less than 12 months of age who received DPT3 from USG supported programs 16.8% 20.8% 35.0% Nutrition Number of children under 5 years of age who received Vitamin A from USG supported programs 18,900 30.400 64,873 Percentage of children under 5 years of age who received Vitamin A from USG supported programs 35% 45% 50% Malaria Percentage of pregnant women who receive IPT2 as part of the ANC visit Indicator not in original contract 50% 60% Maternal Health Percentage of women with one ANC visit 60% 65% 70% Percentage of women with at least four ANC visits 15% 20% 30% Percentage of deliveries with a skilled attendant in USG supported programs 10% 15% 20% Percentage of deliveries by trained traditional birth attendant (TBA) or Maternal and Child Health Worker (MCHW) in USG supported counties 30% 40% 45% Family Planning Number of counseling visits for FP/RH as a result of USG assistance 10,000 15,000 20,000 Hygiene and Sanitation Liters of drinking water disinfected with USG supported point-of-use treatment products 130 mil 140 mil 150 mil Health System Strengthening Number of health personnel trained with USG support in the following program areas: Malaria management, FP/RH, EPI, Diarrhea and ARI, HIV/AIDS, hygiene and sanitation leadership and governance, M&E procedures 1,000 1,690 1,700 Number of people trained in malaria treatment or prevention with USG funds 150 Merged into aggregate training indicator Merged into aggregate training indicator Number of health personnel trained in EPI, diarrhea and ARI management with USG 150 Merged into aggregate Merged into aggregate training 30 USAID commented on 15 March 2011; “2011 targets were later revised based on 2010 results and 2011 approved work-plan after the evaluation team completed the data collection” Sudan Health Transformation Project Phase II (SHTP-II) 57 Mid-Term Evaluation Report – Public Document support training indicator indicator Number of individuals trained in good health and hygiene practices with USG assistance 800 Merged into aggregate training indicator Merged into aggregate training indicator Indicators Currently Being Negotiated Number of pregnant women with known HIV status (includes those who were tested for HIV and received their results Number of HIV-positive pregnant women who received anti-retrovirals to reduce the risk of mother-to-child transmission of HIV Number of individuals from target audience who participated in community-wide event Indicators that will no longer be used starting in FY 2011 Number of ITNs distributed to USG-supported counties Number of people trained in malaria treatment or prevention with USG funds Number of individuals trained in good health and hygiene practices Number of health personnel trained in immunization, diarrhea and ARI management Number of USG-assisted service points experiencing stock-outs of specific tracer drugs Percentage of USG-assisted service delivery points experiencing stock-outs of specific tracer drugs Sudan Health Transformation Project Phase II (SHTP-II) 58 Mid-Term Evaluation Report – Public Document Table 2.2: Quarterly and Cumulative FY 2010 Results for Final Contractual Indicators Target FY 2010 Q1 Oct￾Dec 2009 Q2 Oct￾Dec 2009 Q3 Oct￾Dec 2009 Q4 Oct￾Dec 2009 Cumulative Performance FY 2010 % Target Achieved Child Health Number of children less than 12 months of age who received DPT 3 from USG supported programs 20,750 2,941 3,898 3,228 4,986 15,053 72.5% Percentage of children less than 12 months of age who received DPT3 from USG supported programs 20.8% 5.9% 7.5% 6.2% 9.6% 29.2% 139.1% Nutrition Number of children under 5 years of age who received Vitamin A from USG supported programs 30.400 3,452 42,065 4,915 16,091 66,523 284.3% Percentage of children under 5 years of age who received Vitamin A from USG supported programs 45% 2.7% 32.5% 3.8% 12.4% 51.4% 114.2% Malaria Percentage of pregnant women who receive IPT2 as part of the ANC visit 50% 21.8% 48.7% 53.6% 63.0% 56.3% 112.6% Maternal Health Percentage of women with one ANC visit 65% 6.4% 14.0% 15.5% 16.8% 52.7% 81.1% Percentage of women with at least four ANC visits 20% 1.5% 2.8% 6.5% 7.2% 18.0% 90.0% Percentage of deliveries with a skilled attendant in USG supported programs 15% 0.5% 2.2% 2.4% 1.9% 7.0% 46.6% Percentage of deliveries by trained traditional birth attendant (TBA) or Maternal and Child Health Worker (MCHW) in USG supported counties 40% 2.7% 6.0% 6.8% 7.4% 22.9% 57.3% Family Planning Number of counseling visits for FP/RH as a result of USG assistance 15,000 0 85 961 2,937 3,983 26.6% Hygiene and Sanitation Liters of drinking water disinfected with USG supported point-of-use treatment products 140 million 37.4 million 28.3 million 92.3 million 31.5 million 189.6 million 135.4% Health System Strengthening Number of health personnel trained with USG support in 1,690 37 85 495 942 1,559 92.3% Sudan Health Transformation Project Phase II (SHTP-II) 59 Mid-Term Evaluation Report – Public Document the following program areas: Malaria management, FP/RH, EPI, Diarrhea and ARI, HIV/AIDS, hygiene and sanitation, leadership and governance, M&E procedures Number of community members trained with USG support 2,500 0 215 1,445 626 2,286 91.4% Percentage of all health facilities that provide at least of the 5 of the 7 high impact services using MOH approved standards 94% n/a n/a 72.0% 77.3% 77.3% 82.2% Percentage of USG supported health facilities that submit their HMIS monthly reporting forms within one-month of the reporting period 90% n/a 70.0% 82.0% 94.4% 94.4% 104.9 Indicators Currently Being Negotiated Number of pregnant women with known HIV status (includes those who were tested for HIV and received their results None set 0 0 491 1100 1591 NA Number of HIV-positive pregnant women who received anti-retrovirals to reduce the risk of mother￾to-child transmission of HIV None set 0 0 0 22 22 NA Number of individuals from target audience who participated in community￾wide event 150,000 NA NA NA NA NA NA Indicators that will no longer be used starting in FY 2011 Number of ITNs distributed to USG-supported counties 151,698 886 16,277 6,382 12,804 36,349 24.0 Number of people trained in malaria treatment or prevention with USG funds 150 0 49 114 123 286 445 Number of individuals trained in good health and hygiene practices 520 18 347 70 212 647 124.4 Number of health personnel trained in immunization, diarrhea and ARI management 150 53 16 335 287 691 423.3 Number of USG-assisted service points experiencing stock-outs of specific tracer drugs 35 0 0 21 70 - - Percentage of USG-assisted service delivery points experiencing stock-outs of specific tracer drugs 48.0 NA NA 21.0 42.9 42.9 89.4 Source for Table 2., Management Science for Health. 2010. Sudan Health Transformation Project (SHTP-II): Performance Report for FY 2010, pp. 40-42. Sudan Health Transformation Project Phase II (SHTP-II) 60 Mid-Term Evaluation Report – Public Document Table 2.3: Cumulative FY 2010 Results for Final Contractual Indicators by County Mvolo/ Wau Kapoeta North Panyijar Mundri West Aweil South Tambura Juba Child Health Number of children less than 12 months of age who received DPT 3 from USG supported programs 603 310 1261 1704 1127 1584 1786 Percentage of children less than 12 months of age who received DPT3 from USG supported programs 22.1% 7.4% 189.1% 122.6% 37.3% 69.1% 13.4% Nutrition Number of children under 5 years of age who received Vitamin A from USG supported programs 3952 7676 2583 6927 6584 382 960 Percentage of children under 5 years of age who received Vitamin A from USG supported programs 43.6% 72.8% 154.8% 199.4% 87.3% 6.7% 2.9% Malaria Percentage of pregnant women who receive IPT2 as part of the ANC visit 127.4% 4.9% 58.6% 66.3% 70.7% 74.8% 40.5% Maternal Health Percentage of women with one ANC visit 38.7 33.1 229.1 133.1 66.6 154.9 27.3 Percentage of women with at least four ANC visits 12.5 1.3 40.3 63.4 31.2 39.3 21.7 Percentage of deliveries with a skilled attendant in USG supported programs 6.7% 3.1% 15.5% 32.3% 1.8% 38.5% 2.3% Percentage of deliveries by trained traditional birth attendant (TBA) or Maternal and Child Health Worker (MCHW) in USG supported counties 33.6% 12.1% 111.2% 71.6% 38.6% 75.6% 4.2% Family Planning Number of counseling visits for FP/RH as a result of USG assistance - - - - - - - Hygiene and Sanitation Liters of drinking water disinfected with USG supported point-of-use treatment products - - - - - - - Health System Sudan Health Transformation Project Phase II (SHTP-II) 61 Mid-Term Evaluation Report – Public Document Strengthening Number of health personnel trained with USG support in the following program areas: Malaria management, FP/RH, EPI, Diarrhea and ARI, HIV/AIDS, hygiene and sanitation, leadership and governance, M&E procedures 52 93 61 109 55 325 71 Number of community members trained with USG support 602 614 0 11 202 323 17 Percentage of all health facilities that provide at least of the 5 of the 7 high impact services using MOH approved standards - - - - - - - Percentage of USG supported health facilities that submit their HMIS monthly reporting forms within one-month of the reporting period - - - - - - - Sudan Health Transformation Project Phase II (SHTP-II) 62 Mid-Term Evaluation Report – Public Document Table 2.3 (Continued): Cumulative FY 2010 Results for Final Contractual Indicators by County Malakal Twic East Wau Tonj South Mundri East Terekeka Total Child Health Number of children less than 12 months of age who received DPT 3 from USG supported programs 2,045 1,278 777 581 849 968 15,053 Percentage of children less than 12 months of age who received DPT3 from USG supported programs 39.5% 36.6% 16.1% 16.4% 43.0% 24.8% 29.3% Nutrition Number of children under 5 years of age who received Vitamin A from USG supported programs 17,855 2,101 39 6,401 10,402 661 66,523 Percentage of children under 5 years of age who received Vitamin A from USG supported programs 138.1% 24.1% 0.3% 72.3% 210.5% 6.8% 51.7% Malaria Percentage of pregnant women who receive IPT2 as part of the ANC visit 53.2% 47.3% 42.8% 48.3% 58.0% 53.5% 56.3% Maternal Health Percentage of women with one ANC visit 42.3% 58.3% 62.8% 70.9% 57.9% 24.0% 52.9% Percentage of women with at least four ANC visits 11.1% 17.7% 14.6% 12.6% 15.5% 7.3% 18.1% Percentage of deliveries with a skilled attendant in USG supported programs 6.0% 4.5% 2.5% 10.4% 6.2% 8.1% 6.9% Percentage of deliveries by trained traditional birth attendant (TBA) or Maternal and Child Health Worker (MCHW) in USG supported counties 8.1% 49.3% 6.0% 24.7% 37.8% 20.0% 22.8% Family Planning Number of counseling visits for FP/RH as a result of USG assistance - - - - - - - Hygiene and Sanitation Liters of drinking water disinfected with USG supported point-of-use treatment products - - - - - - - Health System Strengthening Sudan Health Transformation Project Phase II (SHTP-II) 63 Mid-Term Evaluation Report – Public Document Number of health personnel trained with USG support in the following program areas: Malaria management, FP/RH, EPI, Diarrhea and ARI, HIV/AIDS, hygiene and sanitation, leadership and governance, M&E procedures 108 79 25 85 98 260 1,533 Number of community members trained with USG support 11 15 0 0 0 279 2,121 Percentage of all health facilities that provide at least of the 5 of the 7 high impact services using MOH approved standards - - - - - - - Percentage of USG supported health facilities that submit their HMIS monthly reporting forms within one-month of the reporting period - - - - - - - Source for Table 2.3, Management Science for Health. 2010. Sudan Health Transformation Project (SHTP-II): Performance Report for FY 2010, pp. 43-44. Sudan Health Transformation Project Phase II (SHTP-II) 64 Mid-Term Evaluation Report – Public Document ANNEX 3: SUMMARY MATRIX FOR EVALUATION TEAM FIELD OBSERVATIONS County/Partner Infrastructure Staffing (including salary information) Capacity building Procurement Patients referral mechanism M&E and data quality Community involvement and the activities of VHC Distribution and utilization of IEC materials Health System Strengthening Mundri East/ MRDA Most of facilities were constructed by other INGOsin mid 70s, and 90s. PHCU in Bitti operats in dilapidated tukul. Members acquired building materials and were begging AAHI for construction PHCU assistance. Extraordinarily dedicated facility staff. No SCP’s M&E officer. 1/3-1/2 of all staff are paid by the government, while the remaining staff are paid by the SCP No standardized curriculum of training except for LDP and management training; no on job￾training. Most training took place in Mundri town, which took staff away from their work at facilities. MSH removed budget for institutional trainings. No evidence of procurement by MSH for the SCP, except one vehicle in Aug. MSH still has many essential types of equipment stalled in the procurement phase. SCP has logistical capacity to procure its proposed equipment in 1 week-1month. The lack of equipment had a negative service delivery impact on all facilities. There were data errors present at all facilities visited. No consensus on GOSS OR MSH'S intended methodology to collect, monitor or manage data quality. MSH showed no assistance to mentor SCPS, and/or have a regular site visits. There are very little community￾based healthcare activities in the communities. More work seemed to be shouldered by non-SHTP-II affiliated CBDs (community based distributors). Many CHPs & HHPs were not active. Few IEC materials at facilities, and mostly displayed at around time of field visit. No IEC materials in local languages IECs were much too text-heavy for communities with low literacy levels. Many IECs are lacking. CHD has little engagement with health facilities and communities - transport is the main reason. No coordination at the county level for the various organizations operating in the counties. Sudan Health Transformation Project Phase II (SHTP-II) 65 Mid-Term Evaluation Report – Public Document Mundri West/ AAH-I Most of facilities were constructed by other INGOs in mid 70s, and 90s. Skeleton size of program staff, and no M&E Officer. Extraordinarily dedicated facility staff No standardized curriculum of training except for LDP and management training; no on job￾training. Most training took place in Mundri town, which took staff away from their work at facilities. MSH removed budget for institutional trainings. No evidence of procurement by MSH for the SCP, except one vehicle delivered in August. SCP submitted extensive equipment requests in February and is still waiting for their materials. Many essential equipment still stalled in the procurement phase. . There were data errors present at all facilities visited. No consensus on GOSS OR MSH'S intended methodology to collect, monitor or manage data quality. MSH showed no assistance to mentor SCPS, and/or have a regular site visits. Few IEC materials at facilities, and mostly displayed at around time of field visit. No IEC materials in local languages IECs were much too text-heavy for communities with low literacy levels. Many IECs are lacking. CHD has little engagement with health facilities and communities - transport is the main reason. No coordination at the county level for the various organizations operating in the counties. Tonj South/ CCM Tonj has 7 PHCCS and 7 PHCUs. A cold room with six refrigerators and two freezers was constructed by UNICEF in Thiet 69 staff (65 paid by SCP). SCP agreed to pay staff above the GOSS scale. Lack of State health funds for salaries There on￾the-job training and supervision given to staff (14 CHWs). SCP sent two CHWs to a GOAL health training institute. Leadership development training was well-liked by CHD There was adequate drug supply at each facility. Extreme delay in procurement of basic supplies. Inadequate commonly used malaria and pneumonia drug. SCP has supplied all facilities with MOH patient registers and is conducting on the job training and monitoring in the correct use. SCP plans to recruit M&E Officer to strengthen internal M&E VHC members know their roles and responsibilities, spend time at eh health facility, sign off when drugs arrive, engage in enforcement of not steaing drugs Excellent coordination and collaboration between SCP and CHD. SCP's supervision is quite strong. Sudan Health Transformation Project Phase II (SHTP-II) 66 Mid-Term Evaluation Report – Public Document Mvolo/ Save The Children Mvolo has 2 PHCCs, 8 PHCUs - 2 are finished. PHCC labs under￾equiped and lacking essential reagents for testing. All facilities supported by SCP are part of government system. Government salary payments are a major problem. Only one MCHW at the PHCCs and PHCUs visited. TA for training provided by SCP is consistent with government guidelines. However, training guidelines for PHCCs and PHCUs staff are still not standardized and no refresher training for staff. There was drug shortage at some PHCUs. Yeri has all vaccines now. No phenobarbital for limiting nodding disease. No paracheck kits in any lab facilities visited. Motor bikes were to be provided to PHCCs but none were in sight. County health office compiles facility health reports, submits to State level. Facility workers reported that forms of SHTP-I were easier to use than forms introduced by MSH. "Stockers" not available in pharmacy to track record of drug inventory. TBAs are working in Mvolo County but the extent to which they have been trained in midwifery is not clear Wau/JSI There are 37 health facilities in Wau. 12 receive support under SHTP-II. Many of the facilities are rented. Molio PHCU is being constructed, but poorly. Lokolio two room structure is inadequate for PHCC. Lack of staff, especially qualified Nurses. All facility-based health workers are paid by the government. JSI is not providing any salary support. - no CHWs. Most TBAs are poorlyt trained (only two days trainings) Lack of Lab testing equipment (paracheck and VCT Testing). Although there are rats in drug store, yet the drugs are supplied and carefully arranged. Referrals are big problem in Wau. Roads are poor and transportation is often unavailable (the State Hospital has only one ambulance. SCP has provided support in providing registers to record information on service delivery, and has provided training in the use of the registers. There is no VHC in canning factory PHCU. The idea of forming VHC had not yet reached the area. In Molio VHC was formed three years ago, but training was given only recently. The Acting County Health Officer requested for another vehicle to be used for making supervisory trips. At present, they relay on the only JSI vehicle to conduct supervision. Sudan Health Transformation Project Phase II (SHTP-II) 67 Mid-Term Evaluation Report – Public Document Terekeka/ADRA Most HF visited were very basic. PHCU consisted of one single room for all services and lacked basic furniture. PHCC better equipped. BSF is funding the construction of a CHD office due to be completed in December). Both PHCC and all three PHCU visited were understaffed particularly with regard to maternal health. Collaboration with community based health workers: TBAs, HHP, and VHC SCP conducted 10 trainings reaching 182 health workers, 40% of the target. Delay due to contract negotiations and difficulties in selecting the HF. SCP proactive in development of training material; not waiting for MSH. Limited on the job training; SCP mainly trains groups of HF staff in Terekeka town. SCP itself needs further capacity building in EPI management. SCP trained CHD on leadership development. The CHD has received significant support from MDTF through NPA and is equipped with two computers, a v-sat, a printer, a radio, a vehicle and a motorbike. Procurement of essential drugs is fully taken care of by the state while distribution of the kits is facilitated by ADRA. Many essential equipment for HF still stalled in the procurement phase which affects the trust of SCP’s and health facility staff in MSH’s project management. There is no coordinated referral system yet from HF to higher levels. A motorbike ambulance at Muni PHCC out of order. Sometimes ADRA offers own car. Affordability of transport to Juba problem. No data collected on referrals. The CHD has a hard top land cruiser but it is not provided as ambulance. Surveillance Officer of CHD attended LDP training and improved supervision of HF as a result. MSH is hardly supervising SCP at field level. SCP staff find MSH supervision checklist too long to be useful and have adapted a checklist developed for their CHF project. In general the reliability of data is questionable at all levels. VHC and HHP trained and supervised. Community sensitization through VHC focuses on hygiene and less on awareness about danger signs of malaria, ARI, dehydration in under fives. VHC monitoring drug supply, HF performance and helping clean the site. Collaborate with TBAs. But community health component getting too limited support from the project. IEC posters on the HF walls and well￾understood by the CHW. Wide variety of topics. Unclear however how the material is used by HHP. No clear IEC plan at SCP or HF level Well informed CHD, but SCP is not strategically strengthening the CHD aside from providing training to some senior members of the CHD. However management training of CHD in pipeline. No ADRA & CHD regular joint supervision, but joint planning for the running of EPI campaigns; CHD is not coordinating actions of health partners in county (ADRA, AMREF, ZOA). Sudan Health Transformation Project Phase II (SHTP-II) 68 Mid-Term Evaluation Report – Public Document Malakal/ IMA Three (Warajok, Asosa and Lwakat PHCUs) of the 7 facilities in Malakal has serious structural breakdown that require immediate attention. Leaking roofs continuously destroys documents and compels workers to unnecessary frequent cleaning and washing of facility utilities. All the facilities visited were well staffed, some were over staffed. However it’s the CHD strategy to keep them on job for referral to other upcoming health centers and also to introduce 24 hours service delivery through night and day shifts. CMW volunteers are gradually getting enrolled in the government pay system. SCP is working closely with the CHD. Sharing work plans and conducting Joint Supervisory visits to the health facilities. SCP is strengthening M&E system for CHD; mentoring the data focal person on data collection procedures. Several Trainings have been conducted on HIV/AIDS, Malaria and diarrhea case management. Most of the 7 facilities supported under SHTP-II in Malakal county have basic equipment like Thermometers, Examination beds, BP machine etc. SCP has put to MSH list of equipments required for the facilities. A vehicle out of the most equipment has been procured. Most of the facility basic equipments are donation from UNFPA. The PHCUs and PHCCs are not in anyway inter￾linked. PHCUs refer patients directly to the Hospital. Malakia PHCC does not coordinate with the other PHCUs. Efforts have been invested in ensuring proper data collection using the available HMIS tools. However, staffs seem to have challenges coping with newly introduced HMIS tools. Intensive training is required for better results. The VHCs in Malakal are very aware of their roles; however there seems to be lack of clear cut difference in roles and responsibilities between HHPs and VHCs. Although VHC were trained on their roles and responsibilities, most of their works revolve around cleaning of health facilities. There were Posters glued on most facility walls. Most were printed in English with a few Arabic versions from the north. SCP has recommended some CHD staffs for the recently concluded LDP training by MSH in Juba. They acknowledged the importance of the training in their office management. Sudan Health Transformation Project Phase II (SHTP-II) 69 Mid-Term Evaluation Report – Public Document County/Partner Implementation of high seven impact Child health Nutrition Water and Sanitation Malaria Maternal health (ANC, delivery, and postnatal) Family planning HIV prevention and testing Mundri East/ MRDA There is successful and strong EPI system. CHD is not much involved in Child Health activities. No single height/weight chart. Few facilities have working scales. No significant under￾nutrition. No evidence of WASH. The facilities were doing adequate preventive and curative health education of malaria. In most case, malaria drugs ran out before new kits arrive. PSI is also actively providing under-5 malaria treatment in 4 out of 7 facilities visited. Births typical happen at PHCCs. Women out of the immediate vicinity of PHCCs give birth at home. There is lack of delivery equipments; there are semi-active TBAs in all communities (2 -5 each). Lack of refresher trainings. No family planning activities. MSH had no staff assigned to family planning compliance monitoring and could not provide any plans for the future to ensure compliance. Although, MRDA had instructed staff not to distribute pill taken to facility based on GOSS policy, yet 5 female clients from two facilities had already received birth control pills in the interim. No HIV activities. The design of the HIV component does not seem appropriate for the context: facility￾based testing and PMTCT only, and no community awareness or community-based prevention activities planned. There are ethical concerns surrounding the testing plans. HIV counselors were trained in prevention messaging; they are poised to deliver these messages in a facility-setting only without plans to engage in community outreach. Sudan Health Transformation Project Phase II (SHTP-II) 70 Mid-Term Evaluation Report – Public Document Mundri West/ AAH-I There is successful and strong EPI system. CHD is not much involved in Child Health activities. No single height/weight chart. Few facilities have working scales. No significant under￾nutrition. No evidence of WASH. No evidence of WASH. The facilities were doing adequate preventive and curative health education of malaria. In most case, malaria drugs ran out before new kits arrive. PSI is also actively providing under-5 malaria treatment in 4 out of 7 facilities visited. Births typical happen at PHCCs. Women out of the immediate vicinity of PHCCs give birth at home. There is lack of delivery equipments; there are semi-active TBAs in all communities (2 -5 each). Lack of refresher trainings. No family planning activities. MSH had no staff assigned to family planning compliance monitoring and could not provide any plans for the future to ensure compliance. No HIV activities. The design of the HIV component does not seem appropriate for the context: facility￾based testing and PMTCT only, and no community awareness or community-based prevention activities planned. There are ethical concerns surrounding the testing plans. HIV counselors were trained in prevention messaging; they are poised to deliver these messages in a facility-setting only without plans to engage in community outreach. Sudan Health Transformation Project Phase II (SHTP-II) 71 Mid-Term Evaluation Report – Public Document Tonj South/ CCM 570 children under the age of one received third dose of DPT in Sept 2010. No stock out of cotrimoxisol for treatment of pneumonia seen. No vaccination monitoring charts in all facilities visited of CHD. Vitamin A was found in all the facilities and record showed use. ORS and zinc tablets were available in each facility. Nutrition programs in the facilities are run by WVI. CHD has a nutritionist. Fortified milk, porridge and plumpy nut are provided. MSH advised SCP to focus on hand washing, clean water, hygiene and promotion. Most of the facilities have no-existent or substandard latrines and hand pumps. No hand washing facilities in any of the facilities visited. Health education and hygiene promotion messages are given to youth and women groups. Prevention and treatment of malaria is taking place. LLITNs are being distributed through ACSI for mothers attending first ANC. 1,178 LLITNs distributed and there was stock out of malaria RDT kits at the beginning of July but supply was received in September. No job aids/algorithms were present at any of the facilities for the treatment of malaria. TT, LLITNs, IPT, Vitamin A, iron folate and stethoscopes were available and regularly distributed at the facilities. Only Thiet PHCC has delivery room and bed. No midwives or MCHWs in any of the facilities, Thiet and Mabior Yar have TBA - at Thiet, about 10- 20 babies are delivered per month. Knowledge of healthy breast feeding practices seems low. KAP survey done by SCP showed high level of resistance to FP. Survey also revealed that decision is made by men, not women. FP contraceptives were in CCM store but not distributed. CCM staff met a lot of resistance initially but found it more effective to talk about child spacing Sudan Health Transformation Project Phase II (SHTP-II) 72 Mid-Term Evaluation Report – Public Document Mwolo/ Save The Children EPI seems to be functioning well in most sites; each PHCC has two outreach teams that travel in the region from Monday-Friday providing vaccinations. Many mothers are not bringing their children back for all scheduled vaccinations. OXFAM is active in Mvolo digging wells for clean water supply. There were no WASH activities in Mvolo that were seen. Water sources were sometimes placed too far from PHCU or PHCC. Pit latrines had been constructed in most areas but usage remains questionable. Malaria rapid testing kits are not always available at PHCCs. Many mothers were previously afraid to come to the PHCC to deliver, this is changing now. The Yeri PHCC had bay cribs - not seen anywhere. Yeri PHCC had stock-out of oxtoxin and ergometrine, recently lost a mother with twins due to uncontrolled bleeding during delivery. VCT department in Yeri was awful. Sudan Health Transformation Project Phase II (SHTP-II) 73 Mid-Term Evaluation Report – Public Document Wau/JSI There is organized EPI and careful maintained EPI register in Olinka. Water and sanitation facilities in most service sites are inadequate. Bednets were not distributed to some centers. Lack of delivery beds. Canning Factory PHCU has unusually dedicated midwife. Nearly all deliveries now take place at the PHCU since local women have come to trust the services of this community midwife. Community midwife at Molio lack equipment (not even a pair of scissors or simple dressing materials). It was not clear how any kind of competent ANC or maternity care could be provided at this site. There is also lack of maternity equipments in Olinka PHCC. Family planning and reproductive health services in Wau are inadequate at the present time. While oral pills and condoms are available in local pharmacies, contraceptives are not available at the present time in PHCUs and PHCCs. There is no HIV counseling and testing. Sudan Health Transformation Project Phase II (SHTP-II) 74 Mid-Term Evaluation Report – Public Document Terekeka/ADRA 60% of fridges at county level were found out of order; record keeping of fridge temperatures, vaccine supply and stock appeared poor and personnel not very competent to oversee the cold chain. At PHCC level a functional cold chain and records of routine immunization on a daily basis. Vitamin A (supplied by UNICEF) achievement against the target is < 10%. Malnutrition does not seem a big problem among Mundari. But growth monitoring is not done in any of the HF visited. No MUAC practice, no weighing scales available and limited health education on nutrition. SCP invested in hand washing stations & ORS corners in each HF. Water purification tabs available in several of the HF as well as (Carter Center) water filters for prevention of guinea worm. Waste mngt. at HF level appeared problematic. None of the HF visited has own borehole except Terekeka PHCC. Latrines at HF absent or poor. No latrine seen at household level. Provision ITN was irregular and reached only 50% of the target; rains are main hindering factor. ITN used for multiple purposes e.g. animals. Health workers in charge of HF aware of the treatment guidelines. Epidemiological morbidity reports sent to GOSS reveal that malaria is mainly unconfirmed, very little differentiation in diagnosis. All the facilities visited had ACTs for all the age categories PSI and Malaria Consortium not in Terekeka Except for Terekeka PHCC which has two midwives, pregnant women mainly depend on the services provided by TBA. However, ANC and basic delivery services by a TBA are not well recorded. According to SCP report approx. 80% of all deliveries take place at community level but monthly reports of the individual health facilities suggest an even higher number. Basic equipment for quality antenatal care and safe delivery is generally insufficient. There is little demand for FP in Terekeka. No family planning methods available aside from condoms. However, even condoms are hardly demanded and/or promoted. Mundari women consider child spacing method of breastfeeding till child is 2.5 years during which both spouse abstain from sexual contact more appropriate; Many girls marry at age of 15 which is risky for safe delivery. Little is done with regard to HE, VCT, treatment, PMTCT etc. No condom demand reported, no promotion of condoms, no HIV testing except for Terekeka PHCC where pregnant women are tested. Training on HIV/AIDS and PMTCT is done but without further interventions. Sudan Health Transformation Project Phase II (SHTP-II) 75 Mid-Term Evaluation Report – Public Document Malakal/ IMA All the facilities visited have well coordinated EPI sessions twice a week: Mondays and Wednesdays. Wau shilluk PHCU among others has ORS corner, nevertheless the other facilities have ORS sachets in stock. Although Vitamin A capsules were seen in all the facilities visited, yet there was no record keeping. Growth monitoring was also available in Malakia PHCU and Lwakat PHCU. The SCP leveraged resources with the BRIDGE team to install Bio￾sand water filtering facilities for health facilities with no clean water points. IMC through BRIDE is also constructing Latrines for Malakia PHCC and Wara Jok PHCU. All the facilities visited had ACTs for all the age categories and LLITNs are distributed to Pregnant Mothers and Under five children during Immunization sessions. Malakal had a good number of CMWs assisting deliveries. However the quality of training administered to them remains to be queried. On average there are 3 - 4 TBAs per facility, assisting in home based deliveries and reporting to the facility of attachment. No deliveries take place in all the faculties visited. Family planning messages are shared with clients during Health Education sessions at the facility. Oral contraceptives from UNFPA are available for clients who voluntarily want to use them. Malakia PHCC has a good VCT center; positive clients are referred to Malakal County Hospital for treatment. All facilities have condoms in store. Posters containing HIV/AIDS messages are glued on most facility walls. Sudan Health Transformation Project Phase II (SHTP-II) 76 Mid-Term Evaluation Report – Public Document ANNEX 4: KEY DOCUMENTS REVIEWED Beesley, Mark. 2010. Off the Road: The Mapping of Health Training Facilities in Southern Sudan. Juba, MOH/GOSS and Technical Resource Centre, Liverpool Associates in Tropical Medicine, July. Michael, Janet, Michael Andreini, Khadijat Mojidi, Willa Pressman, Yogesh Rajkotia, and Mary Ellen Stanton. Southern Sudan Maternal and Reproductive Health Rapid Assessment. Washington, DC: USAID/Washington, September. Ministry of Health, Government of Southern Sudan (GOSS). 2006. Prevention and Treatment Guidelines for Primary Healthcare Centers and Hospitals. Juba: MOH/GOSS, Ministry of Health, Government of Southern Sudan (GOSS). 2006. Basic Package of Health and Nutrition Services for Southern Sudan. Juba: MOH/GOSS. Ministry of Health, Government of Southern Sudan (GOSS). 2006. Technical Guidelines for Family Planning Services in Southern Sudan. Juba: MOH/GOSS, September. Ministry of Health, Government of Southern Sudan (GOSS). 2007. Southern Sudan Essential Medicine List. Juba: MOH/GOSS. Ministry of Health, Government of Southern Sudan (GOSS). 2007. Health Policy for the Government of Southern Sudan 2006-2011. Juba: MOH/GOSS. Ministry of Health, Government of Southern Sudan (GOSS). 2007. Sudan Household Health Survey (SHHS) - 2006. Juba: MOH/GOSS, October. Ministry of Health, Government of Southern Sudan (GOSS) and UNAIDS. 2008. Southern Sudan: HIV/AIDS Integrated Report (2006-2007). 2008. Juba: MOH/GOSS and UNAIDS. Ministry of Health, Government of Southern Sudan (GOSS). 2008. Southern Sudan HIV/AIDS Strategic Framework (2008- 2012) – Draft. Juba: MOH/GOSS. Ministry of Health, Government of Southern Sudan (GOSS). 2009. Maternal, Neonatal, and Reproductive Health Strategy, 2009. Juba: MOH/GOSS, Ministry of Health, Government of Southern Sudan (GOSS). 2009. Basic Package of Health and Nutrition Services for Southern Sudan, Final Draft. Juba: MOH/GOSS, January. Ministry of Health, Government of Southern Sudan (GOSS). 2009. Southern Sudan Antenatal Care Clinics Sentinel Surveillance Report. Juba: MOH/GOSS, September-December. Ministry of Health, Government of Southern Sudan (GOSS). 2009. Family Planning Policy. Juba: MOH/GOSS, November. Ministry of Health, Government of Southern Sudan (GOSS) and John Snow International. 2009. Health Education and Promotion: National Policy for Health Promotion 2009. Juba: MOH/GOSS and John Snow International. Ministry of Health, Government of Southern Sudan (GOSS). 2010. Guidelines for the Prevention of Mother to Child Transmission of HIV. Juba: MOH/GOSS. Ministry of Health, Government of Southern Southern Sudan (GOSS). 2010. Maternal and Newborn Health Formative Care: A Baseline Formative Research Report. Juba: MOH/GOSS and UNICEF, and Nairobi, Kenya: Acacia Consultants, March. Ministry of Health, Government of Southern Sudan (GOSS). 2010. Southern Sudan Antenatal Care Clinics Sentinel Surveillance Report, September-December, 2009. Juba: MOH/GOSS, HIV/AIDS/STI Directorate. Ministry of Health, Government of Southern Sudan. Nd. Family Planning Policy. Juba: MOH/GOSS, November. Sudan Health Transformation Project Phase II (SHTP-II) 77 Mid-Term Evaluation Report – Public Document Ministry of Health, Government of Southern Sudan. Nd. Maternal, Neonatal, and Reproductive Health Strategy, 2009-2016 (Draft). Juba: MOH/GOSS. Ministry of Health, Government of Southern Sudan. Nd. Health Policy: Government of Southern Sudan, 2007-2011. Juba: MOH/GOSS. Ministry of Health, Government of Southern Sudan. Nd. Prevention and Treatment Guidelines for Primary Healthcare Units. Juba: MOH/GOSS. Peprah, Dorothy. 2009. Draft Hygiene Promotional Strategy: Sudan Health Transformation Project – II. New York: International Relief Committee, June. Shepherd-Banigan, Megan, Franklin Baer, Carrie Hessler-Radelet, & Diana Silimperi. 2008. Health Sector Relief to Development Transition Gap Analysis: Southern Sudan. Arlington, Virginia: Basic Support for Institutionalizing Child Survival (BASICS) for the United States Agency for International Development (USAID). Stoops, Norah. 2010. New Reporting Formats and Registers Training Course. Juba: IMA World Health and HISP, September. Stoops, Norah, Mahlatse Mokoena, and Mo Ali. 2010. Foundation District Health Information Course: August/September 2010. Juba: IMA World Health. Management Sciences for Health. 2009. Micro Grants Program Manual, 2009-2012. Juba: MSH, July. Management Sciences for Health. 2009- 2010. Sudan Health Transformation Project: - Quarterly Progress Reports (April-June, 2009; July-September, 2009; October-December, 2009; January-March, 2010; April-June, 2010). Juba: MSH. Management Sciences for Health. 2010. Sudan Health Transformation Project (SHTP-II): Performance Report for FY 2010. Juba: MSH, October. Management Sciences for Health. 2010. Sudan Health Transformation Project – Phase 2: Project Monitoring and Evaluation Plan (February 2009-February 2012). Juba: MSH and MOH/GOSS, September. Management Science for Health. 2010. Sudan Health Transformation Project (SHTP-II): Performance Report for FY 2010. Juba: MSH, October. Unpublished Consultants Report. 2010. Rapid Assessment and Recommendations for Government Payments of Health Worker Salaries in Southern Sudan. August. Neuse, Margaret, Connie Davis, Victor Masbayi, Mary Harvey, and Yogesh Rajkotia. 2008. Sudan Health Transformation Project (SHTP) Assessment Report. Washington, DC: IT Shows, Inc, March. Rajkotia, Yogesh, Stephanie Boulenger, and Willa Pressman. 2007. Southern Sudan Health System Assessment. Bethesda, Maryland: Health System 20/20 Project, Abt Associates, Inc., July. Unpublished Consultant Report. 2010. Rapid Assessment and Recommendations for Government Payments of Health Worker Salaries in Southern Sudan. August. USAID/Sudan. 2007. Southern Sudan Maternal and Reproductive Health Rapid Assessment, Khartoum: USAID/Sudan, November. USAID/Sudan. 2008. Sudan Health Transformation Project – Phase 2: Request for Task Order Proposals (RFTOP). RFTOP Number 650-09-313, October 14. Sudan Health Transformation Project Phase II (SHTP-II) 78 Mid-Term Evaluation Report – Public Document ANNEX 5: CONTACT LIST OF KEY INFORMANTS Ministry of Health, Government of Southern Sudan (GOSS) Dr. Samson Baba, DG for External Assistance and Coordination, MOH/GOSS Ms. Rebecca Alum William, Program Officer for Primary Healthcare, MOH/GOSS Dr. Sarah Goldsmith, Health Cluster Co-Lead, External Assistance and Coordination Directorate Office, MOH/GOSS, sshealthcoordination@gmail.com Management Science for Health Matt Iwanowicz, Interim Director of Finance, Management Science for Health, miwanoxitz@msh.org Navindra Persad, M&E Principal Technical Adviser, Management Sciences for Health, npresad@msh.org Kip Eckroad, M&E Director, SHTP-2, Management Science for Health, keckroad@msh.org Dan Nelson, Prime Technical Adviser, Management Science for Health, dnelson@msh.org Sam Gonzaga, Wash Adviser, International Relief Committee, sgonzoga@theirc.org Melissa Brill, Community Mobilization Adviser, Management Sciences for Health, melissabrill@yahoo.com Edward Luka, PHC Advisor, Management Sciences for Health, eluka@msh.org Felix Loro, M&E Technical Adviser, Management Sciences for Health, floro@msh.org Joseph Miller, M&E Officer, Management Sciences for Health, josephmiller@yahoo.com John Rumunu, Deputy Director and TB Specialist, Management Sciences for Health, jrumunu@msh.org Acol Ayom Dor, CMC, Management for Sciences for Health, International Relief Committee, acolayom@sudan.theirc Susan Aye, M&E Officer, Management Sciences for Health, aeagai@yahoo.com Rebecca Drale, Program Assistant, Management Sciences for Health, rdrale@msh.org Dr. Fred Hartman, Acting Chief of Party, SHTP-II, Management Sciences for Health, fhartman@msh.org SHTP-II Subcontracting Partners Key Contacts IMC Maarten Romijn, Deputy Country Director Email: mromijn@imcworldwide.org Adventist Development and Relief Agency, Sudan – Southern Sector (ADRA) Awadia Ogillo, Associate Country Director Email: associatecd@adrasouthsudan.org / aogillo@yahoo.co.uk Save the Children Joanna Michler, Program Manager Email: J.Michler@savethechildren.org.sd CARE John Perry, Director of Programs Email: jperry@ci.or.ke Action Africa Help International (AAHI) Dr. Omer MOHamed, PHC Coordinator Email: dromeryahia@yahoo.com International Relief Committee (IRC) Dr. Justice Bazirake, Senior Health Coordinator Email: Justice.Bazirake@theirc.org Dr. Kahi Vincent, Health Coordinator Email: Vincent.Kahi@theirc.org John Snow International (JSI) Margaret Itto, Country Director, Email: ittomargaret@yahoo.co.uk Sudan Health Transformation Project Phase II (SHTP-II) 79 Mid-Term Evaluation Report – Public Document Mundri Relief and Development Association (MRDA) Light Wilson Aganwa, MRDA Director Email: lightaga@yahoo.com Comitato Collaborazione Medica (CCM) Timothy Koe, CCM Country Representative Email: koemal@yahoo.com The World Bank Multi-Donor Trust Fund Dr. MOHammed Ali Kamil, Senior Health Specialist, Health, Nutrition, and Population Unit, Africa region, The World Bank, Juba, mkamil@worldbank.org Basic Services Fund (BSF) Klaziena (Kate) Louwes, Teamleader, Mott McDonald, Basic Services Fund, klouwes@bsf-secretariate-sd.org Wim Groenendijk, Policy Development/M&E Officer, Mott McDonald, Basic Services Fund, wim.groenendjik@bsf￾secretariate-sd.org Norwegian People’s AID, Sudan Program (NPAID) Jan Ledang, Country Director, Norwegian People’s AID (NPA), Juba, janl@npaid.org Dr. Atem Nathan Riak, Deputy Team Leader, Norweigan People’s AID, Sudan Program, atemda@npaid.org IMA World Health Sarla Chand, Vice President, International Programs, IMA World Health, sarlachand@imaworldhealth.org Mo Ali, Project Coordinator and M&E Development Specialist, IMA World Health, ma2766@gmail.com USAID/Juba, Southern Sudan Susan Fine, USAID Director, Southern Sudan Mission, USAID/Juba Charles Lerman, Director, Office of Health, USAID/Juba Cliff Lubitz, Deputy Health Officer, USAID/Juba Kawa Tong, Health Officer, USAID/Juba Dr. Martin Swaka, Health Specialist, USAID/Juba United Nations Population Fund (UNFPA) Fikru Zeleke, Reproductive Health Adviser, UNFPA/Juba Joint Donor Team Anne Lindberg, Team Leader, Basic Services, Joint Donor Team UK Department for International Development (DFID) Mark Mallalieu, Head DFID Juba Office Sudan Health Transformation Project Phase II (SHTP-II) 80 Mid-Term Evaluation Report – Public Document ANNEX 6: SHTP-II EVALUATION WORK PLAN DAY DATE LOCATION/TASKS OCTOBER Friday 15 Team Planning Meeting (day 1) Saturday 16 Team Planning Meeting (day 2) Sunday 17 OFF Monday 18 Team Planning Meeting (day 3) Tuesday 19 Juba interviews; refine methodology Wednesday 20 Pre-test questionnaires at ADRA office and Juba PHCC/VHC; review (Juba) Thursday 21 Juba meetings Friday 22 Juba meetings Saturday 23 Juba meetings Sunday 24 OFF (Team II departs for Mvolo) Monday 25 Team III departs for Tonj South Tuesday 26 3 teams in the field Wednesday 27 3 teams in the field Thursday 28 Teams return from Mundri East and Mundri West and Mvolo Juba meetings Friday 29 Juba meetings Saturday 30 Team III returns from Tonj South Sunday 31 OFF NOVEMBER Monday 1 Juba meetings – midterm check-in Tuesday 2 Two teams depart for Malakal and Terekeka; remaining team – Juba meetings Wednesday 3 Team departs for Wau Thursday 4 OFF Friday 5 Team returns from Terekeka; Saturday 6 Teams return from Malakal and Wau Sunday 7 OFF Monday 8 Juba meetings Tuesday 9 Findings, Conclusions, and Recommendations Workshop Day 1. Wednesday 10 Findings, Conclusions, and Recommendations Workshop Day 2 Thursday 11 Prepare for USAID Internal Debriefing - and report drafting. Friday 12 USAID Internal Debriefing (MSI office - Juba). Draft evaluation report. Saturday 13 Report drafting (Juba) Sunday 14 Report drafting (Juba) Monday 15 Report drafting (Juba) Tuesday 16 Briefing for MSH (MSH office) and Report drafting (Juba). Wednesday 17 Preparation for Presentation to Govts, etc - and report drafting (Juba) Thursday 18 Presentation to government(s), MSH, and other key stakeholders and submit draft Report to MSI. Friday 19 Travel to home base Sudan Health Transformation Project Phase II (SHTP-II) 81 Mid-Term Evaluation Report – Public Document ANNEX 7: INTERVIEW GUIDES FOR EVALUATION INTERVIEW GUIDE FOR SHTP-II ASSESSMENT Questionnaire Sub-Sets GOSS MSH NGO Subcontractors State and County Health Managers Community and Village Community Groups SHTP-II Facility-Based Program Managers SHTP-II Facility-Based Service Providers Beneficiaries (Clients) Served by SHTP-II Observation Guide for Assessing SHTP-II Service Facilities Sudan Health Transformation Project Phase II (SHTP-II) 82 Mid-Term Evaluation Report – Public Document MEETINGS WITH GOSS MOH LEADERS INTRODUCTION OF TEAM AND PURPOSE OF VISIT Assess implementation of the Sudan Health Transformation Project-II Identify its strengths and areas where changes may be needed Consider future areas of USAID investment in the health sector in Southern Sudan If needed, give overview of SHTP-II. ATTENDEES Name, positions. Specific information about them Experience in GOSS, MOH; prior experience; how long have you been in your position, areas of expertise. Questions 1. To begin, can all of you describe your areas of specialization, relationship to the SHTP-II, and the extent to which you interact with the project? 2. Could you describe your overall conception for health system design and strengthening in Southern Sudan? 3. How has SHTP-II contributed to Health Sector development in Southern Sudan? What has the project provided? How? At central MOH? [May want or need to prompt re. policies and guidelines development; development of management capacity, etc.] At State level? At County level? In delivery of primary healthcare services, especially high impact services? Training? Infrastructure? Water and Sanitation? Other contributions? 4. What do you believe have been the major accomplishments of SHTP-II to date? Are there certain health areas covered by the project that have been particularly successful? What activities or initiatives were most responsible for this success? 5. Can you cite some examples of how SHTP-II assistance has helped to strengthen health services? For example, by - Standardizing the range of essential services at facilities? - Improving training curricula for services providers? - Upgrading the quality of services? - Improving the supply and distribution of essential drugs and equipment - Increasing community participation in health planning (including disease prevention and the delivery of services 6. Are there some aspects of the project that have not been moving ahead as planned? - Are there certain health services covered by the project that have been slower to develop? Sudan Health Transformation Project Phase II (SHTP-II) 83 Mid-Term Evaluation Report – Public Document - If yes, what might account for these problems and how might they be addressed in the remaining 15 months of the project? 7. In your opinion, are there program elements of SHTP-II that should be given greater emphasis (effort) between now and the end of the project in 2011? 8. How well has SHTP-II coordinated its work with other stakeholders working in the Health Sector in Southern Sudan? Has the project participated in and made meaningful contributions to the Health and Nutrition Consultative Group? What specifically could it do to improve its coordination and support? 9. What headway has been made by GOSS in transitioning Southern Sudan’s health system away from emergency humanitarian assistance to a more developed and stable health structure capable of providing consistent long-term care to the people of Southern Sudan? 10. How has the project been encouraging greater Sudanese participation and ownership of the health system? 11. It is anticipated that USAID will continue to support the Health Sector in Southern Sudan after the end of the SHTP-II. What are your suggestions for the areas that USAID should support in the future? Service delivery—in what aspects; geographical focus; service focus and high impact PHC programming; community mobilization and involvement; BCC programming. Capacity building—at what levels; for what aspects. National level policy formulation and implementation Infrastructure Support for models of service delivery and management, e.g., development of MOH unit for contracting out to NGOs Systems development: logistics; other. Other components 12. Are there new program services or system strengthening strategies (e.g., with respect to decentralization, integration, human resource allocations, home-based delivery of care, privatization) that you would like to field-test through operational program research? What are operational research capacities in Southern Sudan at the moment and how might these be enhanced with the MOH? Sudan Health Transformation Project Phase II (SHTP-II) 84 Mid-Term Evaluation Report – Public Document INTERVIEW GUIDE FOR SHTP-II ASSESSMENT MEETINGS WITH PRIME CONTRACTOR (MSH) INTRODUCTION OF TEAM AND PURPOSE OF VISIT Assess implementation of the Sudan Health Transformation Project Identify its strengths and areas where changes may be needed Consider future areas of USAID investment in the health sector in Southern Sudan If needed, give overview of SHTP-II. ATTENDEES Name, positions. Specific information about them, as feasible Experience in health service delivery,--where and how long; technical and management training; areas of expertise. Questions Program Operations 1. Please describe how you are currently working with MOH/GOSS? Who do you work with at MOH? Your subcontractors? USAID? 2. How has SHTP-II been coordinating its activities with state and country-level medical officials? Who in SHTP-II is responsible for this? Please describe these activities. Are there aspects of these associations that need further strengthening? 3. We understand that a human resource assessment has recently been completed for SHTP-II. Can you share with us any findings from this report that might be relevant to our assessment? 4. What do you believe have been the major accomplishments of SHTP-II to date? Are there certain health interventions covered by the project that have been particularly successful? What initiatives were most responsible for this success? 5. Are there some aspects of the project that have not been moving ahead as planned? In your opinion, what might account for these issues and how might they be addressed in the remaining 15 months of the project? 6. How has the project been encouraging greater Sudanese participation and ownership of the health system? Program Services 7. At the present time, what services are typically offered in SHTP-II facilities? Is the project now offering a package of essential services that are standard for all facilities participating in SHTP-II, or are some components still underdeveloped in some settings? 8. How well attended are the facilities? In a typical day, how many clients are seen at your facilities? Could you provide us copies of this information? 9. How has the project been generating greater demand for the utilization of health services in SHTP-II catchment areas? 10. What has been the project’s approach to BCC and community-based advocacy outreach activities? Have there been any internal evaluations of these efforts by MSH? Who are the people responsible for these activities in your office? 11. How has SHTP-II worked to enhance health sector infrastructure? Could you provide us examples. What future upgrades can be anticipated in the remaining 15 months of the project? Sudan Health Transformation Project Phase II (SHTP-II) 85 Mid-Term Evaluation Report – Public Document 12. Have standardized training materials and procedural manuals been developed by the project? To what extent are they being utilized by the project? Can you give us examples of these are being used in county facilities? Could we please have copies of these? 13. Have CHW/MCHW Training Institutes providing pre- and in-service training for focus county personnel been established? If yes, how many are there and where? How many are now operational? 14. Have standard training curriculums been developed for Home Health Promoters to be utilized in focus counties? If yes, can we see copies? SHTP-II Management 15. How many supervisory field visits are made on average to the field per quarter. Are trip reports generated from these visits? Do we have all of these? 16. Does SHTP-II serve on management and technical working groups? How does it interact with other major donor programs such as the Multi-Donor Trust Fund and the Basic Services Fund? How effective has this collaboration been? 17. How has the Core Group Management meeting system been operating? Has this proven to a beneficial mechanism for SHTP-II? Has the committee meet frequently enough with adequate participation to make it an effective mechanism? 18. In what ways has SHTP-II strengthened NGO capacity in Southern Sudan? Has the project been able to transfer lessons learned to other partners (other local NGOs, community-based, organizations, and FBOs)? 19. Have your subcontractors been able to leverage resources and support from other programs? 20. Have been there been any problems in disbursing funding to subcontractors in a timely fashion? 21. Have there been any problems in managing direct costs, such as labor, office space, non-expendable and expendable equipment and supplies, information technology, and banking and financial transactions satisfactory? 22. Do you find that a cost-reimbursement fixed fee contract is a satisfactory contracting mechanism for your project? Might there be preferable mechanisms that you could suggest? Are performance-based subcontracts working effectively for NGO subcontractors participating in the project? Human Resource Allocations 23. How do staff in your facilities compare with government standards and approved staffing patterns? What is the typical allocation of clinical staff in SHTP-II facilities? Are there minimal staffing requirements in each SHTP-II facility, or are these left to the discretion of individual subcontractors? 24. Have there been problems in having staff payed in-full and on time in SHTP-II facilities? If yes, have these problems now been rectified? And how were they resolved? 25. Did SHTP-II develop a staff development and mentoring plan for building the sustainable capacity of the staff? Commodities and Logistics 26. Has the project experienced problems with stock-outs of essential supplies and equipment? Has SHTP-II contributed to strengthening existing logistical and supply chain systems for essential supplies and commodities? 27. How has the SHTP-II been working with the SPS Project? Can you give a few examples? 28.. Have SHTP-II subcontractors been able to identify and budget for plans to co-locate in MOH and county health department premises? How many subcontractors have made this move? 29. Have procurement of supplies, equipment and vehicles for subcontractors been handled effectively and efficiently? If yes, can you give us few examples of how this was handled? Monitoring and Evaluation Sudan Health Transformation Project Phase II (SHTP-II) 86 Mid-Term Evaluation Report – Public Document 30. How is the SHTP-II M&E system currently operarting? Is is generating valid and timely information at this point? Is it being kept up-to-date? Is there any feedback to subcontractors on indicators and, if so, is this information being used to improve the program performance? How is M&E data being validated in the field? 31. Were baseline assessments for any program areas undertaken at the start of the project? Project Design Issues 32. In your opinion, are there any issues with respect to the original design of the project that should be reconsidered based on experience to date? Was the original contractual design feasible given the resources and time available to implement all project elements? 33. Does SHTP-II II have well-reasoned objectives? Are these objectives still valid or should they be revised? 34. Were the identification and definition of indicators satisfactory in the original contract? . Were appropriate targets set? . Were the number and location of geographic sites appropriate in the context of achieving results? Sudan Health Transformation Project Phase II (SHTP-II) 87 Mid-Term Evaluation Report – Public Document INTERVIEW GUIDE FOR SHTP-II ASSESSMENT MEETINGS WITH NGO SUBCONTRACTORS INTRODUCTION OF TEAM AND PURPOSE OF VISIT Assess implementation of the Sudan Health Transformation Project Identify its strengths and areas where changes may be needed Consider future areas of USAID investment in the health sector in Southern Sudan If needed, give overview of SHTP-II. ATTENDEES Name, positions. Experience in health service delivery,--where and how long; technical and management training; areas of expertise. Questions 1. To begin, can you describe your roles and responsibilities in your organization? 2. How many supervisory field visits are made on average to the field per quarter. Are trip reports generated from these visits? Do we have all of these? 3. To what extent have you been able to incorporate government standards pertaining to human resources, management, and service provision? 4. Can you provide some background on your work and service sites. Catchment Area(s) For service sites managed: Clients/day, month for INGO or LNGO Services provided (look at and get charts with statistics as available) Outreach to the community—how, how often? Availability of adequate staff (managerial and service provider) Availability of essential drugs, equipment, and other supplies Availability of water, sanitation facilities Status and functioning of the Health Management Committees Health Problems/Issues Management Problems/Issues 5. How do you support NGO and/or your service sites? What has the project provided? How? How/in what way has SHTP-II supported your management and delivery of primary healthcare services? Essential drugs? Training? Water and Sanitation? Community mobilization and communication? Health policy strengthening and roll-out? Technical guidance on developing consistent standards for training and service provision? Development of management capacity? Other contributions? 6. From your experience, what have been the strong elements of SHTP-II’s work? Sudan Health Transformation Project Phase II (SHTP-II) 88 Mid-Term Evaluation Report – Public Document Are these elements that you think should continue? Why? 7. What are areas where you think that SHTP-II has not done very well? Are they areas where you think assistance is still needed? If so, how would you change what SHTP-II has done? 8. How is the SHTP-II M&E system currently operarting? Is is generating valid and timely information at this point? Is it being kept up-to-date? Is there any feedback to subcontractors on indicators and, if so, is this information being used to improve the program performance? How is M&E data being validated in the field? 9. Were baseline assessments for any program areas undertaken at the start of the project? 10. Have you been able to transfer lessons learned to other partners? How is this experience documented and disseminated? Has the program been able to leverage resources and support from other programs? 11. Have there been any problems in managing direct costs, such as labor, office space, non-expendable and expendable equipment and supplies, information technology, and banking and financial transactions satisfactory? 12. Do you find that a cost-reimbursement fixed fee contract is a satisfactory contracting mechanism for your project? Might there be preferable mechanisms that you could suggest? Are performance-based subcontracts working effectively for NGO subcontractors participating in the project? 13. Are there other activities or areas that you think SHTP-II should be engaged in from now and the end of the project in 2009? 14. It is anticipated that USAID will continue to support the Health Sector in Southern Sudan after the end of the SHTP-II. What are your suggestions for the areas that USAID should support in the future? Service delivery—in what aspects; geographical focus; service focus and high impact PHC programming; community mobilization and involvement; BCC programming. Infrastructure Systems development: logistics; other. Capacity building—at what levels; for what aspects. Support for NGOs? At the County level? At the State level? National level policy formulation and implementation Coordination among International and Local NGOs Other components 15. Is the SHTP-II model of building NGO (parastatal) primary healthcare facilities around Southern Sudan the best model to pursue? Should other models be considered, or should the SHTP-II approach be intensified and more successful SHTP-II elements scaled-up? What should the next steps be moving forward? Sudan Health Transformation Project Phase II (SHTP-II) 89 Mid-Term Evaluation Report – Public Document INTERVIEW GUIDE FOR SHTP-II ASSESSMENT MEETINGS WITH STATE HEALTH OFFICIALS INTRODUCTION OF TEAM AND PURPOSE OF VISIT Assess implementation of the Sudan Health Transformation Project Identify its strengths and areas where changes may be needed Consider future areas of USAID investment in the health sector in Southern Sudan If needed, give overview of SHTP-II. ATTENDEES Name, positions. what are your primary responsibilities in your organization? Experience in Southern Sudan and with GOSS, MOH; prior experience; areas of expertise. Questions 1. What is your impression of the NGO project activities support by SHTP-II in your state? 2. How do you believe SHTP-II and its NGO partners have contributed to Health Sector development in your state? Can you cite some examples of useful contributions? In delivery of primary healthcare services? Infrastructure? Training? Water and Sanitation? At MOH [May want or need to prompt re policies and guidelines development; development of management capacity, etc.] Other contributions? 3. From your experience, what have been the strong elements of the project’s work? Are these elements that you think should continue? Why? 4. What areas do you think the project has not done very well? Are they areas where you think assistance is still needed? If so, how would you change what the project has done? 5. In your opinion, how well has the project coordinated its work with other stakeholders working in your area? What specifically could it do to improve its coordination and support? 6. In your area, is there good coordination between state and county medical staff. How often do you meet (frequency)? Such as? Strengthening the provision of program services Upgrading human resource capacity Building infrastructure Ensuring the adequate supply of essential commodities Promoting the provision of pharmaceuticals through commercial outlets Other issues 7. In your opinion, what could be done to strengthen coordination between state and county levels of the health system? Are there priority needs with respect to financial management? 8. Is there a functioning monitoring and evaluation capacity in your state as yet? What steps are being taken to build M&E capacity in your state? How do you anticipate that a well-managed M&E system will contribute to improved management of health services in your state (county). Sudan Health Transformation Project Phase II (SHTP-II) 90 Mid-Term Evaluation Report – Public Document 9. Is a surveillance system for infectious disease active in your state? How current is the information that is being compiled? How effectively is it being used to monitoring changing health conditions in your area? 10. Do you have a rapid response capacity for dealing with disease outbreaks? Do you have sufficient stocks for drugs for emergencies? 11. What are your expectations for the roles NGOs will be playing play in service delivery? What is the longer-term vision for NGOs? For what time period do you see NGOs playing a major role in service delivery in Southern Sudan? Sudan Health Transformation Project Phase II (SHTP-II) 91 Mid-Term Evaluation Report – Public Document INTERVIEW GUIDE FOR SHTP-II ASSESSMENT MEETINGS WITH COUNTY HEALTH OFFICIALS INTRODUCTION OF TEAM AND PURPOSE OF VISIT Assess implementation of the Sudan Health Transformation Project Identify its strengths and areas where changes may be needed Consider future areas of USAID investment in the health sector in Southern Sudan If needed, give overview of SHTP-II. ATTENDEES Name, positions. what are your primary responsibilities in your organization? Experience in Southern Sudan and with GOSS, MOH; prior experience; areas of expertise. Questions 1. What is your impression of the NGO project activities in your county? What are your expectations for the roles NGOs will play in service delivery in the future? For what time period do you see NGOs playing a major role in service delivery in Southern Sudan? 2. How do you believe SHTP-II and its NGO partners have contributed to Health Sector development in your county? Can you cite some examples of useful contributions? At County level? In delivery of primary healthcare services? Infrastructure? Training? Water and Sanitation? At central MOH? [May want or need to prompt re policies and guidelines development; development of management capacity, etc.] Other contributions? 3. Did you recently attend the Leadership Management Training organized through the auspices? Did you find this training useful and how will you utilize it in your work? 4. From your experience, what have been the strong elements of the project’s work? Are these elements that you think should continue? Why? 5. What areas do you think the project has not done very well? Are they areas where you think assistance is still needed? If so, how would you change what the project has done? 6. Are there specific program service areas that are especially under-utilized that require further strengthening through greater demand generation efforts? For example, family planning and reproductive health services and maternity care? Others? 7. In your opinion, how well has the project coordinated its work with other stakeholders working in your area? What specifically could it do to improve its coordination and support? 8. Are community groups, such as women’s groups, youth groups, and church groups involved in priority-setting and health promotion activities in your state (county)? Can you cite a few examples? 9. In your area, is there good coordination between state and county medical staff. How often do you meet (frequency)? Such as? Strengthening the provision of program services Upgrading human resource capacity Sudan Health Transformation Project Phase II (SHTP-II) 92 Mid-Term Evaluation Report – Public Document Building infrastructure Ensuring the adequate supply of essential commodities Promoting the provision of pharmaceuticals through commercial outlets Other issues 10. In your opinion, what could be done to strengthen coordination between state and county levels of the health system? Are there priority needs with respect to financial management? 11. Is there a functioning monitoring and evaluation capacity in your state (county) as yet? What steps are being taken to build M&E capacity in your state (county)? How do you anticipate that a well-managed M&E system will contribute to improved management of health services in your state (county). 12. Is a surveillance system for infectious disease active in your areas? How current is the information that is being compiled? How effectively is it being used to monitoring changing health conditions in your area? 13. Do you have a rapid response capacity for dealing with disease outbreaks? Do you have sufficient stocks for drugs for emergencies? Sudan Health Transformation Project Phase II (SHTP-II) 93 Mid-Term Evaluation Report – Public Document INTERVIEW GUIDE FOR SHTP-II ASSESSMENT MEETINGS WITH COMMUNITY AND VILLAGE GROUPS INTRODUCTION OF TEAM AND PURPOSE OF VISIT Assess implementation of the Sudan Health Transformation Project Identify its strengths and areas where changes may be needed Consider future areas of USAID investment in the health sector in Southern Sudan If needed, give overview of SHTP-II. ATTENDEES Number and general description. Gender representation? Whether they can clearly explain the role they play in the community and village groups to which they belong? . Questions 1. How often does your community/village committee meet? How many people typically come to these meetings? 2. Have you received training relevant to your participation on the committee? 3. Have you received any training materials as part of this instruction? 4. Can you please describe what your community/village committee does? Can you describe the role you play on the community/village committee? 5. Does your Health Committee undertake information and education programs on health and the availability of services in your area? 6. Does your Health Committee have educational materials on health to distribute to the the community? Can you describe these materials? Are any selection criteria used to decide who receives these materials? 7 Where do people in your area go for health services when they need them? How do they get to the services? 8. In your opinion, how well received are the project clinic facilities in the communities and surrounding areas in which they are situated? 9. What do people in your area know about the PHCU or PHCC? What services are provided there? Know how to get there? If not, why not? If yes, what did they think? Of quality? Other aspects (waiting time, cost, availability of drugs)? 10. Are there other sources of healthcare in your area? If yes, can you describe these services? Do many people use them? 11. Please describe your engagement with your CHW (or other key health personnel). And your Home Health Promoter (HHP). 12. Does the county medical officer often attend your Health Committee meetings. How often? 13. Can you please provide examples of their support. 14. How do you address challenges or difficulties with health staff that may arise? 15. Is there a functioning monitoring and evaluation capacity in your state (county) as yet? What steps are being taken to build M&E capacity in your state (county)? How do you anticipate that a well-managed M&E system will contribute to improved management of health services in your state (county). 16. Is a surveillance system for infectious disease active in your areas? How current is the information that is being compiled? How effectively is it being used to monitoring changing health conditions in your area? Sudan Health Transformation Project Phase II (SHTP-II) 94 Mid-Term Evaluation Report – Public Document 17. Do you have a rapid response capacity for dealing with disease outbreaks? Do you have sufficient stocks for drugs for emergencies? Sudan Health Transformation Project Phase II (SHTP-II) 95 Mid-Term Evaluation Report – Public Document INTERVIEW GUIDE FOR SHTP-II ASSESSMENT MEETINGS WITH FACILITY-BASED STAFF INTRODUCTION OF TEAM AND PURPOSE OF VISIT Assess implementation of the Sudan Health Transformation Project Identify its strengths and areas where changes may be needed Consider future areas of USAID investment in the health sector in Southern Sudan If needed, give overview of SHTP-II. ATTENDEES Name, positions. Experience in health service delivery—where and how long; technical and management training; areas of expertise. Questions 1. Background on your work and service site. Catchment Area Hours of Operation Services provided (look at and get charts with statistics as available) Availability of staff, drugs, other supplies Availability of water, sanitation facilities Status and functioning of the Health Management Committee Problems/issues 2. What services are offered at your facility? Are services typically provided in all of the following areas? Child Health Nutrition Malaria Hygiene and Sanitation Practices Maternal Health Family Planning HIV/AIDS Prevention and Testing Services 3. How has the project supported you and/or your service site? What has the project provided? How? How/in what way has your NGO supported delivery of primary healthcare services? Essential drugs and equipment? Training? Infrastructure? Water and Sanitation? Community mobilization and communication? Are you aware of any other activities of SHTP-II? At central MOH? [May want or need to prompt re policies and guidelines development; development of management capacity, etc.] At State level? Other contributions? 4. How often have you had supervisory visits from your NGO headquarters office? How frequent are these visits? Sudan Health Transformation Project Phase II (SHTP-II) 96 Mid-Term Evaluation Report – Public Document 5. Now I would like to ask you some additional questions on the range of health services that are offered at your facility? What family planning and other reproductive health services are provided at your facility? Are contraceptive methods available, and if so which ones? Pills IUDs Injectables Implants Condoms M&F Sterilization Other RH Services HIV/VCT Services Other STI Screening (Diagnosis and Treatment) Post-abortion care Fistual services Male involvement Other What maternity care services are provided at your facility? Antenatal services: How often should women be seen for antenatal services? Do women typically have more than one antenatal visit in your facility? Are women provided with tetanus toxoid as part of their antenatal service? Delivery services: Are more women coming to prefer facility-based deliveries in your catchment area? Is any emergency obstetric care available at your facility for handling delivery complications? Are postnatal services for mothers and children offered at your facility? Are any family planning services offered to mothers in the six month period following delivery? Is LAM provided as a contraceptive method as part of your postnatal care? Does your facility offer all child immunizations (BCG, DBT, polio, measles, etc.) at your facility? What vaccines do you currently have in stock? Is ORS readily available for children with diarrhea in your area? What medicines do you prescribe for bloody diarrhea cases? What medicines do you provide for acute respiratory infections? Is Vitamin A supplementation provided for children? Can women receive iron supplementation tablets at your facility? 6. In a typical day, how many clients are seen at an SHTP-II facility? 7. Are your PHCC and PHCUs are currently staffed with appropriately trained cadres for providing IEC and counseling on: Vitamin A supplementation, Exclusive breastfeeding to 6 months and good maternal nutrition, Monitoring the nutritional status of children under seven, and nutritional supplementation to reverse malnutrition. How would you judge the effectiveness of these outreach efforts? 8. Does your facility offer HIV counseling and testing? Please describe? Sudan Health Transformation Project Phase II (SHTP-II) 97 Mid-Term Evaluation Report – Public Document 9. Are there civil society groups and clubs (e.g. churches, sports clubs, and drama groups) being trained in your area to provide peer education in abstinence and being faithful at population meeting spots? 10. Are your staff working in partnership with Home Health Promoters and TBAs to provide health outreach programs in Family planning and other reproductive health services Maternity care Immunization, nutrition, and prevention/treatment of diarrhea and pneumonia among children HIV/AIDS Malaria prevention and case management Improved drinking water supply, hygiene, and sanitation 11. What infrastructure improvements have been made to your facility by the project? What future upgrades would you like to see at your facility? 12. Is your facility adequately staffed at the moment? 13. Have there been problems in having staff payed in-full and on time in health facilities? If yes, have these problems now been rectified? And how were they resolved? 14. Have you had problems in keeping essential drugs and equipment adequately supplied. Have you had stock-outs over the past year? What do you do in case you experience stock-outs and shortages? 15. Are clients referred to higher levels of care from your facility? If yes, what health conditions are most usually referred? Where do they typically go? Have referral mechanisms been strengthened as part of your program efforts? In what way? 16. Is there a functioning monitoring and evaluation capacity in your facility as yet? What steps are being taken to build M&E capacity in your facility? How do you anticipate that a well-managed M&E system will contribute to improved management of health service provision in your facility? 17. What community outreach programs exist to encourage greater interest in using your facility? 18. Has project funding been used to organize community-based BCC programs in your area? If yes, could you briefly describe these programs? How effective do you believe they have been? 19. What steps have been taken to ensure opportunity for women’s participation in County Health Department management, Village Health Committees and Water & Sanitation activities? 20. Do men often come to your clinics? If not, why not. Do they usually go to other facilities, rely on traditional healers and medicines, or go without care much of the time? Sudan Health Transformation Project Phase II (SHTP-II) 98 Mid-Term Evaluation Report – Public Document INTERVIEW GUIDE FOR SHTP-II ASSESSMENT MEETINGS WITH BENEFICIARIES (CLIENTS) SERVED BY PROJECT FACILITIES INTRODUCTION OF TEAM AND PURPOSE OF VISIT Assess implementation of the Sudan Health Transformation Project Identify its strengths and areas where changes may be needed Consider future areas of USAID investment in the health sector in Southern Sudan If needed, give overview of SHTP-II. ATTENDEES Name, positions. Questions 1. How often do you come to this clinic? 2. How far did you have to travel to come here? 3. Do you often bring your children to this facility for preventive care and treatment? 4. If coming for treatment, what type of illness did they have? 5. Have you been satisfied with the services you have received here? If not, why not. 6. Are the health workers at the facility courteous to you? 7. Do they clearly answer all the questions you have about your health needs? 8. How long do you typically have to wait to be seen by a health worker? 9. Do you judge the facility to be clean and comfortable? 10. Do you have enough privacy in the facility when being seen by a service provider? 11. Were you able to get the care that you needed at this facility? 12. If not, why not? - Health problem not treatable or referable at facility - Drugs for treatment not available - Health worker not available - Facility not open - Other 13. Do you find the cost of services at this facility reasonable? Are you sometimes reluctant to come because of the cost? Do you have friends and neighbors that do not become of they fear having to pay too much money? 14. What do you most like about coming to this clinic? 15. What do you most dislike about coming to this clinic? 16. What improvements would you like to make to the way the clinic is operating? Sudan Health Transformation Project Phase II (SHTP-II) 99 Mid-Term Evaluation Report – Public Document INTERVIEW GUIDE FOR SHTP-II ASSESSMENT MEETINGS WITH OTHER DONORS/STAKEHOLDERS INTRODUCTION OF TEAM AND PURPOSE OF VISIT Assess implementation of the Sudan Health Transformation Project Identify its strengths and areas where changes may be needed Consider future areas of USAID investment in the health sector in Southern Sudan If needed, give overview of SHTP-II. ATTENDEES Name, positions. Specific information about them, as feasible Experience in Southern Sudan and with GOSS, MOH; prior experience; areas of expertise. Questions 1. Can you briefly describe the main components of you health program in Southern Sudan? How would you characterize the role of your organization in relation to the efforts of other donors? 2. Do you know about the activities of the USAID-funded SHTP-II project? 3. From what you know about SHTP-II, do you believe it has been making important contributions to the provision of health services in Southern Sudan? Health system strengthening? 4. What do you believe have been some of the project’s important contributions? At central MOH? [May want or need to prompt re policies and guidelines development; development of management capacity, etc.] At State level? At County level? In delivery of primary healthcare services? Training? Infrastructure? Water and Sanitation? Other contributions? 5. From your experience, what have been the strong elements of SHTP-II’s work? Are these elements that you think should continue? Why? 6. What are areas where you think that SHTP-II has not done very well? Are they areas where you think assistance is still needed? If so, how would you change what SHTP-II has done? 7. How well has SHTP-II coordinated its work with other stakeholders working in the Health Sector in Southern Sudan? What could it do to improve its coordination and support role? 8. What future priority needs would you like your organization to strengthen? In your opinion, where is the greatest need in terms of service delivery and the project management? 9. Are there other activities or areas that you think SHTP-II should be giving greater attention (emphasis) during the remaining period of operation? Sudan Health Transformation Project Phase II (SHTP-II) 100 Mid-Term Evaluation Report – Public Document OBSERVATION GUIDE FOR SHTP-II ASSESSMENT VISITS TO HEALTH SERVICE SITES INTRODUCTION OF TEAM AND PURPOSE OF VISIT Assess implementation of the Sudan Health Transformation Project Identify its strengths and areas where changes may be needed Consider future areas of USAID investment in the health sector in Southern Sudan If needed, give overview of SHTP-II. ATTENDEES Name, positions. 1. Outside area, around the site Cleanliness Evidence of appropriate garbage disposal? Medical waste disposal adequate? Ease of access of site to community? Availability of water and clean sanitation facilities? Availability of soap and facilities for hand washing? 2. Inside the service site Waiting area: Suitable? Clean? People able to sit? Educational materials? Crowded? Exam room(s): Suitable? Clean? Private? Other spaces for records, drugs, etc.: Suitable? Clean? Organized appropriately? Preserve privacy and restrict access, as needed? 3. Availability of staff and commodities Staffing: are there sufficient numbers for the client load? Do they have the requisite training and skills? Have they been paid regularly? Do they have a positive approach to their work and client care? Do they understand the need to work in the community? Drugs and other commodities: Are there adequate supplies of essential drugs and other supplies given the client load? Are they accessible? Are the supplies well-organized to support sound logistics management? 4. Organization of services Does the client flow facilitate access and reduce waiting time? Do all staff members have clear roles and responsibilities? Do they make the clients feel welcome and cared for? Is there a system for identifying those clients in need of emergency or time-sensitive services? Are clinic records being systematically maintained? Are they being used to monitor the performance of the facility? 5. Quality of services If possible, observe service provision. Does provider apply most up-to-date international guidelines in providing that specific service? Does the record-keeping system support high quality services? Are the records being kept? How are they being utilized? For example, are they being used to monitor the performance of the facility? Does the client get adequate information about what the provider is doing or will do? Does the client get adequate information about what s/he needs to do for follow up? Sudan Health Transformation Project Phase II (SHTP-II) 101 Mid-Term Evaluation Report – Public Document Does the provide counsel client to identify other services that may be needed? And then provide them, or refer to someone else? Catchment Area Clients/day, month Services provided (look at and get charts with statistics as available) Outreach to the community—how, how often? Status and functioning of the Health Management Committee