Photo by Pace Moreno Bongiovanni SUBJECT: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO At the request of the United States Agency for International Development (USAID), this publication was prepared independently by International Business and Technical Consultants, Inc. (IBTCI) and written by Swati Sadaphal and Annette Bongiovanni. Photo by Pace Moreno Bongiovanni Photo by Pace Moreno Bongiovanni IMPACT EVALUATION March 20, 2016 Impact Evaluation Results-based Financing in the Democratic Republic of Congo USAID/Democratic Republic of Congo AID-660-M-13-00001/GS-10F-0309P Authors: Swati Sadaphal and Annette Bongiovanni Submitted: March 20, 2016 Prepared for: Raymond Grant, Contracting Officer Representative United States Agency for International Development/DRC USAID/DRC 1980 Isiro Avenue Kinshasa/Gombe, DRC Prepared by: International Business & Technical Consultants, Inc. (IBTCI) 8618 Westwood Center Drive Suite 400 Vienna, VA 22182 USA Tel: +1 (703) 749-0100 Cover Photos and Credits: Children from a village in the DRC: Pace Moreno Bongiovanni Child playing soccer in a Congolese village: Pace Moreno Bongiovanni Mother and child: Pace Moreno Bongiovanni This evaluation report was made possible by the support of the United States Agency for International Development (USAID) under contract number AID-660-M-13-00001/GS-10F-0309P. The contents of this publication are the responsibility of the authors and do not necessarily reflect the views of USAID or the United States Government. RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO IMPACT EVALUATION March 20, 2016 04 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO ACKNOWLEDGMENTS IBTCI and the evaluation team would like to take this opportunity to thank the many individuals and institutions whose talent, dedication, and generosity have made this report possible. While we hope what follows is thorough, we acknowledge that it will inevitably fail to be exhaustive. The team would like to thank the USAID/DRC staff members who provided essential support for this evaluation. Our sincere appreciation goes to the Health Team Leader Meri Sinnitt, Program Management Specialists (Courtney Babcock, John Bernon, Kai Beard, Elena Facchini and Raymond Grant), Monitoring and Evaluation (M&E ) Specialist Antoine Mafwila, Health System Strengthening Specialist Dr. Godefroid Mayala, and IHP Technical Advisor Dr. Richard Matendo. Together, they provided consistent and critical guidance. We would also like to thank the entire Integrated Health Project (IHP) staff, and particularly Delmond Kyanza, Ousmane Faye, and Sam Mbuyamba, who greatly contributed to the evaluation teams’ understanding of the IHP PBF pilot activity. In addition, we would like to thank the Ministry of Public Health and the dozens of directors, managers, and healthcare providers who offered us their support and expertise. We’d particularly like to thank Dr. Mukengeshayi Kupa, Ministry of Public Health (MSP) Secretary-General, who provided invaluable support and facilitated activities at the provincial, district, and zonal levels. Special thanks also go to Dr. Celestin Bukanga, Head of MSP’s Central RBF Unit, in addition to his team members. We are extremely grateful for the support from the Ministry of Public Health officials in East Kasaï, West Kasaï, Katanga, and South Kivu provinces, health districts and health zones. These Democratic Republic of Congo (DRC) government representatives were accommodating and assisted the implementation of the field study immeasurably. The team applauds the work of the study coordinator, Zephyrin Kanyinda. We would also like to thank the field supervisors, Jean-Paul Mwamba, Jean-Pierre Ilunga, Lionel Mambandu, Nadine Masamba, Pascal Bukasa, Dr. Thierry Kamand A Tshikuz, Liliane Ahombo, Junior Nsikuetu, Campo Mandembo Likwela, Edwige Onan, Chouchouna Bualungu, and Richard Muzang, who diligently monitored the data collection process. We are grateful to Jean-Lambert Mandjo Aholoma for his field management oversight and keeping everyone safe. We would also like to thank all of the data collectors and interviewers whose hard work provided the raw material for this evaluation. We are grateful to have communications eased by our interpreters, Hubert Kinwa, Yulu Alphonse Kabama and Richard Mungala. We would also like to thank Kapil Ahmed, a biostatistician, who helped during sampling strategy development. We are also grateful to Professor Patrick Kayembe, Dr. Jorge Velasco and Armand Utshudi, whose technical contributions during the data analysis and report writing process were invaluable. We would like to extend our sincere thanks to Salima Mutima, Susan Kupperstein, Anshikka Singh, Rachel Brown, and Faustin Ikoko, for their work in facilitating and managing the administrative and logistical components of the project, and Nikita A. Gurudas and Tyler Wallace for the report preparation, secondary analyses, and assistance with data visualization. The team would like to finish by thanking the study participants and respondents whose generosity and candor was absolutely essential for this evaluation. It is in this spirit of gratitude and humility that we now proceed with the report. n i TABLE OF CONTENTS LIST OF TABLES AND FIGURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii LIST OF ANNEXES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv ACRONYMS AND ABBREVIATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vi EXECUTIVE SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Evaluation Purpose & Rationale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Project Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Evaluation Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Evaluation Design and Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Quantity and quality of MPA services at health centers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Health promotion activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Health facility management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Costs associated with a potential replication of the pilot RBF model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Results differ for various groups (heterogeneity) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Factors contributing to or limiting the desired results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Unintended consequences of the intervention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 In Sum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Evaluation Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Evaluation Rationale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Evaluation Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 BACKGROUND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Country Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Results-based Financing (RBF) in the DRC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Project Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Integrated Health Project (IHP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 EVALUATION METHODS............................................................................. 14 Evaluation Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Site Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Survey Sampling Strategy and Sample Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Household survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Evaluation Team . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Data collection tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Additional data collection following the preliminary data analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Ethical Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Data Quality Assurance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Training and pilot testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Data quality check procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 ii n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Strengths of Evaluation Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Limitations of Evaluation Methodology................................................................ 19 Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 EVALUATION FINDINGS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Analytical Domain: Quantity and Quality of Health Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Facility sample characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Household sample characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Overview of significant findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Preventive MPA services (quantity and quality) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Preventive MPA service utilization pattern among households (quantity) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Preventive MPA services among households (quality) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Curative MPA service utilization pattern among households (quantity and quality) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Curative MPA services (quality) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Health promotion activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Health facility management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Analytical Domain: Cost Associated with a Potential Replication of the Pilot RBF Model . . . . . . . . . . . . . . . . . . . . . . . 40 Analytical Domain: Results Differ for Various Groups (Heterogeneity) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Household survey results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Facility survey results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Quality of services results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 RBF scores analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Analytical Domain: Contextual Factors Contributed to or Limited the Desired Results . . . . . . . . . . . . . . . . . . . . . . . . 46 Analytical Domain: Unintended Consequences of the Intervention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Positive consequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Negative consequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 DISCUSSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Interpretations of RBF Successes and Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Equity of RBF Incentive Payments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Measuring Quality of Care . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 CONCLUSIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 RECOMMENDATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 In Sum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 BIBLIOGRAPHY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 n iii LIST OF TABLES AND FIGURES Figure 1. Disaggregation of RBF payments for GRHs and HCs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Figure 2. Timeline of evaluation activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Figure 3. Map of selected sites for RBF impact evaluation........................................................... 15 Figure 4. Illustration of difference in difference (DID) estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Figure 5. Percentage of women receiving ANC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Figure 6. Percentage of women receiving TT injections during ANC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Figure 7. Percentage of women examined by a skilled health worker after childbirth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Figure 8. Percentage of women who received first dose of vitamin A within 2 months of childbirth . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Figure 9. Percentage of children (0–23m) who received a dose of vitamin A within 6 months . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Figure 10. Source of vitamin A . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Figure 11. Percentage of current users of any family planning methods and modern contraceptive users . . . . . . . . . . . . . . . . . . . . . 28 Figure 12. Type of modern contraceptive method used . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Figure 13. Received family planning advice from a health worker in the past one year . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Figure 14. Percentage of respondents who sought treatment in a local health center when needed health service . . . . . . . . . . . . . 30 Figure 15. Cost of services paid during last visit to a local HC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Figure 16. Reasons given for not vising a health facility when needed health service, 2015 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Figure 17. Percentage of women gave birth in a health facility and percentage assisted by a skilled health worker . . . . . . . . . . . . . . 31 Figure 18. Quality of Care Measurement Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Figure 19. Percentage of children under 5 receiving appropriate treatment for diarrhea or PP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Figure 20. Percentage of children receiving appropriate treatment for malaria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Figure 21. Percentage of women ever tested for HIV and received results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Figure 22. Percentage of women ever tested for HIV and received results while pregnant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Figure 23. Percentage of households reporting at least one LLITN and child slept under a bed net last night . . . . . . . . . . . . . . . . . . 36 Figure 24. Percentage of households making water potable before use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Figure 25. Percentage of households with improved water source and toilets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Figure 26. Trend analysis: Health Center FOSCAOF score, Intervention vs. Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Figure 27. Trend analysis: GRH FOSCAOF score, Intervention vs. Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Figure 28. Health Center FOSCAOF scores by health zone, Baseline vs. Endline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Figure 29. GRH FOSCAOF scores by health zone, Baseline vs. Endline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Figure 30. Perceived quality of care received during a recent HC visit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Figure 31. Client satisfaction with the quality of care received during a recent HC visit (within 3 months) . . . . . . . . . . . . . . . . . . . . 39 Figure 32. Costs associated with 2 years’ RBF implementation in seven health zones . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Figure 33. RBF scores (Q2 2014 & Q2 2015) for HCs visited during the Midterm Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Table 1. Weighting of FOSACOF scores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Table 2. Selected sites for RBF Impact Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Table 3. MPA and CPA services (revised v2014) for IHP implementation and RBF-contracted service delivery indicators . . . . . . . . 22 Table 4. Household characteristics of intervention and comparison sites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Table 5. RBF services contracted and the difference the pilot RBF intervention made on HC service delivery . . . . . . . . . . . . . . . . 24 Table 6. Difference the pilot RBF intervention made on the health center service delivery indicators . . . . . . . . . . . . . . . . . . . . . . . 25 Table 7. Quantity of preventive MPA services at health centers: Availability of services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Table 8. Quantity of preventive MPA services at health centers: HC client statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Table 9. Quantity of curative MPA services at health centers: Availability of services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Table 10. Quantity of curative MPA services at health centers: Client statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Table 11. Knowledge of childhood illness symptoms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Table 12. Quantity of promotional MPA services at health centers: Availability of services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Table 13. Quality of MPA services at health centers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Table 14. RBF funding: Integrated Health Project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Table 15. Initial budget vs. executed budget in seven HZs (10/2013–9/2015) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 iv n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO LIST OF ANNEXES (provided in a separate volume) ANNEX A Statement of Work ANNEX B RBF Indicators for Health Zone Management Team ANNEX C RBF Indicators for Health Centers ANNEX D Data Collection Sampling Procedure ANNEX E Field Implementation Plan ANNEX F Reference List ANNEX G Survey Questionnaires ANNEX H Training Manual ANNEX I Data Analysis Plan ANNEX J Health Facility Survey Sample Distribution ANNEX K Household Survey Sample Distribution n v ACRONYMS AND ABBREVIATIONS AAP Agence d’Achat de Performance/Purchasing Agent AIDS Acquired Immune Deficiency Syndrome ANC Antenatal care ARI Acute respiratory infection ART Antiretroviral therapy ARV Antiretroviral BC Bureau de Coordination/IHP Coordination Bureau BCC Behavior Change Communication CBO Community-based Organization CCM Community Case Management CDF Congolese Francs CHW Community health worker C-IMCI Community-based integrated management of childhood illness CODESA Comité de Développement Sanitaire/Health Development Committee CPA Complementary Package of Activities CPN Consultation Prénatale/Antenatal Care (ANC) CPON Consultation Postnatale/Postnatal Care (PNC) CPR Contraceptive prevalence rate CSO Civil society organization DHS Demographic and Health Survey DID Difference in difference DPS Division Provinciale de la Santé/Provincial Division of Health DPT Diphtheria, pertussis, tetanus DRC Democratic Republic of Congo ECZ Équipe Cadre de Zone/Health Zone Management Team (HZMT) EPI Expanded Program on Immunization FBO Faith-based organization FGD Focus group discussion FOSA Formation Sanitaire/Health Training FOSACOF Formations Sanitaires Complètement Fonctionnelles/Fully Functional Service Delivery Point FP Family planning GDRC Government of the Democratic Republic of Congo GRH Hôpital Général de Référence (HGR)/General Reference Hospital HC Centre de Santé (CS)/Health Center HF Health facility (includes health centers, GRHs and health posts) HIV Human Immunodeficiency Virus HSSS Health System Strengthening Strategy HZ Zone de Santé (ZS)/Health Zone HZMC Health Zone Management Committee IBTCI International Business and Technical Consultants, Inc. IEC Information, Education and Communication IHP Projet de Santé Intégré (PROSANI)/Integrated Health Project IMCI Integrated management of childhood illnesses IR Intermediate Result IRC International Rescue Committee ITN Insecticide treated bed net KAPs Knowledge, Attitudes, and Practices KII Key Informant Interview LDP Leadership Development Program LLITN Long-lasting insecticide treated net LQAS Lot Quality Assurance Sampling MCH Maternal and Child Health M&E Monitoring and Evaluation MIP Médecin Inspecteur Provincial/Provincial Medical Inspector MNCH Maternal, Newborn and Child Health MONUSCO United Nations Organization Stabilization Mission in the Democratic Republic of the Congo MPA Minimum Package of Activities MSH Management Sciences for Health MSP Ministère de la Santé Publique/Ministry of Public Health NGO Non-Governmental Organization NHDP National Health Development Plan vi n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO NIS Système Nationale d’Information Sanitaire/ National Health Information System of the DRC NTD Neglected tropical disease ORS Oral rehydration Salts ORT Oral rehydration therapy OSC Overseas Strategic Consulting, Ltd. PARSS II Projet d’Appui à la Réhabilitation du Secteur de la Santé/ Health Sector Rehabilitation and Support project PEPFAR President’s Emergency Plan for AIDS Relief PMP Performance Monitoring Plan PMTCT Prevention of Mother-to-Child Transmission of HIV PNC Prenatal consultation/care PNDS Plan National de Développement Sanitaire/ National Health Development Plan PNLP Programme National de Lutte contre le Paludisme/ National Malaria Control Program PNLS Programme National de Lutte contre le SIDA/ National AIDS Program PP Presumed pneumonia PPS Probability proportional to population size RBF Financement Basé sur les Résultats (FBR)/Results￾Based Financing RH Reproductive health STI Sexually transmitted infection TB Tuberculosis TBA Traditional birth attendant TFR Total fertility rate TT Tetanus toxoid UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund USAID United States Agency for International Development USD United States dollars USG United States Government WASH Water, sanitation, and hygiene WHO World Health Organization Definitions Minimum package of activities (MPA): MPA includes curative, preventive, promotional, and community outreach activities provided by a versatile team of health center staff, who are themselves supervised by the health zone management steam. MPA is to be provided at all health centers. Complementary package of activities (CPA): The complementary package of activities (CPA) includes the full MPA and the preventive, curative, and promotional activities that are organized within the framework of internal medical services, surgery, gynecology, obstetrics, and pediatrics. Management-related activities such as management of hospital health information; human, material, and financial resources; action research and supervision of health zone personnel are also included. CPA is to be provided at all general reference hospitals. Halo bias: Responses filtered through a desire to provide favorable answers to some questions more than others. Recall bias: Responses with questionable accuracy of recall. Recall bias is pronounced, for example, concerning medications obtained at a health clinic during a childhood illness. Respondent bias: Responses filtered through a lens of the perceived advantage or disadvantage that might follow a particular response. EXECUTIVE SUMMARY n 1 EXECUTIVE SUMMARY Evaluation Purpose & Rationale I nternational Business and Technical Consultants, Inc. (IBTCI) was contracted by the United States Agency for International Development/Democratic Republic of Congo (USAID/DRC) in December 2012 to conduct multiple external evaluations of the Integrated Health Project (IHP or PROSANI for the French acronym). These included a performance evaluation of IHP and a rigorous impact evaluation of the IHP’s pilot results￾based financing (RBF) activity. This evaluation report presents the final RBF impact evaluation results. The audience of the RBF impact evaluation includes: the USAID/DRC Mission and the implementing partners, Management Science for Health (MSH) and its consortia; the Ministère de la Santé Publique (MSP or Ministry of Public Health); donors involved in RBF programming and evaluation such as the World Bank; and other stakeholders focused on RBF interventions and DRC’s health system development. USAID/DRC will use the impact evaluation results to make an evidence-based decision to inform policy and program changes, including whether to continue and/ or expand the RBF intervention to other areas within the DRC. Project Background As part of a financing strategy under the IHP, MSH implemented a pilot RBF intervention in seven health zones (HZs) in four targeted provinces. The target population benefiting from pilot RBF activity was estimated at 955,427 inhabitants in seven HZs. RBF was implemented at three levels of intervention within the MSP health system: 1. The central level in Kinshasa including the MSP RBF Unit, USAID/DRC, and IHP 2. The provincial level including the MSP Provincial Health Office, and the provincial IHP 3. The operational level including the zone management team (Equipe Cadre de Zones (ECZs) and the health facilities (General Reference Hospitals (GRHs) and Health Centers (HCs)). RBF activities commenced in all seven intervention HZs in November 2013, following the execution of performance contracts with IHP, facility directors, and MSP managers: individual contracts were signed with 118 HCs, seven GRHs, and seven ECZs. MSP’s RBF unit staff were trained on RBF processes and mentored during quarterly verification visits and data validation during the first six months of implementation. Each RBF cycle lasted for a period of three months (one quarter). The performance of each ECZ team was assessed using thirteen indicators. These were mostly contingent upon the performance of the health facilities under their purview, and the ECZ’s supervision of them. At the GRH level, there was one (composite) indicator, MSH’s Fully Functional Service Delivery Point (FOSACOF for the French acronym) facility assessment tool. FOSACOF was primarily focused on inputs and did not include any measurement of service delivery. Lastly, HCs were measured against 16 indicators, many of which reflect services delivered. One MSP provincial-level officer, accompanied by one IHP staff member, conducted the verification of the ECZ level RBF results. Technical verification of reported data from the health facilities was conducted jointly by one member from the ECZ team and one IHP staff member. Two civil society organizations (CSOs) per HZ, under a fixed-price contract with IHP, conducted household interviews to verify clients and ensure patient satisfaction. Once results were verified, payments were processed. The maximum RBF incentives possible for a hospital each quarter was $12,000 and for a health center was $910. Payments, aimed at incentivizing staff performance, were distributed as follows: staff incentives (60%), investments (i.e., infrastructure, equipment) (30%), and operations (i.e.,registers, office supplies) at the HC and GRH levels (10%). 2 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Evaluation Questions The final RBF impact evaluation aimed to answer the following key questions: 1. Is there evidence of change among health centers in the quantity and quality of services? 2. What difference did the RBF intervention make? 3. Is the model worthy of being scaled up to other health zones? 4. What costs are associated with a potential replication of the model? 5. Were the desired results achieved? 6. Do results differ for various groups? (heterogeneity) 7. What contextual factors contributed to or limited the desired results? 8. What are the unintended consequences of the intervention? Evaluation Design and Methods The RBF impact evaluation is a prospective, quasi-experimental design with intervention and comparison groups with measurements taken at baseline (2013), midline (2014) and endline (2015). Baseline data collection in the intervention and comparison HZs was conducted between May and September 2013, prior to the commencement of the pilot RBF intervention in November 2013. The midterm process evaluation was conducted in October 2014 at four of the seven RBF intervention health zones. The endline data were collected between May and September 2015, covering the same intervention and comparison sites studied during the baseline, to ensure pre/post intervention comparability. The evaluation used a mix of quantitative and qualitative data collection strategies. The utilization of qualitative or quantitative methods provided different types of data and addressed different evaluation questions. Data collected via quantitative strategies (household surveys and health facility surveys) are primarily used for evaluating whether there is a relationship between the pilot RBF intervention and an effect (mainly in terms of the quantity and quality of health services). The quantitative methodology measured changes in key outcome indicators between the baseline and endline results. At midterm, a process evaluation used qualitative methods including key informant interviews and focus group discussions at pilot RBF intervention sites. The RBF Midterm Assessment Report (http://pdf.usaid.gov/ pdf_docs/PA00KFHS.pdf), published in April 2015, provides detailed analysis that responds to evaluation questions relating to the contextual factors and unintended consequences of the pilot RBF intervention (Sadaphal S. and Bongiovanni A., 2015). MSH, in consultation with the MSP, selected seven health zones for the pilot RBF intervention. The intervention site selection was based on nine selection criteria described in the RBF Manual 2011. All seven pilot RBF intervention health zones (100%) were selected to be included in the intervention group of the evaluation study. The IBTCI evaluation team selected one comparison health zone for each of RBF’s intervention health zones. A total of 176 facilities (167 HCs + 9 GRHs) and 186 facilities (172 HCs + 14 GRHs) were surveyed at baseline and endline respectively. At the endline household survey, a total of 3,610 respondents and at baseline, 3,588 respondents completed the household questionnaires. Quantitative data related to the costs of implementing the RBF approach were collected using a Microsoft Excel®-based cost analysis tool. This instrument was shared with the MSH Kinshasa office, which in turn shared it with their IHP coordination offices (Bureaux de Coordination or BCs) to input cost data from the seven intervention sites. The quantitative survey data were analyzed using STATA version 12. The impact evaluation used the difference in difference (DID) estimation methodology, chi square and Student t testing, and multivariate regression. Preliminary data analysis was presented at the USAID debrief on December 8, 2015. In consultation with USAID/ DRC, additional information from key informants (n=26) was collected at intervention and comparison health zones. The purpose was to reconfirm the absence of any unknown confounding factors affecting the impact results. One of the limitations of the methodology was that the intervention and comparison groups were statistically different with regard to socio-demographic profile, primarily in terms of employment status and place of residence. The comparison group had more urban areas than did the intervention group. The evaluation team mitigated these differences by using the DID methodology and using multivariate regression analysis. The household and health provider interviews might be subject to respondent bias. Halo bias may be a factor since health providers may have reported what they should do instead of what they actually do. To mitigate these biases, the questionnaires included some measures of direct observation. Also, the survey team was trained extensively on interviewing skills and avoidance of probing to minimize respondent and halo bias. EXECUTIVE SUMMARY n 3 Findings Quantity and quality of MPA services at health centers The minimum package of services (MPA) provided by HCs and the complementary package of services (CPA) provided by GRHs included the following categories and activities, each paired to a contracted service delivery indicator: Preventive MPA Services, Curative MPA Services, Health Promotion Services, and Management and Other Activities. Preventive MPA services Quantity: The availability of preventive maternal and child health and family planning services was maintained at high coverage levels (≥90%) between the two groups, except for growth and development monitoring services for children under five years (76%). RBF appears to be a contributing factor for maintaining the coverage of growth and development monitoring services in the intervention group. The average number of women attending one antenatal consultation increased significantly in both the groups, whereas postnatal consultations decreased significantly in both groups. The number of children receiving the diphtheria, tetanus, and pertussis (DPT1) vaccine over the past year, and family planning (FP) consultations, increased significantly only in the intervention group. The average number of children receiving full immunization at the health center did not show any significant changes in both the groups. Household survey showed that more than 85% of women reported receiving at least one antenatal consultation. There was a significant decrease by 3% in the comparison group, with no significant change in the intervention group. RBF appears to be the contributing factor in preventing the decline in the intervention group. There was, however, a significant decline in the percentage of women receiving at least four antenatal consultations in both groups. Contact with a HC nurse in the previous month was significantly linked to the number of women receiving at least four antenatal consultations. In the intervention group, 74% of mothers received at least one dose of tetanus toxoid (TT) and 47% received at least two doses while pregnant with their youngest child. There were significant reductions in women receiving TT while pregnant in the comparison group. RBF appears to be the contributing factor in preventing the decline in the intervention group. Quality: The examination of women by a skilled birth atten￾dant after childbirth decreased significantly in both groups. In the intervention group, about 26% of mothers received an initial dose of vitamin A within two months of childbirth. There was a significant reduction of Vitamin A provision in the comparison group. RBF appears to be the contributing factor in preventing the decline in the intervention group. A positive association was found with women who had contact with an HC nurse. Similarly, RBF’s contribution is verified by data showing how the youngest child received at least one dose of vitamin A within six months of birth in the intervention group. The reported source of vitamin A given by 65% of respondents was “primarily during a campaign,” followed by 25% “during a routine visit.” Use of any type of contraceptives increased significantly in the comparison group but not in the intervention group. However, modern contraceptive use increased significantly in the intervention group, significantly higher than in the comparison group. RBF’s contribution is seen as increasing the use of modern contraceptive. The most common method used was male condoms (61%), followed by injections (20%) and pills (13%). Long-lasting methods accounted for only 1%. Those who had contact with an HC nurse were found to be most likely to use a modern method. Mothers’ reception of FP ad￾vice from a health worker improved significantly in both groups. The most common source of FP information in both groups was an HC nurse (75%), a community health worker (CHW) (9%), and a birth assistant (9%). Eighty-six percent (86%) had the con￾tact with a health worker at a local health center. Curative MPA services Quantity: The availability of normal childbirth services, nutritional rehabilitation and care, and de-worming services improved significantly in both groups, whereas prevention of mother-to-child transmission of HIV (PMTCT) services declined. The referral services for obstetrics complications improved significantly in the intervention group, but not in the comparison group. RBF appears to be a contributing factor for this increase of high-risk obstetric referrals. The average number of outpatients coming to HCs and pregnant women referred to a GRH for obstetrics complications increased significantly in the intervention group compared to the comparison group. RBF appears to be a contributing factor for these improvements. The average number of pregnant women screened for HIV and normal births attended by facility personnel significantly improved in both groups. More than 80% of mothers had visited a local HC within the past three months of survey. A significant increase in the HC service utilization rate was observed for both groups. The RBF intervention appears to be a factor for positive change in health service utilization patterns. The cost of services during the recent visit in both groups was similar. There was, however, an increase in both groups in the overall cost of services paid during the last visit to a health facility. Costs for the initial consultation did not significantly increase in either the intervention or comparison group. The two most common reasons given by both groups for 4 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO not visiting a health facility were the inability to pay for services and the distance from a health center. The prevalence of facility births remained high at 84% in the intervention group, although there was a significant decline in the comparison group. Childbirth in a health facility is strongly associated with contact with an HC nurse. The presence of a skilled birth attendant during childbirth was similar in both groups. Quality: Malaria, diarrhea, and pneumonia are major morbidity and mortality factors in children under 5. One of IHP’s activities is to support integrated management of childhood illnesses (IMCI) and improve the quality of care related to childhood illnesses. For the purposes of this study, a Quality of Care Measurement Index was developed and is described in detail in the main report. Among several other variables, this Index was utilized to analyze the quality of care for children who were reported to have diarrhea, presumed pneumonia (PP), or suspected malaria two weeks before the household survey. About 37% of children reportedly suffered from diarrhea in the two weeks preceding the survey. In both groups, 50% of mothers of children with diarrhea gave them oral rehydration salts (ORS). Mothers who administered any form of ORS for diarrhea were found to have been in contact with an HC nurse in the previous month. Fifty-five percent (55%) of children reportedly suffered from a cough or PP in the last two weeks preceding the survey. Fifty-nine percent (59%) of children with PP were taken to a health facility. At the health facility, 66% of children with PP were treated with antibiotics, similar to the comparison groups (65%). Forty-five percent (45%) of children reportedly suffered from fever or suspected malaria in the two weeks before the survey, and 68% of children with fever were taken to a health facility. Twenty-nine percent (29%) of children with fever had their blood drawn for malaria testing, an increase in the intervention group. Malaria testing reduced in the comparison group. RBF appears to be a factor in maintaining significant higher rates of malaria testing in the intervention group. Antimalarial treatment rates declined significantly in both groups—by 30% at endline. The body of the report provides results for a host of variables that comprise three proximate determinants that contribute to appropriate care and treatment of illnesses. Those proximate determinants are health-seeking behaviors, structural inputs, and personnel. Health promotion activities Services related to general hygiene, sanitation, and exclusive breastfeeding improved significantly in the intervention group, with no change in the comparison group. Health promotion ser￾vices related to food hygiene and safety, improvement of latrines, use of ORS in diarrhea, and genital fistula prevention improved significantly in both groups, whereas iodized salt promotion improved significantly in the comparison group, with no change in the intervention group. Other health promotional services related to condom promotion and insecticide-treated bed nets for malaria prevention were maintained at high coverage levels. The availability of information, education, and communication (IEC) materials related to FP and vaccination improved significantly in the intervention health zones due to RBF’s intervention. However, IEC materials related to acute respiratory treatment decreased significantly in both groups—more in the comparison group than in the intervention group. RBF’s intervention was a factor in preventing a major decline in the intervention group. The availability of human immunodeficiency virus (HIV) prevention IEC material was significantly improved in the intervention group. IEC materials related to child growth monitoring, maternal and child health, nutrition, malaria, and diarrhea improved significantly in both groups. In households, the availability of at least one long-lasting insecticide-treated bed net (LLITN) declined significantly by 30% in both groups. In the households when LLITN was observed, more than 90% reported that their child slept under the bed net the previous night. Their findings were similar in both groups. The percentage of households receiving advice on the importance of water, sanitation, and hygiene (WASH), use of improved sources of water, and use of a method to make water potable increased significantly in the intervention group. The changes observed could be attributed to the RBF intervention. However, the availability of improved toilets did not show any change between the intervention and comparison group. Health facility management Facility supervision by MSP staff in the three months preceding the survey improved significantly in the intervention group, with 99% supervision rates. The average nurse-to-catchment popu￾lation ratio in the intervention and comparison health zones is 0.26 per 1000 population in 2015. Reports of delays in supply of medications or supplies decreased significantly in the intervention group. Stockouts for contraceptives were more frequent in the comparison group, while the health facilities in the intervention group had lower stockout rates. Although RBF’s intervention appeared to prevent contraceptive stockouts; however, stocks of vaccines remained low and unchanged in both groups. The availability of continuous water supply in the health facility building and access to transportation at all times improved significantly in the intervention group. This change could be attributed to the RBF intervention. Water sources in the form of rainwater cisterns or improved wells and the availability of continuous electricity by use of solar panels improved significantly in the intervention group. Infection control practices of the intervention HCs improved significantly, EXECUTIVE SUMMARY n 5 attributable to the RBF activities. The availability of disinfectant for infection control was reported in 98% of HCs. There was a significant decline in “boiling” as the predominant method for equipment sterilization and, at the same time, an improvement in “autoclave” and “dry or steam heat” as predominant methods. The predominant method of medical waste disposal in the intervention group at endline was incineration (at 71%)— outside incinerator (26%) or in an indoor incineration (45%)— compared to the burial method (28%). This was a significant improvement from the comparison HCs, where incinerator use was 29% and the burial method was 43%. FOSACOF Scores: GRHs improved their overall scores to 87% in the intervention group versus 57% in the comparison group by the end of the 6th quarter and Luiza GRH had highest percentage improvement for FOSACOF scores (+66%). Similarly, the overall scores improved for HCs from 33% at baseline to 72% at the end of 6th quarter in the intervention group; HCs in Bibanga health zone had highest percentage improvement for FOSACOF scores (+70%). Client satisfaction: Approximately 89% of household respondents expressed satisfaction with the overall quality of services received, with a significant increase in the comparison group and no change in the intervention group. The perception of interpersonal skills (a nurse listening attentively to the client and treating them professionally) improved significantly for both groups. The perception of the amount of time the service provider spent per client did not change for the intervention group but significantly improved for the comparison group. Costs associated with a potential replication of the pilot RBF model The original budget estimate was $1,284,680 for the preparatory phase (2011–2013) and $4,986,591 for the implementation phase (2013–2015), totaling $6,271,271 for an estimated pop￾ulation size of 960,000. This allowed for an estimated yearly in￾vestment of $2.60 per capita. The preparatory phase for the pilot RBF activity started in IHP’s third year. A total of $1,120,168 (17% of the total budget = $6,304,181) was invested in this phase. The actual cost for a two-year implementation increased by 4% to $5,184,013 representing a $2.70 yearly per capita expenditure. An important budget line item contributing to the budget increase was the operating cost at the central level (51%). The total budget for purchase of services amounted to $1,687,579, one third (33%) of the budget. It decreased 10% from the original budget, as did that of the HCs (9%). Hospi￾tal budgets increased 16%. Thus, the purchase of services went down from the initial estimate of $0.98 per capita to $0.88 per capita. The payment of services went to 118 HCs (15%), seven GRHs (12%), the ECZs (2%), and excellence purchases (payment of services rewarding good performance) in all facilities (4%). Two thirds of the resources (67%) were utilized for the management and verification of RBF activities. Findings from the 2014 RBF Midterm Assessment revealed that maximum RBF incentive paid to a hospital in one quarter was $12,000 and the maximum paid to a health center was $910 if the facility met or surpassed all indicator targets. Results differ for various groups (heterogeneity) An overview of the household survey results disaggregated by HZ, Table 14 in the main report, indicated that Wembo Nyama had the best performance, with significant improvements in nine of the total of 22 service delivery and behavioral variables (41%) evaluated. Lomela had a more diverse outcome, with significant improvements in seven of the total of 22 service delivery and behavioral variables (32%). Lomela only had one significant reduction at endline related to improved water source at the household level. Among the variables with no significant changes between baseline and endline, downward trends were exemplified for two variables in Kayamba, which showed by far the least success in achieving positive results. It had only one significant positive result across all 22 variables—improved water source at the household level. It also fared most negatively in the number of variables which measured lower at endline than at baseline. RBF scores for the health centers in four intervention health zones visited during the RBF Midterm Assessment were compared. Data from the RBF web portal was downloaded for Quarter 2, 2014 and Quarter 2, 2015 to observe changes in RBF scores over a one-year period. Overall, the RBF scores decreased over time for almost all HCs in all health zones. The least variation was seen in Wembo Nyama and the most variation was seen in Bibanga. Factors contributing to or limiting the desired results Data review from the RBF Midterm Assessment highlighted that there was a buy-in from various stakeholders for the IHP’s pilot RBF intervention. IHP RBF’s design was compliant with the MSP’s policies and directives. There were a few bottlenecks: lack of formal training of new facility staff; gaps between the reported data and the validated data; and lack of sufficient full-time dedicated staff at central and coordination office level to monitor and prevent delays in the payment of incentives. These were corrected immediately by MSH, based on the recommendations provided after this assessment. The coverage of households counter-verified remained less in remote areas. The creation of champion communities by IHP, to act as counter-verification 6 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO agencies when CSOs could not monitor health facilities, was not fully implemented. Before RBF, most service providers relied heavily on user fees to cover the operating costs of the facilities as well as to pay bonuses, or “primes”, to staff. Many staff were not even on the civil service payroll, deriving their remuneration solely from fees charged to patients. In the facilities visited by the Midterm Assessment team, less than 10% of staff members received salaries. RBF incentives provided the means for subsistence, augmenting user fees charged from clients. Many factors such as civil and political unrest and geographical accessibility might have had an effect on RBF implementation. Cultural factors impeding repeat antenatal, vaccination, and postnatal care visits were mentioned by 40% of key informants interviewed in 2016. Supply chain breakdowns, limited or absent electricity and running water supplies, medical waste management, and lack of money transfer services also affect implementation. Unintended consequences of the intervention One of the positive consequences of RBF was that accountability and transparency at operational levels were both promoted. The process of measuring, verifying, and validating data was instilled in the intervention zones. On probing, none of the respondents alluded to any unintended negative effects of RBF related to gaming, distortion, or cherry-picking. The RBF model did not address the differences in socioeconomic status of the target populations, the type of organization, or geographic variations. Thus, for example, the cost of living in Katanga province was not taken into consideration. In Bibanga, even though Human Immunodeficiency Virus (HIV) and Acquired Immunodeficiency Syndrome (AIDS) and tuberculosis (TB) activities are very limited, the HCs were still required to report on these indicators every quarter. There was also evidence of dissent at the provincial and national level over non-inclusion in RBF contracts. The central MSP RBF and Provincial Division of Health were highly supportive of the IHP RBF model and wanted more participation. More than 60% of national-level respondents noted that the current level of investment nationally is insufficient and that additional funding resources would be needed for RBF. Conclusions EVALUATION QUESTION 1: Is there evidence of change among health centers in the quantity and quality of services that is attributable to the RBF model? ■ RBF had a positive impact on quantity and quality of services delivered in HCs. Such services include outpatient curative consultations (new and old cases), antenatal consultations, institutional deliveries, obstetric referral, modern contraceptive use, and tetanus toxoid vaccination. Similarly, FOSACOF scores at HCs and GRH level improved, which indicate the inputs for providing quality care are better in the RBF group. ■ RBF activity did not have a negative effect on the availability or quality of non-incentivized services. ■ Recent contact with a nurse is an explanatory variable for several positive health-seeking behaviors, such as use of ORS for childhood diarrhea, using a bed net for malaria prevention, and getting a child tested for malaria. Contact with a health provider was more frequent in the health center and most people are already using the formal health system. The positive health-seeking behavior does not reflect increased outreach by the health workers. ■ Given that prices for services remained stable between baseline and endline and yet women paid more for services during their last visit to a health facility, this implies women are obtaining more services per visit than in the past. This is corroborated by the increase number of curative and preventative visits. EVALUATION QUESTION 2: What difference did the RBF intervention make? ■ RBF induced behavior change among HC staff and HZ managers, which in turn translated into increased demand for and utilization of services by the catchment population. ■ Community or client contact with an HC nurse improved because of the RBF intervention and was an important factor influencing the community’s positive health-seeking behaviors. EVALUATION QUESTION 3: Is the model worthy of being scaled up to other health zones? ■ The IHP RBF model at the health center level is worthy of scale-up. The pilot is well-designed at the HC level and can strengthen core health system functions, increasing value for money and accountability of the health system. EXECUTIVE SUMMARY n 7 ■ The RBF service delivery indicators at GRHs were introduced after the data collection period of this evaluation and therefore this intervention needs to be evaluated for an in-depth understanding of its effectiveness before the model is scaled up at the hospital level. ■ There is a “buy-in” from the MSP country stakeholders at all levels of the health system for the expansion of the IHP RBF model. Service providers have a strong desire for the RBF program to continue, as it provides a source of income within their manageable control. ■ The current capacity of MSP to function independently as a purchaser or verifier of services is not fully developed at this time. EVALUATION QUESTION 4: What costs are associated with a potential replication of the model? ■ The total budget associated with RBF implementation covering seven health zones for a two-year duration was $6,304,181, with $1,120,168 expended during the preparatory phase and $5,184,013 during the implementation phase. ■ Of the total executed budget of $5,184,013, 24% of that distribution was for human resources, 2% for investments (e.g., vehicles and equipment), 33% for service purchases (incentive payments), 15% for technical verifications, 11% for community verifications, 10% for regulatory improvements, and 5% for operating costs. ■ The budget used in this pay-for-performance scheme was estimated at $2.70 per capita per year, which is typical in similar contexts with output budgets ranging between $2 and $3 per capita per year. ■ There is a large discrepancy between the maximum RBF incentives possible for a hospital each quarter ($12,000) and a health center ($910). EVALUATION QUESTION 5: Were the desired results achieved? ■ Desired results of improved quantity and quality of services were achieved. While successes were not achieved across all indicators, there was considerable positive impact across a host of indices. This is most impressive, given the very short implementation period assessed (1.5 years of RBF implementation) and the difficulties faced in the DRC context and environment. ■ Sizable gains in some key coverage indicators were achieved through the introduction of RBF. The changes introduced are not uniform, however, and there are large variations across HZs and MPA service components. EVALUATION QUESTION 6: Do results differ for various groups (heterogeneity)? ■ Results differed between intervention HZs. Wembo Nyama is clearly the highest performing HZ, followed closely by Lomela. Bibanga is more comparable to these higher performing zones, while Nundu, Kanzenze and Luiza have similar, albeit average, results. Kayamba is by far the poorest performing zone. EVALUATION QUESTION 7: What contextual factors contributed to or limited the desired results? ■ IHP’s RBF design was compliant with the MSP’s policies, which contributed to its successful implementation at HCs, GRHs, and ECZs and achieved desired results. ■ IHP encountered a wide range of difficult environmental factors, such as security and accessibility limitations that hindered RBF implementation and negatively influenced the results in a variety of ways. The nurse-to-population ratio is far below the WHO recommendation of at least 2 nurses per 1000 population. Notwithstanding, the design of the pilot RBF was feasible. ■ The existing health system has serious resource management issues. There is an inadequate budget to provide regular (or any ) salaries to health staff, support for regular facility supervision is lacking and most of the facilities are not equipped to the standard level of infrastructure and equipment. EVALUATION QUESTION 8: What are the unintended consequences of the intervention? ■ RBF implementation clearly had positive unintended consequences. RBF introduced concepts of quality of care, target setting, business planning, work planning, and technical verification to ensure data quality. ■ Negative unintended consequences related to gaming, cherry-picking, and distortion were not found, but cannot be ruled out. ■ The opportunity to receive payment for their services motivated health providers to comply with RBF procedures and guidelines. ■ IHP RBF, by design, does not address the differences in target population socioeconomic status, type of organization, and geographic variations. The RBF design lacked demand-side incentives such as fee exemptions or waiver schemes financed through RBF for the benefit of the poor. On the supply side, the RBF design did not offer bonuses for remote environs. This could lead to unintended inequities. 8 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Recommendations ■ The RBF model at the HC level should be scaled up by USAID/DRC in a cascade fashion. The RBF model at the GRH level should first be solidified and the new model tested and evaluated before scale-up. ■ Continue payment for targets achieved for service delivery indicators at both the HC and GRH level. More attention is needed to setting targets, to ensure the criteria for their development is well-defined and understood and the rates are appropriate. ■ Consider the Quality of Care Measurement Index, presented in the main report, to provide a more comprehensive and well-rounded means for measuring the quality of care, rather than relying on the FOSACOF tool as a proxy for quality measurement, and to a lesser degree to personnel proximate determinants. FOSACOF does not address demand-side proximate determinants such as health-seeking behaviors. Moreover, there are no linkages between proximate determinants and treatment outcomes. ■ Promote and incentivize nursing outreach to the communities to educate the public on healthy behaviors and to seek care from trained providers. ■ RBF mechanisms and principles need to be integrated into the government financial system. Moving forward, the financial sustainability of successful RBF programs needs to be considered. The MSP, USAID and other stakeholders could consider harmonizing other donors’ funds as a part of a comprehensive financing strategy for the continuation of RBF in DRC. MSP should convene multi-disciplinary groups to develop a strategic plan (short-term and long-term) to address shortage of nurses in the health centers. ■ An analysis of the total ceilings of RBF incentives deserves some thought. Consideration of premiums to ensure equity should include: the total catchment populations served; the remoteness and geographic distribution of the inhabitants; the types of services provided (e.g., laboratory, minor versus major surgery, blood transfusions, facilitation of HC to GRH referrals (i.e. ambulances), etc.); and the heterogeneity of the facilities. More research is needed to identify equity factors that need to considered to establish ceilings of RBF incentives. ■ Consider the parity of RBF funds between the GRHs and HCs, based on an analysis of other RBF resource allocations at the facility level; consider a ratio that would account for more equity, including adjustments for remote catchment populations and local cost of living. In addition, GRHs should be held to the same rigorous expectation to meet service delivery targets as are HCs. ■ MSH should record the experience and lessons learned of the extended preparatory phase. In this way, if the process were to be repeated it could be modified in an informed manner to minimize the financial and time investment. ■ The MSP RBF Unit should convene roundtable discussions with the ECZs and selected health facility staff from each of the seven HZs to analyze the RBF results achieved. Using this roundtable forum, stakeholders could take an opportunity to newly analyze and interpret the data collected, with the benefit of knowledge on the ground and the internal and external factors that affect the RBF pilot’s outcomes. ■ Consider the advantages of employing a positive deviance approach to performance improvement and afford opportunities for Wembo Nyama and Lomela ECZs to showcase their best practices and lessons learned. ■ The malaria program should be involved in all zones, analyzing and solving the problem related to the treatment of children suspected of having malaria. This may warrant special attention to supply chain management of antimalarial. In Sum The global development community would benefit from a results-based approach that focuses on the outcomes achieved more so than the processes (Savedoff W., 2015). Governments and their constituents are best poised to determine their respective pathways leading to desired results. Indeed, the health system is embedded within a broader national framework, yet it is a place to start. Enjoying good health is not only a right but arguably a necessary ingredient to prosperity. In a very brief time period, this USAID-funded RBF program has demonstrated great promise. Each HZ and facility has taken its own pathway to achieve objectively measurable results. Of interest would be to examine the various and innovative approaches that originated at the health zone level which led to the successful results. EXECUTIVE SUMMARY n 9 Evaluation Purpose In September 2010, the United States Agency for International Development (USAID)/Democratic Republic of Congo (DRC) awarded the five-year $139,767,129 Integrated Health Project (IHP or PROSANI in French), Cooperative Agreement #AID￾OAA-A-10-00054, to Management Sciences for Health (MSH) and its partners, the International Rescue Committee (IRC) and Overseas Strategic Consulting Ltd (OSC). The five-year IHP (October 2010–September 2015) supported the DRC National Health Development Program (Plan National de Développement Sanitaire, PNDS) and had two components—“Health Services” and “Other Health Systems”—that were designed to create better conditions for, and increase the availability and use of, high-impact health services, products, and practices in 78 health zones (HZs) (formerly 80) in four of the DRC’s original 11 (now 26) provinces: Kasaï Occidental, Kasaï Oriental, Katanga,1 and South Kivu. As part of a financing strategy under the IHP, MSH implemented a pilot RBF intervention in seven HZs in the targeted provinces. This report evaluates the impact of this RBF intervention. USAID/DRC will use the impact evaluation results to make an evidence-based decision to inform policy and programming, including whether to continue and/or expand the RBF intervention to other areas within the DRC. 1. Kasaï Occidental, Kasaï Oriental, and Katanga provinces have since been partitioned into 10 new provinces. Evaluation Rationale International Business and Technical Consultants, Inc. (IBTCI) was contracted by USAID/DRC in December 2012 to conduct multiple external evaluations of IHP. These included a performance evaluation of IHP and a rigorous impact evaluation of the IHP’s pilot RBF activity. The impact evaluation of the IHP’s pilot RBF activity responds to the USAID’s evaluation policy of January 2011. The policy states that any project activity involving untested hypotheses and/or demonstrating new approaches, anticipated expansion in scale or scope through the United States (US) foreign assistance or other funding sources should undergo impact evaluation, if feasible. This evaluation report presents the final RBF impact evaluation results.2 Intended Audience The audience of the RBF impact evaluation is the USAID/DRC Mission, the implementing partners (MSH and its consortia), the Ministère de la Santé Publique (MSP) (Ministry of Public Health), donors involved in RBF programming and evaluation such as the World Bank, and other stakeholders focused on RBF interventions and DRC’s health system development. 2. The final performance evaluation of IHP is presented as a separate evaluation report (Sadaphal S., Bongiovanni A.: Final Performance Evaluation of the Integrated Health Project in the Democratic Republic of Congo, March 2016). INTRODUCTION Evaluation Questions The RBF impact evaluation was guided by the following questions. A complete description of this evaluation’s statement of work is provided in Annex A. 1. Is there evidence of change among health centers in the quantity and quality of services that is attributable to the RBF model? 2. What difference did the RBF intervention make? 3. Is the model worthy of being scaled-up to other health zones? 4. What costs are associated with a potential replication of the model? 5. Were the desired results achieved? 6. Do results differ for various groups (heterogeneity) ? 7. What contextual factors contributed to or limited the desired results? 8. What are the unintended consequences of the intervention? 10 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO BACKGROUND Country Context The Democratic Republic of Congo is a fragile state located in the heart of Central Africa and is home to approximately 74.88 million Congolese, occupying 2,345,410 square kilometers in sub-Saharan Africa (United Nations, 2015). The country is home to the world’s second-largest rain forest, and vast mineral resources. Unfortunately, these human and material riches are dramatically underutilized due to decades of armed conflict, economic mismanagement, and political instability. Consequently, the DRC is currently one of the poorest and least developed nations on earth. Some 63.6% of the population lives below the poverty line, and the country is ranked 176th out of 188 countries on the United Nations Development Program Human Development Index (United Nations Development Programme, 2015). In 2014, the DRC had a Gross Domestic Product of $33.12 billion, 1.3% inflation, and a per capita income of $380, one of the lowest in the world (The World Bank, 2014). Although the economic situation is universally dire, the bottom 20% of the population claims a mere 5% of the annual income, indicating significant wealth disparity (The World Bank, 2012). The country’s macroeconomic indicators have been consistently positive over the past several years, illustrated by an average growth rate of 7.4% between 2010 and 2013 and the successful reigning in of inflation from a catastrophic 34% in 2009 to a sustained 1% from 2013–2015 (The World Bank, 2016). However, upon closer inspection it is apparent that these macroeconomic statistics are predominantly driven by a tremendous surge in extractive industries and high commodity prices, which yield little job growth and highly concentrated wealth. Indeed, 80% of the population is engaged outside of the formal labor market and 73% of the population is unemployed. Of the 9,000 Congolese students graduating from university each year, fewer than 100 will find a job in the formal market (African Development Bank, 2012). Similarly, the time required to start a business has declined precipitously from 84 days in 2011 to 11 days in 2016, an indication of substantial regulatory reform. However, the World Bank ranks the DRC 184th out of 189 countries in the 2016 “Doing Business” annual report (The World Bank, 2016). This serves as yet another indication that structural and macro level reforms are not significantly enhancing the business environment or translating into improved job opportunities for the local population. The DRC remains on the International Monetary Fund and World Bank’s Heavily Indebted Poor Countries list, and depends on the special assistance that status affords (International Development Association, International Monetary Fund, 2016). There are currently 1.6 million internally displaced persons in the DRC, a result of the ongoing conflict between the national army, backed by the United Nations Organization Stabilization Mission in the Democratic Republic of the Congo (MONUSCO) Force Intervention Brigade, and several dozen different armed groups (United Nations Security Council, 2016). In addition to the tremendous loss of life, human suffering, and wasted human potential, this state of perpetual internal conflict destroys domestic infrastructure, stifles foreign investment, and hinders service delivery across sectors. The situation in the DRC constitutes an open-ended and multifaceted state of emergency, nowhere more keenly evident than in the health of the nation’s population. On the one hand, the conflicts claim the lives of countless young men. On the other, poor reproductive, maternal, newborn, and child health services decimate the country’s women and children. As a result, a staggering 52% of the population is 15 years old or younger, and the maternal mortality ratio is 846 per 100,000 live births, well above the Millennium Development Goals target of 250 per 100,000 for 2015 (Ministry of Monitoring, Planning and Implementation of the Modern Revolution; Ministry of Public Health; ICF International, 2014). The current fertility rate is 6.6 and the population is projected to increase by a monumental 58% to 103.7 million by 2030, highlighting the need for immediate and significant strategic reform (Ministry of Monitoring, Planning and Implementation of the Modern Revolution, Ministry of INTRODUCTION n 11 Public Health, ICF International, 2014; United Nations Population Fund, International Confederation of Midwives, World Health Organization, 2014). Diarrheal diseases, lower respiratory infections and malaria are the top three causes of death, accounting for 12.5%, 11.9%, and 5.9% of all deaths, respectively (Institute for Health Metrics and Evaluation, 2013). Nutrition indicators are equally disturbing. An alarming 23% of children under 5 are “severely stunted,” a condition associated with chronic and lifelong health complications (Ministry of Monitoring, Planning and Implementation of the Modern Revolution; Ministry of Public Health; ICF International, 2014). In response to this ongoing crisis, the national government has taken significant steps in the past decade to decentralize the health sector, identify the most effective health interventions, and coordinate with private, bilateral, and multilateral donors. In 2006, the Ministry of Health unveiled the expansive National Health System Strengthening Strategy, a robust reform agenda intended to restructure and fortify the health sector. The associated Compendium of Health Zone Standards established a Minimum Package of Activities (MPA) for health centers as well as a Complimentary Package of Activities (CPA) for general hospitals. These two packages collectively constitute the country’s Essential Package of Health Services and serve to establish a set of standardized priorities and evaluation criteria (Wright, 2015). Infrastructure development in the DRC must remain a significant policy and investment priority, particularly given the need for reliable transportation networks for healthcare service delivery. Unfortunately, only 3,000km of the nation’s roads are paved, while more than 30,000km are unpaved. There is no national network of roads connecting economic and demographic centers, with both paved and unpaved roads taken into consideration. Of the paved and unpaved roads, only 70% and 42% are in good or fair condition, respectively (Foster & Benitez, 2010). However, there has been significant emphasis on road building since 2002 and the DRC has received considerable external investment. The nation’s fledgling railways remain dilapidated and mostly used for cargo. Fortunately, the DRC has and extensive internal water network, which offers enormous potential for transportation services and energy production. Although the DRC currently suffers from extensive power shortages, the nation has the potential to be the largest power exporter on the continent. Its expansive waterways offer remarkable opportunities for hydropower production at a low cost. Very little of this potential is currently being harnessed. The national electricity company, Société Nationale d’Electricité, operates with hidden costs amounting to 595% of revenue, and suffers a distribution loss rate of 40%. The resulting energy environment is particularly unstable. The nation experiences 213.5 days of power outages per year, compared to an average of 11.1 for fragile states. These frequent blackouts have forced approximately 40% of firms to purchase backup generators, and continues to exact a substantial toll on healthcare facilities (Foster & Benitez, 2010). According to the World Bank “Getting Electricity” measure, the DRC ranks 174th out of the 189 countries studied (The World Bank, 2016). The expansion of mobile phone connectivity and usage in the DRC is a considerable success worth noting. Mobile cellular subscriptions have risen from 30.6% in 2012 to 53.5% in 2014. The resulting connectivity enables citizens to communicate with healthcare providers and businesses in remarkable and impactful ways. The population depends heavily on mobile phone service, while only 3% of the population uses the Internet (The World Bank, 2016). The DRC is a nation that has endured decades of instability. While many indicators suggest that the nation is on a slow and steady path toward development across sectors, progress is by no means guaranteed. If infrastructure spending and inefficiency remain at their current level, more than 100 years are estimated to elapse before the country’s current infrastructure deficit is redressed (Foster & Benitez, 2010). Sustained foreign interest, innovative technologies and continued fiscal responsibility are essential if the DRC is to achieve its economic and human potential. Results-based Financing (RBF) in the DRC In 2005, Cordaid in Kasaï and South Kivu and HealthNet TPO in North Kivu initiated the implementation of results-based financing schemes. A study of these two interventions covering the period 2006–2009 highlighted the direct involvement of the MSP. Results showed increases in utilization of services such as assisted deliveries, vaccinations, and family planning. There were important differences in results among the different communities. Factors related to the socio-political situation of the country affected system elements, such as the supply chain network (Toonen J, 2009). The World Bank initiated the $335 million Health Sector Rehabilitation and Support Project (Projet d’Appui à la Réhabil￾itation du Secteur de la Santé (PARSS II)) covering 83 HZs in ten districts across five provinces of the DRC. From 2009 to 2013, an impact evaluation was conducted on RBF activities in Haut-Katanga district in South Kivu. Results showed that RBF can be an efficient and effective approach to improve the supply of priority health services; that it did not lead to a deterioration of the quality or availability of non-targeted health services; that it led to an overall increase in the level of health worker moti￾vation; and that incentives linked to quantity did not lead to significant changes in the coverage or quality of services provided (The World Bank, 2014). 12 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Project Description Integrated Health Project (IHP) Improving the quality and use of essential health services at the national level is a key component of the Government of DRC (GDRC)’s strategy to reduce poverty. It is also one of the MSP’s essential goals. Several reforms are currently under way to improve health system performances. A new National Health Development Plan (DRC MSP, 2010) for 2011–2015 was developed in March 2010. The five-year IHP (October 2010–September 2015) supported the PNDS with the aim of improving the enabling environment for, and increasing the availability and use of, high￾impact services, products and practices for FP, maternal, newborn and child health (MNCH), nutrition, malaria, tuberculosis (TB), Human Immunodeficiency Virus (HIV) and Acquired Immune Deficiency Syndrome (AIDS), and water, sanitation and hygiene (WASH) in target HZs. The health sector in the DRC is currently divided into three tiers, with central policy generating institutions at the top, intermediate provincial health departments and districts in the middle, and HZs at the field level. Health centers (HCs) aim to provide the first line of healthcare within each HZ, while general reference hospitals (GRHs) located at the HZ level provide the second line. CHWs play a vital role, serving as spokes linking patients in remote locations to the resources available at health center hubs. IHP’s technical strategy to integrate activities across health system sectors, levels, and geography is driven by strengthening people-centered health systems. At the heart of the strategy lies outreach to providers, health authorities, community organizations, and families with evidence‐based techniques they can use to impact the health system in ways they experience as meaningful and sustainable. IHP worked with many bilateral and multinational donors, faith‐based organizations (FBOs), and non-governmental organizations (NGOs) to implement MPA and CPA. In addition, IHP staff provided technical assistance for specialized health systems strengthening interventions at all levels. IHP established eight provincial level coordination offices or IHP Bureau de Coordination (BC) to facilitate project implementation in HZs and at the assisted health facilities (i.e., HCs and GRHs).3 The project worked in 78 target HZs and provided varying levels of support to 1,476 facilities (1,398 HCs and 78 GRHs). A summary description of IHP’s pilot RBF activity A pilot RBF activity commenced in seven intervention HZs in November 2013, following the execution of performance contracts with seven ECZs, seven GRHs, and 118 HCs. 3. These coordination bureaus are also referred to as “Supervision Areas” as described in the MSH Baseline Survey Report, August 2011. The MSP RBF Unit staff were trained on RBF processes and mentored during the first six months of implementation. All participating staff, such as ECZs and health facility providers, received five days of training in November 2013 to guide them on the implementation of the RBF activity. According to several providers interviewed, during those trainings the IHP leadership advised the providers to lower the costs of their services to encourage uptake. A manual for managing and tracking RBF activities was distributed in August 2014 and later updated in April 2015. The manual established clear guidelines that are summarized herein. RBF is implemented at three levels of intervention within the MSP health system: 1. The central level in Kinshasa, namely the MSP RBF Unit that defines and ensures compliance with policy, USAID/ DRC and IHP leadership; 2. The provincial level, which includes Provincial Health Office (Division Provinciale de la Santé or DPS) and the sanitary districts that regulate activities, and the provincial IHP BC; and 3. The peripheral or operational level, which includes ECZs which sign the RBF contracts with IHP; the facility level (GRHs and HCs); and the community that receives assistance from CSOs and the health development committee (CODESA). Two verification processes are convened every three months. First, there is a technical verification of services delivered in the previous three months, undertaken by a team comprised of the MSP and IHP staff. The quantitative verification conducted at the ECZ level includes a member of the district health office; sometimes, DPS is also present. Thirteen indicators are employed to assess the ECZ team; they are mostly contingent upon the performance of the health facilities under their purview and the ECZ’s supervision of them (see Annex B: RBF Indicators for Health Zone Management Teams). At the GRH level, there is only one (composite) indicator measured: MSH’s Fully Functional Service Delivery Point (FOSACOF) facility assessment tool. FOSACOF is primarily focused on inputs and does not include any measurement of service delivery or other outcomes. The eighth module, “Clinical Quality,” accounts for 22% of the total FOSCAF score. This module measures inputs such as availability of the national clinical protocols (“Ordinogram”), but does not measure actual outputs or outcomes of clinical care. Table 1 presents the weighting of the nine FOSACOF modules. BACKGROUND n 13 Lastly, HCs are measured against 16 indicators, many of which reflect services delivered (see Annex C: RBF Indicators for Health Centers). It is worth noting that the HC’s FOSACOF score contributed 29–32% of the total RBF Incentive Score. If the HC is compared to the four indicators that measure HIV and AIDS and TB services, FOSACOF comprises 29% of the total score; otherwise, it comprises almost one third of the total RBF score.4 Targets for each of the RBF indicators are negotiated annually, taking into account annual plans, baseline studies, and the results of the preceding year. During the 2014 RBF midterm assessment, key informants only spoke of projecting targets based on results achieved in the previous year. Approximately one month after the completion of the technical verification, CSOs, which have been directly contracted through IHP (purchasing agent), conduct household interviews among a randomly selected group of HC patients. CSOs are given the names of patients who had been treated in the HC in the past three months, so that they can directly verify with those patients (or the mother/guardian) that the visit indeed took place as recorded. The household respondent is simultaneously asked a series of client satisfaction questions, which are then translated into a score that serves as one of the 16 RBF indicators used to assess the HC performance. Once the verifications have been completed at the different levels of the health system, the results are validated at the central level by IHP and the RBF unit. Once results are validated, payments are processed and distributed to the coordination offices and distributed at the HZ and health facility levels. The health facilities receive a payment every three months, based on the 4. HCs that did not receive IHP support for HIV and AIDS and TB services because of low prevalence rates were not expected to be judged against those respective four indicators. However, findings from the 2014 RBF Midterm Assessment revealed inconsistencies in this application. attainment of targets that have been established in the contract. According to findings from the 2014 RBF Midterm Assessment, achievement of all targets at the HC level would yield a quarterly payment of $910 USD, and achievement targets set for the one GRH indicator (FOSACOF) would result in a payment of $12,000 USD. The achievement of indicators was capped at 120% of set target. CSOs also receive a payment through a grant from IHP, based on the successful completion of the household interviews. The number of household interviews varies from month to month, because it reflects the patient volume realized in the previous quarter. The payments to the CSOs are at a fixed amount. However, payments aimed at incentivizing staff performance are distributed as follows: 1. Incentives for the ECZ include staff incentives (70%), investments (20%), and operations (10%) 2. Incentives for health facilities (GRH and HC) are distributed as follows: staff incentives (60%), investments (30%), and functioning of the facility (10%) (see Figure 1) 3. At the health facility level, the distribution of the staff incentives paid directly to personnel take into account the employment categories (e.g., nurse degree, sage femme, etc.), individual performance appraisal, and presence in the workplace (attendance). An Index Tool was devised to make these calculations. Table 1. Weighting of FOSACOF scores Module No. Evaluation variable Maximum Points Overall score (%) 1 Infrastructure 225 7.9 2 Equipment 150 5.3 3 Essential Medicines/ supplies 300 10.5 4 Personnel 300 10.5 5 Continuing Education 150 5.3 6 Community Outreach 400 14.0 7 Community Support 400 14.0 8 Clinical Quality 635 22.2 9 Management 295 10.3 Total 2,855 100 10% Infrastructure 30% 1. Professional rank 2. Performance appraisal 3. Staff attendance CRITERIA FOR APPROPRIATING INCENTIVE PAYMENT TO FACILITY STAFF Staff incentives 60% Operations Figure 1. Disaggregation of RBF payments for GRHs and HCs 14 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO EVALUATION METHODS Evaluation Design This impact evaluation is a prospective, quasi-experimental design with intervention and comparison groups. The measurements were taken at baseline (2013), midline (2014) and endline (2015). Figure 2 provides a detailed timeline and illustrates the methods used at a specific point in time. Baseline data collection in the intervention and comparison health zones were conducted between May and September 2013, prior to the commencement of the pilot RBF intervention in November 2013. The RBF Midterm Assessment was conducted by the Team Leader and Evaluation Specialist along with the Study Coordinator and Research Assistants in October 2014 at four of the seven RBF intervention HZs. The endline data were collected between May and September 2015, covering the same intervention and comparison sites studied during the baseline to ensure pre/post intervention comparability. The evaluation used a mix of quantitative and qualitative data collection strategies. The utilization of qualitative or quantitative methods provided different types of data and addressed different evaluation questions. Data collected via quantitative strategies (household surveys and health facility surveys) were mostly used for evaluating whether there is a relationship between the pilot RBF intervention and an effect (mainly in terms of the quantity and quality of health services). The quantitative methodology measured changes in key outcome indicators between the baseline and endline results. To ensure comparability of findings, the quantitative methodology used the same survey instruments and sampling frame for both baseline and endline data collection. At midterm, a process evaluation used qualitative methods, including key informant interviews and focus group discussions at pilot RBF intervention sites. The RBF Midterm Assessment Report, published in April 2015 (http://pdf.usaid. gov/pdf_docs/PA00KFHS.pdf), provides a detailed analysis that responds to evaluation questions relating to the contextual factors and unintended consequences of the pilot RBF intervention. Pilot RBF HZs selected (MSH with MSP) Sept 2010 Sept 2015 PY 1 PY 2 PY 3 PY 4 PY 5 Comparison HZs selected (IBTCI) Baseline Evaluation* Midterm Assessment‡ Endline Evaluation* Integrated Health Project implemented in target provinces RBF Pilot intervention Jan-May 2013 May-Sep 2013 Oct 2014 May-Sep 2015 *Quantitative study ‡ Qualitative study Figure 2. Timeline of evaluation activities BACKGROUND n 15 KINSHASA Kasaï Occidental Kolwezi Uvira Luiza Mwene Ditu Kole Kamina TSHIUDILOTO LOMELA Tshumbe MINGA KAMIJI BIBANGA WEMBONYAMA NUNDU LUBONDAIE KABONGO KANZENZE MUTSHATSHA KAYAMBA Kasaï Oriental South Kivu Katanga Figure 3. Map of selected sites for RBF impact evaluation Table 2. Selected sites for RBF Impact Evaluation Site No. IHP Coordina￾tion Office Intervention HZ Comparison HZ 1 Luiza Luiza Lubondaie 2 Mwene Ditu Bibanga Kamiji 3 Kole Lomela Tshudiloto 4 Tshumbe Wembo Nyama Minga 5 Kamina Kayamba Kabongo 6 Kolwezi Kanzenze Mutshatsha 7 Uvira Nundu Uvira Key: o = coordination office ▲ = RBF intervention zone ★ = RBF comparison zone Site Selection MSH, in consultation with the MSP, selected seven HZs for the pilot RBF intervention (see Figure 3 and Table 2). The intervention site selection was based on nine selection criteria, described in the RBF Manual 2011 (Management Sciences for Health, 2011). All seven pilot RBF intervention HZs (100%) were selected to be included in the intervention group of the evaluation study. The IBTCI evaluation team selected one comparison HZ for each of RBF’s intervention HZs. Comparison HZs, sharing a geographical border with the intervention zones, were randomly selected after excluding zones which had 1) a preexisting RBF intervention, 2) known security issues, and/or 3) no physical access by land. Table 2 presents the selected intervention and comparison health zones for this impact evaluation. Survey Sampling Strategy and Sample Size Household survey The probability proportional to population size (PPS) sampling method was used to select households for the household surveys. The data from household surveys were obtained from mothers of children aged 0–23 months. The household sample size was determined based on one of IHP’s seven outcome indicators reported to USAID/DRC in their Performance Monitoring Plan (PMP), and which reflects one of the priority program elements for IHP in terms of funding. This maternal health indicator (percent facility births) is the “proportion of mothers of children aged 0–23 months who delivered their youngest child in a health facility.” Other PMP outcome indicators had been considered, such as those relating to the treatment of malaria illness and vaccination coverage rates for DPT. However, the sample sizes yielded were quite large (almost double) and therefore not within the budget of this study. In order to detect a 10% difference on the selected maternal health indicator (institutional delivery) and modern contraceptive users with 80% power and 95% confidence, a sample size of 226 households for each RBF intervention and comparison health zone was needed. A sample size of 3,164 household respondents was finally required for the seven intervention health zones, as well as the seven comparison health zones. Across the entire sample, an additional 15% of household respondents were included in anticipation of non￾response cases. A household was defined as a domestic unit consisting of the members of a family who live together, along with non-relatives, and where at least one member is a mother of a child aged 0–23 months. Households to be interviewed were selected using a cluster survey approach. After randomly selecting a village in the health area,5 the survey team proceeded to a predefined point (such as the village chief’s home) and selected the first 5. The MSP catchment populations are comprised of health zones. Each health zone is comprised of approximately 15–20 health areas. There is one general reference hospital in each health zone and one health center in each health area. 16 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO household at that starting point. The person answering the door at the household was asked if there were a mother with a child aged 0–23 months available for interview. If there was an eligible respondent, that household was selected for completion of interview—assuming the woman did not decline. After the interview, the interviewers were instructed to follow the defined strategy established by their supervisor (i.e., go to the nth household in X direction). When an eligible respondent was not present in the household, or was not willing to be interviewed, the interviewer went to the very next household to determine if there was an eligible respondent. This process continued until all required households in the predetermined direction were completed for the village. For women who were not present at the home on the first attempt to interview, a second attempt was made to interview them by returning later in the day. This helped to avoid selection bias of non-working women only. Facility survey The health facility sample was selected using the Lot Quality Assurance Sampling (LQAS) methodology, which has a 92% precision level and 95% confidence intervals. A “facility” was defined as a fixed structure where health services are provided to the communities residing in nearby areas. Considering the total facility number (N) of 298 in 2013, i.e., the total HCs in the previously planned eight intervention and eight comparison HZs, 25 HCs were originally selected for both the intervention and the comparison areas within each HZ.6 The 25 sampled HCs were then distributed in the corresponding HZs of intervention and comparison sites according to PPS. Using this systematic procedure, a total of 200 health centers were selected (109 intervention centers and 91 comparison centers). All 14 GRHs were also selected, one in each zone (see Annex D for Data Collection Sampling Procedure). 6. The MSP changed its decision on IHP’s inclusion of RBF intervention sites in 2013. Consequently, after baseline data were collected, Bukavu BC was dropped as an IHP intervention site for the RBF. Nevertheless, this did not affect the overall required sample size for the evaluation. Evaluation Team The evaluation team included Annette Bongiovanni, Team Leader; Swati Sadaphal, Evaluation Specialist/Data Analyst; and Zephyrin Kanyinda, Health Economist/Survey Coordinator. Fieldwork was supported by eight field supervisors. During data collection, eight survey teams, consisting of one supervisor and three or four interviewers and data collectors, were deployed to conduct the surveys in the sampled intervention and comparison zones. In each BC, a new team of data collectors/interviewers was selected to ensure that the data collection team had local citizens who were familiar with the local languages and culture (See Annex E for Field Implementation Plan). Logistical and administrative support was provided by staff based in Kinshasa and the IBTCI home office. Data Collection The endline data collection period was from May to September 2015, with six work days per week. Each week of data collection had the following schedule: five full days of survey administration in the intervention and comparison health zones, and one or two days of travel to the next health zones at the end of each survey administration week. The household survey and health facilities’ surveys were conducted in all of the intervention and comparison study sites. All questionnaires were conducted in local languages at the household level, and in French or local languages at the facility level, depending upon the respondent’s preference. In addition to providing native speakers of local languages, the recruitment of a locally based data collection team facilitated mapping and identification of remote HCs and villages and access to local transportation. Data collection tools The document review provided a foundation for developing technical and operational strategies for the evaluation, and designing survey questionnaires and interview guides. Each quantitative survey had sections with questions designed to collect discrete, categorical, and ordinal data. Annex F provides a list of documents and other references used by the team and Annex G includes the survey questionnaires. The bibliography is also included at the end of this report for the reader’s convenience. A structured facility survey questionnaire with open- and closed-ended questions was used to collect information. The questionnaire included questions for data collectors, to record facility observations notes on infrastructure, medications, and documentation reviews. The facility survey collected data in the following analytical domains: ■ Availability (quantity) of MPA plus services as they are supposed to be provided at all HCs in the IHP’s target zones ■ Quality of care in key family health service areas ■ Facility operations and management including infrastruc￾ture, staffing, and facility statistics including patient atten￾dance, availability of supplies, equipment, and personnel. EVALUATION METHODS n 17 The household survey questionnaire was administered to the sampled households in the intervention and comparison groups. A structured questionnaire comprised of fifteen modules with open- and closed-ended questions was used (adapted from the IHP Baseline Household Survey, 2011). The questionnaire included questions for interviewers to record observations and verify relevant household assets related to health. The household survey collected data in the following analytical domains: ■ Individual characteristics ■ Household characteristics ■ Health services utilization characteristics ■ Knowledge and behavior related to key family health issues ■ Perceptions of the quality of care and services provided by health facilities Quantitative data related to the costs of implementing the RBF approach were collected using a Microsoft Excel®-based cost analysis tool designed by the evaluation team. This instrument was shared with the MSH Kinshasa office, which in turn shared it with their IHP coordination offices to input cost data from the seven intervention sites. The MSH Kinshasa team compiled the information received and sent the completed tool to IBTCI in October 2015. Online information on the RBF web portal7 was also reviewed as a reference and to validate the cost findings. Additional data collection following the preliminary data analysis Preliminary data analysis was presented at the USAID debrief on December 8, 2015. In consultation with USAID/DRC, additional information was collected through key informant interviews (n=26) at intervention and comparison health zones. The main purpose of gathering additional data was to reconfirm the absence of any unknown confounding factors affecting the impact result (for example, simultaneous RBF implementation by other partners in intervention or comparison HZs).8 The Study Coordinator, who is a health economist, conducted key informant interviews for obtaining pilot RBF cost-related information during the additional data collection activity. 7. http://www.fbrsanterdc.cd/ 8. At the RBF impact evaluation design stage in 2013, one of the screening criteria for comparison health zones was to exclude health zones with existing RBF intervention. However, given the time lapse, the updated information was necessary. Ethical Considerations Oral informed consent (read from a written document) was administered to inform respondents of the purpose, process, potential risks, use, and confidentiality of the information and their right to refuse to participate at any time. Facility respondents were interviewed in private in a facility’s consultation room. Household respondents were interviewed at their home. All interviewers received training in ethical protocols to ensure that no identifying characteristics of respondents were recorded during data collection. Unique identifiers were used in place of client names. Additionally, the electronic database was kept in password-protected computers used only by the senior evaluation team members. Respondents did not receive any form of inducement or incentive to participate in the study, and the survey team reiterated its external role to service delivery and the MSP. All respondents were informed they could cease participation at any point during the survey or interview process. Data Quality Assurance Training and pilot testing About 80% of the supervisors, data collectors and interviewers recruited were those who participated during the baseline surveys in 2013. A five-day supervisor training was organized from April 6–10, 2015 in Mitendi, near Kinshasa. The questionnaires and methodology were pilot tested at the end of the five-day training in peri-urban health facilities. Some minor revisions were made to the instruments afterwards. For each province, a group of interviewers and data collectors was trained and selected, so that surveys could be conducted in local languages. The field supervisors, assisted by the Study Coordinator, conducted the recruitment of interviewers/data collectors. All selected interviewers/data collectors went through a three-day practical survey training, with one day devoted to the facility survey, one day focused on the household survey, and the last day dedicated to practices sessions using the role-play method. Annex H provides an English translation of the French training manual given to all interviewers and data collectors. When available, nurses and doctors were assigned the role of data collector, given their familiarity with the health facilities. 18 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Data quality check procedures The entire survey team collected data in one province, before moving to another province. This approach facilitated greater communication and problem solving, if needed, between survey sub-teams. Data were collected using paper-based questionnaires by the interviewer/data collector, and his/her work was double-checked by their supervisors at the end of each day. Interviewers/data collectors maintained regular communication with supervisors, mostly in person but when necessary through mobile communication devices, to inform him/her of any survey issues needing to be addressed. Supervisors were present in the villages and the health facilities when data were being collected. They would directly observe the interviewers/data collectors during data collection. In addition, they would randomly select 5% of the completed instruments each day and re-interview selected modules. The interviewers/data collectors were not able to predict which modules would be re-interviewed. During the re-interviews of selected modules of selected respondents, the supervisors were accompanied by an uninterested party who could translate into the local language. The supervisors maintained regular communication with the Study Coordinator to relay all relevant information from the field, in case there were technical matters that had to be addressed. Communication between the supervisors and the Study Coordinator was done on a daily basis to ensure constant oversight and management in the data collection process. Each week, at least two random and unannounced data verification tasks were conducted by the Study Coordinator. In addition, senior evaluation team members conducted data quality assurance field visits. This enabled them to participate in field supervisor training, supervise the pilot study, monitor the trainings of data collectors and interviewers, and provide technical input during direct observation of the data collection. For example, between April and September 2015, three senior evaluation team members made three separate field visits, each for approximately three weeks. At the end of the data collection period, the paper questionnaires were sent to the IBTCI home office for safe storage and data quality checks. The data from the paper questionnaires were computerized using Microsoft Excel® by six experienced data entry administrators at IBTCI’s home office under the direct su￾pervision of the Evaluation Specialist. A random sample of 10% of all paper questionnaires were checked for data issues, which were nominal (for example, misspelled names of health zones). Data cleaning and post-data entry coding were also part of the data quality assurance procedures. Incomplete responses, missing values, or outliers were handled using standard data quality as￾surance techniques including dummy variables, checking for data variance, and data normality using statistical analysis. Data Analysis A data analysis plan was developed and guided by the overall objective of the impact evaluation to measure changes that can be attributed to the RBF program (See Annex I for the Data Analysis Plan). The quantitative data were analyzed using STATA version 12. Endline survey data for health facilities and households were appended with their respective baseline data to create combined databases for the impact analysis. The principal procedure for data analysis involved descriptive statistics on each variable at baseline and endline, making comparisons between the intervention and comparison sites, and testing for any significant differences (calculated as p<=0.05 for significant testing at 5% level and p>0.05 and <=0.10 for significant testing at 10% level). Logistic regression was used to adjust any confounding factors (age, literacy level, employment status of the head of household, residence) as deemed necessary. The unit of analysis of the health facility survey is the HZ, which is disaggregated by the type of health facility (HC or GRH). The unit of analysis of the household survey is the HZ. The data analysis for impact evaluation used the Difference in difference (DID) estimation methodology. The DID estimate gives a reliable estimate of the difference between the intervention group or comparison health zones at different points in time (at baseline and endline), controlled for the various socioeconomic and demographic variables. Figure 4 on the following page shows an illustration of the DID estimation method. The red and solid grey lines represent the intervention groups and comparison groups, respectively. The differences in initial starting points are taken into account in the DID method by comparing each group with itself over time. The dashed red line represents the counter-factual case. The counter-factual case represents the predicted outcome in the intervention group if no pilot RBF had been implemented. The difference between the counter-factual case and the intervention group is the effect of the pilot RBF intervention. The DID estimator uses information from the comparison group to predict the counter-factual. The pilot RBF intervention is considered effective if outcomes in the intervention group improve more (or decline less) than they would have if the pilot RBF had not been implemented. Thus the RBF effect, the difference between the intervention group and its counter-factual of non-RBF, is the central estimate used for the quantitative component of this impact evaluation. The data gathered through the cost analysis tool were analyzed using Microsoft Excel® to calculate cost associated with each budget line item and cost per capita inhabitant. EVALUATION METHODS n 19 Strengths of Evaluation Methodology The quasi-experimental evaluation methodology, with qualitative and quantitative data, provides an opportunity to examine evaluation results in the context and ways in which the pilot RBF intervention is currently being implemented—and also useful for a similar intervention which might be designed in the future. Any results seen or not seen are occurring in more “natural” or “usual practice” scenarios, which is not possible if randomized control trials are utilized. This means that the evaluation results may be more generalizable to other similar settings. Another important strength of this evaluation methodology is that it includes characteristics of both service delivery points and their respective catchment populations. Routine monitoring data related to service utilization collected at health facilities are all too often not representative of the population. Also, there is greater heterogeneity vis-à-vis access to health services and health-seeking behavior within a population; routinely collected data within the health system, which measures only users of the formal health sector, results in uneven data quality. Thus, one of the main advantages of data collected through household surveys is that it provides the perspective of non-users of the health system as well as users. A probability sampling method used in this sample selection provides a sound representative sample of all the households and facilities in the targeted health zones. It is, however, subject to some degree of random variance, due to one particular group of households/ facilities being selected rather than some other; this random variance decreases as sample size increases. The sample size used in this evaluation, with 3,610 households and 186 facilities, is large enough to provide reliable estimates. Limitations of Evaluation Methodology The evaluation team hoped to select the comparison zones that were as similar as possible to the intervention zones; in essence, one zone should not be statistically different from the other. Given the limited resources, the cost of studying the potential comparison zones before selection was prohibitive. Therefore, the team was obliged to assume that zones sharing a contiguous border with the intervention zones were likely to share similar characteristics. However, the baseline study results showed that the intervention and comparison groups were statistically different with regard to socio-demographic profile in terms of employment status and place of residence. The comparison group had more urban areas than the intervention group. The evaluation team mitigated these differences during the final data analysis by using the DID methodology (described in detail above) and using multivariate regression analysis. As in all questionnaire-based surveys, some of the data collected are influenced by the respondent’s accuracy of recall, among other factors, hence there is a potential for recall bias. Questions for which recall bias is most pronounced is most likely for recall of medications obtained at a health clinic during a childhood illness. Similarly, during the facility surveys, the health provider interviews might have been subject to respondent bias, whereby the answer is filtered depending upon the perceived advantages or disadvantages that might follow a particular response. Further, the halo bias might have been a factor in both the household and the health facility surveys, where respondents are likely to provide favorable answers to some questions more than others. Interviewer bias is possible, because for each BC a different team of interviewers was utilized. To minimize these limitations, the survey team was extensively trained in interviewing skills and avoidance of probing for open-ended questions. Daily direct observation by supervisors reduced the likelihood of these biases. Moreover, onsite training by the senior evaluation team members for extended periods of time focused on the reduction of bias (for example, using neutral body language and tone of voice). Additionally, the survey instruments had observation or verification prompts, for the interviewer to record his/her direct observations or verify reported statements (e.g., current stock of drugs, facility infrastructure). Interviewer bias was minimized by using most of the data collection teams who participated during the baseline. Additionally, the supervisor to data collector/ interviewer ratio was low (1:4). This allowed for continuous supervision at the village level. The cost data used for the analysis were sent by the MSH Kinshasa office, which was responsible for Figure 4. Illustration of difference in difference (DID) estimation BASELINE ENDLINE IMPACT INDICATORS IHP effect IHP+RBF effect RBF effect (DID) Intervention Comparison 20 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO the implementation of the activity. This does not allow for any verification with other sources of data outside MSH. However, the cost data formats received were comparable to what RBF practitioners recommend for the implementation of RBF. Constraints The MSP changed its decision on IHP’s inclusion of Minga Health Zone as an RBF intervention site after IBTCI’s baseline evaluation report was finalized. Consequently, Minga HZ was dropped and replaced with Wembo Nyama HZ for the pilot RBF intervention implementation in November 2013.9 Therefore, the baseline status of key indicators were re-analyzed by categorizing Minga HZ in the comparison group, and Wembo Nyama HZ in the intervention group for the purpose final impact analysis. 9. The baseline evaluation report 2013 should not be compared to the baseline status of key indicators presented in this final evaluation report, 2015, because the baseline status of key indicators were re-analyzed by categorizing Minga HZ in the comparison group, and Wembo Nyama HZ in the intervention group. EVALUATION METHODS n 21 EVALUATION FINDINGS Overview The main findings in this section are organized based on the analytical domains of the impact analysis as opposed to the illustrative eight evaluation questions provided in the contract. The findings of evaluation questions number 1, 2, 3, 5 are described under the analytical domains of “Quantity” and “Quality” of services. Evaluation question 4 is described under the “RBF associated costs” and evaluation question 6 is described under the “Heterogeneity” analytical domain. Analytical domains of “Contextual factors” and “Unintended consequences,” corresponding to the evaluation questions 7 and 8 respectively, were extensively studied during the qualitative study carried out in October and November 2014 and reported in more detail in the RBF Midterm Assessment, April 2015. Findings from this midterm qualitative study and IHP program reports (2013–15) have been described wherever applicable. The number of cases included in the results (N=number) is stated at the heading of each table or in the introductory paragraph of a section, and percentages stated in the corresponding text and figures are derived from the total sample of household or health center surveys. Analytical Domain: Quantity and Quality of Health Services The findings for this analytical domain are derived from an analysis of the quantitative data included in the facility survey, household survey, and the relevant qualitative data from the RBF Midterm Assessment Report and project documents. This report describes evidence of any changes among health centers that can be attributed to the RBF model (evaluation question 1); what difference (improvements or declines) the RBF intervention made (evaluation question 2); whether the model is worthy of being scaled up in other health zones (evaluation question 3); and if the desired results were achieved (evaluation question 5). Quantity of health care discussed below includes availability and utilization of preventive and curative MPA services at the health centers, and preventive and curative service utilization among households. Quality of health care discusses preventive and curative MPA services. IHP’s key activities under intermediate result (IR) were to improve access to and availability of facility-based and community-based MPA and CPA health services and products. As part of a financing strategy under the IHP, MSH tested the pilot RBF intervention in seven selected health zones. The pilot RBF aimed for a rapid scale-up of health services (quantity) and improved quality of care at selected health centers and GRHs. Objectively verifiable indicators (for both quantity and quality) were developed in order to measure the level of achievements of results. Table 3 on the next page provides the revised list of MPA and CPA services provided to the evaluation team in 2015, which were supported by IHP at the pilot health facilities. The table also lists the specific service delivery indicators (for health centers and GRHs) included under RBF contracts between IHP and various health facilities. Achievement of results on these indicators was verified and validated by IHP headquarters before any disbursement of incentives under the RBF scheme. Facility sample characteristics A total of 176 facilities and 186 facilities were surveyed in baseline and endline respectively. Annex J provides details of the distribution of facilities surveyed in each health zone at baseline and endline. When interpreting the findings, it is important to note that the facility sample was mainly comprised of health centers (94%), and that GRHs represented only 6% of the total sample of facilities. Ninety percent (90%) of the facilities were located in rural areas and the most typical respondent to the questionnaire was a registered nurse (93%) in both groups. While both groups were predominantly rural, the intervention group contained more rural facilities than the comparison group (p=0.01). 22 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Table 3. MPA and CPA services (revised v2014) for IHP implementation and RBF-contracted service delivery indicators MPA services at health centers RBF-contracted service delivery indicators Preventive activities ■ Monitoring growth and development of children under 5 ■ Antenatal consultations (ANC) ■ Family planning consultations ■ Postnatal consultations ■ Extended vaccination program 1. Rate (number) of coverage with DPT-HepB Hib3 2. Proportion (number) of pregnant women who received 2 doses of TT injections 3. Number of new FP users 4. Rate (number) of use of ANC services 5. Rate (number) of ANC visit coverage, revisits 6. Rate (number) of postnatal care visits Curative services ■ Curative care ■ Screening and treatment of chronic illnesses (TB, leprosy, diabetes, AIDS, etc.) ■ Nutritional rehabilitation ■ Small medical and surgery interventions (minor surgeries) ■ Natural deliveries 7. Rate (number) of use of curative services at the health center 8. Proportion (number) of high-risk pregnancies that were referred 9. Rate (number) of attended childbirths 10. Rate of TB detection/number of cases of TPM+ detected (3 sputum tests) 11. Number of clients who received voluntary HIV counseling and testing 12. Number of pregnant women tested for HIV Health promotion services Activities for the promotion of healthy behaviors: ■ Use of condoms ■ Sanitation ■ Exclusive breastfeeding ■ Healthy eating, nutritional, and cooking habits ■ Use of iodized salt ■ Social marketing and distribution of LLITNs ■ Hygienic latrines ■ Oral hydration therapy, etc. 13. Proportion (number) of LLITNs distributed Management and other activities ■ Resource management (human, material, financial) ■ Continuing education/training of personnel ■ Supervision of health leaders (meetings, field visits) ■ Management of health information 14. Percent (number) of medication orders approved by the quantification committee 15. Health center’s overall FOSACOF score 16. Community satisfaction score Community activities ■ Measures for food sanitation ■ Capture and management of springs, wells, supply of drinking water, community water treatment ■ Vector control: use of LLITNs, trapping of flies and mosquitoes ■ Exclusive breastfeeding ■ Support for orphans ■ Gardening, fish farming, livestock breeding None CPA services at general reference hospitals RBF-contracted service delivery indicators CPA (preventive, curative, and promotional activities) are organized within the framework of internal medical services, surgery, gynecology, obstetrics, and pediatrics. In addition, it includes: ■ The various reviews of a reference laboratory ■ Medical imaging ■ Equipment sterilization ■ Rehabilitation activities 1. GRH’s overall FOSACOF score composed of nine sub-components, including: ■ Infrastructure ■ Medical equipment ■ Essential supplies and medications ■ Staff ■ Training ■ Relationship between the GRH and other components of the HZ ■ Complementary package of activities ■ Clinical quality ■ Management EVALUATION FINDINGS n 23 Household sample characteristics At endline survey, a total of 3,610 household respondents and at baseline, 3,588 household respondents completed the household questionnaires. Annex K provides distribution of the households surveyed in each health zone at baseline and endline. All respondents (100%) were mothers aged between 15–49 years with children aged 0–23 months. At endline, households in comparison zones were located in urban areas (14%) compared to intervention health zones (4%) and there were lower literacy rates among respondents in the comparison group than in the intervention group (Table 4). Table 4. Household characteristics of intervention and comparison sites Socio-demographic characteristics Intervention Group Comparison Group Baseline (n=1791) Endline (n=1797) Baseline (n=1812) Endline (n=1813) Residence of household* Rural 96% 96% 86% 86% Urban 4% 4% 14% 14% Literacy‡ of respondent* Literate (can read and/or write) 48% 48% 47% 47% Not Literate (cannot read nor write) 52% 52% 53% 53% Mean age of respondent (years) 27.2 27.7 27.4 28.1 Employment status of household head Employed (salaried + self - employed) 97% 95% 96% 95% Other 3% 5% 4% 5% ‡ Literate is defined as the respondent who reported that she can read and write; illiterate is defined as the respondent who reported that she can only read and cannot write, OR cannot read or write. Source: Household surveys, 2013, 2015 Key: *p<=0.05 at endline Overview of significant findings A summary of the significant findings related to the preventive, curative and promotional, management, and community activities described in Table 3 is shown on the following pages. Table 5 (next page) shows the contracted indicators and summarizes the difference the pilot RBF intervention made on the HC service delivery indicators. Key: Significant and positive changes are colored green, significant and negative changes are colored red, and variables with no significant changes between baseline and endline are colored grey. The significant positive changes seen for indicators 3, 6, 7, and 8 were attributable to RBF interventions. None of the indicators showed negative (undesired) results due to RBF intervention. Table 6 on page 25 shows other non-contracted indicators, where significant differences were found, using the same colors as in Table 5. Among the 14 Additional Services indicators, seven showed differences attributable to RBF activities. Among the Promotion indicators, 4/7 showed significant changes attributable to RBF, while among the Facility Management indicators the rate was 8/9. These findings, with these respective numerators and denominators and significance values, will be discussed in more detail in the following pages. 24 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Table 5. RBF services contracted and the difference the pilot RBF intervention made on HC service delivery No. RBF contracted indicators at HCs Indicators evaluated RBF Impact Results (baseline vs. endline) DID Intervention Group Comparison Group 1 Rate (number) of coverage with DPT￾HepB Hib 3 Average number of children who received DPT1 vaccine at the HF over the past year‡‡ 2 Proportion (number) of pregnant women who received 2 doses of TT injections Percentage of pregnant women received at least two TT injections‡ 3 Rate (number) of use of antenatal care services Average number of antenatal care consultations in the past year‡‡ Received at least one antenatal consultation‡ 4 Rate (number) of antenatal consultation coverage, revisits Received at least four antenatal consultations‡ 5 Rate (number) of postnatal care visits Average number of postnatal care consultations in the past year‡‡ 6 Rate (number) of use of curative services at the health center Average number of outpatients (new and old cases) visits in the past year‡‡ Sought care in a local HC when needed health service‡ 7 Proportion (number) of high-risk pregnancies that were referred Average number of pregnant women referred to GRH for obstetrics complications‡‡ 8 Rate (number) of attended childbirths Average number of childbirths attended by facility personnel in the past year‡‡ Percentage of women who had childbirth in a health facility‡ 9 No. of clients who received HIV counseling and testing Percentage of women ever tested for HIV‡ 10 Number of pregnant women tested for HIV Average number of pregnant women screened for HIV in the past year‡ 11 Proportion (number) of LLITNs distributed Household has at least one LLITN‡ 12 Community satisfaction score Client's perceptions of quality of care during last visit (within 3 months) to a health center‡ 13 No. of children with diarrhea and fever treated with ORS and zinc Child with diarrhea treated with oral rehydration salts‡ 14 No. of children with presumed pneumonia treated with Antibiotics Child with presumed pneumonia treated with antibiotics in a health center‡ 15 No. of children with suspected malaria treated with Artemisinin combination therapy (ACT) Child with suspected malaria treated with any antimalarial in a health center‡ Source: ‡ Household and ‡‡Health center surveys, respectively Key: Red font indicates significant result. Changes between baseline & endline and RBF impact: Positive changes with statistical significance (green), no statistically significant changes (grey), negative changes with statistical significance (red) EVALUATION FINDINGS n 25 Table 6. Difference the pilot RBF intervention made on the health center service delivery indicators No. Indicators evaluated RBF Impact Results (baseline vs. endline) DID Intervention Group Comparison Group Additional services 1 Growth monitoring of children <5 years‡‡ 2 Average number of FP consultations in past year‡‡ 3 Received at least one dose of TT at antenatal consultation‡ 4 A skilled health worker examined patient after childbirth‡ 5 Received dose 1 vitamin A, 2 months after childbirth‡ 6 Youngest child received vitamin A within last 6 months‡ 7 Current users of modern contraceptives‡ 8 FP advice from a health worker in the past 1 year‡ 9 Average number sick child consultations for children >5 in past year‡‡ 10 Average number childbirths attended by facility in past year‡‡ 11 Respondents had childbirth in a health facility‡ 12 Emergency obstetric cases referred to GRH‡‡ 13 Pregnant women are offered PMTCT services‡‡ 14 Child with suspected malaria had blood drawn for testing‡ Promotion 1 Promotion of breastfeeding, ORS, vaccination, nutrition‡‡… 2 IEC Materials for respiratory infections‡‡ 3 Household had at least one LLITN‡ 4 Child slept under a bed net last night‡ 5 Health workers gave advice on WASH‡ 6 Household used method to make water potable‡ 7 Household with improved source of water‡ Facility management 1 Received supervision by MSP‡‡ 2 Had delays in medications supplies‡‡ 3 Had oral contraceptives in stock‡‡ 4 Had Depo-Provera stock‡‡ 5 Had electricity supply‡‡ 6 Had water supply inside building‡‡ 7 Had access to transportation‡‡ 8 Had biomedical waste disposal‡‡ 9 Had disinfectant in stock‡‡ Source: ‡ Household and ‡‡Health center surveys, respectively Key: Changes between baseline & endline and RBF impact: Positive changes with statistical significance (green), no statistically significant changes (grey), negative changes with statistical significance (red) 26 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Preventive MPA services (quantity and quality) Availability of preventive MPA services at the health centers (quantity) All health centers surveyed provided preventive maternal and child health and family planning services. The availability of these services was maintained at high coverage levels (≥90%) between the two groups (Table 7), except for growth and development monitoring services for children under 5. RBF appears to be a contributing factor for maintaining the coverage of growth and development monitoring services for under-5 children in the intervention group. Service utilization statistics for preventive MPA services at health centers (quantity) The average number of women coming to health centers for antenatal consultation increased significantly in both the groups between 2013 and 2015, whereas postnatal consultations decreased significantly in both groups (Table 8). The increase in the average number of children receiving the DPT1 vaccine over the past year, and FP consultations, increased significantly in the intervention group. There was no change in the comparison group for these service statistics. The average number of children receiving full immunization at the health center did not show any significant changes in both the groups between 2013 and 2015. Preventive MPA service utilization pattern among households (quantity) Antenatal services: Household survey results show that >85% of women reported receiving at least one antenatal consultation (ANC) (Figure 5). There was a significant decrease by 3% in the comparison group, whereas the indicator shows no significant change in the intervention group. RBF appears to be the contributing factor in preventing the decline in the intervention group for women receiving at least one antenatal Table 7. Quantity of preventive MPA services at health centers: Availability of services Intervention Group (N=167) Comparison Group (N=172) DID analysis Availability of preventive MPA services 2013 2015 p-value 2013 2015 p-value p-value Facility offers child health preventive services 99% 100% 0.9 99% 100% 0.9 1.0 Growth monitoring of children <5 years^ 74% 76% 0.8 77% 65% 0.10^ 0.10^ Immunization for children 100% 97% 0.8 100% 99% 0.9 0.8 Facility offers ANC 99s% 93% 0.7 100% 95% 0.3 0.6 Facility offers postpartum care 93% 92% 0.9 93% 97% 0.7 0.8 Facility offers FP services 98% 99% 0.9 99% 100% 0.9 1.0 Source: Health center surveys, 2013, 2015 Key: Red font indicates significant result. *p<=0.05; ^p>0.05 and <=0.10; DID=difference in difference analysis Table 8. Quantity of preventive MPA services at health centers: HC client statistics Intervention Group (N=167) Comparison Group (N=172) DID analysis Health center client statistics 2013 2015 p-value 2013 2015 p-value p-value Average number of ANC visits in the past year 267 429 0.00* 286 518 0.00* 0.50 Average number of postnatal care consultations in the past year 208 102 0.00* 176 64 0.00* 0.89 Average number of children who received DPT1 vaccine at health facility over the past year 225 307 0.02* 294 329 0.29 0.31 Average number of children who received full vaccination at health facility in the past year 182 194 0.55 312 252 0.39 0.34 Average number of FP consultations in the past year 257 403 0.01* 299 376 0.22 0.38 Source: Health center surveys, 2013, 2015 Key: Red font indicates significant result. *p<=0.05; ^p>0.05 and <=0.10; DID=difference in difference analysis EVALUATION FINDINGS n 27 Figure 5. Percentage of women receiving ANC 20% 40% 60% 80% 100% Comparison group* 89% 86% ▼ 3% Intervention group 88% 89% ▲ 1% BASELINE ENDLINE 20% 40% 60% 80% 100% Comparison group* 50% 46% ▼ 4% Intervention group* 57% 50% 7% BASELINE ENDLINE ▼ Received at least one ANC consultation* (DID p=0.00) Received at least four ANC consultations (DID p=0.34) Figure 6. Percentage of women receiving TT injections during ANC 20% 40% 60% 80% 100% Comparison group* 73% 68% ▼ 5% BASELINE ENDLINE Intervention group 73% 74% ▲ 1% 20% 40% 60% 80% 100% Comparison group* 61% 46% ▼ 15% BASELINE ENDLINE Intervention group* 60% 47% ▼ 13% Received at least one TT injection during ANC* (DID p=0.00) Received at least two TT injections during ANC (DID p=0.34) Figure 7. Percentage of women examined by a skilled health worker after childbirth (DID p=0.75) 10% 20% 30% 40% 50% Comparison group 27% 25% ▼ 2% BASELINE ENDLINE Intervention group^ 32% 29% ▼ 3% care consultation. However, there is a significant decline in the percentage of women receiving at least four ANC consultations in both groups. RBF had no effect on this indicator in the intervention group. Literate women, a rural residence, and contact with the HC nurse in the previous month were significantly linked to women receiving at least four antenatal consultations. Preventive MPA services among households (quality) Antenatal services: About 74% of mothers received at least one dose of Tetanus Toxoid (TT) injection and 47% of mothers received at least two doses of TT vaccination while pregnant with their youngest child in the intervention group. Younger women, a rural residence, contact with the HC nurse, and CHW in the previous month were significantly associated with women receiving at least one TT injection during pregnancy. There were significant reductions in women receiving TT injections while pregnant in the comparison group. RBF appears to be the contributing factor in preventing the decline in the intervention group for women receiving at least one TT injection, but had no effect on women receiving two doses of TT injections (Figure 6). Postnatal services: The examination of women by a skilled birth attendant after childbirth decreased significantly in both groups (Figure 7). There was a significant positive association with women who had contact with an HC nurse, and contact with a CHW, in the previous month. About 26% of mothers received an initial dose of vitamin A within two months of childbirth in the intervention group. There was a significant reduction, by 5% of vitamin A provision, in the comparison group (Figure 8, next page). RBF appears to be the contributing factor in preventing the decline in the intervention group. A positive association was found with women who had contact with an HC nurse. Source (Figures 5–10): Household Surveys 2013, 2015 Key: *p<=0.05; ^p>0.05 and <=0.10; DID=difference in difference analysis Intervention group: Baseline n=1791, Endline n=1797 Comparison group: Baseline n=1812, Endline n=1813 28 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Source (Figures 5–10): Household Surveys 2013, 2015 Key: *p<=0.05; ^p>0.05 and <=0.10; DID=difference in difference analysis Intervention group: Baseline n=1791, Endline n=1797 Comparison group: Baseline n=1812, Endline n=1813 10% 20% 30% 40% 50% Comparison group* 26% 21% ▼ 5% BASELINE ENDLINE Intervention group 25% 26% ▲ 1% Figure 8. Percentage of women who received first dose of vitamin A within 2 months of childbirth* (DID p=0.00) 20% 40% 60% 80% 100% Comparison group* 58% 53% ▼ 5% BASELINE ENDLINE Intervention group 54% 56% ▲ 2% 65% 7% 25% During a Campaign Routine Visit During Illness Don’t Know 3% Figure 9. Percentage of children (0–23m) who received a dose of vitamin A within 6 months* (DID p=0.00) Figure 10. Source of vitamin A Similarly, RBF’s contribution is seen in the fact that the youngest child in a family (aged 0–23 months) received at least one dose of vitamin A within six months of birth in the intervention group. Urban residence was significantly linked to the child receiving at least one dose of vitamin A. Respondents specified the source of vitamin A was 65% during a campaign, followed by 25% during a routine visit, and 3% during illness. An estimated 7% did not know where they received vitamin A (Figures 9 and 10). Figure 11. Percentage of current users of any family planning methods (traditional or modern) and modern contraceptive users 20% 40% 60% 80% 100% Comparison group* 44% 49% 5% BASELINE ENDLINE Intervention group 47% 48% ▲ 1% ▲ 10% 20% 30% 40% 50% Comparison group* 7% 17% 10% BASELINE ENDLINE Intervention group* 7% 22% ▲ 15% ▲ Users of any type of family planning method* DID p=0.05 Current users of modern contraceptives* DID p=0.01 Figure 12. Type of modern contraceptive method used 60% 4.5% 20% Male Condoms Injectible Permanent Other 13% Pills 1.5% Source (Figures 11–13): Household Surveys 2013, 2015 Key: *p<=0.05; ^p>0.05 and <=0.10; DID=difference in difference analysis Intervention group: Baseline n=1791, Endline n=1797; Comparison group: Baseline n=1812, Endline n=1813 EVALUATION FINDINGS n 29 Family planning services: Current users of any type of contraceptives increased significantly in the comparison group, but not in the intervention group. However, modern contraceptive10 use by non-pregnant mothers increased significantly from 7% to 22% in the intervention group, significantly higher than the comparison group (17%). RBF’s contribution is seen as increasing the use of modern contraceptives (Figure 11). The most common method used was male condoms (61%), followed by injections (20%) and pills (13%) Long-lasting methods accounted for only 1% (Figure 12). Literate women and those who had contact with an HC nurse were found to be most likely to use a modern method. Mothers receiving FP advice from a health worker improved significantly in both groups (Figure 13). The most common source of FP information in both groups was an HC nurse (75%), CHW (9%), birth assistant (9%), doctor (2%), and other health worker (5%). 10. Modern contraceptive indicator analysis combined data on 10 methods reported: Male condom, female condom, male sterilization, female sterilization, pills, IUDs, injectables, implants, diaphragm and emergency contraception (morning-after pill). Curative MPA service utilization pattern among households (quantity and quality) Availability of curative MPA services at the health centers (quantity): All health centers surveyed provided curative child health services. The availability of normal childbirth services, nutritional rehabilitation and care and de-worming services improved significantly in both groups, whereas prevention of mother-to-child transmission of HIV (PMTCT) services declined (Table 9). The referral services for obstetrics complications improved significantly in the intervention group between 2013 and 2015, but not in the comparison group. RBF appears to be a contributing factor for this increase of high-risk obstetric referrals. There were no changes observed for the care and treatment services of HIV/TB/STIs, and blood transfusion services in the health centers. 10% 20% 30% 40% 50% Comparison group* 14% 20% 6% BASELINE ENDLINE Intervention group* 15% 23% ▲ 8% ▲ Figure 13. Received family planning advice from a health worker in the past one year (DID p=0.75) Table 9. Quantity of curative MPA services at health centers: Availability of services Intervention Group (N=167) Comparison Group (N=172) DID analysis Availability of curative MPA services 2013 2015 p-value 2013 2015 p-value p-value Facility offers child health curative services 99% 100% 0.3 99% 100% 0.3 1.0 Facility offers normal labor services 85% 96% 0.01* 87% 98% 0.01* 0.8 Facility offers advice to pregnant women for PMTCT 57% 42% 0.05* 50% 38% 0.10^ 0.8 Facility offers emergency obstetrics referrals 66% 86% 0.00* 65% 76% 0.44 0.08^ Facility offers nutritional rehabilitations 30% 48% 0.00* 14% 41% 0.00* 0.6 Care and treatment of worm infestation 40% 80% 0.00* 51% 80% 0.00* 0.9 Facility offers diagnosis/care/treatment of TB 47% 42% 0.62 56% 44% 0.12 0.5 Facility offers diagnosis/care/treatment of HIV 14% 16% 0.62 15% 20% 0.40 0.8 Facility offers diagnosis/care/treatment of STIs 68% 72% 0.61 65% 70% 0.68 0.9 Facility offers blood transfusion services 11% 16% 0.33 15% 22% 0.27 0.52 Facility offers minor surgeries 96% 93% 0.73 89% 94% 0.22 0.14 Source: Health center surveys, 2013, 2015 Key: Red font indicates significant result. *p<=0.05; ^p>0.05 and <=0.10; DID=difference in difference analysis 30 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Table 10. Quantity of curative MPA services at health centers: Client statistics Intervention Group (N=167) Comparison Group (N=172) DID analysis Client statistics for curative services 2013 2015 p-value 2013 2015 p-value p-value Average number of outpatients (new and old cases) visits in the past year^ 2162 3577 0.00* 2077 2598 0.05* 0.10^ Average number of sick child consultations for under-5 children in the past year 656 1229 0.00* 540 869 0.02* 0.20 Average number of childbirths attended by facility personnel in the past year 218 246 0.10^ 211 263 0.00* 0.42 Average number of pregnant women referred to GRH for obstetrics complications* 9 35 0.00* 12 25 0.02* 0.04* Average number of pregnant women screened for HIV in the past year 34 42 0.03* 25 45 0.00* 0.64 Source: Health center surveys, 2013, 2015 Key: Red font indicates significant result. *p<=0.05; ^p>0.05 and <=0.10; DID=difference in difference analysis Endline CDF 5,546 COMPARISON* INTERVENTION* Baseline 3,000 4,000 5,000 CDF 6,000 CDF 5,048 Figure 15. Cost of services paid during last visit to a local HC (DID p=0.22) Figure 14. Percentage of respondents who sought treatment in a local health center when needed health service 20% 40% 60% 80% 100% Comparison group* 60% 80% BASELINE ENDLINE Intervention group* 63% 89% ▲ 26% ▲ 20% 10% 20% 30% 40% 50% Comparison group* 18% 9% BASELINE ENDLINE Intervention group* 18% 5% ▼ 13% ▼ 9% Sought care in an HC first* DID p=0.00 Went to neither an HC nor a GRH when needed* DID p=0.00 Source: Household Surveys 2013, 2015 Key: *p<=0.05; ^p>0.05 and <=0.10; DID=difference in difference analysis Intervention group: Baseline n=1791, Endline n=1797; Comparison group: Baseline n=1812, Endline n=1813 Source: Household Surveys 2013, 2015 Key: *p<=0.05; ^p>0.05 and <=0.10; DID=difference in difference analysis Intervention group: Baseline n=1120, Endline n=1599; Comparison group: Baseline n=1087, Endline n=1450 11% Other 5% Prefer traditional medicine 3% Staff not qualified 1% Nurse not welcoming 44% Not enough money to pay 36% Too far Figure 16. Reasons given for not vising a health facility when needed health service, 2015 Source: Household Surveys 2015 Intervention group: n=90 Comparison group: n=163 52% Not enough money to pay 11% Prefer traditional medicine 8% Other 2% Nurse not welcoming 2% Staff not qualified 25% Too far Intervention Group Comparison Group EVALUATION FINDINGS n 31 Service utilization statistics for curative MPA services at health centers (quantity): Between 2013 and 2015, the average number of outpatients (old and new) coming to health centers and pregnant women referred to GRH for obstetrics complications increased significantly in the intervention group, compared to the comparison group (Table 10). RBF appears to be a contributing factor for these improvements. The average number of pregnant women screened for HIV, and normal births attended by facility personnel, significantly improved in both groups between 2013 and 2015. General health services utilization: Respondent mothers were asked to report their experience of their last visit to a local HC. Approximately 89% of mothers in the intervention group and 80% in the comparison group had visited a local health center within the past three months of survey. This represents a significant increase in the health center service utilization rate than the baseline rate (63% and 60%) for both groups. The practice of not visiting a health facility when needing health care declined significantly in both groups (Figure 14). The RBF intervention appears to be a factor for positive change in health service utilization patterns. Eighty-six percent (86%) had contact with a health worker at a health center. The cost of services paid during the recent visit in both groups was similar. However, there is a significant increase in the cost of services paid in both groups at the endline (Congolese Franc (CDF)=~5,000) compared to the baseline (CDF=~4,000) (Figure 15). It should be noted that this figure reflects the bundle of services the respondent paid for, which could include one or more child’s care and/or the mother’s care. On the contrary, the costs for the initial consult, specific services, and medicines did not significantly increase in either the intervention or comparison group. The median cost for an initial (first) visit was CDF 800 in the intervention and CDF 1050 in the comparison group. The most common two reasons given by both groups for not visiting a health facility were the inability to pay for the service and the distance from a health center (Figure 16). Childbirth services utilization: The prevalence of facility births in a public or a private setting remained high at 84% in the intervention group, although there was a significant decline in the comparison group. Childbirth in a health facility is strongly associated with literacy and contact with an HC nurse. The presence of a skilled birth attendant (doctor, nurse, nurse-midwife) during childbirth improved significantly in both groups. The presence of a skilled birth attendant during childbirth was significantly associated with the employed status of the head of household and an urban residence (Figure 17). Figure 17. Percentage of women gave birth in a health facility and percentage assisted by a skilled health worker 20% 40% 60% 80% 100% Comparison group* 78% 74% ▼ 4% Intervention group 83% 84% ▲ 1% BASELINE ENDLINE 20% 40% 60% 80% 100% Comparison group* 50% 67% Intervention group* 48% 62% ▲ 17% BASELINE ENDLINE ▲ 14% Respondent birth in a public or private health facility* (DID p=0.02) Health worker assisted birth in public or private facility (DID p=0.15) Source: Household Surveys 2013, 2015 Key: *p<=0.05; ^p>0.05 and <=0.10; DID=difference in difference analysis Intervention group: Baseline n=1791, Endline n=1797; Comparison group: Baseline n=1812, Endline n=1813 32 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Curative MPA services (quality) Integrated management of childhood illnesses services Quality of Care Measurement Index: We used iterative searches to identify literature describing the measurement of quality of healthcare services. Based on the review, we developed a quality-of-care index for the data analysis. The model identifies three proximate determinants that are lead to the appropriate care and treatment of illness, and in this case, the treatment of childhood illnesses (i.e. PP, diarrhea, and malaria). As it is not feasible in large surveys to utilize nurses and doctors to directly observe client care, nor to use “mystery clients,” mother’s recall of the appropriate treatment of these three illnesses was used as the best proxy for quality of care. Figure 18 below illustrates the relationship between appropriate care of illness (as described by mothers who had a sick child in the two weeks prior to the household interview) and variables related to the three proximate determinants which could potentially influence the results of the treatment. The first proximate determinant, structural inputs of the health facility, are defined as availability of commodities that include medications, supplies, electricity, equipment, communication system, and transport; and infection control measures in situ such as continuous water source, basic sanitation (latrines), equipment needed for proper sterilization, and the removal of bio-hazardous waste. The second proximate determinant in this quality-of-care index is personnel. The variables selected to measure the capacity and availability of personnel include those directly related to the staff, such as their level of pre-service training and the provider-to-cli￾ent ratio, and more specifically client-to-nurse ratio, as it is quite rare to encounter a doctor working in an HC. Another category of variables under personnel include the tools of supportive super￾vision, such as availability and access to quality information (e.g., health management information system or other data source) and access to motivated staff (health workers in the DRC are not typically paid salaries). The third proximate determinant, health-seeking behaviors, rely on the health information received by the community or clients that relate to changes in their cognitive and psycho-social attributes such as knowledge, attitude and practices which influence client decision-making that drive their behaviors. There is a body of evidence indicating that clients’ perception of care leads to increased utilization of services (Baltussen, R. et al, 2002; Larson, E. et al, 2014; Manzi A. et al, 2014; Nanyonjo A. et al, 2013). Thus, this Index takes note of the myriad variables that influence the perception of quality of care received by facility clients such as the interpersonal communication skills of providers, interpretation of whether prices for services are reasonable, wait times, whether staff are adequate to meet the volume of patients in a facility, overall satisfaction with the care received, and recent contact or exposure to a health provider. This cluster of factors influences a client’s overall viewpoint as to whether she will receive quality care and therefore, influence her health-seeking behaviors. CLIENT DECISION M- AK NI G CL EI NT PERCEPTIONS Contact with a health provider Provider interpersonal skills Reasonable price, wait times Adequate staff available Satisfaction with care Knowledge Attitudes Practices Medications Supplies Equipment Electricity Communication systems Transport COMMODITIES INFECTION CONTROL Continuous water source Basic sanitation Equipment sterilization Bio-hazard waste removal SUPPORTIVE SUPERVISION Access to quality data Access to paid staff STAFF Level of pre-service training Provider: client ratio Quality of Care Measurement Index Health-Seeking Behaviors Structural Personnel Inputs Quality of Care Appropriate Care and Treatment Figure 18. Quality of Care Measurement Index EVALUATION FINDINGS n 33 Malaria, diarrhea and pneumonia are major morbidity and mortality factors in children under 5. One of IHP’s activities was to support IMCI and improve the quality of care related to childhood illnesses. The Quality of Care Measurement Index was utilized to analyze the care received for children who were reported to have diarrhea, presumed pneumonia (PP), or suspected malaria within two weeks prior to the household survey. The logistic regression model used the treatment outcomes (children with diarrhea received oral rehydration therapy; children with PP were treated with antibiotics; children with fever who went to a health facility received antimalarial treatment) as dependent variables. A mother’s literacy, mother’s age, employment status of the head of household, urban/rural residence, contact with a health center nurse, and contact with a CHW in the past month were used as independent variables. The knowledge of fever as a childhood illness symptom increased significantly in the intervention group due to RBF (Table 11). Literacy status was significantly associated with the increased knowledge. The knowledge of symptom of ‘child cannot drink or breastfeed’ decreased significantly at the endline, however, there is no effect of RBF on this knowledge decline. The knowledge of other symptoms such as vomiting, diarrhea, and child’s worsening condition remained unchanged. Diarrhea: About 37% of children aged 0–23 months reportedly suffered from diarrhea in the two weeks preceding the survey. In both groups, more than 50% of mothers of children with diarrhea gave them oral rehydration salts (ORS). This included a solution prepared using either a special packet, prepackaged ORS liquid, or recommended homemade sugar-salt solution (Figure 19). Mothers who administered any form of ORS for diarrhea were found to have been in contact with an HC nurse in the previous month and were literate. A mother’s age, urban/rural residence, and employment were not associated to ORS use. Pneumonia: Fifty-five percent (55%) of children aged 0–23 months reportedly suffered from a cough or presumed pneumonia (PP) in the last two weeks preceding the survey. Fifty-nine percent (59%) of children with PP were taken to a health facility. At the health facility, 66% of children with PP were treated with antibiotics, similar to the comparison groups (65%) (Figure 19, next page). The key factor in bringing the child to the health facility was the employment of the head of household. All other demographic factors such as a mother’s age, place of residence, literacy, or contact with a health worker were not found to play a key role. Forty-five percent (45%) of children aged 0–23 months reportedly suffered from fever or suspected malaria in the two weeks before the survey and 68% were taken to a health facility. Fifty-nine percent (59%) of children with PP were taken to a health facility. Twenty-nine percent (29%) of children with fever had their blood drawn for malaria testing, an increase of 2% in the intervention group at endline in comparison to the baseline (Figure 20, next page), whereas malaria testing reduced in the comparison group at endline. RBF appears to be a factor in maintaining the higher rates of malaria testing in the intervention group (p=0.07). Antimalarial treatment rates declined significantly in both groups, by more than 30% at endline. A rural residence and contact with an HC nurse in the previous month was significantly associated with children receiving malaria blood tests. No significant association with any factor was established relative to receiving antimalarial medications. HIV testing services: Twenty percent (20%) of mothers in the intervention group, and 25% in the comparison group, reported being tested for HIV, significantly higher at endline for both groups. However, those tested and those receiving test results were similar in both groups (Figure 21, next page). DID analysis does not show any effect of RBF on either HIV testing Table 11. Knowledge of childhood illness symptoms Intervention Group Comparison Group DID analysis Mother’s knowledge of the top five reasons to take a child to a health facility right away 2013 (n=1776) 2015 (n=1807) p-value 2013 (n=1812) 2015 (n=1813) p-value p-value Child develops fever* 86% 88% 0.04* 87% 88% 0.09^ 0.05* Child has diarrhea 40% 39% 0.15 37% 37% 0.58 0.20 Child cannot drink or breastfeed 36% 29% 0.00* 37% 29% 0.00* 0.42 Child condition is getting worse 25% 24% 0.65 25% 24% 0.80 0.70 Child has vomiting 18% 18% 0.80 20% 19% 0.70 0.64 Source: Health center surveys, 2013, 2015 Key: Red font indicates significant result. *p<=0.05; ^p>0.05 and <=0.10; DID=difference in difference analysis 34 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Figure 19. Percentage of children under 5 receiving appropriate treatment for diarrhea or PP 20% 40% 60% 80% 100% Comparison group* 44% 51% Intervention group* 47% 53% ▲ 7% BASELINE ENDLINE ▲ 6% 20% 40% 60% 80% 100% Comparison group 68% 65% Intervention group 69% 66% BASELINE ENDLINE 3% ▼ 3% ▼ Diarrhea treated with oral rehydration salts (DID p=0.92) PP treated with antibiotics in a health center (DID p=0.95) Source: Household Surveys 2013, 2015 Key: *p<=0.05; ^p>0.05 and <=0.10; DID=difference in difference analysis Intervention group: Baseline n=725, Endline n=789; Comparison group: Baseline n=893, Endline n=977 Figure 20. Percentage of children receiving appropriate treatment for malaria 10% 20% 30% 40% 50% Comparison group 19% 23% Intervention group 27% 29% BASELINE ENDLINE 2% 4% ▼ ▲ 20% 40% 60% 80% 100% Comparison group* 49% 84% Intervention group* 79% 48% BASELINE ENDLINE 35% ▼ 31% ▼ Child with suspected malaria had blood drawn for testing^ (DID p=0.07) Suspected malaria treated with antimalarials in HC (DID p=0.27) Source: Household Surveys 2013, 2015 Key: *p<=0.05; ^p>0.05 and <=0.10; DID=difference in difference analysis Intervention group: Baseline n=805, Endline n=795; Comparison group: Baseline n=811, Endline n=825 Figure 21. Percentage of women ever tested for HIV and received results 10% 20% 30% 40% 50% Comparison group* 18% 25% 7% BASELINE ENDLINE Intervention group* 16% 20% ▲ 4% ▲ 20% 40% 60% 80% 100% Comparison group 85% 81% Intervention group 76% 79% BASELINE ENDLINE 3% ▲ 4% ▲ Ever tested for HIV (DID p=0.16) Received HIV test results (DID p=0.56) Figure 22. Percentage of women ever tested for HIV and received results while pregnant 10% 20% 30% 40% 50% Comparison group* 14% 18% 4% BASELINE ENDLINE Intervention group* 11% 15% ▲ 4% ▲ 20% 40% 60% 80% 100% Comparison group 89% 88% 1% BASELINE ENDLINE Intervention group 84% 79% ▼ 5% ▼ Tested for HIV while pregnant (DID p=0.89) Received HIV test results when tested while pregnant (DID p=0.35) Source (Figure 21–25): Household Surveys 2013, 2015 Key: *p<=0.05; ^p>0.05 and <=0.10; DID=difference in difference analysis Intervention group: Baseline n=1791, Endline n=1797; Comparison group: Baseline n=1812, Endline n=1813 EVALUATION FINDINGS n 35 or receiving HIV test results. Similar results were seen for HIV testing and receiving test results for pregnant women (Figure 22). A mother’s literacy and urban residence was found to be significantly associated with the practice of HIV testing among mothers with children aged 0–23 months. When interpreting these results, bear in mind that HIV and AIDS interventions were limited to select health zones (Kayamba and Kanzenze). Not surprisingly, the two RBF indicators originally included in the assessment of the services purchased at baseline were subsequently removed by IHP in 2015. Health promotion activities Health promotion services related to general hygiene, sanitation and exclusive breastfeeding for first six months improved significantly in the intervention group with no change in the comparison group (Table 12). Health promotion services related to food hygiene and safety, improvement of latrines, use of ORS in diarrhea, and genital fistula prevention improved significantly in both groups, whereas iodized salt promotion improved significantly in the comparison group with no change in the intervention group. Other health promotional services related to condom promotion and long-lasting insecticide-treated bed nets (LLITNs) for malaria prevention were maintained at high coverage levels. The availability of IEC materials related to family planning and vaccination improved significantly in the intervention health zones due to RBF’s intervention. However, IEC materials related to acute respiratory treatment decreased significantly in both groups—more in the comparison group than the intervention group. RBF’s intervention prevented further decline in the intervention group. The availability of HIV prevention IEC material was significantly improved in the intervention group. IEC materials related to child growth monitoring, maternal and child health, nutrition, malaria, and diarrhea, improved significantly in both groups. Table 12. Quantity of promotional MPA services at health centers: Availability of services Intervention Group (N=167) Comparison Group (N=172) DID analysis Availability of health promotional MPA services 2013 2015 p-value 2013 2015 p-value p-value Facility does condom promotion 96% 98% 0.6 93% 98% 0.13 0.62 Facility does general hygiene and sanitation promotion 81% 98% 0.00* 92% 97% 0.18 0.20 Facility does exclusive breastfeeding promotion 93% 99% 0.05* 96% 99% 0.5 0.63 Facility does food hygiene and safety promotion 81% 95% 0.00* 87% 95% 0.05* 0.63 Facility does iodized salt consumption promotion 27% 35% 0.23 36% 48% 0.09^ 0.76 Facility does improvement of latrines promotion 75% 89% 0.01* 78% 93% 0.02* 0.69 Facility does ORS in diarrhea promotion 80% 99% 0.00* 85% 99% 0.00* 0.83 Facility gives information on fistula prevention 26% 44% 0.02* 28% 46% 0.02* 0.84 Facility distributes LLITN for malaria prevention 96% 93% 0.49 89% 94% 0.36 0.27 Availability of health promotional MPA services Has IEC materials for family planning* 83% 94% 0.03* 91% 89% 0.59 0.05* Has IEC materials for vaccination* 49% 96% 0.00* 45% 83% 0.00* 0.03* Has IEC materials for child growth monitoring 79% 65% 0.05* 75% 62% 0.07^ 0.81 Has IEC materials for maternal nutrition 37% 71% 0.00* 21% 56% 0.00* 0.82 Has IEC materials for pre/postnatal care 76% 90% 0.01* 72% 84% 0.07^ 0.49 Has IEC materials for breastfeeding 57% 86% 0.00* 48% 81% 0.00* 0.96 Has IEC materials for vitamin A 14% 69% 0.00* 9% 61% 0.00* 0.82 Has IEC materials for diarrhea prevention 23% 83% 0.00* 15% 68% 0.00* 0.50 Has IEC materials for acute respiratory infections* 85% 66% 0.00* 85% 36% 0.00* 0.03* Has IEC materials for malaria 72% 90% 0.00* 70% 82% 0.07^ 0.26 Has IEC materials for HIV prevention 34% 51% 0.03* 24% 33% 0.18 0.62 Source: Health center surveys, 2013, 2015 Key: Red font indicates significant result. *p<=0.05; ^p>0.05 and <=0.10; DID=difference in difference analysis 36 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Promotion of long-lasting insecticide-treated bed nets for malaria prevention: The availability in the household of at least one LLITN declined significantly, by 30% in both groups. In the households where LLITN was observed by the interviewer, more than 90% reported that their child slept under the bed net the previous night (Figure 23). The findings were similar in both groups. Bed net use was frequently associated with contact with an HC nurse and employment. Promotion of water, sanitation, and hygiene (WASH): Households receiving advice on the importance of WASH, use of improved sources of water, and use of a method to make water potable (boiling or use of chlorine) improved significantly in the Figure 23. Percentage of households reporting at least one LLITN and child slept under a bed net last night 20% 40% 60% 80% 100% Comparison group* 80% 48% 32% BASELINE ENDLINE Intervention group* 78% 48% ▼ 30% ▼ 20% 40% 60% 80% 100% Comparison group* 92% 95% BASELINE ENDLINE Intervention group* 91% 93% ▲ 2% ▲ 3% Has at least one LLITN (DID p=0.23) Child slept under LLITN last night (DID p=0.64) Figure 24. Percentage of households making water potable before use 10% 20% 30% 40% 50% Comparison group* 12% 21% BASELINE ENDLINE Intervention group* 12% 25% ▲ 13% ▲ 9% 5% 10% 15% 20% 25% Comparison group 12% 3% BASELINE ENDLINE Intervention group 2% 3% ▲ 1% NC 3% Health worker gave advice on the importance of WASH* (DID p=0.04) Household use method to make water potable^ (DID p=0.09) Figure 25. Percentage of households with improved water source and toilets 10% 20% 30% 40% 50% Comparison group 32% BASELINE ENDLINE Intervention group* 26% 30% ▲ 4% NC 32% 5% 10% 15% 20% 25% Comparison group 13% BASELINE ENDLINE Intervention group 4% 3% ▼ 1% NC 13% Improved source of water^ (DID p=0.06) Improved type of toilets (DID p=0.19) intervention group. The changes observed could be attributed to the RBF intervention (Figure 24). However, the availability of improved toilets did not show any change between the intervention and comparison group (Figure 25). The presence of an improved source of water and toilet in a household was significantly linked to an urban residence, employment, and contact with an HC nurse. Health facility management Facility operations and management: Information was collected using the facility survey on the availability of basic infrastructure and facility operations, including supervision, drugs and supplies, infection control, and waste management (Table 13). Results were similar for health facility staff who reported re￾ceiving training over the past three years. Facility supervision by MSP staff in the past three months preceding the survey improved significantly in the intervention group, in contrast to the comparison group, with 99% supervision rates. The aver￾age nurse-to-catchment population ratio in the intervention and comparison health zones is 0.26 per 1000 population in 2015. Stock management and reports of delays in supply of medications or supplies decreased significantly in the interven￾tion group. Stockouts for oral contraceptives and injectables were more frequent in the comparison group, while the health facilities in the intervention group had less stockout rates. RBF’s intervention prevented the contraceptives stockout rates. The predominant sources of procurement for medications and sup￾plies were similar for both groups: central offices of health zones (55%); followed by international NGOs (22%); and private sup￾pliers (23%). However, the stock of vaccines remained low and unchanged in both groups. Because cold chains are difficult to EVALUATION FINDINGS n 37 keep intact, rural health centers provide immunizations once per month across a two or three day period. Vaccines are retrieved from the ECZ once per month. The analysis of facility infrastructure variables shows significant changes in basic infrastructure. The availability of continuous electricity in the health centers improved significantly in the intervention group. The main source of improved electricity was reportedly solar panels. The availability of continuous water supply in the health facility building and access to transportation at all times improved significantly in the intervention group. The water sources, in the form of rainwater cisterns and improved wells, reportedly increased. Bicycles were reported as the primary means of transportation. All the above changes could be attributed to the RBF intervention. Infection control measures in the intervention health centers improved significantly, attributable to the RBF activities. The Table 13. Quality of MPA services at health centers Facility operations and management Intervention Group (N=167) Comparison Group (N=172) DID analysis Staff training, supervision, and stock management 2013 2015 p-value 2013 2015 p-value p-value Received supervision by MSP at least once in 3 months* preceding the survey 83% 99% 0.00* 81% 79% 0.55 0.03* Staff receiving training in the past 3 years 95% 89% 0.11 97% 89% 0.11 0.53 Facility experiencing delays for medications/supplies^ 91% 83% 0.09^ 84% 89% 0.33 0.06^ Facility experiencing delays for contraceptive supplies 27% 34% 0.30 39% 47% 0.27 0.99 Has oral contraceptives in stock today (verified by observer)* 60% 72% 0.10^ 61% 51% 0.21 0.03* Has Depo-Provera injection in stock today (verified by observer)^ 80% 85% 0.42 80% 69% 0.10^ 0.10^ Has DPT vaccine in stock today (verified by observer) 18% 11% 0.14 22% 16% 0.26 0.70 Has OPV vaccine in stock today (verified by observer) 17% 12% 0.29 25% 17% 0.20 0.97 Has measles vaccine in stock today (verified by observer) 17% 12% 0.29 23% 18% 0.36 0.83 Facility infrastructure Has waiting room 82% 84% 0.70 81% 78% 0.67 0.57 Has continuous electricity supply 14% 37% 0.00* 18% 24% 0.19 0.12 Has water supply inside the building* 25% 46% 0.00* 25% 25% 1.0 0.05* Has emergency communication system 9% 13% 0.38 13% 8% 0.27 0.17 Has access to transportation at all times* 81% 79% 0.69 81% 54% 0.00* 0.03* Has toilets 95% 99% 0.15 86% 85% 0.92 0.20 Infection control measures Has bio-medical waste disposal procedures (incineration or pit burial)* 95% 100% 0.04* 95% 93% 0.56 0.00* Has disinfectants currently in stock^ 84% 98% 0.00* 81% 90% 0.10^ 0.10^ Source: Health center surveys, 2013, 2015 Key: Red font indicates significant result. *p<=0.05; ^p>0.05 and <=0.10; DID=difference in difference analysis availability of disinfectant for infection control was reported in 98% of health centers. The most effective method for medical equipment sterilization is the autoclave; its availability improved in surveyed health facilities. There was a significant decline in “boiling” as the predominant method for equipment sterilization and, at the same time, an improvement in “autoclave” and “dry or steam heat” as the predominant methods of sterilization. Similarly, there was a significant change in the type of medical waste management used in the health facilities. The predominant method of medical waste disposal method in the intervention group at endline was incineration (71%), outside burning (26%), or in a proper incinerator (45%), an effective method compared to the burial method (28%). This is a significant change in contrast to the comparison HCs, where incinerator use was 29% and burial method 43%. 38 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO FOSACOF scores: In accordance with the MSP guidelines, quality was specifically tracked in terms of services that RBF￾contracted facilities provided. To do this, a quality evaluation rubric called FOSACOF was used for the Formation Sanitaire, FOSAs (includes HCs, GRHs, and other contracting facilities). IHP promoted FOSACOF as a tool for measuring quality. FOSACOF scores were one of the 16 “paid RBF indicators” at the HC level and the only indicator for GRHs. Of the total RBF score, FOSACOF comprised 29% for the HCs and client satisfaction scores compromised of 12%, compared to single digit percentages for the remaining variables. Figures 26 and 27 above summarize the progress of verified and validated FOSACOF scores for HCs and GRHs respectively from baseline to the sixth quarter of the RBF implementation. Figures 28 and 29 on the following page break out these scores by intervention health zone. GRHs had improved their overall scores by up to 87% in the intervention group, versus 57% in the comparison group, by the end of the 6th quarter and Luiza’s GRH had highest percentage improvement for FOSACOF scores (+66%). Similarly, the overall scores improved for health centers from 33% at baseline to 72% at the end of 6th quarter in the intervention group; health centers in Bibanga HZ had highest percentage improvement for FOSACOF scores (+70%). Client satisfaction: Respondent mothers were asked to report their level of satisfaction with the services received during their last visit (within past three months) to a local health center. Figure 30 shows a visual representation of client satisfaction data. Word clouds are used to depict keywords (tags). The frequency of each tag is shown with font size. This format is used for quickly presenting the percentage of respondent reported satisfaction with her experience with services received during the last visit to a health center. Approximately 89% of mothers expressed satisfaction with the overall quality of services received, with a significant increase in the comparison group and no change in the intervention group at endline. The perception of interpersonal skills (a nurse listening attentively to the client and treating them professionally) improved significantly for both groups. The perception of the amount of time the service provider spent per client did not change for the intervention group but significantly improved for the comparison group (Figure 31). Figure 26. Trend analysis: Health Center FOSCAOF score, Intervention vs. Comparison 20% 40% 60% 80% 100% 33% COMPARISON INTERVENTION BL 27% 33% 43% T1 37% 53% T2 39% 59% T3 41% 60% T4 45% 69% T5 46% 72% T6 Source (Figures 26–29): IHP reports 2013–2015 BL: Baseline, T1=Nov 13–Jan 2014, T2=Feb–Apr 2014, T3=May–Jul 2014, T4=Aug–Sep 2014, T5=Oct–Dec 2014, T6=Jan–Mar 2015 • GRHs n=7, HCs n=118 20% 40% 60% 80% 100% 45% COMPARISON INTERVENTION BL 39% 39% 64% T1 50% 74% T2 51% 78% T3 54% 78% T4 56% 81% T5 57% 87% T6 Figure 27. Trend analysis: GRH FOSCAOF score, Intervention vs. Comparison EVALUATION FINDINGS n 39 20% 40% 60% 80% 100% 30% BASELINE ENDLINE TOTAL 34% 73% 100% Bibanga 44% 73% Nundu 44% 71% Kanzenze 41% 71% Wembo Nyama 26% 70% Luiza 27% 68% Kayamba 26% 56% Lomeda Figure 28. Health Center FOSCAOF scores by health zone, Baseline vs. Endline 20% 40% 60% 80% 100% 47% BASELINE ENDLINE TOTAL 39% 87% 79% Bibanga 55% 81% Nundu 37% 96% Kanzenze 46% 92% Wembo Nyama 15% 81% Luiza 37% 93% Kayamba 37% 85% Lomeda Figure 29. GRH FOSCAOF scores by health zone, Baseline vs. Endline Figure 30. Perceived quality of care received during a recent HC visit price posted reasonable prices negotiable prices compassionate staff staff adequate rooms appropriate medication available distance to HC respectful staff waiting time honest staff Source: Household Survey 2015, Intervention group n=1599, Comparison group n=1450 price posted reasonable prices negotiable prices compassionate staff staff adequate rooms appropriate medication available distance to HC respectful staff waiting time honest staff Intervention Group Comparison Group Figure 31. Client satisfaction with the quality of care received during a recent HC visit (within 3 months) 20% 40% 60% 80% 100% Received care sought Satisfied with care received* Nurse spent enough time* Nurse treated professionally* Nurse listened attentively* BASELINE ENDLINE Source: Household Survey 2015, Intervention group n=1599, Comparison group n=1450 Comparison Group 20% 40% 60% 80% 100% Received care sought Satisfied with care received Nurse spent enough time Nurse treated professionally* Nurse listened attentively* BASELINE ENDLINE Intervention Group 40 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Analytical Domain: Cost Associated with a Potential Replication of the Pilot RBF Model Data for the cost analysis were collected on MS Excel-based forms, developed by the evaluation team and provided to the MSH. The information was collected between May and October 2015 and covered the project’s five-year period. MSH, the only source of data, participated in discussions and verifications whenever necessary. Online information was made available on the RBF web portal and other RBF cost studies were also reviewed as references and to check validity of cost data. A summary of the initial estimated budget and the yearly budget expenditures are shown below (Table 14). Table 15 opposite provides details on how the estimated budget was expended according to the different budget-line items, the percentage of each line item in relation to the initial budget and to the expended budget, and the variation found between the initial budget and what was finally expended. The preparatory phase for the pilot RBF activity started in IHP’s third year. A total of $1,120,168 (17% of the total budget = $6,304,181) was invested in the preparatory phase. The preparatory phase included, among other items, IHP staff time in the preparation of the various materials and monitoring tools and the five-day trainings of the ECZs and health providers in 125 facilities; trainings were convened in each HZ. In November of 2013, the pilot was implemented in seven health zones, simultaneously targeting a population of 955,427 inhabitants. The initial budget estimate for a two-year implementation was $4,986,591. This allowed for a yearly per capita investment of $2.60 as was planned in 2013. However, the actual cost for a two-year implementation increased by 4% to $5,184,013 representing a $2.70 yearly per capita expenditure. A Cordaid/HealthNet PBF pilot RBF intervention in Katanga Province in 2006 budgeted $1.80 per capita and $2.00 in Rwanda. A 2009–2013 impact evaluation of a pay-for￾performance intervention by the World Bank in Haut-Katanga district of DRC indicated an investment of $0.43 investment per year; typically, $2.00–$3.00 are spent per year in similar contexts. An important budget line item, contributing to the highest budget increase, was the operating cost with other IHP activities at the central level (51%). These were accompanied by a decrease in internal supervision, monitoring, and experience sharing (66%). It should be noted that after the RBF Midterm Assessment was completed in 2014, the IBTCI evaluation team recommended that MSH allocate more resources to the central level IHP team to verify the large volume of data generated in the field. The conclusion was that having only one full time staff member and two part-time staff was insufficient to process the payments in a timely fashion. A significant change was the increase in the budget of the central IHP headquarters coordination staff (41%) and the BC office staff (25%). Costs expended to strengthen regulations at the central level also markedly increased (25%). A decrease (50%) was seen in the budget assigned to rewarding personnel who consistently performed well (excellence purchases) at the implementation levels. The total budget for the purchase of services amounted to $1,687,579, one third (33%) of the budget. It decreased 10% from the original budget, as did that of the HCs (9%). Hospital budgets increased 16%. Thus, the purchase of services went down from the initial estimate of $0.98 per capita to $0.88 per capita. The payment of services went to 118 health centers ($764,272 or 15%), seven referral hospitals ($623,921 or 12%) the health zone management team ($111,896 or 2%) and excellence purchases (payment of services rewarding good performance) in all facilities ($187,400 or 4%). Two thirds of the resources (67%) were utilized for the management and verification of RBF activities. Findings from the 2014 RBF Midterm Assessment revealed that the maximum RBF incentive paid to a hospital in one quarter was $12,000 and the maximum paid to an HC was $910—if the facility met or surpassed all indicator targets. Figure 32 graphically explains the distribution of the cost associated with a two-year RBF implementation in seven health zones. Table 14. RBF funding: Integrated Health Project Period Initial budget Executed budget Total 2011 2012 2013 2014 2015 Preparatory phase (2011–13) $ 1,284,680 $ 169,200 $ 291,700 $ 659,268 - - $1,120,168 Implementation phase (2014–15) $ 4,986,591 - - - $ 2,395,401 $ 2,788,612 $ 5,184,013 $ 6,271,271 $ 169,200 $ 291,700 $ 659,268 $ 2,395,401 $ 2,788,612 $6,304,181 EVALUATION FINDINGS n 41 Table 15. Initial budget vs. executed budget in seven HZs (10/2013–9/2015) Direct Costs Initial budget Distribution among line items Executed budget Distribution among line items Difference (%) Preparatory phase (All) $1,284,680 $ 1,120,168 -13% Implementation phase Human Resources: ■ Central coordination staff $352,080 7% $497,200 10% 41% ■ Coordination office (AAP) staff $575,400 12% $721,700 14% 25% ■ External technical assistance $48,000 1% $40,200 1% -16% TOTAL HUMAN RESOURCES $975,480 20% $1,259,100 24% 29% TOTAL CAPITAL INVESTMENTS (vehicles, equipment) $67,900 1% $108,600 2% 60% Service Purchases: ■ HC services $ 844,480 17% $764,272 15% -9% ■ GRH services $ 540,008 11% $623,921 12% 16% ■ ECZ services $ 107,520 2% $111,986 2% 4% ■ Excellence purchases: ECZ, GRH and HCs $ 374,800 8% $187,400 4% -50% TOTAL SERVICE PURCHASES $ 1,866,808 37% $1,687,579 33% -10% Technical verification: ■ Technical verification: health areas $ 612,052 12% $697,852 13% 14% ■ Technical and admin verification: GRHs, ECZs, health areas $ 81,340 2% $86,000 2% 6% TOTAL TECHNICAL VERIFICATION $693,392 14% $ 783,852 15% 13% TOTAL COMMUNITY VERIFICATION & COMMUNITY STRENGTHENING $483,000 10% $ 556,736 11% 15% Regulatory improvements: ■ Strengthening regulations at the provincial and operational level $ 349,450 7% $307,760 6% -12% ■ Strengthening regulation at the central level $ 163,845 3% $204,170 4% 25% TOTAL REGULATORY IMPROVEMENTS $513,295 10% $ 511,930 10% 0% Operating costs for RBF implementation: Shared operating costs: central level $ 95,302 2% $143,670 2% 51% Shared operating costs: BC (AAP) level $ 36,264 1% $44,896 1% 24% Supervision, monitoring, experience-sharing $ 255,150 5% $ 87,650 2% -66% TOTAL OPERATING COSTS $386,716 8% $ 276,216 5% -29% TOTAL IMPLEMENTATION COSTS $ 4,986,051 100% $ 5,184,013 100% 4% 42 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Analytical Domain: Results Differ for Various Groups (Heterogeneity) The results of the household and health facility surveys, including FOSACOF scores, have been analyzed, taking into account the heterogeneity among the seven RBF intervention HZs, each of which had a GRH. There were a total of 118 health facilities divided as follows: Nundu (21), Kanzenze (15), Kayamba (13), Wembo Nyama (15), Lomela (18), Luiza (18), and Bibanga (18). Table 16 on the next two pages shows a list of these selected variables that produced significant overall aggregate results when comparing the baseline to the endline results. In some zones, significant positive changes were attributable to RBF interventions, while in other zones, negative (undesired) results were detected. Significant positive changes are colored green, significant negative changes are colored red, and variables with no significant changes between baseline and endline are colored grey. For the most part, the grey colored variables that were not statistically significant did not show downward trends. However, those grey colored cells marked with a “▼” actually showed negative, albeit not statistically significant, changes. The percent occurrence of positive significant changes and negative significant changes for each category of variables are noted at the bottom of the respective section (e.g., household, facility, and quality of care). The grand total at the bottom of the table combines all 22 variables drawn from each of the three sections. In these totals, the percentages of non-significant changes are also noted at the bottom and within that category, the number of variables with undesired, albeit insignificant, changes (“▼”). It is worth noting that the better-performing HZs have fewer of these downward trends among their grey colored cells. Household survey results In the household survey, the variables that demonstrated the highest positive significant changes were “Improved water sourc￾es” (4/7 zones) and “Visit to health center in last three months” (3/7 zones). The “Child with diarrhea treated with ORS,” “Child with presumed pneumonia treated with antibiotics,” and “Child with suspected malaria had blood drawn” achieved significant positive changes in only 2/7 zones. “Child with suspected ma￾laria treated in health center” yielded negative significant results in 5/7 zones and showed downward trends in the two zones that did not have statistically significant results between baseline and endline (Lomela and Bibanga). In the aggregate, this downward trend between baseline and endline was statistically significant. In other words, more children were being treated for malaria in 2013 before the RBF pilot started than now. Facility survey results In the facility survey, the variables “Average number of outpatient visits,” “Average number of children receiving DPT1,” and “Average number of sick child consultations (<5 years)” showed significant positive changes in 3/7 zones. However, only one health zone, Lomela, showed positive changes for all three of those variables. The “Availability of emergency referral services” data were positive for Kanzenze and Bibanga. The former zone, Kanzenze, located in Katanga Province, is notable for its relatively good access to graded roads. “Nutritional rehabilitation” showed significant positive outcomes in 2/7 zones. Among all the facility survey variables that revealed positive changes in the aggregate for the RBF group, only one variable—“Average number of children 10% Regulatory improvements 11% Verifications (community) 5% Operations 2% Capital investments Service purchases 33% Human resources 24% Verifications (technical) 15% Figure 32. Costs associated with 2 years’ RBF implementation in seven health zones To summarize, of the total executed budget (excluding preparatory phase budget) of $5,184,013, 33% of that distribution was for service purchases, 24% for human resources, 15% for technical verifications, 11% for community verifications, 10% for regulatory improvements, 5% for operating costs and 2% for investments (e.g., vehicles and equipment). EVALUATION FINDINGS n 43 Table 16. Selected variables that on the aggregate of the seven health zones achieved significant differences Bibanga (3) Nundu Kanzenze Wembo Nyama(1) Luiza (3) Kayamba Lomela (2) Household survey (6 variables) Visit to health center in last three months ▼ ▼ Child with diarrhea treated with ORS ▼ Child with presumed pneumonia treated with antibiotics ▼ ▼ ▼ Child with suspected malaria had blood drawn ▼ ▼ Child with suspected malaria treated in health center ▼ ▼ Improved source of water at household ▼ ▼ Percentage of household variables with significant positive changes (# variables/6) 17% 17% 17% 83% 17% 17% 33% Percentage of household variables with significant negative changes (# variables/6) 0 17% 17% 17% 33% 17% 17% Bibanga (1) Nundu Kanzenze (2) Wembo Nyama Luiza (2) Kayamba Lomela (1) Facility survey (7 variables) Average number of outpatient visits in past year ▼ Average number of children receiving DPT-1 vaccine at health facility ▼ ▼ Average number of children receiving full vaccination Average number of sick child consultations (under five years) ▼ Availability of growth and development monitoring ▼ ▼ Availability of emergency referral services Availability of nutritional rehabilitation services ▼ Percentage of facility variables with significant positive changes (# variables/7) 57% 0 29% 14% 29% 0 57% Percentage of facility variables with significant negative changes (# variables/7) 0 0 0 14% 0 0 0 receiving DPT1 vaccine”—had a negative result and for only one health zone, Wembo Nyama. Quality of services results Variables from the Quality of Services were derived from the facility survey. For the most part, there was more heterogeneity among the variables presented here. Half of the variables presented here showed no significant changes between baseline and endline when examining the data disaggregated by HZ. These four variables encompass the FOSACOF scores (aggregate and hospital), training scores above 85%, and reduced delays for receiving medications. The variable “Continuous water supply (in the health facility)” was significantly positive in 3/7 zones, 44 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Table 16, continued Bibanga Nundu (1) Kanzenze Wembo Nyama(1) Luiza (2) Kayamba Lomela (2) Quality of services (9 variables) FOSACOF Scores Hospital FOSACOF Scores Received >85% training score in the past three years Reduction in delays for medications (NB: ▼ indicates improvement here) ▼ Availability of oral contraceptives ▼ Availability of Depo-Provera ▼ ▼ Continuous electricity supply ▼ ▼ Continuous water supply Access to transportation ▼ ▼ Percentage of quality variables with significant positive changes (# variables/9) 0 33% 0 33% 11% 0 11% Percentage of quality variables with significant negative changes (# variables/9) 0 0 0 0 0 11% 0 Bibanga (3) Nundu Kanzenze Wembo Nyama(1) Luiza Kayamba Lomela (2) Totals (22 variables) Percentage of all variables with significant positive changes (# variables/22) 23% 18% 14% 41% 18% 5% 32% Percentage of all variables with significant negative changes (# variables/22) 0 5% 5% 9% 9% 9% 5% Percentage of all variables with no significant change (# variables/22) 59% 59% 59% 50% 50% 55% 55% Percentage of all variables with no significant change but an undesired trend ▼ (# variables/22) 18% 18% 23% 0 23% 32% 9% 100% 100% 100% 100% 100% 100% 100% Key: Positive changes with statistical significance (green), no statistically significant changes (grey), negative changes with statistical significance (red).▼on grey indicates no change between baseline and endline but demonstrates an undesired trend. Superscript number (x) indicates rank of HZ within each category. EVALUATION FINDINGS n 45 while “Continuous electrical supply” was significantly positive in 2/7 zones. Contraceptive availability showed significant positive changes in 2/7 zones for oral contraceptives and in 1/7 zones for Depo-Provera. Similar to the facility survey results above, only one HZ for one variable had a statistically negative result at baseline. This was for the “Availability of oral contraceptives” in Kayamba, where there was significantly less stock of this contraceptive method available at endline. Overview Heterogeneity is observed in close to 68% (15/22) of variables that showed significant results in the aggregate level. Heterogeneity is not observed among 32% (7/22) of supply side (health facility) variables. These included: the average number of fully vaccinated children; availability of growth and development monitoring; and four of the Quality of Service variables, namely FOSACOF scores, high training scores in the past three years (>85%), and reductions in the delays for medications. The common denominator is that no significant changes took place in any of the HZs; change was only observed at the aggregate level. An overview of these disaggregated results indicates that Wembo Nyama has the best performance in the household survey, with significant improvements in 5/6 (83%) of the variables. It also has significant improvements in nine of the total of 22 variables (41%), also putting it in first place among the seven zones. Like all the other zones except Bibanga, the negative results for Wembo Nyama were realized in 2/22 (9%) of the total variables. Nonetheless, in all of the variables presented herein, for 11/22 (50%), there were no changes. There is also no indication of any downward trend. Lomela had a more diverse outcome. It came in second place out of all the other HZs in the household survey with 2/6 (33%) positive outcomes; it was in first place in the facility survey with 4/7 (57%) positive outcomes and in second place in the quality of services category with 1/9 (11%). Among all 22 variables, Lomela only had one significant reduction at endline related to improved water source at the household level. Kayamba, by far, showed the least success in achieving positive results. It had only one significant positive result across all 22 variables—improved water source at the household level. It also faired the worst in the number of variables which were worse at endline. It had 2/22 (9%) significant negative results and was the only one with negative significant results in the quality-of￾service survey (availability of oral contraceptives). Furthermore it was the HZ that showed the most variables 7/22 (32%) with undesired trend. Of the four remaining zones, Nundu and Luiza had 4/22 (18%) significant positive results, Kanzenze had 3/22 (14%) positive results, and Bibanga had 5/22 (23%) positive results. Bibanga was also the only health zone with no significant negative results, 0/22 (0%). When reviewing all of the variables without significant changes (grey colored), there is very little variability; across the seven HZs, the rates ranged from 50%–59%. Similarly, with just two exceptions (Wembo Nyama (0%) and Lomela (9%)), there was little variability among the other five health zones in terms of the total percentage of all variables, with no significant change but an undesired trend. Among those five zones, the range was 18–32%. Not surprisingly, the two outliers here are also the two top-performing zones, Wembo Nyama and Lomela. RBF scores analysis The analysis presented here is a comparison of RBF scores for the health centers in four intervention health zones visited during the RBF Midterm Assessment. Data from the RBF web portal was downloaded for Quarter 2, 2014 and Quarter 2, 2015 to observe changes in the RBF scores over a one-year period. Figure 33 on the following page shows that there are large variations (spread of bubbles and bubble size) in performance (RBF scores) between health facilities within HZs. Overall, the RBF scores decreased over time between Q2, 2014 and Q2, 2015 for almost all HCs in all HZs. The least variation is seen in Wembo Nyama and most variation is seen in Bibanga (Figure 33, next page). The following are the twelve service delivery indicators measured in the RBF Midterm Assessment: 1. Rate of use of curative care services at HC 2. Proportion of high-risk pregnancy referrals 3. Coverage rate for pentavalent vaccine 4. Proportion of pregnant women receiving TT2+ 5. Number of clients receiving family planning counseling 6. Rate of assisted deliveries 7. TB case notification rate/New sputum positive smears 8. Rate of ITNs distribution/Number of ITNs distributed 9. Rate of ANC1 service utilization 10. Number of clients receiving counseling and voluntary HIV screenings 11. Number of pregnant women tested for HIV 12. Rate of ANC4 coverage 46 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Figure 33. RBF scores (Q2 2014 & Q2 2015) for HCs visited during the RBF Midterm Assessment 100% 80% 60% 40% 20% 0 20% 40% 60% 80% 100% Aggregate of 12 service delivery indicators FOSACOF scores RBF SCORE Q2 2014 RBF SCORE Q2 2015 Cikuyi, 88% Polyvalence, 84% Katanda 1, 68% Bibanga, 17% Polyvalence, 96% Bibanga, 75% Katanda 1, 68% Cikuyi, 81% 120% 100% 80% 60% 40% 0 20% 40% 60% 80% 100% Aggregate of 12 service delivery indicators FOSACOF scores RBF SCORE Q2 2014 RBF SCORE Q2 2015 Otohe, 120% Tshekopoto, 107% Tshekopoto, 120% Olota, 97% Otohe, 93% Alhamba 111% Olota, 95% Alhamba, 85% 120% 100% 80% 60% 40% 0 20% 40% 60% 80% 100% Aggregate of 12 service delivery indicators FOSACOF scores RBF SCORE Q2 2014 RBF SCORE Q2 2015 Kamoa, 93% Kantala, 74% Kamoa, 111% Mpala, 115% Kantala, 103% Kamimbi, 116% Mpala, 80% Kamimbi, 75% 120% 100% 80% 60% 40% 0 20% 40% 60% 80% 100% Aggregate of 12 service delivery indicators FOSACOF scores RBF SCORE Q2 2014 RBF SCORE Q2 2015 Kamushilu, 95% Kamayi, 115% Kabuanga, 110% Kamushilu, 120% Kamayi, 69% Kitoko, 120% Kitoko, 100% Kabuanga, 88% Bibonga Wembo Nyama Kanzenze Luiza Source: RBF web portal, n=4 Analytical Domain: Contextual Factors Contributed to or Limited the Desired Results The data from the RBF Midterm Assessment Report,11 additional key informant interviews conducted in Feb 2016, and document review formed the basis of findings described under the analytical domain of contextual factors affecting RBF results. Data review from the midterm quality assessment highlighted that there was buy-in from various stakeholders for the IHP’s pilot RBF intervention. This helped in the successful implementation and progression of pilot activities in the targeted HZs. Based on a review of IHP’s RBF Manual (August 2014) and MSP’s RBF operations guide (October 2012), IHP RBF’s design was compliant with the MSP’s policies and directives concerning the underlying principles, the implementing entities 11. The midterm study design was a non-experimental descriptive process assessment using predominantly qualitative data collection and analysis methods. A modest sample of 44 key informants was interviewed from across four out of seven pilot health zones representing USAID, MSP, IHP, and chief nurses and directors of GRHs. A total of 20 focus group discussions were conducted with the members of CODESAs, CSOs contracted by MSH to do counter verifications at the household level, and TBAs in the villages. and their roles, and the levels of contractualization, as well as the entities that executed a RBF program. There were, however, a few bottlenecks impeding RBF’s progress. Due to the lack of formal training of new facility staff, there were gaps between the reported data and the validated data. A lack of sufficient full-time dedicated staff at central and coordination office levels contributed to insufficient monitoring and some reasonable delays in the payment of incentives. This weakness was corrected immediately by MSH based on the recommendations provided after the RBF Midterm Assessment. The coverage of households counter-verified remained less in remote areas. CSOs were not directly involved in health facility improvement plans. CSOs were physically far from the health facilities they audited and did not have any interventions in the surrounding community. The creation of champion communities by IHP, to act as counter-verification agencies, was not fully implemented. EVALUATION FINDINGS n 47 Before RBF, most service providers relied heavily on user fees to cover the operating costs of the facilities as well as to pay bonuses, or “primes,” to staff. In facilities, bonuses or “primes” were low for those who received them. Many staff were not even on the civil service payroll, deriving their remuneration solely from fees charged to patients. In the facilities visited by the RBF Midterm Assessment team, less than ten percent (10%) of staff members received salaries. RBF incentives provided the means for subsistence besides user fees charged from clients. A review of IHP’s and MSP’s reports highlighted a wide range of environmental factors that may have an effect on RBF implementation. Such factors include civil and political unrest in South Kivu and poor geographic accessibility in certain HZs, such as Lomela and Kayamba. Cultural factors impeding repeat antenatal, vaccination and postnatal care visits were mentioned by 40% of key informants conducted in 2016. The lack of paved roads and transportation resulted in widespread medicine and supply chain breakdowns and thereby limited the population’s access to primary health care and referral services. Electricity supply (by solar panel at best) was rare and piped running water was non-existent in most health areas visited. Chief nurse respondents in all HZs reported difficulties in maintaining cold-chains for vaccines/medications, safe medical waste management, and infection control practices. The transfer of funds was done manually, since banking facilities were not available in the rural locations. Plans called for the funds to be paid into the bank accounts of the contracting entities, but at present, very few health facilities have bank accounts. Analytical Domain: Unintended Consequences of the Intervention The data from the RBF Midterm Assessment Report, and document review, formed the basis of findings described under the analytical domain of unintended consequences. The data were triangulated with quantitative data from the endline surveys. Positive consequences 1. Accountability and transparency at operational levels were promoted. Upon review of the IHP RBF Manual, it is evident that accountability and transparency are built in at each operational level of the RBF intervention. In order to pay for performance, there needs to be a standard for performance to be measured, verified, counter-verified, and validated to ensure that only true performance is compensated. The process of counter￾verification by CSOs involves a sample of facility clients, randomly selected from the different facility registers by IHP, tracked and interviewed at their respective homes. The CSOs then compiled data, analyzed it, and transmitted a report on the community verification activity to IHP. IHP also introduced fictional or “ghost” patient records (20% of the total sample) into the community level data collection plan, as a method of preventing fraud or falsified data from the CSOs. All chief nurses reported that they did not have any direct role during counter￾verifications, and that they only receive reports from IHP at the end of each quarter. In the event of data discrepancies, penalties were applied. The approach to sanctions was not rigid, however, but rather treated on a case-by-case basis. On probing, none of the respondents alluded to any unintended negative effects of RBF related to gaming, distortion, or cherry-picking. 2. Concepts of work planning, business planning, target setting, and technical verification were introduced at the lowest operational level. This was non-existent prior to the pilot RBF. Through implementation of the pilot RBF, IHP introduced concepts of target setting, business planning or work planning, and technical verification. All chief nurses and ECZ managers interviewed reported using IHP RBF tools. To improve performance in the HZ and health facilities, the responsible HZ manager conducted regular facility supervision. This finding is corroborated by almost 100% facility supervision rates by MSP staff seen during endline health facility surveys in the intervention HZs (a significant improvement compared to the comparison HZs). Technical verification of the reported health facility data were performed through on-site facility visits every quarter. IHP played a capacity-building supporting role in this supervision. The HZ manager worked alongside IHP staff as the technical verification team. The team verified that the information reported by the facility corresponded with the information contained in the facility registers. The team also monitored the services being provided using FOSACOF tool. Negative consequences 1. Issue of equity was not addressed. By design, IHP’s pilot RBF model does not address the differences in socioeconomic status of the target populations, the type of organization, or geographic variations. For example, the cost of living in Katanga province is much higher than in Kasaï, yet the same amounts of funds are allocated equally among all HCs and hospitals. In Bibanga, even though HIV and AIDS and TB activities are very limited, the HCs were still required 48 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO to report on these indicators every quarter. The GRH incentive payment was significantly higher than the HCs (maximum $12,000 vs. $910 per quarter) and based only on FOSACOF scores, not service delivery indicators as was required of the HCs. 2. Evidence of dissent at the provincial and national level over non-inclusion in RBF contracts existed. With reference to those who implemented RBF, the MSP’s operations guide looked at the entire health pyramid— beginning with the central level and continuing out to the peripheral level—while the IHP pilot RBF focused only on the operational level (health facilities and ECZs). All central MSP RBF unit respondents expressed their desire to participate in the RBF contracts and their commitment to the success of RBF in the DRC. All Division Provinciale de la Santé/Provincial Division of Health (DPS) and Médecin Inspecteur Provincial/Provincial Medical Inspector (MIP) staff members interviewed were highly supportive of the IHP RBF model. They want the model to be expanded to all IHP-supported HZs and to all three levels of the health system pyramid, including provincial and district levels. More than sixty percent (60%) of the national-level respondents noted that the current level of national budget spending on health is insufficient and that additional funding resources will be needed for RBF. EVALUATION FINDINGS n 49 DISCUSSION Interpretations of RBF Successes and Challenges The pilot RBF aimed for a rapid scale-up of health services (quantity) and improved quality of care at selected health facilities. This section presents a discussion on which health activities achieved desired results, which did not, and why. The combined interventions of IHP and RBF had a synergistic effect on the availability (quantity) and delivery of quality services at the HCs. RBF led to high service utilization among mothers in making a first visit for antenatal care, in getting at least one tetanus toxoid injection during antenatal consultations, and in choosing institutional births. Although not attributable to RBF, there was a significant reduction in women receiving at least four antenatal consultations, in getting at least two tetanus toxoid injections during antenatal consultations, and receiving postnatal care. These mixed results seem to point to a lack of a uniform or effective approach to maternal and child health visits during the critical time period surrounding childbirth. This might also be a reflection of the knowledge and/or motivation of providers to promote and offer these services. Evidence from this study indicates that the RBF pilot intervention is contributing to improving structural inputs, health-seeking behaviors, and to a lesser extent, affecting client perceptions of the quality of care. However, while IHP writ large is influencing treatment outcomes for diarrheal disease, RBF is not having an additive effect, as the comparison group also improved in this realm. Yet, RBF also had impact on the increasing the likelihood that a child was tested with a blood draw for malaria. Unfortunately, this did not translate into better treatment with anti-malarials (in either group), likely due to a shortage in those medications at the facility level. Similarly, the treatment for children with presumed pneumonia receiving antibiotics did not show any improvement. Appropriate care with medications or antibiotics significantly reduced in both groups. Juxtaposed with other data, the situation seems to point to problems related to supply chain management of essential commodities. The infection control practices of the intervention health centers improved significantly and is attributable to the RBF activities. Similarly, there was a significant improvement in the use of most effective infection control methods such as autoclave and disin￾fectants, and for medical waste management (e.g. incinerators) in the health facilities. While not measured directly, nosocomial infections could potentially be on the decrease due to the increased access to continuous water supply in facilities (and improved water supply in the homes); better forms of incineration of biomedical wastes is happening through a shift from the burial method to the use of incinerators, and pit latrines are improved (as was the case in the IHP performance evaluation), so RBF potentiated this effect. RBF had a significant impact on increasing the number of curative visits and referral for obstetrics cases to a higher facility. Although the DID analysis did not show RBF impact on the average number of children receiving the DPT1 vaccine, the average number of FP consultations, and the average number of sick children receiving curative care, still RBF appears to be a factor for improving these service indicators. The duration of the pilot (two years) may have been too short to measure statistically significant changes in these indicators. The intervention HCs seemed to do better over time, suggesting that if the pilot had continued, more significant results might have been achieved. Follow-up measurements in a year or two might show an impact on DID analysis. The low coverage of some indicators could also be the lack of demand-side incentives to the community. Another factor could be related to geographic reach, cultural issues, and economic factors. According to the key informant interviews of February 2016, cultural factors impede access to maternal and child health services. Increases in use of modern FP methods were significant, particularly in the intervention group. There was also a significant gain in family planning counseling services. The increase was associated with the HC nurse, who gave advice to 75% of the cases, followed by the CHW (9%). Again, this points to the importance of these locally-based cadres in increasing coverage. And while advice on FP significantly increased in both groups, the distribution of the use of modern 50 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO contraceptives showed a predominance of male condom use, with a very low use of long-lasting methods. This may in part be due to problems related to a significant reduction of oral contraceptives in Kayamba and to non-significant reductions in availability of pills and Depo-Provera in two zones. Improved supplies and continued training of health professionals (in line with the Tiahrt Amendment, as specifically mentioned in the RBF Manual) could improve contraceptive choice. Cordaid/ HealthNet PBF pilot projects in Kasaï province showed a considerable increase (three to four fold) in FP utilization due to RBF interventions, holding promise for similar results. The malaria service indicators showed a high availability of promotional material and a high proportion of children sleeping under bed nets. Malaria testing increased slightly in the implementation zones. The availability of insecticide-treated bed nets and treatment of malaria with antimalarial declined, again indicative of supply chain problems. Although the FOSACOF instrument is cumbersome and time￾consuming to complete, it provides an effective mechanism to track significant changes and to stimulate health professionals to review an array of inputs that are proximate determinants of quality of care. Some of these aspects, such as those related to improvement of infrastructure, may no longer have to be reviewed in such detail once the situation has been resolved through facility renovation support; the points assigned to this might need to be shifted to promote other interventions. As future activities become better established and face new challenges of finding more resources for payment of services, this and other verification instruments and processes may need to be modified or simplified. A review of client satisfaction shows approximately 89% of mothers expressed satisfaction with the overall quality of services received, with a significant increase in the comparison group and no change in the intervention group at endline. The perception of the amount of time the service provider spent per client did not change for the intervention group, but significantly improved for the comparison group. The perception of interpersonal skills (a nurse listening attentively to the client and treating them professionally) improved significantly for both groups. This points again to the importance in strengthening the support for this resource and other field-level cadres of the health system to achieve significant results. This becomes particularly relevant when reports from the RBF Midterm Assessment indicate that many of the health staff rely primarily on RBF incentives as their salary base. Health promotion services and availability of promotion materials, which for the most part have achieved significant results may nonetheless need to be revisited to achieve impact. Increase of perinatal care, receiving tetanus toxoid vaccine, and increased examination by a skilled health worker after childbirth were significantly associated with contact to the nurse at the HC and, in some cases, to the CHW. Thus, the RBF model affords an opportunity to further involve and support these health cadres, and in particular nurses, to achieve the desired impact. Triangulation of data sources implies that increased knowledge of healthy practices is increasing demand. This is substantiated by the increased total costs for services—which is not, however, reflective of increased cost for services, since fees for services remained constant. This implies that during the last visit to a health facility, mothers were getting more care. It is unknown what the package of services were they were procuring, but since the total cost of the last visit was significantly higher, it implies that more services were purchased. It is possible that the mothers brought other children in for care, or sought care for themselves as well, based on new knowledge from a health care worker since contact with a health worker in the previous month was a common explanatory variable for significant improvements attributed to RBF. The fact that curative visits were significantly increased in the intervention group corroborates this supposition. With regard to facility operations and management, RBF facilities reported higher availability of medications and contraceptives (decreased stockouts) than the comparison group. Availability of supply of water (rain water cistern or improved well), electricity (solar panels), and means of transportation (bicycles) improved in RBF health zones. This difference may be due to higher autonomy to use incentives for facility investments in the form of renovation of health facilities and ensuring medication supply through consistent follow up with drug distribution centers. Another factor could be motivated health facility staff and HZ managers. When salaries and personnel benefits have not been provided regularly by the MSP, RBF became a motivational element that kept the nurses and CHWs inside the facilities to promote and provide the delivery of available services. Results measurement, regular reporting, and verification itself brings about changes in service providers to keep their health facility functional and receive highest incentive possible under their contract. Staff supervision improved significantly in the RBF group, with 99% compliance with quarterly visits. These visits were heavily supported by the IHP project, who accompanied the MSP supervisors on those visits and provided transport and fuel. Of course, the only way the MSP staff at any level could receive their RBF incentive payment was to convene these quarterly supervision visits. Whether this consistency in supervision would continue in the absence of external support remains to be seen. Nonetheless, it is the crux of the success of the RBF intervention. As gleaned during key informant interviews during the 2014 RBF Midterm Assessment, these supervision visits allow for on-the-job training using actual cases. The nature EVALUATION FINDINGS n 51 of the technical verification visit allows for the review of a series of recent patient visits and provides an excellent opportunity for the supervisor to help the provider understand best practices, by examining how the patient was treated and then referring to the national protocols, essentially walking the provider through a better diagnostic and treatment modality. This type of supportive supervision is in the MSP supervisor’s best interest, since his/ her RBF incentive payment is integrally linked to their successful completion of their RBF indicators and hence, incentive payment. To summarize, the RBF intervention was helpful and led to significant improvements in the quantity and quality of certain services. However, the changes introduced are not uniform and there are large variations across HZs and MPA service components. The technical verification of the model affords an excellent opportunity to improve the quality of care through continuous on-the-job training through supportive supervision. Equity of RBF Incentive Payments The RBF Midterm Assessment raised the issue of equity in regard to the RBF incentive payments being equal for all health facilities, without taking into consideration the cost of living from one locale to another. This issue remains and goes beyond intra-facility equity, which is another aspect of equity that warrants attention. There was no evidence presented to determine if the ceiling on the quarterly RBF incentive payments of $12,000 USD per GRH and $910 USD per HC are accurate or fair. In 2005, the health authority of South Kivu, the international NGO Cordaid, and the local NGO Bureau des Oeuvres Médicales started a RBF scheme in two districts covering together a population of 300,000 in order to improve preventive care as well as tuberculosis and HIV and AIDS testing and care (Soeters, Peerenboom, Mushagalusa, & Kimanuka, 2011). Based on a performance contract, health facilities received a fixed amount per targeted action per month, plus a bonus of up to 15 per cent for quality. Remote facilities received an additional 15 per cent. The cost was US $2.40 per capita (Witter et al. 2012). In the World Bank RBF model in the DRC, facilities are treated at the individual level and receive a preestablished price tailored to the type of service rendered. For example, for each fully vaccinated child, the facility would receive $5.00; for each skilled birth attended, the purchaser pays $10.00, for each curative care visit for children under 5 years, the payment is $0.50, and so on. In addition, the total amount for volume of services is adjusted according to the remoteness of the facility (equity bonus = +50%), as urban or peri-urban facilities could otherwise earn a disproportionate amount. The total payment would also be adjusted by a quality correction (+60%) based on a checklist administered at the facility every quarter. Thus, the total incentive payment to each facility is based on the quantity of services they produce (RBF website, 2016). Another aspect surrounding the issue of equity is the disbursement of the incentive payments each quarter within the health facilities. In a health center, there are few staff and all are aware when the HC has received a bonus (as revealed in the key informant interviews in the RBF Midterm Assessment). At the HC level, there is an elaborate Index Tool used to calculate how much money each staff member will receive, based on clearly specified criteria (rank, performance, and attendance). While the funds are to be disbursed in the presence of the CODESA, this was not the actual practice in some settings, according to key informants. On the other hand, GRHs are comprised of numerous staff, and since equity was not relevant to the evaluation questions posited in this study, no data were collected to determine the staff inclusion criteria to be eligible for an RBF payment within a hospital. Measuring Quality of Care Measurement of quality of care is a topic that has eluded global health professionals for some time. Donabadian was one of the forefathers of measuring quality and his model primarily looks at structural inputs and processes (Donabedian, 1988). Indeed, many subsequent models of measuring quality of care have built upon these readily measured variables (Adindu, 2010) and expanded beyond to include program effects, services including interpersonal relationships, and client knowledge and satisfaction (Bruce, 1990). However, truly understanding the quality of care given is an expensive endeavor, as it requires the use of “mystery clients” or direct observation by a clinical provider (nurse or doctor). As well, in rural areas, the patient load is low and would require more time spent at the facilities to garner a sufficient sample size. In settings such as the DRC, this is cost-prohibitive. Given this backdrop, mothers recent recall was used as a proxy for measuring treatment outcomes. Women were asked about any child illnesses in the two weeks prior to the interview to minimize recall bias as much as possible and still have a 52 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO reasonable sample size. Three illnesses were queried: diarrhea, presumed pneumonia, and malaria. The women were then asked about their health-seeking behaviors and their interpretation of the treatment their child received at the health facility (for those who sought care at a health facility). Clearly, a woman’s recall of care is not the gold standard, but it was a viable alternative, since direct observation of care was not an option. Yet, to build a more robust model, this study expanded the measurement of quality to include health-seeking behaviors that are a composite of knowledge and practices—and also added the end users’ perception of the quality of care to the model. The later was a composite of numerous questions on care, ranging from the cost, to wait time and beyond, giving ample opportunities for the woman to be critical, as traditional client satisfaction surveys consistently yield high scores. While there is ample evidence that client satisfaction is a poor proxy in and of itself for measuring quality, there is some evidence that clients who perceive a higher quality of care are more likely to utilize services (Baltussen, Ye, Haddad & Sauerborn, 2002). And while utilization is not directly linked to quality, clients need to be physically present in the facilities before they have a chance at receiving quality care. Another component of this model is our expansion of structural inputs to include basic sanitation and utilities (electricity, waste removal, latrines, etc.), which the evaluation team submits is a proxy for infection control. Studies have shown that time spent walking for water has significant effect on improving child health. Analyses of Demographic and Health Survey data demonstrated that for every 15-minute reduction in time spent walking (each way) to gather water: a) 41% decrease in the incidence of diarrheal disease; b) significant improvements in anthropometric measurements of nutritional status; and c) an 11% reduction in childhood mortality (Pickering & Davis, 2012). If a nurse has access to electricity and therefore good lighting in the dark, it stands to reason s/he is more likely to be able to keep a sterile field when suturing, for example. Clearly, incinerators are a safer means of disposal of bio-hazardous waste. These and other basic utilities, which are a luxury in a rural health center in the DRC, are likely to lead to reductions in nosocomial infections and therefore, improve the quality of care. Further analysis of the relationships between the three factors (structural inputs, health-seeking behaviors, and client satisfaction) and how they impact treatment outcomes would have allowed the evaluation team to test the strength of the model, however, this was not within the scope of this study. DISCUSSION n 53 CONCLUSIONS EVALUATION QUESTION 1: Is there evidence of change among health centers in the quantity and quality of services that is attributable to the RBF model? RBF had a positive impact on quantity and quality of services delivered in health centers. ■ Curative service rates are increased, including births attended in facilities, high-risk pregnancies referred, and malaria diagnosis with a blood draw. ■ Preventive services improved (or did not decline as in the comparison group), including first antenatal visit, one dose of TT, growth monitoring, vitamin A administration (mother and child), and modern contraceptive use. ■ Interventions promoting water supply appear to be effective at both the household and facility level. ■ Recent contact with a nurse is an explanatory variable for several positive health-seeking behaviors, such as use of ORS for childhood diarrhea, using bed net for malaria prevention and getting child tested for malaria. Contact with a health provider was more frequent in the health center and most people are already using the formal health system. The positive health-seeking behavior does not reflect increased outreach by the health workers. ■ Given that prices for services remained stable between baseline and endline and yet women paid more for services during their last visit to a health facility, this implies women are obtaining more services per visit than in the past. This is corroborated by the increase number of curative and preventative visits. The incentivized services at the RBF HCs showed significant increase from baseline and also in relation to the comparison facilities. Such services include: outpatient curative consultations (new and old cases), antenatal consultations, institutional deliveries, obstetric referral, modern contraceptive use, and tetanus toxoid vaccination. Similarly, FOSACOF scores at HCs and GRH level improved, which indicate the inputs for providing quality care are better in the RBF group. RBF activity did not have a negative effect on the availability or quality of non-targeted services. For example, improved water source at the household level and continuous water supply in the facility (a non-incentivized service) increased significantly in RBF health zones. None of the MPA service indicators declined in terms of either quantity or quality due to RBF activities. Hence, there is no indication of any issues with distortion (taking attention away from non-incentivized services, which is a potential risk of RBF approaches). On the contrary, there is sufficient evidence that non-incentivized services improved; this may be because incentivized targets are potentiated. As FOSACOF was the only RBF-incentivized indicator at GRHs and it does not include measurements of the quantity or quality of services, the conclusion above is limited to the health centers. No conclusion can be drawn regarding the RBF impact on hospital services. Moreover, this evaluation question is directed to health centers only. ■ Children are not being routinely treated for malaria upon diagnosis in the health facilities, possibly due to the lack of availability of antimalarial medications in the health facilities. Malaria treatment appears to be a systematic problem across all HZs. Even in the two zones without a statistically significant change for this variable (Lomela and Bibanga), their results are trending downward. Since the variable “child with malaria had blood drawn” had significant positive results in two HZs and non-significant positive results in three HZs, these negative treatment findings appear to point to a lack of provision of antimalarial drugs rather than to service delivery, where blood is drawn. If the problem were due to lack of compliance by the providers, one would expect the children not to be tested at all. This conclusion is supported by facility survey results (more than 80% of facilities in both groups report delays in medication supplies), indicating supplies of antimalarial are down. ■ Interventions promoting water supply appear to be effective at both the household and facility level. Improved water at the household level is matched by continuous water supply at the facility level with just one out-lier. Only 54 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Lomela had a significant decrease in improved water source at the household level. Water source is increasing at both the household and the health facility levels, indicating that RBF’s WASH interventions are having an impact across all HZs. EVALUATION QUESTION 2: What difference did the RBF intervention make? ■ RBF induced behavior change among health center staff and health zone managers, which in turn translated into increased demand and utilization of services by the catchment population. ■ Community or client contact with an HC nurse improved because of the RBF intervention and was an important factor influencing the community’s positive health-seeking behaviors. The financial incentives of the RBF payment mechanism in the intervention group led to an intensification of activities used to increase demand and utilization of health services. Facility staff and MSP managers understood that the payment mechanism was linked to the quantity and quality of services provided at the HC level. IHP interventions were implemented across the two groups and these interventions were introduced in September 2010, prior to the start of the pilot in November 2013. Using a DID analysis, the impact evaluation was able to measure the RBF effect on health facility outcomes and health-seeking behavior of its catchment population. Hence, there is empirical evidence that the RBF intervention made a difference in the quantity and quality of services, as noted under Evaluation Question 1 above. Basic structural inputs such as continuous electrical and water supplies, regular access to transportation, infection control measures in situ such as autoclaves and pressurized steamers, and better handling of bio-medical wastes, including switching from burial methods to incinerators, are all fundamental to the operations of a health facility. While the RBF did not impact all interventions at this nascent stage, it is evident that emphasis has been on the appropriate priorities to provide basic services and referrals. EVALUATION QUESTION 3: Is the model worthy of being scaled up to other health zones? ■ The IHP RBF model at the HC level is worthy of scale-up. The pilot is well designed at the level of HCs and can strengthen core health system functions, increasing value for money and accountability of the health system. ■ The RBF scale-up of service delivery indicators at GRHs was introduced after the data collection period of this evaluation and needs to be evaluated for an in-depth understanding of its effectiveness before the model is scaled up at the hospital level. ■ There is a “buy-in” from the MSP country stakeholders at all levels of the health system for the expansion of IHP RBF model. Service providers have a strong desire for the RBF program to continue. RBF requires significant investment of time and money— and careful implementation, especially in supervision and coordination. A key element of the design is the separation of functions between the purchaser of the services and the verifier of services. For the pilot RBF, USAID and MSH had the roles of purchaser and verifier (data validation before payments were dispersed at the operational level). Another major strength of the MSH RBF model is the robust verification process, both at the facility and the community level. While the RBF Midterm Assessment revealed some weaknesses in the administration of the community verification process, it is an essential component to diminish gaming and collusion. The rigorous data quality control methods at the central level to validate performance galvanizes the integrity of the RBF model. Evidence from the evaluation suggest that, although there is a buy-in from the MSP country stakeholders at all levels for the expansion of IHP RBF model, the current capacity of MSP to function independently as a purchaser or verifier of services is not yet fully developed. EVALUATION QUESTION 4: What costs are associated with a potential replication of the model? ■ The total budget associated with RBF implementation covering seven health zones for a two-year duration was $6,304,181, with $1,120,168 expended during the preparatory phase and $5,184,013 during the implementation phase. ■ Of the total executed budget of $5,184,013, 24% of that distribution was for human resources, 2% for investments (e.g., vehicles and equipment), 33% for service purchases (incentive payments), 15% for technical verification, 11% for community verification, 10% for regulatory improvements, and 5% for operating costs. ■ The budget used in this pay-for-performance scheme was estimated at $2.70 per capita per year, which is typical in similar contexts where output budgets range between $2 and $3 per capita per year. The preparatory phase, lasting more than two years, accounted for almost 18% of the total budget. The expansion or replication DISCUSSION n 55 of this program would benefit from the experience acquired and the materials and tools already prepared, thereby reducing the total costs by almost one-fifth. Human resources accounted for about one-quarter (24%) of the expended budget (not including the preparation costs). Expansion or replication of this program and coordination with other RBF activities, whether funded by the government, donor agencies, or non-governmental organizations, would benefit from economies of scale with potential reduction of costs, particularly at the central level. The budget distribution for service purchases (RBF performance payments given to the provincial health office, the ECZs, and health facilities) was 33%. This appears to be low when compared to results in the RBF retrospective undertaken by Cordaid/HealthNet experiences in RBF pilot projects in the DRC, Tanzania, Zambia, Burundi, and Rwanda, where management inputs represent 30% of the total costs on average (DRC was actually higher than 40%) of the project, with 70% going to the performance payments. Notwithstanding, a comparative analysis with other RBF programs was not undertaken and there is not enough data on RBF programs to judge whether 33% or 70% is a good benchmark. ■ There is a large discrepancy between the maximum RBF incentives possible for a hospital each quarter ($12,000) and a health center ($910). This discrepancy is further compounded by the fact that hospitals have only one indicator—FOSACOF—and do not report out on any services delivered. This is significant, given the goal of IHP is to improve the quantity and quality of services. EVALUATION QUESTION 5: Were the desired results achieved? ■ Desired results of improved quantity and quality of services were achieved. While successes were not achieved across all indicators, there was considerable improvement across a host of indices. This is most impressive given the very short implementation period assessed (1.5 years of RBF implementation) and the difficulties faced in the in the DRC environment. ■ Sizable gains in some key coverage indicators were achieved through the introduction of RBF. However, the changes introduced are not uniform and there are large variations across HZs and MPA service components. All pregnancies have the potential to be high-risk and often times the signs and symptoms do not present themselves until the time of labor and delivery. Given the lack of roads and transport in the rural areas in DRC, waiting until labor starts may be too late to refer a woman to a higher level of care, which is often an uncomfortable three- to five-day journey by foot, bicycle, or at best, a motorcycle. The maternal mortality ratio is 846 per 100,000 live births (well above the Millennium Development Goals target of 250 per 100,000) (Ministry of Monitoring, Planning and Implementation of the Modern Revolution; Ministry of Public Health; ICF International, 2014). Thus, referrals of presumed high-risk pregnancies is an important life-saving intervention on which RBF has had an impact. While trained attendants at birth are not rising, RBF improved the rate of referrals of high-risk emergency deliveries from HCs to a higher level facility. Several years ago, the MSP took a decision to invite the traditional birth attendants to work directly in the HCs. The rationale was not explored in this study, but one could hypothesize this rationale stemmed from the lack of female nurses and even fewer nurse midwives available to staff the HCs. Thus, in an HC, even though a woman giving birth with a TBA does not enjoy the benefits of having a skilled attendant at birth, there is a male nurse present who is aware of the delivery and is monitoring for signs and symptoms of complications during labor. Then, he can either treat or refer the patient to the GRH. This is less likely to happen if the woman is giving birth with a TBA in her home. This is substantiated by the increased rate referral rate for high-risk pregnancies, which is an indicator factoring into the RBF incentive payment. IHP WASH interventions are leading to improvements in water supply and the RBF pilot has significantly potentiated this effect. During the study period, health workers gave advice on the importance of WASH improved across both groups, but significantly more often because of the RBF intervention. RBF also had an impact on the incidence of improved water sources at the household level and increases in the availability of continuous water supplies at the HC level. The triangulation of these three indices provides some evidence to suggest the RBF pilot’s WASH interventions are working well on both the demand and supply side. However, since information on the storage of the water source was not examined, the quality of the water after collection is unknown; therefore, there is the potential for clean water to later become contaminated. 56 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO EVALUATION QUESTION 6: Do results differ for various groups (heterogeneity)? ■ Among the variables that demonstrate statistically significant changes at the aggregate level for the RBF intervention group, Wembo Nyama is clearly the highest performing health zone, followed closely by Lomela. Bibanga is more comparable to these higher performing zones, while Nundu, Kanzenze and Luiza have similar, albeit average, results. Kayamba is by far the poorest performing zone. The selected variables presented to examine heterogeneity were all statistically significant at an aggregate level for the RBF intervention group when compared with the comparison group. However, when those same variables were scrutinized at the HZ level within the RBF group, the heterogeneity between the seven zones is revealed. Wembo Nyama and Lomela contributed to almost half of the positive results. Kayamba’s only positive variable was improved water supply at the household level, whereas this variable was the only problem faced by Lomela. Lomela and Bibanga were the clear forerunners among the facility level variables, yet Wembo Nyama did not fare so well in this category, and was the only HZ demonstrating a problem with decreased children being vaccinated with DPT1 at endline. Among the Quality of Services category, there was only one negative change in one health zone: Kayamba had decreased availability of oral contraceptives at endline. EVALUATION QUESTION 7: What contextual factors contributed to or limited the desired results? ■ IHP’s RBF design was compliant with the MSP’s policies, which contributed to its successful implementation at HCs, GRHs, and ECZs and achieving desired results. IHP’s RBF design took into consideration the MSP’s RBF policy directives concerning the underlying principles, the implementing entities and their roles, and the levels of contractualization, as well as the entities that execute a RBF program. ■ IHP encountered a wide range of environmental factors that hindered RBF implementation and negatively influenced the results in a variety of ways. Notwithstanding, the design of the pilot RBF was feasible considering the difficult environmental conditions of DRC. Environmental factors were related to civil unrest and insecurity, geographic inaccessibility, and poor infrastructure. Indeed, the evaluation team was detained a few times by rebel groups during the baseline data collection. The nurse-to-population ratio is far below the WHO recommendation of at least 2 nurses per 1000 population. ■ The existing health system has serious resource management issues. There is an inadequate budget to provide regular (or any) salaries to health staff, support for regular facility supervision is lacking, and most of the facilities are not equipped to the standard level of infrastructure and equipment. EVALUATION QUESTION 8: What are the unintended consequences of the intervention? ■ RBF implementation clearly had positive unintended consequences. ■ Negative unintended consequences related to gaming, cherry-picking, and distortion were not found, but cannot be ruled out. ■ The opportunity to receive payment for their services motivated health providers to comply with RBF procedures and guidelines. RBF introduced concepts of quality of care, target setting, business planning, work planning, and technical verification. The pilot RBF introduced some technically complex attributes, such as (a) stringent measurements (e.g., FOSACOF) tied to performance payments, (b) business plans, (c) technical verification, (d) individual health worker performance evaluations using the index tool, (e) staff training for the introduction of RBF strategies and tools, and (f) community household surveys for counter verifications. IHP RBF, by design, does not address the aforementioned differences in target population socioeconomic status, type of organization, and geographic variations. The RBF design lacked demand-side incentives such as fee exemptions or waiver schemes financed through RBF for the benefit of the poor. On the supply side, the RBF design did not offer bonuses for remote environs. This could lead to unintended inequities. CONCLUSION n 57 RECOMMENDATIONS EVALUATION QUESTION 1: Is there evidence of change among health centers in the quantity and quality of services that is attributable to the RBF model? ■ Consider the Quality of Care Measurement Index presented herein to provide a more comprehensive and well-rounded means for measuring the quality of care rather than relying on the FOSACOF tool as a proxy for quality. ■ Promote and incentivize nursing outreach to the communities to educate the public on healthy behaviors and to seek care from trained providers. Further analysis of the relationships between the three factors (structural inputs, personnel, and health-seeking behaviors) and how they impact treatment outcomes would allow the evalua￾tion team to test the strength of the model; however, this was not within the scope of this study. Inclusion of measurements of health-seeking behaviors—whether for RBF reimbursement or not—will help to understand the demand side of care. Future evaluations of the RBF program should allow for a thorough as￾sessment of treatment outcomes at the facility level, if feasible, and if not, at the household level. Linking structural inputs, personnel, and health-seeking behaviors to these treatment outcomes will better indicate the quality of care given than by simply measuring individual indicators in a vertical, stove-piped, fashion. EVALUATION QUESTION 3: Is the model worthy of being scaled up to other health zones? ■ The RBF model at the health center level should be scaled up by USAID/DRC in a cascade fashion. However, the RBF model at the hospital level should first be solidified and the new model tested and evaluated before scale-up. RBF mechanisms and principles need to be integrated into the government financial system. Moving forward, the financial sustainability of successful RBF programs needs to be considered. USAID could consider harmonizing other donors’ funds as a part of a comprehensive financing strategy for the continuation of RBF in DRC. Continue payment for targets achieved for structural inputs for service delivery indicators at both the HC and GRH level. More attention is needed to target setting, so the criteria for their development is well defined and understood and the rates are appropriate. EVALUATION QUESTION 4: What costs are associated with a potential replication of the model? ■ An analysis of the total ceilings of RBF incentives deserves some thought. More research is needed to identify equity factors to establish ceilings of RBF incentives. Consideration of premiums to ensure equity should include the total catchment populations served, the remoteness and geographic distribution of the inhabitants, the types of services provided (e.g., laboratory, minor versus major surgery, blood transfusions, etc.), and the productivity and associated outcomes. Additional consideration should be given to the number and qualifications of the staff and the heterogeneity of the facilities with the norm among others. ■ Consider reallocation of the RBF ceiling payments between hospitals and health centers. Define the rationale for large discrepancy between the RBF incentive payments made to the seven hospitals ($623,921 or 12% of the budget) versus those made to the 118 health centers ($764,272 or 15% of the budget), which is just 3% more funds than the budget for GRHs.. Per capita costs in all intervention areas should be reviewed. Alternatively, reallocate the distribution of RBF funds between the GRHs and HCs based on an analysis of other RBF resource allocations at the facility level; consider a ratio that would account for more equity with adjustments for remote catchment populations and local cost of living. In addition, the GRHs should be held to the same rigorous expectation to meet service delivery targets as are HCs. RBF indicators for GRHs should reflect the appropriate functioning of operating suites, blood banks, and specializations, among other services, as expected in the MSP CPA- services. Infection control indicators, including nosocomial infections, also should be included. 58 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO ■ MSP should convene multi-disciplinary groups to develop a strategic plan (short-term and long-term) to address shortage of nurses in the health centers. MSH should record the experience and lessons learned during the extended preparatory phase should this process be repeated. These could be used to minimize financial and time investment. Once regulatory systems are set up, it is expected that these additional funds would be made available for the purchase of services. The MSP and USAID should, upon considering further RBF activities or when coordinating with other stakeholders, analyze whether further regulatory improvements are needed and whether funds that had been programmed for this line item could be made available for purchase of services, and in particular at the HC level. EVALUATION QUESTION 6: Do results differ for various groups (heterogeneity)? ■ The MSP RBF Unit should convene roundtable discussions with the ECZ and selected health facility staff from each of the seven health zones to analyze the RBF results achieved. Using a roundtable forum, stakeholders should take an opportunity to newly analyze and interpret the data collected, with the benefit of knowledge on the ground and the internal and external factors that affect the RBF pilot’s outcomes. This will give them a chance to adjust the implementation of activities, so that better results can be obtained across all HZs going forward. Table 16 in the Findings section could be a starting point of the discussion, to provoke a healthy debate as to why some HZs performed better than others selected variables. The venue, timing, and frequencies of these roundtables is up to the discretion of the MSP. There should be a special emphasis on reinforcing these significant positive achievements while examining the heterogeneity between the zones. Negative outcomes, whether significant or non-significant, should also be addressed. The fora should allow for a sharing of experiences and lessons learned, The “fishbone” or “Ishikawa” analysis is a particularly helpful tool to diagnosis the etiology of problems to tailor relevant solutions. They should also address the causes of the poor malaria treatment, so that relevant solutions can be devised. The scores also seem to corroborate the idea that Kayamba could be the health area that needs more support in achieving results. When considering these results, one must take into account that services such as electricity, water, and transportation, are often beyond the control of the health services. Multi-sectoral approaches, which involve participation of individual health facilities, are often necessary to achieve positive results. However, Wembo Nyama and other HZs did succeed in improving access to water and electricity in their facilities. It would be interesting to learn if the RBF incentive payments in those HZs were allocated towards providing these basic utilities. ■ Consider the advantages of employing a positive deviance approach to performance improvement and afford opportunities for Wembo Nyama and Lomela ECZs to showcase their best practices and lessons learned. Wembo Nyama is the zone with the best results, closely followed by Lomela. ECZ staff and health providers from these HZs could serve as mentors for other zones. Exchange sessions could be convened by either going to other zones or allowing visitors to come to those health zones to review best practices. Kayamba needs special attention to diagnose the source of their constraints and barriers, leading them to fall behind vis-à-vis the other HZs. Kayamba’s ECZ should consider the content and frequency of their supervision of the health facilities, on-the-job training and mentoring and, if pertinent, greater promotion of community involvement through CODESAs. ■ The malaria program should be involved in all zones, analyzing and solving the problem related to the treatment of children suspected of having malaria. This might warrant special attention to supply chain management of antimalarial medications. A broader discussion of supply chain management would also be beneficial in discussing why significant results have not been achieved, or why some results have had significant or non-significant decline. In addition to malaria, this might be considered for provision of oral rehydration salts, antibiotics for pneumonia, vaccines like DPT, delays in medication, oral contraceptives, and Depo-Provera. In Sum The global development community would benefit from a results-based approach that focuses on the outcomes achieved more so than the processes (Savedoff W., 2015). Governments and their constituents are best poised to determine their respective pathways leading to desired results. Indeed, the health system is embedded within a broader national framework, yet it is a place to start. Enjoying good health is not only a right but arguably a necessary ingredient to prosperity. In a very brief time period, this USAID-funded RBF program has demonstrated great promise. Each HZ and facility has taken its own pathway to achieve objectively measurable results. Of interest would be to examine the various and innovative approaches that originated at the health zone level which led to the successful results. REFERENCES n 59 BIBLIOGRAPHY Adindu, A. (2010). Assessing and assuring quality of care in Africa. African Journal of Medical Sciences, 3 (1), 31-36. African Development Bank. (2012). African Economic Outlook 2012: Democratic Republic of Congo. Retrieved from: http://www.afdb.org/fileadmin/uploads/afdb/Documents/Publications/Congo%20Democratic%20Republic%20 Full%20PDF%20Country%20Note.pdf. The AIDSTAR-Two Project. (2011). The PFB Handbook: Designing and Implementing Effective Performance-Based Financing Programs. Version 1.0. Cambridge: Management Sciences for Health. Baltussen, R., Ye, Y., Haddad, S., & Sauerborn, R. (2002). Perceived Quality of Care of Primary Health Care Services in Burkina Faso.Health Policy and Planning, 17 (1), 42-48. Basinga, P., Gertler, P.J., Binagwaho, A., Soucat, A.L., Sturdy, J., & Vermeersch, C.M. (2011). Effect on Maternal and Child Health Services in Rwanda of Payment to Primary Health-Care Providers for Performance; An Impact Evaluation. The Lancet, 1421-1428. Beane, C.R., Hobbs, S.H., & Thirumurthy, H. (2013). Exploring the Potential for Using Results-Based Financing to Address Non￾Communicable Diseases in Low- and Middle-Income Countries. BMC Public Health. Benitez, D., Foster, V. (2011). The Democratic Republic of Congo’s Infrastructure: A Continental Perspective. Washington, D.C.: The World Bank. Bertrand, J.T., Hardee, K., Magnani, R.J., & Angle, M.A. (1995). Access, Quality of Care and Medical Barriers in Family Planning Programs. International Family Planning Perspectives, 64-69 & 74. Bredenkamp, C., Borman, N.d., Mullen, P., Ostiguy, D., Sompwe, E., Wane, W., & Wangata, J. (2011). Dealing with Difficult Design Decisions: The Experience of an RBF Pilot Program in Haut-Katanga District of Democratic Republic of Congo. Washington, DC: The World Bank. Brenzel, L. (2009). Taking Stock: World Bank Experience With Results-Based Financing (RBF) For Health. Washington, DC: The World Bank. Brenzel, L., Costales, M., Durrheim, D., Gordon, S., Hyde, T., Kumar, R.,...Susan. (2010). Impact of New Vaccines Introduction on Immunization & Health Systems. Geneva: Strategic Advisory Group of Experts on Immunization. Bruce, J. (1990). Fundamental elements of quality of care: a simple framework. Studies in Family Planning, 21 (2), 61-91. Bucagu, M., Kagubare, J.M., Basinga, P., Ngabo, F., Timmons, B.K., & Lee, A.C. (2012). Impact of Health Systems Strengthening on Coverage of Maternal Health Services in Rwanda, 2000-2010: A Systematic Review. London: Reproductive Health Matters. 60 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Davies, H. (2006). Measuring and Reporting the Quality of Health Care: Issues and Evidence from the International Research Literature. Edinburgh: NHS Quality Improvement Scotland. Democratic Republic of Congo Ministry of Public Health Department of Primary Health Care Development. (2011). Flowcharts of Care at Health Centers. Kinshasa: Democratic Republic of Congo Ministry of Public Health. Democratic Republic of Congo Ministry of Public Health General Secretariat. (2010).National Health Development Plan (2011- 2015). Kinshasa: Democratic Republic of Congo Ministry of Public Health. Democratic Republic of Congo Ministry of Public Health General Secretariat. (2010). Strategy for the Reinforcement of the Public Health System: Second Edition. 2010: Democratic Republic of Congo Ministry of Public Health. Democratic Republic of Congo Ministry of Public Health RBF Unit. (2014). RBF Implementation Report for the Zones Supported by IHP: Quarter 1 and 2. Kinshasa: Democratic Republic of Congo Ministry of Public Health. Democratic Republic of Congo Ministry of Public Health Secretariat General. (2012). Operational Guide for Results Based Financing in DRC/Health Sector. Kinshasa: Democratic Republic of Congo Ministry of Public Health. Democratic Republic of Congo Ministry of Public Health Secretariat General. (2012). Results Based Financing Training Module for Community Assessment. Kinshasa: Democratic Republic of Congo Ministry of Public Health. Democratic Republic of Congo Ministry of Public Health Secretariat General. (2012). Results Based Financing Training Module for Health Facilities (Hospitals and Health Centers). Kinshasa: Democratic Republic of Congo Ministry of Public Health. Democratic Republic of Congo Ministry of Public Health Secretariat General. (2012). Results Based Financing Training Module for Members of the Central Health Bureau. Kishasa: Democratic Republic of Congo Ministry of Public Health. Democratic Republic of Congo Ministry of Public Health Secretariat General. (2012). Results Based Financing Training Module for Members of the Provincial Level. Kinshasa: Democratic Republic of Congo Ministry of Public Health. Democratic Republic of Congo Ministry of Public Health Secretariat General. (2014). Quality Assurance for Services and Health Care for the Population in the Democratic Republic of Congo. Kinshasa: Democratic Republic of Congo Ministry of Public Health. Development & Training Services, Inc.(dTS) & USAID. (2012). Gender Assessment for the Democratic Republic of Congo. Arlington: Development & Training Services, Inc.(dTS). Dijkzeul, D., & Lynch, C. (2006). Supporting Local Health Care in a Chronic Crisis: Management and Financing Approaches in the Eastern Democratic Republic of the Congo. Washington, DC: National Academy of Sciences. Dijkzeul, D., & Wakenge, C.I. (2010). Doing Good, But Looking Bad? Local Perceptions of Two Humanitarian Organizations in Eastern Democratic Republic of the Congo. Disasters, 1139-1170. Donabedian, A. (1988). The quality of care how can it be assessed? The Journal of the American Medical Association, 260 (12), 1743-1748. Eichler, R., & De, S. (2011). Paying for Performance in Health: A Guide to Developing the Blueprint: Version 2. Bethesda: Health Systems 20/20, Abt Associates Inc. Eygen, L.V., Lerberghe, V.V., Blaise, P., Woelk, G., & Criel, B. (2007). The Challenge of Measuring Quality of Care at Health Centre Level in Africa: The Example of Tsholotsho Health District in Matabeleland North, Zimbabwe. International Journal of Health Planning and Management, 63-89. REFERENCES n 61 Fritsche, G.B., Soeters, R., Meessen, B., Ndizeye, C., Bredenkamp, C., & Heteren, G.v. (2014). Performance-Based Financing Toolkit. Washington, DC: The World Bank. Gupta, S., Peters, D., & Vermeersch, C. (2012). Measurement and Indicators for Impact Evaluations of Supply-Side Results-Based Financing (RBF) for Health Programs. Washington, DC: The World Bank. Hecht, R., Batson, A., & Brenzel, L. (2004). Making Health Care Accountable: Why Performance-Based Funding of Health Services in Developing Countries is Getting More Attention. Finance & Development, 16-19. Huillery, E., & Seban, J. (2014). Pay-for-Performance, Motivation and Final Output in the Health Sector: Experimental Evidence from the Democratic Republic of Congo. Washington, DC: The World Bank. Institute for Health Metrics and Evaluation. (2013). Cause of Death, Democratic Republic of Congo, All Ages, Both Sexes [Data File]. Retrieved from http://vizhub.healthdata.org/gbd-compare/. International Development Association; International Monetary Fund. (2016).Heavily Indebted Poor Countries (HIPC) Initiative and Multilateral Debt Relief Initiative (MDRI) – Statistical Update. Ireland, M., Paul, E., & Dujardin, B. (2011). Can Performance-Based Financing Be Used to Reform Health Systems in Developing Countries? Bulletin of the World Health Organization, 695-698. Islamic Republic of Afghanistan Ministry of Public Health & USAID. (2006). Fully Functional Service Delivery Point (FFSDP) Manual. Kabul: Ministry of Public Health. Larson, E., Hermosilla, S., Kimweri, A., Mbaruku, G.M., & Kruk, M.E. (2014). Determinants of Perceived Quality of Obstetric Care in Rural Tanzania: A Cross-Sectional Study. BMC Health Services Research. Litvack, J.I., & Bodart, C. (1993). User Fees Plus Quality Equals Improved Access to Health Care: Results of a Field Experiment in Cameroon. Social Science & Medicine, 369-383. Manzi, A., Magge, H., Hedt-Gauthier, B.L., Michaelis, A.P., Cyamatare, F.R., Nyirazinyoye, L.,...Ntaganira, J. (2014). Clinical Mentorship to Improve Pediatric Quality of Care at the Health Centers in Rural Rwanda: A Qualitative Study of Perceptions and Acceptability of Health Care Workers. BMC Health Services Research. Meessen, B., Kashala, J.-P.I., & Musango, L. (2007). Output-Based Payment to Boost Staff Productivity in Public Health Centers: Contracting in Kabutare District, Rwanda. Bulletin of the World Health Organization, 108-115. Meessen, B., Soucat, A., & Sekabaraga, C. (2011). Performance-Based Financing: Just a Donor Fad or a Catalyst Towards Comprehensive Health-Care Reform? Bulletin of the World Health Organization, 153-156. Mels, G., Boshoff, C., & Nel, D. (1997). The Dimensions of Service Quality: The Original European Perspective Revisited. The Service Industries Journal, 173-189. Ministry of Monitoring, Planning and Implementation of the Modern Revolution, Ministry of Public Health, ICF International. (2014). Democratic Republic of Congo Demographic and Health Survey 2013-2014: Key Findings. Rockville, Maryland: MPSMRM, MSP and ICF International. Musgrove, P.(2011).Financial and Other Rewards for Good Performance or Results: A Guided Tour of Concepts and Terms and a Short Glossary.Washington, DC: The World Bank. Nanyonjo, A., Makumbi, F., Etou, P., Tomson, G., & Kallander, K. (2013). Perceived Quality of Care for Common Childhood Illnesses: Facility Versus Community Based Providers in Uganda. PLoS ONE. 62 n IMPACT EVALUATION: RESULTS-BASED FINANCING IN THE DEMOCRATIC REPUBLIC OF CONGO Nigeria Federal Ministry of Health. (2013).Nigeria State Health Investment Project (NSHIP) Performance-Based Financing User Manual. Abuja: Nigeria Federal Ministry of Health. Nove, A., Hulton, L., Martin-Hilber, A., & Matthews, Z. (2014). Establishing a Baseline to Measure Change in Political Will and the use of Data for Decision-Making in Maternal and Newborn Health in Six African Countries. International Journal of Gynecology and Obstetrics, 102-107. Oxman, A.D., & Fretheim, A. (2009). Can Paying for Results Help Achieve the Millennium Development Goals? A Critical Review of Selected Evaluations of Results-Based Financing. Journal of Evidence-Based Medicine, 184-195. Pickering, A.J., & Davis, J. (2012). Freshwater Availability and Water Fetching Distance Affect Child Health in Sub-Saharan Africa. Environmental Science & Technology, 46(4), 2391-2397. Projet de Santé Intégré en République Democratique du Congo. (2011). Manuel du financement basé sur les résultats. Cambridge: Management Sciences for Health. Projet de Santé Intégré en République Democratique du Congo. (2013). Manuel du financement basé sur les résultats. Cambridge: Management Sciences for Health. Projet de Santé Intégré en République Democratique du Congo. (2014). Manuel du financement basé sur les résultats. Cambridge: Management Sciences for Health. Renaud, A., & Dr.Semasaka, J.-P. (2014). Verification of Performance in Results-Based Financing (RBF): The Case of Community and Demand-Side RBF in Rwanda.Washington, DC: The World Bank. République Democratique du Congo Ministère de la Santé Publique. (2016, Fevrier 26). Rapports. Retrieved from Financement Basé sur les Resultats Santé République Democratique du Congo: http://www.fbrsanterdc.cd/report/ report/2.html Sadaphal S., Bongiovanni A.: Promoting Access to Quality Health Services: A Midterm Assessment of a Results-Based Financing Intervention in the Democratic Republic of Congo, April 2015. Retrieved from: http://pdf.usaid.gov/pdf_docs/ PA00KFHS.pdf Savedoff, W. (2015). A better model for foreign aid. Current History, 114 (775), 316-321. Scheffler, R.M. (2010). Pay For Performance (P4P) Programs in Health Services: What is the Evidence? World Health Report (2010) Background Paper, No 31. Geneva: World Health Organization. Soeters, R., Peerenboom, P.B., Mushagalusa, P., & Kimanuka, C. (2011). Performance-based financing experiment improved health care in the Democratic Republic of Congo.Health Affairs, 30(8), 1518-1527. The National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. (1978). The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research.Washington, DC: U.S. Department of Health, Education, and Welfare. Toonen, J., Canavan, A., Vergeer, P., & Elovainio, R.(2009). Learning Lessons On Implementing Performance Based Financing, From A Multi-Country Evaluation. Amsterdam: Royal Tropical Institute. Toonen.J.(KIT), Canavan, A.(KIT), Vergeer, P., (KIT) & Elovainio, R.(WHO) (2009). Performance Based Financing: A Synthesis Report Drawing lessons from country study reports Cordaid/HealthNet TPO experiences in PBF pilot projects in: Democratic Republic of Congo Tanzania Zambia Burundi Rwanda National PBF (retrospective study). REFERENCES n 63 United Nations Development Programme. (2015).Human Development Report 2015: Work for Human Development. United Nations, Department of Economic and Social Affairs, Population Division . (2015). World Population Prospects: The 2015 Revision, Key Findings and Advance Tables. Working Paper No. ESA/P/WP.241. United Nations Population Fund, International Confederation of Midwives, World Health Organization. (2014). The State of the World’s Midwifery 2014: A Universal Pathway. A Woman’s Right to Health. United Nations Security Council. (2016). Resolution 2277 (2016). USAID. (2014). Success Story: Results-Based Financing Strengthens Services at Kibila Health Center. Washington, DC: USAID. USAID/DRC. (2014). Implementing Partners Handbook. Kinshasa: USAID/DRC. Witter, S., Kessy, F.L., Fretheim, A., & Lindahl, A.K. (2009). Paying for performance to improve the delivery of health interventions in low and middle-income countries. Protocols Cochrane Database of Systematic Reviews. Witter, S., Toonen, J., Meessen, B., Kagubare, J., Fritsche, G., & Vaughan, K. (2013). Performance-Based Financing as a Health System Reform: Mapping the Key Dimensions for Monitoring and Evaluation. BMC Health Services Research. The World Bank. (2015). Doing Business. Starting a Business: Time (days) [Data file]. Retrieved from http://www. doingbusiness.org/Custom-Query/congo-dem-rep The World Bank. (2015). Doing Business. Getting Electricity: Getting Electricity Rank [Data file]. Retrieved from http://www. doingbusiness.org/data/exploretopics/getting-electricity. The World Bank. (2014). Policy Note 4 - Impact Evaluation on Performance-Based Financing In Haut-Katanga District, Democratic Republic of the Congo 2009-2013. The World Bank. (2009-2015). World Development Indicators. GDP at market prices; Inflation, GDP Deflator; GNI per capita, Atlas Method; Income share held by lowest 20%; GDP Growth; Mobile Cellular Subscriptions; Internet Users [Data file]. Retrieved from http://databank.worldbank.org/data/reports. aspx?source=2&country=COD&series=&period= World Health Organization. (2005). Management of Solid Health-Care Waste at Primary Health-Care Centres: A Decision-Making Guide. World Health Organization. World Health Organization. (2013). Pocket Book of Hospital Care for Children: Guidelines for the Management of Common Childhood Illnesses. World Health Organization. World Health Organization. (2014). Integrated Management of Childhood Illness: Chart Booklet. World Health Organization. World Health Organization. (2015). Water, Sanitation, Hygiene in Health Care Facilities. World Health Organization. World Health Organization/UNICEF Joint Monitoring Programme for Water Supply and Sanitation. (2016). Improved and Unimproved Water Sources and Sanitation Facilities. Retrieved from World Health Organization/UNICEF Joint Monitoring Programme for Water Supply and Sanitation: http://www.wssinfo.org/definitions-methods/watsan￾categories/ Wright, J., Health Finance & Governance Project. (2015). Essential Package of Health Services Country Snapshot: The Democratic Republic of the Congo. Bethesda, MD: Health Finance & Governance Project, Abt Associates Inc. EVALUATION Impact Evaluation of a Results-Based Financing Intervention in the Democratic Republic of Congo Annex Volume March 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by International Business & Technical Consultants, Inc. (IBTCI). It was authored by Swati Sadaphal and Annette Bongiovanni. IMPACT EVALUATION OF A RESULTS-BASED FINANCING INTERVENTION IN THE DEMOCRATIC REPUBLIC OF CONGO March 2016 AID-660-M-13-0000I Cover Photo: East Kasai Credit: Annette Bongiovanni DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Table of Contents ANNEX A: Evaluation Statement of Work ................................................................................................................. 1 ANNEX B: RBF Indicators for Health Zone Management Team .......................................................................... 4 ANNEX C: RBF Indicators for Health Centers ......................................................................................................... 6 ANNEX D: Data Collection Sampling Procedure ..................................................................................................... 8 ANNEX E: RBF Field implementation plan................................................................................................................ 14 ANNEX F: Sources of information.............................................................................................................................. 18 List of documents reviewed: .................................................................................................................................... 18 ANNEX G: Survey Questionnaires ............................................................................................................................. 20 ANNEX H: Training Manual for Data Collection .................................................................................................... 57 ANNEX I: Data Analysis Plan ....................................................................................................................................... 70 ANNEX J: Health Facility Survey Sample Distribution ........................................................................................... 77 Number of Facilities Surveyed: Baseline ................................................................................................................ 77 Number of Facilities Surveyed: Endline ................................................................................................................. 77 ANNEX K: Household Survey Sample distribution ................................................................................................ 78 Number of Household Surveyed: Baseline ........................................................................................................... 78 Number of Household Surveyed: Endline ............................................................................................................. 78 1 ANNEX A: Evaluation Statement of Work RBF: Baseline Evaluation IHP’s Results Based Financing Pilot Activity As part of a financing strategy under the IHP, in consultation with USAID/DRC and other key stakeholders, MSH will begin piloting a supply side results based financing (RBF) model in eight selected health zones in the four provinces of East Kasai, West Kasai, Katanga, and South Kivu. RBF is a management tool that encourages good performance and assesses performance on the basis of quantifiable indicators. Motivating health workers and keeping them in the public sector has been a particular challenge in the health system for many countries, including DRC, especially since many work under difficult conditions and in remote areas. RBF rewards health institutions and individuals working in these institutions for their achievements and thus raises morale. RBF can provide additional funds to Ministry of Public Health (MOH) staff, health centers, and hospitals if they meet or exceed the defined indicators for services, both in terms of quality and quantity. IHP will strengthen the capacity of its staff in the coordination offices (and their MOH counterparts) to accomplish two main activities as part of the RBF pilot: (1) provide technical support for the implementation of the RBF program in all health facilities in their respective health zones, and (2) to conduct verification activities and validation of RBF data. RBF contracts will be signed between IHP and health zones, hospitals, and health centers. Each IHP field office will be responsible for piloting the RBF model in one of their supported health zones, as follows: IHP Field Office RBF Health Zone Kole Lomela Tshumbe Wembonyama Uvira Nundu Mwene Ditu Bibanga Kolwezi Kanzenze Luiza Luiza Kamina Kayamba Existing Information The following background documents are included in the SOW as annexes: Project health zone data Minimum package of health service activities plus and complementary package USAID Evaluation Policy USAID/DRC and IHP will provide the successful contractor with a package of briefing materials upon award, including: Detailed Project Description Project quarterly and annual reports Project Performance Monitoring Plan Report on the first part of IHP’s Performance baseline survey, a population based study on Knowledge & Practices and Coverage of key IHP performance indicators, conducted by MSH in May 2011 (IHP Part I baseline) 2 Evaluation Rationale Results Based Financing Impact Evaluation Rationale USAID’s new evaluation policy requires any activity within a project involving untested hypotheses or demonstrating new approaches that are anticipated to be expanded in scale or scope through US Government foreign assistance or other funding sources to, if feasible, undergo an impact evaluation. The RBF impact evaluation will contribute to USAID‘s understanding of what is working in RBF and the effectiveness of RBF to increase the quantity and quality of health services. As part of the RBF impact evaluation the contractor will carry out a baseline, mid term, and final evaluation according to the following schedule: IHP Results Based Financing Impact Evaluation Schedule IHP RBF Baseline December 2012 IHP Mid Term RBF Evaluation December 2014 IHP Final RBF Impact Evaluation March 2016 Objectives of the Evaluations RBF Impact Evaluation Objectives Mid term Evaluation Objective: To evaluate progress to date, identify areas in project implementation that need improvement and provide recommendations. Final Evaluation Objective: To test hypotheses and measure results and impact of IHP’s RBF pilot over time. Audience and Intended Uses The audience of the RBF impact evaluation will be the USAID/DRC Mission, specifically the health team and program office, the implementing partner, the Ministry of Health, donors involved in RBF piloting, and secondary users like NGOs and other stakeholders. USAID/DRC will use this evaluation to inform policy and learning on RBF. Evaluation Questions Evaluation questions will structure the evaluation process, and USAID intends that these be aligned with the evaluations purpose and expected use. It also intends that each question be answerable with the highest quality and most credible evidence possible, given time and budget constraints. To ensure this, the total numbers of questions included in the SOWs for each evaluation will be limited. RBF Impact Evaluation Questions The baseline and final impact evaluations of the RBF component of the IHP program will provide statistically significant data to measure the baseline status and impact of the RBF interventions. The mid term impact evaluation will be predominantly qualitative, and will aim at determining whether initial characteristics in both treatment and comparison zones are still identical and determine the effectiveness of project implementation. However, the final impact evaluation will answer to the following illustrative questions: Is there evidence of change among health centers in the quantity and quality of services that is attributable to the RBF model? What difference did the RBF intervention make? Is the model worthy of being scaled up to other health zones? What are costs associated with a potential replication of the model? Were the desired results achieved? 3 Do results differ for various groups? (Heterogeneity) What contextual factors contributed to or limited the desired results? What are the unintended consequences of the intervention? What contextual factors contributed to or limited the desired results? Evaluation Design and Methodology USAID/DRC will provide the contractor with a statement of work for each evaluation component (baseline, mid term, and final) conducted under this contract but the contractor will design the methodology, sampling frames, and data collection instruments in consultation with USAID/DRC staff, implementing partners and country counterparts. Illustrative designs are set forth in Section J – Attachments. For the two baseline evaluations a SOW will be provided immediately after award of the contract and will require a collaborative effort between USAID and the contractor due to time sensitivities. The SOWs for all mid term and final evaluations will be provided at least four months prior to the start of each evaluation component. The contractor will be responsible for managing the evaluation and data collection in the field, working directly with program implementers, verifying data quality and preparing clean data sets, for analysis. Evaluation Design RBF Impact Evaluation, using experimental design will construct credible counterfactual scenarios, with the most credible being the random selection of treatment and comparison groups. This evaluation process will commence at the beginning of a project’s RBF implementation, to give evaluators the opportunity to work closely with program implementation staff in designing the evaluation, and for obtaining data throughout the life of the pilot. The impact evaluation will use both quantitative and qualitative methods. 4 ANNEX B: RBF Indicators for Health Zone Management Team N° Indicators Required data Calculation method 1 Proportion (number) of supervision visits completed Numerator: Number of supervision visits completed with reports and terms of reference Denominator: Number of supervision visits planned Number of supervision visits completed with reports and terms of reference x 100/Number of supervision visits planned 2 Proportion (number) of activities in the work plan completed during the quarter Numerator: Number of activities completed Denominator: Number of activities planned Number of activities completed x 100/Number of activities planned 3 Proportion (number) of prompt and complete NHIS reports sent to the Provisional Division of Health Numerator: Number of complete NHIS reports sent on time to the Provisional Division of Health Denominator: Total number of reports expected Number of NHIS reports sent on time to the Provisional Division of Health x 100/total reports expected 4 Proportion (number) of written feedback reports to the Health Centers and the general hospital. Numerator: Number of feedback reports written and sent to the facilities: Denominator: Total number of reports expected Number of feedback reports written and sent to the facilities x 100/total number of reports expected 5 Proportion (number) of data consolidation, validation and analysis meetings held. Numerator: Number of monthly monitoring meetings held Denominator: Total number of monthly monitoring meeting held Number of monthly monitoring meetings held x 100/ Total number of monthly monitoring meetings planned 6 Proportion (number) of COGE meetings for which minutes were sent to the Provincial Health Division Numerator: Number of meetings held, with minutes: Denominator: Total number of meeting planned Number of meetings held with reports x 100 / total number of meetings planned 7 Proportion (number) of COGE decisions implemented Numerator: Number of COGE decisions implemented: Denominator: Total number of decisions made by the COGE Number of COGE decisions implemented x 100/Total number of decisions made by the COGE 8 Is there an operational and up-to-date GESIS database? Is there an operational and up-to-date GESIS database? Is there an operational and up-to-date GESIS database? 9 Proportion (number) of Health Centers without medication inventory shortages (>3 days) Numerator: Number of health facilities with inventory shortages for the item: Denominator: Total number of health facilities in the Health Zone Number of health facilities with inventory shortages for the Item x 100/ Total number of health facilities in the Health Zone 10 Proportion (number) of medication orders approved by the quantification committee Numerator: Number of medication orders approved by the quantification committee: Denominator: Total number of orders submitted by the Health Zone Number of medication orders approved by the quantification committee x 100 / Total number of orders submitted by the Health Zone 5 11 Proportion (number) of community education activities that received support from the ECZ Numerator: Number of community education activities that received support from the ECZ Denominator: Total number of community education sessions Number of community education activities that received support from the ECZ x 100/Total number of community education activities 12 Proportion (number) of WASH activities supported by the ECZ Numerator: Number of WASH activities in the community that received support from the ECZ Total number of WASH activities in the community Number of WASH activities in the community that received support from the ECZ x 100/Total number of WASH activities in the community 13 Total score for Health Centers in the Health Zone Numerator: Scores obtained by the Health Centers Denominator: Total points expected Sum of points obtained by all Health Centers x 100 /Total points expected for all Health Centers 6 ANNEX C: RBF Indicators for Health Centers Indicator Points % Points (2014) % (2014) Points (2015) % (2015) Points (Our Analysis) % (O.A.) Rate of use of curative care services at HC (2014 & 2015) 3 3.5 4 6.2 4 4.9 4 6.7 Proportion of high-risk pregnancy referrals (2014 & 2015) 5 5.8 2 3.1 2 2.4 2 3.3 Coverage rate for pentavalent vaccine (2014 & 2015) 5 5.8 4 6.2 4 4.9 4 6.7 % of children who received full immunization (2015) n/a n/a n/a n/a 4 4.9 n/a n/a Proportion of pregnant women receiving TT2+ (2014 & 2015) 5 5.8 3 4.6 3 3.7 3 5 Number of clients receiving family planning counseling (2014 & 2015) 5 5.8 3 4.6 3 3.7 3 5 Rate of assisted deliveries (2014 & 2015) 3 3.5 2 3.1 2 2.4 2 3.3 TB case notification rate/New sputum positive smears (2014 & 2015) 5 5.8 3 4.6 3 3.7 3 5 Rate of ITNs distribution/Number of ITNs distributed (2014 & 2015) 3 3.5 2 3.1 2 2.4 2 3.3 Proportion of children under 5 with confirmed malaria, who received ACT Treatment (2015) n/a n/a n/a n/a 3 3.7 n/a n/a Number of clients (12-59 months) receiving CPS 2 immunization (2015) n/a n/a n/a n/a 3 3.7 n/a n/a Rate of ANC-1 service utilization (2014 & 2015) 3 3.5 4 6.2 4 4.9 4 6.7 Number of clients receiving counseling and voluntary HIV screenings (2014) 2 2.3 2 3.1 n/a n/a n/a n/a Number of pregnant women tested for HIV (2014) 3 3.5 3 4.6 n/a n/a n/a n/a 7 Rate of ANC-4 coverage (2014 & 2015) 2 2.3 4 6.2 4 4.9 4 6.7 Post-natal consultation rate (2014 & 2015) 3 3.5 4 6.2 4 4.9 4 6.7 Percent of monthly management meetings and drug inventory analyses submitted on time (2014 & 2015) 4 4.6 2 3.1 2 2.4 2 3.3 Number of children between 0- 59 months presenting with both diarrhea and fever given the correct treatment according to national policy (ORS and Zinc) in a health facility. (2015) n/a n/a n/a n/a 4 4.9 n/a n/a Number of children, between 0- 59 months, presenting with pneumonia and treated with antibiotics in a health facility. (2015) n/a n/a n/a n/a 3 3.7 n/a n/a Compliance with data from SNIS reports for indicators not funded by the RBF program. (2015) n/a n/a n/a n/a 5 6.1 n/a n/a Global FOSACOF score (2014 & 2015) 25 29.1 15 23.1 15 18.3 15 25 Global client satisfaction score (2014 & 2015) 10 11.6 8 12.3 8 9.8 8 13.3 8 ANNEX D: Data Collection Sampling Procedure Overview As part of a financing strategy under the Integrated Health Project, in consultation with USAID/DRC and other key stakeholders, Management Sciences for Health (MSH) has piloted a supply side results based financing (RBF) model in eight selected health zones in the four provinces of East Kasai, West Kasai, Katanga, and South Kivu in DRC. The RBF impact evolution baseline study employed both quantitative and qualitative methods. The quantitative surveys were comprehensive, and designed to collect data for indicators relevant to the evaluation including health center and hospital’s service assessment in terms of quality and quantity and client’s assess, knowledge, health seeking behaviors, and perceptions of quality of care. Specifically, the surveys consisted of a Health Facilities Survey and a Household Survey. SAMPLE DESIGN We selected one comparison health zone for each of MSH’s eight intervention health zones. Comparison zones sharing a geographical border with the intervention zones were randomly selected after excluding zones which had 1) a pre-existing PBF intervention, 2) known security issues, and/or 3) no physical access by land. This impact evaluation method followed a regression discontinuity that uses an eligibility cutoff to identify and differentiate groups. Sampling Method The RBF Facility survey used Lot Quality Assurance Sampling (LQAS) methodology, which is a cost effective robust approach to obtain information from representative sample of health facilities in the targeted health zones of intervention and comparison area. LQAS has played a valuable role for local-level program Monitoring and Evaluation. LQAS is applied under the assumption of randomness on occurrence of cases (disease condition), unimmunized children (defectives) etc. For RBF Household survey, sample sizes were determined by running power calculations based on selected indicators for intervention and comparison areas, which aimed to differentiating between the samples at 10% level. This procedure is necessary in an impact evaluation because it calculates the minimum sample size needed for the evaluation to measure an impact. Factoring the program implementing guidelines and minimum measures of outcome changes along with other sampling parameters allowed us to conduct power calculations to obtain the appropriate sample size needed for the impact evaluation. Sampling Frames For Site Selection: All facilities, which include all public health centers (160+138=298) and general referral hospitals (16) within eight selected intervention zones, as well as similar number of comparison zones, which share a geographic border with the RBF intervention zone under 4 provinces in DRC. Health zone coordination centers are not included in the sampling frame because these are administrative units and not service providers. For Household Selection: An up to date population statistics was used for drawing a representative sample of the catchment population/households within a zone cluster. The population statistics consisted of Health Zone, Health Center, corresponding Villages’ name and number of population according to 2011 estimates. The individuals or households was related to the priority health service areas of family planning; maternal, newborn, and child health (MNCH); nutrition; malaria; tuberculosis (TB); and HIV/AIDS. 9 For Key Informant Selection: Qualitative data collection activities have focus more on programmatic and managerial approaches in implementing the Integrated Health Project and the RBF program. So USAID/DRC staff, provincial planning authorities, health zone management teams, facility management teams, community organizations were initially included in the sampling frame. Sampling Procedure Health facility sample was selected using the LQAS methodology with a 92% precision level and 95% confidence intervals. Considering a population (N) size of 298, i.e. total health centers in the 14 selected health zones, 25 health centers (HC) were originally determined against each health zone of both intervention and comparison areas. The selected 25 sample HCs was then distributed in the corresponding health zones of intervention and comparison areas according to probability proportion of size, which shown in table below. Using this systematic procedure, a total of 200 health centers were selected (86 intervention centers and 103 comparison centers) and all 14 General Reference Hospitals were also selected (one in each zone). It is noted here that on some changing of population size, that was happened for replacing a health zone due to security reason, no outstanding changed was found on overall sample size. Table 1: Facility Sample Size of RBF Impact Evaluation No. Intervention Comparison Total sampled health facility (HC+ Health Zone GRH) (HZ) Sample size (HC) Health Zone (HZ) Sample size (HC) 1 Luiza 12 Lubondayi 13 (25+2) = 27 2 Bibanga 15 Kamiji 10 (25+2) = 27 3 Lomela 16 Tshudiloto 9 (25+2) = 27 4 Kayamba 7 Kabongo 18 (25+2) = 27 5 Kanzenze 13 Mutshatsha 12 (25+2) = 27 6 WemboNyama 11 Minga 14 (25+2) = 27 7 Nundu 12 Uvira 13 (25+2) = 27 Total 86 103 189 The household sample size calculation was done based on the maternal health indicator (% facility births), i.e. “proportion of mothers of children 0-23 months who delivered their youngest child in a health facility, which is 79.3%”. Considering 10% differences on the given indicator with 80% power and 95% confidence, a size of 226 household samples for each RBF intervention and comparison health zone was determined. A sample size of 3,616 household respondents was finally needed for 7 intervention health zones, as well as 7 comparison health zones. Over the total sample size, additional 15% household respondents also included to come back with the non-response cases. 10 Process for Household Sample Distribution The process of distributing household sample corresponding to each HC was done HZ by HZ both in the intervention and comparison areas. Then selection process followed a random procedure for systematic sampling of units. Moreover, HCs, Villages and Households/Respondents for interview were selected in the following ways: • List the name of sampled HCs using systematic sample procedure and extracted corresponding number of villages for each selected HC • Determined a representative village sample size (25) for each HZ using the LQAS methodology of maximum sample outcomes irrespective of total village size • List the name of sample villages according to probability proportion of village size and selected total population of those sample villages against each HC The total sample households were distributed proportionately among selected villages corresponding to respective sample HCs under each HZ of intervention, as well as comparison areas. Table 2: Example of Household sample sizes for RBF Impact Evaluation Survey-Intervention area Luiza HZ Sl. No Sampled HC (Systematic sampling) Name of the Sampled HC No. of Villages Village Sample Size (PPS) Name of the Sampled Village (Random selection) Pop (2011) % of total Vill. Pop Sampled HH size* Total HH size with 15% NR sample 1 1 Bambaie 10 1 Nsefu 548 3.43 8 9 2 3 Kakala 21 3 Luyinda II, Nsombi, and Nkabu 1659 10.40 23 27 3 4 Kakamba 6 1 Mbote 249 1.56 4 4 4 6 Kamushilu 10 1 Mbalu 1248 7.82 18 20 5 7 Kanda Kanda 13 2 Kanda Kanda and Kambayi 2004 12.56 28 33 6 9 Kasonga 15 2 Muambi and Kasela 956 5.99 14 16 7 10 Kabuanga 19 3 Lubundi, Kalembu, and Samany 2432 15.24 34 40 8 12 Kitoko 9 1 Muanda 1308 8.20 19 21 9 13 Moma 17 3 Kambimba, Moma, and Karunda 1580 9.90 22 26 10 15 Mukuandiang a 20 3 Kalala, Bibalu, and Tshisambi 1353 8.48 19 22 11 17 Mukungu 18 3 Isuku, alombi, and Ikomba 864 5.41 12 14 12 18 Tutante 11 2 Kasungu and Sampika 1757 11.01 25 29 169 25 15958 100.00 226 260 11 Table 3: Changes Made to Sampled Health Zones and Health Centers during Data Collection, 2013 Province Health zone Comments and Changes East Kasai Tshudiloto Loto replaced Booke and Okala replaced Longongo because military entered those has to arrest poachers. As a precaution data were not collected in Mobikili but there were no health areas remaining to replace. Additional households were selected in remaining health areas South Kivu Nundu Due to insecurity in health areas: Lutabara, Abala, Munene, Lamba, Lusenda and Nakiele some villages were replaced. RBF HEALTH ZONE SELECTION GRID Ratings Selection criteria Geographic accessibility 1 Can be reached by foot only 2 Can be reached by foot and by bicycle 3 Can be reached by foot, by bicycle and by motorcycle 5 Can be reached by foot, by bicycle, by motorcycle and by car/truck Security situation E Poor security situation (armed groups active in the majority of the Health Zone) 2 Pockets of uprisings, not armed 3 Rumors and threats exist, armed and unarmed uprisings 5 No uprisings, completely peaceful Availability of CPA services 1 There is a general hospital 2 There is a general hospital with less than 4 units 3 There is a general hospital with 4 units 4 There is a general hospital with 4 units and radiology or imaging Availability of MPA services 1 There is a curative treatment unit 2 There is a curative and preventive (vaccination) treatment unit 3 There is a curative, and preventive (vaccination) treatment unit and a promotional unit 4 There is a curative, preventive (vaccination) treatment unit, a promotional unit and community participation activities. Existence of RBF or payments of employee bonuses by other donors E There is a performance contract or bonus contract for employees in the Health Zone 4 In the past, there was no performance contract in the Health Zone 12 Donors 0 There are support partners in the Health Zone providing comprehensive support, other than IHP, in all health areas 1 There are support partners in the Health Zone, with comprehensive support in some Health Areas 3 There are support partners in the Health Zone, with partial support in some Health Areas 5 There are no support partners other than IHP in the Health Zone Previous use of RBF E Previous use of RBF 4 No previous use of RBF Population covered by the Health Zone: 1 Fewer than 50,000 residents 2 Between 50,000 and 74,999 residents 3 Between 75,000 and 99,999 residents 4 100,000 residents or more Rate of use of curative services 4 < 12.5% 3 25 to 12.5 % 2 37.5 to 25% 1 50 to 37.5% Rate for 3rd dose of pentavalent 5 < 30% 4 30 to 50% 3 50 to 60% 2 60 to 80% 1 < 80% Rate of childbirth attended by healthcare personnel 5 < 30% 4 30 to 50% 3 50 to 60% 2 60 to 80% 1 < 80% NHIS promptness and completeness rates 0 < 30% 1 30 to 50% 2 50 to 60% 3 60 to 80% 4 < 80% 13 Percentage of COGE minutes sent to the DPS 5 < 30% 4 30 to 50% 3 50 to 60% 2 60 to 80% 1 < 80% Proportion of supervision visits completed. 5 < 30% 4 30 to 50% 3 50 to 60% 2 60 to 80% 1 < 80% Existence of a GESIS database: 0 The BCZ has no specific software tool 1 The BCZ has a specific software tool 2 The BCZ has a specific software tool, and the ECZ members are trained in GESIS 3 The BCZ has a specific software tool, and the ECZ members are trained in GESIS, and some reports have been entered 4 The BCZ has a specific software tool, and the ECZ members are trained in GESIS, and some reports have been entered, and the GESIS database is up to date 14 ANNEX E: RBF Field implementation plan Month Week Date Activities SA May 1 5/11/2015 Travel to Kananga and meeting with Provincial authorities. Luiza 5/12/2015 Interviewers and data collection selection 5/13/2015 Meeting with MSH delegates and Logistic for the training 5/14/2015 Training in Kananga 5/15/2015 Training in Kananga 5/16/2015 Training in Kananga 5/17/2015 Training in Kananga 2 5/18/2015 Travel to Luiza 5/19/2015 Data collection in Luiza 5/20/2015 Data collection in Luiza 5/21/2015 Data collection in Luiza 5/22/2015 Travel to Lubondaie 5/23/2015 Data collection in Lubondaie 5/24/2015 Data collection in Lubondaie 3 5/25/2015 Data collection in Lubondaie 5/26/2015 Travel to Luiza 5/27/2015 Travel to Mueneditu with supervisors and Travel to Kananga Mueneditu 5/28/2015 Contact with interviewers and data collector 5/29/2015 Training in Muene Ditu 5/30/2015 Training in Muene Ditu 5/31/2015 Training in Muene Ditu June 4 6/1/2015 Training in Muene Ditu 6/2/2015 Travel to Kamiji 6/3/2015 Data collection in Kamiji 6/4/2015 Data collection in Kamiji 6/5/2015 Data collection in Kamiji 6/6/2015 Travel to Mueneditu 6/7/2015 Travel to Bibanga 5 6/8/2015 Data collection in Bibanga 6/9/2015 Data collection in Bibanga 6/10/2015 Data collection in Bibanga 6/11/2015 Travel to Mbujimayi 15 Month Week Date Activities SA 6/12/2015 Travel to Mueneditu 6/13/2015 Travel to Kinshasa Kinshasa 6/14/2015 Break 6 6/15/2015 Break 6/16/2015 Break 6/17/2015 Break 6/18/2015 Break 6/19/2015 Break 6/20/2015 Break 6/21/2015 Travel to Lodja and Travel to Tshumbe. Tshumbe 7 6/22/2015 Interviewers and data collection selection 6/23/2015 Training in Tshumbe 6/24/2015 Training in Tshumbe 6/25/2015 Training in Tshumbe 6/26/2015 Training in Tshumbe 6/27/2015 Travel to Minga 6/28/2015 Data collection in Minga 8 6/29/2015 Data collection in Minga 6/30/2015 Data collection in Minga July 7/1/2015 Data collection in Minga 7/2/2015 Travel to Wembo Nyama 7/3/2015 Data collection in Wembonyama 7/4/2015 Data collection in Wembonyama 7/5/2015 Data collection in Wembonyama 9 7/6/2015 Travel to Tshumbe 7/7/2015 Travel to Lodja 7/8/2015 Debriefing with Annette in Lodja Kole 7/9/2015 Debriefing with Annette in Lodja 7/10/2015 Interviewers and data collection selection 7/11/2015 Training in Lodja 7/12/2015 Training in Lodja 10 7/13/2015 Training in Lodja 7/14/2015 Training in Lodja 7/15/2015 Travel to Lomela 7/16/2015 Data collection in Lomela 7/17/2015 Data collection in Lomela 16 Month Week Date Activities SA 7/18/2015 Data collection in Lomela 7/19/2015 Travel to Tshudiloto 11 7/20/2015 Data collection in Tshudiloto 7/21/2015 Data collection in Tshudiloto 7/22/2015 Data collection in Tshudiloto 7/23/2015 Travel to Lodja 7/24/2015 Travel to Mukamba 7/25/2015 Travel to Mbujimayi 7/26/2015 Travel to Kinshasa 12 7/27/2015 Break Kinshasa 7/28/2015 Break 7/29/2015 Break 7/30/2015 Travel to Lubumbashi Kolwezi 7/31/2015 Meeting with provincial authorities and MSH August 8/1/2015 Travel to Kolwezi 8/2/2015 Interviewers and data collection selection 13 8/3/2015 Training in Kolwezi 8/4/2015 Training in Kolwezi 8/5/2015 Training in Kolwezi 8/6/2015 Training in Kolwezi 8/7/2015 Travel to Mutshatsha 8/8/2015 Data collection in Mutshatsha 8/9/2015 Data collection in Mutshatsha 14 8/10/2015 Data collection in Mutshatsha 8/11/2015 Travel to Kanzenze 8/12/2015 Data collection in Kanzenze 8/13/2015 Data collection in Kanzenze 8/14/2015 Data collection in Kanzenze 8/15/2015 Travel to Kolwezi 8/16/2015 Travel to Kolwezi (Adm & Fin) 15 8/17/2015 Travel to Lubumbashi 8/18/2015 Travel to Kamina Kamina 8/19/2015 Contact and meeting with local authorities 8/20/2015 Interviewers and data collection selection 8/21/2015 Training in Kamina 8/22/2015 Training in Kamina 17 Month Week Date Activities SA 8/23/2015 Training in Kamina 16 8/24/2015 Training in Kamina 8/25/2015 Travel to Kayamba 8/26/2015 Data collection in Kayamba 8/27/2015 Data collection in Kayamba 8/28/2015 Data collection in Kayamba 8/29/2015 Travel to Kabongo 8/30/2015 Data collection in Kabongo 17 8/31/2015 Data collection in Kabongo September 9/1/2015 Data collection in Kabongo 9/2/2015 Travel to Kamina 9/3/2015 Travel to kanmina and Financial and Administrative 9/4/2015 Break 9/5/2015 Travel to Bukavu From Lubumbashi Uvira 9/6/2015 Break 18 9/7/2015 Travel to Uvira and contact with local authorities 9/8/2015 Interviewers and data collection selection 9/9/2015 Training in Uvira 9/10/2015 Training in Uvira 9/11/2015 Training in Uvira 9/12/2015 Training in Uvira 19 9/13/2015 Travel to Nundu 9/14/2015 Data collection in Nundu 9/15/2015 Data collection in Nundu 9/16/2015 Data collection in Nundu 9/17/2015 Travel to Uvira 9/18/2015 Data collection in Uvira 9/19/2015 Data collection in Uvira 9/20/2015 Data collection in Uvira 20 9/21/2015 Travel to Bukavu 9/22/2015 Travel to Goma 9/23/2015 Travel to Kinshasa 18 ANNEX F: Sources of information List of documents reviewed: 2013 Human Development Report, United Nations Development Programme Congolese Pygmies’ Access to Healthcare: Barriers to Healthcare Access for Pygmies in Three Health Districts in South Kivu, DRC, June 2013 Disponsibilité des Medicaments MSH Prosani à la Cadmeko au 01 Avril 2013 DRC Demographic and Health Survey 2007: Key Findings, Macro International, Inc. DRC/Integrated Health Project – Performance Monitoring Plan, March 1, 2012 Health Systems 20/20 Final Project Report, 2012 Grant Miller and Kimberly Singer Babiarz, Pay-For-Performance Incentives in Low –and Middle Income Country Health Programs, 2013. Health Programs, 2013 PBF in Action Theory and Instruments Course Guide, CORDAID – SINA 2009 V181109 Stratégie de Renforcement du Système de Santé (SRSS), MOH, 2010b Plan National de Development Sanitaire (PNDS) (National Health Development Plan); MOH, 2010a Module de Formation Des Equipes Cadre De DPS En Financement Basé Sur Le Résultat, MOH, 2011 Module de Formation Des Formations Sanitaires En Financement Basé Sur Le Résultat, MOH, 2011 Module de Formation Des Equipes Cadre de BCZ En Financement Basé Sur Le Résultat, MOH, 2011 Module de Formation Des ASLO En Financement Basé Sur Le Résultat, MOH, 2011 IHP Minimum Package of Health Service Activities‐plus and Complementary Package of Health Service Activities-plus List Integrated Health Project (IHP), Manuel PBF, 2012, updated August 2014 Management Sciences for Health & USAID. (2011). Integrated Health Project in the Democratic Republic of Congo Baseline Survey Final Report. Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2011). Integrated Health Project in the Democratic Republic of Congo Manual for Results Based Financing . Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2011). Integrated Health Project in the Democratic Republic of Congo Performance Monitoring Plan. Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2011). Integrated Health Project in the Democratic Republic of Congo Quarterly Report: Year 1, Quarter 2 (January-March 2011). Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2011). Integrated Health Project in the Democratic Republic of Congo Quarterly Report: Year 1, Quarter 3 (April-June 2011). Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2011). Integrated Health Project in the Democratic Republic of Congo Quarterly Report: Year 1, Quarter 4 (July-September 2011). 2011: Management Sciences for Health. Management Sciences for Health & USAID. (2011). Manual for Performance Based Financing: Design and Implementation of Effective Performance Based Financing Programs. Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2012). Integral Health Project in the Democratic Republic of Congo Quarterly Report: Year 2, Quarter 2 (January-March 2012). Cambridge: Management Sciences for Health. 19 Management Sciences for Health & USAID. (2012). Integrated Health Project in Democratic Republic of Congo Quarterly Report: Year 2, Quarter 3 (April-June 2012). Cambridge: Mangement Sciences for Health. Management Sciences for Health & USAID. (2012). Integrated Health Project in Democratic Republic of Congo Quarterly Report: Year 2, Quarter 4 (July-September 2012). Cambride: Management Sciences for Health. Management Sciences for Health & USAID. (2012). Integrated Health Project in the Democratic Republic of Congo Year 2 Workplan. Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2012). Integrated Health Project in the Democratic Republic of Congo Quarterly Report: Year 2, Quarter 1 (October-December 2011). Cambride: Management Sciences for Health. Management Sciences for Health & USAID. (2013). DRC Update on Diarrhea & Pneumonia Situation. Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2013). Integrated Health Project in the Democratic Republic of Congo Year 3 Workplan. Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2013). Integrated Health Project in the Democratic Republic of the Congo Manual for Results Based Financing Programs. Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2014). Integrated Health Project in Democratic Republic of Congo Quarterly Report: Year 4, Quarter 2 (January-March 2014). Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2014). Integrated Health Project in the Democratic Republic of Congo Manual for Performance Based Financing. Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2014). Integrated Health Project in the Democratic Republic of Congo Manual for Results Based Financing Programs. Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2014). Integrated Health Project in the Democratic Republic of Congo Quarterly Report: Year 3, Quarter 3 (April-June 2013). Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2014). Integrated Health Project in the Democratic Republic of Congo Quarterly Report: Year 4, Quarter 3 (April-June 2014). Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2015). Integrated Health Project in the Democratic Republic of Congo Annual Report: Year 4. Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2015). Integrated Health Project in the Democratic Republic of Congo Quarterly Report: Year 5, Quarer 1 (October-December 2014). Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2015). Integrated Health Project in the Democratic Republic of Congo Quarterly Report: Year 5, Quarter 2 (January-March 2015). Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2015). Integrated Health Project in the Democratic Republic of Congo Quarterly Report: Year 5, Quarter 3 (April-June 2015). Cambridge: Management Sciences for Health. Management Sciences for Health & USAID. (2015). Integrated Health Project in the Democratic Republic of Congo Year 5 Workplan. Cambridge: Management Sciences for Health. 20 ANNEX G: Survey Questionnaires Household survey questionnaire (ENGLISH) EVALUATION OF THE RESULTS-BASED-FUNDING COMPONENT OF THE MSH INTEGRATED HEALTH PROJECT (PROSANI) QUESTIONNAIRE FOR MOTHERS OF CHILDREN 0-23 MONTHS INFORMATION GRID FOR MOTHERS OF CHILDREN 0-23 MONTHS This questionnaire must be administered to mothers whose youngest child is between 0-23 months 01 Village Name and Number Name_________________________/____/____/ 02 Household number in the village /____ /____/______/ 03 Health Area name and number Name _________________________/____/____/ 04 Health Zone name and number Name ___________________________/____/____/ 05 Supervision Area name and number Name ___________________________/____/____/ 06 Province name and code West Kasai......................................... 1 East Kasai.......................................... 2 Katanga.............................................. 3 South Kivu.......................................... 4 07 Residential area Urban ..........................................................1 Semi-Urban …………………………………2 Rural .............................................................3 08 Mother's name and number in the village Name ___________________________/____/____/ ________________________________________ 09 Child's date of birth ___ ___ / ___ ___ / ___ ___ ___ ___ Day Month Year 10 Child's age (in months) /____ /____/______/ 11 Interviewer's name and code Name ___________________________/____/____/ 12 Day / Month / Year of interview ____ ____ / ____ ____/ ____ ____ ____ ____ 21 INFORMED CONSENT Hello. My name is ___________________, and I work with IBTCI/CESD. We are conducting a survey, and we would like you to participate in it. I would like to ask you some questions about your health and the health of your youngest child under two years old. This information will help the IHM/Prosani project plan health services and evaluate whether they comply with the goals for improving children's health. The survey usually takes ___ minutes. Regardless of the information that you provide to us, your answers will remain completely confidential and will not be revealed to anyone. Participation in this survey is voluntary, and you may decide not to answer personal questions, or any of the questions. However, we hope that you will participate in this survey since your answers are very useful to us. Do you have any questions for me about the survey? Interviewer's signature: ______________________________________ Date: ____________________ THE RESPONDENT AGREES TO BE INTERVIEWED ............................................................1 THE RESPONDENT DOES NOT AGREE TO BE INTERVIEWED ....................................................................... 2 ──END 1 Yes, permission given. Go to the "Household composition" module to begin the interview. 2 No, permission not given. Discuss this outcome with your Supervisor. 13 Audited on-site by (Name and code): Name ___________________________/____/____/ 14 Data entry agent (Name and code): Name ___________________________/____/____/ 22 Module 1: HOUSEHOLD COMPOSITION CM For all members of the household For all children age 5-24 years For all persons age 15 or older CM1 CM2 CM3 CM4 CM5 CM6 CM7 CM8 CM9 CM10 CM11 N° Order Name Relationship 1= Head of household 2= Spouse 3= Son/daughter 4= Other relative 5= Not a relative Sex: 1= Masc 2= Fem Age (in comple te years) Employment Status 1= Young child 2= Child, not in school 3= Student 4= Salaried worker 5= Self￾employed worker 6= Retired 7= Homemaker 8= Unemployed 9= Other, not employed Current school attendance Is (name) attending school during the 2012- 2013 school year? 1= Yes 2= No [Go to CM9] Level of education What level is he/she in? 0=Preschool/Kin dergarten 1= Primary 2= Secondary or higher 3= Informal program 9 = Don’t know Literacy status 1= Cannot read or write 2= Can read, cannot write 3= Can read and write Highest level of education 0=Preschool/ Kindergarten 1= Primary 2= Secondary or higher 3= Informal program 9 = Don’t know Marital status 1= Single 2= Married 3= In a relationship/co mmon law marriage 4= Divorced /Separated 5= Widowed 1 2 3 4 5 6 7 8 9 10 11 12 23 N° Module 2: WATER AND SANITATION EA EA 1. What is the primary source of the water that your household members drink? Tap In the house …................................. 11 [Go to EA3] …..In a field, street or outdoor area 12 [Go to EA3] Neighbor's tap ..................................... 13 [Go to EA3] Public tap/fountain 14 Pump-operated well; Water hole ................... 21 Dug well Protected well.......................................31 Unprotected well ................................32 Spring water Protected spring.................................. 41 Unprotected spring............................. 42 Surface water (river, dam, Lake, pond, canal, irrigation ditch) …………...81 Other (specify) _______________________ 96 EA 2. Where is this water source located? In the house.................................... 1 In the street/outside .............................. 2 Somewhere else............................................... 3 EA 3. Do you do anything to make the water cleaner before drinking it? Yes .................................................... 1 No ................................................... 2 [Go to EA5] Don’t know......................................... 8 [Go to EA5] EA 4. What do you normally do to purify the water that you drink? Ask again: Anything else? Record everything that is mentioned. Boil it..................................... A Add bleach or chlorine.......... B Filter the water through a cloth..... C Use a filter (ceramic, sand, composite, etc.) ........................... D Solar disinfection .......................... E Let the water settle..........................F Other (specify) _______________ X Don’t know...................................... Z EA 5. What type of toilets do the members of your household typically use? Flush toilets, with or without a water tank Connected to a sewer system ............... ……………..11 Connected to a septic pit...............................................12 24 N° Module 2: WATER AND SANITATION EA If "flush toilet" or "manual flush," ask again: Where does the waste water go? If necessary, ask permission to see the toilets. Connected to latrines ...................................................13 Connected to something else..................................... 14 Connected to an unknown location/not sure/don't know where? 15 Improved, ventilated pits/latrines (LAV)…………….....21 Pit latrines with a slab .................................................22 Pit latrines without a slab/open hole………….. ……. 23 Composting toilets .......................................................31 Buckets..........................................................................41 Raised toilets/latrines....................................................51 No toilets, outdoors used as toilet ................................95 A [Go to EA9] Other (specify) _______________________ 96 EA 6. Do you share these toilets with other people who are not members of your household? Yes..............................................................1 No.............................................................2 [Go to EA9] EA 7. Do you share these toilets only with members of other households who you know, or can anyone use them? Only other households (not public)...........1 Public toilets ..............................................2 [Go to EA9] EA 8. In total, how many households use these toilets, including your household? Number of households (if less than 10)..................... 0 Ten households or more...............................................10 Don't know........................................................................98 EA 9. Please show me where the members of your household wash their hands most of the time. Seen ......................................................1 Not seen............................................... 2 [Go to EA12] EA 10. Check if there is water in the designated hand- washing area Verify whether there is water in the tap, pump, basin, bucket, water container or similar item. Water available ..................................... 1 Water not available................................... 2 EA 11. Note whether there is soap or another cleaning product in the location designated for hand washing. Circle everything that is mentioned. Bar of soap................................ A Cleaner (powder, liquid, paste)...... B Liquid soap.................................... C Ashes/mud/sand........................... D Nothing.......................................... Y 25 N° Module 2: WATER AND SANITATION EA EA 12. Do you have soap or another cleaning product (or other local products used as cleaning products) in your house, for washing hands? Yes .................................................... 1 No ...................................................... 2 [Go to Module 3] EA 13. Can you please show it to me? Record observations. Circle everything that is mentioned. Bar of soap................................ A Cleaner (powder, liquid, paste).......... B Liquid soap...................................... C Ashes/mud/sand........................... D Did not show/refused to show....................Y Other (specify)............................................X EA 14. In the past 12 months, have health workers given you advice or information on the importance of potable water and hygiene? Yes .................................................... 1 No ...................................................... 2 [Go to Module 3] EA 15. If yes, who gave you this advice or information? Doctor ……………………………………..A Nurse ……………………………………...B Midwife …………………………………....C Health agent or community health worker........................D Other (specify) ……………………………E EA 16. Have you put the advice or information that you received from the health workers into practice? Yes .................................................... 1 No ...................................................... 2 [Go to 3] 26 MODULE 3: CONTRACEPTION CO This module applies to women ages 15-49 who are not pregnant. Now, I would like to ask you some questions about family planning; the various means or methods that a couple can use to postpone or avoid pregnancy. Circle code 1 for each method that is spontaneously mentioned. Then read the name and description of each method that is not spontaneously mentioned. Circle code 1 if the respondent is familiar with the method, and code 2 if the respondent is not. CO 1. What means or methods have you heard of? For the methods that are not spontaneously mentioned, ask Have you ever heard of (name the method)? CO 2. FEMALE STERILIZATION-Women can have an operation to avoid having more children. Yes.................................... 1 No ..................................... 2 CO 3. MALE STERILIZATION- Men can have an operation to avoid having more children. Yes.................................... 1 No ..................................... 2 CO 4. PILLS- Women can take a pill every day to avoid becoming pregnant. Yes.................................... 1 No ..................................... 2 CO 5. IUD-Women can have a doctor or nurse place an intra-uterine device internally. Yes.................................... 1 No ..................................... 2 CO 6. INJECTIONS- Women can have a health worker give them an injection to avoid becoming pregnant for one month or more. Yes.................................... 1 No ..................................... 2 CO 7. IMPLANTS-Women can have small rods inserted under the skin of their upper arm, which keeps them from becoming pregnant for a year or more. Yes.................................... 1 No ..................................... 2 CO 8. CO1g CONDOM-Men can put a rubber cover over their penis during sexual intercourse. Yes.................................... 1 No ..................................... 2 CO 9. CO1h FEMALE CONDOM- Women can put a barrier in their vagina before sexual intercourse. Yes.................................... 1 No ..................................... 2 CO 10. CO1b DIAPHRAGM- Women can put a diaphragm in their vagina before sexual intercourse. Yes.................................... 1 No ..................................... 2 CO 11. SUPPOSITORIES, FOAM OR GEL- Women can put a suppository, gel or cream in their vagina before sexual intercourse. Yes.................................... 1 No ..................................... 2 CO 12. PERIODIC ABSTINENCE/RHYTHM/CALENDAR- A sexually active woman can avoid becoming pregnant by avoiding sexual intercourse on the days of the month when she has the highest chance of becoming pregnant. Yes.................................... 1 No ..................................... 2 CO 13. WITHDRAWAL- Men can be careful to withdraw before ejaculation. Yes.................................... 1 No ..................................... 2 CO 14. MORNING-AFTER PILL-Women can take pills up to three days after sexual intercourse to avoid becoming pregnant. Yes.................................... 1 No ..................................... 2 CO 15. Have you heard of other means or methods that women or men can use to avoid pregnancy? Yes.................................... 1 27 MODULE 3: CONTRACEPTION CO (If yes, specify): ________________ No …………………2 CO 16. Some couples use various means or methods to postpone or avoid pregnancy. At this time, are you doing anything or using a method to postpone or avoid pregnancy? Yes.................................... 1 No ..................................... 2 [Go to CO18] CO 17. What are you doing at this time to postpone or avoid pregnancy? Do not suggest a response If the respondent mentions more than one method, circle each method Female sterilization .........................................A Male sterilization .............................................B IUD ..................................................................C Injection............................................................D Implants ...........................................................E Pills ..................................................................F Male condom...................................................G Female condom...............................................H Diaphragm .......................................................I Foam/gel..... ....................................................J Lactation Amenorrhea Method (LAM)..............K Periodic abstinence/Rhythm/Calendar.............L Withdrawal...................................................... M Morning after pill.............................................. N Other (specify) _______________________ X CO 18. In the past 12 months, have you discussed family planning practices with your husband/partner, friends, neighbors or relatives? Yes ................................1 No................................ 2 [Go to Module 4] CO 19. With whom have you talked about it? With anyone else? Record everything that is mentioned. Husband/Partner..................................A Mother...................................................B Father....................................................C Sisters.................................................D Brothers.................................................E Sons......................................................F Daughters.............................................G Mother in law.........................................H Friend(s)……………………………………I Neighbor(s).............................................J Other (specify)____ ____________ X CO 20. In the past 12 months, have you talked with a health worker about family planning, or received advice or information from a health worker about family planning? Yes .................................................... 1 No ...................................................... 2 [Go to Module 4] CO 21. If yes, who gave you this advice or information? Doctor ……………………………………..A Nurse ……………………………………...B Birth assistant ………………………..…..C Health agent or community health worker D Other (specify) …………………………….E 28 N° Module 4: HIV/AIDS VS VS 1 I do not want to know the results, but have you ever been tested to see if you have the AIDS virus? Yes .................................................... 1 No .................................... 2 [Go to 7] VS 2 When did you have your most recent HIV/AIDS test? More than 12 months ago.......................1 12-23 months ago.................................. 2 2 years ago or more .............................. 3 VS 3 I do not want to know the results, but did you receive the results of the test? Yes ....................................................1 No..................................................... 2 Don't know......................................... 8 VS 4 Did you have the HIV test while you were pregnant with this child who is now 0-23 months old? Yes .................................................... 1 No .................................... 2 [Go to VS6] Don't know .....................................8 VS5 If yes, did you receive the results of this test? Yes .................................................... 1 No .................................................... 2 Don't know .....................................8 VS 5 If no, why do you think you did not have this test while you were pregnant? 1 The health workers or health facility did not offer it. 2 Test is not available at this time. 3. I refused. 4. My husband/partner did not want me to have the HIV test. VS 6 Are you currently married, or are you currently living with a man, as if you were married? Yes, currently married........... 1 Yes, living with a man ............ 2 No, not in a relationship........... 3 N° Module 5: MATERNAL HEALTH SM This module applies to all women who have a child between 0-23 months. Note the child's name here _____________________ When you ask the following questions, use the child's name where indicated. SM 1 Did you receive prenatal care during your pregnancy with (name)? Yes........1 No ......2 [Go to SM7] SM 2 Who treated you? Ask again: Anyone else? Make sure to find out the type of person seen, and circle all of the responses given. Health professional: Physician............................. A Nurse………………….... .. B Birth assistant..................... D Midwife................................ E Other person Traditional birth assistant...... F Community health agent........G Other (specify) ___________ X SM 3 How many times did you receive prenatal care during your pregnancy with (child's name)? Number of times __ __ Don't know ..................98 SM 4 Do you have a booklet or other document that all of your vaccinations are recorded in? Yes (booklet seen).......... .1 No (booklet not seen)...........2 No ……………… …3 29 May I please see it? If you are shown the booklet, use it to respond to the following questions. Don't know ...........................8 SM 5 When you were pregnant with this child (say child's name), did you receive an injection in your arm or shoulder to prevent the baby from contracting tetanus, meaning convulsions after birth? Yes ... .1 No....... .2 [Go to SM7] Don't know ... ...8 [Go to SM7] SM 6 How many times did you receive this tetanus injection during the pregnancy with this child (name)? If 7 times or more, indicate "7" Number of times......................... ______ Don't know...........................8 SM 7 Who assisted you during the birth of (say child's name)? Ask again: Anyone else? Ask again, to find out the type of person who attended the birth, and circle all responses mentioned. If the respondent states that no one assisted her, ask again, to determine whether no other adult was present at the birth. Health professional: Doctor.................... A Nurse.... .. B Birth assistant.........D Midwife................... E Other person Traditional birth assistant................. F Community health worker….........G Relative / Friend........ H Other (specify) _________________ X No one.............................. Y SM 8 Where did you deliver (say child's name)? Ask again, to find out the type of location. If you cannot determine whether the location is a public or private facility, write the name of the facility. (Name of facility) _____________________________________ At home Your home.............................11 Someone else's home............12 Public medical sector Government hospital..............21 Clinic/Government health center ............22 Government health post..........................23 Other (specify) ___________________ 26 Private medical sector Private hospital......................................31 Private clinic.......................................32 Private maternity hospital...................33 Other private medical facility (specify)______________________ 36 Other (specify) ________________ 96 SM 9 After (say child's name) was born, did a health professional or another person examine you? Yes ....1 No........2 [Go to SM12] SM 10 How many weeks or days after the birth did you have your first health examination? Circle ‘1’ and indicate”00" days for the day of the birth. Number of days after delivery.....1 __ __ Number of weeks after delivery....2 __ __ Don't know............................................98 SM 11 Who examined you? Health professional: Doctor........................................ A 30 Ask again: Anyone else? Make sure to find out the type of person seen, and circle all of the responses given. Nurse.......................................... B Birth assistant............................. D Midwife........................................ E Other person Traditional birth assistant………...F Community health agent..............G Traditional healer.........................H A mother in the neighborhood/village..I Other (specify)............................... X SM 12 In the months after the birth, did you receive a dose of Vitamin A like this one? Show the capsule Yes ....1 No ..................2 [Go to Module 6] SM 13 How long after (name)'s birth did you receive the first dose of Vitamin A? Less than 1 month ...................1 Between 1 and 2 months .........2 Between 2 and 6 months .........3 More than 6 months ...................4 Don't know.................................8 Module 6: DISEASE SYMPTOMS SY SY1 Sometimes, children become seriously ill and must be taken to a health facility right away. What types of symptoms would lead you to take your child to a health facility right away? Ask again: Any other symptoms? Circle all the symptoms mentioned, but do NOT suggest any responses. Child cannot drink or breastfeed.................. A Child's condition is getting worse................. B Child develops a fever................................. C Child is breathing rapidly............................... D Child has difficulty breathing.......................... E Child has blood in his/her stool.......................F Child has difficulty drinking................. ... G Child has diarrhea......................................... H Child is vomiting............................... .......I Other (specify) _____________________ X Other(specify) _____________________ Y Other (specify) _____________________ Z Module 7: CONTACT WITH HEALTH SERVICES CS-SIS CS1 In the last few months, how many times did you come into contact with the following health professionals in a health center or the community house? Physician Nurse/midwife Community health worker Growth monitor Trained birth assistant Traditional healer Coding categories Frequently Sometimes Never (4 times or more) (1-3 times) (0 times) 1 2 3 1 2 3 1 2 3 1 2 3 1 2 3 31 Other (specify) ____________________ 1 2 3 1 2 3 If at least one of the health contacts above was mentioned, go to the next question. Otherwise, go to question CS4. Where did you meet with this health professional? At my house....................A At a health center..............B In the community................C CS2 How did you come into contact with this person or these people? • When he/she was doing a routine visit in the community • During a health education campaign • During visits to announce a future activity • When I went to the health center for care • Other (specify)_______________ A B C D X CS3 What health practices have you learned through contact with these health professionals? Ask again: Any other practices? Record everything that is mentioned. • Exclusive breastfeeding • Good nutrition • Vaccinations • Prevention and treatment of diarrhea • Prevention and treatment of acute respiratory infections • Prevention and treatment of malaria Education on use of family planning methods • Prevention and treatment of HIV/AIDS • Other (specify): __________________ A B C D E F G H X CS4 From whom do you usually obtain general information or advice about health or nutrition? Record everything that is mentioned. Formal network • Physician • Nurse/midwife • Auxiliary midwife • Community health worker • Growth monitor • Trained birth assistant Informal network • Spouse/partner • Mother/adoptive mother • Sister • Grandparent • Aunt • Friend/neighbor • Traditional healer • Village elders • Other (specify)_______________ A B C D E F G H I J K L M N X CS5 In the last month, did you receive any health messages • Community health workers? 1 1 2 2 32 through the following channels? • Doctor or nurse? • Family member? • Radio? • Magazine/newspaper? • Television? • School? • Text message? • Other: (specify) 1 1 1 1 1 1 1 2 2 2 2 2 2 2 CS6 When was your last visit to a health center? • Last week • Last month • Three months ago • Six months ago • Last year • Don't know 1 2 3 4 5 6 CS7 Was it a local health center? Yes No 1 2 If no, go to CS9 CS8 How much did you pay for these services? /______________/ FC CS9 Where did you go first, to seek advice or treatment the last time you needed health services? Health Center General hospital Traditional healer Pharmacy 1 2 3 [Go to CS12] 4[Go to CS12] CS10 The last time that you sought out a health service that you wanted, did you receive it? If no, go to CS 12 Yes No 1 2 CS11 If yes, were you satisfied? Yes No 1 2 CS12 If not at a health center or hospital, why did you not go to a health center or hospital? Too far Not enough money to pay the bill Staff not qualified Nurse not welcoming I prefer traditional medicine Other (specify)______________ 1 2 3 4 5 6 CS13 At your last health center visit, did the nurse spend as much time as you wanted with you? Yes No 1 2 CS14 Do you think that the nurse/doctor treated you professionally and gave you proper care? Yes No 1 2 CS15 Did the nurse/doctor listen attentively to you and let you ask the questions that you wanted to ask him/her? Yes No 1 2 Module 8: VITAMIN A SUPPLEMENTATION VA VA1 Did (name) receive a dose of Vitamin A like this one in the past 6 months? Yes../.......... 1 No....................... 2 [Go to Module 9] 33 Show the most common types of vials/capsules/syrups. Don't know................. 8 [Go to Module 9] VA2 When did (name) receive the most recent dose? Write the date of the most recent vitamin dose as shown in the vaccination booklet Write "44" for the day if the booklet shows that a Vitamin A dose was given, but that the date was not recorded; leave the month and year blank. Day..........................__ __ Month..........................__ __ Year ..................... __ __ __ __ No mention of Vitamin A in the booklet.....98 No booklet/booklet not seen ....... 99 VA3 When did (name) receive the vitamin A? Campaign .......................1 Routine........................... 2 Illness.............. ...3 Other ..............................6 Module 9: INITIAL BREASTFEEDING AP This module applies to all women who have a child between 0-23 months. Write the child's name here_________________ When you ask the following questions, use the child's name where indicated. AP1 Have you breastfed (name)? Yes.... 1 No ................................2 [Go to Module 10) AP2 How soon after birth did you put (name) to the breast for the first time? If less than one hour, write "00" hours. If less than 24 hours, write the time in hours. Otherwise, write the time in days. Immediately ........................................00 Hours .....................................................1 __ __ Days......................................................2 __ __ Don't know/Don't remember ..............98 Module 10: INSECTICIDE-TREATED MOSQUITO NETS MI MI1 Does your household have mosquito nets that can be used for sleeping? Yes.................................... 1 No ..................................... 2␣ [Go to Module 11] MI2. How many mosquito nets does your household have? If the household has 7 mosquito nets or more, write "7" Number of mosquito nets: ___ MI3 Did (child's name) sleep under a mosquito net last night? Yes................................... 1 No ..................................... 2 Ask the respondent to show you the mosquito net that the child slept under on the night before the interview. MI4 MI4 Mosquito net seen? Seen ................1 Not seen .......... 2 [Go to MI6] 34 MI5 Check or ask about the brand/type of mosquito net Long-lasting insecticide-treated mosquito net: Permanet (Serena).................. 11 Olyset ....................................... 12 Net Protect ............................... 13 Duranet .................................... 14 Interceptor................................ 15 Other (specify) ____________ 16 Brand unknown ............................ 18 Other mosquito net (specify)......... 31 Do not know brand/type................ 98 MI6 Where did you obtain this mosquito net? During a campaign...................... 1 In a health center......................... 2 Other (specify)_______________6 Number of months ............ ___ ___ More than 36 months ...................95 Don't know/not sure .................. 98 MI7 For how many months has your household had this mosquito net? If less than one month, write "00" Number of months.............. ___ ___ More than 36 months ...................95 Don't know/not sure ................... 98 MI8 See MI5 for the type of mosquito net 1 Long-lasting (11-18) Go to module 11 2 Other MI9 When you obtained this mosquito net, was it already treated with insecticide that kills or repels mosquitoes? Yes.................................... 1 No...................................... 2 Don't know/not sure .......... 8 MI10 Since you have had this mosquito net, has it been soaked or immersed in a liquid that kills or repels mosquitoes? Yes ......................................1 No .........................2 [Go to Module 11) Don't know/not sure.8 [Go to Module 11] MI11 How many months have passed since the last time the mosquito net was soaked or immersed? If less than one month, write "00" Number of months ............. ___ ___ More than 24 months ............... 95 Don't know/not sure .................. 98 35 Module 11: TREATMENT OF DIARRHEA TD TD1 Has (name) had diarrhea in the past two weeks? Yes ................... 1 No .....................2 [Go to module 12] Don't know........8 [Go to module 12] TD2 I would like to know what quantity of liquids (name) was given while he/she had diarrhea (including breast milk) While (name) had diarrhea, was he/she given less than usual to drink, about the same amount or more than usual? If less, ask again: was he/she given much less than usual to drink, or a little less than usual? Much less...................... .1 A little less ...........................2 About the same amount .......3 More...................................... 4 Nothing to drink....................5 Don't know ...........................8 TD3 While (name) had diarrhea, was he/she given less than usual to eat, about the same amount, more than usual or nothing at all? If less, ask again: Was he/she given much less than usual to eat, or a little less than usual? Much less.................. .....1 A little less ...........................2 About the same amount ......3 More.................................... 4 Stopped eating................... .5 Never given anything to eat..6 Don't know ...........................8 TD4 While he/she had diarrhea, did you give (name) one of the following products to drink: Read the name of each product out loud, and record the response before going to the next product: [A] A liquid prepared using a special packet called (local name for the ORS solution packet)? [B] A prepackaged ORS liquid for diarrhea? [C] A homemade liquid (salty-sweet solution (SSS) recommended by the government)? Yes No Don't know 1 2 8 ORS liquid packet........ 1 2 8 Prepackaged ORS liquid...1 2 8 Recommended homemade liquid. 1 2 8 TD5 Was he/she given anything else to treat the diarrhea? Yes....................... 1 No ........................ 2 [Go to module 12] Don't know............8 [Go to module 12] 36 TD6 What (else) was he/she given to treat the diarrhea? Ask again: Anything else? Write all of the treatments given. Write the names of all of the medications mentioned. Antibiotic liquid or pills................... A Anti-motility agent.......................... B Zinc................................................ C Other (not an antibiotic, anti-motility agent or zinc) G Unknown liquid or pills................... H Antibiotic injection........................... L Non-antibiotic ................................M Unknown injection........................ N Intravenous................................... O Home remedy/medicinal herbs...... Q Other (specify) _____________________ X MODULE 12, TREATMENT OF PRESUMED PNEUMONIA PP PP1 In the last 2 weeks, did (name) have a cough? Yes ........................... .. 1 No................................ 2 [Go to module 13] Don't know.......................8 [Go to module 13] PP2 When (name) was sick, did he/she breathe faster than usual, with short and rapid breaths, or did he/she have trouble breathing? Yes .................................................... 1 No ........................... .... 2 [Go to module 13] Don't know....................8 [Go to module 13] PP3 Did you receive advice or treatment for the disease outside of your home? Yes .................................................... 1 No .................................. 2 [Go to module 13] Don't know......................8 [Go to module 13] PP4 Where did you seek advice or treatment? Ask again: Anywhere else? Circle all locations mentioned. But do NOT suggest responses. Ask again, to find out the type of location. If you cannot determine whether the location is in the public or private sector, write the name of the location. Public Sector State hospital..................................... A State health center............................ B State health post........................ ... C Community health agent.......... .... D Mobile/community clinic.................... E Other public facility (specify)...... H Private medical sector Private hospital/clinic............................I Private doctor...................................... J Private pharmacy................................. K Mobile clinic................................... ...L Other private medical facility (specify)..O Other source Relative/Friend.............P Store............................................ ...Q Traditional practitioner....................... R Churches........................................... S Other (specify)_______________ X PP5 Was (name) given a medication to treat this illness? Yes................................ 1 No .................................. 2 [Go to module 13] Don't know.......................8 [Go to module 13] PP6 What medication was (name) given? Ask again: Antibiotic: 37 Any other medication? Ask again: Any other medication? Circle all medications given. Write the brand name of all of the medications given. ______________________________________ (Medication Name) Pill/Liquid.............................. A Injection................................. B Antimalarials ........................................ M Paracetamol/Panadol/Acetaminophen............ P Aspirin...................................................... Q Ibuprofen................................................... R Other (specify) _____________________ X Don't know.....................................................Z QUESTIONNAIRE CHILDREN 0-23 MONTHS WITH FEVER Module 13: MALARIA TREATMENT TP TP1 In the past 2 weeks, has (name) had a fever? Yes ......................... ..... 1 No .............................. 2 [Go to module 14] Don't know..................8 [Go to module 14] TP2 At any time when (name) was sick, was blood drawn from his/her fingertip or heel to perform a test? Yes .................................................... 1 No....................... ........... 2 Don't know............... ...................8 TP3 Did you receive advice or treatment anywhere or from anyone for this illness? Yes .................................................... 1 No ................................. 2 [Go to TP8] Don't know........................8 [Go to TP8] TP4 Was (name) taken to a health facility during his/her illness? Yes .................................................... 1 No............................. .... 2 [Go to TP8] Don't know.......................8 [Go to TP8] TP5 Was (name) given a medication for fever or malaria at this health facility? Yes .................................................... 1 No ................................. 2 [Go to TP7] Don't know.....................8 [Passer a TP7] TP6 What medication was (name) given? Ask again: Any other medication? Circle all medications mentioned. If medications were given, write the names of all of the medications. __________________________________________ (Medication names) Antimalarials: SP(*)...........................................................A Chloroquine.................... ..................... …….B Amodiaquine/Camoquin/ Flavoquine............C Quinine .........................................................D Combination with Artemisinin(**) .......... ....E Antimalarial/community-based care facility.....F Other antimalarial (specify) _______H Antibiotics Pills/Liquid................................... I Injection................................................... J Other medications: Paracetamol/ Panadol /Acetaminophen .P Aspirin.................................................. Q Ibuprofen .................................................R Other (specify) _______________________ X Don't know............ ..... Z TP7 Was (name) given a medication for fever or malaria before being taken to a health facility? Yes .................................................... 1 [Go to TP9] No............................................... 2 [Go to TP10] Don't know...................................8 [Go to TP10] 38 TP8 Was (name) given a medication for fever or malaria during this illness? Yes .................................................... 1 No .................................. 2 [Go to TP10] Don't know...................................8 [Go to TP10] (*) SP = Falcidox, Malariadexin, Fansidar, Paludose, etc. (**)Combination with Artemisinin = Serenadose, Luther, Coartem, Co-arinate, Co-arsucam, Artemod, Arsumoon, etc. ␣ TP9 What medication was (name) given? Ask again: Any other medication? Circle all medications mentioned. If medications were given, write the names of all of the medications. __________________________________ (Medication Names) Antimalarials: SP(*)………………………………………... A Chloroquine ............................................ B Amodiaquine/Camoquin/Flavoquine ...... C Quinine................................................... D Combination with Artemisinin(**).............. E Antimalarial/community-based care facility ...F Other antimalarial (specify) ___________H Antibiotics Pills/Liquid.................................................I Injection .................................................. J Other medication Paracetamol/Panadol/Acetaminophen...... P Aspirin...................................................... Q Ibuprofen................................................ R Other (specify) _____________________ X Don't know .................................................Z (*) SP = Falcidox, Malariadexin, Fansidar, Paludose, etc. (**)Combination with Artemisinin = Serenadose, Luther, Coartem, Co-arinate, Co-arsucam, Artemod, Arsumoon, etc. TP10 Check TP6 and TP9: Antimalarials mentioned (codes A - H)? Yes.␣ 1 [Continue with TP11] No.␣ 2 [End of interview] TP11 How long after (name)'s fever began did he/she take (name of antimalarial mentioned in TP6 or TP9) for the first time? If more than one antimalarial was mentioned in TP6 or TP9, list all antimalarials mentioned. Write how long after the fever began was the first antimalarial given. Same day ..................................................0 Next day .................................................1 2 days after the fever began....................2 3 days after the fever began....................3 4 or more days after the fever began...... 4 Don't know ..............................................8 Module 15: Perceptions of quality of care Patient perceptions of quality of care QC Have you used local health services (health center or General Hospital) in the last 3 months? If yes, ask the following questions: Impression of care Not at all favor able Unfav orable Neutr al Favorab le Very favorabl e (1) (2) (3) (4) (5) QC1 Health workers' behavior and practices QC1.1 Show compassion and support for patients QC1.2 Show respect for patients QC1.3 Are friendly/welcoming to patients 39 QC1.4 Are honest QC1.5 Attentively listen to patients QC2 Appropriateness of resources and services QC2.2 The rooms are appropriate QC2.3 The waiting time is reasonable QC2.4 There are enough nurses QC2.5 Medications are available at all times QC3 Finances and cost of care QC3.1 Prices can be negotiated QC3.2 Prices are reasonable QC3.3 Have you seen treatment prices posted? QC3.4 Do you think that you paid the actual price that you should have paid? QC3.5 Medications can easily be obtained. QC3.6 The distance from the center is reasonable for us (not too far). QC3.7 Nurses take enough time for patients. 40 Facility Questionnaire (ENGLISH) Questionnaire: Availability of Services and Equipment N° Question: N° Question: 01. GPS location : 02. Data collector code: 03. Name of health structure _________________ 04. Facility code /____________/ 05. Name of supervision area 06. Supervision area code 07. Name of health zone 08. Health zone Code 09. Type of facility: • Health center • General Referring Hospital (GHR) 1 2 010. Province name and code • West Kasai • East Kasai • Katanga • South Kivu 1 2 3 4 011. Name and code of Supervisor 012. Urban/Rural: • Urban • Semi-urban • Rural 1 2 3 013. Respondent's professional category: • Physician • State Registered Nurse • State Registered Midwife Health Technician • Technical Health Agent • Physician in training/volunteer physician • Other trainee/volunteer • Other (specify): _________________ 1 2 3 4 5 6 7 014. Date and time of survey • Date: /____ /____/______/ • Time: /_____o'clock :_______/ minutes Guide to the survey on availability of services and equipment Locate the Head Nurse and the Health Center Director/ Head Physician of the General Hospital, and introduce yourself as follows: Hello. My name is ____________________. I represent IBTCI/CESD, a research organization working with USAID in cooperation with the Ministry of Health. We are conducting a survey on health facilities supported by USAID through MSH/IHP (Prosani), with the goal of identifying ways to improve services. We would like to interview you about the situation at this facility, and the availability of services and equipment. Be assured that our conversation will remain strictly confidential, and you will not be identified in any way. At any time, you may choose to stop the interview or refuse to answer a question. May I proceed? Yes... No… If no, go to the end of the questionnaire. Module 1: General Information (IG) on the Health Facility N° QUESTIONS CODE IG1. Simply observe and note Do you see a sign or poster showing the availability of the following services (circle all appropriate responses)? A) Family planning services. B) Infant health services C) Prenatal care D) STI/AIDS consultations E) Prices for services Yes, Yes, No OUTSIDE INSIDE 1 2 1 2 1 2 1 2 1 2 41 N° QUESTIONS CODE If prices are NOT posted: Skip to IG3 IG2. If yes, what is the posted price for: • Initial visit forms • Malaria case • Diarrhea case • Prenatal consultation • Childbirth • Family planning • Other (specify): __________________________ • Other (specify): __________________________ • Other (specify): __________________________ • Other (specify): __________________________ /______________/ FC /______________/ FC /______________/ FC /______________/ FC /______________/ FC /______________/ FC /______________/ FC /______________/ FC /______________/ FC /______________/ FC /______________/ FC /______________/ FC IG3. Is there a waiting room for patients? Yes No 1 2 IG4. How many days is this facility open to outpatients? (Outpatients are those who are receiving preventive or curative care and going home the same day). Number of days per week Number of days per month Don't know /______/ days /______/ days 98 IG5. Does this center provide care 24 hours per day? Yes No 1 2 IG6. At what time do outpatient care hours begin? /________ / IG7. At what time do outpatient care hours end? /________ / IG8. Do you have a continuous electricity source in the building? Yes No If no, go to IG11 1 2 IG9. If yes, is it: A generator A solar panel SNEL (public electricity) Other (specify): _____________________ 1 2 3 4 IG10. Do you have electricity today? Yes No 1 2 IG11. Do you have a continuous water supply inside the building? Yes No If no, go to IG14 1 2 IG12. If yes, is it: 42 N° QUESTIONS CODE A water hole An improved well REGIDESO (public water supply) A rainwater cistern Other (specify): 1 2 3 4 5 IG13. Do you have running water today? Yes No 1 2 IG14. Does the facility have an emergency communications system? Yes No If no, go to IG17 1 2 IG15. If yes, is it: A telephone Radiotelephone Motorola/VHF radio Internet A short wave radio Other (specify): 1 2 3 4 5 6 IG16. Does this equipment work? Yes No 1 2 IG17. Does the facility have access to a transportation method at all times? Yes No If no, go to IG19 1 2 IG18. What type of transportation method does the facility have at this time? A motorcycle A vehicle A bicycle Other (specify): 1 2 3 4 IG19. Are there toilets? Yes No 1 If no, go to IG21 IG20. If yes, what type of toilet/shower? Septic pit latrines Pit toilet Other (specify): 1 2 3 IG21. What disinfectant(s) is/are used in the health facility Cyteal (Chlorhexidine - gluconate) Dakin’s solution Bleach (Sodium hypochlorite/Chlorine solution/JIK solution) Denatured alcohol Other (specify): _________________ 1 2 3 4 5 IG22. Do you currently have disinfectants in stock? Yes 1 43 N° QUESTIONS CODE No 2 IG23. What procedure is used to decontaminate medical equipment after initial use? Soak in a disinfectant solution, then brush and scrub with soap and water Scrub with soap and water, then soak in a disinfectant solution Only scrub with soap and water Only soak in a disinfectant solution Clean with soap and water Equipment is never decontaminated Equipment is never reused Other (specify): ________________________________________________ 1 2 3 4 5 6 7 8 IG24. What procedure is used to sterilize the medical equipment before it is reused? Dry heat sterilization (Poupinel) Autoclave Boiling Steam sterilization Chemical method Other (specify): _____________________________ 1 2 3 4 5 6 IG25. Are there procedures for disposing of biomedical waste? Yes No Don't know 1 2 98 IG26. How does the facility dispose of waste? Incineration in an incinerator Outdoor incineration Burial Thrown outside Other (specify): ___________________________ 1 2 3 4 5 IG27. Data collector's comments IG28. Supervisor's comments Module 2: Personnel Statistics (SP) Professional Categories SP1: Number of available personnel currently working Number: Don't know=98; Not determined=9 9 a) Physicians b) Nurses A1 c) Nurses A2 d) Nurses A3 e) Certified birth assistants f) Lay midwives/Village birth assistants g) Community liaisons h) Laboratory technicians i) Nutritionists 44 j) Physiotherapists k) Other (specify) : ___________________________ Other (specify): ___________________________ Other (specify): ___________________________ Category SP2 : Sex: What services does (NAME) provide? SP7: Has (Category) received training during the previous 3 years? SP3. Family planning SP4. Prenatal/ postnata l care SP5. Child health SP6. STI/AID S a) Physicians M....1 F….2 Yes.... 1 No...2 Yes....1 No...2 Yes.... 1 No...2 Yes....1 No...2 Yes....1 No...2 [ to b)] b) Nurses A1 M....1 F….2 Yes.... 1 No...2 Yes....1 No...2 Yes.... 1 No...2 Yes....1 No...2 Yes....1 No...2 [ to b)] c) Nurses A2 M....1 F….2 Yes.... 1 No...2 Yes....1 No...2 Yes.... 1 No...2 Yes....1 No...2 Yes....1 No...2 [to d)] d) Nurses A3 M....1 F….2 Yes.... 1 No...2 Yes....1 No...2 Yes.... 1 No...2 Yes....1 No...2 Yes....1 No...2 [to e)] e) Certified birth assistants M....1 F….2 Yes.... 1 No...2 Yes....1 No...2 Yes.... 1 No...2 Yes....1 No...2 Yes....1 No...2 [to f)] f) Lay midwives/Village birth assistants M....1 F….2 Yes.... 1 No...2 Yes....1 No...2 Yes.... 1 No...2 Yes....1 No...2 Yes....1 No...2 [to g)] g) Community liaisons M....1 F….2 Yes.... 1 No...2 Yes....1 No...2 Yes.... 1 No...2 Yes....1 No...2 Yes....1 No...2 [to h)] h) Laboratory technicians M....1 F….2 Yes.... 1 No...2 Yes....1 No...2 Yes.... 1 No...2 Yes....1 No...2 Yes....1 No...2 [to i)] i) Nutritionists M....1 F….2 Yes.... 1 No...2 Yes....1 No...2 Yes.... 1 No...2 Yes....1 No...2 Yes....1 Non...2 [to j)] j) Physiotherapists M....1 F….2 Yes.... 1 No...2 Yes....1 No...2 Yes.... 1 No...2 Yes....1 No...2 Yes....1 No...2 [to i)] k) Other (specify) : __________________________ _ M....1 F….2 Yes.... 1 No...2 Yes....1 No...2 Yes.... 1 No...2 Yes....1 No...2 Yes....1 Non...2 [to j)] Other (specify): __________________________ _ M....1 F….2 Yes.... 1 No...2 Yes....1 No...2 Yes.... 1 No...2 Yes....1 No...2 Yes....1 No...2 [to i)] Other (specify): __________________________ _ M....1 F….2 Yes.... 1 No...2 Yes....1 No...2 Yes.... 1 No...2 Yes....1 No...2 Yes....1 Non...2 [to j)] Module 3: Preventive Care (PREV) 45 N° QUESTIONS CODES PREV1. How many outpatients (total) were seen at this facility in the past 12 months? (Total for the past 12 months for which data are available). Number Don't know /_______ / 98 PNC PREV2. Does this facility offer prenatal consultations (PNC)? Yes No 1 2 [Go to Prev20] PREV3. How many days per week are PNC patients seen at this facility? Number per week Number per month Don't know /_______ / /_______ / 98 PREV4. How many prenatal consultations were given at this health center in 2012? Number Don't know /_______ / 98 PREV5. In the past 6 months, how many meetings did this health facility hold with traditional birth assistants? Number Don't know /_______ / 98 PREV6. Throughout 2012, did you routinely prescribe iron and folic acid? Interviewer: verify written records. If there are no written records, ask the director. Record the responses, noting whether or not you verified the written records. Yes No Don't know 1 2 98 PREV7. Do pregnant women receive Vitamin A during prenatal consultations? Yes No 1 2 PREV8. Do women who come to the health facility for prenatal care receive prenatal or maternal health cards? If no, go to PREV10 Yes No 1 2 PREV9. Where is the maternal health card kept once it has been issued to the mother? It is given to the mother so that she can bring it to the next visit. 1 It is kept at the health facility. 2 One copy is given to the mother and the other copy is kept at the hospital. 3 Other (specify): __________________________ 4 PREV10.Can you show me the cards belonging to specific patients? Seen Not seen 1 2 PMTCT PREV11.During prenatal consultations, are women advised to be tested for HIV for PMTCT purposes? If no, go to PREV20 Yes No 1 2 PREV12.Does the center routinely offer the HIV test to pregnant women to prevent mother to child transmission? Yes No 1 2 PREV13.Does the center have personnel trained in PMTCT? Yes No 1 2 PREV14.How many pregnant women were screened for HIV in 2012? Number Don't know /__________/ 98 PREV15.What services does the center offer to pregnant women who test Referred to the General Hospital for treatment. 1 46 N° QUESTIONS CODES positive for HIV, and to their babies? Given cotrimoxazole. 2 Given family planning advice. 3 Children born to HIV-positive mothers given cotrimoxazole. 4 Given nutritional advice. 5 Other (specify): 6 PREV16.How many women who tested positive for HIV were given ARVs to prevent transmission of HIV to their babies? Number Don't know /__________/ 98 PREV17.How many babies born to HIV-positive mothers received anti-retroviral treatments for HIV prevention at birth? Number Don't know /__________/ 98 PREV18.Were the partners/husbands of pregnant women who tested positive for HIV asked to be tested? Yes No 1 2 PREV19.How many partners/husbands of women who tested positive for HIV agreed to be tested? Number Don't know /__________/ 98 CHILDBIRTH AND POSTPARTUM PREV20.Does this facility have a maternity unit? Yes No 1 2 PREV21.Is the maternity unit open 24 hours per day? Yes No 1 2 PREV22.Do the health facility's employees offer labor and delivery services in the health facility, in the community, or both? Only in the health facility. 1 Only in the community 2 In both the health facility and the community. 3 PREV23.How many deliveries did this facility's personnel attend in 2012? Number Don't know /__________/ 98 PREV24.Is the health facility capable of managing emergency caesareans? They can be managed now 1 They can usually be managed, but not now 2 They must be transferred 3 PREV25.For health centers, do you have written instructions that pregnant women can refer to in the event of obstetric complications? Yes No 1 2 [Go to Prev27] PREV26.If yes, how many women were referred to the General Hospital for obstetric complications in 2012? Number Don't know /__________/ 98 PREV27.Does this health facility offer postpartum care? Yes No 1 2 PREV28.Are the postpartum services offered only at specific times, during outpatient consultation hours, or both? Only at specific times 1 During outpatient consultation hours 2 Both at specific times and during outpatient consultation hours 3 PREV29.In 2012, on how many days were women allowed to have postpartum care? Number Don't know /__________/ 98 PREV30.What postpartum care is given to women? 1 2 3 4 5 47 N° QUESTIONS CODES PREV31.In 2012, how many clinical information sessions did the health facility have for postpartum clients? Number Don't know /__________/ 98 PREV32.In 2012, how many maternal deaths did the health facility have? Interviewer: verify written records. Number Don't know /__________/ 98 FAMILY PLANNING PREV33.Does this facility offer family planning services? (Family planning includes methods and advice for spacing or limiting births). Yes No 1 2 PREV34.How many days per week are family planning services available? Number of days per week Number of days per month Don't know /__________/ /__________/ 98 PREV35.How many family planning consultations took place at this health center in the past 12 months? Number Don't know /__________/ 98 PREV36.What family planning methods are given to women at this health center? Oral contraceptives 1 Condoms 2 Injections 3 IUD 4 Cycle Beads 5 Natural lactation method 6 Other (specify): _____________________ 7 CHILD HEALTH CARE PREV37.What types of child health care does this facility offer? (Preventive and curative infant care included.) Note all types of care Curative care 1 Preventive care 2 Health promotion care 3 PREV38.What types of preventive care does your facility offer? Immunizations 1 Growth and weight gain tracking 2 Other (specify): 3 PREV39.How many days do you offer preventive care for children? Number per week Number per month Don't know /__________/ /__________/ 98 PREV40.Is there a vaccine education plan in place for this year? Yes No 1 2 PREV41.How many of the following storage methods does this site have for vaccine storage? Refrigerator with a refrigerated section 1 Ice chest 2 Freezer 3 Vaccine carrier 4 Ice packs 5 None 6 PREV42.Are temperature variations recorded? Interviewer: If yes, ask to see these. Yes, seen 1 Yes, not seen 2 No 3 PREV43.In the past 7 days, on how many days was the temperature recorded? Number Don't know /__________/ 98 PREV44.In the past 7 days, how many times was the temperature recorded, in total? Number Don't know /__________/ 98 PREV45.In the past 7 days, on how many days did you record temperatures above 80C or below 20C? Number Don't know /__________/ 98 PREV46.Does every child who starts the vaccination program receive a vaccination card? Yes No 1 2 48 N° QUESTIONS CODES PREV47.Where are the vaccination cards kept after the vaccination program starts? They are given to the parent or guardian so that he/she can bring them to the next visit 1 They are kept at the health facility 2 One copy is given to the parent and the other copy is kept at the hospital. 3 Other (specify): ______________________________ 4 PREV48.Can you show me the cards belonging to specific patients? Seen Not seen 1 2 PREV49.Does this facility have a child vaccination unit? Yes No 1 2 PREV50.How many days per week are vaccination services available? Number Don't know /__________/ 98 PREV51.How many children received a first diphtheria/tetanus/pertussis (DTP1) vaccine in this health center in 2012? Number Don't know /__________/ 98 PREV52.In 2012, how many children received the full vaccination? Number Don't know /__________/ 98 PREV53.How many consultations for sick children took place in this facility in the past 12 months? How many children 0-11 months /_______ / How many children 12-23 months /_______ / How many children 24-59 months /_______ / How many children 0-24 months /_______ / How many children 0-59 months /_______ / PREV54.How many preventive care visits for children took place in this facility in the past 12 months? How many children 0-11 months /_______ / How many children 12-23 months /_______ / How many children 24-59 months /_______ / How many children 0-24 months /_______ / How many children 0-59 months /_______ / STIs PREV55.Does this facility offer sexually transmitted infection (STI) consultations? If no, go to PREV69 Yes No 1 2 PREV56.How many days are STI consultations available? Number per week Number per month Don't know /_______ / /_______ / 98 PREV57.What protocol does the center use for managing STIs? Etiologic method 1 Syndrome approach 2 Other (specify) __________________ 3 Don't know 98 PREV58.Did the center have a shortage of STI medications in 2012? Yes No 1 2 49 N° QUESTIONS CODES Don't know 98 PREV59.Does the center currently have medications for managing STIs? Yes No Don't know 1 2 98 PREV60.How many STI consultations took place in the center in 2012? Number Don't know /_______ / 98 HIV AIDS PREV61.Does this facility offer HIV/AIDS consultation services? Yes No 1 2 [Go to Prev67] PREV62.How many days are HIV/AIDS consultations available? Number per week Number per month Don't know /_______ / /_______ / 98 PREV63.How many HIV/AIDS consultations took place at this health center in the past 12 months? Number Don't know /_______ / 98 PREV64.Is HIV/AIDS prevention information given to the public? Yes No Don't know 1 2 98 PREV65.If yes, what information? Information about prevention Information about management Information about decreasing the effects of the disease 1 2 3 PREV66.What media are used? Posters Leaflets Video messages Audio messages Information provided during prenatal consultations and preschool health visits Community education activities Other (specify): __________________________ 1 2 3 4 5 6 7 PREV67.Does the facility have written instructions for handling rape and other gender-based violence? Yes No Don't know 1 2 98 PREV68.Does the facility have PEP (post￾exposure prophylaxis) kits? Yes No Don't know 1 2 98 MALARIA PREVENTION PREV69.Does the health facility have a malaria-prevention program involving distribution of insecticide-treated mosquito nets? Yes No 1 2 PREV70.If yes, to whom are the insecticide￾treated mosquito nets distributed? To pregnant women To women with children 0-23 months Other (specify): _________________ 1 2 3 PREV71.Are the insecticide-treated mosquito nets distributed for free, or sold? Distributed for free Sold 1 2 50 N° QUESTIONS CODES PREV72.Are the insecticide-treated mosquito nets currently in stock? Yes No 1 2 Module 4: Curative Care (SC) N° QUESTIONS CODES MALARIA, ARIs and HIV/AIDS SC1. Does this facility offer care/treatment for children with the following diseases? Malaria Yes No Don't know 1 2 98 Acute Respiratory Infections Yes No Don't know 1 2 98 HIV/AIDS Yes No Don't know 1 2 98 SC2. How many days is treatment offered to sick children? Number per week Number per month Don't know /_______ / /_______ / 98 SC3. Does the facility have a written protocol for treating childhood diseases? Yes No Don't know 1 2 98 SC4. If yes, verify that the document exists Exists, and is posted Exists, and is not posted 1 2 SC5. Have the personnel assigned to these treatments been trained in IMCI? Yes No Don't know 1 2 98 SC6. Does the facility have the required medications and supplies in stock? Yes No Don't know 1 2 98 SC7. Are the medications to treat these diseases provided to children within the facility itself, or do the parents have to obtain them from an outside pharmacy? They must be obtained from the facility's pharmacy They must be obtained outside the center It depends on the parents Don't know 1 2 3 98 SC8. Does the cost of treatment for these diseases include the consultation and medications? Yes No Don't know 1 2 98 SC9. How many consultations for sick children took place in this facility in the past 12 months? How many children 0-11 months /_______ / How many children 12-23 months /_______ / How many children 24-59 months /_______ / How many children 0-24 months /_______ / How many children 0-59 months /_______ / CHRONIC DISEASES SC10. What tuberculosis-related services does this health facility offer? Interviewer: A Tuberculosis Room is a space in the health facility where people who have tuberculosis can be Only services related to diagnosis 1 Only services related to treatment 2 Both diagnosis and treatment services 3 none 4 51 N° QUESTIONS CODES seen and treated. If none, go to SC15 SC11. Is there a room reserved for tuberculosis patients in the health facility? Interviewer: A Tuberculosis Room is a space in the health facility where people who have tuberculosis can be seen and treated. Yes No 1 2 SC12. Are people diagnosed with tuberculosis monitored at the community level? Yes No 1 2 SC13. Are people in contact with tuberculosis patients asked to be tested? Yes No 1 2 SC14. Is there a tuberculosis registry? If yes, ask to see it.. Yes No 1 2 SC15. Does the center have the document listing the Standards and procedures for reproductive health services, adopted in 1997? Yes No 1 2 [Go to SC17] SC16. May I see a copy of this document? Seen Not seen 1 2 SC17. How are people who test positive for HIV managed and treated? The General Hospital physician writes a prescription The facility only renews a prescription written by the doctor Nothing is done for these patients The facility does not do testing Other (specify): ___________________________________________ 1 2 3 4 5 SC18. Does the facility have written instructions for HIV testing, and for managing patients living with HIV? Yes No 1 2 SC19. Does the facility have written instructions for blood transfusions? If no, go to SC21 Yes No 1 2 SC20. Does the facility do blood transfusions? Yes No 1 2 SC21. If no, what does the facility do if a child or a pregnant woman needs a blood transfusion? Transfer the patient to the General Hospital Transfer the patient to another medical facility Don't know 1 2 98 SC22. Does the facility have written instructions for managing cases of the following diseases? : Leprosy Trypanosomiasis Lymphatic filariasis Onchocerciasis Schistosomiasis Verminosis 1 2 3 4 5 6 SC23. Does the facility perform minor surgery? Yes No 1 2 SC24. How many minor surgeries did you perform in 2012? Number Don't know /_______ / 98 SC25. Does the facility perform nutritional rehabilitation? Yes No 1 2 HEALTH PROMOTION ACTIVITIES SC26. Does the facility perform the following health promotion activities? SC27. Promoting the use of condoms Yes 52 N° QUESTIONS CODES No Don't know SC28. Hygiene and sanitation Yes No Don't know SC29. Exclusive breastfeeding of children for the first six months of life Yes No Don't know SC30. Food hygiene/food safety Yes No Don't know SC31. Consumption of iodized salt Yes No Don't know SC32. Improvement of latrines Yes No Don't know SC33. Promotion of oral rehydration for diarrhea in children Yes No Don't know SC34. Information on fistula prevention Yes No Don't know Module 5: Supervision (SUP) N° Question: Responses Code SU1. A supervision visit is a visit from a Ministry of Health representative who comes to observe the facility in order to help personnel improve services. When was your facility's last supervision visit? Last month During the last three months During the last six months More than six months ago Never [Go to module 6] Don't know [Go to module 6] This month 1 2 3 4 5 6 7 SU2. What took place during this supervision visit? CIRCLE ALL RESPONSES MENTIONED. ASK QUESTIONS. Anything else? Review of files/reports Meetings Inspection/delivery of equipment Observation of patient consultations Discussion of problems Discussion about personnel Other________________________ Nothing Don't know 1 2 3 4 5 6 7 8 98 Module 6: Equipment and Materials (EM) I would like to ask you a few questions about this facility's equipment and materials: N° Question: Responses Code EM1. What method is most often used for high-level disinfection or sterilization of medical equipment and Hot plate Steam sterilizer (steamer) 1 2 53 N° Question: Responses Code supplies? Chlorhexidine Bleach Other None Don't know 3 4 5 6 98 EM2. How do you dispose of your contaminated syringes and sharps? (CIRCLE ALL RESPONSES MENTIONED) Incineration Burial Trash Reuse Septic pit Special pit Other____________________ Don't know 1 2 3 4 5 98 EM3. When did you last inventory your medications, equipment and supplies? Month . . . . . . . Year. . . . . . Don't know /_________ / /__________/ 98 EM4. Where does your facility generally obtain medications and supplies? Government supplier Private supplier International supplier/NGO Central Office for the health zone Other____________________ Don't know 1 2 3 4 5 98 EM5. Do you sometimes experience delivery delays for medications and supplies? Yes No 1 2 [EM7] EM6. What is the most frequent cause of delivery delays for medications and supplies? Inadequate transportation Fuel shortage Administrative difficulties Shortage of personnel Financial problems Central storage location depleted Other ___________________ Don't know 1 2 3 4 5 6 7 98 EM7. Where does your facility generally procure or receive contraceptives? Government supplier Private supplier International supplier/NGO Central Office for the health zone Other____________________ Don't know 1 2 3 4 5 98 EM8. Do you sometimes experience delivery delays for contraceptives? Yes No 1 2 [Go to module 7] EM9. What is the most frequent cause of delivery or collection delays for contraceptives? Inadequate transportation Fuel shortage Administrative difficulties Shortage of personnel Financial problems Central storage location depleted Other ___________________ 1 2 3 4 5 6 7 54 N° Question: Responses Code Don't know 98 Module 7: Inventory of Supplies and Medications for Reproductive Health ASK QUESTION No. 530 FOR EVERY ITEM. IF IT IS NOT AVAILABLE, GO TO THE NEXT ITEM ITEM 530 Do you have an inventory card for (item)? 531 Is (item) stored according to expiration date? 532 Are the (items) protected from rain, sun, harmful temperatures, rats and other animals and harmful insects? a) Contraceptives Yes....1 No...2 [Go to b)] Yes ....1 No....2 Yes ....1 No....2 b) Medications for treating STIs Yes....1 No...2 [Go to c)] Yes ....1 No....2 Yes ....1 No....2 c) Vaccines Yes....1 No...2 [Go to d)] Yes ....1 No....2 Yes ....1 No....2 d) Other medications Yes....1 No...2 Yes ....1 No....2 Yes ....1 No....2 ASK QUESTION No. 530 FOR EVERY ITEM. IF IT IS NOT AVAILABLE, GO TO THE NEXT ITEM ITEM 530 Do you have an inventory card for (item)? 531 Is (item) stored according to expiration date? 532 Are the (items) protected from rain, sun, harmful temperatures, rats and other animals and harmful insects? a) Contraceptives Yes....1 No...2 [Go to b)] Yes ....1 No....2 Yes ....1 No....2 b) Medications for treating STIs Yes....1 No...2 [Go to c)] Yes ....1 No....2 Yes ....1 No....2 c) Vaccines Yes....1 No...2 [Go to d)] Yes ....1 No....2 Yes ....1 No....2 d) Other medications Yes....1 No...2 Yes ....1 No....2 Yes ....1 No....2 Module 8: Availability of Family Planning Methods and Vaccines Now I would like to ask you a few questions about the family planning methods and vaccines that are available at this facility. After these questions, I will need to see your inventory of contraceptives and vaccines. ASK QUESTION No. 533 FOR EACH FAMILY PLANNING METHOD OR VACCINE. IF IT IS NOT AVAILABLE, GO TO THE NEXT METHOD OR VACCINE. METHOD/VACCINE 533 Is (method/vaccine) currently available in this facility? 534 Have you had a shortage of (method/vaccine) or were you unable to 535 VERIFY THROUGH VISUAL INSPECTION: DID YOU SEE TWO 55 offer it during the past 6 months? UNEXPIRED UNITS OF (METHOD/VACCINE)? a) Combination birth control pill (Lo-femenal) Yes....1 No...2 [Go to b)] Yes ....1 No....2 Seen ........1 Not seen....2 b) Progesterone-only birth control pill (Ovrette) Yes....1 No...2 [Go to c)] Yes ....1 No....2 Seen ........1 Not seen....2 c) Injection (Depo-provera) Yes....1 No...2 [Go to d)] Yes ....1 No....2 Seen ........1 Not seen....2 d) IUD kit Yes....1 No...2 [Go to e)] Yes ....1 No....2 Seen ........1 Not seen....2 e) Spermicide Yes....1 No...2 [Go to f)] Yes ....1 No....2 Seen ........1 Not seen....2 f) Condom Yes....1 No...2 [Go to g)] Yes ....1 No....2 Seen ........1 Not seen....2 g) BCG vaccine Yes....1 No...2 [Go to h)] Yes ....1 No....2 Seen ........1 Not seen....2 h) Polio vaccine (OPV) Yes....1 No...2 [Go to i)] Yes ....1 No....2 Seen ........1 Not seen....2 i) DTP vaccine Yes....1 No...2 [Go to j)] Yes ....1 No....2 Seen ........1 Not seen....2 Measles vaccine Yes....1 No...2 Yes ....1 No....2 Seen ........1 Not seen....2 Module 9: Availability of IEC Materials Do you currently have educational materials on family planning, maternal and infant health and STIs/AIDS? SERVICE 601 Image box 602 Brochures 603 Posters a) Family planning Yes.............1 No............2 Yes.............1 No............2 Yes.............1 No............2 b) Prenatal/postnatal care Yes.............1 No............2 Yes.............1 No............2 Yes.............1 No............2 c) Safe motherhood (childbirth) Yes.............1 No............2 Yes.............1 No............2 Yes.............1 No............2 d) Prevention/treatment of HIV/AIDS Yes.............1 No............2 Yes.............1 No............2 Yes.............1 No............2 e) Prevention/treatment of other STIs Yes.............1 No............2 Yes.............1 No............2 Yes.............1 No............2 f) Maternal nutrition Yes.............1 No............2 Yes.............1 No............2 Yes.............1 No............2 g) Monitoring children's nutrition and weight Yes.............1 No............2 Yes.............1 No............2 Yes.............1 No............2 h) Breastfeeding Yes.............1 No............2 Yes.............1 No............2 Yes.............1 No............2 i) Prevention of diarrheal illnesses Yes.............1 No............2 Yes.............1 No............2 Yes.............1 No............2 j) Acute respiratory infections Yes.............1 No............2 Yes.............1 No............2 Yes.............1 No............2 k) Malaria Yes.............1 No............2 Yes.............1 No............2 Yes.............1 No............2 l) Vaccination Yes.............1 No............2 Yes.............1 No............2 Yes.............1 No............2 56 m) Vitamin A Yes.............1 No............2 Yes.............1 No............2 Yes.............1 No............2 n) Adolescent reproductive health Yes.............1 No............2 Yes.............1 No............2 Yes.............1 No............2 n) Men's reproductive health Yes.............1 No............2 Yes.............1 No............2 Yes.............1 No............2 During the interview, the respondent consulted written records and reports: All the time 1 Sometimes 2 Rarely or never 3 Final result of the survey on availability of services and equipment • Complete • Partially complete • Refused • Authorized respondent not found • Facility not found • Other (specify): ___________________ 1 2 3 4 5 6 Interviewer's comments: Supervisor's comments Time completed: /____ /____/______/ 57 ANNEX H: Training Manual for Data Collection (English Translation) I. OVERVIEW OF THE INTEGRATED HEALTH PROJECT (IHP) AND THIS EVALUATION The Integrated Health Project (IHP in English, PROSANI in French) is five-year project, financed by the United States Agency for International Development (USAID) in support of the DRC's National Health Development Plan (NHDP). The project's objective is to create and support an environment conducive to the supply of healthcare services, products and practices in the 80 targeted health zones located in the four provinces of Kasai Occidental, Kasai Oriental, Katanga and Sud-Kivu. The IHP project was begun in October 2010 and will be completed at the end of September 2015. This project is supported by the Management Sciences for Health (MSH) consortium and its partners (International Rescue Committee and Overseas Strategic Consulting Ltd) in the amount of $139,767,129. The IHP has two components. Component 1 corresponds to the first strategic pillar on which the DRC's National Health Plan is focused. Specific emphasis is given to strengthening health zone capacity to be able to supply healthcare services such as family planning, maternal, neonatal, child health, nutrition, malaria, tuberculosis, neglected tropical disease, HIV/AIDS, water, sanitation and hygiene by focusing on the services offered as well as the demand for these services. The objectives of Component 1 are: 1) improve access and availability of the main healthcare services within the Minimum Package of Activities and the Complementary Package of Activities-Plus, (MPA-plus/CPA-plus); 2) improve the quality of MPA-Plus/CPA-Plus services; and 3) improve the knowledge and practices that lead toward behavior that encourages good health. In July 2011, within the IHP evaluation strategy framework, the NGO Management Science for Heath (MSH) and its partners conducted the first part of the baseline household survey on knowledge, practices and coverage in the main health zones. Component 2 of the project corresponds to the second strategic pillar within the DRC's National Health Plan’s six priority areas: development of human resources; drug management; healthcare funding; infrastructure construction/repairs; equipment and new technologies; and improved health system management. The expected result for this component is improved leadership and improved governance as well as improved resource supply. The baseline evaluation and impact evaluation of the IHP program's RBF component will provide a large volume of statistical data that will permit the measurement of the baseline and impact of the RBF interventions. To achieve statistically significant results, all of the sites selected (health areas and villages) must be included in the two evaluations. USAID has contracted with IBTCI and its partners to conduct an IHP performance evaluation and an impact evaluation for the Results-Based Financing (RBF) pilot project which will be implemented by IHP in some of the health zones. 58 The recipients (or audience) of this IHP performance evaluation are the USAID Mission in DRC and the implementing entity (MSH and partners). USAID will use the results to modify its current integrated health strategy and share lessons learned with other partners and implementing entities. An executive summary with recommendations will be provided to the Ministry of Health. II. OBJECTIVES AND EVALUATION QUESTIONS The goal of this study is to establish units of measure that will serve as the basis for evaluating IHP achievements and efficiency. The study will provide data and information on the actual status of access, use and the priority health services offered as well as the quality of services offered in health centers and general reference hospitals within the zones included in the IHP. The following priority healthcare services are: 1) family planning, maternal health, neonatal health, child health; 3) nutrition, malaria and tuberculosis; 4) HIV/AIDS; and water, sanitation and hygiene. The evaluation is specifically focusing on the following objectives: 1. Documenting data on the availability and use of the main family health services: the Minimum Package of Activities (MPA-plus) and the Complementary Package of Activities-Plus (CPA-Plus) at Health Centers (HC) and at Reference General Hospitals (RGH) for (female) patients of childbearing age (15-49 years), for children aged 0-23 months, children aged 24-59 months, and for pregnant women, within four provinces: Kasaï Orientale, Kasaï Occidentale, Katanga, and Sud Kivu; 2. Evaluating the quality (including patient satisfaction) of the main health services offered: The Minimum Package of Activities (MPA-plus)/ Complementary Package of Activities-Plus (CPA-Plus) in Health Centers (HS), and Reference General Hospitals (RGH); 3. Evaluating system management and planning capacity for health service offerings in the targeted health zones in four provinces; 4. Evaluating obstacles and bottlenecks (including beliefs, fears and perception) within the framework of using information services, educational services and communication services (SIEC); Evaluating community support and information awareness, communication and supply of main family health services offered, and, 5. Using performance indicators to identify IHP supervision areas that perform well and those that perform poorly. A. Questions Related to the IHP Performance Evaluation The performance evaluation will allow USAID/DRC to determine which project components and aspects are successful and why they are successful, and what challenges the project faces, so as to be able to make modifications or corrections at the mid-term. To respond to the evaluation objectives above, USAID/Kinshasa have come up with specific questions for each of the two evaluations. The quantitative surveys (for which you are responsible for collecting information) will provide responses to the numerous evaluation questions, but not to all of the questions. Team leaders and experts will conduct a survey of key informants to obtain responses to certain of the evaluation questions which are the most important for a qualitative evaluation. Furthermore, to contribute to the qualitative component, certain supervisors will be invited to facilitate focus groups with members of civil society organizations, health service providers and/or women of childbearing age and mothers of young children who are the direct beneficiaries of the IHP. 59 The six following questions were specifically designed for the IHP Performance Evaluation: 1. To what extent has the project improved the access and availability of MPA-plus and CPA￾plus services in the target health zones? 2. Has the project improved the quality of main family health services in the target zones? 3. Definition of the indicator for the use of curative services. Has there been an improvement in the knowledge, attitudes, practices and positive behaviors related to health in the target health zones? 4. Have leadership and governance in the health sector improved in the target provinces? 5. What are the external factors impeding IHP activities and which impede achieving the best possible results? 6. What is the perception of the IHP and what is its reputation? B. Evaluation of Results-Based Financing Component The RBF component of IHP has not yet been launched. IHP project staff are waiting for RBF baseline data to be collected before implementing it. The RBF component is a pilot project for a new intervention that will provide financial incentives to the Health Zones, to the Ministry of Health's offices and to healthcare establishments (hospitals and health centers). These financial incentives will be awarded every quarter and based on the performance of participating health zones, the Ministry of Health management team and the participating health facilityies. There are specific indicators and related objectives that they should achieve. The Health Zone managers and civil society organizations involved in the IHP will evaluate data quality every quarter to verify the reliability and validity of the results shared by the healthcare establishments and Health Zone managers. There are no further RBF-related interventions. There are eight RBF health zones which must pursue the same goals and objectives as all the other IHP Health Zones. The only difference is that the RBF Health Zones will receive a monetary reward for performing well. If the RBF intervention is considered a success, USAID plans to extend this intervention to the other health zones. The RBF impact evaluation will contribute to USAID and the Ministry of Health's understanding of what works with regard to RBF and its effectiveness in increasing the quantity and quality of healthcare services. The specific evaluation questions for the RBF component of the intervention are as follows: 1. Is there qualitative and quantitative proof of change regarding services provided by health centers that are attributable to RBF? 2. What differences have resulted from the RBF component of the intervention? 3. Does the RBF model merit being expanded to other health zones? 4. What are the costs related to possible replication of the RBF Model? 5. Were the expected results achieved? 6. Do the results vary from one group to another? 7. What are the factors that contributed to limiting the expected results or the constraints that limited the expected results? 8. What are the unexpected consequences of this intervention? 60 III. TRAINING OVERVIEW • The general objective of the training workshop is to provide supervisors, data collectors and interviewers with the skills required to conduct the surveys in an efficient and reliable manner. • Training is an important part of the survey preparation; it ensures the accuracy and reliability with which data collection is carried out, the data entry procedures, the data analysis and the precise nature of the survey results. This is why all personnel involved in data collection, supervision or data entry must be trained to ensure the reliability and precision of data collection, filling out data collection forms and transferring data using an appropriate method. • The training also promotes awareness among survey personnel on the importance of generating quality data. Group training allows for a common understanding of the terms and definitions used in the survey, as well as the procedures used during data collection via different survey tools and different approaches. The specific objectives of the training are: • Understanding the context and the basis for the IHP performance evaluation and the RBF impact evaluation; • Discussing the general data collection process within the IHP framework; • Encouraging participants to become familiar with the data collection tools; • Applying the required techniques and efficient use of the tools; • Developing and understanding the field implementation plan; and • Clarifying logistical challenges related to field work. The trainer's pre-workshop responsibilities • Most importantly, become well-familiar with all aspects of the survey methodology; this requires detailed study of the survey manual and its tools; organize training-related logistics, including pilot survey data collection; • Plan the training schedule and the manner in which each session will be organized; Confirm that there is logistical support and transportation available for the pilot survey location; • Organize meals and refreshments for the training and lodging as needed for participants who do not reside in the area. IV. ROLE OF INTERVIEWER AND DATA COLLECTOR A. Interviewer responsibilities The survey will be based on interviews with healthcare personnel, mothers of children aged between 0 to 23 months, mothers of children aged between 24 to 59 months and pregnant women: • interviews with mothers after consultations • interviews with healthcare personnel about equipment and supplies in the facility, essential drugs, etc. • interviews with mothers of children aged 0 to 23 months within the household. It is important to note that the scientific value of the information collected during these surveys depends in large part on the interviewer's skill. The role of the interviewer is key to the success of this project. For that reason, the interviewer must: • Follow exactly the instructions on completing the questionnaire; 61 • Conduct interviews only with the individuals targeted by the survey. Do not change the way questions are asked; • Provide context for the survey so that questions are well understood and do not suggest responses when it is not required; • Ask all questions in the order in which they are provided on the questionnaire; • Use clear, simple and concise language and do not ask superfluous questions when it is not necessary. B. How to conduct an interview An interview engages an individual's memory. Therefore, it is important to be tactful and to ask questions clearly and in an unambiguous fashion. To be able to conduct an interview well is an art; it is not solely a technical process. The interview should be made as interesting and as positive an experience as possible. a) Establishing a rapport with the respondent 1) Make a good first impression • Make the respondent feel at ease when you meet him or her for the first time. Choosing your words wisely can make all the difference. • The physical appearance of the interviewer should inspire confidence in the individual being surveyed (well dressed, be able to present mission orders (if needed), polite language and attitude, etc.) • Smile (if needed) at the beginning of the interview and begin by saying, "Hello," and then introduce yourself. 2) Positively address the topic at hand 3) Emphasize the confidential nature of the responses 4) Reply to questions in a clear and unambiguous manner 5) Know how to deal with a situation when faced with resistance b) Tips for conducting a successful interview • Remain neutral during the interview • Never suggest answers to the respondent • Do not change the text or the order of the questions • Treat hesitant respondents tactfully • Do not rush the interview • Do not let the respondent distract you Do not fall into the trap of responding in detail to questions asked by a talkative respondent. Instead politely ask him or her to continue the interview and agree to address the other issues after it is complete. C. Qualities of a good interviewer a) Ethical qualities • To perform his/her work, the interviewer must be a strong professional to avoid the following: o Filling out the questionnaires him/herself with assumed responses, specifically when the interviewer is not able to find the individual to be interviewed and doesn't want to make the effort; o Filling out the questionnaire haphazardly when responses should be written down with precision; o Suggesting responses to individuals who are indecisive to save time or to slant survey 62 results to reflect his or her personal opinion. b) Social qualities With a natural and confident manner, the interviewer should be courteous, appropriately dressed and tactful. S/he should also strive to be friendly and straight forward, and not give the person being interviewed the unpleasant impression that he or she is being interrogated; rather, the interview should be a friendly conversation. The interviewer should avoid showing any boredom or annoyance. c) Technical Criteria • There are four technical criteria: o Sufficient knowledge of the survey methodology to be able to respond to certain questions or objections that the individuals being interviewed may express; o A thorough understanding of how to ask questions; o Sufficient knowledge of the survey subject matter and its purpose; o The interviewer should be familiar with all terms and be able to explain them properly, if needed, and should also be sufficiently knowledgeable about the purpose of the survey to be able to explain it to the individuals being surveyed. D. Expected behavior of interviewers and data collectors All field personal are expected to diligently collect high-quality information and do so in quantities that are sufficient to avoid compromising the technical aspects of this study. For example, when data is of poor quality, such results will not be used in the study. If we collect less data than expected, we will not be able to achieve statistically significant data. Therefore, we ask that you pay specific attention to these types of details. Each day, collect as much data as possible. Honesty: We understand that, from time to time, you will make mistakes and that is normal. However, we do expect you to provide transparency. Inform your supervisor of any mistakes that have been made. Your supervisor will be able to help you resolve the problem. In summary, honesty is required. If we find out that you are not honest, you may be removed from the team and we will replace you with someone else. Honesty also applies to the data that you collect. For example, if you tell your supervisor that you are collecting data in the village that has been assigned to you, but in reality, you collect data in a different village, we will find out because you are required to activate the GPS locator for each questionnaire. If we discover that you have been dishonest, you will receive a warning from your supervisor. Completeness: There will be situations where you will be unable to complete the survey questionnaire. In this case, you should record the information that you have collected. But you must return to that location as soon as possible to complete data collection. In the space provided for comments, please note the reason for the interruption. If you cannot fully fill out the questionnaire, note as well the efforts that you have taken to try and complete the questionnaire and also note why the questionnaire was not fully completed. The interviewers and data collectors who repeatedly provide incomplete responses will receive a warning from their supervisor. The supervisor will offer advice on how to be more efficient. Respect: For our team, treating everyone with respect is crucially important. You must be respectful to your supervisor. If you disagree with your supervisor’s advice, you are welcome to express your opinions and suggestions. However, if you cannot come to an agreement with your supervisor on a specific question, you must abide by your supervisor’s decision. If you do not respect your supervisor's guidance, you will 63 receive a warning. Even if rare, it is possible that unresolved problems may exist between you and you supervisor. If you firmly believe that an issue merits attention and that your supervisor is not taking the required steps, you are welcome to bring this issue to the attention of the field coordinator or experts. However, we ask that you try to resolve the problem at the supervisory level and not bring the question to a higher level except when you think that it is absolutely necessary to do so. Respecting our respondents is crucial. You represent our institutions and we want you to display a professional attitude. The survey questionnaire cannot be administered without having read the informed consent to the patient and without having received the patient’s voluntary consent. A respondent is free to refuse to participate in the study for any reason. The respondent is also free to terminate an interview when it is ongoing. Every time that a respondent refuses to participate or asks that an ongoing interview be terminated, you are asked to clearly explain the reasons for this in your notes and to do so in French. If a respondent complains about a lack of respect that you have shown, you will receive a warning. Termination: Any interviewer or individual collecting data who has received three warnings will be asked to leave the team and will be replaced by another interviewer or data collector. V. THE IMPORTANCE OF QUALITY DATA A. Why is the quality of the data important? It is normal to have issues when collecting and entering data. Validating and cleaning up data is indeed a long process, but it is essential for reliable results. We would like to bring to your attention the following fundamental reasons why we encourage recording high-quality data: • We need reliable and valid data to support the evaluation conclusions as well as the recommendations that we formulate. These recommendations may result in modifications being made to IHP interventions and its related work plan. If we base our recommendations on inaccurate data, this could cause cost overruns, and not only in financial terms; it could also result in Congolese citizens not receiving the quality health care that they need and to which they have a right. • Our detractors and opponents will look for weaknesses in the survey methodology and results. If we have poor quality data, our results will be rejected and we will lose credibility with USAID and our colleagues. • The results will be accessible to the public on the Internet and can be used by others, to, for example, make international comparisons or in new research. If we report inaccurate data, this could affect other researchers and, even, possibly, the beneficiaries of international aid in the DRC and other countries. • Future policy decisions may depend on the information generated from these surveys. If the policy is based on erroneous conclusions and recommendations, Congolese citizens may be deprived of quality healthcare. 64 B. Problems related to data collection • Incorrect costs of services provided: Health facilities should provide a list showing the costs of each type of service provided. The costs should be decided on in collaboration with the district-level government and with the CODESA. Considering that the cost of services is generally a sensitive subject, we ask that you verify the data that you collect from other sources, using triangulation. When there are discrepancies, please note them in writing, in French, in your survey tools in the section reserved for comments. • Unreadable or incomplete data collection forms, or both: Since our data will be gathered with cell phones using MAGPI software, this problem should be limited. However, we must still be able to read your notes. We require, therefore, that all information be written in French. • Numeric data entry errors: Incorrect numbers and other data may be incorrectly entered on the questionnaire. Again, MAGPI will prevent certain errors, but not all (for example, incorrect numbers as responses, extra or missing zeros, etc.). Please verify your work and specifically when entering numbers in the survey tools. • Translation from French to local languages: MAGPI tools are only programed in French. The three survey tools will be translated into three local languages (Swahili, Tshiluba and Lingala), but only on paper. You should read the questions to the respondent in the local language, unless the respondent speaks French fluently. You will enter their responses directly into MAGPI during the interview. C. Data entry errors Errors committed during data collection must be verified and corrected or omitted from the results. The quality of data is more important than the quantity of data. No purpose is served by having a large volume of data that is of poor quality. In certain surveys, a lot of erroneous or unreliable data had to be excluded from analysis. Sometimes, new data collection was required. Unreliable data compromises overall survey results and is a waste of precious resources in terms of the time and effort spent to collect data that could not be used. The problems related to data that are currently seen in surveys are attributable to several issues: • Supervisors, data collectors and interviewers have been insufficiently or poorly trained. • Field work has not been properly carried out (inadequate supervision, lack of attention to detail by data collectors or by interviewers, lack of quality control before submitting filled out forms, lack of understanding instructions, etc.) • The data was not verified at each step in the survey process. • A data verification function was not used or suspect values were not verified. • Human error. Problems related to data can, therefore, be avoided. To do so: • carefully study the interviewer's manual and tools at each stage and carefully follow instructions. • choose an individual who is capable and reliable and make sure that this individual has been well￾trained in survey methodology. • encourage personnel to openly communicate uncertainties related to survey procedures or suspect data. • verify data collection forms for accuracy and completeness after each data collection visit, at the end of each day of working in the field and before entering data. • conduct random testing on data entered into the tools (to identify unusual or outlying results that 65 require verification). D. Quality control To guarantee quality data, supervisors will conduct random evaluations of data quality. Every day, the supervisor will set a goal of re-interviewing 5% of the total data gathered during the day. For example, the supervisor will randomly select households and health facilities to re-administer some sections of the questionnaire. Each time, the supervisor will select different sections. The supervisor will compare his/her results with the results that were gathered by the interviewer and data collectors. If data collectors or interviewers are claiming to collect data that, in reality, is not being collected, and/or if it is discovered that the data collection has many errors, the data collectors or interviewers will receive a warning. VI. SAMPLE SELECTION A. RBF Impact Evaluation: Three types of survey will be organized: • Household surveys • Interviews and Focus-groups with Key Informants: MPH, MSH, stakeholders, CODESA, community leaders, etc. • Collection of program data and results from healthcare facilities (HC and RGH) Sampling Plan Only RBF Health Zones are relevant (zones where thes intervention is). In addition to these 8 Health Zones programmed for RBF, there will be 8 additional Health Zones selected near to the Health Zones where intervention is occurring (comparison zones). Selecting households to survey: The methodology has four (4) stages: 1. For each Health Zone where intervention is occurring, also choose a Health Zone for comparison. (completed) 2. For each of the 16 Health Zones involved, choose the villages in which the survey will be organized: random selection that takes into account each village's demographic weight. (completed) 3. For each village selected, randomly select the first household to be surveyed. (to be completed in the field) 4. Within each household, identify if the eligible individuals (women with children aged 0 to 23 months) are present. (to be completed in the field) The protocol for selecting households and individuals to be surveyed has two stages: 1) the selection of the first household, and the selection of the eligible individual. Selecting the 1st household: 1st Scenario: If the list of all village households is available:  Assign a number to each household  Calculate the SAMPLING INTERVAL (number of households to be surveyed divided by the number of total households)  Choose the first household at random  Follow the sampling interval to identify other households 66 2nd Scenario: If the number of households in the village is fewer than 19 (our sample per village): All eligible households where a mother resides with children between 0 to 23 months will be surveyed 3rd Scenario: If the village has too many households and it is difficult to count them:  Quickly estimate the number of households  Subdivide the village into sections so that each section has about the same number of households  Randomly select a section and follow the instructions outlined in Scenario 1 Selecting the individuals to be surveyed within the selected households Once a household has been selected: • Ask if a child between 0 to 23 months lives there with his/her mother. If Yes, this household has been selected as the first one in the survey; the number of eligible individuals are then counted and the interview process is begun with the mother of the youngest child aged 0 to 23 months. • If the household does not have a child between 0 to 23 months residing with his/her mother, the household that is closest to the one initially selected is contacted next. In this new household, the same question is asked. If an eligible individual is present, the interview is begun, or, again, the closest household is contacted. This process is continued until an eligible first household is found. • Once the first eligible household has been found and the interview completed, the interviewer returns to the first household selected at random to apply interval sampling and identify the second household. In the second household, the eligible person will be located and the process will proceed as with the first household. The process will continue until the entire defined sample has been completed. • If all the village's households are visited and a sample is not obtained (the predetermined number of households), the closest village is to be visited next and the sampling interval continued by beginning with the first household (the household closest to the previous village). • If two children are eligible within the same household, priority will be given to the mother with the youngest child. Then, the interviewer will ask questions about the other child by using the appropriate questionnaire. VII. DATA COLLECTION TOOLS A. QUESTIONNAIRE TYPES 1. HEALTH FACILITY DATA COLLECTION QUESTIONNAIRES: HEALTH CENTERS AND GRH This tool is for health facility data collectors, and specifically in Health Centers and General Reference Hospitals. It will also be used to evaluate both IHP performance and RBF impact. Within this questionnaire, there are the following modules: Module 1: General information about the health facility (GI) Module 2: Personnel statistics (PS) Module 3: Preventive care (PREV) Module 4: Curative care(CC) Module 5: Supervision (sup) 67 Module 6: Equipment and materials (EM) Module 7: Inventory of materials and medication for reproductive health Module 8: Availability of family planning methods and vaccines Module 9: Availability of materials (IEC) When this tool is being used, the data collectors will ask questions of the Head Nurse or designated representative. For certain information, the use of documentation is required (NHIS report, inventories, etc.) or even direct observation (for example, displayed tariffs). 2. HOUSEHOLD QUESTIONNAIRE This questionnaire is used to evaluate RBF impact and for interviewing mothers of children between 0 to 23 months within households. The women surveyed are those who have been found in eligible households, selected at random with strict adherence to the methodology described above. Within this questionnaire, there are the following modules [sic. The interviewer must read the woman the information on informed consent and receive her consent before beginning the interview. It is important for the interviewer to put the woman at ease. This questionnaire is subdivided into the three follow sections. The first section is for mothers of children between 0 to 23 months (modules 1 to 12). The second section is for mothers of children between 24 to 59 months (modules 13 to 14). The third section is for all of the mothers (module 15). Within this questionnaire, there are the following modules: Module 1: Household residents Module 2: Water and sanitation Module 3: Contraception Module 4: HIV/AIDS Module 5: Maternal health Module 6: Disease symptoms Module 7: Contact with healthcare services Module 8: Vitamin A supplements Module 9: Initial breastfeeding Module 10: Insecticide-treated nets Module 11: Treatment of diarrhea Module 12: Treatment of suspected pneumonia Module 13: Treatment of malaria Module 14: Responding mother's behavior regarding children's health Module 15: Perception of healthcare quality The interviewer must read the woman the information on informed consent and receive her consent before beginning the interview. It is important that the interviewer put the woman at ease and avoid her being influenced by health center personnel. B. INSTRUCTIONS FOR FILLING OUT QUESTIONNAIRES 1. Asking questions It is important to ask each question exactly as it is written in the questionnaire. It is for this reason that, when you ask the question, you should confirm that the person being interviewed heard the question clearly and had no difficulties understanding. At times, you will need to repeat the question to be sure 68 that the person being interviewed has understood. In such cases, do not paraphrase the question—repeat it as written. If after having repeated the question, the respondent still does not understand, ask the question again in a different way. When you change the way you ask the question, however, be careful to not change its original meaning. In some cases, you will need to probe—asking additional questions to obtain a complete response. If that is the case, be careful that your probing is "neutral" and does not suggest any particular response. Such probing requires a significant amount of tact and diplomacy and will be one of the more challenging aspects of your work. 2. Recording responses There are two types of questions: (a) closed questions (coded), for which a set of responses are defined in advance and are assigned a code and (b) open questions, the questions for which the responses are not defined in advance. a. Closed questions (Coded questions) In the questionnaire, there is a list of responses to coded questions. To record the response given, you will simply need to circle the number (code) that corresponds to the response. Example: GI1. Have arrangements been made for the disposal of biomedical waste? Yes No Don't know 1 2 98 If no arrangements have been made for the disposal of biomedical waste, "click NO" There are questions for which multiple responses are possible. In such cases, all responses given by the respondent must be circled. What types of health care services does this facility provide to children? (Preventive and curative pediatric care, included.) Note all types of care provided. Curative care 1 Preventive care 2 Promotional care 3 If the center offers more than one type of care, "click all responses given" b. Open questions (not coded) The responses to certain questions cannot be predefined. To record the responses to this type of question, you must write the response in the space provided, rather than circling a given code. In general, the response will be a number that you will note down (for example, the of age of the individual being surveyed, a quantity, etc.) or a date; sometimes it will be a sentence that the respondent has uttered. However, in other cases, you must write the response exactly as it was given to you. Be careful to write the responses exactly as given, do not modify terms and do not excessively summarize. Example: In 2012, how many days a week is postpartum care available to women? Number Don't know /__________/ 98 69 c. Mixed questions (coded and not coded) Some questions contain set responses as well as space to write responses that do not correspond to any of the predefined responses. Example: HC2 When have you come into contact with this (these) individual(s)? • When the individual(s) was/were conducting a routine community visit • During a health awareness campaign • During a visit where future activities were announced • When I left the health center for services • Other (specify) __________________________________ • A • B • C • D • X VIII. LOGISTICS a. Supplies: (in plastic bags): plastic bags will be supplied to all interviewers to be used to protect cell phones, paper questionnaires, etc. b. Site visit schedule: Each team member will receive a schedule of sites to be visited. This will include health zones, health facilities (hospitals and health zones). With regard to the RBF evaluation, the villages selected for the household survey will also be included on this schedule. The Expert will schedule field visits in conjunction with the supervisors and the interviewers. To do so, they should be assisted by the relevant Health Zones manager or MSH manager c. Transportation: a means of transportation will be made available to all evaluation team members. This is to ensure that everyone is on time when arriving at survey locations and, also, that everyone can leave the locations on time. These methods of transportation will depend on the specific field conditions. d. Lodging and meals: when the interviewers are required to spend the night away from their normal place of residence, a lodging allowance will be provided. The amount of this allowance is set by the CESD and is affected field conditions. A per diem will also be provided to each interviewer for daily meals when he/she works outside his/her normal place of residence. e. Salary: A salary will be provided to each interviewer on a pro rata basis according to the actual number of days worked. A day worked is one when the interviewer collected data. Travel days or rest days are not considered to be work days and will not be paid. f. Safety: interviewers must remain aware of the security situation. Supervisors must be notified of any information or event related to security and, in conjunction with the expert, take any necessary measures. Describe here what should happen in the event of an unexpected situation, including individual safety concerns [sic]. If a village or site where a survey is to take place is not accessible, the interviewer should inform the supervisor so that appropriate measures can be taken. 70 ANNEX I: Data Analysis Plan Final Evaluation Health facility survey Compare changes between baseline and final surveys for intervention and comparison HZ Change in Availability (quantity) of key MPA services at facilities: % change between baseline and final for intervention and comparison HZ Change in Accessibility (quality) of key MPA services at facilities: % change between baseline and final for intervention and comparison HZ % changes in quality of key family health interventions: between baseline and final for intervention and comparison HZ • Maternal health • HIV/AIDS • Family Planning • Child Health • Malaria Test of significance for quality of services scores for intervention and comparison HZ: change from baseline vs final: Difference –in- Difference (DID) Household survey Compare changes between baseline and final surveys for intervention and comparison HZ % Changes in client knowledge and practices on key health issues % Changes in health seeking and utilization practices on key health issues % Changes in client satisfaction with health facilities, health workers Test of significance for household knowledge, practices and perceptions (client satisfaction) scores: change from baseline vs final: Difference –in- Difference (DID) Household Survey Evaluation Questions Assessment Domains Source of data (quantitative) Modules Data analysis Household Characteristics Assess similarities between respondent and household profiles in Intervention and Comparison HZ Household survey Module 1: Household composition CM3-6, 11: Respondent’s type, gender, age, employment status, marital status CM7-8: Child profile in household (age 5-24 years), level of education CM9-11: Education profile of adults of the household Respondent characteristics • Age of respondent • Sex of respondent • Type of respondent Compare household characteristics between intervention and comparison sites • Education level • Employment status • Literacy status 71 Household Survey Evaluation Questions Assessment Domains Source of data (quantitative) Modules Data analysis Access to health services Assess reasons for visits, type of service, satisfaction with health visit Intervention and Comparison HZ Household survey Module 7: Contact with health services CS1: frequency of contact with health providers CS2 Reason for health care worker (HCW)contact CS3: Type of health education received through HCW CS4-5: Type of source/medium/HCW for any health messages CS6-15 health services access reasons and satisfaction with services received: last visit to a health centre % with frequent contact, sometimes , no contact, by type of health provider, location Top 3 reasons Top 3 messages Top 3 sources % with frequent visits, % with more than one year visits Cost of visit Type of health facility Top 3 Reasons for visit to traditional healer or pharmacy % satisfied with time spent by nurse % satisfied with professional skills of HCW % satisfied with interpersonal skills of HCW Perceptions of services provided by health facility (HC or GHR) Assess satisfaction with services at local health facility in the last 3 months Intervention and Comparison HZ Household survey Module 15: (Client satisfaction questions : Likert Scale) QC 1: interpersonal skills of providers, QC2, QC3: quality of care, access and availability of provider and service Calculate Overall satisfaction with the services received by indicators • Satisfaction with facility infrastructure • Satisfaction with health worker behavior • Satisfaction with health worker time for patients • Satisfaction with cost of care • Satisfaction with access to medication • Satisfaction with access to health facility (distance) Compare intervention vs. comparison: by type of facility, location of facility, HZ, SA, Province Knowledge and practice on key health topics; and and health seeking behaviors of key family health care services: Compare Intervention and Comparison HZ Household survey Module 3: Contraceptives CO 1 – CO15 : knowledge of different method of contraception CO 16 – CO17 : practice of using FP method CO 18 –CO21: source of FP information Calculate Overall knowledge score on FP method Calculate knowledge score : Modern method vs. Traditional method % using FP method, by Type of methods(Modern method vs. Traditional method) Client knowledge source: by source of information, HZ, SA, age of respondent 72 Household Survey Evaluation Questions Assessment Domains Source of data (quantitative) Modules Data analysis • Contrac eptives • HIV/AI DS • Materna l Health • Child health • Malaria • WASH Compare Intervention and Comparison HZ Household survey Module 4: HIV/AIDS VS1-3: practice of using HIV testing VS4-6: practice of using HIV testing during pregnancy % HIV tested and received results % HIV tested and received results during pregnancy Compare Intervention and Comparison HZ Household survey Module 5: Maternal Health SM1-6: access to prenatal care SM7-8 -: access to safe delivery services SM9-13 -: access to postnatal care % receiving prenatal care, by type of provider, number of visits, % received TT vaccination during pregnancy, number of times vaccinated % received postnatal care, by type of provider, time of postnatal care % delivering at home vs. facility, by type of facility (public, private), by type of provider % received Vit A after pregnancy, time received first dose Compare Intervention and Comparison HZ Household survey Module 6: Child health SY1: knowledge of serious childhood illness symptoms Module 8: Vitamin A supplementation VA1-3: child access to Vita A Module 9: Initial breastfeeding AP1-2: practice of early breastfeeding Module 11: Childhood diarrhea TD1-6: knowledge and practice of responding to childhood diarrhea % answered all symptoms Top 3 answers % received Vit A in past 6 months, by source % started immediately, within 4 hours % giving nothing to drink, about the same amount, less than usual % use ORS products % use homemade liquids % giving child nothing to eat % giving any treatment, by type of treatment % receive advice or treatment for pneumonia, source of advice/treatment % given a medication, by type of medication 73 Household Survey Evaluation Questions Assessment Domains Source of data (quantitative) Modules Data analysis Module 12: Childhood Pneumonia PP1-6: Client knowledge and practice of responding to childhood pneumonia Module 13: Malaria treatment /childhood fever TP1-11: Client knowledge and practice of responding to childhood fever % blood tested if had fever % received treatment for fever, by medication type, day treatment began, timing of treatment %taken to HF if had fever , Compare Intervention and Comparison HZ Household survey Module 10: Malaria intervention MI1-2, 4, 6, 7: Client access to bed nets MI5, 8-11: Client access to ITNs MI3: client practice of use of bed nets % of household have bed nets, by number, confirmed presence by data collector, type of bed net, source of bed net, duration of net ownership % of households with ITNs, time since treated Compare intervention vs. Comparison HZ Compare Intervention and Comparison HZ Household survey Module 2: Water and Sanitation EA 1-2: Household source of drinking water EA 3-4: Household practice of purifying drinking water EA 5-8: Household practice on use of toilets EA 9-13: Household practice on hand-washing and use of soap EA 14-16: household source of knowledge on hygiene Top 3 sources of drinking water Top 3 sources of drinking water Top 3 type of toilets used % use shared toilets, no toilets, personal toilets % have hand-washing area %have access to soap % received info from health workers, type of health worker 74 Facility Survey Evaluation Questions Assessment Domains Source of data (quantitative) Modules Data analysis Facility Characteristics Assess similarities between facility profile in Intervention and Comparison HZ Facility survey: HCs and GRHs Compare characteristics between intervention and comparison sites • Location: urban/rural • Type: HC/HGR Availability (quantity) of key family health care services minimum package of Activities/Compl ementary Package of Activities plus (MPA/CPA-plus) Assess range and type of services provision including Preventive care, Curative, Water/Sanitation/Hygiene (WASH) services and products MPA+ services: supposed to be provided at all HCs CPA+ services: supposed to provided by all GRHs (This is a limitation since as of mid-March MSH did not have data about CPA-Plus yet. The facility questionnaire used for the survey didn't incorporate CPA-Plus.) Compare Intervention and Comparison HZ Facility survey: HCs and GRHs Module 3 questions Module 4 questions Percentage of facilities (HC and GRH) offering full range of MPA+), by health HZ Availability (quantity) of community or health promotion interventions to improve accessibility to key family health care services Assess community-based support and outreach with respect to information, communication, and delivery of key family health care services Compare Intervention and Comparison HZ Facility survey: HCs and GRHs Module 4: SC26-34 Module1: IG4,5 Module 3 PREV22 Module 3 PREV40 % of the health centers surveyed provide health promotion/information on: condom use; hygiene and sanitation; exclusive breastfeeding; food hygiene/food safety; oral rehydration for diarrhea; and, fistula prevention) % of facilities open 24 hours a day % of facilities open 7days a week • % facility offering labor and delivery services in community • % facility has vaccine education plan • % facility offering TB contact tracing, monitoring in community 75 Module 4 SC12, 13 questions Module 3 PREV71 Module 10 • % of the health centers surveyed distribute ITNs, free of charge • % of facility offer Growth monitoring has growth monitoring IEC materials General quality of Health Facility Derive quality score by combining quality of health facility variables Compare Intervention and Comparison HZ Health facility infrastructure, equipments, communication , transport, waste management Facility survey Module 1 questions combine Q IG: 3, 8, 11, 13, 14, 16, 17,19,22, 25 % of facility with adequate infrastructure for being functional today (Presence of waiting room, electricity, running water, source of water, equipment work, transport, emergency communication, toilets, disinfectant, waste disposal ) Derive quality score on health facility infrastructure Staff Characteristics Staff training and skills Module 2 questions: SP 1-7 Type of staff,: number (%), gender, training status Characteristics of staff: type of staff, number (%), gender, training status Percentage health workers with specific skills (FP, HIV, STI, IMCI, PMTCT) by type of facility Infection Comparison Practices Module 1: IG 21, 23-24, 25 Module 6: EM 1-2 The top three methods of decontamination of medical equipment The top three methods of sterilization of medical equipment The top three methods of sharp waste disposal The top three methods of waste disposal Facility Supervision Module 5: SU1-2 Facility supervision rate: number and % of facilities with at least once a month supervision visit Essential Drugs, supplies management and monitoring systems Module 6: EM5-6 Percent of health facilities experienced delivery delays for medications and supplies The top three reasons why medications and supplies are delivered delayed The top three source of procurement Quality of key family health care services provision Assess whether a particular key health services offered in the facility is adequate Facility survey: HCs and GRHs Module 1, PREV 33 Module 1, PREV 36 Module 1, PREV 34 % of facility adequate for providing services 76 • FP services • Maternal: prenatal care, PMTCT, emergency obstetrics, • Child health: IMCI, vaccinatio n • HIV/AIDS • Malaria Similar analysis for vaccination service, PMTCT, Maternal health, IMCI Compare Intervention and Comparison HZ Module 2 SP3 Module 1, IG 1 Module 6 EM7 Module 8 Module 6: EM8, EM9 Module 10 Module 7 77 ANNEX J: Health Facility Survey Sample Distribution Number of Facilities Surveyed: Baseline Number of health facilities Province BC Health Zone Intervention HZ Comparison Total Name n % Name n % n % W. Kasai Luiza Luiza 12 13% Lubondaie 14 16% 26 30% E. Kasai Mwene-ditu Bibanga 16 18% Kamiji 11 13% 27 31% E. Kasai Lodga (Kole) Lomela 17 19% Tshudiloto 8 9% 25 29% E. Kasai Tshumbe Minga 15 17% Wembonyama 12 14% 27 31% Katanga Kamina Kayamba 6 7% Kabongo 17 20% 23 27% Katanga Kolwezi Kanzenze 12 13% Mutshatsha 11 13% 23 27% S. Kivu Uvira Nundu 12 13% Uvira 13 15% 25 29% 90 51% 86 49% 176 Number of Facilities Surveyed: Endline HZ Intervention HZ Comparison Total Province BC Name n % Name n % n W. Kasai Luiza Luiza 13 14% Lubondaie 13 14% 26 E. Kasai Mwene Ditu Bibanga 16 17% Kamiji 11 12% 27 E. Kasai Lodga Lomela 17 18% Tshudiloto 9 10% 25 E. Kasai Tshumbe Wembonyama 12 16% Minga 15 13% 27 Katanga Kamina Kayamba 8 8% Kabongo 18 20% 26 Katanga Kolwezi Kanzenze 14 15% Mutshatsha 12 13% 26 S. Kivu Uvira Nundu 13 14% Uvira 15 16% 28 4 7 7 93 50% 7 93 50% 186 78 ANNEX K: Household Survey Sample distribution Number of Household Surveyed: Baseline Number of respondents Province BC HZ Intervention HZ Comparison Total Name n % Name n % N % W. Kasaï Luiza Luiza 235 13% Lubondaie 254 14% 489 27% E. Kasaï Muene￾Ditu Bibanga 260 15% Kamiji 260 14% 520 29% E. Kasaï Kole Lomela 260 15% Tshudiloto 260 14% 520 29% E. Kasaï Tshumbe Minga 245 14% Wembonyama 260 14% 505 28% Katanga Kamina Kayamba 256 14% Kabongo 259 14% 515 28% Katanga Kolwezi Kanzenze 260 15% Mutshatsha 259 14% 519 29% S. Kivu Uvira Nundu 260 15% Uvira 260 14% 520 29% 1776 49% 1812 51% 3588 Number of Household Surveyed: Endline HZ Intervention HZ Comparison Total Province BC Name n % Name n % N % W. Kasaï Luiza Luiza 260 14% Lubondaie 260 14% 520 14% E. Kasaï Muene Ditu Bibanga 260 14% Kamiji 260 14% 520 14% E. Kasaï Kole Lomela 260 14% Tshudiloto 260 14% 520 14% E. Kasaï Tshumbe Wembonyama 256 14% Minga 258 14% 514 14% Katanga Kamina Kayamba 260 14% Kabongo 259 14% 519 14% Katanga Kolwezi Kanzenze 259 14% Mutshatsha 258 14% 517 14% S. Kivu Uvira Nundu 252 14% Uvira 259 14% 511 13% 4 7 7 1,807 50% 7 1,813 50% 3,620