EVALUATION Promoting Access to Quality Health Services: A MidtermAssessment of a Results-Based Financing Intervention in the Democratic Republic of Congo April 2015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by International Business & Technical Consultants, Inc. (IBTCI). It was authored by Swati Sadaphal, and Annette Bongiovanni. PROMOTING ACCESS TO QUALITY HEALTH SERVICES: A MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION IN THE DEMOCRATIC REPUBLIC OF CONGO April 13, 2015 AID-660-M-13-0000I Cover Photo: Bridge to Tshudi Loto Health Zone Credit: Annette Bongiovanni DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION I ACKNOWLEDGMENTS The assessment team wishes to thank everyone in the Democratic Republic of Congo (DRC) who accommodated our needs, in particular the many individuals who went out of their way amid busy schedules to provide various types of support. The team wishes to thank Dr. Mukengeshayi Kupa, Ministry of Public Health (MSP) Secretary-General, who provided invaluable support enabling the team to conduct its activities at the provincial, district, and zone levels. We would like to give special thanks to Dr. Celestin Bukanga, Head of MSP’s Central RBF Unit, and all of his team members for their interest and participation. The cooperation and input from all MSP respondents in the provincial and health zone headquarters visited is highly commended. The team also wishes to thank USAID/DRC staff members who supported this assignment, in particular Kai Beard, Elena Facchini and Dr. Richard Matendo. We also appreciate the support of the staff of the Integrated Health Program (IHP); we especially thank Delmond Kyanza, Ousmane Faye, and Tchim Tabaro for sharing valuable time and knowledge to improve the team’s understanding of the IHP PBF Pilot activity. We also wish to thank the IHP Coordination Bureau staff and MSP’s health zone managers in Mwene Ditu, Luiza, Tshumbe, and Kolwezi for making themselves readily and enthusiastically available to us throughout the assignment. Finally, we were enormously grateful for the chance to meet and talk with health facility staff, community volunteers and community members in all four health zones visited. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION II Table of Contents Table of Contents .............................................................................................................. ii List of Tables and Figures ................................................................................................ iii ACRONYMS...................................................................................................................... iv EXECUTIVE SUMMARY .................................................................................................. 1 I. EVALUATION PURPOSE AND OBJECTIVES .......................................................... 9 Program Background ...................................................................................................10 II. EVALUATION METHODS AND LIMITATIONS ..................................................12 Evaluation Team ..........................................................................................................12 Study Design .................................................................................................................12 Site Selection Strategy ................................................................................................12 Ethical Considerations ................................................................................................. 15 Data Management ........................................................................................................15 Data Analysis ................................................................................................................15 Limitations ....................................................................................................................15 III. FINDINGS ..................................................................................................................17 OBJECTIVE 1: Effect of PBF intervention on the quantity of services ..................17 OBJECTIVE 3: Achievements to-date of the PBF activity ......................................21 OBJECTIVE 4: Availability of funds at the operational levels ................................. 23 OBJECTIVE 5: Contextual factors which might influence PBF intervention results ........................................................................................................................................25 OBJECTIVE 6: Unintended consequences related to PBF implementation .........28 IV. CONCLUSIONS ........................................................................................................30 OBJECTIVE 1: Effect of PBF intervention on the quantity of services ..................30 OBJECTIVE 2: Effect of PBF intervention on the perceived quality of services ...30 OBJECTIVE 3: Achievements to-date of the PBF activity ......................................30 OBJECTIVE 4: Availability of funds at the operational levels ................................. 30 OBJECTIVE 5: Contextual factors which might influence PBF intervention results ........................................................................................................................................31 OBJECTIVE 6: Unintended consequences related to PBF implementation .........31 V. RECOMMENDATIONS .............................................................................................32 ANNEX I. Statement of Work ......................................................................................34 ANNEX II. Detailed Methodology and Data analysis Plan .........................................38 ANNEX III. Data Collection Instruments .....................................................................56 ANNEX IV. List of Documents Reviewed ..................................................................113 ANNEX V. Sources of Information .............................................................................115 ANNEX VI. Field Implementation Plan ......................................................................119 ANNEX VII. Additional Data Analysis ........................................................................124 MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION III List of Tables and Figures Figure 1: Sites selected .................................................................................................. 14 Figure 2: SNIS data review for period July-Sept 2014 ................................................. 18 Figure 3: Formal Sources of Health-related Information ........................................... 19 Table 1: Comparison of perceptions of quantity of care between baseline and midterm survey ................................................................................................................ 22 Table 2: Percentage performance payments earned by HFs and ECZs during Q1 and Q2 of the PBF Intervention ..................................................................................... 25 MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION IV ACRONYMS AIDS Acquired Immune Deficiency Syndrome ANC Antenatal Care ARI Acute Respiratory Infection BC Bureau de Coordination/IHP Coordination Bureau CSO Community Service Organization CDF Congolese Francs CODESA Comité de Développement Sanitaire/Health Development Committee CPA Complementary Package of Activities CPN Consultation Prénatale/ Antenatal Care (ANC) CPON Consultation Postnatale/ Postnatal Care (PNC) CSO Civil Society Organization DPS Division Provinciale de la Santé/Provincial Division of Health DRC Democratic Republic of Congo ECZ Equipe Cadre de Zone/Health Zone Management Team (HZMT) FGD Focus Group Discussion FOSA Formation Sanitaire FOSACOF Formations Sanitaires Complètement Fonctionnelles/Fully Functional Service Delivery Point FP Family Planning GRH General Reference Hospital/ Hôpital Général de Référence (HGR) HC Health Center/ Centre de Santé (CS) HF Health Facility (includes health centers, general referral hospitals and health posts) HIV Human Immunodeficiency Virus HZ Health Zone/ Zone de Santé (ZS) IBTCI International Business and Technical Consultants, Inc. IHP Integrated Health Project/Projet de Santé Intégré (PROSANI) KII Key Informant Interview M&E Monitoring and Evaluation MCH Maternal and Child Health MIP Médecin Inspecteur Provincial/Provincial Medical Inspector MNCH Maternal, Newborn and Child Health MPA Minimum Package of Activities MSH Management Sciences for Health MSP Ministère de la Santé Publique/Ministry of Public Health NTD Neglected Tropical Disease PBF Performance Based Financing PMP Performance Monitoring Plan MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION V PMTCT Prevention of Mother-to-Child Transmission of HIV RBF Results-Based Financing/Financement Basé sur les Résultats (FBR) RH Reproductive Health SNIS Système Nationale d'Information Sanitaire/National Health Information System of the DRC TB Tuberculosis TBA Traditional Birth Attendant USAID United States Agency for International Development USG United States Government MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 1 EXECUTIVE SUMMARY INTRODUCTION The Integrated Health Project (IHP) or Projet de Santé Intégré (PROSANI), supported by United States Agency for International Development/ Democratize Republic of Congo (USAID/DRC), is currently implementing a performance based financing (PBF) pilot intervention in seven Zone de Santé (ZS) or health zones (HZs) (Bibanga, Kanzenze, Kayamba, Lomela, Luiza, Nundu and Wembonyama) across four provinces: East Kasai, West Kasai, Katanga, and South Kivu. PBF is one of the potential strategies of the Ministère de la Santé Publique (MSP) or Ministry of Public Health to achieve health system strengthening strategy goals. The MSP is responsible for creating policies, standards and procedures, and ensuring monitoring and evaluation of PBF activities. MSP’s central PBF unit has the role of regulator for PBF implementation at the national level. At the intermediate level, the MSP audits and evaluates quality assurance, and provides technical capacity building. At the operational level, it performs quality evaluations of health facilities (HFs), builds capacity, and monitors the implementation of the annual operational plan. In the literature, the term PBF is often used interchangeably with results-based financing (RBF) or Financement Basé sur les Résultats (FBR), which is the broader rubric under which PBF resides. PBF indicates that the intervention is focused on the supply-side, which encompasses service delivery. For the purpose of this report, PBF is used throughout the text and RBF is used in the title and introduction section only. USAID/DRC contracted International Business and Technical Consultants, Inc. (IBTCI) to conduct an independent, impact evaluation of its PBF program implemented by IHP. The impact evaluation methodology uses a prospective quasi-experimental design with interventions and comparison groups covering all seven PBF HZs, with measurements taken at baseline and endline. IBTCI completed the baseline evaluation in 2013. Data collection for a midterm assessment was conducted from October 2014 to November 2014. The main focus of this mid￾term assessment was on programmatic and management approaches (structural and procedural measures) to sufficiently capture any intermediate effects resulting from the PBF intervention. The following are the PBF Midterm Assessment objectives: 1. Assess the initial effect, if any, the PBF intervention has had on the quantity of services. 2. Assess the initial effect, if any, the PBF intervention has had on the perception of the quality of services. 3. Assess the achievements to-date of the PBF objectives and describe any bottlenecks that are impeding progress. 4. Determine any changes between the planned versus the actual availability of funds at the operational levels (i.e., HZ management and facilities). 5. Identify and analyze contextual factors which might influence the results of the PBF intervention. 6. Gather preliminary information and describe unintended consequences related to the implementation of the PBF intervention. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 2 The audience of the mid-term assessment is primarily the USAID/DRC Mission, specifically the Program Office and Health Team, and the implementing partner, Management Science for Health (MSH). USAID/DRC and MSH will use this assessment to inform programming and learning so as to strengthen the next phase of PBF pilot intervention by making any mid-course corrections as needed. METHODOLOGY The mid-term study design was a non-experimental descriptive process assessment using predominantly qualitative data collection and analysis methods. The assessment utilized purposeful sampling at select sites and a wide range of stakeholders at the national, provincial, district, and community levels. The team reviewed various documents from MSP, IHP and USAID to assess the process and current extent of PBF implementation. Selected indicators from the Système Nationale d'Information Sanitaire (SNIS) or National Health Information System of the DRC, and RBF web portal (www.fbrsanterdc.cd) were also reviewed. A modest sample of 44 key informants was interviewed from across three provinces representing USAID, MSP, IHP, and chief nurses and directors of Hôpital Général de Référence (HGR) or general reference hospitals (GRH). A total of 20 focus group discussions (FGDs) were conducted with the community leaders— Comité de Développement Sanitaire (CODESAs or Health Development Committee), Civil Society Organizations (CSOs) contracted by MSH to do counter verifications at the household level, and Traditional Birth Attendants (TBAs) in the villages. Clients who visited health facilities (HFs) during the past month were interviewed at their respective homes. A total of 259 facility clients completed the interviews. The client questionnaire collected data on the following: health services utilization characteristics; cost of services received; health education services received and; perceptions of the quality of care and services provided by HFs. Facility mini-surveys were conducted in five HFs (one GRH and four Centre de Santé (CS) or health centers (HCs)) in each of the four HZs visited (N=20). The facility survey questionnaire included questions for research assistants to record facility observations on the costs of services posted at the HFs, and to note indicator data collected through facility registers and chart reviews. Quantitative and qualitative data were analyzed using the following analytical domains: progress to-date; relevance of PBF; availability of funds at the operational level; ownership and management capacity; contextual factors and; unintended consequences. A comprehensive process evaluation assessing the fidelity of PBF implementation was not justified given the nascent stage of the pilot intervention, and therefore resources were not allocated for such a study. The assessment results are meant to be a description of sites visited and not representative of all PBF sites supported by USAID. Nevertheless, the use of direct observation, a household survey among recent facility clients, in-depth interviews with managers, administrators, and providers, and FGDs with communities allowed us to identify achievements and potential gaps within the implementation process, and thus, inform future programming; the overarching purpose of the assessment. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 3 MAJOR FINDINGS Progress to-date PBF activities commenced in all seven intervention HZs in November 2013, following the execution of performance contracts with IHP, facility directors and MSP managers: individual contracts were signed with 118 HCs, seven GRHs and seven Equipe Cadre de Zone (ECZ) or health zone management teams (HZMT) during October and November, 2013. The first cycle of the PBF implementation period began on November 1, 2013, with the second on February 1, 2014, and the third on May 1, 2014. During each PBF cycle, technical verification visits were conducted jointly by one member from the ECZ team and an IHP staff member to verify facility-reported data. Two CSOs per HZ were contracted to implement data counter verification activities. A MSP provincial-level officer, along with an IHP staff member, conducted the verification of the ECZ-level PBF results. MSP’s PBF unit staff were trained on PBF processes and mentored during quarterly verification visits and data validation during the first six-months of PBF implementation. IHP designed a PBF web portal to improve coordination, monitoring, and communication among MSP, IHP and other partners. The PBF web portal provided accessibility to all quantity and quality performance information on the PBF facilities for each quarter (Q). More than 90 percent of respondents noted PBF as an appropriate intervention for the DRC health system. However, bottlenecks were observed that impeded the PBF progress. Staff recruited after November 2013 did not receive formal training on PBF reporting tools at the service delivery level, and the IHP PBF manual did not include detailed instructions on how to complete various PBF forms. All CSOs visited were based at the level of HZ- they do not have a permanent presence in all health areas encompassed within a HZ. They are physically far from the HFs they audit and do not have any interventions in the surrounding communities. FGDs with CSO respondents noted that this affected their quality of work because they faced difficulty in identifying clients due to non-familiarity with the local population and geography. IHP planned to use local organizations but staff could not identify any suitable CSOs located in, or close to, the health areas. According to IHP respondents, there were only two full-time staff members dedicated to PBF pilot implementation in the IHP headquarters in Kinshasa. There were three other IHP staff members who contributed between 20 and 80 percent of their time; however this contribution was ad hoc and not codified. Relevance of PBF PBF is increasing the quantity of services provided, but there are large variations across HZs and between types of health services. SNIS trends analysis of selected quantitative indicators (July-Sept 2014) shows a slight but steady improvement in the service utilization rates for services contracted under PBF, such as antenatal care (ANC), child vaccination, curative services and family planning (FP). PBF led to noticeable changes in health worker behaviors. 90 percent of CODESA FGDs reported that facility staff is present at facilities more often than before the PBF implementation started. They introduced strategies to increase demand and utilization of health services primarily through decreasing user-fees and increasing community outreach activities. Facilities reported the lowering of user fees by 30-50 percent, except for the Kanzenze GRH, where prices remained unchanged (the Kanzenze GRH is managed by a Catholic Mission). User-fees varied from site to site. For example, the fees for an initial visit for MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 4 a child varied from 800 – 1,200 CDF ($0.86 – 1.30) and for an adult, 1,200 – 5,000 CDF ($1.30- 5.40). HF staff and ECZ management, along with CODESA input, ostensibly decide the user￾fees. However this was not always the case, according to CODESA members. More than 80 percent of chief nurse respondents reported an increase in client attendance due to the decrease in user fees. Providers and/or community health workers and CODESA members motivated communities to mobilize sick individuals, pregnant women, and children to seek health center services. This finding is corroborated by the household survey results. The most common channel for health and nutrition related information was the health worker (66 percent), followed by family (20 percent), and other (14 percent). Among health workers, nurses were the most common source of health information (75 percent), followed by community health workers (59 percent), and TBAs (37 percent). All facilities reported that they had technical verification activity conducted by an ECZ team member and an IHP staff after each quarter ended. The two-person team stayed in the HFs between one and three days. Some of the verification activity teams were accompanied by a provincial-level MSP staff member. The technical verifiers used PBF tools (work plans, guideline documents, facility targets, and data verification forms) to check adherence to standards, norms and guidelines, and to verify reported data. The technical quality was measured by the Formations Sanitaires Complètement Fonctionnelles (FOSACOF) or Fully Functional Service Delivery Point tool. FOSACOF scores are one of the 16 "paid indicators" at the health center level, weighted at 29% of the total performance score, or 32% percent (when 12 indicators are measured, dropping HIV and TB indicators). Hospitals' payments are based 100 percent on the quarterly FOSACOF score. The technical verifiers also discussed ways to improve interpersonal skills of providers, the availability of basic equipment, medicines and supplies, and the facility’s data trends. The client household survey results revealed satisfaction with the service received during their last facility visit: 84 percent of clients were ‘very satisfied’, 13 percent were ‘somewhat satisfied’ and three percent were ‘dissatisfied’ with the service. FP and child vaccination services are provided free of charge in all surveyed facilities, while other services including services of pregnant women, child birth, curative services and medications, are paid services. There were large variations in the user fees paid by the surveyed clients for various health services. The user fee amount depended upon the type of health service available, and differed by HZ. About 26 percent of survey respondents availed free-of-charge services. Among those who attended paid services (including all types and age categories), 48 percent paid user fees between 100 CDF ($0.10) and 1,500 CDF ($1.62), 38 percent paid between 1,500 CDF ($1.62) and 5,000 CDF ($5.40) and only 11 percent paid more than 5,000 CDF ($5.40). Anecdotal information gathered in FGDs in remote areas indicated an average household income of approximately 40,000-50,000 CDF ($43.20 - 54.00) per month. 80 percent reported that prescribed medication was available at the HF during the last visit. 16 percent of clients interviewed reported that in the past six months, there was a time when they needed health services but did not visit a health center or hospital. The most common reason given was insufficient money to pay the bill (70 percent). Regarding perceptions that clients have of health workers’ behaviors and practices; more than 90 percent were satisfied. Only 60 percent of clients believed that the facility rooms are appropriate and 79 percent reported that wait-time was reasonable. 75 percent of clients reported that the prices of health services were reasonable MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 5 and 57 percent reported that the cost of care is negotiable. 45 percent of clients were dissatisfied because the prices of services were not posted, and about 25 percent were dissatisfied with the availability of medications at the HF. Availability of funds at the operational level As of Oct 2014, the 118 HCs targeted by IHP received $163,564; seven GRHs received $147,811; and seven ECZs received $24,694 as PBF incentives. All HFs and ECZs visited received their PBF payments for the first two Qs. The general trend is an increase in facility funds by 10 percent and ECZ funds by 19 percent, from Q1 to Q2, improved providers/staff performance. At the time of the assessment, none of the HFs surveyed received incentives for the third cycle of PBF which were expected within one month of the completion of the counter verification, as per the PBF contracts. About 19 percent of chief nurses mentioned that prolonged delays in payments could lead to de-motivation of HF staff. However, respondents were not aware of the counter verification, and understood payment would be received within one month of the technical verification. This misinterpretation of their contracts led them to perceive longer delays in the receipt of payment than what was agreed. Respondents were aware of the IHP guidance on the allocation of the PBF funds: 60-70 percent of funds are allocated for staff incentives, 10-20 percent of the funds are to be expended on ‘investments,’ manifested in expenditure on infrastructure, and 10 percent to cover operational costs (i.e., patient registers). All HFs reported that they used the IHP Index Tool to calculate and distribute the incentives across all HF staff. Ownership and management capacity IHP’s PBF design is compliant with MSP policies and directives vis-à-vis a) MSP’s underlying PBF principles, b) the implementing entities and their roles, c) the levels of contractualization, and d) the entities that execute a PBF program. IHP introduced concepts of target setting, creating business plans, work-plans, and technical verifications. New tools for facility management and supervision were added with the introduction of the PBF intervention. All central MSP’s RBF unit respondents expressed their commitment to the success of PBF in DRC. The activities such as joint trainings, tools development, and monitoring and evaluation of pilot sites were mentioned by the RBF central unit respondents as the learning opportunity which will help them in the future, even if the IHP support ends. However, three out of nine IHP respondents noted that the role of the central RBF unit is still more of a “coordinator” rather than a “regulator”. All Division Provinciale de la Santé (DPS) or Provincial Division of Health and Médecin Inspecteur Provincial (MIP) or Provincial Medical Inspector staff members interviewed were highly supportive of the IHP PBF model, despite the fact they did not receive any PBF incentives. They want the model to be expanded to all IHP-supported HZs and to all three levels of health system pyramid, including provincial and district contractualization, to improve ownership of PBF implementation on the ground. CODESA members are expected to participate in the management of their HCs and conversely, making HCs accountable to the communities they serve. All CODESA FGDs noted that they help facility staff by encouraging pregnant women and people who are ill to attend services, or by gathering community members and children on immunization day. CODESA members mentioned that they are not involved in planning and monitoring and evaluation of HC’s activities. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 6 Contextual factors Document review revealed that IHP encounters a wide range of environmental challenges that may hinder PBF implementation, and negatively influence the results in a variety of ways. These factors include civil and political unrest in South Kivu, and poor geographic accessibility in certain health HZs such as Lomela and Kayamba. Supply of electricity (by solar panels at best) is rare and running water is non-existent in the health areas visited. Chief nurse respondents in all HZs mentioned difficulties in maintaining cold-chains for vaccines/medications, safe medical waste management, and infection control practices. Unintended consequences Upon review of the IHP RBF Manual, it is evident that accountability and transparency are built￾in at each operational level of the PBF intervention. In order to pay for performance, it needs to be measured, verified, counter-verified and validated to ensure that only true performance is compensated. On probing, none of the respondents alluded to any unintended negative effects of PBF related to gaming, distortion, or cherry-picking. By design, IHP’s PBF model does not address the differences in socioeconomic status of the target populations, the type of organization, or geographic variations. For example, Katanga Province’s cost of living is much higher than in Kasai, yet the same amounts of funds are allocated equally among all HCs and hospitals. In Bibanga, even though HIV/AIDS and TB activities are very limited, the HCs were still required to report on these indicators every quarter. MAJOR CONCLUSIONS The PBF pilot intervention is progressing as planned. MSP’s PBF unit staff were trained on PBF processes and mentored during at the beginning of the implementation. IHP designed a PBF web portal to improve coordination, monitoring, and communication among MSP, IHP and other partners. A joint technical verification process was used by senior MSP officials and IHP Bureau de Coordination (BC) or IHP Coordination Bureau staff to assess the performance at the service delivery point level. Community-level counter-verifications were carried out by CSOs, independently of HFs and HZ management. However, there are a few bottlenecks impeding the progress. Due to the lack of formal training of new facility staff, there are gaps between the reported data and the validated data. A lack of sufficient full-time dedicated staff at the central level and at the BC level is contributing to insufficient monitoring and some reasonable delays in the payment of incentives. The CSOs are not directly involved in health facility improvement plans. The creation of champion communities by IHP, to act as counter verification agencies, is not yet fully implemented. There was a slight, but steady improvement in the quantity of services contracted under PBF, but there were large variations across HZs and between the types of health services. Reduction of user fees and increases in community outreach activities were the two major strategies used by HF staff to improve client volume. PBF intervention was supporting the quality of services improvements. There was an immediate behavior change noted among health providers who are now present at the HF more often than before PBF started. Lowering of user-fees improved financial accessibility of health services. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 7 All HFs and ECZs received performance bonuses based on their performance levels during Q1 and Q2. The total payments for HFs and ECZs increased between the two Qs. There was, however, a delay in payments for Q3. IHP PBF design was compliant with the MSP's policies and directives. IHP collaborated directly with the MSP's PBF unit to implement its pilot activity promoting ownership. CODESAs were aware of PBF objectives but their role was either unclear or only limited to mobilizing community agents to gather patients and encourage them to use the HCs. IHP encountered a wide range of environmental factors that hindered PBF implementation and negatively influenced the results in a variety of ways. These factors were related to civil unrest and insecurity, geographic inaccessibility, poor infrastructure, and lack of sufficient national budget spending on health. PBF implementation clearly had positive unintended consequences. PBF introduced concepts of quality of care, target-setting, business-planning, work-planning, and technical verification. Verifications and counter-verification processes supported transparency, accountability and improvements in data reporting. However, IHP PBF by design did not adequately address the aforementioned differences in target population socioeconomic status, type of organization, and geographic variations. This could lead to unintended inequities. Negative unintended consequences related to gaming, cheery-picking, and distortion were not found, but these cannot be ruled out. MAJOR RECOMMENDATIONS The recommendations presented below are for IHP, listed in descending priority order, to improve ongoing activities of the PBF implementation. We do not recommend any major modifications to the pilot intervention in view of the forthcoming end line data collection for the impact evaluation study. Continue PBF pilot implementation IHP should continue with implementation of the PBF pilot intervention at all 118 HFs, seven GRHs and seven ECZs. Immediately appoint full-time staff dedicated to PBF The IHP PBF team needs to be staffed-up immediately at the central and BC offices to handle the workload, especially considering the need for data verification and validation. Strengthen PBF trainings at the operational level Validity and reliability of PBF data at the service delivery level needs continued attention. Priority target audiences for technical verification training should be focused on the newly recruited chief nurses, especially on PBF calculations and the use of Index Tool to calculate bonuses. They need more guidance on how to assess staff performance, and in general, the overall application of the tool. Provide written guidance for medical record review component MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 8 The medical record review component is a critical component of the process on which more attention should be paid in terms of the provision of written guidance in the PBF manual. Medical record review (cross-checked with other documents such as the partogram, medication distribution list, etc.) demonstrates written confirmation that the norms and protocols set out in the tools are adhered to, for example, the Ordinogram. Add performance indicators to measure the quantity of care at GRH The GRH contracts would benefit from having a more rigorous standard than just the FOSACOF. There should be service indicators that are proxies to assess GRH progress toward compliance with CPA Plus services. It is important to keep the FOSACOF as one indicator, but other service related indicators are needed. Client satisfaction scores could be a highly subjective indicator. The satisfaction scores should not be given higher weight than the health outcomes indicators. Provide additional training to CSOs on budgeting CSOs need to better understand and anticipate the average cost per unit (household visited) for budgeting purposes. An illustrative budget to accompany the $4,500 award would be helpful. Develop local community champions for counter verifications As far as possible, local community organizations should be selected to perform counter verifications. In the absence of such organizations, IHP could develop strategies for identifying motivated community leaders and developing their skills in creating local community champion organizations for counter-verifications. Build capacity of CODESA to participate in facility management CODESAs capacity should be strengthened in planning, and monitoring and evaluation of HC activities. CODESA should be able to participate in the management of their HC and conversely, make HCs accountable to the communities they serve. Assess the problem of inequity There is a need to carefully assess the equity between a) types of facilities (HC versus hospital); b) cost of living between provinces and/or HZs; and c) baseline status of the HC infrastructure so the requirements are more balanced between and within the various facilities in different locations. The problem of inequity can be mitigated by rewarding improvement, in addition to absolute achievement, as well as additional incentives for HCs, specifically serving remote or poor disadvantaged populations. In order to have comprehensive and equitable coverage in a health area, the Health Posts should be able to receive bonuses. This might be too complicated to develop guidance or directives in the contracts, so an initial assessment study could be conducted to address this issue. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 9 I. EVALUATION PURPOSE AND OBJECTIVES The Integrated Health Project (IHP) or Projet de Santé Intégré (PROSANI), supported by United States Agency for International Development/ Democratize Republic of Congo (USAID/DRC), is currently implementing a performance based financing (PBF) pilot intervention in seven Zone de Santé (ZS) or health zones (HZs) (Bibanga, Kanzenze, Kayamba, Lomela, Luiza, Nundu and Wembonyama) across four provinces: East Kasai, West Kasai, Katanga, and South Kivu. PBF is one of the potential strategies of the Ministère de la Santé Publique (MSP) or Ministry of Public Health to achieve health system strengthening strategy goals. The MSP is responsible for creating policies, standards and procedures, and ensuring monitoring and evaluation of PBF activities. MSP’s central PBF unit has the role of regulator for PBF implementation at the national level. At the intermediate level, the MSP audits and evaluates quality assurance, and provides technical capacity building. At the operational level, it performs quality evaluations of health facilities (HFs), builds capacity, and monitors the implementation of the annual operational plan. In the literature, the term PBF is often used interchangeably with results-based financing (RBF) or Financement Basé sur les Résultats (FBR), which is the broader rubric under which PBF resides. PBF indicates that the intervention is focused on the supply-side, which encompasses service delivery. For the purpose of this report, PBF is used throughout the text and RBF is used in the title and introduction section only. USAID/DRC contracted International Business and Technical Consultants, Inc. (IBTCI) to conduct an independent, impact evaluation of its PBF program implemented by IHP. The impact evaluation methodology uses a prospective quasi-experimental design with interventions and comparison groups covering all seven PBF HZs, with measurements taken at baseline and endline. IBTCI completed the baseline evaluation in 2013. Data collection for a midterm assessment was conducted from October 2014 to November 2014. The main focus of this mid￾term assessment was on programmatic and management approaches (structural and procedural measures) to sufficiently capture any intermediate effects resulting from the PBF intervention. The following are the PBF Midterm Assessment objectives: 1. Assess the initial effect, if any, the PBF intervention has had on the quantity of services. 2. Assess the initial effect, if any, the PBF intervention has had on the perception of the quality of services. 3. Assess the achievements to-date of the PBF objectives and describe any bottlenecks that are impeding progress. 4. Determine any changes between the planned versus the actual availability of funds at the operational levels (i.e., HZ management and facilities). 5. Identify and analyze contextual factors which might influence the results of the PBF intervention. 6. Gather preliminary information and describe unintended consequences related to the implementation of the PBF intervention. The audience of the mid-term assessment is primarily USAID/DRC Mission, specifically the Program Office and Health Team, and the implementing partner, MSH. USAID/DRC and MSH MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 10 will use this assessment to inform programming and learning so as to strengthen the next phase of PBF pilot intervention. PROGRAM BACKGROUND The Government of Democratic Republic of Congo (DRC) adopted the Health Systems Strengthening Strategy (Stratégie de Renforcement du Système de Santé) to implement financing reforms, avoid wastage, and achieve national health objectives. Among the strategies to achieve financing reforms, RBF emerged as the most promising strategy, compared to other types of financing, namely input financing. RBF is a strategy for attaining positive health results through financial incentives. RBF schemes can be developed for both supply (health worker, facility, district health team, community) and demand (patient/client) sides of the health system.1 A demand-side RBF intervention may give households cash incentives to receive preventive care services or to encourage completion of treatment. A supply-side RBF can contribute to increasing the quality of care and range of services, and generating positive health outcomes in two primary ways: first, by incentivizing providers to put more effort into specific activities with explicit performance targets, and second, by increasing the amount of resources available to finance the delivery of health services. Motivating health workers to provide quality services and keeping them in the public sector has been a particular challenge for the health system in DRC, as in many other countries. Fixed salaries with raises that are not tied to performance often lead to low productivity, poor quality, absenteeism, and lack of innovation. Moreover, payment of fees by clients for health services tends to result in greater attention to fee-generating services such as curative care, at the risk of preventive care and quality of services. RBF is designed to be a more productive alternative to input financing. Rather than granting an advance payment, RBF pays for outputs. It is a transfer of money or other material incentives from an external supporter to a recipient, contingent upon the beneficiary performing a measurable action or reaching a predetermined target. Recipients can be either health care providers or consumers, depending on the needs and goals of the specific project. This creates new performance incentives for employees, empowers health facilities to allocate resources to where they are most needed, and increases demand for essential health services. Additionally, RBF helps finance the under-funded health sector. RBF projects were implemented beginning in 2002 in the DRC, when donors resumed their support for the country’s health sector after decades of civil war and socioeconomic crisis. Currently, various forms of RBF initiatives exist in all 11 provinces and in 189 of the 515 HZs. As part of a financing strategy under the USAID-funded IHP, MSH is piloting a supply-side RBF model, or PBF, in seven selected HZs (Bibanga, Kanzenze, Kayamba, Lomela, Luiza, Wembonyama, and Nundu) across four provinces of East Kasai, West Kasai, Katanga, and South Kivu. MSH has adopted a specific type of RBF intervention model, PBF. The objective IHP intends to meet through its PBF intervention is a rapid scale-up of health services and improved quality through PBF contracts mechanisms. IHP’s PBF model operates at three levels: 1) the national level (MSP and IHP’s Kinshasa-based team); 2) the provincial level 1 USAID. (2008) Paying for Performance in Health: A Guide to Developing the Blueprint. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 11 (District health office and facilities and IHP’s Coordination Bureau-BC offices); and 3) the periphery or operational level, including ECZ, GRHs, and HCs. At the central level, the MSP plays a regulatory and supervisory role with regard to the implementation of PBF at the provincial level and the harmonization of the program across provinces. At the intermediary/institutional level, coordination offices of IHP are responsible for the distribution of funds to the contracting HZ management committees, general hospitals, HCs and community organizations; supervision of the coordination offices; monitoring and evaluation; and developing a PBF model at the national level in collaboration with MSP. At the periphery/operational level, the HZ management committee has a regulatory and supervisory role with regard to monitoring of activities and ensuring quality of services, training and capacity building, promotion of community activities, and the coordination of RBF contract and performance payment services. Contracting and performance payments are made at the operational/periphery level, and the GRHs and HCs are responsible for ensuring quality performance and delivery of priority health services. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 12 II. EVALUATION METHODS AND LIMITATIONS EVALUATION TEAM The assessment team included Annette Bongiovanni, Team Leader, and team members Zephyrin Kanyinda, RBF Specialist, and Swati Sadaphal, Data Analyst. Field work was supported by two French Interpreters (Hurbert Kinwa and Alphonse Yulu Kabamba), and a team of six Research Assistants (Mike Kilolo, Charles Kassongo, Thierry Tshikuz, Thierry Junior Nsikuetu, Adolphe Kamangu and Jacques Kabongo) who administered HF and household surveys. Logistic and administrative support was provided by staff based at IBTCI home office and by Germaine Kawal in the field. Once in the provinces, our assessment team divided into two sub-teams in order to maximize the geographic reach of the study. STUDY DESIGN The study design was a non-experimental descriptive process assessment using predominantly qualitative data collection and analysis methods triangulated with quantitative data (both primary and secondary). The unit of analysis is stakeholders at various levels of the health system (i.e., central—province—health areas—community). The sample size of respondents and facilities selected were decided in light of allocated resources and keeping in mind the intention to gain an in-depth insight into respondents’ perceptions of PBF implementation (see Annex II: Detailed Methodology and Analysis plan). SITE SELECTION STRATEGY The assessment utilized purposeful sampling to select sites and a wide range of stakeholders at the national level, provincial health authorities, HZ management teams, IHP BC staff, health workers, Comité de Développement Sanitaire/Health Development Committee (CODESA), local civil society organizations (CSOs), and traditional healers, such as traditional birth attendants (TBAs). The site selection criteria were as follows: 1. Current intervention sites with PBF implementation lasting at least six months; 2. No recent history of security concerns; 3. HCs accessible by road within 24 hours from the HZ headquarter; and 4. One GRH, and two high and two low performing HCs in each HZ. The team had planned to go to the highest performing HCs and lowest performance HCs based on their total PBF scores. However, IHP provided the team with the community satisfaction survey scores for each HC2 . Based on these scores, we selected two HCs with the highest and two HCs with the lowest scores and considered them high and low performing HCs respectively. 2 PBF Q2 reports submitted by CSOs to IHP MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 13 Sampling from any comparison sites was not included, since the emphasis of the assessment is descriptive nature, that is, a non-experimental design. Based on the above selection criteria, the following HZs and HFs were selected for data collection (Figure 1): Figure 1: Sites selected MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 14 DATA COLLECTION METHODS The qualitative data collection methods included desk review of existing documents and data, key informant interviews (KIIs), and focus group discussions (FGDs). Quantitative mini￾surveys of the HFs visited, and their clients who visited the facility in the past month (interviewed at their home), were conducted, which provided informative data (e.g., perception of community regarding quality of care, utilization of services etc.). Appropriate data collection tools, including survey questionnaires and discussion guides for KIIs and FGDs were developed in French and English (see Annex III: Data collection instruments and Annex IV: Field Implementation Plan). The back translation method and pre-tests were used to ensure quality. The KII and FGD instruments included a standard module of basic questions asked of all respondents and tailored modules for the type of respondent (e.g., MSP, IHP, HC provider, etc.). Desk review The evaluation team reviewed various documents from MSP, IHP and USAID, including the DRC IHP quarterly reports, updated RBF manual, MSP-RBF unit’s RBF review report, and all other related documents to assess the process, and current extent of, PBF implementation (See Annex V for a complete list of documents reviewed). Selected indicators from Système Nationale d'Information Sanitaire (SNIS) or National Health Information System of the DRC and PBF web portal (www.fbrsanterdc.cd) were also reviewed. KIIs A modest sample of 44 key informants were interviewed from across three provinces representing USAID; MSP officials at various levels in the health system; chief nurses and directors of GRHs, the central level administrators, managers, and technical advisors to the IHP PBF activity. (See Annex VI for a complete list of key informants interviewed.) FGDs A total of 20 FGDs were conducted with the community leaders—eight CODESAs, eight CSOs contracted by MSH to do the counter verifications at the household level, and the four small groups of TBAs in the villages. Household Mini-survey Clients who lived in the village where the HC was located and who visited the HC during the past month were interviewed at their respective homes. A total of 259 facility clients completed the interviews. The client questionnaire collected data on following: health services utilization characteristics; cost of services received; health education services received; and perceptions of the quality of care and services provided by HFs. Facility Mini-survey Facility mini-surveys were conducted in five HFs (one GRH and four HCs) in each HZ (N=20). The facility survey questionnaire included questions for data collectors to record facility observations on the cost of services posted at the HFs, and note indicator data collected through facility records and charts reviews. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 15 ETHICAL CONSIDERATIONS Verbal informed consent was administered to inform respondents of the purpose, process, potential risks, use, and confidentiality of the information, and their right to refuse to participate at any time. Client respondents were interviewed at their home to ensure privacy. All interviewers received training in ethical protocols to ensure that no identifying characteristics of respondents were recorded during data collection. Respondents did not receive any form of inducement or incentive to participate in the study and the survey team reiterated their external role in service delivery. All respondents were informed they could cease participation at any point during the survey or interview process. Survey data collection forms were stored securely by all team members during data collection. At the end of the data collection period, the paper questionnaires were sent to IBTCI’s home office for secured storage. DATA MANAGEMENT Each day, at least two random and unannounced data verification tasks were conducted by a senior evaluation team member. At the end of the data collection period, the paper questionnaires were sent to IBTCI home office for secured storage and data entry. A random sample of 20 percent of all paper questionnaires were checked for data quality issues, which were nominal. Data from the paper questionnaire were entered manually into spreadsheet formats in Microsoft Excel. Data were then converted into STATA data files. Data Analysis A data analysis plan was developed and guided the overall objective of the midterm assessment (Annex II). Quantitative data were analyzed using STATA Version 12. Analysis of the qualitative information was carried out with the software Atlas.ti Version 7. The quantitative and qualitative data were analyzed using following analytical domains: progress to-date; relevance of PBF; availability of funds at the operational level; ownership and management capacity; contextual factors, and unintended consequences. The analysis began with a first reading of the interview transcripts to acquire familiarity with the data. Categories and sub-categories were developed, modified and extended on the basis of what themes emerged. The qualitative information was then coded, compared, and re-categorized as new themes or issues emerged. Analyst triangulation was applied across all qualitative data sets. An additional valuable source of triangulation is provided by comparing findings across data sources (interviews, FGDs, and surveys) and across respondents (national, provincial and HZ officials, health providers, and community). Reference was also made to the baseline quantitative and qualitative analysis, latest available Performance Monitoring and Evaluation Plan results, and relevant government data sources (SNIS, PBF web portal) to elucidate understanding of the emerging mid-term qualitative findings. Statements that were indicative of general tendencies in the responses were selected for quotation. After complete data triangulation and the final result interpretation, the subsequent conclusion and recommendations were drafted. Limitations The primary limitation of the assessment methodology relates to the political and security situation in DRC. Remote and insecure areas are not included in sampling, leading to a sampling MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 16 bias. There is a selection bias for the household survey sample, since clients residing in the village where the HC is located were likely to have better access to health services. Other biases also exist such as “halo” bias whereby respondents tend to provide favorable impressions and perspectives of the activities. Other manifestations of such respondent bias include understating the actual situation or circumstances in anticipation of receiving donor support. Interviewer bias is also a concern, especially in a qualitative study. The nature of semi￾structured interview instruments for the KIIs left room for interpretation by the interviewers, especially if they asked the same questions in different ways and/or probed for answers. To mitigate these biases, the number of team members present during KIIs and FGDs was limited. As well, our experienced team reduced these biases as they worked together to develop the instruments. The team members maintained regular communication to relay all relevant information from the field, in case there were technical matters that had to be addressed, or any particular questions that were more prone to biases than others. During data analysis, at least two assessment team members conducted the analysis separately and compared findings. A comprehensive process evaluation for assessing the fidelity of PBF implementation was not justified given the nascent stage of the pilot intervention and therefore, resources were not allocated for such as study. As this was a small formative assessment of high and low performing PBF facilities, the results were meant to be a description of sites visited and not representative of all PBF sites supported by USAID. Nevertheless, the use of direct observation, a household survey among recent facility clients, in-depth interviews with managers, administrators, and providers, as well as FGDs with communities allowed the identification of achievements or potential gaps in the implementation process, and informed future programming, which was the overarching purpose of the assessment. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 17 “Before only a few people were coming, but now we see a larger number of people compared to last year. Look at the graph on the wall.” Chief Nurse III. FINDINGS In this section, key findings are presented according to each of the assessment objectives. OBJECTIVE 1: Effect of PBF intervention on the quantity of services Evidence suggests that PBF is increasing the quantity of services provided, but there are large variations across HZs and between the types of health services. SNIS data trend analysis of selected quantitative indicators (July-Sept 2014) from all 16 HFs (combined) shows a slight but steady improvement in the service utilization rates for key family health services (Antenatal Care (ANC), child vaccination, curative services and new acceptors of modern family planning (FP) among women ages 15-49 years) (Figure 2). Trend analysis of SNIS data from each HC visited show similar results (see Annex VII: Additional data analysis graphs). Almost all respondents (99 percent), including HC staff, CODESA, CSOs, TBAs, MSP and IHP reported an increase in numbers of patients visiting HFs to avail services, particularly for ANC, vaccination, FP and services for sick children. Ninety percent of CODESA FGDs reported that facility staff was present at facilities more often than before the PBF implementation started. They introduced strategies to increase demand and utilization of health services by the population they serve, primarily through decreasing user-fees and increasing community outreach activities. Facilities reported lowering of user fees by 30-50 percent, except for Kanzenze GRH where prices remained unchanged (the Kanzenze GRH is managed by a Catholic Mission). User-fees varied from site to site. For example, the fees for an initial visit for a child varied from 800 – 1,200 CDF ($0.86 – 1.30) and for an adult, 1,200 – 5,000 CDF ($1.30- 5.40). HC staff and ECZ mangers, along with CODESA, 34% 34% 36% 85% 99% 100% 21% 21% 25% 102% 80% 104% 0% 20% 40% 60% 80% 100% 120% July Aug Sept Service utilization rate Months Figure 2: SNIS data review for period July-Sept 2014 Curative Care ANC Modern FP DTP3 vaccine MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 18 “More people are coming to the health facility because prices have been lowered, more people can afford to go to the health facility.” CODESA member “I met with women who delivered their child and said they received good advice on FP – so I should mention –they are explaining how to care for the child and use FP. The first time we went, we didn’t notice any cases of FP but the second time, we heard some nice stories regarding the health staff that was doing outreach and explaining to them how family planning works”. CSO member who participated in counter verification activity usually decided the user-fees. However, according to CODESA members, this was not always the case. Due to the lowering of the user-fees, an increase in client attendance was reported by more than 80 percent of chief nurse respondents. Providers and/or community health workers and CODESA members motivated communities to mobilize sick individuals, pregnant women, and children, to seek HC services. This finding is corroborated by the household survey results. The most common channel for health and nutrition related information was the health worker (66 percent), followed by family (20 percent) and other (14 percent). Among health workers, nurses were the most common source of health information (75 percent), followed by community health workers (59 percent) and TBAs (37 percent) (Figure 3). MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 19 OBJECTIVE 2: Effect of PBF intervention on the perceived quality of services All facilities reported that they had a technical verification activity once during each quarter (Q), when senior MSP officials and IHP BC staff visited the HF. Using PBF tools, the following were discussed during these technical verification visits: adherence to standards, norms and guidelines; ways to improve interpersonal skills of providers; ways to ensure the availability of equipment, medicines and supplies; and facility data trends. Formations Sanitaires Complètement Fonctionnelles/Fully Functional Service Delivery Point (FOSACOF) was the only indicator (composite) measured at the GRH and therefore, the quantity of care was not being measured at this level. FOSACOF, although useful in monitoring the quality of care inputs, does not measure outcomes of the care offered by the providers. On the day of the household client survey at each facility’s health area, research assistants, with the health help of a chief nurse, prepared a list of clients to be surveyed based on pre￾determined selection criteria. The main criteria were to select female clients who attended the facility service for herself or for her child in the past month. Only two out of 259 households surveyed mentioned that they did not receive any HF services in the last month. The most common reason for a facility visit was curative care (63 percent), followed by vaccination (18 percent), ANC (16 percent) and FP services (three percent). The client household survey results revealed satisfaction with the services received during their last facility visit: 84 percent of clients were ‘very satisfied’, 13 percent were ‘somewhat satisfied’ and three percent were ‘dissatisfied’ with the service. FP and child vaccination services are provided free of charge in all surveyed facilities while other services, including services for pregnant women, child birth, curative services and medications, are all paid services. About 26 percent of client respondents received the above mentioned free services. There were large variations in the user-fees paid by the surveyed clients for various health services. The user-fee amount depended on the type of health service available, and differed by the HZ. Among those who used paid services (including all types and age categories), 48 percent paid user fees between 100 CDF ($0.10) and 1,500 CDF ($1.62), 38 percent paid between 1,500 CDF ($1.62) and 5,000 CDF ($5.40) and only 11 percent paid more than 5,000 CDF ($5.40). Anecdotal information gathered in FGDs in remote areas indicated an average household income of approximately 40,000-50,000 CDF ($43.20 - 54.00) per month. 80 percent reported that prescribed medication was available at the HF at the last visit. 16 percent of respondents reported that in the last six months, there was a time when they needed health services, but did not visit a HC or hospital. The most common reason given was that they did not have enough money to pay the bill (70 percent). There is an overall improvement in client perceptions on aspects of health worker behaviors and practices, appropriateness of facility resources, and cost of care measured at baseline and midterm (Table 1). More than 90 percent of all clients were satisfied with provider behavior and practices. Only 60 percent of clients believed that facility rooms are appropriate, and 79 “They [the verifiers] are looking at the medical records and comparing to the national protocols”. Director GRH MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 20 percent reported that wait-time is reasonable. 75 percent of clients reported that the fees for health services were reasonable and 59 percent reported that the cost of care was negotiable. 45 percent of clients expressed dissatisfaction with the fact that the pricing of services was not posted, and about 25 percent were dissatisfied with the availability of medications (or lack thereof) at the HF. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 21 Table 1: Comparison of perceptions of quality of care between baseline and midterm survey Perceptions of quality of care Disagree Neither agree or disagree Agree Baseline Midterm Baseline Midterm Baseline Midterm Perceptions on health workers’ behavior and practices Show compassion and support for patients 28% 6% 52% 2% 20% 92% Show respect for patients 27% 6% 54% 1% 19% 93% Are friendly/welcoming to patients 28% 5% 52% 2% 20% 93% Are honest 21% 4% 18% 2% 62% 94% Attentively listen to patients 20% 4% 15% 1% 65% 95% Nurses take enough time for patients. NA 8% NA 2% NA 90% Perceptions on appropriateness of facility resources The rooms are appropriate 36% 39% 15% 2% 49% 59% The waiting time is reasonable 32% 12% 49% 9% 18% 79% There are enough nurses 40% 25% 15% 4% 45% 71% Medications are available at all times 36% 25% 19% 5% 45% 70% Perceptions on cost of care Prices can be discussed 42% 37% 20% 4% 38% 59% Prices are reasonable 48% 21% 32% 4% 20% 75% Seen treatment prices posted 36% 45% 7% 8% 37% 47% Think paid the actual price that should have paid 30% 8% 20% 7% 50% 79% Medications can easily be obtained 39% 16% 17% 6% 44% 78% The distance from the center is reasonable for us (not too far). 25% 25% 10% 0% 65% 75% OBJECTIVE 3: Achievements to-date of the PBF activity PBF activities began following the signing of performance contracts with the 118 HCs, seven GRHs and seven ECZs between October and November, 2013. The first cycle of the PBF implementation period began on November 1, 2014, the second began on February 1, 2014, and the third, on May 1, 2014. During each PBF cycle, verification visits were conducted jointly by one member from the ECZ and an IHP staff member to verify facility-reported data. 14 community-based organizations were contracted to implement data counter-verification activities in seven HZs with two CSOs per HZ. Following the signing of the contracts, IHP trained CSOs on data collection methods. The CSOs received management tools to use during their data collection and counter-verification activities. Counter-verifications took place immediately after the verifications. Following both verifications, IHP central staff validated the reported data and finalized the payments for each contract. Each cycle of service delivery, reporting, verification and payment took about four to five months. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 22 “I would sum it up that the data from the HZs have become reliable, the quality of data has improved. Another aspect lies in the resolve and determination of the staff, which has improved. And all this has contributed to improved health service.” MSP HZ Manager Another achievement is related to capacity building of MSP-PBF unit, to function as the PBF ‘Regulator.’ IHP designed a PBF web portal for improved coordination, monitoring, and communication among MSP, IHP and other partners. The PBF web portal provides accessibility to all quantity and quality performance information on the PBF facilities for each Q. MSP’s PBF unit staff were trained on PBF processes and mentored during quarterly verification visits and data validation during the first 6 months of the PBF implementation. IHP also supported the PBF unit to carry out a 6 month review and report on the progress of the PBF intervention. More than 90 percent of respondents noted that PBF was an appropriate intervention for the DRC’s health system. Stakeholders understood the importance of accurate data reporting and monitoring as per performance contracts in order to improve PBF incentives to be received in subsequent Qs—even those with minimal or non-formal education level workers in rural HCs. However, there are a few bottlenecks impeding the PBF progress, as highlighted below. Bottlenecks impeding PBF Progress Lack of sufficient training on PBF reporting tools at service delivery level Sixty percent of IHP respondents mentioned that the Q1 cross-review of facility data showed problems such as poor use of data collection tools (incomplete pages and months, rushing, excess information, etc.), abnormal or inconsistent data in the registry (sex, age, address, etc.), the absence of certain key information, and the reporting of unverifiable cases during the verification process. These situations resulted in data invalidation over the course of the Q1 verification process. By the subsequent Qs, the aforementioned challenges were addressed through scheduled joint supervisory visits by MSP and IHP staff. The major problems they noted were the gaps between data reported by the SNIS, the self-reported data, and the verified data, as well as between the verified data and the validated data. Chief nurses who were recruited after November 2013 did not receive formal training on PBF reporting tools, and RBF manual did not include detailed instructions on how to complete various PBF forms. CSOs are distant from the communities All CSOs visited were based at the level of the HZ- they do not have a permanent presence in all health areas encompassed within a HZ. They were physically far from the HFs they audit and did not have any interventions in the surrounding community. Two out of eight FGDs with CSO respondents noted that this affected their quality of work because they faced difficulty in identifying clients due to non-familiarity with the local population and geography. CSO auditors needed to spend more time building rapport with the community to implement audits. There was no evidence that CSOs participated in the activities related to health promotion and facility MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 23 “Coordinating Offices are set up in such a way that no one is appointed exclusively for PBF.” IHP staff member improvement plans, and renovation of facility infrastructure. Three out of eight CSOs interviewed did not have previous experience working in health–related interventions and started working in health after receiving contracts from IHP. Two out of nine IHP respondents reported that the creation of local champion communities by IHP, to act as counter-verification agency, was not yet fully implemented. Payment of CSOs as a lump-sum flat rate All CSOs are paid an all-inclusive lump-sum fee of $4,500 for counter-verification activities per Q. On probing, none of the CSOs interviewed revealed how much it costs to complete one counter-verification activity in a given health area. However, two out of eight CSOs interviewed did raise the issue of having a constrained budget to implement activities. The roads and transportation conditions vary throughout DRC. Some remote areas are inaccessible by road and can only be travelled by bicycle and foot. To conduct counter-verifications in these areas, more manpower and time were needed. For example, the two CSOs in one province appeared to stop collecting data when they ran out of money. They reportedly visited roughly half of the households assigned, when 80 percent contact was expected (allowing for the 20 percent fictitious patients). Lack of full-time dedicated staff at IHP Kinshasa and BC Offices According to IHP respondents, there were only two full-time staff members assigned for PBF pilot implementation at IHP Kinshasa. There were three other IHP staff members who contributed 20 to 80 percent of their time. However, this contribution was ad hoc and not codified. There were no full-time staff members assigned for PBF at the IHP BC offices. OBJECTIVE 4: Availability of funds at the operational levels The maximum quarterly performance payments at the operational levels were as follows: $2,400 for ECZs, $12,054 for GRHs and $910 for health centers. As of October 2014, the 118 HCs received $163,564; seven GRH received $147,811; and seven ECZs received $24,694 as PBF incentives. These figures include payments for Q1 and Q2 only. All health facilities surveyed and ECZs received their PBF payments for Q1 and Q2 (Table 2). The general trend was an increase in funds from Q1 to Q2 as providers and/or staff improved their performance. The total payments for HFs increased by 10 percent, and for ECZs increased by 19 percent between Q2 and Q1. At the time of the assessment, none of the HFs or ECZs visited received incentives for the third cycle of PBF, which were expected within one month of completion of counter￾verification, as per the PBF contracts. Complaints of delayed incentive payments were expressed suggestively in every HZ visited. About 19 percent of chief nurses mentioned that prolonged delays in payments could lead to de-motivation of health facility staff. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 24 Table 2: Percentage performance payments earned by HFs and ECZs during Q1 and Q2 of the PBF intervention Health Facility % performance payments earned Q1 % performance payments earned Q2 Difference (Q2-Q1) Bibanga GRH Bibanga 91% 91% 0% HC Cikuyi 81% 96% 15% HC Bibanga 26% 32% 6% HC Station 64% 100% 36% HC Katanga 1 35% 81% 47% Kanzenze GRH Kanzenze 95% 93% -2% HC Kamoa 98% 98% 0% HC Mpala 30% 85% 55% HC Kantala 40% 66% 25% HC Kamimbi 54% 92% 38% Wembonyama GRH Wembonyama 90% 100% 10% HC Olota 85% 89% 4% HC Ahamba 68% 79% 11% HC Tshekopoto 46% 79% 33% HC Otohe 46% 95% 49% Luiza GRH Luiza 63% 77% 13% HC Kamayi 63% 89% 26% HC Kamushilu 78% 84% 7% HC Kitoko 53% 85% 32% HC Kabuanga 70% 95% 26% TOTAL 79% 89% 10% ECZ % performance payments earned Q1 % performance payments earned Q2 Difference (Q2-Q1) ECZ Bibanga 68% 89% 21% ECZ Kanzenze 91% 96% 5% ECZ Wembonyama 54% 80% 25% ECZ Luiza 49% 73% 24% TOTAL 66% 84% 19% Before PBF, most service providers relied heavily on user fees to cover the operating costs of the facilities as well as to pay bonuses, or the “Prime”, to staff. In facilities, bonuses or “Prime” were low for those who did receive them, while many posted staff were not even on the civil service payroll, deriving their remuneration solely from fees charged to patients. In the facilities MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 25 visited by the assessment team, less than 10 percent of staff members received salaries. No health facility, however, reported a decrease in the overall facility net income due to lowering of user fees since PBF started. Respondents reported that PBF amounts received were divided into three parts; 60-70 are percent allocated for staff incentives, 10-20 percent of the funds are to be expended on ‘investments’ which has been manifested in expenditure on infrastructure (i.e., repairing facility building, building pit latrines and burial pits for medical waste management) and 10 percent to cover operational costs (i.e., patient registers). OBJECTIVE 5: Contextual factors which might influence PBF intervention results Compliance of the IHP RBF Manual with the MSP PBF Unit's operations guide Based on a review of IHP’s PBF manual (August 2014) and MSP’s PBF operations guide (October 2012), IHP PBF’s design was compliant with the MSP's policies and directives concerning the underlying principles, the implementing entities and their roles, and the levels of contractualization, as well as the entities that executed a PBF program. The IHP’s PBF implementation manual was modeled on the requirements included in the MSP’s operations guide prepared in October, 2012. In terms of PBF implementation principles, IHP PBF emphasized best practices; the Ministry's guide to the basic principles of a successful PBF program included the following:  Separation of functions;  Quality of care;  Cooperation among actors;  Public-private partnerships;  Independent management of health facilities;  Contractualization;  Financial viability of health facilities; and  Strengthening the community's voice. Concerning the actors who implemented PBF, the MSP’s operations guide looked at the entire health pyramid, beginning with the central level and continuing out to the peripheral level, while IHP PBF, at this stage, focused on the operational level only. IHP PBF created contracts with HZ actors and involved actors from the central and intermediate levels in supervision. Concerning implementation entities, the operational guide listed a series of entities at the central, intermediate and peripheral levels that must be stakeholders in PBF implementation. It specified entities that should direct the strategy and regulations. It also specified that funds should be directed to national fiduciary agencies and provincial fiduciary agencies in terms of funding. The IHP PBF implementation manual did not clearly mention these entities' involvement in implementation, although the IHP Office at Kinshasa served as the national fiduciary agency. In the institutional structure, the IHP coordination offices were verification and coaching entities for health facilities, but did not serve as provincial fiduciary agencies. Insofar as PBF implementation required a clear separation of functions, with each function being specific, the operational guide mentioned five major functions: 1) the regulator (at various levels of the health pyramid); 2) the service providers; 3) the verifiers; 4) the purchaser, and 5) the payer, with a clear distinction between the roles of purchaser and payer. The IHP MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 26 implementation manual clearly separated the various functions listed in the operational guide, and the national office served as both purchaser, for the negotiation of indicators, and payer, for directing funds to HFs. Although the guide assigns an agency to the role of verifying the services that were purchased, the IHP manual mentions a joint verification method, involving MSP actors at the intermediate and peripheral levels, and the provincial coordination offices. Community verification appeared in both documents, as did technical verification. According to the MSP operations guide, quality was specifically tracked in terms of the services that facilities provided. To do this, the quality evaluation rubric was to be used for Formation Sanitaire -FOSAs (includes health centers, general reference hospitals and for other contracting facilities). These quality evaluations could lead to a quality bonus, of which at least 50 percent would be assigned to investments. The IHP manual promoted FOSACOF as a tool for measuring quality, and FOSACOF scores were one of the 16 "paid indicators" at the HC level, weighted at 29 percent of the total performance score. Hospitals' payments were based 100 percent on the quarterly FOSACOF score. Ownership by stakeholders All central MSP PBF unit respondents expressed their commitment to the success of PBF in the DRC. The activities such as joint trainings, tools development, and monitoring and evaluation of pilot sites were mentioned as the learning opportunity which would help them in future, even if IHP ends. However, three out of nine IHP respondents noted that the role of the central PBF unit was still more of a “coordinator” rather than a “regulator”. All Division Provinciale de la Santé/Provincial Division of Health (DPS) and Médecin Inspecteur Provincial/Provincial Medical Inspector (MIP) staff members interviewed was highly supportive of the IHP PBF model. They want the model to be expanded to all IHP-supported HZs and to all three levels of health system pyramid, including provincial and district levels. Linkages between community and health centers All CODESA FGDs noted that they helped facility staff by encouraging pregnant women and people who are ill to attend services, or gathering community members and children on immunization day. But respondents were not involved in the planning or the monitoring and evaluation of HC activities. Only 25 percent of CODESA reported that they were consulted on how to use PBF incentives for improving facility operations and infrastructure. It was observed that strongest sites had deep linkages between the HC and the CODESA. Environmental factors A review of IHP and MSP’s reports highlighted a wide range of environmental factors that may have an effect on PBF implementation. Such factors include civil and political unrest in South Kivu and poor geographic accessibility in certain HZs, such as Lomela and Kayamba. The lack of “The foundation of sustainability is there. However, it’s too early to talk about sustainability, and people are not clear about this issue. On the ground, things happen slowly.” IHP staff member MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 27 “Management of data on the web portal- (sighs) for us at the province we don’t have internet and we have to go to cybercafé and it becomes quite complex”. MSP Provincial Officer “We do not have entire control. For example, there is no banking system and roads are poor. The national system needs an overhaul. It has a long way to go.” IHP staff member “Too much time is involved in managing money and reports. For example: when an ECZ manager comes to collect money for incentives, the whole day of his is spent here in this office.” MSP Provincial Officer “Without external funding, since the government budget line isn’t adopted yet, the chances of success are very low, too low.”’ MSP PBF Unit Official “Success will depend on the working and living conditions for the HF staff because as long as there is money they will work and produce results.” IHP “The Health system pillars are in place, but clear allocation of resources is needed.” USAID paved roads and transportation resulted in widespread medicine and supply chain breakdowns and the population’s limited access to primary health care and referral services. Supply of electricity (by solar panel at best) was rare and running water was non-existent in the health areas visited. Chief nurse respondents in all HZs reported difficulty in maintaining cold-chains for vaccines/medications; safe medical waste management; and infection control practices. IHP was charged with calculating the amount due to each HF. ECZ was responsible for transferring the funds to the HF and ensuring that a receipt with the signature of the chief nurse was sent to IHP. The transfer of funds was done manually since banking facilities were not available in the rural locations. Plans called for the funds to be paid into the bank accounts of the contracting entities, but at present, very few health facilities have bank accounts. Health providers had to spend time and money to travel in order to receive funds in the presence of the IHP coordinating office. More than 60 percent of the national-level respondents noted that the current level of national budget spending on health is insufficient and additional funding resources will be needed for PBF success in the future. Other major factors for PBF success reported were the existence of effective primary health services and local infrastructures. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 28 “There is a computed Index Tool to note the physical presence, the receipt of staff salary/bonus, the performance (monthly performance evaluations), and education. You might find someone highly educated but is not performing or showing up for work and he/she might make less incentive than another lower, educated colleague.” Director GRH OBJECTIVE 6: Unintended consequences related to PBF implementation PBF fosters a foundation of transparency and accountability On IHP’s PBF manual review, it was evident that accountability is built-in at each level of the PBF activity. In order to pay for performance, performance needs to be measured, verified and validated in order to ensure that only verified and validated performance is rewarded. There can be two sources of risk: 1) poor data quality; and 2) explicit fraud. To address the data quality risk, facility-level, technical verification was carried out by the ECZ team, working together with IHP, as part of regular facility supervision. The team verified that the information reported by the facility corresponded with the information contained in the facility registers. They also monitored the quality of services being provided using FOSACOF tool. IHP played a supporting role in this supervision. The technical verification team selected a random sample of patients from the facility registers to be used for the community verification. The process of counter-verification by CSOs involves a sample of facility clients, randomly selected from the different facility registers by IHP, tracked and interviewed at their respective homes. The CSOs then compiled data, analyzed and transmitted a report on the community verification activity to the IHP. IHP also introduced fictional or “ghost” patients (20 percent of the total sample) into the community level data collection plan, as a method of preventing fraud or falsified data from the CSOs. All chief nurses reported that they did not have any direct role during counter-verifications, and that they only receive reports from IHP at the end of each Q. However, transparency between the HC and community was variable and perhaps not well assessed during the technical verification visits. Transparency between the chief nurses and their staff and also with CODESA was unclear. The monthly index tool for staff performance reporting is a complex tool, and could lead to subjective assessments if not carefully monitored. The Index Tool was not readily understood by all. Penalties were built in the event of data discrepancies. The approach to sanctions was not rigid, however, but rather treated on a case-by-case basis. In general, it was envisaged that the first instance of fraud would result in a written warning; a repeat occurrence with a 20 percent reduction in the value of the performance-based payment; a second repeat would reduce incentives by 50 percent; and on the third, cancellation of the contract. On probing, a case was brought up of fraud detection and subsequent penalty in the Bibanga HZ. PBF contributions to facility management IHP introduced concepts of target-setting, business-planning or work-planning, and technical verifications. All chief nurses and ECZ managers interviewed reported that they use IHP PBF MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 29 “We see all type of patients whenever they come, this is our job. PBF only taught us to do our job efficiently.” Chief Nurse tools. However, documentation from one of the HCs showed that the ECZ (or IHP) wrote that the nurse lacked analytical skills and capacity to diagnose, and did not have the Ordinogram available, etc. Yet the nurse received an 87 percent on FOSACOF Module 8: Clinical Skills. On the day of interview, the same nurse was unable to recognize neonatal tetanus and did not appreciate the urgency of the patients’ status (labored shallow respirations, grey/white color, unresponsive) for an urgent referral to the GRH. The PBF manual doesdid not provide written guidance for the medical record review component. Inequity by PBF design Based on document review and field observations, we noted that IHP PBF, by design, did not adequately address the differences in a) target population socioeconomic status, b) type of organization and c) geographic variations. For example, Katanga Province’s cost of living is much higher than in Kasai, yet the same amount of funds was allocated equally among all HCs and hospitals. HIV/AIDS and TB activities were very limited in Bibanga pilot zones, still all HCs, even with no activity, were required to report on these indicators every quarter. The GRH incentive payment was significantly higher than the HCs (maximum $ 12,000 vs. $910 per Q) and based only on FOSACOF scores, not service delivery indicators as required of the HCs. IHP respondents reported that there were plans to introduce client satisfaction indicators to measure GRH performance. It was observed that the contracts did not protect the Health Post staff. It was up to the chief nurses to devise a sub-contract with health post nurses. There were anecdotes of health posts that shut down after PBF and complaints from other health post nurses who felt they did more work but did not receive any of the bonuses. Negative effects of PBF (gaming, cheery picking, distortion) On probing, none of the respondents alluded to any negative effects of PBF related to gaming, cherry picking or distortion. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 30 IV. CONCLUSIONS OBJECTIVE 1: Effect of PBF intervention on the quantity of services PBF resulted in a slight, but steady improvement in the quantity of services contracted under PBF, but there are large variations across HZs and between the types of health services. Reduction of user-fees and increases in community outreach activities are the two major strategies used by the HF staff to improve client volume. OBJECTIVE 2: Effect of PBF intervention on the perceived quality of services PBF intervention is supporting improvements in the quality of services. There was an immediate behavior change noted among health providers who are now present at the health facility more often than before PBF started. Lowering of user-fees improved financial accessibility of health services. There was an overall improvement in facility perceptions by clients on aspects of health worker behaviors and practices, appropriateness of facility resources, and cost of care. OBJECTIVE 3: Achievements to-date of the PBF activity The PBF pilot intervention is progressing well. PBF activities were being implemented in 118 HCs, seven GRHs and seven ECZs. MSP’s PBF unit staff were trained on PBF processes and mentored during at the beginning of the implementation. IHP designed a PBF web portal to improve coordination, monitoring, and communication among MSP, IHP and other partners. A joint technical verification process was used by senior MSP officials and IHP BC staff to assess the performance at the service delivery point level. Community-level counter-verification was carried out by CSOs, independently of HFs and HZ management. However, there were a few bottlenecks impeding PBF progress. Due to the lack of formal training of new facility staff, there were gaps between the reported data and the validated data. A lack of sufficient full-time dedicated staff at central and BC level contributed to insufficient monitoring and some reasonable delays in the payment of incentives. The CSOs were not directly involved in HF improvement plans. The creation of champion communities by IHP, to act as counter￾verification agencies, was not fully implemented. CSOs did not seem to be tracking their expenditures or willing to share the actual cost. The coverage of household counter-verified remained less in remote areas, and the chances of gaming or fraud at the level of CSO may increase. OBJECTIVE 4: Availability of funds at the operational levels All HFs and ECZs received performance bonuses based on their performance levels during the Q1 and Q2. The total payments for HFs and ECZs increased between the two Qs. However, there was a delay in payments for Q3. PBF implementation was accompanied by the simultaneous reduction of user-fees. Before PBF, most service providers relied heavily on user￾fees to cover the operating costs of the facilities as well as to pay staff bonuses or “Prime.” Now, PBF amounts are divided into three parts; staff incentives, investments, and operational cost. There was no effect on the overall facility net income due to lowering of user-fees since PBF started. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 31 OBJECTIVE 5: Contextual factors which might influence PBF intervention results IHP’s PBF design was compliant with the MSP's policies and directives concerning the underlying principles, the implementing entities and their roles, and the levels of contractualization, as well as the entities that execute a PBF program. It was widely accepted by all stakeholders interviewed as an appropriate intervention for the DRC health system. Ownership by the country stakeholders was built-in to the IHP’s PBF design. IHP collaborated directly with the MSP's PBF unit to implement its pilot activity promoting ownership. Work is still needed when it comes to strengthening CODESAs to improve community participation and accountability at the local level. CODESAs were aware of PBF objectives but their role was either unclear or only limited to mobilizing community agents to gather patients to encourage them to use the HCs. They were not involved in PBF payment allocations for facility operations, resources planning, or monitoring and evaluation of HCs. IHP encountered a wide range of environmental factors that hindered PBF implementation and negatively influenced the results in a variety of ways. These factors were related to civil unrest and insecurity, geographic inaccessibility, poor infrastructure, and lack of sufficient national budget spending on health. OBJECTIVE 6: Unintended consequences related to PBF implementation PBF implementation clearly had positive unintended consequences. PBF introduced concepts of quality of care, target-setting, business and work planning, and technical verifications. Verifications and counter-verification processes supported transparency, accountability and improvement in data reporting. The technical verification process served other purposes simultaneously, such as capacity building of the MSP staff at the provincial and district levels, and especially at the ECZ and service provider levels. The ECZ had a vested interest in conducting individualized capacity building and supportive supervision during technical verifications because their indicators were, in part, dependent upon the success of the HC indicators. This was one of the strengths of the IHP’s PBF approach. However, on the other hand, the verification mechanism risked creating a conflict of interest at the HC level, insofar as the ECZs served as the verifiers, but were also contracted to conduct those same centers' performance. IHP PBF by design does not adequately address the aforementioned differences in target population socioeconomic status, type of organization and geographic variations. This could lead to unintended inequities. Performance of GRHs is measured solely on the quality, as the quantity of care is not being measured. Having one third of the overall health facility score weight assigned to FOSACOF seemed high. Those centers which were poor in infrastructure were at a bit of a disadvantage when it came to getting a higher score. It is possible, although it did not appear to be a problem that PBF implementation led to substantial negative unintended consequences related to gaming, cheery-picking, distortion. However, this possibility cannot be ruled out. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 32 V. RECOMMENDATIONS The recommendations present below are for IHP, listed in descending priority order, to improve ongoing activities of the PBF implementation. We do not recommend any major modifications to the pilot intervention in view of the forthcoming endline data collection for the impact evaluation study. Continue PBF pilot implementation IHP should continue with implementation of the PBF pilot intervention at all 118 HFs, seven GRHs and seven ECZs. Immediately appoint full-time staff dedicated to PBF The IHP PBF team needs to be staffed up immediately at the central and BC offices to handle the workload, especially considering need for data verification and validation. Strengthen PBF trainings at the operational level Validity and reliability of PBF data at the service delivery level needs continued attention. Priority target audiences for technical verification training should be focused on the newly recruited chief nurses, especially on PBF calculations and use of the Index Tool to calculate bonuses. They need more guidance on how to assess staff performance, and in general, the overall application of the tool. Provide written guidance for medical record review component The medical record review component is a critical component of the process for which more attention should be paid, in terms of the provision of written guidance in the PBF manual. Medical record review (cross-checked with other documents such as the partogram, medication distribution list, etc.) demonstrates written confirmation that the norms and protocols set out in the tools are adhered to, for example, the Ordinogram. Add performance indicators to measure the quantity of care at GRH The GRH contracts would benefit from having a more rigorous standard than just the FOSACOF. There should be service indicators that are proxies to assess GRH progress toward compliance with Complementary Package of Activities (CPA) Plus services. It is important to keep the FOSACOF as one indicator, but other service related indicators are needed. Client satisfaction scores could be highly subjective indicator. The satisfaction scores should not be given a higher weight than the health outcomes indicators. Provide additional training to CSOs on budgeting CSOs need to better understand the average cost per unit (household visited) to anticipate for budgeting purposes. An illustrative budget to accompany the $4,500 award would be helpful. Develop local community champions for counter verifications As far as possible, local community organizations should be selected to perform counter￾verifications. In the absence of such organizations, IHP could develop strategies of identifying MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 33 motivated community leaders, and develop their skills in creating local community champion organizations for counter-verifications. Build capacity of CODESA to participate in facility management CODESAs capacity should be strengthened in planning, and monitoring and evaluation of HCs’ activities. CODESA should be able to participate in the management of their HC and conversely, making HCs accountable to the communities they serve. Assess the problem of inequity There is a need to carefully assess the equity between a) types of facilities (HC versus hospital); b) cost of living between provinces and/or HZs; and c) baseline status of the HC infrastructure so the requirements are more balanced between and within the various facilities in different locations. The problem of inequity can be mitigated by rewarding improvement, in addition to absolute achievement, as well as additional incentives for HCs that specifically serve remote or poor disadvantaged populations. In order to have comprehensive and equitable coverage in a Health Area, the Health Posts should be able to receive bonuses. This might be too complicated to develop guidance or directives in the contracts, an initial assessment study could be conducted to address this issue. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 34 ANNEX I. STATEMENT OF WORK RESULTS-BASED FINANCING MIDTERM ASSESSMENT SCOPE OF WORK I. BACKGROUND Results-based financing (RBF) is a strategy for attaining positive health results through generally, financial incentives. RBF schemes can be developed for both supply (health worker, facility, district health team, community) and demand (patient/client) sides of the health system.3 A demand-side RBF intervention may give households cash incentives to receive preventive care services or to encourage completion of treatment. A supply-side RBF can contribute to increasing the quality of care and range of services, and generate positive health outcomes in two ways: first, by incentivizing providers to put more effort into specific activities with explicit performance targets, and second, by increasing the amount of resources available to finance the delivery of health services. Motivating health workers to provide quality services and keeping them in the public sector has been a particular challenge for the health system in DRC, as in many other countries. Fixed salaries with raises that are not tied to performance often lead to low productivity, poor quality, absenteeism, or lack of innovation. Moreover, payment of fees by clients for health services tends to result in greater attention to fee-generating services such as curative care, at the risk of preventive care and quality of services.4 RBF projects were implemented beginning in 2002 in DRC, when donors resumed their support for the country’s health sector after decades of civil war and socioeconomic crisis. Currently, various forms of RBF initiatives exist in all 11 provinces and in 189 of the 515 health zones. As part of a financing strategy under the USAID-funded Integrated Health Program (IHP), Management Sciences for Health (MSH) is piloting a supply-side RBF model in seven selected health zones (Bibanga, Kanzenze, Kayamba, Lomela, Luiza, Minga, and Nundu) in the four provinces of East Kasai, West Kasai, Katanga, and South Kivu. MSH’s has adopted a specific type of RBF intervention model, Performance Based Financing (PBF). The World Bank differentiates PBF as a particular form of RBF whereby 1) payment is made to providers of the services, not the beneficiaries (at any level of the health system, including managers), 2) only financial incentives are distributed (with some limited exceptions), and 3) remuneration is contingent upon degree to which certain targets have been met in terms of approved quality through predefined protocols and processes.5 The objective IHP intends to meet through its PBF intervention is a rapid scale up of health services and improved quality through grants and contracts mechanisms. IHP’s PBF model operates at three levels: 1) the national level (Ministère de la Santé Publique (MSP) and IHP’s Kinshasa-based team); 2) the intermediary level (District health facilities(office) 3 USAID. (2008) Paying for Performance in Health: A Guide to Developing the Blueprint. 4 Ibid 5 Musgrove, Philip. (2011) Financial and Other Rewards for Good Performance or Results: A Guided Tour of Concepts and Terms and a Short Glossary. The World Bank. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 35 and IHP’s Bureau de Coordination (BC) offices); and 3) the periphery or operational level, including Health Zone Management Committees (ECZ), General Referral Hospitals (GRH), and Health Centers (HC). At the central level, the MSP plays a regulatory and supervisory role with regard to the implementation of PBF at the provincial level and the harmonization of the program across provinces. At the intermediary/institutional level, coordination offices of IHP are responsible for the distribution of funds to the contracting health zone management committees, general hospitals, health centers and community organizations; supervision of the coordination offices; monitoring and evaluation; and developing a PBF model at the national level in collaboration with MSP. At the periphery/operational level, the health zone management committee has a regulatory and supervisory role with regard to monitoring of activities and ensuring quality of services, training and capacity building, promotion of community activities, and the coordination of RBF contract and performance payment services. Contracting and performance payments are made at the operational/periphery level, and the GRHs and HCs are responsible for ensuring quality performance and delivery of priority health services. USAID/DRC RBF IMPACT EVALUATION USAID/DRC has contracted International Business and Technical Consultants, Inc. (IBTCI) to conduct an independent IHP PBF impact evaluation. The impact evaluation methodology uses a prospective quasi-experimental design with intervention and comparison groups covering all seven PBF health zones, with measurements taken at baseline and endline. Per the scope of work for this contract, the final impact evaluation will answer the following illustrative questions: 1. Is there evidence of change among health centers in the quantity and quality of services that is attributable to the PBF model? 2. What difference did the PBF intervention make? 3. Is the model worthy of being scaled up to other health zones? 4. What costs are associated with a potential replication of the model? 5. Were the desired results achieved? 6. Do results differ for various groups? (heterogeneity) 7. What contextual factors contributed to or limited the desired results? 8. What are the unintended consequences of the intervention? OBJECTIVES FOR THE RBF MIDTERM ASSESSMENT IBTCI completed the RBF baseline evaluation in 2013. A midterm qualitative assessment is planned to be conducted during 2014 that will focus more on programmatic and managerial approaches in implementing the PBF intervention. The main objectives of the mid-term assessment are listed below: 1. Assess the initial effect, if any, the PBF intervention has had on the quantity of services. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 36 2. Assess the initial effect, if any, the PBF intervention has had on the perception quality of services. 3. Assess the achievements to-date of the PBF objectives and describe any bottlenecks that are impeding progress. 4. Determine any changes between the planned versus the actual availability of funds at the operational levels (i.e., HZ Management Team and facility). 5. Identify and analyze contextual factors which might influence the results of the PBF intervention. 6. Gather preliminary information and describe unintended consequences related to the implementation of the PBF intervention. To address each of the above objectives, anticipated data collection methodology includes: document review, key informant interviews, focus group discussions, and direct observations. SPECIFIC TASKS 1. Illustrative methodology and finalization of the workplan:  Literature review of key documents (e.g., applicable USAID sections and IHP project documents; relevant legislation and regulations regarding the health sector; IHP quarterly reports, documents and studies/evaluations on RBF in DRC, among others.);  With the input of USAID and MSH, design the sampling plan including selection criteria for sites and respondents;  Develop midterm study tools for: key informant interviews using semi￾structured questionnaires; focus group discussions (if applicable) using FGD facilitator guide  Prepare a list of key informants;  Facilitate tools translation where necessary;  Plan the logistics for fieldwork; and,  Seek appropriate approvals from local authorities, and create systems for ethical data collection and handling (e.g., standardized informed consent (written) with verbal consent by respondents) 2. Fieldwork  Conduct data collection, and ensure data quality through field coordination and supervision of teams. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 37 3. Analysis of key findings, conclusions, and recommendations  Data entry and analysis of collected data (e.g., content analysis)  Perform data triangulation  Prepare the draft report and synthesize findings into recommendations for system improvement;  Make any necessary revisions based on Mission input; and  Submit draft and final reports. 5. Structure of the midterm report  Executive Summary: Key findings and recommendations  Introduction and Background  Summary of the current status of IHP PBF implementation  Objective of the midterm assessment  Presentation of findings  Conclusion  Recommendations  References  Annexes: Annexes may include but are not limited to 1) Tables/Graphs/Figures; 2) Methodology/ Tools;3) Scope of Work; and 4) List of key informants, sites visited, document reviewed. DELIVERABLES 1. Work plan/schedule: To be submitted three days prior to the start of fieldwork 2. Debriefing to USAID and MSH: A debriefing on preliminary key findings for the PBF midterm in Kinshasa following the termination of fieldwork and data entry. 3. Draft midterm report: The draft midterm reports will be submitted in electronic versions no later than 36 business days after the completion of data collection. 4. Final midterm report: The final midterm report will be completed within ten working days after the Mission submits comments on the first draft of the report. LEVEL OF EFFORT AND TIMING The midterm assessment will require approximately 12-14 weeks of effort, based on a six-day work week in the field. The research team members will include the Team Leader, RBF Specialist and Project Director/Data Analyst. The estimated time for conducting both the fieldwork including de-brief is three weeks. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 38 ANNEX II. DETAILED METHODOLOGY AND DATA ANALYSIS PLAN STUDY DESIGN The study design is a non-experimental descriptive process assessment using predominantly qualitative data collection and analysis methods. The unit of analysis will be stakeholders at various levels of the health system (i.e., central—province—health areas-community)6 . We will employ a comparative analysis of outlier sites—those which have embraced PBF and have done well on the quarterly indicator checks. We will want to visit the centers that have been trained on the PBF intervention, but for some reason have not mobilized the effort yet. We propose a modest sample of key informants from across three7 provinces representing the MSP voice at various levels in the health system. Also represented will be the central level administrators, managers and technical advisors to the IHP project writ large and PBF more specifically. At the other end of the spectrum, we will convene numerous focus group discussions (FGD) with the community leaders—the community members (CODESA) the community service organizations (CSO) contracted by MSH to do the quality verifications at the household level, and the traditional birth attendants and other traditional healers in the villages and interview with clients who visited health facilities during the past month. These people will have an interesting story to tell. Set on a backdrop of a comprehensive document review, the key informant interviews (KII) and FGD respondent data will be triangulated with facility survey data and PBF indicators reported to IHP. As this is small formative assessment of high and low performing PBF facilities, the results are meant to be descriptive and not representative of the PBF sites supported by USAID. SAMPLING STRATEGY The assessment will rely on purposeful sampling for assessment sites and a wide range of stakeholders at the national, provincial health authorities, health zone teams (ECZ), IHP BC staff, health workers, CODESA and local community organizations and traditional healers (i.e., traditional birth attendants -TBAs). The site selection criteria are as follows: 5. Current intervention sites with PBF implementation duration at least 6 months (According to IHP/MSH, all pilot sites started implementing PBF around Nov/Dec 2013) 6. No recent history of security concerns (S. Kivu currently has security concerns) 6 We are configuring the Field Implementation Plan (FIP) now and it seems doubtful we could go to the District levels. The district capitals are not necessarily going to be located close to the outlier clinics where we want to visit (i.e., high versus low performing PBF intervention). 7 Ideally, we want to go to three of the four provinces—all which are safely accessible. However, we might not have the time because of logistics. We will know after we have a draft of the FIP. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 39 7. Health facilities reachable by road within 24 hours’ time from the health zone headquarter Based on the above selection criteria, the proposed health zones selection for data collection are (Figure 1): 1. Luiza (Province: W. Kasai, BC: Luiza), 2. Bibanga (Province: E. Kasai; BC: Mwene-Ditu) 3. Wembo Nyama (Province: E. Kasai; BC: Tshumbe) 4. Kanzenze (Province: Katanga, BC: Kolwezi) The site and respondent selection will be finalized with USAID/DRC and IHP/MSH. Sampling from any comparison sites is not included, since the emphasis of the assessment will be of descriptive nature, that is, a non-experimental design due to budget limitations. This study will delve into issues surrounding the implementation of PBF therefore respondents from the comparison groups would not be able to relay those experiences. Figure 1: Proposed sites for data collection: Province, BC, and Health Zones SAMPLE DISTRIBUTION At the central level, the primary stakeholders targeted will be the MSP team in charge of Performance Based Financing; the IHP senior management team including the person responsible for the management of the PBF intervention and selected USAID staff familiar with the project. Once at the provincial level, respondents will include the Provincial Medical Officers; HZ Management Team, selected GRH staff, HC staff, CODESA members, MSH￾contracted CSO representatives, and traditional healers including TBAs. Figure 2 below provides an illustrative example of the sites to be visited within each of the three provinces. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 40 Figure 2: Sample Section at the IHP/BC Level: Luiza BC Example ASSESSMENT TEAM The assessment team will include Annette Bongiovanni, Team Leader, Zephyrin Kanyinda, RBF Specialist and Swati Sadaphal, Project Director/Data Analyst. Field work will be supported by two French Interpreters (Hurbert Kinwa and other-TBD), and a team of four data collectors to administer observation checklist during health facility observations and conduct focus group discussions. Logistic and administrative support will be provided by staff based at IBTCI home office. Once in the provinces, the team will divide into two sub-teams, A and B in order to maximize the geographic reach of the study. DATA COLLECTION METHODS AND TOOLS The data collection methods include desk review of existing data (IHP Project PMP) and documents, key informant interviews (KII), health facility structured observations, and focus group discussions (FGD) and Costing tool. Mini-survey s of the health facilities visited and their clients who attended the facility in the past one month (interviewed at their home) will also be conducted and provide limited but hopefully informative data (e.g., posting of costs of services; availability of supplies and equipment needed to be compliant with the PBF indicators, perception of community regarding quality of care, utilization of services etc.). The MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 41 proposed sampling strategy, data collection methods for each type of data source are provided in Table 1 for illustrative purposes. Appropriate data collection tools, including facility observation checklist, semi-structured questionnaires and discussion guides for KII and FGD will be developed in French and English. The back translation method and pre-tests will be used to ensure quality. The sample size of respondents and facilities to be visited are decided in keeping with the intention to gain an in￾depth insight into respondents’ views and perceptions of PBF implementation. Table 1: Data sources and data collection methods8 Multi Levels Data Sources Qualitative data collection methods KIIs/FGDs/Surveys Total number/ # participants National Level USAID KII 3 Central MSP (Management & Technical) KII 5 IHP-PBF staff KII 5 Provincial Levels Provincial MSP staff (W. Kasai, E. Kasai, Katanga) 1 KII per province 3 IHP BC teams 2-3 KII per province 3 (6-9) Health Zone Levels Health Zone Management Teams (ECZ) Luiza, Kanzenze, Mwene-Ditu, Tshumbe 1 FGD per Health Zone 1 KII Health Zone Manager 4 (32) 4 Facility Level Health facilities (GRH and HC) Facility mini-Survey 20 (4GRH, 16 HC) Health centers (HC) staff 1-2 KII per HC 4 HCs per HZ 16-32 HCZ Team & GRH staff 1 FGD per HZ 4 (20) Community Level Traditional Birth Attendants (TBA) 1 FGD/KII per HC 16 Beneficiary (facility client) Mini-survey: 10 clients interviewed in their household per HC 10*4=40 Beneficiary (CODESA members) 1 FGD per HC 16 Civil Society Organization (CSO) members 2 FGD per HZ 8 (48) 8 This table will be completed and updated after the Field Implementation Plan has been finalized. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 42 The data collection tools (instruments) will be tailored to the type of information collected. There will be one semi-structured instrument for KIIs and a facilitators’ guide for the FGDs. The KII instrument will include a standard module of basic questions asked of all respondents and tailored modules for the type of respondent (e.g., MSP, IHP, HC provider, etc.) There also will be a structured facility survey instrument to record direct observations and a structured client interview questionnaire. LIMITATIONS The primary limitation of the assessment methodology relates to the political and security situation in DRC. Remote and unsecure areas are not included in sampling leading to a sampling bias. Other biases also exist such as “halo” bias whereby respondents will tend to provide favorable impressions and perspectives of the activities. Other manifestations of such respondent bias include understating the actual situation or circumstances in anticipation of receiving donor support. Interviewer bias is also a concern especially in a qualitative study. The nature of semi￾structured interview instruments for the KIIs leave room for interpretation by the interviewers, especially if they ask the same questions different ways and/or probe for answers. To mitigate these biases, we have limited the number of team members who will be conducting the KIIs and FGDs. As well, our experienced team will reduce these biases as they work together to develop the instruments. Further, during data collection, they will debrief daily to address any particular questions that are prone to biases more than others. Such outliers will be removed, if need be. During data analysis, at least two assessment team members will conduct the analysis separately and will compare and contrast their findings at a later stage. Using Atlas.ti software, team members will be able to discern how each other is coding the written transcripts and communicate differences during that phase of the study as well. A comprehensive process evaluation assessing the fidelity of PBF implementation is not possible given the allotted budget. Nevertheless, the use of direct observation, mini-surveys among facility clients as well as the interviews with managers, administrators, providers and community organizations will allow us to identify achievements or potential gaps in the implementation process and inform future programming which is the overarching purpose of the assessment ETHICAL CONSIDERATIONS IBTCI adheres to strict ethical guidelines as delineated in the IBTCI Ethical Standards and Protocols for Field Research which is based on the US National Institutes of Health guidance on the projection of human subjects. We will obtain standardized verbal voluntary consent from all participants of interviews and focus groups. The Belmont Report’s Ethical Principals of respect for persons, beneficence and justice are covered in our standardized Informed Consent form (USG, 1978). To protect confidentiality, no respondent identifying information will be collected on the data collection forms; only information necessary for data analysis, such a provider cadre/role, site type, region, etc. All interviews and FGDs will be pre-arranged. KIIs will take approximately 1.5 hours each. Hard copy data collection forms will be stored securely by all Team members during data collection. The Team Leader will determine the best way to store hard copy data to assure MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 43 data security, and to maintain confidentiality and privacy. All electronic data will be password protected to insure data security and to maintain confidentiality. DATA MANAGEMENT AND ANALYSIS PLAN Data Collection by Analytical Domains Below, each mid-term assessment objective is categorized according to analytical domains. A brief background of on the data source and collection process and in some cases, illustrative questions are included. The data analysis plan is based on the Midterm Assessment objectives and impact evaluation questions. However, impact evaluation questions 3 and 6 (described in the introduction section) will not be addressed in this study. Each Midterm Assessment objective is reflected by an analytical domain(s). The data collection instruments will be designed to reflect these domains to facilitate data analysis. During the course of the interviews, there is likely to be cross-referencing of the domains. This is the nature of qualitative interviewing and our interviewers will handle the coding of the domains based on the best fit between the response and the study objective. Analytical Domain: RELEVANCE OF PBF (quantity & quality of services) Objective: Assess the initial effect, if any, the RBF intervention has had on the quantity of services. Data Source: In addition to qualitative questions included in the Key Informant Interviews (KII) and the focus group discussions (FGD) with beneficiaries, we will apply facility checklist to collect data from health facility records on the trends in facility MCH services statistics before and after the PBF started. We will examine whether the PBF program affects the quality of MNCH services delivered in terms of: a) patient attendance, b) availability of medicine, c) consumables, d) changes in facility functioning such as infrastructure, e) informal or formal changes in user fees for health services, f) additional services, and g) supervision? NB: It might be easier to gather this information at the level of the Health Zone assuming their records are more easily obtained than from the health facilities themselves. We will collect year end data for 2013 and compare with current statistics July/August 2014. Objective: Assess the initial effect, if any, the RBF intervention has had on the perception quality of services. Data Source: Stakeholders at each level (National, HZMT, Health Facility, and Community) will be asked through KIIs and FGDs and mini-survey to ascertain their views on whether PBF leads to improvements in quality of facility services? We will also examine any effects on staff motivation and job satisfaction after PBF started. Health service providers will be probed to reflect on if and how the intervention has contributed to improve their working conditions, to increase their motivation, and to enable them to provide quality services to their communities. Community (CODESAs) and facility MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 44 clients will be probed to reflect on if and how the intervention is perceived by the communities whether facilitated better access (financial) to family health services and perception on the quality of these services. Analytical Domain: PROGRESS Objective: Assess the achievements to-date of the PBF objectives and describe any bottlenecks that are impeding progress. Data Source: Key Informant Interviews (KII), focus group discussions (FGD), record review will be the choice of methods to measure overall progress at this early stage of implementation. The protocols delineated in the IHP (PROSANI) Manual will be our main reference to index the expected implementation process. We will collect information from health providers regarding the reliability of the government paying their monthly salaries. We will describe any differences there might be in progress when data are disaggregated according to providers who receive regular salaries and those who have not been paid in the past year. Information regarding payment of salary will primarily be based on the providers recall as our team has no means of verifying whether government employees are paid. We will differentiate between payments for monthly per diem and salary. Our results will not be empirical as they are subject to a very small sample size and the data might not be valid. How did facilities use the resources gained from PBF? How did they adapt their operations in order to respond to the changed incentives introduced by PBF? Analytical Domain: BOTTLENECKS Objective: Assess the achievements to-date of the PBF objectives and describe any bottlenecks that are impeding progress. Data Source: Key Informant Interviews (KII), focus group discussions (FGD), record review will be the choice of methods to measure any bottlenecks or constraints which are impeding implementation. The protocols delineated in the IHP (PROSANI) Manual will be our main reference to index the expected implementation process. Explore the social and cultural setting of service delivery, shedding light on why providers manage the clinical encounter the way they do, what are facilitating and hindering elements to the delivery of quality care (within and beyond the PBF intervention), and what elements are responsible for motivation and satisfaction (within and beyond the PBF intervention) Part of examining the bottlenecks will include identifying if there are vulnerabilities to corruption in the model. Analytical Domain: COSTS Objective: Determine any changes between the planned versus the actual availability of funds at the operational levels (i.e., HZ Management Team and facility). Data Source: Using the costing tool applied during the baseline, we will gather data again from the sites selected to ascertain the current costs incurred to implement PBF in a health zone. Also of interest will be to reveal any initial information indicating changes in user fees. As MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 45 well as the actual costs, we will explore the process of distributing the incentives. Gather evidences on the PBF progress compared with provincial, ECZ and IHP work plans for PBF activities, reporting, funding mechanism (financing planning, accounting and reporting-records, capacity building plans). We expect to find some nominal information through direct observation of health facilities which relate to the costs (i.e., posting of user fees at the facility level). As well, the KII and FGD instruments will touch upon the effect the PBF intervention might have on user fees, additional costs expended to allow for PBF implementation, etc. Analytical Domains: OWNERSHIP & MANAGEMENT CAPACITY Objective: Identify and analyze contextual factors which might influence the results of the PBF intervention. Data Source: To achieve this objective, we will approach the analysis through the lens of two analytical domains: management capacity and ownership. Ownership Ownership Sub-objective: Assess the level of ownership of the PBF intervention among all stakeholders from national to facility and community so the government can adopt and support this approach at scale. Illustrative questions will include: a) What national policy guidelines and procedures on PBF exist? b) What is the perception of stakeholders at each level (National, HZMT, Health Facility, Community) on sustainability of PBF intervention? c) What are the plans for PBF scale￾up? Ownership Sub-objective: Assess the engagement of the CODESA members. We will explore the role of CODESA in PBF program and note any changes that might have been sparked by the PBF intervention. Data from this domain will contribute to the demand-side contingencies delineated in our Sustainability Measurement Framework presented in the PBF baseline report. Management Capacity Management Sub-objective: Assess the degree to which the local health authorities’ and CSOs are following through with their commitment as determined in their contracts with IHP. Illustrative questions will include the following: a) What is the role of each in PBF implementation? b) Are formal contracts in place as planned (with HZ Management Teams, Service providers (?), and Civil Society Organizations (CSO) collecting data at the community level)? c) Are data verifications done according to contracts, d) quality assurance checks to ensure the reliability and validity of the data collected by CSOs, etc.? We will also ask questions and look for any indications of conflicts of interest the CSOs might have which could potentially impede their neutrality. Management Sub-objective: Assess the IHP Leadership Development Program’s influence on the PBF component. Illustrative questions include: Are there differences in PBF pilot facilities with or without LDP training in regards to facility operations, management, supervision, coverage of MCH services, and patient attendance of MCH services? Management Sub-objective: Assess the Health Zone Management Team’s capacity to monitor services (e.g., through interventions such as the data verification process with the CSOs). Illustrative questions MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 46 include: a) What is the role of HZMT in PBF roll out? b) Has HZMT received PBF training? By whom, when, any refresher training plans? and c) What is the process of monitoring & evaluation and supervision by HMZT? Management Sub-objective: Assess the extent to which IHP BC teams are supporting the implementation of the PBF intervention. Illustrative questions are: What is the role of IHP BC teams in PBF program? What are the current management achievements or gaps, so far? Analytical Domain: UNINTENDED CONSEQUENCES Objective: Gather preliminary information and describe unintended consequences related to the implementation of the RBF intervention. Data Source: We look at the PBF indicators which are incentivized to see a) which services are included and b) the relative weighting among those services which are incentivized. Our KIIs and FGDs at all levels of the health system are likely to elucidate some of the unintended consequences of the PBF intervention. In particular we will explore questions with key informants and focus group respondents related to: 1. Prioritization of some services over others (e.g., less attention on non-MCH services) NB: We will look at which of the services are incentivized and the weight given to some services over others. 2. Effect on staff motivation, if any. 3. Effect on the workload of HZ Managers and service providers—are there other functions and services of these MOH staff which have received less attention now that RBF is being implemented? Has the overall workload been increased (e.g., increased hours worked)? 4. Deployment of HZ Managers and service providers: Has there been any switching out of existing (trained) staff for other staff in order to reap the benefits of the RBF programs directly (e.g., rewarding “favorite” employees by assigning them to RBF sites despite the need for a change and/or qualifications of the incoming staff. Data Management and Quality Assurance Prior to data collection, all data collection instruments will be field tested to ensure clarity and logical flow of questions. As needed, these tools will be revised to insure the highest level of data quality. Furthermore, the Team will collect qualitative data (KIIs and FGDs) in pairs, and will then compare notes as a quality assurance check. Research Assistants (RA) will conduct mini-surveys among facility clients at their homes and direct observations in health facilities using a structured survey tool as mentioned earlier. The RAs will be trained by the assessment team in data collection instruments, research ethics, interview skills, confidentiality. Each RA will report to the supervising team member at the completion of day’s work in the field. While in the field the two sub-teams will attempt to communicate whenever possible to review progress, and summarize emerging themes (findings) from qualitative data and plan for next steps. This will ensure that any discrepancies in qualitative data collection are addressed in timely manner. The Team Leader will provide overall supervision. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 47 All KIIs and FGDs will be conducted in the working language of the respondents (in French when possible, otherwise, a local language) and verbally translated to English by trained interpreters working with the assessment team members. All verbal material (interviews and FGDs) will be translated into English for analysis. Transcripts and translations will be checked for content consistency and accuracy. All collected and analyzed data will be saved in a secure location at the IBTCI Head Office until one year after the completion of the impact evaluation. Data Analysis Method Analysis of the qualitative information will be carried out with the software ATLAS Ti version 7. Using grounded theory approach, qualitative analysis will rely on an inductive standard comparison method (Glaser BG, Strauss AL 1999). The analysis will begin with a first reading of the interview transcripts to acquire familiarity with the data. Categories and sub-categories will be developed, modified and extended on the basis of what themes emerge as the analysis proceeds. The main coder Mr. Zephyrin Kanyinda, being RBF specialist and health economist, highly familiar with the DRC context and proficient in local language, will undertake preliminary coding of data sets. The qualitative information is then coded, compared and re-categorized as new themes or issues emerge. Analyst triangulation will be applied across all qualitative data sets. An additional valuable source of triangulation is provided by comparing findings across data sources (interviews, FGDs, and observations) and across respondents (national, provincial and HZ officials, health providers, and community). When needed, the assessment team will refer to the baseline quantitative and qualitative analysis, latest available PMP results and relevant government data sources (SNIS) to elucidate understanding of the emerging mid-term qualitative findings. During the process of writing up the findings the main coder will translate the quotes from French to English. Statements that are indicative of general tendencies in the responses will be been selected for quotation. After complete data triangulation and the final result interpretation, the subsequent conclusion and recommendations will be drafted. IHP PBF PILOT SITES RBF CONTACT LIST PROSANI, MSH Niveau central o Le coordonnateur de l’équipe FBR : Delmond Kyanza, dkyanza@msh.org, +243 (99 59 05 990) o Le Conseiller Technique Senior chargé de renforcement des capacités et assurance qualité : Freddy Tshamala : ftshamala@msh.org, +243 (97 00 01 689) o Le Conseiller Technique Senior chargé du suivi et évaluation : lkatambayi@msh.org, +243 (99 59 05 990) o Le responsable des contrats : Joel Amisi, jamisi@msh.org, +243 (99 59 05 990) Bureaux de coordination o Mwene Ditu : MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 48  Directeur : Didace Demba : ddemba@msh.org, +243 (97 00 07 772)  Point focal FBR : Jean-Pierre Bianga : jbianga@msh.org, +243 (99 41 70 356)  Représentante provinciale Mbuji-Mayi : Francine Ngalula : fngalula@msh.org, +243 (97 00 07 770) o Luiza :  Directeur ai et Point focal FBR : Matthieu Lutondo : mlutondo@msh.org, +243 (97 10 16 188)  Chargé de suivi et évaluation : Freddy Mukeba : fmukeba@msh.org, +243 (99 59 05 466)  Représentant provincial Kananga : Jean Kanowa : jkanowa@msh.org, +243 (99 59 04 470) o Kole et Tshumbe :  Directeur : Raphael Tshinzela : rtshinzela@msh.org, +243 (99 59 05 447)  Point focal FBR Kole : Emery Kapingani : ekapingani@msh.org, +243 (97 10 46 283)  Point focal Tshumbe : Albert Okitalutumba : aokitalutuma@msh.org, +243 (97 00 01 686) o Kamina:  Directeur : Sylvain Kasonga : skasonga@msh.org, +243 (97 00 07 776)  Point focal FBR : Alexis Ndumbi : andumbi@msh.org, +243 (99 19 27 624)  Chargé Assurance qualité : Faustin Bushabu : fbushabu@msh.org, +243 (97 00 07 766)  Représentant provincial Lubumbashi : Augustin Mwala : omwala@msh.org, +243 (99 59 05 465) o Kolwezi :  Directeur : Adamo Fumie : AdamoFumie.Bonay@rescue.org , +243 (99 52 00 206)  Point focal FBR : Francis Kambol, Francis.Kambol@rescue.org , +243 (99 52 00 319)  Représentant provincial Lubumbashi : Augustin Mwala : omwala@msh.org, +243 (99 59 o Nundu MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 49  Directeur : Jean Mpiana : Jean.Mpiana@rescue.org , +243 (99 52 00 703)  Point focal FBR : Luc Mweze Masirika: Luc.MwezeMasirika@rescue.org , +243 (99 77 42 050)  Représentant provincial Lubumbashi Barhobagayana Janvier, Barhobagayana.Janvier@rescue.org , +243 (99 52 00 700) MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 50 Introduction of the IHP Performance Based Financing Intervention Name of the PBF Health Zone Name of the Health Facility (GRH and health centers) Name of the Civil Society Organization BIBANGA (E Kasai, BC: Kamina) HGR BIBANGA Action communautaire pour le développement intégral du Kasai "ACDIK" CS BUFUA CS KATANDA 2 CS MOLOLA CS LUKANGU CS KALUNDA CS KAPONJI CS CIKUYI CS KABALA1 Action pour le Développement Durable et Intégré de Mwene Ditu "ADDIM" CS BIBANGA CS KATABUA CS STATION CS KASTHIAPANGA CS CIBILA CS MANJA CS KATANDA 1 CS CILUILA CS KABALA 2 KANZENZE (Katanga, BC: Kolwezi) HGR KANZENZE Planète santé CS Munanga CS Nzilo CS Mpala CS Musokatanda CS Tshala CS Kamimbi CS Tshamundenda 2 CS Kantala Aide pour la Scolarisation des Enfants en Milieu Rural " ASEMIR" CS NSEKE MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 51 CS TSHAD 1 CS KANZENZE CS WAKIPINJI CS MULOMBA CS KAMOA CS WALEMBA KAYAMBA (Katanga, BC: Kamina) HGR KAYAMBA Action des Femmes pour le Développement et la protection de l’Enfant "AFEDEPE" CS Mwala CS Kafuku CS Kisaho CS Kalamba CS Lufuishi CS Kibila CS Kamayi (moitié) Christ Fondation School for Life "CFSL" CS Kahako CS Lwamba sakadi CS SuluaLowa CS Mudindwa CS Kayi CS Mombela CS Kamayi (moitié) LOMELA (E. Kasai, BC: Kole) HGR LOMELA Association des Femmes pour le Développement du Sankuru"AFEDES" CS LOKALA CS IPEMBE CS DIAMAMBA CS ODILA CS MUKUMARI CS LOMELA PILOTE CS SHAMBI CS POKAONGO CS YANGUNDA CS EMAKOKO Union pour le Développement Intégral de Lomela "UDILO" CS ELINGAPANGO CS BAYAYA CS YOMBO CS ALANGA MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 52 CS ONYANGONDO CS VANGO CS IKOTO CS EDJOLA CS SHAIE LUIZA (W Kasai, BC Luiza) HGR LUIZA CENTRE MUKUNDA ISASA KAKALA KAMAYI KAMUSHILU KAZEA KITOKO MOMA MPIKAMBUJI MUBINZA TUTANTE Comité Paysan pour les Travaux de Développement "CONTRADE" BAMBAIE ISASA KABUANGA KAKAMBA KANDA KANDA KAPANGA KASONGA MUKUANDJANGA MUKUNGU NUNDU (s. Kivu, BC: Uvira) HGR NUNDU Association pour le Développement Intégré et Intégrale de Fizi "ADIF" CS I'AMBA CS MUNENE CS SWIMA CS AKE CS ABEKA CS KABUMBE CS LWEBA CS KILUMBI CH NAKIELE CS KANGULI MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 53 Kamati la Maji Safi "KMS" CS KABONDOZI CS MBOKO CS NUNDU Pilote CS KABOKE II CS KENYA CS LUSENDA CS MUKOLWE CS BITOBOLO CS LUTABURA CS ABALA CS PUNGU WEMBONYAMA (E. Kasai, BC: Tshumbe) HGR WEMBONYAMA Association des Jeunes du Sankuru pour la lutte contre le VIH Sida "AJSS" DIMANGA ODUKU OHAMBE OLOTA ONALOWA OSOMBA OTOHE TSHEKO POTO Action pour le Développement Intégré du Sankuru au CONGO "ADIS-CONGO" AHAMBA DIKOKA LOMEMBE LONDEKE LUSHIMAPENGE SHENGA VANGASHILO MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 54 TABLE 1B: DETAILED FIELD VISIT PLAN FOR TEAM B LWIZA AND BIBANGA (TEAM B) Date Activity Site Persons to be contacted Time frame 10/09/2014 Travel to Kananga Kananga from 8:30 to 11:30 am 10/09/2014 Meeting with meeting with IHP office Kananga IHP office from 1:00 to 2:30 pm 10/09/2014 Meeting with the provincial MOH morning Kananga MoH (MIP) from 3:00 to 3:30 pm 10/10/2014 Travel to Luiza Luiza from 6:00 am to 5:00 pm 10/11/2014 Meeting with IHP local office Luiza IHP local office. from 8:30 to 10:30 am 10/11/2014 Meeting with IHP BCZ medical doctor Luiza BCZ Medical Doctor from 11:00 am to 1:30 pm 10/11/2014 Visiting GRH Luiza GRH Director and staff from 3:00 to 5:00 pm 10/12/2014 Travel and data collection to Kamayi Kamayi chief nurse from 6:00 to 4:00pm 10/13/2014 Travel and data collection to Kabwanga Kabwanga chief nurse from 6:00 to 4:00pm 10/14/2014 Travel and data collection to Kamushilu Kamushilu chief nurse from 6:00 to 4:00pm 10/15/2014 Travel and data collection to Kitoko Kitoko chief nurse from 6:00 to 4:00pm 10/16/2014 Travel to Mwene Ditu by car Mwene Ditu from 11:00 to 4:30 pm 10/17/2014 Meeting with IHP BC Mwene Ditu IHP BC coordinator from 9:00 to 11:00 am 10/17/2014 Travel to Mbuji -Mayi Mbuji - from 12:00 to MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 55 Mayi 16:00 pm 10/18/2014 Meeting with provincial MoH Mbuji - Mayi provincial MoH from 9:00 to 11:00 am 10/19/2014 Travel to Bibanga Bibanga from 8:00 am to 2:00 pm 10/20/2014 Meeting with IHP BCZ medical doctor Bibanga BCZ Medical Doctor from 8:30 am to 10:30 am 10/20/2014 Visit GRH Bibanga GRH Director and staff from 11:00 am to 1:30 pm 10/20/2014 Travel to Cikuyi Cikuyi from 3:00 pm to 6:00 pm 10/21/2014 Data collection to Cikuyi Cikuyi chief nurse from 8:00 am to 4:00pm 10/22/2014 Travel and data collection to Katanda 1 Katanda 1 chief nurse from 6:00 am to 4:00pm 10/23/2014 Travel and data collection to station Station chief nurse from 6:00 am to 4:00pm 10/24/2014 Travel and data collection to Bibanga Bibanga chief nurse from 6:00 am to 4:00pm 10/25/2014 Travel to Mbuji -Mayi Mbuji - Mayi from 8:00 am to 2:00pm 10/26/2014 Travel to Kinshasa Kinshasa germaine MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 56 ANNEX III. Data Collection Instruments FOCUS GROUP DISCUSSION GUIDE: CODESA OR TBAS Discussion questions: CODESA or TBAs Evaluation objective 1. Assess the initial effect, if any, the PBF intervention has had on the quantity of services. [Analytical domain: relevance of PBF] 1. What can you tell us about the following key family health services at your health facility? (Probe for all questions below to ascertain if there are any differences observed today compared to one year ago?) A. What services are available at the health facility for pregnant women? A. Who provides antenatal services? What happens at a typical ANC visit? (Note: we are aiming to see the type of clinical care received and if any medicaments are disbursed.) B. What services are available at the health facility for childbirth? A. Who cares for the patient during childbirth? (Note: if the TBA attends the birth, is the health facility nurse present? where does delivery occur, e.g., the pregnant women’s home? the health facility?) C. What services are available at the health facility for child immunizations? D. What services are available at the health facility for family planning? E. What services are available for childhood diarrhea disease? A. In the past year, have there been any changes in the way children with diarrhea are treated? F. What services are available for fever or malaria? A. In the past year, have there been any changes in the way malaria patients are treated? B. Do you receive malaria bed nets? Who provides them? Name of Interviewer/Facilitator: ____________________ Date: ____________ Province: _____________BC: ____________Health Zone: ____________Village: ___________ Name of Facility/Organization: ___________________ Type of Facility: GRH __ HC__ GPS coordinates: Longitude: ______________Latitude: ______________ Type of respondents: CODESA  TBA  CSO  Contact information (Telephone no.) of respondent (one representative from the focus group): _________ MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 57 (Probe: note all the services that respondents mention without prompting them for each type of health service above. Refer to MPA list (provided at the back of this FGD guide) and probe specific activities listed in MPA under ANC, postnatal care and child health, malaria etc.) Evaluation objective 2. Assess the initial effect, if any, the PBF intervention has had on the perception of the quality of services. [Analytical domain: relevance of PBF] 2. Have you noticed any changes in the cost of services provided at this facility one year ago compared with today? If so, describe. A. What is the cost of childbirth at present? _____________, what was the cost one year ago_____________? B. What is the cost of immunization at present? ___________, what was the cost one year ago_____________? C. What is the cost of a curative visit at present? _________ What was the cost one year ago__________________________? 3 Where do patients usually obtain their medicaments? A. Are the necessary medicaments available at the health facilities? B. If there is more than one source for obtaining medicaments, please describe. C. Has the availability of medicaments changed in the past year? If so, give some illustrative examples of the cost of a medicament one year ago and that same medicament today? NB: probe to explore if patients might prefer to obtains their meds from one source over another source. The point is to better understand any issues with the distortion of prices for medicaments at the local level. 4 Are patients charged for medicaments? If so, what are examples of the costs? NB: probe to explore: Do you think the prices are reasonable? Do you know of any other place where you can get the medications at a lower price than provided at the health facility? Are prices similar in other health facilities? 5 What can you tell us about the functioning of the health facility? (Probe for all questions below: any difference observed from current and one year ago?) A. What is a usual wait-time to see the health provider? B. What do you think about the infrastructure of the health facility? C. What do you think about the health care provider’s professional skills? D. What do you think about the way health care provider interacts with patients? (probe regarding his interpersonal skills) For TBA FGD: additional questions 1. do you receive training from the health facility staff in: ANC care? MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 58 Child birth? Postnatal care? 2. Do you receive child birth kits from the health facility? If yes, what is the content? --------------------------------------------------------------------------------------------------------------------- FOCUS GROUP DISCUSSION GUIDE: CIVIL SOCIETY ORGANIZATIONS (CSO) Discussion Questions: CSO Evaluation objective 1. Assess the initial effect, if any, the PBF intervention has had on the quantity of services. [Analytical domain: relevance of PBF] 1. What do you think about the availability of key family health services in health facilities which has PBF roll out? A. services for pregnant women B. childbirth C. immunization D. family planning E. fever or malaria F. childhood diarrhea G. cough more than 2 weeks in a child or childhood pneumonia (Probe for all above: any difference observed from current and one year ago before PBF started?) (Probe: note all the services that respondents mention without prompting them for each type of health service above. Refer to MPA list (provided at the back of this FGD guide) and probe specific activities listed in MPA under ANC, postnatal care and child health, malaria etc.) Evaluation objective 2. Assess the initial effect, if any, the PBF intervention has had on the perception of the quality of services. [Analytical domain: relevance of PBF] Name of Interviewer/Facilitator: ____________________ Date: ____________ Province: _____________BC: ____________Health Zone: ____________Village: ___________ Name of Facility/Organization: ___________________ Type of Facility: GRH __ HC__ GPS coordinates: Longitude: ______________Latitude: ______________ Type of respondents: CODESA  TBA  CSO  Contact information (Telephone no.) of respondent (one representative from the focus group): MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 59 2. What do you think about the quality of key family health services in health facilities since the PBF has been implemented? A. wait-time B. cost of services C. infrastructure D. availability of medications E. provider professional skills F. provider intercommunication skills (Probe for all above: any difference observed from current and one year ago before PBF started?) G. Do you think health facility staff are focusing on the key family health services that are under PBF scheme at the expense of other services? (distortion) H. Do you think health facility staff is falsely reporting on patient or cases to increase their PBF incentive? (gaming) I. Do you think health facility staff are focusing on providing high quality services to patients or clients availing services that are under PBF scheme at the expense of other patients or clients? (cherry-picking) Evaluation objective 3: assess the achievements to-date of the PBF objectives and describe any bottlenecks that are impeding progress. [Analytical domain: progress] 3. What is the role of CSO in PBF implementation? 4. What are your achievements to-date in PBF? Probe: are there any bottlenecks impeding PBF implementation progress? 5. What is the process for community audits (also known as the verification process)? Evaluation objective 4. Determine any changes between the planned versus the actual availability of funds at the operational levels (i.e., HZ management team and facility). [Analytical domain: costs] 6. What is the funding source for verifications? How much does it cost? Evaluation objective 5: identify and analyze contextual factors which might influence the results of the PBF intervention. [Analytical domains: ownership and management capacity] 7. What are your biggest challenges and constraints in implementing community verifications under PBF roll out? Evaluation objective 6. Gather preliminary information and describe unintended consequences related to the implementation of the PBF intervention. [Analytical domain: unintended consequences] 8. Are you aware of whether there are any counter-verifications of the CSO? (Does anyone do any quality control checks to confirm you are providing accurate information?) 9. Who is aware of how your CSO conducts its verification checks? MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 60 A. How do you report your findings to PROSANI and the health zone management team? 10. What measures, if any, does your CSO institute to avoid conflict of interest in verification process? KEY INFORMANT INTERVIEW GUIDE: ECZ MANAGER Evaluation objective 1. Assess the initial effect, if any, the PBF intervention has had on the quantity of services. [Analytical domain: relevance of PBF] 1. How would you characterize effects, if any, of PBF intervention on the quantity of key family health services? A. services for pregnant women B. childbirth C. immunization D. family planning E. malaria F. childhood diarrhea G. childhood pneumonia (Probe for all above: any difference observed from current and one year ago before PBF started?) (Probe: refer to MPA list (provided at the back of this guide) and probe specific activities listed in MPA under ANC, postnatal care and child health, malaria etc.) Evaluation objective 2. Assess the initial effect, if any, the PBF intervention has had on the perception of the quality of services. [Analytical domain: relevance of PBF] Name of Interviewer/Facilitator: ____________________ Date: ____________ Province: _____________BC: ____________Health Zone: ____________Village: ___________ Name of Facility/Organization: ___________________ Type of Facility: GRH __ HC__ GPS coordinates: Longitude: ______________Latitude: ______________ Name of respondent:________________ Contact information of respondent: _________ Telephone: _________ Email: _____________ Type of respondents: ECZ Manager  MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 61 2. How would you characterize the effects, if any, of PBF intervention on the quality of key family health services? A. cost of services B. wait-time C. infrastructure D. availability of medications E. provider professional skills F. provider intercommunication skills (Probe for all above: any difference observed from current and one year ago before PBF started?) Evaluation objective 3: assess the achievements to-date of the PBF objectives and describe any bottlenecks that are impeding progress. [Analytical domain: progress] 3. What is your organization’s (ECZ/HZMT) role in IHP’S PBF intervention? 4. What support do you receive from IHP in implementing PBF activities? Probe: have you received PBF training? By whom, when, any refresher training plans? Any assistance with work plans? Probe: have they performed their role in ways that facilitates implementation? 5. What is your opinion about achievements of the PBF objectives thus far? Probe: where has progress been the strongest and why? Probe: where has it been the most challenging or weakest? 6. are there any bottlenecks impeding PBF implementation progress? Probe: what could have been done differently? Probe: if those measures were in place, would it mitigate the bottleneck (just described) Evaluation objective 4. Determine any changes between the planned versus the actual availability of funds at the operational levels (i.e., HZ management team and facility). [Analytical domain: costs] 7. Tell me about the availability of funds at the ECZ/HZ management team level to implement PBF activities? Probe: are there any changes between planned versus the actual availability? 8. Tell me about the availability of funds at the facility level to implement PBF activities? Probe: are there any changes between planned versus the actual availability? 9. Has the PBF intervention had any effect on user fees? Probe: Are there any preliminary information indicating changes in user fees? Evaluation objective 5: identify and analyze contextual factors which might influence the results of the PBF intervention. [Analytical domains: ownership and management capacity] MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 62 10. What do you think about ECZ/HZMT workload before and after PBF started? 11. What is your opinion about sustainability of PBF activities with IHP funding? Probe: how about without IHP donor funding? Evaluation objective 6. Gather preliminary information and describe unintended consequences related to the implementation of the PBF intervention. [Analytical domain: unintended consequences] 12. What do you think about the process of technical verification for health facility (HC or GRH) and ECZs levels? Probe: talk to me about the transparency of the PBF intervention? Probe: are measures in place to ensure transparency? If yes, describe. Probe: talk to me about any conflict of interest you are aware of? Probe: are measures in place to control for conflict of interest? If yes, describe. 13. What is your opinion on the reliability and validity of the data collected by CSOs, etc.? Probe: are CSOs supervised? If yes, describe: by whom, how? How often? Etc. Probe: are there any quality control measures in place to ensure the reliability and validity of the CSO data? Probe: what do you think how transparency and conflict of interest are managed? 14. Do you think health facility staff are focusing on the key family health services that are under PBF scheme at the expense of other services? (Distortion) 15. Do you think health facility staff is falsely reporting on patient or cases to increase their PBF incentive? (gaming) 16. Do you think health facility staff are focusing on providing high quality services to patients or clients availing services that are under PBF scheme at the expense of other patients or clients? (cherry-picking) 17. How high a priority of PBF activities in your day to day work (in a scale of 1-10, 1 being the lowest and 10 being the highest) :______ Probe: why? 18. How would you rate the likelihood of PBF scheme to be a successful intervention in future (in a scale of 1-10, 1 being the lowest and 10 being the highest): _____ Probe: Please explain - why? Name one critical factor for its success or failure? MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 63 KEY INFORMANT INTERVIEW GUIDE: HF DIRECTOR (GBH) OR CHIEF NURSE (HC) Evaluation objective 1. Assess the initial effect, if any, the PBF intervention has had on the quantity of services. [Analytical domain: relevance of PBF] 1. How would you characterize effects, if any, of PBF intervention on the quantity of key family health services? A. services for pregnant women B. childbirth C. immunization D. family planning E. malaria F. childhood diarrhea G. childhood pneumonia (Probe for all above: any difference observed from current and one year ago before PBF started?) (Probe: refer to MPA list (provided at the back of this guide) and probe specific activities listed in MPA under ANC, postnatal care and child health, malaria etc.) Evaluation objective 2. Assess the initial effect, if any, the PBF intervention has had on the perception of the quality of services. [Analytical domain: relevance of PBF] 2. How would you characterize the effects, if any, of PBF intervention on the quality of key family health services? A. cost of services B. wait-time C. infrastructure D. availability of medications E. provider professional skills Name of Interviewer/Facilitator: ____________________ Date: ____________ Province: _____________BC: ____________Health Zone: ____________Village: ___________ Name of Facility/Organization: ___________________ Type of Facility: GRH __ HC__ GPS coordinates: Longitude: ______________Latitude: ______________ Name of respondent: ________________ Contact information of respondent: _________ Telephone: _________ Email: _____________ Type of respondents: GRH Director  HC Chief Nurse/Manager  MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 64 F. provider intercommunication skills (Probe for all above: any difference observed from current and one year ago before PBF started?) Evaluation objective 3: assess the achievements to-date of the PBF objectives and describe any bottlenecks that are impeding progress. [Analytical domain: progress] 3. What is your opinion about IHP’s role with PBF activities? Probe: have they performed their role in ways that facilitates implementation? 4. Did you receive any training on PBF before the intervention began? Describe Probe: how many days, what did you learn, did you receive materials, do you feel prepared? 5. What is your opinion about achievements of the PBF objectives thus far? Probe: where has progress been the strongest and why? Probe: where has it been the most challenging or weakest? 6. Are there any bottlenecks impeding PBF implementation progress? Probe: what could have been done differently? Probe: if those measures were in place, would it mitigate the bottleneck (just described)? Evaluation objective 4. Determine any changes between the planned versus the actual availability of funds at the operational levels (i.e., HZ management team and facility). [Analytical domain: costs] 7. Tell me about the availability of funds at the facility level to implement PBF activities? Probe: are there any changes between planned versus the actual availability? 8. Has the PBF intervention had any effect on user fees? Probe: Are there any preliminary information indicating changes in user fees? 9. Do you have any recommendations for PROSANI staff and/or ECZ based on your PBF experiences? Evaluation objective 5: identify and analyze contextual factors which might influence the results of the PBF intervention. [Analytical domains: ownership and management capacity] 10. What is your opinion about the role of the ECZ team in PBF? 11. What is your opinion about the role of the CSO in PBF? Evaluation objective 6. Gather preliminary information and describe unintended consequences related to the implementation of the PBF intervention. [Analytical domain: unintended consequences] 12. What do you think about the process of technical verification for health facility (HC or GRH)? Probe: talk to me about the transparency of the PBF intervention? Probe: are measures in place to ensure transparency? If yes, describe. Probe: talk to me about any conflict of interest you are aware of? MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 65 Probe: are measures in place to control for conflict of interest? If yes, describe. 13. What is your opinion on the reliability and validity of the data collected by CSOs, etc.? Probe: are CSOs supervised? If yes, describe: by whom, how? How often? Etc. Probe: are there any quality control measures in place to ensure the reliability and validity of the CSO data? Probe: what do you think how transparency and conflict of interest are managed? 14. How long have you worked in this facility? __________________ Probe: when did you start work here: before or after PBF started? 15. How were you selected or posted to work in this facility? Probe: what was the selection process? Do you think you were specifically placed in this facility because it is a PBF facility? 16. Do you routinely receive your base salary from the MSP? Explain (NB: differentiate between salaries and monthly per diem for living expenses) Probe: since PBF has started, has this changed the receipt of your salary from the MSP? Explain Probe: if there has been a change in payment of your MSP salary since the PBF was started, to what do you attribute this change? 17. Did your facility receive any PBF incentives during last quarter 1? ______(y/n/dk) During last quarter 2? ______ (y/n/dk) If yes, probe: if PBF incentives distributed among health staff? And how? Were any incentives used to improve facility infrastructure or buy materials (including medications or supplies) for the health facility? 18. Do you think health facility staff are focusing on the key family health services that are under PBF scheme at the expense of other services? (distortion) 19. Do you think health facility staff is falsely reporting on patient or cases to increase their PBF incentive? (gaming) 20. Do you think health facility staff are focusing on providing high quality services to patients or clients availing services that are under PBF scheme at the expense of other patients or clients? (cherry-picking) 21. how high a priority of PBF activities in your day to day work (in a scale of 1-10, 1 being the lowest and 10 being the highest) :______ Probe: why? 22. How would you rate the likelihood of PBF scheme to be a successful intervention in future (in a scale of 1-10, 1 being the lowest and 10 being the highest): _____? Probe: please explain - why? Name one critical factor for its success or failure? MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 66 KEY INFORMANT INTERVIEW GUIDE: USAID/MSP/IHP Evaluation objective 1. Assess the initial effect, if any, the PBF intervention has had on the quantity of services. [Analytical domain: relevance of PBF] 1. How would you characterize effects, if any, of PBF intervention on the quantity of key family health services? A. services for pregnant women B. childbirth C. immunization D. family planning E. malaria F. childhood diarrhea G. childhood pneumonia (Probe for all above: any difference observed from current and one year ago before PBF started?) (Probe: refer to MPA list (provided at the back of this guide) and probe specific activities listed in MPA under ANC, postnatal care and child health, malaria etc.) Evaluation objective 2. Assess the initial effect, if any, the PBF intervention has had on the perception of the quality of services. [Analytical domain: relevance of PBF] 2. How would you characterize the effects, if any, of PBF intervention on the quality of key family health services? A. cost of services B. wait-time C. infrastructure D. availability of medications E. provider professional skills F. provider intercommunication skills (Probe for all above: any difference observed from current and one year ago before PBF started?) Name of Interviewer: ____________________ Date: ____________ Name of Facility/Organization: ___________________ Name of Respondent: ____________________ Job title of Respondent: __________________ Contact information of respondent: Telephone: _________ Email: ___________________ Type of respondents: USAID  MSP  IHP  MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 67 Evaluation objective 3: assess the achievements to-date of the PBF objectives and describe any bottlenecks that are impeding progress. [Analytical domain: progress] 3. What is your organization’s role in IHP’s PBF intervention? 4. What is your opinion about IHP’s role with PBF activities? Probe: have they performed their role in ways that facilitates implementation? 5. What is your opinion about achievements of the PBF objectives thus far? Probe: where has progress been the strongest and why? Probe: where has it been the most challenging or weakest? 6. are there any bottlenecks impeding PBF implementation progress? Probe: what could have been done differently? Probe: if those measures were in place, would it mitigate the bottleneck (just described) Evaluation objective 4. Determine any changes between the planned versus the actual availability of funds at the operational levels (i.e., HZ management team and facility). [Analytical domain: costs] 7. Tell me about the availability of funds at the HZ management team level to implement PBF activities? Probe: are there any changes between planned versus the actual availability? 8. Tell me about the availability of funds at the facility level to implement PBF activities? Probe: are there any changes between planned versus the actual availability? 9. Has the PBF intervention had any effect on user fees? Probe: Are there any preliminary information indicating changes in user fees? Evaluation objective 5: identify and analyze contextual factors which might influence the results of the PBF intervention. [Analytical domains: ownership and management capacity] 10. What is the role of IHP in MSP’s capacity building at national, provincial and health zone level? 11. What are the coordinating mechanisms for IHP’s PBF activities with MSP? How is that working? Has it been an effective partnership with MSP? 12. Do you have any specific evidence at this point that there is increased “ownership” of the PBF by province governments or the national government? 13. What is your opinion about sustainability of PBF activities with IHP funding? Probe: how about without IHP donor funding? Evaluation objective 6. Gather preliminary information and describe unintended consequences related to the implementation of the PBF intervention. [Analytical domain: unintended consequences] MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 68 14. What do you think about the process of technical verification for health facility (HC or GRH) and ECZs levels? Probe: talk to me about the transparency of the PBF intervention? Probe: are measures in place to ensure transparency? If yes, describe. Probe: talk to me about any conflict of interest you are aware of? Probe: are measures in place to control for conflict of interest? If yes, describe. 15. What is your opinion on the reliability and validity of the data collected by CSOs, etc.? Probe: are CSOs supervised? If yes, describe: by whom, how? How often? Etc. Probe: are there any quality control measures in place to ensure the reliability and validity of the CSO data? Probe: what do you think how transparency and conflict of interest are managed? 16. Do you think health facility staff are focusing on the key family health services that are under PBF scheme at the expense of other services? (distortion) 17. Do you think health facility staff is falsely reporting on patient or cases to increase their PBF incentive? (gaming) 18. Do you think health facility staff are focusing on providing high quality services to patients or clients availing services that are under PBF scheme at the expense of other patients or clients? (cherry-picking) 19. how high a priority of PBF activities in your day to day work (in a scale of 1-10, 1 being the lowest and 10 being the highest) :______ Probe: why? 20. How would you rate the likelihood of PBF scheme to be a successful intervention in future (in a scale of 1-10, 1 being the lowest and 10 being the highest): _____? Probe: please explain - why? Name one critical factor for its success or failure? ------------------------------------------------------------------------------------------------------------------------ Additional interview questions: MSP national MSP nat-1 has the IHP PBF pilot contributed to the national RBF approach? Describe. MSP nat-2 are there other organizations supporting your PBF program other than USAID, and IHP? What is the nature of that support? MSP nat-3 what are the similarities or dis-similarities between IHP and MSP approaches for PBF? Or how does IHP RBF approach differ from MSP approach? MSP nat-4. What, if any, is the MSP’s policy on PBF? Describe MSP nat-5. Does the PROSANI PBF approach influence/inform national policy, strategy, or guidelines on PBF? If yes, describe. MSP nat-6. What is your opinion of PBF and its effect, if any, on strengthening the health system? MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 69 MSP nat-7. Has the PROSANI PBF model had any effect on strengthening the health system? If yes, describe. Additional interview questions: MSP provincial/DPS MSP dps-1 how do you identify and select the community based organizations to perform counter-verifications? Probe: is there a systematic approach to selecting CSO (e.g., RFP, selection criteria, etc.) MSP dps-2 how do you determine which employees will benefit from the performance bonuses? Probe: do you sometimes transfer employees to an RBF health facility? If yes, what is the rationale for those transfers? Additional interview questions: IHP- national IHP NAT 1. What have been the biggest lessons learned with the RBF program? IHP NAT 2. In what specific ways are you using lessons learned from PBF to do things differently in future? IHP NAT 3. Are there differences in PBF pilot facilities with or without LDP training in regard to: a) facility operations, b) management, c) supervision, d) coverage of MCH services, and e) patient attendance of MCH services? Additional interview questions: IHP- BC IHP BC 1. What is the process of distributing the incentives? IHP BC 2. How do you ensure the appropriate staff receives their incentives? IHP BC 3. How much “buy-in”/ownership do you believe exists at the ECZ level for the concept of PBF? Is there any specific evidence of this buy-in? Probe: are formal contracts in place as planned (with HZ management teams, service providers and CSO collecting data at the community level? Probe: are data verifications done according to contracts? IHP BC 4. Do the majority of health providers routinely receive their base salaries from the MSP? Explain NB: differentiate between salaries and monthly per diem for living expenses IHP BC 5. Since PBF has started, has this changed the distribution of health provider salaries from the MSP? Explain IHP BC 6. If there has been a change in payment of MSP salaries since the PBF was started, to what do you attribute this change? Ihp-11 do you have any recommendations for PROSANI senior management in Kinshasa and/or ECZ based on your PBF experiences? --------------------------------------------------------------------------------------------------------------------- IHP MPA-plus (the MPA-plus is to be provided at all health centers in the 80 targeted health zones) MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 70 Preventative activities Growth and development monitoring for children under 5. Prenatal counseling PMTCT, including counseling, HIV testing, antiretroviral prophylaxis, FP counseling, and cotrimoxazole, nutrition counseling, and referrals for treatment, Cotrimoxazole for exposed infants FP counseling and services (condoms, orals, injectables, intrauterine devices, standard day method cycle beads, lactational amenorrhea method (lam) and referrals for long-acting and permanent methods) Postnatal counseling Immunizations: BCG (tuberculosis), OPV (polio), dpt-hepb-hib (diphtheria, pertussis, tetanus, hepatitis b, haemophilus influenza type b), VAR (measles) Universal precautions for infection prevention and blood safety Distribution of IPTP and LLINS HIV information Vitamin a, other micronutrient supplementation Curative activities Clinic-based IMCI including treatment of malaria and acute respiratory infection (ARI), diarrhea: Testing and treatment of chronic diseases, including NTDS HIV/AIDS: PMTCT and blood transfusion testing, monitoring patients on antiretroviral therapy who have been diagnosed at GHR, management of opportunistic infections (cotrimoxazole) and related nutritional support devices. TB: sputum collection and forwarding to diagnostic and treatment centers; TB treatment diagnosis and treatment (referrals as indicated) for other NTDS: leprosy, trypanosomiasis, lymphatic filariasis, hookworm, roundworm, whipworm, shistosomiasis, onchocerciasis) other curative care not elsewhere cited Nutritional rehabilitation minor surgery Normal labor and delivery services including practice of active management of third stage labor (AMTSL), availability of OxyContin, and newborn care kits. IPTP for pregnant women and children under 5 STI syndromic treatment and referrals Post-exposure prophylaxis (pep) and appropriate counseling for victims of S/GBV: facility survey Acute respiratory infection treatment Promotional activities Condom use for dual protection environmental sanitation exclusive breastfeeding MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 71 Healthy eating and food handling use of iodized salt Improved latrines Oral rehydration therapy and diarrheal disease control fistula awareness and prevention Management/administrative activities Increase availability of essential services to underserved populations (e.g., increase coverage) management of resources (human, material, financial) Continuous health personnel training Training and mentoring of (community) outreach workers (meetings, site visits) Linkages with and referrals from private health providers in the health zones (if such exist) management of health information Management of pharmaceutical information Community activities Community-based IMCI (C-IMCI) including early recognition and referral for danger signs disease surveillance: TB, NTDS, etc. Food safety and food handling Potable water improvements: spring and well capping, improved water distribution systems, community water treatment Disease control: use of LLITNS, tsetse control, environmental sanitation, etc. Community-based information-education-communication and distribution of FP commodities: standard day method cycle beads, orals, condoms, and referrals for other methods Community awareness and prevention S/GBV vegetable gardens, fish farming, livestock production MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 72 CLIENT SURVEY: HOUSEHOLD VISITS Note to RAs: from the out-patient registers, select 20 clients that visited the facility in the month of September. A client is either a woman or child of any age who attended facility for any type of service in the month of September 2014. For each selected client, note down details in the client details box of the client questionnaire. Visit each client at his/her home to interview. Select only patients who are located in the same village/town where the health center is located. Select patients from the following registries (in the order listed below): 1 st curative child visit 2 nd: curative female visit 3 rd: well child visit (e.g., immunizations) 4 th antenatal care for pregnant women 5 th family planning for women If there were more than 20 patient visits of women and children during the month of September, you will need to select every “xth” patient from the registry. To calculate “x”, divide the number 20 into the total number of women and child visits in September. For example, if there were 45 child visits and 15 women visits in September, there were a total of 60 patients. We need a total of 20 patients to be interviewed therefore, you would divide 20 into 60 (60/20 = 3) and select every “3rd” patient in the registries. A.client or guardian/mother of client (if client is a child) is identified at the home address recorded from the health facility & is present at home at the time of interview: proceed with the client questionnaire (given below) by asking questions to client or his/her representative B. client or guardian/mother of client (if client is a child) is identified at the home address recorded from the health facility, not present at home at the time of interview stop C. client or guardian/mother of client (if client is a child) could not be identified at the home address recorded from the health facility stop Client questionnaire Consent form: Hello. My name is ____________________. I represent IBTCI a company working with USAID in cooperation with the ministry of public health. We are conducting a survey on health facilities supported by USAID through MSH/IHP (PROSANI), with the goal of identifying ways to improve services. We would like to interview you about the situation at this facility, and the availability of services. Be assured that our conversation will remain strictly confidential, and you will not be identified in any way. At any time, you may choose to stop the interview or refuse to answer a question. May I proceed? Yes... no… If no, go to the end of the questionnaire. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 73 Information grid for client 01 Village name _________________________ 02 Household number in the village /____ /____/______/ 03 Health area name _________________________ 04 Health zone name 05 Supervision area name ___________________________ 06 Province name and code West Kasai......................................... 1 East Kasai.......................................... 2 Katanga.............................................. 3 07 residential area Urban ..........................................................1 Semi-urban …………………………………2 Rural .............................................................3 08 Mother’s (client’s) name Name __________________________ __________________________________________ 09 Child's date of birth ___ ___ / ___ ___ / ___ ___ ___ ___ day month year 10 Child's age (in months) /____ /____/______/ 11 Interviewer's name Name __________________________ 12 Day / month / year of interview ____ ____ / ____ ____/ ____ ____ ____ ____ 13 Mother’s (client’s) age (in years) /___/___/ Q1. Did you visit the health facility (mention the name of the health facility) last month (the month of September)? Yes ......................................... 1 date: __/__ No ......................................... 2 MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 74 NR/not sure......................................... 3 If no or not sure, skip to cs13 If the answer is ‘yes’ ask the following questions: Q1.a did you receive care at this visit? Yes ......................................... 1 what type of service did you receive? ___________________ No ......................................... 2 Q1.b did your child receive care at this visit? Yes ......................................... 1 what type of service did your child receive? _____________ What is the age of your child? ____ No ......................................... 2 Continue asking the respondent the following questions about her last visit to the health facility: Facility Visit Details # questions response cs01 Did you receive the service at a health facility? If yes, go to cs02 If no or not sure, skip to cs14 yes (name of facility) ....................................... 1 no.........................................2 nr/not sure......................................... 3 cs02 What was the reason for the facility visit? _________________________________ cs03 How would you rate your overall satisfaction with the service that you received at your last visit? very satisfied......................................... 1 somewhat satisfied......................................... 2 neutral.........................................3 somewhat dissatisfied.......................4 very dissatisfied...................................5 nr/not sure .........................................6 cs04 The last time that you sought out a health service that you wanted, did you receive it? yes no cs05 At your last health center visit, did the nurse spend as much yes no 1 2 MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 75 time as you wanted with you? nr/not sure 3 cs06 Do you think that the nurse/doctor treated you professionally and gave you proper care? yes no nr/not sure 1 2 3 cs07 Did the nurse/doctor listen attentively to you and let you ask the questions that you wanted to ask him/her? yes no nr/not sure 1 2 3 cs08 How much did you pay during this visit? (nb: differentiate between payments for services and payments for medications.) cost of service:____________ cost of medication(s):_______ other means of payment (non-cash): yes or no if yes, specify ___________ cs11 Were all the prescribed medications always at the HC? all......................................... 1 some......................................... 2 none......................................... 3 nr/not sure......................................... 4 cs12 Were the necessary supplies and equipment available at the HC? yes no if no, explain ______________ nr/not sure cs13 In the past six months, did you receive health services at another health facility (not this one)? yes......................................... 1 no......................................... 2 nr/not sure......................................... 3 MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 76 cs14 If yes, what was the reason for visiting a different health center? cs15 In the past six months, was there a time when you needed health services but did not visit a health center or hospital? yes………………..1 no…………………2 If yes, what was the reason you did not seek health services at a health facility? too far......................................... 1 not enough money to pay the bill......................................... 2 staff not qualified......................................... 3 nurse not welcoming......................................... 4 i prefer traditional medicine......................................... 5 difficulty in getting transport........................................ 6 other (specify)______________ .............................7 nr/not sure......................................... 8 HEALTH EDUCATION SERVICES he01 What health practices, if any, have you learned through contact with these health professionals? Ask again: Any other practices? Record everything that is mentioned.  exclusive breastfeeding  good nutrition  vaccinations  prevention and treatment of diarrhea  prevention and treatment of acute respiratory infections  prevention and treatment of malaria  education on use of family planning methods  prevention and treatment of HIV/AIDS  other (specify): __________________ a b c d e f g h x he02 From whom do you usually obtain general information or advice about health or nutrition? ask again: Formal network  physician  nurse/midwife  auxiliary midwife  community health worker a b c d MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 77 Any other information or advice? Record everything that is mentioned.  growth monitor  trained birth assistant Informal network  spouse/partner  mother/adoptive mother  sister  grandparent  aunt  friend/neighbor  traditional healer  village elders  other (specify)_______________ e f g h i j k l m n x he03 In the last month, did you receive Any health messages through the following channels?  community health workers?  doctor or nurse?  family member?  radio?  magazine/newspaper?  television?  school?  text message?  other: (specify) yes 1 1 1 1 1 1 1 1 no 2 2 2 2 2 2 2 2 PERCEPTION OF QUALITY OF CARE Patient perceptions of quality of care qc Have you used local health services (health center or general hospital) in the last 3 months? yes…………………1 no……………………….2 If yes, ask the respondent the degree to which they agree or disagree with the following statements: If no, end the interview. MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 78 Impression of care Strongly disagree Disagree Neither agree nor disagree Agree Stron gly agree (1) (2) (3) (4) (5) qc1 Health workers' behavior and practices qc1.1 Show compassion and support for patients qc1.2 Show respect for patients qc1.3 Are friendly/welcoming to patients qc1.4 Are honest qc1.5 Attentively listen to patients qc1.6 Nurses take enough time for patients. qc2 Appropriateness of resources and services qc2.2 The rooms are appropriate qc2.3 The waiting time is reasonable qc2.4 There are enough nurses qc2.5 Medications are available at all times qc3 Finances and cost of care qc3.1 Prices can be discussed qc3.2 Prices are reasonable qc3.3 Have you seen treatment prices posted? qc3.4 Do you think that you paid the actual price that you should have MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 79 paid? qc3.5 Medications can easily be obtained. qc3.6 The distance from the center is reasonable for us (not too far). Thank you very much for participating in our survey. Your input is valuable to us. Time interview ended: ____________ MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 80 HEALTH FACILITIES MINI SURVEY IHP PBF mid-term assessment N° Question: 01. Name of health structure _________________ GPS location: 02. 03. Name of supervision area (coordination bureau) __________________________ Name of health zone: 04. ___________________________ 05. Type of facility:  health center  general referring hospital (HGR) 1 2 Province name and code 06.  West Kasai  East Kasai  Katanga 1 2 3 07. Respondent's professional category:  physician  state registered nurse  state registered midwife health technician  technical health agent  physician in training/volunteer physician  other trainee/volunteer  other (specify): _________________ 1 2 3 4 5 6 7 8 9 Date and t 08. ime of survey  date: /____ /____/______/  time: /_____o'clock :_______/ minutes Guide to the survey on availability of services and equipment Locate the head nurse and the health center director/ head physician of the general hospital, and introduce yourself as follows: Hello. My name is ____________________. I represent IBTCI a company working with USAID in MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 81 cooperation with the ministry of public health. We are conducting a survey on health facilities supported by USAID through MSH/IHP (PROSANI), with the goal of identifying ways to improve services. We would like to interview you about the situation at this facility, and the availability of services. Be assured that our conversation will remain strictly confidential, and you will not be identified in any way. At any time, you may choose to stop the interview or refuse to answer a question. May I proceed? yes... no… If no, go to the end of the questionnaire. A. COSTS OF SERVICES AND FACILITY OPERATIONS N° QUESTIONS CODE 09. Simply observe and note Do you see a sign or poster showing the availability of the following services (circle all appropriate responses)? a) family planning services. b) infant health services c) prenatal care d) child birth e) prices for any other services yes yes no outside inside 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 010. if yes, what is the posted price for:  initial visit forms  malaria case  diarrhea case  prenatal consultation /______________/ fc /______________/ fc /______________/ fc /______________/ fc MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 82 N° QUESTIONS CODE  childbirth  family planning  other (specify): _________________ _________  other (specify): _________________ _________  other (specify): _________________ _________  other (specify): _________________ _________ /______________/ fc /______________/ fc /______________/ fc /______________/ fc /______________/ fc /______________/ fc 011. Ask interviewee: how many days is this facility open to outpatients? (outpatients are those who are receiving preventive or curative care and going home the same day) Number of days per week Number of days per month Don't know /______/ days /______/ days B. FACILITY SERVICE STATISTICS 012. Ask interviewee: when did the PBF intervention start in this health facility? Month: ____________ Year: ____________ 013. Ask interviewee: did you receive any training on PBF intervention? MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 83 Yes 1 No 2 No response 3 Don’t know 4 If yes, when did you receive the training? ________ (month/year) How long did the training last? ______________ (hours/days) 014. Ask interviewee: please tell us about the durations for quarter 1 and quarter 2. Or in other words, when did quarter 1 and quarter 2 start under PBF intervention and when ended or currently ongoing? A. PBF quarter 1 (q1): from: ______(month), to: _______(year) Did you receive any payment for quarter 1? Yes 1 No 2 No response 3 Don’t know 4 If yes, how much payment received? ______ B. PBF quarter 2 (q2): from: ______(month), to: _______(year) Did you receive any payment for quarter 2? Yes 1 No 2 No response 3 Don’t know 4 If yes, how much payment received? ______ 015. Ask interviewee for facility registers to answer below questions. Review facility registers and note following facility statistics for q1 and q2 (after PBF started) N° Indicator Q1 Q2 Source document/register Comments Number of A. outpatients visit at the health center Number of B. pregnant women registered to receive ANC MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 84 Number of C. pregnant women received TT2 Number of D. pregnant women tested for HIV Number of E. pregnant women tested for HIV and received results Number of child F. birth attended by trained personnel Number of G. new family planning consultations Number for H. children who received DTP3 Number of I. consultations for sick children Number of LLINS J. (bed nets) distributed Number of clients K. who received voluntary HIV testing and counselling 015. Status of survey  complete  partially complete  refused  authorized respondent not found  facility not found 1 2 3 4 5 MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 85  other (specify): ___________________ 6 Interviewer's comments: Supervisor's comments: Time completed: /____ /____/______/ --------------End of questionnaire------------- Following are the PBF health center indicators which are tabulated quarterly by the IHP and health zone management team: PBF HEALTH CENTER INDICATORS: Rate (number) of use of curative services at the health center Proportion (number) of high-risk pregnancies referred Rate (number) of coverage with DTP-HEPBHIB3 (pentavalent) Proportion (number) of pregnant women who received TT2+ Number of clients who received family planning counseling Rate of childbirth attended by healthcare personnel Rate (number) of tuberculosis detection MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 86 Proportion (number) of LLITNS distributed Rate (number) of use for prenatal consultation 1 services Number of clients who received voluntary HIV counseling and testing Number of pregnant women tested for HIV Rate (number) of CPN 4 coverage (recentered) Rate (number) of CPON 2+ use % (number) of monthly medication management and inventory reports analyzed and sent on time to the central office for the zone Health center's overall FOSACOF score Overall satisfaction score for health center patients MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 87 FRENCH VERSION GUIDE DES GROUPES DE DISCUSSION : CODESA OU À DÉTERMINER QUESTIONS DE DISCUSSION : CODESA OU À DÉTERMINER Objectif d'évaluation No. 1. Évaluer l'effet initial de l'intervention du FBP sur la quantité de services, le cas échéant. [Domaine analytique : pertinence du FBP] 3. Que pouvez-vous nous dire sur les services clés suivants de santé familiale dans votre centre de soins? (Sondage pour toutes les questions ci-dessous afin d'établir l'existence de différences observées aujourd'hui par rapport à un an auparavant)? A. Quels services sont disponibles au centre de soins pour les femmes enceintes? A. Qui fournit des services prénatals? En quoi consiste une visite prénatale typique? (Remarque: nous cherchons à voir le type de soins cliniques reçus et si des médicaments sont donnés) B. Quels services sont disponibles au centre de soins pour l'accouchement? A. Qui s'occupe de la patiente pendant l'accouchement? (Remarque: Si le (à déterminer) assiste à la naissance, l'infirmière du centre de soins est-elle présente? Où a lieu l'accouchement, par ex. : Au domicile de la femme enceinte? Au centre de soins)? C. Quels services sont disponibles au centre de soins pour les vaccinations infantiles? D. Quels services sont disponibles au centre de soins pour le planning familial? E. Quels services sont disponibles pour les maladies diarrhéiques de l'enfance? A. au cours de l'année écoulée, y a-t-il eu des changements au niveau de la manière dont les enfants atteints de diarrhée sont traités ? F. Quels services sont disponibles pour la fièvre ou le paludisme ? A. Au cours de l'année écoulée, y A-T-Il eu des changements au niveau de la manière dont les patients atteints de paludisme sont traités? Nom de l'intervieweur/du facilitateur : ____________________ Date : ____________ Province : _____________BC : ____________Zone sanitaire : ____________Village : ___________ Nom de l'établissement/organisme : ___________________ Type d'établissement : GRH __ HC__ Coordonnées GPS : Longitude : ______________Latitude : ______________ Type de répondant : CODESA  À déterminer  OSC  Coordonnées (No. de téléphone) du répondant (un représentant du groupe de discussion) : MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 88 B. Recevez-vous des moustiquaires de lits contre le paludisme? Qui vous les fournit? (Sondage: Notez tous les services mentionnés par les répondants, sans les solliciter au sujet de chaque type de service de soins ci-dessus. Reportez-vous à la liste MPA (fournie au dos de ce guide des groupes de discussion) et sondez des activités spécifiques figurant dans le MPA sous les soins prénatals, les soins postnatals et la santé infantile, le paludisme, ETC. Objectif d'évaluation no. 2. Évaluer l'effet initial de l'intervention du FBP sur la perception de la qualité des services, le cas échéant. [Domaine analytique : pertinence du FBP] 4. Avez-vous remarqué des changements au niveau du coût des services fournis dans cet établissement par rapport à un an auparavant? Si c'est le cas, décrivez. D. Quel est le coût d'un accouchement à l'heure actuelle? _____________. Quel était le coût il y a un an_____________? E. Quel est le coût des vaccinations à l'heure actuelle? ___________. Quel était le coût il y a un an_____________? F. Quel est le coût d'une visite curative à l'heure actuelle? ___________. Quel était le coût il y a un an_____________? 6 Où les patients obtiennent-ils habituellement leurs médicaments? D. Les médicaments nécessaires sont-ils disponibles aux centres de soins? E. En cas de plusieurs sources d'obtention de médicaments, veuillez décrire. F. La disponibilité des médicaments A-T-ELLE changé au cours de l'année écoulée? Si c'est le cas, donnez des exemples pour illustrer le coût d'un médicament il y a un an par rapport à son coût aujourd'hui pour le même médicament? N.B.: Sondage pour explorer si les patients préfèreraient obtenir leurs médicaments d'une source plutôt que d'une autre. Le but consiste à mieux comprendre tout problème de distorsion des prix des médicaments au niveau local. 7 Les patients doivent-ils payer pour leurs médicaments? Si c'est le cas, quels sont des exemples de coûts? N.B.: Sondage à explorer: Pensez-vous que les prix sont raisonnables? Connaissez-vous un autre endroit où vous pouvez vous procurer les médicaments à un prix plus bas que celui du centre de soins? EST-CE que les prix sont similaires dans d'autres centres de soins? 8 Que pouvez-vous nous dire sur le fonctionnement du centre de soins? (Sondage pour toutes les questions ci-dessous: Avez-vous observé des différences entre maintenant et un an auparavant)? A. Quel est le délai d'attente habituel pour voir le prestataire de soins? B. Que pensez-vous de l'infrastructure du centre de soins? C. Que pensez-vous des compétences professionnelles du prestataire de soins de santé? MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 89 D. Que pensez-vous de la manière dont le prestataire de soins interagit avec les patients? (Sondage concernant ses compétences en communications interpersonnelles) Pour le groupe de discussion à déterminer: Questions complémentaires 3. Recevez-vous une formation de la part du personnel du centre de soins en: Soins prénatals? Accouchement? Soins postnatals? 4. Recevez-vous des trousses d'accouchement du centre de soins? Si c'est le cas, quel en est le contenu? GUIDE DES GROUPES DE DISCUSSION : ORGANISATIONS DE LA SOCIÉTÉ CIVILE (OSC) QUESTIONS DE DISCUSSION: OSC Objectif d'évaluation no. 1. Évaluer l'effet initial de l'intervention du FBP sur la quantité de services, le cas échéant. [Domaine analytique : pertinence du FBP] 11. Que pensez-vous de la disponibilité des services clés suivants de santé familiale dans des centres de soins qui pratiquent le financement basé les performances (FBP)? H. Services pour femmes enceintes I. Accouchement J. Vaccination K. Planning familial L. Fièvre ou paludisme M. Diarrhée infantile N. Toux durant plus de 2 semaines chez un enfant ou pneumonie infantile Nom de l'intervieweur/du facilitateur : ____________________ Date : ____________ Province : _____________BC : ____________Zone sanitaire : ____________Village : ___________ Nom de l'établissement/organisme : ___________________ Type d'établissement : GRH __ HC__ Coordonnées GPS : Longitude : ______________Latitude : ______________ Type de répondant : CODESA  À déterminer  OSC  Coordonnées (No. de téléphone) du répondant (un représentant du groupe de discussion) : MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 90 (Sondage pour tout ce qui précède : Avez-vous observé des différences entre maintenant et un an auparavant avant le début du FBP?) (Sondage : Notez tous les services mentionnés par les répondants, sans les solliciter au sujet de chaque type de service de soins ci-dessus. Reportez-vous à la liste MPA (fournie au dos de ce guide des groupes de discussion) Et sondez des activités spécifiques figurant dans le MPA sous les soins prénatals, les soins postnatals et la santé infantile, le paludisme, etc. Objectif d'évaluation no. 2. Évaluer l'effet initial de l'intervention du FBP sur la perception de la qualité des services, le cas échéant. [Domaine analytique : pertinence du FBP] 12. Que pensez-vous de la qualité des services clés de santé familiale dans les centres de soins depuis que le FBP a été mis en application? J. Délai d'attente K. Coût des services L. Infrastructure M. Disponibilité des médicaments N. Compétences professionnelles des prestataires de soins O. Compétences d'intercommunication des prestataires de soins (Sondage pour tout ce qui précède: Avez-vous observé des différences entre maintenant et un an auparavant avant le début du FBP?) P. Pensez-vous que le personnel du centre de soin se concentre sur les services clés de santé familiale faisant partie du programme FBP aux dépens des autres services? (DISTORSION) Q. Pensez-vous que le personnel du centre de soin fait de faux rapports sur les patients ou les cas afin d'augmenter leur prime FBP? (JEUX) R. Pensez-vous que le personnel du centre de soin se concentre sur la fourniture de services de haute qualité aux patients ou aux clients utilisant des services faisant partie du programme FBP aux dépens d'autres patients ou clients? (PICORAGE) Objectif d'évaluation no. 3. Évaluer les réalisations à ce jour des objectifs FBP et décrire tout goulot d'étranglement en entravant le progrès. [Domaine analytique: progrès] 13. Quel est le rôle de l'OSC dans la mise en application du FBP? 14. Quelles sont vos réalisations à ce jour en matière de FBP? Sondage: Y A-T-IL des goulots d’étranglement entravant les progrès d'une mise en application du FBP? 15. Quel est le processus en matière d'audits communautaires (aussi connus sous le nom de processus de vérification)? Objectif d'évaluation no. 4. Déterminer tous les changements entre la disponibilité de fonds prévus par rapport aux fonds réels aux niveaux opérationnels (Par ex.: centre et équipe de gestion HZ). [Domaine analytique : couts] 16. Quelle est la source de financement des vérifications? Quel est le coût? MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 91 Objectif d'évaluation no. 5. Identifier et analyser des facteurs contextuels susceptibles d'influencer les résultats de l'intervention FBP. [Domaines analytiques : capacité de prise en charge et de gestion] 17. Quels sont vos plus grands défis et contraintes pour la mise en œuvre de vérifications communautaires en vertu du déploiement du FBP ? Objectif d'évaluation no. 6. Rassembler des informations préliminaires et décrire les conséquences imprévues liées à la mise en œuvre de l'intervention FBP. [Domaine analytique : conséquences imprévues] 18. Savez-vous s'il y a des contre-vérifications de la part de l'OSC? (Quelqu'un réalise-t-il des vérifications de contrôle de la qualité pour confirmer que vous fournissez des informations exactes?) 19. Qui est au courant de la manière dont votre OSC réalise ses contrôles de vérification? A. Comment communiquez-vous vos résultats à PROSANI et à l'équipe de gestion de la zone sanitaire? 20. Quelles mesures, le cas échéant, votre OSC organise-t-elle pour éviter tout conflit d'intérêts au cours du processus de vérification? GUIDE D'INTERVIEW D'INFORMATEUR CLÉ : RESPONSABLE ECZ Objectif d'évaluation no. 1. Évaluer l'effet initial de l'intervention du FBP sur la quantité de services, le cas échéant. [Domaine analytique : pertinence du FBP] 1. Comment caractériseriez-vous les effets, le cas échéant, d'une intervention FBP sur la quantité de services clés de santé familiale? A. Services pour femmes enceintes B. Accouchement C. Vaccination Nom de l'intervieweur/du facilitateur : ____________________ Date : ____________ Province : _____________BC : ____________Zone sanitaire : ____________Village : ___________ Nom de l'établissement/organisme : ___________________ Type d'établissement : GRH __ HC__ Coordonnées GPS : Longitude : ______________Latitude : ______________ Nom du répondant : ________________ Coordonnées du répondant : _________ Téléphone : _________ Email : _____________ Type de répondant : Responsable ECZ  MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 92 D. Planning familial E. Paludisme F. Diarrhée infantile G. Pneumonie infantile (Sondage pour tout ce qui précède : avez-vous observé des différences entre maintenant et un an auparavant avant le début du FBP ?) (Sondage : reportez-vous à la liste MPA (fournie au dos de ce guide) et sondez des activités spécifiques figurant dans le MPA sous les soins prénatals, les soins postnatals et la santé infantile, le paludisme, etc.) Objectif d'évaluation no. 2. évaluer l'effet initial de l'intervention du FBP sur la perception de la qualité des services, le cas échéant. [Domaine analytique : pertinence du FBP] 2. comment caractériseriez-vous les effets, le cas échéant, d'une intervention FBP sur la qualité de services clés de santé familiale ? G. Coût des services H. Délai d'attente I. Infrastructure J. Disponibilité des médicaments K. Compétences professionnelles des prestataires de soins L. Compétences d'intercommunication des prestataires de soins (Sondage pour tout ce qui précède : avez-vous observé des différences entre maintenant et un an auparavant avant le début du FBP?) Objectif d'évaluation no. 3. Évaluer les réalisations à ce jour des objectifs FBP et décrire tout goulot d'étranglement en entravant le progrès. [Domaine analytique : progrès] 3. Quel est le rôle de votre organisation (ECZ/HZMT) dans l'intervention FBP de l'IHP? 4. Quel soutien recevez-vous de l'IHP pour la mise en œuvre des activités FBP? Sondage: Avez-vous reçu une formation FBP ? Par qui, quand, des plans de formation d'appoint? De l'aide avec des plans de travail? Sondage: Ont-ils exécuté leur rôle de manière à faciliter la mise en œuvre? 5. Quel est votre avis sur les réalisations des objectifs FBP jusqu'à présent? Sondage: Où les progrès ont-ils été les plus marquants et pourquoi? Sondage: Où ont-ils été les plus difficiles ou les plus faibles? 6. Y A-T-IL des goulots d’étranglement entravant la mise en œuvre du FBP? Sondage: Qu'est-ce qui aurait pu avoir été fait différemment? Sondage: Si ces mesures étaient en place, réussiraient-elles à atténuer le goulot d'étranglement (juste décrit) MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 93 Objectif d'évaluation no. 4. Déterminer tous les changements entre la disponibilité de fonds prévus par rapport aux fonds réels aux niveaux opérationnels (par ex. : Centre et équipe de gestion HZ). [Domaine analytique : coûts] 7. Parlez-moi de la disponibilité de fonds, au niveau de l'équipe de gestion ECZ/HZ, pour mettre en œuvre les activités FBP? Sondage: Y A-T-IL des changements au niveau de la disponibilité entre ce qui était prévu et la réalité? 8. Parlez-moi de la disponibilité de fonds, au niveau du centre, pour mettre en œuvre les activités FBP? Sondage: Y A-T-IL des changements au niveau de la disponibilité entre ce qui était prévu et la réalité? 9. L'intervention FBP A-T-ELLE eu un effet sur les coûts d'utilisation? Sondage: Y A-T-Il des informations préliminaires indiquant des changements au niveau des coûts d'utilisation? Objectif d'évaluation no. 5. Identifier et analyser des facteurs contextuels susceptibles d'influencer les résultats de l'intervention FBP. [Domaines analytiques : capacité de prise en charge et de gestion] 10. Que pensez-vous de la charge de travail d'ECZ/HZMT avant et après le début du FBP? 11. Quel est votre avis sur la durabilité des activités FBP avec le financement IHP? Sondage: Qu'en serait-il sans le financement des donateurs de l'IHP ? Objectif d'évaluation no. 6. Rassembler des informations préliminaires et décrire les conséquences imprévues liées à la mise en œuvre de l'intervention FBP. [Domaine analytique : conséquences imprévues] 12. Que pensez-vous du processus de vérification technique des centres de soins (HC ou GRH) et des niveaux ECZ? Sondage: Parlez-moi de la transparence de l'intervention FBP? Sondage: Des mesures sont-elles en place pour assurer la transparence? Si oui, décrivez. Sondage: Parlez-moi de tout conflit d'intérêts dont vous auriez connaissance? Sondage: Des mesures sont-elles en place pour contrôler l'absence de conflits d'intérêts? Si oui, décrivez. 13. Quel est votre avis au sujet de la fiabilité et de la validité des données recueillies par les OSC, etc.? Sondage: Les OSC sont-elles supervisées? Si oui, décrivez : par qui, comment? À quelle fréquence? Etc. Sondage: Y A-T-IL des mesures de contrôle de la qualité en place pour assurer la fiabilité et la validité des données des OSC? Sondage: Que pensez-vous de la manière dont la transparence et les conflits d'intérêts sont gérés? 14. Pensez-vous que le personnel du centre de soin se concentre sur les services clés de santé familiale faisant partie du programme FBP aux dépens des autres services ? (distorsion) 15. Pensez-vous que le personnel du centre de soin fait de faux rapports sur les patients ou les cas afin d'augmenter leur prime FBP? (jeux) MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 94 16. Pensez-vous que le personnel du centre de soin se concentre sur la fourniture de services de haute qualité aux patients ou aux clients utilisant des services faisant partie du programme FBP aux dépens d'autres patients ou clients? (picorage) 17. Quel est le degré de priorité des activités FBP dans votre travail quotidien (sur une échelle de 1 à 10, 1 étant le plus faible et 10 le plus élevé) :______ Sondage : Pourquoi ? 18. Quelle chance de succès donneriez-vous au programme FBP à l'avenir (sur une échelle de 1 à 10, 1 étant la plus faible et 10 la plus élevée) : _____ Sondage: Veuillez expliquer - pourquoi ? Citez un facteur crucial à son succès ou à son échec? GUIDE D'INTERVIEW D'INFORMATEUR CLÉ : DIRECTEUR HF (GBH) OU INFIRMIER EN CHEF (HC) Objectif d'évaluation no. 1. évaluer l'effet initial de l'intervention du FBP sur la quantité de services, le cas échéant. [Domaine analytique : pertinence du FBP] 1. Comment caractériseriez-vous les effets, le cas échéant, d'une intervention FBP sur la quantité de services clés de santé familiale? A. Services pour femmes enceintes B. Accouchement C. Vaccination D. Planning familial E. Paludisme F. Diarrhée infantile G. Pneumonie infantile (Sondage pour tout ce qui précède: Avez-vous observé des différences entre maintenant et un an auparavant avant le début du FBP?) Nom de l'intervieweur/du facilitateur : ____________________ Date : ____________ Province : _____________BC : ____________Zone sanitaire : ____________Village : ___________ Nom de l'établissement/organisme : ___________________ Type d'établissement : GRH __ HC__ Coordonnées GPS : Longitude : ______________Latitude : ______________ Nom du répondant : ________________ Coordonnées du répondant : _________ Téléphone : _________ Email : _____________ Type de répondant : Directeur GRH  Directeur/Infirmier en chef HC  MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 95 (Sondage: Reportez-vous à la liste MPA (fournie au dos de ce guide) et sondez des activités spécifiques figurant dans le MPA sous les soins prénatals, les soins postnatals et la santé infantile, le paludisme, etc.) Objectif d'évaluation no. 2. Évaluer l'effet initial de l'intervention du FBP sur la perception de la qualité des services, le cas échéant. [Domaine analytique : pertinence du FBP] 2. Comment caractériseriez-vous les effets, le cas échéant, d'une intervention FBP sur la qualité de services clés de santé familiale? A. Coût des services B. Délai d'attente C. Infrastructure D. Disponibilité des médicaments E. Compétences professionnelles des prestataires de soins F. Compétences d'intercommunication des prestataires de soins (Sondage pour tout ce qui précède: Avez-vous observé des différences entre maintenant et un an auparavant avant le début du FBP?) Objectif d'évaluation no. 3. évaluer les réalisations à ce jour des objectifs FBP et décrire tout goulot d'étranglement en entravant le progrès. [Domaine analytique : progrès] 3. Quel est votre avis sur le rôle de l'IHP concernant les activités FBP? Sondage: Ont-ils exécuté leur rôle de manière à faciliter la mise en œuvre? 4. Avez-vous reçu une formation sur le FBP avant le début de l'intervention ? Décrivez. Sondage: combien de jours, qu'avez-vous appris, avez-vous reçu des documents, vous sentez-vous préparé 5. Quel est votre avis sur les réalisations des objectifs FBP jusqu'à présent? Sondage: Où les progrès ont-ils été les plus marquants et pourquoi? Sondage: Où ont-ils été les plus difficiles ou les plus faibles? 6. Y A-T-IL des goulots d’étranglement entravant la mise en œuvre du FBP? Sondage: Qu'est-ce qui aurait pu avoir été fait différemment? Sondage: Si ces mesures étaient en place, réussiraient-elles à atténuer le goulot d'étranglement (juste décrit) Objectif d'évaluation no. 4. Déterminer tous les changements entre la disponibilité de fonds prévus par rapport aux fonds réels aux niveaux opérationnels (par ex. : centre et équipe de gestion HZ). [Domaine analytique : coûts] 7. Parlez-moi de la disponibilité de fonds, au niveau du centre, pour mettre en œuvre les activités FBP? Sondage: Y A-T-IL des changements au niveau de la disponibilité entre ce qui était prévu et la réalité? 8. l'intervention FBP a-t-elle eu un effet sur les coûts d'utilisation? MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 96 Sondage: Y A-T-IL des informations préliminaires indiquant des changements au niveau des coûts d'utilisation? 9. Avez-vous des recommandations pour le personnel de PROSANI et/ou ECZ en fonction de votre expérience avec le FBP? Objectif d'évaluation no. 5. Identifier et analyser des facteurs contextuels susceptibles d'influencer les résultats de l'intervention FBP. [domaines analytiques : capacité de prise en charge et de gestion] 10. Quel est votre avis sur le rôle de l'équipe ECZ dans le FBP? 11. Quel est votre avis sur le rôle de l'OSC dans le FBP? Objectif d'évaluation no. 6. Rassembler des informations préliminaires et décrire les conséquences imprévues liées à la mise en œuvre de l'intervention FBP. [domaine analytique : conséquences imprévues] 12. Que pensez-vous du processus de vérification technique des centres de soins (HC ou GRH)? Sondage: Parlez-moi de la transparence de l'intervention FBP? Sondage: Des mesures sont-elles en place pour assurer la transparence? Si oui, décrivez. Sondage: Parlez-moi de tout conflit d'intérêts dont vous auriez connaissance? Sondage: Des mesures sont-elles en place pour contrôler l'absence de conflits d'intérêts? Si oui, décrivez. 13. Quel est votre avis au sujet de la fiabilité et de la validité des données recueillies par les OSC, etc.? Sondage: Les OSC sont-elles supervisées? Si oui, décrivez: Par qui, comment? À quelle fréquence? Etc. Sondage: Y A-T-IL des mesures de contrôle de la qualité en place pour assurer la fiabilité et la validité des données des OSC? Sondage: Que pensez-vous de la manière dont la transparence et les conflits d'intérêts sont gérés? 14. Depuis combien de temps travaillez-vous dans cet établissement? __________________ Sondage: Quand avez-vous commencé à travailler ici: Avant ou après le début du FBP? 15. Comment avez-vous été sélectionné ou nommé pour travailler dans cet établissement? Sondage: Quel était le processus de sélection? Pensez-vous que vous avez été spécifiquement placé dans cet établissement parce qu'il s'agit d'un centre FBP? 16. Votre salaire de base provient-il systématiquement du MSP? Expliquez. (N.B.: Faites la distinction entre les salaires et les indemnités journalières pour frais de subsistance) Sondage: Depuis le début du FBP, cela A-T-IL changé le versement de votre salaire par le MSP? Expliquez. Sondage: S'il y a eu un changement au niveau du versement de votre salaire par le MSP depuis le début du FBP, à quoi attribuez-vous ce changement? 17. Votre établissement A-T-IL reçu des primes FBP : au cours du dernier 1er trimestre? ______ (o/n/nsp) MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 97 au cours du dernier 2e trimestre? ______ (o/n/nsp) Si oui, sondage : Des primes FBP sont-elles été distribuées parmi le personnel de soins? Comment? Des primes sont-elles été utilisées pour améliorer l'infrastructure du centre ou acheter du matériel (y compris des médicaments ou des fournitures) pour le centre de soins? 18. Pensez-vous que le personnel du centre de soin se concentre sur les services clés de santé familiale faisant partie du programme FBP aux dépens des autres services? (distorsion) 19. Pensez-vous que le personnel du centre de soin fait de faux rapports sur les patients ou les cas afin d'augmenter leur prime FBP? (jeux) 20. Pensez-vous que le personnel du centre de soin se concentre sur la fourniture de services de haute qualité aux patients ou aux clients utilisant des services faisant partie du programme FBP aux dépens d'autres patients ou clients? (picorage) 21. Quel est le degré de priorité des activités FBP dans votre travail quotidien (sur une échelle de 1 à 10, 1 étant le plus faible et 10 le plus élevé) :______ Sondage: Pourquoi ? 22. Quelle chance de succès donneriez-vous au programme FBP à l'avenir (sur une échelle de 1 à 10, 1 étant la plus faible et 10 la plus élevée): _____ Sondage: Veuillez expliquer - pourquoi? Citez un facteur crucial à son succès ou à son échec? MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 98 Enquête auprès des clients: Visites des ménages Note aux ar: du registre des patients en consultation externe, sélectionnez 20 clients ayant visité le centre de santé pendant le mois de septembre un client est une femme ou un enfant de n'importe quel âge s'étant rendu dans un centre de santé pour n'importe quel type de service pendant le mois de septembre 2014. pour chaque client sélectionné, notez les détails dans la case détails du client du questionnaire client. rendez visite à chaque client à son domicile pour l'interview. Sélectionnez uniquement des patients situés dans le même village/ville où se trouve le centre de santé. Sélectionnez des patients des registres suivants (dans l'ordre indiqué ci-dessous): 1) Visite pour soins curatifs de l’enfant 2) Visite pour soins curatifs de la femme 3) Visite pour soins préventifs de l'enfant (par ex. : vaccination) 4) Soins prénatals pour femmes enceintes 5) Planning familial pour femmes S’il y avait plus de 20 visites de patients femmes et enfants pendant le mois de septembre, vous devrez sélectionner chaque « xe » patient du registre. Pour calculer « x », divisez le nombre total de visites de femmes et enfants en septembre par le nombre 20. Par exemple, s'il y a eu 45 visites d'enfants et 15 visites de femmes en septembre, il y a eu un total de 60 patients. Comme il nous faut un total de 20 patients à interviewer, il vous suffit de diviser 60 par 20 (60/20 = 3) et de sélectionner chaque « 3e » patient des registres. A.Le client ou tuteur/mère du client (si le client est un enfant) est identifié à l'adresse du domicile enregistré au centre de santé et se trouve à la maison au moment de l'interview : procédez avec le questionnaire client (donné ci-dessous) en posant des questions au client ou à son représentant. B. Le client ou tuteur/mère du client (si le client est un enfant) est identifié à l'adresse du domicile enregistré au centre de santé, mais ne se trouve pas à la maison au moment de l'interview. arrêtez C. Le client ou tuteur/mère du client (si le client est un enfant) n'a pas pu être identifié à L'ADRESSE DU DOMICILE ENREGISTRÉ AU CENTRE DE SANTÉ. ARRÊTEZ Questionnaire client Formulaire de consentement : Bonjour. Je m'appelle ____________________. Je représente IBTCI, une société travaillant avec USAID en coopération avec le ministère de la santé publique. Nous effectuons une enquête sur les établissements de soins de santé avec l'appui d'USAID et par l'intermédiaire de MSH/IHP (PROSANI), avec comme but l'identification de manières d'améliorer les services. Nous aimerions vous interviewer au sujet de la situation dans cet établissement, ainsi que la disponibilité des services. Soyez assuré que notre conversation restera strictement confidentielle et que vous ne serez identifié d'aucune manière. À MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 99 tout moment, vous pouvez choisir d'arrêter l'interview ou refuser de répondre à une question. M’autorisez-vous à commencer ? oui... non… Dans la négative, passez directement à la fin du questionnaire Grille d'informations du client 01 Nom du village ________________________ 02 Numéro de ménage dans le village /____ /____/______/ 03 Nom de l'aire de santé _____________________ 04 Nom de la zone de santé 05 Nom de l'aire de supervision __________________________ 06 Nom et code de la province Kasaï Occidental................................. 1 Kasaï Oriental..................................... 2 Katanga................................................ 3 07 Milieu de résidence Urbain...........................................................1 Semi-urbain..................................................2 Rural .............................................................3 08 Nom de la mère (du client) Nom _________________________ _____________________________________ 09 Date de naissance de l'enfant ___ ___ / ___ ___ / ___ ___ ___ ___ jour mois année 10 Âge de l'enfant (en mois) /____ /____/______/ 11 Nom de l'intervieweur nom ____________________________ 12 Jour / mois / année de l'interview ____ ____ / ____ ____/ ____ ____ ____ ____ 13 Âge de la mère (du client) (en années) MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 100 /___/___/ q 1. Avez-vous visité le centre de santé (mentionnez le nom du centre de santé) le mois dernier (le mois de septembre)? Oui ........................................ 1 date : __/__ Non ........................................ 2 Nr/pas sûre ............................ 3 Si non ou pas sûre, passez à cs13 Si la réponse est « oui », posez les questions suivantes : q1.a Avez-vous reçu des soins lors de cette visite ? Oui ........................................ 1 quel type de service avez-vous reçu? ___________________ Non ........................................ 2 q1.b votre enfant A-T-IL reçu des soins lors de cette visite? Oui ......................................... 1 quel type de service votre enfant A-T-IL reçu? _____________ Quel âge a votre enfant? ____ Non ......................................... 2 Continuez à poser les questions suivantes à votre répondante au sujet de sa dernière visite au centre de santé : DÉTAILS DE LA VISITE AU CENTRE No. Questions Réponse cs01 Avez-vous reçu le service dans un centre de santé? Si oui, passez à cs02 Si non ou pas sûre, passez à cs14 oui (nom du centre) ......................................... 1 non.....................................................................2 nr/pas sûre....................................................... 3 cs02 Quelle était la raison de la visite au centre? _________________________________ cs03 Comment noteriez-vous votre satisfaction globale avec le service que vous avez reçu lors de votre dernière visite? Très satisfaite................................... 1 Quelque peu satisfaite...................... 2 Neutre................................................3 Quelque peu déçue...........................4 MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 101 Très déçue.........................................5 NR/pas sûre .....................................6 cs04 La dernière fois que vous aviez besoin d'un service de santé, l'avez-vous reçu? oui non cs05 Lors de votre dernière visite au centre de santé, l'infirmier A-T￾IL passé autant de temps que vous vouliez avec vous? oui non nr/pas sûre 1 2 3 cs06 Pensez-vous que l'infirmier/docteur vous a traitée professionnellement et donné les soins nécessaires? oui non nr/pas sûre 1 2 3 cs07 L'infirmier/le docteur vous A￾T-IL écoutée attentivement et laissée poser toutes les questions que vous vouliez? oui non nr/pas sûre 1 2 3 cs08 Combien avez-vous payé durant la visite? (N.B.: Faites la distinction entre les paiements pour les services et ceux pour les médicaments.) Coût du service : ____________ Coût des médicaments : ______ Autres moyens de paiement (pas en argent liquide): oui ou non si oui, spécifiez_____________ cs11 Tous les médicaments prescrits étaient-ils toujours au centre de santé? Tous...................................... 1 Certains................................. 2 Aucun.................................... 3 NR/pas sûre.......................... 4 cs12 Les fournitures et le matériel nécessaires étaient-ils oui MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 102 disponibles au centre de santé? non si non, expliquez ______________ nr/pas sûre cs13 Au cours des six derniers mois, avez-vous reçu des services de santé dans un autre centre de santé (pas celui-ci)? oui.......................................... 1 non......................................... 2 nr/pas sûre............................ 3 cs14 Si oui, quelle était la raison pour aller dans un autre centre de santé? cs15 Au cours des six derniers mois, y A-T-IL eu un moment où vous aviez besoin de services de santé, mais n'êtes pas allée dans un centre de santé ou un hôpital? oui…………..…..1 non………………2 Si oui, quelle était la raison pour laquelle vous n'avez pas fait appel à des services de santé dans un centre de santé? Trop loin...................................................... 1 Pas assez d'argent pour payer la facture.....2 Personnel non qualifié................................. 3 Infirmier pas accueillant............................... 4 Je préfère la médecine traditionnelle........... 5 Difficulté d'obtention de transport................. 6 Autre (spécifiez) _____________ .................7 NR/pas sûre.................................................. 8 Services d'éducation en santé he01 Quelles pratiques de santé, au besoin, avez-vous apprises grâce au contact avec ces professionnels de la santé? Demandez à nouveau: D’autres pratiques?  Allaitement exclusif  Bonne nutrition  Vaccinations  Prévention et traitement de la diarrhée  Prévention et traitement des infections respiratoires aiguës  Prévention et traitement du paludisme  Éducation sur l'utilisation des méthodes de planning familial a b c d e f g MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 103 Enregistrez tout ce qui est mentionné.  Prévention et traitement du VIH/SIDA  Autre (spécifiez) : __________________ h x he02 Auprès de qui obtenez-vous habituellement des informations générales ou des conseils sur la santé ou la nutrition? demandez à nouveau: d'autres informations ou conseils? Enregistrez tout ce qui est mentionné. Réseau Officiel  médecin  infirmier/sage-femme  sage-femme auxiliaire  personnel de santé communautaire  moniteur de croissance  assistant d'accouchement qualifié réseau informel  époux/partenaire  mère/mère adoptive  sœur  grand-parent  tante  ami/voisin  guérisseur traditionnel  anciens du village  autre (spécifiez)______________ a b c d e f g h i j k l m n x he03 Au cours du dernier mois, avez￾vous reçu des messages de santé à travers les canaux suivants?  Personnel de santé communautaire?  Docteur ou infirmier?  Membre de la famille?  Radio?  Magazine/journal?  Télévision?  École?  Sms? oui 1 1 1 1 1 1 1 1 non 2 2 2 2 2 2 2 2 MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 104  Autre: (spécifiez) 1 2 PERCEPTION DE LA QUALITÉ DES SOINS Perception de la qualité des soins par les patients qc Avez-vous utilisé des services de santé locaux (centre de santé ou hôpital général) au cours des 3 derniers mois? oui…………………1 non.…………………….2 Si oui, demandez aux répondantes de préciser la manière dont elles approuvent ou désapprouvent les déclarations suivantes? Si non, arrêtez l'interview. Impression des soins Pas du tout d'accord Pas d'accord Ni d'accord ni pas d'accord D'accord Tout à fait d'accord (1) (2) (3) (4) (5) qc1 Comportement et pratiques du personnel de santé qc1.1 Démontre compassion et soutien aux patients qc1.2 Montre du respect aux patients qc1.3 Est amical et accueille bien les patients qc1.4 Est honnête qc1.5 Écoute attentivement les patients qc1.6 Les infirmiers consacrent suffisamment de temps aux patients. qc2 Convenance des ressources et des services qc2.2 Les salles sont adéquates qc2.3 Le temps d’attente est MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 105 raisonnable qc2.4 Les infirmiers sont en nombre suffisant qc2.5 Les médicaments sont disponibles à tout moment qc3 Finances et coût des soins qc3.1 Les prix sont négociables qc3.2 Les prix sont raisonnables qc3.3 Avez-vous vu le tarif des soins affiché? qc3.4 Pensez-vous que vous avez payé le vrai prix que vous auriez dû payer? qc3.5 Les médicaments peuvent être obtenus facilement. qc3.6 La distance du centre est raisonnable pour nous (pas trop loin). Merci beaucoup d'avoir accepté de participer à notre enquête. Votre avis est important pour nous. Heure à laquelle l'interview s'est terminée : ____________ MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 106 MINI-ENQUÊTE AUPRÈS DES CENTRES DE SOINS ÉVALUATION DU FBP DE L'IHP À MI-PARCOURS N° Question: 012. Nom de la structure de soins _________________ Localisation GPS: 013. 014. Nom de la zone de supervision (bureau de coordination) __________________________ Nom de la zone sanitaire: 015. ___________________________ 016. Type d'établissement:  Centre de soins (HC)  Hôpital général de renvoi (HGR) 1 2 Nom et code de la province 017.  Kasaï Occidental  Kasaï Oriental  Katanga 1 2 3 018. Catégorie professionnelle du répondant:  Médecin  Infirmier diplômé d'état  Technicien de santé/sage￾femme diplômée d'état  Agent de santé technique  Médecin stagiaire/volontaire  Autre stagiaire/volontaire  Autre (spécifiez) : _________________ 1 2 3 4 5 6 7 8 9 Date et heure de l'enquête 019.  date: /____ /____/______/  heure: /_____ h_______/ min Guide pour l'enquête sur la disponibilité des services et du matériel Trouvez l'infirmier en chef et le directeur/médecin en chef du centre de soins ou de l'hôpital général et MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 107 présentez-vous comme suit : Bonjour. Je m'appelle ____________________. Je représente IBTCI, une société travaillant avec USAID en coopération avec le ministère de la santé publique. nous menons une enquête sur les établissements de soins de santé avec l'appui d'USAID et par l'intermédiaire de MSH/IHP (PROSANI), avec comme but l'identification de manières d'améliorer les services. Nous aimerions vous interviewer au sujet de la situation dans cet établissement, ainsi que la disponibilité des services. Soyez assuré que notre conversation restera strictement confidentielle et que vous ne serez identifié d'aucune manière. À tout moment, vous pouvez choisir d'arrêter l'interview ou refuser de répondre à une question. M'autorisez-vous à commencer? oui... non… Dans la négative, passez directement à la fin du questionnaire. C. Coût des services et du fonctionnement de l'établissement N° Questions Code 020. Simplement, observez et notez Voyez-vous un panneau ou une affiche indiquant la disponibilité des services suivants (encerclez toutes les réponses appropriées)? a) Services de planning familial b) Services de santé infantile c) Soins prénatals d) Accouchement e) Prix pour tout autre service oui oui non dehors à l'intérieur 0 1 2 0 1 2 0 1 2 0 1 2 0 1 2 021. Dans l'affirmative, quel est le prix affiché pour :  Visites initiales /______________/ fc MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 108 N° Questions Code  cas de paludisme  cas de diarrhée  consultation prénatale  accouchement  planning familial  autre (spécifiez) : __________________________  autre (spécifiez) : __________________________  autre (spécifiez) : __________________________  autre (spécifiez) : __________________________ /______________/ fc /______________/ fc /______________/ fc /______________/ fc /______________/ fc /______________/ fc /______________/ fc /______________/ fc /______________/ fc 022. Demandez à la personne interviewée : combien de jours ce centre est-il ouvert aux patients externes? (les patients externes sont ceux qui reçoivent des soins préventifs ou curatifs et qui rentrent à la maison le même jour) Nombre de jours par semaine Nombre de jours par mois Ne sait pas /______/ jours /______/ jours 98 D. Statistiques des services de l'établissement 012. Demandez à la personne interviewée : Quand a démarré l'intervention FBP dans ce centre de soins? Mois : ____________ Année : ____________ MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 109 013. Demandez à la personne interviewée: avez-vous reçu une formation sur l'intervention FBP? oui 1 non 2 pas de réponse 3 ne sait pas 4 Si c'est le cas, quand avez-vous reçu la formation? ________ (mois/année) Quelle a été la durée de la formation: ______________heures/jours 014. Demandez à la personne interviewée: Veuillez nous indiquer les durées pour le 1er trimestre et le 2e trimestre. Ou, en d'autres termes: Quand est-ce que le 1er et le 2e trimestre ont démarré sous l'intervention FBP? ONT-ILS déjà été mis en œuvre cette année ou SONT-ILS encore en cours de mise en œuvre? C.1er trimestre (t1) FBP: de : __________________ (mois à mois): _______ (année) Avez-vous été payé pour le 1er trimestre ? oui 1 non 2 pas de réponse 3 ne sait pas 4 Si oui, quel était le montant du paiement ? ______ 5. 2e trimestre (t2) FBP: de: ___________________ (mois à mois) : _______(année) Avez-vous été payé pour le 2e trimestre ? Oui 1 Non 2 Pas de réponse 3 Ne sait pas 4 Si oui, quel était le montant du paiement ? ______ 015. Demandez à la personne interviewée de vous donner les registres de l'établissement afin de pouvoir répondre aux questions ci-dessous. Examinez les registres de l'établissement et notez les statistiques suivantes de l'établissement pour t1 et t2 (après le début du FBP) N° Indicateur t1 t2 Document/registre source Commentaires Nombre de visites de L. patients externes au centre de soins MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 110 Nombre de femmes M. enceintes inscrites pour bénéficier de soins prénatals Nombre de femmes N. enceintes ayant reçu tt2 Nombre de femmes O. enceintes testées pour le VIH Nombre de femmes P. enceintes testées pour le VIH avec résultats reçus Nombre Q. d'accouchements Assistés par du personnel qualifié Nombre de nouvelles R. consultations de planning familial Nombre d'enfants S. ayant reçu DTP3 NomT. bre de consultations d'enfants malades Nombre de MIILD U. (moustiquaires de lits) distribuées Nombre de clients V. testés et conseillés volontairement pour le VIH MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 111 015. État de l'enquête  Terminée  Partiellement complétée  Refusée  Répondant autorisé introuvable  Établissement introuvable  Autre (spécifiez) : ___________________ 1 2 3 4 5 6 Commentaires de l'intervieweur : Commentaires du superviseur : Date et heure d'achèvement : /____ /____/______/ --------------Fin du questionnaire------------- Vous trouverez ci-dessous une liste des indicateurs FBP des centres de soins qui sont tabulés trimestriellement par l'IHP et l'équipe de gestion de la zone sanitaire: Indicateurs FBP des centres de soins: Taux (nombre) d'utilisation des services curatifs au centre de soins Proportion (nombre) de grossesses à haut risque référées Taux (nombre) de couverture avec DTP-HEPBHIB3 (pentavalent) Proportion (nombre) de femmes enceintes ayant reçu TT2+ Nombre de clients ayant reçu des conseils de planning familial Taux d'accouchements assistés par du personnel de santé Taux (nombre) de détection de tuberculose MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 112 Proportion (nombre) de miild distribuées Taux (nombre) d'utilisation de services de consultation prénatale 1 Nombre de clients testés et conseillés volontairement pour le VIH Nombre de femmes enceintes testées pour le VIH Taux (nombre) de couverture PNC 4 (recentré) Taux (nombre) d'utilisation de PONC 2+ % (nombre) de rapports mensuels de gestion et d'inventaire de médicaments ayant été analysés et envoyés à temps au bureau central de la zone en question Score FOSACOF global du centre de soins Score de satisfaction globale des patients du centre de soins MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 113 ANNEX IV. List of Documents Reviewed List of Documents Reviewed for the Midterm RBF Evaluation 1) Title: GUIDE OPERATIONNEL DE L’APPROCHE DU FINANCEMENT BASE SUR LES RESULTATS DANS LE SECTEUR DE LA SANTE Author: REPUBLIQUE DEMOCRATIQUE DU CONGO, MINISTERE DE LA SANTE PUBLIQUE SECRETARIAT GENERAL Date: October 2012 2) Title: MODULE DE FORMATION DES ACTEURS DU NIVEAU PROVINCIAL SUR LE FINANCEMENT BASE SUR LES RESULTATS Author: REPUBLIQUE DEMOCRATIQUE DU CONGO MINISTERE DE LA SANTE PUBLIQUE SECRETARIAT GENERAL Date: October 2012 3) Title: MODULE DE FORMATION DES ACTEURS DU BUREAU CENTRAL DE LA ZONE DE SANTE SUR LE FINANCEMENT BASE SUR LES RESULTATS Author: REPUBLIQUE DEMOCRATIQUE DU CONGO MINISTERE DE LA SANTE PUBLIQUE SECRETARIAT GENERAL Date: October 2012 4) Title: MANUEL DE FORMATION DES PRESTATAIRES DES ETABLISSEMENTS DE SOINS (HOPITAUX ET CENTRES DE SANTE) SUR LE FINANCEMENT BASE SUR LES RESULTATS Author: REPUBLIQUE DEMOCRATIQUE DU CONGO MINISTERE DE LA SANTE PUBLIQUE SECRETARIAT GENERAL Date: October 2012 5) Title: MODULE DE FORMATION DES VERIFICATIONS COMMUNAUTAIRES SUR LE FINANCEMENT BASE SUR LES RESULTATS Author: REPUBLIQUE DEMOCRATIQUE DU CONGO MINISTERE DE LA SANTE PUBLIQUE SECRETARIAT GENERAL Date: October 2012 6) Title: RBF Manual, Integrated Health Project in the Democratic Republic of Congo MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 114 Author: MSH Date: January – March 2014 7) Title: Résultats du premier semestre du programme FBR mis en œuvre par MSH/PROSANI avec le financement de l’USAID (PPT presentation) Author: PROSANI Date: November 2013 – April 2014 8) Title: RBF implementation report for the zones supported by IHP Author: The Ministry of Public Health's RBF Unit Date: 2014 MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 115 ANNEX V. Sources of Information LIST OF CONTACTS S. NO. NAME POSITION ORGANIZATION LOCATION TELEHONE EMAIL 1 Albert Mbuyi SECRETORY GENERAL CSO-CENTER MUKUNDA LUIZA 995616604 2 Bernard Kasaka Mwana Dijashi CHIEF NURSE KITOKO HC LUIZA 3 Dr. Denis Mpika DIRECTOR, HGR HGR LUIZA LUIZA 0994361081, 0822919191 4 Dr. Fernand Cibwabwa Gyembe MCZ MEDICAL OFFICER ECZ LUIZA LUIZA 0995267924, 0812467169 FERNANDCIBW ABWA@GMAIL. COM 5 Fortunant Malala CHIEF NURSE KAMISHILU HC LUIZA 995617482 6 Francois Tshibangu MEMBER CSO-CONTRADE LUIZA 0977538341, 0810528475 TSHIBANGUFR ANC@GMAIL.C OM 7 Gustave Kabutakapua HEAD OF OFFICE, RESOURCE MANAGEMENT, W. KASAI MSP-MIP KAPANGA 0816004396, 0993592844 8 Lucieu Kabeya Badibanga CHIEF NURSE KAMAYI HC LUIZA 990801145 9 Mukadi Kampemba CHIEF NURSE KABUANGA HC LUIZA 992923304 10 Dr. Jean Michel Mutombo BCZ MEDICAL OFFICER BCZ BIGANGA BIBANGA 0997330360, 0852749943 11 Dr. Nestor Tshiteku DIRECTOR- HGR HGR BIBANGA BIBANGA 0991254344, 0856129660 DOCTORBON HEUR@GMAIL. COM 12 Espor Mbuyi Bukasa CHIEF NURSE KATANDA-1 HC BIBANGA 0998348649, 0851777285 MWENE￾DITU 998626359 14 Jean Crispin Kazadi CHIEF NURSE CIKUYI HC BIBANGA 0856706507, 0992952526 15 Joachim Kazadi COORDINATOR CSO-ADDIM MWENE￾DITU 0854351473, 0816065786 JOACHIMKAZA DI@YAHOO.FR 16 Kanyinda Kalenga David CHIEF NURSE STATION HC BIBANGA 0970184199, 0852274995 17 Leonard Kayembe MEMBER CSO-ADDIM MWENE￾DITU 0856167852, 0994839145 MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 116 18 CHIEF NURSE BIBANGA HC BIBANGA 19 Chango Chachi Joseph CHIEF NURSE OTOHE HEALTH CENTER OTOHE 20 Djamba Kipata GRH DIRECTOR GRH WEMBO NYAMA 993020938 21 John Shutsha CHIEF NURSE OLUTA HEALTH CENTER OLUTA 998264296 22 Jose Pastor HEAD OF CODESA CODESA OLUTA 23 Londola Lodi Guy HEALTH ZONE MANAGER MSP-HEALTH ZONE MANAGEMENT WEMBO NYAMA 993900027 ANTONIEKALU LAMBI2013@G MAIL.COM 24 Nygoia HEAD OF CODESA CODESA OTOHE 25 PRESIDENT CSO-ADIS AHAMBA 992128045 26 Bertrand Charla Muzala PRESIDENT CODESA KAMOA 081 087 6008 27 Claude Soneka Makayi CHIEF NURSE KANTALA HC KANTALA 082 143 9645 28 Dr. Job Matumba DIRECTOR CSO-PLANET SANTE KAMOA 0970 016 043, 081 273 5363 29 Freddy Kyungu PRESIDENT CSO-ASEMIR KANTALA, MPALA, KAMIMBI 082 327 9520 30 Kalongo Mwanza CHIEF NURSE KAMIMBI HC KAMIMBI 099 845 2974 31 Kashala Tshijika Cesar CHIEF NURSE MPALA HC MPALA 099 131 7391 32 Pascal Mukumbi Mwambu CHIEF NURSE KAMOA HC KAMOA 819379427 33 Adamo Fumie Bonay DIRECTOR OF IHP BC KOLWEZI IHP BC KOLWEZI 995200206 ADMAOFUMIE.B ONAY@RESCU E.ORG 34 Daniel Wutshu FOCAL FBR IHP HC TSHUDI LOTO 35 Didace Demba DIRECTOR, FOCAL FBR IHP BC MWENE-DITU 36 Dr. Francine Ngalula TECHNICAL ADVISOR IHP BC MBUJI MAYI 37 Frances Kambo PBF FOCAL POINT AND CAPACITY BUILDING SPECIALISTS IHP KOLWEZI BC PROSANI/IRC STAFF (NOT MSH) KOLWEZI 099 5200319 FRANCES.KAMB O@RESCUE.OR G 38 Freddy Mukeda M&E SPECIALIST IHP BC LUIZA 243971016187 39 Gilbert Andrianandrasana TECHNICAL DIRECTOR OF IHP/PROSANI IHP KINSHASA 40 Matthieu Lutendo SR. CAPACITY BUIDLING SPECIALIST, FOCAL FBR IHP BC LUIZA 0971016188, 0998910705, MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 117 0818081768 41 Derek Kahongo WEB-PORTAL IN-CHARGE, IT COORDINATOR IHP KINSHASA 970007782 42 Freddy Tshamala CAPACITY BUILDING MANAGER AND QA ADVISOR IHP KINSHASA 970001689 43 Ousmane Faye CHIEF OF PARTY IHP KINSHASA MLUTONDO@ MSH.ORG, LUTONDO@Y AHOO.FR 44 Rafael Shinzela DIRECTOR OF BC IHP BC LODJA AND TSHUMBE 995905472 RTSHINZELA@ MSH.ORG 45 Delmond Kyanza SR. TECHNICAL ADVISOR IHP KINSHASA 46 Tchim Tabaro DEPUTY CHIEF OF PARTY IHP KINSHASA 47 Dr. Lina Piripiri MCH SPECIALIST USAID KINSHASA 48 Dr. Godefroid Mayala RH AND HSS SPECIALIST USAID KINSHASA 49 Meri Sinnitt HEALTH TEAM LEADER USAID KINSHASA 50 Charley Tchomba Tulia CHIEF OF RESOURCE MOBILIZATION OF RESOURCES MSP-CENTRAL RBF UNIT KINSHASA 813204275 CTCHOMBAT@ YAHOO.FR 51 Dr. Celestin Bukanga CHIEF COORDINATOR CENTRAL RBF UNIT MSP-CENTRAL RBF UNIT KINSHASA 52 L. Shamashanga CHIEF OF ADMINISTRATION AND FINANCE MSP-CENTRAL RBF UNIT KINSHASA EMERMUKENAT @YAHOO.FR 53 Dr. Raymond Cambele M&E OFFICER IN-CHARGE MSP-CENTRAL RBF UNIT KINSHASA 243-81-1630467 54 Dr. Makaya CHIEF OF PLANNING MSP-CENTRAL RBF UNIT KINSHASA MAKAYADAMA SE@YAHOO.FR 55 Dr. Mukena MIP MSP-MIP MBUJI MAYI 0997318403, 0816556549 56 Eric Mukomena Sompwe PROVINCIAL MEDICAL OFFICER (MIP) MSP-KATANGA PROVINCIAL HEALTH OFFICE KATANGA 0998 281 568, 0815 059 229 DRERICSOMP@ YAHOO.FR 57 Jasque Kabamba DISTRICT MEDICAL OFFICER MSP-DISTRICT MEDICAL OFFICE KANZENZE 995926363 58 Jean Jasques Muluka Nsengadps, PROVINCIAL MEDICAL OFFICER (MIP) MSP-PROVINCIAL HEALTH OFFICER, DPS KANZENZE 0997105590, 0814043234 MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 118 59 Jean Martin Malaba DISTRICT HEALTH OFFICER MSP-HEALTH ZONE SANKURU 243 819 617 879 60 Jeannette Boniche Rosales DIRECTOR OF KANZENZE HOSPITAL MSP KANZENZE 243 816 770 836 JANETBONICHE @GMAIL.COM 61 Londola Lodi Guy HEALTH ZONE MANAGER MSP-HEALTH ZONE MANAGEMENT WEMBO NYAMA 993900027 ANTONIEKALU LAMBI2013@G MAIL.COM 62 Bruno Mwenya HEALTH ZONE MANAGER MSP-HEALTH ZONE CENTRAL BUREAU, KANZENZE, KOLWEZI KOLWEZI 979660417 63 Antonie Kalulambi PRINCIPAL ADMIN, CHIEF OF STAFF, W. KASAI MSP-PROVINCIAL MOH KAPANGA 0997406379, 0823516133 64 Dr. Desire Iseloko CHIEF MEDICAL OFFICER￾DISTRICT DPS BIBANGA MBUJI MAYI 0995040900, 0823654185 DESIRE_ISELOK O@YAHOO.FR MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 119 ANNEX VI. Field Implementation Plan The field work and data collection will start on or about October 4, 2014 with six working days per week in-country. The team will be divided in two sub-teams (team A and team B) for simultaneous data collection to cover 4 health zones in three provinces. A detailed FIP is provided in table 1, 1a and 1b below: TABLE 1: FIELD IMPLEMENTATION PLAN WEEK DATE DAY ACTIVITIES WEEK 0 10/4/2014 SAT TRAVEL FROM USA TO KINSHASA WEEK 1 10/5/2014 SUN TRAVEL FROM USA TO KINSHASA 10/6/2014 MON IN-BRIEF WITH USAID (AM) AND TEAM PLANNING MEETING (PM) 10/7/2014 TUE TEAM SPLITS IN TWO SUB-TEAMS (A AND B) AND TRAVELS TO FIELD SUB-TEAM A SUB-TEAM B TRAVEL TO LODJA FROM KINSHASA AIRPORT, MEETING WITH LODJA HEALTH DISTRICT IHP KII IN KINSHASA 10/8/2014 WED TRAVEL TO TSHUMBE, MEETING WITH IHP BC (PM) USAID KII IN KINSHASA 10/9/2014 THU TRAVEL TO WEMBONYAMA, TRAVEL TO KANANGA AND MEETING WITH IHP OFFICE (PM) 10/10/2014 FRI DATA COLLECTION IN HEALTH CENTERS TRAVEL TO LUIZA (PM) 10/11/2014 SAT DATA COLLECTION IN HEALTH CENTERS VISIT GRH AND MEETING WITH IHP LUIZA WEEK 2 10/12/2014 SUN SUNDAY TRAVEL TO LODJA SUNDAY 10/13/2014 MON DATA COLLECTION IN HEALTH CENTERS DATA COLLECTION IN HEALTH CENTERS 10/14/2014 TUE TRAVEL TO KINSHASA DATA COLLECTION IN HEALTH CENTERS 10/15/2014 WED MOH KII IN KINSHASA DATA COLLECTION IN HEALTH CENTERS 10/16/2014 THU MOH KII OR ANY REMAINING KII IN KINSHASA DATA COLLECTION IN HEALTH CENTERS MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 120 10/17/2014 FRI TRAVEL TO LUBUMBASHI FROM KINSHASA AIRPORT TRAVEL TO MUENEDITU BY CAR 10/18/2014 SAT MEETING WITH PROVINCIAL MHO AND IHP OFFICE (LUBUMBASHI) MEETING WITH IHP BC WEEK 3 10/19/2014 SUN TRAVEL TO KOLWEZI TRAVEL TO MBUJIMAYI 10/20/2014 MON MEETING WITH KOLWEZI HEALTH DISTRICT AND IHP BC AND TRAVEL TO KANZENZE IN THE AFTERNOON MEETING WITH PROVINCIAL MHO IN THE MORNING AND TRAVEL TO BIBANGA 10/21/2014 TUE VISIT GRH AND MEETING WITH ECZ TEAM. DATA COLLECTION IN HEALTH CENTERS VISIT GRH AND MEETING WITH ECZ TEAM AND TRAVEL TO THE FIRST HC 10/22/2014 WED DATA COLLECTION IN HEALTH CENTERS DATA COLLECTION IN HEALTH CENTERS 10/23/2014 THU DATA COLLECTION IN HEALTH CENTERS DATA COLLECTION IN HEALTH CENTERS 10/24/2014 FRI DATA COLLECTION IN HEALTH CENTERS DATA COLLECTION IN HEALTH CENTERS 10/25/2014 SAT TRAVEL TO LUBUMBASHI FROM KOLWEZI DATA COLLECTION IN HEALTH CENTERS WEEK 4 10/26/2014 SUN TRAVEL TO KINSHASA TRAVEL TO KINSHASA 10/27/2014 MON OUT-BRIEFING WITH USAID 10/28/2014 TUE STAKEHOLDER'S PRESENTATIONS (IHP AND RBF EVALUATION FINDINGS) 10/29/2014 WED TRAVEL TO USA: ANNETTE 10/30/2014 THU DATA ANALYSIS OR ANY FOLLOW-UP KII IN KINSHASA 10/31/2014 FRI DATA ANALYSIS OR ANY FOLLOW-UP KII IN KINSHASA 11/1/2014 SAT TRAVEL TO USA: SWATI MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 121 TABLE 1A: DETAILED FIELD VISIT PLAN FOR TEAM A WEMBO NYAMA AND KANZENZE (TEAM A) Date Activity Site Persons to be contacted Time frame 10/07/2014 Travel to Lodja Lodja from 8:00 am to 11:00 am Meeting with Lodja health district Lodja Dr. Malaba from 2:30 to 4:30 pm 10/08/2014 Travel to Tshumbe Tshumbe JHP BC coordinator from 6:00 am to 3:00 pm Meeting with IHP BC coordinator Tshumbe from 4:00 to 5:30 pm 10/09/2014 Travel to Wembo Nyama Wembo Nyama from 8:00 to 10:00 am Meeting with IHP BCZ medical doctor Wembo Nyama BCZ medical doctor from 10:30 am to 12:30 pm Visiting GRH Wembo Nyama GRH director from 1:00 pm to 2:00 pm Travel to Olota Olota chief nurse from 3:00 pm to 5:00 pm 10/10/2014 Data collection to Olota Olota chief nurse from 8:00 am to 4:00 pm Travel to Tshekopoto (Zephyrin) Tshekopoto chief nurse Travel to Ahamba (Annette) Ahamba chief nurse 10/11/2014 Data collection in Tshekepoto HC Tshekopoto chief nurse from 8:00 am to 4:00 pm Data collection in Ahamba HC Ahamba chief nurse from 8:00 am to 4:00 pm MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 122 Travel to Otohe HC (Annette and Zéphyrin) Otohe chief nurse 10/12/2014 Data collection in Otohe (Annette and Zéphyrin) Otohe chief nurse from 8:00 am to 4:00 pm Travel to Tshumbe Tshumbe 10/13/2014 Travel to Lodja Lodja from 11:00 to 4:30 pm 10/14/2014 Travel to Kinshasa Kinshasa from 11:00 to 4:30 pm 10/15/2014 KII MoH at Kinshasa 10/16/2014 KII MoH or remaining KII at Kinshasa 10/17/2014 Travel to Lubumbashi from Kinshasa airport Lubumbashi 18/10/2014 meeting with provincial mho and IHP office (Lubumbashi) Lubumbashi MIP from 10:00 to 12:30 pm Meeting with IHP office Lubumbashi Lubumbashi Dr. Augustin Mwala from 2:00 to 4:00 pm 19/10/2014 Travel to Kolwezi and meeting with Kolwezi health district and IHP BC. Kolwezi IHP BC coordinator medical district manager 20/10/2014 Travel to Kanzenze and visit GRH and meeting with ECZ team. Kanzenze BCZ medical doctor GRH manager 21/10/2014 Travel and data collection in health centers Kamimbi Kamimbi chief nurse from 6:00 am to 4:00 pm 22/10/2014 Travel and data in health centers Kamoa Kamoa chief nurse from 6:00 am to 4:00 pm 23/10/2014 Travel and data in health centers Kantala Kantala chief nurse from 6:00 am to 4:00 pm MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 123 24/10/2014 Mravel and data in health centers Mpala Mpala chief nurse from 6:00 am to 4:00 pm 25/10/2014 Travel to Lubumbashi from Kolwezi Lubumbashi 26/10/2014 Travel to Kinshasa Kinshasa MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 124 ANNEX VII. Additional Data Analysis SNIS 2014 Indicators MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 125 MIDTERM ASSESSMENT OF A RESULTS-BASED FINANCING INTERVENTION 126