An Evaluation and Cost-effectiveness Analysis of a Collaborative Improvement Intervention for Pre-eclamspia/Eclampsia Care in Mali RESEARCH AND EVALUATION REPORT SEPTEMBER 2014 This report was prepared by University Research Co., LLC (URC) for review by the United States Agency for International Development (USAID). It was authored by Astou Coly, Karim Sangare, Edward Broughton, Sabou Djibrina, Zakari Saley, Abdoulaye Sylla, Maina Boucar, Ibrahima Mahamadou Dicko, and Haneefa Saleem of URC. This study was carried out under the USAID Health Care Improvement Project, which is made possible by the generous support of the American people through USAID. RESEARCH AND EVALUATION REPORT An Evaluation and Cost-effectiveness Analysis of a Collaborative Improvement Intervention for Pre-eclampsia/Eclampsia Care in Mali SEPTEMBER 2014 Astou Coly, University Research Co., LLC Karim Sangare, University Research Co., LLC Edward Broughton, University Research Co., LLC Sabou Djibrina, University Research Co., LLC Zakari Saley, University Research Co., LLC Abdoulaye Sylla, University Research Co., LLC Maina Boucar, University Research Co., LLC Ibrahima Mahamadou Dicko, University Research Co., LLC Haneefa Saleem, University Research Co., LLC DISCLAIMER The views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Acknowledgements: The authors thank the head of the Division of Reproductive Health at the DNS and his team, the Regional Director of Health in Kayes and teams from the Kayes, Diéma, and Yélimané CSREF. The authors also thank the health team of the USAID mission in Mali, HCI technical advisor Kadiatou N'Diaye, all facility teams, coaches and data collectors who participated in this study. HCI is managed by University Research Co., LLC (URC) under the terms of Contract Number GHN-I￾03-07-00003-00. URC’s subcontractors for HCI include EnCompass LLC, FHI 360, HEALTHQUAL International, Initiatives Inc., Institute for Healthcare Improvement, and Johns Hopkins University Center for Communication Programs. For more information on HCI’s work, please visit www.usaidassist.org. Recommended citation: Coly A, Sangare K, Broughton E, Djibrina S, Saley Z, Sylla A, Boucar M, Dicko IM, Salem H. 2014. An evaluation and cost-effectiveness analysis of a collaborative improvement intervention for pre-eclampsia/eclampsia care in Mali. Research and Evaluation Report. Published by the USAID Health Care Improvement Project. Bethesda, MD: University Research Co., LLC (URC). Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali • i TABLE OF CONTENTS List of Figures and Tables ..................................................................................................................................................i Abbreviations .......................................................................................................................................................................ii EXECUTIVE SUMMARY ................................................................................................................................................. iii I. INTRODUCTION ..................................................................................................................................................... 1 II. METHODOLOGY ..................................................................................................................................................... 2 A. Study Design ................................................................................................................................................................ 2 B. Description of the Collaborative Improvement Intervention ......................................................................... 3 C. Sampling ........................................................................................................................................................................ 4 D. Data Collection........................................................................................................................................................... 5 E. Analysis ......................................................................................................................................................................... 6 III. RESULTS ...................................................................................................................................................................... 7 A. Characteristics of the Study Sample ...................................................................................................................... 7 B. Adherence to Standards for Pre-eclampsia and Eclampsia Care (Unadjusted Analyses) ......................... 9 C. Regression Analyses ................................................................................................................................................ 12 D. Cost-effectiveness Analysis .................................................................................................................................... 13 IV. DISCUSSION ............................................................................................................................................................ 14 A. Limitations .................................................................................................................................................................. 16 V. CONCLUSION AND RECOMMENDATIONS............................................................................................... 16 REFERENCES .................................................................................................................................................................... 17 List of Figures and Tables Figure 1.Geographical location of intervention district (Diéma) and control district (Yélimané) ................. 3 Figure 2. Components of intervention cost .............................................................................................................. 14 Table 1. Activities in intervention and control sites ................................................................................................. 3 Table 2. Planned and actual timeline of intervention and study activities ............................................................ 4 Table 3. Number of charts reviewed in health centers in the intervention and control districts ................. 7 Table 4. Characteristics of patients in the intervention and control districts .................................................... 7 Table 5. Key characteristics of providers in intervention and control sites at baseline and end-line ........... 8 Table 6. Mean PEE knowledge score among providers by key provider characteristics ................................. 9 Table 7. Number of cases of pre-eclampsia and eclampsia detected by providers based on chart reviews ............................................................................................................................................................................... 10 Table 8. Unadjusted changes to standards of pre-eclampsia/eclampsia care by study arm and data collection time based on charts review ...................................................................................................................... 11 Table 9. Adherence to pre-eclampsia screening standards estimated by chart review and observations . 12 Table 10. Adherence to pre-eclampsia screening standards ................................................................................. 12 Table 11. Total adherence to pre-eclampsia/eclampsia screening and treatment standards ........................ 13 Table 12. Changes in adherence for the intervention and control group ......................................................... 13 Table 13. Incremental cost-effectiveness ratio results in CFA francs ................................................................. 14 ii • Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali Abbreviations AIDS Acquired immune deficiency syndrome AMTSL Active management of the third stage of labor ANC Antenatal care ASACO Association de santé communautaire (community health association) ASSIST USAID Applying Science to Strengthen and Improve Systems Project AZT Azidothymidine, also called zidovudine CFA Communauté française d'Afrique (French Community of Africa) CI Confidence interval CSCOM Community health center CSREF Referral health center DALY Disability-adjusted life-years DD Differences-in-differences ENC Essential newborn care EONC Essential obstetric and newborn care HAART Highly active antiretroviral therapy HCI USAID Health Care Improvement Project HIV Human immunodeficiency virus MOH Ministry of Health PEE Pre-eclampsia/eclampsia PMTCT Prevention of mother-to-child transmission of HIV OR Odds ratio URC University Research Co., LLC US United States USAID United States Agency for International Development VAP Ventilator-associated pneumonia WHO World Health Organization Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali • iii EXECUTIVE SUMMARY Introduction Empirical evidence for the effectiveness of collaborative improvement interventions in low- and middle￾income countries is limited. Due to operational restrictions, most evaluations have used uncontrolled pretest/post-test designs that cannot rule out other plausible causes for observed improvements. The USAID Health Care Improvement (HCI) Project and its successor, the USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project, have implemented a collaborative improvement intervention to improve the quality of essential obstetric and newborn care (EONC) services, including active management of the third stage of labor and essential newborn care, in two health districts (Diéma and Kayes) in the Kayes Region of Mali since early 2010. HCI started implementing a second improvement intervention aimed at the quality of clinical practice regarding pre-eclampsia and eclampsia care in 2011. This evaluation sought to determine the costs and effects of this collaborative improvement intervention and compare them to the costs and effects of a basic clinical training only. The specific research questions were: 1) Do pregnant and delivering women in collaborative improvement intervention facilities receive better care (screening/diagnostic and treatment of pre-eclampsia/eclampsia) than those in basic clinical training-only facilities? 2) What is the incremental cost-effectiveness of the collaborative improvement intervention compared to the basic clinical training-only intervention, in terms of adherence to pre￾eclampsia/eclampsia screening and management standards? Methodology This evaluation used a controlled longitudinal design. Intervention sites were facilities participating in the EONC and pre-eclampsia/eclampsia (PEE) improvement collaborative in Diéma District (seven community health centers and the district referral hospital), and the control sites were facilities in Yélimané District (six community health centers and the district referral hospital). As part of the study, control facilities received basic clinical training on PEE. However, additional trainings were subsequently conducted by the Ministry of Health at some sites. At the request of the Ministry of Health, only facilities with at least one physician were included in the PEE intervention. To ensure comparability, we also applied this restriction to the control facilities. Three data collection methods were used: patient chart reviews, structured observation of providers, and a self-administered questionnaire for providers. Due to the March 2012 coup d’état and ensuing suspension of HCI activities in Mali from March to August 2012, modifications were made to the initial evaluation design. As a result, data were collected four times using chart review, and twice (baseline and end line) using observations and self-administered questionnaires between February 2011 and June 2013. In addition, unlike most HCI-supported collaborative improvement activities, the implementation of the intervention included only one learning session and four coaching visits which included but did not focus on PEE. Data on adherence to PEE screening and management standards were calculated based on chart reviews and observations. Hierarchal regression models with differences-in-differences analyses were used to adjust for clustering of observations by site and baseline differences in terms of adherence to pre￾eclampsia/eclampsia screening and management standards. Potential confounders such as woman’s age and parity were also controlled for in the regression models. Costs for the intervention were obtained from project records and used as inputs for the decision model, along with the results from the logistic regression analysis. We used the perspective of the payer iv • Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali of the intervention. The two outputs used were costs for compliance to screening and PEE management standards. The study was approved by the National Ethics Committee of Mali and the URC Institutional Review Board. Findings A total of 1756 charts were reviewed: 893 in the intervention district and 863 in the control district. Only 32 pre-eclampsia and 20 eclampsia cases were detected during the evaluation based on chart reviews. To validate data collected from chart reviews, observations were conducted at baseline and end line to estimate adherence to standards for pre-eclampsia screening. Baseline scores for pre￾eclampsia screening based on observations appear higher than the mean score obtained from chart review in the intervention facilities (0.96 versus 1.20) and lower than the score obtained from chart reviews in the control districts (0.57 versus 1.02). At end line, the scores in both districts appeared slightly higher based on observation than chart review. Data from observations confirmed that end line scores were higher in the intervention than the control group (4.20 versus 3.12; p<0.001). Regression analyses showed a modest effect of the intervention on adherence to pre-eclampsia screening standards and overall adherence to PEE screening and management standards. On average, the intervention group improved by 0.02 points for adherence to screening and 0.38 points for overall adherence per month (p<0.001). In addition, the intervention group had a 7% higher odds of scoring at least as high as the 75th percentile for overall adherence to PEE standards (p=0.035). The intervention was also associated with 6% higher odds of scoring at least the 75th percentile or above for screening standards. However, this odds ratio was only marginally significant (p=0.05).The differences in scores for screening adherence and overall adherence attributable to the intervention between baseline and end￾line are 0.46 and 8.8 points, respectively. The incremental cost-effectiveness ratios were 524,000 CFA francs per additional patient screened according to standards and 453,000 CFA francs per additional patient managed according to PEE standards. Conclusions and Recommendations This controlled evaluation contributes to the much-needed evidence base for the effectiveness of collaborative improvement interventions. While it demonstrated a positive effect of a collaborative improvement intervention on pre-eclampsia and eclampsia care, the result was weaker than expected. Several factors may have led to under-performance of the intervention, including the fact that only one learning session and few coaching visits were organized during the intervention, the suspension of intervention activities in Mali due to the political situation, and implementation by the Ministry of Health of pre-eclampsia and eclampsia interventions in the control district. Future evaluations are needed to assess the effectiveness of improvement collaboratives with more certainty to determine factors that promote or hinder the effectiveness of collaborative improvement interventions in a variety of settings. For approximately one unit of Gross Domestic Product per capita, we expect one additional patient to receive screening compliant with standards of care and one additional patient to receive care compliant with overall care standards. It was not possible to link these process measures to specific health outcomes; therefore, this result leaves the cost-effectiveness of the program dependent on willingness to pay for these process outcomes. The 95% confidence interval for this estimate indicates this is not a robust result, and there is a small possibility that the strategy without the improvement intervention may be as or more cost-effective than the strategy with the improvement intervention. This suggests that the investments in this relatively low level of inputs aimed at improving health system performance may not yield acceptably efficient results. Other studies have shown that improvement interventions can be successful, but such studies involved more intensive activities for supporting personnel in the targeted facilities. Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali • 1 I. INTRODUCTION Most research regarding the effectiveness of collaborative improvement approaches has focused on high-income countries and obtained mixed results. A systematic review of the effectiveness of collaborative improvement interventions showed moderate positive effects on outcome measures in seven out of the nine studies that met study criteria and no significant effect on the outcomes of interest in the remaining studies (Schouten et al. 2008). The studies included in the review were limited to high￾income countries with a longer history of improvement initiatives, including the United States, the United Kingdom, other European countries, Canada, and Australia. In these high-resource settings, collaborative improvement interventions were associated with reductions in neonatal deaths, pain, infant infections, and treatment costs for infants and improvements in quality of life, patient satisfaction, patient knowledge, self-management behaviors, and surfactant treatment for premature infants in the delivery room. However, the authors found that the positive effect of collaborative improvement programs could not be predicted with great certainty. They argued that research should be conducted to understand why some improvement interventions and some organizations participating in them are successful while others are not (Schouten et al. 2008). Research examining factors that influence the effectiveness of collaborative improvement activities, again primarily limited to high-income settings, found that some aspects of teamwork enhanced short-term success of improvement collaboratives, as well as participation in specific collaborative activities, including gathering data. However, the authors found little empirical evidence of positive effects of leadership support, sufficient resources, or time on collaborative effectiveness (Hulscher et al. 2013). Empirical evidence for the effectiveness of collaborative improvement interventions in low- and middle￾income countries is limited, but growing. A health systems strengthening intervention aimed at accelerating highly active antiretroviral treatment (HAART) initiation in South Africa using a collaborative improvement approach demonstrated increases in HIV testing by over 300% and monthly HAART initiation by 185% among clinics participating in the intervention (Webster et al. 2012). Another study conducted in South Africa examined the impact of a combination of approaches to health systems strengthening, including a Breakthrough Series Collaborative, on prevention of mother-to-child transmission (PMTCT) of HIV (Youngleson et al. 2010). Findings from this study showed a reduction in the proportion of HIV-exposed infants testing positive for HIV and increases in antenatal AZT, PMTCT clients on HAART at the time of labor, intrapartum AZT, and postnatal HIV testing from baseline assessment. In addition, a collaborative improvement intervention conducted in 18 secondary and tertiary care hospitals in Thailand to reduce rates of ventilator-associated pneumonia (VAP) resulted in a decrease in the VAP rate from 13.3 to 8.3 per 1,000 ventilator days in 12 months and a reduction by more than half in the costs of antibiotic treatment for VAP (Unahalekhaka et al. 2007). However, these studies, like many others examining the effectiveness of collaborative improvement approaches, was limited by the absence of a comparison group, which makes it difficult to attribute results to the intervention. An evaluation of the USAID Health Care Improvement (HCI) Project summarizing the results of collaborative improvement in 12 low- and middle-income countries by over 1300 teams during 1998- 2008 has shown that teams were able to achieve large increases in compliance with health care standards and in some cases, health outcomes, in maternal, newborn and child health, HIV/AIDS, family planning, malaria, and tuberculosis, regardless of the baseline level of compliance (Franco & Marquez 2011). This multi-country evaluation used data from 27 settings and is believed to be the largest body of evidence on the effectiveness of improvement collaboratives in low- and middle-income countries. However, due to operational restrictions, this evaluation also used an uncontrolled pretest/post-test design that cannot rule out other plausible causes for observed improvements such as secular trends. 2 • Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali More recently, HCI conducted it first controlled study in Uganda which showed that a collaborative improvement intervention was associated with higher odds of improvement in terms of compliance with maternal and newborn performance indicators. This evaluation also reported an incremental cost￾effectiveness of $20 per additional increase in key indicators of quality in maternal and neonatal care. However, in this study, missing data were considered as indicating non-compliance as removing those missing data resulted in a sample too small to detect statistically significant differences. Given that findings were based on a strong assumption regarding missing data, the authors recommended conducting additional controlled evaluations to assess the effectiveness and efficiency of improvement collaboratives with more certainty (Broughton et al., 2014). We sought to address the need for controlled evaluations to assess the effectiveness of the collaborative improvement approach by comparing the costs and outcomes for the clinical management of pre-eclampsia and eclampsia in collaborative improvement facilities to facilities with no collaborative improvement intervention and monitor changes in quality performance over time in Mali. HCI and its successor, the USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project, have implemented an essential obstetric and newborn care (EONC) collaborative improvement intervention, including active management of the third stage of labor (AMTSL) and essential newborn care (ENC), in two health districts (Diéma and Kayes) in the Kayes Region of Mali since early 2010. At the time this study was started, the intervention included 41 facilities that had surpassed 80% average compliance with AMTSL and ENC standards and were working on maintaining or improving performance. HCI started implementing a second collaborative phase aimed at improving clinical practice for pre-eclampsia and eclampsia care in facilities in the Diéma and Kayes districts at the end of February 2011. We conducted a study to determine whether a collaborative intervention to improve the quality of pre￾eclampsia and eclampsia care has an added value above clinical training alone. We aimed to determine the costs and effect of this quality improvement collaborative intervention and compare them to the costs and effects of a basic clinical training only. The basic clinical training provided an overview of screening, classification, treatment, and monitoring for pre-eclampsia/eclampsia to health providers. The specific evaluation questions were: 1. Do pregnant and delivering women in collaborative improvement intervention facilities receive better care (screening/diagnosis and treatment of pre-eclampsia/eclampsia) than those in basic clinical training-only facilities? 2. What is the incremental cost-effectiveness of the collaborative improvement intervention compared to the basic clinical training-only intervention in terms of adherence to pre￾eclampsia/eclampsia screening and case management standards? II. METHODOLOGY A. Study Design This evaluation used a controlled longitudinal design. Intervention sites were facilities participating in the EONC and pre-eclampsia/eclampsia (PEE) improvement collaborative in the Diéma District, and the control sites were facilities in the Yélimané District. Control facilities received basic clinical training on PEE. Figure 1 shows the geographical location of the intervention and control districts. Activities conducted in the intervention and control sites are listed in Table 1. The clinical training-only sites switched to the collaborative intervention so that at the end of the study all sites were part of the intervention. This allowed the delivery of an intervention, thought to be beneficial, to all sites. Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali • 3 Figure 1.Geographical location of intervention district (Diéma) and control district (Yélimané) Baseline data were to be collected in May 2011 and end-line in May 2012 with additional data collected every three months. However, this evaluation was disrupted by the March 2012 coup d’état and ensuing suspension of HCI activities in Mali from March to August 2012. Modifications were made to the initial evaluation design. As a result, baseline data collected were collected in May 2011 and end-line data in June 2013 with additional data collected in October 2011 and March 2012. Table 1. Activities in intervention and control sites Intervention Control Clinical training in pre-eclampsia/eclampsia Yes Yes Quality improvement methodology training Yes No Coaching visits Yes No Learning session Yes No B. Description of the Collaborative Improvement Intervention Although this evaluation focused on the effectiveness of the PEE collaborative, facilities participating in the PEE collaborative were also part of the EONC collaborative. All study sites received clinical training on pre-eclampsia screening and management and eclampsia management. In addition, intervention sites received improvement methodology training and coaching visits and participated in one learning session.  Improvement methodology training: Intervention sites received improvement methodology training at the beginning of the intervention. The training included an introduction to improvement methods, improvement collaborative and quality improvement teams functioning, and data collection and data monitoring. Providers who participated in the training were asked to share what was learned with colleagues from their respective sites.  Coaching visits: Facilities received visits from coaches with expertise in improvement methods and PEE. Coaches were Niger and Mali-based HCI improvement advisors and district supervisors from the Mali Ministry of Health. During these visits, coaches provided on-site support to improvement teams and helped team use data to identify problems and potential changes to address these problems. Although coaching visits are usually conducted monthly or bi-monthly in 4 • Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali HCI intervention programs, only four coaching visits were conducted in the intervention sites from February 2011 to June 2013 due to the suspension of HCI activities after the coup as well as issues related to logistics. Importantly, although these coaching visits included PEE, their focus was AMSTL and ENC.  Learning sessions: As part of a collaborative, teams from different facilities meet periodically to share their experience implementing changes to meet their shared goals to improve the quality of services. This peer-to-peer learning allows teams from different facilities to share what works and what did not and fosters healthy competition between the sites. Such meetings also allow successful changes to be spread to other sites. Although learning sessions were expected to occur every quarter, only one learning session was organized for facilities in the intervention district in April 2011. It is worth noting that that learning session focused on the EONC intervention but did include a segment on the PEE collaborative. Prior to the intervention, pre-eclampsia diagnosis were almost solely based on external signs such as feet edema, and women were not routinely screened for high blood pressure. The intervention placed emphasis on proper screening and documentation. The intervention also emphasized the importance of sites having equipment and medicine needed for pre-eclampsia screening and management. In most cases, community groups known as ASACOs that were linked to community health centers, provided financial support for equipment and medicine. Table 2 lists the planned and actual study activity dates. As mentioned above, the timeline of collaborative intervention and evaluation activities was disrupted by the political situation and ensuing halt of USAID activities in Mali. Table 2. Planned and actual timeline of intervention and study activities C. Sampling Facilities: Facilities participating in the intervention were those in Diéma District participating in the previous improvement collaborative that focused on EONC. These eight facilities include the district referral hospital and seven functional community health centers (CSCOMs). At the request of the Ministry of Health, only sites with at least one medical doctor could be part of the collaborative. This is because in Mali, only physicians are allowed to administer magnesium sulfate which is needed for the prevention and management of eclampsia. Eight control facilities (district referral hospital and six CSCOMs) were selected from the health district within the Kayes Region that was most similar to Activity Planned Dates Actual Dates Coaching visits in intervention district Monthly February 2011 July-August 2011 February-March 2012 Learning session in intervention district Quarterly April 2011 Data collection in both districts May 2011 May 2011 Data collection 2 in both districts August 2011 October 2011 Data collection 3 in both districts November 2011 March 2012 SUSPENSION OF HCI FIELD ACTIVITIES FROM MARCH TO AUGUST 2012 Data collection 4 in both districts February 2012 June 2013 Data collection 5 in both districts May 2012 Did not occur Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali • 5 Diéma in terms of location (i.e., rural) and socio-economic status of the population covered. Within the control district, Yélimané, all facilities with at least one physician were selected as control sites. Chart review: The charts and records of women receiving antenatal care or delivering during the data collection month were reviewed. In cases where there were not enough records in the current month to meet the desired sample size, records from the previous month(s) were also included. This was often the case for CSCOMs. The unit of analysis was the individual mother receiving care (i.e., the chart of the mother receiving care). Sample size was calculated based on an alpha of 0.05, power of 80% to detect a difference of 20% in the indicators between the intervention and control groups and accounting for a design effect for clustering by facility. The expected sample size was approximately 210 clinical records per group for a total of 420 clinical records for each time period. Systematic random sampling was used to select the same number of medical records from each site. The required sample size was met or exceeded during each data collection for both groups except at end line for the control group (the sample was 195 instead of the required 210 records). Observations: All health workers who provided services for antenatal care or delivery care on the day of baseline or end line data collection and who provided consent and whose patient provided consent were observed while providing services. Self-administered questionnaires: All health workers who provided antenatal care or delivery care service and who were present on the day of baseline or end line data collection were included if they consented to participate in the evaluation. D. Data Collection PEE Care Data Data on PEE care were collected using the following methods:  Medical chart reviews: Patients’ charts were reviewed and adherence to PEE standards of care was assessed using a checklist. Adherence scores were given separately for: 1) pre-eclampsia screening/diagnostic; 2) surveillance (referral hospitals only); 3) laboratory exams (referral hospitals only); 4) treatment; 5) obstetrical treatment (referral hospitals only); and 6) total adherence. Medical charts were reviewed during each of the four data collections. Information was also collected on available patients’ characteristics, such as age and parity.  Observations: In order to validate the information obtained from chart reviews, observers assessed the extent to which health providers adhered to standards of care for PEE. Findings from observations were recorded on a checklist. Observations included both the process used in conducting examinations and specific procedures as well as the content of the information conveyed to the patient (i.e., advice and signs to watch for). A maximum of five clients were observed for each provider. Observations were conducted at baseline and end line  Self-administered questionnaires for providers: Self-administered questionnaires included questions related to providers’ characteristics and training as well as questions on screening and management of PEE. A PEE knowledge score was given based on the number of questions answered correctly by the provider. Questionnaires were administered at baseline and end line. Each data collection team included a physician and an experienced midwife. Teams visited each site to review medical charts, conduct direct observations, and administer questionnaires. Data collectors were trained to ensure proper use of data collection tools and respect ethical considerations. Data collection tools were pretested prior to study implementation. 6 • Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali Cost Data The cost of implementation of the collaborative improvement intervention was collected from the accounting records of the HCI Project and entered into the collaborative costs data sheet for coaching and learning sessions. Costs were considered from the perspective of the implementer of the intervention; therefore, we did not include the opportunity costs to patients and their caregivers or others. The time horizon considered was the length of the intervention. Incremental clinical costs due to the implementation of the intervention were collected by the HCI team consulting with staff at the facility. Any expenses incurred directly as a result of the intervention were included. Intervention￾related costs did not include those associated with standard clinical practice such as magnesium sulfate. The cost of job aids (e.g., reminder posters) was included if it resulted from changes implemented because of the intervention. The incremental costs was divided by the number of deliveries and women receiving ANC visits in the facilities during the course of the intervention. Costs for clinical training were collected from accounting records and entered into the clinical training costs data sheet. All costs were collected in 2012 CFA francs. Given that most costs occurred within a year of the period of the intervention, discounting was not applied in the model. Ethical Considerations Data collectors who reviewed medical charts did not interact with patients and did not collect any identifying data during the chart reviews. Data from medical charts, observations, and questionnaires were assigned a random number. The clinical information collected from the medical charts was part of routine care and was not collected solely for the purpose of research. Women who were observed while receiving medical care were given information regarding the study and asked to provide informed consent. Providers observed and those who took the self-administered questionnaire also signed an informed consent form. The study was approved by the National Ethics Committee of Mali and the University Research Co., LLC (URC) Institutional Review Board. E. Analysis PEE Care Data Data were analyzed using STATA (version 11.1, StataCorp). Hierarchal regression models with differences-in-differences analyses were used to adjust for clustering of observations within sites and baseline differences in terms of adherence to pre-eclampsia/eclampsia screening and management standards. The differences-in-differences (DD) approach adjusts for differences between the intervention and control groups and measures the differences in trends in terms of the dependent variables of interest over time. Potential confounders such as woman’s age and parity were also controlled for in the regression models. Linear regressions were obtained with adherence to standards as the dependent variable. Independent variables were: group (intervention or control), time period (month 1, month 6, month 11, and month 24), and the interaction between group and period. The coefficient for this interaction and its 95% confidence interval represents the difference in the change over time in the intervention group compared to the change over time in the control group. Differences between intervention and control groups in terms of dependent variables such as adherence to norms were assessed. Logistic regression models and corresponding odds ratios (OR) and 95% confidence intervals (CI) were also obtained to serve as inputs for the cost-effectiveness analysis. Cost-effectiveness Data Decision tree analysis was used to determine the incremental cost-effectiveness of the collaborative improvement intervention. Inputs for the effectiveness of the intervention were derived from the results obtained from the analysis of data from this study described above. Monte Carlo simulations were used to determine the confidence intervals around the point estimates. Analyses were conducted for two outcomes: changes in adherence to screening standards and overall adherence to PEE management Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali • 7 standards. Given that the non-collaborative improvement intervention sites experienced significant changes in these outcomes during the same period, the results reflect the difference not from baseline but from these substantially improved non-collaborative intervention sites. For the program effectiveness inputs, odds ratios from regression analyses controlling for potential confounders were converted into probabilities with the appropriate distributions applied to account for the uncertainty of these inputs. III. RESULTS A. Characteristics of the Study Sample Data were collected at four different times between May 2011 and June 2013. A total of 1756 charts were reviewed (Table 3) in both districts; 893 in the intervention district and 863 in the control district. Table 3. Number of charts reviewed in health centers in the intervention and control districts Baseline (Month 1) Data collection 2 (Month 6) Data collection 3 (Month 11) End-line (Month 24) Total Intervention 235 233 230 195 893 Control 222 217 214 210 863 Total 457 450 444 405 1756 Patients: Table 4 shows characteristics of patients whose charts were reviewed for each data collection point. The mean age of patients was 24.6 in the intervention district and 24.1 in the control district. On average, patients had 2.8 children in both districts. The majority of charts reviewed in the intervention and the control districts were from community health centers. Overall, the districts did not differ in terms of patient characteristics, such as age and parity, or regarding types of facility where they sought health services, with the exception of parity for the third data collection and age at end line data collection (p<0.05). Table 4. Characteristics of patients in the intervention and control districts Baseline Data collection 2 Data collection 3 End line Interv. Control p-value Interv. Control p-value Interv. Control p-value Interv. Control p-value Mean age of patients 24.4 23.9 0.415 23.6 24.5 0.134 24.2 24.02 0.733 24.1 26.7 0.008 Mean parity 2.8 2.8 0.909 2.5 2.7 0.375 3.0 2.5 0.0356 3.0 3.3 0.354 Type of facility (%) - CSCOM - Referral hospital 81.9 18.0 86.4 13.6 0.202 80.7 19.4 87.6 12.5 0.052 82.2 178. 88.3 11.7 0.081 83.3 16.7 89.7 10.3 0.081 Providers: Table 5 shows the characteristics of providers present at the intervention and control sites during baseline and end line data collection. The majority of providers were from community health facilities and had been working at the facility, on average, for 10 years in intervention facilities and six years in control facilities. Most providers were medical doctors or matrones (auxiliary midwives with primary school education who are trained for nine months). Providers in the intervention group were more likely to have received trainings on eclampsia/pre-eclampsia than those in the control group (p<0.05). 8 • Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali Table 5. Key characteristics of providers in intervention and control sites at baseline and end-line BASELINE END-LINE Characteristic Intervention Control p value Intervention Control p value Type of facility Community health center (CSCOM) 28 9 0.318 18 21 0.777 Referral health center (CSREF) 9 6 7 10 Provider's sex Male 24 8 0.534 17 22 0.760 Female 13 7 7 7 Type of provider Medical doctor 10 6 0.363 6 4 0.720 Midwife 6 2 3 4 Obstetric nurse 5 4 4 8 Graduate nurse 0 0 0 2 First cycle nurse 2 1 2 3 Matrone 14 2 9 10 Other 0 0 1 0 Mean duration in current function (years) 10.19 6.20 0.228 10.19 6.23 0.061 Mean duration at current facility (years) 5.55 3.59 0.375 7.51 5.69 0.351 Training Pre-eclampsia/ eclampsia training 20 0 <0.001 33 11 0.014 EONC training 10 4 1.000 20 6 0.064 AMTSL training 28 2 <0.001 48 17 0.001 Essential newborn care training 31 10 0.260 53 26 0.027 Postnatal care counseling training 13 6 0.760 28 21 0.544 In an attempt to assess providers’ ability to diagnose and manage PEE cases, providers present at each facility during baseline and end line data collection were asked to answer questions regarding pre￾eclampsia screening and management as well as eclampsia management. Table 6 shows mean knowledge scores for pre-eclampsia/eclampsia screening and management. At baseline, PEE knowledge was low overall, and there was no difference in knowledge between providers in control facilities and those in intervention facilities. Knowledge was lower among women, matrones, and obstetric nurses. Knowledge was higher among providers at referral hospitals, doctors, midwifes, and male providers (p<0.05). However, at end line, providers in the intervention district had significantly higher PEE knowledge than those in the control districts. Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali • 9 In an attempt to assess providers’ ability to diagnose and manage PEE cases, providers present during baseline and end-line data collection were asked to answer questions regarding pre-eclampsia screening and management as well as eclampsia management. Table 6 shows mean knowledge scores for pre￾eclampsia/eclampsia screening and management. At baseline, PEE knowledge was low overall and there was no difference in knowledge between providers in control facilities and those in intervention facilities. Knowledge was lower among women, matrones, and obstetrics nurses. Knowledge was higher among providers at referral hospitals, doctors and midwifes and male providers (p<0.05). However, at end-line, providers in the intervention district had significantly higher PEE knowledge than those in the control districts. Table 6. Mean PEE knowledge score among providers by key provider characteristics BASELINE END LINE Mean p value Mean p value Intervention Group Control 28.4 0.318 54.8 <0.001 Intervention 33.6 73.1 Type of Service CSCOM 28.7 0.021 61.5 0.440 CSREF 40.4 66.2 Profile Doctor 43.8 <0.001 78.4 0.067 Midwife 42.8 68.4 Obstetric nurse 26.9 63.2 Graduate nurse __ 36.8 First cycle nurse 40.4 55.8 Matrone 16.4 58.2 Other __ 47.4 Gender Male 25.2 <0.001 62.2 0.844 Female 43.2 63.5 PEE Training No Training 27.5 0.938 51.6 0.045 Any Training 35.8 72.0 B. Adherence to Standards for Pre-eclampsia and Eclampsia Care (Unadjusted Analyses) Cases of Pre-eclampsia and Eclampsia Based on Chart Review Table 7 shows the number of pre-eclampsia and eclampsia cases diagnosed based on chart reviews. Only 32 pre-eclampsia and 20 eclampsia cases were detected during the evaluation. The largest number of cases was detected at end line in the intervention district (13 pre-eclampsia cases and 10 eclampsia cases). 10 • Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali Table 7. Number of cases of pre-eclampsia and eclampsia detected by providers based on chart reviews Baseline Data collection 2 Data collection 3 End line Total Inter. Control Inter. Control Inter. Control Inter. Control Pre￾eclampsia 6 1 3 1 1 5 13 2 32 Eclampsia 3 2 1 2 1 1 10 0 20 Adherence Based on Chart Review Table 8 lists key indicators for PEE care adherence for the intervention and control districts for each data collection point. Data on adherence to laboratory standards were missing for the first and third data collections, and data for adherence to surveillance were also missing for the third data collection. The data show that the vast majority of women were not screened properly for pre-eclampsia. Baseline screening adherence are only 0.96 and 1.02 (out of 6), in the intervention and control group, respectively. Although these scores increase over time, even at end line, screening for pre-eclampsia is not optimal (3.36 and 2.79 the intervention and control group, respectively). At baseline, scores for adherence to surveillance standards for pre-eclampsia/eclampsia and adherence to treatment norms were higher in the intervention group compared to the control group. However, overall adherence to PEE care standards was higher in the control group compared to the intervention group (p<0.05). Differences between the intervention and control districts were significant for most indicators for subsequent data collections. Surprisingly, some unadjusted scores remained higher in the control group compared to the intervention group during the second and third assessment. However, by end line assessment, three out of the six adherence scores, including overall adherence, were significantly higher in the intervention district compared to the control district (p<0.05). Adherence to treatment standards was higher in the control facilities. Adherence Based on Observations Observations were conducted at baseline and end line to estimate adherence to standards for pre￾eclampsia screening. It is worth noting that observations were conducted on the day of data collection while chart reviews included visits from the current month as well as previous month in some cases. Although we attempted to observe management of pre-eclampsia and eclampsia cases, too few cases were observed to calculate these indicators. Table 9 shows the comparison of adherence to pre-eclampsia standards of care as estimated from chart reviews and observations. Baseline adherence scores obtained from observations were found to be significantly higher in the intervention district compared to the baseline district (p<0.001). Baseline scores based on observations appears higher than the mean score obtained from chart review in the intervention facilities (0.96 versus 1.20) and lower than the score obtained from chart reviews in the control districts (0.57 versus 1.02). At end line, the scores in both districts appeared slightly higher based on observation than chart review. Data from observations confirmed that end line scores were higher in the intervention than the control group (4.20 versus 3.12; p<0.001). End line scores obtained from observations appeared higher than those computed from chart reviews. However, as shown in Table 9, data from observations were based on a smaller sample than data from chart review. Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali • 11 Table 8. Unadjusted changes to standards of pre-eclampsia/eclampsia care by study arm and data collection time based on charts review Baseline Data collection 2 Data collection 3 End-line I C p value I C p value I C p value I C p value Adherence to screening standards 0.96 1.02 0.186 1.01 1.44 <0.001 1.60 2.25 <0.001 3.36 2.79 <0.001 Adherence to surveillance standards 3.29 1.0 0.046 3.75 0.13 <0.001 4.15 2.00 <0.001 Adherence to laboratory standards 0.5 0.13 0.188 1.00 1.03 0.914 Adherence to obstetrical management standards 1.29 0.60 0.116 1.25 0 <0.001 1.50 1.40 0.895 1.60 2.00 <0.001 Adherence to treatment standards 2.20 0.53 0.002 1.5 0.13 0.0383 0.60 1.67 0.069 1.32 1.67 <0.001 Total adherence (%) 19.59 21.59 0.0410 21.10 30.27 <0.001 32.11 45.40 <0.001 57.58 47.93 <0.001 I=Intervention; C=Control. 12 • Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali Table 9. Adherence to pre-eclampsia screening standards estimated by chart review and observations BASELINE END-LINE Intervention Control p value Intervention Control p value Chart review Mean score Number 0.96 235 1.02 222 0.186 3.36 195 2.79 210 <0.001 Observation Mean score Number 1.20 93 0.57 67 <0.001 4.20 70 3.12 58 <0.001 C. Regression Analyses Hierarchical regressions were conducted with differences-in-differences analyses to adjust for clustering of observations within sites and baseline differences in terms of adherence to pre-eclampsia/eclampsia screening and management norms. Given the low number of pre-eclampsia and eclampsia cases detected (Table 7), regression analyses focused on the following two key indicators: 1. Adherence to pre-eclampsia screening standards 2. Total adherence to pre-eclampsia/eclampsia screening and management standards. Table 10 shows results for the linear regression and logistic regression model for adherence to pre￾eclampsia screening standards, while Table 11 shows results for the linear regression and logistic regression model for total adherence to pre-eclampsia/eclampsia screening and management standards. The coefficients and corresponding p values are those for the linear models with the outcome as a continuous variable. The OR and 95% CI are for the logistic regression model with the dependent variable dichotomized based on its 75th percentile value. Logistic regression models were obtained to provide inputs for the cost-effectiveness model. All regression models controlled for age, parity and clustering by site. Regression analyses show a very modest effect of the intervention on adherence to pre-eclampsia screening standards and overall adherence to pre-eclampsia/screening and management standards. The statistically significant interaction terms reflects that overall, the intervention group is doing better by 0.02 points for adherence to screening and 0.38 for overall adherence per month ( p<0.001). In addition, the logistic regression models show that the intervention group was associated with 7% higher odds of scoring in the 75th percentile or above for overall adherence to PEE standards ( p=0.035). The intervention was also associated with 6% higher odds of scoring in the 75th percentile of above for screening standards. However, this OR was only marginally significant (p=0.05). Table 10. Adherence to pre-eclampsia screening standards Linear Regression Model Logistic Regression Model Coefficient 95% CI OR 95% CI Time 0.066 0.060; 0.073 1.23 1.19-1.27 Group -0.043 -0.825; -0.0472 0.14 0.17-1.15 Interaction Time x Group 0.0198 0.0103-0.0293 1.06 1.00-1.13 Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali • 13 Table 11. Total adherence to pre-eclampsia/eclampsia screening and treatment standards Linear Regression Model Logistic Regression Model Coefficient 95% CI Odds ratio (OR) 95% CI Time 0.960 0.834-1.086 1.22 1.19-1.27 Group -9.12 -15.00; -3.26 0.15 0.016-1.52 Interaction Time X Group 0.382 0.207-0.558 1.07 1.00-1.13 Based on the linear models, values of the two indicators and the difference associated with the intervention between baseline and end line data collected were calculated (Table 12). Differences in differences analyses shows that between baseline (month 1) and data collection 4 (month 24), the differences in scores for screening adherence and overall adherence attributable to the intervention are 0.46 and 8.80 points, respectively. Table 12. Changes in adherence for the intervention and control group INTERVENTION CONTROL Diff-in-Diff Adjusted scores Baseline End-line Diff Baseline End-line Diff Adherence to screening standards 0.86 2.84 1.98 1.28 2.81 1.53 0.46 Total adherence 19.01 49.90 30.89 27.75 49.84 22.09 8.80 D. Cost-effectiveness Analysis We conducted decision-tree analysis to estimate the incremental cost-effectiveness of the collaborative improvement intervention in the facilities in which it was conducted, on improving compliance to PEE standards of screening and management compared to the non-intervention sites. Inputs used for the effectiveness were the attributable changes as odds ratios from the logistic regressions shown in Tables 10 and 11. The program cost data inputs were obtained from the implementers’ accounting records and totaled 18,700,000 CFA francs. The breakdown of these costs is shown in Figure 2. A total of 2530 women received services in the participating facilities during the period of the intervention. The cost of the intervention per woman receiving services was 7,390 CFA francs per delivery. Estimates for incremental cost effectiveness are 524,000 CFA per additional childbirth patient screened for PEE according to standards and 453,000 CFA per additional childbirth patient managed to overall adherence to PEE standards. The 95% confidence intervals derived from Monte Carlo simulations crossed into negative numbers by approximately 5%, indicating a high likelihood that the result is in the upper right quadrant of the cost-effectiveness plane, meaning the determination of whether or not the intervention is cost-effective depends on the willingness to pay of the program funders (Table 13). 14 • Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali Figure 2. Components of intervention cost Salaries & Per diems, 60 Transport, 24 Supplies, 5 Other travel expenses, 10 Room rental, 1 Table 13. Incremental cost-effectiveness ratio results in CFA francs Point estimate Lower 95% CI Upper 95% CI Adherence to screening standards 524000 -28000 1090000 Overall adherence to standards 453000 -18000 924000 IV. DISCUSSION This evaluation shows a very modest potential effect of a collaborative improvement intervention on adherence to pre-eclampsia screening standards and on overall adherence to pre-eclampsia/ eclampsia screening and management standards. It is worth noting that the control group received additional inputs from the Ministry of Health. Findings show that the intervention group is doing better by 0.02 points for adherence to screening and 0.38 for overall adherence per month (p<0.001) and is associated with 6 to 7% higher odds of scoring in the 75th percentile or above for pre-eclampsia/eclampsia standards of care. The differences in scores for adherence to pre-eclampsia screening and overall adherence to PEE care attributable to the intervention between baseline and end line are 0.46 and 8.8 points, respectively. Although the effect detected is modest, this evaluation used a controlled study design and contributes to the much-needed evidence base for the effectiveness of improvement collaboratives (Franco & Marquez 2011, Broughton et al. 2013). The relatively weak effect of the intervention might be explained by several factors. First, the screening and management of PEE is a complex process that requires technical expertise as well as the availability and correct use of equipment for assessment of blood pressure and proteinuria and medicine such as magnesium sulfate, which are not always available in resource-limited settings. Secondly, the way the intervention was implemented may have diluted the effect. Only one learning session was held in the intervention district during a two-year period, and although this session included PEE activities, its focus was EONC (specifically AMTSL and ENC). Learning sessions provide opportunities to promote shared learning and rapid disseminations of best practices to facilities participating in an improvement Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali • 15 collaborative and are usually held on a quarterly basis over the life of the collaborative. In addition, while coaching visits for HCI-assisted program are usually conducted on a monthly basis, only four coaching visits were conducted during the evaluation period, and due to logistics, coaching visits were only conducted in sites where performance problems had been identified. Furthermore, although these coaching visits included PEE activities, they focused on EONC. The fact that only one learning session and four coaching visits occurred during a two-year period and which did not focus on PEE may have led to the weak intervention effect observed. Third, due to the political situation, HCI’s activities were suspended in Mali from March to August 2012. The interruption of the intervention is likely to have contributed to the small intervention effect. Fourth, activities conducted in the control sites may have diluted the intervention effect. Because PEE was found by the Ministry of Health to be particularly high in the control district, the MOH conducted maternal and neonatal trainings and provided equipment in both districts during the evaluation, including one of the control sites. In addition, the district officer from the control district who was present at an HCI-organized meeting with participants from all districts in Kayes Region had expressed his commitment to focus on the screening and management of PEE, which was highest in his district. This along with relative proximity of the two districts may have led the cross-contamination of the control group and higher than expected performance in the control sites. We were unfortunately not able to document more specific information regarding activities implemented by the MOH in the control district. Finally, staff turnover may have played a role in the evaluation results, as the physicians trained in improvement methodology in two facilities in the intervention district, were known to have transferred to other facilities. The two facilities remained in the intervention group although the new providers had not received improvement training. Values for the indicator for adherence to screening standards measured in this evaluation are lower than those reported previously by HCI in Mali. This may be explained by the fact that during each data collection and at each site, data collectors for the evaluation reviewed charts for the current as well as previous months until they met the required sample size, whereas the project only reviews five charts each month at each site and only reviews charts of PEE cases. Restricting the chart reviews to PEE cases artificially inflates the value of adherence for screening standards, as PEE cases are more likely to have been appropriately screened. While we cannot assume that data collected in this evaluation are the gold standard for this indicator, this difference highlights the need to focus on the quality of data collected by the project. The intervention should aim to review a number of charts that are more likely to be representative of that month’s performance and take a sample from all pregnant and delivering women receiving antenatal care that month. The economic analysis result indicates that for the cost of approximately $1,000 per additional mother screened and managed to compliance with standards which is about the same as the annual GDP per capita of Mali. If adherence with standards was equated to the saving of one disability-adjusted life year (DALY), then this program would be considered by the WHO and World Bank as cost-effective compared to business-as-usual (WHO-CHOICE). However, we could find no literature linking this level of compliance with clinical standards to specific improvements in DALYs averted. The 95% confidence interval for this estimate indicates this is not a robust result, and there is a small possibility that the strategy without the improvement intervention may be as or more cost-effective than the strategy with the improvement intervention. This suggests that the investments in this level of inputs aimed at improving health system performance but spread over a longer period of time due to conflict in the region may not yield acceptably efficient results. Other studies have shown that improvement interventions involving more intensive activities for supporting personnel in the targeted facilities can be successful, including studies done in the region on programs for improving EONC in facilities (Broughton et al. 2012; Broughton et al. 2013a). More research is needed to determine if there is a threshold for investment in activities aimed at improving health system performance in maternal care and more generally. 16 • Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali A. Limitations These findings should be interpreted in light of the study’s limitations. The original study design was altered due to the interruption of HCI activities as well issues related to logistics. In addition, most of the data were collected from existing data sources (patient records). This evaluation revealed that even essential socio-demographic characteristics such as age and parity were missing in some records. Additional confounders that could have had an effect on the indicators measured, such as history of diabetes or high blood pressure, were missing from almost all records. Furthermore, this evaluation is assuming that the information entered in patients’ records is accurate. However, studies have shown that the quality of records at maternity hospitals in similar settings is questionable (Broughton et al. 2013b; Hermida et al. 2011; Ndira et al. 2008). Another limitation is that it was not possible to randomly assign facilities within each district to the intervention or control group without increasing the likelihood of cross-contamination. To adjust this, an effort was made to use differences-in-differences analysis to adjust for baseline differences. V. CONCLUSION AND RECOMMENDATIONS This evaluation contributes to evidence regarding the effectiveness of collaborative improvement by demonstrating a modest effect of a collaborative improvement intervention for improving the quality of pre-eclampsia and eclampsia care in Mali using a controlled study design. Although this evaluation found that the intervention is only associated with a 7% increase in the odds of adherence to standards for PEE care, findings reflect the effect of a collaborative in a “real world setting”. One would expect the intervention to have an even larger effect without deviations from the initial study design and cross￾contamination and with more frequent learning sessions and coaching visits as initially planned. Further research is need to assess the effectiveness of the improvement collaborative approach with more certainty. Future controlled evaluations should ensure that the intervention is implemented in an optimal manner. In addition, HCI’s comparative evaluations have thus far focused on maternal and child care; we recommend evaluating the effectiveness of collaboratives for a variety of clinical areas and investigating factors that promote or hinder the effectiveness of collaborative improvement interventions. The cost-effectiveness results indicate that it may not be worthwhile to implement an improvement intervention that is as attenuated as this one was. More evidence is required to determine where the threshold for improvement intervention activities is to achieve substantive improvements in health system performance. Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali • 17 REFERENCES Broughton E, Namajji C, Vaid S, Karamagi E, Byabagambi J. 2014. A comparative evaluation and cost￾effectiveness analysis of collaborative improvement for maternal and newborn care services in Uganda. Research and Evaluation Report. Published by the USAID health Care Improvement Project. Bethesda, MD: University Research Co., LLC (URC). Broughton E, Saley Z, Boucar M, Alagane D, Hill K, Marafa A, Asma Y, Sani K. 2013a. Cost-effectiveness of a quality improvement collaborative for obstetric and newborn care in Niger. Int J Health Care Qual Assur; 26(3):250-61. Broughton EI, Ikram AN, Sahak I. 2013b. How accurate are medical record data in Afghanistan's maternal health facilities? An observational validity study. BMJ Open; Apr 24;3(4). pii: e002554. doi: 10.1136/bmjopen-2013-002554. Broughton EI, López SR, Aguilar MN, Somarriba MM, Pérez M, Sánchez N. 2012. Economic analysis of a pediatric ventilator-associated pneumonia prevention initiative in Nicaragua. Int J Pediatr; Article ID 359430. Franco LM, Marquez L. 2011. Effectiveness of collaborative improvement: evidence from 27 applications in 12 less-developed and middle-income countries. BMJ Qual Saf; 20(8):658-665. Hermida J, Broughton EI, Franco LM. 2011. Validity of self-assessment in a quality improvement collaborative in Ecuador. Int J Qual Health Care; 23:690–6. Hulscher, M., Schouten, L., Grol, R., & Buchan, H. 2013. Determinants of success of quality improvement collaborative: what does the literature show? Qual Saf Health Care; 22:19-31. Ndira SP, Rosenberger KD, Wetter T. 2008. Assessment of data quality of and staff satisfaction with an electronic health record system in a developing country (Uganda): a qualitative and quantitative comparative study. Methods Inf Med; 47:489–98. Schouten L, Hulscher M, Everdingen J, Huijsman R, Grol R. 2008. Evidence for the impact of quality improvement collaborative: systematic review. BMJ Qual Saf; 336(7659);1491-1494. Unahalekhaka A, Jamulitrat S, Chongsuvivatwong V, Ovretveit J. 2007. Using a collaborative to reduce ventilator-associated pneumonia in Thailand. The Joint Commission on Quality and Patient Safety; 33:387– 394. Webster, PD, Sibanyoni M, Malekutu D, Mate KS, Venter WD, Barker PM, Moleko W. 2012. Using quality improvement to accelerate highly active antiretroviral treatment coverage in South Africa. BMJ Qual Saf; 21(4):315-324. WHO-CHOICE. Cost-effectiveness thresholds. http://www.who.int/choice/costs/CER_thresholds/en/. Youngleson MS, Knurunziza P, Jennings K, Arendse J, Mate KS, Barker P. 2010. Improving a mother to child HIV transmission programme through health system redesign: Quality improvement, protocol adjustment and resource addition. PLoS One; 5(11) :e13. 18 • Comparative evaluation of pre-eclampsia/eclampsia care improvement in Mali USAID HEALTH CARE IMPROVEMENT PROJECT University Research Co., LLC 7200 Wisconsin Avenue, Suite 600 Bethesda, MD 20814 Tel: (301) 654-8338 Fax: (301) 941-8427 www.hciproject.org