Photo Credit: John Nicholson, JSI EVALUATION REPORT USAID/INDIA VRIDDHI: SCALING UP INTERVENTIONS IN RMNCH+A MID-TERM PERFORMANCE EVALUATION June 2018 This publication was prepared independently by Dr. Beulah Jayakumar, Meredith Feenstra, Leah Ghoston, and Dr. Francis Raj of Social Impact, Inc. at the request of the United States Agency for International Development. Photo Credit: John Nicholson, JSI USAID/INDIA VRIDDHI: SCALING UP INTERVENTIONS IN RMNCH+A MID-TERM PERFORMANCE EVALUATION EVALUATION REPORT June 2018 Submitted to: Chandan Samal, Contracting Office’s Representative USAID/India Program Support Office AID-486-I-14-00001 (ALME IDIQ) / 72038618F00002 (Task Order) DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. i ABSTRACT The United States Agency for International Development (USAID)/India contracted Social Impact, Inc. to conduct a mid-term performance evaluation of Vriddhi: Scaling Up Interventions in Reproductive, Maternal, Neonatal, Child, and Adolescent Health (RMNCH+A). The Vriddhi project provided technical and managerial assistance to the state and national governments of India focusing on improving quality of RMNCH+A services within healthcare facilities. This evaluation examined the project’s performance and effectiveness and provide insights and lessons learned to inform USAID/India’s future RMNCH+A programming. The evaluation collected data through an exhaustive document review, analysis of secondary quantitative data, and 88 key informant interviews with 114 total respondents. Among the many project components, the project dedicated the largest share of its time and resources to improving the quality of care during the intra-partum and immediate post-partum period. The “Care around Birth” approach was designed to reduce preventable maternal and infant mortality and effectively built the capacity of health providers to deliver high-impact obstetric and newborn interventions. However, the project faced numerous challenges, including perennial deficiencies in human resources and infrastructure, which limited its overall impact. The project’s supply-side mandate and its primary focus on maternal health also limited its scope and resulted in missed opportunities for demand generation and an inability to address issues of access to healthcare services. This evaluation report also provides recommendations to USAID and implementing partners for the remainder of the project and future work in the RMNCH+A sector. ii ACKNOWLEDGMENTS The Evaluation Team extends its gratitude to many individuals and organizations for their support of this evaluation and report. This evaluation would not have been possible without the support of the IPE Global and JSI staff, particularly Gunjan Taneja and Rajeev Gera. We are appreciative of their openness to learning and their dedication to the success of this evaluation. We are very grateful to the USAID/India Mission, especially Chandan Samal and Sachin Gupta, for their insights and guidance. We thank Ankita Rawat for her logistical assistance during in-country data collection. Finally, we appreciate the support of the Social Impact headquarters team, Lilly Liu, Adam Reisman, and Charity Whitehead. iii TABLE OF CONTENTS ABSTRACT I ACKNOWLEDGMENTS II EXECUTIVE SUMMARY VI I. INTRODUCTION & PROJECT BACKGROUND 1 II. EVALUATION PURPOSE & QUESTIONS 2 III. EVALUATION METHODS & LIMITATIONS 2 DATA COLLECTION METHODS 2 DATA ANALYSIS 3 LIMITATIONS 4 IV. FINDINGS & CONCLUSIONS 4 EQ1 FINDINGS 4 EQ1 CONCLUSIONS 10 EQ1.I FINDINGS 11 EQ1.I CONCLUSIONS 11 EQ1.II FINDINGS 12 EQ1.II CONCLUSIONS 13 EQ1.III FINDINGS 13 EQ1.III CONCLUSIONS 15 EQ2 FINDINGS 15 EQ2 CONCLUSIONS 17 EQ2.I FINDINGS 18 EQ2.I CONCLUSIONS 20 EQ2.II FINDINGS 20 EQ2.II CONCLUSIONS 21 EQ3 FINDINGS AND CONCLUSIONS 21 EQ4 FINDINGS AND CONCLUSIONS 23 V. RECOMMENDATIONS 24 ANNEX I. EVALUATION STATEMENT OF WORK 26 ANNEX II. LIST OF RESPONDENTS 33 ANNEX III. DATA COLLECTION INSTRUMENTS 37 ANNEX IV. CAB DATA VALIDATION 46 ANNEX V. ENDNOTES 47 iv ACRONYMS AFHC Adolescent-Friendly Health Clinics AH Adolescent Health AHD Adolescent Health Days AMTSL ANC ANM ARSH ASHA BLISS Active Management of the Third Stage of Labor Antenatal Care Auxiliary Nurse Midwife Adolescent Reproductive and Sexual Health Accredited Social Health Activist Block level Implementation Support Strengthening CaB DP Care around Birth Development Partner ECP Emergency Contraception EmOC Emergency Obstetric Care ET Evaluation Team EQ Evaluation Questions ESB Ensuring Spacing at Birth ESW Experience Sharing Workshops FOGSI Federation of Obstetric and Gynecological Societies of India FP Family Planning FRU First Referral Units GoI Government of India HBNC Homebased Newborn Care HDC Home Delivery of Contraceptives HPD High Priority District HR Human Resources HRP HMIS IEC IFA IP High-Risk Pregnancy Health Management Information System Information, Education and Communication Iron and Folic Acid Implementing Partner IMA Indian Medical Association IUCD Intrauterine Contraceptive Device JSI John Snow, Inc. JSSK Janani Shishu Suraksha Karyakaram JSY Janani Suraksha Yojana KII Key Informant Interview KMC Kangaroo Mother Care LaQshya Labor Room Quality Improvement Initiative LBW Low Birthweight MCH MCTS Maternal and Child Health Mother and Child Tracking System MIS Management Information System v MNCH Maternal, Newborn and Child Health MOHFW Ministry of Health and Family Welfare NAP Newborn Action Plan NRU National RMNCH+A Unit ORS Oral Rehydration Salts PE Performance Evaluation PIP Program Implementation Plans PMSMA Pradhan Mantri Surakshit Matritva Abhiyan PMU Program Management Unit PPFP Postpartum Family Planning PPIUCD Postpartum Intrauterine Contraceptive Devices PSE PTK QI Private Sector Engagement Pregnancy Testing Kits Quality Improvements RBSK Rashtriya Bal Swasthya RKSK Rashtriya Kishor Swasthya Karyakram RMC Respectful Maternity Care RMNCH+A Reproductive, Maternal, Neonatal, Child, and Adolescent Health SAMMAN Safe Motherhood and Newborn Health SI Social Impact SNCU Special Newborn Care Unit SOP Standard Operating Procedure SRH Sexual and Reproductive Health SRU State RMNCH+A Units SS Supportive Supervision TIP Technical Intervention Packages UHND Urban Health and Nutrition Days UNICEF United Nations Children’s Fund USAID United States Agency for International Development USD United States Dollars WHO World Health Organization WIFS Weekly Iron Folic Acid Supplementation vi EXECUTIVE SUMMARY PROJECT OVERVIEW Scaling Up Interventions in Reproductive, Maternal, Neonatal, Child and Adolescent Health (RMNCH+A), or Vriddhi, is a $25 million technical assistance project of the United States Agency for International Development (USAID) to the RMNCH+A Strategy of the Government of India (GoI). Implemented by IPE Global and John Snow Inc. (JSI) from 2014–2020, Vriddhi works with national, state, and district governments to reduce preventable maternal, neonatal, and child mortality in 26 high priority districts (HPDs) across six states where USAID is the lead development partner (DP). Four strategic outputs support the achievement of this objective: i. Improve availability and quality of RMNCH+A services in government health facilities ii. Strengthen evidence for RMNCH+A services iii. Incubate RMNCH+A good practices for scale-up iv. Involve multiple stakeholders in delivery of RMNCH+A services EVALUATION PURPOSE & METHODS USAID/India contracted Social Impact, Inc. (SI) to conduct a mid-term performance evaluation of Vriddhi from March to May 2018 to 1) assess Vriddhi’s results and overall progress against objectives, 2) evaluate its key strategies and approaches for scale-up and develop recommendations for the remainder of the project, and 3) provide insights from Vriddhi to inform USAID/India’s future programming. The evaluation team (ET) used a mixed-methods approach, reviewing project documents, RMNCH+A literature, and secondary quantitative data; and conducting 88 key informant interviews with national, state, and district government officials, implementing partners (IPs), and healthcare providers from the Centre and ten purposively sampled HPDs in the six states. While the evaluation was designed to explore the project in￾depth, the sample is not representative of the population and the evaluation is unable to directly attribute impact to the project. The ET addressed possible selection bias by randomly selecting facilities and providers where possible, and minimized recall bias and the potential conflation of Vriddhi-sponsored activities with others by appropriately focusing the lines of inquiry. FINDINGS AND CONCLUSIONS BY EVALUATION QUESTION (EQ) This section presents high-level findings and conclusions by evaluation question. The body of the report that follows explores each question in greater detail. EQ1: To what extent has the project achieved the objectives and expected results in terms of enhanced institutional capacity of GoI and private sector networks to scale up proven high-impact RMNCH+A interventions? Key informants perceived Vriddhi’s training and support for its Care around Birth (CaB) package to be of high quality, leading to successful delivery of high-impact intra-partum and postpartum interventions with respectful maternity care (RMC) through confident and competent providers in 141 high caseload facilities across the 26 HPDs. Facility mentorship and quality improvement (QI) teams raise the prospect of continued delivery of these services, but lack of staff and infrastructure limited CaB delivery and building managerial competencies. Facilities delayed implementation of Kangaroo Mother Care (KMC) or did not offer it to all eligible newborns because they misunderstood that the practice required equipment and specialized care. Vriddhi trained CaB providers in postpartum family planning. Vriddhi supported GoI initiatives of adolescent-friendly health clinics (AFHCs) and weekly iron folic acid supplementation (WIFS), leading to improved uptake of these services, but there were no activities to identify and address sexual and reproductive health needs of adolescents, particularly those of boys. Thus, Vriddhi built the institutional capacity in terms of technical competencies of health providers to deliver high-impact interventions for maternal and newborn health but faced limitations due to human resource (HR) and vii time constraints. Interventions for other aspects of RMNCH+A were proportionately smaller in scope than the CaB initiative. EQ1.i: To what extent has the program been successful in influencing various GoI efforts on improving RMNCH+A, specifically through the support of the National RMNCH+A Unit (NRU)? The NRU successfully rolled out the Supportive Supervision (SS) mechanism and coordinated the analysis and use of SS data, thus unifying DPs around the RMNCH+A Strategy. Recently, the NRU also developed the RMNCH+A App and Health Atlas. Intentional transfer of SS-related responsibilities (from Vriddhi staff to government officials) has not occurred, mainly due to time constraints of state and district health managers. EQ1.ii: To what extent has the program engaged private providers or stakeholders in delivering RMNCH+A services, and what are the effectiveness and results of this engagement? Vriddhi’s private sector engagement (PSE), led by JSI, underwent multiple changes over time. The project conducted a landscape assessment of private providers and held workshops at national and state levels to orient private providers on high-impact interventions and national guidelines. Vriddhi has not had sufficient time to utilize the landscape assessment and build relationships between GoI and private providers. In conclusion, Vriddhi has not been successful in engaging private stakeholders in delivering RMNCH+A services. EQ1.iii: To what extent have the results of this project contributed to the scale-up of effective RMNCH+A solutions to achieve significant reductions in preventable morbidity and mortality in women and children? Scale-up of CaB was led by the respective states, with plans to cover up to 90 percent of births in the six states. Vriddhi supported the scale-up plans, budgets, and monitoring systems. CaB has also informed the GoI Labor Room Quality Improvement (LaQshya) Initiative and Vriddhi has set up its project management unit. There is no evidence to support the inference that improved quality of services will lead to reductions in mortality and morbidity without concomitant efforts to reach the most vulnerable who are furthest from care. Thus, key interventions of Vriddhi have been significantly scaled up, but access barriers need to be addressed to impact mortality. EQ2: How actively has the project scaled up the approaches/models/interventions from the five￾by-five matrix used by the GoI in the national program? In addition to its core interventions drawn from the 5x5 matrix, Vriddhi also pilot-tested the operational feasibility of two models that address key causes of maternal and newborn mortality and supported the strengthening of others that improve availability and demand. However, the pilots require more careful documentation to enable successful scale-up. Thus, Vriddhi scaled up interventions from the 5x5 matrix that complemented its focus on addressing mortality. EQ2.i: What have been the results of the project’s approaches on increased off-take of the RMNCH+A services in High Priority Districts? Vriddhi’s express outcome of interest was improving quality of facility-based services, which some stakeholders assume will lead to increased uptake of services. However, evidence from literature and from Vriddhi’s experience in strengthening availability show that a range of demand- and supply-side barriers need to be addressed, in order to increase the uptake of services. Thus, the successful improvements that Vriddhi brought about in the quality of care are unlikely to have a direct impact on the uptake of these services, without simultaneous efforts to address barriers related to availability and accessibility. EQ2.ii: What have been the results of the project’s approaches on technical assistance for improved program planning, monitoring, and implementing at the district and state levels? viii Vriddhi supported states and districts in analyzing gaps and drafting and implementing responsive plans. Key informants from states and district governments reported appreciation for Vriddhi’s input, but none see it as a strategic opportunity to develop their own capacities to sustain managerial competencies after Vriddhi’s withdrawal. They also lack time for ongoing monitoring of project implementation to inform future plans. The results of Vriddhi’s support, therefore, are likely to be limited to the project’s lifetime. EQ3: What key achievements and challenges helped/hindered the outcomes of the project? Vriddhi enhanced the technical competencies of a large cohort of service providers and built the capacity of high caseload facilities to deliver a set of high-impact interventions from the RMNCH+A 5x5 matrix. In the process, it demonstrated to district teams what it takes to manage the delivery of these services with quality. However, high-burden locations face perennial HR deficiencies at all levels, including severe shortage of staff, lack of required training and skills, poor utilization of available skillsets, the heavy burden of administrative tasks and frequent transfers and rotations preventing skill and knowledge transfer within some facilities and consistent staff mentorship. While HR and infrastructure gaps fell outside of Vriddhi’s direct mandate, these challenges significantly affect what Vriddhi is able to achieve. Vriddhi lacked a strategic, coordinated response to address these gaps or how it would achieve its objectives within these constraints, which was a missed opportunity that may threaten the overall impact and sustainability of the project.. A lack of communication between project staff and government entities often led to misunderstanding of the full scope of the project and staff taking up initiatives outside of the project mandate while trying best to be responsive to government requests. Overall, Vriddhi’s broad-reaching scope was overly ambitious within the project timeline, as each component of RMNCH+A could be a standalone project in itself, and each would require considerable resources and strategy to be effectively addressed. As a result of Vriddhi’s central focus on CaB, programmatic attention to other technical areas was imbalanced or lacked sufficient investment (both time and resources) to be fully effective. EQ 4: What lessons can be drawn from this program in terms of key strategic approaches and impacts that should inform USAID’s future health interventions? 1. Projects timelines need to realistically align with project objectives, which often involve proving the feasibility and effectiveness of a core approach, and building in-house capacities. 2. Improving the quality of services may affect but does not directly impact uptake of services. The huge inequalities in accessing facility-based care also need to be addressed. 3. There is no mechanism to assess the completeness of Health Management Information System data on health outcomes and service utilization. Population-level measures will be needed to accurately assess gaps and plan responsive measures. 4. The project has to consider building managerial competencies for delivering CaB services, further training in CaB facilities, and capacities at state and district levels for planning, data management, and monitoring, in order to sustain the momentum of change. RECOMMENDATIONS FOR USAID 1. Limit the scope of initiatives with lofty but time-bound goals to improve population health outcomes, acknowledging where technical areas are complementary and where activities and staff are not overstretched while being fully responsive to government priorities. Instead, focus on prioritizing a limited number of key technical areas where activities have sufficient feedback loops and opportunities for learning and continuous improvements. 2. Align expectations of initiatives such as Vriddhi and LaQshya that have the sole mandate of supply￾side strengthening, given that context-sensitive combinations of interventions that address demand and supply-side barriers will be needed to impact population level uptake and health outcomes. 3. Support integrated health programs that address both supply- and demand-side issues and which speak to and learn from each other. Build national capacity to generate local evidence that identify ix issues impacting the availability of services, geographic and financial accessibility, and acceptability of services, and design context-specific interventions to address them. 4. Build national capacity for stronger measurement strategies, such as effective coverage, a quality￾corrected measure of population-level outcomes, as a metric for monitoring health system improvements at the population level. This combines need, use, and quality, and has been increasingly advocated as a measure of Universal Health Coverage.i 5. Prioritize strategic support to states in targeting adolescent health improvements, collecting end user and beneficiary feedback more systematically, and utilizing private sector cells within government. 6. Utilize the private sector landscape assessment to inform a scaled-up strategy for the GoI at national and state levels under a dedicated PSE project and with a more substantial timeline. 7. Coordinate with other development partners to bring together a diverse, inter-sectoral team to develop a long-term and creative set of solutions to the perennial, complex issues with HR at program management, facility and outreach levels, which existing policies/projects have failed to address effectively. RECOMMENDATIONS FOR VRIDDHI (NATIONAL) 8. Ensure that lessons from CaB implementation and the initial scale-up efforts feed into LaQshya PMU through experience-sharing workshops and site visits. 9. Continue support for the adoption of KMC in the scale-up phase of CaB in each state, with a focus on clarifying facility staff interpretation of guidelines, especially in facilities without Special Newborn Care Units (SNCUs), deemphasizing dependence on equipment and supplies, and emphasizing the importance of continued KMC practice in the home. 10. Work with national and state health teams to increase the uptake and use of the RMNCH+A App and Health Atlas. 11. Incorporate recent developments in performance standards for RMC and their measurement in CaB scale-up locations. Experiential learning and data from Vriddhi will contribute to this evolving field. 12. Incorporate metrics around client satisfaction in ongoing interventions to include feedback on facility infrastructure, treatment by medical staff within the facility, timeliness of care, cleanliness, availability of space, staff communication during delivery with mothers and families, and information provided regarding follow-up after leaving the facility. RECOMMENDATIONS FOR VRIDDHI (STATE AND DISTRICT) 13. Continue supporting states to understand the nuances of operationalizing their unique scale-up plans. Invest in detailed documentation of “what it takes” to operationalize CaB, Advance Distribution of Misoprostol and Community-based Management of Newborn Sepsis, and other interventions in disaggregating information by state, district and facility, to shine the spotlight on both the strategic and day-to-day operational details in which Vriddhi has excelled. 14. Support scale-up efforts by titrating the level of support (training, monitoring, assistance to planning and budgeting for supplies) to the current capacity of states and districts to absorb the interventions in CaB scale-up and Newborn Action Plan (NAP) implementation. This also includes refining the mechanisms for client feedback and developing long-term accountability measures for providers and staff. 15. Dedicate time and resources to invest in building capacity for strategic planning in state governments, using ongoing feedback from the current implementation around monitoring and reporting and utilizing data for decision-making, and also feed this information into efforts that address HR issues. x 16. Ensure that key training focal persons have been identified within state and district governments who are committed to train new staff in new facilities and provide refresher training for those already trained in CaB and in utilization of key lifesaving equipment for mothers and newborns. 17. Clearly communicate the project’s mandate (and its limitations) to district and state government. This communication should include identifying a consistent communications mechanism between staff and government counterparts for requesting and providing technical support within the project’s mandate. 18. Regulate the age-appropriate peer education model across all states and emphasize community level information campaigns to reach male and female adolescents where they feel most comfortable and less stigmatized, especially in discussing Sexual and Reproductive Health (SRH). 19. Ensure the incentivization of ASHAs and service providers for providing counseling for postpartum family planning (PPFP) and provision of a basket of choices. 20. Generate state and district-wide initiatives to capture and analyze data of both in- and out- of school youth in order to inform a comprehensive adolescent health strategy for initiatives to target the key identified areas where adolescents are in need of information. 21. Establish state government focal points as champions to initiate a full-fledged private sector development strategy and fully implement it, providing a clear engagement and communication channel between public and private actors. Relationship building takes time and must be sustained in order to keep private sector interests. 1 I. INTRODUCTION & PROJECT BACKGROUND India’s maternal, infant, and child mortality rates are among the highest in the world.ii The Government of India (GoI) has made considerable progress reducing mortality over the last few decades, including the implementation of the National Health Mission in 2005, which yielded significant improvements in the survival and health of mothers, newborns, and children. However, these advances have varied significantly across states and high mortality rates persist. To address these gaps and improve outcomes more uniformly in underperforming areas, the GoI identified 184 High Priority Districts (HPDs) across all states to target its efforts within its comprehensive Reproductive, Maternal, Neonatal, Child and Adolescent Health (RMNCH+A) Strategy. The RMNCH+A Strategy aimed to improve RMNCH+A indicators in these districts using an integrated programming approach, improving healthcare service delivery at various stages of life, including adolescence, pregnancy, post-natal, and newborn periods, with a focus on the unique needs of marginalized and vulnerable populations. In June 2012, the Governments of India, Ethiopia, and the United States, along with the United Nations Children’s Fund (UNICEF), convened a Summit on “Global Child Survival Call to Action: A Promise to Keep.” The GoI later held a Summit on Call to Action for Child Survival, formally launching its RMNCH+A Strategy. As part of this Strategy, the GoI identified high-impact interventions for each of the five health areas, as well as five cross-cutting and health systems strengthening interventions, and the essential commodities for each intervention. The strategy directed each Indian state to focus its efforts on the most vulnerable and disadvantaged groups within its borders. The GoI also designated one donor or development partner (DP) to lead the targeted efforts for each state and its HPDs. In May 2014, the United States Agency for International Development (USAID) began providing technical and managerial support to the GoI to implement the RMNCH+A Strategy through its Scaling Up Interventions in RMNCH+A technical assistance project. Scaling Up Interventions in RMNCH+A is a four￾year, $25 million (USD) activity that concluded in May 2018, but has been extended through May 2020. Scaling Up Interventions in RMNCH+A—now commonly referred to by its Hindi name, Vriddhi—is implemented by IPE Global in partnership with John Snow, Inc. (JSI).1 IPE Global works in partnership with the GoI to provide technical and managerial assistance to the national, state, and district governments, with an emphasis on the effective scale-up of high-impact interventions identified in the GoI five-by-five matrix. USAID works in 26 HPDs across six states in India where USAID is the lead DP: Delhi, Haryana, Himachal Pradesh, Jharkhand, Punjab, and Uttarakhand. The project works with government systems at various levels to expand these interventions to other HPDs and non-HPDs, sharing best practices and lessons learned, and engaging with the private sector and civil society organizations. Vriddhi’s overarching goal is to reduce preventable maternal, neonatal, and child mortality in its HPDs, with a primary objective of increasing uptake of RMNCH+A services in these HPDs. There are four strategic outputs that support the achievement of this objective: i. Improve availability and quality of RMNCH+A services in government health facilities ii. Strengthen evidence for RMNCH+A services iii. Incubate RMNCH+A good practices for scale-up iv. Involve multiple stakeholders in delivery of RMNCH+A services 1 Vriddhi’s original design included a consortium of implementing partners. Jhpiego and Save the Children withdrew after Year 1 due to country regulations preventing IPE Global from sub-granting to non-Indian organizations. Jhpiego was unable to register as an Indian entity and although Save the Children was a registered Indian entity, the value of the planned sub-award exceeded the ceiling on the size of subgrants Indian entities can receive. 2 II. EVALUATION PURPOSE & QUESTIONS USAID/India contracted Social Impact, Inc. (SI) to conduct a mid-term performance evaluation (PE) of the Vriddhi project. The purpose of this PE is to: 1. Assess Vriddhi’s results and overall progress against its objectives and performance indicators. 2. Evaluate the key strategies and approaches Vriddhi adopted in scaling up its efforts and develop recommendations for the remainder of the project. 3. Provide insights and lessons learned from Vriddhi’s various components that will help inform USAID/India’s future designs in the RMNCH+A sector. This PE seeks to answer the following evaluation questions (EQ): EQ 1 To what extent has the project achieved the objectives and expected results in terms of enhanced institutional capacity of GoI and private sector networks to scale up proven high-impact RMNCH+A interventions? i. To what extent has the program been successful in influencing various GoI efforts on improving RMNCH+A specifically through the support of the National RMNCH+A Unit? ii. To what extent has the program engaged private providers or stakeholders in delivering RMNCH+A services, and what are the effectiveness and results of this engagement? iii. To what extent have the results of this project contributed to the scale-up of effective RMNCH+A solutions to achieve significant reductions in preventable morbidity and mortality among women and children? EQ 2 How effectively has the project scaled-up the approaches/models/interventions from the five-by-five matrix used by the GoI in the national program? What have been the results (effectiveness) of the project’s key approaches on: i. Increased off-take of the RMNCH+A services in High Priority Districts, and ii. Technical assistance for improved program planning, monitoring, and implementing at the district and state levels? EQ 3 What are the key achievements and challenges that helped/hindered the outcomes of the project? EQ 4 What lessons can be drawn from this program in terms of key strategic approaches and impacts that should inform USAID’s future health interventions? The primary audience for this evaluation report is USAID/India, with the expectation that the evaluation’s results will inform future strategy and programming within the Mission’s health portfolio. The secondary audience includes the implementing partners (IPs), IPE Global and JSI, who may be able to integrate lessons learned into their future implementation efforts. Finally, a tertiary audience is the GoI and the development community who are also active in implementing the RMNCH+A Strategy throughout India. III. EVALUATION METHODS & LIMITATIONS DATA COLLECTION METHODS Desk Review: Prior to data collection, the evaluation team (ET) conducted a thorough desk review of all available project documents and relevant RMNCH+A literature to understand the situational and programmatic context. The ET developed an analysis framework, organizing desk review findings by document type and source, mapping them against each evaluation question. This framework allowed the ET to highlight gaps in the literature that could be further probed during data collection. The ET also utilized the desk review findings to inform the data collection protocols and as a source of triangulation for primary qualitative data collected during fieldwork. Field-Based Data Collection: The ET conducted field-based data collection from March 26-April 16, 2018. Upon arrival in New Delhi, the ET held an in-brief meeting with USAID to discuss the evaluation workplan, clarify evaluation questions and USAID’s plans for utilization of the evaluation findings and 3 recommendations. After USAID’s approval of the workplan and the final data collection protocols (see Annex III), the ET began qualitative data collection. The ET collected data from all six USAID focal states and the Centre (New Delhi). Due to time constraints and geographical distribution, the ET visited a sample of 10 HPDs – see Table 1 for the distribution of sampled HPDs by state. Table 1: Sampled High Priority Districts by State Haryana Punjab Himachal Delhi Jharkhand Uttarakhand Total Total HPDs 5 5 4 2 7* 3 26* Sampled HPDs 1 2 1 1 3 2 10 HPDs Mewat Muktsar, Barnala Chamba Northwest Delhi Gumla, Dumka Godda Haridwar Tehri Garhwal 10 *UNICEF worked intensively in 4 of the 11 HPDs in Jharkhand; therefore, the number of HPDs for USAID project interventions is 26. The ET used a purposive sampling approach for the selection of HPDs and key informants. The sampling criteria for HPDs included: i) geographic location, including rural and urban districts and distribution across the state, ii) project activities in the district, including pilots of newer models and approaches, iii) stakeholder presence in the district, and iv) district performance against RMNCH+A indicators (both high￾and low-performing districts). The final sample of key informants was relatively balanced by sex (61 male, 53 female) and included government officials (at the national, state, and district levels), IP staff, USAID officials, public healthcare providers, and private healthcare providers. Table 2 details respondents by type and state. Using semi-structured key informant interview (KII) guides, the ET conducted 88 KIIs with 114 total respondents (with multiple respondents in some interviews). The ET also reviewed labor room registers in some of the facilities the team visited in each state to gather data on project intervention practices and national-level Health Management Information System (HMIS) quantitative indicators. At the conclusion of fieldwork, the ET presented preliminary findings to USAID at an outbriefing on April 16. Table 2: Key Informants by Respondent Type and Location DATA ANALYSIS The ET systematically analyzed both qualitative and quantitative data. During fieldwork, the ET took KII notes in real-time, cleaning and synthesizing electronic notes and summaries after site visits each day. The ET used content and comparative analyses to identify patterns and elucidate themes and contextual factors among qualitative data. The ET used these themes to generate a coding structure and systematically coded all qualitative notes in a Microsoft Excel-based coding structure, organized by evaluation question and sub￾question. The ET disaggregated qualitative data by stakeholder type, respondent sex, and location, to capture divergences between men and women and geographic perspectives. The ET also reviewed quantitative project data from baseline and monitoring reports and HMIS data to identify progress in meeting indicator targets at the time of the evaluation, noting any regional considerations or variability. The ET analyzed programmatic and HMIS quantitative data in Microsoft Excel. Centre Haryana Punjab Himachal Delhi Jharkhand Uttarakhand Total Government Officials 3 2 8 5 4 9 9 40 Development Partners 3 1 4 Public Providers 2 2 1 2 8 5 20 Private Providers 2 1 3 Project Staff 4 4 3 3 2 3 2 21 Total 10 8 13 9 8 23 18 88 4 LIMITATIONS Representativeness: While the ET conducted a large number of KIIs for an evaluation of this scope, the overall qualitative sample is still small compared to the project’s intended reach and cannot be considered representative of all HPDs or of the broader population to a statistically significant degree. While the ET mitigated this by analyzing population-level data (such as HMIS data), these data points do not directly align with Vriddhi interventions, and in the absence of a counterfactual, the ET is unable to assign attribution at the population- or outcome-level to Vriddhi. Selection Bias: Given the purposive nature of the sample and the coordination support provided by the implementing partner, there is some risk that the ET was guided toward key informants that had favorable experiences of the project or toward high-performing facilities. The ET attempted to minimize this by randomly selecting public facilities and private providers whenever possible. However, all public facilities had prior knowledge of planned data collection, which could have created an opportunity for preparation prior to the ET’s visit. Recall Bias: The ET observed recall bias (difficulty remembering specifics from the past or blending experiences into composite memories) from respondents who were unable to remember specifics about trainings they received or described non-Vriddhi interventions or activities. The ET took steps to mitigate this bias by asking targeted questions about Vriddhi interventions and clearly noting when respondents referred to non-Vriddhi activities so responses were not conflated in analysis. IV. FINDINGS & CONCLUSIONS This section presents findings and conclusion by evaluation question and sub-question. In some cases, they are further divided by intervention or project component. Section V presents recommendations overall. EQ1 FINDINGS To what extent has the project achieved the objectives and expected results in terms of enhanced institutional capacity of GoI and private sector networks to scale up proven high-impact RMNCH+A interventions? This section presents evaluation findings and conclusions related to the performance of Vriddhi’s key RMNCH+A interventions. Care around Birth (CaB): Vriddhi designed and implemented the Care around Birth (CaB) approach to deliver high-impact, evidence-based interventions during the intra-partum and immediate post-partum period, addressing the major drivers of maternal and neonatal mortality. CaB combined elements of quality of service delivery with health systems strengthening, using a provider- and patient-centered approach. Vriddhi implemented CaB in 141 high caseload facilities, covering about 70 percent of all births in the 26 HPDs. Key CaB activities included a baseline facility assessment, design of training packages, two-tier training of Vriddhi staff and service providers, establishing facility-level mentorship and quality improvement (QI) teams, handholding visits, skills drills, procurement of CaB-related equipment and supplies, and the collection, reporting, and local use of CaB-related data. CaB’s technical training content was highly standardized and organized into technical intervention packages (TIP) 1 and 2. TIP 1 covered high-impact interventions related to routine obstetric and newborn care and standard labor room management. TIP 2 trained providers on the management of complications and Kangaroo Mother Care (KMC). Vriddhi staff used assessments of provider knowledge and competencies to tailor trainings to local needs. Training material converted existing guidelines for these known interventions into standard operating procedures (SOPs) and combined them with adult learning and experiential learning approaches, with a focus on skills. Vriddhi developed content for respectful maternity care (RMC) through consultations, still a nascent concept in India. RMC promoted a positive facility experience for patients, promoting rapport between women and health providers and creating an 5 environment free from abuse, disregard for privacy, and discrimination. RMC principles were woven throughout all CaB interventions. Vriddhi designed its training pedagogy to be interactive and participatory and included drills to practice how to deliver the interventions, with the aim of developing skills and competencies. Most public provider key informants cited the hands-on style of the training as a key strength of the trainings, and the majority of key informants described the quality of both the style and content of the trainings favorably, resulting in improved skills and confidence. Following the initial trainings, Vriddhi identified mentors in each facility and trained them to conduct on-the-job coaching for new staff that join the labor room by rotation or transfer from other facilities, which has an added benefit of reinforcing the skills of those who were previously trained. Vriddhi also established skill stations with mannequins to help mentors conduct “low-dose, high-frequency” drills for managing complications. CaB established and streamlined labor room processes, such as organizing supplies in trays or on carts for easy retrieval and use, enabling providers to deliver the high-impact interventions efficiently and effectively. Vriddhi also facilitated revisions to labor room registers in each state to include critical data points. Vriddhi helped form cross-functional QI teams in each CaB facility and coached the teams to review case sheets to identify gaps in the quality of services, agree on remedial actions, and document them. CaB-trained service providers, as well as state and district managers, characterized the training approach and delivery to be of high quality. Trainers were regarded as highly skilled technical experts who delivered interactive trainings. Facility and program managers reported that the drills were particularly helpful in facilities with high turnover of staff. However, in several facilities the ET visited, the drills were conducted only when Vriddhi staff were present, and the mannequins and simulation equipment were not present in other facilities. The ongoing coaching was widely considered more important than the initial training to build and maintain competencies, and much of CaB’s success can be attributed to the coaching done by Vriddhi district staff. Providers in CaB facilities reported that the training, mentoring, support visits, and streamlined processes increased their confidence in managing routine labor processes and complications and making more appropriate referrals. Many facility respondents explained that they had previously referred such patients to higher facilities without any attempt to manage them. They report that they had learned some of these interventions in theory but did not implement them consistently. They do so now as they have learned “why they do what they do,” and also because of regular encouragement from Vriddhi staff, despite viewing these interventions as “increased” workload. Delivering the interventions has necessitated frequent interactions between providers and mothers and newborns, and the provision of RMC has enhanced the quality of these interactions.. Involvement in QI teams has improved the problem-solving capabilities of providers and facility managers, and managers are better able to monitor the delivery of services and supplies. In several locations, these teams continue to function without support from Vriddhi staff. Measuring CaB Results: There is anecdotal evidence that CaB’s most notable achievement is the successful identification and management of obstetric and newborn complications. However, measuring this impact is not straightforward because of a number of artefactual effects resulting from Vriddhi’s concurrent interventions (such as improvements in the capturing and reporting of data on complications and deaths over time, and greater readiness to admit and manage women with complications) and by extraneous factors (such as the clinical profile at admission, and the availability of an obstetrician or anesthetist to provide comprehensive emergency care). Management of birth asphyxia is one intervention that was not likely affected by these factors. CaB data shows that the proportion of newborns with “Turning training into actual improvements in performance is a challenge but Vriddhi has supported that process quite well.” – Senior health official, Uttarakhand “We used to depend on the doctor to do newborn resuscitations, but now we can do it, and the doctor trusts us to do it.” -Staff nurse, Himachal Pradesh 6 asphyxia who were successfully resuscitated rose sharply from around 70 percent in 2015 to early 2016 to above 90 percent from mid-2016.iii This increase is likely due to CaB training and support, but there is no counterfactual to support this attribution. Nevertheless, the ET conducted a data validation exercise and found that project data on facility performance was highly accurate, with an average variance of -1.22 between facilities and the district reports and no variance between district and project-wide reports. Annex IV provides the detailed results of this exercise. In the ET’s review of Vriddhi’s measurement of RMC, we found that it does not reflect recent advances in this fieldiv or experience in measuring RMC in similar contexts.v CaB Limitations and Challenges: Several respondents reported that CaB demonstrated that it is possible to improve service delivery with existing infrastructure and human resources (HR) in public facilities, by focusing on improving and institutionalizing technical competencies of service providers. However, the ET found that these improvements were largely made possible by the continual improvisation and innovation of Vriddhi staff at the district level. They found creative solutions for a range of issues that threatened successful ongoing implementation of CaB, such as procuring essential supplies locally and retaining trained staff in the labor room. District health managers reported that Vriddhi staff rapidly and assertively escalated issues, when appropriate. The ET found some evidence for facility and district-level managers internalizing these competencies by observing Vriddhi staff in action but did not find any evidence for an intentional effort to build these competencies alongside the technical ones. The ET also observed that Vriddhi primarily used its own staff as CaB trainers in its direct implementation locations, in an effort to prevent dilution of quality and potential delays in obtaining approvals for using health department staff as trainers. District health managers viewed Vriddhi’s management information system (MIS) to be of limited use, as it includes only CaB facilities. These issues are poised for change through the scale-up of CaB to entire districts and states, which is discussed further under EQ2. The ET found severe constraints in HR and infrastructure in facilities. A complex web of factors has resulted in several facilities functioning with one-half to one-third of the required number of provider, with most labor rooms lacking dedicated, permanent staff (rather staff float between departments to cover staff shortages as needed). Staff shortages also affected CaB’s implementation in facilities with high caseloads. This choice of Level 2 and 3 (L2 and L3) facilities for CaB implementation was guided by the GoI’s focus on these facilities for supportive supervision. However, the chronically understaffed L2 and L3 facilities, which are not staffed proportional to their caseload and population served, resulted in an uneven distribution of CaB-trained providers and functionally increased the burden of already high caseload facilities. The increased workload of existing labor room teams sometimes resulted in staff being unable to deliver CaB interventions consistently or at all. The use of partograph (the intervention that requires the most sustained effort) is a clear example of this. It is the only intervention that was at less than 70 percent during the last quarter of 2017 while others were trending at 90 percent or higher performance.vi Facility mentors reported that these HR constraints limit the extent to which they are able to mentor junior staff or new hires. Kangaroo Mother Care (KMC): KMC is an evidence-based practice of skin-to-skin contact between pre-term or term low birthweight (LBW) infants and their mothers or other caregivers that supports temperature regulation, promotes bonding, and encourages breastfeeding and weight gain.vii Although the GoI released KMC guidelines in 2014, its uptake was limited to a few large facilities across the country, led by individual champions. Vriddhi sought to institutionalize KMC, beginning with the Haridwar and Gumla District Hospitals,viii and later rolled it out in all CaB facilities as part of TIP 2 (as late as August 2017 in some areas). World Health Organization (WHO) guidelines suggest that KMC be initiated in the hospital so mothers can learn and feel confident continuing the practice at home. KMC can be implemented in any setting that allows the mother to comfortably learn the practice and does not require special equipment or supplies. Vriddhi’s KMC training prescribed a designated room or corner where KMC could be practiced privately, 7 and many facility respondents cited a lack of adequate space as a major barrier. Vriddhi staff at state and national levels reported that no significant equipment is required, but those at the periphery explained that their abilities to implement KMC were constrained because they did not have reclining chairs or H￾wraps (an H-shaped piece of fabric used to bind the baby around the mother’s torso). Other facilities reported delaying the implementation of KMC because they needed to budget for the equipment. Despite these misconceptions, a few facilities reported practicing KMC with minimal resources. KIIs with IPs and providers suggest a disconnect between how KMC was taught and how it was ultimately implemented. Vriddhi primarily, though not exclusively, implemented KMC practices in Special Newborn Care Units (SNCUs), although the SNCUs themselves were not established by USAID. WHO guidance suggests that while some newborns below 1.8 kg may be too unstable for KMC, they should be evaluated on a case-by-case basis, and newborns below 1.8 kg who are sufficiently stable could benefit the most from KMC in places that lack incubators or other methods of warming (i.e. not SNCUs). Vriddhi-trained facility staff reported that newborns below 1.8 kg would only be started on KMC once stabilized in an SNCU. In a district hospital without a functioning SNCU, the medical officer explained that they did not provide KMC to a baby that weighed less than 1.8 kg only because the newborn was on antibiotics. The tendency to only practice KMC in SNCUs also meant that babies in the regular labor or recovery wards who were not SNCU-eligible did not receive the benefits of KMC. The duration of KMC practice was particularly rushed in high caseload facilities, with some mothers only practicing KMC for 30-45 minutes at a time. Early and exclusive breastfeeding (a key feature of KMC) was reportedly encouraged but not consistently implemented. A nurse explained that they encourage mothers to breastfeed as soon as possible, but some newborns delivered by C-section are supplemented with formula. To increase the likelihood of KMC’s continuation after discharge from the hospital, facility staff reported training other family members, including fathers and mothers-in-law, in the practice. Hospital staff also explained that mothers often report housework responsibilities or childcare for other children as a barrier to practicing KC at home. While most facility respondents reported keeping records of KMC while the mother and newborn were in the hospital (such as number and duration of KMC sessions) KMC follow￾up and tracking after discharge from the hospital was relatively minimal. Nearly all the facilities the ET visited reported having no mechanism to track KMC and relied only on informal reports from ASHAs. Overall, despite its inconsistent application, providers in public facilities reported favorable outcome improvements in breastfeeding and newborn weight gain as a result of KMC. A staff nurse in Himachal Pradesh noted that “before we weren’t really thinking that KMC was necessary, but now we see the results from doing it with underweight babies.” Additionally, some private providers reported implementing the practice after Vriddhi training. Adolescent Health: The GoI rolled out the National Adolescent Health Strategy in 2014, under the name Rashtriya Kishor Swasthya Karyakram (RKSK) to address and improve nutrition, sexual and reproductive health (SRH), mental health, and prevent violence and substance misuse. Project staff, government officials and public providers identified key adolescent health (AH) concerns as nutrition, stress, sexually transmitted infections, contraception and unplanned pregnancy, and the ability to access information and resources. Project staff indicated that girls were commonly afflicted by anemia and menstrual concerns, while boys were most commonly afflicted by obesity and substance use. High rates of drug abuse among boys were of particular concern in Punjab. Government and project staff agreed that the district or state administration’s prioritization of AH is what dictates action, but despite the identified AH concerns, the government lacks champions that can propel the AH agenda at both state and district levels. Both government officials and Vriddhi staff agreed that AH does not appear to be a priority of the government and that Vriddhi has not holistically addressed it. Government policies and strategies to directly target AH improvements did not come to fruition during the project, despite the project’s advocacy for and completion of formative research toward an adolescent communication and health promotion strategy. Many respondents felt that AH needs have been neglected by the government or 8 have been overshadowed by the focus on maternal and newborn health. Vriddhi helped finalize tools and micro-plans for the RKSK pilot in Uttarakhand, which are now being used nationwide. While the district Adolescent Health Officers are responsible for monitoring and supervising the RKSK initiatives at the state level, the ET found that Vriddhi staff were the only persons doing so. Vriddhi supported three initiatives under RKSK: Adolescent Health Days (AHD), Adolescent-Friendly Health Clinics (AFHCs) and Weekly Iron Folic Acid Supplementation (WIFS). The project’s approach to targeting AH centered on improving the availability and quality of AFHCs as noted in the 2016 AFHC Strengthening Plans for each district. AFHCs existed in the high caseload facilities in all states except Punjab before Vriddhi, but many were often unstaffed or lacking in equipment or commodities. The baseline for these plans identified how many clinics were operational or sanctioned, registers used, equipment shortfalls, availability of HR, information, education and communication (IEC) materials and commodities, and services offered. The clinics are outlets for both male and female youth to receive information not available in school, especially about mental health and menstruation. To date, Vriddhi’s involvement in the AFHCs has focused on developing new clinics, identifying clinic infrastructure needs, branding clinics and advising on more user-friendly clinic locations, supervising clinic activities, monitoring commodity availability and developing IEC materials to increase uptake, all completed in the HPDs per the specifications in the Strengthening Plan. The Plan recommended continuous monitoring and mentoring to be done by both the Vriddhi staff and the facility manager, however the ET found that AFHC monitoring was only being done by Vriddhi staff. Vriddhi increased the number of operational clinics from 79 in October 2015 to 100 in June 2017. The ET found that the clinic uptake ranged greatly and is affected by the stigma surrounding the clinics, the fear of the hospital environment and lack of privacy, and adolescent shyness. Exit interviews on client satisfaction have not consistently been conducted across districts, which could provide substantive data on uptake trends for female or male adolescents; however, data from 2017 showed that 97 percent of attendees were satisfied overall with their experience, with more than half of respondents being girls. The ET found that the AFHC counselors are not always appropriate for adolescent needs in terms of sex (most clinics do not have both female and male counselors), age, and counseling experience, and they lack comfort discussing SRH issues. Government officials and Vriddhi staff agreed that the peer-to-peer counseling approach is most useful to reach adolescents, but this model has not been rolled out in all states. Existing AFHC counselors are not exclusively staffed in the AFHCs, sometimes serving other roles, and are often stretched thin between clinical responsibilities and counseling. Vriddhi held a four-day training for staff nurses and Auxiliary Nurse Midwives (ANMs) on RMNCH+A counseling for maternal and newborn health, family planning and adolescent health, two days of which was designated for family planning, with less than a half-day for adolescent health. The AFHC baseline showed all four selected Haryana facilities had counselors, and none of the district hospitals in Punjab had counselors. While the placement of dedicated AH counselors within AFHCs was a recommended activity in the Strengthening Plan, the Punjab and Haryana state governments cut the budget for new counselors. There are processes for outreach in place where adolescent counselors engage with students and community members, both in and out of schools, to share information about and generate demand for the AFHCs. One such initiative, AHDs, are held monthly and provide counseling to youth and their parents on hygiene, nutrition, and substance abuse. Vriddhi staff believed that AHDs have increased the number of adolescent visits made to the Uttarakhand AFHCs. “Maternal health is the priority of the state, so why shouldn’t adolescent health be a priority? Maternal health can only be achieved if adolescent health is achieved, because it is from early years that you need to start addressing the health.” - Senior health department official, Punjab 9 The state government also utilized project staff to collect data from the other RKSK initiative’s activities. Data for WIFS, the school-based iron supplementation program, was difficult to retrieve, as many teachers were not fully informed as to the purpose of iron supplementation and did not respond well to being responsible for “mixing health and education” or collecting data on distribution of health commodities. In Punjab, the project conducted a rapid assessment of WIFS to identify gaps in implementation; recommendations from this assessment were piloted in Barnala district to improve WIFS by standardizing the supply of iron tablets at the district level and creating a compliance checklist. Vriddhi staff trained counselors and teachers on supplement distribution to support the WIFS program and identified nodal persons within schools for accountability and to increase teachers’ comfort following WIFS protocols. There is limited to no availability of population data on adolescent health behaviors or trends from the government or generated by the project. Both the government and Vriddhi’s AH interventions largely do not address the health needs of out-of-school youth or of boys, and the uptake of AH services by boys is perceived to be lower than girls. A student-based health survey was done by Vriddhi in Haryana only, and included questions about health behaviors around exercise, diet, substance use, and health information to inform the state’s AFHC approach. Survey questions about SRH behaviors were limited based on the state’s preferences. The data on iron and calcium supplementation for adolescents at the community level is collected by ASHAs/ANMs solely for reporting and is not analyzed for trends. While exit surveys from AFHCs were initiated under Vriddhi, there is no consistently collected state-level data of direct feedback from adolescents or their parents at the community level or from students at the school level on experiences with WIFS or from services received at AFHCs. There is no data on uptake of clinic services by out-of-school adolescents. Reproductive Health & Family Planning: Vriddhi’s original plans for reproductive health programming included home-based distribution of contraceptives, an activity that was largely driven by Jhpiego. After Jhpiego left the consortium at the end of Year 1, IPE Global shifted its family planning (FP) activities to better align with CaB. Vriddhi trained nurses in CaB facilities to counsel and offer postpartum family planning (PPFP) methods, with primary emphasis on postpartum intrauterine contraceptive devices (PPIUCD). Vriddhi also responded to the dearth of FP-related IEC material in facilities, developing a poster describing the Basket of Choices (of family planning options), distributing them in all six states, and translating into Punjabi (in Punjab State). District managers in Punjab and Delhi reported that these posters enable providers to offer comprehensive FP counseling. According to CaB HMIS data, 21 percent of all women who delivered in CaB facilities elected to receive PPIUCDs, although this trend has not changed much between 2015 and 2017 (the project’s implementation period).ix In one district, Vriddhi successfully advocated for inclusion of GoI-approved performance incentives for providers in the district budgets, but these incentives did not yield any observable effects on uptake. Counseling and distribution of FP methods were part of Urban Health and Nutrition Days (UHNDs) that Vriddhi supported in Northwest Delhi. The ET found anecdotal evidence that this effort yielded improved demand. Interviews with government and IP officials suggest that current FP initiatives (at both the state and district level) are insufficient to increase both demand and supply. While the RKSK strategy was designed to address adolescent SRH, government and IP key informants agreed that it did not sufficiently engage the topic. The AFHC Strengthening Plan for facilities in all states recommended that all AFHCs have stocks of pregnancy test kits and emergency contraception (ECP), and endline data from December 2017 showed a 52 percent increase in clinics with kits and 46 percent increase in clinics with ECP from the baseline. Condom and oral contraceptive availability increased by 39 percent and 44 percent respectively from the baseline, but several key informants noted that contraceptives are still not widely offered or available at AFHCs. AH counselors do not discuss SRH during school outreach, and open conversations about youth “You can only achieve the MNCH+A if you address ‘R’ first. It’s important to focus on how to make reproductive health improve so that you have a healthy mother and healthy child.” – Senior health department official, Punjab 10 SRH generate resistance within communities or facilities, as a Vriddhi staff member in Northwest Delhi observed: “We do not feel it is the appropriate time to talk to adolescents about sex education and family planning.” In the face of this discomfort around adolescent SRH and with teenage childbearing at 7.9 percent,x both state and district officials in Delhi, Punjab, and Haryana noted the need for a separate national strategy to address teen pregnancy and behavior change. The strategies and activities that currently exist also lack specific approaches for engaging male adolescents in their own reproductive health needs. EQ1 CONCLUSIONS CaB built capacity of health providers to deliver high-impact obstetric and newborn interventions in facilities but faced some limitations. Through its CaB approach and TIP training packages, Vriddhi successfully built and institutionalized technical capacity in facilities across all HPDs to deliver high-impact interventions that reduce maternal and newborn mortality. QI teams and facility mentorships have the potential to continue reinforcing these competencies. However, severe HR and facility infrastructure constraints limited CaB’s success and reach. CaB’s focus on L2 and L3 facilities missed an opportunity to engage Level 1 (L1) facilities. The addition of a pared-down version of basic obstetric and newborn care in L1 facilities alongside the current CaB package for L2 and L3 facilities could have led to a more optimized distribution of caseload, improvements in quality of care across all levels, as well as a possible increased uptake of services. RMC improved overall patient experience, but the intervention did not take advantage of recent advances in its measurement. Vriddhi prioritized building technical capacity over managerial capacity, out of necessity. Vriddhi was mandated to demonstrate CaB’s feasibility and effectiveness as proof of concept in a relatively short period of time. This resulted in Vriddhi staff prioritizing the immediate need of building up of technical competencies in service providers, over the strategic need of building managerial competencies. These managerial competencies are equally critical for the “process reengineering” that will enable any quality improvement initiative to successfully impact service delivery. KMC’s impact was limited by misaligned implementation. KMC’s ultimate operationalization was more equipment-dependent and high-maintenance than intended. This “clinicalization” resulted in significant numbers of eligible newborns not receiving KMC because it was not implemented as widely as it should have been, and mothers’ perception of equipment dependence could result in fewer newborns receiving KMC after leaving the hospital. In addition, KMC was implemented relatively late in the project period, so it had a shorter opportunity to take hold within facilities. Adolescent and reproductive health were proportionally smaller in scope and impact. Given Vriddhi’s strong emphasis on CaB, the relegation of AH was not unreasonable and was responsive to the priorities and comfort levels of the government. Nevertheless, there were missed opportunities, with minimal effort made to reach boys and out-of-school adolescents and a lack of consistent, comprehensive data within states to quantify AH needs to inform a clear, comprehensive approach under the current health strategy. Ongoing data collection of AH activities, including WIFS and AFHCs, is a challenge for the government to uphold. AFHCs have promise but must seriously re-evaluate the comprehensiveness of the services provided. Counseling is useful for adolescents but long-term behavioral change requires more intention and effort. Adolescents benefit most from spaces where they can relate with peer counselors and have their privacy respected. Given the significant impact of birth spacing on maternal and infant survival and the GoI’s specific focus on PPFP, Vriddhi’s inclusion of FP within CaB has been appropriate. Vriddhi’s support to promotion of FP uptake through AFHCs, however, has not been optimal. 11 EQ1.I FINDINGS To what extent has the program been successful in influencing various GoI efforts on improving RMNCH+A, specifically through the support of the National RMNCH+A Unit? To answer this sub-question, the ET examined the supportive supervision mechanism and its implementation by the National RMNCH+A Unit (NRU). One of Vriddhi’s primary activities was to establish and standardize a uniform supportive supervision mechanism that could be used in all states, providing external monitoring to help government authorities understand facility readiness to effectively deliver RMNCH+A services. Vriddhi trained district staff of DPs to conduct supportive supervision visits to health facilities using the supportive supervision checklist, which was developed prior to Vriddhi’s inception by another DP. According to IP interviews, the GoI heavily directed Vriddhi’s mandate to include the uptake of the NRU—which had been established and funded by USAID prior to Vriddhi—and the supportive supervision checklist and mechanism. Another DP key informant indicated that Vriddhi aligned its activities more closely with GoI priorities than other DPs. In response to the GoI mandate, the NRU took the lead in finalizing the standardized supportive supervision checklist that was later used by all DPs. After its rollout, the NRU functionally coordinated the efforts of all DPs working on the RMNCH+A Strategy and compiled the data from all DP supportive supervision visits at the Centre. Many respondents (DPs, IP staff, and government officials) expressed that DPs commonly operate in siloes, focused on their individual mandates with little coordination. However, the unification around the RMNCH+A Strategy, with the coordination from the NRU, was widely lauded as a successful initiative, with one DP saying the NRU “created a single entry point for all the partners and a focused agenda for them to work on.” Most government officials perceived the supportive supervision process to be useful, explaining that it helps them understand the performance of the states and districts, and provides feedback loops to the Centre. However, some DPs were critical of the supportive supervision process, with one donor representative explaining, “There is hardly any support, it’s just supervision. What I’m seeing is supervision.” In addition, the overall technical and managerial capacity of the national government was reportedly low, and nearly all interviewed government officials reported being too busy to conduct supportive supervision visits themselves. They instead relied heavily on Vriddhi project staff to visit facilities and report back to them. On the other hand, GoI officials report that they view supportive supervision as designed to be conducted by DPs as a third party “truth-telling” exercise. IP staff also reported that the NRU was often stretched in many different directions, supporting various GoI campaigns or initiatives, some of which were outside of the scope of Vriddhi’s mandate. Some staff explained that the NRU being physically distanced from the Ministry of Health and Family Welfare (MOHFW) (it is housed within the IP offices) provided a buffer to this, giving them the space needed to focus on their activities and prioritize Vriddhi initiatives. For example, the NRU has recently invested significant time and resources in developing an RMNCH+A app and a Health Atlas, with plans to roll them out nationwide. However, the ET is unable to assess their uptake or quality due to the nascence of their implementation at the time of data collection. EQ1.I CONCLUSIONS The NRU worked hand in hand with the GoI throughout the duration of Vriddhi’s implementation, providing crucial technical and managerial capacity building and coordination support. However, the strategic influence flowed more from the GoI to the NRU, rather than in the other direction, with the NRU often supporting GoI initiatives that were sometimes outside of Vriddhi’s intended scope. In addition, although supportive supervision visits were a GoI mandate from the outset, reliance on project staff to conduct supportive supervision indicates that the process has not been well taken up by government officials, possibly due to time and capacity constraints. This is a direct threat to the sustainability of this practice, because if project staff leave, it seems likely government officials will not keep up the practice. 12 Furthermore, the evaluation found no efforts made to identify responsible parties within the GoI that will take over the supportive supervision data collection and analysis process after Vriddhi’s conclusion. EQ1.II FINDINGS To what extent has the program engaged private providers or stakeholders in delivering RMNCH+A services, and what are the effectiveness and results of this engagement? Vriddhi’s private sector engagement (PSE) objectives underwent multiple iterations. This aspect of the project initially envisioned as a multi-partner design but was assigned to JSI after the withdrawal of the other partners. Further, shifts in Vriddhi’s focus and priorities from demonstration models of good practices to a wider PSE strategy development were thought to disrupt a comprehensive PSE design, marking a “…change from what was initially planned and designed.” In late 2016, Vriddhi conducted a landscape assessment to explore knowledge, attitudes, and practices of 200 private providers across the six states, which included registered and recognized private MCH providers. The assessment found that private facilities had a bias in treating poor patients and were plagued by challenges in HR, equipment, financial viability, client attrition, and training. These providers also lacked standard client care guidelines, accreditation, and did not consistently offer FP services. The assessment findings were disseminated at a February 2017 meeting to inform a strategic roadmap but there has been no traction on the development of a formal strategy since then. Government and IP key informants agreed that the private sector is in need of in-service training, especially in semi-urban and rural locations which have a higher burden of maternal and newborn mortality, where the high impact interventions promoted the CaB are needed and where standards are not universally followed. In line with GoI plans to extend certification/accreditation to private facilities, the project conducted the first-ever forum to bring national and state public and private providers together. Vriddhi initiated a national-level orientation workshop on the high impact intervention models, national guidelines, and protocols for 51 pediatricians and gynecologists in Delhi. The forum highlighted increased willingness and openness from state levels to engage private providers, but the political will to engage is lacking. While some private providers are interested in engaging with the public sector, others lack interest. Those who are interested lack guidance on how to start a collaboration or feel it is up to the other party to initiate a collaboration and governmental financial support is expected for PSE to occur. Engaging with state medical colleges and professional associations (like FOGSI and IMA) on a broad platform is necessary for accreditation or collaboration, but there is no current strategy for collaboration with them. Following the national level forum, Vriddhi held state workshops to cover the same topics. Project staff explained that it was a challenge to get providers to attend due to the duration and location; furthermore, engagement of the private sector is a challenge in states with a smaller presence of private providers like Himachal Pradesh. The providers who attended had mixed reviews of efficacy of the Vriddhi workshop, with some noting that the information shared did not affect their practice and others acknowledging that they had been out of touch with standards and practice guidelines. Time was a major challenge in the attempt to craft a comprehensive PSE strategy. The process of relationship and trust building, planning, coordination, and scheduling, including selection of the pilot districts for demonstration modeling, was lengthy, and left minimal time to begin a comprehensive strategy development. This was further restricted by JSI’s limited staff at the state and district levels. Finally, the reinstatement of the Mexico City policy by the U.S. Government in late 2017 created restrictions that required JSI to halt its collaboration with gynecologists. The GoI initiated Pradhan Mantri Surakshit Matritva Abhiyan (PMSMA), a program to support antenatal care and safe motherhood and engage private providers to provide care to clients at public facilities on a monthly basis. Vriddhi created monitoring checklists and trained providers on counseling at discharge and high-risk pregnancy (HRP) identification. Project staff found that the standardization of these practices has 13 improved under this initiative; however, PMSMA is fraught with issues including unmet or impractical objectives (for example, requiring multiple ultrasounds during pregnancy or detailed examination of HRPs), supply inconsistency, and unclear guidance to private providers regarding their involvement. EQ1.II CONCLUSIONS Increases in institutional deliveries and reductions in maternal and neonatal mortality are dependent on better quality at public facilities and better accountability at private facilities. Vriddhi had insufficient time to comprehensively roll-out its PSE strategy, as trust and relationship-building take time. The Landscape Assessment provided key insights to private sector knowledge, attitudes, and practices, but no changes have been institutionalized as a result. EQ1.III FINDINGS To what extent have the results of this project contributed to the scale-up of effective RMNCH+A solutions to achieve significant reductions in preventable morbidity and mortality among women and children? Implementation of CaB in 141 high caseload facilities covered an estimated 70 percent of all births in the 26 HPDs. Although Vriddhi anticipated that the states would take up key elements of CaB, there were no set plans for scale-up beyond its targeted reach. This allowed the states to set the pace of scale-up based on the absorptive capacity of each location, among other considerations. CaB’s scale-up has taken place within four expanding concentric circles: 1. Within CaB facilities: Staff rotation and transfers, along with movement of residents, offer facility mentors the opportunity to train and coach an ever-increasing number of providers. The skill stations in hospitals in Delhi, for instance, continue to train residents during their relatively short time at these facilities, and train providers from lower-level facilities in the locality. With continued support and supervision, this has indefinite potential to impact service delivery. 2. Within the HPDs, beyond CaB facilities: Vriddhi facilitated quarterly Experience Sharing Workshops (ESWs) at the district level, where providers from CaB facilities shared the benefits of CaB and the constraints in implementing it. These ESWs helped district managers take remedial action and gave the non-CaB facilities and districts the motivation to seek CaB training and support in their areas. This has led to Vriddhi providing training and support to the non-CaB facilities in several districts. Punjab and Haryana have scaled up the use of trays in labor rooms, a best practice that CaB promoted. Jharkhand expanded CaB to the remaining 15 facilities in the districts where Vriddhi is present. QI teams have been initiated at other non-CaB facilities within HPDs in Punjab and Delhi. Vriddhi has created links for mentorship between district hospitals, maternity waiting homes, and family health centers in Delhi to improve the referral process and continuity of care. This initiative also has the benefit of reducing the delivery load burden at the district hospital. 3. Beyond HPDs, to the rest of the state: CaB has been scaled up to varying extents in all six states. CaB implementation in Punjab went from 25 facilities to 100, with plans for state-wide scale up. Four of the five HPDs in Punjab are no longer considered high priority, which the government attributes to Vriddhi’s success. Haryana has held CaB trainings in seven non￾HPDs, which along with the five HPDs represents more than half of all districts. Himachal Pradesh included CaB in pre-service training for 140 non-HPD nurses, trained all HPD and non-HPD SNCU staff in KMC, and scaled up training and monitoring to eight non-HPDs; along with the three HPDs already trained this represents 92 percent of the state. Delhi scaled up CaB to all 64 delivery points and 11 districts in the state. Uttarakhand combined CaB with fixed-day antenatal care (ANC) services at sub-centers to develop a “We would like to do a six-month review of these districts no longer considered HPDs. We don’t want them to go back on the successes they made. We will need these ‘graduated’ districts to share their best practices with the new (prioritized districts).” - Government official, Punjab 14 sophisticated and ambitious scale-up plan extending to the year 2020. Termed the Safe Motherhood and Newborn Health (SAMMAN) Initiative, this embedded program has scaled up to all facilities in the three HPDs in the last fiscal year and has approved budgets to scale up to four additional districts in the current year. Statewide coverage is planned by 2019-2020. Vriddhi assisted in planning and budgeting for the scale-up in all states, and trained state-level health staff in CaB and the rollout of the revised labor room registers and micro-plans for facility-level mentorship. The project also helped establish an online portal for reporting and data management under SAMMAN. However, the limited transfer of managerial competencies discussed under EQ1, as well as the constraints for state and district health officials to continue gap analysis and responsive planning, are likely to affect the quality of implementation in the scale-up locations. 4. Influencing and informing national efforts: Vriddhi trained district-level DP staff in six HPDs in Jammu and Kashmir in CaB. Practices from CaB rollout and scale-up have informed the development of LaQshya, the GoI’s national Labor Room Quality Improvement Initiative. Although this has not been explicitly acknowledged, the fact that MOHFW wanted Vriddhi to set up the project management unit (PMU) for LaQshya is a testament to this fact. LaQshya’s primary focus is on infrastructure and HR, in addition to technical skills-building, so the extent of incorporation of CaB learning into LaQshya remains to be seen, although the PMU can be expected to guide this process. Scale-up of other interventions: Vriddhi led the development of costed Newborn Action Plans (NAP) in Haryana and Jharkhand, in coordination with other DPs and the state health system. Impact on mortality: The project’s theory of change assumes that improving the quality of obstetric and newborn care in facilities will lead to reductions in maternal and newborn mortality. The premise for this theory is that increases in institutional deliveries brought about in the past decade through GoI efforts have not resulted in concomitant reductions in maternal and neonatal deaths. While the premise is indeed true, studies, including an impact evaluationxi,xii have not found a clear association between institutional births and maternal mortality, and that the poorest women do not equally benefit from programs promoting institutional births. These studies also cite “residual” confounders that are hard to control for, such as road networks and cultural factors. Concluding that improving quality of care around birth in public facilities will, in itself, lead to significant reductions in mortality ignores several important nuances. The studies referenced above, while unable to draw definitive conclusions due to statistical reasons, point out several likelihoods for increased institutional births not resulting in reduced mortality, in addition to the lack of quality care in public facilities: 1. The most vulnerable women are not entering programs such as the Janani Suraksha Yojana (JSY), or face significant non-financial barriers 2. The narrow focus on intrapartum care, leading to missed opportunities to identify risks in the antenatal period, which would actually reduce the burden of intra-partum complications that could lead to death. 3. The proven higher mortality associated with C-section rates higher than 15 percent, which could account for higher mortality amongst births in private institutions, where C-section rates are as high as 50 percent. Tellingly, the study by Randive et al. concludes “…Although the JSY succeeded in raising institutional birth proportions significantly; the same has not translated into significant reduction in the MMR. It is likely that a weak supply side has led to a situation in which increased access to institutional birth has not resulted in reduction in maternal deaths, as mothers are not receiving appropriate or adequate care. It is also possible that the JSY failed to draw mothers with life-threatening complications into institutions, resulting in most of such women continuing to deliver at home, contributing to persistent maternal mortality.” These observations can be schematically represented in an adaptation of the Tanahashi modelxiii, in Figure 1below. Interventions to improve quality of care in public facilities (“C”) cannot make a significant impact 15 on mortality until “B” is expanded to include the most vulnerable (blue shaded areas under “A”). It is important to note that this graphic does not account for the trends of births in private institutions, where available data are very limited. Figure 1: Impact of quality of care versus reach of services The other piece of evidence for this analysis is from the data on maternal deaths. Vriddhi’s efforts have improved their reporting to 50 percent of estimated deaths, and most of these deaths happen a) in the home, b) in transit and c) in private facilities. There is likely a larger proportion of in-home and in-transit deaths in the remaining half that are currently unreported. Therefore, while improving the quality of care around birth in public institutions is a very important intervention toward reducing mortality, it alone has limited effects. It is reductionistic to not concomitantly address the barriers to access that continue to assail the poorest, who are farthest from institutions and who bear the highest burden of maternal mortality, as well as the quality of care around birth in private institutions. EQ1.III CONCLUSIONS CaB has been significantly scaled up but its effect on mortality is difficult to attribute. CaB has been scaled up to cover up to 90 percent of all births in the six focal states. While scale-up efforts are led by each state, Vriddhi has been instrumental in developing plans, budgets, and monitoring systems. However, the limited capacity of states and districts for ongoing monitoring and responsive planning is likely to adversely impact the quality of implementation in scale-up locations. Furthermore, the benefits of these service quality improvements on the poorest and the most vulnerable women can only be affected if they are brought into the healthcare system. Thus, the inference that improved quality of service delivery in public facilities alone will result in reduced mortality and morbidity must be made with caution and moderation. EQ2 FINDINGS How actively has the project scaled up the approaches/models/interventions from the five-by-five matrix used by the GoI in the national program? To answer this evaluation question, the ET examined the effectiveness of each of Vriddhi’s pilot interventions and the extent to which they have been scaled up. Community-Based Advance Distribution of Misoprostol: The GoI recommends a single oral dose of misoprostol to prevent postpartum hemorrhage during home births. At the request of the state governments, Vriddhi helped establish the pilot in one block in Mandi district of Himachal Pradesh that had a high rate of home births. The project provided training and mentorship to Accredited Social Health 16 Activists (ASHA) on advance distribution of the tablet to pregnant women, counseling and follow-up, procurement, monitoring, data collection and reporting mechanisms. Vriddhi developed training and communication material and led the rebranding and repackaging of the drug, to include a tracking mechanism and an insert with instructions on its use. The project also adapted GoI guidelines on eligibility criteria to include all pregnant women, because many women in the area who intended to deliver in a facility ultimately delivered at home due to poor accessibility. ASHAs were responsible for tracking total home and institutional deliveries, the number of tablets distributed and consumed, and the number of unused tablets that were returned. The pilot ran for nearly two years, from May 2016 to February 2018, and distributed the drug to 74.5 percent of all registered pregnancies. The rest were not given the drug because of their expressed intention to deliver in facilities. About 83 percent of those who received the tablet and delivered in home or in transit consumed the drug.xiv During the pilot, institutional births increased to 60 percent, from 11 percent during the preceding year. There was also a significant increase in the proportion of registered pregnant women whose outcomes were known. These incidental benefits are likely due to the frequent, high-quality contacts between ASHA and pregnant women. The ET interviewed frontline workers in the pilot site, who expressed appreciation for the hand-holding and increased confidence in their ability to counsel women and their families. Insights into acceptability and consumption of the drug (given that its consumption was not universal among home/transit deliveries) are not available, which would be useful to inform future programming. The model itself could be seen as too sophisticated to fit within the existing state and district health system, because of elements such as the tracking of consumption. The pilot was assessed internally from time to time but has had no formal assessment. Vriddhi is currently supporting the scale-up of this intervention to 11 blocks across two districts, all of which have poorly accessible facilities, and account for 38 percent of all home deliveries in the state. It will be important to document this process and the adaptations made along the way. Vriddhi supported the implementation of this model in Jharkhand but has been hampered by procurement-related challenges. There is a varying appetite at state levels for acceptance and roll out of Misoprostol, which ranges from excitement about potential use to desires for outcome-level data. Homebased Newborn Care (HBNC), Gentamicin, and KMC: Vriddhi conducted a pilot that combines these three evidence-based interventions for newborn survival in one block each in Haridwar district, (Uttarakhand) and Gumla district (Jharkhand). Vriddhi selected these two sites using pre-defined criteria such as weighted ranking of HPDs. The pilot included: training ASHAs and ANMs in the algorithm for Possible Serious Bacterial Infection, establishing referral linkages, supply chain management, data management and use, and training ASHA in promoting KMC in the home. The ET found the pilot to be functioning very well in Gumla. Record keeping was excellent and showed that several newborns born in facilities and had kept the cord stump dry ultimately developed cord stump infection, but these were identified and treated. These three interventions have been included in the Jharkhand NAP, and Vriddhi supported the scale-up training in other districts. But the ET found that they were not being implemented in a sub center in Jama block of Dumka district, where the ANM had been trained, but did not have the needed supplies nor was carrying out the intervention to any extent. Strengthening Antenatal Care (ANC) services: In Punjab, Haryana, Vriddhi supported the development of criteria and reporting formats to enable ANMs to better identify HRPs at sub centers across the state. This resulted in increase in identification of HRPs. In Uttarakhand, Vriddhi supported the design and implementation of the state health department’s Block Level Implementation Strengthening Support (BLISS) in one block of Haridwar district. BLISS includes fixed-day ANC at subcenters, identifying HRPs, supporting families to develop birth plans, improving the visibility of services at sub centers through display material and regular coaching of ANMs at the block-level facility. Encouraged by the increases in ANC uptake, the state government has incorporated this model into the SAMMAN scale-up initiative. 17 Strengthening Urban Health and Nutrition Days (UHNDs): Vriddhi strengthened elements of the UHND related to counseling on nutrition, safe motherhood and providing information on facility-based services, thus helping them deliver the full scope of services. The project provided additional training to ANMs and ASHA and monitored their implementation. This has resulted in improved acceptance of UHND services in communities. ASHAs and ANMs felt the Vriddhi training helped them to “know more about comprehensive care instead of just focusing on immunization” however they admitted to not having enough information about nutrition. Vriddhi staff also use UHNDs to collect client feedback on their birth experiences in the nearby CaB facilities. This support was expanded statewide in January 2018. Reporting and Reviewing of Maternal and Child Deaths: Per GoI regulations, all reported maternal deaths and one-tenth of child deaths are to be investigated using verbal/social autopsy tools and the findings from the investigations are to be reviewed at district and state levels. Historically, these deaths have been chronically underreported. With repeated advocacy to state and district managers, Vriddhi has contributed to an increase in the reporting of maternal deaths to about half of the estimated deaths in 2017-2018. Vriddhi helped district decision makers understand the rationale for maternal death reviews. For child deaths, Vriddhi created guidelines in Haryana to take photos of stillborn babies to check over-reporting of macerated intrauterine deaths and ensure accountability of providers within HPDs. Vriddhi staff was part of the national team that reviewed and updated guidelines for maternal death reviews. In many states and districts, maternal deaths are reviewed regularly and responsive actions are finalized. In others, frequent transfer of officials makes it difficult to maintain this as priority, or data is collected but responsive action is not taken. The providers’ reluctance to report maternal deaths for fear of reprisal and blame from family members or facility management is a significant constraint, and it was noted that providers were generally uncomfortable discussing deaths or were quick to note that mothers with extreme complications are usually referred so as to absolve them of wrongdoing. If deaths do occur, MDRs are often difficult to carry out due to inability to locate patients or family members and duplication of data, as well as a lack of HR to conduct reviews or analyze results in a timely manner. EQ2 CONCLUSIONS Vriddhi chose models and approaches from the 5x5 matrix that complemented its focus on addressing key drivers of mortality and morbidity (the interventions that align with Vriddhi’s activities are highlighted in red in Figure 2). It has piloted those that needed evidence on operational feasibility and has scaled them up to varying extents, based on the absorptive capacity and buy-in of the host states. The pilots were carefully designed, but the level of attention to detail, both in their design and implementation, might not be possible to maintain on a larger scale. The WHO ExpandNetxv approach to scale-up recommends that innovations first be assessed if they have the correct attributes for scale-up. It is therefore imperative to examine the pilots carefully and document the elements of design and operations in order to determine what essential features need to be maintained and supported at scale. Analyzing the trade-offs between the ideal and the feasible will be crucial to the success at scale. “If we take the risk to refer and something happens to the mother in transit or even upon arrival, it’s on us. If we take the risk to not refer her and manage her illness instead and something happens to her, we are also responsible.” - Staff nurse, Haryana 18 Figure 2: Elements from the 5x5 matrix in Vriddhi interventions Reproductive Health Maternal Health Newborn Health Child Health Adolescent Health • Focus on spacing methods, particularly PPIUCD at high caseload facilities • Focus on interval IUCD at all facilities, including subcenters, on fixed days • Home delivery of contraceptives (HDC) and ensuring spacing at birth (ESB) through ASHAs • Ensuring access to pregnancy testing kits (PTKs - "Nischay Kits") and strengthening comprehensive abortion care services • Maintaining quality sterilization services • Use MCTS to ensure early registration of pregnancy and full ANC • Detect high risk pregnancies and line list including severely anemic mothers and ensure appropriate management • Equip delivery points with highly trained HR and ensure equitable access to EmOC services through FRUs; Add MCH wings as per need • Review maternal, infant, and child deaths for corrective actions • Identify villages with high numbers of home deliveries and distribute Misoprostol to selected women in 8th month of pregnancy for consumption during 3rd stage of labor; Incentivize ANMs for home deliveries • Early initiation and exclusive breastfeeding • Home-based newborn care through ASHA • Essential Newborn Care and resuscitation services at all delivery points • Special Newborn Care Units with highly trained human resources and other infrastructure • Community-level use of Gentamicin by ANM • Complementary feeding, IFA supplementation, and focus on nutrition • Diarrhoea management at community level using ORS and Zinc • Management of pneumonia • Full immunization coverage • Rashtriya Bal Swasthya (RBSK): screening of children for 4Ds (birth defects, developmental delays, deficiencies, and disease) and their management • Address teenage pregnancy and increase contraceptive prevalence in adolescents • Introduce community-based services through peer educators • Strengthen ARSH clinics • Roll out National Iron Plus Initiative, including weekly IFA supplementation • Promote menstrual hygiene Health Systems Strengthening Cross-Cutting Interventions • Caseload based deployment of HR at all levels • Ambulances, drugs, diagnostics, reproductive health commodities • Supportive supervision and use of data for monitoring and review, including scorecards based on HMIS • Health education, demand promotion, and behavior change communication • Public grievances redressal mechanism; client satisfaction and patient safety through all around quality assurance • Bring down out of pocket expenses by ensuring JSSK, RBSK, and other free entitlements • ANMs and Nurses to provide specialized and quality care to pregnant women and children • Address social determinants of health through convergence • Focus on underserved villages, urban slums, and blocks • Introduce difficult area and performance-based incentives EQ2.I FINDINGS What have been the results of the project’s approaches on increased off-take of the RMNCH+A services in High Priority Districts? Vriddhi’s theory of change rested on the assumption that improving the availability and quality of obstetric and newborn care (Output 1), along with other supply-side interventions, will lead to increased uptake of RMNCH+A services (Outcome/Objective). However, the project’s design and implementation principally focused on addressing the quality of intra- and postpartum care in facilities through the CaB package. As there were no major interventions to improve service availability, it is evident that the express outcome of interest of the project was to improve the quality of facility-based services. The project’s outcome-level indicators are measures of facility performance and not population-level indicators, which is appropriate for this set of outcomes. USAID and IP staff clearly expressed that the supply-side focus on improving quality of services was intentional, along with the decision not to focus on demand generation or improving availability or accessibility of obstetric and newborn care services. KIIs with MOHFW leadership revealed that Vriddhi was born out of their request for technical assistance for the implementation of the RMNCH+A strategy. The bottleneck analysis presented in the RMNCH+A strategy, however, identifies a range of demand- and supply-side barriers and avenues for addressing them. It therefore appears that the technical assistance was deemed necessary only for the service quality aspects of the RMNCH+A strategy, and not for addressing other barriers. The ET also found among MOHFW respondents the assumption that improving quality of services will lead to increased population-level uptake of those services. DPs, on the other hand, clearly stated that while service quality improvement is 19 critical, one cannot expect uptake to improve as a result of supply-side interventions alone, and expectations/indicators must be adjusted accordingly. Literature shows that a range of demand- and supply-side barriers, both monetary and non-monetary, limit the utilization of health services. xviii xvi Results from the Ballabgarh Health and Demographic Surveillance site shows that improving the financial and geographical accessibility of services led to an increase in service uptake.xvii Vriddhi’s own experience in the BLISS pilot in Haridwar and in the AFHC clinics shows that increased availability and visibility of services leads to improved uptake. An MNCH Working Paper (2013) on District Health System Strengthening from UNICEF shows that access barriers limit uptake, along with supply-side barriers such as poor quality. Interestingly, the RMNCH+A Strategy and the UNICEF report (mentioned above) utilize the Tanahashi model xix of health systems bottlenecks, which clearly shows that issues related to availability and accessibility of service points need to be addressed first, to increase the target beneficiaries that are brought into the system, so that quality improvements can have an effect on them. Figure 3: WHO Tanahashi Model Thus, interventions focused exclusively on improving the quality of services are unlikely to result in an increased uptake of the services, unless they are combined with efforts to improve the availability and visibility of services and efforts to address context-specific geographic and financial access barriers, and acceptability issues. Even the RMNCH+A Strategy and in the five-by-five matrix do not have a well thought out strategy for addressing context-specific barriers to access and community and household level behavior change, but are heavily oriented toward improving facility-based services. The ET reviewed the completeness of data on institutional births in the national HMIS, comparing the data with the population-based estimates of births that program managers used. Although the HMIS includes data from public and private facilities and communities, district program managers reported that the database did not capture up to one-quarter of the estimated births in HPDs. The ET therefore concluded that the HMIS is not a reliable source of data to assess changes in the uptake of services. The ET also noted that across all states, facilities did not collect or analyze client feedback on a regular basis to determine client perceptions of improvements in the quality of care, which can serve as a proxy for improved uptake of services. Client satisfaction has largely been omitted from Vriddhi’s activities and 20 metrics, which also affects the ability to discern exactly what the client experience was and what motivated clients to choose to receive services at facilities. As one Vriddhi staff in Himachal Pradesh noted, “It is difficult to measure respect with a suggestion box. The client or patient should actually be involved but they are not involved in the QI process currently.” EQ2.I CONCLUSIONS The project’s overall outcomes, as stated in its theory of change, are incongruous with its express goalpost of improving the quality of services, especially for obstetric and newborn care. The technical assistance provided under the RMNCH+A strategy is synonymous with supply-side strengthening. The successful improvements that Vriddhi brought about in the quality of obstetric and newborn care are unlikely to have a direct impact on the uptake of these services, and any relationship between the two is tenuous at best, and its expectations need to be moderated accordingly. Barriers related to financial, geographical and socio-cultural accessibility must be addressed, along with supply-side interventions, to improve the uptake of services. The perception of some stakeholders that improved quality of services will result in an increase in uptake of those services does not appear to be grounded in evidence. EQ2.II FINDINGS What have been the results of the project’s approaches on technical assistance for improved program planning, monitoring, and implementing at the district and state levels? Program Planning: In coordination with the NRU discussed in EQ1.I, Vriddhi leads state RMNCH+A units (SRU) that review the current year’s performance and gaps identified by the supportive supervision visits. The SRUs also work with state managers to incorporate responsive measures in state program implementation plans (PIPs), revising and optimizing the health action plans the districts submit. This support extends to non-HPDs, where Vriddhi does directly implement programming. Vriddhi’s State Technical Leads provide close support to the presentation of state plans at the national level, where they are supported also by Vriddhi national staff. They also support the drafting of new initiatives in the form of proposals, to be submitted at the national level, including scale-up plans for CaB beyond Vriddhi locations, such as the SAMMAN initiative in Uttarakhand. Vriddhi teams in Haryana and Jharkhand led the development of costed NAPs for the state. Vriddhi advocates with state level first and then worked through them to reach districts for issues that needed directive and reassurance from the state level. Vriddhi staff and other DPs reported that district teams lack the capacity to identify gaps and turn them into operational plans and that “the micro-level of making it happen on the ground is not there.” HR deficiencies (see EQ3 for further discussion) are major contributors to this lack of capacity. District officials, on the other hand, explained that Vriddhi supports them in identifying and presenting district needs in the health action plans, though they are not always accepted at the state level. Program Implementation and Monitoring: Vriddhi staff organized workshops at state levels for state and district managers to foster better understanding of the details in the annual plans. The ET found that state-level managers and officials were generally unable to carry out supportive supervision visits due to other urgent priorities and administrative work. Additionally, the budgets proposed for field visits are often cut at the national level based on historical spending data (i.e., if state officials were unable to conduct supportive supervision visits in the previous year because of time constraints, their budget for supportive supervision visits in the next year would be reduced), which can perpetuate a cycle of underperformance. As a result, key informants explained that Vriddhi staff from all levels conduct most supportive supervision visits. Vriddhi has worked with district and facility managers to identify creative solutions for local procurement of supplies and equipment required for implementing CaB and other interventions. They encouraged and facilitated joint supervision visits with the district managers and civil surgeons, although these were rare, due to time constraints. Vriddhi staff at the district level also advocated with district managers to include 21 government-approved performance-based incentives for nurses and ASHAs in the district action plans. One district incorporated pay raises for contractual nurses, both of which boosted staff morale and brought these issues to the attention of district managers. Vriddhi staff shared anecdotes of ASHA and ANM counseling women more consistently to accept PPIUCDs as a result of these incentives. State level officials and program managers reported learning the process of gap analysis and budgeting appropriate responses to address the gaps. They also expressed new knowledge of the inputs required for planning and budgeting of Cab and other interventions. However, given the minimal time expended on monitoring and supportive supervision visits, it is unlikely that there will be ongoing identification of gaps and responsive planning. Interviews with project and government staff illustrate a situation in which project staff actually conduct the majority of strategic project implementation, with less focus on building the capacity of state and district program officers and medical officials to do the work. The availability and commitment from government officials, in particular, is a significant barrier to this – without a willingness from government officials to take the time to have their capacity built, it is virtually impossible to do so. Relatedly, decision￾makers, managers, and providers at all levels expressed strong appreciation for the extra manpower Vriddhi staff provide, viewing them as “part of the team,” which is not conducive to seeing their input as time-bound and using it to strategically build competencies in their own teams. Government officials, in particular, expressed expecting ongoing indefinite support from a DP (USAID or another) alongside their teams. EQ2.II CONCLUSIONS Vriddhi’s involvement in supporting states and districts in planning, implementation and monitoring was critical and will leave a gap when the project ends. In general, state-level teams lack the time and district￾level teams lack the competencies to assess gaps and plan with the level of detail needed to implement complex programs. The decentralized planning and budgeting that was envisioned in the National Health Mission has not taken off. Overall, the ET found that while state and district officials deeply appreciate Vriddhi’s input, they do not see these as strategic opportunities to develop in-house competencies. These attitudes are unlikely to change unless the underlying, perennial HR constraints are addressed. EQ3 FINDINGS AND CONCLUSIONS What key achievements and challenges helped/hindered the outcomes of the project? KEY ACHIEVEMENTS CaB resulted in improved facility performance, evidenced by documented increases in the delivery of the high-impact interventions over time. Although this increase is less pronounced in facilities with very high caseloads and staff shortages, there is evidence of improved management of complications. Vriddhi’s CaB intervention has seen significant scale-up, covering up to 90 percent of all births in the six states. While the scale-up effort is led by the respective states, Vriddhi has been instrumental in developing plans, budgets, and monitoring systems. QI teams and facility mentorship have the potential to continue quality improvements after project closure. However, Vriddhi continues to provide close support, and the sustainability of these interventions in the absence of this support has not yet been tested. RMC has the potential to improve patient experience of care, and over time, improve the uptake of services, especially when combined with appropriate demand-side interventions. Vriddhi enhanced the technical competencies of a large cohort of service providers and built the capacity of high caseload facilities to deliver a set of high-impact interventions from the RMNCH+A 5x5 matrix. In the process, it demonstrated to district teams what it takes to manage the delivery of these services with quality. The project also brought RMC and facility and block-level mentorship into mainstream conversations on quality of care. 22 Vriddhi gave states the flexibility to choose areas of focus that aligned with their priorities for RMNCH+A, with CaB at the center. The credibility of project staff contributed to successful advocacy for the uptake of project interventions, and several government representatives said they embraced them as technical support and not “fault-finders,” who were responsive to requests at the district and state level. Project support to campaigns was timely, effective, and lent credibility. There was strong coordination among DPs with minimal activity overlap. The NRU played a key role coordinating the efforts of all DPs, ensuring complementary collaboration. The project generated a significant amount of high-level data that can be used for decision making during the remainder of the project and in the future. KEY CHALLENGES Perennial HR shortages limit uptake and impact: Key informants nearly universally agreed that staffing of state and district health teams and facilities is a major challenge. Almost all project locations face perennial HR deficiencies, with many vacant positions among managers and service providers across states and districts. The vacant positions create an increased workload on current teams or result in work being left undone. High-burden locations hold little incentive for attracting and retaining talent and are often referred to as “punishment posts.” Staff rotations and transfers dilute capacity building efforts. The potential of the facility mentorship initiated under CaB is threatened as the mentors themselves are overworked and are unable to spend time mentoring new providers. ANMs (trained in skilled birth attendance) are moved from subcenters to facilities to deal with shortages, which adversely impacts frontline work. HR deficiencies also result in distribution of providers that are not optimized to the level of facility or the scope of its services. Government officials lack key managerial competencies: Staff in managerial positions in states and districts lack the required training and skills to effectively manage large-scale programs. Those with training in public health and management are not optimally utilized but are drawn into administrative tasks. Rotations and frequent transfers of managers and officials give rise to the need to repeatedly invest in building these relationships. Changes in state leadership also influence or shift activity and technical support priorities. Gaps in inter-departmental communications thwart clear identification of priorities. There was a lack of communication noted in several states between Vriddhi staff and state governments of the project’s mandate and the technical support it could provide. This often resulted in staff doing work beyond the project’s mandate, demonstrating a need for improved relationship building and project clarity with the government. Vriddhi’s scope was too broad, limiting overall project impact. Addressing every aspect of RMNCH+A is an enormous task. Despite the value of designing health programs holistically, Vriddhi’s broad-reaching scope to design and scale up CaB, along with interventions in AH, FP, and PSE, was overly ambitious. Any one of these components could be a standalone project in itself, requiring considerable resources and strategy to be effectively addressed. Both government and Vriddhi staff agreed that with the focus on CaB, programmatic attention to the other technical areas was imbalanced or lacked sufficient investment (both time and resources) to be fully effective. Many key informants noted that adolescent programming was overshadowed by MNCH, and AH and PSE require separate projects with more directed focus. Due in part to the breadth of Vriddhi’s scope, project staff often split time among numerous initiatives simultaneously and the project’s commitment to being responsive to all government requests limited time for adequate learning and feedback. Some state program officers were eager to engage staff in ‘scale-up’ without identifying expectations, reflecting on lessons learned, or adequately addressing quality control concerns within districts. As one state program officer noted, “…we should instead focus on doing a limited number of things well in a limited number of places.” 23 Infrastructure shortfalls and equipment shortages constrain the ability to provide care. Shortages were reported in access to blood banks, water, electricity, bed space, room space, and CaB training equipment, like mannequins. Additionally, supply chain issues and coordination challenges at the periphery created challenges for maintenance of essential medicines or other supplies. While HR and infrastructure gaps fell outside of Vriddhi’s direct mandate, these challenges significantly limited what Vriddhi could achieve. Vriddhi lacked a strategic, coordinated response to address these gaps or how it would achieve its objectives within these constraints, which was a missed opportunity that may threaten the overall impact and sustainability of the project. EQ4 FINDINGS AND CONCLUSIONS What lessons can be drawn from this program in terms of key strategic approaches and impacts that should inform USAID’s future health interventions? Project timelines did not realistically align with the achievement of project objectives. Projects such as Vriddhi bear a double burden of proving the feasibility and effectiveness of a core approach (including changes in facility performance or population-level indicators) while building in-house competencies. As this is challenging to achieve within typical project timelines, the balance tilts in favor of building the needed technical competencies that will provide proof of concept for the core intervention, as was the case in Vriddhi. Improving quality of services does not directly impact uptake of services. Efforts to improve quality of facility-based services are a critical body of work, but they need to be combined with efforts to address the huge inequalities in accessing facility-based care, to improve the uptake of services. The highest burden of maternal mortality rests on the poorest people, who also are often the furthest from care. Barriers related to financial, geographical and sociocultural accessibility need to be addressed for these groups, along with supply-side interventions, for quality improvement initiatives to impact mortality and morbidity. The perception by some stakeholders that improved quality of services will result directly in an increase in uptake of those services is not grounded in evidence. Development partners are best placed to provide this clarity. Also, projects that are mandated to improve service quality should not be expected to improve uptake and reduce mortality. National HMIS does not capture the entire population. Despite ongoing efforts, capturing data from private facilities and communities on health outcomes and service utilization continues to be a challenge and there are no mechanisms in place to assess the completeness. HMIS data are therefore not a reflection of population-level changes, especially for those segments that are beyond the reach of routine services. Population-level measures will be needed to accurately assess gaps and plan responsive measures. CONSIDERATIONS FOR SUSTAINABILITY Managerial competencies for service delivery: As noted earlier, the focus on building technical competencies and proving the feasibility of CaB came at the expense of intentional development of managerial competencies to oversee the delivery of the complex services. This will require additional time and concerted effort to help facilities assume the day-to-day operational details of delivering CaB and other interventions. Training: Additional training for staff in HPD CaB facilities will be needed, as well as training of new staff in new facilities. Staff will require refresher training on CaB, along with training on utilization of key maternal and newborn health equipment like ultrasounds, radiant warmers and diagnostic kits, to fully utilize resources available in facilities. Institutional Capacity: Even though there have been select improvements in institutional capacity, government officials largely view Vriddhi as providing hands and feet to get the work done, and the government is not empowered (in terms of skills developed) or available (in terms of time constraints and HR) to do the work on its own in the long-term. Sustained behavior change will require regular monitoring 24 and provision of feedback to both providers and staff and development of long-term accountability measures. This has been noted as an area of weakness at both district and state level. Government entities are reliant on DPs to provide this level of support through all stages in the project cycle. Vriddhi staff appear to hold the prime responsibility of doing the work and a clear transfer of skills and knowledge to those held accountable in government is not apparent. Data management, monitoring and reporting: There is a concern about the continuation of monitoring, data management, feedback and reporting loops and supportive supervision after the end of the project, despite the production and roll-out of checklists, SOPs and guidelines by the project. The teams at state and district levels noted their inability to collect, analyze, and use data, due to their time constraints. Vriddhi did provide support in this area, but mainly did data management on its own. The likelihood that the GoI will maintain this level of data collection and analysis is low. The ability to use data for decision making dramatically improved under the project and the data generated during the last few years can impact the development of future strategy and programming. However, the ability of the state teams to have the time and resources to continue collecting and utilizing data, especially for future PIPs is noted as a challenge in the absence of Vriddhi. There were strong feelings among almost all government and Vriddhi respondents that it is premature to withdraw project support at this time, with some districts feeling they need additional support, and others feeling as if they have reached their stride and fear interrupting their momentum. V. RECOMMENDATIONS RECOMMENDATIONS FOR USAID 1. Limit the scope of initiatives with lofty but time-bound goals to improve population health outcomes, acknowledging where technical areas are complementary but where activities and staff are not overstretched, while being fully responsive to government priorities. Instead, focus on prioritizing a limited number of key technical areas where activities have sufficient feedback loops and opportunities for learning and continuous improvements. 2. Align expectations of initiatives such as Vriddhi and LaQshya that have the sole mandate of supply￾side strengthening, given that context-sensitive combinations of interventions that address demand and supply-side barriers will be needed to impact population level uptake and health outcomes. 3. Support integrated health programs that address both supply- and demand-side issues and which speak to and learn from each other. Build national capacity to generate local evidence that identify issues impacting the availability of services, geographic and financial accessibility, and acceptability of services, and design context-specific interventions to address them. 4. Build national capacity for stronger measurement strategies, such as effective coverage, a quality￾corrected measure of population-level outcomes, as a metric for monitoring health system improvements at the population level. This combines need, use, and quality, and has been increasingly advocated as a measure of Universal Health Coverage.xx 5. Prioritize strategic support to states in targeting adolescent health improvements, collecting end user and beneficiary feedback more systematically, and utilizing private sector cells within government. 6. Utilize the private sector landscape assessment to inform a scaled-up strategy for the GoI at national and state levels under a dedicated PSE project and with a more substantial timeline. 7. Coordinate with other development partners to bring together a diverse, inter-sectoral team to develop a long-term and creative set of solutions to the perennial, complex issues with HR at program management, facility and outreach levels, which existing policies/projects have failed to address effectively. RECOMMENDATIONS FOR VRIDDHI (NATIONAL) 25 8. Ensure that lessons from CaB implementation and the initial scale-up efforts feed into LaQshya PMU through experience-sharing workshops and site visits. 9. Continue support for the adoption of KMC in the scale-up phase of CaB in each state, with a focus on clarifying facility staff interpretation of guidelines, especially in facilities without SNCUs, deemphasizing dependence on equipment and supplies, and emphasizing the importance of continued KMC practice in the home. 10. Work with national and state health teams to increase the uptake and use of the RMNCH+A App and Health Atlas. 11. Incorporate recent developments in performance standards for RMC and their measurement in CaB scale-up locations. Experiential learning and data from Vriddhi will contribute to this evolving field. 12. Incorporate metrics around client satisfaction in ongoing interventions to include feedback on facility infrastructure, treatment by medical staff within the facility, timeliness of care, cleanliness, availability of space, staff communication during delivery with mothers and families, and information provided regarding follow-up after leaving the facility. RECOMMENDATIONS FOR VRIDDHI (STATE AND DISTRICT) 13. Continue supporting states to understand the nuances of operationalizing their unique scale-up plans. Invest in detailed documentation of “what it takes” to operationalize CaB, Advance Distribution of Misoprostol and Community-based Management of Newborn Sepsis, and other interventions in disaggregating information by state, district and facility, to shine the spotlight on both the strategic and day-to-day operational details in which Vriddhi has excelled. 14. Support scale-up efforts by titrating the level of support (training, monitoring, assistance to planning and budgeting for supplies) to the current capacity of states and districts to absorb the interventions in CaB scale-up and NAP implementation. This also includes refining the mechanisms for client feedback and developing long-term accountability measures for providers and staff. 15. Dedicate time and resources to invest in building capacity for strategic planning in state governments, using ongoing feedback from the current implementation around monitoring and reporting and utilizing data for decision-making, and also feed this information into efforts that address HR issues. 16. Ensure that key training focal persons have been identified within state and district governments who are committed to train new staff in new facilities and provide refresher training for those already trained in CaB and in utilization of key lifesaving equipment for mothers and newborns. 17. Clearly communicate the project’s mandate (and its limitations) to district and state government. This communication should include identifying a consistent communications mechanism between staff and government counterparts for requesting and providing technical support within the project’s mandate. 18. Regulate the age-appropriate peer education model across all states and emphasize community level information campaigns to reach male and female adolescents where they feel most comfortable and less stigmatized, especially in discussing SRH. 19. Ensure the incentivization of ASHAs and service providers for providing counseling for PPFP and provision of a basket of choices. 20. Generate state and district-wide initiatives to capture and analyze data of both in- and out- of school youth in order to inform a comprehensive adolescent health strategy for initiatives to target the key identified areas where adolescents are in need of information. 21. Establish state government focal points as champions to initiate a full-fledged private sector development strategy and fully implement it, providing a clear engagement and communication channel between public and private actors. Relationship building takes time and must be sustained in order to keep private sector interests. 26 ANNEX I. EVALUATION STATEMENT OF WORK Mid-term Performance Evaluation of the Project: Scaling Up Interventions in Reproductive, Maternal, Neonatal, Child and Adolescent Health (RHMNCH+A) I. PROGRAM INFORMATION a. Program Title: Scaling Up Interventions in Reproductive Health, Maternal, Neonatal, Child and Adolescent Health (RHMNCH+A) b. Start-End Dates: May 26, 2014 – May 25, 2018 c. Budget: $24,885,399 million d. Program Description: The purpose of USAID/India’s assistance is to support Government of India (GoI) and private sector to scale up proven high-impact RMNCH+A interventions to reduce morbidity and mortality in support of India's efforts towards achieving the then Millennium Development Goals (MDGs 4 and 5) and now Sustainable Development Goal (SDG) 3. Reproductive, maternal, neonatal, child plus adolescent health are a key priority area that encompasses the continuum of care for child survival services, including newborn and child health, maternal and reproductive health. In the area of RMNCH+A, the focus is on prioritizing support to achieve substantial impact on child survival, in support of the Call to Action for Child Survival, announced by the Governments of the U.S., India, and Ethiopia, and UNICEF in June 2012. The overarching purpose of the project is to scale up key high impact interventions for each thematic area as summarized in 5x5 matrix23 by GoI under RMNCH+A strategy. These form the core areas of involvement for the project. The program has five major objectives: under RMNCH+A strategy. These form the core areas of involvement for the project. The program has five major objectives: • Objective 1: Increase off-take of RMNCH+A services in High Priority Districts (HPD) • Objective 2: Involve Multiple Stakeholders (Including Private Players) • Objective 3: Support Monitoring and address bottlenecks for RMNCH+A service delivery • Objective 4: Enhance Capacity of States and Districts to provide quality RMNCH+A services • Objective 5: Promote, Pilot, learn from and support scaling up of RMNCH+A good practices Since the introduction of the National Reproductive Health (NRHM) in 2007, there have been significant improvement in national-level health indicators; however, gains vary widely across states. To prioritize and focus its efforts, the GoI identified 184 High Priority Districts (HPDs) across all states for special interventions under a comprehensive RMNCH+A Strategy, which is aimed at improving maternal and child health indicators in these districts under an integrated program approach that includes service delivery in various life stages including the adolescence, pre-pregnancy , childbirth and postnatal period, childhood and through reproductive age The RMNCH+A strategic approach has been developed to provide an understanding of ‘continuum of care’ to ensure equal focus on various life stages. The RMNCH+A appropriately directs the States to focus their efforts on the most vulnerable population and disadvantaged groups in the country. It also emphasizes on the need to reinforce efforts in those poor performing districts that have already been identified as the high focus districts. To harmonize the support 33 The RMNCH+A 5 x 5 Matrix identifies five high-impact interventions across each of the five thematic areas, five cross-cutting and health systems strengthening interventions, and, the minimum essential commodities across each of the thematic areas. The 5 x 5 Matrix is an important tool for explaining the strategy in simple terms, organizing technical support, and monitoring progress with the states and high-priority districts. 27 as well as resources from various partner organizations, GoI has also designated different donors and District Partners (DPs) to lead the partners’ efforts in various states and HPDs in respective states. Under RMNCH+A strategy, GoI has identified key high impact interventions to improve maternal and child health indicators. The intent of this project is to provide technical and managerial assistance to GoI in implementing the RMNCH+A Strategy at the national level and in 30 * HPDs of the 6 states where USAID is the lead partner. These states are Jharkhand, Uttarakhand, Punjab, Haryana, Himachal Pradesh and Delhi. To achieve sustainable scale-up, the project work with government systems at various levels and also prepares and engages medical colleges, private sector and civil society organizations. The project has established a National Resource Unit (NRU) to provide technical and managerial support to government entities at national, state and district levels to enhance institutional capacity of governments to scale-up proven, high impact RMNCH+A interventions. The project support ensures that all interventions are systematically conceptualized with scaling up being an integral part of the planning from the beginning. The project collaborates with governments at various levels to identifying bottlenecks in implementation of key interventions and overcome them for effective implementation. The project has a technical team at national level to provide technical support to Ministry of Health and Family Welfare (MOHFW) and backstop project teams in six states. It also set up a State Resource Unit and a district-level coordinator in each of 30 high priority districts. However, the project is not intended to limit its efforts to the 6 states and 30 HPDs only. From the national level, the project team engages government and other stakeholders and helps create opportunities and platforms for sharing and cross￾learning. The project team provides support to governments in other states through respective lead partners, especially in high-burden states of Uttar Pradesh, Bihar, Madhya Pradesh, Rajasthan, Odisha etc., in adapting innovative interventions and scaling them up in respective states. The team at national level contributes towards capacity building of state program managers from all states. As the prime recipient (PR) of this project grant – IPE Global Private Limited is also working in two other states, Bihar and Odisha, through other grants. Learning experience and good practices from this project are also shared to help those two states by leveraging resources from those grants in respective states. Similarly, in the state, although, district-level coordinators are placed only in HPDs but, the project ensures that learning from these districts is incorporated in the state plan for other districts as well. State Resource Units (SRUs) are set up and involved in state-level capacity building of staff from other districts and monitoring of their performance. The project realize its purpose by prioritizing scale up of evidence based interventions to address the major drivers of maternal and child mortality, by tracking quality of services in high case-load delivery points, contextualizing and adapting identified global, national and local best practices to state specific scenarios with scale in mind, developing innovative catalytic models that can be scaled up at state and national levels, and engaging multiple stakeholders in order to achieve scale. II. TASK ORDER STATEMENT OF WORK a. Evaluation Purpose: The purpose of the mid-term performance evaluation is to: i. Assess the results achieved and overall progress made by the project against its objectives and specifically on the performance indicators established for the project. ii. Evaluate the key strategies and approaches adopted by the project in scaling up its efforts, and develop recommendations for the remainder of the project. iii. Provide insights and lessons learned from various components of the project that will help inform USAID/India’s future designs in RHMNCH+A health sector. 28 b. Evaluation Questions: This evaluation will answer the following questions: 1. To what extent the project has achieved the objectives and expected results in terms of enhanced institutional capacity of GoI and private sector networks to scale up proven high-impact RMNCH+A interventions? i. To what extent has the program been successful in influencing various GoI efforts on improving RMNCH+A specifically through the support of National RMNCH+A Unit? ii. To what extent has the program engaged private providers or stakeholders in delivering RMNCH+A services and what are the effectiveness and results of this engagement? iii. To what extent have the results of this project contributed to the scale up effective RMNCH+A solutions to achieve significant reductions in preventable morbidity and mortality among women and children? 2. How effectively has the project scaled up the approaches/models interventions from the five by five matrix used by the GoI in the national program? In particular, the evaluation shall address the following sub-questions. i. What have been the results (effectiveness) of the program’s key approaches on: ii. increased off-take of the RMNCH+ A services in High Priority Districts, and iii. technical assistance for improved program planning, monitoring, and implementing at the district and state levels? 3. What are the key achievements and challenges that helped/hindered the outcome of the project? 4. What lessons can be drawn from this program in terms of key strategic approaches and impact that should inform USAID’s future health interventions? III. EVALUATION DESIGN AND METHODOLOGY The evaluation will cover all the initiatives undertaken in (RHMNCH+A) project since 2014. Evaluation methods, including information on evaluation design and data collection, are presented below. The evaluation team will need to present a well- thought methodology for answering the evaluation questions while keeping in mind USAID’s evaluation policy and guidance. a) Evaluation Design This performance evaluation will, to the extent possible, adhere to the USAID Evaluation Policy (http://www.usaid.gov/evaluation)http://www.usaid.gov/evaluation)evaluation guidelines for more rigorous evaluation, using mixed methods that incorporate both quantitative and qualitative methods. . The evaluation should utilize mixed methods that include both qualitative and quantitative data collection. b) Gender Compliance In compliance with Revised ADS 205 released in 2016 on Integrating Gender Equality and Female Empowerment in USAID’s Program Cycle, ADS 201 on program Cycle Operational Policy (2016), Gender Equality and Female Empowerment Policy (2012), and Gender Sensitive Evaluation (2014) – best and promising Practices in engendering evaluation; the evaluation will be gender responsive. The evaluation will address/ respond to gender-sensitive indicators, sex-dis-aggregated data and attention to gender inter￾relations are required elements of USAID evaluations as highlighted in http://pdf.usaid.gov/pdf_docs/PA00K43P.pdf. c) Data collection and analysis 29 The contractor is required to complete the following table and submit it as part of their technical proposal. Data Collection and Analysis Matrix Evaluation Question Data Source Data Collection Methods/Sampling Data Analysis Methods 1. To what extent has the project achieved the objectives and expected results in terms of enhanced institutional capacity of GoI and private sector networks to scale up proven high impact RMNCH+A interventions? • Project documents • IPE Global project staff • USAID/India • Vriddhi project staff • JSI project staff • GoI (MOHFW/ NRU/SRU) • Health facility staff • Private sector/Academic partner staff • Qualitative/ Quantitative desk review • KIIs - Project staff (IP, Partners and USAID), GoI staff, provider-level staff • Qualitative: Content analysis for identifying project successes and challenges, thematic organization and coding • Quantitative: Summary and descriptive statistics, cross￾verification 2. How effectively has the project scaled up the activities/model/ interventions from the five by five matrix used by the GoI in the national program? • Project documents • IPE Global project staff • USAID/India • Vriddhi project staff • JSI project staff • GoI (MOHFW/ • Qualitative/ Quantitative desk review • KIIs - Project staff (IP, Partners and USAID), GoI staff, provider level staff • Qualitative: Content analysis for identifying project successes and challenges, thematic organization and coding • Quantitative: 3. What are the key achievements and challenges that helped/hindered the outcome of the project? • Project documents • IPE Global project staff • USAID/India • Vriddhi project staff • JSI project staff • GoI (MOHFW/ NRU/SRU) • Health facility staff • Private sector/Academic partner staff • Qualitative desk review • KIIs - Project staff (IP, Partners and USAID), GoI staff, provider-level staff • Qualitative: Content analysis for identifying project successes and challenges, thematic organization and coding • Quantitative: summary statistics • Validation of log frame assumptions Should the evaluation team deem it necessary to collect quantitative data using a sample survey, the evaluation team will need to include a section in the evaluation plan that clearly depicts how the survey will be conducted, the sample frame to be used, sample size, quality assurance, data analysis plan, etc. The evaluation team will review documentation provided by USAID and the RMNCH+A implementing partner, and any relevant research they collect. An instrument will be developed to codify and organize 30 data from the document review for analysis according to the evaluation questions. The evaluation team will be accountable for ensuring data analysis methods are in line with best practices. For both quantitative and qualitative data, the evaluation team will need to articulate methodologies for analyzing collected information, including any qualitative data analysis software programs to be used. For qualitative data specifically, the evaluation team will need to ensure that data are recorded and transcribed. It is expected that the evaluation team will present initial findings from the document review against the evaluation questions as part of the Team Planning Meeting at the beginning of the evaluation. Desk review of documents: USAID/India will provide the team with all relevant country and program specific documents including proposals, Project Monitoring & Evaluation plan (PMEP), progress reports, monitoring indicators data and other relevant documents for conducting this desk review. The evaluation team is expected to collect and collate relevant documents, reports, and data, and all team members are expected to review these documents in preparation of the evaluation design and for the team planning meeting. Specific Tasks Specific tasks to be undertaken by the evaluation team in carrying out the evaluation include, but not limited to: • Review of the program’s Contract documents. • Review of all program reports and annual work plans. • Review of baseline data, Project Monitoring & Evaluation Plan (PMEP), targets and performance reports as provided in the quarterly reports. • Review of USAID/India’s Country Development Cooperation Strategy, the relevant Development Objective and RMANCH+A’s role therein. • An in brief and out brief with USAID/India’s Mission Director, Health, Program Office, and USAID’s Agreement Officer’s Representative (AOR) of the RMNCH+A activity and other related USG representatives. • Meetings and interviews with IPE Global (the implementing Partner). • Meetings and interviews with the GoI and State Government counterparts and partners. • Meetings and interviews with associated institutions and other relevant stakeholders associated in the program in each region of the country. • Further information includes Project Publications: Technical Updates (informs and updates staff in the field on technical areas of RMNCH+A), RMNCH+A bulletin (chronicles RMNCH+A events, news, opportunities and provide relevant research summaries) and Promising Practice, and IPE Global quarterly and annual project reports. d) Reporting Requirements and Deliverables The following are deliverables required of the Contractor under this Task Order: • Initial meeting of USAID/India’s COR of the Evaluation contract and USAID/India’s technical office’s AOR of the RMNCH+A and others to discuss about the evaluation questions, methodologies, and management. In addition, logistical details such as number and location of data collection and meeting scheduling, etc. will be discussed. Team Planning Meeting (TPM): A one-day team planning meeting will be held by the evaluation team at a convenient place in New Delhi before the field data collection begins. This will be facilitated by the evaluation team leader and will provide USAID/India with an opportunity to present the purpose, expectations and agenda of the assignment. The evaluation team will provide to USAID/India’s technical and Program Office an evaluation design Plan which will include a detailed work plan, a projected timeline 31 and preliminary itinerary for the site visits, a detailed description of the evaluation methodology and data collection and analysis methods which will be used (including draft data collection instruments). This evaluation design will be reviewed and approved by the COR within 5 business days. In addition, the TPM will also: • Clarify team members’ roles and responsibilities • Establish the timeline, share experiences and firm up the evaluation methodology • Finalize the methodology guidelines including tools and questionnaires for the filed survey. The evaluation team may commence data collection immediately following the TPM, after they receive a written approval of the evaluation design plan from the COR. • Site Visits for data collections: The evaluation team will conduct a thorough review of the Program through site visits and collect data as planned and approved by the evaluation COR. Site visits will be planned taking into consideration factors like geographical diversity, representation of various implementation agencies, and the scale of the interventions. • Mid-term Briefings. The Evaluation Team Leader will have a discussion remotely with the COR and representatives from the technical offices and Program Offices to provide an update on status, key issues affecting project implementation, and any initial findings/tentative conclusions. In addition, the team will provide to the COR bi-weekly email updates on status and key issues. • Debrief Presentation: The evaluation team will make a PowerPoint presentation of preliminary findings and conclusions of the evaluation to USAID/India and key stakeholders prior to departing from India • Draft Report: The contractor will submit a draft written report in English within fourteen working days of the Presentation. The report should clearly describe findings, conclusions, and recommendations, and should incorporate comments and questions raised during the Presentation. An electronic version of the report will be provided to the COR for dissemination among relevant Mission staff, implementing partners and stakeholders for review and comment. USAID will provide consolidated comments on the draft report within two weeks of submission. • Submission of all raw and processed data, which becomes the property of USAID/India. • Final Report (due within seven working days after receiving written comments from USAID). The team will submit a Final Report in English that appropriately incorporates and/or addresses all comments and feedback. See below for an outline of the final report. • Both an electronic version and five copies of a written version will be provided to the COR for dissemination among relevant Mission staff and stakeholders. The evaluation COR will submit one electronic copy of the Final Report to the Development Experience Clearinghouse at http://dec.usaid.govhttp://dec.usaid.gov after final approval. • The contractor will submit all raw data sets (quantitative and coded qualitative data) to Development Data Library as per the requirements and guidance provided in ADS 579. The Final Report will have the following contents: • Table of Contents (1 page); • Executive Summary – concisely state the most salient findings and recommendations (2 pages); • Introduction – Purpose, audience, and synopsis of task (1 page); • Background – Brief overview of development context and problem, USAID strategy and activities implemented in response to the problem, purpose of the evaluation (2-3 pages); • Methodology – Describe evaluation methods, including constraints and gaps as well as ways 32 to address or minimize their impact on the evaluation (1 page); • Findings/Conclusions/Recommendations – (8-10 pages); • Success Stories – Individual success stories (2-3) which illustrate how USAID project activities have improved lives of people at the bottom of the pyramid that can be added in the Annex. • Annex – Document the evaluation methods; schedules; bibliography of documents reviewed; list of respondents, and SOW - all materials should be succinct, relevant and readable. e) Criteria to ensure the quality of the evaluation report (Please refer USAID Evaluation Policy & How￾TO NOTE- Preparing Evaluation Reports) • The evaluation report should represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not, and why. • Evaluation reports shall address all evaluation questions included in the scope of work. • The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline need to be agreed upon in writing by the COR. • Evaluation methodology shall be explained in detail, and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report. • Evaluation findings will assess outcomes and impact on males and females. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay or the compilation of people's opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be supported by a conclusion derived from a specific set of findings. • Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action 33 ANNEX II. LIST OF RESPONDENTS Affiliation Respondent Type Sex State District/ City NIPI (Norway India Partnership Initiative) DP Male Delhi Centre BMGF (Bill and Melinda Gates Foundation) DP Male Delhi Centre UNICEF DP Male Jharkhand Ranchi USAID/India DP Male Delhi Centre USAID/India Development Partner Male Delhi Centre Department of Health and Family Welfare Government Female Delhi NW Delhi Department of Health and Family Welfare Government Male Delhi Centre Department of Health and Family Welfare Government Female Delhi NW Delhi Department of Health and Family Welfare Government Female Delhi NW Delhi Department of Health and Family Welfare Government Female Delhi NW Delhi Ministry of Health and Family Welfare Government Male Delhi Centre Ministry of Health and Family Welfare Government Male Delhi Centre Directorate of Family Welfare Government Male Delhi NW Delhi Department of Health and Family Welfare Government Female Haryana Chandigarh Department of Health and Family Welfare Government Female Haryana Mewat Department of Health and Family Welfare Government Male Himachal Pradesh Chamba Department of Health and Family Welfare Government Male Himachal Pradesh Chamba Department of Health and Family Welfare Government Male Himachal Pradesh Chamba Department of Health and Family Welfare Government Male Himachal Pradesh Shimla Department of Health and Family Welfare Government Female Himachal Pradesh Shimla Department of Health and Family Welfare Government Male Jharkhand Ranchi Department of Health and Family Welfare Government Male Jharkhand Ranchi Department of Health and Family Welfare Government Male Jharkhand Ranchi Department of Health and Family Welfare Government Male Jharkhand Ranchi Department of Health and Family Welfare, NHM Government Male Jharkhand Gumla Department of Health and Family Welfare, NHM Government Male Jharkhand Gumla Department of Health and Family Welfare Government Male Jharkhand Dumka Department of Health and Family Welfare Government Male Jharkhand Dumka Department of Health and Family Welfare Government Male Jharkhand Ranchi 34 Affiliation Respondent Type Sex State District/ City Department of Health and Family Welfare Government Male Punjab Chandigarh Department of Health and Family Welfare Government Female Punjab Chandigarh Department of Health and Family Welfare Government Female Punjab Chandigarh Department of Health and Family Welfare Government Male Punjab Chandigarh Department of Health and Family Welfare Government Male Punjab Barnala Department of Health and Family Welfare Government Female Punjab Barnala Department of Health and Family Welfare Government Female Punjab Muktsar Department of Health and Family Welfare Government Female Punjab Muktsar Department of Health and Family Welfare Government Female Uttarakhand Dehradun Department of Health and Family Welfare Government Male Uttarakhand Dehradun Department of Health and Family Welfare Government Male Uttarakhand Dehradun Department of Health and Family Welfare Government Male Uttarakhand Dehradun Department of Health and Family Welfare Government Female Uttarakhand Haridwar Department of Health and Family Welfare Government Male Uttarakhand Haridwar Department of Health and Family Welfare Government Male Uttarakhand Haridwar District Chief Medical Office Government Male Uttarakhand Tehri Garwhal District Chief Medical Office Government Male Uttarakhand Tehri Garwhal District Chief Medical Office Government Male Uttarakhand Tehri Garwhal District Chief Medical Office Government Female Uttarakhand Tehri Garwhal Private clinic Private Provider Female Uttarakhand Dehradun Private clinic Private Provider Male Jharkhand Ranchi Private clinic Private Provider Female Jharkhand Ranchi Vriddhi staff, National Project Male Delhi Centre Vriddhi staff, State Level Project Female Delhi NW Delhi Vriddhi staff, District Level Project Female Delhi NW Delhi Vriddhi staff, State Level Project Female Haryana Chandigarh Vriddhi staff, State Level Project Male Haryana Chandigarh Vriddhi staff, State Level Project Female Haryana Chandigarh Vriddhi staff, District Level Project Male Haryana Mewat Vriddhi staff, District Level Project Male Himachal Pradesh Chamba Vriddhi staff, State Level Project Male Himachal Pradesh Shimla 35 Affiliation Respondent Type Sex State District/ City Vriddhi staff, State Level Project Male Himachal Pradesh Shimla Vriddhi staff, State Level Project Male Punjab Chandigarh Vriddhi staff, District Level Project Female Punjab Barnala Vriddhi staff, District Level Project Female Punjab Muktsar Vriddhi staff, National Project Male Delhi Centre Vriddhi staff, National Project Male Delhi Centre Vriddhi staff, National Project Male Delhi Centre Vriddhi staff, State Level Project Male Uttarakhand Dehradun Vriddhi staff, District Level Project Male Uttarakhand Haridwar Vriddhi staff, District Level Project Male Jharkhand Gumla Vriddhi staff, District Level Project Male Jharkhand Dumka Vriddhi staff, District Level Project Male Jharkhand Godda Sultanpuri (level undefined) Public Provider Male Delhi NW Delhi Sultanpuri (level undefined) Public Provider Female Delhi NW Delhi Sultanpuri (level undefined) Public Provider Female Delhi NW Delhi Sultanpuri (level undefined) Public Provider Female Delhi NW Delhi Primary Health Center- Ujina (L2) Public Provider Female Haryana Mewat Nuh CHC (L2) Public Provider Female Haryana Mewat Chowari (L2) Public Provider Female Himachal Pradesh Chamba Chowari (L2) Public Provider Female Himachal Pradesh Chamba Jama CHC (L2) Public Provider Female Jharkhand Dumka Jama CHC (L2) Public Provider Female Jharkhand Dumka Jama CHC (L2) Public Provider Female Jharkhand Dumka Jama CHC (L2) Public Provider Female Jharkhand Dumka Jama CHC (L2) Public Provider Female Jharkhand Dumka Harrakha Subcentre (L1) Public Provider Female Jharkhand Dumka Harrakha Subcentre (L1) Public Provider Female Jharkhand Dumka Harrakha Subcentre (L1) Public Provider Female Jharkhand Dumka Dumka District Hospital Public Provider Female Jharkhand Dumka Godda CHC (L2) Public Provider Male Jharkhand Godda Pathargama CHC (L2) Public Provider Female Jharkhand Godda Poraiyahat CHC (L2) Public Provider Female Jharkhand Godda 36 Affiliation Respondent Type Sex State District/ City Poraiyahat CHC (L2) Public Provider Female Jharkhand Godda Gumla District Hospital (L3) Public Provider Male Jharkhand Gumla Gumla District Hospital (L3) Public Provider Female Jharkhand Gumla Gumla District Hospital (L3) Public Provider Female Jharkhand Gumla Gumla District Hospital (L3) Public Provider Female Jharkhand Gumla Solangbira Subcentre (L1) Public Provider Female Jharkhand Gumla District Hospital Barnala (L3) Public Provider Female Punjab Barnala Civil Hospital Malout (L3) Public Provider Female Punjab Muktsar CHC Laksar (L2) Public Provider Female Uttarakhand Haridwar SHC Jaitpur (L3) Public Provider Female Uttarakhand Haridwar SHC Jaitpur (L3) Public Provider Female Uttarakhand Haridwar Haridwar District Women's Hospital (L3) Public Provider Female Uttarakhand Haridwar Haridwar District Women's Hospital (L3) Public Provider Female Uttarakhand Haridwar Baurari District Hospital (L3) Public Provider Female Uttarakhand Tehri Garwhal Baurari District Hospital (L3) Public Provider Female Uttarakhand Tehri Garwhal Baurari District Hospital (L3) Public Provider Female Uttarakhand Tehri Garwhal Narendra Nagar Subdistrict Hospital (L2) Public Provider Female Uttarakhand Tehri Garwhal Narendra Nagar Subdistrict Hospital (L2) Public Provider Female Uttarakhand Tehri Garwhal 37 ANNEX III. DATA COLLECTION INSTRUMENTS Key Informant Interview Protocols Interview date and location: Interviewer: [Where appropriate] Agency or Company: [Where appropriate] Title(s): Name(s): Sex: Consent Statement: Thank you for taking the time to meet with us today. My name is [NAME]. I am a researcher from an organization called Social Impact, a company that is based in the United States. Our team is in India to conduct an independent evaluation about Vriddhi: Scaling Up RMNCH+A Interventions, which is funded by USAID. We would like to conduct a brief interview today to learn about your perspectives on project performance, outcomes and lessons learned. This information will be used in a final report for USAID that will be publicly available. It is important to understand that while we would like your help in this study, you do not have to take part if you do not want to, and you do not have to answer any questions if you do not feel comfortable answering. If you decide to take part, your responses will be kept strictly confidential. This means that your name will not be mentioned anywhere in the report, and will not be provided to anyone, including at your company or place of work. Any personal information we collect today will be stored in a secure computer file. The objective of this research is to improve the performance of projects like Vriddhi and to ensure its success in the future. The information may be used by other organizations as well. The interview is expected to take between 45-60 minutes. Do you have any questions? You may ask questions at any time. If you have questions or concerns about the research after we leave today, you can contact [insert ET name and contact information for each sub-team]. By saying “yes,” and participating in this study, you are indicating that you have heard this consent statement, had an opportunity to ask any questions about your participation and voluntarily consent to participate. Will you participate in this interview? You may answer yes or no. ◻ Yes, I will participate ◻ No, I will not participate 38 Project Staff (all) 1. What steps did the project take to establish and deepen working relationships with the MOHFW? What were the key enablers? What lessons have you learnt? 2. What parameters has the project used in defining scale? How have the complexities in the operating environment shaped the project’s understanding of scale? Could you give specific examples from the project? 3. Scaling up public health interventions is not a straightforward task. To what extent do you think the project utilized efficiencies within the public health system and the private sector, to achieve scale? What inefficiencies have hindered progress? 4. To what extent has the scope of the project contributed to achieving impact (reduced mortality and morbidity from preventable causes) at scale? What realities of implementation have constrained this? What factors have enabled? 5. To what extent has the project contributed to scaling up effective interventions in target locations? What would you do differently in new HPDs? What key differences, if any, do you find in the operating environments of HPDs and non-HPDs? 6. In what ways have the project’s efforts in the six states and HPDs contributed to increased uptake of services? 7. What do you look forward to accomplishing in the next two years? What would you do differently, and why? Project Director, Vriddhi 1. Please describe the evolution in the project’s design, from its inception to date, and the contextual and other factors that necessitated the changes. In retrospect, how useful do you think these changes were in reaching its objectives? 2. Please describe the inception phase of the project (first year). What challenges did the project face at the time, and how were they managed? What lessons did the project learn through this process? 3. What steps did the project take to establish and deepen working relationships with the MOHFW? What were the key enablers? What lessons have you learnt? 4. Please describe the process used to identify implementing partners (such as JSI). What considerations went into the selection of the agencies and assigning of tasks to them? What lessons have you learnt? 5. What factors enabled the process of designing the project’s key strategies such as Care around Birth (CaB) and Supportive Supervision (SS)? What challenges did the project encounter, and how were they managed? 6. How did the project ensure alignment with the 5*5 matrix of the RMNCH+A strategy? How did you manage the timelines of the project in the light of the release of MOHFW guidelines and campaigns related to interventions in the 5*5 matrix? 7. Please describe your key takeaways from the rollout of the project’s major strategies. What were the major challenges? What factors enabled the rollout? 8. What parameters has the project used in defining scale or shaping the project’s understanding of scale? Could you give specific examples from the project? 9. Scaling up public health interventions is not a straightforward task. To what extent do you think the project utilized efficiencies within the public health system and the private sector, to achieve scale? What inefficiencies have hindered progress? 10. To what extent has the scope of the project contributed to achieving impact (reduced mortality and morbidity from preventable causes) at scale? What realities of implementation have constrained this? What factors have enabled? 11. What, in your view, are factors related to key personnel, that have contributed to or hindered the success of the project? 39 12. In what ways has the USAID/India Mission supported the project in reaching its goals? What have been the key enablers? What could have been done better? What lessons have you learnt? 13. Overall, what assumptions held up, and what failed? What could have been done better, in terms of design and implementation, to identify and manage key assumptions? 14. What do you look forward to accomplishing in the next two years? What would you do differently, and why? Project Technical Lead, Vriddhi 1. How did the project ensure alignment with the 5*5 matrix of the RMNCH+A strategy? What challenges did the project face and how were they overcome? What lessons did the project learn through this process? 2. How did you manage the timelines of the project in the light of the release of MOHFW guidelines and campaigns related to interventions in the 5*5 matrix? 3. Please describe the process of designing the CaB strategy. What role did each of the stakeholders play and what factors went into its design? 4. Please describe the process of designing the SS approach. What role did each of the stakeholders play and what factors went into its design? 5. Please describe the process used in identifying the various technical models the project selected for incubation and their geographic location. What factors contributed to these choices? (Describe specifics for each model by turn) 6. Some models were proposed but later dropped. What factors led to those decisions? 7. The project has carried out advocacy for inclusion of evidence-based interventions in national policy. What is your assessment of the extent of success in these efforts? What were the key challenges and to what extent have they been overcome? 8. Please describe your key takeaways from the rollout of the project’s major strategies. What were the major challenges? What factors enabled the rollout? Do you have specific observations for the various technical service areas – maternal health, child health, family planning, adolescent health? Team Leader, National RMNCH+A Unit (+ Knowledge Hub) 1. The NRU has supported the development of guidelines and standards in RMNCH+A. Please describe the process by which they were developed. What factors enabled the successful development, and what constraints did you face? What lessons have you learnt? 2. The project has aligned itself with several GoI initiatives such as the Intensive Diarrhea Control Fortnight and Mission Indradhanush. How did the project achieve alignment? What constraints did you encounter and how did you address them? Is there any state-specific learning? 3. The project has incubated several evidence-based models. What process does the project (and other stakeholders) use to assess the scalability of each? 4. Please describe the key lessons you have learnt in carrying out SS visits in HPDs and non-HPDs, managing SS data and ensuring its use. 5. Scaling up public health interventions is not a straightforward task. To what extent do you think the project utilized efficiencies within the public health system and the private sector, to achieve scale? What inefficiencies have hindered progress? 6. To what extent has the project built institutional capacity of the state and district implementing units? What factors enabled progress in this area and what factors impeded progress? How can we better foresee challenges and build in mechanisms to address them? 7. Please describe your experience in supporting the development of project implementation plans (PIPs) at state and district levels? What factors enabled progress in this area and what factors impeded progress? How can we better foresee challenges and build in mechanisms to address them? 40 8. What is your assessment of the current capacity at state and district levels, for planning, monitoring and performance management? Any location-specific factors? 9. What are the top three ways in which the project has impacted service delivery? What factors have enabled success in these areas? What challenges were overcome? 10. The project has initiated several technology-based solutions, especially in the area of data management. To what extent have they achieved their stated purpose? To what extent has their use been institutionalized? What factors should be considered in scaling them up? 11. The project has carried out advocacy for inclusion of evidence-based interventions in national policy. What is your assessment of the extent of success in these efforts? What were the key challenges and to what extent have they been overcome? 12. To what extent has the project contributed to scaling up effective interventions in target locations? What would you do differently in new HPDs? What key differences, if any, do you find in the operating environments of HPDs and non-HPDs? 13. What is your perspective on the constitution, scope and functioning of the NRU? What factors have enabled effective and efficient functioning of the Unit? What challenges did you encounter and how did you manage them? JSI: Project Lead 1. What is your assessment of the success of the rollout of Kangaroo Mother Care (KMC)? What factors have enabled this level of success? What challenges remain? a. Use of KMC by mothers of newborns with low birth weight continues to remain low. What needs to be done to further improve its uptake? b. What is your assessment of the skill and proficiency of service providers in 2. Please describe your experience with developing the private sector engagement strategy. What factors have affected its rollout in target states and how? What more remains to be done, in engaging the private sector for service provision, especially in CaB? 3. The project has carried out advocacy for inclusion of evidence-based interventions in national policy. What is your assessment of the extent of success in these efforts? What were the key challenges and to what extent have they been overcome? State Technical Leads, Vriddhi 1. Please describe your experience in supporting the development of project implementation plans (PIPs) at state and district levels? What factors enabled progress in this area and what factors impeded progress? How can we better foresee challenges and build in mechanisms to address them? 2. Please describe the key lessons you have learnt in carrying out SS visits in HPDs and non-HPDs, managing SS data and ensuring its use. To what extent is this process institutionalized? 3. What is the extent of success in implementing the following interventions? What challenges and opportunities remain? What lessons have you learnt? a. CaB technical interventions package I – rollout: Home-based Newborn Care, Postpartum Family Planning, Kangaroo Mother Care, Maternal Death Review b. CaB technical interventions package II – rollout (if begun) c. Adolescent-friendly health clinics d. Weekly iron-folate supplementation rounds e. Support to routine immunization f. Utilization of oral rehydration solution and zinc 4. Please describe the composition of MDR/MDSR teams and the QI teams in facilities. What opportunities exist, to enable the findings from MDSRs to inform actions taken by the QI teams? 5. What factors have enabled progress in piloting of the (good practice – specific to this state) and what factors have impeded progress? What considerations would affect its state-wide uptake? 41 a. Jharkhand: Support for Newborn Action Plan; support to rapid assessment of immunization; misoprostol distribution; special newborn care corner b. Punjab: Identifying high-risk pregnancies through frontline workers c. Uttarakhand: Block-level implementation strengthening support (BLISS) d. Delhi: Support to urban health and nutrition days e. Himachal Pradesh: Misoprostol distribution f. Haryana: Improving family planning access 6. Training government service providers has been a major activity of the project. What major successes have you experienced in this area? What lessons have you learnt from organizing and carrying out training events? 7. What is your assessment of the current capacity at state and district levels, for planning, monitoring and performance management? Are there any location-specific factors? Is there any difference in capacity growth at different facility levels (LI/2/3)? In what ways did the project contribute to improving this capacity? 8. What are the top three ways in which the project has impacted service delivery in this state? What factors have enabled success in these areas? What challenges were overcome? What key differences, if any, should be factored in for HPDs and for non-HPDs? 9. Please describe your experience in engaging the private sector players in service delivery and working with associations of private providers. What is the extent of success in bridging service delivery gaps through the private sector? What challenges remain? What opportunities are yet to be made use of? (Only in Delhi, Dehradun and Ranchi) 10. What constraints did you encounter in aligning project implementation with GoI initiatives such as the Intensive Diarrhea Control Fortnight and Mission Indradhanush? How did you address them? 11. What steps did the project take to establish and deepen working relationships with the Mission Director’ office and the state program offices? What were the key enablers? What lessons have you learnt? District Technical Officers, Vriddhi 1. Please describe the successes and constraints in supporting PIP development in your district. How did you overcome the constraints? What lessons have you learnt from this experience? 2. Training government service providers has been a major activity of the project. What major successes have you experienced in this area? What lessons have you learnt from organizing and carrying out training events? 3. What factors have enabled progress in piloting of the (good practice – specific to this district, if applicable) and what factors have impeded progress? What considerations would affect its state￾wide uptake? a. Jharkhand: Support for Newborn Action Plan; KMC (Gumla) support to rapid assessment of immunization (Godda); misoprostol distribution (Dumka); home based newborn care (Gumla) b. Punjab: Identifying high-risk pregnancies through frontline workers, WIFS (Barnala) c. Uttarakhand: Block-level implementation strengthening support, home based newborn care & KMC (Haridwar) d. Delhi: Support to urban health and nutrition days (NW district) e. Haryana: Improving family planning access (Mewat) 4. Please describe the composition of MDR/MDSR teams and the QI teams in facilities. What opportunities exist, to enable the findings from MDSRs to inform actions taken by the QI teams? 5. To what extent has the project built institutional capacity of the district implementation unit? What factors enabled progress in this area and what factors impeded progress? How can we better foresee challenges and build in mechanisms to address them? 42 6. What is your assessment of the current capacity of this district for planning, monitoring and performance management? 7. What are the top three ways in which the project has improved service delivery? What factors have enabled success in these areas? What challenges were overcome? 8. In what ways has the project built the skills and competencies of service providers, and supplies/inventory management in facilities? (Chamba, HP- are there any differences noted in competency improvement among different facility levels?) What challenges remain? 9. What are your observations on the project’s engagement of frontline workers? What challenges and opportunities does the project continue to encounter in maximizing reach of services through them? 10. Please describe your experience in engaging the private sector players in service delivery. What is the extent of success in this effort? What challenges remain? What opportunities are yet to be made use of? Service Providers – Public Facilities 1. What specific training or skill-building efforts of the project have you participated in? In what ways have they benefited your work? 2. What challenges do you face in employing these skills and competencies in the services you provide? What can be done to ensure that you are able to put them to good use? 3. Do you think there is an increase in people utilizing RMNCH+A services? a. Which areas have seen the most increase and why? b. Which areas have seen the least improvement, and why? 4. Specific models that providers are involved in: a. SAMMAN/BLISS/PMSMA: Please outline the level at which this model is implemented or supported (type of facility; block or district level). In what ways has the implementation of this model increased the uptake of services (identification and follow up of high risk pregnancies, increase in ANC coverage etc) b. Special newborn care unit (SNCU): In what ways has the training helped improve the quality of care provided to newborns here? What changes have you observed in the survival of newborns and other outcomes? (Please provide examples/stories). What aspects of skills and competencies are difficult to implement or maintain? c. Provider trained in TIP 1 package: In what ways has the training helped improve the quality of care provided to mothers and newborns here? What changes have you observed in the survival of mothers and newborns and other outcomes? (Please provide examples/stories). What aspects of skills and competencies are difficult to implement or maintain? d. Provider involved in Urban Health and Nutrition Days (UHND): How has the training from Vriddhi improved the quality of services in UHNDs? Do you think these changes have changed the uptake of services in these camps? What more can be done to increase uptake of services? e. Adolescent friendly health centres (AFHCs): In what ways has the training from Vriddhi improved your skills and competencies in providing health services to adolescents? Has the uptake of services improved as a result? Could you provide examples from recent months? What more can be done to get more adolescents use these services? f. Misoprostol distributors: Please outline your activities related to misoprostol distribution. To what extent do women (and their families) accept the intervention? What challenges have you faced? What more can be done to further increase the number of women that accept and use misoprostol? 43 Service Providers – Private Facilities 1. What specific training or skill-building efforts of the project have you participated in? In what ways have they benefited your work? 2. What challenges do you face in employing these skills and competencies in the services you provide? What can be done to ensure that you are able to put them to good use? 3. How can the project further support you to expand the reach of your services, especially to the poor and disadvantaged? What factors hinder such growth? 4. Do you think there is an increase in people utilizing obstetric, newborn and child services? a. Which areas have seen the most increase and why? b. Which areas have seen the least improvement, and why? State Directors and Program Officers, Dept. of Health & Family Welfare 1. Please describe your experience with the support that the project has provided the state machinery. In what specific ways have these interventions built the institutional capacity of the department in planning, monitoring and delivering RMNCH+A services and manage performance? What challenges and opportunities remain? What lessons were learnt? 2. Please describe the key lessons you have learnt in carrying out SS visits in HPDs and non-HPDs, managing SS data and ensuring its use. What further support would the state and districts require, in order for this system to become established? 3. What factors have enabled progress in piloting of the (good practice – specific to this state) and what factors have impeded progress? What considerations would affect its state-wide uptake? a. Jharkhand: Support for Newborn Action Plan; support to rapid assessment of immunization; misoprostol distribution; special newborn care corner b. Punjab: Identifying high-risk pregnancies through frontline workers c. Uttarakhand: Block-level implementation strengthening support d. Delhi: Support to urban health and nutrition days e. Himachal Pradesh: Misoprostol distribution f. Haryana: Improving family planning access 4. In what ways did the project help the state directorate in achieving its goals with respect to special campaigns such as Mission Indradhanush? 5. Training government service providers has been a major activity of the project. In what ways have these trainings built skill and competency in service providers? What areas of capacity need further strengthening? 6. What are the top three ways in which the project has impacted service delivery in this state? What factors have enabled success in these areas? What challenges were overcome? What key differences, if any, should be factored in for HPDs and for non-HPDs? 7. Looking ahead, what areas of capacity need to be further developed, in order to increase the uptake of RMNCH+A services in the state? District Officers, Department of Health and Family Welfare 1. What is your assessment of the current capacity of this district for planning, monitoring and performance management? To what extent has the project built institutional capacity at the district level? What factors enabled progress in this area and what factors impeded progress? How can we better foresee challenges and build in mechanisms to address them? 2. What are the top three ways in which the project has improved service delivery in this district? What factors have enabled success in these areas? What challenges were overcome? 3. In what ways has the project built the skills and competencies of service providers, and supplies/inventory management in facilities? (In Chamba, HP - are there any differences in provider capacity growth at different facility levels (L1/2/3)? What challenges remain? 44 4. What are your observations on the project’s engagement of frontline workers? What challenges and opportunities does the project continue to encounter in maximizing reach of services through them? 5. Please describe your experience with the support that the project has provided at the district level. Have these interventions built the institutional capacity at your department’s level in planning, monitoring and delivering RMNCH+A services and manage performance? What challenges and opportunities remain? What lessons were learnt? 6. What factors have enabled progress in piloting of the (good practice – specific to this district) and what factors have impeded progress? a. Jharkhand: Support for Newborn Action Plan; support to rapid assessment of immunization and QI team (Godda); misoprostol distribution, MDR and responses (Dumka); home based newborn care and KMC (Gumla) b. Punjab: Identifying high-risk pregnancies through frontline workers, WIFS (Barnala), CaB trainer; c. Uttarakhand: Block-level implementation strengthening support, home based newborn care & KMC (Haridwar), QI team (Tehri Garhwal) d. Delhi: Support to urban health and nutrition days (NW district) e. Haryana: Improving family planning access, MDR and responses (Mewat) f. Himachal Pradesh: QI team (Chamba) 7. Training government service providers has been a major activity of the project. In what ways have these trainings built skill and competency in service providers? What areas of capacity need further strengthening? 8. What are the top three ways in which the project has impacted service delivery in this district? What factors have enabled success in these areas? What challenges were overcome? 9. Looking ahead, what areas of capacity need to be further developed, in order to increase the uptake of RMNCH+A services in the district? Deputy Commissioners, MOHFW 1. Please describe the ways in which the project has supported the goals of the Ministry with regard to RMNCH+A (maternal health, child health, other). 2. A major area of work of the project has been in the CaB strategy. To what extent has the rollout been successful? What challenges constrained progress and how have they been overcome? 3. The project has supported the development of guidelines such as the monitoring indicators for the PMSMA campaign, and maternal death reviews. What was the value addition of the project in these areas? 4. In what ways have the project’s efforts in the six states and HPDs contributed to increased uptake of services? 5. Training government service providers has been a major activity of the project. In what ways have these trainings built skill and competency in service providers? What areas of capacity need further strengthening? 6. Looking ahead, what specific areas need further support, in order to achieve scale in RMNCH+A services? USAID/India Mission 1. How was the transition from the earlier initiative(s) managed, and the programmatic focus maintained? 2. To what extent to the project’s design and approaches reflect USAID’s priorities? Have there been areas where better alignment is needed? 3. What is your view on the focus of the project to build institutional capacity – in MOHFW and in the state Directorates? What challenges remain? 45 4. Scaling up public health interventions is not a straightforward task. To what extent do you think the project utilized efficiencies within the public health system, to achieve scale? What inefficiencies have hindered progress? 5. What specific factors need to be taken into consideration for scaling up specific interventions to other high-burden states? In what ways can the project’s current approach help those future efforts? What changes are needed? 6. Did the project optimally utilize the opportunities that the private sector affords, that was delineated by earlier USAID initiatives for the private sector? What more can be done? 7. To what extent has the NRU been effective in providing technical oversight and support to the design and implementation of project interventions? What strategic changes are needed in its structure and functioning that might improve its effectiveness? 8. How would you rate USAID/India’s support to the project in reaching its goals? What have been the key enablers? What could have been done better? What lessons have you learnt? Development Partners (Bill and Melinda Gates Foundation, The Norway India Partnership Initiative, UNICEF) 1. What are your perspectives on the structure and functioning of the NRU and their influence on the strategic objectives of the Unit? How has your experience in working with the team been? 2. In your perspective, to what extent has the rollout of CaB strategy of Vriddhi been successful? What could have been done better? 3. In your perspective, to what extent has the rollout of the Supportive Supervision strategy of Vriddhi been successful? What could have been done better? 4. To what extent do you think the project utilized efficiencies within the public health system, to achieve scale? What inefficiencies have hindered progress? To what extent is the project’s approach to scale in line with GoI’s thinking? 5. What has your experience been in working with Vriddhi in states where USAID is the lead DP? How effective is the unified response team/coordination committee? How effective is Vriddhi’s leadership and management for coordinated efforts? (only for UNICEF) 6. To what extent has institutional capacity been built in states where you work alongside USAID in RMNCH+A? (only for UNICEF) 7. Based on your experience in states where you are the Lead DP, what does it take to build institutional capacity in the public healthcare delivery system? What efficiencies can we take advantage of? What specific lessons you have learnt? a. Specific lessons for care around birth (intrapartum and immediate postpartum care) 8. Based on your experience in states where you are the Lead DP, what does it take to achieve and maintain scale (sufficient to lead to demonstrable reductions in mortality)? How do you ensure alignment between DPs and the government in order to achieve scale? 9. What, in your experience, are the key considerations for building national-level capacity – for scale up and for quality improvement? 46 ANNEX IV. CAB DATA VALIDATION The ET conducted a data validation exercise in 10 facilities across the 6 HPDs, the purpose of which was to ascertain the accuracy of the data collected and reported through Vriddhi’s CaB MIS. The ET selected 3 indicators and compared the data at 3 levels: facility records (labor room registers), district report (from Vriddhi district staff), and project report (CaB MIS report). The 3 indicators were: 1. Number of deliveries in the facility for a predetermined month/year 2. Number of deliveries (#1 above) that had completed partographs 3. Number of deliveries (#1 above) in which the mother was administered oxytocin within a minute of delivery The ET tabulated these numbers from the labor room registers of these facilities. The team later obtained data for these indicators for the corresponding reporting periods from the district reports and the project’s CaB MIS. Deliveries that occurred in transit from home to facility were recorded in the labor room registers as facility births, but not in Vriddhi records. The table below provides the data obtained: Chamba Himachal Muktsar Punjab Barnala Punjab Mewat Haryana Mewat Haryana Tehri Garwhal Uttarakhand Tehri Garwhal Uttarakhand Gumla Jharkhand Dumka Jharkhand Godda Jharkhand Facility CHC Chowari Malout Civil Hospital Barnala District Hospital CHC Nuh PHC Ujina Baurari District Hospital CHC Narendra Nagar Gumla District Hospital CHC Jama CHC Pathargama Level L2 L3 L3 L2 L2 L3 L2 L3 L2 L2 Date of facility data collection 3/30/2018 4/6/2018 4/5/2018 4/10/2018 4/10/2018 4/4/2018 4/4/2018 4/7/2018 4/9/2018 4/10/2018 Month/Year of Record Jan-18 Mar-18 Mar-17 Mar-18 Mar-18 Sep-17 Sep-17 Sep-17 Feb-18 Sep-17 Facility Records (F) DTO Report (D) Project Report (P) F D P F D P F D P F D P F D P F D P F D P F D P F D P F D P # of deliveries 11 10 10 117117 117397396 396141141141 66 66 66 59 59 59 27 27 27 348 293293 86 85 85 204 203 203 # of completed partographs 11 10 10 62 62 62 333333 333136136136 63 63 63 29 29 29 19 17 17 170 168168 81 81 81 100 93 93 # administered oxytocin within 1 minute of delivery 10 10 10 117117 117396396 396136136136 63 63 63 59 59 59 26 27 27 295 291293 85 85 85 201 202 202 Variance was calculated between facility and district and between district and project as follows: Variance = {(Data in destination record – Data in source record)/Data in source record} * 100 Variance was calculated by indicator and by facility and averages were computed. The average variance between facility records and district reports was -1.22 and the average variance between district and project reports was 0.02. 47 ANNEX V. ENDNOTES i Ng M, Fullman N, Dieleman JL, Flaxman AD, Murray CJL, et al. (2014) Effective Coverage: A Metric for Monitoring Universal Health Coverage. PLoS Med 11(9): e1001730. doi:10.1371/journal.pmed.1001730 ii See UNICEF India Maternal Health Factsheet. Accessed March 16, 2018, UNICEF. State of the World's Children, 2016 iii CaB MIS, Vriddhi Project, IPE Global iv USAID MCHIP. Standards for RMC https://www.k4health.org/toolkits/rmc/standards-rmc Accessed May 16, 2018 vUSAID MCHIP. RMC: What to measure and how to measure it. https://www.k4health.org/sites/default/files/figo_africa_rmc_measurement.pdf Accessed May 16, 2018 vi CAB MIS, Vriddhi Project, IPE Global vii World Health Organization, Department of Reproductive Health and Research. 2003. Kangaroo Mother Care: A Practical Guide. viii JSI contribution under Vriddhi for implementation of Kangaroo Mother Care in District Hospitals. USAID, IPE Global, JSI. Undated ix CAB MIS, Vriddhi Project, IPE Global x MOHFW and International Institute for Population Sciences. National Family Health Survey 4, 2015-16 xi Randive B, Diwan V, De Costa A (2013) India’s Conditional Cash Transfer Programme (the JSY) to Promote Institutional Birth: Is there an Association between Institutional Birth Proportion and Maternal Mortality? PLoS ONE 8(6): e67452. doi:10.1371/journal.pone.0067452 xii India's Janani Suraksha Yojana, a conditional cash transfer programme to increase births in health facilities: an impact evaluation Lim, Stephen S et al. The Lancet, Volume 375, Issue 9730 2009 - 2023 xiii Tanahashi T. Health service coverage and its evaluation. Bull World Health Organ. 1978;56(2):295–303. pmid: 96953 xiv Suraksha – Protecting Mothers. Community based advance distribution of Misoprostol – a pilot in Himachal Pradesh. USAID, Govt of Himachal Pradesh, National Health Mission, IPE Global, 2018 xv ExpandNet WHO. Beginning with the end in mind: planning pilot projects and other programmatic research for successful scaleup. 2011 http://www.expandnet.net/tools.htm Accessed May 16, 2018 xvi Jacobs B, et al Addressing access barriers to health services: an analytical framework for selecting appropriate interventions in low-income Asian countries. Health Policy and Planning, July 2012. https://doi.org/10.1093/heapol/czr038 xvii Amudhan S. et al. Effectiveness of demand and supply side interventions in promoting institutional deliveries – a quasi-experimental trial from rural north India. Int’l Journal of Epidemiology, June 2013 https://doi.org/10.1093/ije/dyt071 xviii Maternal, Newborn and Child Health Working Paper. Reaching Universal Health Coverage through District Health System Strengthening: Using a modified Tanahashi model sub-nationally to attain equitable and effective coverage, UNICEF 2013 xix Same as (xiii) above xx Ng M, Fullman N, Dieleman JL, Flaxman AD, Murray CJL, et al. (2014) Effective Coverage: A Metric for Monitoring Universal Health Coverage. PLoS Med 11(9): e1001730. doi:10.1371/journal.pmed.1001730