USAID/India Health of the Urban Poor Program Final Evaluation Report September 2015 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Ranjani Gopinath, Kerry Richter, Kumkum Srivastava and Gabrielle Plotkin of Social Impact, Inc. ii USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT FINAL EVALUATION OF USAID/INDIA’S HEALTH OF THE URBAN POOR PROGRAM DRAFT EVALUATION REPORT September 2015 AID-386-TO-15-00002 AID-486-I-14-00001 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States government. ACKNOWLEDGEMENTS The evaluation team wishes to thank the Population Foundation of India’s Program Management Unit staff and partner organizations for generous support in providing information and documentation, particularly Dr. Sainath Banerjee. Special thanks to Ms. Sumoni Mukherjee and Mr. Gajinder Pal Singh Seerah, HUP team members, for assisting quantitative data collation. The team also thanks the HUP state and city teams and partners for facilitating the site visits and providing detailed information on local activities, in addition to state government and municipal officials for making time to meet and answer our many questions and follow-up requests for additional information. We thank officials from the government of India, external donors and USAID staff members for their thoughtful responses to the team’s questions. Finally, the team wishes to thank the community members who graciously gave time to participate in at times lengthy focus group discussions and one-on-one interviews, which provided rich information for the team’s analysis. The team is grateful for the support of Mannat Travels and Tours and Dr. Ash Pachauri for in-country logistical and local support. The Evaluation Team September 2015 ii USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT CONTENTS ACRONYMS ........................................................................................................................................................................................... iv EXECUTIVE SUMMARY...................................................................................................................................................................... vi EVALUATION PURPOSE AND EVALUATION QUESTIONS............................................................................................ vi PROGRAM BACKGROUND........................................................................................................................................................ vi EVALUATION QUESTIONS, DESIGN, METHODS AND LIMITATIONS ..................................................................... vii FINDINGS AND CONCLUSIONS............................................................................................................................................. vii SUMMARY OF MAJOR RECOMMENDATIONS.....................................................................................................................ix 1. INTRODUCTION.............................................................................................................................................................................1 EVALUATION PURPOSE ................................................................................................................................................................1 EVALUATION QUESTIONS ..........................................................................................................................................................1 PROGRAM BACKGROUND.........................................................................................................................................................2 EVALUATION METHODS & LIMITATIONS ............................................................................................................................3 METHODOLOGY ........................................................................................................................................................................3 DATA SOURCES ..........................................................................................................................................................................4 LIMITATIONS AND THREATS TO VALIDITY....................................................................................................................5 FINDINGS AND CONCLUSIONS ...................................................................................................................................................6 FINDINGS: EVALUATION QUESTION 1 (MID-TERM RECOMMENDATIONS) .........................................................6 FINDINGS: EVALUATION QUESTION 2 (EFFECTIVENESS OF SCALE UP) ...............................................................11 FINDINGS: EVALUATION QUESTION 3 (EXTENT OF CONVERGENCE)................................................................21 FINDINGS: EVALUATION QUESTION 4 (EXTENT OF POLICY LEVEL INFLUENCE) ...........................................24 5. LESSONS FOR USAID’S FUTURE URBAN PROGRAMMING...........................................................................................26 RECOMMENDATIONS......................................................................................................................................................................27 ANNEXES...............................................................................................................................................................................................32 ANNEX I: EVALUATION STATEMENT OF WORK............................................................................................................32 ANNEX II: EVALUATION DESIGN, METHODOLOGY, AND LIMITATIONS ............................................................38 ANNEX III: DATA COLLECTION TOOLS .............................................................................................................................41 ANNEX IV: LIST OF INFORMATION SOURCES: DOCUMENTS AND RESPONDENTS.......................................54 ANNEX V: DATA COLLECTION SCHEDULE ......................................................................................................................63 ANNEX VI: MATRIX OF DIVERSITY ........................................................................................................................................64 ANNEX VII: QUANTITATIVE DATA........................................................................................................................................69 ANNEX VIII: TA PROCESS INDICATORS ..............................................................................................................................74 ANNEX IX: OUTCOME OF TA PRODUCTS ........................................................................................................................77 ANNEX X: COMPARISON OF THE CHANGE IN URBAN BUDGETS OF HUP AND NON-HUP STATES, 2010–2013..........................................................................................................................................................................................79 ANNEX XI: GEOGRAPHICAL COVERAGE AND ACTIVITIES OF HUP CONSORTIUM..................................80 iii ANNEX XII: ASSESSING THE ACHIEVEMENTS OF THE DEMONSTRATION CITIES ............................................82 ANNEX XIII: ACHIEVEMENTS AGAINST QUANTIFIABLE OUTCOMES ....................................................................85 iv USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT ACRONYMS ADB Asian Development Bank ANC Antenatal care ANM Auxiliary Nurse Midwife ASHA Accredited Social Health Activist AWC Anganwadi Center AWW Anganwadi Worker BCC Behavior Change Communication BCT Bhoruka Charitable Trust CBO Community Based Organization CCC City Coordination Committee CEDPA Centre for Development and Population Activities CG Chhattisgarh CHP City Health Plan(ning) CII Confederation of Indian Industries CPM(U) City Program Management (Unit) EAG Empowered Action Group ET Evaluation Team FGD Focus Group Discussion GIS Global Information System GOI Government of India HIV/AIDS Human Immunodeficiency Virus/ Acquired Immunodeficiency Syndrome (H)MIS (Health) Management Information System HUP Health of the Urban Poor ICDS Integrated Child Development Services IEC Information, Education, and Communication IIHMR International Institute of Health Management and Research IIPS International Institute for Population Studies INR Indian National Rupees JNNURM Jawaharlal Nehru National Urban Renewal Mission MAS Mahila Swasthya Samiti (Women’s Health Committee) MCH Maternal and Child Health MCTS Mother & Child Tracking System MIA Micro Insurance Academy MO Medical Officer MOHFW Ministry of Health and Family Welfare MOU Memorandum of Understanding MIS Management Information System MP Madhya Pradesh MTE Mid-term Evaluation NCD Non-Communicable Diseases NGO Nongovernmental Organization NHSRC National Health System Resource Centre NRHM National Rural Health Mission NUHM National Urban Health Mission PFI Population Foundation of India v PHED Public Health Engineering Department PIP Project Implementation Plan POU Point of Use PPP Public-Private Partnership PwC Price Waterhouse Coopers RCH Reproductive and Child Health (national program) RMNCH+A Reproductive, Maternal, Newborn and Child Health + Adolescents RSBY Rashtriya Swasthya Bima Yojana (national health insurance scheme) SIFW State Institute of Health and Family Welfare SIFPSA State Innovation in Family Planning Services Planning Agency STI Sexually Transmitted Infections TA Technical Assistance TAG Technical Advisory Group TRG Technical Resource Group TB Tuberculosis UHC Urban Health Center UHND Urban Health and Nutrition Days UHRC Urban Health Resource Centre ULB Urban Local Body UK Uttarakhand UP Uttar Pradesh UPHC Urban Primary Health Center USAID United States Agency for International Development USD United States Dollar WASH Water and Sanitation Hygiene WB West Bengal WCC Ward Coordination Committee WCD Women and Child Development VA Vulnerability Assessment vi USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT EXECUTIVE SUMMARY India has made progress in achieving health outcomes over the last decades, especially in rural areas; but the urban poor have generally not benefited. The delivery of health services in urban areas has been sub￾optimal and fragmented. As a policy response, in May 2013, the Government of India (GOI) launched the National Urban Health Mission (NUHM) to strengthen health service delivery in urban areas. Since 2001, the United States Agency for International Development (USAID) has engaged in an active partnership with the GOI’s Ministry of Health and Family Welfare (MOHFW). In 2009, USAID initiated the “Health of the Urban Poor” (HUP) program. Within the context of the imminent launch of NUHM, the program was designed to support the central, state, municipal, and community health structures develop innovative policies and program strategies to better meet the health needs of the urban poor. EVALUATION PURPOSE AND EVALUATION QUESTIONS The purpose of this final evaluation of the HUP program is threefold. First, it explores the program’s progress against its objectives and recommendations made in the midterm evaluation (MTE) conducted in 2012. Second, it evaluates the strategies and success of efforts to scale-up the program’s results from the first phase. Finally, this evaluation seeks to obtain insights and document lessons learned from the various components of the program. The evaluation aims to answer four primary questions, including: 1. To what extent the program has successfully addressed the major recommendations from its mid-term evaluation in its second phase? 2. How effectively has the program scaled up the activities/model interventions from the five cities in phase one to 18 cities in the second phase? In particular, the evaluation shall address the following sub-questions. 3. To what extent has the program been successful in ensuring the convergence of various GOI efforts on improving urban health? 4. To what extent has the program influenced policy-level changes with regard to improving urban health at the national, state and city levels? 5. What lessons can be drawn from this program in terms of key strategic approaches and impact that should inform USAID’s future urban health focus? These include several sub-questions addressed in the body of the report. Given the overlap in the content of question 5 with the sub-questions from question 2, the evaluation team and USAID decided to subsume the findings, conclusions, and recommendations pertaining to question 5 under question 2. PROGRAM BACKGROUND The Cooperative Agreement (for the total sum of 513,601,582 Indian Rupees (10,778,627 United States Dollars) was awarded to the Population Foundation of India (PFI) to provide support to USAID/India’s HUP program. HUP is the first USAID award made directly to an Indian non-governmental organization (NGO). GOI approval had not been obtained prior to awarding the Cooperative Agreement for HUP in September 2009, thus delaying program implementation for ten months. The program was originally planned for four years; however, one of the program design’s chief assumptions—which ultimately did not hold true—was a timely rollout of NUHM. At mid-term therefore, in the absence of NUHM and with the accompanying reluctance of state governments to initiate urban health activities beyond those mandated under the GOI’s urban reproductive and child health (RCH program), the HUP had been implemented for 18 effective months. Based on mid-term recommendations HUP was extended to September 30, 2015 in order to overlap with the launch of NUHM in 2013 and facilitate its rollout. The vii extension phase was redesigned to provide TA support to the national and state governments in rolling out NUHM, as well as to scale up lessons from its phase I cities to at least 20 cities. EVALUATION QUESTIONS, DESIGN, METHODS AND LIMITATIONS The evaluation was carried out by a four-member team from Social Impact, Inc. during July and August 2015. The team consisted of a Team Leader/Senior Evaluation Specialist, a Senior Public Health Specialist, a Senior Management, Governance and Private Health Sector Expert and a Research Assistant. The evaluation uses a qualitatively-dominant albeit mixed-methods approach involving (1) a desk review of available primary and secondary documents; (2) site visits to cities involving semi-structured key informant interviews, observation and focus group discussions, (3) semi-structured key informant interviews at the national level; and (4) quantitative analysis of data reported by the program. FINDINGS AND CONCLUSIONS The evaluation found that the HUP program has several accomplishments and achievements, despite encountering many challenges—including delay in the launch of the NUHM, no clear incentive for states to address urban health issues, and a 10-month delay in HUP program approval by GOI. Another fundamental challenge is the nature of the program design, which includes a broad range of activities to be provided through a technical assistance (TA) approach in eight states and five municipalities representing diverse environments. However, HUP helped to delineate national- and state-level policies on urban health, in addition to broadening the participation of relevant stakeholders in the development and implementation of NUHM policies, program priorities, and operational strategies. The key findings and conclusions in response to the evaluation questions are as follows: 1. Progress against mid-term recommendations: Findings: TA: HUP created state-level preparedness for the NUHM launch and developed a comprehensive primary health service framework. It realized the potential matrix delineated by the MTE, except for addressing intra-health coordination. However, the post-NUHM potential for an integrated and convergent urban health model—one which delineates the role of each stakeholder and the funding source for each planned intervention—could not be found in any state. HUP could not overcome the disinclination and the nascent capacities of the states to innovate beyond the NUHM guidelines. Private Public Partnerships (PPP): HUP could not move forward on the PPPs in the absence of interest within the governments to participate. The importance of the missed opportunity to build on what previously existed within the governments is now clear in light of the barriers being faced by the NUHM to recruit and retain health functionaries in the urban areas. Convergence: HUP facilitated the creation of convergent platforms. HUP incorporated water, sanitation, and hygiene (WASH) indicators into its monitoring plans and successfully integrated WASH components with the country’s supplementary nutrition program. There has been ample progress and realized outcomes on integration, as reflected by the WASH interventions in the Project Implementation Plans (PIP) of seven states. The coordination with other programs and stakeholders has however been minimal, limited to invitations to advocacy and dissemination workshops. Demonstration sites: HUP has systematically documented the capacity of state and municipal urban health systems, tracked beneficiary level indicators, and is in the process of completing the endline survey as recommended by the MTE. Although several research studies were carried out by HUP, operational research to determine meaningful Mahila Aarogya Samitis (MAS) incentives and impact assessment of demonstration sites have not been carried out. viii USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Conclusions: HUP has progressed on most of the MTE recommendations. However, there has been limited cross learning and the challenges detailed above limited the progress on PPP and convergence objectives. 2. Effectiveness of Scale-up Approaches Findings: Four of the five scale-up approaches have been incorporated into the NUHM framework. These include formation of women’s groups (MAS) at the community level, implementation of urban health and nutrition days (UHND), city health planning (CHP), and establishment of city coordination committees (CCC) as convergent platforms for implementing the NUHM. Similar structures at the ward level, the Ward Coordination Committees (WCC), were not adopted by the NUHM. The fifth approach, the Health Management Information System (HMIS), though customized for all eight HUP states has not been streamlined at the national level and disaggregated data for urban slums is not being collated and analyzed by the NUHM. In all, 4,267 (58% of target) MAS were formed, 118,148 (97%) UHNDs were held, eight (61%) CCCs were formed and 379 (1895%) CHPs were developed. Conclusions: Both quantitative and qualitative data show the eight states with scale-up cities to be in various stages of operationalization. Stakeholder perceptions indicate that states with an HUP presence have moved ahead in an accelerated manner on operationalizing these approaches as compared to other states. This could not be quantitatively ascertained owing to lack of access to indicator data from non￾HUP states. Key lessons learned include that NUHM is a nascent program and current capacities to roll out are low; linkages with other stakeholder organizations would help bolster implementation; different cities have differing needs as well as opportunities and while the NUHM framework provides flexibility, it is inadequately interpreted by governments. Furthermore, alternate/complementary models of service delivery that address the dearth of human resources for urban health will be required going forward. The current strategies of NUHM are not addressing specific health needs of men, which is interdependent with that of women and families. The structure of MAS is effective and empowers women, and WCCs are an excellent model for decentralized urban health and development management. 3. Effectiveness of Convergence Findings: HUP has facilitated the delineation of convergence processes, advocated for convergent platforms, and supported the formation of convergence structures at the state and city level. The convergence efforts of HUP resulted in WASH practices being adopted by the Health and Women and Child Development Departments. The model of point of use (POU) water testing and treatment developed by HUP has been adopted in seven states. Although the convergent actions were evidently strong at the community and the ward level (in the few locations where WCCs have been formed), there is an overall perception among the stakeholders that convergence has been initiated but not achieved at state and city level. HUP’s efforts to facilitate convergence with other urban development initiatives were limited. Conclusions: Convergence has occurred to a great degree at the grassroots level. Ward level coordination has worked well where there has been successful leveraging of local representatives’ participation. The most successful result created by HUP’s convergence efforts is the integration of WASH component in PIPs. Joint CHP with mapping of key focus areas on urban health and health determinants is an effective approach, but it is not currently integrated and comprehensive. HUP was not able to bring about convergence of various health organizations and urban development programs. ix 4. Extent of policy influence Findings: The evaluation found that HUP’s TA has led to outcomes that include demonstration sites that informed the NUHM framework, advocacy that created an enabling environment for urban health, replication and scale up of urban health programming, a demand for TA, a high level of appreciation for the TA among HUP states, and increased allocation (122%) of funds to urban health. However, the evaluation also found that sustainable capacities are yet to be achieved at the state level. The challenges to influencing policy have been the states’ limited readiness, their low level of interest in urban programming, the short period of implementation, and the delayed launch of NUHM at the state level. Conclusions: Despite all delays and impediments, HUP has produced a large and varied base of evidence and policy-level work around urban health. Not all TA efforts have reached their intended outcomes owing to contextual factors. Although HUP has created only one model of urban primary health care delivery, it has delineated processes and established mechanisms for effective roll out of this model through the NUHM. The implementation of the NUHM framework is in various stages of maturity. Community processes and advocacy have been the strong elements of HUP, compared to the convergence and PPP aspects of the program. SUMMARY OF MAJOR RECOMMENDATIONS Recommendations for USAID 1. Design a convergence mechanism such as pooled funding from USAID sources for programs seeking to address cross-cutting health issues which require multi-sectoral approaches and recruit regional hubs of high-powered TA teams, as opposed to central TA teams for such programs. Complement these teams with a small hand-holding team at the state level. Rationalize team size based on a preparedness analysis of states when introducing newer strategies or interventions to ensure ownership and results. 2. Invest in documenting and disseminating HUP lessons, products, and models to reach a wider set of audience (state governments, non-health departments involved in urban development, organizations working with marginalized population, donors and other countries in the process of strengthening urban health programs). Important among the lessons that must be disseminated include: participation of men in urban health, convergence processes, and the need for a comprehensive urban health model. Some suggested avenues for dissemination are national and international conferences, regional workshops for disseminating process documentation within NUHM, physical as well as virtual resource centers linked to the NUHM website, and courses and curricula for developing urban health cadres. Another important avenue for investment is the HMIS software, as identified by the HUP team. 3. Expand the scope of urban health in the medium term to demonstrate a model which includes comprehensive health services, disaster management, epidemic management, environmental health, solid waste management, community insurance models, and lifestyle issues. In the short term, advocate for a comprehensive urban services package, men’s health, and innovative models of primary health care delivery to overcome current gaps in human resources. Examples include primary health care models which are led by nurse practitioners, models managed by communities and models involving partnerships with the private sector (social franchising). 4. Consider avenues for supporting NUHM. Options include: x USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT o Option 1: Extend the program by 18 months until the Asian Development Bank TA takes over to ensure continued velocity of NUHM and support the maturation of a program that has been nurtured by USAID for over two decades. o Option 2: Create HUP phase II with renewed objectives that address emerging challenges to the NUHM, health determinants, and urban development; in the interim period, engage with national and state level NUHM through a development partners’ forum to inform and advocate for the successes of HUP. o Option 3: Start afresh; invest in the creation of large scale private sector models which are co-funded by large municipal corporations to serve the urban vulnerable population, especially in mega cities. Recommendations for NUHM 1. Ensure continued engagement of MAS and ASHAs through partnership with community-based organizations to carry out vulnerability assessment and community action; introduce mechanisms for formally identifying members as community volunteers and linking them to employment, entrepreneurial, insurance schemes, and development opportunities; develop revised norms for ASHA incentives and mechanisms and issue financial guidelines to MAS. 2. Establish platforms to address men’s health issues such as substance abuse, tuberculosis, sexually transmitted diseases e.g. the human immunodeficiency virus, and leverage men’s participation in community health and action. 3. Consider incorporation of the WCC strategy within the NUHM framework as these can be leveraged for micro-planning, monitoring, and mobilizing services. 4. Orient and advocate to states the flexibilities within the NUHM framework and encourage innovations by creating flexible funding mechanisms. 5. Develop a customized behavior change communication strategy following identification of specific urban needs. The placement of Public Health Managers at the urban public health centers will accelerate this process. Phase out recommendations for HUP 1. Document and disseminate the successes of vulnerability mapping and WCCs to create a demand. Document experiences as well as processes to ensure their utility and effective implementation. Process documents include: processes for forming MAS, conducting a UHND, managing WCCs, carrying out CHP, and convergent decision making. 2. Delineate an exit strategy. Initiate institutional processes at GOI level for expanding public health center services/human resource norms and strengthening intradepartmental coordination. At the city level, the exit strategy must include the identification of a support agency to provide needs￾based TA, NGO partners to support community processes, a city resource center equipped with a set of HUP products and process documents, and key multi-sectoral officials at the state level with which city teams could link. 1 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT 1. INTRODUCTION EVALUATION PURPOSE The purpose of this final evaluation of the United States Agency for International Development/India’s (USAID/India) Health of the Urban Poor (HUP) program is threefold. First, it explores the program’s progress against its objectives and recommendations made in the midterm evaluation (MTE) conducted in 2012. Second, it evaluates the strategies and success of efforts to scale-up the program’s results from the first phase. Finally, this evaluation seeks to obtain insights and document lessons learned from the various components of the program. This information will help inform USAID/India’s future designs in urban health programming. EVALUATION QUESTIONS Specifically, the evaluation seeks answers to the following five questions: 1. To what extent the program has successfully addressed the major recommendations from its mid-term evaluation in its second phase? 2. How effectively has the program scaled up the activities/model interventions from the five cities in phase one to 18 cities in the second phase? In particular, the evaluation shall address the following sub-questions. 2.1 What have been the results (effectiveness) of the program’s key approaches, i.e. governance, institutional capacity strengthening and public-private partnerships? 2.2 What are the key achievements of the program and what are the key factors facilitating the achievements? 2.3 What are the key challenges and how could they be overcome? 2.4 What are key lessons learned in scaling up activities/model interventions? 3. To what extent has the program been successful in ensuring the convergence of various Government of India (GOI) efforts on improving urban health? 4. To what extent has the program influenced policy-level changes with regard to improving urban health at the national, state and city levels? 4.1 To what extent have cities or states adopted these models? 5. What lessons can be drawn from this program in terms of key strategic approaches and impact that should inform USAID’s future urban health focus? Given the overlap in the content of question 5 with the sub-questions from question 2, the evaluation team and USAID decided to subsume the findings, conclusions, and recommendations pertaining to question 5 under question 2. The primary intended users of this evaluation are USAID/India, USAID/Washington, and the GOI at the national and state levels. In particular, the USAID/India Health Office, Program Support Office, and Mission management are interested in findings and recommendations concerning the progress of health innovations and partnerships within this program. Secondary audiences of this evaluation include: local institutions, other donors, and other missions worldwide. Answers to these questions should provide useful guidance on the extent of success achieved by the program, the facilitators and impeders influencing the success, and how HUP strategies and lessons might be best deployed to strengthen the implementation of urban health programming in the country. 2 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT The evaluation was commissioned through USAID’s Asia Learning, Monitoring and Evaluation Support Project IQC under Task Order AID-386-TO-15-00002 with Social Impact, Inc. (See Annex I for a copy of the complete Scope of Work.) PROGRAM BACKGROUND India has made good progress in achieving health outcomes over the last decades, especially in the rural areas, but the urban poor have generally not benefited. India is urbanizing rapidly, and the urban poor, estimated to number around 77.5 million, are one of the country’s fastest-growing and most vulnerable population segments. The delivery of health services in urban areas has been sub-optimal and fragmented. Past interventions have tended to be in the form of vertical programs focusing on particular diseases, rather than investments made to strengthen broader urban health systems. The facilities established under GOI’s urban Reproductive and Child Health (RCH) Program along with the limited urban health facilities established by urban local bodies (ULBs) have been the mechanism for delivering health in the urban context. These facilities suffer from weak referral linkages, are underutilized, vary in norms and quality, and have limited scope of services, such as in community outreach and health promotion. As a policy response, in May 2013, the GOI launched the National Urban Health Mission (NUHM) to strengthen health service delivery in urban areas1. In 2009, USAID/India’s Office of Population, Health and Nutrition initiated the HUP program. The Cooperative Agreement (513,601,582 Indian Rupees [INR]; 10,778,627 United States Dollars [USD]) was awarded to the Population Foundation of India (PFI)2. HUP is the first USAID award made directly to an Indian non-governmental organization (NGO). PFI’s consortium partners on the HUP program include: Plan India; the Institute of Health and Management Research -Jaipur (IIHMR); the Bhoruka Charitable Trust (BCT); the Centre for Development & Population Activities (CEDPA); the International Institute for Population Sciences (IIPS); and the Micro Insurance Academy (MIA). In addition, the program had one or more community-based implementing partner organizations in each of the cities in which HUP operated. Within the context of the imminent launch of NUHM, the HUP program was designed to support the achievement of improved delivery and utilization of maternal, child health and nutrition services, including the promotion of water supply, sanitation, and hygiene services to urban poor communities. HUP was expected to develop models and policies to address the needs of the urban poor through the provision of technical assistance (TA) to state, municipal, and community level institutions. The program was originally planned for four years and builds on nearly 15 years of USAID/India’s work on urban health. HUP program objectives include the following: 1) provide quality TA to the GOI, states, and cites for effective implementation of the NUHM; 2) expand partnerships in urban 1 Supporting National Urban Health Mission; Program Safeguard Systems Assessment; Dec 14, Asian Development Bank 2 The list of consortium partners and their geographic coverage is provided in Annex 11. 3 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Onset of HUP: Oct 2009 Onset of Activities: June 2010 Mid term Evaluation: July 2012 NUHM Launch: May 2013 Onset of NUHM Activities: Jan -Jun 2014 Final Evaluation: Jul-Aug 2015 health, including engaging the commercial sector in public-private partnerships (PPP) activities; 3) promote the convergence of different GOI urban health and development efforts; and 4) strengthen the evidence-based rigor of city-level demonstration and learning efforts in order to improve program learning. One of the program design’s chief assumptions—which ultimately did not hold true—was the rollout of NUHM with the onset of the HUP. Furthermore, HUP could not initiate activities for the first ten months owing to delays in necessary approvals from the GOI. At mid-term therefore, in the absence of NUHM and the accompanying reluctance of state governments to initiate urban health activities beyond those mandated under the urban RCH program, HUP had been implemented for 18 effective months. The possibility of ending the HUP contract in the face of delays was indicated to the MTE team. Based on mid-term recommendations, HUP was extended to September 30, 2015 in order to overlap with the launch NUHM in 2013 and facilitate its rollout. The extension phase was redesigned to provide TA support to the national and state governments, in rolling out NUHM, as well as scale up the lessons from the five demonstration approaches to at least 20 cities (in addition to the existing five HUP cities – see map). These included 18 cities in the existing HUP states and two mega cities outside the HUP states like Bangalore and Kolkata3. HUP thus sought a more comprehensive vision of health following the recommendations of the MTE and adjusted its activities to renew a focus on water, hygiene, and sanitation (WASH); improving the supply side of urban health service delivery; and scaling up the demonstration models to scale-up cities. State City State City Bihar Patna Uttarakhand (UK) Dehradun Odisha Cuttack and Rourkela Uttar Pradesh (UP) Lucknow and Kanpur (Nagar) Chhattisgarh (CG) Raipur and Bhilai Jharkhand Dhanbad, Ranchi & Jamshedpur Rajasthan Jodhpur, Kota & Ajmer Madhya Pradesh (MP) Gwalior, Indore, Bhopal & Jabalpur Karnataka Bengaluru West Bengal (WB) Kolkata EVALUATION METHODS & LIMITATIONS METHODOLOGY The evaluation was carried out by a four-member team during July and August 2015. The evaluation team (ET) consisted of a Team Leader/Senior Evaluation Specialist, a Senior Public Health Specialist, a Senior Management, Governance and Private Health Sector Expert and a Research Assistant from Social Impact, Inc. The evaluation uses a qualitatively-dominant albeit mixed-methods approach involving (1) a desk review of available primary and secondary documents; (2) site visits to cities involving semi- 3 The HUP approaches, which are part of the NUHM framework and which were tested by HUP in the five demonstration cities are (i) community empowerment through slum women’s groups called Mahila Arogya Samiti (MAS), (ii) outreach services in urban slums through Urban Health & Nutrition Day (UHND), (iii) Health Management Information System (HMIS) based on name based tracking of slum women and children through Mother & Child Tracking System (MCTS), (iv) City Health Planning (CHP), and (v) convergence platforms at city and ward level through City/Ward Coordination Committees (CCC/WCC). 4 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT structured key informant interviews, observation and focus groups; (3) semi-structured key informant interviews at the national level and (4) quantitative analysis of data reported by the program. The ET examined the questions through a gendered lens through the inclusion of specific gender-focused questions the data collection tools and consideration of HUP’s differential effects on men and women during the analysis. The detailed evaluation design and methods are provided in Annex II: Evaluation Design and Methods. DATA SOURCES Desk review of documents: USAID/India provided the ET with documents including proposals, the MTE report, monitoring indicators, and other relevant documents for conducting this desk review. HUP also provided an extensive group of documents at the national, state, and city level. These included training modules, guidelines, national and local-level studies, quarterly reports, and others. For more detail on the documents reviewed, see Annex IV: List of Information Sources. Site visits: Given the time available to complete field visits, the team sampled six cities for site visits using a purposive sampling technique. The ET developed a city matrix to detail the program efforts and, through this, identified the following parameters for selecting the cities for the site visits:  Phase of implementation;  Availability of demonstration model;  Availability of PPP models that can be reviewed; and  Specific convergence activities of interest. To address the importance allocated to the scale-up efforts reflected in the evaluation questions, the ET sampled one Phase I city, three scale-up cities, and two demonstration cities. The team also attempted to include as many HUP states as possible to better understand the diverse contexts and implementation environments. The cities thus selected were Bhopal, Cuttack, Jodhpur, Bhilai, Pune, and Agra. At the request of USAID and PFI, Dehradun (UK) was also visited. Interviews were included at the state level in Bhopal (MP), Raipur (CG) and Bhubaneswar (Odisha), taking advantage of these capital cities’ proximity to the sampled cities. In addition, the ET conducted a number of telephonic interviews with key stakeholders in other cities/states at the request of USAID and PFI. These were Bihar (Patna), Rajasthan (Jaipur), and two non-HUP cities, Bengaluru and Kolkata. In all, data was collected from eight cities (two demonstration cities, four scale-up cities and two non-HUP cities) and seven HUP states. See Annex V: Data Collection Schedule for additional detail on the places visited and people consulted. Qualitative data was collected through semi-structured interviews, focus group discussions (FGD) and observation of the Urban Health Nutrition Days (UHND) and Urban Primary Health Centres (UPHC). The ET conducted interviews with a diverse group of stakeholders who include representatives of the Ministry of Health and Family Welfare (MOHFW), Ministry of Woman and Child Development (WCD), ULBs. Municipal Departments, TA agencies (National Health System Resource Centre/State Institute of Health and Family Welfare/State Health Resource Centres/Indian Institute of Public Health) and facility and community-based government health workers and stakeholders in urban slum areas such as Medical Officers, Auxiliary Nurse Midwives (ANM), Accredited Social Health Activists (ASHA), Anganwadi Workers (AWW), Mahila Aarogya Samiti (women’s groups, or MAS) members. The team further conducted in-depth interviews with representatives of PFI and HUP partners; USAID, and other organizations working in urban health. In all, the team conducted 158 interviews (102 men, 56 women), 15 FGDs (4 men, 187 women), and 13 facility/event observations. The guidelines for interviews and FGDs along with the checklist for observations are provided in Annex III: Data Collection Tools. Quantitative data: The program’s endline survey has not yet been conducted, limiting the quantitative analysis to that generated by the program’s Health Management and Information System (HMIS). This 5 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT system includes outcome indicators in reproductive, maternal and child health and WASH, and output (program) indicators such as the number of MAS groups formed. Data from non-HUP states and cities was not available for comparison. For a detailed description of the indicators, see Annex II. Data Analysis: The findings of the evaluation are based on an analysis of the data collected from several sources, including detailed notes from interviews and site visits, extensive document review and an assessment of quantitative results generated by the program’s HMIS. At the end of the in-country evaluation, the ET presented preliminary findings to USAID/India and PFI. Qualitative data was triangulated with the quantitative results, including the feedback given during the in-country de-briefings, and presented in the report to arrive at conclusions. Details of data analyzed are provided in Annex II. LIMITATIONS AND THREATS TO VALIDITY As with any short-term performance evaluation, the ET was restricted by its limited fieldwork schedule. The team had roughly three weeks together in country for start-up meetings, data collection in eight cities, and data analysis. The evaluation is also limited by the limited data available to triangulate results. HUP conducted a baseline survey that measured relevant health outcomes (e.g. utilization of health services, access to water and sanitation, etc.), however, the endline survey has not yet been conducted, eliminating the possibility of the evaluation team to compare the figures over time. The varied launch dates for NUHM in the states mean that there is no data available on key NUHM indicators to compare between non-HUP and scale-up cities. The evaluation faces internal threats to validity as well. The ET largely relied on HUP staff to identify respondents and make appointments for the fieldwork. For the FGDs and site visits, there may be selection bias. While the ET requested that they visit a “typical” facility and MAS group at each site, HUP may be motivated to choose the more cohesive and mature MAS groups and the more developed facilities. Lastly, as is typical in evaluations of this type, an external threat to validity exists through response bias, whereby respondents felt a natural tendency to provide answers that they believed the interviewer wanted to hear. While the factors presented above do raise concerns for the validity of the evaluation findings, the strength of the evaluation design should help to overcome these weaknesses. The ET spoke to a large number of key informants at all levels and visited a large number of sites. It also reviewed a wealth of documentation from and about the program and attempted to obtain as much quantitative data as possible to measure whether there were objective improvements in urban health over the course of the program. These data sources provided an opportunity for the team to triangulate its findings and limit the influence of the threats to validity mentioned above. See Annex II for additional details on limitations and how the ET addressed them. 6 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT FINDINGS AND CONCLUSIONS The objectives of HUP were modified in the extension phase accordingly: # Original Objective Extension Objective 1 Provide TA to urban programming Provide TA to the national and state governments in rolling out NUHM 2 Demonstrate models of urban health programming Scale up the lessons from the demonstration cities4 to at least 20 cities 3 Strengthen convergence of urban health activities Expand the implementation of convergence to address health determinants 4 Promote models of PPPs within urban health Serve as a platform to convene influential stakeholders around urban health issues This section presents the efforts made by HUP to meet these two sets of objectives and the outcomes emanating from such efforts; and provides the answers to the evaluation questions. The following matrix delineates the organization of the section. Evaluation question HUP Objective Evaluated Extent of mid-term recommendations addressed? All four objectives/program management Effectiveness of the scale up, results of approaches, achievements, challenges and key lessons? Objectives 2 and 4 Success in ensuring convergence? Objective 3 Success in influencing policy change and adoption of models by cities and states? Objective 1 Lessons and key strategic approaches for USAID’s future urban health focus FINDINGS: EVALUATION QUESTION 1 (MID-TERM RECOMMENDATIONS) Evaluation question 1: To what extent the program has successfully addressed the major recommendations from its MTE in its second phase? TECHNICAL ASSISTANCE Recommendations: At midterm, limited by the disinclination of the states to address urban health beyond the urban RCH mandate, HUP had been supporting governments develop action plans for implementing urban RCH programs. However, in some states where the environment was conducive (Odisha, MP, and CG), HUP had created a comprehensive urban health plan and had supported the operationalization of the plan, for example the Mukhyamantri Shahari Swasthya Karykram5. The MTE, recognizing the efforts as well as the additional potential in states such as Rajasthan, recommended that HUP continue to facilitate the development of a state urban health vision, and develop an urban health strategy to implement a comprehensive urban plan and urban health model. Furthermore, identifying 4 The HUP approaches, which are part of the NUHM framework and which were tested by HUP in the five demonstration cities are (i) community empowerment through slum women’s groups called Mahila Arogya Samiti (MAS), (ii) outreach services in urban slums through Urban Health & Nutrition Day (UHND), (iii) Health Management Information System (HMIS) based on name based tracking of slum women and children through Mother & Child Tracking System (MCTS), (iv) City Health Planning (CHP), and (v) convergence platforms at city and ward level through City/Ward Coordination Committees (CCC/WCC). It must be noted that WCCs are not a part of the NUHM framework. 5 The Chief Minister’s Urban Health Program 7 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT that the design of HUP required it to focus on maternal and child health (MCH)/RCH elements, the MTE team recommended that HUP redefine the scope of health, articulate this vision, and reorient staff. Findings: Comprehensive urban health vision and plan: In response to the above MTE recommendations, HUP developed urban health and plans (Odisha and CG); supported reorganization of state health society to include stakeholders from other relevant departments (UK); generated evidence through facility assessments6 and studies7, addressed both health and health determinant issues under urban health (all states); incorporated health determinant approaches in state NUHM Project Implementation Plan (PIP) for 2013-14 (Rajasthan); advocated and oriented urban health functionaries on comprehensive public health issues and needs assessments; and influenced policy level decisions to expand the scope of urban health to include water and sanitation activities (CG, UK, and Rajasthan). With the onset of NUHM, HUP supported the development of PIPs by involving a number of stakeholders both at the central level and at the state levels in eight Empowered Action Group (EAG) states. At the central level, HUP drafted the guidelines for PIPs and developed the financial norms for NUHM fund allocation. HUP thus supported eight state and 359 city PIPs. In addition, HUP facilitated the formation and orientation of state and city level program management units (CPMU). HUP’s success is reflected in its being nominated as a member in the NUHM steering committees and other similar committees in all the states8. HUP facilitated the establishment of a Memorandum of Understanding (MOU) for training functionaries through technical agencies with the State Institute of Health and Family Welfare (SIHFW, Rajasthan) and the State Innovation in Family Planning services project agency (SIFPSA, UP). Although comprehensive urban health vision and plans were limited to two of the eight states, a plethora of evidence generated and advocacy for urban health accelerated the uptake of NUHM in HUP states. As described by senior decision makers who interviewed in the HUP states, “urban health started with HUP,” and “HUP has advocated for and created an inclination for urban health activities.” Findings: Redefining and expanding the scope of ‘health’: In addition to preventive and promotive RCH issues, a comprehensive public health approach to urban health requires that communicable diseases (such as malaria, tuberculosis [TB], polio, meningitis, sexually transmitted infections [STIs], HIV/AIDS, avian and swine flu, Dengue, Chikunguniya), NCDs (such as cardiovascular disease, diabetes, and metastatic disease) and social determinants of health are addressed. It must be noted that HUP was advised by USAID to expand the scope of health to include Reproductive Maternal Neonatal Child Health plus Adolescents (RMNCH+A), as opposed to MCH, which had been the focus at MTE. HUP’s support to the development of primary health services bouquet within the NUHM framework has created a pathway for addressing comprehensive services9. Although these are yet to be completely implemented through UPHCs, the ET found several UPHCs that provide services beyond RMNCH. The definition of health has been expanded from merely MCH to more comprehensive services especially at the UPHCs. The framework has yet to be completely operationalized and has been delayed owing to vacancies within the UPHCs. At the community level, the services continue to focus on MCH 6Anganwadi Centres (AWCs), urban health facilities 7Point of Use (PoU), Policy papers, Burden of Disease and State fact sheets 8 In Bihar HUP has been nominated as a member of the Executive Committee SHS. In MP HUP is a member of the State Urban Planning Unit. In Odisha HUP has been nominated as a member of the State Health Mission. In Uttarakhand HUP-PFI has been designated as the technical agency to provide support for NUHM. 9 NUHM Framework for Implementation; http://nrhm.gov.in/nhm/nuhm/nuhm-framework-for￾implementation.html 8 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT interventions (antenatal [ANC] and postnatal care, safe delivery services for pregnant women and child immunization) and a few nutritional services. The community level services do not cater to men’s health issues10. However, owing to HUP’s advocacy, services such as distribution of chlorine tablets, testing of water quality, identification of mosquito breeding sites have been added to the community level services. Similarly, in Pune the UHNDs include screening NCDs and in CG pregnant women are screened for gestational diabetes. While the framework for urban health services is comprehensive, it does not include strategies to address substance abuse (a recognized urban phenomenon) under UPHC service norms; nor does it delineate strategies for intra-departmental coordination for implementing various vertical health programs. Furthermore, while mental health is articulated as a primary level service, a Counselor’s position has not been envisioned at the UPHC level. Conclusions: HUP created state level preparedness for the launch of NUHM and developed a comprehensive primary health service framework. It has realized the potential matrix delineated by the MTE except for addressing intra-health coordination. However, the post-NUHM potential for an integrated and convergent urban health model—one which delineates the role of each stakeholder and the funding source for each planned intervention—could not be found in any state. The nascent capacities of the states to innovate beyond the NUHM guidelines could not be overcome by HUP. PUBLIC-PRIVATE PARTNERSHIPS (PPP) Recommendations: The MTE recommended that HUP strengthen PPP cells under the National Rural Health Mission (NRHM); support the development of specific PPP guidelines for urban health; disseminate and document successful PPP models; and encourage adoption of such models. Findings: While a PPP policy for urban health was developed in Odisha and in states such as Rajasthan and UK, PPP policies for NRHM have been utilized for forging PPPs under NUHM. Although PPP guidelines for urban health were developed in Rajasthan, they were not adopted by the state. The PPP cell under the CG NRHM is fully functional and the Mobile Medical Unit for urban health is being implemented in PPP mode by the state. HUP facilitated this process. HUP published a list of existing best practices in PPPs for urban health through its partner, CEDPA. This was not widely disseminated, however. State level respondents in UP and Rajasthan were not aware of this publication. Similarly, PPP models within HUP states were documented and shared with the state officials. The urban health centers operational in a PPP mode in UK were assessed and documented. Efforts were made to promote Corporate Social Responsibility (CSR) in urban health and WASH; however, this has not led to any documented results. Various stakeholders in the government articulated the lack of inclination among states to pursue PPPs for urban health. The current impetus is to establish a public health system for the urban areas. The respondents further articulated the mistrust that exists between the private and public sector as one of the impeding factors for this disinclination, “We are not happy with the services provided by the private partners,” said one official in UK. Another official in MP stated, “We want to strengthen the public systems.” Similarly, the private sector (including the NGOs) experiences systemic impediments while partnering with the government programs. Delays in fund disbursement and inflexibility of terms of reference are cited as influencing factors. However, the ET’s interviews also uncovered examples within the public health sector of institutional mechanisms that could alleviate some of the unease that exists. These include the establishment of a Private Public Interface Agency structure by the national TB program; and 10 Discussed under Question 2 9 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT the provision of family planning and reproductive health services through social franchise networks in UP and Haryana. Conclusions: HUP could not move forward on the PPPs in the absence of sufficient interest within the governments. The importance of this missed opportunity to build on what previously existed within the governments is now clear in light of the barriers being faced by the NUHM to recruit and retain health functionaries in the urban areas11. CONVERGENCE Recommendations: The MTE recommended that HUP support the establishment of convergent platforms such as Ward and City Level Coordination Committees (WCCs and CCCs); strengthen cross learning between states; and facilitate the engagement of NGOs and other urban health stakeholders. Findings: HUP facilitated the creation of convergent platforms; alternatively, urban health decision makers were incorporated into existing societies and committees such as State and District Coordination Committees/Joint Steering Committees/State Health Societies/Missions. At the community level, HUP approaches facilitate convergence as well. The UHNDs promote convergent activities and the MAS are recipients of convergent actions. HUP developed Operational Guidelines for CCCs, WCCs, Urban Health Cells, UHND, and MAS, some of which (CCCs, UHND, and MAS) have been adopted at the central and state levels. While CCCs have been formed or are in the process of formation across the eight HUP states, the WCCs exist only in CG, Cuttack, Pune, and Agra. The NUHM implementation framework does not mandate the states to establish WCCs. Furthermore, the CCCs are not performing at optimal level yet. The functionality of convergence structures is discussed under Evaluation Question 2. HUP organized several visits for state officials (both HUP and non-HUP) and HUP team members to demonstration cities. The non-HUP states include Maharashtra, Karnataka, WB, and Tamil Nadu. HUP facilitated exposure visits of an Ethiopian team to Bhubaneswar as well. These visits were cited by officials in Karnataka and WB as having influenced the adoption of urban health approaches, especially UHNDs and MAS. Additional outcomes of the information exchange between states include implementation of micro birth plans and post-natal care checklists; formation of model anganwadi centers (AWCs); introduction of colored referral slips, household listing and mapping, PPP activities, and improved linkages with Nutrition Rehabilitation Centers. However, the coordination with other programs and stakeholders has been minimal and limited to invitations to advocacy and dissemination workshops. The central government has issued guidance pertaining to domain and coverage of activities to health organizations in India. This has been cited as the limiting factor by HUP for influencing engagement of various partners in urban health. Conclusions: Convergence has been initiated; however, the structures are not strong and processes not yet entrenched. The inability to create a network of agencies supporting the NUHM beyond HUP support has been a missed opportunity. WASH Recommendations: The MTE recommended that HUP revisit and prioritize the scope of WASH, restate performance indicators, and leverage partnerships with other NGOs and ULBs to expand the WASH agenda. 11 Discussed under question 2 10 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Findings: HUP incorporated WASH indicators into its monitoring plans and integrated WASH components with the country’s supplementary nutrition program, the Integrated Child Development Services (ICDS). There was ample progress and outcomes on integration as is reflected by the WASH interventions in the PIPs of seven states. Activities for WASH have been limited to three components: hand washing, improving water quality at Point of Use (POU), and improving toilet use. However, additional WASH activities have been initiated at the state level. These include: WASH activities incorporated into the mid-day school meal scheme in UK; the WASH communication material adopted by the Public Health and Engineering Department (PHED); establishment of community toilets in Jamul, CG through a PPP between Department of Urban Development, ACC Cements Pvt. Ltd. and NGOs; and identification of vulnerable pockets for water-borne and water-based vector-borne diseases and subsequent sanitary inspections and drinking water quality monitoring in Odisha and CG. State and city officials reported reductions in mortality and morbidity from water-borne outbreaks as a result of these actions, however, this could not be corroborated without official data. The availability of chlorine tablets at the community level was a norm in all the sites visited; the two Indian ET members have not experienced this in their 50 collective years of experience. Conclusions: HUP was successful in incorporating WASH as an integral part of health—an occurrence yet rare in the older rural health sector in India. Missed opportunities include strategies for solid waste management—traditionally the role of ULBs—and menstrual hygiene. While HUP cites USAID’s guidance on not including solid waste management issues, the ET did not find any efforts to address the latter either directly or through organizations working on the issue. DEMONSTRATION MODELS Recommendations: The MTE recommended that HUP systematically document the capacity of state and municipal urban health systems, track beneficiary-level indicators, carry out an endline survey, conduct operational research to determine meaningful MAS incentives, and conduct an impact assessment of demonstration sites. Findings: HUP supported the analyses of the urban health scenario in the cities which have been used for developing the city health plans (CHP) and the State Health Plans. Analyses include assessments of municipal hospitals, urban health posts, UPHCs, AWCs, and facility-based and community outreach workers (ANMs, ASHAs, MAS/community groups etc.) However, not all gaps identified and needs cited by the cities have been approved by the state or by the center. Although HUP has carried out and documented several research studies, it has not attempted process documentation of the city demonstration models. HUP revised its monitoring system to capture three types of indicators—some RMNCH indicators, WASH indicators, and process indicators of the scale up cities. These were finalized with USAID following a series of discussions. However, the HUP Mother & Child Tracking Systems (MCTS) are not linked to the government’s facility-based MIS tracking system. The government’s MIS in the urban areas is currently performing at sub-optimal levels and does not disaggregate data for urban slums. Hence HUP did not attempt to link these two systems. Consequently, the beneficiary records maintained by HUP cannot be cross-linked with the facility records. The effort made by HUP to address MIS at the central level has met with systemic challenges. The ET was unable to learn about the current challenges and future MIS plans of the NUHM from the perspective of the NRHM MIS official. The forthcoming HUP endline survey will measure changes in basic impact indicators over the life of the program (since the baseline survey). Data will be available in September. The post MTE status of demonstration cities is detailed in Annex VII: Demonstration Cities. 11 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Conclusions: While the MTE recommendations have been carried out, the phasing down of activities in the demonstration cities have meant reduced availability of resources to carry out suggested operational research activities. MANAGEMENT AND GOVERNANCE OF THE HUP PROGRAM Recommendations: The MTE recommended that HUP leverage the strengths of HUP partners, promote cross-learning, and ease administrative and reporting processes among others. Findings: Although cross learning was encouraged by HUP through quarterly program review meetings and annual thematic workshops, staff members in several states cited the need for a strengthened effort. The teams reported receiving some benefit from the strengths of each partner, however articulated a need for further support especially in the public health domain. HUP, based on MTE recommendations reduced administrative burdens and streamlined the approval and budgetary procedures which was deeply appreciated by the partners. The MTE recommended that HUP state plans need to be agreed upon by the state government. However, HUP work plans were being approved by GOI (as per the contractual obligation) and HUP state level staff reported this as a limiting factor in making major changes to the plans in response to state-specific needs. Nevertheless, activities and stakeholder perceptions suggest that HUP teams have been responsive to state needs. Ranking of cities by their progress on key indicators was recommended by the MTE to ensure healthy competition among HUP teams. This has been carried out; however, staff reported little communication between the teams from different partner organizations. The Technical Advisory Group (TAG) met every six months, reviewed the program, and provided guidance. A Technical Resource Group (TRG) for NUHM was formed at the national and state levels as well. HUP was included as a member of the TRG at both levels. The program’s internal reporting was reduced from a weekly to a monthly schedule. This resulted in reducing management loads and was reportedly appreciated by all the partners. Conclusions: While most program management recommendations were met, the gaps in support available to the state level HUP teams could not be completely bridged as the HUP worked with limited resources and a larger scope following the MTE. SUMMARY CONCLUSION FOR QUESTION 1 HUP has complied with most of the MTE recommendations. However, there has been limited cross learning, and the challenges identified in the above section limited the progress on PPP and convergence objectives and some program management issues. FINDINGS: EVALUATION QUESTION 2 (EFFECTIVENESS OF SCALE UP) Question 2: How effectively has the program scaled up the activities/model interventions from the five cities in phase one to 18 cities in the second phase? In particular, the evaluation shall address the following sub-questions. 2.1 What have been the results (effectiveness) of the program’s key approaches, i.e. governance, institutional capacity strengthening and PPPs? 2.2 What are the key achievements of the program and what are the key factors facilitating the achievements? 2.3 What are the key challenges and how could they be overcome? 2.4 What are key lessons learned in scaling up activities/model interventions? 12 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT The quantifiable results committed to demonstrate the achievement of objective 2 of HUP and the results are presented in Annex VIII: Achievements Against Quantifiable Outcomes. HUP approaches tested by HUP in the five demonstration cities are (i) community empowerment through MAS, (ii) outreach services in urban slums through UHND, (iii) HMIS based on name based tracking of slum women and children through MCTS, (iv) CHP, and (v) convergence platforms at city and ward level through CCCs and WCCs. Results of the MAS and UHND approaches are discussed in this section to assess effectiveness of community level institutional mechanisms. Similarly, results of HMIS are discussed to assess institutional strengthening and results of CCCs, WASH, and PPPs are discussed to assess effectiveness of governance approaches. Findings: UHND (Community institutional mechanism) HUP’s monitoring data show that the scale-up cities have held a total of 118,148 against a target of 90,832 for October 2014-June 2015, which is 30% over target (Figure 2.1, Annex VII) 12. While most of the scale-up cities exceeded their targets for the past nine months, Rourkela, Lucknow, and Kanpur lag behind and Cuttack has not reported data for this period. Two cities, Patna and Dehradun, did not have targets set because NUHM had not yet been launched in those cities. Data on the number of beneficiaries attending UHNDs were not available. Evidence of HUP’s results in establishing the UHNDs is also shown by the fact that HUP’s Operational Guideline on Urban Health & Nutrition Day (UHND) were incorporated within the national guidelines for MAS and outreach activities under NUHM, which was prepared by the National Health Resource Center (NHSRC). During site visits, the ET observed variations in the comprehensiveness of services offered and the convergence achieved in the UHNDs. UHNDs are organized jointly with ICDS with MAS groups, and in some places, the elected representative, the CPMU’s community mobilizers, and ASHA mentors (Raipur)13. While family welfare services and curative care are offered in Bhopal and Raipur, in some cities UHNDs are restricted to ANC, take home rations, growth monitoring, and immunization. It must be noted that growth monitoring was restricted to weighing of babies and did not include plotting of the growth chart and counseling based on weight. Furthermore, the take home ration was being distributed at the time of receipt of rations as the AWCs lacked storage space. Micro-plans have been developed for organizing UHNDs in several HUP cities. However, in one of the UHND observed by the ET, the activities had been split across three different days owing to non-convergent planning. The ET found that Health Education was limited and restricted to inter-personal communication; ANMs said that the UHND attendees did not want to spend time on Health Education as it increased their waiting time for services. The ET filled out checklists to document the presence of staff, medical supplies, and drugs at the four UNHDs it visited. ANMs, AWWs, ASHAs, and MAS members were present at all of the UHNDs observed. Most of the beneficiaries attending the UHNDs were mothers with children under age 5 and pregnant women; some women of reproductive age not accompanied by children, a few children older than age 5 and some grandmothers also attended, but no men were observed. Only one UHND had both an examination table and a privacy screen; at one the team observed examinations performed on a 12 Annual targets are adjusted by 0.75 to reflect the 9-month period of measurement. 13 Institutional structures under the NUHM 13 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT mat on the floor in the AWC. A hemoglobin meter was available at two of the UHNDs; none had urine examination kits, but there were also no toilet facilities. Iron Folic Acid tablets, deworming tablets like Albendazole, paracetamol tablets, chloroquine tablets, and Oral Rehydration Solution packets were universally available; condoms and oral contraceptive pills were available at 3 out of 4 of the UHNDs. As discussed earlier, one community observed did vaccinations on a different day than the UHND; the other UHNDs all were doing vaccinations as well. Other services included ANC check-ups (all); provision of supplementary nutrition (all); and referral services; only 2 of the 4 UHNDs observed were providing health check-ups, pre-school, nutrition and health education, or WASH activities. The ET also observed that infection control methods (using gloves, cleaning of injection site, etc.) were not always being followed by ANMs at the UHNDs. Water was not available for hand washing. Record keeping was poor and not systematic. The ET team did not find any men accompanying children and women or seeking care through the UHNDs although some provide curative care as well. Hence the UHND profile of providing RH/MCH services only appears to be well established in the communities. HUP conducted an assessment of UHNDs in Ranchi, Jharkand that examined the degree of coordination between health and ICDS services14. The team found a lack of coordination at both the management and implementation level. The study made a number of recommendations for improvement and convened a meeting of supervisory level and district officials of both the health and ICDS department to discuss the findings. The study provides evidence of HUP’s efforts in convergence to implement the UHNDs. Conclusions: The UHNDs are operational, however they do not currently provide the intended bouquet of services. At present, they are limited to family planning, child health, ANC, some curative services, and distributing chlorine tablets. Strategies to address men’s health through UHNDs is absent by design. Mandated services to promote adolescent health services; comprehensive RH services; counseling; and nutrition health education are not entrenched as yet. Coordinated planning with the WCD department is needed in some locations. Record keeping and MIS are not systematic currently. The ET concludes that this is the first stage in the maturity matrix of the UHND. With appropriate support and technical inputs, they can gradually mature to provide comprehensive services. Findings: MAS (Community institutional mechanism) Monitoring data shows that the scale-up cities are at various stages of implementation for establishing the MAS in each slum community (Figure 2.2, Annex VII). Half of the 18 cities exceeded their targets for MAS establishment by June 2015, while four cities (Patna, Dhanbad, Lucknow, and Kanpur) had not started implementation by that time. Five cities (Jhodpur, Ranchi, Jamshedpur, Gwalior, and Indore) had started establishing MAS groups but only were at 15-84% of their targets. It should be noted that this indicator only reflects whether the MAS has been formed, but not whether they are yet functional. The ET observed the MAS at various stages of formation, capacitation and financial functionality in their site visits. In Raipur and Bhilai, the MAS visited had started receiving capacity building inputs, whereas in Cuttack and Jodhpur the MAS had just been formed and opened their accounts, awaiting training and also guidelines for expending the amount. Some MAS show a high level of cohesiveness and are actively taking community actions. The ET visited strong 14 Gupta, P.K. (n.d.). UHND Assessment: Ranchi Urban Jharkhand. Unpublished report. 14 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT MAS in Bhopal that is spearheading community action for WASH, MCH service utilization, and financial support to identified vulnerable community members. The trained MAS groups met by the ET were mapping the community for beneficiaries and resources; actively mobilizing communities to access services; testing water at POU; reporting drainage and solid waste management problems; monitoring sanitation facilities; cleaning mosquito breeding grounds; promoting hand washing and use of hygienic practices for water handling; and reporting; monitoring epidemic outbreaks such as diarrhea and hepatitis and; demanding remedial actions from the system. The MAS women reported an increase in utilization of services by the communities and decreases in epidemic mortality. MAS are being formed either with the support of NGOs or through the ASHA network (ASHA, her supervisors, and ASHA district coordinator with ICDS supervisors). In Jodhpur, the MAS requirement and identification was based on a structured analysis of geographic rationalization of functional areas. In Cuttack, NGOs were subcontracted to form MAS groups for 500 INR per group. While this small amount was being used to hold three meetings with community women, including orientation and selecting MAS members, it did not provide for any hand-holding support for mapping, community actions or utilization of the funds available. A framework for engaging the interest and continued motivation of MAS does not exist. The FGDs with MAS revealed that the women had a strong commitment to improving health, water quality, and sanitation in their communities. The ET found that these groups showed a sense of agency in addressing issues, and that they knew the channels to be used for communicating problems. However, the MAS women expressed that there is a barrier to accessing and informing men in their community. “We get our diseases from men, we need services and education for them as well,” remarked one MAS member in Pune. They voiced a need for a similar community platform for men; and services for addressing men’s health. In Bhopal and Bilai, MAS members stated, “We don’t want men in our group; they should however form a group for men as well to address HIV, Leprosy, TB and substance abuse.” Furthermore, one of the groups from an unauthorized slum questioned the utility of group formation when the slum faces the threat of demolishment. The current absence of linkages between the MAS and socio-developmental systems/elected representatives/legal system is a threat to the sustainability of the groups. This is compounded by the absence of mechanisms for continued capacity building and incentivizing their participation. HUP technical partner CEDPA carried out a situation assessment of Behavior Change Communication (BCC) needs for urban health. Although this was shared with the national and state governments, owing to the current focus on infrastructure development, there has not been much focus on BCC needs and commensurate strategies. Conclusions: Wherever MAS has received capacity-building inputs, members are empowered and community actions are visible. There is a huge demand for information and skill-building among MAS members, fulfilling a clear need in the communities. Thus the structure of MAS has been effective and the ET saw concrete evidence of how the MAS groups were improving health, water, and sanitation in their communities. However, government does not have the inherent capacity or resources to establish and build capacity in the MAS, as ASHAs are inexperienced with limited knowledge and skills. Moreover, delays in the issuance of financial guidelines and training of MAS have the potential to cause group attrition. While the MAS have created empowerment opportunities for women, it is important to create avenues for men to participate in community action for health and WASH issues. Men’s participation can further bolster the health of women and children. The ET concludes that this is the first stage in the maturity matrix of the MAS. With appropriate support and capacity building, the MAS can gradually result in effective collective community action. However, the current set-up is not sustainable and greater inputs are needed, most feasibly through engagement of community based organizations (CBOs). 15 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT The current practice of using replicated rural and disease specific BCC tools in the urban areas will prove to be inadequate. Findings: CCC and WCC (Governance) By June 2015, eight of the scale-up cities had a CCC and five other cities in HUP states were forming them. However, these are not completely functional yet. The CCC members have been oriented to NUHM and in some cities received WASH training. Members of CCCs whom the ET met said that they had not held regular meetings in the last six months. Although there is a perception amongst the stakeholders that CCCs are not completely proactive, there are examples of decisions taken by CCCs collectively: delineating roles and responsibilities of the Municipal Corporation, health department, and WCD in managing NUHM; transfer of property from corporation to the health department for UPHCs; and allocation of funds for medicines in urban health centers. In one case, the CCC provided a platform for seeking stakeholder inputs to development of CHPs. However, the CCCs have not been able to create linkages with insurance programs, livelihood urban infrastructure programs, or employment. In Rajasthan, the state is utilizing the joint steering committees formed under NRHM as CCCs. In UP, coordination committees under the municipalities were constituted but meetings have not been regularly held. These committees were involved mainly in planning. WCCs are not part of the NUHM framework. Of the 234 WCC formations targeted by HUP, only 30 had been established by June 2015 (all in Cuttack). The rest are in various stages of establishment. It must be noted that MP has not agreed to the establishment of WCCs and CG has pre-existing WCCs (called Ward Kalyan Samitis) from its erstwhile urban program. HUP did not set targets for WCCs in the states of UP, UK, and Rajasthan. In UP the WCCs are in process of being formed, while in UK the government has deferred the formation of WCCs. In Rajasthan, the process of formation of WCCs was deferred due to panchayat elections in the state. They will be formed by the year end and budget provision will be made in 2016-17 for their capacity building. The ET met WCC members in Bhilai, Raipur, and Cuttack. The ET found that WCCs are leveraging social groups and community level leadership for promoting health seeking behavior, infrastructure, and monitoring of developmental inputs (including WASH) at the ward level. The WCC meeting attended in Cuttack saw the participation of the Ward Corporator (elected representative), a municipal representative, PHED official, opinion leader of the community, ANMs, ASHAs, AWWs, and teachers. A Ward Corporator in Bhilai suggested the opportunity to leverage untied funds is available with the Corporator for health action. However, this will require modification of state guidelines to avoid audit objections. WCCs in Pune have been addressing issues beyond health such as stray dogs, traffic jams, and electricity connections. Similarly in Bhilai, a park was created by the WCC in an area which was being used for open defecation earlier. Other examples include sanction of funds to build a railing along a canal to prevent accidents and the establishment of child toilets. The ET found ward level health committees as well as nutrition committees in MP; similarly, there exist other ward level platforms for other developmental areas. These have the potential for being leveraged for health purposes as well. Conclusions: The CCCs and WCCs, wherever formed, have played an important role in coordination and convergence and in development of city plans. Orientation, field visits, and mentoring of the committees have helped in developing a perspective on urban health. The ET’s observation of WCCs found them to be an excellent platform for convergence and immediate action in the urban context, as wards provide some degree of social cohesiveness in the otherwise disparate community fabric of the 16 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT cities. WCCs have identified and resolved local problems with local solutions and support. They provide a political platform for Ward Corporators/Councilors which can be effectively leveraged for health nutrition and WASH action, including financial resources. However, WCCs have as yet been established in only one scale-up city by HUP, and very minimal capacity inputs have been available as of yet. Both WCCs and CCCs have not reached their full potential and will need to be supported further to reach maturity and have sustainability. Findings: CHP (Governance) The program management structure at the city level is a CPMU. The CPMU has a City Program Manager, a Community Processes consultant, and an MIS staff member. The diversity of the structure and how they coordinate with the district health society (a forum under the NRHM) is delineated in Annex VI: Matrix of Diversity. The CPMU is responsible for the development of CHPs under the guidance of the CCCs and the district health society. HUP has helped develop CHPs in 18 +2 scale-up cities. It also facilitated the plans and drafted the budgets for 359 non-HUP cities in HUP states in 2014-15 and 94 in 2015-16. Sample plans reviewed by the ET reveal evidence-based development of strategies and activities. Evidence included the identification of the slum population, epidemiological data, WASH profiles, systemic status, and in some cities vulnerability assessments (VAs) as well. These are partly drawn from the State Urban Health Plans conducted in each state by EPOS Health India for PFI. VAs are also being carried out in Odisha, CG, and MP and will feed into future CHPs. Mapping and VAs conducted by HUP have led to identification of required services, rationalization of facilities, HR planning, and outreach routes of health workers. The states are in the process using the maps for planning, implementation, and monitoring the increase in accessibility and availability of quality services to the vulnerable populations. However, there have been challenges to accessing maps already created under the Jawaharlal Nehru National Urban Renewal Mission (JNNURM). Also, there are limitations to using Global Information System (GIS) mapping in the absence of VAs, which have not been completed yet in some of the HUP states. In CG, the maps developed by HUP were used by ASHAs to map the Hepatitis A epidemic and allocate evidence-based and timely support to communities. City officials credited this mapping effort with helping to achieve zero mortality in the most recent epidemic. HUP support in MP has led to an increase in allocation for urban health from 30 million INR in the first PIP to 630 million INR in the latest PIP. While allocations for urban health have increased based on actual needs in all HUP states, the central government observes that the expenditure against allocated funds has been slow in all HUP states compared to the other states of India. This observation has to be viewed in the context of all HUP states being historically weak performing states, however. Conclusions: HUP has successfully created an environment and commitment for urban health programming through orientation of stakeholders, evidence generation, handholding support, demonstration, and cross visits. Demand for city mapping has been generated, and these are being utilized for decision-making, although there are challenges that prevent universal use of mapping Findings: HMIS (Institutional Strengthening) HUP reports that HMIS has been customized for all scale-up cities. Customization included segregating data for slum and vulnerable populations, incorporation of WASH indicators, and formats for all levels of functionaries. However, these customized formats are not being used as the states have decided to follow the formats prescribed by the GOI. The GOI is not amenable to customizing MIS for the urban region and has decided to follow the rural indicators for ease at the systemic level. The current MIS does not segregate urban data for slums and non-slum areas. 17 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT With regard to monitoring urban health at the community level, HUP has simplified the monitoring system for the ANM by creating a daily diary in CG. The diary facilitates systematic service provision and monthly reporting to the MIS by the ANM. However, the diary and the guidelines for use have yet to be documented and disseminated to other cities and states. Conclusions: Although HUP has made efforts to support governments in customizing HMIS, these have not been implemented due to barriers at GOI level in institutionalizing urban HMIS. Findings: WASH in Scale-up Cities Monitoring data shows that in 2013-14, six of the 15 scale-up cities reporting exceeded their targets for improved drinking water (Patna, Bhilai, Ranchi, Jamshedpur, Gwalior, and Bhopal) and five of these for improved sanitation in all cities but Bhilai (Figures 2.3 & 2.4, Annex VII) 15. Other cities fell short of these targets, particularly in Rajasthan and UP. The ET observed the progress on integrating WASH strategies into all of the program approaches and, most notably, on incorporating these strategies into NUHM. WASH has been integrated as a component of state and city level PIPs. In each scale-up state, HUP conducted a WASH profile study with a needs assessment and policy review. HUP prepared a City Sanitation Plan for the City of Mussoorie in coordination with Urban Development. POU training occurred in Odisha and CG, facilitating improved management of epidemics such as Hepatitis A. As part of the Smart City initiative, the city of Durg has a mobile application as well as a WhatsApp group for reporting water contamination and solid waste dumps requiring clearing. These are widely used by MAS and ASHAs through the smart phones provided to the ANMs. Finally, although USAID guidance to HUP did not include solid waste management, community, ward, and city level action has led to focus on this issue. Conclusions: WASH has been institutionalized as an integral component in health and WCD outreach. This can be considered as a major achievement as similar inclusion is yet to be witnessed in the rural health sector. HUP has created models of sustainable platforms for continued WASH interventions through state urban health strategies, school health, within urban development, and ICDS interventions, through partnerships between municipal corporations and the private sector. POU models and mapping exercises wherever implemented have had huge impact on epidemic control. Findings: Primary Health Care Delivery Although this is not one of the key approaches that was the focus of HUP, it is important to understand the functioning of the health care delivery structures in the urban context. This discussion sets the context for PPPs in urban health care as well. The NUHM framework envisages a three tiered structure for delivering primary health in the urban areas. UPHCs will be formed for every 50,000 people. These will be supported by first referral units the Community Health Centres for every 250,000 people. One ANM, based at the UPHC will provide weekly outreach services to a population of 10,000. She will be supported by an urban ASHA for every 2,500 people and MAS for every 500 people. The ANM along with the ASHA is expected to support the MAS to promote community level collective action. The Medical Officer (MO) based at the UPHC provides overall public health management leadership to the designated urban area. The number of UPHCs established and made functional is 49, against 54 planned (91%) in HUP cities. Although some UPHCs have been upgraded or renovated by the municipal corporations, they have not 15 WASH indicator data is not available for 2014-15 18 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT been handed over to the health department yet16. The functional UPHCs do not have a complete complement of human resources; some provide services beyond RMNCH such as curative care for TB, NCDs, and communicable diseases, though this is not uniform across the centers. All UPHCs visited by the ET had ample supplies of medicines and supplies. In some states, the UPHCs function for a limited time during the day (MP, Rajasthan, and Odisha) while in others (CG) these function as 24/7 facilities with in-patient care. While the MO is expected to provide overall public health leadership for the covered urban community, the MOs are not adequately capacitated to take on this role. The UPHCs conduct monthly meetings with ASHAs however. While the position of public health managers has been envisioned under the NUHM framework, they had not been recruited by any of the states as the states are yet to receive the center’s approval. First referral facilities are being established and are not yet functional in most of the cities. Training sessions are being organized for UPHC staff members; however these are in various stages of completion. The CPMU teams currently support the UPHCs to plan outreach and facility-based services. Access to primary health care is still a challenge as infrastructure is being developed. CG has a structure (sub-centre – Swastya Suvidha Kendra) for a population of 10,000. Instead of a UPHC-based ANM providing outreach services, the ANM provides services through this physical structure and is available to the community during fixed hours. Some states have introduced mobile clinics to reach pockets unreached by UPHCs. Progress on indicators is not yet at optimal levels; however communities and health workers perceive significant increases in outreach activities and utilization of services following the inception of NUHM. Estimated total number of mothers and children reached through services in scale up cities from April 2013 to June 2015 Number of Pregnant women received 3 ANC checkups Deliveries conducted at Facility Number of newborn breastfed within 1 hour Number of fully immunized infants (9-11 months) 229,860 208,964(62%) 199,901 159,135(42%) 183,995 142,746(43%) 166,143 222,729(155%) The ET’s rough calculation found that against an estimated 0.43 million pregnant women and 0.56 million infants in the HUP cities, HUP reached 0.2 million (46.5%) with ANC services, 0.15 million (34%) with facility-based delivery services, and 0.22 million (40%) infants with immunization services17. Several challenges affect the establishment of UPHCs. The rationalization of facilities has to be completed, transitioning facilities from one department to the other. The localization of facilities based on needs is ongoing. Finding a place to build new facilities is a challenge in the urban context. The deliberate decision of the NUHM to pay the urban MO a lower salary package than the rural MO has made it difficult to recruit MO. Similarly, it has been difficult to recruit ANMs. Larger cities such as Bengaluru and Kolkata, where competing economic opportunities exist, have found it difficult to engage ASHAs and MAS as well. Most members of the urban marginalized households are not available in the day as they seek employment opportunities. The urban ANMs are reluctant to provide outreach services during the evening hours owing to security concerns. Similarly, the UHNDs and UPHCs which do not provide services during the evening hours will are potentially inaccessible to the population. 16 Prior to the launch of NUHM, health facilities were led and managed by various department including health, municipal bodies, ESI among others and each facility had varying service norms and lacked linkages with a higher facility. 17 Slum population of each city was calculated applying proportion of slum population for the state (except in the case of Bengaluru and Kolkata where city level data was available. Applying estimates of pregnant women and infants to this population, the above figures were calculated. 19 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT The potential for demonstrating an urban health delivery model which complements the public provision with models of private provision has been a missed opportunity for HUP. Findings: PPP The scale-up cities visited by the ET provided a few examples of active PPP interventions. HUP facilitated a partnership in Jamul, CG, between ACC Cement (for community toilet infrastructure), a community based NGO (for institutionalizing community monitoring) and the municipal corporation (for the land). The toilet was being used by 100 households for a monthly fee of 50 INR. However, the ET found an aggressive effort by the ward corporators to establish individual toilets under the Swach Bharatmission18. Mobile medical units have been established in CG and Odisha to serve the unreached urban slum population through state efforts. HUP has supported the government to develop service norms and route maps for the mobile clinic. Other support for PPP initiatives provided by HUP includes:  PPP cells under NRHM facilitated in CG and Odisha and oriented on PPP avenues.  In UP, mapping of all private sector players along with meetings with the Confederation of Indian Industries (CII) to leverage CSR funds. CII has decided to adopt Kanpur and will raise funds from the corporate sector.  Assessment of an outreach model in UP and discussions facilitated with Tata Motors for its implementing.  In Bihar, support to strengthen the government’s partnership with NGOs for the management of UPHCs (including Terms of Reference, service norms, and monitoring of implementation). This has included training of NGOs in NUHM, WASH, and service norms.  In Rajasthan, special PPP guidelines for service delivery and mobilization of the corporate sector. Three partners had been finalized: Rotary Club, Narayan and Lupin; however owing to change in the government, the partnerships could not be materialized.  In UK, implementation of UHNDs through the Ambuja Cement Foundation and promotion of best practices in health and WASH through Azim Premji Foundation. The ET also identified ongoing efforts by other agencies in leveraging private participation through social franchising networks, and subsidies by larger private hospitals in UP. In general however, states and cities were disinclined towards PPP. States such as UP and Rajasthan wanted more focus on infrastructure development through the private sector rather than service delivery. Furthermore, many of the governments have had poor previous experiences with PPP. For example, in UP, there was a PPP with a solid waste facility which failed. In Rajasthan, the government had poor experiences contracting out UPHCs to NGOs. HUP reported a high degree of skepticism among government officials regarding the motives of the private sector in service delivery, especially regarding misuses which has occurred with unnecessary procedures such as abortions and cesareans. There is no PPP policy or cell in most states, and government officials consider the private sector difficult to regulate. To date, government has mainly been a contracting agency and not a partner to a private sector player. Also, the cost of care in existing partnership models is still prohibitively high for the beneficiary (example: in Maharashtra where 50% of the cost is borne by the patient). The quantifiable results committed to demonstrate the achievement of objective 4 of HUP and the results are presented in Annex VIII. Conclusions PPP: HUP could not move forward on creating strong models for PPP as there was little appetite among the national and state governments to form partnerships. However, some examples of PPP have been implemented in a structured manner, such as the community toilet partnership with ACC Cement in Jamul. HUP also advocated for incorporation of micro-insurance models which would 18 GOI initiative to clean India 20 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT have been complementary to the Rashtriya Swasthya Bima Yojana (RSBY). However, it is possible that advocacy for piloting these models was not adequate. Existing insurance models in demonstration cities have not been linked to the HUP slum areas, which is perhaps a missed opportunity. FACTORS FACILITATING ACHIEVEMENTS The robust and evidence based advocacy efforts of HUP created an environment for prioritizing urban health. HUP successfully leveraged positive political environments to support the creation of the comprehensive framework for implementing urban health interventions (CG and Odisha). The evidence generated in the demonstration sites influenced the incorporation of community level and systemic mechanisms as an integral part of the NUHM framework. The responsiveness of the HUP teams to the states’ needs, beyond the program mandate, contributed to the rapid operationalization of NUHM along with creating a demand for technical assistance among non-HUP states. CHALLENGES Challenges of the HUP design: The restrictions on USAID funding guided the RMNCH focus of HUP, perhaps limiting the actions to achieve comprehensive urban health care model. Furthermore, as opposed to creating five different models in five demonstration cities, HUP chose to demonstrate one model of urban health delivery (CCC, CHP, UPHC, UHND, and MAS) This, as clarified by both USAID and HUP, was to generate evidence of success of these approaches in five different geographies, which would facilitate adoption by the government. While the strategy has worked and indeed the model has been adopted by the NUHM, the emerging challenges in implementing publically provided services would have benefited from complementary privately provided models, especially in the urban context where there is a thriving private sector. Challenges to initiating NUHM activities: The most notable challenge for HUP has been the late rollout of NUHM. This was compounded by national, state and panchayat elections; changes in governments in various states have resulted in delays of implementation of NUHM activities in states such as Rajasthan, Delhi, and UK. In addition, there were several challenges to the adoption of HMIS at the national level. States’ inclination to wait for NUHM guidelines has delayed activities such as MIS. Lack of national guidelines for urban HMIS, the unavailability of MCTS for states to adopt, and inadequate advocacy of NUHM from the HMIS team within the health department are some other contributing factors. Delayed recruitments of the CPMU teams and inability to leverage NRHM structures for urban areas in the absence of CPMU teams further compounded the delays. Challenges specific to urban areas: Human resource issues also affected implementation in many of the scale-up states. Many key informants noted that the lower salaries for doctors for UPHCs have made it difficult to form an urban cadre. The inability to recruit doctors, staff nurses, and ANMs has resulted in delays in scale-up of services and UHNDs. Where women have competing (and better paying) economic opportunities, it is difficult to recruit and sustain MAS members and ASHAs. Advocacy efforts for alternative models to address the dearth of human resources have not emerged, owing to the limited efforts by HUP in the context of the levels of interest within the government. Systemic Challenges: All government departments with health (and social determinants) components articulate annual plans, which are not considered or coordinated under the CHPs, creating duplication of efforts. It will require larger systemic and financial reforms to facilitate joint budgeting efforts by various departments. Introducing PPP has been a great challenge since the states are disinclined to form partnerships and there is lack of preparedness towards PPP. Convergence efforts were sometimes hampered by power dynamics between the hierarchies in ULBs and health. Challenges emanating from HUP capacity: HUP teams post-MTE have been reduced to two members at the state level and four members supporting all the scale-up cities. In addition, the 21 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT uncertainty about the end date for HUP resulted in frequent turnover in the HUP team. Public health capacities were perceived as inadequate at the state level within HUP teams. Despite efforts of the HUP teams to facilitate evidence-based planning and implementation, several states were not amenable to change, perhaps suggesting the need for a high-powered team to influence the government’s decision making process. HUP could not create a viable business demonstration model to overcome the government’s reticence. Furthermore, the state’s inadequate internal capacity to operationalize NUHM could have been bolstered by creating a network of technical agencies, which would have supplemented the small HUP teams. An example is the failure to involve the Indian Institute of Public Health in Odisha and at the central level. While several organizations were invited to participate in workshops, cogent strategies to engage their participation however are not evident. HUP could not influence the use of NRHM structures to manage NUHM in the initial stage of operationalization when state level NUHM teams were yet to be designated and recruited. This was not envisioned in all the state and therefore not attempted. Lessons from NRHM implementation, specifically the ability to leverage the flexible framework of NUHM to innovate, was not influenced either. Finally, an exit strategy is yet to be articulated by the HUP. Almost all the respondents at the state level expressed their concern that the HUP was coming to an end prematurely, especially in the light of the recent launch of NUHM. EFFECTIVENESS OF SCALE UP: OVERALL CONCLUSIONS AND LESSONS While stakeholder perceptions indicate that states with an HUP presence have moved ahead in an accelerated manner on operationalizing these approaches as compared to other states, the ET was unable to obtain NUHM indicator data from non-HUP states to do a comparison. Both quantitative and qualitative data show the eight states with scale-up cities to be in various stages of operationalization. A key factor in their progress is the differing levels of government support for NUHM and its approaches. Some of the key lessons are:  Scale up would have been accelerated if TA assistance was combined with implementation support, as NUHM is a nascent program and current capacities to roll out are low. In the context of limited HUP resources for scale-up cities, linkages with other stakeholder organizations would have helped bolster implementation support to the city administration.  Different cities have differing needs as well as opportunities and while the NUHM framework provides flexibility, it is inadequately interpreted by governments. Furthermore, mega cities have a specific set of problems which are difficult to address with the community approaches of HUP/NUHM.  There is a need to demonstrate alternate/complementary models of service delivery that address the dearth of human resources for urban health.  The structure of MAS is effective and empowering for women; but men also need to be mobilized and should be important agents in health and WASH issues.  WCCs are an excellent model for decentralized urban health and development management.  Governments will require support from community-based organizations to strengthen community processes and there is an articulated need for continued TA to operationalize the NUHM, especially as the capacities of the governments and intermediary organizations have reached optimal levels. FINDINGS: EVALUATION QUESTION 3 (EXTENT OF CONVERGENCE) Evaluation Question 3: To what extent has the program been successful in ensuring the convergence of various GOI efforts on improving urban health? Convergence Objectives of HUP A schematic representation of the areas covered by government departments and where HUP interventions are targeted is presented here. 22 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Findings: Convergence Processes HUP facilitated convergence between the ministries at the national level and in states with the departments of Health and Family Welfare, WCD, Urban Development department, ULBs and, in some states, PHED. Advocacy for convergence was conducted at the national and state levels. National level convergence meeting was organized with the Ministries of Health, Urban Development, and WCD and a joint statement was issued underscoring the need for NUHM. The various processes for convergence in the states include convergent forum meetings, workshops, joint orientations on urban health and health determinants for NUHM, joint assessments and joint monitoring visits with various related departments. In the states of Rajasthan, Bihar, and CG the Education Department and elected representatives were also involved. A compendium of government orders on inter-sectoral convergence was compiled in UK to advocate for convergence. A roundtable on Safe Water and a WASH summit was organized jointly with the Government of Rajasthan. In CG, the program identified nodal persons within each department to increase convergence and accessibility. Convergence Platforms: The current platforms for convergence have been identified or have been formed with support from HUP at the state, district, and ward levels (State Health Societies/State Coordination/State Steering Committees, District Health Societies/District Coordination/State District Steering Committees, and Ward Committees). The CCCs and WCCs have been envisaged as Apex Bodies for the convergence of various stakeholders involved in the delivery of MNCHN and WASH services. These committees consist of elected representatives, health officials, sanitary officials, education official, municipal officials, MAS members and NGOs. The current status of CCCs and WCCs; and convergence with WASH interventions has been discussed under question 1 and 2. For detail on achievements against quantifiable indicators, see Annex VIII. The convergence efforts of HUP resulted in WASH practices being adopted in AWCs and UPHCs. This process was facilitated through joint assessments and planning. Examples include the replication of WASH Information Education Communication (IEC) material prototypes in Haridwar in partnership with Punjab National Bank; adoption of IEC materials by the Raipur Municipal Corporation to control the epidemic in the city; WASH IEC materials printed and distributed through PHED budget; WASH module utilized to train PHED personnel and hosted on the PHED website; onset of the global hand washing day campaign at urban AWCs in Dehradun city; and incorporation of WASH training within the mid-day meal scheme of the education department. HUP has evolved an important model of integrating WASH with MNCH at community level through promotion of POU water testing and treatment. The model has been adopted in seven HUP states. The convergent activities facilitated by the UHND and MAS approaches have been delineated under question 2. Although the convergent actions were evidently strong at the community and the ward level (wherever WCCs are available), there is an overall perception amongst the stakeholders that convergence has been initiated but not achieved at state and city level. A senior state official articulated the need for a facilitation agency to bring about convergence of various government departments. Stakeholders also 23 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT articulated the need for various departments to understand each other’s programs and provisions; and how these could be leveraged for health. Challenges and Limitations: HUP’s efforts to facilitate convergence with other urban development initiatives were limited. HUP did not specifically bring about convergence of other urban developmental programs such as JNNURM, Swach Bharat Mission, the Smart City campaign, or the National Skill Development Mission. The convergence with the Urban Development Department at the national level has been minimal. The issue of convergence was succinctly summarized by a senior government official, “No external agency can bring about total convergence; it can happen only if the issue of convergence is brought under the Prime Minister’s Office!” The program neither identified avenues of financing health from funds available under urban development guidelines (example funds available with Ward Corporators) nor did it converge with RSBY to create link communities to insurance opportunities. Mechanisms to create convergence for linking community-based schemes also did not happen. HUP teams agreed that attempts to leverage support of the JNNURM structures and interventions were not successful. PIP development was conducted jointly with Health, ICDS, ULBs, and locally elected leaders. However, the PIPs did not consider the inputs available to health from the budgets of other relevant departments. Other institutional structures at community, ward and city level (for example ward nutrition committees in MP, nigarani committees for ICDS in Rajasthan, private sector coordination meetings at collectorate level in Bhilai) were not identified as convergence platforms and therefore not leveraged. The 74th amendment of the Indian Constitution mandates the formation of ward level development committees; however, very few states have formed them. This route to advocating for WCCs within the NUHM was not identified by HUP. HUP does not capture process indicators to ascertain the performance of CCC and WCCs. In the absence of performance indicators, it is not possible to assess the effectiveness of the convergence structures19. Convergence has not materialized with other donor programs and technical agencies other than those mandated by the HUP program. The HUP team reportedly tried to establish connections with other USAID programs initially with MCH-Star, M-CHIP, VISTAR, but there has been no perceivable effort to utilize the TA group effectively, or to bring other existing and potential donors together. CONCLUSIONS Convergence has occurred to a great degree at the grassroots level between the AWWs, ASHAs, and the MAS members. Ward level coordination has worked well due to the successful leveraging of local representatives’ participation. Successful convergence has also been influenced by the presence of strong municipal corporations. Involvement of ULBs and other stakeholders has led to better public health outcomes in urban health program. The most successful result created by HUP’s convergence efforts is the integration of WASH component in PIPs. Joint CHP with mapping of key focus areas on urban health and health determinants is an effective approach but it is not currently integrated and comprehensive. HUP was not able to bring about convergence of various health organizations to support NUHM and was not able to address Intradepartmental coordination as discussed under question 1. In sum, the ET found that while institutional mechanisms for convergence exist, capacities and further advocacy will be needed to achieve convergent outcomes. 19 Availability of comprehensive and formally approved convergence plans, comprehensive resource mobilization plan, responsibility assignments, minutes of half yearly review, record of private providers, charitable organizations, trusts, NGOs, CBOs, Social clubs meetings, minutes of monthly coordination meetings among others 24 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT FINDINGS: EVALUATION QUESTION 4 (EXTENT OF POLICY LEVEL INFLUENCE) Evaluation Question 4: To what extent has the program influenced policy-level changes with regard to improving urban health at the national, state and city levels? To what extent have cities or states adopted these models? Findings: HUP’s efforts at the central level aimed at creating an enabling environment included policy products20; advocacy efforts (workshops and cross visits); support to the formation of a TRG at the ministry level which plays an advisory role to the NUHM; a convergence meeting between three key ministries; and evidence generation21. Similar efforts were made at the state and city levels including TRG-like structures in Rajasthan, UK, and UP; stakeholder consultations; studies/assessments/policy reviews, compendiums, exposure visits; analysis of gender and nutrition/WASH; review of existing governance structures in Jaipur, Pune, and Bhubaneswar; facility assessments in UP, Jharkhand, CG, and UK; GIS mapping; and VAs. Prior to the inception of the NHUM HUP supported urban health planning under the NRHM through embedded staff members by reviewing the urban component of state NRHM PIPs; developing a NUHM framework, including the costing norms; preparing the annotated draft for the NUHM framework incorporating learning from 6 countries and 3 states of India; and drafting the National Urban Health proposal for INR 22,000 crores (USD 4.4 billion), covering 221 million people across 779 cities. At the state level, HUP similarly supported the development of evidence-based NRHM PIPs for the urban health component. HUP supported the launch of NUHM since its inception. It developed tools for operationalizing a model CHP, MAS, urban ASHAs, and UPHCs. It developed training modules for all levels of functionaries. In states where scale-up models were models accepted, it further supported GIS and vulnerability mapping. HUP support resulted in the incorporation of WASH interventions into all HUP state PIPs. HUP facilitated meetings of national health officials with 63 ULBs which resulted in a change to the original plans of handing over NUHM to ULBs, as the meeting revealed that ULBs lacked capacities and the health infrastructure was minimal. A matrix of HUP’s policy level products and their outcome (adoption status by the government) has been provided in Annex IX: Outcome of TA products. HUP made a concerted effort to build capacities at national, state, and city level through skill sharing, orientations and training in VAs to all 35 State NUHM Nodal Officers in partnership with NHSRC, two regional workshops for 25 State Mission Directors, 10 Municipal Commissioners, and more than 30 state and city level health officials; and cross visits to urban health models and good practices of Mumbai, Pune, Kolkata, Surat, and Chennai. HUP partnered with SIHFW in Rajasthan and SIFPSA in UP for training urban health functionaries22. It organized capacity building sessions for city and state program managers in Rajasthan, UP, and UK; oriented ULBs to NUHM; translated and customized training modules in MP, Odisha, CG, UK, Rajasthan, and UP. The support provided by HUP has been much appreciated by all urban health functionaries unequivocally. One city program manager remarked, “HUP guidelines were a bible to me, as I was completely uninformed about NUHM.” 20 Training material, guidelines, fact sheets, and also advocacy material 21Micro Health Insurance schemes in urban India: A compendium; Study on Gender and Nutrition in Urban Programs and Policies; Study on Gender and WASH in Urban Programs and Policies among others 22 Reproductive Child Health Officers, City Health Officials, Medical Officers, training academy staff and newly recruited NUHM along with training of trainers of ANM, MAS and ASHA. 25 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Furthermore, HUP provided implementation support through joint monitoring visits and embedded national level HUP staff providing support beyond HUP states (now 35 states). The ET found that HUP contributed to institutional strengthening which facilitated the rollout of NUHM. The program facilitated the formation of various committees, incorporation of NUHM in the existing governance structures and strengthening the PMUs. One state senior official said “It was because of HUP that we were able to roll out NUHM, as they supported us to carry out the ground work.” The advocacy of HUP resulted in identification of nodal officers for urban health prior to the launch of NUHM. In the states where the launch of NUHM was delayed, HUP provided direct support and introduced all the NUHM activities through the PIPs for 2013-14 and 2014-15. The process indicators for TA provision are provided in Annex VIII: TA process indicators. For detail on achievements against quantifiable indicators, see Annex VIII. Analyzing TA effectiveness: TA can take varied forms and measuring its effectiveness often poses a challenge. There are several performance indicators for assessing achieving the four HUP objectives. The HUP program developed an MIS system to capture program data, in addition, a baseline survey was conducted and will be repeated at endline to measure outcomes at the end of the program. At mid-term HUP was mainly tracking inputs and outputs, and indicators did not convey much information on the relevance and effectiveness of initiatives. Effective TA is usually determined by criteria of timeliness, relevance, sustainability, flexibility, responsiveness, cutting-edge quality (uses latest developments/ updates/ approaches in the specified area), and efficiency (resources, costs). HUP’s TA has been timely and appreciated by the states. The TA was relevant as it strategically was spent on strategic advice/technical work compared to administrative work. The TA provided was considered of optimal quality although the TA for PPP was not considered cutting edge by the GOI. Clear cut communication protocols were put in place to make the TA responsive to state needs. However, TA could not achieve optimal flexibility as HUP state plans were being approved by the GOI. The ET has not explored and therefore will not comment on the efficiency of the TA. In terms of TA outcomes, HUP demonstrated the change through its demonstration cities and facilitated the introduction of NUHM policy. It has showcased the reform process to a wide audience and created an environment for urban health building on USAID legacy. It has facilitated the replication and scale up of urban programming in the country, generated demand for TA at the state level and satisfied its clients at the state level. It has facilitated 122% increase in allocation of urban health budgets (from 170.8 crore INR 2013-14 to 378.5 crore INR in 2014-15) in some of the EAG states. For detailed comparison of NUHM budget allocations and expenditure please see Annex X: Comparison of the Change in Urban Budgets of HUP and Non-HUP States, 2010–2013. However, it has not created sustainable capacities yet at the state level, nor created a network of organizations which can continue to provide TA beyond HUP. The NUHM has recently received a USD 300 million loan from the Asian Development Bank (ADB). The loan is accompanied with TA support which has begun through a consortium with PFI and PriceWaterhouse Coopers (PwC). The team’s conversation with the ADB representative revealed that while this is being considered a TA design phase (18 months) to be followed by comprehensive TA support both contracted by NUHM and ADB separately, the current TA support from PwC does include implementation support. This is expected to support the GOI, but does not have ample provision for state level TA. What was not accomplished: Facilitating participation of Urban Development Ministry at the national level; optimal participation of WCD at the state level; identification of a network of TA partners for state governments to manage NUHM beyond HUP; identification and leveraging of existing structures and practices for creating convergence at city and ward levels (examples under Q3); advocacy for incorporating complementary community insurance models to counter government’s resistance to 26 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT creating parallel efforts to RSBY; establishing mechanisms and guidelines for ensuring convergence of all vertical health programs within the urban model; including substance abuse as one of the service components under urban health; incorporating the position of Counselor under UPHC staffing norms although the mental health program is expected to be delivered through UPHC as well. Challenges to influencing policy: The challenges to influencing policy have been the low level of readiness of the states, their limited interest in urban programming, the short period of implementation (refer to the time line in the program background section), and finally the delay in launch of NUHM at the state level. Therefore, some of the efforts were unsuccessful. These include: dedicated HMIS for urban health (disinclination and readiness of government); models and policy decisions on PPPs in urban health (disinclination); capacity building of intermediary agency to ensure sustainability (short period of implementation); and although perceived as faster than non-HUP states by some stakeholders, delays in operationalization of NUHM (political changes). Emerging needs: Owing to challenges being encountered in recruiting health human resources for the urban sector, there is an emerging need as well as an interest at the state level to develop models of private sector participation such as social franchising models, community insurance schemes, voucher schemes for urban poor with linkages to local private providers, and a need for a public private interface agency to support government strengthen partnerships with the private sector. The mega cities will need alternate models for recruiting and retaining ASHAs as well as MAS. Nurse practitioner-led urban health delivery models need to be explored as well. Conclusions: Despite delays and impediments, HUP has produced a large base of evidence and policy level work around urban health. The states are deeply appreciative of the support received through HUP; in the words of one director, “urban health in the state started with HUP.” A senior official from another state commented, “There is only one agency working on urban health” Even officials in the non￾HUP states indicated an understanding of HUP’s value stating, “We need HUP in Bengaluru as well.” HUP products and efforts have resulted in creation, speedy roll out and guided implementation of NUHM. Not all TA efforts have reached their intended outcomes owing to contextual factors. Although HUP created only one model of urban primary health care delivery, it delineated processes and established mechanisms for effective roll out by the government. The implementation of the NUHM framework is in various stages of maturity matrix. Community processes and advocacy have been the strong elements of HUP compared to convergence and PPP aspects of the program. The ET found minimal evidence of coordination between HUP and other complementary programs. Finally, there is a continued need for TA to the NUHM at the national and state level. TA areas include capacities in intermediary agencies to support NUHM, capacities in states to manage NUHM, scaling up, monitoring, and support community processes and planning. 5. LESSONS FOR USAID’S FUTURE URBAN PROGRAMMING Evaluation Question 5: What lessons can be drawn from the program in terms of key strategic approaches and impact that should inform USAID’s future urban focus?  The three goals a) support NUHM, b) inform NUHM design and c) inform urban health programming, are incremental in nature. A program design with higher level goals provides flexibility to overcome challenges and address emerging and future needs.  Focus on MCH/RMNCH as mandated by USAID funding sources limits the flexibility of program design. 27 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT  The focus on gender in this program has been prioritized to women’s needs and participation. Men living in poverty and working in stressful urban conditions, often are isolated from family structures and become marginalized from the health perspective. This needs to be taken in consideration when designing future urban programs.  There is a strong emerging need for alternate mechanisms to address urban health care delivery.  Flexible programs, sufficient time, and flexible funds for innovations are required to demonstrate sustainable large scale models.  There is a potential for renewed interest in pursuing PPPs as human resources for urban health has been a challenge for NUHM.  There is a critical need for technical support to state governments to effectively roll out and institutionalize NUHM. TA is required for policy, planning, monitoring & evaluation, and feedback. RECOMMENDATIONS The following recommendations are based on the findings of the evaluation. Recommendations for USAID 1. Design convergence mechanism such as pooled funding from USAID sources for programs which seek to address cross cutting health issues and recruit regional hubs of high powered TA teams The design of HUP as discussed has been influenced by the USAID’s funding sources which are predominantly RMNCH oriented. The potential of the program to influence USAID core objectives of water and sanitation, solid waste management, and environmental health can be further realized when a mechanism for pooled USAID resources funding is developed. This is especially important when addressing multi-sectoral efforts. It is often the experience of organization recruiting to place TA teams that experts are unwilling to move to smaller cities. The second challenge is to find the correct mixture of technical expertise and influencing power that is required to engage senior government decision makers. USAID may consider the model of regional TA hubs as opposed to central TA teams to overcome these challenges. TA teams should consist of skilled urban health Public Health Experts, Social Scientists, Anthropologists, and Health Economists. These teams could be complemented with a small hand-holding team at the state level. Rationalize team size based on a preparedness analysis of states when introducing newer strategies or interventions to ensure ownership and results. 2. Invest on documenting and disseminating of HUP lessons, products, and models. While HUP has made concerted efforts to disseminate HUP products and experiences, there is a need to reach a wider set of audience (state governments, non-health departments involved in urban development, organizations working with marginalized population, donors, and other countries in the process of strengthening urban health programs) with HUP lessons and outcomes. Important among the lessons that must be disseminated include, participation of men in urban health, convergence processes and the need for a comprehensive urban health model. As NUHM is in its infancy, all Indian states can benefit from these lessons. Some suggested avenues for dissemination are national and international conferences, regional workshops for disseminating process documentation within NUHM, physical as 28 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Option 1Close HUP in Sept '15 as planned and invest in interventions beyond NUHM Benefits: On time and budget Costs: Delays in NUHM operationalization and potential loss of sustainability as project received 1 year post NUHM for implementation Option 2Extend HUP beyond by 18-24 months until ADB takes over - to build sustained NUHM capacities and create a base for PPPs Benefits: ROI high; creation of fertile environment for USAIDs future PPP initiatives Costs: Higher budget outlay required Option 3Create a new program which address emerging needs; and facilitate large scale private sector models co-funded by Municipal Corporations for vulnerable population in mega cities Benefits: First movers advantage; higher leveraging of resources. Cost: Needs high powered urban health and development teams well as virtual resource centers linked to NUHM website, courses and curriculum for developing urban health cadres. Another important avenue for investment is the HMIS software. 3. Expand the Scope of Urban Health Given the USAID’s urban commitment and legacy, the ET recommends that USAID build on the HUP experience in the medium term to demonstrate an expanded model of urban health which includes comprehensive health services, disaster management, epidemic management, environmental health, solid waste management, community insurance models and lifestyle issues. More immediately, USAID should consider advocating for:  Expansion of the urban health package to include substance abuse, gender based violence, and coordinated implementation of all vertical programs. This has to be accompanied with commensurate human resource norms;  Focus on men’s health issues in the urban context;  Partnership with CBOs to support to community processes;  Innovative models of primary health care delivery to overcome current gaps in human resources. Examples include primary health care models which are led by nurses practitioners, models managed by communities, and models involving partnerships with the private sector (social franchising); and  Revised incentive norms for ASHAs and MAS for preventing attrition; for ASHAs this will include higher rates of incentives and for MAS, a mechanism of capacity building, reward and recognition. 4. Consider avenues for supporting NUHM Options include:  Option 1: Extend the program by 18 months until ADB TA takes over to ensure continued velocity of NUHM and support maturity of the USAID program.  Option 2: Create HUP phase II with renewed objectives which address emerging challenges to the NUHM, health determinants and urban development; in the interim period engage with national and state level NUHM through development partners’ forum to inform and advocate for the successes of HUP.  Option 3: Start afresh; invest in creation of large scale private sector models which could be co-funded by large Municipal Corporations for urban vulnerable population especially in mega cities. 29 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Recommendations for NUHM 1. Ensure continued engagement of MAS and ASHAs through:  Partnership with CBOs for establishing and building the capacity of MAS followed by continued mentoring through ASHA for facilitating community action and VA;  Introduction of mechanisms for formally identifying members as community volunteers, link them to insurance schemes, employment, entrepreneurial and development opportunities;  Development of revised norms for ASHA incentives and mechanisms; and  Issuance of financial guidelines to MAS. 2. Establish platforms to address men’s health issues, and leverage men’s participation in community health and action Substance abuse, TB, as well as STIs are proven to be health issues of the urban male. The migrant male without the benefit of family support is further marginalized in the event of sickness. Reaching men through ASHAs or ANMs (women) has its limitations. Consider promoting men’s groups or build on the slum development groups of JNNURM to ensure access to care and men’s participation in health. Community level resource persons (men) could be identified and trained for this purpose. 3. Consider incorporation of the WCC strategy within the NUHM framework Senior officials of NUHM at the central level stated that the ward level structures have limited capacity for decision making. While this may be true, in the urban context, a semblance of cohesiveness is found at the ward level, making it amenable to collective action. The high visibility available to elected representatives leading health actions provides a good opportunity for leveraging micro-planning, monitoring and mobilizing services. The ET has recommended HUP to document the WCCs and disseminate the results widely to create a demand for this structure. The NUHM should consider incorporating the WCC concept within the NUHM framework following the perusal of evidence. 4. Orient and advocate to states the flexibilities within the NUHM framework and encourage innovations by creating flexible funding mechanisms As was experienced with NRHM, states are inclined to follow central guidelines although NUHM framework provides ample flexibility. While continuing to inform the states about the flexibilities consider a flexible funding mechanism to accommodate innovations by the states for example introducing an additional budget line. 5. Develop customized BCC strategy following identification of specific urban needs The variables of temporary settlements, migration, and availability of informal private providers along with specific urban health needs demand a specific BCC strategy. The replication of the existing rural BCC tools for urban areas will require customization based on the urban BCC needs. Newer areas of health including, gender based violence, solid waste management, substance abuse, mental health 30 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT (violence and depression) will need the development of newer tools. The placement of Public Health Managers at the UPHC will accelerate this process. Phase out Recommendations for HUP 1. Document and disseminate the successes of vulnerability mapping and WCCs and processes Document the successes of WCCs and disseminate the results widely to create a demand for this structure. While documenting include processes involved in the formation of the WCC, the suggested structure, the role of each member, suggestive actions that could be taken and provide case studies of WCCs in CG and Pune. Similarly, while states are moving forward with VA exercises, delineate the way to use the VA effectively for planning urban health needs. If resources permit develop a training module for using VA maps. Document the process of forming MAS, conducting a UHND, managing WCCs, carrying out CHP, and convergent decision making to ensure the capacities for leading on these processes are developed within the CPMUs. 2. Delineate an exit strategy Document and share a strategy with the NUHM on expanding current UPHC services and process for ensuring the convergence of vertical health programs at the UPHC level. If time and resources permit, initiate structured processes (such as directives from NUHM) for institutionalizing the changes. The exit strategy at the city level must include the identification of:  A support agency to provide need based TA;  NGO partners to support community processes;  City resource center equipped with a set of HUP products and process documents; and  Key multi-sectoral officials at the state level with which city teams could link. 31 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT ANNEXES I. EVALUATION SCOPE OF WORK II. EVALUATION DESIGN AND METHODOLOGY III. DATA COLLECTION TOOLS IV. LIST OF INFORMATION SOURCE: DOCUMENTS AND RESPONDENTS V. DATA COLLECTION SCHEDULE VI. MATRIX OF DIVERSITY VII. QUANTITATIVE DATA ANALYSIS VIII. TA PROCESS INDICATORS IX. OUTCOME OF TA PRODUCTS X. COMPARISON OF THE CHANGE IN URBAN RICH BUDGETS OF HUP AND NON-HUP STATES, 2010-2013 XI. GEOGRAPHIC COVERAGE AND ACTIVITIES OF HUP CONSORTIUM XII. DEMONSTRATION CITIES XIII. ACHIEVEMENTS AGAINST QUANTIFIABLE OUTCOMES 32 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK a. Program Project Title: Health of the Urban Poor (HUP) b. Start-End Dates: October 1, 2009 - September 30, 2015 (Phase 1 from October 2009 to September 2013; Phase 2 from October 2013-September 2015) c. Budget: $10,778,627 d. Program/Project Description: Urban health issues have received little attention in the past as compared to rural health programs in India. In 2010, the World Health Organization’s (WHO) World Health Day theme, "1000 cities 1000 lives" brought a much needed focus on public health issues in urban health. USAID/India has been active in the urban health sector in India since 2002. Building on USAID/India’s past efforts in urban health, the Health of the Urban Poor (HUP) Program was launched in the year 2009 with the purpose of providing technical assistance to the Government of India’s (GOI) National Urban Health Mission (NUHM), the National Rural Health Mission (NRHM), and the Reproductive and Child Health II (RCH II) program of the Ministry of Health and Family Welfare (MOHFW). These Missions are classified as sub-Missions under the National Health Mission, which aims to bridge the rural￾urban healthcare gap through increased community ownership, decentralization of the programs to the district level, and urban clusters improving primary health care. Technical assistance is provided through program learning, institutional strengthening, and assistance in policy formulation, development of operational guidelines, implementation, capacity building, and the strategic dissemination of urban health knowledge. Through strategic pilot interventions and demonstration projects which highlight comprehensive maternal and child health and nutrition interventions, and the promotion of safe water, sanitation and hygiene services, HUP has been paving the way to improve the health status of the urban poor by working closely with GOI counterparts at the center, state, and city levels. HUP’s primary objectives are to: 1. Provide quality technical assistance to the GOI, states and cities for the effective implementation of the NUHM. 2. Expand Partnerships in urban health including engaging the commercial sector in Public- Private Partnerships (PPP) activities. 3. Promote the convergence of different GOI urban health and development efforts. 4. Strengthen urban planning activities through evidence-based city-level demonstration and learning. HUP is implemented by the Population Foundation of India (PFI) as the prime recipient and a consortium of sub￾implementing partners including Plan India; the Institute of Health and Management Research -Jaipur (IIHMR); the Bhoruka Charitable Trust (BCT; the Centre for Development & Population Activities (CEDPA); the International Institute for Population Sciences (IIPS); and the Micro Insurance Academy (MIA), providing technical support. The key project strategies include: 1. Need-based technical assistance for the operationalization of urban health programs within the public health system at all national, state and city levels. 2. Convergence at all levels for improved health, nutrition, water, sanitation and hygiene through institutional capacity building. 3. Capacity building for high quality accessible and sustainable health, nutritional, water and sanitation services. 4. Leveraging resources. 5. Gender equity. 6. Community empowerment (improving negotiation skills development). 7. Fostering strategic alliances and partnerships at all levels. 8. Demonstration, documentation, and the systematic replication of successful urban health intervention models. I. PROGRAM INFORMATION 33 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT The geographical focus of the project is at the national and state levels in Bihar, Chhattisgarh, Jharkhand, Madhya Pradesh, Odisha, Rajasthan, Uttarakhand and Uttar Pradesh. At the national level, HUP supported the MOHFW to formulate the NUHM which was launched in the year 2013. At the state level, the project provided technical support to state governments with the implementation of urban health programs. In addition, the project has initiated models for urban health improvement in five cities (Agra, Bhubaneshwar, Delhi, Jaipur and Pune). The project has also created opportunities for dialogue and interaction on various issues related to urban health implementation and policy making amongst key stakeholders including policy makers, managers, academics and civil society organizations. In June 2012, USAID/India conducted a mid-term evaluation of the HUP program to assess and evaluate its key objectives and strategies including technical assistance, private sector partnerships and convergence of different GOI urban health and development efforts. An in-depth assessment of the best practices models created under this program was also carried out. The mid-term evaluation of HUP concluded that ‘it is important to manage the risk of creating unmet demands through HUP in partnering states in the event NUHM is delayed beyond program closure. If HUP is successful in creating demand for urban health services, USAID should develop a strategy to involve resources from government, donors, and the private sector to meet the urban health demands in the absence of NUHM. While a long-term engagement strategy would be the best option, interim measures could include an extension of HUP or the design of a follow-on award to begin immediately after the closure of HUP’. Due to the initial project delay and to overlap with the launch of NUHM, a decision was taken to extend the program beyond its planned end of project date (September 30, 2013) to September 30, 2015. The goal of extending the HUP program has been to support the GOI, state governments, and city administrations in institutionalizing and scaling up proven RMNCH interventions among the urban poor (including interventions for nutrition, safe water and sanitation). This will be achieved through the following objectives: 1. Strengthen the capacity of the GOI, state and city administrations to plan, implement and monitor urban health services [corresponding to Objective-1 (Technical Assistance) of the Cooperative Agreement]. 2. Expand the integration efforts to address health determinants among the urban poor through the departments of Health, Women and Child Development (WCD), Urban Department (UD), and the municipal administrations [corresponding to Objective-3 (Convergence) of the Cooperative Agreement]. 3. Share lessons learned from the HUP models and approaches beyond the selected and focused Empowered Action Group (EAG) states and cities [corresponding to Objective-4 (City Demonstration) of the Cooperative Agreement]. 4. Serve as a platform to convene influential stakeholders around urban health issues [corresponding to Objective-2 (Public Private Partnerships) of the Cooperative Agreement]. The activities of HUP in the extension phase (2013-2015) differ from the previous phase (2009- 2013) as follows: 1. Focus on systems strengthening for urban health (including nutrition and WASH services) i.e. a shift from demand generation to supply side strengthening. 2. Hand over HUP’s demonstration model to the respective cities and states and instead focus on providing technical assistance to states and other cities with scaling up this model. 3. Establish an “urban health knowledge resource center” at the central, regional and state levels, to ensure future sustainability. II. EVALUATION PURPOSE The purpose of the final evaluation is to: 1. Evaluate the overall progress made by the program against its objectives and specifically on the major recommendations from the mid-term assessment of the program. 2. Evaluate the key strategies and approaches adopted by the program in scaling up its efforts in its second phase. 3. Provide insights and lessons learned from various components of the program that will help inform USAID/India’s future designs in urban health. 34 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT a. Intended Uses or other Audiences for the Evaluation: The primary intended users of this evaluation are USAID/India, USAID/Washington and the GOI at the national and state levels. In particular the USAID/India Health Office, Program Support Office, and Mission management are interested in findings and recommendations concerning the progress of health innovations and partnerships within this project. Local institutions, other donors and other missions worldwide are the secondary audience of the evaluation. b. Evaluation Questions: This evaluation will answer the following questions: 6. To what extent the program has successfully addressed the major recommendations from its mid-term evaluation in its second phase? 7. How effectively has the program scaled up the activities/model interventions from the five cities in phase one to 18 cities in the second phase? In particular, the evaluation shall address the following sub￾questions. 2.5 What have been the results (effectiveness) of the program’s key approaches, i.e. governance, institutional capacity strengthening and public-private partnerships? 2.6 What are the key achievements of the program and what are the key factors facilitating the achievements? 2.7 What are the key challenges and how could they be overcome? 2.8 What are key lessons learned in scaling up activities/model interventions? 8. To what extent has the program been successful in ensuring the convergence of various GOI efforts on improving urban health? 9. To what extent has the program influenced policy-level changes with regard to improving urban health at the national, state and city levels? 4.1 To what extent have cities or states adopted these models? 10. What lessons can be drawn from this program in terms of key strategic approaches and impact that should inform USAID’s future urban health focus? III. TECHNICAL REQUIREMENTS FOR EVALUATION a. Data collection and Analysis Methods USAID/India anticipates a ‘mixed method’ evaluation methodology that would include both quantitative and qualitative approaches. USAID/India, Evaluation COR will need to review and approve the data collection methodologies and tools prior to the start of the assignment. The evaluators shall consider a range of possible methods and approaches for determining, collecting, and analyzing the information needed to assess the evaluation objectives. The evaluators will review the program’s objectives, performance management plan, and the log-frame to assess the performance of the program against the targets set for key indicators. The evaluators must also review the mid-term assessment report and assess the performance on the key recommendations. The evaluation must answer all of the questions above. We require that the specific methodology will be discussed at length and refined during the evaluation planning phase and the program design Team Planning Meeting (TPM). Desk review of documents: USAID/India will provide the team with relevant country and program specific documents including proposals, mid-term evaluation report, monitoring indicators and other relevant documents for conducting this desk review. The evaluation team will collect and collate relevant international documents, reports, and data. All team members are required to review these documents to prepare for the TPM. This desk review will help to organize the materials for the external evaluation team analysis and review of progress to date, develop appropriate data collection tools and methodologies, and inform their analysis during the field work and report writing stages. 35 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Data sources: Data sources that the team will be expected to utilize, review and analyze include the program design documents, proposal, annual work plans, and M&E data including relevant baseline information on program sub-components, evaluation reports, and other related documents and reports. Additional relevant documents related to urban health programming in India, such as urban health policies, statistics, and relevant international standards may be utilized as supporting documents as appropriate. b. Composition, Technical Qualifications and Experience Requirements of the Evaluation Team USAID seeks a three-member evaluation team comprised of a Team Leader (Senior Evaluation Specialist), a Senior Public Health Specialist, and a Senior Management, Governance and private Health Sector Expert. All team members must have relevant prior experience in India, familiarity with USAID’s objectives, approaches, and operations, and prior evaluation/ assessment experience. Collectively, the team must have experience in evaluating urban health programs. The responsibilities and technical qualifications and required experience of individual team members identified are given below: a. Senior Evaluation Specialist (Team leader): The Team Leader must have extensive experience in leading and managing large-scale health evaluations both in the public and health sector. S/he must have a good understanding of project administration, financial and management skills, including an understanding of USAID. S/he should have excellent English language writing, editing and communication skills. In addition to proven ability to provide this leadership role, involving a technically and logistically complex program, s/he must have substantial and demonstrated expertise in evaluation techniques involving projects with technical assistance, training, advocacy, and partnership components. S/he should be familiar with the functioning of large donor-funded programs in India. The person must have ability to lead a diverse team of technical and management experts and to interface with various stakeholders ranging from government to non￾government organizations, donors and beneficiaries. A minimum of 12 years’ experience in design, management and evaluation of health programs is required.S/he will oversee the overall design of the evaluation framework, including methodology determinations; organization of calendar/travel/meetings; overseeing the desk study, interviews, and other data collection; and analyzing the data with input from team members to draft the evaluation report and presentation. b. Senior Public Health Specialist: This Senior Public Health Specialist must have extensive and strong experience in designing, implementing, and evaluating public health programs with a focus on maternal and child health (MCH), and water sanitation and hygiene (WASH) projects. S/he should be an expert in integrated public health programming in the context of urban health programs. S/he must be familiar with the public and private actors in the health sector and have a good grasp of the relevant national programs. A minimum of 12 years of experience in the design, management and evaluation of public health programs including urban and private health sector is required. Excellent writing and communication skills are required. Having an excellent understanding of USAID operational, management, and technical approaches including health systems strengthening will be an added advantage. Along with the team leader, s/he will contribute to the overall drafting of the evaluation framework and participate in the desk study, interviews, and other data collection; and analyzing the data with input from team members to draft the evaluation report. c. Senior Management, Governance and private Health Sector Expert (Local): This expert must have an extensive experience in managing and governance of health and non-health programs. Specifically, s/he must have an excellent understanding of project administration, governance and management in the health and non-health sectors in India. This expert will assess the overall governance of the HUP project at the state and the national level and must have a thorough knowledge of the project governance of large donor-funded programs which manage networks of NGOs and institutions; experience working with government and various management issues related to such projects is required. This expert also will be responsible for assessing private commercial sector involvement in the project and assess the public-private partnerships (PPPs) piloted by the project. S/he should assess and analyze the processes of identification of opportunities for partnerships and mechanisms to accelerate participation, as well as the sustainability and scalability of PPPs S/he will be responsible for assisting in coordinating the desk study, interviews, and other data collection, and providing overall support to the team. 36 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT c. Gender: Collectively, the evaluation team members must address gender concerns in the evaluation. The evaluators shall pay attention to gender issues during activity implementation and describe how both women and men involved were affected by the context or the work undertaken. IV. EVALUATION MANAGEMENT a) Roles and Responsibilities: Overall Guidance: The Evaluation COR and the Contracting Officer (CO) will provide overall direction to the evaluation team.  The Contractor will be responsible for obtaining visas and country clearances for travel for the consultants if required.  The Contractor will be responsible for coordinating and facilitating assessment related team planning meetings, field trips, interviews, and other meetings in conjunction with USAID and the HUP Program.  The Contractor will be responsible for international and in-country logistics as applicable such as transportation, accommodations, communications, office support, etc.  The evaluation team will receive support from USAID/India in selecting priority organizations and places to visit during the evaluation. The evaluation team is expected to schedule interviews or other modes of data collection with key stakeholders, though USAID/India can assist in providing contact information. b) Schedule: The duration of the evaluation shall not exceed 10 weeks. The evaluation team is expected to provide a schedule (in a tabular form) defining when specific steps in the evaluation process will occur and when the deliverables are due. Team Planning Meeting (TPM): A one-day team planning meeting will be held by the evaluation team at a convenient place in New Delhi before the evaluation begins. This will be facilitated by the evaluation team leader, and will provide USAID/India with an opportunity to present the purpose, expectations and agenda of the assignment. The evaluators shall come prepared with a draft set of tools and guidelines and a preliminary itinerary for the proposed evaluations. In addition, the TPM will also:  Clarify team members’ roles and responsibilities  Establish the timeline, share experiences and firm up the evaluation methodology  Finalize the methodology guidelines including tools and questionnaires to be used by the team. Site Visits and Interviews: Conduct a thorough review of the Program through site visits and interviews. Interview questionnaire will be prepared in advance and finalized during the TPM. Site visits will be planned taking into consideration factors like geographical diversity, representation of various implementation agencies, and the scale of the interventions. c) Delivery Schedule: CLIN DELIVERABLES DUE DATE 1 Work Plan: The work plan will be submitted to the Evaluation COR at USAID for approval after the team is confirmed prior to departure for the field. The team will meet with USAID/India Program Support and the Health team after arrival in Delhi and prior to starting field data collection process. 15 days 2 Interim briefings, including status reports: The team leader will provide weekly status reports to USAID on work plan implementation via email by OOB Monday (beginning of the next week). The evaluation team will provide a mid-point briefing to the USAID/India team, including evaluation and technical members, to clarify any outstanding queries that may have emerged since the initiation of the evaluation process by phone and e￾mail. - Debriefing with USAID: The evaluation team will be required to 35 days 37 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT debrief the Mission Director and Deputy Mission Director on the observations and recommendations after the field visit and draft analysis is over. 3 Debriefings with other stakeholders/implementing partner: The team will independently present the major findings of the evaluation to the USAID partner (as appropriate and as defined by USAID) and /or GOI in New Delhi and state government officials. The debriefing will include a discussion of findings, conclusions and recommendations. The evaluation team will consider partner comments and draft report accordingly, as appropriate. 40 days 4 Draft Evaluation Report: The evaluation team will present a draft report not to exceed 30 pages of its findings and recommendations to the USAID/India’s Evaluation COR. 50 days 5 Final Evaluation Report: The final report, with the Executive Summary must be received by the Evaluation COR, within seven working days after receiving the final comments on the draft evaluation report from the USAID/India team. The final report should include an executive summary of no more than three pages, a main report with findings, conclusions and recommendations not to exceed 30 pages, a copy of this statement of work, evaluation tools used to collect information to answer the evaluation questions, and a list of persons and organizations contacted. 70 days 38 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT ANNEX II: EVALUATION DESIGN, METHODOLOGY, AND LIMITATIONS Approach The ET adopted a utilization based approach which focused on an in-depth analysis of the program’s underlying logic and causal linkages. Without assuming a linear cause-and-effect relationship between a program’s inputs and activities and desired outcomes, the team recognizes that a multitude of factors and interactions influences a program’s impact and looks to identify those factors which have acted as facilitators or impeders. The team made an attempt to document evidence of contribution to each program objective, determine if and how these contributions interact, and investigate any instances where there is a little to no evidence of contribution. Furthermore, where objectives have not been met, the evaluation team explored key issues that have constrained HUP’s performance. Additionally, the approach allowed the team to identify any unanticipated consequences of the program (positive or negative) and lay the ground work for future urban program designs while assessing the integration of gender perspectives within the program. Methods The evaluation team began its fieldwork together in Delhi, where they met with USAID and PFI (the implementing agency). The team then divided into two sub-teams to optimize their time for the fieldwork. The first team (Team A) visited Bhopal, Bhilai/Raipur and Cuttack/Bhubaneswar, while the second team (Team B) visited Dehradun, Jodhpur, Pune and Agra. Telephonic interviews were split between the teams. Qualitative approach: Each evaluation question was explored in the context of the program objectives to assess project accomplishments and impact. The qualitative fieldwork used key informant in-depth interviews, focus group discussions and observation to collect data in each site. Prior to beginning fieldwork the ET developed interview guidelines, a focus group discussion guide and a facility visit checklist, as seen in Annex 3: Evaluation Tools. In addition to visiting Urban Primary Health Centers (UPHC), the ET observed one urban health nutrition day (UHND) in each city. Focus group discussions were conducted with the Mahila Aarogya Samitis (MAS) in each city; some FGDs were also conducted with ASHAs (Accredited Social Health Activists) and ANMs (Auxiliary Nurse Midwives), sometimes in mixed groups. Qualitative data analysis: The two teams conducted debriefs at least every two or three days with each other as part of a rolling analysis in order to discuss evidence collected, patterns, and discrepancies to help answer the evaluation questions. Upon completion of the data collection, the team analyzed the data for relevance to the evaluation questions and the SOW of the evaluation. This allowed for triangulation, in which the team analyzed data related to an evaluation question using different methods and then across the different research sites. These themes were used to draw conclusions and make recommendations regarding future programming. Quantitative data analysis: The quantitative data was analyzed in two ways. Within each of the scale-up cities and states, the number achieved for each indicator was calculated as a percentage of the annual target for that indicator. Second, the percentage of the slum population reached by the program in scale-up cities was compared to that in non-HUP cities.23 HUP undertook a baseline survey at inception which included measurement of key reproductive and child health behaviors, morbidities, utilization of health services, access to water and sanitation, status of women’s empowerment and adequacy of health facilities. The comparative end line assessment to measure progress has not been completed. This has limited the quantitative analysis to data generated by the project’s Health Management and Information System (HMIS) and the HMIS of the National Rural Health Mission24 reported quarterly by the project for each. In addition, water, sanitation and hygiene (WASH) indicators have been reported by the project as well. Quantitative data thus available for analysis was at the output level and includes: Gender Integration: The ET addressed gender concerns throughout the evaluation process. Each member of the team was sensitized to gender issues through the USAID online course on gender. The need for gender analysis and available data was discussed during the team planning meeting. The ET met with the USAID gender Specialist 23 Pending receipt of this data 24 Sources of data as cited in the project’s M&E plan for the extension phase. 39 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT to further delineate the evaluation questions through the gender lens. While gender disaggregated quantitative data was not available, the ET ensured gender sensitive interview settings for all the focus group discussions held with women community members. In addition a gender score card tool has been applied to evaluate the final evaluation report. For scale up cities Indicator Source of data Number of Pregnant women received 3 ANC checkups National Rural Health Mission’s (NRHM) Health Management Information System (HMIS) report Deliveries conducted at Facility Number of newborn breastfed within 1 hour Number of fully immunized infants (9-11 months) Water Sanitation and Hygiene (WASH) Indicators Access to improved source of drinking water Ministry of Urban Development’s Service Access to improved sanitation Level Benchmark Data for each city For demonstration cities Indicator Source of data Number of pregnant women receiving complete ANC (3 ANC checkups, 100 IFA tablets consumed & 2 TT) HUP HMIS Number of institutional deliveries HUP HMIS Number of women receiving at least 2 post natal care visits HUP HMIS Number of fully immunized infants (up to measles) HUP HMIS Number of newborns breast fed within one hour of birth HUP HMIS Number of children exclusively breast fed for the first six months HUP HMIS Number of currently married women aged 15-49 years using any modern method of contraception HUP HMIS Water Sanitation and Hygiene (WASH) Indicators* Number of people gaining access to an improved drinking water source (Urban) (Men & Women) Rapid Household Census in HUP slums Number of people gaining access to an improved sanitation facility (Urban) (Men & Women) Rapid Household Census in HUP slums Limitations and Threats to Validity As with any short-term performance evaluation, the team was restricted by its limited fieldwork schedule. The team had a total of two days for start-up meetings, 11 days for the site visits plus an additional five days in Delhi to complete national level interviews and complete data analysis. Despite this, the team added three additional cities for site visits and four additional cities for telephonic interviews. The inclusion of these additional cities limited the time allocated to other tasks, especially data analysis. Furthermore, some national and state level stakeholders were unable to participate in the evaluation as they were not available during the brief fieldwork period. One particular meeting, with the MIS official of Ministry of Health and Family Welfare (MOHFW) which could not be carried out would have clarified the status and the future plans for urban MIS. This report does not have the benefit of these inputs. The evaluation is also limited by the limited data available to triangulate results. The available data was not gender disaggregated limiting the gender analysis. HUP undertook a baseline survey at inception which included measurement of key reproductive and child health behaviors, morbidities, utilization of health services, access to water and sanitation, status of women’s empowerment and adequacy of health facilities. The comparative end-line assessment to measure progress has not been conducted yet and so the team could not compare this post-intervention data with the baseline survey. The varied launch dates for NUMH in the states mean that the scale-up cities have different periods of active programming depending on when NUHM reporting began. Data was not available to attempt a comparison of non￾HUP states and cities with the HUP scale-up states and cities on key indicators of NUMH progress. 40 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT We acknowledge internal threats to validity as well. While the ET did its best to present itself as independent and neutral experts, it was reliant on HUP to identify respondents and make appointments for the fieldwork. A HUP staff member accompanied the team to the interview site and sometimes introduced us to the respondent, but did not stay for the interview. For in-depth key informant interviews, the ET identified who was to be interviewed and HUP made arrangements based on our list. However for the FGDs and site visits to facilities, there may be selection bias in the choice of the MAS group to participate and/or the choice of facility. While the ET requested that they visit a “typical” facility and MAS at each site, HUP may be motivated to choose the more cohesive and mature MAS groups and the more developed facilities. Lastly, as is typical in evaluations of this type, an external threat to validity exists through response bias, whereby respondents felt a natural tendency to provide answers that they believed the interviewer wanted to hear. While the factors presented above do raise concerns for the completeness and the internal and external validity of the evaluation findings, the strength of the evaluation design should help to overcome these weaknesses. The ET spoke to a large number of key informants at all levels and visited a large number of sites. It also reviewed a wealth of documentation from and about the program and attempted to obtain as much quantitative data as possible to measure whether there were objective improvements in urban health over the course of the program. These data sources provided an opportunity for the team to triangulate its findings and limit the influence of the threats to validity mentioned above. 41 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT ANNEX III: DATA COLLECTION TOOLS Tool 1: FGD Guide for MAS Topic Discussion/Transitions Introduction and greeting Namaste, I am Facilitator’s and this is Observer’s name Purpose of FGD The name of the ‘implementing agency’ has been working on a project for the urban poor to improve maternal, neonatal and child health. Because you are members of the MAS we would like to learn from you about your opinions about maternal, neonatal and child health issues. Your ideas can help us understand how our project performed and inform us about any changes that we can incorporate in our future projects to improve maternal and child care. May we begin the discussion? (Allow for those who do not want to participate in the discussion leave). No right or wrong answers We would just like to know your frank opinion. There is no right or wrong answers to any of the questions. This is not a test. We just want to learn from you. The idea is for everyone to share their honest opinions and experiences so that we can learn from you and your experiences to strengthen projects in the future. Although you are members of the same group, your experience may differ. The group members will allow for all differing views to be presented. Length of time The discussion would take about an hour and a half. During that time we will be asking some questions about different topics related to the project. We are interested in hearing what you think and feel. Talking to one another As we will be discussing about each of your opinions, it will be important that we do not talk at once because we will want to hear each other so we should not talk together. Everybody should try and participate and everybody will be given a chance to put forth their views. If you have any queries we will try to address them at the end of the discussion. Explain note taking Confidentiality (Name of Observer/reporter) will be writing down some of the things that we will be talking so we can remember later. Does anyone object? We are the only ones who will know your names, we will not use any names in our reports Checking understanding Clarify Does everyone understand what I have said? Does anyone have any questions? Participants introductions (Warm up) Please introduce yourselves 1. When was the MAS formed? 2. Why did you want to join the MAS? What determines your continued participation in the MAS? 3. What are some of the major health problems of this community? 4. Where does this community seek curative care from? How far are these facilities? 5. Where does the community seek preventive care from? How far are these facilities? 6. Are there any other facilities nearby? Why they are not utilized? 7. Who are the public providers visiting this community? (Probe for ANMs, health visitors, TB workers, other NGO workers) 8. What are the health inputs being provided by the Aanganwadi centers in the community? 9. What inputs have you received from the HUP program? 10. Can you give us some examples of how you have personally benefitted as a woman? 11. What is the role of men in ensuring health in your community? 12. What are the other health inputs that if introduced by HUP, will benefit the community? 42 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT 13. Looking back at the time when the MAS had not been formed, what are some of the significant changes that have occurred since in i. Health problems ii. Health seeking behavior iii. Services available iv. Availability of water v. Water hygiene vi. Availability of sanitation facilities vii. Utilization of sanitation facilities Thank you indeed for your time. Is there any question you have for us? Respond to best of knowledge or inform IP to find the answer and respond. 43 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Tool 2: Observation Checklist for UHND Observe Comments Staff and beneficiaries present Yes No ANM Yes No AWW Yes No HUP front line worker Yes No MAS members Yes No Number of children under 5 yrs of age Number of pregnant women Other beneficiaries Instruments and equipment Yes No Weighing scale – Adult & Child Yes No Examination table and Bed screen Yes No Hemoglobin meters Yes No Urine examination kits Yes No Gloves Yes No Stethoscope and blood pressure instrument Yes No Measuring tape Yes No Foetoscope Yes No Vaccine carrier with ice packs Yes No AD Syringes and syringe cutter Yes No Drugs and contraceptives Yes No Iron Folic Acid (IFA) tablets- Adult and Child segregated by male and female Yes No Deworming tablets like Albendazole Yes No Paracetamol tablets Yes No Chloroquine tablets Yes No Condoms Yes No Oral Contraceptive Pills Yes No Oral Rehydration Solution (ORS) packets Yes No Vaccines Yes No TT Yes No Polio Yes No Hepatitis B Yes No DPT Yes No BCG Yes No Measles Yes No Vitamin A solution Yes No Services provided Yes No Supplementary nutrition Yes No Immunization Yes No Health check-up Yes No Antenatal care check up (BP, weight, per abdominal check and Hb estimation) Yes No Referral services Yes No Pre-school education Yes No Nutrition and health education Yes No WASH activities Yes No 44 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Tool 3: In-depth Interview Guide for other Urban Program Implementers Name:______________________________Designation:________________ Introduction: We have been asked to evaluate the HUP project with the aim to understand its effectiveness and identify the lessons learnt. We are independent evaluators working for the firm Social Impact. Please be assured that any information we discuss will remain confidential and your name will not be attributed to any of the findings in our study. We expect the interview to take one hour. Are you willing to participate in the interview? 1. What are some of the challenges to implementing urban health programs? 2. How has your program overcome these challenges? 3. Does your program interact with the HUP program being implemented by PFI? What structures exist for the exchange of lessons between HUP and your program? 4. What has been the role of HUP in understanding of health needs of the urban poor? To what extent has HUP been able to influence urban health care delivery under NRHM? 5. Are you aware and if so can you describe the type of TA that the HUP Project has provided in urban health at the municipal level? 6. What capacities have been built by the HUP to promote improving joint planning by various stakeholders? 7. What efforts have been made by HUP in promoting convergence for urban health? 8. What are some of the gender mainstreaming approaches adopted by HUP? What has been the outcome of these approaches? 9. What efforts have been made by HUP in developing and disseminating methodologies for city level health planning? 10. In your opinion what are some of the significant successes of the HUP? What could have been done better by the program? 45 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Tool 4: In-depth Interview Guide for Intermediary Agency Name:______________________________Designation:________________ Agency:___________________________________________ Introduction: We have been asked to evaluate the HUP project with the aim to understand its effectiveness and identify the lessons learnt. We are independent evaluators working for the firm Social Impact. Please be assured that any information we discuss will remain confidential and your name will not be attributed to any of the findings in our study. We expect the interview to take one hour. Are you willing to participate in the interview? 1. What is the function of this organization as an internal TA organization to the NUHM? 2. What structures are in place to fulfill this role? 3. We understand that the HUP works closely with your organization. Can you describe the type of TA that the HUP Project has provided to this organization to effectively manage urban health programs? 4. Has the TA from the HUP Project been appropriate for addressing the needs of this organization? Please provide examples of what other inputs would have been useful? 5. Has this TA been provided in a timely and effective manner? If not, why not? 6. What kind of capacity building inputs have you received from HUP? 7. What are some of the gender mainstreaming approaches adopted by HUP? What has been the outcome of these approaches? 8. What additional inputs do you think are required to perform your role effectively? 9. Has the HUP established a resource centre within this organization? What kinds of resources are available and how are these resources utilized? 10. How do you support PPP initiatives? 11. How do you support convergence activities? 12. How do you support dissemination of lessons across the country/state? 13. What are the continuing challenges to implementing urban health programs? 46 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Tool 5: In-depth Interview Guide for Urban Development Department Name:______________________________Designation:________________ Agency:___________________________________________ Introduction: We have been asked to evaluate the HUP project with the aim to understand its effectiveness and identify the lessons learnt. We are independent evaluators working for the firm Social Impact. Please be assured that any information we discuss will remain confidential and your name will not be attributed to any of the findings in our study. We expect the interview to take one hour. Are you willing to participate in the interview? 1. What is the role of your organization in ensuring health for the marginalized population of urban areas? 2. Please describe the systems in place within this department to effectively converge for promoting urban health and WASH. 3. What are some of the challenges you face in promoting health care and WASH (probe for funds, human resources, information for monitoring and capacities)? 4. We understand that the HUP works closely with your organization. Can you describe the type of support that the HUP Project has provided? What according to you was the most appreciated support? What other support could have been given by the HUP to help you achieve your goals? 5. How do you support PPP initiatives for health and WASH? Please describe any such partnerships you may have? How has HUP supported in promoting PPPs at the city/ward level? 6. What are some of the gender mainstreaming approaches adopted by HUP? What has been the outcome of these approaches? 7. To what extent has HUP been able to influence urban health care delivery and WASH under NUHM? Other aspects of urban health policy? 8. Are you aware of HUP’s extension into new cities in the last few years? If so, please describe the achievements that you may be familiar with. What made these achievements possible? 9. What are some of the gender mainstreaming approaches adopted by HUP? What has been the outcome of these approaches? 10. Have you observed any changes in the frequency or quality of joint planning by various state departments and municipal corporations on urban health and WASH issues in the last few years? 11. Can you suggest ways in which future programs might more effectively support initiatives in urban health and WASH? What challenges might they face? 47 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT 12. Tool 6: In-depth Interview Guide for Stakeholders at City and Ward Level Name:______________________________Designation:________________ Agency___________________________________________ Introduction: We have been asked to evaluate the HUP project with the aim to understand its effectiveness and identify the lessons learnt. We are independent evaluators working for the firm Social Impact. Please be assured that any information we discuss will remain confidential and your name will not be attributed to any of the findings in our study. We expect the interview to take one hour. Are you willing to participate in the interview? 1. When was the city/ward level committee for health established? Who are the members of this committee? 2. What are the principal goals or activities of your committee? Please describe the systems in place to effectively converge for promoting urban health. 3. How is health care and WASH inputs delivered to the marginalized population in the city? What are some of the challenges you face in delivering health care (probe for funds, human resources, information for monitoring and capacities)? 4. We understand that the HUP works closely with your organization. Can you describe the type of support that the HUP Project has provided to the municipality/ward? What according to you was the most appreciated support? What other support could have been given by the HUP to help you achieve your goals? 5. What are some the challenges in converging to implement urban health and WASH initiatives? How are these being overcome? 6. How do you support PPP initiatives? Please describe any such partnerships you may have? How has HUP supported in promoting PPPs at the city/ward level? 7. What are some of the gender mainstreaming approaches adopted by HUP? What has been the outcome of these approaches? 8. Please describe any efforts to share information with other similar groups to yours in other parts of the city or state. 9. Has HUP support been a part of information exchanges? If so, can you describe that support? 10. Please describe some of the outcomes this committee has generated till date - in improving how urban health programs are implemented. 11. How would you rate the quality of HUP support received by your committee? 48 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT 12. Tool 7: In-depth Interview Guide for IP and Technical Partners Name:______________________________Designation:________________ IP/ Technical partner:___________________________________________ Introduction: We have been asked to evaluate the HUP project with the aim to understand its effectiveness and identify the lessons learnt. We are independent evaluators working for the firm Social Impact. Please be assured that any information we discuss will remain confidential and your name will not be attributed to any of the findings in our study. We expect the interview to take one hour. Are you willing to participate in the interview? I. General 1. To what extent has the program successfully addressed the major recommendations from its mid-term evaluation in its second phase? What were the recommendations which you did not address and why? 2. Can you describe the major highlights of the phase II activities and additional efforts made in the demonstration cities? 3. Can you describe the key HUP approaches (governance, institutional capacity strengthening and public￾private partnerships) and how effective have they been in the successful implementation of HUP and scaling up? 4. What specific activities were planned and implemented for the non-HUP cities? 5. What are some of the gender mainstreaming approaches adopted by HUP? What has been the outcome of these approaches? 6. What are the key achievements of the program and what are the key factors facilitating the achievements? 7. In your view, what are some of the main urban health challenges that the HUP Project has addressed and how have these been addressed? 8. What has been the key learning from the HUP project? 9. What issues still need to be addressed and what in your opinion should be done to address them? 10. What measures have been introduced since the midterm to facilitate ease of management of HUP? II. HUP Technical Assistance to National, State, and Municipal Governments 1. Please describe the highlights of TA provided at national/state/city level (whichever appropriate)? 2. Describe the system in place to determine what TA to provide and how? Probe for whether HUP annual work plans are aligned with state/city plans? 3. Which TA has been most appreciated by the public partner and why? 4. What efforts have been made under HUP to build the capacity of the internal TA agency? What are some of the challenges HUP faced in doing so and how were they overcome? 5. What are the systems or institutional mechanisms that HUP has put in place to coordinate other urban health initiatives? 6. What efforts have been made by the HUP to build the capacities of the cities to develop city plans? What are some of the challenges HUP faced in doing so and how were they overcome? 7. Support to the demonstration city has been scale down. What are some of the HUP activities and outcomes that have sustained and what has not been sustained and why? What additional measures could have been taken to sustain these efforts? 8. Please describe the efforts that have been made by HUP in developing and disseminating lessons from the phase I. III. Private Public Partnerships 1. What are some of the key achievements of HUP in promoting PPP in urban health? 2. Did HUP explore social marketing initiatives for urban health? If not why not? 3. What were the challenges in leveraging PPPs, especially funds under corporate social responsibility (CSR)? 4. What are some the platforms that have been created by HUP to sustain the linkages between private partners and the government? 49 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT 5. What are some of the key lessons from the assessment of PPP models that have been incorporated at the policy level? IV. Convergence 1. To what extent have HUP’s efforts brought various State departments and ULBs (Department of Health and Family Welfare, WCD, and PHED) and Municipal Corporations to converge in planning and implementing urban health initiatives for the urban poor? 2. What effort has been made towards converging with the Jawaharlal Nehru National Urban Renewal Mission (JNNURM)? 3. How has HUP worked with slum development committees? Have convergence programs promoted by HUP been undertaken by these committees? If so, can you cite a few examples? 4. What were some of the key challenges in promoting convergence and how were they overcome? 5. In your opinion, what urban health convergence initiatives should be given greatest priority in future years? For example, in the fields of maternal and child health, environmental health (water and sanitation), infectious disease, non-communicable disease, and nutrition? 6. Are there ways in which future urban health convergence activities could be strengthened? V. HUP Project Management Systems 1. What were some of the key challenges to effectively managing HUP’s (1) project MIS and monitoring and review procedures; (2) financial and procurement systems; (3) delegating roles and responsibility; and (4) team deployments and capacity building of PFI and its partnering organizations? 2. What are some the changes (if any) in the working relationships between PFI and its sub-partners that have been introduced since midterm? How has this worked? 3. Please cite examples of how has the project’s MIS informed the implementation of project interventions? 4. Have HUP management procedures been effective in analyzing and resolving implementation bottlenecks? If yes, can you cite a few examples? And finally  In your opinion which is the most significant contribution of HUP to urban health care? To what extent has HUP been able to enable better understanding of the health needs of the poor and to influence urban health care delivery?  Do you have any other specific observations or recommendations for HUP that can be useful to inform USAID’s future urban health focus programming? 50 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Tool 8: In-depth Interview Guide for Private Sector Partners Name:______________________________Designation:________________ PPP partner:___________________________________________ Introduction: We have been asked to evaluate the HUP project with the aim to understand its effectiveness and identify the lessons learnt. We are independent evaluators working for the firm Social Impact. Please be assured that any information we discuss will remain confidential and your name will not be attributed to any of the findings in our study. We expect the interview to take one hour. Are you willing to participate in the interview? 1. Please share with us the current status of Private – Public partnerships (PPP) in urban health and WASH? 2. What processes and institutional structures exist to promote and establish PPP? 3. Can you give examples of successful PPP initiatives and how these were made possible? 4. In your opinion, are current PPP activities being adequately assessed with respect to effectiveness, the potential for replication, and scale-up? 5. Please describe your institution’s initiative as a PPP partner in HUP? 6. Could you describe the nature of support you have received from the HUP? – Probe for capacity building, systems and processes, monitoring and evaluation and linkages with other partners. 7. Who are your partners in implementing this initiative? What systems are in place to link and coordinate activities with the government? 8. Describe how you manage information generated by your initiative. How is it being used? What improvements are required for better monitoring? 9. What challenges were faced in implementing PPP and how were these addressed? 10. What additional inputs are required for better implementation of PPP in HUP? Are there ways in which social marketing initiatives could be more effectively deployed in support of urban health? 11. What are some of the gender mainstreaming approaches adopted by HUP? What has been the outcome of these approaches? 12. In your opinion, what are some of the key successes of the HUP? To what extent has the program influenced policy-level changes for PPP with regard to improving urban health and WASH at the national, state and city levels? 13. What lessons can be drawn from the PPP initiatives that should inform USAID’s future urban health and WASH focus? 51 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Tool 9: In-depth Interview Guide for Health and WCD Ministry at National and State level Name:______________________________Designation:________________ Agency:___________________________________________ Introduction: We have been asked to evaluate the HUP project with the aim to understand its effectiveness and identify the lessons learnt. We are independent evaluators working for the firm Social Impact. Please be assured that any information we discuss will remain confidential and your name will not be attributed to any of the findings in our study. We expect the interview to take one hour. Are you willing to participate in the interview? 1. Through your work with NUMH, could you please tell us about what aspects of the HUP program that you have been involved with, including:  HUP representation in the NUHM Technical Resource Group (TRG);  implementation planning support;  national and regional workshops;  technical assistance on WASH;  technical assistance on HMIS and health indicators;  other types of technical assistance;  lessons learned workshops;  others? 2. For each of these aspects that you have been involved with, has the NUMH support been relevant and valuable for NUMH? Has it addressed the main challenges faced by NUMH in implementing urban health programming? 3. What is your experience with guidelines and other resource materials developed by HUP for NUHM? From your perspective have these guidelines etc. been user friendly and suitable for NUHM staff? Were training workshops and other support provided for the guidelines and other resource materials effective? 4. What are some of the inputs of the HUP in PPP for urban health? From your perspective, how these efforts influenced the urban health program delivery? What are the additional efforts that could have been made to strengthen PPP further? 5. What are some the design level aspects which in your opinion were helpful in meeting the challenges (Probe for TA at national/state and city level; PPP strategy; establishment of demonstration cities; convergence strategy; and community level support) 6. From your perspective what is the main contribution made by HUP to the NUMH? Overall what aspects are valuable? Overall what aspects have been problematic? Please highlight issues related to project design, to program structure, to governance systems and strategies, and to technical aspects. 7. Have HUP’s governance systems and strategies proved to be effective? Has the HUP project been effective in streamlining governance systems and strategies for NUHM and other urban health programs? Can you cite examples? a. What outcomes has the HUP created through its TA strategy? What additional effort should have been made? b. What outcomes has HUP created through its PPP strategy? What additional effort should have been made? c. What outcomes has HUP created through its convergence strategy? What additional effort should have been made? d. What are some of the gender mainstreaming approaches adopted by HUP? What has been the outcome of these approaches? 8. Has HUP effectively disseminated lessons learned, such as situational analysis of urban slums and other vulnerable populations, on urban health to NUHM and other programs? 9. Has HUP incorporated gender mainstreaming approaches into its program effectively? 10. What else could HUP have done to more effectively improve urban health? Additional WCD questions: 52 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT 11. What are the key contributions of HUP to maternal and child health and nutrition in urban marginalized areas? Has the HUP technical assistance addressed the key challenges for MCH nutrition? 12. What are the institutional mechanisms that have been put in place by HUP to promote convergence between various stakeholders in nutrition and health? Have HUP’s efforts at bringing convergence between WCD and other government agencies been effective for urban health programming? What have been the successes and the barriers to promoting convergence? 13. Ha HUP supported PPPs for urban maternal and child nutrition? Please describe the support and comment on its effectiveness. 14. Are HUP’s contributions to WCD’s urban health programs sustainable? Why or why not? 53 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT 15. Tool 10: In-depth Interview Guide for other Donors Name:______________________________Designation:________________ Agency:___________________________________________ Introduction: We have been asked to evaluate the HUP project with the aim to understand its effectiveness and identify the lessons learnt. We are independent evaluators working for the firm Social Impact. Please be assured that any information we discuss will remain confidential and your name will not be attributed to any of the findings in our study. We expect the interview to take one hour. Are you willing to participate in the interview? 1. Could you please tell us about the programs in urban health that your organization has currently? What is the scope of the intervention (how many cities, which cities, target groups)? (Or if donor only) What are the investments that your organization has made in urban health and urban WASH initiatives? 2. How do these programs interface with the NUHM? 3. Does this program have any partnerships or joint programming with HUP? Please tell us about these. 4. What do you see as the main challenges in urban health programming? How has your organization worked to overcome these challenges? (Experiences with HUP if knowledgeable about HUP) 5. Please tell us in what capacity you have had interactions with HUP, besides the above. 6. How has the HUP addressed challenges in urban health and WASH programming? What are some the design level aspects which in your opinion were helpful in meeting the challenges (Probe for TA at national/state and city level; PPP strategy; establishment of demonstration cities; convergence strategy; and community level support). 7. Have HUP’s governance systems and strategies proved to be effective? Has the HUP project been effective in streamlining governance systems and strategies for NUHM and other urban health programs? Can you cite examples? a. What outcomes has the HUP created through its TA strategy? What additional effort should have been made? b. What outcomes has HUP created through its PPP strategy? What additional effort should have been made? c. What outcomes has HUP created through its convergence strategy? What additional effort should have been made? d. What are some of the gender mainstreaming approaches adopted by HUP? What has been the outcome of these approaches? 8. Has HUP effectively disseminated lessons learned, such as situational analysis of urban slums and other vulnerable populations, on urban health to NUHM and other programs? 9. What are some of the gender mainstreaming approaches adopted by HUP? What has been the outcome of these approaches? 10. Overall what would you say is the main contribution of the HUP program? 11. What else could HUP have done to more effectively improve urban health and WASH? 12. Has there been knowledge transfer and lessons learned shared between you program and HUP? 54 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT ANNEX IV: LIST OF INFORMATION SOURCES: DOCUMENTS AND RESPONDENTS List of Key Informant Interviews Government Officials and External experts SNo . Name of person Sex Designation Institutional Affiliation Location 1 Mr Nikunja.B.Dhal, IAS M Joint Secretary, Urban Health Ministry of Health & Family Welfare, GOI NewDelhi 2 Dr Preeti Pant F Director NRHM Ministry of Health & Family Welfare, GOI NewDelhi 3 Mr C.K.Mishra M Additional Secretary & MD (NRHM) Ministry of Health & Family Welfare, GOI NewDelhi 4 Mr JP Mishra M Consultant and ex ED SHRC Chattisgarh Delhi 5 Dr. Saroj K. Adhikari F Assistant Director Ministry of Women and Child Development, GoI. Delhi 6 Mr Suresh Chauhan M State Urban Health Officer State Program Management Unit- Health Dehradun 7 Ms. Kshama Bahuguna, F CDPO Ddn City ICDS, DWCD Dehradun 8 Ms Najma Mansuri, F FieldSupervisor ICDS, Jakhan Rajpur Road ICDS, DWCD Dehradun 9 Mr Rakesh Bist M City Urban Health Officer City Program management Unit- Health Dehradun 10 Mr Sukhveer Singh M Project Executive Ambuja Cement (PPP Partner) Dehradun 11 Mr. Ramchander Punker M Chief Chemist, PHED PHEd Jodhpur 12 Ms Priyanka Sharma F Junior Chemist PHEd Jodhpur 13 Mr Rajneesh Barashar M Sanitary Inspector, Ward 25 Jodhpur Municipal Corporation Jodhpur 14 Mr Rajesh Barashar M Sanitary Inspector, Ward 26 Jodhpur Municipal Corporation Jodhpur 15 Dr. Y.S. Rathore M Deputy Director Health, Jodhpur Division office, Health and FW Jodhpur 16 Mr. Rakesh Sharma M DPM, Jodhpur District Health and FW Jodhpur 17 Mr Ashish Mathews M Urban Health Planning Consultant District Health and FW Jodhpur 18 Ms Leela Birsa F ANM, Incharge Ward 24 District Health and FW Jodhpur 19 Ms. Usha F ASHA, Isaiyon Ka kabristan II District Health and FW, DWCD Jodhpur 20 Ms. Daryo Kanwar F AWW, Isaiyon Ka kabristan II DWCD Jodhpur 21 Ms. Sharda Vyas F LS, Bhagat Ki Kothi DWCD Jodhpur 22 Ms. Nisha Ameta F Urban Health Programme Manager, District Health and FW Jodhpur 23 Dr. H.R. Goyal M RCHO, Nodal Officer, Urban health program District Health and FW Jodhpur 24 Ms. Simla Baresa F CDPO, DWCD, Jodhpur WCD Jodhpur 55 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT 25 Dr. Pankaj Shukla M Deputy Director Urban Health Bhopal 26 Mr. Faiz Ahmed Kidwai M Mission Director Urban Health Bhopal 27 Dr. Manjula Agarwal F Medical Officer UPHC Balseraniya Bhopal 28 Mr Mohit Shrivastav M Social Mobilizer Bhopal 29 Mr Suhendra Dhakar M CPM Bhopal 30 Mr Abhishek Payasi M Community Officer Consultant Bhopal 31 Dr. Veena Sinha F CMHO Bhopal 32 Ms. Anita Dugaya F DPM Bhopal Bhopal 33 Mr Akshay Shrivastava M Joint director ICDS Bhopal 34 Ms. Neelam Shami Rao F Commissioner ICDS Bhopal 35 Dr. Kiran Shejwar F Regional Joint Director Bhopal 36 Mr Raza Faraz M Divisional Program Manager 37 Ms. Pushpa Awasthy F Program Officer UNICEF Bhopal 38 Dr. S.P. Shrivastava M UADD, Bhopal municipal corporation Water Aid Bhopal 39 Dr. Goutam Sadhu M Project Coordinator (PC), Rajasthan and Chhattisgarh HUP-IIHMR Raipur 40 Mr. Sanjoy Samaddar M Former Project Director (PD) HUP-IIHMR Raipur 41 Mr. Hemant Kumar Mishra M MIS officer Cum Officiating Project Director HUP-IIHMR Raipur 42 Mr. Rakesh Kumar M Water and Sanitation Specialist HUP-IIHMR Raipur 43 Mr. Yagyesh Srivastava M Finance Officer HUP-IIHMR Raipur 44 Dr. .Subhash Pandey M Former State Nodal Officer, NUHM Department of Health and Family Welfare (DoHFW), GoCG Naya Raipur 45 Mrs. Cristina Lal F Joint Director (JD), Department of Women and Child Development (DoWCD), GoCG Naya Raipur 46 Dr. Nand Lal Choudhary M Deputy Director Department of Women and Child Development (DoWCD), GoCG Naya Raipur 47 Dr. Kamlesh Jain M State Nodal Officer, NUHM Department of Health and Family Welfare (DoHFW), GoCG Raipur 48 Dr. Satish Tajne M State Program Manager (SPM), NUHM Department of Health and Family Welfare (DoHFW), GoCG Raipur 56 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT 49 Dr. Geeta Sinha F Medical officer In-charge, UPHC, Baikunthdham, Bhilai Department of Health and Family Welfare (DoHFW), GoCG Bhilai 50 Mr. Rajendra Arora M Speaker Bhilai Municipal Corporation Bhilai 51 Mr. Sanjiv Dubey M City Consultant, NUHM, Bhilai & Durg Department of Health and Family Welfare (DoHFW), GoCG Bhilai 52 Dr. Pranav Arya M Regional Manager, CSR ACC cement private Limited, Jamul, Bhilai Jamul 53 Mr. Jamil Khan M Deputy Manager, CSR ACC Limited, Jamul Jamul 54 Mr. Rekh Ram Banchore M Chairman Jamul Nagar Palika Jamul 55 Mr. Net Ram Chandrakar M Chief Municipal Officer (CMO), Jamul Nagar Palika Department of Urban Administration and Development, GoCG Jamul 56 Mr. A. K. Lohiya M Engineer, Jamul Nagar Palika Jamul 57 Mr. D. N. Mishra M President Anmol Jan kalian Vikas samiti, Raipur Jamul 58 Mrs. Kavita Biswal F Ward Councilor Elected Representative, ,Jamul Nagar Palika Jamul 59 Mrs. Madhavi F Care taker Jamul Community Toilet Jamul 60 Mr. A. K. Sundrani M Commissioner, Durg Municipal Corporation Department of Urban Administration and Development, GoCG Jamul 61 Ms. Sarita Barbe F ANM on UHND site Department of Health and Family Welfare (DoHFW), GoCG Raipur 62 Dr. Prabeer Chaterjee, M Executive Director (ED) State Health Resource Center, Chhattisgarh Raipur 63 Dr. B Das M Civil Surgeon, District Hospital, Raipur Department of Health and Family Welfare (DoHFW), GoCG Raipur 64 Dr. Amrit Chopra, M Health Officer, Raipur Municipal Corporation. Raipur 65 Mr. Swatantra Rahangdale, M City Program Manager, Raipur Department of Health and Family Welfare (DoHFW), GoCG Raipur 66 Ms. Anshul Thudgar M City Program Manager, Raipur Department of Health and Family Welfare (DoHFW), GoCG Raipur 67 Mr. Praful Singh Kushwah M Project Associate State Health Resource Center, Chhattisgarh Raipur 68 Mr. Ashok Kumar Pandey, M District Program Officer (DPO), Raipur City Department of Women and Child Development (DoWCD), GoCG Raipur 69 Dr. Anshuman Dash, M Health Officer, and Ex. State Program Manager, Mukhya Mantri Shahari Swasthya Karyakram (MSSK) Raipur 70 Mr. Ramchandra Singh Bisht, M WASH Consultant UNICEF, Chhattisgarh Raipur 57 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT 71 Ms. Mansha Jose F Consultant, Water Aid, Chhattisgarh Raipur 72 Mr. Purushottam Panda M Consultant, CCDU, Chhattisgarh Raipur 73 Dr. Bibhakar Bhattacharyya M Nodal officer for urban health GOWB Kolkata 74 Dr. Ajay Bhattacharya M Add. MD GOWB Kolkata 75 Dr Amrita Khurana F RMO, Wanorie Hospital Pune Municipal Corporation(PMC) Pune 76 Dr Minal Vaichalkar F Rama Bai Ambedkar Maternity Hospital, Ambi Odha Pune Municipal Corporation(PMC) Pune 77 Mr Vinod Yadavrao Jadhav M City Programme Manager Pune Municipal Corporation(PMC) Pune 78 Ms Sandhya Nagarkar F CDPO, Shivaji Nagar DWCD Pune 79 Dr.Sanjeev Wavare M Astt. Medical Officer Health Pune Municipal Corporation(PMC) Pune 80 Dr.Anjali Sabane F Deputy Medical Officer Health Pune Municipal Corporation(PMC) Pune 81 Mr Jayant Kumar Bhosekar M Assistant Commissioner Pune Municipal Corporation(PMC) Pune 82 Mr Pathak M Assistant Engineer, DUDA Nagar Nigam Agra 83 Ms Meera Devi F Link Worker CURE Agra 84 Dr Daya Gupta F UPHC, Ramnagar Deptt of Medical & Health Agra 85 Mr Ashok Kushwaha M Ward Councillor ULE Ward No 80 Agra 86 Ms Mohini Kushwaha F Community Development S (CDS), DUDA, Nagar Nigam Agra 87 Ms Mariamma F Lady Health Visitor Deptt of Medical & Health Agra 88 Mr Vinod M Chief Sanitary Inspector ULE Ward No 80 Agra 89 Ms Hemlata F Link Worker Harjupura, Ward 80 CURE Agra 90 Ms Sunita Mahour F Link Worker Patiram Bagichi, Ward 80 CURE Agra 91 Ms Pushpa Rani F Link Worker Tajganj, Ward 81 CURE Agra 92 Mr Abhishek Prakash M Addl Mission Director UP Deptt Medical & Health Lucknow 93 Mr MR Gautam M General Manager, SPMU (Urban Health) UP Deptt Medical & Health Lucknow 94 Mr Sanjeev Pandey M Block Social Mobilizer UNICEF Agra 95 Ms Ayesha Khatoon F Community Social Mobilizer UNICEF Agra 96 Mr Suresh Chand M Executive Engineer, DUDA Nagar Nigam Agra 97 Dr Raju Jotkar M Technical Director NHM and Nodal person Urban Health Deptt of Medical & Health, Maharashtra Mumbai 98 Dr. M. L. Jain M Director SIHFW Govt of Rajasthan Jaipur 99 Ms. Shikha Sharama F Consultant Urban Health Directorate of Medical & Health, Govt of Rajasthan Jaipur 58 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT 100 Ms. Hayman Win F Social Sector Expert ADB Manila 101 Mr. Gyana Dash M Commissioner Cuttack Municipal Corporation Cuttack 102 Dr. Prafulla Ku. Behera M Chief District Medical Officer Cuttack District Cuttack 103 Dr. Pradipta Kumar Pradhan M City Health Officer Cuttack Municipal Corporation Cuttack 104 Dr. Mrs. Dipika Duttaray F Additional District Medical Officer (PH) Cuttack District Cuttack 105 Mr. Sudhanshu M City Program Manager City Program Management Unit, NUHM Cuttack 106 Dr. Arun Sao M Incharge, UPHC Naya Bazar Cuttack 107 Mrs. Jyotsna Rani Singh F Anganwadi Worker Ramgarh Anganwadi Center Cuttack 108 Mrs. Minati Manjhi F ANM Ramgarch, Ward No-10 Cuttack 109 Mr Pradeep Swain M Regional Coordinator Social Awareness Institution Cuttack 110 Dr. Lipika Nanda F Director Indian Institute of Public Health Bhubaneswar 111 Mr Bhuputra Panda M Associate Professor Indian Institute of Public Health Bhubaneswar 112 Mr. Ranjan Kumar Biswal M Ward Coordination Committee Ward No 10 Cuttack 113 Mr Ranyan Biswan M Health Coordinator 114 Ms Roopa Mishra, IAS F NUMH Mission Director NHM, Odisha Bhubaneswar 115 Mr. Saroj Samal M Additional Mission Director NHM, Odisha Bhubaneswar 116 Dr. Dinabandhu Sahoo M Joint Director ( Technical ) NHM, Odisha Bhubaneswar 117 Dr. D.K. Panda M Team, SHSRC Lead NHM, Odisha Bhubaneswar 118 Mr. Sukant Kumar Mishra M State Programme Manager , Urban Health NHM, Odisha Bhubaneswar 119 Dr. Radha Reddy M NHM, Odisha Bhubaneswar 59 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT List of HUP personnel and partners interviewed SNo. Name of person Sex Designation Placement Location 1 Dr. Jatin M PFI New Delhi 2 Dr. Bharti Dangwal F HUP-Project Director Plan India Dehradun 3 Mr. Nitin Bisht M HUP-MIS Officer Plan India Dehradun 4 Dr. Ambey Shrivastava M MIS Officer, HUP PFI, Rajasthan Jaipur 5 Dr. Dhirendra Kumar M Professor and PD HUP IHMR Jaipur 6 Dr. Hemlata Yadav F Project Officer, Bhoruka Charitable Trust Jaipur 7 Mr. Prabhat Jha M Acting Project Director; Convergence Advisor HUP (PFI) Bhopal 8 Mr. Praeshur Nath Mishra M Finance & Admin HUP (PFI) Bhopal 9 Ms. Pragya Dube F Urban Health Consultant HUP (PFI) Bhopal 10 Dr. Anil Paranjape M Executive Director CASP Pune 11 Mrs. Smita Barwe F Project Officer CASP Pune 12 Mrs Vijaylakshmi Tulpule F Branch Manager FPAI Pune 13 Ms. Sandhya Joshi F Project Coordinator HUP FPAI Pune 14 Ms. Rashmi Shirhatti F City Coordinator, HUP Plan India Pune 15 Ms. Leena Rajan F NGO CBO Coordinator HUP Plan India Pune 16 Mr. Jayanto Choudhury M HUP Plan India Pune 17 Ms. Trupti Kulkarni F Finance and Adminstrative officer HUP Plan India Pune 18 Mr. Rajesh Kumar M Project Officer CURE Agra 19 Mr. Lince Alencherry M HUP Coordinator CURE Agra 20 Ms. Neeru Sharma F Cluster Coordinator CURE Agra 21 Ms. Geeta Pipal F Cluster Coordinator CURE Agra 22 Mr. Anil Kumar M Cluster Coordinator CURE Agra 23 Mr. Ravi Kashyap M President Shri Nirotilal Buddha Sansthan(SNBS) Agra 24 Ms. Khushboo Gupta F Programme manager Shri Nirotilal Buddha Sansthan(SNBS) Agra 25 Mr. Kushal Pal Singh M Convergence Advisor PFI Lucknow 26 Mr. Nitin Dwivedi M PPP specialist PFI Lucknow 27 Dr. Sainath Banerjee M Chief of Party PFI New Delhi 28 Mr. Gautam Chakraborty M Public Health Economist and acting Project PFI New Delhi 60 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Director 29 Mr. Gajinder Pal Singh M MIS manager PFI New Delhi 30 Mr. Pradeep Panda M Director of Research and Implementation Micro Insurance Academy (MIA) New Delhi 31 Ms. Sneha Siddham F Sr Programme Manager Urban Health Plan International New Delhi 32 Mr. Mohammed Asif M Director Plan International New Delhi 33 Mr. Smarajit Chakraborty M Project Director HUP, Odisha Bhubaneswar 34 Mr. Partha Roy M City Coordinator HUP, Odisha Bhubaneswar 35 Mr. Niladri Chakraborty M NGO/CBO Coordinator HUP, Odisha Bhubaneswar 36 Mr. Sheikh Nausad Akhtar M MIS Officer HUP, Odisha Bhubaneswar 37 Mr. Basudev Panda M Documentation Officer HUP, Odisha Bhubaneswar 61 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT List of Other Donors, Non-Governmental Organizations Met SNo. Name Organization Designation 1 Dr. Jyoti Vajpayee BMGF Senior Program Manager FP 2 Mr. Abhilash Philip PSI Senior Program Manager 3 Hayman Win ADB Project Officer 4 Dr. Anshuman UNICEF, Chattisgarh Health Specialist 5 Mr. Manusha Jose Wateraid MP Program Executive 6 Mr. Purshottam Panda Swach Bharat Mission Communication and Capacity Development Unit 7 Dr. Pranab Arya ACC Cements Regional CSR Coordinator, East India 8 Mr. Srinivas Bhavaraju ACC Cements CSR Project Disha lead 9 Dr. Rajni Vaid NHSRC Lead, Community Processes 10 Dr.. Prabir Chaterjee SHRC, Chattisgarh Director 11 Dr. DK Panda SHRC Odisha Team Leader 12 Mrs. Vidya Bhalle CASP 13 Prof. Ranjit Roy Choudhuri PFI Board Member 14 Mr. Harsh Mandar TRG member 62 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Selection of Documents Reviewed The ET reviewed an extensive number of documents through the course of the evaluation, too numerous to include here. The list below is by no means exhaustive, but provides the reader with an idea of the types of documents read and analyzed by the team. HUP Reports  HUP Baseline Report  HUP Annual Report: October 2009-September 2010  HUP Annual Report: October 2010-September 2011  HUP Annual Report: October 2011-September 2012  HUP Annual Report: October 2012-September 2013  HUP Annual Report: October 2013-September 2014  Achievements of the Health of the Urban Poor (HUP) program  HUP 2009-2010 Work Plan  HUP Extension Phase Work Plan  Achievements of the Health of the Urban Poor (HUP) program Guidance Produced by HUP  Operational Guidelines for City Coordination Committee  Operational Guidelines for Ward Coordination Committee  Operational Guidelines for Mahila Arogya Samiti (MAS)  Operational Guidelines for Urban Health and Nutrition Day (UHND)  User Manual for HUP HMIS  Micro Insurance Health Schemes in Urban India, A Compendium  PPP for Delivery of Heal Services, A Compendium  WASH IEC materials MTE-Related Documents  HUP Mid-Term Evaluation Report  MTE Compliance Report Government Documents  Pre-NUHM (NRHM) and post-NUHM city PIPs  NRHM and NUHM frameworks  City Health Plans Other  Letters of invitation/appreciation from HUP cities and states  HUP-generated TORs and MOUs for PPPs and other collaborations (e.g. with training institutes such as SIHFW in Rajasthan) 63 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT ANNEX V: DATA COLLECTION SCHEDULE HUP EVALUATION USAID/India TIMELINE Sunday Monday Tuesday Wednesday Thursday Friday Saturday July 19 20 21 22 23 24 25 In-briefing with USAID; Team meeting Meeting with PFI: meeting with Ministry of Health Team 1 Dehradun Team 2 Bhopal Team 1 Jodhpur Team 2 Bhopal Team 1 Jodhpur Team 2 Bhopal Team 1 Jodhpur Team 2 Bhopal 26 27 28 29 30 31 August 1 Team 1 Pune Team 2 Raipur Team 1 Pune Team 2 Raipur Team 1 Pune Team 2 Raipur Team 1 Pune Team 2 Bhubaneshwar Team 1 Agra Team 2 Bhubaneshwar Team 1 Agra Team 2 Bhubaneshwar Team 1 Agra Team 2 Bhubaneshwar 2 3 4 5 6 7 8 Team return to Delhi Telephonic interviews and analysis in Delhi Telephonic interviews and analysis in Delhi Telephonic interviews and analysis in Delhi Telephonic interviews and analysis in Delhi Analysis in Delhi Analysis and preparing for dissemination 9 10 11 12 13 14 15 Presentation to PFI Final presentation to USAID and Ministry of Health 64 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT ANNEX VI: MATRIX OF DIVERSITY Diversity among HUP states State/City Population data (Census 2001) Environment Achievements of HUP Budget Overall ET Observations Urban Pop. in lakhs Pop. of Cities Total Slum Pop. % of slum pop. Inclinati on for urban health NUHM launch date Inclinatio n for PPPs CCCs in place MAS formed UHND conducted WCCs in place Allocation (10m INR) 2013-2016 Expenditure (10m INR) 2013-2016 Bihar 86,81,800 48,14,512 5,31,481 11 Low Jan 2015 Moderate 1/1 0% NA 0% 46.98 0.1 (0.32%) Has low importance for urban health and limited current capacities for managing NUHM reflected in the slow roll out of NUHM. Manages UPHCs through PPP. Chattisgarh 41,85,747 26,04,933 8,17,908. 31 High Jul 2013 Moderate 2/2 98.6% 95.4% 0% 49.26 16.28(46.1%) State was proactive in implementing urban program prior to the launch of NUHM, is currently transitioning state funded activities under NUHM funds. Moderate capacities at city level and strong community process including WCCs Jhark hand 59,93,741 24,22,943 3,01,569 12 Low Feb 15 DK 3/3 17.7% 85.9% 0% 23.23 0 (0%) Did not sample the state Madhya Pradesh 1,59,67,14 5 95,99,007 24,17,091 25 Moderate Oct 13 Low 4/4 59.4% 95.3% 0% 102.07 15.69(15.3%) Has moderate capacities for implementing NUHM, however strong leadership, has ward level structures for coordinating nutrition interventions, however not leveraged. Odisha 55,17,238 28,38,014 6,29,999 22 High Feb 14 Moderate 2/2 97.9% 14.3%* 25.4% 43.06 3.56(8.2%) Has articulated low capacities to implement NUHM; HUP cities benefit from strong municipal leadership; opportunities for leveraging CSR forums at district level missed Rajasthan 1,32,14,37 5 76,68,508 12,94,106 16 Moderate Oct 14 Low NA 84.2% 95.2% NA 97.4 11(11.2%) State was implementing urban program prior to NUHM and is currently transitioning state funded activities under NUHM funds. Manages some UPHCs through PPP in Jaipur city. Has moderate capacities for implementing NUHM, Uttarakhand 21,79,074 10,10,188 1,95,470 19 High Aug 15 Moderate 1/1 100% 13344+ NA 10.71 5.87(54.8%) State was implementing urban program prior to NUHM and is currently transitioning state funded activities under NUHM funds. Has moderate capacities for implementing NUHM; Manages UPHCs through PPP Uttar Pradesh 3,45,39,58 2 2,12,56,87 0 43,95,276 20.7 Low Feb 14 Low NA 0% 8.6% NA 206.73 44.3(21.4%) Has articulated low capacities to implement NUHM; UP has started NUHM interventions in all the 131 cities. The state is first focusing on developing the infrastructure and hiring personnel. MAS and ASHAs will be selected later. 65 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Bengaluru (Karnataka) 43,01,000 431000 10.02 High Jan 14 High NA NA NA NA 64.68 7.01(10.8%)@ As a mega city Bengaluru has experienced difficulties in recruiting urban health functionaries and establishing MAS. The municipal Corporation takes the lead on implementing NUHM. Capacity for implementing moderate Kolkata (West Bengal) 45,73,000 14,85,000 32.47 High Aug 13 Low NA NA NA NA 166.21 6.81(4.1%)@ Kolkata had a strong urban health programme prior to the NUHM launch. Rapid Action team is very active for Dengue and other diseases and focuses on mosquito breeding sites. Kolkata is facing problems in recruiting ANMs because ANM training colleges are not functioning. *: Data not reported by HUP for Cuttack; +:Target not set; @: Data for the state and not the cities 66 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Diversity among Demonstration Cities Parameters Pune Agra Jaipur Population of the city 3115431 1585704 3046163 Slum Population 1189000 750000 402920 Number of slums (listed and unlisted) 564 432 238 Wards 90 (Administrative 15) 90 91 (Administrative 8) Type of slum Population Slums established long time back. The people residing here are mainly industrial workers and work for fixed hours. The slums have evolved over the years and people have now small pucca houses. These were village habitations which have now become slums. People are laid back and are daily wagers. Slums established long time back. The people residing here are mainly workers in business houses. The women have businesses at home. The slums have evolved over the years and people have made pucca houses Sanitation A number of the houses have toilets of which there are substantial individual toilets. Community toilets have also been built. Children still defecate in the open. Through RAY the government is now building individual toilets. The slums have community toilets as well. However, people still defecate in the open. 70% have individual toilets made through RAY or through their own money.. But the old people and children still practice open defecation. Municipal Corporation Been in existence from 1950. Have sufficient funds. A large number of health facilities are under the corporation. Each administrative unit has an executive head a senior official - The Assistant Commissioner each having their independent offices which are self sufficient. Do not have sufficient funds. Each ward is headed by a Ward Councillor who lacks an independent office. The Ward Corporators nominate members to form ward committees. Do not have sufficient funds and there are no health facilities or AWCs under the corporation. Each ward is headed by a Ward Councillor who lacks an independent office. The Ward Corporators nominate members to form ward committees. Health Facilities under Municipal Corporation Have own ANMs and health facilities.ICDS is not under PMC. No health facilities or AWCs under the corporation No health facilities or AWCs under the corporation. Ward Committee meetings Three WCCs formed in the HUP intervention areas. Meetings Held every month or once in 2 months mostly in the ward office. All the departments attend the meetings under the Assistant Commissioner. MAS members, Link workers, NGOs are also members. Work is reviewed for each months decisions taken. Decisions are taken on the spot. Examples of decisions: toilets, garbage collection, electricity, child toilets, traffic jams, railings for security, location of AWCs etc; WCC has asked HUP for a proper handing over with orientation of officials, records and HUP material. WCC meetings held in the slum area. Ward Committee nominees, LHV, SI, AWWs, MAS members, Link workers attend the meeting facilitated by NGO. Meetings are held every month.The decisions are mainly related to water connections and availability, toilet cleaning, requirement of individual toilets, garbage collection etc; The quality of the WCCs depends on the Ward Councillor. BCT has tried to initiate meetings but they have not been successful as there is no government mandate. Scaling Up of WCCs to non Scaling-up of Ward Coordination Committees (WCC) has occurred in 4 non-HUP intervention None None 67 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT HUP area wards at the request of PMC. MAS The members are empowered and speak very confidently at the WCCs. They have got a number of issues sorted out collectively. The members are empowered and speak very confidently at the WCCs. They raise issues and are dependent on the Ward Councillor to take up the issues.Previous groups made under UHRC were used by one NGO. MAS members Agra had a skype call with MAS Jaipur for making functional MAS. UHND Presence of Pune Municipal Corporation (PMC) Doctors in Urban Health and Nutrition Day (UHND). Team comes for every UHND with 2 ANMs, vaccines, medicines, syringes etc. A vehicle has been provided for the team. Services include, growth monitoring, ANC, PNC, Immunization, general checkups and treatment of minor illnesses (mainly women and children), referral, some counselling. THR is given as soon as it comes since there is no place in AWC for storage. Good coordination amongst AWWs, ANMs, Link workers and MAS. Services include taking weight, ANC, Immunization, THR (mainly women and children). Reasonably good coordination amongst AWWs, ANMs, Link workers and MAS. Mainly Immunization and some elements of ANC Special UHND Introduction on Urban Health and Nutrition Day (UHND) on Non communicable diseases - screening for high BP, blood sugar is done for both men and women, referral None None ICDS services Upgraded 16 AWCs to model AWCs in HUP intervention area. AWWs trained for providing services. AWWs oriented for urban health issues Leveraging private sector Deepak foundation provided training for AWWs, helped them in developing model AWCs. None None Risk Pooling Effective and active members Active Recently started NGOs facilitating the HUP processes 2 NGOs selected for HUP implementation. Had local presence before HUP. They continue to work through other sources of funding. 2 NGOs selected for HUP implementation. Had local presence before HUP, in fact one NGO had worked in UHRC and had developed resource material and has provided support to HUP cities to establish MAS and Link workers(Communitization) in other cities. The NGOs continue to work through other sources of funding. One NGO has an office in the ULB’s office as they have become partners for supporting RAY for identification of houses, toilets required, garbage sites, creating awareness etc; The NGO met did not have local presence in Jaipur before HUP. They continue to work in the slums even after the project has been completed, with another project on blindness. Communitization process accepted by Pune Municipal Corporation, First amongst all HUP cities Link worker and MAS under HUP were absorbed by Pune Municipal Corporation (PMC) under NUHM The MAS and Link workers have not been selected in the state as yet. However it has been decided that the exiting Link workers will get preference in being selected as ASHAs (30% extra weightage). The HUP Link workers have all been selected as ASHAs and all the MAS have been absorbed in NUHM. Learning from each HUP programme Comprehensive UHND services, WCC functioning, Model AWCs How to form MAS, Techniques of conducting household listing and mapping and how to organize UHNDs including preparing the due list, involving community members in the process and implementation 68 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Spread of HUP Programme MAS, Link Workers, comprehensive UHND services, WCC adopted by 26 cities of Maharashtra, Tamil Nadu, Bangalore and Pune is the learning site for Maharashtra MAS, Link Workers, UHND adopted by UP and Agra is the Learning site for UP MAS, Link Workers, UHND, MCTS adopted by Rajasthan and Jaipur is the Learning site for UP 69 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT ANNEX VII: QUANTITATIVE DATA 19 126 129 125 131 129 126 108 109 128 129 130 121 12 11 130 0 20 40 60 80 100 120 140 Figure 2.1: Percentage of target reached in scale-up cities for UHND, October 2014-June 2015 (* no target set; † no data available) 70 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT 0 98 97 97 100 68 84 100 0 38 15 100 48 25 84 81 0 0 58 0 20 40 60 80 100 120 Figure 2.2: Percentage of target reached in scale-up cities for MAS, cumulative to June 2015 71 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT 206 64 452 31 19 25 94 417 681 110 49 129 59 10 36 65 0 100 200 300 400 500 600 700 800 Figure 2.3: Percentage of target reached in scale-up cities for number of people gaining access to improved drinking water, Oct 2013-Sep 2014 (*: data not available) 72 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT 102 193 41 52 16 42 59 130 103 131 27 24 0 0 0 31 0 50 100 150 200 250 Figure 2.4: Percentage of target reached in scale-up cities for number of people gaining access to an improved sanitary facility, Oct 2013-Sep 2014 (*: data not available) 73 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Percentage of target achieved for RMNCH indicators for scale-up cities, October 2013-June 2015 % of Pregnant women received 3 ANC checkups Deliveries conducted at Facility % of newborn breastfed within 1 hour % of fully immunized infants (9-11 months) Patna 152.7 154.3 155.0 355.8 Cuttack 13.4 53.9 59.1 32.9 Rourkela 20.0 46.4 50.2 35.4 Raipur 66.3 3.2 7.8 127.1 Bhilai 44.1 3.1 5.1 68.5 Jodhpur 109.5 91.3 84.6 91.7 Kota 58.1 52.9 57.5 51.7 Ajmer 36.9 64.7 62.8 39.4 Dhanbad 8.4 14.4 7.6 18.6 Ranchi 26.0 33.6 35.0 59.7 Jamshedpur 13.4 20.6 24.6 24.7 Dehradun 84.4 34.7 43.1 69.6 Gwalior 42.5 33.7 31.7 71.5 Indore 0.0 0.0 0.0 0.0 Bhopal 23.5 8.2 8.1 38.6 Jabalpur 25.8 6.7 19.4 50.6 Lucknow 111.8 51.4 50.5 265.1 Kanpur (nagar) 130.0 27.4 27.4 288.2 Bengaluru 164.6 97.7 97.7 282.6 Kolkata 8.3 67.5 43.0 48.1 Total 52.4 37.5 37.4 81.3 74 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT ANNEX VIII: TA PROCESS INDICATORS Oct 2013 - Sep 2014 Oct 2014 - June 2015 Indicators Source of Verification Bihar CG Jh MP Od Raj UP UK Bihar CG Jh MP Od Raj UP UK Objective 1: Technical Assistance Objective 3: Convergence HUP provided assistance to state in development of NUHM PIP HUP program MIS Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes 0 0 No No Yes Number of City Level NUHM PIPs prepared with HUP's assistance HUP program MIS 15 56 14 70 12 33 131 6 42 19 17 70 2 0 0 6 Number of city level NUHM PIPs that integrated WASH with HUP assistance HUP program MIS 15 41 14 50 42 0 131 6 42 19 17 70 0 0 0 6 Number of city level NUHM PIPs that integrated nutrition with HUP assistance HUP program MIS 15 41 14 50 42 33 131 6 42 19 17 70 0 0 0 6 Number of Meetings / Consultations / Trainings / Workshops at national level by HUP to support the government in customizing HMIS for urban context Reports / Minutes N.A N.A N.A N.A N.A NA NA 0 0 N.A 0 0 NA NA Number of Meetings / Consultations / Workshops / Training on Customization of HMIS for urban context conducted by state government or city administrations with HUP assistance Reports / Minutes 0 4 4 0 0 3 7 0 5 0 5 0 1 3 4 0 Customized HMIS format for urban areas developed by Government published HMIS format for urban health 0 0 0 0 0 0 0 0 20 0 0 0 0 0 0 0 75 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT government with HUP assistance (National Level Target: 7 Cities) systems under NUHM Number of cities adopting customized HMIS in facilities covered under NUHM State / City Progress Reports submitted to NHM 0 132 0 0 0 0 0 0 3 99 3 0 0 0 0 0 MAS guideline developed by government with HUP assistance MAS guideline made available by government for use in NUHM roll￾out in states Yes Yes Yes Yes Yes Yes Yes Yes 0 0 Yes 0 0 0 0 0 Number of Meetings / Consultations / Workshops / Training on Customization of MAS guidelines conducted by state government or city administrations with HUP assistance Reports / Minutes 15 0 1 0 5 0 4 4 7 0 4 7 1 0 0 1 Number of MAS formed in the state under NUHM with HUP's assistance HUP program MIS 0 999 0 2192 0 0 0 0 0 3223 133 1690 1525 3600 0 63 Number of HUP-assisted cities where MAS / other groups formed under NUHM as per MAS guideline State / City Progress Reports submitted to NHM 0 180 0 696 0 0 0 0 17 15 5 801 1082 4 0 4 Number of MAS /other groups formed under NUHM in HUP assisted cities 0 16877 0 692 0 0 0 0 0 1455 73 894 1082 1764 0 63 UHND guideline developed by government with HUP assistance No No No No No 0 0 0 Yes Yes 0 0 0 0 0 Number of Meetings / Consultations / Workshops / Training on Customization on UHND 7 2 2 0 8 2 2 1 8 0 5 0 3 2 3 0 76 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT conducted by state government or city administrations with HUP assistance State conducting UHNDs under NUHM with HUP's assistance (National Level Target : 8 States) - - - - - 0 0 0 0 0 8 0 0 0 0 0 Number of facilities / sites where UHND was started under NUHM in HUP assisted cities 0 0 14 0 0 0 0 4596 0 549 538 40 1 96 91 972 Number of UHNDs conducted under NUHM in a quarter in HUP assisted cities 0 3995 350 0 0 0 0 4596 0 13493 5259 13368 0 8479 150 8748 CCC/WCC formed in the state under NUHM with HUP's assistance (National Level Target: 8 states) - - - - - 0 Yes 0 10 Yes 0 0 0 0 0 0 CCC/WCC Guideline or TOR developed by state government / city administration with HUP assistance Yes Yes Yes Yes Yes 0 Yes 0 Yes 3 0 0 0 0 0 Number of CCC/WCC formed by city administration with HUP's assistance 0 0 1 0 1 0 0 2 0 0 1 5 2 0 0 0 Number of intermediary agencies identified and supported by HUP to provide TA to government on urban health issues 0 1 0 0 0 1 1 0 0 0 0 0 0 0 0 0 77 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT ANNEX IX: OUTCOME OF TA PRODUCTS List of materials Why developed? Whether adopted by Government? Reason for Outome Advocacy material City Urban Health Profiles (Agra, Bhubaneswar, Jaipur, Pune) This was compilation of city level health demographic, administrative, health and WASh statistics; meant to be a ready reckoner for project cities. Yes These were project documents, and some of the data were taken in the City PIPs developed under NUHM. State Fact Sheets (Reanalysis NFHS - III) , 2011 - 8 HUP States This showed disaggregated health statistics from NFHS-3, highlighting the vulnerabilities of the urban poor that gets overshadowed by the generally good statistics of the urban areas. Yes These disaggregated data for the urban poor formed the main justification of NUHM and these statistics were used in the NUHM document as well as government presentations on NUHM. HUP Baseline Report and Factsheets, IIPS (Bhubaneswar, Jaipur, Pune) These were undertaken in 2011 to arrive at city level health and WASH indicators, covering both slum and non-slum populations. Yes Apart from the data, that were used in respective city PIPs; the tools and methodology were adopted by MP for state-wide health survey in 50+ cities as these form comprehensive evidence in the state CSR Factsheets on Urban Health These highlighted the issues of Urban Health and WASH, and also identified potential areas for intervention through CSR. No These were developed as project advocacy documents to attract more CSR investments in HUP states. The government is still not looking at CSR and its priority is channelizing its own investments in urban health. Behavior Change Communication for Urban Health, A Situation Assessment, CEDPA.pdf This was a landscaping exercise by HUP to identify potential BCC models that may used for urban health. No The report was shared with all HUP states, but the focus under NUHM is infrastructure and HR, and not so much on IEC/BCC at this point of time. The Burden of Disease among India's Urban Poor - A Study of Three Cities This was an attempt at profiling the burden of disease in cities (by slum and non-slum population), along with associated health seeking behavior and out-of-pocket expenditure (OOP). The idea was to build a case for the need of micro-health insurance for protecting the poor against catastrophic health expenditure. No Govt. insurance schemes (like RSBY) focused on hospitalisation and secondary/tertiary healthcare, whereas NUHM focused on primary healthcare through governments own delivery system. Hence it was perceived that there was no role for insurance under NUHM. Government planned to address the high out of pocket expenditure through public provisioning of primary healthcare. Micro Health Insurance Schemes in Urban India, A Compendium - MIA.pdf This documented micro-health insurance models for urban poor communities. The idea was to develop a micro-insurance model for slum population in HUP intervention slums. But the idea could not be carried forward as MoHFW was not interested in insurance models for primary healthcare. No Review of Governance Structures - Bhubaneswar, Jaipur and Pune This was undertaken as part of the model City Health Plans in 3 HUP cities. This formed part of the situation analysis and was used for planning the institutional mechanisms for urban health. No This document has proved a very important tool to understand the distinct role of a) Health b) ICDS c) Urban Development and their institutional arrangements in the cities starting from ground level up to the city level. Since the NUHM decided that the program will be led by the health department this was not utilized. However in the mega cities where NUHM will be implemented through the ULB this should prove to be beneficial. Rapid Assessment of Five cities - EPOS.docx As the IIPS baseline estimated indicators at the city level, the rapid assessment was undertaken in early 2012 for HUP project are level indicators. No These were meant for HUP project assessment and covered only HUP intervention slums; was not perceived as useful by HUP for city level planning under NUHM. State Urban Health Reports - 8 HUP States These were prepared as compendium of demographic and health indicators useful for urban program planning, based on secondary data sources like Census, NSSO, NFHS, AHS, etc. Yes The data from these compendiums were extensively used for State NUHM PIPs as these were the only available urban data. IEC Material Maternal and Child These were developed to be used by HUP Link No Health department already has enough IEC material from various sources. 78 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Health Workers and MAS members during community based trainings and meetings. Training Material Training Manuals for MAS Yes Contents of these training modules incorporated in the MAS induction module prepared by NHSRC and circulated to all states through MoHFW, as they were specifically requested by the NUHM. Orientation Module for MAS These were based on ASHA training modules under NRHM, adapted for urban contexts. Used for capacity building of HUP MAS members MAS Training Module- Maternal Health MAS Training Module- Child Health MAS training Module- Family Planning MAS Training Module- Nutrition Training Manuals for Frontline Workers Yes Contents of these training modules incorporated in the Urban ASHA induction module prepared by NHSRC and circulated to all states through MoHFW, as they Orientation & Induction Training were specifically requested by the NUHM. Module I These were based on ASHA training modules under NRHM, adapted for urban contexts. Used for capacity building of HUP Cluster Coordinators and Link Workers Maternal Health Training Module Module II Newborn & Child Health Module III WASH-Urban Health Training Module Module IV Nutrition & Growth Monitoring Training Module FAMILY PLANNING Urban Health Training Module VII Communicable and Non Communicable Training Module VIII(draft) Guidelines Operational Guideline on Mahila Arogya Samiti (MAS) These guidelines were based on the NUHM (draft) Implementation Framework, and used for HUP city demonstration activities. Yes The contents of MAS guideline were incorporated in the national guidelines on MAS and Outreach activities under NUHM, prepared by NHSRC, as they were specifically requested by the NUHM. Operational Guideline on Urban Health & Nutrition Day (UHND) Yes The contents of UHND guidelines were incorporated in the national guidelines on MAS and Outreach activities under NUHM, prepared by NHSRC, as they were specifically requested by the NUHM. Operational Guideline on City Coordination Committee (CCC) and Ward Coordination Committee (WCC) Yes States like Odisha, MP and Jharkhad adapted guidelines prepared by HUP and also included the plan of formation WCC and CCC with specific number as they were inclined to create these structures. Policy Document City Health Plan (Jaipur, Bhubaneswar, Pune, Agra) The attempt was to develop a model comprehensive city health plan (CHP), addressing both health and health determinants through clinical care, institutional and community based approaches. Yes Elements of the Situation Analysis and Mapping components of CHP were incorporated in the national guideline of State and City PIPs for NUHM as this was perceived as evidence based methodology. Nutrition Situation in the Gender Context in Urban India This document attempted to review the existing systems, key policies and schemes which are in operation to address the problem of malnutrition in children and women No As the Health ministry is not dealing with Nutrition programs (which is under WCD ministry through ICDS), NUHM has no space to incorporate the study findings on Nutrition and Gender. However adequate convergence and coordination of the WCD departments at central and state level would have created a positive outcome. 79 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT ANNEX X: COMPARISON OF THE CHANGE IN URBAN BUDGETS OF HUP AND NON-HUP STATES, 2010– 2013 State 2011-2012 Allocation 2012-2013 Allocation 2013–2014 Allocation Expenditure% Allocation 2014-15 Expenditure% % Change in Allocation (13/14 vs 14/15) % Change in Expenditure (13/14 vs 14/15) Bihar 1.08 1.08 13.91 0.0 16.93 0.6 21.7 0.6 Chattisgarh+ 0.8 40 10.97 18.4 24.3 58.6 121.5 40.2 Jharkhand 0.8 0.24 6.73 0.0 16.5 0.0 145.2 0.0 MP 1.53 2.36 23.36 1.2 78.71 19.6 236.9 18.4 Odisha 2.93 2.76 19.33 0.0 23.73 15.0 22.8 15.0 Rajasthan 7.68 5.67 40.8 0.0 56.66 19.4 38.9 19.4 UP 16.74 20.63 54.72 0.0 152.01 22.6 177.8 22.6 Uttarakhand 3.69 8.01 1.00 0.0 9.71 52.0 871.0 52.0 Assam NA NA 4.88 0.6 33.18 19.3 579.9 18.7 Manipur 0.80 0.88 0.77 0.0 2.11 23.7 174.0 23.7 Punjab 1.90 3.45 25.06 0.0 39.07 15.2 55.9 15.2 Telangana NA NA 0 0.0 48.63 0.0 0 0.0 Figures in INR Crores (Ten Million); Source, NUHM financial data for 2013-15; date from MTE report: 2011-13 Budgets for 2015-16 has not been released for most of the states +Funds allocated by the state government 80 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT ANNEX XI: GEOGRAPHICAL COVERAGE AND ACTIVITIES OF HUP CONSORTIUM Partners Geographical coverage Activities 1. Population Foundation of India, New Delhi States for Technical Assistance & Implementation 1. Uttar Pradesh 2. Madhya Pradesh 3. Odisha City covered for City Health Demonstration and Learning Program Bhubaneswar, Agra & Delhi. 1. Undertake implementation of HUP in assigned states. 2. Provide technical assistance in the three states. 3. Provide overall technical assistance and capacity building. 4. Overall responsibility for implementation of HUP in Bhubaneswar city health demonstration and learning program. 2. Plan India, New Delhi States for Technical Assistance & Implementation 1. Bihar 2. Uttarakhand 3. Jharkhand City covered for City Health Demonstration and Learning Program – Pune (Maharashtra) 1. Undertake implementation of HUP in assigned states. 2. Provide technical assistance in the three states. 3. Provide overall technical assistance and capacity building. 4. Overall responsibility for implementation of HUP in Pune city health demonstration and learning program. 3. Institute of Health Management and Research (IIHMR), Jaipur, Rajasthan. States for Technical Assistance & Implementation 1. Rajasthan 2. Chhattisgarh 1. Undertake implementation of HUP in assigned states. 2. Provide technical assistance in the two states 3. Provide overall technical assistance and capacity building. 4. Bhoruka Charitable Trust (BCT), Jaipur, Rajasthan. City covered for City Health Demonstration and Learning Program – Jaipur (Rajasthan) 1. Overall responsibility for implementation of HUP in Jaipur city. 5. CEDPA India, New Delhi. States for Technical Assistance 1. Bihar 2. Uttarakhand 3. Jharkhand 4. Rajasthan 5. Chhattisgarh 6. Odisha 7. Madhya Pradesh 8. Uttar Pradesh Cites covered for City Health Demonstration and Learning Program: 1. Pune (Maharashtra) 2. Bhubaneswar (Odisha) 3. Jaipur (Rajasthan) 1. Provide TA for mainstreaming of gender issues in the eight states 2. Provide TA for PPP related issues. 3. Provide support in reviewing IEC/BCC in the urban context. 81 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT 4. Micro Insurance Academy (MIA), New Delhi. States for Technical Assistance 1. Bihar 2. Uttarakhand 3. Jharkhand 4. Rajasthan 5. Chhattisgarh 6. Odisha 7. Madhya Pradesh 8. Uttar Pradesh Cites covered for City Health Demonstration and Learning Program: 1. Pune (Maharashtra) 2. Bhubaneswar (Odisha) 3. Jaipur (Rajasthan) 1. Provide technical support in developing Community Insurance Models. 2. Support scaling up of these models across the eight states. 5. International Institute of Population Sciences (IIPS), Mumbai, Maharashtra. States for Technical Support 1. Bihar 2. Uttarakhand 3. Jharkhand 4. Rajasthan 5. Chhattisgarh 6. Odisha 7. Madhya Pradesh 8. Uttar Pradesh Cites covered for City Health Demonstration and Learning Program: 1. Pune (Maharashtra) 2. Bhubaneswar (Odisha) 3. Jaipur (Rajasthan) 1. Carry out research and advocacy. 2. Develop tools, conduct baseline, mid￾term and end-line studies. 82 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT ANNEX XII: ASSESSING THE ACHIEVEMENTS OF THE DEMONSTRATION CITIES HUP undertook several efforts to assess the achievements of the demonstration cities. Rapid Assessment: In 2013 HUP commissioned EPOS Health India to conduct a Rapid Assessment (RA) in the form of a household survey in HUP slums in the five demonstration cities.25 This was essentially a mid-term survey, using similar methodology to the baseline survey, and conducted before HUP was scaled down in the demonstration cities.26 It measured the key indicators for the demonstration cities on a household basis by interviewing a sample of households with women who had given birth in the past two years from the HUP slum areas (about 400 households per city). Cohort Analysis: HUP also conducted a cohort analysis using its HMIS data from the demonstration cities to measure the key indicators. The RMNCH data in the HMIS was collated from Mother and Child Tracking (MCT) registers filled in by HUP frontline workers and reported quarterly. The differences between the two data sources may be expressed as follows, using one indicator as an example: RA: Using a representative household sample, what is the estimated percentage of women in this population who delivered their last baby at a facility? CA: Of the pregnant women identified by HUP frontline workers, what percentage delivered their baby at a facility? The two measures are not exactly the same. The RA measure relies on systematic sampling to contact a representative group of women, and the data is obtained by asking the women about the details of their last pregnancy during a single interview. The CA measure is calculated from data compiled over a period of time while the details of the woman’s pregnancy were monitored by the HUP frontline worker. It should be noted also that if a woman had two births during the past two years, she would be counted twice in the HMIS, whereas the RA only collected information about a single pregnancy from each woman. These two measures resulted in the data below: RA: Percentage of women surveyed in 2013 who said they had an institutional delivery for their last birth (from RA survey) CA:Percentage of women who had an institutional delivery of those who delivered between July 2012 and September 2013 (from project HMIS) 25 EPOS Health India Pvt. Ltd. (2013). Rapid Assessment In Five Cities: Agra, Bhubaneswar, Delhi, Jaipur & Pune. 26 The Rapid Assessment report is dated 2013, but there is no indication of when the data was collected. 83 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT While the two measures differ slightly, the relative difference between cities is roughly similar. Delhi had the lowest percentage of women delivering in a facility and Pune the highest for both measures. Agra showed a difference of 9% between the two measures, while the other cities ranged from 0.1 to 2.7% different. We conclude that the two methods for measuring the project indicators serve to confirm and triangulate the assessment of project achievements. The results for both the RA and CA analysis for some key RMNCH indicators that are available in both the Rapid Assessment survey and the HMIS are presented in the table below. Rapid Assessment, 2013 Cohort Analysis (HMIS), July 2012-September 2013 Agra BBSR Delhi Jaipur Pune Agra BBSR Delhi Jaipur Pune Percentage of pregnant women receiving complete ANC (3 ANC checkups, 100 IFA tablets consumed & 2 TT) 2.9 9.6 25.2 12.2 31.4 8.7 59.9 17.9 11.4 43.9 Percentage of pregnant women who received 3 or more ANC checkups 63.5 43.3 73.6 87.3 97.2 67.2 85.9 35.3 82.8 71.1 Percentage of institutional deliveries 73.8 95.5 55.4 87.6 97.0 82.2 96.3 58.1 84.0 96.9 Percentage of women receiving at least one post natal care visit 74.4 71.8 73.1 87.9 65.0 77.9 84.2 48.3 82.6 74.6 Percentage of currently married women aged 15-49 years using any modern method of contraception† 37.8 36.5 21.1 26.6 24.3 38.9 42.0 34.5 43.6 21.2 †HMIS data is calculated by number of visits rather than dates; figures represent the average for 36 visits. RA CA Agra 73.8 82.2 Bhubaneswar 95.5 96.3 Delhi 55.4 58.1 Jaipur 87.6 84.0 Pune 97.0 96.9 Total 81.9 91.1 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 100.0 RA CA 84 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Complete ANC: The HUP demonstration cities showed varied results in achieving complete ANC for the pregnant women in the project areas. HUP’s own MIS shows that 60% of women in Bhubaneswar received complete ANC during this period, while the RA estimate is much lower (10%). The results for the other demo cities are more similar between the two measures, and range from 3% (Agra—RA) to 44% (Pune—HMIS). Measuring complete ANC involves several components—meaning that it is a composite measure involving a series of questions in the RA questionnaire and the tracking of a series of events for a single woman in the HMIS. For this reason there may be a number of errors in both measurements. Still, it may be concluded that less than half of the pregnant women received complete ANC (with the possible exception of Bhubaneswar). Three ANC visits: Measuring whether a woman had three ANC check-ups is more straightforward than the complete ANC measure. Aside from the same high figure in the Bhubaneswar HMIS, the measures are more comparable. It can be concluded that at least two-thirds of the women in the project areas of four of the cities had at least three ANC visits, with the exception of Delhi. We would conclude that HUP’s community health mechanisms, including the MAS and UHNDs, were fairly successful in reaching pregnant women in the areas they served. This quantitative finding confirms the qualitative evidence that we observed during the site visits. Institutional deliveries: As discussed above, there is close corroboration between the RA and HMIS measures for this indicator. While in Delhi only 55-58% of women delivered in a facility, the other cities achieved from 75- 95% on this indicator. This confirms the statements of the MAS women that we met who said that nearly every woman is giving birth in a facility nowadays. At least one post-natal visit: The findings on post-natal care follow logically from the fact that most women gave birth in a facility; a high percentage was achieved in this area. Unfortunately the rapid assessment did not measure whether women had two PNC visits. Use of a modern contraceptive method: HUP calculated the percentage of couples using contraception by the number of the HUP frontline worker’s visits, not the dates. In the table the data presented is the average of 36 visits. The figures are roughly comparable to the RA estimate in Agra, Bhubaneswar, and Pune, but the HMIS measure is about 10% higher in Delhi and Jaipur. Overall contraceptive use ranged from a low of 21% (Delhi RA & Pune) to 44% (Jaipur HMIS). 85 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT ANNEX XIII: ACHIEVEMENTS AGAINST QUANTIFIABLE OUTCOMES Achievements on committed quantifiable scale-up indicators Output 2 : Lessons learned from HUP models and approaches shared beyond EAG states and cities Indicators Quantifiable Results and Remarks Agencies identified /strengthened for rolling out urban health models and processes – Two at national level and four at state /city level National Level: 1 and State Level: 3 NHSRC at national level; UP signed MOU with SIFPSA and Rajasthan with SIHFW for supporting NUHM related trainings. Odisha handed over HUP knowledge products to SHSRC, which provided an "urban health corner" for such products. Linkages established between state PMU / urban health cell and city PMU in seven cities National Level: NA and State Level: 302 This has not had HUP’s direct contribution. The linkages – which are reporting and monitoring in nature - have been established as required by the NUHM. However, HUP supported planning process for these cities Tools and guidelines of HUP project (MAS, City planning and HMIS) customized and adopted for scale up in seven cities National Level: 19 and State Level: All 8 states All the HUP states have customized and adopted tools and guidelines of HUP program. Urban Health Knowledge Management Resource Centre established - two at national level and four at state level. National Level: 1 and State Level: 2 National Level: PFI State level: Bihar and Odisha Achievements on committed quantifiable PPP indicators Output 2 : HUP serves as a platform to convene influential stakeholders around urban health issues Indicators Quantifiable Results and Remarks Consultations/workshops on partnerships in urban context – One each at national and international level and four at state level National Level: 2 WASH round table at PFI WASH round table at Jaipur New partnerships facilitated in four states /cities State Level: 1 Bihar - HUP assisted the state in operationalising 46 UPHCs through PPP, of which 15 are in the scale-up city of Patna Urban PPP cell / task force established in two states HUP state teams in Jharkhand, Odisha, Rajasthan and Bihar prepared TORs for the PPP cell but it did not go through as states were not interested in a urban specific PPP cell Ten PPP models assessed National Level: 6 and State Level 0 PPP Knowledge Resource Centre established within the TA institutions – Two at national level and four at state level 0 Achievements on committed quantifiable convergence indicators Output 3 : Expanded implementation of convergence to address health determinants among the urban poor through the departments of Health, Women and Child Development (WCD), Urban Department (UD) and municipal administrations Indicators Quantifiable Results and Remarks Workshops on urban governance and convergence – Two at national level and four at state level National level: 2 workshops (Regional workshops on NUHM at Mumbai in August 2013 and Kolkata in September 2014; where municipal commissioners from 6 and 8 large cities shared implementation challenges with 16 and 14 state NHM Mission Directors) State level: 8 workshops (orientation on role of ULBs in NUHM in all HUP states) Guidelines on models of convergence issued in four states Guidelines on UHND: 2 states (Bihar, Chhattisgarh) Guidelines on CCC/WCC: 3 states (UP, Bihar, Jharkhand) Tools and guidelines of HUP project convergent models and process (UHND, CCC/WCC) adopted for scale up in six cities UHND adopted in: 11 cities (2 in Chhattisgarh, 2 in Jharkhand, 3 in MP, 2 in Rajasthan, 1 in UP and 1 in Uttarakhand) CCC adopted in: 7 cities (1 in Jharkhand, 4 in MP, 2 in Odisha) WCC adopted in: 1 city (Rourkela) City level multi-stakeholder coordination committees set up in six cities 7 cities (1 in Jharkhand, 4 in MP, 2 in Odisha) 86 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT Achievements on committed quantifiable TA indicators Output 3 : Expanded implementation of convergence to address health determinants among the urban poor through the departments of Health, Women and Child Development (WCD), Urban Department (UD) and municipal administrations Indicators Quantifiable Results and Remarks Institutions identified to provide TA on urban health -Two at national level and four at state level One at national level: NHSRC and 2 at state level – SIHFW Rajasthan and SIFPSA in UP Training of Trainers (TOTs) for government functionaries on NUHM implementation initiated in four states Three at national level 1. NIHFW (2013) State nodal Officers training 2. Mumbai (2013) State Nodal officers training 3. West Bengal (2 trainings: Kolkata and Siliguri) State and city nodal officers State NUHM Program Implementation Plans (PIPs) prepared in eight states All eight City health plans prepared in four cities Facilitated 337 city plans Ten percent increase in budget allocation for urban RMNCH primary health care services in the eight states Facilitated 122% increase in allocation of urban health budgets (from 170.82 crore INR 2013-14 to 378.5 crore INR in 2014-15) Customized urban HMIS adopted by four states All 8 states have started reporting on NHM HMIS where the reporting hierarchies are customised for urban areas. However the data being collected is not segregated for urban slums. Guidelines for community groups and urban ASHAs adopted by four states /cities 1 at National level ASHA guideline prepared by NHSRC, jointly with HUP, and circulated to all states. 5 at state leve: HUP states like MP, Jharkhand, Odisha, Chhattisgarh and Rajasthan adapted the MAS & ASHA guideline, assisted by HUP state teams. Twenty percent of required number of ASHAs deployed in urban areas of eight states 8658/18831; 46% in 8 HUP states (as on January 2015, as per MoHFW reports) 87 USAID/INDIA HEALTH OF THE URBAN POOR FINAL EVALUATION REPORT U.S. Agency for International Development American Embassy New Delhi - 110 021 Phone: 91-11-2419-8000 Fax: 91-11-2419-8612 / 2419-8454