DISCLAIMER: The views of the expressed in this document do not necessarily reflect the views of the United States Agency for International Development or the United States Government USAID/INDIA NISHTHA – TRANSFORMING COMPREHENSIVE HEALTHCARE IN INDIA MIDTERM PERFORMANCE EVALUATION FINAL EVALUATION REPORT Collaborating, Learning & Adapting in India Mechanism (CLAIM), Task Order No. 1 IDIQ # 72038619D00001, Task Order # 72038620F00003 OCTOBER 26, 2022 DISCLAIMER: The views of the expressed in this document do not necessarily reflect the views of the United States Agency for International Development or the United States Government Contract Title: Collaborating, Learning and Adapting in India Mechanism (CLAIM) Contract Number: 72038619D00001; TO No. 72038620F00003 Submitted: October 26, 2022 Contractor: Panagora Group Authors: Dr. Elvira Beracochea, Team Leader; Dr. Chesta Sharma, Deputy Team Leader; Ashish Yadav, Evaluator; Manish Kumar, Evaluator; and Dr. Aman Borkar, Data Analyst. DISCLAIMER: The views of the expressed in this document do not necessarily reflect the views of the United States Agency for International Development or the United States Government ACKNOWLEDGMENTS The evaluation team wishes to thank the guidance and support of Chandan Samal, Subrato Mondal and Dr. Anuradha Jain of USAID/India. We are very thankful for the time and contribution of all those interviewed, especially the Community Health Officers (CHOs) and other state and district authorities that gave their valuable time and experience to inform this evaluation. We particularly, want to acknowledge the contribution and support of the NISHTHA team, who has been a close partner throughout this evaluation. We would like to specifically acknowledge the following team members who helped streamline this midterm performance evaluation: Dr. Swati Mahajan, Former Chief of Party, NISHTHA Dr. Ravi Kumar AV, Deputy Chief of Party, NISHTHA Dr. Jatin Dhingra, Lead Innovations, NISHTHA Madhya Pradesh Team Dr. Jyoti Benawri, State Team Lead, NISHTHA Mr. Anand Bairagi, Advisor M&E, NISHTHA Dr. Vaibhavrao Vilasrao Patil, Senior Program Officer, NISHTHA Dr. Roshni Dilbagi, Senior Program Officer, NISHTHA Mr. Pratik Modi, Senior Program Officer, NISHTHA Dr. Navin Vijay Kamble, Program Officer, NISHTHA Mr. Adarsh Sukumaran, Program Assistant, NISHTHA Chhattisgarh Team Dr. Prafull Kumar Sharma, State Program Officer, NISHTHA Mr. Rashid Ahmed Khan, Senior Program Officer, NISHTHA Dr. Pratiksha Pal, Senior Program Officer, NISHTHA Dr. Balu Nath Mote, Senior Program Officer, NISHTHA Ms. Sujata Pandey, Program Officer, NISHTHA Mr. Gaurav Kapoor, M&E Officer, NISHTHA Nagaland Team Mr. Rapungna Kishio Pairson, State Program Officer, NISHTHA Ms. L. Ethel Ngullie, Data & Program Associate, NISHTHA Assam Team Dr. Meena Jagannath Chavan, State Team Lead, NISHTHA Ms. Chayanika Saikia, Program Officer, NISHTHA Dr. Puneet Bhatt, Program Officer, NISHTHA Ms. Bhavna Sharma, Data & Program Associate, NISHTHA Meghalaya Team Ms. Annie Sushma Suchiang, State Program Officer, NISHTHA Ms. Lasumi Dakani Shylla, Senior Program Officer, NISHTHA CONTENTS Acknowledgments............................................................................................................................................................. 1 Contents .............................................................................................................................................................................. 2 Acronyms............................................................................................................................................................................. 1 Abstract ................................................................................................................................................................................ 3 Executive Summary ........................................................................................................................................................... 4 Introduction ........................................................................................................................................................................ 1 Project Background........................................................................................................................................................... 2 Evaluation Purpose ............................................................................................................................................................ 3 Evaluation Questions (EQs)............................................................................................................................................ 4 Methodology ....................................................................................................................................................................... 5 Data Collection Tools............................................................................................................................................. 5 Sampling ...................................................................................................................................................................... 7 Confidentiality ........................................................................................................................................................... 7 Data Analysis Methods............................................................................................................................................ 7 Limitations .................................................................................................................................................................. 8 Findings................................................................................................................................................................................. 9 1.............. To what extent has NISHTHA’s technical assistance supported GOI to achieve the overall objective of increasing access to quality comprehensive health care services at primary level?.................... 9 2......................Based on the available evidence, what is the progress toward the establishment of LLs, upgradation and operationalization of health facilities and access to quality comprehensive health care services at primary level?...............................................................................................................................................13 3...... Were the intervention activities and approaches implemented appropriately to address gender differences/gaps? What evidence exists to substantiate the reduction of gender gaps? ................................22 4............What are the accomplishments and challenges? How can the implementation be adjusted to address those challenges and speed up the interventions that have lagged behind (if any)?.........................23 5......................................................What are the key learnings out of the project implementation so far? 29 6........ What are the perceptions of the nearby community that is served by the HWC supported by NISHTHA?.........................................................................................................................................................................29 Conclusions.......................................................................................................................................................................33 Recommendations ...........................................................................................................................................................37 Annex 1: Evaluation Team (ET)....................................................................................................................................40 Annex 2. Evaluation Design Matrix .............................................................................................................................41 Annex 3: List of Documents Reviewed......................................................................................................................42 Annex 4. List of Key Informants..................................................................................................................................43 Annex 5: Sampling ...........................................................................................................................................................44 Annex 6: Data Collection Tools..................................................................................................................................50 KII: USAID, IP, MoHFW, State Stakeholders................................................................... 50 KII: NISHTHA State Teams ............................................................................................... 53 KII: Learning Labs (Field Interviews)................................................................................. 54 KII: Learning Labs (Partner INterviews)........................................................................... 56 KII: CHO Leader .................................................................................................................. 57 HWC Checklist and CHO Interview................................................................................................................58 Online CHO Survey ..............................................................................................................................................63 Community Interview ...........................................................................................................................................68 Annex 7. Timeline and Deliverables............................................................................................................................71 Annex 8. Tables................................................................................................................................................................72 Annex 9. NISHTHA’s Theory of change....................................................................................................................78 Page | 1 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov ACRONYMS AB-HWC Ayushman Bharat – Health and Wellness Centre AIDS Acquired Immune Deficiency Syndrome ANM Auxiliary Nurse Midwife ANO Assistant Nodal Officer ASHA Accredited Social Health Activist AVPN Asian Venture Philanthropy Network AYUSH Ayurveda, Yoga, Unani, Siddha, Homeopathy BSc Bachelor of Science CCCH Certificate Course in Community Health CGMSC Chhattisgarh Medical Services Corporation Limited CHO Community Health Officer CLAIM Collaborating, Learning and Adapting in India Mechanism CMHO Chief Medical & Health Officer CMO Chief Medical Officer CNES Center for North East Studies & Policy Research COVID Coronavirus Disease CPHC Comprehensive Primary Health Care DPM District Program Manager DTL Deputy Team Leader ELMS Electronic Learning Management System EQ Evaluation Question ET Evaluation Team FCR Findings, Conclusions, Recommendations FP Family Planning GOI Government of India GRAAM Grassroots Research and Advocacy Movement HIV Human Immunodeficiency Virus HLFPPT Hindustan Latex Family Planning Promotion Trust HLL Hindustan Latex Limited HMIS Health Management Information System HR Human Resource HWC Health and Wellness Centre IEC Information Education Communication IGNOU Indira Gandhi National Open University IIHMR International Institute of Health Management & Research IP Implementation Partner IT Information Technology JAS Jan Arogya Samiti KI Key Informant KII Key Informant Interview LL Learning Lab LOE Level of Effort LRP Learning Resource Package MCH Maternal and Child Health MCSP Maternal and Child Survival Program MD-NHM Mission Director – National Health Mission MDR Multidrug Resistant M&E Monitoring & Evaluation Page | 2 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov MEL Monitoring, Evaluation & Learning MMU Mobile Medical Unit MNCH Maternal, Newborn and Child Health MO Medical Officer MOIC Medical Officer in Charge MoHFW Ministry of Health & Family Welfare MoU Memorandum of Understanding MWRA Married Women in the Reproductive Age NCD Noncommunicable Disease OPD Outpatient Department PBI Performance Based Incentive PHC Primary Health Centre PSC Program Study Centre PSE Public Sector Enterprise SC Sub Centre SHP School Health Program SNO State Nodal Officer SPM State Program Officer SPO State Program Officer SSV Supportive Supervision ST Scheduled Tribe TB Tuberculosis TBI Team Based Incentive TL Team Leader TOC Theory of Change UPHC Urban Primary Health Centre UNICEF United Nations Children’s Fund USAID United States Agency for International Development USD United States Dollar WASH Water Sanitation & Hygiene WHO World Health Organization Page | 3 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov ABSTRACT The midterm performance evaluation of the NISHTHA activity answered six questions regarding the effectiveness of NISHTHA’s technical support to the Government of India (GOI) to implement a comprehensive primary health care (CPHC) system, operationalize health and wellness centers (HWCs) and facilitate public and private strategic partnerships. NISHTHA has provided effective and responsive technical assistance and contributed to the operationalization of 25,978 HWCs and the training and deployment of 18,490 CHOs. While NISHTHA has not been able to access Health Management Information System (HMIS) data to quantify increased access to CPHC—especially maternal, newborn and child health (MNCH), family planning (FP) services and tuberculosis (TB) treatment—key informants report that states are on track and value its support and innovations: CHO leadership training, i-Learn platform, Naga TeleHealth, etc. The online leadership training program empowered CHOs to improve health services in their HWC. Eight learning labs (LLs) were planned to address various operationalization challenges; three are operational, four have just started and one has not started yet. To improve access to quality CPHC, and especially MNCH/FP/TB services, it is recommended that NISHTHA continue supporting the GOI and states to implement a continuous quality improvement and to address a number of systemic challenges: integration of MNCH/FP, management and retention of CHOs, financial management and payment of performance incentives, health information reporting and use, and medicine supply. It is also recommended that NISHTHA improve its M&E system to monitor its performance, develop an effective exit strategy, support states to scale up interventions, and transfer its innovative problem-solving capacity. Page | 4 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov EXECUTIVE SUMMARY Background. The midterm performance evaluation of the NISHTHA activity was conducted from May to August 2022. The purpose of the evaluation was to assess progress and identify areas for improvement. With a budget of about U$13 million, NISHTHA was designed to achieve three main objectives: 1. Support the national and 12 state governments in their efforts to implement GOI’s vision of a CPHC system through effective and responsive technical assistance. 2. Establish an ecosystem for innovations and a strong learning agenda for designing, incubating and testing innovative solutions to address some of the key intractable barriers that impede the delivery of CPHC. 3. Facilitate strategic partnerships between the public and private sectors through establishment of platforms, alliances and advisory groups for CPHC in India. Evaluation methodology. This was a mixed-methods participatory evaluation that allowed the evaluation team (ET) to work closely with the NISHTHA team. Data collection methods included: document review, Key Informant Interviews (KII), HWC survey, online survey of CHOs and interviews with community members near upgraded HWCs. The ET analyzed NISHTHA’s monitoring and evaluation (M&E) indicator data, but had no access to HMIS data. This prevented a comprehensive analysis of secondary data to measure performance of various key interventions—e.g., increased access to quality services, increased successful TB treatment. The findings are largely based on quantitative data collected by the ET. Findings. The findings are presented by evaluation question: 1. To what extent has NISHTHA’s technical assistance supported GOI to achieve the overall objective of increasing access to quality comprehensive health care services at primary level? NISHTHA has provided effective and responsive technical assistance to GOI and selected states and is viewed as a strong partner. It has contributed to the operationalization of 25,978 HWCs and the training and the deployment of 18,490 CHOs. Although states have received different types of support, all states are on track to achieve their targets for upgrading HWCs. Improved access to quality CPHC was reported, but quantitative data are not available. It is not possible to say how many HWCs now deliver quality CPHC and improved MNCH/FP and TB services. The analysis of the activity’s Theory of Change (TOC) and M&E indicators showed that outcomes, as defined in the TOC, are not measurable and some lack corresponding M&E indicators. M&E indicators are mostly output indicators and not fully aligned to measure progress of the activity’s three objectives, or of how well the outputs lead to the main MNCH/FP and TB outcomes. 2. Based on the available evidence, what is the progress toward the establishment of LLs, upgradation and operationalization of health facilities and access to quality comprehensive health care services at primary level? LLs are small-scale operational research projects to solve problems, test innovative solutions or improve quality of care of the CPHC model. Eight LLs have been planned; three are ongoing, four have just started and one has not started yet. The dissemination and scale-up of their results have not been planned yet. States supported by NISHTHA reported to be on track to achieve their upgradation and operationalization targets, and this was confirmed by available public data. States vary in their achievement of the different components of the HWC model, but NISHTHA does not yet have Page | 5 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov an M&E system to monitor and compare state performance. Key Informants (KIs) reported that quality of care and community utilization vary from facility to facility and state to state. NISHTHA has reported to be assisting facilities to receive their National Quality Assurance Standards (NQAS) certification, but it is not known how many HWCs will be certified. 3. Were the intervention activities and approaches implemented appropriately to address gender differences/gaps? What evidence exists to substantiate the reduction of gender gaps? NISHTHA has integrated various gender-related topics in CHO trainings and developed a “gender guidance” for the integration of gender in CPHC. For example, the leadership course for CHO is meant for leadership quality enhancement of CHO who are 70% females on an average in the country. NISHTHA is working on integrating RMNCH+A services in HWC, 90% of the beneficiaries are girls and women, and the School Health Program (SHP), which NISHTHA supported had curriculum for empowerment of adolescent especially girls. In addition, through strengthening of wellness programs in NE NISHTHA is working with adolescents’ engagement through innovative strategies to address some issues like adoption of safe measures of FP, avoid teenage pregnancies and drug abuse. However, the evaluation question asked for evidence of reduction of gender gaps which is not a specific objective of NISHTHA. NISHTHA reported that it is not required to monitor how many states or HWCs apply gender guidance. KIs in some states reported concerns regarding early marriage and teen pregnancy, while matrilineal states do not perceive gender gaps affecting health care delivery or HWC performance. 4. What are the accomplishments and challenges? How can the implementation be adjusted to address those challenges and speed up the interventions that have lagged behind (if any)? NISHTHA has established effective working relationships with state health teams and has been reported to be instrumental in the Ayushman Bharat – Health and Wellness Centre (AB/HWC) program. It has contributed to the upgrading and operationalization of HWC through their assistance to train, deploy and support over 18,000 CHOs. This is a new cadre of health workers that has not been integrated into the human resource system of the states and that lacks a continuous education program. A recent assessment of CHOs by the GOI found that only 5 percent of CHOs scored above 75 percent in competence. NISHTHA has also introduced performance-based incentives (PBIs) for CHOs and the HWC team, and other tools and innovations in the last year. NISHTHA has effectively supported the GOI, which has focused on expanding the coverage of noncommunicable diseases (NCDs) screening. NISHTHA has developed operationalization documents, but it is not known yet how many of the operationalized HWC provide integrated MNCH/FP and TB services in the context of CPHC. Data are not yet available to show increments in MNCH/FP/TB coverage in supported districts. Several systemic weaknesses exist that affected the activity: CHOs being reassigned to respond to COVID, high turnover of state and district leadership, lack of access to HMIS data, stockouts of medicines and poor infrastructure (electricity, internet access, etc.), to name a few. 5. What are the key learnings out of the project implementation so far? As a phasic process that was initiated by the previous activity, the Maternal and Child Survival Program (MCSP), NISHTHA requires continuous support and monitoring to make decisions and implement solutions. USAID reported that by building on the achievements of MCSP, they were able to focus on the GOI’s priorities and contribute to the AB-HWC program. KIs reported, too, that NISHTHA has made important contributions to the AB-HWC program and developed tools and trainings as well as provided timely technical assistance. However, lack of access to quantitative data did not allow the evaluation team to measure its progress of this phasic process in the first half of the life of the activity and how much it has contributed to increasing MNCH/FP/TB coverage, which are the specific outcomes of the activity yet. We cannot confirm progress to the activity’s outcomes because there was no recent HMIS data to prove it in accordance with the activity’s indicators. Page | 6 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov 6. What are the perceptions of the nearby community that is served by the HWC supported by NISHTHA?1 1 In the course of the evaluation, this additional question was added to gather perceptions of the ultimate beneficiary of NISHTHA’s support; that is, the local communities served by the upgraded HWCs. Most informants had used their local HWC for NCD, Maternal and Child Health (MCH) or COVID vaccination services. They all reported improvements: availability of staff, medicines, daily services, longer hours and new posters. Only 5 percent were dissatisfied with the care due to lack of medicine or tests they needed. The informants suggested improvements in the delivery of emergency services, more staff and diagnostic services (e.g., X￾ray and ultrasound) and inpatient beds for patients in remote areas. These findings are consistent with the recent assessment of the AB-HWC program by the GOI and partners. Conclusions. NISHTHA has supported the GOI and select states to achieve the upgradation and operationalization of approximately 25,978 HWCs and the training and deployment of 18,490 CHOs, as planned for its Objective 1. As part of Objectives 1 and 2, NISHTHA has developed an innovative online CHO leadership program that was reported to have improved the performance of CHOs. NISHTHA has developed innovative partnerships that created a number of innovations, such i￾Learn and Naga Telehealth, which have been reported to expand access to quality CPHC as planned under Objective 3. However, NISHTHA’s TOC and M&E indicators do not allow the activity to measure how many of the supported HWC and CHOs are delivering quality CPHC or how much access to MNCH/FP and TB services—the main activity outcomes—has been increased. Three of eight planned learning labs have been implemented; four started in 2022, and one is yet to start. NISHTHA is well regarded by the Ministry of Health and Family Welfare (MoHFW) and supports states and other partners. In conclusion, the qualitative evidence gathered has demonstrated that NISHTHA is on track to achieve Objectives 1 and 3 and has made progress to achieve Objective 2. However, we were not able to triangulate these findings with quantitative data. 1. Recommendations for NISHTHA to accelerate and sustain health outcomes Technical assistance 1. Given the number of partners in each state and district, NISHTHA should support states to coordinate the contribution of each partner as part of one integrated state plan that keeps all parties mutually accountable and avoids duplication. 2. Document the leverage strategy and technical assistance model that facilitated the training and deployment of CHOs so this Health System Strengthening (HSS) modality can be replicated in other districts to accelerate the dissemination and scale-up of HWC interventions. 3. Accelerate improvement by developing scale-up strategy of innovations. For example, twin low￾performing HWCs with model HWCs to facilitate improvement and accreditation. A similar approach may be used with districts. 4. Develop and implement an effective handover and exit strategy in every state and at GOI level. Ensure effective handover of Technical Assistance (TA) outputs to states so all HWC standard operating procedures (SOPs), tools and manuals are available on https://ab-hwc.nhp.gov.in/ 5. While still supporting an integrated and comprehensive primary health care (PHC) model, focus on the effective integration of MNCH/FP and TB diagnosis and treatment in supported districts, and monitor their effective integration in CPHC outputs and outcomes. Page | 7 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov 6. Redesign/evaluate the TOC and M&E framework to adapt to NISHTHA’s objectives and intended impact. The current 35 M&E indicators are mostly health service delivery outputs and need to include additional indicators that measure the progress of health system strengthening proves to allow regular monitoring and course correction, based on quantitative evidence. Learning Lab Innovations 1. Develop scale-up plan for the successful LLs. 2. Document and facilitate dissemination of other innovations that have been developed beyond LLs, such as those related to COVID, and that may be adopted/adapted to improve access to quality CPHC, especially at State level. 3. Facilitate technical assistance for states and districts to adopt/adapt and sustain relevant best practices and innovations, including applying LL findings. Sustainability and management of the new cadre of CHOs 1. Given that the CHOs are a new cadre under contract, contracted for only 3 years and not part of the formal health system, there is a need to assist States to integrate CHO production, deployment, orientation, supervision and retention into the state HR management procedure manual and system . 2. Assist the GOI and states to develop an advanced and robust continuous education and supportive supervision program to support all CPHC health workers, especially CHOs. 3. Institutionalize the CHO Leadership program to ensure peer–peer mentoring and CHO– auxiliary nurse midwife (ANM)/accredited social health activist (ASHA) mentoring. 4. Advocate with GOI and states to disburse timely PBIs and team-based incentives (TBIs) and ensure efficient and regular payments and keep workers motivated. 5. Advocate with GOI to review the contract mechanism to ensure sustainability and visible career pathways for CHOs. 6. Assist all States to ensure that all CHOs in supported districts have access to untied funds and the support of Jan Arogya Samiti (JAS) and facilitate community participation and engagement in HWC activities and programs Delivery of quality CPHC package of 12 services 1. Strengthen monitoring and supervision of operationalization status, especially MNCH/FP/TB components, to ensure that quality CPHC package of 12 services is being delivered at all supported HWCs. 2. Continue support to HWC to receive NQAS accreditations, as well as developing a framework for quality CPHC service delivery at supported HWCs. 3. Operationalize team building at HWC level and improve collaboration between CHOs and ANMs, ASHAs, and medical officers (MOs) to deliver integrated MNCH/FP and TB diagnostic and treatment services, along with other services. 4. Assist supported districts to monitor and increase coverage and utilization of FP, MNCH and TB services integrated as part of the CPHC package, especially in remote and vulnerable communities. 5. Contribute to integrating multiple reporting and digital tools and apps and improve reporting and monitoring of program outcomes and avoid duplication. 6. Support the GOI to develop an effective referral system for integrated continuum of care from primary to secondary and tertiary care. Page | 8 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov 7. Assist states to engage in designing and implementing targeted community mobilization activities for increased uptake of CPHC services by specific minorities or marginalized populations not being reached yet. Partnerships 1. Explore for-profit private-sector partnerships to support specific components under the CPHC agenda in the states and develop a report with recommendations of partnerships for the GOI to expand CPHC access and coverage. 2. Explore and make recommendations for sustainable public private partnerships (PPP) models, especially in the scaling up of innovations, supply chain and ambulance transport. 3. Explore partnerships with Christian groups and other religious organizations, especially in the northeast regions. 4. Create academic partnerships and facilitate learning exchange forums with Christian groups and other religious organizations to ensure widespread dissemination of NISHTHA’s lessons and best practices. 5. Support states to strengthen relationships with community organizations to drive the community mobilization agenda for CPHC. 2. Recommendations for USAID 1. Work with GOI to allow NISHTHA access HMIS data that will enable them to track and report progress on HSS outcome indicators. If that is not possible, encourage NISHTHA to explore and implement other options such as conducting regular LQAS surveys, establishing sentinel sites, and gathering additional data during facility supervision visits, which NISHTHA already performs. Examples of progress indicators that would be desirable to monitor to see growth and progress of the activity’s three main objectives every quarter are: a) # and % of CHOs that meet competency score in supported districts b) # and % of HWCs that deliver quality MNCH/FP and TB services c) # and % of HWCs that have implemented a community engagement program and have increased footfall d) # and % of districts that meet selected CPHC coverage targets: such as MNCH/FP/TB coverage indicators e) # of % of supported districts and facilities that implement NISHTHA’s innovations 2. Continue assisting the GOI’s CPHC agenda and development of several remaining CPHC components until supported states are sustainable and do not require assistance. This may include improving community engagement, linkages with secondary and tertiary levels, and effective CPHC financing. 3. USAID/India should continue its HSS agenda in alignment with USAID’s revised HSS vision in future activities. 4. The endline evaluation should include contribution analysis and measurable outcome harvesting to demonstrate the effectiveness of the pathways between NISHTHA’s interventions and desired health outcomes. Page | 1 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov INTRODUCTION USAID/India contracted Panagora Group to conduct a midterm performance evaluation of a USAID/India primary health care activity implemented by JHPIEGO Corporation, called NISHTHA: Transforming Comprehensive Healthcare in India from May to August 2022. The NISHTHA (“dedication” in Hindi) activity was designed to support the GOI’s CPHC policy to provide universal access to primary health services through a national network of HWCs. Under this initiative, 150,000 sub centers (SCs), PHCs and urban primary health centers (UPHCs) are to be transformed into HWCs offering a set of 12 preventive, supportive, curative, and rehabilitative services. NISHTHA runs from November 2019 to November 2024 and has recently passed its halfway point. This report summarizes the findings of the midterm performance evaluation, analyzes the achievements and challenges of NISHTHA and makes recommendations to sustain these achievements to ensure that it will achieve its goal. Page | 2 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov PROJECT BACKGROUND Since 2018, through MCSP (USAID’s flagship program), JHPIEGO—the implementing partner (IP)—has assisted the GOI and 12 states to transform CPHC. Under MCSP, JHPIEGO: 1. Developed roadmaps for scale-up and implementation of operational plans for transforming SCs and PHCs (both rural and urban) into HWCs through a phased approach 2. Trained and deployed more than 2263 CHOs by successfully establishing quality-training ecosystems to ensure their competency development in the 12 states, leading to establishment of 70 Program Study Centers (PSCs) with 700 academic counselors, where nurses are trained before being posted to HWCs upon graduation 3. Established 1263 HWCs with basic infrastructure; supportive supervision/mentoring mechanisms; and systems that ensure efficient, integrated client-focused primary health care 4. Reached a substantial population of marginalized groups, especially Scheduled Tribes (STs) from central (Chhattisgarh, Jharkhand and Odisha) and northeastern states. The NISHTHA activity was awarded in 2019 with a budget of U$ 13 million and with the goal to build on the work undertaken by MCSP and to continue assisting with transformation, redesign and re￾engineering of primary health care in India. This goal is aligned with the goal of the USAID/India Partnerships Program for provision of equitable, comprehensive and client-centered primary health care. Particularly, NISHTHA is to contribute to improved MNCH/FP- and TB-related health outcomes for India’s marginalized and vulnerable populations, especially women and girls. The midterm performance of NISHTHA was evaluated based on the progress it has made toward its three main objectives: 1. Support the national and state governments in their efforts to implement GOI’s vision of a CPHC system through effective and responsive technical assistance 2. Establish an ecosystem for innovations and a strong learning agenda for designing, incubating and testing innovative solutions to address some of the key intractable barriers that impede the delivery of CPHC 3. Facilitate strategic partnerships between the public and private sectors through establishment of platforms, alliances and advisory groups for CPHC in India With a budget of $13 million USD, NISHTHA is expected to assist to upgrade and operationalize an estimated 30,050 health facilities into functional HWCs and establish a series of LLs across the country to improve the quality of CPHC, and thus improve access to quality CPHC services to approximately 143 million people. Page | 3 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov EVALUATION PURPOSE The purpose of this participatory midterm performance evaluation of NISHTHA was to assess the progress in achieving its objectives to date and identify areas for improvement, suggest adjustments to existing programs and recommend priorities for future implementation. The evaluation supported learning, innovation and adaptation by working closely with the NISHTHA team and identifying challenges incurred by the activity to date, especially during the ongoing COVID-19 pandemic. The evaluation formulated several recommendations for facilitating improvements and solving problems and for improving the activity’s capacity to respond to current and future challenges. It also made recommendation for future USAID/India program activities. Page | 4 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov EVALUATION QUESTIONS (EQS) The evaluation aimed to answer the following questions: 1. To what extent has NISHTHA’s technical assistance supported GOI to achieve the overall objective of increasing access to quality comprehensive primary health care services through HWC 2. Based on the available evidence, what is the progress toward the establishment of LLs, upgradation and operationalization of health facilities, and access to quality comprehensive health care services at primary level? 3. Were the intervention activities and approaches implemented appropriately to address gender differences/gaps? What evidence exists to substantiate the reduction of gender gaps? 4. What are the accomplishments and challenges? How can the implementation be adjusted to address those challenges and speed up the interventions that have lagged behind (if any)? 5. What are the key learnings out of the project implementation so far? 6. What are the perceptions of the nearby community that is served by the HWC supported by NISHTHA?2 2 In the course of the evaluation, this additional question was added to gather perceptions of the ultimate beneficiary of NISHTHA’s support— that is, the local communities served by the upgraded HWCs. Several sub-questions were used to evaluate NISHTHA’s progress and guide the development of the data collection instruments (see Annex 2). Page | 5 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov METHODOLOGY This was a mixed-method participatory evaluation whereby the ET (refer to Annex 1 for the team’s composition) worked closely with the NISHTHA teams at national and state levels to gather qualitative and quantitative evidence to evaluate progress to date and answer the EQs. A lot has changed since the design of the current TOC of the NISHTHA activity, including the impact of COVID. Therefore, we undertook a participatory approach that involved discussions with the NISHTHA team to understand the context of different components. The ET analyzed NISHTHA’s M&E indicator data but had no access to HMIS data. This prevented a comprehensive analysis of secondary data to measure performance of various key interventions—e.g., increased access to quality services, increased diagnosis and successful TB treatment, improved management of Acute Respiratory Infection (ARI), better outcomes of high-risk pregnancies, and reduced unmet need. Thus, the findings are largely based on quantitative data collected by the ET. The preliminary findings were presented and discussed with the NISHTHA team to ensure that recommendations to improve the performance of the NISHTHA activity were practical and realistic. A detailed evaluation design matrix is available in Annex 2. The evaluation included the following tasks: 1. Document review included: existing GOI documents on AB-HWC; state-specific documents on HWCs; USAID/India and NISHTHA’s IP’s documents; and available literature and secondary data on HWC and CPHC service delivery in India (see Annex 3) 2. Development and pre-testing of data collection methodology and tools 3. Review of HWC performance indicator analysis through NISHTHA’s M&E information systems, and assessment of select HWCs by ET in the five selected states 4. KIIs with USAID/India stakeholders, NISHTHA IP, NISHTHA partners, and KIIs with GOI-, state-, district- and block-level stakeholders 5. Collection and review of primary and secondary data from a purposive sample of HWCs in five states, through an HWC checklist, and interviewing the CHOs at HWCs and MOs at PHCs/ UPHCs 6. Conduct an online survey of CHOs in 10 intervention states. 7. Survey of community members using a convenience sample in the areas served by HWC supported by NISHTHA 8. Data analysis, presentation of preliminary findings and preparation of the draft evaluation report 9. Review evaluation report based on USAID/India feedback and final report submission DATA COLLECTION TOOLS Various data collection tools were employed to gather relevant information and answer the EQs: KII guides, HWC checklist, CHO interview guide, online survey of CHOs and community survey interview guide. The final data collection tools are included in Annex 6. A detailed sampling methodology for selecting the SC-HWCs is provided in the Annex 5. KIIS The ET conducted 117 KIIs with USAID/India; senior members of the IP team and their state counterparts in five states; GOI authorities; and state, district, block and facility staff. The interviews with USAID/India and the NISHTHA team helped to decide on the best participatory approach and reach agreement on roles throughout the evaluation process. The purpose of the KIIs was to assess the progress, challenges and lessons learned across all three activity objectives and various interventions. Some KIIs were done remotely; others were face-to-face, depending on the interviewee’s availability and COVID regulations at the time of the interview. Preliminary KIIs with USAID/India and the IP team were Page | 6 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov unstructured to inform the study design, sampling methodology and the analysis frame of the evaluation. A list of the parties interviewed is included in Annex 4. HWC ASSESSMENT AND INTERVIEW WITH CHOS The ET conducted interviews with 38 CHOs deployed in the visited SC-HWCs in the five sample states: Assam, Chhattisgarh, Madhya Pradesh, Meghalaya and Nagaland. The purpose of these interviews was to assess the upgradation of the HWC, and the overall satisfaction of the CHOs with the support provided and challenges faced. The HWC assessment used a checklist that was adapted from a Supportive Supervision (SSV) checklist used by NISHTHA to monitor the operationalization status of SC-HWCs. The checklist also assessed the capacity of the selected facilities to deliver quality CPHC. HWC capacity was defined with the NISHTHA team and included human resources available diagnostic services, medicines, contraceptives and supplies; and recordkeeping and reporting practices (see Box 1) to deliver the package of 12 services that comprise the current CPHC model. Photographs of relevant components and service areas were taken to corroborate and strengthen the findings. Photos were taken without patients, as the purpose was to assess the physical readiness of the facility to deliver relevant quality services. Box 1: Key Components of HWC Checklist • Status of Human Resource availability • Key training received • Availability of expanded services • Diagnostics availability • Physical infrastructure status • Shortage of drugs • Status of telemedicine • Status of wellness activities • JAS and Untied Fund status • Information technology (IT) adequacy (reporting) • Status of incentives to staff ONLINE SURVEY WITH CHOS The ET conducted a short online survey with 3073 CHOs in 11 NISHTHA intervention states: Arunachal Pradesh, Assam, Chhattisgarh, Madhya Pradesh, Manipur, Meghalaya, Mizoram, Nagaland, Odisha, Sikkim and Tripura. The purpose of the survey was to understand the trainings provided and how these trainings prepared the CHOs to provide comprehensive primary care services to improve community health, and to measure the degree of satisfaction with their work. The ET developed a structured questionnaire to gather the required information from CHOs and measure their level of satisfaction with their work at HWCs. The major focus of the questionnaire was on CHO trainings, their experience of working at HWCs and with the community, and challenges faced (see Box 2). The questionnaire was developed in alignment with NISHTHA team. Box 2: Key Components of CHO Survey • Trainings received • Perceptions on trainings received • Additional training needs • Common health issues in the community • Availability of basic resources • Key challenges • Suggestions for improvement SURVEY OF COMMUNITIES ADJACENT TO HWCS Based on a convenience sample of NISHTHA HWCs and adjoining villages, the ET conducted interviews with 154 people from 15 villages in four states: Assam, Chhattisgarh, Madhya Pradesh and Meghalaya. The purpose of these interviews was to gather information on community perceptions of their HWC and its services. Interviewed respondents represent six key demographic categories: pregnant women; Page | 7 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov women with a child less than 2 years of age; married women in the reproductive age (MWRA) who were neither pregnant nor with a child less than 2 years of age; adult men; elderly women; and elderly men. As this was not intended to be a representative sample of NISHTHA HWCs and adjoining population centers, we were careful in reporting and interpreting the survey results. A detailed sampling methodology for selecting the respondents and the villages is provided in Annex 5. SAMPLING NISHTHA spans across 12 states in India: Arunachal Pradesh, Assam, Chhattisgarh, Jharkhand, Madhya Pradesh, Manipur, Mizoram, Meghalaya, Nagaland, Odisha, Sikkim, and Tripura. It provides technical assistance and support in Direct Intervention Districts (see Box 3) and in other select districts (soft technical assistance) in the states. For the purpose of this evaluation and in collaboration with USAID/India, five states were selected: Assam, Chhattisgarh, Madhya Pradesh, Meghalaya and Nagaland. To capture critical learnings from different geographies, purposive sampling was used to shortlist the states from both the northeastern territory as well as the mainland. Based on NISHTHA’s HWC ranking scores, a representative number of sample sites—i.e., the total number and by type of facilities (HWC, PHC and UPHC)—were chosen randomly. Representation of top performing HWCs was ensured to understand key levers that can help leapfrog other HWCs. Detailed description of the sampling methodology is in Annex 5. Box 3: Direct Intervention Districts Direct intervention districts were considered for sampling – • 3 districts (out of 35) in Assam • 13 districts (out of 32) in Chhattisgarh • 4 districts (out of 52) in Madhya Pradesh • all districts (11) in Meghalaya • all districts (16) in Nagaland CONFIDENTIALITY The ET anonymized the notes from the interviews and coded the responses for analysis to protect the identity of those interviewed at all levels: national, state, district and community. The ET analyzed all audio recordings to respect respondent confidentiality and will be erased after the report is approved. DATA ANALYSIS METHODS Qualitative data were analyzed using Dedoose, a web-based qualitative data analysis program that helps organize findings to identify the main themes that emerge under each EQ. The ET conducted a thematic content analysis of the qualitative data gathered through documents and interviews and organized the findings in a table format by EQ. Next, key themes were organized to develop a findings, conclusions and recommendations (FCR) table. The FCR table helped inform the causal pathways used and assess results achieved so far. When appropriate or necessary, illustrative quotes of each theme are presented in the evaluation report. The identity of the informant is not disclosed, only the stakeholder group and state they represent, if the perspective is unique to that stakeholder group. For the analysis of the results of the survey of people in communities adjacent to the HWCs, we tabulated the responses and reported on percentages. Given that it was not a representative sample of people in HWC catchment areas, we did not calculate margin of error, confidence intervals or other statistics. Data analysis was conducted throughout the course of this evaluation to identify preliminary findings and conclusions jointly with the NISHTHA team. A presentation of the preliminary findings and discussion was held with the NISHTHA team, and an out-briefing presentation was given to USAID/India before Page | 8 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov the departure of the team leader (and was provided online) to review findings. The feedback received during these presentations and further discussions has been included in this report. LIMITATIONS No evaluation is free of bias. However, bias was identified and mitigated by having the meaningful participation of the NISHTHA team and prevent information bias. By selecting sites that would allow the ET to evaluate NISHTHA’s interventions, the ET visited project sites where NISHTHA provides full and soft technical assistance. The ET worked with the NISHTHA team to select a sample of the HWCs upgraded by the activity based on M&E data to ensure they are truly representative of the activity work to date. The ET also worked with the USAID/India mission to select sample villages for community interviews. The ET worked closely with USAID and NISHTHA teams and with project stakeholders to organize KIIs. To mitigate the risk of selection bias, prior to launching data collection, the ET requested that the NISHTHA staff members provide a universal list of stakeholders in advance of the KIIs to detect and prevent selection bias. However, availability of national authorities was limited because the evaluation coincided with the World Health Assembly, and many potential informants were in Geneva. Despite that, all major stakeholders available were interviewed and heard. Recall bias was mitigated by focusing on the current HWC and LL performance and not asking about previous years. The community was also asked for their observations on changes in the HWC for the past year, using a convenience sample of HWCs and adjoining villages. There may well be bias in this respect, as we chose respondents from villages adjacent to the HWCs, where it could be expected that villagers had more frequent use and knowledge of HWC services. Villagers in more distant areas may have quite different responses. Given these limitations, we did not attempt to triangulate the results of the community interviews with the KIIs and other data sources. Rather, they have been presented in a separate section. As explained above, the ET analyzed NISHTHA’s M&E indicator data but had no access to HMIS data. This prevented a comprehensive analysis of secondary data to measure performance of various key interventions—e.g., increased access to quality services. The findings are largely based on quantitative data collected by the ET. Page | 9 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov FINDINGS 1. TO WHAT EXTENT HAS NISHTHA’S TECHNICAL ASSISTANCE SUPPORTED GOI TO ACHIEVE THE OVERALL OBJECTIVE OF INCREASING ACCESS TO QUALITY COMPREHENSIVE HEALTH CARE SERVICES AT PRIMARY LEVEL? 1.1. NISHTHA is seen as an integral part of the AB-HWC program. However, the implementation of CPHC is complex and involves numerous stakeholders, of which the GOI is the main policy maker and leader, and NISHTHA is one of many partners (see Box 4). The GOI has invested $5.5 billion USD in the AB-HWC program and has set targets for each state to upgrade its SCs/PHCs/ UPHCs into HWCs. NISHTHA is reported to have built on the results of the previous project (MCSP) and has made an effort to coordinate with these stakeholders. Below is the reported vision of the GOI for the AB￾HWC program. Box 4. Partners reported to be working in the Barpeta District, Assam: • WHO – supporting quality initiatives (NQAS) • UNICEF – immunization • JHPIEGO – HIV/AIDS (RISE); HWCs (NISHTHA) • Hindustan Latex Limited (HLL) – laboratory services • Krishna Digital – free digital X-ray service • Spandan Foundation – free CT Scan Service • Center for North East Studies & Policy Research (CNES) – boat clinics • Hindustan Latex Family Planning Promotion Trust (HLLFPPT) – mobile medical unit (MMU) “We made three important decisions: First, we decided 65 percent of our investment to go to PHC and to move from selective PHC to Comprehensive PHC. Second, we decided to bring CPHC closer to the people at a ratio of 3 to 5K people, led by a strong PHC led by an MO. And third, to ensure community participation. … We are one with NISHTHA in all this.” (MOHFW) Along with its supportive role to the central GOI, NISHTHA also plays a significant role at the state level: “I was MD – NHM [Mission Director – National Health Mission] for 2 years, 2020–2021. I worked on comprehensive health care. We were the worst performing state in the country. JHPIEGO played a vital role in handholding us for setting up PHCs and enabled us to understand the components of CPHC. They helped us with our strategies and brought out good results. We were able to deliver as per the target through the hub￾and-spoke model where NISHTHA￾JHPIEGO played a big role. They helped us get equipped for setting up. Had it not been with their support, we would not have been able to deliver results. It is overall positive feedback.” (Former MD – NHM, Madhya Pradesh) Page | 10 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov 1.2. NISHTHA reports that it provides states with effective technical assistance to upgrade HWC and helps the states meet their targets (see Figure 1). This has been confirmed in the five sampled states where NISHTHA is reported to be an integral partner in alignment with the GOI’s priorities. At this time, the GOI’s focus is on screening of NCDs as part of the CPHC package. “The team helped address all the building blocks: team, training, linkages with the secondary facilities, especially for the NCDs, and three forms of cancer—breast, cervical and oral.” (MOHFW) “Lots of support has been provided by NISHTHA in NCD streamlining in terms of screening and systematizing reporting.” (District Joint Director, Assam) 1.3. NISHTHA provides various kinds of programmatic support, which is highly regarded by MoHFW and states. NISHTHA has been reported to be successful in establishing systemic learning, providing technical resources, establishing partnerships with different stakeholders and introducing a number of innovations in the AB-HWC program. “They have worked right from the inception of the program to operations and to training. The keep on providing guidance and continuous support. We are really appreciative of the team; they are always available.” (Chief Medical Officer [CMO], Nagaland) 1.4. Despite the positive qualitative findings on NISHTHA’s support to the GOI, a lack of quantitative data prevents USAID from measuring the progress, especially to the goals of improving MNCH/FP and TB outcomes. This may be partly attributed to the fact that the TOC and the M&E system of the activity have a number of design limitations to measuring the effectiveness of NISHTHA’s support to the GOI and the states (i.e., to achieve the objective of increasing access to quality CPHC and increase MNCH/FP and TB outcomes). 1.4.1.The first objective of NISHTHA is to support the GOI’s efforts and the states, which are the prime implementers of the AB-HWC program. NISHTHA’s support contributes to improved facilities in select districts in select states, but the TOC does not make clear NISHTHA’s contribution in the context of all systemic components of the AB-HWC program. For example, NISHTHA’s TOC states the main output of this objective is strengthened State HR capacity and a number of optimally functional HWCs that provide client-centered CPHC. NISHTHA is contributing to these outputs but not solely responsible for their achievement. Likewise, M&E indicator #23 requires NISHTHA to report the number of beneficiaries that accept FP methods at PHC level. However, increasing acceptance requires more than training CHOs on FP methods. Other health systemic factors—such as human resources management, medicine supply, health planning and financing, and health information systems—also contribute to improving health service delivery outputs and health outcomes, especially increasing MNCH/ FP and TB service delivery, the main outcome of NISHTHA. This evaluation was designed to measure NISHTHA’s performance and was not able to do a contribution analysis to ascertain the effectiveness of its causal pathways. 1.4.2.The TOC does not include measurable outputs of NISHTHA’s support. For example, one output is “strengthened system capacity,” but the activity has not quantified how much capacity will be strengthened and how it will be measured. Another output is “Optimally functional HWC,” but there is no definition of “optimally functional”, what the supported HWC will be able to do, or how many of these optimally functional HWCs it will support states to achieve every year of the activity. Page | 11 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov 1.4.3.Some outcomes as defined in the TOC are not measurable, and some lack corresponding M&E indicators. For example, CPHC is defined as care that is “community￾owned, continuous, integrated, equitable, decentralized and technology-enabled.” Out of all the numbers of upgraded HWC, it is not known how many of the HWCs NISHTHA has supported states to achieve so far are community owned, continuous, integrated, equitable, decentralized and technology enabled. Also, according to the TOC, NISHTHA is supposed to ensure that CPHC is “more resourced, resilient, responsive, and client centered,” and to increase satisfaction and utilization of care that leads to increased MNCH/FP and TB treatment coverage. The strategies in TOC do not explain how support to the GOI and states will lead to making CPHC more resourced, resilient, responsive, and client centered or quantify how many HWCs will demonstrate they are more resourced, resilient, responsive, and client centered. 1.4.4. NISHTHA’s ability to report on CPHC indicators is limited due to two reasons: The GOI does not allow partners to access HMIS data, and NISHTHA has not found ways to use alternative data sources for effective monitoring, such as LQAS surveys during their monthly HWC supervision visits. In addition, NISHTHA is asked to report on indicators for which it is only making a partial contribution, such as “pills cycles distributed” (see Figure 2). NISHTHA does not supply contraceptive pills or directly implement FP activities in supported HWCs that help distribute these pills. To continue reporting on these indicators, NISHTHA reports the same figures (in the last three quarters) but not actual data (see Figure 2). 1.4.5.No baseline exists for the MNCH/FP/TB outcomes in the TOC (which are addressed only through LLs), and their results will be limited to select areas and not all the states supported by NISHTHA. LLs are small-scale research projects conducted in a few districts in select states, and their results are limited to those districts. For example, to address TB services uptake, the Family Care Giver Model is being tested in two of the 52 districts in Madhya Pradesh. Similarly, to help reduce FP unmet need, NISHTHA will start this year conducting research to integrate and expand MNCH/FP services in Meghalaya only, not in the other 11 states. 1.5. The HWC survey found a number of improvements in the sampled facilities supported by NISHTHA. Also, KIs reported NISHTHA has successfully assisted States to increase utilization of HWC. The improvements were reported to be mainly the result of NISHTHA’s support to planning of the upgrading process at state and district levels, and the training and deployment of CHOs. Among these improvements, NISHTHA was reported to have supported access to the Untied Funds for CHOs to improve their facilities. NISHTHA also supported states in the design and Page | 12 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov implementation of) and TBIs (see Figure 3) and the creation of a portal for CHOs to enter their monthly performance achievements. However, 18 percent of CHOs and 50 percent of ANMs and ASHAs have reported not to have received incentives in the previous cycle. KIs also reported that the process that ensures efficient and timely PBI and TBI payment has not been worked out yet. KIs also suggested other topics for lfurther improvement in the sampled HWCs, such as to facilitate community participation and improve internet connectivity (45 percent of CHOs reported irregular interconnectivity at their HWCs). “Daily OPD improved. Out-of-pocket expenditure of patient has reduced. Family planning training … villagers are becoming aware about the services … yoga awareness has improved. Identified acute malnourished child through chart … awareness about ORS.” (CHO, Chhattisgarh).” 1.6. NISHTHA has been reported as “instrumental” in the training and deployment of CHOs (Figure 4) and especially in the leadership certification program. A total of 18,490 CHOs have been deployed across the 12 NISHTHA intervention states. The online CHO survey reported that CHOs were 84 percent female, 28 years of age in average, with 73 percent having a B.Sc. degree in nursing, and about 59 percent have been on the job for less than two years—that is, during the life of NISHTHA. The leadership program has trained 314 CHOs, or about 26 percent of the target. CHOs were reported to be responsible for implementing the CPHC program in the communities they serve. Their preparation for this job was reported as effective and appropriate by the majority of those on the job. It is expected that their effective performance will lead to the achievement of NISHTHA’s CPHC outcomes in the supported districts, but NISHTHA does not monitor their performance. 1.7. NISHTHA’s support is reported to contribute to improving quality of CPHC by supporting states in obtaining their NQAS certification of supported facilities in select states. 38 57 65 68 80 97 100 100 116 124 128 165 84 0 50 100 150 200 Odisha Assam Jharkhand Tripura Chhattisgarh Manipur Meghalaya Nagaland Mizoram Arunachal Pradesh Madhya Pradesh Sikkim Overall “NQAS is being rolled out for facility certifications. One UPHC has been certified. We have promised 90 facilities’ certifications in the state.” (CMO, Nagaland) Page | 13 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov 2. BASED ON THE AVAILABLE EVIDENCE, WHAT IS THE PROGRESS TOWARD THE ESTABLISHMENT OF LLS, UPGRADATION AND OPERATIONALIZATION OF HEALTH FACILITIES AND ACCESS TO QUALITY COMPREHENSIVE HEALTH CARE SERVICES AT PRIMARY LEVEL? The following section describes the progress and performance of LLs and HWCs, and access to quality comprehensive primary health care services in the intervention states based on secondary data, KIIs and sampled HWCs. 2.1. LEARNING LABS 2.1.1. NISHTHA had planned to establish eight LLs. Out of eight, three LLs have been implemented, four have just started, and one is yet to be initiated (see Table 1). A detailed progress status is presented in Annex 8. The LLs intend to seek out locally relevant solutions to key challenges related to access to affordable quality CPHC. NISHTHA reported that the LLs got delayed due to COVID, which engaged the entire states’ health machinery to manage the pandemic response. COVID also delayed implementation because the health staff had to focus on responding to COVID cases, and CHOs were deployed to deliver vaccination services instead of CPHC provision. “COVID has hampered the HWC operationalization process in terms of training, as entire health staff was mobilized to manage the pandemic in the state/district.” (DPM, Meghalaya) “It [COVID] affected the quality of work because supervision was lacking. All CHOs were engaged in vaccination drive. I was heading the district at that time.” (Assistant Nodal Officer [ANO], Chhattisgarh) Table 1. Status of NISHTHA’s Learning Labs as of May 2022 S.No. Learning Lab Start Date Progress Status 1 CHO Leadership Program (All intervention states) Aug 2020 ● 314 CHOs trained 2 Community monitoring and social audit mechanism for HWCs (2 districts in Jharkhand) Jan 2021 ● 35 JAS constituted ● Bank account not yet opened for untied fund ● Patient satisfaction survey done (Mar 2022) ● Community monitoring activities yet to be in place (e.g., health visioning and health planning, social audit, annual public dialogue) 3 Family care model for TB and multidrug-resistant (MDR) TB (2 districts in Madhya Pradesh) July 2021 (official launch) January 2022 (on-ground Launch) ● 77 HWCs selected, 77 CHO trained and 468 ASHAs oriented ● 272 TB patient reached, and 84 family members trained 4 Operationalizing AYUSH HWC (2 districts in Odisha) Apr 2022 ● State approvals sought, LRP material preparation in progress ● On-ground operationalization yet to start 5 Strengthening supply chain mechanism (SCM) across primary health care settings in Meghalaya (6 districts in Meghalaya) Mar 2022 (state approval) Apr 2022 (tentative start) ● State approvals has been sought ● On-ground implementation yet to start 6 Increasing uptake of FP services among the young and low parity in two districts of Meghalaya May 2022 ● On-ground implementation yet to start 7 Achieving continuum of care through digital Interventions TBD ● Yet to start 8 Midwifery care at HWC (3 districts in Arunachal Pradesh) Jan 2022 ● On-ground implementation yet to start Page | 14 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov 2.1.2. CHO Leadership Program, the first LL, was initiated in August 2020 in partnership with Indian Institute of Health Management Research (IIHMR) to build capacities of CHOs on leadership skills. This is a virtual program that is being delivered by the experts of IIHMR university and a pool of master trainers trained by IIHMR. A total of 88 master trainers have been trained in two batches (Sept–Oct 2021). To date, three cohorts and 314 out of about 1200 (26 percent) CHOs have been trained (see Figure 5). Cohort IV is ongoing, and 100 CHOs from Arunachal Pradesh, Manipur, Mizoram, Meghalaya, Nagaland, and Sikkim may have completed their training by July 2022 (see Table 2). Table 2: Progress of CHO Leadership Program Training Cohort CHOs Enrolled State Representation Period Status Cohort I 103 AP, AS, MN, MZ, MG, TR, SK, OD & MP Nov 20 to Feb 11, 2022 Completed Cohort II 110 JH, CHG Feb 5 to Apr 23, 2022 Completed Cohort III 101 MP Mar 12 to May 28, 2022 Completed Cohort IV 100 AP, MN, MZ, MN, MG, NG, SK Apr 23 to Jul 2, 2022 Completed Total 314 (Source: NISHTA's Report "Leadership Certification Program for Community Health Officer") Interviewed CHOs appreciated the curriculum and the mixed pedagogy of the program. It was reported that the leadership program has helped CHOs to improve planning of service delivery and outreach activities. Interviewed CHOs reported fewer medicine stockouts, as they are able to efficiently plan and use the demand and stocks. The program was also reported to have helped to improve motivation and team management. CHOs reported being able to communicate more effectively with the community, resulting in increased outpatient department utilization (OPD), increased NCD screening and increased trust of the community. NISHTHA team was reported to constantly provide support in terms of problem-solving and other support through WhatsApp groups, which was highly appreciated by the CHOs. KIs provided a number of recommendations for improving and scaling up the program (see Box 5). 2.1.3. The LL for Community Monitoring and Social Audit Mechanism for HWCs started in January 2021 in collaboration with a local implementation partner, Grassroots Research and Advocacy Movement (GRAAM). The LL is to last 18 months and is being implemented in two districts: four blocks in West Singhbhum and three blocks in Kunti in Jharkhand—that is, across 35 SHC-HWCs (20 in West Singhbhum and 15 in Khunti districts, respectively). NISHTHA has trained participants and constituted JAS, but no data exists on the results of the monitoring and the work of the various JAS yet (see Table 3). MNCH/FP/TB monitoring has not started. Box 5. Key recommendations of CHOs for improving the leadership program and for further scale-up • Offer both onsite and virtual training to overcome network issues in remote areas and improve participation and interaction • Arrange for smaller batch size to increase participation and efficiency of learning • Include hands-on practice and field visit for practical learning • Develop a robust M&E to measure and track participants’ achievements and overall performance of program for continuous program improvement Page | 15 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov Table 3: Progress Status of Community Monitoring and Social Audit Mechanism LL (March 2022) S.No Key Intervention Progress/ Status 1 Health visioning exercise for developing Health Plans ● Orientation of implementation partner is done ● Activity is yet to initiate 2 Development of HWCs wide health plan ● Plan for community health plan for each HWC developed ● Activity yet to start on-ground 3 Constitution of JAS ● 35 JAS as per guidelines have been constituted ● Bank account for untied funds not opened yet 4 Developing community monitoring and social accountability system ● State-, district- and block-level officials have been oriented on JAS and its key role ● Community monitoring framework finalized ● IVRS system launched in Dec 2021 but not yet functional ● Other planned systems yet to be initiated on-ground 5 Training and capacity building of JAS ● Trained on oriented on roles and responsibilities 6 Sensitizing PRI, Gram Sabha member, local leaders on JAS ● Not initiated yet 7 Initiating community monitoring processes with active community participation ● Village Health Sanitation and Nutrition Committee (VHSNC) community monitoring in progress ● Activities like IVRS, monthly performance review meeting by JAS, patient satisfaction survey, social audit, annual public dialogue yet to initiate on￾ground 8 Layering of Reproductive, Maternal, Newborn, Child Health and Adolescent (RMNCHA), TB and FP services monitoring through JAS ● Yet to initiate 2.1.4. The design of the Family Care Model for TB and MDR TB Learning Lab started in July 2021, and its implementation started in January 2022—in partnership with Noora Health—in the high-burden districts of Guna and Khandwa in Madhya Pradesh. Seventy CHOs3 will be trained as master trainers, who will further train ASHAs and other health staff. The trained ASHAs will give counselling and other support to TB patients and their families to provide care and improve healthy behaviors. The LL target is to enroll 1500 TB patients and their families, who will be provided with mobile-based support though automated WhatsApp messages. So far, the LL has trained 77 CHOs and oriented 486 ASHAs in Guna and Khandwa districts (see Table 4). The LL has been able to reach only 18 percent of the targeted TB patients to date, and only 84 family members have been trained. KIs have made recommendations to accelerate and improve the implementation of this LL (see Box 6). 3 Source: Concept note on “Family Care Model for TB and MDR TB.” Box 6. Key recommendations of CHOs for further scale-up of TB Family Care Model • Shift focus to ASHAs, as they are from the community and also are the primary caregivers at the community level • Strengthen community awareness and outreach to create and sustain health behaviors. • Integrate model with NIKSHAY portal to avoid duplication and improve follow-up of TB patients • Strengthen M&E to track progress, report outcomes and be able to make corrections Page | 16 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov Table 4: Total Number of HWC and Persons Participating in TB Family Care Model as of May 2022 Activities Planned Guna Khandwa HWCs Selected 70 57 20 CHOs Trained 70 57 20 ASHAs Oriented — 377 91 No. of Family Members Trained 1500 14 74 No. of Active TB Patients Reached 1500 155 117 (Source: NISHTHA TB family care model presentation) 2.2. UPGRADATION AND OPERATIONALIZATION OF HWCS 2.2.1. HWC upgradation is largely on track in terms of the number of the targeted health facilities that were to be converted into HWCs. All intervention states except for Tripura (73 percent), Jharkhand (75 percent), Odisha (84 percent) and Manipur (95 percent) have been able to achieve their upgradation targets (see Annex 8 for table of state targets vs. achievements). NISHTHA has reported to have facilitated the roadmap to operationalization—e.g., supporting the adoption of HWC guidelines, standardizing internal branding, enrolling and training CHOs, providing Information Education Communication (IEC) materials, systematizing reporting and drug indenting, incentive payment system. To contribute to the upgradation and operationalization of the HWCs, NISHTHA provides technical assistance (TA) and expertise to the GOI and States. In discussions with the NSIHTHA team in Madya Pradesh, the team reported five many interventions to upgrade HWCs: 1. TA to states to develop their program implementation plans (PIP) and HR management for recruiting CHOs; 2. Establish LLs; 3. Support M&E and Supervision;4. Create linkages and partnerships; and 5. Support in Health System Strengthening. 2.2.2. Several challenges were identified in the operationalization process. For example, KIIs with medical officers in charge (MOICs) in states like Chhattisgarh, Madhya Pradesh and Assam expressed limited understanding of HWC operationalization guidelines. This indicates the need for orienting MOICs on the HWC operationalization process as well. HR and equipment shortages were also reported, which prevents the provision of adequate expanded services—e.g., mental health, ENT, ophthalmology, oral health, geriatric and palliative health care, and trauma care. Underutilization of Ayurveda Yoga Unani Siddha & Homeopathy (AYUSH) was reported to prevent the participation of these professionals in the CPHC model.The GOI’s Operational Guidelines for CPHC provides guidance on key criteria toward the operationalization of HWCs. However, there is variation in progress by state when it comes to components like infrastructure, availability of essential medicines, diagnostics, availability of expanded services, etc. This explains the different strategies and types of interventions NISHTHA employs in each supported state. 2.2.3. CHO training and deployment are largely on track in eight states (80 percent to 100 percent achievement). Odisha is lagging behind at 38 percent, while Tripura, Jharkhand and Assam are at Page | 17 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov 68 percent, 65 percent and 57 percent, respectively (see Figure 4 in Evaluation Question 1). KIs reported concerns regarding retention, and various reasons were cited including security of female CHO, non￾availability of residential facility at HWCs and frequent transfers. KIs also highlighted that there are frequent CHO-ANM team coordination issues, which causes conflicts and require better and clear definition of roles. NISHTHA supports states to arrange for basic training and provides supervision and on-the-job training. As per the findings from the online survey, 74 percent of CHOs received six months of basic training and a certificate course in community health (CCCH) prior to joining. Induction training was conducted for the CHOs on various topics, such as national health programs, roles and responsibilities, and on working together with primary health care teams, after which they were deployed at HWCs. The majority (96 percent) of the CHOs interviewed mentioned that the basic course was easy and well organized. About 81 percent of the CHOs found the basic training had prepared them either extremely well or very well for providing CPHC. On-the-job training is also provided in supported states and includes a mix of technical and soft skill capacity-building areas (see Table 5). In the sampled state, good coverage exists for the basic training (89 percent), training on expanded services (82 percent), diarrhea and pneumonia (71 percent), FP (76 percent), and MCH areas (76 percent). AYUSH training has not been initiated in the sampled states. Training on TB Champion (overall 47 percent) coverage is very low in Assam (14 percent) and Nagaland (20 percent). JAS training (overall 53 percent) has not been conducted in Nagaland, and its coverage in Madhya Pradesh is only 18 percent. Drug and Vaccine Distribution Management System (DVDMS) training coverage (overall 18 percent) is very low in all the states, viz. 0 percent in Nagaland and Assam, 10 percent in Chhattisgarh and 20 percent in Meghalaya. The DVDMS software is not in use as reported by CHOs in all the sampled states. Overall Assam lags behind in most of the trainings. The CHO survey highlighted challenges regarding their training program and its applicability on the job and the need to continue improving the program based on CHO needs. Their main priority (88 percent) was to have the skills for HWC to improve utilization and footfalls. Amongst the CHOs who mentioned the basic training was not so useful, 57 percent mentioned that they find it difficult to apply their knowledge in their daily work. Others rated the basic training as too technical (20 percent) or difficult to understand (13 percent), and some just felt they didn’t get proper training (10 percent). Additionally, the KIs mentioned that the virtual on-the-job training has some challenges, such as poor network, low CHO participation, lack of hands-on practice, and interference with OPD timings. KIs highly recommended onsite training over virtual training for CHOs and other staff, including ANMs and ASHAs. It was also suggested that NISHTHA needs to support further refresher training, provision of handouts for quick reference, additional technical guidance and tools, etc. The CHO Online Survey respondents expressed a need for continuous education to update their skills and ensure the quality of the CPHC delivery. Figure 7 shows topics for continuous education suggested by the CHO survey respondents. For example, two-thirds of the respondents mentioned that they Page | 18 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov would like to learn about “how to ensure treatment adherence for NCDs,” and 61 percent of the CHOs would like to learn about “how to interpret laboratory tests.” Figure 7. Reported Topics for Future Training by 3073 CHO Respondents 2.2.4. Infrastructure and amenities were reported to limit the upgradation and operationalization process. Key infrastructure (e.g., OPD Space, Lab, medicine dispensing, patient waiting area) is largely in place in most of the sampled states. However, in Meghalaya it was observed that HWC buildings have inadequate space and that services are being provided from a two-room facility. Of the five sampled HWCs in Meghalaya and Nagaland, only two had separate toilets for males and females. Electricity was available in all the visited HWCs; however, in Meghalaya frequent and lengthy power cuts hamper service delivery, especially teleconsultations, which are essential for MOs to authorize CHOs to prescribe required medicines for patients. Figure 8. Infrastructure of Sampled HWCs (Source: Data from HWC Checklist and CHO Interview, N=38 HWCs/CHOs) In Meghalaya, none of the facilities visited had piped drinking water, and the CHO and the team have to fetch water from nearby borewell. As the facility is in a hilly terrain, sometimes it takes more than 30 Page | 19 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov minutes to bring water to the facility. Water is either boiled or filtered to make it safe for drinking. Similar observations were also made in some HWCs visited in Chhattisgarh. “Twenty-five centers are there, but they are not as per the guidelines. We are looking forward to improving them, as their infrastructure is very poor.” (CMHO, Chhattisgarh) 2.2.5. States reported that the complete package of expanded services as per government HWC guidelines is not being delivered in almost all the sampled states. Expanded services include ophthalmic care, ENT, oral care and mental health screening. Lack of training, equipment and medicine were reported to be the key reasons for not providing these services. Assam is lagging in most services. Chhattisgarh has reported 100 percent training on expanded services; however, due to lack of equipment and medicines, the services are not being delivered. GOI’s increased focus on NCD is reflected in the high percentage of screening and management of NCD cases across the states. Figure 9. Reported Availability of Expanded Services (Source: Data from HWC Checklist and CHO Interview, N=38 HWCs/CHOs) Medicine shortages are reported across all sampled states due to limited supply from the state headquarters. Most of the HWCs reported stockouts and that medicine logistics is a challenge. CHOs reported they have to bring the monthly stock from designated stores themselves. Most CHOs use their personal vehicles or public transport (bus) to bring the medicine stock. Medicine supply is reported as one of the major challenges, especially in remote or hard-to-reach locations in Meghalaya due to lack of, or limited, transport facilities. An increase in service delivery requires an increase in medicines and supplies purchased and distributed. Some KIs have reported that NISHTHA has supported states in including the HWC essential drug list in the procurement plan and process; however, adequate supply from state headquarters to the last mile needs to be strengthened. NISHTHA is planning an LL on supply chain in the state of Meghalaya only. 2.2.6. The supply chain problems that affect medicines also affect the diagnostics. Diagnostics were largely available for most tests (e.g., hemoglobin, urine test for pregnancy, multi￾parameter urine, blood sugar, malaria) in all the sampled states (see Annex 8). However, Assam has a shortage of glucometer strips, and CHOs have purchased the strips using untied funds. Visual Inspection with Acetic Acid for cervical cancer screening (VIA) hepatitis B and syphilis tests were not available in 86% 60% 82% 100% 40% 74% 0% 20% 40% 60% 80% 100% Assam Chhattisgarh Madhya Pradesh Meghalaya Nagaland Total Figure 10.Medicine Shortage Page | 20 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov most states. It was reported that CHO training on VIA and Hepatitis B has not been done, and for that reason test kits have not been provided yet. “There is inadequacy of lab equipment—analyzer, centrifuge, auto-analyzer, microscope and also of other equipment: auto scope (for basic ear care service), slit lamp (basic eye care services).” (MOIC, Assam) 2.2.7. Availability of internet access and teleconsultation were reported to affect the delivery of quality care (Figure 11). All states have provided a laptop or a tablet to the CHOs; however, internet connectivity was reported as an issue in every state. Very few of the HWCs have been provided with state internet connections (e.g., in Madhya Pradesh and in Meghalaya) in the facility. CHOs use their personal mobile phone data to access teleconsultation services to be able to communicate with the MO and prescribe required medicines. Findings from the community perception survey depict very low awareness of teleconsultation services, and none of the respondents interviewed ever used the service. Direct observation by the ET shows that 66 percent of the HWCs do not have any display depicting availability and timings of teleconsultations. It was reported that ASHA workers have created awareness in the community to increase demand for teleconsultation. Figure 11: Provision of IT Infrastructure and Teleconsultation (Source: Data from HWC Checklist and CHO Interview, N=38 HWCs/CHOs) 2.2.8. Naga Telehealth is reported to be working well in Nagaland in partnership with Karma Primary Health Care Services Private Limited. They provide telemedicine consultation to CHOs through their own platform customized for Nagaland, which will be transitioned to e-Sanjeevani soon, ensuring the sustainability and dissemination of this innovation. A total of 1500 teleconsultations were reported to have been completed so far in Nagaland. 2.2.9. Electronic Learning Management System (eLMS) platform (i-Learn) is another of NISHTHA’s innovations. It was reported to have been launched in Manipur, Meghalaya and Nagaland. Learning Resource Package (LRP) for CHOs on TB; management of labor; HWC portal; and Water, Sanitation and Hygiene (WASH) has been launched on i-Learn. The LRP aims at building capacities of Page | 21 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov CHOs in efficient and effective management of their HWCs. However, the ET did not find any evidence from primary data to corroborate this finding. 2.2.10. NISHTHA was reported to have developed a system for PBIs that motivate CHOs and TBIs for ANMs and ASHAs to meet their performance targets. The majority of CHOs had not received incentives in the month previous to this evaluation, which is often a cause of demotivation. No CHO in Assam, Madhya Pradesh or Nagaland had received their incentives. Except for Chhattisgarh (90 percent), fewer than 50 percent of ANMs and ASHAs have received their TBIs. “Motivation for CHOs is very important. They have come from cities and are now working in remote locations. They need to be given awards, recognitions, etc. They should get their PBIs on time.” (Joint Director, District, Assam) Figure 12. Reported PBI and TBI Payments in Sampled States (Source: Data from HWC Checklist and CHO Interview, N=38 HWCs/CHOs) 2.2.11. NISHTHA has reported to have started assisting states and HWCs to receive their NQAS certification. However, access to quality CPHC is not measured by NISHTHA M&E progress indicators. The ET was unable to verify how many HWCs had been certified; however, from the KIIs we understood that NISHTHA has just started this support and there was a very small number. Such an indicator would allow NISHTHA to report how many of the supported HWCs have successfully been certified. 2.2.12. Monitoring and reporting progress toward the planned results is limited due to lack of access to HMIS data and not having an alternative system. NISHTHA’s access to HMIS is limited both at national and state level. The activity staff reported using a checklist to track components of upgradation and operationalization progress and gaps at the HWC level. The data from the checklist is shared with respective districts to support HWC improvement. These data were not shared with the ET because it was not possible for NISHTHA to obtain authorization by the respective states. Therefore, it is not possible to comment on the status and adequacy of HWC operationalization monitoring. KIs at state and district level reported the need for NISHTHA to have a comprehensive long-term and short￾term implementation plan jointly agreed to by the state and, subsequently, a joint monitoring mechanism Page | 22 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov in place to track progress. District-level KIs also emphasized the need to have increased and regular supervision by NISHTHA team, especially in the intervention district where their team is not physically stationed. “NISHTHA, with their agenda, should have [a] joint meeting with development partners and state so as to be able to understand what area of focus will be there in the coming year, and also to have more collaboration between the development partners. Also, joint review meeting on what been achieved as planned.” (Ex. SNO, Chhattisgarh) “For the first year, NISHTHA’s support was good, but now the presence of project team members is not very regular. NISHTHA has more focus on the district where they are physically present.” (DPM Chhattisgarh) “Increased monitoring of HWCs by NISHTHA team will help to timely identify challenges.” (DNO, Meghalaya) 3. WERE THE INTERVENTION ACTIVITIES AND APPROACHES IMPLEMENTED APPROPRIATELY TO ADDRESS GENDER DIFFERENCES/GAPS? WHAT EVIDENCE EXISTS TO SUBSTANTIATE THE REDUCTION OF GENDER GAPS? 3.1. NISHTHA reported to have developed a guidance document to address gender and social inclusion as part of CPHC, and another on how to make HWCs gender responsive. It is not known or monitored how many states or districts have adopted these guidelines. 3.2. NISHTHA has reported that gender is one of the topics in the trainings they have developed. In fact, all NISHTHA activities were reported to address gender in an integrated way. For example, the leadership course for CHO is meant for leadership quality enhancement of CHO who are 70% females on an average in the country. NISHTHA is working on integrating RMNCH+A services in HWC, 90% of the beneficiaries are girls and women, and the School Health Program (SHP), which NISHTHA supported had curriculum for empowerment of adolescent especially girls. In addition, through strengthening of wellness programs in NE NISHTHA is working with adolescents’ engagement through innovative strategies to address some issues like adoption of safe measures of FP, avoid teenage pregnancies' and drug abuse. However, the evaluation question asked for evidence of reduction of gender gaps which is not a specific objective of NISHTHA. 3.3. The CHO Leadership Training includes mostly female CHOs. A CHO informant reported that when female CHOs play leadership roles in the community, this empowers girls to see them as role models, and they then want to study and go into health careers. 3.4. NISHTHA design lacks an M&E gender-related progress indicator that requires it to report on gaps identified and addressed. Also, NISHTHA lacks an M&E progress indicator to measure reduction on gender gaps. The TOC does not have an outcome to reduce gender gaps 3.5. Gender-related issues vary by state and community, so one solution may not fit all. For example, some KIs in states with matriarchal or matrilineal traditions reported that their states do not have gender challenges. CHOs in these states being mostly female are expected to understand gender issues, including the need for privacy when providing FP counseling and breast cancer screening. In contrast to this perception, other district- and facility-level staff mentioned the importance of adolescent health services, menstrual hygiene and reaching tribal and vulnerable populations. There are several subjects that are reported not to be addressed or that need higher Page | 23 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov focus by HWCs, such as Gender-based violence (GBV), mental health, postpartum depression, anemia, and early marriage and teen pregnancy. KIs suggested for NISHTHA to consider them for inclusion in the CPHC package. “All the surveillance coordinators are females. There are no discrimination for girl child. Sex ratio is in favor of females. A mother is a queen.” (JHPIEGO, Nagaland) “Breast cancer screening is easier with female CHOs. Currently, the infrastructure may not be great for privacy.” (SPM, Assam) “No gender-based violence but early marriages are prevalent. ASHA workers are running awareness programs for early marriage prevention.” (DPM, Chhattisgarh) 3.6. About 74 percent of facilities have separate male and female toilets for staff and patients. However, in states like Meghalaya and Nagaland, only two out of five facilities have male and female toilets for staff and patients. 3.7. NISHTHA/JHPIEGO helped to develop an innovative adolescent school health program that is now being implemented in 10 states and 218 districts. The program includes emotional wellbeing, menstrual hygiene and physical health, and it is particularly focused on girls. The MoHFW uses the training materials developed by NISHTHA and the training dashboard to keep track of trained teachers. The program is now being implemented by the USAID Momentum Project. The MOHFW informant reported that linkages between HWC and the local school have not been created yet and that empowering CHOs to work with schoolteachers is a great need to maximize the investment in the Adolescent School Health Program. 3.8. The answer to this evaluation question is based on qualitative findings that show that NISHTHA has integrated gender in its activities. However, the evaluation question asked about evidence to substantiate reduction in gender gaps, which we could not substantiate with quantitative data. This is because reduction gender gaps owing to NISHTHA interventions is not being monitored by the program at present. For example, there is no indicator of the number of male and female adolescents that are served by the supported HWCs. We understand this is difficult due to the lack of HMIS data. An alternative indicator would need to be considered. For example, during its supervisory visits, NISHTHA may work with CHOs to track adolescents that receive FP services or that are at risk of or victims of abuse. 4. WHAT ARE THE ACCOMPLISHMENTS AND CHALLENGES? HOW CAN THE IMPLEMENTATION BE ADJUSTED TO ADDRESS THOSE CHALLENGES AND SPEED UP THE INTERVENTIONS THAT HAVE LAGGED BEHIND (IF ANY)? 4.1. ACCOMPLISHMENTS In terms of the activity’s objectives, NISHTHA has made progress in all three objectives. 4.1.1. Objective 1. NISHTHA has supported the intervention states to operationalize AB-HWCs in line with the functionality criteria and target set by GOI. NISHTHA was also reported to have contributed to building the capacity of the new cadre of CHOs. A total of 25,968 HWCs have been upgraded and operationalized, and 18,490 CHOs have been trained and deployed at HWCs across 12 intervention states as of March 2022. Additionally, 116 training sites have been established as of March 2022. Page | 24 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov ● Training and deployment of CHOs at operational HWCs: Basic trainings were conducted for the CHOs on various topics, such as national health programs, roles, and responsibilities, and on working together with primary health care teams, after which they were deployed at HWCs. Basic training enables CHOs to get a better understanding of the program and helps them in providing comprehensive primary health care services and prepares them in performing preventive and promotive services. ● Technical assistance: NISHTHA has reported providing states with technical assistance on different operationalization components for establishing well-functioning HWCs: o Selection, counseling and admission of CHO trainees; timely completion of CCCH; conducting trainings; and ensuring deployment of CHOs across the intervention states o Upgradation and operationalization of HWCs: See selected manuals and guidelines developed by NISHTHA in photo 1. o Developing and contextualizing state￾specific wellness calendars and rollout of the wellness strategy for celebrating the wellness days o Establishment of LLs to test innovative ideas and fill in service gaps o Rolling out of teleconsultation services o Establishing JAS o Development of district performance rankings Photo 1. NISHTHA’s Selected Manuals and IEC Materials However, the ET was not able to evaluate ranking data or how many of the supported HWCs have all components in place. ● Reported increased utilization of HWCs in supported districts. Service uptake of the HWCs is reported to have increased. Increased institutional deliveries were also reported by KIs in Chhattisgarh and Nagaland. Although this is not part of the activity’s TOC, NISHTHA’s support has also been reported to have improved NCD (e.g., cancer, diabetes, and hypertension) screening, diagnosis, and treatment. Unfortunately, without HMIS data or NISHTHA’s supervision indicators, we cannot measure how much utilization has increased in NISHTHA’s supported HWCs. “Services have improved, especially in the diagnostic space. HWCs are doing 30–40 tests, even PHCs are not doing so many tests.” (SPO, Nagaland) 4.1.2. Objective 2. Innovative approaches are being implemented through various LLs and state￾focused technical assistance. NISHTHA has designed eight LLs to address various operational challenges. Three out of the eight LLs are in implementation phase, four have just started, and one is about start. CHO Leadership and Mentoring Model has completed the training of 314 (26 percent) CHOs from 12 intervention states as of May 28, 2022. These CHO leaders are reported to be Page | 25 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov equipped with skills such as leadership, management, conflict resolution, interpersonal communication, critical thinking, and public health planning to help them achieve the desired results through a team-based approach. CHOs’ communication skills have also been reported to have improved, which has resulted in increased trust in the community, thereby increasing OPD numbers and NCD screenings. “My communication skill has improved. Now I give more time to the patients to further understand their problems (not just limited to giving medicines). I have been able to help community members beyond their health issues, e.g., for pensions, information on ration card, etc. I have been able to create a very good rapport and [gain the] community’s trust.” (CHO Leader, Chhattisgarh) ● Additionally, to ensure continued CHO learning, NISHTHA’s i-Learn platform (Photo 2) has been launched in Manipur, Meghalaya and Nagaland. LRP for CHOs on TB, management of labor, HWC portal and WASH has been launched on i-Learn. The LRP aims to build capacities of CHOs in efficient and effective management of their HWCs. ● Community Monitoring and Social Audit Mechanism for HWCs: Thirty-five JASes have been constituted as per the guidelines, and state-, district- and block-level officials have been oriented on JAS and its key role. ● The Family Care Model for TB and MDR/TB: This model is implemented at 77 SHC HWCs and has built the capacity of 100 CHOs and 500 ASHAs at HWCs. This model has reached 20 percent of the targeted TB patients (300 against target of 1500) to date, and 120 family members have been trained. ● Naga Telehealth: This is implemented in Nagaland in partnership with Karma Primary Health Care Services Private Limited. They provide telemedicine consultation to CHOs through their own platform customized for Nagaland, which will be transitioned to e-Sanjeevani soon. A total of 1500 teleconsultations have been completed so far. 4.1.3. Objective 3. NISHTHA is an effective and responsive partner in alignment with the GOI AB￾HWC strategy and guidelines. KIs reported an effective working relationship between NISHTHA and state teams in all sampled states. ● Partnerships: NISHTHA has reported to have developed innovative approaches in partnership with Indira Gandhi National Open University for PSCs (for CCCH), IIHMR University (CHO Leadership Model), Noora Health (Family Care Giver Model for TB) and Karma (Naga Telehealth). ● Corporate Social Responsibility: NISHTHA’s Public Sector Enterprise partner Asian Venture Philanthropy Network, organized a South Asia Summit in February 2022 to explore future private-sector engagement in improving CPHC. ● Leverage: NISHTHA has reported that it leveraged $178.6 million USD (see Table 6) and unlocked $216.8 million USD from public and private sector. Photo 2. NISHTHA’s i-Learn platform Page | 26 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov Table 6: Reported Leveraged Funds by NISHTHA Quarter Leverage Reported (USD) Leverage Source Jan–Mar 2020 $1,052,496 BMGF, WHO, UNICEF Apr–Jun 2020 $73,090,000 NHM-PIP Jul–Sep 2020 $3,254,452 BMGF, VIVA Healthcare Oct–Dec 2020 $150,000 NISHTHA-CPHC Jan–Mar 2021 NA NA Apr–Jun 2021 $98,468,200 NHM-PIP & Unilever-WASH Jul–Sep 2021 NA NA Oct–Dec 2021 $2,330,000 BMGF, UNICEF Jan–Mar 2022 $264,810 J&J, WASH Total (USD) $178,609,958 4.2. CHALLENGES Key informants in sampled states and HWC reported several challenges to the effective delivery of quality CPHC services in the upgraded HWCs: 4.2.1. High Leadership Turnover. Frequent change in leadership at state levels and different agendas at the center and at the state level affect the program and the technical assistance provided by NISHTHA. On the other hand, stakeholder management at different levels of administration is also reported as a challenge. “Frequent change in leadership, it makes the work harder.” (JHPIEGO) 4.2.2. Regarding the systemic challenges, KIs report that district officials are often too overburdened to focus on HWCs. Some also reported not having received any guidelines from the state on how to implement the program on the ground. Lack of medicines and supplies, and PBI and TBI payments are also barriers. “Diagnostics equipment is a challenge area for improving quality of services. State purchasing agency (CGMSC) lacks capacity and are not able to timely procure and supply medicine and equipment to districts.” (State Nodal Officer [SNO] HWC, Chhattisgarh) “Timely supply of all equipment is needed. Easy-to-use equipment like digital hemoglobinometer is needed, user-friendly equipment is needed. HWCs do get supply of diagnostic kits, but since they do not get used, they get expired. Labor rooms are not there. Only a few SHC-HWCs are conducting deliveries. Regular medicine supply is needed. They have received only one-time medicine supply.” (CMO, Nagaland) “For supply chain, CHOs have to come and pick supplies. There are no supplies to the last mile.” (CMO, Nagaland) 4.2.3. Improving HWC demand. KIs reported the need to increase footfall and strengthen community engagement to support the HWC. Lack of awareness amongst communities and Page | 27 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov community outreach still prevent expanding coverage to all. Other reported challenges to increase access were bad roads, lack of transport (including ambulances) and limited IEC activities and poor awareness in villages. “The main challenge is to deliver CPHC to a society that does not know what that is, and who has lifestyles that need to change.” (Additional Secretary, MoHFW) “Public dealing is difficult. To explain benefits of meds and patients’ resistance is hard to handle. There are preconceived notions regarding medicine.” (CHO, Chhattisgarh) ● Improving access to remote communities. Remote communities and poor infrastructure prevent HWC from reaching the last mile. Alternative outreach and mobile services may need to be considered until the goal of being 3 to 5 kilometers from every village has been fulfilled. Expansion of CPHC requires more planning for deciding where to build the next HWC. Most states also have specific communities that continue to be left out of the programs, either due to their low social status or remote locations (or both). At this time, NISHTHA does not have a list of all the vulnerable and left out communities. “The state is struggling with bringing down its MMR and IMR. We need good training module. … More focus needed in hilly districts, migratory population, Char areas, tribal population. Specific training may be needed.” (SNO HWC, Assam) 4.2.4. HR management: CHOs are contract employees bonded for three years, and no career path or job security exists for them at this time. CHOs’ lack of accommodation at HWCs continues to be a major area of concern in all states. As a result, many CHOs spend their own money to reach HWCs, as their homes are not near HWCs. Lack of continuous education and motivation program may be affecting CHO retention and performance. Lack of retention strategy and a definite career path for CHOs (who are bonded for three years) may lead to many leaving the program after that period. The CHO cadre is new and not yet integrated into the CPHC team and HR system. The staffing indicators have not been estimated, and some CHOs may be serving populations that are too large to manage effectively. Some posts may need several CHOs to meet community needs. Many HWCs have large coverage areas and lack a sufficient number of designated staff. For example, one HWC visited had a population of over 30,000 and one CHO to serve them. “They do not feel accepted. Medical officers do not acknowledge/accept them. ANM are age-wise and experience-wise senior. They resist the new information and techniques by CHOs. They feel discouraged.” (Principal, Nursing College, Madhya Pradesh) “Capacity building of all staff is needed, especially for NCDs. They need refresher trainings. Current duration of training is not enough. Even ASHAs and MOs need training.” (CMO, Nagaland) “Some of the CHOs are really good, but infrastructure of HWCs is really bad. Motivation for CHOs is very important. They have come from cities and are now working in remote locations. They need to be given awards, recognitions, etc. They should get their PBIs on time.” (CMO, Nagaland) AYUSH doctors, despite being present at PHCs/UPHCs, reported to continue to be overlooked for appropriate trainings. “AYUSH doctors are somehow ignored with respect to capacity building on National Health Programs since last 11 years.” (MO, Chhattisgarh) Page | 28 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov 4.2.5. Infrastructure: Despite upgradation and operationalization in full swing, many HWCs lack basic amenities—water, electricity (including backup), cleanliness/hygiene, toilets (separate for male/female, for patient/staff)—and infrastructure: absence of designated rooms for services (lack of privacy), which continues to discourage people from accessing services and prevents CHOs from delivering quality CPHC. “No room for eye care services, no designated room for immunization, no storeroom in the facility, no attached toilet in the labor room. Water availability is an issue.” (MO, Chhattisgarh) 4.2.6. Linkages between CPHC and secondary and tertiary facilities have been inconsistent. This makes referrals and continuum of care difficult. 4.2.7. GOI’s AB-HWC focus is mostly on NCDs, compared to MNCH/FP or TB. KIs also reported that GOI’s focus is currently on NCD screening and treatment. This focus takes away from other critical service areas, including MNCH, FP and TB, which are the main outcomes of NISHTHA. “Focus more on awareness and advocacy. Maternal death has increased [due to] … lack of ANC in part, and to high-risk pregnancy. We need more CHOs and speed up the process.” (CMHO, Chhattisgarh) “Barpeta is an aspirational district, and there are many challenges in delivering health services: teenage pregnancy is very high, home deliveries are high, connectivity is an issue, remote HWCs do not have residential quarters for ANMs/CHOs, riverine areas population is difficult to reach, FP acceptance is very poor, anemia is very high, and many more. NISHTHA can help the district though planning and innovative approaches.” (DPM, Assam) 4.2.8. Health Information Systems Challenges ● Lack of access to HMIS data makes improvement of CPHC difficult. Information is essential to manage the activity and make informed decisions. ● CHOs now manage numerous registers, apps and portals that take a lot of their time. Data reporting continues to suffer in most states. Reporting in physical registers as well mobile applications creates a double burden on CHOs. Too many applications/platforms are also reported to burden the CHOs. “Data is still going in registers. We need to transfer it on app, but that’s duplication of effort. … Right now, we have too many platforms.” (District ANO HWC, Chhattisgarh) ● A need exists for an integrated digital approach that allows databases to share information and track progress. “Need to improve how we track the numbers of HWC and how we demonstrate impact.” (USAID) ● Supportive supervision. Some HWCs have no regular supervision visits, as they have no representatives from the NISHTHA team in some of the districts. Additional digital tools may expand NISHTHA’s support to the HWCs in most need of support. ● Teleconsultations and digital health tools need to be integrated into one common platform to reduce the reporting burden of CHOs. Most districts have poor internet access and limited or absent telemedicine services. Page | 29 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov 5. WHAT ARE THE KEY LEARNINGS OUT OF THE PROJECT IMPLEMENTATION SO FAR? 5.1. The strategy of building on the achievements of MCSP has been reported to have helped create continuity of effort and given momentum to the new activity toward achieving large numbers of upgraded HWCs. 5.2. The TOC needs to be reviewed regularly to ensure the effectiveness of the causal pathways and the contribution of the activity to the CPHC program are clearly defined. 5.3. The activity is in alignment with the GOI program, but its contribution is not easily measured and quantified due to lack of HSS output, outcome, and progress indicators, and lack of access to HMIS data. 5.4. Playing a technical consulting and facilitating role was reported to have allowed NISHTHA to be responsive and rapidly assist states with health service delivery problem￾solving. However, a multi-stakeholder coordination strategy at the state level is needed to ensure specific focus and ownership by different partners. In states like Assam, stakeholder coordination continues to affect NISHTHA’s outputs. 5.5. A PHC activity needs to also consider linkages of the PHC level with secondary and tertiary levels. These linkages are essential to support CHOs and ensure they perform and achieve all their goals. Most states do not have an effective referral and counter-referral system to ensure continuity of care. 5.6. Supply and demand both need to be developed. Any health care program is only as good as its utilization by the community. HWC services are primarily being utilized because they are closer to the community, and not necessarily because a dedicated IEC agenda around expanded services have been implemented. To improve access as well as quality (due to increased awareness), both demand and supply-side issues need to be addressed. 5.7. Monitoring and increasing the supply of medicines and other supplies needs to occur when increasing utilization to prevent stockouts. 5.8. Despite responsive technical guidance, the GOI and the states need to have a long-term vision for CPHC. This includes ensuring CHOs and the comprehensive primary health care team are motivated, focusing on all CPHC services to ensure outcomes for all beneficiaries, and engaging with community to ensure utilization of services provided. 6. WHAT ARE THE PERCEPTIONS OF THE NEARBY COMMUNITY THAT IS SERVED BY THE HWC SUPPORTED BY NISHTHA? The ET conducted interviews with 195 respondents in 15 villages in four states (Assam, Chhattisgarh, Madhya Pradesh and Meghalaya) to elicit community perceptions on HWC strengthening. Of these, 154 respondents reported having used their local HWC. The respondents represent six demographic categories:, pregnant women; women with a child less than 2 years of age; married women in the reproductive age group who were neither pregnant or with a child less than 2 years of age; adult men; elderly women; and elderly men. Of the 195 respondents, 94 percent have visited a health care provider in the past one year. Of the ones who visited a health care provider, 50 percent have visited a private provider (formal or informal). Approximately 35 percent of the respondents have visited a PHC or a higher government center or hospital as well. As we discussed in the methodology section above, the results of the community interviews should be interpreted with care, as the respondents were selected using a convenience sample of HWCs and villages adjacent to the centers. They should be seen as reflecting the views of people in these localities only and cannot be extrapolated to the entire universe of the NISHTHA/HWC catchment area. The selection of villages adjacent to the HWC could in itself introduce considerable bias. Nevertheless, they do provide some insights and, in some aspects, they coincide with findings of a recent GOI study that Page | 30 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov shows that communities notice improvements but more is needed to ensure quality care that meets all the health needs of the community. Awareness of HWC and utilization. Ninety-six percent of the respondents were aware of their HWC. Of these, 82 percent (154 out of 195) had visited their HWC at least once in the past year to access various services. These respondents were shortlisted for detailed interviews. Among the different demographic categories, adult men (72 percent) are less likely to visit the HWC despite being aware of it. Preferential access of health care services from a private doctor; perception of HWC services as not being good; lack of need of any services; and presence of nearby higher government facilities were some of the reasons reported for not visiting the HWC. Summary of perceptions. Below findings represent 154 respondents who are aware of their HWC and paid at least one visit to the HWC in the past year. 6.1. Reasons for selecting the HWC for services. Nearby location is reported as the most common reason for accessing services at HWC. Availability of tests; recommendation by family/friends/relatives/neighbors/ASHAs; availability of free COVID vaccination; registration for ANC services; and easy access for women have also been reported as some of the reasons for accessing services at HWC. Figure 13: Reasons for Selecting HWC (Survey of 154 Community Members) 6.2. Health issues for which HWC was visited. NCD-related services (38 percent) are the top choice of respondents when it comes to visiting HWCs. Other services such as MCH vaccination (35 percent), other diseases (33 percent) and children’s illness (15 percent) have also been reported. 6.3. Human resource who attended the respondent at the HWC. The majority of respondents reported that CHO attended to them at the HWC. However, ANMs continues to attend to people when they come to visit the HWC, especially for MCH services. 6.4. Changes observed at HWC. Of the 154 respondents, 66 percent observed changes in the HWC in the past two years, including increased availability of medicines, services and diagnostics; new staff; cleaner facilities; and improved infrastructure. Other changes they noticed include: HWC opens daily, HWC is open for longer hours, better care is being provided, and the presence of new posters. Page | 31 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov 6.5. Satisfaction with the HWC services. Approximately 65 percent of the respondents reported to be satisfied with the services at HWC. Availability of good services, medicines and tests (70 percent) and availability of staff who understand people’s problems (19 percent) have been reported as the reasons for satisfaction. Ease of access and cleanliness were also reported as reasons by some of the respondents. Five percent respondents reported dissatisfaction with the services due to non-availability of services and medicines and non-availability of staff at the time of need. Figure 14: Reason for Satisfaction with HWC Services (Survey of 154 Community Members) 6.6. Areas for improvement at HWC. Approximately 31 percent of respondents suggested improvement in services, primarily delivery and emergency services (see Figure 15). Among human resource issues, respondents suggest presence of a doctor, regular availability of CHO and other staff as areas for improvement. Respondents also feel more medicines and diagnostic services like X-ray and ultrasound should be available. Among infrastructure improvements, inpatient beds were the prime area of concern. Figure 15: Areas for Improvement at HWC (N = 154) 6.6. Time, mode of transport and money spent to access services at HWC. Of the 154 respondents who access services at the HWC, 86 percent go on foot. As most HWCs are located near to the villages, most respondents take 5–10 minutes on foot to reach the HWC. As a result, most of the respondents do not have to spend any money to reach the HWC. Respondents also use bicycle, auto rickshaw or their personal vehicle to reach the HWC. Page | 32 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov 6.7. Visits to the community by HWC staff. Of the 154 respondents who access services at the HWC, 61 percent get visits from the ASHA/ANM (see Figure 16). Page | 33 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov Figure 16: Purpose of HWC Staff Visits to the Community (N = 154) 6.8. Usage of telemedicine. Of the 154 respondents, all but two reported that the HWC did not use telemedicine to provide any services. 6.9. Referral services. Of the 154 respondents, only 10 percent received referral services (for issues ranging from NCD treatment, diagnostic services, eye care, and child birth services) from the HWC. This would indicate the HWC are fulfilling their PHC role as gatekeepers for effective and appropriate referrals. 6.10.Perceptions of why other people do not visit HWC. Of the 154 respondents, 43 percent reported that some people do not visit the HWC. However, most were unclear about who such people are. Some respondents said that people who have money, people who prefer private doctors, people who prefer going to higher government centers, or people who are not aware of the HWC are the ones who do not go to the HWC. 6.11.Encouraging other people to access services at HWC. Of the 154 respondents, 54 percent may encourage other people to access services at HWC. CONCLUSIONS Conclusions are presented by evaluation question and are based on the findings presented above: 1. To what extent has NISHTHA’s technical assistance supported GOI to achieve the overall objective of increasing access to quality comprehensive health care services at primary level? NISHTHA has supported the GOI to the extent of the following findings: 1.1. NISHTHA has supported the GOI to achieve the upgradation and operationalization of approximately 25,978 HWCs and the training and deployment of 18,490 CHOs. However, NISHTHA does not have access to HMIS data and NISHTHA’s M&E does not currently allow it to measure how many of these HWCs deliver quality CPHC and how many CHOs deliver services according to quality standards. 1.2. The evidence gathered has demonstrated that NISHTHA is on track to achieve Objectives 1 and 3 and has made progress in the last year to achieve Objective 2. The activity has reported to be working to accelerate implementation. Page | 34 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov 1.3. NISHTHA’s support to the GOI has been reported as essential at state and district levels and to have leveraged funding from GOI sources. They support states in planning and budgeting for CPHC. They have also increased access to Untied Funds by CHOs. 1.4. The original TOC has not been updated to reflect the causal pathways that lead to the desired outcomes of improved MNCH, increased TB diagnosis and treatment, and reduced unmet FP need. Lack of access to HMIS quantitative data and of selected M&E outcome indicators is a barrier to measure and monitor performance. 1.5. NISHTHA is effectively assisting the GOI to upgrade facilities in supported states. Improving and sustaining quality of care delivered by HWC is the next challenge. 1.6. NISHTHA has developed effective online training and PBI tools. Rationalizing the existing health information tools and avoiding duplication is important to reduce the burden on HWC staff. 1.7. Multiple digital interventions/solutions have been developed by NISHTHA. However, the Digital Health “package” of an effective HWC has not been developed yet. The related LL has not started yet, mostly due to COVID. 2. Based on the available evidence, what is the progress toward the establishment of LLs, upgradation and operationalization of health facilities, and access to quality comprehensive health care services at primary level? 2.1. CHO training and deployment is the most important achievement. ET’s survey of CHOs gave us clear evidence. Meeting the targets for HWC upgradation is another critical achievement. 2.2. Implementation of LLs is reported to be very valuable to the GOI because it helps solve operational problems and remove bottlenecks and barriers to efficient health service delivery. However, only three labs are in progress, and four others have just started. 2.3. The TB model is being tested in two districts of Madhya Pradesh. No scaling-up plan exists yet, and this may prevent expansion. 2.4. Monitoring and comparing performance across states and districts was reported not to be done due to lack of access to data, including HMIS data. This prevents identifying which states need more technical support and focused technical assistance, and making midcourse corrections. 2.5. Quality of training needs to continue improving and be part of a continuous education program that allows the new cadre of CHOs to fulfill their role and implement a CPHC in their communities. 2.6. The sustainability of the Leadership Program is not secured yet. Master trainers are available to continue the training, but no plan exists to sustain the training after the end of the activity. 2.7. COVID has provided an opportunity to develop a number of tools and learnings that may be applicable to other programmatic areas. 2.8. With respect of the comprehensiveness of care, the evaluation suggests that the GOI has developed a comprehensive package and CHOs have been trained on how to deliver the 12 services that are part of that package. The GOI has been focused on expanding some services that are new and for which there are several unserved patients in the community (e.g., breast cancer, hypertension, diabetes). NISHTHA has supported the GOI and states in this effort. However, in the context of the current focus, it is not known how MNCH/FP and TB services are integrated and delivered as part of the comprehensive package. For example, in an integrated CPHC model of service delivery, a woman of reproductive age may come to the HWC for cough and fever and should be tested for malaria and COVID, and also screened for cervical, breast and mouth cancer and receive FP, mental health and Page | 35 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov GBV counseling. She may also be asked about the vaccination and health status of her children, and if there are others with chronic cough in the household to screen for TB. At this time, NISHTHA has reported to be developing the CPHC service delivery model through a number of LLs, most of which have just started. In sum, lack of MNCH/FP/TB HMIS data prevents us from measuring progress toward the MNCH/FP/TB outcomes, which are the main outcome of NISHTHA. 3. Were the intervention activities and approaches implemented appropriately to address gender differences/gaps? What evidence exists to substantiate the reduction of gender gaps? 3.1. NISHTHA does not have a target or an indicator that monitors gender gaps and their effective reduction. 3.2. NISHTHA has developed guidance documents on gender in the operationalization of HWCs. It is not known how many states have adopted the guidance and have implemented it yet. 3.3. A gender approach to ensure women’s participation and involvement in their local HWC has not been developed yet. 3.4. Seventy percent of CHOs are women, and accommodation and security is a concern in remote facilities. 3.5. Each state had different gender issues regarding women and girls to address in the context of MNCH/FP service delivery that have not been addressed and that were not included in NISHTHA’s scope of work and results framework. 4. What are the accomplishments and challenges? How can the implementation be adjusted to address those challenges and speed up the interventions that have lagged behind (if any)? 4.1. There are a number of achievements. Here we highlight three: 4.1.1. NISHTHA has successfully built on the achievements of MCSP. Continuity of strategic support to CPHC is reported as a success factor to support the GOI achieve its HWC upgradation targets. 4.1.2. NISHTHA developed an effective CHO training and deployment program. 4.1.3. NISHTHA is responsively supporting states in achieving their targets with respect to HWC upgradation and operationalization. 4.2. A number of challenges to the success of the HWC program exist, but only a few that NISHTHA can help address with the resources available: 4.2.1. Ensuring the sustainability of the CHO program is one of them. 4.2.2. Addressing and institutionalizing the successful integrated delivery of MNCH/FP services as part of the CPHC model is the other. For this to happen, the TOC will need to be adjusted and the implementation of LLs accelerated. 4.2.3. Improve the use of digital technologies, rationalization of digital tools to support delivery, and reporting of MNCH/FP and TB services. 4.2.4. Although there has been improvement in strengthening systemic weaknesses, there are challenges to the implementation of NISHTHA such as lack of medicines that may become more marked with increase in demand for services. NISHTHA has started to implement a LL to address the challenges in medicine supply chain in Meghalaya state only. 5. What are the key learnings out of the project implementation so far? Page | 36 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov 5.1. USAID is perceived as strong partner in building an effective CPHC system. KIs from the GOI and visited states reported that the support and technical assistance was instrumental to the HWC program. 5.2. The CHO cadre is new and under contract, and CHOs are looking for a career path and recognition and timely incentives to perform their duties. 5.3. CPHC outputs are not only dependent on NISHTHA’s technical support. They also depend on the supply of medicines and diagnostics and the generation of demand side and community uptake. NISHTHA’s contribution to outputs and outcomes needs to be defined and quantified based on clear causal pathways to deliver them. 5.4. The TOC of the activity needs to be reviewed to ensure that the original design assumptions are still valid and allow the activity to monitor outputs, outcomes and overall progress as well other necessary health indicators. 5.5. NISHTHA does not have a baseline and does not have access to HMIS data or gather its own data to measure progress and make decisions. NISHTHA is reported to effectively and responsively be facilitating the implementation of the AB-HWC program, though. Its effect on footfall and access to quality CPHC is not known. Without access to HMIS data, NISHTHA has limited data for decision making. NISHTHA cannot monitor its impact in terms of footfalls, use of services, access to comprehensive services. 6. What are the perceptions of the nearby community that is served by the HWC supported by NISHTHA? 6.1. Most of the community members interviewed in villages immediately adjacent to the HWCs have made use of their local HWC, mainly due to local availability and for the services they provide. 6.2. About two-thirds of the respondents have noticed improvements in the last two years, such as increased availability of medicines, services and diagnostics; new staff; cleaner facilities; and improved infrastructure, and 65 percent are satisfied with the care received. Only 5 percent were really dissatisfied due to the lack of medicines or staff at the time of service. 6.3. Areas of improvement included expanding services—providing emergency services, more staff and medicines. 6.4. Fifty-four percent of respondents would encourage others to access the services of the HWC. 6.5. Without a representative sample and a baseline measure to compare these figures, it is hard to draw more general conclusions about changes in HWC services. Page | 37 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov RECOMMENDATIONS Recommendations for NISHTHA to accelerate and sustain health outcomes Technical assistance 1. Given the number of partners in each state and district, NISHTHA should support states to coordinate the contribution of each partner as part of one integrated state plan that keeps all parties mutually accountable and avoids duplication. 2. Document the leverage strategy and technical assistance model that facilitated the training and deployment of CHOs so this HSS modality can be replicated in other districts to accelerate the dissemination and scale-up of HWC interventions. 3. Review and follow up the recommendations made the CHOs in the evaluation survey: a. Offer both onsite and virtual training to overcome network issues in remote areas and improve participation and interaction b. Arrange for smaller batch size to increase participation and efficiency of learning c. Include hands-on practice and field visit for practical learning d. Develop a robust M&E to measure and track participants’ achievements and overall performance of program for continuous program improvement e. Shift focus to ASHAs, as they are from the community and also are the primary caregivers at the community level f. Strengthen community awareness and outreach to create and sustain health behaviors. g. Integrate model with NIKSHAY portal to avoid duplication and improve follow-up of TB patients h. Strengthen M&E to track progress, report outcomes and be able to make corrections 4. Accelerate improvement by developing scale-up strategy of innovations. For example, twin low￾performing HWCs with model ones to facilitate improvement and accreditation. A similar approach may be used with districts. 5. Develop and implement an effective handover and exit strategy in every state and at GOI level. Ensure effective handover of TA outputs to states so all HWC SOPs, tools and manuals are available on https://ab-hwc.nhp.gov.in/ 6. While still supporting an integrated and comprehensive PHC model, focus on the effective integration of MNCH/FP and TB diagnosis and treatment in supported districts, and monitor their effective integration in CPHC outputs and outcomes. 7. Redesign/evaluate the TOC and M&E framework to adapt to NISHTHA’s objectives and intended impact. The current 35 M&E indicators are mostly health service delivery outputs and need to include additional indicators that measure the progress of NISHTHA’S health system strengthening process to allow regular monitoring and course correction based on quantitative evidence. 8. Monitor the number of supported HWCs that are certified and quality of care of services provided. Innovations 1. Develop scale-up plan for the successful LLs. 2. Document and facilitate dissemination of other innovations that have been developed beyond LLs, such as those related to COVID, and that may be adopted/adapted to improve access to quality CPHC, especially at state level. Page | 38 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov 3. Facilitate technical assistance for states and districts to adopt/adapt and sustain relevant best practices and innovations, including applying LL findings, particularly MNCH/FP/TB service delivery and supply chain improvements. Sustainability and management of the new cadre of CHOs 1. Given that the CHOs are a new cadre under contract, bonded for 3 years only and not part of the system, there is need to assist States to integrate into the state HR manual how to manage CHO production, deployment, orientation, supervision and retention. 2. Assist the GOI and states to develop an advanced and robust continuous education and supportive supervision program to support all CPHC health workers, especially CHOs. 3. Institutionalize the CHO Leadership program to ensure peer–peer mentoring, and CHO￾ANM/ASHA mentoring. 4. Advocate with GOI and states to disburse timely PBIs and TBIs and ensure efficient and regular payments and keep workers motivated. 5. Advocate with GOI to review the contract mechanism to ensure sustainability and visible career pathways for CHOs. 6. Assist all States to ensure that all CHOs in supported districts have access to untied funds and the support of JAS, and facilitate community participation and engagement in HWC activities and programs Delivery of quality CPHC package of 12 services 1. Strengthen monitoring and supervision of operationalization status especially of the MNCH/FP/TB components to ensure that a quality CPHC package of 12 services is being delivered at all supported HWCs. 2. Continue support to HWC to receive NQAS accreditations as well as develop a quality framework for improving quality of CPHC service delivery at supported HWCs. 3. Operationalize team building at HWC level and improve collaboration between CHOs, ANMs, ASHAs and MOs to deliver integrated MNCH/FP and TB diagnostic and treatment services, along with other services. 4. Assist supported districts to monitor and increase coverage and utilization of FP, MNCH and TB services integrated as part of the CPHC package, especially in remote and vulnerable communities. 5. Contribute to integrate and rationalize multiple reporting and digital tools and apps and improve reporting and monitoring of program outcomes and avoid duplication. 6. Support the GOI to develop an effective referral system for integrated continuum of care from primary to secondary and tertiary care. 7. Assist States to engage in designing and implementing targeted community mobilization activities for increased demand and uptake of CPHC services by specific minorities or marginalized populations not been reached yet. 8. Assist States to identify and address infrastructure and other barriers to access HWCs and achieve universal coverage. Partnerships 1. Explore for-profit private-sector partnerships to support specific components under the CPHC agenda in the states and develop a report with recommendations of partnerships for the GOI to expand CPHC access and coverage. Page | 39 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov 2. Explore and make recommendations for sustainable PPP models especially in the innovations, supply chain areas and ambulance/transport. 3. Explore partnerships with Christian groups and other religious organizations especially in the northeast regions. 4. Create academic partnerships and facilitate learning exchange forums with Christian groups and other religious organizations to ensure widespread dissemination of NISHTHA’s lessons and best practices. 5. Support states to strengthen relationships with community organizations to drive the community mobilization agenda for CPHC. Recommendations for USAID 1. Encourage NISHTHA to work with GOI to access HMIS data that will enable them to track and report progress on HSS outcome indicators. If that is not possible, encourage NISHTHA to explore and implement other options such as conducting regular LQAS surveys, establishing sentinel sites, and gathering additional data during facility supervision visits, which NISHTHA already performs. Examples of progress indicators that would be desirable to monitor to see growth and progress of the activity’s three main objectives every quarter are: a) # and % of CHOs that meet competency score in supported districts, b) # and % of CHOs in supported districts that are satisfied with their work, c) # and % of HWCs that meet selected CPHC targets, especially MNCH/FP, d) # and % of HWCs that deliver quality MNCH/FP and TB services, such as MNCH/FP/TB coverage indicators e) # and % of HWCs that have implemented a community engagement program and increase footfall f) # and % of HWCs that have implemented an effective referral and counter-referral system with secondary and tertiary level facilities g) # and % of districts that meet selected CPHC coverage targets. h) # of % of supported districts and facilities that implement NISHTHA’s innovations 2. Continue assisting the GOI’s CPHC agenda and development of a number of remaining CPHC components until supported states are sustainable and do not require assistance. This may include: a) Community engagement and participation interventions to increase HWC utilization and coverage b) Sustainable linkages of CPHC with secondary and tertiary levels to sustain continuum of care c) Sustainable CPHC financing strategy integrating health insurance with national and state funding to reduce out-of-pocket expenditures that are reported to be still high 3. USAID/India should continue its HSS agenda in alignment with USAID’s revised HSS vision in future activities. 4. The endline evaluation should include contribution analysis and measurable outcome harvesting to demonstrate the effectiveness of the pathways between NISHTHA’s interventions and desired health outcomes. Page | 40 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov ANNEX 1: EVALUATION TEAM (ET) The evaluation team is comprised of five team members, as follows: Dr Elvira Beracochea – Team Leader (TL). Dr. Beracochea (MD, MPH) is an experienced primary health care evaluation expert. She has over 20 years of evaluation experience, assessing health systems performance and the effectiveness of aid programs and projects. She is the Editor and co-author of “Improving Aid Effectiveness in Global Health” (Springer 2015). The primary responsibilities of the TL are to lead the ET in activity planning, team coordination meetings, developing evaluation methodology, desk reviews, supervision of data collection, review work products, and delivering presentations to USAID and other stakeholders, drafting and finalizing all deliverables. The TL will manage assignments of the ET within the level of effort (LOE) assigned to each team member and within the timeline outlined in the deliverables table. Chesta Sharma – Deputy Team Leader (DTL) & Evaluation Specialist. Dr. Sharma (PhD, MHA) is a Health Systems professional with core expertise in Health Systems Strengthening, Public Health, Primary Care, Stakeholder Analysis, Digital Health, Innovations, RMNCH+A, Health Human Resources, Quality of Care & Health Systems Research. Dr. Sharma will assist the TL in managing the ET in activity planning, team coordination meetings, developing data collection tools, contributing document summaries to the desk review, supervision of field data collection by Evaluation Specialists, designing presentations to USAID and other stakeholders, supervising drafting of products by Evaluation Specialists, and drafting sections of the evaluation report. The Team also had the participation of two Evaluation Specialists, and a Data Analyst. ● Mr. Ashish Yadav, Evaluation Specialist. Mr. Ashish is an evaluation specialist with 19 years of experience in the domain of technical assistance, strategy and project design, research, large program evaluation, M&E and capacity building. Ashish has led project/assignments for National Ministries, State Government, bi/multilateral agencies (USAID, WHO, UNDP, UNICEF, ILO), Development Banks (Asian Development Bank, KfW and The World Bank) and International Non-Profits. He has 12+ years of experience managing Public Health programs - RMNCH+A, Nutrition, Immunization, Health Systems, Innovations, Primary Care, Community Engagement, SBCC, Capacity Building ● Mr. Manish Kumar, Evaluation Specialist. Mr. Kumar is a Public Health professional with a MBA in Health Management with 17 years of program management experience implementing large public health programs, stakeholder management, supply chain management, health care financing, developing Monitoring, Evaluation (M&E) and learning systems, managing and conducting studies, operation research, impact evaluation, and surveys. ● Dr. Aman Borkar. Data Analyst. Dr. Borkar (PhD) has over 13 years of work experience in teaching & corporate research. He is currently working as an Assistant Professor at Delhi Skill and Entrepreneurship University. He is an expert in Operations Management and Research Methodology. One logistician, Ms. Ankita Babbar and one transcriptionist, Ms. Neha Sharma supported the ET in relevant activities. Page | 41 NISHTHA – Midterm Performance Evaluation –Final Report USAID.gov ANNEX 2. EVALUATION DESIGN MATRIX Table 1: Evaluation Design Matrix EQs Information Required and Data Source (s) Data Collection Methods Data Analysis Method 1. To what extent has NISHTHA’s technical assistance supported the GOI to achieve the overall objective of increasing access to quality comprehensive health care services at primary level? ● Stakeholder interviews ● Progress reports and M&E data ● Documents describing the theory of change and the process for operationalizing HWC upgrading and establishing effective learning labs ● Perspective of each stakeholder ● M&E data ● HWC facility interviews and observations ● CHO survey ● Comparison of achievement to targets ● Comparison to access at the start of the upgrading and after fully functional HWC is established ● Degree of satisfaction reported by users 2. Based on the available evidence, what is the progress towards the establishment of learning labs, upgradation and operationalization of health facilities and access to quality comprehensive health care services at primary level? ● Data extraction sheet by EQ ● Activity outputs and outcomes ● KIIs ● HWC statistics and performance indicators ● Document review ● KIIs ● M&E data ● Site visits ● Comparison of progress to targets ● Perspective by stakeholders ● Analysis of activity M&E indicators and system ● Assess use of the M&E data for programmatic decisions ● Assess monitoring of QI and comprehensiveness 3. Were the intervention activities and approaches implemented appropriately to address gender differences/gaps? What evidence exists to substantiate the reduction of gender gaps? ● Data collection by gender ● Views of stakeholder on gender gaps ● KIIs ● CHO interviews and survey ● Assess appropriateness of each intervention based on outcome and views of various stakeholders ● Descriptive statistics of Exit Interviews 4. What are the accomplishments and challenges? How can the implementation be adjusted to address those challenges and speed up the interventions that have lagged behind (if any) ● Activity outcomes to date ● IP’s and stakeholders’ challenges ● Document review ● KIIs ● CHO survey ● List challenges identified in the document review ● Qualitative analysis by stakeholder group 5. What are the key learnings out of the project implementation so far? ● Document lessons learned by NISHTHA and by ET ● Document Review ● KIIs ● Triangulation of findings in relation to activity’s objectives and outcomes 6. What are the perceptions of the nearby community that is served by the HWC supported by NISHTHA? ● Community interviews ● Interviews ● Quantitative data analysis ● Triangulation of findings with systemic findings 42 | NISHTHA - Final Performance Evaluation USAID.gov ANNEX 3: LIST OF DOCUMENTS REVIEWED 1. AB-HWC Assessment in 18 States, March 2022 2. AB-HWC GOI Documents 3. Annual & Quarterly Reports 4. Annual Work Plans 5. CHO Training Material, Process Documents, and Guidance Booklets 6. Cooperative Agreement 7. Documents on Financial Data 8. Documents on Learning Labs 9. Documents on Partnerships 10. Documents on Wellness Activities, & Toolkits 11. Government MoUs, Directives, Notices, Letters, and Guidelines 12. HWC Operational Guidelines, Process, Brochures, Guidance Notes & Tools 13. IEC Material, Community Mobilization Material, Branding, Campaigns 14. Independent Studies 15. MEL Plan 16. NISHTHA Performance Indicators 17. NISHTHA Stakeholders List 18. Organogram of the NISHTHA team and their job descriptions 19. Publications 20. Ranking of HWCs and the scoring criteria 21. Theory of Change 43 | NISHTHA - Final Performance Evaluation USAID.gov ANNEX 4. LIST OF KEY INFORMANTS Table 2: Overall Distribution of KIIs Details on KIIs Number NISHTHA (Delhi & State Teams) 12 USAID 3 Ministry of Health & Family Welfare (MoHFW) 4 NISHTHA Learning Labs (Field Interviews) 8 NISHTHA Learning Labs (Partner Interviews) 4 State Stakeholders - Assam 10 State Stakeholders - Chhattisgarh 30 State Stakeholders – Madhya Pradesh 25 State Stakeholders - Meghalaya 5 State Stakeholders - Nagaland 16 Total 117 State KIIs (86) covered a total of 91 stakeholders across different levels of administration, i.e., State, District, Block, Facilities & Health Committees (refer graph….) Figure 3: Distribution of State Level Key Informants 44 | NISHTHA - Final Performance Evaluation USAID.gov ANNEX 5: SAMPLING A mix of Purposive, Random and Convenience sampling method was applied to arrive at the final SC￾HWC, PHC-HWC, and UPHC-HWC sample size and sites (refer table 2 and table 3). A detailed list of visited centers is available in annex 5. Table 3: Summary of HWC Sample Methodology Steps Details Type of Sampling 1 A representative number of sample sites, i.e., the total number and by type of facilities was chosen. ● For smaller states like Meghalaya and Nagaland; and states with limited intervention sites like Assam, a total sample of 10 or below was chosen. ● For bigger states like Chhattisgarh and Madhya Pradesh, a total sample of 17 and 15 was chosen respectively. Purposive & Convenience 2 All HWCs have been assigned ranks by NISHTHA team based on a scientific criterion. Based on the ranking scores, ET subdivided all HWCs (in the three categories of health facilities, i.e., SCs, PHCs, and UPHCs) into four sub-categories ● Category 1: Score of 0-25 ● Category 2: Score of 25-50 ● Category 3: Score of 50-75 ● Category 4: Score of 75-100 Based on the representation of the HWCs in the above scoring criteria, the total sample was further subdivided to represent differential performance of HWCs (refer annex 4). Post subdivision, HWCs were randomly selected. Purposive & Random 3 All randomly selected HWCs were then cross checked for two additional criteria. ● Representation of top performing HWCs to understand key levers that can help leapfrog other HWCs. ● Choosing sites that may be convenient to reach out for the ET. This was done specifically due to limited time for the ET, and anticipated COVID restrictions in the states. Purposive & Convenience HWC SAMPLING METHODOLOGY, SIZE AND SITES As on April 30, 2022, in the sample states, a total of 17,894 HWCs are operational. The upgraded and operational HWCs are across three categories of health facilities, i.e., SCs, PHCs and UPHCs. Table 4: List of Operational HWCs in Sample States States SCs PHCs UPHCs Total Assam 2493 912 53 3458 Chhattisgarh 3737 745 52 4534 Madhya Pradesh 7886 1155 162 9203 Meghalaya 257 95 17 369 Nagaland 248 75 7 330 Total 14621 2982 291 17894 Table 5: List of Operational HWCs in Direct Intervention Districts in Sample States In Direct Intervention Districts SCs PHCs UPHCs Total Assam 166 101 4 271 Chhattisgarh 1142 315 17 1474 45 | NISHTHA - Final Performance Evaluation USAID.gov In Direct Intervention Districts SCs PHCs UPHCs Total Madhya Pradesh 435 98 8 541 Meghalaya 164 79 16 259 Nagaland 162 49 7 218 Total 2069 642 52 2763 Table 6: HWC Sample Size and Distribution States SCs PHCs UPHCs Total Assam 7 2 1 10 Chhattisgarh 10 6 1 17 Madhya Pradesh 11 4 1 16 Meghalaya 5 3 1 9 Nagaland 5 2 1 8 Total 38 17 5 60 SURVEY OF COMMUNITY MEMBERS SAMPLING METHODOLOGY, SIZE AND SITES A mix of Purposive and Random sampling method was applied to arrive at the list of 15 HWCs in the four sample states, i.e., Assam (2 villages), Chhattisgarh (4 villages), Madhya Pradesh (7 villages) and Meghalaya (2 villages). A total of 154 respondents across different demographic categories were interviewed across 15 villages from four NISHTHA intervention states. The nearest village (distance or time to reach) to the HWC was selected for interviewing the community. A minimum of 10 respondents (one per household) in each village were to be covered. A detailed field movement plan is available in annex 6. Table 7: Respondents per Village Sample S. No. Respondents Representative Sample 1 Pregnant Woman/ Woman with children below 2 years of age 2 (1 each) 2 MWRA 2 3 Elderly (above 65 years of age) 2 (one Woman, one Man) 4 Adult Men (below 65 years) 4 Total per village 10 Steps for Data Collection: 1. The total number of households (including hamlets) were divided by the total sample size, i.e., 10 to arrive at the factor ‘n’. 2. The field movement began with the main cluster of households in the village. 3. Household Zero – Village of the Panchayat Head/ Village Head/ Village Committee Chairman/ Anganwadi Centre/ Religious Centre. 4. The selection of households began with movement towards the right covering each lane and each household. 5. Every nth household (beginning from Household Zero) was picked as the sample household. Only one target respondent was picked from each sample household. If target respondent was not available in the sample household, nth + 1 household was picked and the next nth household was picked after that. 6. If hamlets were present, the closest hamlet was picked first, after the main cluster of households were covered. 46 | NISHTHA - Final Performance Evaluation USAID.gov 7. If the total sample did not get covered in round 1 across the village, respondents were picked purposively in consultation with Village Committee Chairman/ Anganwadi Worker/ ASHA. 8. The respondents were picked in the order given in table 4. List of Sample Villages Table 8: List of Sample Villages No. State District SHC – HWC Village Household Population 1 Assam Nagaon Lahkar Adarsha 188 703 2 Assam Nagaon Borholla Borholla 207 934 3 Chhattisgarh Bilaspur Majhwani Majhwani 240 1227 4 Chhattisgarh Janjgir Champa Tiwaripara Tiwaripara 536 3394 5 Chhattisgarh Janjgir Champa Deori Deori 675 2340 6 Chhattisgarh Janjgir Champa Birra B Birra B 430 3814 7 Madhya Pradesh Rajgarh Lashkarpur Lashkarpur 200 NA 8 Madhya Pradesh Rajgarh Khuri Khuri 240 NA 9 Madhya Pradesh Barwani Panya Punarnivas 100 1282 10 Madhya Pradesh Barwani Dudhkheda Bormala 64 11 Madhya Pradesh Barwani Sustikheda Patel Faliya 90 775 47 | NISHTHA - Final Performance Evaluation USAID.gov No. State District SHC – HWC Village Household Population 12 Madhya Pradesh Barwani Moyada Moyada 170 13 Madhya Pradesh Barwani Khadki Khadki (including hamlets) 209 NA 14 Meghalaya West Jaintia Hills Darrang Darrang 123 1498 15 Meghalaya West Jaintia Hills Sanaro Sanaro (including hamlets) 347 2085 Brief Profile of each Sample Village A. Assam 1. Adarsha (Lahkar HWC) – Adarsha is spread across a radius of 1.5 kms on the eastern side of Lahkar HWC. As a result, the HWC is easily accessible by most households on foot. A small market on the western side of Adarsha, also provides access to private pharmacies. Dhing FRU (Block PHC) is located at a distance of 3 kms from the village. Despite villagers being aware of the HWC, most of them prefer accessing services at the block PHC, it being a higher center. Villagers feel that HWC should be upgraded to the level of FRU in order to be beneficial. 2. Borholla (Borholla HWC) – Borholla is spread across a radius of 5 kms on the northern & eastern side of the HWC. Though it is not easily accessible by foot by all households, it being located in a remote location (in a tea estate), the center becomes a preferred choice for the villagers. Irregular attendance of CHO at the HWC, lack of inpatient facilities, and lack of emergency transport, were some of the immediate concerns of the villagers. B. Chhattisgarh 3. Majhwani (Majhwani HWC) - Majhwani village is situated 28 km away from sub-district headquarter Kota and 35 km away from district headquarter Bilaspur. Most of the respondents go to the HWC to avail medical services including ANC services, children immunization and covid vaccination. Few of them also go to Ratanpur CHC, 17 km away from the village. 4. Tiwaripara (Tiwaripara HWC) - Most of the people from school chowk area in the village do not go to HWC as CHC is located nearby and they get better and complete medical services at the CHC. Non-availability of medicines and irregularity of the staff at the HWC were some of the issues raised by the villagers. Almost all from Bhaderiya Para area do not know about the HWC which is just 1 km from their locality. Most of them go to private doctors for medical services. 5. Deori (Deori HWC) - Deori is a small village/hamlet located 7 km towards west from district headquarters Janjgir. Most villagers’ complaint about the irregularity of the staff at the HWC. Few of them expressed unhappiness as most of the time staff come to the facility only on RI days (Tuesday 48 | NISHTHA - Final Performance Evaluation USAID.gov and Saturday). The HWC was locked on the day of the visit. Most villagers prefer to go to PHC Saragaon which is located 1-2 km from the village. Most respondents are not aware about the various healthcare services available at the HWC. 6. Birra (B) (Birra B HWC) - Birra is located 28 KM towards South from District headquarters Janjgir. Specific section of the community goes to HWC for ANC services, children immunization and covid vaccination. Others go to PHC Birra which is located 2-2.5 km from their locality. C. Madhya Pradesh 7. Lashkarpur (Lashkarpur HWC) - HWC is adjacent to the village (100-150 meters). A mix population of OBC (Nagar) and SC community reside in the village. Few private doctors (quacks) reside in the village and a large sector of community prefer going to them due to 24X7 availability of services. The OBC community own land and are engaged in agriculture/ dairy activities and mostly prefer going to private doctors. There is lack of awareness of the availability of CHO and healthcare services availability at the HWC. It’s mostly considered for MCH related services. 8. Khuri (Khuri HWC) - Khuri village is with-in 500 meters of the HWC. The village is predominantly inhabited by OBC community, who own land and their main source of livelihood is agriculture. A very small population of SC community in the village are agricultural laborers. Community is aware of the HWC in the village but not about the services. For minor ailments community prefer going to private doctors/ clinics in the nearby market (1-1.5 kms), for anything major, people prefer going to either private or government hospitals a Beoara (CHC), Rajgarh (District Hospital) or to Bhopal (mostly to private hospital). There is a perception that government facilities do not provide adequate services in terms of availability of doctor, medicines and diagnostic tests. 9. Punarnivas (Panya HWC) - Punarnivas hamlet was established to provide habitation to communities living in low-lying areas of Narmada River. The village is inhabited by Bhilala tribal community who own medium to small piece of land. It is a general perception that a good doctor or a health facility is that which provide injection and IV fluid for any ailment. People prefer going to private doctors at nearby market at Manwada as they are available 24x7. For any major ailment the preference is either CHC at Anjar or DH/ Private Hospital at Barwani. There is a general lack of trust for government facilities. Community is not aware of the services available at HWC. 10. Bormala (Dudhkheda HWC) - Bormala is the closest hamlet to the HWC (200- 300 meters) primarily belonging to Barela tribe. These are mostly landless laborer who work at brick kilns and also as daily wage casual labor. Many time the community migrate to another state or district for 4-5 months to work in brick kilns. Community is aware of HWC in their vicinity but not aware of the range of available health services. They mostly access the health facility for MCH services. For other ailments, community prefers to go to private clinic of the CHC at Balwadi. 11. Patel Faliya (Sustikheda HWC) - Patel Faliya is spread out in large areas with clusters of small number of households at a location. The community is consisting of ST (Barela Tribe) and SC. Agricultural land holding is very small and most of the families migrate out for 4-5 months in a year as contract agricultural laborers to Gujarat. As a result, awareness about HWC services is low. The nearest PHC is at Manimata which is about 2-3 kilometers from the hamlet and is preferred. The CHO is not from the local area and is not well versed with the local dialect. Also, there is general perception that injections and IV fluids provide immediate relief which is not available at HWC. 12. Moyada (Moyada HWC) - The HWC is within the village and the village is inhabited by Bhilal tribes. The HWC opens mostly twice a week as CHO does not come regularly. People are aware of 49 | NISHTHA - Final Performance Evaluation USAID.gov HWC in the village but do not access its facilities as for most days it is closed. The preference is going to PHC at Jalwania (7 kms), or to private clinic or CHC at Salkhera (5 kms). 13. Khadki (Khadki HWC) - The HWC is about 500-600 meters away from the village. The village community is a mix of OBCs (Patil and Yadav) and SCs. Dominated by OBCs, who are land owners and have source of livelihood as agriculture, contractors, commercial vehicles, etc.; while SCs are mostly landless, and are contract laborers. Rajpur, the block headquarter is in close vicinity of the village (3 kms) where people have ease of access to CHC, private clinics and private hospitals. For minor ailments HWC or other government facility is preferred, that too specially for children. For other matters there is a clear preference of private doctor/ private clinic run by government doctor. Community is generally not aware of the facilities being available at HWC beyond MCH services. Women prefer going to HWC as it has ease of access and they are not dependent on family members. D. Meghalaya 14. Darrang (Darrang HWC) – Darrang is spread on a steep hill (accessible by foot) across a radius of 3 kms on the northern side of the HWC. The village has a very remote location and is one of the last villages on the route. The nearest PHC is about 5 kms away. Due to its remote location, the center becomes a preferred choice for the villagers. Non-availability of CHO in the evening (preferred time for villagers), lack of in-patient facilities, and lack of emergency transport, were some of the immediate concerns of the villagers. 15. Sanaro (Sanaro HWC) – HWC is located at the heart of the four hamlets which are spread across a radius of 5-7 kms around the HWC. Most of the village is non-accessible by vehicle and is spread on a hillock. The CHO belongs to a different tribe and does not understand the local dialect. Most people prefer going to nearby PHC, private doctors, quacks for services. Profile of Sample Respondents from Community Perception Survey: In the sample, 52% respondents were women while 48% respondents are men. Most of the respondents belong to the farming/ agricultural community. Forty two percent women are exclusively housewives. One-third were illiterate, while another one-third were educated up to primary level. Only 7% of respondents had either graduate or post graduate degrees. Majority (64%) of the respondents have a family size that ranges 4-8 members. 50 | NISHTHA - Final Performance Evaluation USAID.gov ANNEX 6: DATA COLLECTION TOOLS KIIS KII: USAID, IP, MOHFW, STATE STAKEHOLDERS KII Consent Form My name is _________________, and as you know, I am an independent consultant working for the India Claim project that evaluates projects for USAID/India. Thank you for making the time to talk with me today. USAID/INDIA has asked us to evaluate the progress of the NISHTHA project to identify opportunities for learning from its achievements and address any challenges. You were suggested as a key person to inform this assessment and we greatly appreciate your perspective, experiences and views on the successes, challenges, barriers and lessons learned from your experience. Our interview will take about one hour. Before we begin, I want to inform you that any information or examples we gather during this interview process will not be attributed to any specific person, or otherwise attributed to you. If we include quotations from our interview in the final assessment report, they will be attributed only to a stakeholder group (such as the MOH Partner, Healthcare provider, etc.). You are also free to not respond to any of our questions or to stop or pause the interview at any time. If you are comfortable, I would like to record this interview to ensure that I do not miss any important points when writing up my summary. Please know that anything you say during the interview will be kept confidential within the evaluation team, and that our interview notes and any recordings will be erased when the evaluation report is completed. Do I have your permission to record? Yes: ________ No: ________ (Do not ask this question to very senior government officials) Do I have your permission to begin? Yes: ________ No: ________ Before we begin, do you have any questions about this interview? Key Informant Interview Guide Interviewer: Interview Date: Start Time: End Time: Interviewee’s Name: First: Last: Current Position: Org: Contact email: Phone (Optional): State/ District: Facility: 51 | NISHTHA - Final Performance Evaluation USAID.gov Has the KI affirmed Informed Consent? Y____ N____ (Interviewer's initials) Respondent's Unique ID: ______________________ Questions / Topics Responses I. Background 1. What activities are you responsible for in ____________ (the stakeholder organization)? 2. How long have you worked in _________ (the organization)? 3. In your opinion, how has the NISHTHA project helped you in this facility/ district/ state? 4. How did COVID affect the CPHC work in the facility/ district/ state? 5. Beyond COVID, what have been the main challenges in the last two years? II. Core Questions for All 1. How has NISHTHA supported the facility/ district/ state in HWC & CPHC related work? Have they been responsive to the needs of the state? How well has NISHTHA helped solve problems and innovate? Has NISHTHA helped develop partnerships with other organizations? 2. How many facilities have been upgraded/ operationalized in the district/state? For the facility, ask has the operationalization been complete? Have you been involved in the establishment and implementation of learning labs? Has NISHTHA helped improve access to quality comprehensive health care services at primary level? Ask for NQAS and IPHS standards. If yes, how much has access improved? How much has quality improved? 3. What districts/facilities are performing better, and why? Which are the model HWCs in state/ district? Why are they performing well? 4. Has NISHTHA helped address gender differences/gaps? What specific interventions were used to reduce gender gaps? Did NISHTHA focus on vulnerable groups? If yes, which/ how? 5. Regarding challenges, how can the implementation of NISHTHA be adjusted to address those challenges and speed up the interventions that have lagged behind (if any)? 52 | NISHTHA - Final Performance Evaluation USAID.gov Questions / Topics Responses 6. What are the key learnings out of improving CPHC in India so far? (Focus on asking for NISHTHA’s contribution) – key contributions/ good practices 7. Going forward, which other programmatic interventions should NISHTHA focus on? 8. What needs do the HWCs have that are not met yet? What gaps are there for HWCs to address? 9. What opportunities does the COVID pandemic present? 10. Which HWC should we visit to learn how the upgrading process works? III. Questions tailored for different stakeholder groups: USAID, IP Team, and MOH (national, state, ______) 1. How do you access additional funds to upgrade HWCs s? What is NISHTHA’s role in supporting the GOI/ State use of funds for different purposes? Is there a specific process/ pathway that NISHTHA followed for unlocking state funds? Probe for untied fund/ PBI/ TBI 2. How does NISHTHA or the government stakeholders help leverage the unlocked money? 3. Has the state included any innovations in the PIPs and/ or the innovation summit? Will your state be able to sustain these innovations or require additional funds or support? How do you get these innovations into the PIPs and the summit? (Focus on NISHTHA innovations) 4. How has NISHTHA supported the government to achieve such progress in a short span and especially the trust of the state and local governments? 5. How was NISHTHA able to select and groom CHOs as part of the TA to the state? 6. What are some of the factors that enabled the NISHTHA team to leapfrog the different components? 7. How has your organization/ partner organizations been part of the upgrading process? Was this the best way to get involved? 8. What other opportunities do you see to accelerate the upgrading of HWCs? 9. What other opportunities do you see to increase access to quality comprehensive primary care? Other comments: Time at End of Interview: Interviewer’s observations and main comments: 53 | NISHTHA - Final Performance Evaluation USAID.gov KII: NISHTHA STATE TEAMS KII Consent Form My name is _________________, and as you know, I am an independent consultant working for the India Claim project that evaluates projects for USAID/India. Thank you for making the time to talk with me today. USAID/INDIA has asked us to evaluate the progress of the NISHTHA project to identify opportunities for learning from its achievements and address any challenges. You were suggested as a key person to inform this assessment and we greatly appreciate your perspective, experiences and views on the successes, challenges, barriers and lessons learned from your experience. Our interview will take about one hour. Before we begin, I want to inform you that any information or examples we gather during this interview process will not be attributed to any specific person, or otherwise attributed to you. If we include quotations from our interview in the final assessment report, they will be attributed only to a stakeholder group (such as the MOH Partner, Healthcare provider, etc.). You are also free to not respond to any of our questions or to stop or pause the interview at any time. If you are comfortable, I would like to record this interview to ensure that I do not miss any important points when writing up my summary. Please know that anything you say during the interview will be kept confidential within the evaluation team, and that our interview notes and any recordings will be erased when the evaluation report is completed. Do I have your permission to record? Yes: ________ No: ________ Do I have your permission to begin? Yes: ________ No: ________ Before we begin, do you have any questions about this interview? Discussion Guide 1. What has been the overall journey of the state from prior to MCSP Project to MCSP Project to NISHTHA? 2. Broad areas of discussion ● Technical assistance to the state for operationalization of HWCs (Input, Process, Outputs) – Discuss all areas of assistance, these could be programmatic like TB, MCH, etc. ● COVID Support ● Establishment of Learning Labs ● Key Partnerships – Government, Private, Inter-state ● Gender and vulnerable/ marginalized groups ● Any other areas of discussion that may be unique to the state – innovations, state level structures, AYUSH HWCs, etc. ● MLE Plans 54 | NISHTHA - Final Performance Evaluation USAID.gov KII: LEARNING LABS (FIELD INTERVIEWS) KII Consent Form My name is _________________, and as you know, I am an independent consultant working for the India Claim project that evaluates projects for USAID/India. Thank you for making the time to talk with me today. USAID/INDIA has asked us to evaluate the progress of the NISHTHA project to identify opportunities for learning from its achievements and address any challenges. You were suggested as a key person to inform this assessment and we greatly appreciate your perspective, experiences and views on the successes, challenges, barriers and lessons learned from your experience. Our interview will take about one hour. Before we begin, I want to inform you that any information or examples we gather during this interview process will not be attributed to any specific person, or otherwise attributed to you. If we include quotations from our interview in the final assessment report, they will be attributed only to a stakeholder group (such as the MOH Partner, Healthcare provider, etc.). You are also free to not respond to any of our questions or to stop or pause the interview at any time. If you are comfortable, I would like to record this interview to ensure that I do not miss any important points when writing up my summary. Please know that anything you say during the interview will be kept confidential within the evaluation team, and that our interview notes and any recordings will be erased when the evaluation report is completed. Do I have your permission to record? Yes: ________ No: ________ (Do not ask this question to very senior government officials) Do I have your permission to begin? Yes: ________ No: ________ Before we begin, do you have any questions about this interview? Key Informant Interview Guide Interviewer: Interview Date: Start Time: End Time: Interviewee’s Name: First: Last: Current Position: Org: Contact email: Phone (Optional): State/ District: Facility: Has the KI affirmed Informed Consent? Y____ N____ (Interviewer's initials) Respondent's Unique ID: ______________________ 55 | NISHTHA - Final Performance Evaluation USAID.gov Questions / Topics Responses I. Background 1. What activities are you responsible for in this learning lab (the stakeholder organization)? 2. When was the Learning lab initiated/rolled out? 3. How has the NISHTHA project helped in creating the learning labs? 4. Did COVID affect the Learning labs? 5. Beyond COVID, what have been the main challenges while planning and implementing the Learning labs? II. Core Questions for All 1. How does this Learning lab function? Can you elaborate on the issues that this Learning lab addresses in your state/ district? Who all were involved and their specific roles? 2. What is the current status of implementation? What is the intended coverage of the Learning lab? 3. How do you monitor the progress of the learning lab? Do you have any reporting system in place? Has the Learning lab helped improve access to quality comprehensive health care services at primary level? 4. Has the Learning lab helped address vulnerable and hard to reach populations and system gaps? What specific interventions were used to reduce these gaps? (system/services/situational/workforce/resources) 5. How can the implementation of Learning lab speed up the NISHTHA interventions that have lagged behind (if any)? 6. What are the key learnings out of learning lab? 7. What other opportunities do you see to accelerate the roll out? What is planned to sustain the strategies/ solutions/ interventions of the Leaning lab in future? 8. What other opportunities do you see to increase access to quality comprehensive primary care through these learning labs? Other comments: Time at End of Interview: Interviewer’s observations and main comments: 56 | NISHTHA - Final Performance Evaluation USAID.gov KII: LEARNING LABS (PARTNER INTERVIEWS) KII Consent Form My name is _________________, and as you know, I am an independent consultant working for the India Claim project that evaluates projects for USAID/India. Thank you for making the time to talk with me today. USAID/INDIA has asked us to evaluate the progress of the NISHTHA project to identify opportunities for learning from its achievements and address any challenges. You were suggested as a key person to inform this assessment and we greatly appreciate your perspective, experiences and views on the successes, challenges, barriers and lessons learned from your experience. Our interview will take about one hour. Before we begin, I want to inform you that any information or examples we gather during this interview process will not be attributed to any specific person, or otherwise attributed to you. If we include quotations from our interview in the final assessment report, they will be attributed only to a stakeholder group (such as the MOH Partner, Healthcare provider, etc.). You are also free to not respond to any of our questions or to stop or pause the interview at any time. If you are comfortable, I would like to record this interview to ensure that I do not miss any important points when writing up my summary. Please know that anything you say during the interview will be kept confidential within the evaluation team, and that our interview notes and any recordings will be erased when the evaluation report is completed. Do I have your permission to record? Yes: ________ No: ________ (Do not ask this question to very senior government officials) Do I have your permission to begin? Yes: ________ No: ________ Before we begin, do you have any questions about this interview? Key Informant Interview Guide Interviewer: Interview Date: Start Time: End Time: Interviewee’s Name: First: Last: Current Position: Org: Contact email: Phone (Optional): State/ District: Facility: Has the KI affirmed Informed Consent? Y____ N____ (Interviewer's initials) Respondent's Unique ID: ______________________ 57 | NISHTHA - Final Performance Evaluation USAID.gov Discussion Guide 1. Background and establishment of Learning Labs 2. What activities are you responsible for in this learning lab? 3. How did you work with the NISHTHA team? Did you have a liaison or counterpart? What was their role? 4. How has the NISHTHA project helped in creating the learning labs? 5. How does this Learning lab function? 6. Key Learnings (best practices) 7. What results have you achieved so far? 8. What were the challenges? 9. How did the Learning Lab improve the HWC and CPHC? 10. Does the Learning Lab focus on gender interventions? 11. Plan to sustain the strategies/solutions/interventions of the Leaning lab in future 12. Knowing what you know now, what would you do differently? KII: CHO LEADER KII Consent Form My name is _________________, and as you know, I am an independent consultant working for the India Claim project that evaluates projects for USAID/India. Thank you for making the time to talk with me today. USAID/INDIA has asked us to evaluate the progress of the NISHTHA project to identify opportunities for learning from its achievements and address any challenges. You were suggested as a key person to inform this assessment and we greatly appreciate your perspective, experiences and views on the successes, challenges, barriers and lessons learned from your experience. Our interview will take about one hour. Before we begin, I want to inform you that any information or examples we gather during this interview process will not be attributed to any specific person, or otherwise attributed to you. If we include quotations from our interview in the final assessment report, they will be attributed only to a stakeholder group (such as the MOH Partner, Healthcare provider, etc.). You are also free to not respond to any of our questions or to stop or pause the interview at any time. If you are comfortable, I would like to record this interview to ensure that I do not miss any important points when writing up my summary. Please know that anything you say during the interview will be kept confidential within the evaluation team, and that our interview notes and any recordings will be erased when the evaluation report is completed. Do I have your permission to record? Yes: ________ No: ________ (Do not ask this question to very senior government officials) Do I have your permission to begin? Yes: ________ No: ________ Before we begin, do you have any questions about this interview? 58 | NISHTHA - Final Performance Evaluation USAID.gov Key Informant Interview Guide Interviewer: Interview Date: Start Time: End Time: Interviewee’s Name: First: Last: Current Position: Org: Contact email: Phone (Optional): State/ District: Facility: Has the KI affirmed Informed Consent? Y____ N____ (Interviewer's initials) Respondent's Unique ID: ______________________ Discussion Guide 1. Background on selection process, duration of the course, course content, mode of teaching, etc. 2. Utility of the course and its benefits in the daily routine. What more can be added to help you in your daily work routine? 3. Any challenges you face while accessing the course, understanding the content, etc. Explore additional areas like 4. How has the NISHTHA program helped in your initial training and now with the Leadership training? 5. Do you get continuous supervision and support by the NISHTHA team? 6. Overall service delivery (use key thematic areas from HWC checklist) 7. What are some of the key achievements in your work as a CHO? 8. What are some of the key challenges that you face in your work as a CHO? 9. Do you need support in additional areas? If yes, what are those, and what kind of support do you need? 10. Anything else that you would want to share HWC CHECKLIST AND CHO INTERVIEW A. General Information 1. Name of HWC: 6. MOIC Name: 2. Catchment Population under HWC: 7. Name of CHO Interviewed 3. Block: 8. Phone No. of CHO: 4. District: 9. Date of visit: 5. State: 10. NIN of the facility: 11. Type of Facility: PHC/UPHC/SHC 59 | NISHTHA - Final Performance Evaluation USAID.gov B.1. Status of Human Resource (Interview and fill) Provider No. available 1. CHO (F )/ (M) 2. MPW (F) / (M) 3 ASHA (F) / (M) 4. ANM(F) / (M) For CHO – Which of these trainings have you received in last three years Mark those mentioned AYUSH □ Yes □ No i-Learn □ Yes □ No CHO Leadership Program □ Yes □ No Induction Training □ Yes □ No Collaborative Framework for TB □ Yes □ No Family Care for TB □ Yes □ No TB Champions □ Yes □ No TB for CHOs □ Yes □ No JAS □ Yes □ No Diarrhea and Pneumonia □ Yes □ No MCH □ Yes □ No DVDMS □ Yes □ No NQAS □ Yes □ No e-Sanjeevani □ Yes □ No Family Planning □ Yes □ No Expanded Services □ Yes □ No Saksham □ Yes □ No WASH □ Yes □ No Which of the following expanded services are available at this HWC? Mark those mentioned Basic ophthalmic care services (eye care) □ Yes □ No Basic ear, nose, throat (ENT) care services □ Yes □ No Basic oral health care □ Yes □ No Basic geriatric health care services □ Yes □ No Screening & management of mental health ailments □ Yes □ No Emergency Medical Services □ Yes □ No Screening for Cancer, hypertension and diabetes □ Yes □ No CHO’s Training T.1 Do you think the basic training you received prepared you for the work you do at this HWC? □ Yes □ No □ Partially T.2 What more could have been added to your trainings to help you work better? T.3 Is the supervision you receive helpful? If yes, how? If not, what else do you need? □ Yes □ No T.4 What other support do you need to help you deliver quality health care services? T.5 What challenges do you face while delivering HWC services the people in this district/Block have? CHO’s equipment: What equipment do you use at work? C.1 Desktop/ Laptop / Tablet for CHO Not Available □ Available but not functional □ Available and functional □ C.2 Phone □ Yes □ No C.3 Stethoscope □ Yes □ No C.4 Adult scale □ Yes □ No C.5 Infant Scale □ Yes □ No 60 | NISHTHA - Final Performance Evaluation USAID.gov CHO’s equipment: What equipment do you use at work? C. 6 Other: C.7 What tests can you do at this HWC? Mark all that are mentioned. Probe for the rest. 1 Hemoglobin – Digital hemoglobinometer / Sahli’s haemoglobinometer □ Yes □ No 2 Human chorionic gonadotropin (HCG) (Urine test for pregnancy) – Rapid card test (dipstick) □ Yes □ No 3 Urine test for PH, specific gravity, leucocyte esterase glucose, bilirubin, urobilinogen, ketone, haemoglobin, protein, nitrite – Multiparameter Urine Strip (dipstick) □ Yes □ No 4 Blood Sugar (Glucometer) □ Yes □ No 5 Malaria (Rapid test) □ Yes □ No □ NA 6 HIV (Antibodies to HIV 1&2) – Rapid card test □ Yes □ No 7 Visual Inspection Acetic Acid (VIA) - Manual □ Yes □ No 8 HbsAg test for Hepatitis B – Rapid card test □ Yes □ No 9 Syphilis – Rapid test kit □ Yes □ No 10 Sample collection for Sputum for Tuberculosis □ Yes □ No 11 Other: D. Physical Infrastructure the CHO is able to use D.1 OPD room □ Available □ Not Available D.2 Availability of furniture: Table for CHO Chairs Almirah/Rack Examination table □ Available □ Not Available □ Available □ Not Available □ Available □ Not Available □ Available □ Not Available Available □ Not Available D.3 Laboratory/ diagnostics services □ Available □ Not Available D.4 Pharmacy/ medicine dispensation □ Available □ Not Available D.5 Space/ room identified for Wellness activities including Yoga sessions □ Available □ Not Available D.6 Waiting area with sitting arrangements for patients/attendants □ Available □ Not Available D.7 Separate functional toilets for males and females □ Available □ Not Available D.8 Whether residential facilities available for MPW / CHO □ Available □ Not Available D.9 Electricity □ Available □ Not Available D.1 0 Power back up □ Available □ Not Available D.1 1 Safe drinking Water supply □ Available □ Not Available D.1 2 Functional Handwashing corner (designated) with availability of running water and soap /cleaning hand hygiene agents □ Available □ Not Available D.1 3 Provision of BMW management □ Available □ Not Available D.1 4 Color coded waste bins □ Available □ Not Available D.1 5 IEC/Poster on BMW displayed at the facility. □ Available □ Not Available D.1 6 Bio-medical waste disposal mechanism (Hub-cutters, Sharps pit and burial pits or biomedical waste collection mechanism for disposal at higher facility) □ Available □ Not Available D.1 7 External branding as per guidelines □ Available □ Not Available 61 | NISHTHA - Final Performance Evaluation USAID.gov E. Drugs (Check for availability) E.1 Are you lacking any medicines at this time? If yes, which one? □ Yes □ No E.2 Does this HWC have indenting cycle that is followed to request medicines? (may be Fortnightly, Monthly, Quarterly or as required) □ Yes □ No E.6 If yes, do you participate in the preparation of the request? □ Yes □ No E.7 Do you know if the DVDMS or any other software is being used for medicine stock management □ Yes □ No F. Telemedicine services F.1 Is the Internet connectivity, OK? □ Yes □ No F.2 Do you have arrangements for teleconsultation made – linked with hub site- PHC/CHCs/DH or Medical College □ Yes □ No F.3 Is the teleconsultation schedule has been prepared and displayed □ Yes □ No F.4 Whether teleconsultation is being provided? □ Yes □ No If No, Skip to section G F.5 Total consultations in the last month F.6 Platform utilized for teleconsultation 1. e-Sanjeevani OPD 2. e-Sanjeevani.in (AB-HWC) 3. State specific app. 4. Any other (Specify) G. Are you comfortable providing Wellness services? G.1 Staff/Resource person identified for wellness activities (Yoga/Zumba/Aerobics etc.) □ Yes □ No G.2 Schedule for wellness activities is prepared and displayed at the facilities □ Yes □ No G.3 Health days celebrated in previous month as per wellness activity calendar □ Yes □ No G.4 How many wellness sessions were conducted in previous month? (Validate from wellness reporting through HWC app/portal) I. Jan Arogya Samiti (JAS)- Ask and Fill I.1 Whether Jan Arogya Samiti constituted? □ Yes □ No I.2 If yes, whether monthly meetings for JAS held? (Check for MoM for previous month) □ Yes □ No I.3 Whether untied fund received? □ Yes □ No I.4 Whether signatories for untied is as per JAS guidelines? □ Yes □ No I.5 Is monthly progress report including fund utilization being filled under JAS? □ Yes □ No I.6 How have you used the un-tied fund? 62 | NISHTHA - Final Performance Evaluation USAID.gov J. AB-HWC Reporting J.1 Whether HWC app installed? □ Yes □ No J.2 Whether regular daily reporting is being done? (Check for last 20 days reporting in HWC app/portal in previous month) □ Yes □ No J.3 Whether regular monthly service delivery reporting being done? (check entries for last month if the visit is conducted after 15th day and check entries for the month before the last month if the visit is conducted before 15th day of the month.) □ Yes □ No J.4 Use of other reporting applications: 1. NCD app 2. IHIP 3. FPLMIS 4. NIKSHAY 5. ANMOL □ Yes □ No □ Yes □ No □ Yes □ No □ Yes □ No □ Yes □ No K. Incentives K.1 Has CHO received Performance Based Incentive (PBI) till month before the previous? □ Yes □ No □ NA K.2 If no, till which month PBI has been given? K.3 Has Team Based Incentive (TBI) been given to ANM and ASHA till last cycle? □ Yes □ No □ NA K.4 If no, till which month TBI has been given? L. Service delivery statistics (data of previous quarter) Is the CHO able to report: L.1 No. of PW registered for ANC □ Yes □ No L.2 No. of PW received 4 or more ANC check-ups □ Yes □ No L.3 Total number of institutional deliveries □ Yes □ No L.4 Total no. of High-Risk Pregnancies identified □ Yes □ No L.5 Total no. of children under 24 months of age who received the first dose of the Pentavalent vaccine □ Yes □ No L.6 Total no. of children under 24 months of age who received the third dose of the Pentavalent vaccine □ Yes □ No L.7 No. of Childhood Diseases – Diarrhea □ Yes □ No L.8 No. of Childhood Diseases – Acute Respiratory identified L.10 No. of FP commodities distributed (disaggregated by each commodity) (i) Condoms (ii) OCP (iii) Cent chroman (iv) Injectables L.11 Total number of outpatient department visits L.12 Total number of TB Patients under treatment L.13 Number of patients suffering from NCDs (any) undergoing treatment L.14 Number of referrals to Higher level facility L.15 Total number of TB Patients under treatment 63 | NISHTHA - Final Performance Evaluation USAID.gov ONLINE CHO SURVEY Online CHO survey The NISHTHA project has been supporting the Government of India to implement the Ayushman Bharat program, focusing on the upgrading of Health and Wellness Centers (HWCs) for the last three years. USAID funds NISHTHA and would like to evaluate how well NISHTHA's support has worked so far; and find ways to keep improving the project. The purpose of this survey is to understand how well NISHTHA has achieved as per intended plan and to determine what works well and what could be improved in a program or initiative. As a CHO, you have been responsible for delivering services at the HWC and we would like to know your experience and suggestions for the future. This survey is anonymous, and your name would remain confidential with the evaluation team. The survey takes less than 15 minutes, and everyone's participation is very important. We appreciate your taking the time to share your experience with us. If you have any questions, please feel free to contact the Evaluation Team Leader at eberacochea@panagoraindia.net 1. Name of State o Arunachal Pradesh o Assam o Chhattisgarh o Jharkhand o Madhya Pradesh o Manipur o Meghalaya o Mizoram o Nagaland o Odisha o Sikkim o Tripura 2. Age of respondent _______________________________ 3. Gender of respondent o Male o Female o Prefer not to say o Other: _________________ 4. Education Qualification of respondent o General Nursing & Midwifery (GNM) o B. Sc Nursing o Post Basic B. Sc o Ayurveda Practitioner o Other: _________________ 64 | NISHTHA - Final Performance Evaluation USAID.gov 5. Type of facility o SHC o PHC o UPHC o Other: _________________ 6. When did you start working with this HWC? o 2019 o 2020 o 2021 o 2022 7. Which basic training did you receive before joining as CHO? o Six Months Certificate Course in Community Health (CCH) o Integrated B. Sc Nursing Course o Other: _________________ 8. When did you receive your basic training? o 2019 o 2020 o 2021 o 2022 9. How would you rate the training overall? o Excellent o Very Good o Good o Fair o Poor 10. Was the content well organized and easy to follow? o Yes o No 11. How useful was the training content? o Extremely useful o Very useful o Somewhat useful o Not so useful o Not at all useful 12. Was the basic training helpful in enhancing knowledge and skills in providing community health care services beyond what you had previously acquired? o Yes o No 13. How well did your basic training prepare you to provide comprehensive primary care service at the level of Sub-Centre? o Extremely well o Very well o Somewhat well o Not so well o Not at all well 65 | NISHTHA - Final Performance Evaluation USAID.gov 14. With reference to Q 13, If not so well or not at all well, what is the reason? (Mark all that apply) o Too technical o Difficult to understand o Difficult to apply on daily basis o Other: _________________ 15. How well did your basic training prepare you to perform preventive and promotive services to improve community health? o Extremely well o Very well o Somewhat well o Not so well o Not at all well 16. Was the training helpful to improve laboratory services in your HWC? (Mark all that apply) o Increased sample processing o Helped understanding the procedures better o Still performing investigations as usual o Other: _________________ 17. Do you think HWC meets all healthcare needs of the community? o Yes o Somewhat o No 18. If somewhat or no, please mention the reasons behind this. _________________________________________ 19. Is there a need for any communication or outreach activity to improve client footfall at HWC? o Yes o No 20. What other trainings you have received? (Tick all that apply) o AYUSH o i-Learn (Learning Management System for CHOs) o CHO Leadership Program o Induction Training o Collaborative Framework for TB Family Care for TB o TB Champions o TB for CHOs o JAS (Jan Arogya Samiti) Diarrhea and Pneumonia o MCH (Maternal & Child Health) o DVDMS (Drugs and Vaccine Distribution Management System) o NQAS (National Quality Assurance Standards) o e-Sanjeevani o Family Planning Expanded Services Saksham o WASH (Water Sanitation and Hygiene) o Other: _________________ 66 | NISHTHA - Final Performance Evaluation USAID.gov 21. Which training was most helpful? (Please write name of trainings as there can be more than one answer) ___________________________________________________________________________ 22. What other topics would you like to learn about? (Mark all that apply) o How to interpret laboratory tests o How to ensure treatment adherence for NCDs o How to engage the community o How to encourage mothers and their family to deliver in the facility o How to prevent malnutrition o How to manage the budget o How to improve teleconsultations o Other: _________________ 23. What types of issues / health problems do people have in the community? (Mark all that apply) o Maternal health o Infant diarrhea o ARI o Malnutrition o Diabetes o Hypertension o Cancer o Anemia o Occupational health problems o Dental health o Mental health o Poverty o Gender Based Violence o COVID o Other___________________ 24. Do you have the basic resources (staff, medicines, testing kits and equipment) to provide quality care to your patients? o Yes o No o Other___________________ 25. Overall, how satisfied are you with your work? o Very satisfied o Somewhat Satisfied o Neither satisfied nor dissatisfied o Somewhat dissatisfied o Very dissatisfied 26. What are the key challenges that you face in the Health and Wellness Centre Operations? (Mark all that apply o Lack of community support o Lack of security o Lack of diagnostics Lack of medicines o Lack of support by ANM or ASHA 67 | NISHTHA - Final Performance Evaluation USAID.gov o Lack of supportive supervision o Unreliable internet access for tele consultations o Power cuts o Other___________________ 27. Based on your experience as a CHO, what would you like to suggest to government officials or institutions who train future CHOs? ___________________________________________________________________________ ___ 28. Do you have any comments or suggestions for NISHTHA team? ______________________________________________________________________________ ____ 29. You are welcome to share your name and phone number if you want to be contacted for further information on your experience. 68 | NISHTHA - Final Performance Evaluation USAID.gov COMMUNITY INTERVIEW Introduction: My name is (__________________); I am conducting a survey to find out what health services you have used in the last two years and what recommendations you have to improve healthcare services at the HWC. Confidentiality and consent: I am going to ask you questions about your opinions on the HWC. Your answers are completely confidential. Your name will not be written on this form, and will never be used in connection with any of the information you tell me. Do you consent to participate in this survey? Identification Details Name of HWC Village Hamlet Block District State Respondent Details 1. Respondent Category 1. Pregnant Woman 2. Mother with child less than 2 years 3. Married women in the reproductive age group 4. Adult men (below 65 years) 5. Elderly Female (above 65 years) 6. Elderly Male (above 65 years) 2. Age (Years) 3. Gender (Male/Female) 4. Marital Status 1. Unmarried 2. Married 3. Divorced 4. Widowed 5. Level of Education (Education completed) 1. Primary 2. Secondary 3. Graduate 4. Post Graduate 5. Illiterate 6. Size of the Family including adult and children 7. Occupation Perceptions on Quality S.No. Question and Filters Response Options Skips 1. In general, how would you describe your own health? 1. Excellent 2. Good 3. Poor 4. Not Sure 5. Decline to answer 2. Have you ever visited a doctor or a health facility in last 1 year? 1. Yes 2. No If ‘No’, skip to Q5 3. If yes, where have you visited? (Name of facility/ doctor) 4. For what reasons did you visit the health facility? 5. Are you aware of the HWC in your village? 1. Yes 2. No If yes, skip to Q7 6. If no, end the interview. 7. If yes, have you ever visited the HWC (Name of the HWC)? 1. Yes 2. No If yes, skip to Q9 69 | NISHTHA - Final Performance Evaluation USAID.gov S.No. Question and Filters Response Options Skips 8. If, no, why not? (Seek response and end the interview) A. Not required to visit B. Not sure about services C. No confidence D. Not sure about availability of doctor E. No sure if this open F. Not good G. Other…………….. (Specify) 9. If, yes, when the last time you visited, how long does it take to reach (Name of HWC) Time: ……………. 10. How did you reach the HWC? Ask for any money incurred if private vehicle Mode of Transport: ……….. 11. How many times in the last 1 year you visited (Name of HWC)? 12. Has anyone from the HWC visited your household? 1. Yes 2. No 12A. If yes, what was the purpose of the visit? 3. 13. Why did you select (Name of HWC) for your last visit and not any other? (Multiple response, select all that applies) A. Good reputation B. Recommended by family/ friend/ relative C. Low cost of treatment D. Location is accessible E. Availability of services F. Availability of medicines G. Availability of tests H. Not cured by previous provider I. Referred from the previous provider J. Bad experience at the previous provider K. Other………. (specify) L. Don’t Know 14. What services you have availed in the last 1 year? (Multiple Response) A. MCH Services (ANC/PNC) B. Delivery C. Child Immunization D. Family Planning E. Sick Child F. NCD screening G. NCD related consultation (telemedicine) H. TB Screening I. Medicine for condition under treatment (……………….) J. Wellness activity/ day K. Others ……………... (Specify) 15. Who attended to you on your services 1. CHO 2. ANM 3. ASHA 4. Others (Specify) ____________ 16. Was the service availed were adequate? 1. Yes 2. No 17. Did the (Name of the HWC) facilitate use of telemedicine to access higher level of care? (Ask for use of phone/ camera to consult the doctor) 1. Yes 2. No 18. Were you ever referred to CHC/ DH or other higher facility in case the services were not available at (Name of HWC)? 1. Yes 2. No If no, skip to Q21 19. If, yes, for which services you were referred? 20. Was your problem solved? 1. Yes 2. No 70 | NISHTHA - Final Performance Evaluation USAID.gov User Satisfaction S.No. Question and Filters Response Options Skips 21. Have you observed any changes in your (name of HWC) in last 2 years? 1. Yes 2. No If no, skip to Q23 22. What changes you have observed? (Multiple response) A. Converted from non-functional to functional B. New Staff C. Open daily D. Open for loner hours E. More services available F. Medicine availability G. Diagnostics availability H. More clean I. Other ………………. (specify) 23. Are you satisfied with the services you have received at (Name of HWC) 1. Very satisfied 2. Satisfied 3. Neither satisfied nor dissatisfied 4. Dissatisfied 5. Very Dissatisfied 24. Please specify reason for your above response. 25. Do you feel some people from this village do not visit (Name of HWC) at all? 1. Yes 2. No If no, skip to Q27 26. If yes, who are they? (mention reason for not visiting the HWC) 27. Would you encourage your friend/ relative/ family, neighbors to visit (Name of HWC) for health services? 1. Yes 2. No 3. Not Sure 28. Any suggestions to improve the services at (name of HWC) 71 | NISHTHA - Final Performance Evaluation USAID.gov ANNEX 7. TIMELINE AND DELIVERABLES Table 9 Preliminary Evaluation Timeline and Deliverables TASK / DELIVERABLE DATE (2022) Begin document review April 25 Orientation and getting acquainted meeting with NISHTHA team May 6 Draft Inception report submitted to USAID/India May 11 Inception report approved by USAID May 13 Team leader arrives in country May 17 Team planning meeting May 20 USAID & IP KIIs May 17-20 Full evaluation team onboarded May 20 Final list of specific KII respondents to be developed with NISHTHA team and submitted to USAID May 23 KII protocols finalized and pre-tested May 20 KIIs and facility data collection May 23-June 10 Community interviews July Presentation of Preliminary Findings to USAID/India (and another to the IP?) June 16 Final team meeting to discuss next steps and report writing June 17 Team leader departs June 18 Data analysis finalized August 10 Draft evaluation report submitted to USAID/India August 29 USAID/India reviews draft report and provides comments TBD Evaluation team reviews comments and finalizes report TBD Final evaluation report submitted to USAID/India 72 | NISHTHA - Final Performance Evaluation USAID.gov ANNEX 8. TABLES Table 10: EQs and Illustrative Sub-Questions QUESTIONS SUB-QUESTIONS 1. To what extent has NISHTHA’s technical assistance supported the GOI to achieve the overall objective of increasing access to quality comprehensive health care services at primary level? 1.1. Has the design of the activity allowed it to achieve the overall objectives? 1.2. How well has the activity supported the GOI to implement quality comprehensive healthcare services at primary level in selected states? 1.3. How much has NISHTHA supported the GOI to increase access? 1.4. How many CHOs report to have been empowered to deliver healthcare through HWCs? 2. Based on the available evidence, what is the progress towards the establishment of learning labs, upgrading and operationalization of health facilities and access to quality comprehensive health care services at primary level? 2.1. How many Learning Labs have been established and how well do they perform? 2.2. How many health facilities have been upgraded and how well do they perform? 2.3. How well has the activity operationalized the mandatory components at health facilities? 2.4. How well has the activity operationalized the upgrading of HWCs to deliver quality comprehensive primary healthcare services? 2.5. Is the current activity monitoring effective to measure progress towards the planned results? 3. Were the intervention activities and approaches implemented appropriately to address gender differences/gaps? What evidence exists to substantiate the reduction of gender gaps? 3.1. How has the activity identified gender gaps? 3.2. What gaps were identified in the various states? 3.3. What interventions and approaches has the activity developed to address identified gender gaps? 3.4. How well has the activity measured the effectiveness of its gender gap reducing interventions? 3.5. What gender gaps remain to be addressed? What opportunities exist to address them? 4. What are the accomplishments and challenges? How can the implementation be adjusted to address those challenges and speed up the interventions that have lagged behind (if any) 4.1. What are the main accomplishments of the activity so far? 4.2. What evidence is there that the activity is on track to achieve the planned results? 4.3. What challenges need to be addressed for the activity to achieve the planned results (if any)? 4.4. How can activity implementation be accelerated (if needed)? 4.5. What are some of the missed opportunities that can strengthen the program further? 5. What are the key learnings out of the project implementation so far? 5.1. What lessons have been learned so far? 5.2. How has the activity applied and shared its lessons learned? 5.3. What lessons can be used to improve implementation? 6. What are the perceptions of the nearby community that is served by the HWC supported by NISHTHA? 6.1. Which are the services for which community access HWC services? 6.2. Has the community observed any changes in the HWC in recent times? 6.3. Is the community satisfied with the services at the HWC? 6.4. What improvements does the community want at the HWC? 73 | NISHTHA - Final Performance Evaluation USAID.gov Table 11. NISHTHA Stakeholder Matrix Type of Stakeholder At what level does the stakeholder function? What role does this stakeholder play? What matters to this stakeholder? What assets does this stakeholder bring to the activity? How can this stakeholder be better engaged (if needed)? GOI Centre Owner of the AB-HWC program Successful implementation of AB-HWC program; provision of CPHC services to community Strategy, Guidelines, Infrastructure, Human Resources, Supplies Identification & collaboration on need gap areas State Governments State Responsible for implementation of the AB-HWC program Successful implementation of AB-HWC program; provision of CPHC services to community Infrastructure, Human Resources, Supplies Identification & collaboration on need gap areas International Institute of Health Management & Research (IIHMR) University State Designing & Executing CHO Leadership Training Program Successful execution of the training program University with proven academic experience for leadership & management training Partnership with state level institutes to create a larger pool of leadership trainers as well as programs Noora Health State Designing of Family care giver model for TB and drug resistant TB patients, including IEC materials and training program Successful execution of the model Strategy, Stakeholder management & Training of CHOs Existing challenges to align goals in the partnership. Stakeholder can be engaged at facility, district and state level. Karma Nagaland Development of and technical support to Naga Telehealth, a telemedicine platform Successful implementation and utility of the platform Telemedicine platform, technical know￾how and training of CHOs Development of IEC materials to be provided; Development of a strategy to involve local doctors to bridge language barrier between patients and doctors. Indira Gandhi National Open University (IGNOU) Centre & State Memorandum of Understanding (MoU) between GOI and IGNOU for developing Certificate in Community Health (CCH) course. Identification & Accreditation of PSCs in states; Successful enrolment of students, execution of course, and graduation of CHOs IGNOU has a network of regional centers, and Learner Support centers. Each regional center has number of PSCs responsible for implementation of the course Establishment of coordination mechanisms among the key stakeholders (GOI, IGNOU, PSCs and JHPIEGO) 74 | NISHTHA - Final Performance Evaluation USAID.gov Table 12. Summary of NISHTHA’s Learning Labs, Location and Status S.No Learning Lab Location and Target Start Date Status 1. CHO Learning Program 1200 CHOs across 12 project intervention states ● Arunachal P – 25 ● Assam – 110 ● Nagaland – 25 ● Mizoram – 25 ● Meghalaya – 25 ● Tripura – 110 ● Sikkim – 25 ● Jharkhand – 110 ● CHG – 110 ● MP – 221 ● Odisha – 387 ● August 2020 Total Trained 314 ● Cohort-I – 103 (Nov 21 – Feb 22) ● Cohort II – 110 (Feb – Apr 22) ● Cohort III – 101 (Apr to Jul 22) 2. Family Care Model for TB and MDR TB 1500 TB patients & their families across 2 districts of Madhya Pradesh (Guna & Khandwa) ● July 2021 (Launched) ● January 2022 (on-ground implementatio n ● 77 HWCs selected ● 77 CHO trained ● 468 ASHAs oriented ● 84 Family Member Trained ● 272 TB patient reached (as on 7th May 2022) 3. Operationalizing AYUSH HWC, Odisha 25 dispensaries (10 in Puri and 15 in Ganjam districts) ● April 2022 ● Operationalization data not available on HWC progress ● IEC materials been prepared (in English language) ● Learning material is being prepared) (NISHTA has to operationalize 2500 AYUSH centers across its 12 intervention states) 4. Community Monitoring and Social Audit Mechanism for HWCs 2 districts in Jharkhand ● 4 blocks (20 HWCs) in West Singbhum 3 blocks (15 HWCs) Khunti ● Dec 2020 ● 35 JAS constitutes and monthly meetings in progress ● Patient satisfaction survey done in March 2022 (data not available to comment) ● Community monitoring activities yet to be in place – IVRS service, Health visioning & health planning, social audit, annual public dialogue ● Achievement data not available 5. Strengthening supply chain mechanism (SCM) across 6 NHM District Warehouses and 150 SHC-HWCS ● Landscape Assessment Sep 2021 ● Landscape Assessment Completed ● State approval sought 75 | NISHTHA - Final Performance Evaluation USAID.gov S.No Learning Lab Location and Target Start Date Status primary healthcare settings in Meghalaya (Districts - East Khasi Hills, West Khasi Hills, Ri Bhoi, West Jaintia Hills, West Garo Hills, Southwest Garo Hills) ● Approval by State- March 2022 Polit Start – April 2022 (tentative) ● Implementation yet to initiate 6. Increasing uptake of FP services among the young and low parity in two districts of Meghalaya 50 HWCs across 2 districts (West Garo Hills and Ri Bhoi) ● May 2022 ● Partner Identified ● Implementation is yet to start 7. Achieving Continuum of Care through Digital Interventions Not yet finalized Not yet finalized Yet to start 8. Midwifery care at HWC 17 HWCs across 3 districts in Arunachal Pradesh ● Tirap – 4 HWCs ● Longding – 5 HWCs ● Changlang – 8 HWCs ● January 2022 ● Facility assessment conducted ● 17 Facilities Identified ● Implementation yet to start Table 13. Summary of Selected Results of the Operationalization of HWC in Sampled States. HWC Target Achievement Other Achievements Innovations Partnerships Key Challenges Madhya Pradesh 106% High acceptance of digital portals MoU for AYUSH HWC PBI framework and portal Family care giver model for TB Noora Health for TB Learning Lab High Neonatal Deaths (29 per 1000 live birth) Inconsistent quality across facilities & districts Weak referrals system due to fewer doctors for Teleconsultations Chhattisgarh 117% Internal branding by state adapted nationwide Exclusive HWC window for NCD at DHs and CHCs TB Mitanin for promoting adherence to TB Rx No partnerships UPHC/ PHC’s MO and staff not trained on expanded services Medicine shortages at HWC level CHO capacity building on inventory and financial management Nagaland 111% HWC branding customized with culturally appropriate symbols Naga Telehealth Recognition and Reward for CHOs Community kitchens Community newsletter Karma for Naga Telehealth CIHSR as hub for Telehealth Remote locations of HWCs which are not well connected with transport/ emergency services Poor data network 76 | NISHTHA - Final Performance Evaluation USAID.gov HWC Target Achievement Other Achievements Innovations Partnerships Key Challenges Improved health seeking behavior by communities highlighting community contributions Proactive Village Health Committees Assam 100% First HWC-SHC NQAS assessment NCD Shushrush App NCD Tickler bag for patient tracking AB-Aushadhi pouch No partnerships High Maternal Deaths (215 per 100,000 live birth) and Child Deaths (33 per 1000 live birth) Bureaucratic challenges – recent focus on CDR-MDR Lack of collaboration amongst multiple development partners Meghalaya 108% First state to rollout iLearn CHO-ANM onsite training for increased collaboration Credit hours for continuous education (State Nursing Council) through iLearn World Bank for HWC infrastructure development Remote locations of HWC with irregular electricity and absence of transport/ emergency services Table #. HWC Upgradation State-wise target vs achievement (As per GOI's target (ROP 2018)) State Target (2019-20) Achievement (2019-20) Target (2020-21) Achievement (2020-21) Target (2021-22) Achievement (2021-22) Arunachal Pradesh 79 120 (152%) 149 211 (142%) 234 276 (118%) Assam 1175 1192 (101%) 2209 2211 (100%) 3477 3478 (100%) Chhattisgarh 1311 1845 (141%) 2465 3099 (126%) 3881 4523 (117%) Jharkhand 977 988 (101%) 1836 1597 (87%) 2891 1685 (58%) Madhya Pradesh 2863 3032 (106%) 5382 6313 (117%) 8474 8996 (106%) Manipur 110 113 (103%) 206 206 (100%) 324 326 (101%) Meghalaya 116 121 (104%) 217 249 (115%) 342 369 (108%) Mizoram 95 81 (85%) 178 180 (101%) 280 292 (104%) Nagaland 101 101 (100%) 189 218 (115%) 297 330 (111%) Odisha 1696 1615 (95%) 3189 1637 (51%) 5021 5030 (100%) Sikkim 38 40 (105%) 72 84 (117%) 113 125 (111%) Tripura 257 274 (107%) 483 278 (58%) 760 548 (72%) 77 | NISHTHA - Final Performance Evaluation USAID.gov State Target (2019-20) Achievement (2019-20) Target (2020-21) Achievement (2020-21) Target (2021-22) Achievement (2021-22) Achievement 8818 9522 (108%) 16575 16283 (98%) 26094 25978 (100%) 78 | NISHTHA - Final Performance Evaluation USAID.gov ANNEX 9. NISHTHA’S THEORY OF CHANGE