i June 15, 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by CAMRIS International. Midterm Evaluation of the Health for Life Core Activity Final Report ii MIDTERM EVALUATION HEALTH FOR LIFE CORE ACTIVITY IMPROVING THE CAPACITY OF THE GOVERNMENT OF NEPAL TO PLAN, MANAGE, AND DELIVER QUALITY AND EQUITABLE MATERNAL, NEWBORN, CHILD HEALTH, AND FAMILY PLANNING SERVICES June 15, 2016 Contract Number: AID-367-C-15-00001 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. iii CONTENTS Acronyms and Abbreviations...................................................................................................... 1 Evaluation Team Members ........................................................................................................ 3 Acknowledgments ...................................................................................................................... 3 Executive Summary ................................................................................................................... 4 Background................................................................................................................................ 9 Evaluation Questions and Methodology ....................................................................................13 Findings and Conclusions .........................................................................................................17 Recommendations ....................................................................................................................33 Annex 1: Evaluation Statement of Work ....................................................................................40 Annex 2: Getting to Answers Matrix ..........................................................................................78 Annex 3: Documents Reviewed ................................................................................................82 Annex 4: Persons and Organizations Interviewed......................................................................83 Annex 5: Data Collection Instruments........................................................................................84 Annex 6: Performance Management Plan ...............................................................................104 Annex 7: H4L’s Statement of Differences in regard to the Midterm Evaluation.........................111 Annex 8: Response to H4L’s Statement of Differences by the Evaluation Team......................118 1 ACRONYMS AND ABBREVIATIONS ANC Antenatal care ANM Auxiliary nurse midwife AWPB Annual work plan and budget BCC Behavior change communication CAP Community Action Promoter CAP/R Community Action Promoter/Researcher CB-IMCI Community-based integrated management of childhood illness CB-IMNCI Community-based, integrated management of newborn & childhood illnesses CBO Community-based organization CB-NCP Community-based newborn care package CEONC Comprehensive emergency obstetric and neonatal care CH Child health CHD Child Health Division COP Chief of Party COR Contracting Officer’s Representative CPR Contraceptive prevalence rate CTEVT Center for Technical Education and Vocational Training CYP Couple-years of protection DAG Disadvantaged group DDC District Development Committee DHO District Health Office DoHS Department of Health Services DPHO District Public Health Office DSA Daily subsistence allowance EDP External development partner EHA Essential hygiene actions ENA Essential nutrition actions FCHV Female community health volunteer FGD Focus group discussion FHD Family Health Division FP Family planning FY Fiscal year G2G Government to government GBV Gender-based violence GESI Gender equality and social inclusion GIS Geographic information system GIZ German Federal Enterprise for International Cooperation GON Government of Nepal H4L Health for Life core activity H4Lcore Health for Life core activity HF Health facility HFOMC Health Facility Operation and Management Committees HISPIX Health Information System Performance Index HMIS Health management information system HR Human resources HSS Health system strengthening HTSP Healthy timing and spacing of pregnancy IEC Information, education and communications IPCC Interpersonal communication and counseling IR Intermediate result IRHDTC Integrated Rural Health Development Training Center IUCD Intrauterine contraceptive device JAR Joint Annual Review KII Key informant interview LHGSP Local Health Governance Strengthening Program LMD Logistics Management Division LMIS Logistics management information system M&E Monitoring and evaluation MD Management Division MDAG Marginalized and disadvantaged group MDG Millennium Development Goal 2 MEL Monitoring, evaluation, and learning MNCH Maternal, newborn, and child health MNCHN Maternal, newborn, and child health and nutrition MoFALD Ministry of Federal Affairs and Local Development MoHP Ministry of Health and Population MOLD Ministry of Local Development (now MoFALD) MTE Midterm evaluation MWRHD Mid-Western Regional Health Directorate NDHS Nepal Demographic and Health Survey NFHP Nepal Family Health Program NGO Non-governmental organization NHEICC National Health Education, Information, and Communication Center NHRC Nepal Health Research Council NHSP Nepal Health Sector Programme NHSS Nepal Health Sector Strategy NHSSP Nepal Health Sector Support Programme NPC National Planning Commission NPR Nepali rupees NRH Nutritional Rehabilitation Homes OCMC One-stop Crisis Management Centre OPD Outpatient department ORC Outreach clinic ORS Oral rehydration salts PHA Public health analytics PHC Primary health care PHCC Primary health care center PHCRD Primary Health Care Revitalization Division PHO Public Health Office PMP Performance management plan PNC Postnatal care PPFP Post-partum family planning PPH Post-partum hemorrhaging PPP Public-private partnership PY Project year QA Quality assurance QACG Quality Assurance Coordination Group QAWC Quality Assurance Working Committee QAWG Quality Assurance Working Group QI Quality improvement QIT Quality Improvement Team RH Reproductive health RTI Research Triangle Institute SBA Skilled birth attendant SDG Sustainable Development Goal SRH Sexual and reproductive health SWOT Strengths, weaknesses, opportunities, and threats TA Technical assistance TOCAT Technical and organizational capacity assessment tool TWG Technical Working Group TWG-HIM Thematic Working Group – Health Information Management UNFPA United Nations Population Fund USAID United States Agency for International Development USG United States Government VDC Village Development Committee WCDO Women and Children Development Office WHO World Health Organization 3 EVALUATION TEAM MEMBERS Dr. Elvira Beracochea, Team Leader Dr. Yasho Pradhan, Family Planning/Maternal, Newborn, and Child Health and Nutrition Expert, New ERA Dr. Devi Prasai, Health Systems Expert, Community of Evaluators Ms. Pinky Singh Rana, Gender Equality and Social Inclusion Expert Dr. Prakash Pant, Demographer and Statistician Mr. Nirakar Acharya, Field Coordinator, New ERA Ms. Meena Sitaula, Data collector and Focus Group Moderator, New ERA RN. Sareeta Pradhan, Data collector and Focus Group Moderator, New Era Mr. Umesh Chaudhary, Data collector and Focus Group Moderator, New ERA ACKNOWLEDGMENTS The evaluation team wishes to thank the support of the MEL team, particularly Jonathan Jones for his feedback and support on qualitative research, Manorama Adhikari for her support with the analysis of focus groups, Lorene Flaming and Scott Thomas for their feedback and editing of the final report, Heather Sullivan for her assistance with the graphics and Shibesh Regmi for his overall support in keeping this project on track. We are very grateful for the support and feedback provided by USAID/Nepal. We particularly want to thank and acknowledge the support of Sabita Tuladhar, Daniel Verschneider, Daniel Sinclair and Murari Adhikari. We also want to thank the H4L team that took lots of their time to share their work and challenges, particularly the district staff that accompanied the evaluation team and was ready to answer all our questions. We also want to acknowledge with gratitude the time the colleagues at the MoHP, DoHS, DPHO staff and the health post healthcare providers gave us to share their experiences and views on working with the assistance of the H4L activity. Finally, we want to dedicate our work to the FHCVs and mothers and community leaders that also took time of their busy lives to participate in the interviews and focus groups discussions. 4 EXECUTIVE SUMMARY This is the report of the midterm evaluation (MTE) of the Health For Life core activity (H4L) conducted by the Monitoring, Evaluation, and Learning (MEL) project from January 26 to March 31, 2016. This evaluation was postponed due to an Indian blockade that has affected the landlocked country of Nepal, limiting supplies of medicines, fuel, and all other goods. The blockade was lifted during the evaluation, but its effects endure. EVALUATION PURPOSE The purpose of this MTE was to assess the effectiveness of the H4L Core activity of the H4L project and recommend midcourse corrections. The MTE had two main questions to answer (Exhibit 1): How effective the project had been to address the gaps of the health system, and how flexible it had been to respond to the needs of the Government of Nepal (GON) to plan, manage, and deliver quality and equitable maternal, newborn, and child health and nutrition (MNCHN) and family planning (FP) services. PROJECT BACKGROUND The H4L project is to build on the accomplishments of prior projects with “a greater focus on health system strengthening (HSS) and measuring progress in system improvements1 .” Its goal is to improve the GON’s stewardship to plan, manage, and deliver quality and equitable MNCHN/FP services. To strengthen the health system, the project is to improve governance at local and district levels; develop evidence-based health policies; strengthen the GON’s stewardship; institutionalize a quality improvement (QI) system; improve the capacity of health workers and community volunteers to deliver MNCHN/FP services; improve knowledge, behavior, and use of services by target marginalized and disadvantaged groups (MDAGs) integrating the GON’s gender equality and social inclusion (GESI) policy; and strengthen the capacity of a local logistics contractor to improve the logistics system and the supply of medicines. The project is to work at national level and in 14 districts and has the potential to impact 4.3 million Nepalese citizens. Work is to be conducted in collaboration with other USAID-funded projects, particularly the Suaahara project, which aims to improve nutrition in the same target districts. EVALUATION METHODOLOGY The MTE used a number of quantitative and qualitative methods, including review of the project’s documents and other health sector documents and more than 100 key informant interviews, which included interviews with USAID staff, Ministry of Health and Population (MoHP) and district staff, health workers, members of Health Facility Operations and Management Committees (HFOMC), QI teams, Village Development Committees (VDCs), female community health volunteers (FCHVs), and H4L staff. The MTE team also observed a district review meeting and a very small number of services delivered in 10 selected health facilities in 5 of the 14 project-supported districts visited (Dang, Banke, Bardiya, Surkhet, and Dailekh). A facility readiness assessment was conducted using the project’s readiness assessment validated tool, and a services satisfaction survey was conducted using a systematically random community sample of households (instead of the planned exit interview), given the small 1 Project’s Scope of Work 1. How is H4Lcore activity addressing health system gaps identified in the problem statement and activity baseline? 2. To what extent is the structure of H4Lcore activity sufficiently flexible to respond to the maternal, newborn, and child health priorities of the MoHP as outlined in the Nepal Health Sector Plan II, 2010–2015? Exhibit 1: Key Evaluation Questions 5 numbers of patients observed in every facility. Fifteen focus group discussions (FGD) were conducted with mothers of infants, youth, and women of MDAGs—three in each of the five districts visited. An email survey was sent out to all 108 H4L staff and a strengths, weaknesses, opportunities, and threats (SWOT) analysis was conducted with the H4L leadership team and USAID staff. Findings and performance indicators in the project’s performance management plan (PMP) were verified and data checked with the H4L team. FINDINGS AND CONCLUSIONS The overall evaluation of the project’s effectiveness indicates that project—with a team of over 100 staff and a limited budget of US$18.2 million—has achieved considerable progress in assisting to improve local governance through improved planning and management of health facilities in the project￾supported VDCs. However, less progress has been achieved regarding the delivery of quality and equitable MNCHN/FP services by these facilities. Progress regarding the improvement of quality of health care service delivery and demand generation tasks as stated in objectives 4, 5 and 6 of the project has been less significant in comparison to the widespread revitalization of HFOMCs. A number of factors were reported to have affected the implementation of project activities regarding these objectives including: issues with the project design and lack of a common understanding of health system strengthening approaches among all stakeholders involved; two major earthquakes that diverted resources to affected areas; a blockade from the neighboring India that prevented the importation of medicines and the supplies to manufacture them; and high staff turnover among project and USAID staff. All, this seems made project focus mainly on strengthening governance, creating of quality improvement teams and training health providers as the main strategies to strengthen the health system. Despite these limitations, there is preliminary evidence of the increase in the demand for maternal health services by MDAGs in the sentinel VDCs, where a new tracking system is being implemented using community action promoters and researchers (CAP/Rs) hired jointly by the project and the local HFOMCs. The scope of work (SOW) of the H4L core project is comprehensive and addresses most of the building blocks of a health system, because it was designed to address several aspects of the health system that needed strengthening and to be a much larger activity. However, despite this design, the funding was reduced at the time of the contract to fund other activities by the mission. The number and breadth and depth of the deliverables and activities were not reduced proportionally. This reduction in funding was reported to have limited the breadth and depth of the activities and contributed to limiting the progress in some objectives. Another observation about the SOW is that it is focused on inputs and activities but not so much on defining the outcomes that those activities must deliver. In other words, it says much about what must be done and how but not so much about what the results must be. The deliverables are mostly outputs too and do not define the minimum level of performance the GON is to achieve in the planning, management and delivery of MNCHN/FP services at national, regional, district and facility levels as a result of the H4L core project. In addition to the reduced funding issue, a number of design issues were identified as well. First, the analysis of the project’s SOW and its health system strengthening (HSS) approach shows that it addressed most of the World Health Organization (WHO) building blocks but not the health financing component, which would limit the capacity of the GON to implement the project’s strengthening activities in addition to its recurrent operational budget without additional funding to do so. In addition, the need to transition and still provide humanitarian assistance and support to the MOHP while providing development assistance was not anticipated. The scope was later expanded with additional funding but only to respond to the needs of ten additional districts affected by the two major earthquakes, but not the original 14 target districts. As mentioned above, the country has also been affected by political issues and a blockade from neighboring India that has increased the cost of fuel and other goods and limited the country’s financial capacity to supply medicines as well as to plan, budget and channel funding for HSS activities. 6 Second, the analysis of the project’s Performance Management Plan (PMP) shows that a number of outputs and outcomes are not expressed in quantifiable terms and the intermediate results (IRs) are not linked to the project’s activities (see Annex 6) not allowing timely monitoring of the project’s progress. In addition, the impact indicators may not show the expected impact as they have whole population denominators, while the project is to improve quality of services and equitable access by MDAGs within selected districts. Consequently, when successful, the project’s impact may improve quality and equity in the target populations but not achieve population-level impact. When talking of equity and MDAGs in the context of the H4L core activity, it is important to understand that the selection of project districts was based on the DAG mapping score rather than HDI. The DAG mapping includes the 7 indicators that are social determinants of health status: food sufficiency, marginalized households, condition of primary school, condition of health post, participation of women, Dalits and Janajati in decision making, prevalence of gender discrimination, and vulnerability. The main achievement for which H4L project is well-known and recognized is that it has revitalized 97% of HFOMCs in the target districts and expanded participation of MDAGs in these committees, as well as improved health planning at VDC level. Improvements in planning at VDC level have not been observed at district level, where the project’s assistance was reported to have been limited to helping districts conduct gap analyses and conduct review meetings but has not included assistance to districts to implement their activities to address at last some of those gaps through the revitalized HFOMCs. In addition, the project is also well-known for having facilitated the creation of QI committees and working groups at national and district levels, and QI teams in the project-supported facilities. All this has contributed to having increased the readiness score of these facilities to deliver quality care in accordance to the readiness assessment tool developed by the project. In regards to improving the capacity of district, and local health workers and community volunteers to deliver quality MNCHN/FP services, the project has provided extensive training of health workers in the first two years of the project in several areas including interpersonal communication. However, building capacity is more than training. The project has been reported to not having implemented other activities to ensure trained staff implements what they learned. The capacity of the districts to follow up on training to help health staff implement new healthcare delivery processes and to supervise health workers and community volunteers to ensure quality and equitable delivery of services according to health care guidelines and quality standards was reported not to have been strengthened. The project has demonstrated leadership and played a major role in the design of the new health policy and the Nepal Health Sector Strategy 2015–2020 (NHSS) but the implementation strategy of the project has not yet been aligned to the NHSS. The same can be said about the new costed FP policy and plan and the project’s approach to improve the planning, management and delivery of FP services in the target districts. The project has worked to remain flexible to implement requested changes and respond to requests from the GON and to the changing priorities. However, these changes are reported to take several months for USAID to approve. RECOMMENDATIONS Given the current limitations and degree of progress, we recommend an adjustment in the SOW to allow the project to focus on objectives 4, 5 and 6 in support of objectives 2 and 3 that is the GON stewardship to implement the NHSS pillars, outputs and outcomes in the 14 target districts over the last two years of the project. The project’s work plan in PY4 and PY5 should implement that activities listed below to strengthen the health system at the district level and demonstrate improved district and VDC performance in terms of improved quality and coverage of MNCHN/FP services. Building on the successful engagement of VDCs and the revitalization of HFOMCs and the QI teams in each facility, the project should assist District Health Offices (DHOs) to work with these stakeholders to improve the delivery of health care 7 services according to the MoHP guidelines and in line with the new health policy and the NHSS in the target 14 districts. In addition, in September 2015, USAID approved its “Vision for Health Systems Strengthening 2015- 2019) 2 .” In line with the document, we recommend that the project align its activities with the HSS core functions and focus on HSS outcomes of improving coverage and quality of essential services, as well as equity and responsiveness to local needs, particularly MDAGs. In this way, the project will be able to focus on demonstrating performance improvement at district, health facility and community levels by assisting districts to improve quality and equity of facility-based MNCHN/FP services and expand community-based, integrated management of childhood illness (CB-IMCI), and community-based services by FCHVs, particularly in MDAG areas. To do this, the project should assist DHOs, HFOMCs, and QI teams to monitor utilization and productivity of health staff and facilities and implement a community-based demand generation approach that addresses local GESI barriers. The project’s PMP should be adjusted to support a dashboard for monitoring of project activities, outputs, and outcomes on a quarterly basis. The project should assist the DHOs to monitor facility management procedures, such as which HFOMCs monitor medicine and equipment inventories, the number of HFOMCs that monitor health worker performance and whose facility performance scorecards show they meet healthcare delivery coverage targets and quality standards. The project should also assist QI teams to monitor and report indicators of improved quality of health care delivery of MNCHN/FP services and facilitate the sharing of best practices with other QI teams in other VDCs in the district. To strengthen the capacity of the district to do all this, the project should improve MNCHN/FP program management at district level. The project should accompany the annual district planning process to include support to districts to improve program management in the Project Year (PY) 4 work plan. By the end of the project, districts and HFOMCs should be able to demonstrate the capacity to monitor and increase access and quality of health care services provided to MDAGs and all citizens in their VDC. The overall outcome should be that all 14 project supported districts perform at the top level of the MoHP scale and demonstrate improved quality and equity in the planning, management, and health care delivery of MNCHN/FP services. With an increased focus on improving district and health facility performance outcomes and expansion of community-based services, the project should plan and coordinate activities with the Suaahara project to ensure integration of health and nutrition activities at facility and community levels. The project should also leverage its success in governance to plan and coordinate its work at district level with the Sajhedari Bikas project. In this way, USAID and the GON will be able to see and monitor an integrated and well-articulated U.S. Government strategy that delivers measurable outputs and outcomes through reliable progress indicators, as well as ensure a smooth handover of governance, health and nutrition activities in PY5. USAID has invested in Nepal’s health sector for more than 25 years, and a review of these achievements and lessons learned was ongoing at the time of this MTE. This review will also add insights about actions to strengthen the Nepali health system. Our findings indicate that through previous projects, several successful pilots have been conducted and new interventions and standard technical assistance have been tested. What remains is development of the health system to make all these routine services. USAID should consider the adoption of a 15- to 20-year roadmap of higher-level and more complex health reform and development assistance of which H4L is an important instrument. Progressively and consistently USAID activities will help build and hopefully, graduate a sustainable decentralized health system capable of scaling up interventions, managing its own public health programs, and of ensuring the delivery of high-quality equitable health care services that meet local 2 https://www.usaid.gov/what-we-do/global-health/health-systems/usaids-vision-health-systems-strengthening 8 needs. In this way, USAID will have made an important contribution to Nepal’s attainment of the targets of Sustainable Development Goal (SDG) 3: Ensuring Healthy Lives through Universal Health Coverage by 2030. 9 EVALUATION PURPOSE In December 2015, USAID/Nepal commissioned an external midterm evaluation (MTE) of the Health for Life core activity (H4L) implemented by Research Triangle Institute (RTI) and partners Jhpiego and the Integrated Rural Health Development Training Center Nepal (IRHDTC/Nepal). The main purpose of the evaluation was to assess the effectiveness of the H4L activity toward meeting its objectives, document how it is making a difference in Nepal’s health system, and recommend any changes to improve the project’s performance. The secondary purpose is to recommend changes in the project model for health system strengthening (HSS), with a focus on achieving Sustainable Development Goal (SDG) 3 (see Exhibit 2). USAID will use the findings and recommendations to make changes to the H4L activity to ensure the best use of the remaining years of the project, and share lessons with the Government of Nepal (GON) and other stakeholders. The findings will also be used to inform design of the follow-on project. The MTE only assesses the H4L core activity. For ease of communication, we refer to it as H4L. It also briefly considers the work in 14 other districts, which were added to address the April 2015 earthquake and a GON local governance initiative, as it affects the design and implementation approach of the H4Lcore. BACKGROUND PROJECT DESCRIPTION The main purpose of the H4L core project is to strengthen the capacity of the GON to plan, manage, and deliver quality and equitable maternal, newborn, and child health and nutrition (MNCHN) and family planning (FP) services. As the project mainly focuses on HSS, the project objectives were designed for strengthening the capacity of the health system to improve the planning, management, and delivery of MNCHN/FP services. Following the successful achievement of the health targets of the Millennium Development Goals (MDGs) by the GON, this project is focused on improving equity to reach those not having access to quality MNCHN/FP health services—marginalized and disadvantaged groups (MDAGs). It is implemented over a five year period (December 2012 to December 2017) with a budget of $18.2 mission. The main goal of the H4L project is that the MoHP will be fulfilling its stewardship role in working with counterparts at local levels to continually build structures and systems that deliver high-impact and equitable services to citizens. The design hypothesis is that the GON’s capacity to plan, manage, and deliver quality and equitable Exhibit 3: H4L Objectives 1. Improve health system governance at district and sub-district level health facilities 2. Develop and implement a national evidence-based health policy 3. Strengthen national level stewardship of the health sector 4. Institutionalize a nationwide system for quality improvement 5. Improve capacity of district and local health workers and community volunteers to deliver quality MNCHN/FP services 6. Improve knowledge, behavior, and use of health services among target populations 7. Strengthen the capacity of the local logistics contractor • Assess the effectiveness of the core activity • Document how the project is making a difference • Recommend changes to the project model Exhibit 2: Evaluation Main Purpose 10 MNCHN/FP services would be improved through a number of HSS interventions that included most of World Health Organization (WHO)-recommended building blocks of health systems, with the exception of financing: 1. local health governance 2. data for decision-making and evidence-based policies 3. human resources (HR) capacity 4. quality improvement 5. logistics systems 6. knowledge and behavior change Consequently, the project is designed to achieve seven objectives (see Exhibit 3) and it was expected that by midpoint, the H4L core project would have demonstrated effective ways of improving the capacity of GON to plan, manage, and deliver MNCHN/FP services so that by the end of the project, a measurable improvement in the health system performance would be achieved. The H4L core project is also designed to build on the achievements of prior U.S. Government projects to improve access to MNCHN/FP services, such as the Nepal Family Health Plan (NFHP) I and II, and to function as part of a portfolio of USAID-funded projects in the health sector that include governance, HIV/AIDS, nutrition, and social marketing. H4L is designed to provide technical assistance at national, regional, and district levels to improve the quality and equitable coverage of the health care delivery of MNCHN/FP services in 14 districts. The H4L core project was expanded to include work in 6 demonstration districts—2 that were already part of the 14 original target districts and 4 more new ones—for the implementation of a local governance initiative of the GON known as the “Collaborative Framework (CF).” In addition, 10 more districts were added to provide assistance following the earthquakes of 2015. The technical assistance at national and regional levels and in the 14 districts includes a number of interventions and activities known as H4L core, which are the focus of this evaluation. The MTE did not include the CF or the assistance to the earthquake affected districts. HEALTH SYSTEMS STRENGTHENING H4L is a project designed to strengthen the health system in Nepal. Although there is no general consensus, WHO’s views on the six building blocks of HSS are generally accepted. A lot has been learned about what works from HIV/AIDS projects. The work of the Global Fund and the recent Ebola epidemic in West Africa have also highlighted the importance of countries having effective health systems that meet the needs of their populations. In this section, we will define what a health system is and provide a short summary of the generally accepted view of the strengthening process. In September 2015, USAID published its “Vision for Health System Strengthening” that includes the overarching goal of ensuring sustained equitable access to essential and high quality health services. The vision includes four strategic outcomes: financial protection, essential services, population coverage and responsiveness as well as 6 functions that address WHO’s six building blocks. 11 Exhibit 4 shows the structure of a health system with three levels. The base of the pyramid is the primary health care (PHC) level, where most of the patient-centered health services are delivered through a network of public, private, and civil society and non-profit facilities. PHC includes facility-based services provided by frontline health workers and community-based services provided by community health volunteers or paid workers, and the management and referral linkages between facilities and communities. In Nepal, female community health volunteers (FCHVs) provide community-based services. H4L is to improve the health system to impact performance at the PHC level. The secondary level includes referral facilities for specialized care, medical education, and research institutions. H4L is to improve the training of health workers at the national and regional training centers, as well as the performance of selected ANMs schools. Assistance to improve the National Health Research Center’s capacity to conduct research into HSS challenges is also part of the H4L project. The third level is where governance structures plan and manage the implementation of health policy reforms that are the core of HSS activities. At this level, the health system manages a number of sub￾systems: health surveillance and management information, financing, human resource, supply chains for essential medicines, supplies and equipment; and quality patient-centered service delivery—the six WHO building blocks. Coordination of development assistance also takes place at the governance level. For the system to work, all the six blocks need to be aligned, so that “vertical’ programs address the health priorities of each country. Vertical programs are converted into “horizontal” packages of essential services to meet the local needs of communities at the point of service. Most countries opt for some form of “basic packages of services” to deliver quality health care funded through various models of insurance schemes and/or public and out-of-pocket or private funding. This pyramid structure is evident at national, district and local levels in Nepal. The MoHP is the governance level; the secondary level is public and private hospitals that provide specialized care and the medical, nursing and auxiliary nurse midwife (ANM) schools that train the future health workforce; and the PHC level is the network of public and accredited private facilities delivering services. This structure is also repeated at district level, where the District Health Office (DHO) represents the governance level, the district hospital and local training schools are the secondary level, and the network of PHC centers and health posts are the base of the pyramid. At the local level, the Village Development Committee (VDC) and the Health Facility Operation and Management Committee (HFOMC) play the local governance role, in alignment with DHO’s plans and MoHP’s policies and program guidelines. All three levels work to provide the services needed by the population in their respective coverage areas. Health system strengthening (HSS) includes a number of strategies and interventions to strengthen the performance of the pyramid structures that comprise the six WHO building blocks of a well-functioning health system. USAID’s new vision for HSS includes approaches, interventions and indicators to measure progress of the HSS process. Below is a description of the main activities and interventions usually included in HSS projects. Phase 1: Laying the foundation for effective health system performance. Decentralization policies are passed and district implementation guidelines of decentralized planning and management are tested and implemented. PHC is strengthened through revised guidelines and procedures that are simplified for increased efficiency, integrated vertical programs are converted into service packages and Exhibit 4: Structure of a Health System 12 standardized and consistently implemented across all public facilities. Health facility and health worker performance are supervised. Re-engineering of health facility delivery processes is started to bring facilities up to pare in regard to quality standards, and effective health service delivery processes are in place and monitored to reach and account for the unserved groups. Health worker performance targets are set and effective supervision and support are in place. Health management information systems (HMIS) and surveillance programs are standardized, and streamlined processes are put in place. Various financing mechanisms are tested, and a national policy is developed. Effective accreditation of private health providers and training schools is implemented. Phase 2: Management improvement. Ownership and institutionalization of new programs—quality improvement (QI), gender equality and social inclusion (GESI), gender-based violence (GBV), trauma, cancer, etc.—are achieved through effective governance, effective standard operating procedures, and program management. MNCHN/FP and other relevant programs based on the country’s epidemiological profile are managed efficiently and coverage continues to be expanded. Management and quality of hospital health care and an efficient referral system are strengthened and in place. Standardized quality health service delivery packages are in force in all public and private health facilities. HMIS and surveillance are able to provide timely information for decision making and have the capacity for data quality control. Effective research capacity is built and demonstrated to inform and monitor policy reform and programmatic decisions. Effective intra- and inter-sectoral coordination is demonstrated. System structures demonstrate improved management capacity; ability to measure and compare performance measures; and set and achieve performance targets, checks, and controls. A national health insurance scheme and other appropriate financing mechanisms are in place. Phase 3: Scale up. Improved health program and overall system efficiency are measured and achieved. Equity and growing coverage expansion, citizen participation, accountability, and transparency are measured and managed. Synergy is increased with all inter- and intra-sectoral stakeholders at national and local levels, and health system performance targets are progressively met. The system is able to monitor performance deviations and make corrections. Financial and humanitarian assistance may be required to achieve efficient health system performance standards. Phase 4: Sustainability, self-continuous reform, and improvement. Self-improvement capacity and demonstrated health reform capacity are in place, and only focused and intermittent technical assistance is required. Financial and humanitarian assistance may be required to respond to natural or man-made crises. 13 EVALUATION QUESTIONS AND METHODOLOGY The MTE sought to answer two key questions that guide the evaluation process (see Exhibit 5 A full set of evaluation questions and sub-questions is available in Annex 2. To ascertain the degree of progress, achievements, and gaps, and the degree of effectiveness of the H4L project, which is in its third year of implementation, the MTE team used the following “life cycle” model of an ideal five-year HSS project and the principles of the Paris Declaration on Aid Effectiveness (see Exhibit 6). This approach assesses how the different parts of the building blocks of a health system are being strengthened during the life of the project. In the life cycle model, the MTE team understands PY 1 to 3 as the “effectiveness” years, in which the project is to deliver outputs and outcomes through interventions of documented and measurable effectiveness. By “documented”, we mean that we can find evidence of the progress the project has made, and by “measurable”, we mean that the project’s benchmarks and indicators are able to measure progress. The first three years of a project are typically years of intense HSS intervention testing, training, and implementation to ensure maximum MNCHN/FP program coverage and proactive harmonization with other stakeholders working in the health sector. By PY 3, the project must start the transition to strengthen structures that will sustain the achievements of the previous years, by institutionalizing and systematizing interventions. PY 4 is the “sustainability” year, in which the project team starts to hand over its effective interventions to district, regional, and national counterparts and reduce support to an “as needed” basis. The focus of PY 5 is to complete the handover that will sustain the results and demonstrate accountability for these results. At the time of the MTE, it was not expected that the project will have strengthened the health system in any of the HSS areas, but will have shown progress along its objectives and benchmarks. Nepal and the U.S. Government have both endorsed the principles of aid effectiveness of the Paris Declaration and both strive for country ownership, harmonization, alignment, management by results, and mutual accountability (see Exhibit 6). The MTE assessed the project’s progress in light of the progressive fulfilment of these principles as they relate to the increased stewardship role of the GON. However, we understand that the project was not designed to fulfil the principles of the Paris Declaration or to follow the stages of the life cycle, so these are used to identify opportunities for improvement and not to measure the contractor’s performance. 1. How is H4Lcore addressing health system gaps identified in the problem statement and activity baseline? 2. To what extent is the structure of H4Lcore activity sufficiently flexible to respond to the MNCHN priorities of the MOHP as outlined in the NHSP II, 2010-2015? Ownership Partner countries exercise effective leadership over their development policies and strategies and coordinate development actions. Alignment Donors base their overall support on partner countries’ national development strategies, institutions, and procedures. Harmonization Donors’ actions are more harmonized, transparent, and collectively effective Managing for Results Results are achieved by managing resources and improving decision-making Mutual Accountability Donors and partners are accountable for Exhibit 5: Key Evaluation Questions Exhibit 6: Principles of the Paris Declaration 14 During the initial years of the project, the Nepal Health Sector Strategy (NHSS) 2015–2020 was developed with assistance from external development partners (EDPs), and replaced NHSP II (2010– 2015). H4L contributed significantly to the new strategy, which acknowledges the importance of development partners as well as the increasing complexity of having numerous organizations and agencies assisting in the health sector. The NHSP II document guided the project from 2013 to 2015, and NHSS is considered as the main health policy to guide the strengthening of the Nepali health system from 2015 to 2020. Alignment of the project with this policy document is expected. The NHSS has four strategic pillars: 1. Quality health services; 2. Equity and access in health; 3. Health systems reform; and 4. Multi-sectoral approach; and calls for nine interrelated outcomes including related interventions to achieve them and outputs focused on the delivery of a basic essential package of services to all Nepalis: 1. Rebuilt and strengthened health systems: Infrastructure, HRH management, Procurement and supply chain management. 2. Improved quality of care at point-of-delivery 3. Equitable utilization of health care services 4. Strengthened decentralized planning and budgeting 5. Improved sector management and governance 6. Improved sustainability of health sector financing 7. Improved healthy lifestyles and environment 8. Strengthened management of public health emergencies 9. Improved availability and use of evidence in decision-making processes at all levels METHODOLOGY The evaluation team drew on a mix of methods for the evaluation, including document reviews, interviews with key informants, and visits to five project districts. Following an extensive document review (see Annex 3) and of the project’s Results Framework and scope of work (SOW), the MTE team conducted a total of 103 interviews with key informants across 10 districts. The 103 interviews include 57 with MoHP representatives at national, regional, district, and facility levels, as well as members of the communities. Forty-six interviews were conducted with staff from H4L subcontractors, from other USAID-funded projects, from USAID, and from other external development partners. All interviews were conducted using a semi￾structured interview guide. An email survey was sent out to over 100 H4L staff, and 46 confidential replies were received and analyzed only by the Team Leader to ensure confidentiality. Meetings to conduct a strengths, weaknesses, opportunities and threats (SWOT) analysis were held with the H4L team at headquarters and with selected USAID staff. Collectively, the H4L project covers12 districts of the Midwestern region and 2 districts of western region District Selection – Total 5 3. Balance between high- and low-performing districts with focus on identifying the traits of the positive deviant districts and the barriers and challenges in the low-performing district 4. Representation of marginalized populations with high concentration of marginalized pop 5. Representative of all interventions VDC selection – total 10  High-priority VDC  Has a birthing center  Large target populations and diversity  One near district base and the other no more than two hours’ drive were selected in discussions with the DHO.  Wide range of activities: five are general TA and five are sentinel  One high- and one low-performing VDC in terms of MNCHN/FP indicators if data are available Exhibit 7: Field Sites Selection Criteria 15 of Nepal. Given available time and resources, data collection was limited to five districts. The criteria used to select the districts were developed in consultation with USAID/Nepal, the district teams, and the H4L team (see Exhibit and 8). In each district, the MTE team interviewed the District Public Health Office (DPHO) and other members of the district team available at the time of the visit. The MTE team visited 10 priority VDCs, 2 in each district. In each district, interviews were conducted with VDC members, health workers in these facilities, members of the HFOMCs, members of Quality Improvement Teams (QIT), members of local mothers’ groups, adolescents and marginalized populations, and FCHVs to gather their views of H4L results. The selection of the VDCs to be visited was finalized during the visit to the corresponding DPHO, based on a review of local information for the VDCs and MTE selection criteria. Of the 10 selected priority VDCs (Exhibit 8), 5 had received general technical assistance (TA) and 5 were sentinel sites which had a community action promoter (CAP) and researchers (CAP/Rs). In each facility, a facility readiness assessment tool (see Annex 5) was used to assess the facility’s readiness to deliver quality MNCHN/FP services. The MTE team validated this tool before deployment. In addition, the team observed the number patients in each facility during the time of the visit and the use of the health care delivery guidelines. However, observations could not be done in a standardized way given the low utilization of the health facilities. The team also took photographs of quality of care issues which were discussed during the SWOT analyses and various briefing meetings with the H4L Team and the mission. An exit interview of patients attending the health facility on the day of the visit was planned, but could not be conducted given the low utilization of the health facilities (the team typically observed only three to five patients or clients attending the facility during a three- to four-hour visit). Instead, a community survey using randomized systematic household sampling was used to identify households with a mother whose child was delivered in the last 12 months, to find out her views and satisfaction with the services received. Data collectors were instructed to turn right from the health facility and choose the third household on the left and inquire about whether there was a mother with an infant, request consent if affirmative, and conduct the interview. The data collectors were also instructed to continue visiting every third household until they reached the end of the village, then turn back, always visiting households on their left, until they reached 15 interviews. In isolated VDCs in Dailekh and Surkhet, this proved difficult, and fewer interviews were obtained. A total of 95 mothers were interviewed from the 10 VDC communities where a selected health post was visited. The team also conducted 15 focus group discussions (FGDs) with mothers of children below 1 year of age, youth, and members of MDAGs— three in each of the five districts visited. The full set of data collection tools used for this evaluation is available in Annex 5. LIMITATIONS The findings of the MTE are based on a cross-sectional review of the project’s activities, and their interpretation may not be comprehensive or representative of the project’s numerous achievements. This MTE was designed to answer questions regarding the project’s ability to address gaps and to respond flexibly to the needs of the GON in relation to the planning, management, and delivery of MNCHN/FP services. The MTE was not designed to assess the performance of the Nepali Health System in regard to its capacity to plan, manage and deliver quality and equitable health services, but the degree of progress of the Project in influencing and improving it. Moreover, the findings from the fieldwork are based on a non-representative sample of 5 districts out of 14, and of 10 priority VDCs (5 sentinel and 5 general TA) out of 140 priority VDCs (out of 627 total project-supported VDC sites). Although, this is a common trade off with qualitative data collection, and Dang: Goltakuri and Gadawa * Bardiya: Suryapatuwa and Deudakala * Banke: Saigaun and Titihinya * Dailekh: Gamaud and Badalamji * Surkhet: Guthu and Sahare * * sentinel site Exhibit 8: Districts and VDCs Visited 16 we believe the data highlight themes and issues that emerged in most of the studied districts, we cannot conclude that this is representative of the situation in all the remaining districts. Preliminary findings were triangulated from several sources and shared with the H4L team to validate some of our observations and to correct factual or interpretation errors. We conclude that the findings will give the H4L team a fresh perspective on issues that will require new strategies or renewed efforts. Interviews are ideal to explore perceptions of gaps and needs (real or not) and offer the flexibility to address them in depth and detail. It is important to hear about a given topic in the interviewee’s own words regarding how well the project has managed to meet its design objectives. Because interviews take significant time and resources, it is typically not possible to interview enough people to develop findings that are representative of a whole population. For that reason, although the team interviewed 103 people and reviewed 46 email replies to the email survey, we may not have been able to capture all achievements, gaps and needs or understand all the causes and reasons for the observed project’s degree of progress and effectiveness. The MTE triangulated the data from different sources and discussed preliminary findings with the project team and USAID to better ascertain and understand the main findings related to overall progress toward the project’s objectives and the challenges the project is facing. As noted above, because the number of patients/clients that visited the facilities was very small, the research did not allow for standardized observations, so the reported observations are to be considered anecdotal and might be the focus of more detailed study and follow up by the H4L team and USAID. Attribution of the findings may prove difficult given the increasing number of external development partners in the health sector in Nepal and in the target districts. Contribution is more appropriate in such a complex programming environment, where a variety of actors, factors, enablers and constraints come together to affect change. For that reason, qualitative data collection focused on contribution and complementarity of the H4L activity to improving the quality and equity of MNCHN/FP services. Having the H4L team accompany some of the field visits added to the context of our observations. However, to ensure more open and confidential discussion of challenges and gaps, we conducted interviews of beneficiaries using one-on-one private settings, while one member of the MTE team interviewed the H4L staff separately. Despite the strengthening work of the H4L team to improve the health information system, the HMIS in Nepal has been reported to have limitations. Since the H4L team uses the HMIS to report on several of its intermediate results (IRs) in its PMP, the MTE mined and made secondary analysis of the data in the HMIS. However, without doing a thorough data quality analysis, the HMIS findings may be indicative of further research needs and not of gaps or systemic weaknesses or needs. We have strived to triangulate data from various sources to prevent this and to qualify HMIS-related findings. One way we did this was to thoroughly review the PMP with the H4L team and to analyze the data from the CAP￾CAP/R and compare with HMIS data. 17 FINDINGS AND CONCLUSIONS In a HSS project, the key word is “performance.” This word summarizes the project’s focus and expected impact. The H4L project is an HSS project that seeks to improve the performance of the GON’s MNCHN/FP programs and the performance of the health system to deliver quality and equitable MNCHN/FP services. Therefore, we present the findings in answer to the evaluation questions in the context of how well the project has so far addressed gaps in health system performance and improved the performance of governance and evidence-based policies in the health sector, quality improvement, data use, capacity of health workers to deliver quality MNCHN/FP services, and utilization by target communities. These findings are presented in alignment with the high-level questions and sub-questions in the SOW (Annex 1). PROJECT DESIGN AND IMPLEMENTATION Summary of Findings on Project Design and Implementation: The overall finding of the project design is that it was designed to be a much larger project, but funding was reduced at the time of the contract (from approximately US$28 million to US$18 million). The original design was not adjusted, and it still addressed the need for specific inputs and processes to strengthen the health system at national, regional and district levels. These inputs and processes are in line with most of the recognized building blocks of a health system as promoted by WHO, except health financing, which was not included in the design. This limitation was reported to have limited the ability of the project to implement some of its strengthening activities at district and facility levels, where disadvantaged populations live and where there are scarce resources to be mobilized. In addition, the analysis of the project’s Results Framework (RF) showed that the impact indicators may not be appropriate measures of project impact because they are measures of impact on the general population, while the project targets selected districts and communities of MDAGs within districts. The project activities may not have measureable impact to move the needle at country level, but would show impact if equitable access were measured as an IR. Disaggregated indicators of coverage and access to quality services by the target populations may be more appropriate. Project implementation has been uneven—better for some objectives such as 1, 2, and 7 and less so for objectives 3, 4, 5, and 6 due several factors including issues with the project design, reduced funding, and lack of a common understanding of health system strengthening approaches among all stakeholders involved; two major earthquakes that diverted resources to affected areas; a blockade from the neighboring India that prevented the importation of medicines and the supplies to manufacture them; and high staff turnover among project and USAID staff. All this let the project to initially focus mainly on strengthening governance, creating of quality improvement teams and training health providers and supporting the development of the NHSS mainly. Project management and oversight are limited because output and outcome indicators are not linked to achieving measurable improvements of MNCHN/FP program performance indicators, quality and equitable coverage of MDAGs. This disconnect makes the project’s activities under each objective seem to run in parallel to, instead of synergistically contributing to, strengthening the planning, management, and delivery of MNCHN/FP services. Outputs, outcomes, and IRs do not allow for timely monitoring of the effectiveness of planning, management, and delivery capacity at national, regional, district, and facility levels. The sources of the data for some IRs are not always reliable, and the reporting periods also varied, making performance comparisons across districts and VDCs hard. The lack of appropriate output, outcomes, and impact indicators, coupled with three changes in Contracting Officer’s Representatives (CORs), may have prevented USAID from monitoring effectively and implementing timely course corrections. 18 HOW IS H4LCORE ADDRESSING HEALTH SYSTEM GAPS IDENTIFIED IN THE PROBLEM STATEMENT AND ACTIVITY BASELINE? Overall Progress The project has achieved or is on track to achieve objectives 1, 2, and 7 by project completion. The MTE showed that the project is well-known and recognized for having revitalized a high number of HFOMCs (97%) that are now functional in the project’s target districts. The project also successfully provided support to the Family Health and the Child Health Divisions to monitor NHSP II, and in developing the new NHSS 2015–2020. The project also assisted the Management Division which is in charge of local health governance, quality of care and HMIS. In addition, the project has reported to have played an important leadership role in support of the Collaborative Framework in pilot districts. The project has also provided TA to the local logistics contractor in the form of two consultancies—one in leadership and another in financial management. Twenty-five percent of the project staff and 36% of the project resources are deployed at district level; districts reported more support is needed from the project, and that only two staff at this level is not enough to address their challenges. MoHP directors and stakeholders acknowledged the effort of the project to reach isolated and remote VDCs: “Most donors want to work in the cities. We are happy the H4L project is working in isolated rural areas where the need is most high.” – MoHP Director In contrast, the MTE indicates project performance related to objectives 3, 4, 5, and 6 has been more limited. In regard to objective 3, the project reported to have hired the services of a consultant to help develop the NHSS that was approved in June 2015. However, in interviews the project was reported not being able to respond to the implementation of the NHSS by reorganizing and focusing its activities to implement it in the target districts. H4L seems not much involved in the development of NHSS Implementation Plan which must guide the future H4L activities. Review of the project’s work plans and interviews with staff show that the application of a basic package of services, as well as the actions required to achieve the nine outcomes of the NHSS, do not seem to have changed the approach of the project or its activities in PY3. Also, with regard to the benchmarks under objective 3, the project reported it is in the process of exploring public-private partnerships (PPPs), but none has been formalized yet, and we do not know how these PPPs will help advance the implementation of the NHSS. For more detailed progress information, see the project’s SOW where deliverables for which evidence of progress was found are marked with “” and those for which we were not able to find enough evidence or progress was less significant are marked with “”. Objective 4 calls for the institutionalization of a nationwide QI system. The project has developed QI Teams, and these teams were observed to be functional—that is, they meet regularly, have developed an action plan and keep meeting minutes, but do not have performance targets. In addition, the MTE assessment showed that the facilities demonstrate high levels of readiness based on the project’s readiness assessment tool. However, despite being ready, there are problems in the quality of services delivered, such as cleaning practices and handwashing observed not to follow infection prevention standards or health workers observed not to follow clinical guidelines. These quality-of-care issues seem not to have been corrected despite the training, coaching, and support reported to have been provided by the project. In addition, functional linkages between facility and community-based services were reported not to have been strengthened. For example, FCHVs reported not being supported by health workers in their respective facilities in the implementation of community-based integrated management of newborn and childhood illnesses (CB-IMNCI) or FP counseling. Objective 5 is to improve the capacity of district and local health workers, as well as of FCHVs, to deliver quality MNCHN/FP services. The project reported to have met its training targets one year early, but this 19 achievement was not recorded in the national training information system. Since this training is not linked to supervision and measuring of health worker performance, it is not clear if and how these trained staff will help districts manage and deliver better quality and equitable health services. The number of FCHVs trained in the target MDAG populations was not tracked. Community-based approaches, such as CB-IMCI, were reported to not have been expanded and this was a concern at national and district levels. This lack of progress might be due in part to limited funding and in part to the perception of the project staff that they are an HSS project, not a health service delivery project. This misunderstanding of the interconnectedness of HSS objectives and the building blocks of a health system, along with overemphasis on governance activities related to the revitalization of HFOMCs, may have hindered the project’s progress regarding this objective. “We are a health system strengthening project; we do not do service delivery.” – Project staff Objective 6 is to improve knowledge, behaviors, and use of services by the target populations. The project has demonstrated increases in the use of health facilities in sentinel sites through the use of improved tracking using dedicated personnel who are provided with a smart phone and the free CommCare app. Exhibit 9 shows the coverage of MDAGs by district and all the numbers are not very big, they do show trends and areas of inequity that need to be addressed. Despite the usefulness of providing valuable data such as the consistent low coverage of MDAGs in Jakarkot district (see highlighted data) where more project support would be required to bring it to par, the quality of services provided by the CAP/Rs, and the coverage of this intervention in comparison to other options such as the current MoHP’s community midwives pilot test, has not been evaluated by the project yet. Other interventions to increase demand have been implemented but not sustained or evaluated. More on this objective will be presented in the corresponding section. District Dalit Janajati Madhesi Muslim Brahmin/ Chhetri Other Overall Total % N % N % N % N % N % N % N Arghakhachi 77.8 9 75.0 40 - - - - 84.2 57 - 3 80.7 109 Banke 90.0 30 100.0 50 87.9 33 93.2 44 93.1 29 - 6 92.7 192 Bardiya 100.0 12 97.1 35 70.0 10 87.5 8 95.0 20 83.3 6 92.3 91 Dailekh 86.7 30 - - - - 100.0 2 85.2 61 83.3 6 85.9 99 Dang 92.3 13 90.4 52 100.0 5 100.0 2 96.2 52 100.0 4 93.8 128 Jajarkot 42.9 7 50.0 2 - - - - 36.4 11 60.0 5 44.0 25 Jumla 86.5 37 100.0 3 - - - - 83.3 72 - 2 85.1 114 Kalikot 68.8 16 - 1 - - - - 76.2 21 83.3 12 76.0 50 Kapilvastu 65.8 38 85.5 62 61.3 75 81.0 79 97.8 46 - 1 77.7 301 Pyuthan 72.4 58 67.8 87 - - 100.0 3 81.6 38 100.0 15 74.6 201 Rolpa 66.7 30 63.2 57 - - - - 61.8 34 87.5 8 65.1 129 Rukum 73.7 38 63.6 11 - - - - 70.5 78 - 24 70.2 151 Salyan 58.3 12 66.7 3 - - - - 73.4 64 55.6 9 69.3 88 Surkhet 86.1 101 97.2 72 - - - - 87.1 93 100.0 14 90.0 280 Total 78.7 431 82.7 475 70.7 123 86.2 138 82.2 676 81.7 115 81.1 1958 Note: All delivery conducted in the institutions are done by SBA. Data on what proportion of delivery among home delivery are conducted by SBA is not available H4L possesses a comprehensive GESI strategy, which includes specific activities that aim to address and enhance access and utilization of health resources, particularly by MDAGs. However, project staff reported that activities to integrate and roll out the strategy at district and sub-district levels have not been implemented. Respondents noted that to achieve sustainable change in the health system, change will need to go beyond health care provision and address the local social determinants of Exhibit 9: Percentage distribution of women who had institutional delivery and a live birth by ethnicity 20 health, including GESI barriers (family, social, cultural, institutional, economic, and geographical) that affect both service seekers and service providers. “When it comes to GESI everyone says they have understood, but just making a couple of women and Dalits representatives participate is not enough. – Ministry of Federal Affairs and Local Development informant Efforts to enhance accountability of HFOMCs regarding GESI barriers and to identify and institutionalize GESI-related best practices are not visible; neither are endeavors to reach and address communities’ and MDAGs’ GESI needs. The NHSS states it will seek to foster the potential for “greater participation of women, poor, and disadvantaged groups” through structures such as healthy mothers’ groups and HFOMCs.3 These remain to be undertaken in the project-supported districts. The project was reported to be expected to play a greater role in developing Implementation Plan for NHSS including addressing GESI strategies in healthcare delivery and demand. Finally, we want to reiterate that output, outcome, and impact indicators do not accurately measure results by districts, and within districts, by target populations. The project’s PMP does not measure progress of community-based activities, and there are no IRs related to addressing GESI as part of the project’s strengthening of health care management and delivery and demand generation activities. See specific comments on each indicator of the PMP in Annex 6. We conclude that the project has focused on strengthening governance and facility readiness in the hope of achieving better health outcomes, and this has limited its ability to achieve objectives 3, 4, 5, and 6. Their understanding of “not being a service delivery project” is against what WHO recommends (and the now published USAID’s vision on HSS) and along with the lack of additional funding to implement quality improvements, may have limited the implementation of tasks and deliverables under objectives 3, 4, 5, and 6. Consequently, the project has focused on governance and system inputs and not processes at the expense of strengthening program management and service delivery: e.g., the project assists the regions and districts to conduct reviews but does not help with implementing procedures to solve the problems presented in these reviews, and the projects helps establish QITs but does not assist to implement their QI plans or procedures to improve the delivery of quality services according to MoHP guidelines. We believe there are opportunities for the project to take advantage and assist the districts, HFOMCs, and QITs to improve support to and motivate health workers to apply guidelines and training to deliver quality health care services. Strengthening the health system and service delivery are not mutually exclusive; in fact, the first is the means to achieve the second, and the project has to achieve measurable improvement in both to ensure the GON gets on the path towards a strengthened health system. We also conclude that the presence of project staff and funding at district and facility levels is insufficient to achieve MNCHN/FP program management improvements and QI goals. The project progress has also been limited by a number of challenges outside its control: • The 2015 earthquakes • District needs for technical and financial support are growing because flexi funds are being reduced as a result of the GON diverting funding to earthquake affected districts • Lack of elections at local level have reduced local capacity to sustain improvements • Border closures that have led to stockouts of essential medicines and commodities • Data quality and HMIS problems have been observed that make measuring IRs difficult The project’s ability to measure progress through output and outcome indicators and MNCHN/FP￾related indicators may be because IRs are not linked to planning, management, and delivery of 3 NHSP III, p 25 21 MNCHN/FP programs or project interventions. The MTE team spent several hours going over the PMP with the H4L team, allowing us to conclude that the project uses it to report to USAID but not to manage its activities or measure its progress. Reporting periods to USAID are not aligned with the availability of data make the data incomplete. We noticed the lack of IRs to measure quality of health care delivery in project districts or coverage among target MDAGs. Detailed descriptions and comments on the IRs and the PMP are in Annex 6. LOCAL HEALTH GOVERNANCE Per the project’s PMP, we noted that revitalization of HFOMCs was achieved in 97% of the 627 facilities supported by the project—an impressive achievement at midpoint in the life of the project. The HFOMCs visited by the MTE were observed to be functional because they meet regularly, keep minutes, have mobilized funding from the VDC, and are advocating to attract users. Also, transparency is reported to be improving: “Handful [of] powerful persons used to carry the fund before, but now it is determined on the basis of need.” – Key Informant Interview (KII), Community In addition, the MTE team observed that the HFOMCs in the visited districts have been able to mobilize additional funding from the VDCs. This is mainly because of the increased participation of the local people, injecting the health agenda in WCF meetings, compliance of MCPM (minimum conditions for performance measures) and evidence based planning. The amount, varied, however, as it is dependent on the availability of funding in each VDC. Some seem to have more funds than others, and also some of the VDCs do not release fund directly to HFOMCs but manage the expenditure themselves and therefore, the expenditure and income are not entered in to inventory book (Exhibit ) which explains some of the “no” in the exhibit. Selected questions from the Facility Readiness Assessment tool Surkhet Dailekh Bardiya Banke Dang Gut hu Sah are Badal amji Gam aud Suryap atuwa Deud kala Titihi niya Saig aun Gad awa Golt akuri In total, what amount (NPR) of support was the HFOMC able to mobilize from VDCs and other organizations? 171, 500 245, 000 200, 000 275, 000 27080 0 120, 000 150, 000 200, 000 433, 790 128, 800 Were any of the supports received from VDC entered in the inventory book (income and expenditure) in the last 12 months? no yes yes yes no no yes yes yes yes Exhibit 10: Reported Additional Funding from VDCs 22 The project reported that it has assisted to implement VDC plans that now prioritize the health sector. A clear upward trend toward increased resources being directed to the sector was observed in the 10 VDCs visited (see Exhibit 11). The 10 visited VDCs were reported to fund the HFOMCs as per the priority order of the integrated VDC plan. They also reported that after the revitalization of HFOMC, VDC-level health planning has been integrated with the VDC plan. Before that, health planning was treated as a separate entity. “A few projects used to get in the VDC through the window but every activity should come through the ward citizen forum” – Vasti Vikas Vela (village development meeting) It was reported in the visited VDCs that a range of 30 to 40 people now assemble in each ward to discuss on the health agenda and activities. We estimate that as a VDC has nine wards, a total of 270 to 360 people have directly participated in the planning process at a VDC level. Before that, only a handful of VDC staff used to plan health-related activities. It was also reported that now there is virtually no bypass of the ward citizen forum. Members of HFOMC started to visit PHC outreach clinics and extend cooperation to offer health care services. They were reported to ask about the problems of the poor and MDAGs and listen. “Now we close the window, all should come from the door.” – KII, Community “Before the H4L, the members of HFOMC were reactive, after the revitalization they became proactive.” – KII, Community). “Health post was supposed to be opened at 10 am and be closed at 2 pm, arriving at 11am and leaving at 1pm was tolerable. Now it opens at 10 am and closes at 3 pm,” – KII, HFOMC Member Another achievement of improved governance is the collaboration between the MoHP and the Ministry of Federal Affairs and Local Development (MoFALD), through the Collaborative Framework, a policy that was developed with partners, including H4L. It is too early to evaluate the progress of the Collaborative Framework, but initial reported progress is positive. 37.4 38.1 2013/4 2014/5 2015/6 The share of Village Development Committees funding directed toward health activities increased from 2013 to 2016. 37.4 38.1 40.7 Exhibit 11: VDC and MoHP Share of HFOMC funding in 10 Visited VDCs 23 Despite these achievements at VDC level, governance strengthening activities at district level are reported to be limited to gap analyses and administrative support by the project staff. DHO staff reported perceiving district project staff to be less skillful and junior compared to their own experience. The transparency of financial information, mutual accountability, and quality of social audit remain as challenges for the project to assist districts. The social audit reports reviewed put emphasis on the health system issues such as creating more positions for healthcare providers, upgrading of health institution, infrastructure development and so on but less on responding to clients’ issues such as cleanliness of health facility, behavior of care providers, availability of medicine and healthcare providers. District and regional reviews and annual reports show systemic gaps that affect performance but the project has been reported not to be able to assist districts to respond to them. The project contributed well at VDC level but limited support to district level was reported with the exception of developing the calendar of operation. In addition, high project staff turnover at district level was reported to have affected the continuity of project activities at district level. Interviews with project staff indicated that GESI interventions have not been integrated as part of strengthening facility management or strengthening MNCHN/FP programs and services at the district level. For example, issues that impact MNCHN/FP services such as early marriage, respectful maternity care, ability of service providers to identify GBV cases to make referrals – have not been integrated into program activities at district and VDC level or included within HFOMC and QIT performance targets to address such gaps. We did not find evidence of the project assisting these governance structures to have targets for health service providers and FCHVs would further mainstream GESI. During interviews, national and district level informants reported that the main gaps affecting the health system are the lack of implementation of CB-IMNCI, insufficient coordination between the numerous stakeholders at national and district levels, lack of skilled health workers, and no district-specific programs to respond to local needs. 4 Document reviews indicated the same. However, the project seems to have been unable to assist the local governance to address these gaps. Efforts to link with and enhance the capacity of GESI institutional structures and GESI focal persons were reported to remain nominal. The project was reported to not have helped districts identify existing barriers or build synergy with relevant government and non-government stakeholders at central, district, and community levels (WCDO, OCMC, NRH, police, community-based organizations [CBOs] working on social and gender discrimination, men’s groups, and migration related CBOs, etc.). Assistance by project was reportedly limited to provide technical assistance to strengthen GESI institutional structures by the creation of a GESI focal point and technical working groups (TWGs) at district and sub-district facilities. “The GESI TWG has become active than before. We try to make sure that GESI concerns are addressed—particularly for the old citizens, domestic violence, marginalized, disabled, freed Kamalari, and others.” – KII, Banke District However, these efforts are reported to be still incipient, and the project reported they have been unable to mainstream context-specific GESI concerns (e.g., Chhaupadi, GBV, migration-related issues, early marriage, etc.). The project staff reported that GESI interventions have not been funded and prioritized for greater change and equity, and this was confirmed in the analysis of the project’s work plans. The project is not implementing a strengthening strategy to improve district management of MNCHN/FP programs. The mean reason for this was that planning is still centralized in Nepal, and districts just implement activities with earmarked funding and have no authority to adapt plans to local needs. In the context of decentralization, this practice would require a change in policy and regulations. This limitation 4 Annual Report 2070/71(2013/2014) Mid-Western Regional Health Directorate, http://www.mwrhd.gov.np/np/downloads/annual71.pdf 24 has not allowed the project to assist districts to better plan to meet district needs. The project has not yet tested models of decentralized planning that would inform further decentralization policies and practices. The Collaborative Framework between MoHP and MoFALD is reported to be effective. However, it is too early to evaluate effectiveness, and future evaluation of the Framework will determine how well it delivers measureable results. In conclusion, despite the emphasis on implementation of governance activities reported at facility level, MNCHN/FP program management processes were reported to have not improved at district level yet and will need to be the focus of the project in next two years. QUALITY IMPROVEMENT The project reports that the vacancies at the MoHP have prevented its ability to support the national QI Steering Committee. Limited technical support is perceived at national level by MoHP counterparts, who seem to see the project as a facilitator of meetings but not as a technical resource. “They help us organize meetings and bring snacks.” –MoHP Official At facility level, the project has successfully formed QITs. Every facility now has a QIT that meets regularly and has a QI plan. The QITs are reported to have improved readiness, which the MTE confirmed through our facility survey. They also have been provided with project-developed guidelines on quality improvement in MNCHN and FP service delivery. However, the MTE observed that health workers do not follow these guidelines and wonder how prepared these QITs are to improve healthcare delivery processes which are essential to deliver quality healthcare5 . In addition, our review of the QITs’ plans shows that they lack priorities and are too ambitious, because the type of activities listed would require additional funding that is not available. These plans also lack a common vision of what quality health care will look like when delivered correctly. The plans do not include activities to address evident quality problems such as lack of handwashing, disorganized medicines and supplies, or lack of compliance with facility cleanliness and infection prevention standards. QITs and health workers were not aware of the need to apply the project’s training on interpersonal communication. In addition, the MTE team observed lack of privacy in the provision of FP services. Low service utilization and low health worker productivity may be a problem too for the project to assist the QITs and districts to address. We observed very few patients and clients in the 10 facilities we visited—three to five patients at the most in the three- to four-hour window of our visit. This observation will require further research by the project team. We had expected about three to four patients to be seen by the ANM per hour. Most facilities visited had two or four ANMs at the time of our visit that looked unoccupied during our visit. We understand that many services are delivered through outreach and a separate assessment of the quality and coverage of these outreach activities may be required to ascertain the real degree of utilization of health services and staff productivity. In short, facility readiness has been improved and sustained by QI teams, but health workers were not observed to follow quality guidelines and protocols. All visited birthing centers were observed not to be clean or ready for use given that they are expected to be ready for a delivery 24/7. We wondered about the ability of the visited health posts to manage medicines, equipment, and supplies given the reported lack of inventory control and management by the HFOMCs and the observed disarray of medicines in some facilities. All 10 visited facilities in the facility readiness survey demonstrated high scores, except some stock outs of essential medicines, which makes us wonder about the appropriateness of this tool 5 Better care for every patient, every time: improving quality in health systems in low income countries. Heiby JR, Armburster D, Jacobs TA. http://onlinelibrary.wiley.com/doi/10.1111/1471-0528.12903/epdf 25 to detect quality issues beyond achieving facility readiness and the project may need to introduce other tools to empower QIT to comprehensively address quality of care issues. The community satisfaction survey revealed that a great majority of respondents (97%) were satisfied with the care received; however, only 67% reported to have used the birthing center in the local health post. It was reported to be used by poor women; those who can afford it deliver at the hospital (HFOMC member and FCHV interviews). Only one of the 95 interviewed women reported to have used FP services at her last visit at the health post. This may be due to recall bias in a random and non￾representative sample, but merits further research regarding missed opportunities to offer FP services. Also during our visit, we observed a number of quality issues—such as the lack of privacy for FP counseling and lack of integration of FP with other services—in all 10 visited facilities that would prevent the delivery of quality integrated FP health care according to standards. Health workers were not observed to follow IMCI guidelines in at three facilities where infants had been brought for consultation. No health staff was seen using the handwashing stations in the 10 visited health posts. Information, education and communications (IEC) posters and other tools were misused to cover or decorate walls (such as the use of a tuberculosis poster in a birthing center) and do not seem to be used to counsel patients or educate the community because they are in areas where education would not be possible. All posters were dusty and have not been cleaned in a while becoming an infection control issue. Medicines and supplies were not stored or inventoried correctly which would prevent quality services. All but one visited birthing centers were observed not to follow infection prevention and cleaning guidelines. Infection control and prevention must be a priority given the high neonatal infection mortality rate in Nepal, which was reported to be the cause of 33% of neonatal deaths according to HMIS and up to 47% of neonatal deaths in a study of six districts (MOHP Verbal autopsy study, 2014). Additionally, according to interviews with members of the QITs, their role seems to be limited to ensuring planning and keeping minutes, appropriately filling out forms, or maintaining readiness scores. They do not seem to be aware of the lack of cleanliness observed in the visited health facilities. QIT plans reviewed do not include ensuring service provision according to guidelines, setting performance and coverage targets for the facility and for each staff, ensuring space is used to meet the needs of the service seekers (such as privacy for patients to have the ability to share problems); referrals for symptoms related to violence against women and girls, malnutrition, or migration-related health problems; or barriers preventing effective performance by health workers. We conclude that the objective of institutionalizing a nationwide QI system needs to be adjusted to be on track. The project is not perceived as influential at national level, and the health facilities visited, although seemingly ready to deliver quality health care, were not observed to do so according to guidelines. Although the number of observed cases was limited to three to five per facility, facility utilization and health worker productivity merit further research and monitoring, because it was consistently low across all 10 visited facilities. We understand that the QI activities are more recent and it takes time for QITs to have full understanding of how QI system works. However, we believe that the project’s next challenge is to assist the QITs to improve the healthcare delivery processes. In sum, the QI strategy of the project requires adjustment. The project lacks appropriate QI IRs to measure the progress of the QI system being implemented. We already mentioned the project’s perception that HSS and quality service delivery are mutually exclusive, when in fact the latter is the goal of the former. This perception may be responsible for the focus of the project on inputs such as the readiness index and the formation of QIT and not on the redesign of the health care delivery processes or the creation of job aides, process checklists, revised supervision tools, and other tools to help health workers, particularly ANMs, follow guidelines and meet QI standards. DATA USE AT LOCAL LEVEL 26 In every visited facility, data were reported to be used as part of the “village health situation analysis” in the VDC planning process, and to mobilize additional funding from VDCs. However, on a consistent basis, facility staff was simply unable to produce or explain data. In addition, there is lack of data on equity in health care delivery, because only one indicator in the PMP addresses GESI by monitoring the number of Dalits that deliver in a health facility. However, the project reported not to use this indicator to monitor equitable access and assist facilities to intervene. Exhibit 12 shows the data obtained from the HMIS regarding the VDCs visited by the MTE team that shows the data quality problems the H4L project and the districts face when managing facilities, and the type of decisions that need to be made. For example, the existence of indicators higher than 100% (see highlighted figures) is a sign that at least the wrong denominator is being used and requires correction, all raising doubts about the numerators as well. Also, though the quality of the data may be in question, when there are big departures from the median, the district must be assisted to address this problem such as Suryapatawa and Badalamji having such low SBA coverage in comparison to the others; or Titihiniya, Saigun and Suryapatawa having such higher LBW that requires looking into the nutrition program in these VDCs. 27 VDC Name 4 ANC visits 4th ANV visits as parentag e of 1st ANC visit Delivery coverage by SBA (as % of Expected pregnancie s) % SBA deliver y among Live birth % home deliver y of total deliver y PNC 1st visits (as % of expected pregnancie s) % of LBW amon g live birth CPR DPT￾Hep B3 TITIHIRIYA (Sentinel) 47.7 78.2 44.5 52.5 - 33.2 15.0 17.35 89.3 SAIGAUN (Priority) 64.2 88.3 63.2 74.4 - 49.4 12.6 40.89 102.8 DEUDAKALA (Sentinel) 39.2 62.9 42.9 50.6 - 10.0 7.4 13.12 96.3 SURYAPATAWA (Priority) 41.0 72.8 8.3 9.8 - 3.1 20.0 21.99 64.7 BADALAMJI (Sentinel) 76.1 103.7 8.7 10.3 - 67.9 2.7 11.17 114.7 GAMAUDI (Priority) 45.6 106.8 57.3 67.8 - 55.2 0.0 14.36 76.5 GADHAWA (Sentinel) 52.3 73.6 67.1 79.0 - 34.8 3.8 45.84 84.4 GOLTAKURI (Priority) 52.3 87.9 30.1 35.4 4.2 25.4 0.0 46.88 87.4 SAHARE (Sentinel) 49.4 76.3 27.8 32.8 - 48.8 2.6 17.84 92.3 GUTHU(Priority) 45.1 64.9 48.8 57.4 - 7.7 3.8 14.14 87.4 Sampled Sentinel VDC 31.3 79.0 24.2 28.5 0.5 33.8 5.7 21.08 94.0 Sampled Priority VDC 49.0 79.0 37.9 44.7 0.5 22.6 6.6 27.14 82.8 Districts reported that the HMIS is cumbersome and not accurate or timely. The H4L team cannot use the data to monitor activities or MNCHN/FP programs but has been able to circumvent the problem using data from 38 sentinel sites (see exhibit 9). We requested some indicators to be calculated using this data, but the project was not able to provide them by the time this report was prepared. The project has assisted the HMIS division of the DoHS with emergency printing of forms. However, the project has not used this opportunity to systematize and address the chronic problem of lack of forms and to prevent this from recurring every year. We understand the project has been looking for creative solutions to this systemic gap. For example, it has assisted the DoHS to estimate the budget for all required forms so it can include this in its annual recurring budget, looked for paperless options, helped coordinate with other donors to share the cost of printing, help find better ways to manage printing procurement efficiently, and looked for options to decentralize the printing by posting forms on the MoHP’s website for districts to download. However, a sustainable solution to the HMIS and its forms has been elusive. Data use by the H4L project has been limited by HMIS weaknesses and a PMP that does not allow timely monitoring despite monthly, semiannual, and annual reporting requirements. In conclusion, progress in this area has been limited and requires adjustment. HMIS gaps make data inaccurate and untimely. Documented best practices in data use have not been developed for use at district or health facility levels. Health facility performance data are not monitored by HFOMC, and health facility staff lack the capacity to use data to improve facility management and health care delivery and coverage. The MTE observed weaknesses in the HMIS and the use of data from health post up to the district level. MOBILIZATION OF COMMUNITY ACTION PROMOTERS Exhibit 12: Performance of visited VDCs by selected indicators. Source HMIS 28 The CAP/Rs have increased tracking of antenatal care (ANC) visits (exhibit 9). Appointment reminder calls to women are reported to have increased the number of women that complete their ANC visits, along with the financial incentive of the Amma project (World Bank- DfID). This has helped increase coverage. CAP/Rs also impart health education, and they report the free CommCare app that tracks pregnant women is easy to use. Despite these advantages, CAP/Rs were perceived as duplicating the function of FCHVs and ANM in tracking pregnant women. Also, they are not aligned with current MoHP policies. At this time, the MOHP is piloting a community midwife model. On the other hand, some HFOMCs reported to really appreciate having the data captured by CAP/Rs and have hired another one. However, CAP/Rs are co-funded by H4L and HFOMC. This funding will phase out at the end of the project, and the cost-effectiveness of this new cadre and its sustainability are in question. Based on these observations, we conclude that the technology is appropriate, easy to use, and provides useful information, and the role played by the CAP/R is necessary to track pregnant women. However, this intervention requires formal evaluation given MoHP’s community midwife initiative and the roles of ANMs and FCHVs in the Nepali health workforce. CAPACITY BUILDING OF HEALTH WORKERS The project developed a capacity building plan in PY 1, and most benchmarks have been achieved. Also, the project reported to have achieved its training targets one year early. However, the project was not able to explain the benefit of this achievement, and it seems the level of health workers’ performance as a result of the project-supported training has not been measured. The project reported that only selected sites have received training in CB-IMNCI. More training is not planned for the remaining of the project, though interviews with national and district staff indicate the need to train health workers in CB-IMNCI particularly. Our assessment showed that the local non￾governmental organization (NGO) subcontractor, whose role is to provide training in rural areas, still has more than 50% of its approximately US$400,000 budget unspent. This NGO reported not being involved in the planning of training activities with the H4L team, and to not know the reason why they have not been asked to continue the CB-IMNCI training. In addition, USAID has reported that these districts will receive CB-IMNCI training from other agencies, but we are not sure how the project will leverage this contribution to building capacity in its target districts. H4L staff reported to conduct various capacity building activities at facility level: training, mentoring, coaching, on-the-job training, etc., but no best practices have been documented or transferred to district staff yet. The numbers of FCHVs and ANMs from MDAGs were reported not to have increased, and most districts are concerned about the aging of the FCHV workforce. There is no up-to-date national or district-wide HR training database, and it is unclear who has been trained and who still needs training. The project did not include the health workers trained by the project into the database to conform to Nepal’s training procedures. They said this was not part of their SOW. We disagree. It is not written in the scope of work but a project like H4L that is to strengthen the system must comply with the system. Project staff report being aware that capacity building is more than training. However, we observed lack of using other approaches to build capacity, such as using the district reviews to share best practices; facilitating exchange visits between low- and high-performing teams; rotating staff and networking; using job aides and new job routines; documenting standard operating procedures; setting individual performance targets and accountability measures after training; or improving procedures for better results of supervision, which are reported to be irregular. Although coaching is reported to be used to help health workers, we did not find a documented coaching manual or procedure that describes how coaching is used to assess its effectiveness and to transfer the coaching capacity to district teams. We did not find evidence that the project is planning to do so yet. Districts reported that the lack of trained skilled birth attendants (SBAs) is a major limiting factor to improve maternal and neonatal outcomes, particularly in MDAGs. The project was reported to have 29 assisted ANM schools to assess their gaps and identified that the capacity of the trainers was not up to standard. The MTE visited one ANM school which reported the project assisted the school to conduct a gap analysis, but it has not helped to address the gaps. We observed they also lacked textbooks and a proper simulation lab. The project reported they are planning to hire a consultant to update the preservice training manual in PY 3. In conclusion, we believe the capacity building targets have not met the capacity gaps identified in the visited districts. CB-IMNCI capacity has not been expanded in project districts yet. The capacity building of QIT has been focused on readiness and not on processes or achieving outcomes. The FCHV program has not been strengthened to improve the quality and equity of their services, and the numbers of FCHVs and ANMs from MDAGs may yet be insufficient to meet the project’s goal of improving the delivery of quality and equitable MNCHN/FP services. The production of new ANMs certified as SBAs is still limited due to weaknesses in ANM schools and the project has not explored SBA certification processes. Stockouts of misoprostol was reported to have hindered the expansion and strengthening of his program. Standards of obstetric care or newborn referral are not reported to be in place in every birthing center yet. The performance capacity of SBAs seems compromised in the project’s districts, which lack a system for onboarding and mentoring new SBAs as well as supervising their existing SBA workforce or planning for the timely succession of retiring ANMs. Health workers without supervision and performance targets may lack incentives and motivation to apply new knowledge and skills. Finally, the leadership of the National Health Training Center reported that it lacks the capacity to respond to all the training needs of the MNCHN/FP programs and districts. DEMAND FOR HEALTH SERVICES The project was designed to increase knowledge, behaviors, and use of health facilities by MDAGs. However, low utilization was observed in all 10 visited facilities, and there is concern that this may be indicative of low perceived quality and low staff productivity. Despite the project’s training on interpersonal communication, the observed behavior of care providers to MDAG women was indifferent. In one case it was not appropriate, when we observed a mother from a marginalized group was forced to clean the floor of the facility with her handkerchief when her child vomited. The correct practice for the ANM would have been to have a janitorial staff clean the floor and assist the mother to comfort her child. All visited facilities lacked proper cleaning supplies adequate for cleaning a health facility and we observed only one facility having a janitorial staff cleaning the floor and it was cone incorrectly. All facilities were dusty, walls were dirty and looked like they had not been cleaned in years, and had dirty linen on examination rooms. In short, they did not look like a welcoming and pleasant place to come that would generate demand. We did not see a demand generation strategy being implemented in the districts visited—particularly to increase use of health facilities by mothers from MDAGs—besides the tracking by CAP/Rs. MDAGs have not been directly reached through other frontline health workers such as FCHVs, and there is little information on progress toward addressing GESI needs at community levels. Increasing demand generation among MDAGs through identification and addressing of GESI barriers remains inadequate. The project reported to have used the following activities to generate demand, but best practices have not been documented: • Promotion of health post activities is reported by HFOMCs • Education of mothers’ groups • Radio Bahas was reported by the District Development Committee (DDC) to be effective but discontinued • SMS texting to adolescents was a one-time event, reaching reportedly 181,577 youth, but no follow up observed at local health facilities or related increases of the demand for services. GIZ was reported to have agreed to evaluate 30 • The school health program was discontinued, but is still talked about by youth • Learning circles implemented by IRHDTC were discontinued after one year From the FGDs, the MTE findings indicate that most mothers remember healthy messages, but early marriage, traditional healing, and in one case, preference to deliver at home, are common behaviors that have not changed in the MDAGs. The project’s SOW requires that it develop a look for ways to improve access to adolescents. In fact, the PMP requires the project to measure the number of youth that know the legal age of marriage. “We discuss different issues such as delivery care, hygiene and sanitation, ANC care, nutrition… food feeding, colostrum milk, newborn care, and chest-to-chest contact to keep the baby warm.” – MDAG FGD “There is no such change in marital age. Children even get married at the age of 14, 15 years. We think that mobile phone had made them easy to communicate.” – MDAG FGD “There are still some people in village who have not visited the health post. They question the need to go there, claiming they delivered all their children at home.” – MDAG FGD FGDs show that youth and adolescents are becoming increasingly aware of their various health concerns through civil society organizations and other stakeholders, but they are not keen to visit health facilities for adolescent sexual and reproductive health (SRH) services. The project reported they have not been able to strengthen FCHV’s abilities to gather information on prevailing GESI barriers. The project has not assisted HFOMCs to establish linkages with existing CBOs, mothers’ and fathers’ groups, and line agencies to address GESI barriers yet. In conclusion, the project lacks a demand generation strategy for MDAGs besides the CAP/Rs in the sentinel sites. Use of health facilities by youth is not monitored by the project or districts and seems not to have improved, since health posts do not consistently attract youth for SRH services. Observed utilization of health facilities is very low, leaving possible idle capacity for demand generation activities. Sustainability options have not been explored for demand generation activities, and best practices have not been transferred to districts to include in their district activity plans. ORGANIZATIONAL CAPACITY BUILDING OF H4L LOGISTICS CONTRACTOR Selected assistance has been provided to Lifeline Nepal, the local NGO that is implementing another contract called H4L Logistics: “H4L has provided two consultants to train our staff, one on leadership and the other on financial management.” –KII H4L Logistics However, we did not find a capacity building plan to develop business skills or systematic capacity transfer to achieve contractor’s organizational growth goals. We also did not find an effective coordination strategy between the two projects. In conclusion, Lifeline Nepal has very knowledgeable and experienced technical staff in logistics. They reported providing technical assistance in logistics, including forecasting and procurement planning as well as logistics management information services (LMIS) to the MoHP. However, it is still a new organization in need of business organizational support and business development assistance to accelerate its growth and impact. USAID/Nepal has a long successful history of supporting local NGOs like the Nepal Technical Assistance Group, a local NGO that has successfully institutionalized and 31 sustained high coverage of Vitamin A. Providing the MoHP a possible strong partner to help outsource LMIS and logistics services could be a way to strengthen MOHP capacity. TO WHAT EXTENT IS THE STRUCTURE OF H4LCORE ACTIVITY SUFFICIENTLY FLEXIBLE TO RESPOND TO THE MNCHN PRIORITIES OF THE MOHP AS OUTLINED IN THE NHSP? National and district staff reported the H4L project is not flexible if requests are outside of the project’s work plan. However, we observed that it does respond to requests, although it takes time, sometimes months, for the project to do so. For example, it responded to the GON’s request to implement the Collaborative Framework in the selected districts, but changing the SOW took more than six months, and harmonizing the daily subsistence allowance (DSA) with GON reportedly took months too. The project also responded to the need for new computers for the National Health Research Council (NHRC), but the computers also took months to procure. The project reported that delays were related to delays in obtaining USAID approval. Lack of funding mechanisms that would allow districts to implement program management and improvements of the quality of healthcare identified by the H4L team has also limited the ability of the project to respond to district needs. It was reported that the project does not respond to changes in district needs, and that timing of the project’s work plan does not allow for the inclusion of district needs identified in the district planning process. Districts reported limited involvement of the H4L project in district work planning, as well as their limited involvement in the H4l work planning. Regarding flexibility to implement the NHSP, the project provided a consultant to help write the NHSS (2015–2020), but has not been active enough to start aligning its activities to support its implementation. We understand the implementation plan is still under development, but we had expected that given H4L’s involvement in the development of the policy and strategy, the project would have aligned its activities to lead the implementation planning and thus provide best practices and evidence to the planning process. The project has supported 25 MoHP personnel in the development of the GESI strategy and incorporated GESI within National Health Education, Information and Communication Center (NHEICC) manuals. However, project staff reported that GESI activities remain unaddressed and underfunded (less than 1% of project funding is allocated to GESI-related activities) when compared to other project interventions due to limited funding. In our review of the project’s work plans, we were not able to perceive continuity to build on and sustain the successful strategies developed by the previous USAID-funded projects, such as treatment of the umbilical cord with chlorhexidine or oral rehydration salts/zinc treatment, which are incorporated in the CB IMNCI package. Training of healthcare providers, implementation of the CB-IMNCI delivery processes, supervision and monitoring of this new package is limited in the target districts, and consequently, coverage and further scaling up remain challenges. In addition, interviews with MoHP directors pointed out that the project’s implementation strategy has not addressed gaps in child health program management, and expansion of an integrated strategy to address GESI issues in maternal health and FP programs and services, or as part of its governance activities. The project has not responded to the evident lack of application Exhibit 13: Proposed VDC Graduation Criteria • HFOMC Functionality Index (5 out of 5) • HFOMC Performance Index (≥4 out of 6) • Timely HMIS reporting (12/year) • Health Facility general readiness index ≥ 60% • 40% institutional deliveries • 45% of eligible couples using modern contraceptives • EPI coverage among eligible children ≥ 90% • Stock out of FP commodities in last 12 months (<8%) • No stock out of tracer drugs in last 12 months (≥ 85%) • Timely submission of LMIS report for prior year (4) 32 of health care delivery guidelines by health workers. Its implementation approach focuses on system inputs but not on the improvement of management or quality of the delivery process, as evidenced by the proposed criteria for VDC graduation (see Exhibit 13). Moreover, the criteria include measures for HMIS and logistics, for which neither the project nor HFOMCs are responsible. The project is responsible for several outcomes not included in the criteria: QI, improving the delivery of services to MDAGs, improving supervision of ANMs, improving the capacity of FCHVs in CB-IMNCI, linking facility and community-based services; working with Suaahara to implement, monitor, and maintain the “Essential Nutrition Actions”; and improving standards of obstetric care in birthing centers. The project was reported to not have found ways to effectively coordinate activities in its SOW with other partners in the 14 districts, such as finding an effective coordinated implementation mechanism with other USAID-funded projects like Sajhedari Bikas, which is also working to strengthen governance, or Suaahara, which is improving nutrition services in the same target districts. The MTE team did not find operational plans or functional linkages, or reports of joint activities to improve synergy and impact of these three USAID-funded projects. We conclude that the project has been able to improve health planning at the local level, but it has not improved the management and delivery of quality, equitable MNCHN/FP services to target populations in priority VDCs and at district level. This means that the project does not have an effective strategy to scale up to the remaining VDCs in the target districts. The project has been able to improve planning capacity, but due to reported contractual limitations, it has not been able to respond to challenges in assisting the GON to manage and deliver quality and equitable MNCHN/FP services. It has, however, strengthened governance and policy development. The project has developed a GESI strategy, which, however, has not been implemented. The project has not started to use its activities to inform the implementation plan of the NHSS yet. Effective functional mechanisms for coordinating planning and implementation activities with other USAID-funded projects and donors working in the target districts reportedly were not observed. We also conclude that the H4L project was designed to effectively strengthen a number of building blocks of the Nepali health system. A contract is a less-flexible mechanism, because the U.S. Government states “what” it needs to be delivered and understands the contractor is an expert in the field and will know “how” to deliver what the contract requires. The project’s SOW includes a long list of “musts” (see annex 1)—that is, things that need to be done to strengthen the Nepali health system. They still remain needs of the Nepali health system and need to get done. For example, the misoprostol pilot needs to be extended to the 14 districts, FP counseling needs to be improved to address missed opportunities, and CB-IMNCI needs to be revitalized to save children’s lives. Certified SBAs need to be posted in every birthing center and need to be supervised to ensure they perform according to standard. The activities included in the SOW of the project are appropriate for an HSS project. However, we believe that the observed lack of progress in some objectives is determined not by the instrument used by USAID, but also by the unanticipated systemic weaknesses that needed to be addressed at national, regional and district level, the earthquakes and border issues, as well as the approach of the contractor, the reduced funding, and organization and distribution of the project team. The observed implementation approach used is that of a governance project and not an HSS project. The MTE team had anticipated the project team to be supporting health systems and programs and assisting to improve the performance of the MNCHN/FP programs. This approach needs to be adjusted to ensure effective strengthening of MNCHN/FP program management and health care delivery. In addition, the project’s PMP, IRs and impact indicators do not allow the project or USAID to monitor and compare progress of improvements in the performance by MNCHN/FP program, districts, and VDCs. 33 RECOMMENDATIONS ADDRESSING HEALTH SYSTEM GAPS IDENTIFIED IN THE PROBLEM STATEMENT AND ACTIVITY BASELINE HSS is part of a continuous and never-ending health reform process6 . No country has achieved health system perfection yet. A recent review of 20 years of health reform of Brazil’s unified health system showed areas of work to be continued, particularly in quality improvement. The United States has recently implemented the Affordable Care Act, and it has taken several years to reach all that it intended to benefit. Nepal is a much smaller country than Brazil or the US, but the geography and the geographic diversity, and social and political challenges it faces make HSS a complex process that may take several project cycles to reach the targets of universal coverage, as set by the SDG 3. Despite this, Nepal has achieved significant progress towards the MDGs and it is expected to continue improving. It is important to develop the capacity for health reform in Nepal. Therefore, we recommend that, in close collaboration with the MoHP and other GON stakeholders such as the MOFALD and the Ministry of Finance, a review be conducted of previous HSS efforts in other countries and a roadmap of HSS for Nepal be developed along the lines of the ideal main phases of HSS described in the HSS phases described on page 10 above and in alignment with USAID’s Vision for Health System Strengthening. This roadmap will help USAID strengthen GON stewardship and ownership of the HSS process. The roadmap will also help USAID to focus and sustain the work of the H4L project, as well as facilitate the design and implementation of future activities. Below are detailed recommendations to help the H4L project address the gaps in the Nepali health system. They aim to help the country get on the road toward a strengthened health system that delivers improved performance and health outcomes. We recommend that these activities be included in the PY 4 work plan. Recommendations to Impact Overall Project Performance 1. The project should prepare its PY4 work plan in alignment with the NHSS and its basic package of services. It should work closely with the Family Health and Child Health Divisions and the target districts to include activities that improve the management of MNCHN/FP programs at national and district levels. These activities will inform the NHSS planning process and initiate implementation of the NHSS in coordination with other USAID-funded activities and other stakeholders. 2. The project has scarce resources and time, and 6 standalone objective. Thus, prioritization is essential in allocating the resources to the objectives. We recommend that the project focus on the deliverables of objectives 4, 5 and 6 in the 14 target districts. In these districts, the project should 6 Suggested readings: 1. Everybody’s business: Strengthening health systems to improve health outcomes. WHO’s Framework for Action. http://www.who.int/healthsystems/strategy/everybodys_business.pdf 2. Monitoring the Building Blocks of Health Systems. www.k4health.org/toolkits/hrh/monitoring-building-blocks-health￾systems-handbook-indicators-and-their-measurement. 3. PEPFAR HSS strategy. http://www.pepfar.gov/about/strategy/ghi/134854.htm 4. Chapter 3. Effective health systems. In “Improving Aid Effectiveness in Global Health,” Beracochea Ed. Springer 2015. 5. Twenty years of health reform in Brazil. https://openknowledge.worldbank.org/handle/10986/15801 6. The affordable care act at 5 years. David Blumenthal, M.D., M.P.P., Melinda Abrams, M.S., and Rachel Nuzum, M.P.H.N Engl J Med 2015; 372:2451-2458. June 18, 2015. http://www.nejm.org/doi/full/10.1056/NEJMhpr1503614 34 include in the PY4 work plan activities to expand its approach to HSS and demonstrate improved performance of MNCHN/FP programs in terms of improved planning and management and service delivery outputs and outcomes and improved coverage by MDAGs in target districts. 3. USAID should work with the project to adjust the project’s IRs and output and outcome indicators in the PMP. Outputs and outcomes must be related to improved performance of MNCHN/FP programs by VDC and district and MDAG. Also, the period of reporting must be adjusted to allow timely data collection and monitoring of project progress and facility and district performance. For example, semiannual reporting which is due in July should cover data from October to March to allow for the HMIS data to be gathered and checked. See Annex 6 for proposed changes to the PMP indicators. The PMP must also have indicators of quality improvement in the delivery of MNCHN/FP services in the target districts. 4. The quality of delivery service and obstetric care requires immediate attention. The project must address the quality problems in the birthing centers, work with HFOMCs to paint the birthing centers with washable paint, and purchase appropriate cleaning supplies, introduce a certification system of SBAs and assist to set standards, improve infection control and cleaning and hygiene practices at birthing centers and health posts in target districts. 5. HFOMCs should be supported to address GESI issues and the mitigation of social, cultural, economic, and institutional barriers to health care access making use of existing tools. HFOMCs should be assisted to collaborate and coordinate with local civil society organizations and relevant stakeholders. 6. The project should also assist districts to improve the financial transparency and accountability by disclosing the financial statements. Client's perspective issues such as cleanliness of the facility, behavior of the care provider, privacy and confidentiality, availability of care providers and medicine should be discussed and decisions made to correct problems during the social audit to make the management and care providers accountable to the clients. 7. Current government efforts on Gender Responsive Budgeting and Gender Auditing should be taken into account by H4L and provide assistance in this regard for the next planning and budgeting cycle that will include the implementation of the NHSS. 8. Coordination should be improved both at central and district levels by increasing the frequency of the coordination meetings between the H4L and the MOHP leaderships. The project should also involve the respective D/PHOs in the PY4 and PY5 annual project planning to address the local needs effectively. The project should also consider joint district level planning and monitoring to develop the mutual accountability and release the additional synergy. 9. The project should assist MoHP to map donor contributions to implement the NHSS, and ensure effective coordination of its activities to avoid duplication and ensure synergy with other stakeholders. 10. The project should align its PY4 and PY5 activities to leverage the work of other USAID-funded projects, mainly Suaahara due to the important connection between nutrition and health, and Sajhedari Bikas to find ways to sustain its governance achievements. Joint district planning and monitoring should be done to achieve additional synergy. Indicators of collaboration such as joint implementation of the “essential nutrition actions” and local governance must be included in the PMP. Recommendations to Address Priority Gaps in Local Health Governance 35 1. The project should take an inventory of its strengths, such as its successful experience revitalizing HFOMCs, creating QITs, and engaging FCHVs, trained health workers, and other stakeholders, to empower VDCs to achieve performance targets and outcomes in terms of improved quality and equity of service delivery in the 14 target districts. The project should assist to set and achieve performance targets and thus enable districts to transparently compare and analyze performance across VDCs. In this way, districts will be able to identify best practices, and help VDCs share them and learn from each other. 2. USAID should consider exploring government-to-government (G2G) support for a micro-matching grant program for HFOMCs with MDAGs, until the flexible fund is introduced in the remaining 12 districts (except in Dang and Jumla where a flexible fund is available). This funding would be used to achieve QI performance goals and coverage targets in MDAGs. This micro-matching grant mechanism could also include districts and test a decentralized financing grant mechanism for future policy development. Appropriate PMP indicators will need to be developed to monitor the effectiveness of this decentralized funding mechanism. 3. The project should implement tools for the HFOMCs to monitor frontline health workers regarding respectful treatment and other GESI barriers to health care access. 4. The project should time its PY 4 planning to align with the district planning that starts in March to assist districts to set and achieve measurable improvement of district performance with mutual accountability indicators in the form of a “district scorecard.” The district scorecard will help FH and CH program managers to compare district performance and improve use of data and decision making. 5. The project should document lessons learned in revitalizing HFOMCs so other districts can replicate the process. Recommendations to Address Priority Gaps in Quality Improvement 1. The project should improve MNCHN/FP program management and results at VDC and district levels by addressing GESI policy and local challenges identified in their district reviews and by assisting districts to assess and improve the quality of health care delivery in health posts and birthing centers. 2. The project should assist to ensure all birthing centers are staffed by certified SBAs and they meet quality standards to provide obstetric and neonatal care. The project should assist the districts to set up performance targets for every ANM and health care provider in every health facility. In this way, each staff will be responsible for reporting coverage rates of a number of families in their assigned wards. The health provider will then report to the HFOMC the number of fully immunized children, number of children with the right weight for age, number women that are up to date in their ANC visits, and the number of MWRA that have FP needs met. Based on the facilities visited by the MTE team, we believe each healthcare provider would be responsible for about 200-300 patients/clients. This is a very reasonable number in terms of productivity. 3. The project should assist the districts to assess the performance of health posts and set performance targets for facilities and staff to be monitored and supervised regularly. The existing self-assessment tool over scores the quality of care thus, an independent external team has to assess the quality of care of the health facility. Staff from one facility can assess another. An alternative approach would be mobilization of the public health nurse and her team to validate the scores and help with decision-making and follow-through of recommendations and improvements. 36 4. The project should assist the districts to improve quality of health care delivery of MNCHN/FP services in priority facilities through the use of a facility scorecard and appropriate MNCHN/FP accreditation standards. 5. With additional funding from a G2G or other appropriate mechanism, the project should assist districts to use all their staff to develop and implement an effective supervision program in all its VDCs, starting with the lowest performers and building on the lessons of top performers or “positive deviants.” 6. The project should assist districts to work with QITs to implement and ensure correct cleaning and infection prevention guidelines. Monitoring of facility maternal and neonatal infection rates by the QIT and HFOMCs must be implemented in every facility. 7. The project should work with districts to develop and implement a plan to expand CB-IMNCI in all 14 districts, particularly MDAGs. Recommendations to Address Priority Gaps in Data Use 1. The project should develop a simple system to improve accuracy and timeliness of data collection and use at VDC level. Improving data collection and use at local level that is the basis of the pyramid of the HMIS will improve quality of data as a whole. The choice of the right denominator to ensure indicators are calculated correctly must be part of this activity. The project must enable districts to do data quality controls and correct data errors such as indicators above 100% and acting upon data that shows disparities such as a higher low birth weight or lower coverage rates that require to be addressed. 2. The project should use the data and experience of VDCs and districts that are “positive deviants” in the use of data to document best practices and promote “champions” in the use of data. 3. The project should assist districts to develop GESI indicators to monitor gender violence, early marriage and pregnancy, traditional beliefs, cultural barriers, and factors preventing access to services. 4. The project should assist the NHRC to conduct research and ensure GESI-related data are available to be used in planning and managing health services and monitoring equitable access as part of the plan to implement the NHSS. 5. The project should assist D/PHOs to select a small number of performance indicators to use data to monitor MNCHN/FP program performance (scorecards) and meet MoHP top performer’s criteria. All project-supported districts should meet these criteria. Recommendations to Decide on the Sustainability of CAP/Rs 1. The project should evaluate this project intervention in comparison to other options, such as expanding the role of ANMs in tracking, the role of FCHVs, and the MoHP’s community midwives program. 2. The pregnant mothers tracking app is free of charge, easy to use, and efficient. Smart phones are becoming increasingly more available and many ANMs and FCHVs were observed to have them. The project should work with other donors and look for options to scale it up to other VDCs as well. 37 3. The project should document best practices in tracking mothers’ ANC, delivery, postnatal care (PNC), and FP appointments and make recommendation to the GON. Recommendations for Improving the Capacity of Health Workers and District Staff 1. The project should document best practices, streamline “how to” procedures through the creation of operations manuals, that will help transfer skills to district and facility staff. Manuals that would help districts to replicate and expand the project’s achievements may include: “How to revitalize a HFOMC,” “How to assist a VDC to plan for health services,” “How to form a QIT and improve quality of care,” “How to coach healthcare providers,” and “How to track antenatal care appointments and improve maternal health care.” These manuals, as well as other tools developed by H4L, may be uploaded on the MoHP’s website and made available digitally to all districts. 2. Each district has about 50 VDCs to manage and needs of MDAGs in the target districts need to be consistently met. For that reason, we recommend that the project expand the number of priority VDCs supported to 20+ per district in PY 4 and to 30+ in PY 5. This will require that the project streamline its “VDC strengthening assistance process” during PY4 based on the results of the first 10 VDCs and use that to add another 10 in PY4. It is expected that the project will have streamlined their process of assistance to VDCs that will allow it to assist districts to bring up to par at least another 10 more VDCs in the first six months of PY5. The project should aim at assisting districts to bring all the VDCs in the 14 districts to the same range of quality standards of performance by the end of the project. 3. The project should build on the experience of IRHDTC to train FCHVs in MDAG wards on CB￾IMNCI, essential nutrition actions (ENA) and other relevant MNCHN/FP services. The prime should work with the local subcontractor to expand its capacity to improve IMNCI and CB-IMCI practices and outcomes in the 14 target districts. 4. The project’s pre-service education expert should give training demonstrations and train trainers at the 4 ANM schools and demonstrate measurable improvement in the performance of these schools from the baseline study. 5. The project should consider sponsoring women from MDAGs to attend local ANM schools. 6. The project should coordinate with existing government line agencies (e.g. WCDO), CSOs, MDAGs’ networks, and other USAID-funded project such as Suaahara and Sajhedari Bikas to share best practices and assist the Family and Child Health Divisions to develop and undertake training for all health facility personnel on GESI, with specific focus on respectful maternity care services, pertinent GESI barriers that need to be addressed, and knowledge on referral points for gender violence, HIV/AIDS, STIs, malnutrition, and obtaining legal support, psycho-social counseling, and help set procedures for police support and referral of victims. Upon training, the project must assist districts to work with HFOMCs to monitor GESI indicators and to set performance targets for every healthcare provider to be responsible for a number wards and to conduct screening for GESI issues in all their assigned communities. 7. The project should organize exchange visits of SBAs between the low- and high-performer health facilities, and coaching by senior top performing and accredited SBAs to help low-performing SBAs perform according to quality standards. The project should work with the districts to institutionalize this practice of sharing and coaching among district staff in the next year’s districts’ budget. 8. The project should assist districts to develop induction training for new ANMs to demonstrate correct and quality SBA skills before posting them. The project should assist NHTC to update its database to have reliable data on who has been trained and who has not. 38 9. The project should coordinate work with DfID’s new pay-for-performance £80 million project to ensure project districts meet the proposed four performance targets: vaccination, contraceptive prevalence rate (CPR), institutional deliveries, and growth monitoring of infants. In this way, the project will be able to assist the 14 target districts to leverage this funding to improve the provision of quality and equitable MNCHN/FP services. Recommendations to Support the Local Logistics Contractor H4L should support the work of Lifeline Nepal by jointly developing and implementing a business development plan that meets Lifeline’s goals of serving the GON. ENSURING THE STRUCTURE OF THE H4LCORE ACTIVITY IS SUFFICIENTLY FLEXIBLE TO RESPOND TO THE MNCHN PRIORITIES OF THE MOHP, AS OUTLINED IN THE NHSP Recommendations 1. USAID should adjust the PMP indicators to measure and monitor the project’s progress towards revised outcomes to help USAID and the H4L team monitor the strengthening process of the health system. 2. The project should adjust PY 4 project planning of its district activities to coincide with the district planning cycle. 3. USAID and the H4L project should meet semiannually with MoHP authorities to report on the project’s progress and jointly review GON needs and requests to include in the project’s work plan. Scheduling requests for support in this manner should reduce the number of unanticipated requests. 4. The project should establish mechanisms to coordinate of implementation activities in the target districts with other USAID-funded project staff in the districts, to have an efficient U.S. Government response. 5. Health care need is context-specific; therefore, centrally planned project activities will be less relevant at the district level. Thus, for the implementation of the NHSS, the project should assist to develop tools and best practices so that the central level allow more operational flexibility to be devolved to the district level. 6. The project should realign H4L budgeting to ensure gender-responsive budgeting is appropriately taken into account by the project and in this way, model and assist MoHP divisions to do the same. 7. USAID should work with the project to ensure adequate project funding for rolling out GESI mainstreaming activities at central, district, and community levels, to ensure that technical aspects of health service delivery align with GESI mainstreaming and integration activities. Unless key issues facing service-seeking MDAGs, women, children, and youth (such as GBV, malnutrition, migration impacts, etc.) and service providers (security concerns, knowledge on GESI matters, etc.) are taken into account, little change may be visible at the end of the project. 8. A separate GESI unit within H4L, with enough staff and appropriate funding, should be established to take into account and advance GON’s GESI priorities and address and document best practices 39 at community levels (social audits, MDAG representation in HFOMCs, etc.) that will inform further policies and future MNCHN/FP program management at district level. 40 ANNEX 1: EVALUATION STATEMENT OF WORK STATEMENT OF WORK, Revised December 29, 2015 MIDTERM PERFORMANCE EVALUATION OF HEALTH FOR LIFE CORE This Statement of Work is to provide the objective and scope of the mid-term performance evaluation of USAID/Nepal’s Health for Life Core activity hereafter referred as “H4Lcore”. Through H4Lcore USAID provides technical assistance to Government of Nepal (GoN) to plan, manage and deliver, quality and equitable, family planning and maternal, newborn, and child health services with a greater focus on health systems strengthening. Awarded to Research Triangle Institute (RTI) and its partners on 17 December 2012, H4Lcore is a five year USAID/Nepal’s bilateral project providing support to the Ministry of Health and Population (MoHP). H4Lcore is a health system strengthening project with seven distinct but interlinked objectives that requires working with multiple stakeholders at national and sub-national levels. In the first two years, the project focused on 14 districts of the Mid￾Western and the Western Development Regions of the country. In the third year, the project was expanded to work in four additional districts for governance activities and in ten districts for earthquake recovery activities. H4Lcore is a major departure from how USAID designed its health systems strengthening activities in the years past. The predecessor of H4L, the Nepal Family Health Program I (2002-2007) and II (2007-2012) placed more emphasis on supportive supervision and facilitated GON’s stewardship and oversight roles (Refer to NFHP Qualitative Evaluation 2007, NFHP Quantitative Evaluation 2007, NFHP II Final Evaluation, July 29, 2011). NFHP I and II were designed to support the Government of Nepal (GoN) in meeting the Millennium Development Goals (MDG) of reducing child deaths and maternal deaths. Nepal made significant progress in improving skilled attendance at births from 18 percent to 36 percent from 2006 to 2011 as measured by Nepal Demographic and Health Surveys (NDHS). Infant and under five deaths declined from 64 to 46 and from 91 to 54 per 1000 live births respectively from 2001 to 2011 and neonatal deaths from 36 to 33 per 1000 live births (NDHS 2001 and 2011). The post MDG- Sustainable Development Goals (SDG) targets reducing maternal deaths to less than 70 per 100,000 live births and newborn deaths to less than 12 per 1000 live births by 2030. USAID’s future activities to support GoN should effectively support health system strengthening to meet these goals. H4L tries to have USAID lead from behind, with more emphasis on building capacity of local health facility operation and management committees (HFOMCs) and creating the enabling environment to tap resources from local bodies for quality family planning, maternal, newborn and child health services. The H4L project is in its third year of implementation. A significant amount of time during the first year was devoted to developing project strategies, capacity building plans and work plan and the actual implementation in the field started in earnest towards the later part of the first year. The evaluation will be conducted and managed through USAID Nepal’s Monitoring, Evaluation and Learning (MEL) contract managed by CAMRIS International. Project Title: Health for Life Core Award Number: AID-367-C-13-00001 Project Period: Dec 2012- Dec 2017 Funding: USAID Total Estimated Cost: $27,981,417 Implementing Partners Prime: RTI Subs: Jhpiego and Integrated Rural Health Development and Training Center (IRHDTC) 41 1) PURPOSE OF THE MID-TERM PERFORMANCE EVALUATION: The purpose of this evaluation is to assess the effectiveness of H4Lcore towards meeting its objectives, document how H4Lcore is making a difference in the health systems, and recommend any changes that could help to improve H4Lcore’s performance in the latter half of the project. The secondary objective of this mid-term evaluation is to make recommendations on changes to the project model for health system strengthening with focus on achieving SDGs. USAID will use the findings and recommendations to make changes to the H4Lcore activities best utilize the remaining years of the project and also share lessons learned with the GoN and other stakeholders. The findings of this evaluation will be utilized in designing the follow-on project. 2) H4LCORE INFORMATION The purpose of H4Lcore is to provide technical assistance and other support services to strengthen the GoN’s capacity to plan, manage and deliver family planning, maternal, newborn, and child health services in an equitable and quality manner. The H4Lcore builds on accomplishments gained from USAID/Nepal’s health sector investment in the past, with a greater focus on health systems strengthening and measuring progress in systems improvements. The principal development hypothesis underlying this activity is “the GoN’s capacity to plan, manage and deliver high quality and equitable family planning, maternal, newborn and child health services can be greatly improved with cross-cutting health system strengthening interventions”. H4Lcore interventions directly address key health system constraints in the following areas: local health systems governance; data for decision-making and evidence-based policy development; human resources; quality improvement systems; logistics systems, and; knowledge and behavior change. 2.1 Goal and Objectives The following is a summary of the H4Lcore contract scope of work as modified February 21, 2014. The full scope of work is attached in Annex 1. The goal of H4Lcore is that by 2017, the MoHP will be fulfilling its stewardship role in working with counterparts at local levels to continually build structures and systems that deliver high-impact, equitable services to citizens. These systems and structures will both enable and require a decentralized and accountable MoHP to practice good governance at national, regional and local levels; make decisions and allocate resources based on evidence; routinely collect timely and accurate information for monitoring and management; and provide even the most vulnerable and marginalized groups with quality-assured appropriate and respectful services. Objective 1: Improve health system governance of district health offices and sub-district level health facilities. Objective 2: Develop and implement national evidence-based health policy. Objective 3: Strengthen national level stewardship of the health sector. Objective 4: Institutionalize nationwide system for quality improvement. Objective 5: Improve capacity of district and local health workers and community volunteers to deliver quality family planning, maternal, newborn and child health, and nutrition services. Objective 6: Improve knowledge, behavior and use of health services among target populations. Objective 7: Strengthen capacity of USAID’s contractor for a logistical support activity 2.2 Indicators H4Lcore contributes to the following high-level indicators: • Under-five mortality rate • Infant mortality rate • Neonatal mortality rate • Maternal mortality ratio 42 • Total fertility rate • Health information system performance index • Health service readiness index • Disparity between advantaged and disadvantaged caste/ethnicity groups quintiles. Indicators for monitoring H4Lcore’s progress include the following: • Health institutions carrying out Social/Public Audits • Resource mobilization by HFOMC • Clinical staff position filled at Health Facilities • Health information system performance index score • Analytical district reviews • Data use at Health facilities • Health service readiness index • Disparity between richest and poorest quintiles • Women protected from Post-partum hemorrhage (PPH) • Skilled Birth Attendance at delivery • Contraceptive Prevalence Rate (CPR) • Auxiliary Nurse Midwives (ANMs) with proficient skills • Stock outs of FP and tracer commodities • Technical and Organization Capacity Score of the H4L Logistics contractor 2.3 Capacity Building Plan/Approach The Contractor must apply a systematic approach to strengthening GON systems and institutionalizing best practices at the local, district and national levels. The unifying theme of all seven objectives is that of capacity development of the public health system, measuring and monitoring progress, and making the necessary change along the way to ensure sustainability. 2.4 Geographic Focus H4Lcore maintains a national and sub-national focus to provide TA pertinent to the seven objectives. The majority of the objectives have national level inputs in quality improvement, governance and capacity building. Objectives 1, 5 and 6 are implemented in the 14 project districts in the Western and Mid-Western Development Regions and highlighted below in Map 1. In late 2014, MoHP requested H4Lcore to provide TA in implementation of Collaborative Framework (Objective 1) in four other districts – Jhapa, Chitwan, Kaski, and Kailali. H4Lcore is providing TA to these four districts since FY 2015/16. In addition, H4Lcore will also roll out QI system (Objective 4) in the four districts. Starting July 2015, H4L also started working in ten of the 14 Earthquake affected Health for Life Core Project Districts for Evaluation 43 districts7 to coordinate governance of health recovery and reconstruction activities. However, this evaluation will not assess H4Lcore’s TA to these new 14 districts. The total population covered by the 14 core districts is approximately 4.3 million which is nearly 15% of the national population, as well as about 630 health facilities and 8,744 female community health volunteers (FCHV). 2.5 Strategy to Reach Hard-to-Reach Populations Towards end of year two, H4Lcore analyzed marginalized/disadvantaged group (M/DAG) population data to identify 140 village development committees (VDCs), ten in each district to focus their intensive technical assistance which covers 24% of the total VDCs in project districts. These VDCs have relatively fewer resources and most have high concentrations of M/DAGs. Additionally, apart from M/DAG concentration, H4Lcore is providing intensive TA to VDCs with low service utilization. Two to three high￾priority VDCs in each district have been chosen as sentinel sites, where information is routinely collected and results monitored at facility and community levels. Sentinel sites in each district have been selected carefully to be representative of high-priority VDCs. Mobile-based tracking of pregnant women in sentinel sites is in pilot stage since March 2015. H4Lcore has partnered with HFOMCs in these VDC to mobilize Community Action Promoters and Researchers (CAP-R) on a progressive cost sharing model. 2.6 Collaboration with Other Projects Collaboration with other USAID partners and activities is essential for H4Lcore to achieve the scale and scope of the aforementioned objectives. H4Lcore works closely with relevant USAID implementing partners, including Suaahara, KISAN, Sajhedari Bikas, Hariyo Ban, Ghar Maa Swasthya, and Saath-Saath to align work plans and discuss synergies between activities and approaches. H4Lcore also requires monitoring ongoing donor programs in the same or similar technical and geographic areas of operation and seeks opportunities for collaboration and complementarity. 3) EVALUATION QUESTIONS The evaluation should include the following two questions: 3.1 How is H4Lcore addressing health system gaps identified in the problem statement and activity baseline? 3.2 To what extent is the structure of H4Lcore activity sufficiently flexible to respond to the MNCH priorities of the MoHP as outlined in the Nepal Health Sector Plan II, 2010-2015? While answering the evaluation questions the evaluator should consider strengths, gaps, challenges, potency of interventions to scale up etc. 4) EVALUATION DESIGN AND METHODOLOGY The evaluation design must consist of quantitative and qualitative methods that provide for a strong analysis to address the evaluation questions. The design must describe in detail what information will be collected for answering each question, what method will be used to collect these information, who will provide the information and how the information will be analyzed to arrive at findings, conclusions and recommendations. The first two evaluation areas are related to H4Lcore’s performance assessed against the objectives, expected results, and the tasks and deliverables as stipulated in the Contract. These results will be useful to USAID, H4Lcore and also to MoHP and other donor partners. The third evaluation area is 7 Kavre, Sindhupalchok, Rasuwa, Nuwakot, Ramechhap, Gorkha, Makwanpur, Dhading, Sindhuli, Dolkha 44 mostly related to the internal management system of the project and will be used for USAID’s internal use only and to provide recommendations to H4Lcore. 5) EVALUATION TIMELINE SN Tasks Week 1 2 3 4 5 6 7 8 9 1 0 1 1 1 2 1 Sign Contract X 2 Develop instruments for evaluation X 3 Desk Review X X 4 Conduct a team planning meeting with USAID and H4Lcore staff X 5 Submit final evaluation plan X 6 Finalize instruments for the evaluation X 7 Collect evaluation data in Kathmandu and outside at the project districts and sites X X X X 8 Analyze data, prepare and deliver presentation of preliminary findings & conclusions X X X X 9 Submit first draft evaluation report to USAID for Review X X X 10 Finalize report X SECTION C - STATEMENT OF WORK In this section, the MTE team highlighted in green the most important factors of the project design, in yellow are the project indicators and in blue are the project deliverables and tasks or “musts.” The team assessed most of them and have marked with “” those “musts” for which we found evidence of progress, and with a “” those for which progress has not been as much and requires attention and adjustment. Given the limitations of the MTE, we may have not gathered all the evidence and will welcome corrections. C.1 PURPOSE AND INTRODUCTION The purpose of the Contract is to provide technical assistance and other support services to strengthen the Government of Nepal’s (GON) capacity to plan, manage and deliver, quality and equitable, family planning and maternal, newborn, and child health services. The Health for Life Core (H4Lcore) activity builds on accomplishments gained from USAID/Nepal’s health sector investment to date with a greater focus on health systems strengthening and measuring progress in systems improvements. The principal development hypothesis underlying this activity is that the GON’s capacity to plan, manage and deliver high quality and equitable family planning, maternal, newborn and child health services can be greatly improved with cross-cutting health system strengthening interventions. H4Lcore interventions, described in Section C.2, directly address key health system constraints in the following areas: local health systems governance; data for decision- making and evidence-based policy development, human resources; quality improvement systems; logistics 45 systems; and, knowledge and behavior change. In addition to these areas of focus, the Contractor will work in the fourteen H4L project districts defined in Section C.3.3 (“14 project districts”) to improve access to, and knowledge of, maternal newborn, child health and family planning services. USAID’s investment in these 14 project districts will complement USAID’s other activities and serve as a learning lab for nation- wide systems improvements. The MoHP has identified six demonstration districts for the implementation of local health governance strengthening activities known as the Collaborative Framework, a joint agreement between the MoHP and Ministry of Federal Affairs and Local Development (MoFALD). Two of the six districts are within H4L core districts and four are outside H4L core districts (See map C.3.1). The Contractor, in collaboration with other partners, will support the implementation of the Collaborative Framework in all six districts. The Contractor will also support service delivery quality improvement activities in these six districts. The H4Lcore activity will complement the GON National Health Sector Program II (NHSP II, 2010- 2015), and embrace new US government initiatives and reforms, including the GHI and the USAID Forward procurement reforms (see H4L reference documents under: http://nepal.usaid.gov/downloads/all-downloads/category/1-health-and-family-planning.html). The Contractor and USAID will contribute to the following shared vision: By 2017, the MOHP is fulfilling its stewardship role in working with counterparts at local levels to continually build structures and systems that deliver high-impact, equitable services to citizens. These systems and structures will both enable and require a decentralized and accountable MOHP to practice good governance at national, regional and local levels; make decisions and allocate resources based on evidence; routinely collect timely and accurate information for monitoring and management; and provide even the most vulnerable and marginalized groups with quality￾assured appropriate and respectful services. C.2 OBJECTIVES AND INDICATORS The Contractor will provide technical assistance and support to the GON and will design and deliver activities that contribute to improvements in the following high-level indicators: • Under-five mortality rate • Infant mortality rate • Neonatal mortality rate • Maternal mortality ratio • Total fertility rate • Health information system performance index • Health service readiness index • Disparity between richest and poorest quintiles. The success of the Contractor will be directly measured against the successful delivery of activities that support the following objectives: Objective 1: Improve health system governance of district health offices and sub-district level facilities Objective 2: Develop and implement national evidence-base policy Objective 3: Strengthen national level stewardship of the health sector Objective 4: Institutionalize nation-wide system for quality improvement Objective 5: Improve capacity of district and local level health workers and community volunteers to deliver high quality FP/MNCH and nutrition services 46 Objective 6: Improve knowledge, behavior and use of health services among target population Objective 7: Strengthen capacity of USAID’s contractor for a logistical support activity The following sections provide more information about the background and aim of activities for each objective, including deliverables, indicators, and capacity building benchmarks. For each of the objectives listed above and described below, the Contractor must develop, in coordination with relevant GON counterparts, a life-of-project Capacity Building Plan that includes a description of current health system capacity in each area, capacity targets for the year and measurable benchmarks against which to track success, including benchmarks from the illustrative lists below. The Contractor will include new and or updated annual capacity building targets and benchmarks in subsequent annual work plans. All capacity building benchmarks described in each Objective section below should be considered in the Capacity Building Plan (see Section C.4.1.21), which is subject to COR approval. Achievement of capacity building objectives will be another important area against which the Contractor will be evaluated. C.2.1 Objective 1: Improve health system governance of district health offices and sub-district level health facilities. C.2.1.1 Overview The central aim of this objective is to build organizational capacity within the MOHP to support decentralization and local governance of health services. For this objective the Contractor must perform two inter-related activities: (1) provide technical assistance and support to district officials to expand and strengthen Health Facility Operations and Management Committees (HFOMCs) composed of community leaders and citizens, including women, disadvantaged castes and minority groups and (2) provide technical assistance and implementation oversight for the expansion of MOHP’s local health governance activities now referred to as the Collaborative Framework, a joint agreement between the Ministry of Health and Population (MoHP) and Ministry of Federal Affairs and Local Development (MoFALD) to build the capacity of local government officials to manage health care services. HFOMCs empower community members to support the GON and health providers to improve their services and also offer an avenue for communities to actively participate in the improvement of health facility operations and thus their own health status. Within HFOMCs, the Contractor must develop mechanisms in a participatory manner that capture clients’ perspectives on quality of and access to care, barriers to access, and provider bias. Quality improvement supported by the Contractor must guarantee confidentiality and informed consent and ensure providers are responsive regarding quality and respective care to all clients. USAID/Nepal’s financial and technical support was instrumental in developing the local health governance strengthening program piloted in 2010 and revised to become the Collaborative Framework in 2013. This kind of policy scale-up and programmatic input will be even more crucial as the country moves forward with decentralization and devolution of health programs over the next five years. The H4Lcore Contractor must build on the work of NFHP II to strengthen the role of local governance in building a stronger health system. The Contractor must adapt its focus of technical and strategic assistance based on Government of Nepal health governance policies and USAID’s development strategy for Nepal. C.2.1.2 Approval 47 During the annual work planning process (see Sections C.7 and F.4) both the Contractor and USAID may propose indicators and capacity building benchmarks either from the lists below or proposes others for inclusion in the final PMP, subject to COR approval. Deliverables and Tasks: • HFOMCs The Contractor must provide technical assistance to and support coordination of the District Health/Public Health Officer (D/PHO), District Development Committee (DDC), Village Development Committees (VDC), and HFOMC representatives to develop criteria for assessing the functionality of HFOMCs, including setting targets and identifying steps for the assessment process. These efforts must support increased acknowledgment of HFOMC contributions to improved health services and must improve accountability of the health system. In order to ensure transparency and strengthen downward accountability, input must be sought from communities through existing local consultative forums and local media and radio. • The Contractor must provide technical assistance and support to the D/PHO to develop a district- specific health plan and VDC health plan that includes HFOMCs. This technical assistance must include guidance on how the D/PHO can play a key role to support and encourage HFOMCs and how to improve or re- energize those that are dysfunctional. District￾level plans should also be incorporated into annual work plan and budgeting to ensure appropriate budget allocations. • In turn, the Contractor must provide support and technical assistance to encourage the D/PHOs to educate their staff about the advantages of having active committees. The Contractor must develop a presentation that can be given by the D/PHO on the positive role that HFOMCs can play in improving services; for example, a before and after look at health facilities with strong HFOMCs. • -The Contractor must provide technical assistance to D/PHOs and MOHP leadership to institutionalize best practices among HFOMCs and/or other civil society organizations that effectively respond to needs identified by communities, especially to increase access for marginalized and disadvantaged (M/DAG) and hard-to-reach populations. • -The Contractor must work with D/PHOs and provide technical assistance to develop best practices and case studies to share at annual district, regional and national review meetings. • -The Contractor must prepare policy recommendations in the form of briefs and case studies reflecting lessons learned from field experience for the MOHP/DOHS, MOFALD, NPC and EDPs. Recommendations must be disseminated through local media, technical working groups and the JAR meetings. Lessons learned must further be incorporated into the “Local Health Facility Management and Operation Guidelines (Revised), 2063.” • -The Contractor must provide technical assistance and collaborative support to the D/PHOs and DDCs to plan for expansion of the functional HFOMCs and required orientation, training, and/or refresher training within the 14 project districts, identified in consultation with the DOHS. • The Contractor must take a district-wide approach but should prioritize support to VDCs that have high levels of poverty, poor health indicators and communities with large numbers of arginalized/disadvantaged populations. • The Contractor, in collaboration with other partners, must also provide technical assistance to form, assess, strengthen and evaluate capacity of prioritized HFOMCs in six demonstration districts for implementation of collaborative framework activities. • -The Contractor must provide technical assistance to the MOHP to highlight and publicize the accomplishments, contributions and lessons learned from the HFOMCs. • The Contractor must provide technical assistance and training to increase and build the awareness of HFOMCs on their role in disaster preparedness and response. The Contractor must include disaster risk reduction in the policy and training documents and guidelines related to HFOMCs and track progress in this area. To build awareness of HFOMCs, the Contractor must collaborate with the D/PHO, DDC and District Disaster Management Committee, as the GON-identified leads for disaster management and response. 48 Deliverables and asks “Governance Program • -The Contractor must rigorously evaluate the progress and lessons learned from the pilot districts and apply this information to support the MOHP, MOFALD, DDCs and D/PHOs to plan program scale-up. • In order to identify policy and implementation challenges, the Contractor must organize and facilitate a review workshop to identify lessons learned with other LHGSP stakeholders, such as GIZ, DFID/NHSSP, WHO and Plan Nepal. At the conclusion of this workshop, the Contractor must produce a brief report with lessons learned, recommendations and agreed upon next steps and institutions/individuals responsible. • The Contractor must provide technical assistance to the MOHP, MOFALD and other stakeholders at the national level to expand local health governance strengthening activities to all 14 program districts, including defining specific benchmarks to track progress in each district for achieving the specific objectives of the national program. • Contractor must play lead role in designing the Collaborative Framework for Local Health Governance and Implementation Guidelines. • -Contractor must focus on health systems strengthening: devising and adapting context dependent innovative approaches to increase access to and coverage of health services through the promotion of evidenced- based participatory health planning and budgeting; promoting multi-sectoral engagement in establishing health as a prioritized local development agenda; managing social determinants of health; creating synergy and better leveraging and utilizing of local resources; and strengthening transparency and accountability. • Contractor must support the MoHP in documenting and disseminating best practices and learning through different forums including the media. This a challenging reform agenda in the health sector of Nepal and there is much to be learned from other countries in nearby regions that have moved forward through similar decentralized approaches. In this context, study tours for selected government officials will encourage institutionalizing such local health governance approaches in a sustainable manner • Together with other partners, the Contractor must provide technical assistance to implement Collaborative Framework activities in six demonstration districts identified by the MoHP (see C.3.3). This will include the formation and strengthening of the District Health Governance Strengthening Task Force (DHGSTF), development of health periodic and/or annual plans and implementation, mobilizing flexible health grants and other resources in meeting health needs of disadvantaged and marginalized groups and monitoring and evaluation of these activities. • The Contractor must document the progress of this expansion and share the findings at the district, regional and national levels with the MOHP and other stakeholders. • The Contractor must provide technical assistance to the MOHP and other stakeholders in the systematic analysis of field experience from pilot implementation districts to capture lessons learned from the pilots. • The Contractor must produce case studies based on implementation experience, as well as policy and technical briefs to be shared with other stakeholders through the EDP Forum and the JAR. • The Contractor must collaborate with NHSSP to expand the GON’s Transaction Accounting and Budget Control System (TABUCS) in H4L districts and to pilot e-AWPB (electronic Annual Work Planning Budgeting) for the D/PHO to use in district-level health annual work planning and budgeting.” C.2.1.3 Indicators or proxies as approved by the COR • Health facilities in focus districts, including the demonstration districts having functioning 49 HFOMCs • HFOMCs generating funds and donations for their health facilities • -Women from marginalized or disadvantaged groups raising issues/concerns in HFOMC meetings in the past 6 month period • DDCs receiving central government block grants for LHGSP activities • -Marginalized and disadvantaged clients of health facilities as a ratio of Marginalized/DAG population in health facility catchment area who visited health facility in last 30 days in HFMSP area • Health facilities that have conducted social audits following GON Guidelines in the last 12 months • Health facilities that have health plans addressing local health needs and priority given to equitable access to services, quality and coverage • -Districts that have comprehensive district-level health plans C.2.1.4 Capacity Building Benchmarks or proxies as approved by the COR (year of completion) • HFOMC functionality criteria finalized and used in HFOMC performance assessments (Y1 and 2) • District capacity building plans to strengthen HFOMCs and health facilities completed and approved (Y1 and 2) • -VDCs and D/PHOs continuously monitoring HFOMC and facility performance and providing feedback (Y2-5) • HFOMC manuals/guides updated to reflect documented best practices and in use by HFOMC members for operations (Y2) • MOHP disburses flexible grants to VDCs on time (Y2-5) • Governance Program Technical Coordination Team and Steering Committee and Health Sector Decentralization Policy Forum meet regularly and perform advisory/oversight functions (Y1-5) C.2.2 Objective 2: Develop and implement national evidence-based health policy. C.2.2.1 Overview MOHP and stakeholders have identified a serious gap in Nepal between research findings and policy. These gaps must be addressed through improved and systematic communication between research bodies and decision- makers at the local and national levels1. The H4Lcore Contractor will provide technical expertise and evidence from the local level to inform national health policy decisions. Through the use of technologies such as Geographic Information Systems (GIS), mobile devices, and web-based reporting; the H4Lcore Contractor must establish and strengthen mechanisms for evidence-based, timely decision making by policy-makers. The H4Lcore Contractor must also continue USAID’s widely recognized technical assistance in health sector strategy development and implementation in the major areas of: maternal, newborn and child health, reproductive health and family planning, quality assurance, logistics systems, information systems and research, behavior change communication, and strengthening human resources for health, especially as relating to crucial cadres such as Auxiliary Nurse Midwives and staffing rural areas. The Contractor must participate in technical working groups and provide technical assistance at the district, regional and national level related to health policy development and program guidance to ensure equitable, affordable and accessible delivery of services. C.2.2 Approval 50 During the annual work planning process (see Sections C.7 and F.4) both the Contractor and USAID may propose indicators and capacity building benchmarks either from the lists below or proposes others for inclusion in the PMP, for the final annual work plan, subject to COR approval. C.2.3 Deliverables and Tasks • The Contractor must provide support and technical assistance to the Management Division in the DOHS to organize annual regional program review meetings that encourage better use of data and analyses at the district, regional and national levels. These meetings must involve improved preparatory planning based on analysis of HMIS data and include district counterparts and stakeholders as well as regional and national representatives from the MOHP, EDPs and others. • -The Contractor must strengthen existing HMIS to improve quality and timeliness of reporting and feedback through testing the appropriate use of technology such as the use of GIS, tablet PCs, SMS based reporting and feedback in the 14 project districts. • -The Contractor must coordinate with USAID/Nepal’s integrated nutrition program, Suaahara, to ensure as broad a geographic coverage as possible and to ensure that lessons learned are institutionalized within the GON system. • The Contractor must also invite the MOHP GESI focal points and WDO to participate in discussions on data analysis and data use and offer technical assistance to them in these areas to build their capacity in program planning and data dissemination. • The Contractor must adapt and use developed public health analytic tools to build data-use capacity among District Health Office Managers in 14 project districts to identify low performing VDCs and health facilities, and to develop specific work plans to improve performance. • -Through technical assistance, training and use of the public health analytic tools, the Contractor must equip D/PHO managers with skills and tools to use available data to assess health systems performance and develop specific, prioritized program action plans to improve performance. • -The Contractor must further provide technical assistance to D/PHOs to enable them to better raise awareness and mobilize communities and community-based organizations to improve health system performance. • -The Contractor must provide support and technical assistance to the Nepal Health Research Council (NHRC) to develop regularly updated national research priorities and to capitalize on technical expertise of Nepali institutions. • The Contractor must provide technical assistance to build collaborative networks between the GON, donors, academia, research institutes and other USAID health programs to identify health research priorities in Nepal and to utilize existing data/information to plan, monitor and evaluate health programs. • Through collaboration with other stakeholders, the Contractor must monitor the current national health sector program, NHSP II, establish new targets for the follow-on, NHSP III, and develop policies based on field experience. • The Contractor must conduct a data quality assessment of two national information systems, HMIS and LMIS, in at least six high-performing districts of Nepal and use the findings to facilitate improvements in the respective systems. • -The Contractor must integrate GESI into research priority setting activities with the Nepal Health Research Council and other stakeholders. C.2.3.2 Indicators or proxies as approved by the COR • D/PHO managers using data analysis skills to develop work plans to assist health facilities with performance improvements • -D/PHO managers using disadvantaged group (DAG) mapping to improve access to 51 services and coverage and improve policy • -Technical and policy briefs produced at the regional and national level to document experiences, best practices and learning at the project field area • Health facilities displaying current year service coverage information on wall charts and graphs • -Workshops conducted by the NHRC to discuss programmatic implications of health services research findings and advocacy for policy amendments • Policies, regulations or guidelines developed and/or refined to improve access to and use of health services drafted with technical assistance from the Contractor • HMIS reporting using advanced technology (SMS, smartphones, GIS, tablet computers) designed and piloted C.2.4 Capacity Building Benchmarks or proxies as approved by the COR (year of completion) • -GIS mapping system designed and approved (Y1) • PHA skills course modified and tested in five districts (within the 14 project districts) with poor performance (Y1-3) • Collaborative network established for identifying national health research priorities under NHRC and MOHP leadership (Y2-5) • -Annual district-level plans, developed based on locally relevant data, submitted to DDC and approved (Y3-5) • -Research priorities identified, research conducted, and findings fed into health policy and programming (Y1-5) 1 Health Research in Nepal: An Assessment of Evidence, MoPH, January 2011. C. 2. 3. Objective 3: Strengthen national level stewardship of the health sector. C.2. 3. 1. Overview The Contractor must support the MOHP to develop its next national health sector program in collaboration with other health sector donors. The H4Lcore Contractor will be required to provide policy guidance that strengthens MOHP stewardship over private as well as public health system actors, particularly in areas such as family planning and newborn health where the sphere of private sector provision of commodities and services is expanding. Eventually this kind of policy framework can lead to better regulatory and quality oversight of private sector services, which currently lack active regulation, as well as possibly involving professional associations and accreditation systems. C.2. 3.2. Approval During the annual work planning process (see Sections C.7 and F.4) both the Contractor and USAID may propose indicators and capacity building benchmarks either from the lists below or proposes others for inclusion in the PMP, for the final annual work plan subject to COR approval. C.2.3.3. Deliverables and Tasks 1. In addition to the tasks related to monitoring implementation of NHSP III, the Contractor must provide technical assistance to ensure that the MOHP revises the national health policy and develops its next national health sector program (NHSP III) in collaboration with other health sector donors in a manner that is evidence￾based, strategic and participatory. 2. –The Contractor, together with NHSSP, must provide technical assistance to the MOHP 52 to link the new NHSP to the AWPB process. 3. In Year 2, the Contractor must support initiation of the strategy formulation process in which senior representatives from the MOHP and EDPs will develop the NHSP III, including a review of drafts; and finalizing a draft NHSP III for review. Together with partners, the Contractor must provide technical assistance for a written synthesis of recent research and data in order to provide an evidence-base on which NHSP III will be developed. 4. –The Contractor must provide technical assistance to GON institutions for development of strategy documents and implementation plans as requested. For example, technical assistance to Nepal’s National Health Training Center for development of a training strategy. 5. The Contractor must work with the GON to ensure the timely and appropriate disbursement of budgeted Redbook funds. To this end, the Contractor must provide TA to the MOHP and must provide two consultants to monitor Redbook activities with the MOHP. 6. –The Contractor must review options for partnerships with the private sector based on NHSSP’s prepared draft, “State Non-State Partnership Policy for the Health Sector in Nepal”. As a result of these recommendations, the Contractor may be required to support senior staff of the MOHP and DOHS, in collaboration with other health donors, to conceptualize a practical policy framework within the MOHP for partnering with the private sector, including contracting services, quality assurance, monitoring of service provision, and reporting arrangements. 7. –The Contractor will identify and provide technical assistance to develop partnerships between the MOHP/DOHS and private sector, which may include partnerships with telecommunication networks and Value Added Service Providers, as well as partnerships in districts to address specific gaps in reaching M/DAG populations. 8. –The Contractor must document these partnerships and share them with the MOHP and EDPs in the form of lessons learned to inform future policies and programs. C.2.3. 4.Indicators or proxies as approved by the COR 9. –Public Private Partnership (PPP) experiences in the health sector a) documented and b) shared with partners 10. PPP Policy Forum activated and made functional 11. Third long term health sector plan (2018-2038) developed and endorsed (cancelled by the GON) 12. Comprehensive health sector plan (2016-2021) developed and endorsed by stakeholders C.2.2.2 Capacity Building Benchmarks or proxies as approved by the COR (year of completion) 1. National Health Sector Strategy workshop held incorporating findings from the Joint Annual Review and DOHS National Review (Y2) 2. National Health Sector Strategy development “write-shops” convened, functional and producing draft strategy for stakeholder review (Y2 and 3) 3. Second Long-Term Health Sector Plan and Nepal Health Sector Program II assessment completed with MOHP (Y2) 4. Participatory feedback forums established at national and local levels to review and provide input to Long-Term and Comprehensive National Health Sector Plans (Y2 and 3) 5. –Public-Private Partnership (PPP) Policy Forum functional (Y2) and regularly meeting to share documented evidence on PPPs (Y3 and 4) C.2.4. Objective 4: Institutionalize nationwide system for quality improvement. C.2.4.1. Overview 53 Building on lessons learned to date and in collaboration with other agencies, the Contractor must support the MOHP to test a quality improvement and assurance program suitable for the conditions in Nepal for feasibility and scale up. This quality improvement system must begin with very simple parameters of quality of care with every level of facility having a list of expected standards of infrastructure, equipment, supplies, and personnel and functions, appropriate to the type of health facility. Standards of clinical care must be available and simple management protocols that define the roles and responsibilities of each facility readily available. The QI and supervision system should be tied to these standards and protocols and enable both facility staff and visiting supervisors to work together to ensure adherence to the established standards. Client satisfaction must also be a factor and HFOMCs given a role assisting facilities to meet quality standards, both of which will help empower and give a voice to clients who come from disadvantaged groups. As staff at all levels become familiar with this kind of quality assurance system, it can over time gradually become more refined and eventually lead to the basis of accreditation programs. NFHP II helped organize Quality Assurance Working Groups (QAWGs) in its implementation districts, and worked at the national level to help develop standards of care and national guidelines for various programs. QAWGs are primarily focused on infection prevention activities and in some cases prioritizing equipment for certain facilities. GIZ is also working on supporting a small-scale effort to develop a quality improvement system at the district level. This work, in addition to the components already supported by NFHP II, must be incorporated into a more comprehensive national system. C.2.4.2. Approval During the annual work planning process (see Sections C.7 and F.4) both the Contractor and USAID may propose indicators and capacity building benchmarks either from the lists below or proposes others for inclusion in the PMP, for the final annual work plan subject to COR approval. C.2.4.3. Deliverables and Tasks 1. By the end of Y2, the Contractor’s technical assistance to the MOHP DOHS Management Division (MD) must result in development of a step-by-step program for an enhanced national quality improvement program, including the benchmarks required to move from one stage to the next. 2. –The Contractor must provide capacity building and technical assistance to the DOHS MD at the national level to update, institutionalize, and support a quality improvement system including monitoring of technical and management functions and updating national medical standards. 3. The Contractor must pilot a simple facility-based quality improvement system, in collaboration with GIZ, in three districts to improve awareness about basic standards and the need for continuous monitoring and improvement. 4. With GIZ, the Contractor must share with MOHP best practices through presentations and briefs from the pilot (e.g., in JAR – Joint Annual Review meetings) to improve QI and supervision systems for facility and community staff. 5. –Beginning in year 3, the Contractor must provide technical assistance to D/PHOs in the 11 remaining core districts to introduce the QA approach. 6. –In Y3, the Contractor must provide technical assistance to D/PHOs in the demonstration Districts (identified in C.3.3) for the implementation and strengthening of the QI system at district and health facilities. The Contractor must also document and assess the results in these districts. C.2.4.4. Indicators or proxies as approved by the COR 1. Health facilities assessed using the QA tool 54 2. –Health facilities being independently assessed and accredited annually based on a national QA assessment tool 3. QAWG meetings held in H4L districts C.2.4.5. Capacity Building Benchmarks or proxies as approved by the COR (year of completion) 4. National Quality Assurance Coordination Group (QACG) activated under the leadership of DOHS DG (Y1) 5. -Consensus achieved on National QA System, including the QA tools, implementation plan, pilot measurements, and accreditation criteria (Y2) 6. Pilot conducted in three districts in coordination with GIZ (Y2-3) 7. Pilot evaluated (Y3) 8. -Pilot results, best practices, and lessons learned documented and shared with National QACG and national stakeholders (Y3) 9. -QA system expanded into all H4Lcore districts, including the Collaborative Framework demonstration districts (identified in C.3.3) (Y3-5) C.2.5. Objective 5: Improve capacity of district and local health workers and community volunteers to deliver quality family planning, maternal, newborn and child health, and nutrition services. C.2.5.1. Overview This objective addresses the human resource constraints in the health system, particularly the low quality of pre- service training and insufficient clinical skills of lower level health workers. It recognizes that long-term solutions require revising pre-service curriculum and training instead of supporting in￾service training to fill gaps. It addresses the related challenge of improving the quality of services, both at the community and sub- district facility levels. For this objective, the H4Lcore Contractor will focus support on the 14 project districts (see section C.3.3). Findings and best practices will be shared by the Contractor with national-level policy makers and external development partners and the Contractor will facilitate incorporation of best practices and lessons learned into national level policies, programs, and training and supervision systems as possible through Objective 2. Community level: USAID and its implementing partners have enjoyed tremendous success in piloting innovations and scaling up programs for the delivery of services in maternal, newborn and child health as well as family planning and reproductive health such as allowing FCHVs to treat pneumonia with cotrim, diarrhea with zinc and oral rehydration solution and preventing post-partum hemorrhage among women who deliver at home using misoprostol (see NFHP II evaluation). The H4Lcore Contractor must support the capacity of the health care system to implement, monitor and maintain such programs at the community level. The Contractor must develop a working methodology in IPCC and health messaging with Suaahara and FtF for FCHVs in their role as community mobilizers for essential nutrition actions (ENA+), essential hygiene actions (EHA) and water/sanitation (see Objective 6). Facility level: The H4Lcore Contractor must coordinate and collaborate with the Family Health Division and Child Health Division in the MOHP to improve oversight and quality of care at the national level and strengthen the referral link between the communities and facilities for maternal and newborn health services, including family planning. Critically important will be the expanded provision of quality long term family planning services such as intrauterine contraceptive devices and implants at all sub-district level facilities that have birthing rooms because family planning is such a key part of reducing maternal mortality and improving child health. The H4Lcore Contractor must expand and strengthen the quality of services delivered by health facilities, as well as community-based approaches such as prevention of postpartum hemorrhage 55 using misoprostol. The Contractor must also provide technical support to the MOHP to improve the quality of antenatal care services to screen for PE/E, and provide TA to the H4L Logistics Contractor to ensure a regular supply of magnesium sulfate for treating eclampsia. The Contractor, in coordination with USAID’s nutrition and food security projects (including but not limited to Suaahara), must provide technical support to improve the quality and availability of nutrition and hygiene counseling services from health facilities and outreach clinics. Pre-service Auxiliary Nurse Midwife (ANM) training: ANMs are included in Nepal’s definition of Skilled Birth Attendants (SBAs) and therefore central to Nepal’s MDG target of 60% of all births attended by SBAs by 2015 and the GON’s related “National Policy on Skilled Birth Attendants” and “National In- Service Training Strategy for Skilled Birth Attendants”.2 Along with nurses, ANMs are the main providers of safe delivery and emergency obstetric care services in rural areas. As of April 2011, 2,268 SBAs have been trained and 18 SBA training sites have been established3. ANMs receive 18 months of training through the Center for Technical Education and Vocational Training (CTEVT) or another government or privately run training institutes with CTEVT affiliation followed by three months of in-service training supported by government and donor funding to supplement skills on safe delivery. A 2006 study4 on pre- service training for ANMs at CTEVT revealed the inadequate skill level of ANMs which is particularly concerning given the rapid expansion of new birthing facilities and increased facility deliveries as a result of the “Aama” program. C. 2.5.2. Approval During the annual work planning process (see Sections C.7 and F.4) both the Contractor and USAID may propose indicators and capacity building benchmarks either from the lists below or proposes others for inclusion in the PMP, for the final annual work plan subject to COR approval. C.2.5.3. Deliverables and Tasks • -In collaboration with CTEVT, the Contractor must conduct a mapping exercise to identify all key stakeholders including the nursing focal person in the MOHP, the Nepal Nursing Council, UNFPA, NHSSP, UNICEF, World Bank and others, and agree on a method for improving the pre-service curriculum and teaching methods for ANM training and upgrading instructional skills. • The Contractor must conduct a rapid assessment of the curriculum in Q2/Y1 and • -a second-phase assessment in Y2 to explore the options for SBA certification processes including certifying ANMs as SBAs. • -In collaboration with the Ministry of Education, MOHP, IOM, PAHS and other academic institutions and CTEVT, the Contractor must update the pre-service curriculum, improve teaching learning practices and establish standards and a more rigorous process for certification. This must include identification of benchmarks for moving forward. • - The Contractor must provide technical assistance and support to academic institutions to use the revised curriculum and training methodology in at least 4 selected ANM schools in H4Lcore districts. • -In Y2 the Contractor must produce a baseline report including assessments based on standards, school- specific action plans, mapping of clinical practice sites and any available information on post-graduation deployment. • -In Y2-3 and in close collaboration with the Ministry of Education, the MOHP nursing focal point, Nepal Nursing Council and CTEVT the Contractor must plan, organize and facilitate a series of workshops to develop a SBA performance-based certification process that tests ANM graduate essential competencies. • -The Contractor must then pilot the SBA certification process in FY4 in the 4 or more selected ANM schools and must provide lessons learned through briefings and 56 documentation to the GON and stakeholders through the SBA forum. 2 Nepal National Policy on Skilled Birth Attendants: http://www.mohp.gov.np/english/files/news_events/6-1-Safe-Motherhood-and SBA-Policy.pdf and Five Year Operational Plan for In-Service Training of Skilled Birth Attendents: http://www.safemotherhood.org.np/pdf/1535- year%20OP%20for%20SBA%20IST_20%20Mar%2009.pdf 3 Workshop to Review the Implementation Status of the National SBA Programme Report. 28-29 April 2011, Park Village Hotel, Budhanilkantha, Kathmandu 4 http://www.nsi.edu.np/pdf/CTEVT%20report.pdf 9 -The Contractor must provide technical assistance to the D/PHOs in the 14 project districts to enhance the skills of FCHVs in community based birth preparedness, newborn care and postpartum family planning counseling and services. 10 - The Contractor must also provide technical assistance to D/PHOs to enhance skills of FCHVs to implement the community based newborn care package in eight districts. 11 -In the 14 H4L districts where Community Based- Integrated Management of Childhood Illness is being revitalized, the Contractor must provide technical assistance and support to ensure selected essential nutrition actions including hygiene and sanitation will be strengthened in the FCHVs’ program of work within communities, including exclusive breastfeeding for six months; adequate complementary feeding from about 6–24 months with continued breastfeeding for at least two years; appropriate nutritional care of sick and referral of severely malnourished children; adequate intake of vitamin A for women and children; and adequate intake of iron for women and children. 12 - The Contractor must work closely with Suaahara and FTF from Y1 to identify and implement program synergies. 13 -The Contractor must provide technical assistance and support to the DOHS to collect and disseminate field-based evidence of best practice as a means to improve the quality and efficacy of family planning, maternal, newborn and child health services. 14 -Following the results of current pilot on misoprostol, the Contractor must identify options for strengthening the misoprostol program for home births in at least 10 of the 14 project districts. In its analysis and recommendations, the Contractor must provide criteria for selecting districts. 15 -The Contractor must provide technical assistance and support to the MOHP to reposition the national family planning program through a renewed drive to make long term contraceptive methods available, especially to postpartum and post-abortion clients; by reaching those clients who are underserved by virtue of their social status and geographic remoteness; and by initiating a stronger effort to encourage healthy timing and spacing of pregnancies. 16 The Contractor must provide technical assistance to the DOHS for an FP consultation to update the national policies and strategies related to family planning in light of progress and 2011 NDHS findings. 17 -The Contractor must provide technical assistance to operationalize the National Adolescent Health Strategy (NAHS) and Implementation Guideline. 18 -With stakeholders in 10 of the project district health offices the Contractor must develop a system to apply the standards of care for obstetrical services at health posts and sub-health posts with birthing centers and strengthen the system of clinical supervision for ANMs and others providing delivery care and long-term family planning methods at those facilities. Standards will include strengthening the use of Active Management of Third Stage Labor, partographs, magnesium sulfate for the management of pre- eclampsia and eclampsia, appropriate referrals, IUCD and implant insertions. 57 19 -The Contractor must use the MNCHN/FP Clinical QI Tools developed by the FHD and CHD. After introducing these tools in 10 project districts, the Contractor must expand the system to the remaining four project districts. 20 -The Contractor’s district staff must provide technical assistance to support D/PHOs to identify and develop plans to reach high seasonal migration communities and those underserved due to social status or geographic location. 21 -The Contractor must identify opportunities to improve the referral network for women and their newborns. In FY1-3 the Contractor must work with the MOHP and other EDPs to improve access to CEONC sites in project districts and to improve referrals. C. 2.5.4. Indicators or proxies as approved by the COR 1. Birthing centers that provide at least one long-term FP method 2. Health facilities receiving at least one documented supervision visit per quarter by DHO officials 3. - Pregnant women who receive the drug misoprostol from trained FCHVs or community health workers for use at home deliveries (in post-partum hemorrhage prevention districts) 4. -Children under 5 who received comprehensive nutritional status monitoring (in OPD CB-IMCI registers) 5. -Deliveries conducted following standard AMTSL 6. -ANM graduates/students assessed and certified as SBAs 7. Births that are attended by a skilled birth attendant (doctor, nurse or ANM) 8. Health workers/FCHVs trained in child health and nutrition. 9. -ANMs/SBAs trained as per revised curriculum and training methodology C.2.5. 5. Capacity Building Benchmarks or proxies as approved by the COR (year of completion) Community 10. -National consensus achieved on modular FCHV continuum of care strategy (Y1) 11. -FCHV training modularized (Y2) 12. -ENA+, misoprostol, and CB-NCP trainings rolled-out (Y2-3) 13. -District data used to determine needs for FCHV training and relevant trainings conducted (Y2-3) Facilities 14. -System developed based on standards of care for quality MNCHN/FP clinical services for sub- health posts and health posts (Y1) 15. HTSP, PPFP and PPIUCD trainings conducted; SBA in-service training for contract ANMs conducted (Y1-2) 16. -High-quality birthing centers assessed and strengthened using MNCHN/FP Clinical QI Tools (Y3) 17. -Mortality targeted interventions tested (Y1-2), documented/evaluated (Y2-3) and scaled-up (Y2-4) ANM education: 18. ANM pre-service rapid assessment completed (Y1) 19. -Second-phase assessment on options for SBA certification such as re-licensing as SBAs (Y1) 20. -ICM-based ANM school standards finalized (Y1) 21. -Auxiliary Nurse Midwife (ANM) curriculum updated to match SBA competencies (Y2) 22. -Four ANM schools strengthened using standards (Y3) 23. -SBA performance-based certification process for ANM graduates piloted and results shared with stakeholders to inform national policy (Y4) C.2. 6. Objective 6: Improve knowledge, behavior and use of health services among target populations. C.2.6.1. Overview 58 A key component of addressing inequitable access to health services is improving the availability of information about health behaviors and using health services and mitigating the factors that exclude certain castes, adolescents, ethnic groups or women. The H4L Contractor must support dialogue at the national level to improve the spread of information on health services, appropriate use of services and GESI-related barriers to service. The Contractor must provide technical assistance to support communication activities related to new initiatives and investments and to test and strengthen existing materials and messages. The Contractor must address specific gaps in current programs and in specific populations not currently being reached. The Contractor must: 1) target key populations currently being missed with education and behavior change efforts (such as adolescents age 10- 19, migrants and select marginalized groups); 2) target missed opportunities in family planning/maternal, newborn and child health/nutrition to provide education and counseling on healthy behaviors primarily through effective interpersonal communication; and 3) use innovative technologies and approaches to reach people. Priority messages include those focusing on the Healthy Timing and Spacing of Pregnancy to help women and families delay or space their pregnancies to achieve the healthiest outcomes for women, newborns, infants, and children, within the context of free and informed choice; Long Acting and Permanent Methods; exclusive breastfeeding; delaying age at first marriage; essential newborn care; and care of sick children. Improved interpersonal skills and counseling in the community and facility will address the missed opportunities to educate post-partum women and post- abortion women about Healthy Timing and Spacing of Pregnancy and Long Acting and Permanent Methods and to reach adolescents with messages on family planning. Improving the quality of counseling is a key challenge. Therefore, new counseling techniques and improved support to day-to-day activities on family planning, Community-Based Integrated Management of Childhood Illness, Community-Based Newborn Care Program and essential nutrition actions must be included in training programs to ensure that FCHVs gain the necessary skills needed to promote positive behaviors at the household level especially among those who are often socially excluded by virtue of their caste or ethnicity. The Contractor will provide technical assistance and support to improve the interpersonal communication and counseling skills of health workers and volunteers by rolling out the IPCC package developed by NHEICC. In coordination with Feed the Future, the Contractor must promote implementation of district level behavior change activities that include key messages to increase coverage of health, nutrition and water, hygiene and sanitation issues in the mass media (local FM radio, IEC materials). In coordination with the Feed the Future program and Suaahara, the Contractor must provide technical support to design context- specific radio messages on health and nutrition in H4L districts. The Contractor must also use the “Learning Circle” methodology used by NFHP II to enhance the effectiveness and reach of Mothers Groups and enable them to become a better mechanism for peer education and behavior change as identified by the NFHP II evaluation. C.2.6.2. Approval During the annual work planning process (see Sections C.7 and F.4) both the Contractor and USAID may propose indicators and capacity building benchmarks either from the lists below or proposes others for inclusion in the PMP, for the final annual work plan subject to COR approval. C. 2.6.3. Deliverables and Tasks 1. The Contractor must either identify and use information from existing formative research and barrier analyses or conduct additional research in at least five districts in order to identify context-specific issues for non-utilization of long-acting and permanent family planning services. The findings from either existing research or research conducted by 59 the Contractor must be clearly incorporated into H4L and MOHP interventions. The Contractor must submit a written summary of how the findings are being used in project interventions to the COR. 2. The Contractor must provide technical assistance to District Health Office Managers in each of the 14 project districts on the use of personal, folk and mass media to promote messages regarding Healthy Timing and Spacing of Pregnancy, Long Acting and Permanent Methods, essential newborn care of sick child, essential nutrition actions and adolescent reproductive health. 3. In Q2/Y1 the Contractor must complete an Adolescent Knowledge, Attitudes and Practices literature review on family planning and must map other EDP/donor/GON activities. 4. -The Contractor must provide technical assistance to the Women Development Officers (WDO), youth and community members to support their efforts to delay early marriage, address GBV, and improve access by adolescents to family planning services in 10 districts of the 14 project districts. 5. The Contractor must use local media to increase awareness of the efforts of HFOMCs, encourage people to become more involved and link them to national media as appropriate. 6. The Contractor must provide technical assistance to support integration of family planning and maternal, newborn and child health messages into existing programs and improve counseling skills among health workers (~500) and volunteers (~2500), including skills to reach adolescents. 7. -The Contractor must provide technical assistance and support for in-service counseling training for FCHVs (~2500) and health workers (~500) aimed at improving prenatal, natal and post-natal home visits. 8. -The Contractor must provide technical assistance, advocacy and support to strengthening peer education and counseling aspects of the Community-Based Integrated Management of Childhood Illness and Community-Based Newborn Care Program using Learning Circle techniques with Mothers Groups in five districts. C.2.6.4. Indicators or proxies as approved by the COR 1. -Parents, adolescents and young adults who understand the reasons for delaying marriage and childbirth 2. Women who know where to obtain long term contraceptive methods 3. -Ratio of health facility clients vs. population of marginalized groups within the catchment area of the health facility 4. -Pregnant women who follow all recommendations for birth preparedness 5. Couple-years of protection in core districts and at national level. 6. -Postpartum and post-abortion women receiving FP information/counseling and services 7. -Community members (especially parents and influential HH members) who can correctly cite at least two danger signs of each: a) pregnancy, b) delivery, c) postnatal, d) neonatal and e) childhood illness 8. -Men who reject negative behaviors such as violence against women 9. -Men’s knowledge of long-term family planning C.2.6.5. Capacity Building Benchmarks or proxies as approved by the COR (year of completion) 10. -Process approved by D/PHO and FMCs and operationalized to allocate the budget earmarked for BCC at the district level (Year 1) 11. -BCC strategies designed and refined to reach groups not commonly targeted such as men and adolescents (Y1 and 2) 12. -Learning Circle initiated and used for CB-IMCI and CB-NCP (Y1 and Y2) 13. -Competent CBOs and FCHVs are capable of planning and delivering high-quality 60 BCC in support of district-level priorities (in 14 core districts) (Y3 and 4) 14. -Strategies to address gender-based violence and discrimination developed for folk and local media (Y1 and Y2) C.2.7 Objective 7: Strengthen capacity of USAID’s contractor for a logistical support activity. C.2.7.1 Overview Through a separate contract, USAID will strengthen the logistics system to ensure availability of key drugs, supplies and commodities at health facility and community levels. A local Nepali organization or company will implement these activities. C.2.7.2 Approval During the annual work planning process (see Sections C.7 and F.4) both the Contractor and USAID may propose indicators and capacity building benchmarks either from the lists below or propose others for inclusion in the PMP, for the final annual work plan subject to COR approval. C.2.7.3 Tasks The Contractor must provide technical assistance in the areas of management and coordination as needed to the implementer of the logistical support activity. The scope and duration of the technical assistance, as well as key interventions and indicators, will be determined through consultations between the Contractor, USAID, and the H4L logistics implementer and will be defined in the work plan. During the final two years of the contract, needs for technical assistance will decline and the Contractor will be expected to provide minimal guidance and monitoring of activities as needed. C.2.8 Indicators or proxies as approved by the COR • Percent of HFs (PHCCs, HPs, SHPs) experiencing stock outs of any contraceptive commodity (Condom, Oral Pills and Injectables) • Percent of HFs (PHCCs, HPs, SHPs) experiencing stock outs of specific tracer drugs (ORS, Vitamin A, Cotrimoxazole (P), Iron tablets) • Percent of HFs (PHCCs, HPs and SHPs) that submitted LMIS reports on time • Performance of H4L-Logistics partner rated by Technical and Organization Capacity Assessment Tool (TOCAT) C.2.8 Additional Activities C.2.8.1 Internship Program The Contractor must create an Internship Program that serves to build the professional capacity of women and marginalized groups from the districts where program activities are targeted. Internship positions should run for between ten and 12 months and must expose participants to work both inside and outside Kathmandu including work at the village level. The Contractor must offer at least two paid internship positions per year of operations. The Contractor must coordinate with Nepali public health training institutions and CTEVT to place interns at the H4L project, health facility and/or community level to gain practical programming, management and technical experience. Each intern must be assigned a mentor and must complete an activity that supports the goal of their assigned portfolio. The Contractor must provide a summary of the internship program in its annual reports. At the end of each internship period, the 61 Contractor must also submit to USAID a brief (1 page) analysis of the skills and knowledge gained by the interns as well as information on the intern’s career goals and future plans. If the internship period coincides with the annual or semi-annual report, the information may be included in that report. Where appropriate, the Contractor should encourage partners to adopt similar internship programs. C.3 IMPLEMENTATION PARAMETERS AND APPROACH C.3.1 Capacity Building Approach The Contractor must apply a systematic approach to strengthening GON systems and institutionalizing best practices at the local, district and national levels. The unifying theme of all seven objectives is that of capacity development of the public health system, measuring and monitoring progress, and making the necessary change along the way to ensure sustainability. H4L’s capacity building approach should align with the Potter/ Brough model5 (right), which postulates that capacity development falls short when it only focuses on tools and skills. It must also encompass improvements especially on strengthening the structure, systems and roles that support the institutional change being sought. H4L should also adopt a monitoring and evaluation approach to health systems strengthening that aligns with WHO’s reflected below (2010). The Contractor must ensure that its interventions align with GON priorities at the local, regional and national level. The Contractor must further tailor its interventions to ensure the highest priority needs are identified and addressed in partnership with GON stakeholders. 5 Systematic capacity building: a hierarchy of needs”, Potter, C. and Brough, R., Health Policy and Planning 19(5): 336– 345, Oxford Press, 2004 62 C.3.2 Mandate for Collaboration Collaboration with other USAID partners and activities is essential for the Contractor to achieve the scale and scope of the aforementioned objectives. During Y1, the Contractor must meet with relevant USAID implementing partners, including Suaahara, Feed the Future (KISAN), Sajhedari Bikas, Hariyo Ban, Ghar Ghar Maa Swasthya, and Saath-Saath (briefly described below), to align work plans and discuss synergies between activities and approaches. The Contractor must clearly identify in its annual work plan the activities that will be conducted in coordination with other partners. During years 1-5 the Contractor must meet with partners at least on a semi- annual basis to, at a minimum, review implementation and strategies. The Contractor must further be flexible to meet with other partners on specific topics at the request of the USAID COR as needed, which may be as frequent as once a quarter. USAID/Nepal’s social marketing program, the Ghar Ghar Maa Swasthya (GGMS) program assists the GON in expanding the depth, reach, and impact of the private sector in social marketing, by providing a low-cost supply of maternal/child health, family planning, and HIV-prevention products and services. GGMS focuses on underserved and most-at-risk populations, scaling up promising rural and community-based marketing initiatives, and engaging non-governmental organizations and commercial distributors to increase product accessibility in hard-to-reach areas. Both GGMS and the H4Lcore Contractor will concentrate on the hard- to-reach regions and populations in the areas of family planning and maternal, newborn and child health. GGMS will continue to develop the private sector while the H4Lcore Contractor will focus on the public sector. USAID/Nepal’s five-year, $46 million integrated nutrition program, Suaahara will improve the nutritional status of women and children during the most critical period (pregnancy to two years of age) through a comprehensive facility and community-based nutrition program targeting the most vulnerable in Nepal. The H4Lcore Contractor must also promote messages that support essential nutrition actions, such as relevant hygiene messages, in districts within the 14 project districts that are not covered under Suaahara and FtF (see Sections C.2.5 and C.2.6). The H4L 63 contractor must align its approaches for ENA+ training and outreach with Suaahara and FtF or must otherwise provide written justification for the difference in approach to the USAID COR. USAID/Nepal’s five-year, $27.5 million HIV/AIDS program, Saath-Saath, which will reduce transmission and the impact of HIV/AIDS, and improve reproductive health among select, most-at risk populations. Through Objective 7, the H4Lcore Contractor must provide TA to the H4L Logistics Contractor to work with Saath-Saath to integrate the HIV/AIDS and health commodity logistics systems for improved efficiency. Donor collaboration and coordination is also important to the success of H4L. The Contractor must monitor ongoing donor programs in the same or similar technical and geographic areas of operation and notify the COR of opportunities for collaboration and complementarity. Monitoring can occur on an as needed basis, given that greater collaboration among activities may be required at different times. Depending on the level of activity in any given region, monitoring and collaboration may extend from periodic informal conversations with other donors to more intensive planning and regular meetings for program coordination and development. The Contractor must work with the COR on a case-by-case basis to determine the level of coordination and collaboration necessary. Furthermore, the Contractor must be prepared to undertake significant collaborative and cooperative efforts with other donors. C 3.3 Geographic focus and target population The Contractor must maintain a national and sub-national focus to cover the range of objectives. The majority of the objectives will have national level inputs in quality improvement, governance and capacity building. Objectives 1, 5 and 6 will be implemented in the 14 project districts in the Western and Mid￾Western Development Regions and highlighted below in Map 1. There are also select local health governance strengthening (Collaborative Framework) and QI activities that will be implemented in six demonstration districts. These demonstration districts are also identified in Map 1. The findings and best practices identified in those districts will be shared with central-level counterparts from the GON and development partners to influence policy and nationwide systems. The 14 project districts (See Map 1), are selected based on the need in those districts (measured in terms of HDI, district performance, HMIS/LMIS data); other programs implemented by USAID and other donors to ensure synergy and avoid duplication; necessary follow-up needed to mature some components of current (NFHP II) programs, and clustering of the program districts to ensure cost￾effective program operation. The total population covered by these districts is 4.3 million which is nearly 15% of the national population, as well as 700 health facilities and 9,000 female community health volunteers. Final number and selection of districts for different interventions described in the task list for each objective will be confirmed following the award and in consultation with the GON. Criteria for selecting sub-sets of districts for specific activities must be proposed by the Contractor before beginning activities and is subject to COR approval. During the first quarter, the Contractor must divide the 14 project districts into groups, or clusters, to roll out H4L activities. Also during the first year, the Contractor must develop work plans with leadership in all H4Lcore districts defining roles, responsibilities and key interventions. Final work plans must be shared with the COR. Map 1: H4L 14 project districts and demonstration districts 64 C 3.4 Roles C.3.4.1 Leadership Role of the Government of Nepal The primary role of the GON through the MOHP is to implement the NHSP II, 2010-2015. The Contractor is responsible for remaining aware of all GON policies relevant to the implementation of this contract. The Contractor shall immediately notify the COR of any GON policy changes impacting the implementation of activities. The Contractor must coordinate and collaborate with the GON to maximize the institutionalization of capacity in the targeted health systems. The Contractor must attend and actively, substantially participate in the annual progress reviews, semi-annual joint approval reviews held by the GON and weekly meetings of external development partners as the key coordination mechanism for ensuring USAID activities and results are widely shared with key stakeholders in the sector. C.3.4.2 USAID The USAID office responsible for the direction and management of this activity is the Office of Health and Family Planning. The COR, located in the Office of Health and Family Planning, will be designated by the Contracting Officer and is responsible for the technical oversight and administration of the contract. HFP will also promote and monitor progress on GON policy initiatives affecting activity implementation and ensure coordination with other donor-funded programs. The USAID Contracting Officer (CO) has the ultimate responsibility of negotiation, award and administration of this contract. C.3.4.3 Contractor The dominant required roles of the Contractor in reaching the objectives described above are outlined briefly below. 1) Technical Expertise. The Contractor must provide quality, proven, and respected technical expertise in health systems strengthening, including logistics management, health information and policy development, quality improvement and health governance as well as the health service delivery areas of family planning, maternal, newborn and child health, nutrition, interpersonal communication and counseling skills, mass media and behavior change programs. The Contractor must further provide consistent, focused technical expertise in the area of capacity development for Nepali government entities, 65 particularly at the district and sub-district levels. 2) Monitoring and Evaluation. In addition to tracking required indicators in the PMP to measure performance against the required objectives, the Contractor must measure technical, organizational, and financial capacity development, particularly at the district and sub-district levels, through the Capacity Building Plans (see Section C.4.1.21) and against defined and approved benchmarks. Ongoing monitoring and evaluation will help USAID/Nepal and the MOHP determine the extent to which improvements are being institutionalized over the course of this activity and identify whether certain activities need to be changed for better results. 3) Convening and Collaboration. The Contractor must work with the GON at all levels to engage in collaborative planning for health across government ministries involved in local health governance. The Contractor must organize, coordinate, prepare and in some cases facilitate meetings with GON officials at the national, district and local levels. The Contractor must use participatory learning exchanges and coaching/mentoring approaches to reach out to and engage women, the poor and marginalized; health facilities and related structures such as facility management committees, NGOs, civil society and community￾based organizations; local and national bodies; and the many related multi-stakeholder forums. At local levels these include Mother’s Groups, Ward Citizen Forums; Users Committees and other such as QAWGs. At the national level, key forums that the Contractor must join, strengthen and/or revitalize include the JAR, technical working groups, professional association meetings, the Local Governance Technical Team and Steering Committee, EDP forums and policy forums about health sector decentralization and public￾private partnerships (PPP). Finally, the Contractor must coordinate with other USAID project partners. During quarter one, the Contractor must meet with other USAID-supported projects such as Feed the Future (KISAN), Sajhedari Bikas, Hariyo Ban, Saath-Saath, Suaahara and Ghar Ghar Maa Swasthya (GGMS) to inform, coordinate and align each other’s implementation strategies and work plans where possible an must confer thereafter at least semiannually. (See C.3.2 for more details on collaboration). 4) Technical Assistance to USAID Logistics Contractor. USAID plans to award a separate contract to a Nepali organization to build the capacity of the MOHP Logistics Management Division (LMD). During the first three years of the logistics contract, the Contractor must provide technical assistance in the areas of management, monitoring and evaluation, and coordination to this logistics activity. After the three years, the H4Lcore Contractor must provide limited technical assistance for monitoring. Ensuring availability of the commodities, training health workers and other components of implementing the logistics activity will not be the responsibility of the Contractor. After award of the H4Lcore contract and H4Llogistics contract, USAID and the Contractor will determine the precise duration and scope of this technical assistance. 66 C.4 CONTRACTOR PROGRAM IMPLEMENTATION RESPONSIBILITIES AND IN- COUNTRY PRESENCE C.4.1 Deliverables The Contractor must accomplish all “Tasks” described below by the date indicated in Section F. The Contractor must develop deliverables for submission to the COR by the due dates listed in Section F. C.4.1.1. Start-up/Deployment Tasks and Deliverables During the Start-up phase, the Contractor must establish key implementation systems and preparation for longer- term field staff deployment for program implementation through the life of the contract. C.4.1.2 Personnel Manual, Bio-Data and Curriculum Vitae Submitted (Deliverable) The Contractor must develop and submit Employee Bio-Data (EBDs) sheets and Curriculum Vitae (CVs) for all Key Personnel not included in the original offer. In its operations, the Contractor must include women and marginalized groups in all levels of planning, decision- making and implementation. It must endeavor to have its staff include women and members of marginalized groups - no less than 33% of the professional staff must be women and/or from marginalized groups. C.4. 1.3 Start-Up/Deployment Work Plan Submitted (Deliverable) The Contractor must prepare and submit a plan for start-up and deployment of staff to Kathmandu, Nepal. C.4. 1.4 Key Personnel Deployed (Task) The Contractor must deploy the Key Personnel team to Kathmandu, Nepal. If Key Personnel cannot be mobilized within 21 days of contract award, they must still be available to attend the 2-day start-up conference in Kathmandu, Nepal, (see Section C.4.1.5) and must be deployed as soon as possible. C.4.1.5 Start-Up Conference in Kathmandu, Nepal (Task) The Contractor must make logistical arrangements for and participate in a startup conference to include USAID/Nepal and Contractor staff. The Contractor’s start-up team members, designated Chief of Party and select home office staff must participate in this session. The Contractor will be responsible for making travel and logistics arrangements for all Contractor staff. The schedule and detailed agenda for this session will be developed by USAID/Nepal in consultation with the Contractor after the award is made. A central part of the conference will be to ensure mutual understanding of the roles of the USAID/Nepal COR and Contractor staff and establish initial communications protocols between the respective parties. The session will also provide an initial overview of other USG assistance to Nepal and an opportunity to establish parameters initial assessments to be undertaken by the Contractor. Representatives from the MOHP/DOHS, other ministries and agencies (MOLD, MOF and NPC), institutions (NHRC and CTEVT), key national-level TWGs/forums, professional associations, EDPs and other USAID project partners will be invited to select sessions. C.4.1.6 Revised Start-Up/Deployment Work Plan Submitted (Deliverable) 67 The schedule/due dates for Contractor completion of all other start-up tasks and deliverables will be finalized at the Start-up Conference. Based upon the discussions and agreements at the Start-Up Conference, the Contractor must submit a Revised Start-Up Work Plan to the COR. These dates will be subject to amendment with COR concurrence. The Revised Start-up Work Plan must cover the first six months of the program and must include roles and responsibilities and the timeline for: (1) All Tasks & Deliverables for first 45 days after effective award date, including a revised Marking Plan (See Attachment 5); (2) A schedule for Contractor Key Personnel recruitment and deployment; (3) A schedule for other staff recruitment and involvement in this process by the COR; (4) Procedures for start-up of program activity, including capacity and processes for the home office role in grant-making and short-term technical assistance arrangements prior to field office deployment; (5) Schedule and proposed mechanism/plans for conducting initial rapid assessments; and (6) See also Section B.5 for additional component description for cancellation costs. C.4.1.7 Communications, Information and Physical Security Plan(s) (Deliverable) The Contractor must develop and submit a plan(s) documenting the procedures and systems in place to ensure the secure management of information related to the program (activities, vendors, grantees, and staff), as well as the physical security of the Contractor’s expatriate, TCN and local staff. This plan(s) must include phone/SMS trees, contingency plans for other forms of communication, and appropriate actions in the event of emergencies. This plan(s) is a requisite for effective program implementation as well as staff security. C.4.1.8 Communications and Information Security Systems Established (Task) The Contractor must procure and provide staff with appropriate and licensed communications equipment (cell phones, satellite phones, laptops, radios, video/still cameras as appropriate) and redundant systems (if needed). The Contractor must procure and provide Contractor staff with appropriate communications equipment and redundant systems as required for program needs and security protocols outlined in the plan(s). C.4.5.1.9 Administrative Procedures (Deliverable) The Contractor must provide written procedures for operations and travel, including, but not limited to: (1) Provisions for arranging and providing for in-country, regional, and international Contractor travel including airline, vehicle rental, hotel accommodations, passports and visas for Contractor personnel. From time to time where limited USG support services are available (i.e. travel up- country, last-minute travel requirements), the Contractor, USAID and other USG staff may accompany contractor personnel using the contractor travel provisions. (2) Translation and interpretation services, as required. C.4.1.10 Offices Identified and Leased (Task) Within 60 days of the effective award date, the Contractor must have secured permanent furnished office facilities as required for the program. 68 This will include an office within Kathmandu Valley and a field office in a location outside the Kathmandu Valley. The location of the field office must be made in collaboration with the COR. C.4. 1.11 Recruitment of Non-Key Personnel (Task) The Contractor must identify and recruit all Key and Non-Key Personnel by the due date noted in Section F. The Contractor must submit a final organizational chart outlining each of the positions and respective lines of authority. C.4.1 Financial Guidelines (Deliverable) and Systems Established (Task) The Contractor must have a bank account, available funding, and financial systems in place to make financial disbursements. The Contractor must develop and deliver written guidelines and systems in consultation with the COR for timely and effective accounting and vendors payments. The Contractor must revise financial records on a monthly basis to take into account fluctuations in the exchange rate, and adjust program financial estimates and spending accordingly. These guidelines and systems must include the following functions and objectives: (1) procedures to maintain sufficient liquidity to meet activity goals (developed during start up); (2) ‘audit ready’ records pertaining to disbursements, accruals, financial liquidity, payroll, vehicle use and maintenance, communications and non-expendable property; (3) tracking and documentation of the overall monthly ‘expenditure rates’ for grant and non- grant expenditures; (4) quarterly accrual estimates reporting in format provided by the COR (due no later than the 20th of March, June, September and December of each year, covering the periods through the end of the reporting quarter); (5) accounting for funds from different sources, including USAID accounts, congressionally mandated earmarks, and other USG funds; (6)Under the bilateral agreement between USAID/Nepal and the GON, development assistance programmatic funds are exempt from taxes to the GON. As such USAID does not finance any identifiable host country taxes. The Contractor is responsible for asserting and obtaining the necessary tax exemptions and reimbursements. Upon request, USAID/Nepal will provide available documentation to support allowable exemption from incurrence of host country taxes on H4L- related activities; travel and per diem reimbursement (consistent with the organization’s established policies and procedures). See also Attachment 6 of this Contract for rates applicable to Government of Nepal staff. The Contractor would update the rates when GoN revises its rates. C.4.1.13 Rapid assessments (Deliverable) In close consultation with the COR, the Contractor must conduct a rapid assessment of pre￾service training of ANMs, and a literature review of knowledge, attitudes and practices of adolescents. The assessment and review should be focused on the identified program target districts and both should include mapping of other donor activities, and serve as pre-entry points for informing activities under this contract. The Contractor must submit a written report of the assessment and review. This assessment and review must not be disseminated outside of USAID without the express agreement of the COR. 69 C.4.1.14 Gender Equality and Social Inclusion (GESI) Strategy (Deliverable) The Contractor must submit a detailed, strategy for gender equality and social inclusion considerations into both programming and activity implementation (the activity’s ‘corporate’ policies or ‘principles’), including staff planning and deployment. The Contractor must mainstream social inclusion considerations and not develop a unique social inclusion program component within this activity. The Contractor must provide a review and summary of process to date against the Contractor’s GESI strategy and related indicators in the annual report. The Contractor must actively work to include GESI considerations in its activities and showcase evidence of the application of the GESI strategy in its annual work plan. Activities may have significantly different effects on different social groups. The Contractor must also briefly outline social exclusion issues and indicate how it will ensure appropriate levels of participation by marginalized groups in its target beneficiaries. This strategy must indicate how the Contractor will analyze the different roles played by men and women and marginalized groups in Nepal – specifically in the target areas – and how gender sensitivity and social inclusion will be incorporated into implementation, and how impact on gender-related and social inclusion issues will be measured. It should go beyond simply noting that ‘gender considerations and/or social inclusion issues will be taken into account.’ In its operations, the Contractor must include marginalized groups, as well as women from all groups, in all levels of planning, decision-making and implementation. It must endeavor to have its staff include women and members of marginalized groups - no less than one-third of the professional staff must be women and/or from marginalized groups. Within the strategy, the Contractor must include a brief GESI analysis that includes thoughtful explanation of the most significant gender issues related to this activity: 1. Are men and women involved and/or affected differently by the program? 2. If so, how will the program address differences in managing for sustainable program impact? Addressing these questions involves taking into account not only the different roles of men and women, but also the relationship and balance between them and the institutional structures that support them. (For details on USAID policy, refer to ADS 201.3.11.6.) C.4.1.15 Anti-Corruption Procedures (Deliverable) The Contractor must develop and implement anti-corruption procedures for this program. These procedures must be consistent with any commonly used practices within the donor community, to minimize the procedures grantees and beneficiaries need to learn to report corruption. The Contractor must develop anticorruption reporting and remediation measures for all aspects of the program, starting with contractor staff and operations, to sub￾contractors, grantees, other partners and vendors. Any and all incidences of fraud or fraudulent activity on the part of sub-partners must and will be reported to the Regional Inspector General. USAID/Nepal is committed to anti-corruption efforts. If, at any point during implementation of activities, one or more of the intended beneficiaries (community groups, civil society organizations, or government officials) significantly impedes progress or completion of an activity (particularly due to requests for bribes, threats to the safety of staff, and so forth), the Contractor must make the beneficiaries aware that the activity will be halted and inform them 70 why the activity will not be completed until the issue is resolved to the satisfaction of USAID and the Contractor. The immediate work-stoppage and discussion with all intended beneficiaries regarding the source of the problem often results in quick resolution and resumption of the intervention and increased beneficiary ownership of the final product. All incidents must be immediately reported to the COR. C.4.1.16 Emergency Preparedness Plan (Deliverable) Disaster risk reduction—addressing vulnerabilities to, and preparation for, anticipated and recurring disasters— requires sound awareness and advocacy within the government, EDPs, civil society and general public. Nepal’s recurring bouts with floods, landslides, droughts, fires, and disease outbreaks, in addition to its vulnerability to a potential large￾scale earthquake necessitate that the U.S. Government and its partners develop sound and comprehensive preparedness and response plans. The Contractor must continually seek creative opportunities for incorporating disaster risk reduction and preparedness in Contractor activities. This is, and will continue to be, an area important to reducing the impact of future natural disasters. The Contractor is expected to ensure that trainings and events include disaster preparedness and risk reduction elements, and that all activities (particularly those related to Health Facility Management Committees) take disaster resiliency (including seismic, flooding and fire) considerations into account. The Contractor must develop an Emergency Preparedness and Response Plan which incorporates preparedness and response for the Contractor and sub-Contractors as well as continuity of operations. C.4.1.17 Performance Management Plan (PMP) and Data Collection Plan (Deliverable and Tasks) Monitoring and evaluation will be an important aspect of this activity. The Contractor must allocate at least five percent of the total estimated cost of the award towards monitoring and evaluation. This five percent covers all seven objectives as well as the Capacity Building Plan. Decisions on the allocation of monitoring and evaluation resources must be made in collaboration with the COR. Once the H4L activity begins, the Contractor will finalize a detailed PMP with inputs from key managers in the DOHS and USAID/Nepal. The Contractor will develop the draft PMP and manage the process of revising and finalizing the PMP, subject to COR approval. The Contractor, in its semi-annual and annual reports, will report on data with the frequency and source specified in the PMP. When possible, all data will be disaggregated by sex. The Contractor must consider further disaggregation by caste, SES and age group when feasible and meaningful for programmatic purposes. The Contractor will also report on costs-per-objective to facilitate the analysis of the cost effectiveness of various programmatic approaches. The Contractor must periodically conduct its own data quality assessment to ensure the accuracy and reliability of the information submitted to USAID/Nepal and the DOHS. USAID/Nepal, as required in its own PMP, will conduct periodic external data quality assessments of all of its on-going programs, including H4L. The Contractor must use various approaches and technologies for program monitoring and evaluation, including use of GIS (see below) for mapping needy areas and targeting 71 program focus, and use of mobile technologies and text messaging for proper recording and timely reporting. USAID encourages the Contractor to be innovative in its approaches to collecting, analyzing and disseminating information. C.4.1.17.1 Requirement to Integrate GIS Mapping into the PMP As per USAID Forward policy, USAID/Nepal is committed to spatially portraying all of its activities. To meet this requirement, the Contractor must develop a proper data collection and performance monitoring system that includes reporting geo-enabled performance management data to USAID Nepal. Implementation of GIS System – Throughout the period of performance, the Contractor must: a) Document digital spatial data according to Federal Geographic Data Committee (FGDC) Level 1 metadata standards (see www.fgdc.gov); b) Deliver to USAID digital copies of spatial data with accompanying metadata; and c) Make spatial data available to the public at the cost of reproduction. All spatial and geographic information system activities financed by USG federal funds must comply with OMB Circular A-16, Executive Order 12906, and the Freedom of Information Act. The Contractor may refer to USAID Automated Directives System chapters 507, 551, and 557 for additional guidance on how USAID uses GIS data. Should the Contractor require, free GIS tools are available from the U.S. Government at http://www.fgdc.gov/metadata/geospatial-metadata-tools. For additional requirements and guidance see Attachment 7 of this Contract entitled” Geographic and Management Information Systems Policy Guidance. C.4.1.18 Evaluations In addition to the aforementioned PMP requirements, two types of formal external evaluations6 will be conducted to ensure that there is an impartial and broad assessment of progress and constraints. An external mid-term performance review will be conducted around January 2015. This assessment will be commissioned by USAID/Nepal to an external agency. The evaluation led by an external team leader, but will involve USAID/Nepal, USAID/Washington, GON staff, and external consultants. The purpose will be to assess progress to date, determine whether any significant changes are needed and to define any adjustments necessary to improve implementation. Data for the assessment of progress will come from the MOHP HMIS, as well as district and activity records. An external final evaluation will be conducted in the final year of the program. This evaluation will also be commissioned to an external agency to assess overall activity performance, determine whether targeted objectives and the purpose of the activity have been met and provide USAID/Nepal and the GON with critical advice about lessons learned and future directions. The focus of the evaluation will be on assessing the development hypothesis which is that the Government of Nepal (GON)’s capacity to plan, manage and deliver high quality and equitable family planning, maternal, newborn and child health services can be greatly improved with cross-cutting health system strengthening interventions. This evaluation will have an impact assessment component, as data from the NDHS 2015 will be available. NDHS data from 2011 and 2016 will be utilized for core program districts and appropriately selected comparison districts to use counterfactual analysis. 72 The Contractor must cooperate with the evaluations, including answering evaluator questions and providing requested documents and access to activities. C.4.1.19 First Strategic and Activity Planning/Team Building Session (Task) The Contractor must make arrangements for and host a Strategic Planning and Team Building meeting in Nepal that includes Contractor personnel and USAID staff. This first session will primarily serve to define a strategy consistent with interests and views of US Government stakeholders in this activity and reflective of the operational environment. The first strategic planning/team building session may be one to two days, or longer, as determined in consultation with the COR. It should be held after delivery of the identified assessments noted in Section C.4.1.13. C.4.1.20 Capacity Building Plan (Deliverable) The Contractor, in consultation with relevant GON counterparts and stakeholders, must develop a life of project capacity-building plan for building Nepali capacity related to the first six objectives. This plan must include a brief assessment of current capacity, and measurable targets for capacity development and select measurable benchmarks to track progress related to capacity development during the activity at the end of year one. The Contractor must include selected capacity building benchmarks and targets listed in Section C.2 in the Capacity Building Plan and incorporated into the PMP as appropriate. Capacity building targets and benchmarks for all following years will be incorporated into annual work plans. The Capacity Building Plan and PMP will be reviewed on at least an annual basis. Both the COR and the Contractor may propose additions and modifications, which will be incorporated into the PMP and Annual Work Plan, subject to COR approval. C.4.1.21 Results Framework (Deliverable) The Contractor must develop a Results Framework for the activity, including identification of all US Foreign Assistance Framework (F) and non-F indicators that must be tracked. F Framework indicators and other non-F indicators (as determined by the Results Framework) will be decided upon in consultation with the COR 6 As defined in the USAID Evaluation Policy, January 11, 2011 C.4.1.22 Review Meeting of First Three Months (Task) If requested by USAID, the Contractor must host and participate in a review meeting of the first three months. Timing, participation and agenda will be determined through discussions between the COR and Contractor. This event will review lessons learned and experience from the Start-up Phase, plans for commencement of subsequent phases, and all deliverables from programming of activities that occurred to- date. C.5 CONTRACTOR PERSONNEL RESPONSIBILITIES The Contractor must hire sufficient numbers of each non-key personnel position to meet overall activity goals, but within program budget limitations. The Contractor must take into consideration that the keys to success for such an activity are systems, management, and logistical/procurement operations. Therefore, the Contractor must strike an appropriate 73 balance between the number of personnel dedicated to the activity, the ratio of program to operational costs, and the need to expand or contract. In its operations, the Contractor must include women and marginalized groups in all levels of planning, decision- making and implementation. It must endeavor to have its staff include women and members of marginalized groups - no less than one-third of the professional staff must be women and/or marginalized groups. C.5.1 Key Personnel & Other Staff Hire and replacement of key personnel are subject to USAID approval. The Contractor must employ Key Personnel with leadership, project and organizational management experience, effective communication skills, sound judgment, and demonstrable skills in the areas of health systems strengthening, capacity building, and maternal, newborn, child health and family planning. In addition to these abilities and the qualifications listed below, the Contractor must ensure that Key Personnel also have experience in the following areas: monitoring and evaluation, gender and social inclusion and adolescent health. Below are the required key skills and capabilities of the key personnel. C.5.1.1 Requirements for Key Personnel C.5.1.1.1 Chief of Party (COP) The COP must provide overall technical and administrative leadership and expertise for the first three years of the Health for Life activity in Nepal. The COP serves as the primary liaison with USAID/Nepal on contract management and technical matters and must adjust programs and operations in response to USAID/Nepal contractual or technical direction from the Mission’s Contracting Officer and the Contracting Officer’s Representative (COR). The COP must also ensure that technical assistance and expertise provided under the contract is timely, responsive, and of high quality. The COP position, at a minimum, requires the following qualifications: • At least 7 years of relevant management, supervisory, and technical experience working in complex development programs related to health systems strengthening that are of similar scope and scale to the H4L activity in Nepal; prior experience as a Chief of Party strongly preferred • Demonstrated technical expertise in one or more relevant areas for institutional capacity-building (i.e. health administration/management, application of modern quality improvement methods to health care, strengthening service delivery, etc.) as applied to one or more of the key technical areas discussed in Section C.2. • Minimum of a Masters Degree in a relevant field, such as public health, health administration, nursing, medicine, etc. • Proven capacity in building and effectively supervising a diverse team of employees • Demonstrated effective interpersonal skills, leadership, creative problem￾solving and ethical management • Prior experience working with international donors and knowledge of Federal Government policies and procedures preferred • Excellent oral and written communication skills • English language fluency is required and Nepali language skills are strongly desirable • Knowledge of Nepal and experience in Asia is strongly desirable 74 C.5.1.1.2 Deputy Chief of Party The Deputy Chief of Party must possess at least a Master’s degree in a relevant discipline and at least 7 years in progressively responsible management and/or advisory roles in complex health programs. S/he should have prior experience as a Deputy Chief of Party or in a senior advisory role; experience with U.S. Government programs and USG program requirements are preferred. S/he should have demonstrated technical expertise in one or more relevant areas for institutional capacity-building (i.e. health administration/management, application of modern quality improvement methods to health care, strengthening service delivery, etc.) as applied to one or more of the key technical areas discussed in Section C.2. Since liaising with a broad range of stakeholders is essential to this position, advanced oral/written communication and interpersonal skills are essential. S/he must be fluent in Nepali and have basic competency in English. C.5.1.1.3 Health Systems Team Leader. The Health Systems Team Leader must possess at least a Master’s degree in a relevant discipline (Health Business Administration, Public Health Administration, Public Administration) and at least 7 years of experience in change management of health systems in developing countries. S/he should have extensive experience in health system analysis and a proven capacity to identify solutions to challenges faced by health systems in developing countries. S/he should have significant experience in performance planning and management of decentralized health systems and may have experience in quality improvement systems, logistics systems and/or experience translating evidence to policy. Since liaising with a broad range of stakeholders is essential to this position, advanced oral/written communication and interpersonal skills are essential. S/he must be fluent in Nepali and have basic competency in English. C.5.1.1.4 Health Services Team Leader The Health Services Team Leader must possess at least a Master’s degree in a relevant technical discipline and at least 7 years of progressively responsible experience working on health service delivery management, quality improvement, reproductive health or human resources for health. S/he must have proven experience managing and motivating a team in a leadership role. Since liaising with a broad range of stakeholders is essential to this position, advanced oral/written communication and interpersonal skills are essential. S/he must be fluent in Nepali and have basic competency in English. C.5.1.1.5 Gender Equity and Social Inclusion Team Leader The Gender Equity and Social Inclusion (GESI) Team Leader must possess a Master’s degree in a relevant technical discipline or a Bachelor’s degree and at least 7 years of progressively responsible experience related to gender and social inclusion programming. S/he must have proven technical capacity to integrate GESI into health activities as well as proven ability to provide training and education to people from a wide range of background on the meaning, importance and implementation of GESI. S/he must have GESI-related experience in behavior change, health service delivery and/or systems strengthening. Since liaising with a broad range of stakeholders is essential to this position, advanced oral/written communication and interpersonal skills are essential. S/he must be fluent in Nepali and have basic competency in English. C.6 CONTRACTOR PERFORMANCE MONITORING 75 The Contractor’s performance will be evaluated based on the completion of specific tasks as outlined in the contract, adherence to the work plan, and reports submitted to the COR. USAID will conduct a number of reviews and evaluations: management reviews, financial reviews, program performance reviews, annual contractor performance reports, and evaluations. The Contractor must cooperate with and contribute to these reviews and evaluations. Progress Reviews Each January, six months prior to the end of the Nepali fiscal year, USAID/Nepal, MOHP representatives, the DOHS and the Contractor will conduct a review of progress and constraints during the past 12-month period as the basis for negotiating the next annual H4L work plan. The review of progress will assess whether specific benchmarks have been met and whether to continue the activities as planned or modify the activities. Costs and benefits of activities will also be assessed during this process. The Contractor will function as the secretariat for organizing these annual sessions on behalf of senior managers from DOHS and USAID/Nepal who will assess progress, discuss constraints and make decisions about the program for the following year. This process will allow for a maximum level of transparency about the responsibilities and commitments of each party and foster inclusive decision making about how to maximize outcomes for funds invested in certain areas. The specific activities within the contract will be adjusted and agreed upon based on this review, subject to COR approval. Initial Management/Financial Reviews USAID reserves the right, and the Contractor must expect, and be prepared for, a management and/or limited financial review by USAID/Nepal at any time to ensure systems (management, administration, finance, procurement, and program) are in place. Such review(s) will focus on program management and performance, including such factors as cost, timeliness, and accountability; and will include field and home office records pertaining to operations and program activities. The Contractor must cooperate with and contribute to a final management and financial review conducted by USAID/Nepal prior to program closeout. Contractor Performance Reports On an annual basis the COR will complete a contractor performance report (CPR). Contractor evaluations will focus specifically on the contractor’s stated responsibilities and contractor-submitted work plans. C.7 RECURRING PLANS AND REPORTS 1. Annual Work Plans –The Work Plan must address the following points in a coherent and concise presentation that can be reviewed quickly by USAID management: (a) the performance objectives and capacity building benchmarks for the period; (b) the expected activities to be undertaken to reach annual objectives including clearly defined approaches and activities focused on measurable impacts in GESI; (a) the entity responsible for the completion of the activity (i.e. Contractor, named sub￾contractor, etc); (b) how the annual objectives and activities contribute toward achieving contract objectives; (e) expected commencement and completion dates of the activities; (f) cost estimates for each major category with a supporting narrative, including proposed international travel; (g) critical assumptions or support needed from USAID, for example waivers or proposed sub-contracts, and other activity partners to accomplish the work; 76 (h) dates on which reports are to be submitted; (i) procurement plan; and (j) revised Marking Plan for specific activities. Work Plans will include a brief summary of the Contractor’s operational and administrative requirements and plans. All Work Plans will include strategies to address GESI-related issues. These Work Plans may be updated as necessary to reflect changes in strategies and/or activities and updates on performance targets, subject to COR review. See also Section B.5 paragraph (c) for additional component of Annual Work Plan which is subject to CO approval. 2. Monthly Report - These reports must briefly detail: a. program highlights, achievements, and major activities; b. problems encountered and proposed remedial actions; c. status of previously identified implementation issues. The format will be determined in consultation with the COR, and is likely to be restricted to no more than 3 pages. These reports are only reoccurring during the first six months of the activity. 3. Semi-Annual Report - The report must include: a. a review of actual accomplishments towards meeting the intended outcomes for the period (as defined in the work plan, GESI Strategy and Capacity Building Plan); b. explanations for why intended outcomes were not met (if applicable); c. Information on management issues, including administrative problems or problems with implementing partners or community groups and the steps taken to resolve them; d. Anticipated future problems, delays, or conditions or constrains that may adversely impact implementation of the project and plan to deal with them; e. Status of finances and expenditures- and when appropriate, analysis and explanation of cost overruns or high unit costs; f. Success stories and good practices. The COR will provide guidance on the format and due date of the first Semi-Annual report during the H4L Start- Up Phase (first 45 days from effective date of award). 4. Annual Report - The report must include: a. A comparison of actual accomplishments against intended outcomes for the period in the Annual Work Plan, Capacity Building Plan, GESI Strategy and Monitoring and Evaluation Plan; b. Information on major challenges and constraints faced during the performance period that resulted in delays of achievement of outcomes, if applicable; c. Highlights of the internship program as described in Section C.2.8.1 d. Cumulative quantitative monitoring and evaluation data, including information on progress 77 towards targets, and explanations of any issues related to data quality; e. Information on the status of finances, including expenditure data based on the budget and accruals, as well as, when appropriate, analysis and explanation of cost overruns or high unit costs; f. Information on management issues, including administrative problems, or problems with beneficiary groups, or implementing partners and what steps or actions were taken to manage these and lessons learned for future; g. Anticipated future problems, delays, or conditions that may adversely impact implementation of the project and what measures are in place to deal with these; h. Information on security issues, especially as these affect program integrity and safety of beneficiary groups and implementing partners; i. Other information such as new opportunities for program expansion, lessons-learned and success stories, and prospects for the following year’s performance. 5. Final Report - This report must include: a. the achievements of the projects, including the aspects that did not work well b. good practices that are replicable in other projects; c. effectiveness of different activity tools and methods; d. recommendation for similar interventions in future; e. collection of success stories; f. case studies highlighting changes or approaches/methods that were effective; g. challenges and obstacles that the program faced and the measures that were helpful in dealing with challenges. The actual format for a final report will be determined in consultation with the COR. [END OF SECTION C] ----------------------END OF MODIFICATION #7------------- ---------- 78 ANNEX 2: GETTING TO ANSWERS MATRIX Evaluation questions Key elements of the question Type of answer needed 8 Data sources/ collection methods Sampling or selection criteria Data analysis method(s) / limitations 1. How is H4Lcore addressing health system gaps identified in the problem statement and activity baseline? 1. Overall progress Descriptive H4L indicators and targets from PMP, annual reports Number and quality of policies developed Benchmarks met Number of deliverables completed Comparison of targets with results 2. Local health governance, Collaborative Framework, VDC health planning challenges Descriptive HFOMCs assisted and functioning Project reports KIIs – MoFALD, DDC, DPHO, HFOMC, VDC Expenditure tracking data Baseline and midterm survey KIIs will be selected from districts in which H4L is performing both well and poorly according to indicators 3. Quality improvement: Comprehensiveness, feasibility, challenges Percentage of encounters that met standards Percentage of satisfied mothers in ex KIIs – Management Division, DDC, DPHO, HFOMC Health facility QI data analysis from pilot areas, baseline and mid-term survey data Exit interviews Same as above 8e.g., descriptive, comparative, normative, cause and effect, etc. 79 Evaluation questions Key elements of the question Type of answer needed 8 Data sources/ collection methods Sampling or selection criteria Data analysis method(s) / limitations Encounter observations Audit of medical records 4. Data use at local level to improve MNCHN/FP service delivery, HMIS analytics Village health situation analysis reports, VDC health plans KIIs- MOFALD, DDC, DPHO, HFOMC, VDC Same as above Structured interviews/risk of subjectivity 5. Mobilization of CAP/Rs to improve marginalized groups’ access to health services. Unit cost analysis Sustainability CAP/R pregnant women tracking data (mobile phone) KIIs – DPHO, HFOMC, health facilities, FCHVs, health workers, Family Health Division, HD, e￾health unit, CAP-R, FGDs with pregnant and postpartum women Same as above 6. Capacity building of health workers (FP, CB￾IMNCI) Trend analysis: HMIS data KIIs – DPHO, FHD, CHD Same as above 7. Demand for health services (Radio Bahas, SMSing for adolescents, learning circle, school SMS text data KIIs – NHEICC, DPHO, GIZ, UNFPA Same as above 80 Evaluation questions Key elements of the question Type of answer needed 8 Data sources/ collection methods Sampling or selection criteria Data analysis method(s) / limitations health, use of FM stations) 1. Organizational capacity building of H4L logistics contractor TOCAT KII – Lifeline 2. To what extent is the structure of H4Lcore activity sufficiently flexible to respond to the MNCHN priorities of the MoHP as outlined in the NHSP? 1. Flexibility to operate in areas of MNCHN/FP 2. Comparison of H4L’s approach with NFHP’s approach to support MNCHN/FP activities and HSS (focus on local governance vs. service delivery model for HSS: need, relevancy with reference to MDGs and SDGs) 3. Review assumptions about what can be accomplished by this type of project: Is H4L able to address the fundamental and underlying causes and issues hindering positive change and improvements in the health system? Is the time frame realistic to make the changes expected? Normative KIIs Same as 1 Structured interviews/risk of subjectivity 81 82 ANNEX 3: DOCUMENTS REVIEWED H4Lcore documents 1. Project description document – SOW 2. M&E plan 3. Semiannual/annual reports 4. Health Facility Readiness Survey – baseline report 5. Assessment reports on ANM, local health governance, adolescent health, 6. Technical and policy briefs 7. GESI strategy 8. Capacity building plan 1. Annual work plans 2. Sentinel sites baseline report Government of Nepal 1. Collaborative Framework 2. NHSP II 3. NHSS 4. MoHP GESI strategy 5. Demographic and Health Survey 2011 6. Annual report, DoHS USAID documents 1. Nepal Family Health Program Qualitative Evaluation 2007 2. Nepal Family Health Program Quantitative Evaluation 2007 3. Nepal Family Health Program II Final Evaluation 83 ANNEX 4: PERSONS AND ORGANIZATIONS INTERVIEWED Personally identifiable information has been removed. 84 ANNEX 5: DATA COLLECTION INSTRUMENTS QUALITATIVE DATA Focus Group Discussions: General Instructions Each FGD will start with an introduction of why the participants have been asked to speak with the team, and why they are being asked three questions about: Access • Why did you come to this facility today? Are there any other places where you can go to get health care? • What are some of the health problems people have in this village? What do people do when they have these problems? Quality • How was the care you received today at this local health facility? • How do you feel about the care you/your children receive? • What change(s) have you noticed regarding the care provided in this clinic in the last year or so? Gaps • What can be improved or you would like to see happen at this facility? • What are some of the problems women have in this village? Moderator Instructions 1. Make sure all topics under the main themes are covered. 2. Do not judge or give your opinion as to how the participants should answer the question. Let them answer and tell their stories for why they think or feel a certain way. 3. Make sure that everyone in the group gets a chance to discuss the question. 4. Probing questions: a. Is that so? What makes you say that? b. Can you give me an example? c. Have any of you experienced something different, similar, anything else? d. Why do you say that? 5. Please use probing questions by each target group if time permits. Note Taker Instructions 1. Write down exactly what people say. 2. If there is a good quote, please take it down word for word. 85 3. When the FDG is done, translate what you have written down. Do not summarize. H4L MTE Document Review Form Document Number: Reviewer: Title of the document: Evaluation Questions Facts and Evidence (quote, reference, and/or page number) Gaps identified in relation to objective 1 Health governance Gaps identified in relation to objective 2 Evidence-based policies Gaps identified in relation to objective 3 Strengthened stewardship of the GON Gaps identified in relation to objective 4 Quality improvement Gaps identified in relation to objective 5 Improved MNCHN/FP capacity Gaps identified in relation to objective 6 Improved knowledge, attitudes, and practices and use by target pops Gaps identified in relation to objective 7 Lifeline strengthened Examples of flexibility Examples of lack of flexibility Effectiveness Project design 86 Document Number: Reviewer: Title of the document: Evaluation Questions Facts and Evidence (quote, reference, and/or page number) Manner of implementation and processes Return on investment Evidence of sustainability and ownership Recommendations for next level of Nepalese MNCHN program Other observations 87 CAMRIS/ MEL Project Midterm Evaluation of H4L Project/USAID Key In-depth Interview with the members of HFOMC and QI teams Please interview as many members as are available Guideline 2016 Informed consent form  Namaste! My name is ............. I am working for CAMRIS, MEL Office to collect data for evaluating the USAID-funded H4L project. During this interview, I will ask you some questions that will be about health system governance and empowerment of staffs and volunteers to deliver the MNCHN services. It is important that you provide correct information. The information given by you will be strictly treated as confidential. Nobody will know whatever we talk about because your name will not be mentioned on this form. All the mentioned information will be used only for the evaluation purpose. This will take about half an hour/an hour.  Your participation in this assessment is voluntarily. There is not any possible risk and direct benefit for you. It depends on your wish to participate in this assessment. You may end this interview at any time you want to. But I hope you will participate in this interview and make it a success by providing correct answers to all the questions.  Would you be willing to participate?  1. Yes 2. No  Date:______/______/2016 Please write the important “quotes” from respondent Objective 1: Health governance General performance questions • What are some of the good things that are happening in this facility? • What are some of the health problems people in this village have when they need to come to this facility? Could you give me an example? • What do you usually do when people have health problems? • What are some of the new things the HFOMC is doing now? • How did you get to do these things? Access to and use of MNCHN services • In your knowledge, why do women and children come for MNCHN/FP services? o Do people in this village know how to time the birth of their children? o What type of health problems do children have in this village? o What does the facility do about these problems? What do they do if someone is in a serious condition? o Have there been any births/deaths lately? Did everything work out as expected? • How is the quality improvement team working? What quality improvements have they introduced? • Are there any lessons learned/best practices that could be shared? • What are some of the new services you want to introduce? Why? What are the services you want to strengthen? • What do you expect from the project? From the DPHO? The MoHP? MoFALD? Overall 88 • Is there any duplication and overlap of work? How do we reduce or avoid this? • Any room for fostering coordination? Thanks To Interviewer: Please gather copies of minutes 89 H4L MTE Key Informant Interview Form Interview Agenda and Minutes Date: Interviewee’s Name: Email: Phone: Cell: Interviewer(s): Questions / Topics “Quote” and Notes Effectiveness What do you think are the main priorities and objectives of the MoHP now? How well do you think USAID contributes and aligns with government’s priorities and policies? Can you give me an example? How familiar are you with the H4L project? What do you think of the project’s activities and interventions to improve quality and coverage of MNCHN/FP services? Are you familiar with policy changes and improvements at national/district/VDC level made in the last two years or so? Are these polices working well in your opinion? Can you list three successes or things that are working well in the health sector? • At district level? • At national level? Flexibility How well has H4L done in addressing the needs of the government at VDC levels? What do you think is the general perception regarding H4L’s ability to address required changes among other donors and implementing partners and counterparts? Gap What else could H4L have done or should do now? What do you think the H4L project should work on that it is not doing now? Have you ever requested help from the H4L project? How did that work out? How well is your own program performing now? How come? What else do you wish you could do to take your program to the next level? GESI 90 Interview Agenda and Minutes Date: Interviewee’s Name: Email: Phone: Cell: Interviewer(s): Questions / Topics “Quote” and Notes What are some of the challenges your program has in regards to GESI? Has H4L helped you in any way in this regard? What effective GESI interventions do you know H4L has helped with through its activities? What else can H4L do to improve GESI/the situation of women/marginal populations? Management How do the H4L staff get along with counterparts and among themselves? How would you describe the style of management of the H4L project? What would you change about the way the H4L project works? Sustainability To what extent has H4L helped improve the capacity or performance of your program? Do you feel you have achieved all the H4L project has to offer? Other questions or comments: 91 Key Informant Interview Checklist for CAP and CAP/R General Information: a. Name: ……………………….. b. Health Facility name: ………………. c. VDC/District:……………. …….. 1. When did you start working as a CAP or CAP/R? How were you selected for the position? …………………………………………………………………………………………..…………………… ……………………………………………………………………..………………………………………… ………………………………………………………………..……………………………………………… 2. Please describe your roles and responsibilities. How do you think it is helping this Health Facility? …………………………………………………………………………………………..…………………… ……………………………………………………………………..………………………………………… ………………………………………………………………..……………………………………………… 3. How are you supporting the FCHVs? What are the differences between your responsibilities and those of the FCHVs? Do you think your being a male/female makes a difference in your work? …………………………………………………………………………………………..…………………… ……………………………………………………………………..………………………………………… ………………………………………………………………..……………………………………………… 4. Have you received any training from H4L? What are some gender equality and social inclusion (GESI) concerns you have been informed about or trained on? How do you address these issues during your work? …………………………………………………………………………………………..…………………… ……………………………………………………………………..………………………………………… ………………………………………………………………..……………………………………………… 5. What are some significant changes you have been able to make in this Health Facility and community? …………………………………………………………………………………………..…………………… ……………………………………………………………………..………………………………………… ………………………………………………………………..……………………………………………… 6. Please indicate areas of improvement for reaching the MDAG population. What are some changes you would suggest in your role as CAP or CAP/R to address these requirements? …………………………………………………………………………………………..…………………… ……………………………………………………………………..………………………………………… ………………………………………………………………..……………………………………………… Thank you for your time. 92 Quantitative Data H4L FACILITY READINESS ASSESSMENT TOOL (not included here) COMPLEMENTED BY THE FOLLOWING FORM H4L MTE CHECKLIST FOR HEALTH FACILITIES INSTRUCTIONS TO DATA COLLECTOR This checklist should be completed by observing the facilities that are available and through discussions with the person in charge of the facility on the day of the visit. In all cases you should verify that the items exist by actually observing them yourself. If you are not able to observe them, then code accordingly. Remember that the objective is to identify the facilities that currently exist and not to evaluate the performance of the staff or clinic. For each item, circle the response or describe as appropriate. 1. Facility Visited (Name) __________________________________________District 2 Name of Researcher: __________________________________________ 3. Name of the respondent and contact no: ____________________________ 4. Checked by QC Supervisor: ______________________________________ 5. Opening days in a week: _________________________________________ 6. Service operation hour: _________________________________________ 7. Is a staff member always available at night for official call? (Applicable to PHCC and hospital only) Yes …..1 (Go to Question 8) No …..2 (Skip to Question 11) 8. If yes, does that staff member live or stay at this health centre when on night call? Yes …..1 No …..2 9. Is there a birthing center in this facility? Yes …..1 No …..2 Comment ________________________________________________________________ 10. Toilets: a) Are toilets available for patients? Yes …..1 No …..2 (If no skip to question 19) b) Are these toilets able to be used at present? 93 Yes …..1 No …..2 Comment ________________________________________________________________ 11. What is the source of power for the health centre? (More than one answer is allowed) a. Electricity grid …..1 b. Generator …..2 c. Kerosene …..3 d. Bottled gas …..4 e. Solar …..5 f. Wood …..6 g. Other …..7 Please specify __________________________ h. No energy source …..8 12. Where is equipment sent for repairs? Repaired here …..1 Sent to the nearest health office …..2 Sent to the district health office …..3 Sent to the district health office …..4 Sent to the regional health directorate …..5 Sent to private/commercial workshop…..6 Not sent for repairs …..7 (Skip to question 14) Don’t know/cannot remember …..8 (Skip to question 14) 13. How long, on average, does it take from equipment being sent and returning from repairs? (Note to interviewer: Fill in one of the two –DAYS OR WEEKS) A. _______________days OR B. _______________weeks 14. Have you had to interrupt services in the last 6 months? Yes …..1 No …..2 (Skip to question 16) 15. If yes, what was the reason? Reason Has this been fixed? 1. Shortage of kerosene Yes …..1 No …..2 Don’t know …..3 2. Electricity Yes …..1 No …..2 Don’t know …..3 3. Shortage of vaccines Yes …..1 94 Reason Has this been fixed? No …..2 Don’t know …..3 4. Shortage of needles and syringes Yes …..1 No …..2 Don’t know …..3 5. Shortage of petrol/diesel Yes …..1 No …..2 Don’t know …..3 6. Shortage of vaccine carriers or ice packs Yes …..1 No …..2 Don’t know …..3 7. Shortage of drugs Yes …..1 No …..2 Don’t know …..3 8. Lack of transport Yes …..1 No …..2 Don’t know …..3 9. Fridge breakdown Yes …..1 No …..2 Don’t know …..3 10. No care provider Yes …..1 No …..2 Don’t know …..3 16. Were PHC outreach activities carried out by the health post in the past three months? Yes …..1 No …..2 How many?................................ Comment ________________________________________________________________ 17. Was a “health education session” (group lecture or discussion with clients) held today? Yes …..1 No …..2 Comment ________________________________________________________________ 18. Is some record maintained for each PHC outreach clinic (ORC)? (NOTE for interviewer only: Obtain and attach an example of a client record form and of the reporting forms used.) Yes …..1 No …..2 (Skip to question 20) 95 Comment ________________________________________________________________ 19. If yes, is this record updated? Yes, for all PHC ORC service offered …..1 Yes for some PHC ORC service offered …..2 No …..3 Comment ________________________________________________________________ 20. Is there a daily activity register? Yes …..1 No …..2 Comment ________________________________________________________________ 21. What is the target population covered by the centre? 1. Children aged 0 – 11 months __________ Don’t Know 999 2. Children aged 12 – 59 months __________ Don’t Know 999 3. Women aged 15 – 49 years __________ Don’t Know 999 4. Expected pregnant women Don’t Know 999 5. Catchment population total __________ Don’t Know 999 22. Are the following materials available? (observe) Clinical protocol Maternal Health Yes No Clinical protocol Child Health Yes No Clinical Protocol Family Planning Yes No Case management guidelines Yes No Partograph Yes No Others 23. Human resources Provider Sanctioned Filled Vacant Stationed Medical Officer Health Assistant/Sr. AHW Staff Nurse Lab Tech Radiographer ANMs Others 96 Contracted person Total 7. Training Provider Total Total Trained on MNCHN Trained by Project Remarks Medical Officer Health Assistant/Sr. AHW Staff Nurse Lab Tech Radiographer ANMs Others Contracted person Total CONCLUSION Thank you for your assistance in this survey. The results will be analyzed, and a summary, without names or identifying characteristics, will be feedback to you and to the managers of the services at district and provincial level. Do you have any questions or other comments? Please feel free to ask them or talk about them now. B. Governance 1. Inclusiveness (Member of HFOMC) Total Male Female Dalits Disadvantaged Janajatis Religious minorities Bramins/Chhetri FCHVs Staffs Total 2. Frequency of the HFOMC meeting: Weekly, Bimonthly, Monthly, Quarterly, Yearly 3. Functionalities (6 criteria) 97 Criteria Yes No Meeting on regular basis (check minutes) Meeting schedule and agenda in advance Participation of women and Dalits in the meeting Provision of service in regular hours (10:00 – 13:00) Social audit conducted 4. Quality improvement committee Planning Meeting review Yes 1 No 2 5. Financial resources Source Base year 2011/12 2012/13 2013/14 2014/15 2015/16 MoHP DDC VDC/Municipality Community Total C. Strengths, weakness, opportunities, and threats Strengths Weaknesses Opportunities Threats 98 H4L MIDTERM EVALUATION RAPID COMMUNITY INTERVIEW WITH MOTHERS OF INFANTS Directions for interviewers: Start at the health post to the right and check every third household on your left for women with children under 12 months of age. If yes, conduct interview. If not, continue to the next third household, always turning to the left. When you reach the end of the village, turn back and continue checking every third household on your left for women with infants. Complete 15. Introduction: My name is (__________________); I am conducting an evaluation of the project that has been working to improve health care for mothers and children in this district. We are interviewing mothers that bring their children to the local health post to find out if they have noticed improvements and get their views on what they need to take care of their children. Have you visited the local health post in the last month? If yes, ask for consent and go on with interview. If not, thank her and go on. Confidentiality and consent: I am going to ask you questions, some of which may be very personal. Your answers are completely confidential. Your name will not be written on this form and will never be used in connection with any of the information you tell me. You may need to know that this exercise is also taking place in other health facilities that are involved in the project in Nepal. I would greatly appreciate your help in responding to this survey. I. Identification Questionnaire number: _____________________(Skip this question) District: ____________________ VDC:________________________________________ Ward: ____________________ Facility:_______________________________________ Location: Rural ----------- 1, Urban------------------2 Name of the interviewer: _____________________ II. Background Characteristics 1. Age: _______ 2. Marital status: Single: -----1, Married: -----2, Separated: -----3, Divorced: ---4, Widowed: --- --5 3. Education: how many years of education have you completed? Illiterate-----1. Primary ------2, Secondary------3, Post secondary-----4, Graduate----------5 4. Occupation: Schooling------1, Unemployed-------2, Housewife------3, Trading/Farming----- 4, Professional-------5 III. Quality and Results 5. How do you know about the services provided in this facility? Radio-----1, Television----2, Relatives------3, Friends-------4, Community awareness activities-----5, FCHVs--------6, Others (Specify) ________________________7 6. How far do you walk to get to the health post from your house? Less than 15 minutes----1, 15 to 30 minutes----2, 30 minutes to 1 hour-----3, more than 1 hour.----4 99 7. What kind of services did you receive the last time you visited this facility? (Circle as many as appropriate) Maternal Health (ANC/PNC)------1, FP------2, IMCI (Acute respiratory infection/CDD)-------3, Nutrition--------4, Immunization----------5, Others (Specify)______________________6 8. Who attended to you on your services: (Circle as many as appropriate) Nurse-----1, Health Assistant/Senior AHW-------2, Village Health Worker/Maternal and Child Health Worker-------3, Others (Specify) ____________4 9. How long did you have to wait for the given services? Less than 15 minutes-----1, 15 to 30 minutes----2, 30 minutes to 1 hour---3, More than 1 hour-------4 IV. Client Satisfaction 10. Are you happy or unhappy with the services you received in the facility? Very satisfied---- 1, Satisfied-------2, Dissatisfied------3, Very dissatisfied--------4 11. Would you please tell me what made you say that? __________________________________________________________________ 12. Did the service provider/s explain your problems properly? Yes--------1, No----------2, Don’t know----------3 13. Was the service provider easy to understand when s/he explained things? Easy to understand------1, Difficult to understand-----2, Don’t know------3, No answer--------4 14. How was the staff's attitude toward you? Friendly-------1, Partially friendly----3, Not friendly------4, Don’t know------4, No information------5 Please describe____________________________________________ 15. Will you encourage your friends or relatives to visit the facility for services? Yes-----1, No-- ----2, Undecided---------3 16. Do you have any suggestions for the improvement of services offered in this facility? Yes- -------1, No-------2, If yes, what are your suggestions? ___________________________________________________________________ 17. Did you receive any referrals? Yes--------1, No-------2, On what? (Please probe) ___________________________________________________________________ V. GESI Concerns 18. Did you receive antenatal care in this health post? Yes-------1, No------2 why not? 100 19. Did you deliver your child in the health post birthing center? Yes-------1, No------2 If not, where?_________________________ 20. Was there privacy for women for the examination? Yes-------1, No------2 21. Did you feel you were treated differently from other mothers? (some probable causes: because of sex, ethnicity and caste, language, culture, economic status, distance) Yes-----1, Never-----2, I did not notice it ------3. 22. If yes, would you tell me what makes you say that? _________________________________________________________________ _________________________________________________________________ _________________________________________________________________ 23. Are you using the family planning services in the health post? Yes--------1, No-------2 If yes, what method? If not, why not? (Please probe) _______________________ Thank you for your time and attention! 101 H4L MTE Team Email Interview: Dear H4L team member, As part of the midterm evaluation, we would like to involve all the members of the H4L team to participate and share their opinions, views, and suggestions. We believe the results of the evaluation will be more useful to you all and help you make the midterm adjustments every project has to make if everyone participates. Our evaluation is a cross-sectional view of the many activities and achievements so far. We would like to understand also the long-term perspective of those involved in the day to day of the project activities. For that reason, we invite you to reflect on your work and add your unique perspective based on your experience on the project and knowledge of the Nepali Health System. I have taken the liberty to send this message to your personal and work emails to ensure you receive it and have an opportunity to get involved in the midterm evaluation of your project. Please open the attached file and take some time today to answer the questions in the form. You can take as much space as you need, so do not be limited by the size of the box in the table. Some questions may not be relevant to you, so feel free to skip them. Please be as frank and factual as you can so that we can understand what works and needs to improve in the strengthening of the Nepali Health System and the work of the H4L project. Please email your file with your answers and comments to me only at elvira@realizingglobalhealth.com. I will personally manage the analysis of the replies. Your answers will be confidential and not shared with anyone outside the MTE team and will be used to inform our own observations. I am writing this email after having visited Banke and Dang districts, and want to thank you for all you do to strengthen the delivery of MNCHN/FP services. I also want to thank you in advance for taking the time to get involved in the midterm evaluation. Sincerely, Elvira H4L MTE Team Leader Team Member Information Contact information Name: Position: Date of hire: Email: Phone: Evaluation Questions Team member’s response a. What is your main role on the project? b. What has been your main contribution to the project so far? 102 Team Member Information Contact information Name: Position: Date of hire: Email: Phone: Evaluation Questions Team member’s response c. How would you rate the project’s performance and the results achieved so far? d. What have been the three main successes of the project? In your opinion, are these sustainable? Why or why not? e. What have been the three main challenges the project has faced or is facing? f. If you had to give a % for the time you spend on an average week on each of the project objectives, what would it be? g. Flexibility to implement your work: 1. Do you have a written job description? If yes, please attach it to this questionnaire. 2. How well does this job description reflect your everyday activities? 3. Do you have written performance goals and targets? If yes, how are they set up and how often are they measured and reviewed? 4. What criteria are used to assess your performance? h. How much flexibility do you have to plan and implement work and meet the objectives in your area of responsibility? I. What do you think will be the main legacy of the project that will be still ongoing one year after the project? Five years after the project? J. What is important for the MTE team to know and consider when evaluating the progress of the project so far? Suggestions for changes that would improve how the H4L project can work in its last two years? None 103 Team Member Information Contact information Name: Position: Date of hire: Email: Phone: Evaluation Questions Team member’s response Other observations None SAMPLE ANALYSIS TEMPLATE Evaluation Question 1: How is H4Lcore addressing health system gaps identified in the problem statement and activity baseline? THEME: SOURCES: REPRESENTATIVE QUOTES: SUBQUESTION: THEME: SOURCES: REPRESENTATIVE QUOTES: ANNEX 6: PERFORMANCE MANAGEMENT PLAN This annex presents the project’s PMP with comments by the H4L team in yellow and by the MTE team in blue S N Indicator Definition Source Baseline Year 3 (2015-16) Target Actual (Jul￾Dec 2015) OBJECTIVE 1-Improve health system governance of district health offices and sub-district level health facilities Intermediate Result 1.1 : HFOMCs hold health providers accountable for delivery of services by better mobilizing and management of local resources 1 Percent of health facilities that undertook Social Audit in the last 12 months HF Readiness Assessment (non￾representative) 30% 25% (155 VDCs) 66.3% (173 of 261 VDCs visited by H4L ) The FY target was established by NSHP II and to be implemented by PHCRD. So we aligned our target in the 14 core districts and set at 155 VDCs. H4L visited 261 VDCs to date (thru Dec 2015) and found that 173 had completed the social audits. The project has assisted with the conduct of social audits. However, it is not known from this indicator the quality of the social audits and how this will help achieve the project’s objectives and it it is used to identify needs of MDAGs. Sub-Intermediate Result 1.1.1 Health Facilities in 14 H4L districts have functioning HFOMCs. Sub-Intermediate Result 1.2.1 Sub-Intermediate Result 1.2.1 Village health plans are prepared using evidence and approved by Village Councils as an integral part of Village Development Plan Sub-Intermediate Result 1.2.2 District Health Governance Strengthening Task Forces (DHGSTFs) are functional and effective 2 Percent of HFOMCs mobilizing resources to support health services in the last FY District Report 59% 75% 97% 97% of 627 HFOMCs mobilized local resources more than NPR 50,000 during the GON’s FY 2015 (GON FY begins July 15). The project has revitalized almost all the HFOMCs in the target districts and these committees have mobilized local resources. It is not clear how they will continue to be supported so they sustain activities and use the funds to improve quality and equitable access to MDAGs. In any case, an indicator of how these HFOMC are managing their facilities would be also helpful to manage MNCH/FP program improvements. A performance scorecard could be developed for this purpose to include % of children vaccinated, % of pregnant mothers that had their ANC appt as planned,% of medicine inventory and records match, etc. Sub-Intermediate Result 1.1.2 Best practices (communitized programs, local resource mobilization, local procurement and logistics management, local human resource management, flexible financing and local innovation) adapted for local health governance in H4L districts 3 Percent of clinical staff positions filled District report 80% 90% 86.5% 86.5% of GON sanctioned positions were filled during the past 6 months, Jul—Dec 2015.The project has not done anything to facilitate the hiring of positions. Positions are all on a yearly basis and need to be advertised annually. The GON is planning a reform to ensure continuity of services. The project is not involved in this activity. Some HFOMCs have hired additional staff using their own funds, but the project is not tracking this indicator which would be more appropriate. OBJECTIVE 2- Develop and implement national evidence-based health policy Intermediate Result 2.1 Guidelines developed and implemented for National and Regional Performance Review Meetings of MoH programs (A Framework for Analysis) 4 Multiple Good Good S N Indicator Definition Source Baseline Year 3 (2015-16) Target Actual (Jul￾Dec 2015) Comprehensiveness of MoH’s information system as measured by Health Information System Performance Index (HISPIX) Very Good The reference sheet shows us measuring the index FY 2015 and at end of project but we have been doing a desk review for each semi-annual and annual report H4L. The indicator was recommended at the project’s inception but we have not found it useful to measuring our progress. The project has used the HISPIX to assess the HMIS but the tool is not a practical tool to monitor the effectiveness of the HMIS and not practical for ongoing management of performance improvement of the HMIS. Alternative HMIS performance measures linked to project activities would help monitor their effectiveness and use of the data. Measures like how many facilities can track their coverage of vaccination, ANC, Supervised deliveries, PNC, FP, and growth monitoring would help the district know which facilities need help and which are doing well. 5 Number of district reviews conducted in accordance with PHA performance District report N/A Acceptabl e (4 of 14 districts) Progressiv e or Model (4 of 14 districts) 3 Acceptable, 9 Progressive and 2 Model (14 districts) Based on a checklist completed by H4L DPCs, we rank districts as unacceptable, acceptable, progressive and model. Although we rank all 14 districts as acceptable or better, D(P)HOs are better utilizing analytics to measure progress but they are not interpreting and prioritizing to achieve better results. It is not clear how this information is used to assist districts. In the project documentation reviewed we did not find the use of “model” districts as positive deviants that would serve as models to low performing ones. The checklist in questions is a tool to organize the preparation of meeting to discuss the PHA tools, how to conduct the meeting and its follow up, but not to prioritize and make decisions that impact performance. It is not clear what happens after the meeting or who is responsible for what. The use of PHA tool would be very timely in preparation of the annual district plan and the project’s PY 4 plan. Sub-Intermediate Result 2.1.1 Data routinely used at all levels to review program performance, identify prioritized actions, and develop plans to improve performance 6 Percent of HFs displaying up-to-date monthly service coverage information on wall on the day of visit HF Readiness Assessment (non￾representative) 18% Midterm survey is underway and to be reported in next annual report 71.3% (n=282) HERD, our subcontractor that is conducting the midterm HF readiness survey, will report their findings in the coming months and we will report the midterm survey results in our next annual report. In the meantime, we are reporting semi-annually that 71% of 282 HFs visited by our staff displayed up-to-date information. However, it is not representative of the 627 HFs in the 14 districts. This coverage to be reported should also be by target population to show equitable access. However, we are not sure this is what is the subcontractor will measure. It would be useful to have an indicator that measures how many of the project supported facilities are meeting their coverage targets. The MTE team was not able to have this information to compare performance levels. Intermediate Result 2.2: NHRC prioritizes, oversees and regulates research based on established practice of Health Sector Planning and Budgeting S N Indicator Definition Source Baseline Year 3 (2015-16) Target Actual (Jul￾Dec 2015) 7 NHRC policy role revised to reflect management of health sector research network and expanded knowledge management with H4L TA Project report N/A NHRC Network and Research Clearing house operationa l Prioritization of published health research articles completed & others in process NHRC suffered from vacant leadership for one and a half years. It was filled about a year ago, which has made a considerable difference. We are definitely behind in building their capacity. We are about to subcontract to help NHRC update their library’s webportal and support research (operations) opportunities that we recommend. The GON needs a policy research agenda that meets the need to develop its health system. And indicator of how the research agenda is being implemented by H4L and other EDPs would be appropriate and useful for the MOHP OBJECTIVE 3-Strengthen national level stewardship of the health sector Intermediate Result 3.1 MoH’s next Health Sector Strategy, Third Long Term Health Sector Plan (2018- 2038) and Comprehensive Health Sector Plan (2016-2021) developed in collaboration with EDPs and TA programs 8 Health Sector Strategy, Third Long-Term Plan, and NHSS-III: 2016-2021 developed with H4L’s TA Project report N/A NHSS III in force guiding MOH and EDPs NHSS-III has been developed and approved by cabinet. NHSS III achieved with much TA from H4L. Third Long-Term Plan cancelled by MOH. We also provided TA to the new National Health Policy, 2014. This is a very important achievement of the project to have achieved this policy document. However, it is not clear how the project will align its activities with the policy it helped develop. An indicator of how the project is advancing the implementation of the NHSS would be more appropriate. Intermediate Result 3.2 State Non-State Partnerships are identified in H4L districts documented and shared for MoH scale up 9 NHEICC and/or D(P)HOs establish partnerships with Telecom and/or FM Radio to develop and implement mHealth and/or radio BCC activities Project report N/A NHEICC and/or D(P)HOs fund mHealth or FM Radio BCC activities Three out of 14 D(P)HOs have contracted local FM in the district to air Radio spots & Jingles (PSA) in this FY H4L’s subcontract to Antenna Foundation for 18 months that supported D(P)HOs and 28 local FM radio stations ended last year. A few (3) D(P)HOs have followed up by contracting local FM radio stations without H4L support. Our mHealth initiative, including subcontract to FocusOne that led to almost 200,000 adolescents engaging in the messaging, has also ended. Other partners are following up. GiZ is planning to conduct a telephone poll to assess results with our TA. It would be useful to find out the reasons for some DPHOs to continue the funding of the radio programming. The ability of these efforts to strengthen the districts’ ability to generate demand for services has not been measured yet. The need for districts to have an effective demand generation remains. OBJECTIVE 4-Institutionalize nationwide system for quality improvement Intermediate Result 4.1 Consensus built on National QI system and piloted in in H4L districts for MoH scale up S N Indicator Definition Source Baseline Year 3 (2015-16) Target Actual (Jul￾Dec 2015) 1 0 Number of district QAWC that carry out action plan in the last reporting period (four￾monthly) District report 0 14 11 Measuring this indicator is based on completion of action plans and implementation of previous trimester action plans. Therefore, implementation is monitored and results tracked. This is an input indicator. The effectiveness of these plans to deliver improved outputs is not known. Intermediate Result 4.2 D/PHO rolls out facility-based QI system in all H4L districts 1 1 General Service Readiness Index of health facility HF Readiness Assessment (non￾representative) 41% 60% 57.39% (n=284) We measure this index by survey baseline, midterm and endline to be representative of the core districts. However, we also measure it each reporting period based on visits to HFs by our district staff. It is, therefore, non-representative. As you see, the average index score from visiting 284 HFs in the past 6 months is 57%. This is an indicator of the ability to deliver but not actual delivery of quality services. Since it measures at midpoint and it is a self-assessed measure, it is not clear how useful this indicator is as a measure of quality improvement and progress of the QI system being implemented. The number of facilities that deliver MNCH/FP services according to standards should be monitored through supervision visits and added to the PMP. OBJECTIVE 5- Improve capacity of district and local health workers and community volunteers to deliver quality family planning, maternal, newborn and child health, and nutrition services Intermediate Result 5.1 Community level health innovations and programs delivering MNCHN/FP services scaled-up and maintained 1 2 Percent of deliveries conducted by skilled birth attendant (doctor, nurse or ANM) HMIS* 42% 65% 44.07% Last FY reported 55%. For the past 6 months HMIS reports only 44%, which still represents incomplete reporting because hospitals, including cooperating private hospitals, are late to report deliveries. For example, to date the Zonal hospital in Banke has not yet reported. This indicator does not track the quality of the service. This indicator is not linked to the project activities. For example, the project works in only 14 districts and does not work to strengthen private hospitals. Also this indicator does not measure coverage of deliveries in MDAGs. This indicator is not a good measure of the project’s progress. The project should report on deliveries in its target districts, disaggregate by target populations, and compare it to the national coverage to have an idea of impact. The sentinel site data should serve as measure of project’s performance and as a proxy for USAID to monitor the project’s progress. Nevertheless, the project should continue assisting to improve use of HMIS data at district and VDC level. 1 3 Percent of newborns receiving postnatal health check-up within 24 hours of birth (New Indicator effective from 2014/15) HMIS* To be updated 56% 47.20% The same applies to this indicator as indicator 12. This is indicator was reported not to be measured separately. The project reported that “it is assumed” that if delivery was supervised by a SBA, the newborn received a postnatal checkup. This indicator should be measured in the 14 target districts and disaggregated by target population to measure coverage and equity, the goals of the project. Sentinel site data should be used to measure project’s performance to allow USAID and the project to monitor progress of project activities to increase access to newborn care. Sub-Intermediate Result 5.1.1 D/PHO managed/implemented Matri Surakshya Chakki program in 1 district and CB-IMNCI in 8 H4L districts 1 4 Number of districts implementing a comprehensive integrated management of childhood illness and newborn care (CB￾IMNCI) package (New Indicator effective from 2014/15). District Report To be updated 7 0 (CHD plan delayed) 4 to 7 districts projected S N Indicator Definition Source Baseline Year 3 (2015-16) Target Actual (Jul￾Dec 2015) Rolling out the new integrated program started last FY and we supported it in 4 districts. Our support will continue. In the past 6 months we supported an additional 3 districts. We expect CHD with our support to achieve implementation in the 7 districts by the end of this FY. Training is not implementation The implementation strategy of CB-IMNCI of the project is not known. Training has been limited and supervision and quality improvement is limited to inputs. The quality of this implementation is not being measured and it should. 1 5 Percent of pregnant women protected from PPH HMIS* 47% 68% 48.31% This indicator is a proxy of IR 12, and this is assumed by HMIS, interrelated with indicator 12 that is underreported, as well as shortages of misoprostol. Recent stockouts due to the blockcade have affected the availability of misoprostol. Five of the 10 districts visited had stockouts at the time of the MTE. In any case, this indicator is a proxy and not measured separately. It is “assumed” that if the delivery was supervised, the woman received misoprostol. In addition, the project is not implementing a strategy to sustain or expand use of misoprostol in the 14 target districts. It is not clear why this indicator is part of the project’s PMP. If it is kept, the project should actively work with QIT to ensure they implement procedures to ensure deliveries meet quality standards including the provision of Misoprostol in the project-supported birthing centers. Intermediate Result 5.2 D/PHO enhanced FCHV knowledge and skills in CB-IMCI, and FP counseling services in all 14 H4L districts 1 6 Number of people trained on FP/RH and MNCH for the reporting period Project report N/A 350 0 (Target for the project was achieved in previous fiscal year) The target of 350 was achieved a year earlier so none was conducted in last 6 months. Perhaps this indicator should be cumulative to avoid confusion. We are retaining the indicator to be responsive should staff turnover demand more of our attention. We are not sure of the rationale for completing the training earlier and how that strengthened the districts to deliver quality and equitable services. Reports from districts indicate lack of trained staff in MDAG areas. The MTE team recommends that the project work with districts to help them monitor staff turnover and develop a training plan to ensure quality MNCHN/FP services 1 7 Annual protection against pregnancy afforded by contraceptives distributed (couple-years protection) HMIS* 250,072 267,980 118,004 We now seem to be on track (last 6 months is about half of the annual target). We also note that we are now in the season for VSC in highly populated Terai. The fuel crisis has probably also had some impact on performance. LAST YEAR THEY EXCEEDED THEIR TARGET (115%) AND THIS YEAR THEY ARE ON TRACK This is an indicator of the country’s performance and not of the project’s. The project reported to have implemented training on interpersonal communication and provided IEC counseling materials. The project does not monitor CYP or CPR in priority VDCs. This indicator should be adjusted measured in the 14 districts supported by the project and disaggregated by MDAG. The project should include improving the quality of FP services in its QI strategy. 1 8 Percent of reproductive age women in union who are currently using a modern method of contraception (New Indicator effective 2014/15) HMIS* To be updated 45% 35.59% The problem with this indicator is the denominator from a census and the significant migration of married men. If the HMIS is to continue to estimate modern CPR we must use eligible couples as a denominator. However, we would expect it to increase in the coming months as for CYP. MIGRATION OF HUSBANDS IS OVERESTIMATING UNMET NEED. This indicator should be linked to project’s interventions and measured in the 14 target districts and disaggregated by MDAG. Sentinel site data should be used to measure project’s performance. Sub-Intermediate Result 5.2.1 Knowledge and skills of FCHVs enhanced 83% 90% S N Indicator Definition Source Baseline Year 3 (2015-16) Target Actual (Jul￾Dec 2015) 1 9 Percent of FCHVs able to recite 3 home rules for treatment of diarrhea HF Readiness Assessment (non￾representative) 98.6% (n=492) FCHVs seem to be knowledgeable but, again, between surveys we are measuring based on our visits to HFs that is not representative. Of 492 FCHVs, most all recited the 3 home rules. 9 PER VDC This indicator should be linked to project’s interventions and measured in the 14 target districts and disaggregated by MDAG. At this time, the project is not implementing an intervention to sustain the capacity of FCHVs, who are the backbone of the Nepali health system. Sub-Intermediate Result 5.2.1 Knowledge and skills of health workers enhanced 2 0 Percent of health workers who performed all of the six actions during counseling HF Readiness Assessment (non￾representative) 40% 70% 82% (n=67) Based on observation of only 67 cases, 83% performed well. Not representative and not necessarily reliable. The baseline, midterm and endline surveys will also report on this indicator. The MTE observed very few patients, 1 o 2 in the visited VDCs coming for FP services and they did not meet standards. The performance of health workers is not monitored by QITeams. We suggest that the project should analyze health worker performance in the 14 target districts and work with DHOs to implement ways to strengthen it. Intermediate Result 5.3 ANM pre-service training improved in 4 ANM schools of H4L districts 2 1 Percent of ANM with high level of proficiency in core SBA skills Project report NA 75% N/A H4L’s TA led to a new curriculum prior to the deadline. We will be evaluating recent graduates in the coming months by the end of this FY. However, our prior evaluation showed both faculty and students demonstrating very low proficiency and competency. This indicator must be observed among the graduating students. We also suggest that the Project’s pre service expert teach at least once a week in an ANM school using the new training materials developed by the project to be a role model, and give demonstration of training skills at the 4 ANM schools supported by the project and observe lessons and give feedback to trainers. OBJECTIVE 6-Improve knowledge, behavior and use of health services among target populations Intermediate Result 6.1 Demand and utilization for MNCHN/FP services increased among M/DAG and Adolescents/Youths 2 2 Percent of Dalit women who delivered at health facility (Revised indicator effective from 2014/15) HMIS * To be updated (NDHS 2011, 26%) 40% 42.14% The baseline is taken from the NDHS 2011. The correct target is 40%, not 45%, and we are achieving it according to the HMIS. The denominator is expected live births among Dalits. This indicator should be linked to project’s interventions and measured in the 14 target districts and disaggregated by MDAG. Sentinel site data should be used to measure project’s performance. Sub-Intermediate Result 6.1.2 GBV and early marriage are mitigated in 10 districts 2 3 Percent of adolescents and youth that know the legal age of marriage for men and women School assessment 10% >90% 63% (Jun 2015) Last FY we reported a post-test of 63%. We will report again at the end of this FY. The project is not working with targets districts to strengthen their SRH program. This indicator is not linked to project’s activities. This indicator should be linked to project’s interventions and measured in the 14 target districts and disaggregated by MDAG. Sentinel site data should be used to measure project’s performance. Intermediate Result 6.2 Missed opportunities reduced at service delivery points to provide education and counseling on healthy behaviors for MNCHN/FP 2 4 Percent of post-partum women receiving counseling on comprehensive FP services Exit interviews (HF readiness assessment) 58% 80% 56.9% (n=58) S N Indicator Definition Source Baseline Year 3 (2015-16) Target Actual (Jul￾Dec 2015) The representative statistic will come from the midterm survey in a few months. In the meantime, 57% are receiving counseling based on 58 exit interviews in the past 6 months during our staff’s TA visits. This indicator is below what is expected. It should be linked to project’s interventions and measured in the 14 target districts and disaggregated by MDAG. Sentinel site data should be used to measure project’s performance. Missed opportunities to provide FP services were observed in the visited VDCs by the MTE. OBJECTIVE 7-Strengthen USAID’s contractor capacity for logistical support Intermediate Result 7.1 Strengthened logistics system 2 5 25a. Percent of USG supported service delivery points (SDPs) (PHCCs, HPs, SHPs) experiencing stock outs of any contraceptive commodity (Condoms, Pills and injectables) in any quarter LMIS** 13% 8% 7% 25b. Percent of USG supported service delivery points (SDPs) (District Hospitals, PHCCs, and Birthing Centers) experiencing stock outs of any contraceptive commodity (Condom, Oral Pills, injectables, Implant and IUCD) (Revised indicator effective from 2014/15) Stock outs for Implants and IUCDs may represent services not provided Condom: 2% Oral pills: 3% Injectables : 4% Condom: 2% Oral pills: 3% Injectables: 4% Self-explanatory. The project is providing organizational development support to the contractor. These indicators are indicators of the contractor’s performance, not of the results achieved by the H4L project. A business development strategy must be developed and implemented to ensure the growth of the local NGO that is implementing the logistics contract so it can succeed beyond the life of the project and continue supporting the MOHP with LMIS services and necessary TA. 2 6 Percent of USG supported HFs (PHCCs, HPs, SHPs) experiencing stock outs of specific tracer drugs (ORS, vitamin A, Cotrimoxazole (P), iron tablets) LMIS** 31% 20% 23.5% Seems to be on track to reach target but may continue to suffer from fuel crisis and other disruptions. As above 2 7 Percent of USG supported HFs (District, PHCCs, HPs and SHPs) that submitted LMIS reports on time in all quarters LMIS** 90% >90% 87% Stagnating perhaps because of fuel crisis and other disruptions. As above Intermediate Result 7.2 Enhanced organization capacity of Lifeline Nepal 2 8 Performance of H4L-Logistics partner￾Lifeline Nepal rated by Technical and Organization Capacity Assessment Tool (TOCAT) TOCAT 2 2.5 N/A Midterm TOACT to be completed by end of this FY. As above ANNEX 7: H4L’S STATEMENT OF DIFFERENCES IN REGARD TO THE MIDTERM EVALUATION General Comments H4L appreciates the opportunity to respond to the MTE Team’s report. We have made some general comments on the evaluation report that are followed by more detailed comments that refer to specific text. As requested by USAID, we have been succinct but we are prepared to elaborate as requested. In general, we found the MTE Report to be a disappointment in terms of its quality, the accuracy of its statements, and a waste of an opportunity to provide practical and informed guidance to achieving the important objectives of the program. The MTE team seemed to have little knowledge or understanding of HSS programs in general and sustainable approaches that lead to strengthened health systems and improved health services, in particular -- either in Nepal or at the global level. The MTE team recognized that the large number of objectives laid out in the H4L contract’s scope of work did not match the budget that was ultimately approved for the H4L project but did not take this into consideration for findings or recommendations. The team made little valuable use of data from the field, such as service utilization data from registers while visiting health facilities. Rather, the team relied on conversations, impressions and observations to develop its findings and recommendations. We did not find a comprehensive synthesis of information that would lead to an integration of findings. Rather we found findings fragmented and sometimes contradictory. The recommendations did not seem to originate from the findings and the sheer number of them would not be practical to implement given the time and budget remaining for the project. We found little evidence that the current context of Nepal was considered in making such recommendations. Furthermore, many did not take into account – or build on – H4L achievements; they also contradicted the sustainable HSS approach embraced by H4L, which is to support government and community efforts and programs – not our own. Nor was sustainability given due diligence in making the recommendations although the sustainability of H4L activities was questioned in the body of the report. Instead of addressing each of the 47 recommendations individually, we would like to discuss them with USAID, strategize together, and address the priorities in our next annual work plans. We would have appreciated recognition and at least a review of major achievements, which are alluded to but not described in any detail. The annex in which the team ticked numerous contractual commitments that H4L completed suggests that some comment in the body of the report is warranted. These include expanding access to services to M/DAG populations; establishing a local and national QI system; improving inclusiveness and local health governance; improving local budgeting and planning for ensuring sustainability; improving the quality of the next National Health Sector Strategy (NHSS); revision of the ANM curriculum; reaching and informing adolescents and youth about RH issues through the innovative and highly successful mHealth initiative and local FM radio broadcasting; introducing new school health programs; and Radio Bahas’ public hearings in hundreds of VDCs. H4L’s review of the report also found that findings by objective often reported on only a few aspects involved with achieving the objective and ignored others. For example, the MTE team focused much more on HFOMC revitalization under Objective 1 but nearly ignored other key interventions involved with local health governance, such as annual VDC health planning and budgeting. For Objective 4, the focus was on QI teams at HFs and committees at higher levels; the team ignored the major effort of consensus building and developing the QI system itself (tools and protocols) at the DoHS. We also take issue with some of the comments on H4L’s approved PMP, which was developed and monitored in close collaboration with USAID. We acknowledge that the PMP does not reflect all that the MTE team would have liked to have seen, but PMPs are not designed to be dynamic or flexible instruments; they are monitoring and measurement tools that are intended to measure change. They reflect best available knowledge at project start-up and on what can be captured within the 5-year timeframe and project budget. Last, we disagree with the repeated suggestion that H4L has not recognized service delivery as a pillar of health systems strengthening. It is clear from the project’s inception -- its documented strategies, assessments and work plans and reports approved by USAID -- that it has made a continuous effort to address service readiness and quality within the programmatic and budgetary limitations of the project. When USAID or H4L identified H4L as a health systems strengthening project and not a service delivery project the intent was to inform the audience of its focus, and to contrast it to previous projects that offered supplemental service delivery support to the government, which was not sustained after those projects ended. Specific Comments • On page 4 of the Executive Summary and elsewhere in the report collaboration with Suaahara and Sajhaderi is emphasized. H4L notes that since its inception, the project team has regularly coordinated and collaborated with both projects as well as with HC3, and, early during the project, with KISAN. • In the Findings and Conclusions of the Executive Summary and throughout the report when addressing health governance, the MTE Team refers to HFOMC revitalization as the main focus of its efforts and achievements. However, health governance achievements should encompass the development of the national Collaborative Framework and establishing and expanding evidence-based health planning and budgeting in VDCs. Citing only revitalization reveals a lack of in-depth understanding of H4L’s TA and approach to health governance in general. HFOMC revitalization is only an initial step to local planning and budgeting and improvement in quality services. The same is true when the Team refers to H4L’s achievements regarding quality improvement (Objective 4) in the same section and throughout the report by referring to our support as only establishing QI teams and training health workers to oversee and implement the QI system. H4L successfully led the effort to develop a national QI system in close collaboration with the DoHS, which included building consensus and ownership; and field testing the system. Rather than establishing QI teams itself, H4L facilitated and supported the DoHS to carry out training, which is in line with H4L’s sustainable approach to strengthening health systems. The QI system, which is currently being field tested, provides the information necessary to take action to address gaps through the local planning and programming process. H4L is also building the capacity of local stakeholders that are in a better position than the central government to respond immediately. The QI health facility assessment itself will not lead to improved quality of care. The Executive Summary and report often refer to H4L conducting only formal training, while not recognizing the inadequacies of training. We point out that H4L was contracted to do specific and required trainings. However, our capacity building approach went well beyond these trainings. H4L has introduced coaching, mentoring, networking, and collaborating as major approaches to capacity building. • Page 18 and elsewhere: the MTE Team is under the impression that H4L is not in alignment with the new NHSS, which we find to be an odd conjecture given our significant contributions to drafting it and managing the process. H4L provided substantial TA to drafting the NHSS from 2014 to 2015–contributing to 6 thematic groups, compiling evidence, drafting chapters, and managing the process as a member of the Program Development Team, led by MOH. Moreover, H4L began January 2013, whereas the NHSS relates to July 2015 to June 2020. H4L’s FY 2015-16 work plan was finalized prior to the approval of the NHSS. In any case, H4L’s work plan clearly reflects implementation of NHSP II and also the new NHSS. It is H4L that has led the way, especially in regard to the four pillars. The MTE Team also concludes that H4L is not engaged in drafting the implementation plan for the NHSS. The implementation plan is still a work in progress and H4L is making contributions. Perhaps the MTE team is confusing the NHSS and its upcoming implementation plan. • In many places in the report, the MTE Team claims that H4L made less progress addressing objectives 5 and 6. We can cite many specific achievements to counter this finding: H4L succeeded in revising the ANM curriculum with CTEVT, conducted the ANM school assessment and supported IMNCI package development and rollout to complete tasks under objective 5. H4L also supported NHEICC by implementing the highly successful mHealth initiative, local FM radio programming, a school health program and conducted hundreds of public hearings in VDCs (Radio Bahas). • On page 5, the MTE Team notes shortcomings of the contract SOW, which focuses more on activities and inputs and less on outcomes and results. H4L notes that outcomes cannot be regularly measured and the PMP (and PPR) must rely on proxy measures which are outputs (e.g., utilization). To ensure results and impact with the resources available to the project, H4L strategically focused its TA in the most marginalized and disadvantaged VDCs to measurably reduce disparities of access to and use of services. In addition, H4L introduced tracking of pregnant women to increase service utilization and reduce inequalities. There is evidence that the focus and innovation has led to results. • The MTE faults H4L for not building the capacity of the D(P)HOs to supervise and monitor. H4L has and continues to engage D(P)HO staff in all its activities both at district and VDC levels. However, H4L has found that TA to D(P)HOs produces limited results to increase equitable service utilization and improved quality of care. Because of these challenges and shortcomings the project has focused more at capacity building at the VDC level to achieve desired impact and results. • The MTE recommends focusing on objectives 4. 5 and 6 because we apparently achieved more progress toward objectives 1, 3 and 7. (We note that the MTE Team has confused objectives 2 and 3 in its report.) H4L does not agree with this finding as our integrated approach focuses on achieving all the objectives associated with strengthening health systems to improving MNCH/RH/FP services. We illustrate this approach below: completing the following cycle is critical to improving quality of care and assuring equitable access to services. As a key strategy of health systems strengthening, H4L conducted evidence-based health planning under the Collaborative (local governance) Framework, which is a new practice in the health sector, but is leading to improved service delivery. Few are working in the health sector in support of health governance but many are supporting the other building blocks of the health systems. Health for Life Annual Capacity-building Cycle • With regard to linkages between the health facility and community, the MTE cites that FCHVs were not supported by workers at HFs. We informed the MTE Team that CB￾IMNCI is now rolling out and they should not expect results at this time. The delay was not due to lack of funding or H4L as the MTE Team speculates. The delay is due to challenges in building a consensus between CHD and partners. This challenge is well known to USAID, other EDPs and implementing partners. H4L adds that HFOMCs represent communities and there are many examples of functional linkages between facility and community. More appropriate examples include engagement of Ward Citizen’s Forums, Mother’s Groups and other community-based organizations in the village health planning and budgeting process. • The evaluation states that the IPCC training H4L supported was not recorded in the national training information system. The reason for this is that IPCC training supported by H4L was led by NHEICC (so is not recorded in the TIMS of NHTC). • As we stated in our general comments, H4L takes issue with the repeated suggestion that H4L has not recognized service delivery as a pillar of health systems strengthening. H4L, as well as USAID, emphasizes that it is a health systems strengthening project to audiences that may expect H4L to have the scope of USAID’s previous projects. In fact, H4L has focused attention on equitable access to and use of services for measuring system strengthening, thereby linking governance, human resources, financing, information, drugs and commodities to service delivery. H4L adapted WHO’s capacity building and HSS model for implementation and it first appears in H4L’s capacity building plan approved during the first year of the project and integrated in work plans thereafter. • Regarding Exhibit 9: we note that the MTE team apparently could not properly interpret the exhibit results for their evaluation. The table shows significant increases in institutional delivery in all sentinel sites, including Jajarkot where institutional delivery was lower than in other districts. The Team cited only “low coverage of M/DAGs in Jajarkot.” When H4L undertook this initiative institutional delivery in Jajarkot was 24.1%; it is now at 44%. • On page 20, the report claims that H4L “project staff reported that activities to integrate and rollout the [GESI] strategy at district and sub-district levels have not been implemented.” We disagree: in fact, disaggregation of data by caste, ethnicity and location has been analyzed and used for local health planning by D(P)HO staff and HFOMCs with support from H4L staff, particularly in H4L’s high-priority VDCs with high concentrations of M/DAGs in all 14 districts. Such planning has led to expansion of PHC/ORC, mother-in-law interactions with pregnant daughters, expansion of service hours, LAFP satellite clinics, construction of additional birthing centers and VSC camps. As a result of GESI TWG meetings, in Banke, Dang and Kalikot, funds have been allocated to organize satellite clinics for LAFP focused in MDAG communities. H4L can only assume that the questioning was misunderstood based on the overwhelming evidence that contradicts this finding. Furthermore, in H4L high-priority VDCs, upon reviewing service registers, evidence can be found that disparities of service utilization have been reduced between advantaged and disadvantage groups. In addition, in sentinel sites where H4L is tracking pregnant women with mobile phones, there is clear evidence of increased equity (see the evaluation’s Exhibit 9 regarding institutional delivery that the MTE Team failed to properly interpret). There is no implementing partner more aggressive than H4L in strategizing and implementing interventions that addresses equity in the health sector in Nepal. Furthermore, H4L argued vehemently during the NHSS development workshops regarding the results framework that key indicators should be analyzed by caste and ethnicity. • On page 20, the MTE team (again) states that H4L has focused on governance at the expense of service delivery. We disagree: there is substantial evidence that H4L has successfully engaged HFOMCs in the annual health planning and budgeting process based on gaps identified by the QI assessment process. This has led to mobilizing local resources to address service delivery gaps and improve the quality of services. Formation and functioning of the QI teams are part of the national QI system and follows MOH guidelines. • On page 21, the MTE criticizes H4L for using the PMP to report progress to USAID but not to manage its activities or measure its progress. H4L disagrees. We use the PMP to measure progress with project implementation as it follows the results framework and our contractual obligations. Reporting periods are specified in the contract. Oddly, the report does not mention the PPR, which reports on service utilization linking IRs to delivery of MNCH/FP services. • On page 22, the MTE reports that there is a “clear upward trend toward increased resources being directed to the sector” in the 10 VDCs visited. These funds are being used to improve quality of service delivery yet the report fails to recognize this and make the connection between strengthened systems and improved services. • The report further states on page 22, “that a range of 30 to 40 people now assemble in each ward to discuss the health agenda and activities. We [the MTE Team] estimate that as a VDC has nine wards, a total of 270 to 360 people have directly participated in the planning process at a VDC level. Before that, only a handful of VDC staff used to plan health-related activities. It was also reported that now there is virtually no bypass of the ward citizen forum. Members of HFOMC started to visit PHC outreach clinics and extend cooperation to offer health care services. They were reported to ask about the problems of the poor and M/DAGs and listen.” H4L cites this as more evidence of systems strengthening that has led to more equitable access to and use of higher quality services. • On page 22, the report diminishes H4L’s role in developing the Collaborative Framework. Contractually, H4L was responsible for evaluating local health governance pilots in 4 districts and scaling them up in 14 H4L districts. H4L carried out an assessment of pilots together with GiZ, documented key experiences and learning related to decentralization and local governance in the health sector (see annex of the Collaborative Framework), and designed a scale-up plan, shared with MoH, MoFALD, USAID and partners. A joint meeting was chaired by the Minister and Secretaries of both Ministries as well as high￾level officials on 24 November 2013. At the meeting, the Minister requested H4L to design a national program based on the assessment. In response, H4L drafted the Collaborative Framework that was reviewed by MoH and MoFALD. On 1 December 2013, the two Secretaries approved and signed the Collaborative Framework in the Minister’s presence. The evaluation report goes on to say that the project has “not yet tested models of decentralized planning that would inform further decentralization policies and practices,” whereas the Collaborative Framework and its implementation guidelines are landmark achievements in strengthening local health governance. • On page 26, the MTE team notes that the data quality of the HMIS is poor but goes on to criticize H4L for data inaccuracies in its reporting as it must rely on HMIS data. H4L has long been an advocate for HMIS reform and has worked steadily to improve data quality and access where feasible. • On page 26, the MTE attacks H4L’s innovative tracking of pregnant women that is increasing equitable use of maternal and newborn care services in 39 sentinel sties. The findings suggest that the initiative involving the CAP and CAP/R duplicates the function of FCHVs and ANMs in tracking pregnant women and is not aligned with current MoH policy. At present the DoHS is reviewing and revising the role of the FCHV and it is questionable whether FCHVs could use the smart phone technology. Of course, as an innovation, the initiative is not yet MoH policy. The MTE team also suggests that the effort may not be sustainable. Currently, the cost is shared by HFOMCs and H4L. Oddly, in the previous 2 pages the evaluation concludes that it is too early to evaluate the success of the Collaborative Framework but it concludes that the cost-effectiveness and sustainability of a more recently implemented initiative is questionable. In addition, the CAP and CAP/R have clear scopes of work, which differ from FCHVs and ANMs for demand generation in all high-priority VDCs to address GESI barriers. During HFOMC meetings, the CAP and CAP/R share their notes regarding barriers, observations and recommendations for further actions. HFOMCs have engaged Ward Citizen Forums that this report itself cited earlier (see page 22 above). • On page 29, the MTE Team claims not to have found a capacity building plan for LLN. However, LLN has a five-year strategic plan (2015-20) developed with TA from H4L following the TOCAT baseline that H4L conducted. H4L’s TA provision has followed LLN’s annual action plan that was subsequently prepared under the five-year strategic plan. H4L has supported LLN with international STTA and a local subcontractor to provide ongoing financial analysis and management. H4L also provided computers and office equipment. ANNEX 8: RESPONSE TO H4L’S STATEMENT OF DIFFERENCES BY THE EVALUATION TEAM The evaluation team appreciates the feedback from the H4L core activity team. The evaluation team offers the following points of response: Evaluation questions: The team would like to emphasize our task, which was to respond to three overarching evaluation questions that were focused on: • The extent to which the project is addressing existing health system gaps • The flexibility of the project to respond to MNCH/FP priorities • The effectiveness of the current management structure and organization Data Collection Methods: The team feels that sufficient data were collected to support all findings, conclusions and recommendations. We used a mixed methods evaluation approach that included quantitative data, as well as in-depth interviews with key informants noted in the Scope of Work. We would like to note that qualitative interview is a well-established research method, enabling the evaluation team to capture the perspective of key stakeholders in their own words. Clarification of Findings: Rich data were captured from interviews with H4L team staff and partners, which led to lessons learned regarding staff turnover. We would like to clarify that the findings focused on the collaboration with Suaahara and Sajhedari Bikas were derived directly from interviews with project staff and thus do not reflect the opinions of the evaluation team. As well, there was consensus among counterparts, District Health Officers, partners, and the H4L core team that the revitalization of the HFOMCs was a central achievement, albeit, not the only one. We did not find the same degree of consensus regarding gender equality and social inclusion, as well as sustainably improving the delivery of MNNCH/FP services to MDAGs. Evidence captured during data collection raises a question about the sustainability of the H4L activities. This is why we raised the issue regarding the sustainability of the community action promoter/researcher (CAP/R), based on the other initiatives being spearheaded by the Ministry of Health. We feel that the tracking of pregnant mothers through established interventions be continued as well as outreach activities to key target populations (such as Radio Bahas, and SMS texting). Indeed, if cost-effective, tracking with the COMMCARE App should be adopted by all 14 project districts and the whole country eventually through a well-planned and executed strategy. Analysis: We would like to note that all qualitative data underwent a rigorous analytical process that involved multiple members of the evaluation team. As such, all findings and conclusions were derived directly from evidence. Monitoring and Evaluation Plan: We concur with the H4L team that the monitoring and evaluation plan is not a flexible tool as currently implemented. We also note that the effectiveness of the PMP has been diminished by the use of proxy measures. The quality of source data also does not allow analysis of trends. As such, the Mission might consider a data quality assessment. Recommendations: In responding to the evaluation questions noted above, we were also tasked with providing evidenced based recommendations to inform mid-course corrections. A central goal of a mid-term evaluation is to uncover programmatic opportunities that might not have been visible at project inception. The team feels that the evaluation recommendations present an opportunity for the H4L team to consider mid-course adjustments. We would like to note that many of our recommendations represent low-cost and no-cost opportunities to improve the health system in Nepal. U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523