HEALTH SYSTEMS STRENGTHENING COMPONENT FINAL EVALUATION REPORT JANUARY 26, 2018 This publication was produced for review by the United States Agency for International Development. It was prepared by Management Systems International, A Tetra Tech Company. HEALTH SYSTEMS STRENGTHENING COMPONENT FINAL EVALUATION REPORT Contracted under Order No. AID-391-C-15-00004 Performance Management Support Contract DISCLAIMER This report is made possible by the support of the American people through the United States Agency for International Development (USAID). The contents are the sole responsibility of the Management Systems International and do not necessarily reflect the views of USAID or the United States Government. Health Systems Strengthening Component: Final Evaluation Report iii ACKNOWLEDGMENTS The evaluation team is grateful to all government staff, NGOs, development partners, and stakeholders interviewed who generously gave their time and insights for this report. Because of their contributions, this report represents a collective effort and incorporates opinions and perspectives of a large community of individuals. The team is also grateful to USAID/Pakistan staff for their support and cooperation. Jennifer Katekaine led the team, while sector specialists Dr. Shahzad Ali Khan, Dr. Muhammad Khalid, and Dr. Qadeer Ahsan provided expert advice and local context in designing and implementing the evaluation and interpreting the data. Researcher Qadeer Tariq, data analyst Arif Mustunsir, and research assistants Tayaba Raza and Abid Ali Soomro contributed to data collection and analysis, and Monitoring and Evaluation Specialist Amna Khan managed the assignment technically and logistically. To all who are committed to improved health service delivery in Pakistan, we offer these observations, analysis, and recommendations in the confidence that further progress will be realized through collective efforts. Health Systems Strengthening Component: Final Evaluation Report iv CONTENTS Acronyms...........................................................................................................................vi Project Summary .............................................................................................................. 1 Executive Summary.......................................................................................................... 2 Evaluation Purpose and Questions.................................................................................. 4 Evaluation Questions.......................................................................................................................... 4 Project Background............................................................................................................................. 4 HSS Project Implementation............................................................................................................. 5 Theory of Change and Intended Results........................................................................................ 6 Evaluation Methods and Limitations............................................................................... 6 Data Collection Methods and Sources........................................................................................... 8 Data Analysis......................................................................................................................................... 8 Limitations............................................................................................................................................. 9 Findings .............................................................................................................................. 9 Findings for Question 1: Effectiveness............................................................................................ 9 Effectiveness of Subsystems............................................................................................................10 Effectiveness of Federal-Level Activities.......................................................................................17 HSS Contribution to Objectives .....................................................................................................19 Findings for Questions 2 and 3: Sustainability and Institutionalization .................................19 Sustainability of DAP.........................................................................................................................19 Sustainability of DHPMT.................................................................................................................20 Institutionalization .............................................................................................................................20 Findings for Question 4: Best Practices, Innovations, and Lessons Learned.......................21 Best Practices and Innovations.......................................................................................................21 Challenges............................................................................................................................................22 Overall Conclusions ........................................................................................................ 24 Recommendations .......................................................................................................... 25 Recommendations for USAID/Pakistan........................................................................................25 Recommendations for the Department of Health ....................................................................25 Annexes............................................................................................................................ 27 Annex 1: Statement of Work .........................................................................................................27 Annex 2: Assignment Work Plan...................................................................................................34 Annex 3: Data Collection Instruments.........................................................................................55 Annex 4: Effectiveness of District-Level Activities..................................................................147 Health Systems Strengthening Component: Final Evaluation Report v List of Tables Table 1: Project Summary................................................................................................................................. 1 Table 2: Key Informant Interviews by Level and Respondent Type....................................................... 7 List of Figures Figure 1: Health System Subsystems.............................................................................................................. 5 Figure 2: Did DAP and MTBF Contribute to Improved Service Delivery?.........................................12 Figure 3: How Did DAP and MTBF Improve Service Delivery?............................................................12 Figure 4: The Sindh Health Information System........................................................................................15 Figure 5: How Did SHIS Improve the Function of Health Systems? ....................................................15 Figure 6: Is the DHPMT Sustainable?...........................................................................................................20 Figure 7: Perceived Implementation Challenges........................................................................................23 Health Systems Strengthening Component: Final Evaluation Report vi ACRONYMS AJK Azad Jammu and Kashmir AMR Antimicrobial Resistance BCC Behavior Change Communication CHX Chlorhexidine CSO Civil Society Organization DAP District Action Plan DGHS Director General Health Service DHIS District Health Information System DHMT District Health Management Team DHO District Health Officer DHPMT District Health and Population Management Team DoE Department of Education DoH Department of Health FP/RH Family Planning/Reproductive Health GB Gilgit-Baltistan GoP Government of Pakistan HSS Health Systems Strengthening JSI John Snow, Inc. LHW Lady Health Worker LMIS Logistics Management Information System M&E Monitoring and Evaluation M&S Monitoring and Supervisory or Monitoring and Supervision MCH Maternal and Child Health MICS Multiple Indicator Cluster Survey MIS Management Information System MNCH Maternal, Newborn, and Child Health MPH Master’s in Public Health MPP Master’s in Public Policy MTBF Medium-Term Budgetary Framework NGO Non-Governmental Organization NHSR&C Ministry of National Health Service Regulation and Coordination NHV National Health Vision NNS National Nutrition Survey Health Systems Strengthening Component: Final Evaluation Report vii OHPN Office of Health, Population, and Nutrition PDHS Pakistan Demographic and Health Survey PHIS Pakistan Health Information System PKR Pakistani Rupees PPHI Peoples Primary Healthcare Initiative PSLM Pakistan Social and Living Standards Measurement PWD Population Welfare Department RMNCH Reproductive, Maternal, Newborn, and Child Health RSPN Rural Support Program Network SCM Supply Chain Management SDG Sustainable Development Goal SHIS Sindh Health Information System USAID United States Agency for International Development WHO World Health Organization Health Systems Strengthening Component: Final Evaluation Report 1 PROJECT SUMMARY TABLE 1: PROJECT SUMMARY Title/Field Project/Activity Information Contract/agreement numbers AID -391-A13-00002 COR/AOR Start date February 2013 Completion date December 2017 Location Sindh province Implementing partner John Snow, Inc. (JSI) USAID/Pakistan Mission Strategic Framework objectives addressed Increased utilization of quality family planning and maternal and child health (MCH) services Health Systems Strengthening Component: Final Evaluation Report 2 EXECUTIVE SUMMARY Evaluation Purpose and Questions This final evaluation of the Health Systems Strengthening (HSS) project, a component of USAID’s Maternal and Child Health (MCH) program, assesses the effectiveness of the project and its individual activities in achieving objectives, the sustainability of benefits, and the institutionalization of interventions. It also identifies best practices, innovations, and lessons learned, which USAID/Pakistan intends to use to inform existing and future programming, including the final work plan for a planned integrated health systems strengthening and service delivery activity. The evaluation answers the following four questions: 1. To what extent did the project achieve its objectives, and how have individual activities contributed to achieving objectives? 2. To what extent are project activities and/or activity benefits likely to be sustained? 3. To what extent has the project institutionalized various interventions within the public sector? 4. What best practices, innovations, and lessons learned can be applied to other existing or future programming in health system strengthening? The evaluation focused on two of the four project objectives, as specified in the statement of work, namely: (1) strengthening systems that will foster improved reproductive, maternal, newborn, and child health (RMNCH) service delivery and outcomes, including accountability and transparency, and (2) strengthening management capacity at the provincial and district levels. Project Background The HSS project sought to develop and support innovative, cost-effective, integrated, and quality programs and services to strengthen systems around RMNCH services for improved outcomes. It provided technical assistance to the health and population sectors at the federal, provincial, and district levels to reform and improve service delivery in a post-devolution operating environment, focusing on provincial-level oversight of the district health departments in Sindh. The HSS project addressed governance, workforce, information systems, and financing. It also provided technical support to strengthen and improve coordination of health functions at the federal level and among federal, provincial, and district governments. Key Findings and Conclusions The project built the leadership and management capacity of health workers. HSS has created a critical mass of professional public health practitioners who, if appropriately deployed, will improve leadership and management at the provincial and district levels. It conducted both long- and short-term courses, and provided technical assistance (largely on-the-job training and hands-on support) for the implementation of the other four subsystems. All participants in the training, most of whom were from the Department of Health (DoH), said the training improved their knowledge, skills, and work practices. The HSS project has facilitated data-driven decision-making. The Sindh Health Information System (SHIS) created by the HSS project has enhanced the quality of reporting as demonstrated by the improved regularity and timeliness of reporting and the accuracy of data. The SHIS has improved access Health Systems Strengthening Component: Final Evaluation Report 3 to and availability of data by consolidating multiple health data sets on a single online dashboard. SHIS enabled district- and provincial-level managers to provide oversight and make evidence-based decisions. The HSS project improved coordination between stakeholders. The project helped revitalize the District Health and Population Management Team (DHPMT), which improved coordination within the districts and between the districts and the province. This has strengthened the planning processes by facilitating joint problem identification, resolution, and planning. It has also contributed to integrating RMNCH activities, such that all vertical programs share information and coordinate their activities. Project activities contributed to overall project objectives. Regular DHPMT meetings, strengthened health information systems, and increased monitoring and supervisory (M&S) capacity led to improvements in reporting and data quality, thus enhancing oversight, transparency and accountability, and coordination between stakeholders and facilitating evidence-based decision-making. Better coordination and monitoring improved health service delivery by increasing access to and utilization, quality, and integration of RMNCH services. HSS interventions also improved leadership and management capacity by creating a critical mass of health professionals and improving knowledge, skills, and job practices. The HSS project has the potential to leave lasting changes in the way health systems function in Sindh. This design created a strong foundation for sustainability, although this foundation is threatened by the delayed release of funds for implementation of the district action plans (DAPs). Stakeholders have institutionalized interventions. The DoH established new structures, processes, and positions for institutionalization of interventions. Stakeholders have also adopted HSS￾originated activities, interventions, and approaches under each of the five subsystems. The project identified best practices in HSS. The HSS project was built on existing structures and designed around identified capacities at all levels. It was implemented through, and in close collaboration with, the provincial DoH, the district health offices, and the federal ministry. This enhanced the project’s relevance to the Government of Pakistan (GoP’s) needs and the GoP’s ownership of the results. Because of this, the government made significant investments in the form of increased budget allocations for the Sindh DAP, office space for monitoring and evaluation (M&E) cells, and human resource commitments. Summary Recommendations 1. In the design of future projects that engage directly with government institutions, USAID should emphasize including relevant administration and finance officials to streamline the release of funds to districts. 2. Future USAID project designs should articulate strategies and approaches to effectively engage all relevant stakeholders. 3. The GoP must address the frequent transfer of key officials and develop a mechanism for succession planning to ensure the sustainability of initiatives. 4. The DoH should ensure equitable engagement of relevant stakeholders such as the Peoples Primary Healthcare Initiative (PPHI), Population Welfare Department (PWD), and Department of Education (DoE). 5. The DoH should nominate provincial administration and finance offices to join the DHPMT. If these offices are involved in planning and costing activities, they will be more likely to support timely release of funds to implement activities. Health Systems Strengthening Component: Final Evaluation Report 4 EVALUATION PURPOSE AND QUESTIONS This final evaluation of the five-year (2013–2017) HSS project, one of five components of USAID/Pakistan’s MCH program, focuses on evaluating the effectiveness of the project in achieving its objectives. The evaluation was commissioned by USAID’s Office of Health, Population, and Nutrition (OHPN) to document the project’s technical performance, including the relevance and contributions of the project’s approaches, processes, and tools. USAID/Pakistan intends to use the best practices, innovations, and lessons learned to inform the final work plan for a proposed integrated health systems strengthening and service delivery activity and other existing and future programming. The primary intended audiences are: (1) USAID/Pakistan’s OHPN and other USAID/Pakistan staff; (2) John Snow, Inc. (JSI) and its HSS consortium partners; and (3) GoP health officials, civil society organizations (CSOs), and other donors and stakeholders in Pakistan’s health sector. Evaluation Questions The evaluation statement of work (Annex 1) specified four questions which are further elaborated in the assignment work plan (Annex 2). 1. To what extent did the project achieve its objectives, and how have individual activities contributed to achieving objectives? 2. To what extent are project activities and/or activity benefits likely to be sustained? 3. To what extent has the project institutionalized various interventions within the public sector? 4. What best practices, innovations, and lessons learned can be applied to other existing or future programming in health system strengthening? Project Background The USAID MCH program consists of five interrelated and mutually supportive components including HSS, family planning and reproductive health (FP/RH); behavior change communication (BCC); maternal, newborn, and child health (MNCH); and health commodities and supply chain management (SCM). USAID/Pakistan intended the HSS component to complement the other four components. The HSS component was a five-year project implemented by JSI together with its consortium partners, CONTECH International and the Rural Support Program Network (RSPN), to provide technical assistance to the health and population sectors at the federal, provincial, and district levels to reform and improve service delivery in a post-devolution operating environment, as well as provide crosscutting health systems support to public partners at the provincial and district levels and in building public￾private approaches, results-based management approaches, and community-based financing schemes. The HSS component coordinates with other MCH program partners to ensure coordination and collaboration in the development of annual work plans. The evaluation focused on two of the four HSS objectives, namely (1) strengthening systems that will foster improved RMNCH service delivery and outcomes, including accountability and transparency, and (2) strengthening management capacity at the provincial and district levels. Health Systems Strengthening Component: Final Evaluation Report 5 The HSS project designed, developed, and delivered interventions in six subsystems of Sindh’s health system: (1) Health Care Commission, (2) leadership and management, (3) DAP and medium-term budgetary framework (MTBF), (4) DHPMTs, (5) M&S system, and (6) the SHIS (Figure 1). The subsystems are interdependent, and the leadership and management subsystem is crosscutting in that it supports human capacity to implement the other subsystems. This evaluation addressed five of the subsystems, the exception being the Health Care Commission. The subsystems are designed around the six World Health Organization (WHO) health system building blocks, namely service delivery, health workforce, essential medicines, information systems, governance and accountability, and health financing. HSS project interventions in the five subsystems covered in the evaluation directly addressed five of the six building blocks, with the exception of service delivery, the strengthening of which was an indirect outcome of interventions associated with the other five building blocks. FIGURE 1: HEALTH SYSTEM SUBSYSTEMS HSS Project Implementation The HSS project initially worked at the provincial level in Sindh and in all districts in Sindh except for Karachi. At USAID’s request, the project expanded activities to the federal level in its third year. At all Health Systems Strengthening Component: Final Evaluation Report 6 three levels, the project provided technical assistance to the health and population sectors to reform and improve service delivery in a post-devolution operating environment, focusing on provincial-level oversight of the districts. The HSS project addressed governance, workforce, information systems, and financing and provided technical support to strengthen and improve coordination of health functions at the federal level and between federal and provincial governments. Within the leadership and management subsystem, the project conducted both long- and short-term training courses, and provided technical assistance for implementation of the other four subsystems. The technical assistance consisted primarily of on-the-job training and hands-on support. The HSS project supported the development of DAPs and the MTBF. Prior to the project, districts did not have plans and received their budgets from the provincial government. These were traditional budgets which increased at a fixed annual percentage that was not based on any information. The HSS project conducted health facility assessments and provided technical support to district health departments to develop district health profiles that identified specific health needs and gaps. The districts used this information to identify district priorities based on standard criteria and develop three￾year rolling DAPs and MTBFs based on needs and costed by activity. From the three-year DAPs, the districts developed annual operational plans. The HSS project revitalized the DHPMT, a forum that brings together all district health stakeholders under one platform to discuss district health issues. Its membership includes the district health office, PWD, PPHI, community representatives, non-governmental organizations (NGOs), and development partners. Prior to the HSS project, District Health Management Teams (DHMTs), the predecessors of the DHPMTs, involved only the district health offices, did not meet regularly, and did not consistently resolve issues. The HSS project also supported the development of M&S tools, checklists, and manuals and trained staff to use them. Finally, the project introduced the SHIS, which integrated all the data sources of vertical programs, including the Lady Health Worker (LHW) program and MNCH, into one dashboard. Prior to HSS, some of the information was paper-based with restricted circulation. Theory of Change and Intended Results The project’s theory of change posits that a direct investment in strengthening health systems will promote country ownership and sustainability, scale up solutions, and promote greater efficiencies. If Pakistan’s health system improves its functions to improve access to health messages and equitable, quality family planning, MNCH, nutrition, and water and sanitation services, the population will adopt healthy behaviors and increase its use of health services, leading to improved health status, particularly MCH outcomes. EVALUATION METHODS AND LIMITATIONS The evaluation used a mixed methods approach consisting of collecting and analyzing both quantitative and qualitative data from project documents, GoP documents, and key informant interviews. Quantitative data from the interviews addressed what happened, while the qualitative data addressed why and how things happened. Table 2 illustrates the distribution of key informant interviews by level and type of respondent, and Annex 3 contains the 12 interview guides the team developed for various types of respondents. Health Systems Strengthening Component: Final Evaluation Report 7 TABLE 2: KEY INFORMANT INTERVIEWS BY LEVEL AND RESPONDENT TYPE Level District/ Location DoHa PWD PPHI DoEb JSI Field Coordinators NHSR&C USAID Development Partner District Hyderabad 4 1 1 1 1 Matiari 4 1 1 1 1 NawabShah 4 1 1 1 Tando M. Khan 4 1 1 1 1 Tando Allahyar 4 1 1 1 1 Jamshoro 4 1 1 1 1 Sujaawal 4 1 1 1 1 Tharparkar 4 1 1 1 Shikarpur 4 1 1 1 Sukkur 4 1 1 1 1 Ghotki 4 1 1 1 Jacobabad 4 1 1 1 1 Larkana 4 1 1 1 Province Sindh 7 Federal Islamabad 1 2 2 Donor and development partner Karachi 1 2 Total 59 13 12 12 11 2 1 4 a. Includes the provincial DoH and the district health offices. b. The district offices of the DoE. Health Systems Strengthening Component: Final Evaluation Report 8 The evaluation began with a team planning workshop in which the evaluation team developed a data analysis plan and the data collection instruments, planned the fieldwork, and presented the evaluation plan to USAID. Data Collection Methods and Sources • Document review: Prior to the start of the team planning workshop in Islamabad, the evaluation team reviewed relevant project documents for all project activities. These documents included quarterly and annual work plans, progress and annual reports, M&E plans, standard operating procedures, and project outputs including capacity assessments and DAPs provided by JSI. The team also reviewed the National Health Vision and the National Strategic Framework for Containment of Anti-Microbial Resistance policy. The review provided the evaluation team with a good understanding of the nature and scope of project activities at each administrative level. • Key informant interviews: The evaluation team interviewed key informants representing project beneficiaries, implementers, and stakeholders. These included representatives of the Ministry of National Health Service Regulation and Coordination (NHSR&C), provincial DoH, district health offices, PWD, PPHI, and implementing and development partners. The team also interviewed USAID staff to gain a clear understanding of the scope of work, project objectives, activities, and outputs. Sampling The evaluation team used cluster sampling to keep the fieldwork practical, ensure adequate geographic coverage, and increase efficiency. A team consisting of an expatriate team leader, three local sector specialists, three field researchers, and one evaluation specialist collected data from a sample of 13 of the 29 project districts. The team selected a convenience sample of DoH, district health office, DoE, PWD, and PPHI managers and purposively selected directors of NHSR&C at the federal level and representatives of development partners in order to capture specific experiences and perspectives about the project’s effectiveness. The evaluation team collected data from 114 respondents, a 95 percent response rate relative to the planned number of interviews. Data Analysis The team organized the analysis around the data analysis plan it developed during the team planning workshop. Quantitative data collected during the interviews provided information regarding perceptions of relevance, contribution, effectiveness, sustainability, and institutionalization of various HSS interventions. The evaluation team used the Statistical Package for the Social Sciences to develop frequencies and cross-tabulations of the quantitative data. The team conducted a thematic analysis of the qualitative data to better understand the why and how of the various aspects of HSS design and implementation. The evaluation team analyzed this data using MAXQDA software. The package facilitates coding, organizing, and extracting patterns of data that fall under identified themes related to the evaluation questions. The evaluation team used deductive and inductive approaches to code the data and then used patterns and trends in the qualitative data to explain the quantitative findings. Health Systems Strengthening Component: Final Evaluation Report 9 Limitations The evaluation faced a number of limitations that potentially affect interpretation of the evaluation results. The most significant of these limitations, and the measures the evaluation team employed to mitigate their effects, are: • Attribution: The HSS project collected baseline data on some activities. Before and after comparisons are not, however, strong evidence of attribution. To partially mitigate this limitation, the team asked interview respondents to compare pre- and post-HSS practices and outcomes and explain what contributed to changes. These comparisons provided explanations of how HSS interventions affected performance and afforded a level of plausible attribution. • Inter-rater reliability: Any time multiple field teams collect data, there is a chance that the different teams will ask, or translate, questions somewhat differently, which threatens the validity and reliability of the data. The evaluation team mitigated this risk by developing the instruments collaboratively, pretesting the instruments before going to the field, and practicing using the instruments. The team also held daily feedback sessions with the research assistants to jointly take notes and summarize the main points. • Nonresponse bias: The team could not always arrange interviews with the selected informants. For example, data collection partially overlapped a provincial polio campaign, which made it difficult to arrange interviews with some of the targeted respondents. Similarly, some selected informants were not available for other reasons. Nonresponse of this nature introduces the potential for bias to the extent that the informants whom the team could not meet had systematically different perspectives than those who were available. To partially mitigate this risk, the evaluation team prioritized completing interviews with district health department personnel before the polio campaign began. It also conducted interviews with several members of each cadre of respondents and triangulated responses with other sources of information. • Generalizability: The findings of this evaluation cannot be generalized to the entire province because the sample districts were not randomly selected. FINDINGS Findings for Question 1: Effectiveness To what extent did the project achieve its objectives, and how have individual activities contributed to achieving objectives? This section first presents evidence of the effectiveness of each of the five subsystems (i.e., leadership and management, DAP and MTBF, DHPMT, M&S, and SHIS) at the provincial and district levels. It then examines the effectiveness of federal-level activities and the cumulative contribution of activities to project objectives.1 1 Annex 4 summarizes evidence of the effectiveness of individual activities. Health Systems Strengthening Component: Final Evaluation Report 10 Effectiveness of Subsystems Leadership and Management Subsystem Project activities to support strengthening leadership and management included both long- and short￾term training and technical assistance (i.e., on-the-job training and hands-on support) for implementing the other four subsystems. The project reported training 3,668 health workers, among whom 77 earned master’s degrees: 70 earned a master’s in public health (MPH), and 7 a master’s in public policy (MPP). The rest were trained in a range of short courses including management (86 trainees), development of DAP (105), development of MTBF (580), district health information system (DHIS) (2,350), M&S checklist (200), online dashboard (200), and DAP software (70). The HSS project developed linkages and partnerships with leading institutes of health management in Pakistan including Health Services Academy Islamabad, Aga Khan University Karachi, Association of Physicians of Pakistani Descent of North America, Institute of Public Health, and Jinnah Sindh Medical University Karachi to provide short-term courses and master’s degree training to officials of the DoH, PWD, and PPHI. Over half (59 percent) of 83 key informants representing the DoH, PWD, and PPHI reported participating in HSS-led training. Because the project was designed primarily to support the DoH, which was the main governmental partner, DoH staff were more likely to report participating in the training (43 of 58 respondents) than were staff of PWD (1 of 12 respondents) or PPHI (5 of 12 respondents). All of those who participated in the training reported that it improved their knowledge, skills, and work practices. Respondents reported gaining knowledge and skills and changing work practices related to M&S, problem identification and prioritization, data-driven decision-making, planning, use of the dashboard, and costing of health interventions for MTBF. Respondents mentioned that training improved trainees’ reporting, thus improving the quality and timeliness of data, which facilitated data-driven decision-making. All respondents also affirmed that technical assistance (i.e., on-the-job training and hands-on support) was useful in supporting relevant health subsystems, i.e., developing the management information system (MIS), preparing the DAP, developing M&S checklists and manuals, and revitalizing the DHPMT. When asked how the technical assistance strengthened the health system, respondents explained that prior to the project, reporting was paper-based and irregular. The MISs developed and deployed by the project provided real-time online access to data, which improved the frequency, timeliness, and accuracy of reports. Technical assistance for the DAP enabled district officials to develop plans and budgets based on identified local needs. Standardized M&S checklists and manuals facilitated structured, consistent, and comprehensive monitoring that improved transparency and accountability. Technical assistance to the DHPMT revitalized the dormant DHMT and expanded membership, which contributed to strengthening coordination between health system partners. While trainees reported that HSS-led training improved their knowledge, skills, and work practices, key informants frequently noted that most of the 77 health professionals who earned MPH and MPP degrees were still in their original jobs and had not yet been deployed in management positions where they could fully utilize their newly acquired skills. Without being appropriately deployed, these trainees may have a limited impact on leadership development. However, the Sindh chief minister only approved the management cadre in January 2018, after the conclusion of the evaluation, so management positions were not available at the time of the evaluation. JSI also developed a deployment plan for trained health workers and submitted it to the DoH in August 2016. Health Systems Strengthening Component: Final Evaluation Report 11 In addition, although the district budgets include a new line item for training, funds for the 2016–2017 fiscal year were only released in the third quarter. The late release of funds delayed training, thereby limiting the continuity of this activity and its contribution to strengthening leadership and management. Conclusions In conclusion, JSI’s training and technical assistance to support leadership and management improved trainees’ knowledge, skills, and job practices and supported relevant health subsystems. The long-term training, i.e., MPH and MPP courses, created a critical mass of professional public health practitioners who, if appropriately deployed, have the potential to improve leadership and management at the provincial and district levels. However, many key informants confirmed that many trained professionals have not yet been deployed to management positions, thus potentially limiting the immediate impact of the training on strengthening leadership and management. Recommendations To more fully benefit from HSS project support, the DoH should: • Develop a deployment plan for the new cadre of health professionals and • Release funds against the DAP budget in a timely manner. District Action Plan and Medium-Term Budgetary Framework Subsystem District health departments develop the DAP as a three-year rolling plan with the intention of identifying and instituting activities that are based on local needs. The DAP includes crosscutting activities contributing to overall service provision such as training, M&S, repair and maintenance, printing, and community events. The DAP costs planned activities to estimate the budgetary requirements for their implementation. Drawing from the DAP, districts develop annual operational plans with details of activities, their timelines, and associated costs for each fiscal year. The DAP was expected to strengthen service delivery by improving M&S, addressing training needs of local staff, facilitating needs-based planning, and mobilizing communities. Of 30 key informants in a position to speak to the contributions of the DAP and MTBF to service delivery, 19 (63 percent) believed that service delivery had improved. Respondents from the district health offices were more likely than those from PPHI and those at the provincial level to believe that the DAP and MTBF had improved service delivery (Figure 2). The difference of opinion may reflect the fact that the district health offices were very involved in the DAP and MTBF interventions, whereas PPHI had little or no involvement. Another potential reason could be different perspectives regarding improvement in service delivery. PPHI respondents may not necessarily have associated improvement in subsystems with improvement in services at the service delivery points. When evaluators asked how the DAP and MTBF had improved service delivery, respondents explained that the DAP and MTBF strengthened planning processes, created new budget lines, informed decision￾making, and supported needs-based budgeting and activity-based costing (Figure 3). The DAP is based on the needs identified in district health profiles and health facility assessments. Resource allocation decisions were based on the identified needs. Five new budget lines, namely training, M&S, community events, printing, and repair and maintenance, were included in the DAP and MTBF. Health Systems Strengthening Component: Final Evaluation Report 12 FIGURE 2: DID DAP AND MTBF CONTRIBUTE TO IMPROVED SERVICE DELIVERY? FIGURE 3: HOW DID DAP AND MTBF IMPROVE SERVICE DELIVERY? The major factor compromising implementation of the DAP was the DoH’s delay in releasing the budget to the districts. At the time of the final evaluation, the DAP budget for fiscal year 2017–2018 had not yet been released to any of the districts. For fiscal year 2016–2017, although PKR 330.9 million was proposed, only half was released to the districts, and that only in the last two quarters. Due to delayed release of the budget, districts were able to utilize only 16.5 percent (PKR 27.3 million) of the released funds.2 Data from the DAP MIS dashboard shows that almost all this expenditure took place during the fourth quarter of 2016–2017, and 69 percent of expenditures were for travel allowances. 2 District Action Plan Implementation Dashboard. Health Systems Strengthening Component: Final Evaluation Report 13 Conclusions Interventions in the DAP/MTBF subsystem improved processes that supported needs-based planning and budgeting and, consequently, service provision. However, the implementation—and consequently the effectiveness—of the DAP is dependent on the timely release of the budget, the lack of which has limited the effectiveness of the DAP in improving service delivery. Recommendations In future similar projects, USAID should ensure that district administration and finance officials are members of the DHPMT to facilitate the timely disbursement of funds at the district level.3 The provincial DoH should identify the barriers to timely release of the DAP budget and engage the provincial finance department to streamline the processes involved in releasing the DAP budget. Monitoring and Supervisory Subsystem The HSS project provided training to 200 health workers and technical assistance to the DoH in developing M&S tools, including checklists and manuals. It developed standard operating procedures for the DHIS, including at the facility level. The project also facilitated the DoH to standardize the DHIS indicators, adding new indicators as necessary. Moreover, the project introduced the M&S system designed to validate data, thus contributing to improving the reliability, accuracy, completeness, and comparability of data. The online M&S checklists completed the system of monitoring the quality of health services with continuous supervisory support to the health service providers. This M&S system is housed in the M&E cell, which is a unit within the DoH. The project established M&E cells in 21 districts and at the Director General Health Service (DGHS) office at the provincial level. All these M&E cells were equipped with information technology equipment and furniture to provide a conducive work environment as well as facilitate data verification and uploading and maintenance of the dashboard. The scope of M&S interventions consisted of the following: • Development of tools, checklists, standard operating procedures, and guidelines; • Standardization of checklists/tools for all programs and their availability online; • Training of district health teams to use online M&S tools; • Hands-on support to implement online M&S plans; • Support to upload completed M&S checklists; and • Validation of data during supervisory visits. When asked, all 58 respondents affirmed that they had used the tools and found them useful. An open￾ended follow-up question asked the same respondents to explain why they found the tools useful. The respondents attributed the usefulness to the fact that the tools are comprehensive (36 percent), facilitate problem identification (33 percent), and are easy to use (31 percent). They explained that the comprehensive tools kept them from missing anything during M&S. Regarding problem identification, they explained that the tools helped the district health officers (DHOs), the LHW and MNCH coordinators, and other supervisors identify and resolve issues or promptly raise them with responsible authorities. These issues may include stock-outs of family planning commodities or the absence of health workers in health facilities. The tools were also easy to understand, hence the ease of use. 3 JSI reported that district finance offices are members of the District Health Planning Committees notified by each district health office. These committees are responsible for planning and costing activities in the DAP. However, the evaluation team did not find District Health Planning Committees in any of the sample districts, nor did the committee come up in any of the team’s interviews. Health Systems Strengthening Component: Final Evaluation Report 14 “The minister of health himself got to know through the online dashboard that I haven’t uploaded pictures on my monitoring and supervision visit and an explanation letter was issued to me.” – LHW coordinator Conclusions Evidence suggests that the training on M&S concepts and the hands-on support for proper utilization of the online M&S system strengthened the M&S capacity of district- and provincial-level health managers, as well as the staff of Sindh DGHS, including the vertical program managers. The improved M&S systems contributed to: • Improved accuracy, timeliness, completeness, and frequency of health facility reporting. • Strengthened management oversight, and the feedback loop within and between departments. Evidence shows that the minister himself logs into the SHIS and uses M&S tools including checklists, supervisory plans, and reports to query issues and provide guidance. • Managers’ increased ability to identify and resolve problems in the context of limited resources. • Changes in mindsets about sharing information, and increased confidence in reported data among program coordinators. Data are verified and therefore more reliable, and there has been increased realization of the need for collaboration and joint problem identification and resolution. DoH has adopted the following HSS approaches for purposes of improving M&S systems: • M&S is included as a new budget line in the DAP budget. • DoH conducts planned M&S visits. However, not all the planned visits were conducted since only a portion of the DAP budget was released by the Finance Department of Sindh, and even what was eventually released was not on time. Sindh Health Information Subsystem SHIS integrated health services, logistics, and surveillance data by bringing together 12 MIS dashboards (Figure 4): M&E, DHIS, M&S, LHW-MIS, MNCH-MIS, family planning MIS, tuberculosis control MIS, vaccine logistics MIS (v-LMIS), contraceptive logistics MIS (c-LMIS), community coverage, DAP MIS, and expanded program for immunization (EPI). The integrated dashboard also provides access to key health￾related national surveys such as the Pakistan Social and Living Standards Measurement (PSLM) survey, Multiple Indicator Cluster Survey (MICS), Pakistan Demographic and Health Survey (PDHS), and National Nutrition Survey (NNS). Health Systems Strengthening Component: Final Evaluation Report 15 FIGURE 4: THE SINDH HEALTH INFORMATION SYSTEM When asked, all 51 respondents said that SHIS improved the use of information at the district level. When asked how SHIS improved the use of information, respondents explained that it improved access to data by making data available online, through mobile apps and desktop computers in the M&E cells, and by placing all the MISs in one place. FIGURE 5: HOW DID SHIS IMPROVE THE FUNCTION OF HEALTH SYSTEMS? Health Systems Strengthening Component: Final Evaluation Report 16 Better access to data strengthened planning processes by making real-time information available; by increasing health officials’ access to data for monitoring, which improved the accuracy, completeness, regularity, and timeliness of reporting; improving M&S capacity; and facilitating evidence-based decision￾making (Figure 5). SHIS IMPROVED USE OF INFORMATION “All the data is now available on the dashboard and we can easily download reports like the LHW program and health facility reports for preparing the DAP and annual operational plan. There have been considerable improvements due to this intervention.” – DHIS coordinator Conclusions The SHIS created by the HSS project has improved reporting, largely facilitated by M&E cells and attributed to shared accountability and transparency since data sets from various programs and departments are now visible to all stakeholders. By making multiple health system data sets available in a single place online, SHIS improved access to data, which has enabled district- and provincial-level managers to provide oversight and make evidence-based decisions. District Health and Population Management Team Subsystem The HSS project revitalized the former DHMT into a broad-based district forum, which was then renamed. The DHPMT was established and notified at the provincial level for all the districts. Its membership includes representatives of district health offices, PWD, PPHI, and district education offices, as well as local government, CSOs, and community representatives. The DHPMT meets every three months to discuss district-level issues pertaining to health service delivery, resolve identified problems, and ensure implementation of DAP operational activities. The minutes of the DHPMT meetings are forwarded to the provincial DoH. The evidence suggests that the revitalized DHPMT has improved health service delivery in all the districts. Most respondents (77 of 80) from the DoH (provincial and district), district DoE, and PWD believed that the DHPMT has improved health service delivery. When asked how, they explained that the DHPMT has been able to: • Provide a forum to resolve district-level issues. Respondents mentioned that most of the district-level issues are resolved through the DHPMT. • Improve coordination among district health offices, PWD, district education offices, and PPHI. Representatives of all these organizations meet in the same DHPMT forum to discuss the health issues that affect them. • Strengthen planning processes through joint problem identification and planning. • Integrate RMNCH activities, such that all vertical programs share information and coordinate their activities. • Improve coordination between the districts and the province, including on management oversight. However, 5 of 12 PPHI respondents disagreed that DHPMT has improved service delivery; this may be because they were less engaged in the project and were thus less aware of the outcomes of DHPMT. Health Systems Strengthening Component: Final Evaluation Report 17 Factors Compromising the Effectiveness of DHPMT Several factors may have compromised the effectiveness of the DHPMT. Although the DHPMT has a broad and diverse membership, stakeholders from organizations such as PPHI, PWD, and DoE believed that their voices were not heard. They explained that the DoH did not allocate as much meeting time to their issues or appear to believe them important. Other stakeholders reported that the DoH did not consult them when setting meeting agendas and dates. Key stakeholders such as the district administration and district finance officers, who are critical for resolution of management and finance-related issues (including release of funds), are not members of the DHPMT. Some respondents complained that there was no follow-up to ensure compliance with decisions taken during DHPMT meetings. Some of the DHOs reported that the provincial DoH did not resolve district￾level issues in a timely manner, which may lead to problems of compliance and impede the accomplishment of goals set in DHPMT meetings. As a result, some issues—such as those related to human resources, medicines, and supplies—remained unresolved. Effectiveness of Federal-Level Activities The federal Ministry of Health was dissolved after passage of the 18th Amendment, which devolved health services to the provincial and district levels. The HSS project supported the reinstated NHSR&C to rethink its mandate, culminating in a realignment to address international health regulations and global health commitments such as the Sustainable Development Goals (SDGs). The evaluation team conducted two interviews with representatives of NHSR&C to explore activities that the HSS project implemented at the federal level. These activities were added in the third year of the project at USAID’s request. Maintenance of Web Portal for Pakistan Health Information System (PHIS) The two federal-level respondents said that the HSS project supported the establishment of the PHIS unit and built the capacity of stakeholders to use the PHIS effectively. The main objective was to develop an integrated online PHIS dashboard—the first of its kind in South Asia—to ensure production, analysis, dissemination, and use of reliable, quality, and timely information on health determinants, health systems performance, and health status for the country. JSI provided technical assistance to upgrade the health information system for Azad Jammu and Kashmir (AJK) and Gilgit-Baltistan (GB) and trained two master trainers from each district in these regions. The master trainers in turn trained 150 health workers at the facility level. In addition, the HSS project developed trainee and trainer manuals and DHIS procedural manuals for AJK and GB. The federal dashboard is linked to the provincial dashboards and to regional routine health information systems and their dashboards. The two federal respondents, both representing NHSR&C, mentioned that NHSR&C has used the national dashboard to develop policy papers based on national data. Human Resources Support to the Health Planning, Systems Strengthening, and Information Analysis Unit The HSS project hired two epidemiologists as short-term consultants for the Health Planning, Systems Strengthening, and Information Analysis unit to analyze, interpret, and package data, and to provide evidence-based information to policymakers for decision-making. This support enabled NHSR&C to perform its post-devolution role. Both federal-level respondents believed that the consultants built the capacity of NHSR&C and reported that the consultants had handed over the PHIS control panels to the newly recruited staff of the PHIS unit. Health Systems Strengthening Component: Final Evaluation Report 18 Development of National Strategic Framework on Antimicrobial Resistance (AMR) Containment The HSS project provided technical assistance to NHSR&C to develop a national strategic framework to contain AMR. The framework is coordinated, collaborative, and sustainable as an AMR containment mechanism, with measurement outcomes under WHO’s One Health approach encompassing stakeholders beyond the health sector. 4 This is a step toward Pakistan’s fulfillment of international health-related accords and honoring of global health commitments. Implementation of the policy has begun, though the outcome and impact of the policy are not yet realized. NHSR&C has carried out laboratory assessments of AMR, supported the development of task forces at the provincial level, and identified pilot districts in which the policy will initially be implemented. Development of Draft National Health Vision (NHV) The HSS project provided technical assistance to NHSR&C to develop the NHV. The project provided consultants for each theme and supported consultations and consensus-building at the provincial and district levels. The NHV provides a road map for the health sector without overriding provincial autonomy, and guides implementation of the SDGs. The final NHV has been endorsed by the provincial and federal ministers of health. The NHV will be implemented through existing structures such the National Strategic Forum and the interagency coordinating forum. As a result of the NHV, the GoP has allocated PKR 38 billion over 5 years to a population of 80 million members of vulnerable households for the implementation of universal health coverage. Development of a Multipronged Strategy to Scale Up Chlorhexidine (CHX) Use The HSS project provided technical expertise, commodities, printed materials, and capacity building for development of a multipronged strategy to scale up the use of CHX, with the aim of reducing neonatal mortality in Pakistan. In collaboration with experts from Nepal, the project developed the strategy and revised training curricula for health providers on the proper use of CHX. Both NHSR&C respondents said that the strategy has been operationalized by forming technical working groups to standardize CHX training, importing CHX tubes, developing action plans, and training master trainers. Formation of Technical Working Groups to Standardize CHX Training The HSS project facilitated the formation of technical working groups to standardize CHX training. The process involved all provincial health departments; representatives of regulatory and professional alliances, such as the Society of Gynecology and Obstetricians of Pakistan and the Pakistan Pediatric Association; and public and private partners. These stakeholders formed the membership of four technical working groups focused on: (1) CHX training manuals, (2) social and behavior change communication, (3) M&E, and (4) local production and registration of CHX in Pakistan. The project facilitated the working groups to develop action plans for initiating pilot projects in the provinces, a communication and advocacy strategy, and curricula for training. Importing CHX Tubes The HSS project, in accordance with its action plan, imported 2.1 million CHX tubes from Nepal and distributed them to the provinces and regions. Both of the federal-level respondents said that this has 4 One Health is an approach to defeating emerging and re-emerging zoonotic diseases by improving cooperation among the animal, public, and environmental health sectors. Under International Heath Regulations (IHR 2005) and the Global Health Security Agenda, all WHO member states are required to enhance their core capacities to respond to all potential public health hazards including through prevention, control, and response to zoonotic diseases. Health Systems Strengthening Component: Final Evaluation Report 19 facilitated ongoing CHX use in 26 districts in Pakistan. There are plans in place to expand to other districts throughout Pakistan in a phased approach. Development of Provincial Action Plans and Training of Master Trainers When asked whether the HSS project supported the provinces and regions in developing national CHX scale-up plans, both federal-level respondents answered in the affirmative. As a result of the project’s support, CHX has been added to the essential list of drugs for the provinces and regions. The provinces and regions have also committed funds in their budgets to purchase CHX. In addition, the project assisted three potential local producers in all technical aspects of start-up, including registration of CHX for local production in Pakistan. The respondents therefore believed that there is no barrier to CHX use at either the district or provincial/regional level. The project also trained 121 master trainers to scale up training in CHX use. HSS Contribution to Objectives This evaluation focused on two of the four project objectives: (1) strengthening systems that will foster improved RMNCH service delivery and outcomes, including accountability and transparency, and (2) strengthening management capacity at the provincial and district levels. The overall findings of the evaluation show that the HSS project improved: • Transparency and accountability by strengthening health information systems, improving coordination between stakeholders, strengthening oversight, and facilitating evidence-based decision-making (Objective 1). Examples include regular DHPMT meetings, improved reliability of data, improved reporting, and the minister’s use of the dashboard to question issues and provide oversight. • Health service delivery by increasing access to and utilization, quality, and integration of RMNCH services (Objective 1). For example, key informants from the health department mentioned that the number of patients in outpatient departments has increased, and the quality of health services has improved due to improved leadership and better monitoring through M&S systems. This enables managers to identify problems and address issues related to the availability of supplies and medicines. • Leadership and management capacity by creating a critical mass of health professionals and improving knowledge, skills, and job practices (Objective 2). The project reported training 3,688 health workers in both long- and short-term courses. Findings for Questions 2 and 3: Sustainability and Institutionalization To what extent are project activities and/or activity benefits likely to be sustained? To what extent has the project institutionalized various interventions within the public sector? Sustainability of DAP All but 1 of the 13 DHOs the evaluation team interviewed believed that the DAP was sustainable. Respondents from the provincial departments were less confident about the DAP’s sustainability (only four of seven believed the DAP was sustainable), mainly due to uncertainty about the release of funds for DAP in the future. Health Systems Strengthening Component: Final Evaluation Report 20 When asked to describe why they believed that the DAP was sustainable, DoH respondents cited the development of transition plans, the appointment of DAP focal persons, and the allocation of a budget for DAP as indicators of its likely sustainability. Sustainability of DHPMT Ninety-seven percent of the 37 district health office representatives and 82 percent of the 11 provincial￾level representatives believed that the DHPMT will be sustained. However, other DHPMT members, such as representatives of PPHI, district education office, and PWD, were less confident about the sustainability of DHPMT, largely because they felt marginalized; they felt that their voices were not heard, the meetings did not attend to their issues, and the DoH did not consult with them when setting meeting agendas and times (Figure 6). FIGURE 6: IS THE DHPMT SUSTAINABLE? When asked why they believed the DHPMT was sustainable, the district health office respondents explained that in most of the districts, the office has nominated a DHPMT focal person, and the DHPMT has been holding quarterly meetings for a year with minimal JSI facilitation. Furthermore, little cost is involved in convening DHPMT meetings, so it requires minimal budget allocation. Representatives of the other partners—PPHI, DoE, and PWD—explained that they were largely unaware of the measures the DoH had taken to ensure the sustainability of DHPMT because they were not engaged at that level. Institutionalization The HSS project expected that all interventions or activities relevant to sustaining the five subsystems introduced in Sindh should, over the five-year period of project implementation, become standard practice in the Sindh DoH. To support this desired outcome, the project provided training and technical assistance in all five subsystems, including on-the-job and hands-on training to a pool of health workers. The project had field staff in each of the districts whose main responsibility was to facilitate the DHPMT; together with the DHOs, they drafted letters of invitation for DHPMT meetings, reminded members of the meetings a week and a day in advance, organized the meetings, prepared the minutes, ensured that the minutes were signed, uploaded them to the DHIS, and facilitated follow-up on action points. In the last year, the project deliberately withdrew from playing most of these roles, leaving the DoH to carry out these responsibilities. Health Systems Strengthening Component: Final Evaluation Report 21 Indicators of Institutionalization New Structures, Processes, and Positions The evaluation team investigated the establishment of new structures, processes, and positions as evidence of institutionalization. The team asked respondents whether, in their opinion, their institutions have incorporated the DAP, MTBF, and DHPMT as normal and regular practices. All 44 respondents from the provincial DoH and the district health offices believed that DAP and MTBF had become regular practices. When asked the same question in relation to DHPMT, all 13 DHOs the team interviewed believed that DHPMT was now a normal and regular practice in their institutions. They supported their beliefs by explaining that: • DoH has nominated focal persons for DAP, MTBF, and DHPMT; • All district health offices hold monthly review meetings in preparation for DHPMT meetings; and • Twelve of 13 districts now have refurbished and functional M&E cells with staff whose main responsibility is to monitor and verify data as well as maintain the online dashboard. Adoption of HSS-Originated Activities, Interventions, and Approaches The team also asked questions related to adoption of HSS-originated activities, interventions, and approaches under each of the five subsystems. Under leadership and management, respondents cited training as a new budget line in the DAP. Under M&S, they also cited a new budget line and noted that planned M&S visits were implemented regularly. In addition, respondents believed that DAP and DHPMT were regularized and standardized. Finally, the evaluation team inferred from the project design and implementation approach that the project had secured political commitment for the interventions and activities that should be institutionalized, as demonstrated by the notification of the DHPMT and the increased budget allocations to implement the DAP. Findings for Question 4: Best Practices, Innovations, and Lessons Learned What best practices, innovations, and lessons learned can be applied to other existing or future programming in health system strengthening? Best Practices and Innovations Best, or good, practices are processes, techniques, or implementation approaches for which the evaluation found sufficient evidence, from multiple sources, of their effectiveness. Some best practices are applicable to a specific activity or context, but many encompass lessons that are more broadly applicable to other activities or contexts. Furthermore, some may be innovative, while others are not. Because it is difficult to cleanly separate best practices, innovations, and lessons learned, this section summarizes HSS project implementation approaches that have proven their efficacy, many of which are broadly applicable as lessons learned. Adopt a systems approach: The program demonstrated the efficacy of a systems approach designed around essential health subsystems modeled on the WHO’s health systems building blocks. The evaluation presents evidence from multiple sources that this approach yielded benefits in the form of strengthened coordination between and across departments as well as between districts and the Health Systems Strengthening Component: Final Evaluation Report 22 province, increased district health financing, improved health facility reporting, increased use of information for decision-making, and improved service delivery. Build on existing government structures: A good test of sustainability is when a country, or part of a country, adopts, regularizes, and standardizes necessary approaches, processes, and tools. This is more likely when the project engages with, and implements through, existing government structures. “For us, sustainability was [the] number one objective. We decided not to start anything that the government cannot continue.” – JSI respondent Its close collaboration with government allowed the HSS project to design interventions based on the absorptive capacity, commitment, and buy-in of its primary development partner, the DoH. The HSS project was built on existing structures and designed around identified capacities and gaps at all levels to ensure that interventions were relevant and appropriate. It was implemented through the provincial DoH and the district health offices in addition to the federal ministry. Because of this, the government made significant investments in the form of increased budgetary allocations for the Sindh DAP process, office space, and human resources. Encourage broad representation in DHPMT: The DHPMT membership is comprised of representatives of the district health office, PWD, PPHI, DoE, local government, CSOs, community leaders, and development partners. The members value the forum because it brings all key stakeholders in health together to interact, discuss and resolve their district-level issues, and coordinate their activities. In addition, the forum provides an avenue for provincial-level oversight and performance review of the districts. Conduct advocacy with policymakers: The HSS project carried out advocacy with policy- and decision-makers at all three levels of government—federal, provincial, and district—which yielded ownership and buy-in. It also enabled government entities to make decisions for which they can hold themselves accountable and understand the requirements of sustaining the benefits once the HSS project ends. Examples of policy advocacy include the initial systems assessment that the project undertook and disseminated, which provided the foundation for the interventions, and the consensus-building workshops held with provincial and district stakeholders during the development of the NHV and AMR policies. Ensure country ownership: The project secured government ownership through continued negotiations, dialogue, and ongoing stakeholder engagement. Beneficiary institutions have adopted and assumed ownership of many HSS project interventions. For example, the minister of health regularly uses the online dashboard to monitor and provide oversight to provincial and district health workers; supervisors are required to upload pictures online for accountability after every supervision visit; the government uses the DHPMT and M&S subsystems regularly; and the government has allocated a budget for implementation of the DAP. Consistent use of the dashboard for performance review and feedback has increased transparency and accountability at all levels. Challenges The HSS project experienced several challenges, some of which have been documented in previous findings. The evaluation team gathered additional evidence of challenges by asking respondents directly what challenges they encountered during implementation. Figure 7 presents the challenges respondents mentioned most frequently. Health Systems Strengthening Component: Final Evaluation Report 23 FIGURE 7: PERCEIVED IMPLEMENTATION CHALLENGES Limited stakeholder engagement: Representatives of PWD, PPHI, and DoE, who as DHPMT members felt that their voices were not heard in DHPMT meetings, were most likely to cite this as a challenge. “The other departments consider teachers and the education department a lower class.” – District education officer Human resource issues: Respondents explained this issue primarily in terms of the challenges posed by frequent transfers of key personnel, which resulted in the loss of knowledge and ownership, often necessitated building new relationships and frequent orientation and retraining, and compromised sustainability. “Repeated transfers and postings in the DoH almost make it look like a game of musical chairs.” – Provincial respondent Respondents also mentioned challenges associated with the shortage of staff, particularly in relation to the polio campaigns which are demanding in terms of staff time and compete with other priority activities. Delays in release of funds: Despite making a budget allocation for DAP, the provincial government released the budget late. For example, only 50 percent of the 2016–2017 budget was released, and even that was released in the third quarter of the year. Delays in resolution of district-level issues: Respondents also cited provincial government delays in resolving district-level issues as a challenge. Health Systems Strengthening Component: Final Evaluation Report 24 OVERALL CONCLUSIONS HSS project implementation was straightforward, consisting largely of training and technical assistance to build capacity, develop tools and processes, and implement interventions in the interdependent subsystems. Consequently, the conclusions relate largely to whether the project’s training and technical assistance were appropriate or relevant to the needs of health system stakeholders, whether they were well implemented, and whether they contributed to strengthening health subsystems and improving service delivery. The training and technical assistance were unquestionably effective in imparting knowledge, developing skills, and contributing to changing work practices; all respondents from all stakeholders interviewed said as much. The effectiveness of these interventions reflects the project’s approach of collaboratively working through existing government structures to engage government counterparts to design interventions that meet identified needs and gaps and account for absorptive capacity, commitment, and buy-in from the primary government partner, the DoH. The project has the potential to foster lasting changes in the way health systems function in Sindh. By collaboratively working within existing government structures to address mutually identified needs, the project ensured the relevance of its interventions, promoted government ownership, and created a strong foundation for sustainability. In fact, the provincial and district departments of health have already taken over many of the project’s facilitation functions; introduced new budget lines to sustain essential activities; and established new structures, processes, and positions to institutionalize interventions. While these conditions bode well for the sustainability of results, the slow release of funds from the province to the districts to implement the DAP remains a challenge that may ultimately threaten sustainability. At the subsystem level, the project-supported long-term training (i.e., for MPH and MPP degrees) created a critical mass of professional public health practitioners who, if appropriately deployed, have the potential to improve leadership and management at the provincial and district levels. However, many key informants confirmed that most of these trained professionals have not yet been deployed to management positions, thus potentially limiting the immediate impact of the training on strengthening leadership and management. The project’s technical assistance for the DAP and MTBF supported processes that facilitated needs￾based planning and budgeting at the district level to address locally identified and prioritized needs, changes which improved service provision. However, implementation, and consequently the effectiveness and sustainability, of the DAP are dependent on timely release of the budget, the lack of which has limited the effectiveness of the DAP in improving service delivery. The project’s assistance in developing M&S tools and processes, and training staff in their use, strengthened the M&S capacity of district- and provincial-level health managers and of Sindh DGHS staff, including the vertical program managers. Reflecting this improved capacity and its effect on system performance and service delivery, the accuracy, timeliness, completeness, and frequency of health facility reporting have improved; better data has facilitated the feedback loop within and between departments; managers have improved their ability to identify and resolve problems in the context of limited resources; and mindsets have changed about sharing information due to increased confidence in its accuracy. Once again, however, the province must release DAP budgets for M&S to reap the benefits of the improved M&S system. The integrated SHIS created by the HSS project has improved reporting and contributed to shared accountability and transparency by making data from various programs and departments visible to all Health Systems Strengthening Component: Final Evaluation Report 25 stakeholders. Consequently, the quality of reports has improved, as demonstrated by their regularity and timeliness and the accuracy of their data. By easing managers’ timely access to reliable data, the SHIS enabled district- and provincial-level managers to provide more effective oversight and make evidence￾based decisions. These results have contributed to the project’s objectives of improving RMNCH service delivery and outcomes by improving transparency and accountability and strengthening management capacity. Improved coordination between stakeholders though the DHPMT and improved data as a result of the project’s support to the SHIS and M&S subsystems have improved transparency and accountability and enabled evidence-based decision-making. In tandem with needs-based planning and budgeting facilitated by the DAP/MTBF process, these interventions improved service delivery by directing resources to identified local needs and by improving monitoring, which enables managers to identify and address issues related to the availability of supplies and medicines. Finally, the project demonstrated the efficacy of a systems approach designed around essential health subsystems modeled on the WHO’s health systems building blocks. Evidence from multiple sources suggests that this approach yielded benefits in the form of strengthened coordination between and across departments as well as between districts and the province, increased district health financing, improved health facility reporting, increased use of information for decision-making, and improved service delivery. RECOMMENDATIONS Recommendations for USAID/Pakistan 1. Delays in releasing funds compromised many aspects of implementation and threatens the sustainability of activities and results. In the design of future projects that engage directly with the government institutions, USAID should emphasize including relevant administration and finance officials to streamline the release of funds to districts. Although this is the responsibility of the government, not USAID, including it specifically in the design may promote a mechanism to address such issues. 2. Limited engagement with relevant stakeholders was one of the factors compromising the project’s success. Future USAID project designs should articulate strategies and approaches to effectively engage all relevant stakeholders. Engagement may take the form of involving stakeholders in developing work plans, customized trainings for their capacity building, or engaging them more equitably in project activities as appropriate. Recommendations for the Department of Health 1. The frequent transfer of government officials, especially at higher levels, has emerged as a challenge in many projects, including HSS. Frequent transfers can compromise government ownership, slow implementation, or require implementing partners to establish new relationships with key government partners. To address this common issue, the GoP must address the frequent transfer of key officials and develop a mechanism for succession planning to ensure the sustainability of initiatives. 2. The DoH should ensure equitable engagement of relevant stakeholders such as PPHI, PWD, and DoE. Processes that fail to engage and empower relevant stakeholders may not incorporate valuable perspectives that could improve performance. As the department responsible for Health Systems Strengthening Component: Final Evaluation Report 26 provincial health systems, the DoH has a responsibility to effectively engage these diverse stakeholders. 3. The DoH should nominate provincial administration and finance offices to join the DHPMT. If these offices are involved in planning and costing activities, they will be more likely to support timely release of funds to implement activities.5 4. The provincial DoH should resolve district issues in a timely manner. 5 District administration and finance offices participate in the District Health Planning Committees that are responsible for planning and costing activities included in the DAP. Health Systems Strengthening Component: Final Evaluation Report 27 ANNEXES Annex 1: Statement of Work Health Systems Strengthening Component: Final Evaluation Report 28 Health Systems Strengthening Component: Final Evaluation Report 29 Health Systems Strengthening Component: Final Evaluation Report 30 Health Systems Strengthening Component: Final Evaluation Report 31 Health Systems Strengthening Component: Final Evaluation Report 32 Health Systems Strengthening Component: Final Evaluation Report 33 Health Systems Strengthening Component: Final Evaluation Report 34 Annex 2: Assignment Work Plan Health Systems Strengthening Component: Final Evaluation Report 35 Health Systems Strengthening Component: Final Evaluation Report 36 Health Systems Strengthening Component: Final Evaluation Report 37 Health Systems Strengthening Component: Final Evaluation Report 38 Health Systems Strengthening Component: Final Evaluation Report 39 Health Systems Strengthening Component: Final Evaluation Report 40 Health Systems Strengthening Component: Final Evaluation Report 41 Health Systems Strengthening Component: Final Evaluation Report 42 Health Systems Strengthening Component: Final Evaluation Report 43 Health Systems Strengthening Component: Final Evaluation Report 44 Health Systems Strengthening Component: Final Evaluation Report 45 Health Systems Strengthening Component: Final Evaluation Report 46 Health Systems Strengthening Component: Final Evaluation Report 47 Health Systems Strengthening Component: Final Evaluation Report 48 Health Systems Strengthening Component: Final Evaluation Report 49 Health Systems Strengthening Component: Final Evaluation Report 50 Health Systems Strengthening Component: Final Evaluation Report 51 Health Systems Strengthening Component: Final Evaluation Report 52 Health Systems Strengthening Component: Final Evaluation Report 53 Health Systems Strengthening Component: Final Evaluation Report 54 Health Systems Strengthening Component: Final Evaluation Report 55 Annex 3: Data Collection Instruments Instrument for District Health Officer (DHO) Pakistan Perform; Health Systems Strengthening Component; Final Evaluation A1. Language: A2. Date: A3. Venue: DHO Office A4. City/district: A5. Institution/Organization: A6. Respondent Name Title and Gender: A7. Start time: End time: A8. Moderator name: A9. Note taker name: Instructions to interviewer: Read the introduction below INTRODUCTION My name is . I work for a research organization based in Islamabad. We are conducting a study to determine the extent to which the USAID-funded Health Systems Strengthening project has been successful in meeting its objectives. The interview will take approximately 40–60 minutes. We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No _______________) If respondent says yes, continue the interview. If no, try to motivate respondent by answering their questions and explaining the importance of recording the interview. If respondent does not agree to the recording, do not record the interview. Health Systems Strengthening Component: Final Evaluation Report 56 SECTION 1: Introduction 1. What are your roles and responsibilities in this department? 2. For how long has your department worked with HSSC? years months 3. Did you or your department participate in any JSI HSSC interventions or activities related to district action planning and the medium-term budgetary framework? 1 Yes 2 No ........................... GO TO Q 6 3 Don’t know........... GO TO Q 6 4. IF YES, in which interventions or activities? 5. IF YES, What was your specific role in these interventions or activities? 6. Did you or your department participate in any JSIHSSC interventions or activities related to the district health and population management team? 1 Yes 2 No ............................. GO TO Q 9 3 Don’t know............. GO TO Q 9 7. IF YES, What was your specific role in these interventions or activities? 8. Did you or your department participate in any JSI HSSC interventions or activities related to the Sindh health information system? 1 Yes 2 No ............................... GO TO Q 11 3 Don’t know............... GO TO Q 11 9. IF YES, What was your specific role in these interventions or activities? Health Systems Strengthening Component: Final Evaluation Report 57 10. Did you or your department participate in any JSI HSSC interventions or activities related to leadership and management capacity building? 1 Yes 2 No ............................. GO TO Q 13 3 Don’t know............. GO TO Q 13 11. IF YES, in which interventions or activities? 12. IF YES, What was your specific role in these interventions or activities? 13. Did you or your department participate in any JSI HSSC interventions or activities related to the monitoring and supervision system? 1 Yes 2 No .....................................GO TO Q 16 3 Don’t know.....................GO TO Q 16 14. IF YES, in which interventions or activities? 15. IF YES, What was your specific role in these interventions or activities? I keep checks and balances, for instance, EPI program tools SECTION 2: Technical assistance for the development of district action plans (DAPs) to improve coverage of basic health services, referral linkages among different levels of the health system, based on district needs. (DAP); District level capacity building in costing DAPs and transitioning from traditional budgeting processes to medium term budgetary frameworks. (DAP & MTBF) Now I’m going to ask you some more detailed questions about the district action plans and the planning process. 16. In your opinion, why did JSI HSSC introduce the district action plans? Health Systems Strengthening Component: Final Evaluation Report 58 17. Did the district action plan or the planning process affect health service delivery in your district? PROBE: NEEDS BASED BUDGETING, HEALTH FACILITY ASSESSMENT, PRIORIZATION OF NEEDS, RMNCH, ALLOCATION OF RESOURCES, REFERRAL LINKAGES AT DIFFERENT LEVELS. 1 Yes 2 No 3 Don’t know..........GO TO Q 19 18. IF YES, How (positively/negatively) did the plan or the planning process affect health service delivery and how did the plan or planning process contribute to the change? IF NO, Why did the plan or the planning process not affect health service delivery? I’m going to ask you about five different tasks related to developing district action plans. For each task, I want to know the extent to which the tasks have affected evidence-based decision making; did the task make evidence-based decision-making much better, better, worse or did it have no effect? Task 1: Let’s talk first about the effect of training to medium-term budgetary framework trainers. 19. To what extent did training to medium-term budgetary framework trainers affect costing and budgeting; did it make it much better, better, worse, or did it have no effect? 1 Much better 2 Better 3 No effect 4 Worse 5 Don’t know.................GO TO TASK 2 20. How did the training to medium-term budgetary framework master trainers affect costing and budgeting? Task 2: Now I’d like to ask you about the effects of district analytical profiles. 21. To what extent do you think district analytical profiles affected evidence-based decision making; did it make it much better, better, worse, or did it have no effect? 1 Much better 2 Better 3 No effect 4 Worse 5 Don’t know.................GO TO TASK 3 22. Can you explain how district analytical profiles made evidence-based decision making better/worse/have no effect. Health Systems Strengthening Component: Final Evaluation Report 59 Task 3: Now let’s talk about the effects of operational plans. 23. To what extent do you think operational plans affected evidence-based decision making; did it make it much better, better, worse, or did it have no effect? 1 Much better 2 Better 3 No effect 4 Worse 5 Don’t know.................GO TO TASK4 24. Can you explain how operational plans made evidence-based decision making better/worse/have no effect. Task 4: Next, I’d like to ask you about activity-based costing. 25. To what extent do you think activity-based costing affected evidence-based decision making; did it make it much better, better, worse, or did it have no effect? 1 Much better 2 Better 3 No effect 4 Worse 5 Don’t know.................GO TO TASK 5 26. Can you explain how activity-based costing made evidence-based decision making better/worse/have no effect. Task 5: Finally, let’s talk about needs-based budgeting. 27. To what extent do you think needs-based budgeting affected evidence-based decision making; did it make it much better, better, worse, or did it have no effect? 1 Much better 2 Better 3 No effect 4 Worse 5 Don’t know.................GO TO Q 29 28. Can you explain how needs-based budgeting made evidence-based decision making better/worse/have no effect. Health Systems Strengthening Component: Final Evaluation Report 60 29. Did you face any challenges transitioning from the traditional budgeting to the medium term budgetary framework? 1 Yes 2 No 30. IF YES, Please describe the challenges you faced and how you resolved the challenges. 31. Besides the things we’ve just discussed, has district action planning affected any other aspects of your work? 1 Yes 2 No 32. IF YES, Can you describe what other effects the plan had on your work and why these effects are important? (PROBE: REDEFINED ROLE, NEW BUDGET LINES) 33. You’ve mentioned several ways the district action plans have affected your work (MENTION SOME EXAMPLES IF YOU CAN REMEMBER THEM). Can you tell me the two or three most significant benefits of introducing the plan? I’d like to talk now about if and how you will continue the planning process after JSI HSSC ends. 34. In your opinion, which, if any, JSI HSSC activities need to be continued for the district action planning process to continue after JSI HSSC ends? Why are these important? 35. In your opinion, is the district likely to continue to develop plans after JSI HSSC ends? 1 Yes.......................GO TO Q 37 2 No 3 Don’t know 36. IF NO, Why not? (PROBE: AFFORDABILITY/FINANCIAL CONSTRAINTS, CHALLENGES) Health Systems Strengthening Component: Final Evaluation Report 61 37. Have you made any changes in staffing, or the roles and responsibilities of existing staff, to support continuing district planning activities without JSI HSSC support (e.g., field coordinators)? 1 Yes 2 No ....................................... GO TO Q 39 3 Don’t know....................... GO TO Q 39 38. IF YES, Please explain the changes you’ve made. 39. Have you made plans to ensure that you retain or acquire the skills necessary to continue essential activities to support the planning process? (PROBE: REFRESHER TRAINING, ON-THE-JOB-TRAINING OR MENTORING, TRANSITION PLANS FOR TRANSFERS OR HIRE NEW STAFF) 1 Yes 2 No 3 Don’t know 40. Does your 2017-18 budget include resources for continuing district action planning activities? 1 Yes 2 No ....................................... GO TO Q 42 3 Don’t know....................... GO TO Q 42 41. IF YES, Please explain. Which activities are included in the budget? 42. Has your institution incorporated the district action planning process as part of your normal practice? 1 Yes 2 No ....................................... GO TO Q 44 3 Don’t know....................... GO TO Q 44 43. IF YES, Please explain. SECTION 4: Revitalizing District Health & Population Management Teams (DHPMTs) to conduct district-level management, oversight, and integration of RMNCH activities. 44. Has the role of district health and population management teams changed during the past 5 years? 1 Yes 2 No 3 Don’t know......................GO TO Q 46 – I wasn’t there 5 days ago 45. IF YES, How? IF NO, Why not? Health Systems Strengthening Component: Final Evaluation Report 62 46. In your opinion, has the district health and population management team improved the overall performance of the district health service delivery system? (PROBE: QUALITY, ACCESS, UTILISATION OF RMNCH SERVICES) 1 Yes 2 No 3 Don’t know......................GO TO Q 48 47. IF YES, How? IF NO, Why not? Please explain? Now I’d like to ask about if and how changes in the district health and population management team have affected various aspects of the health system. For each question, please tell me whether the district health and population management team has made things much better, better, worse, or had no effect. 48. To what extent did the district health and population management team affect the district performance review system? Did it make it much better, better, worse, or did it have no effect? 1 Much better 2 Better 3 No effect...........................GO TO Q 50 4 Worse 5 Don’t know......................GO TO Q 50 49. Please explain how you think the district health and population management team affected the district performance review system. Information sharing is really helpful for effective decision making 50. To what extent did the district health and population management team affect the integration of RMNCH activities, LHW, EPI, MNCH, nutrition programs? Did it make it much better, better, worse, much worse, or did it have no effect? 1 Much better 2 Better 3 No effect...........................GO TO Q 52 4 Worse 5 Don’t know......................GO TO Q 52 51. Please explain how you think the district health and population management team affected the integration of RMNCH activities, LHW, EPI, MNCH, nutrition programs. Health Systems Strengthening Component: Final Evaluation Report 63 52. To what extent did the district health and population management team affect the district￾level management, oversight? Did it make it much better, better, worse, or did it have no effect? 1 Much better 2 Better 3 No effect...........................GO TO Q 54 4 Worse 5 Don’t know......................GO TO Q 54 53. Please explain how you think the district health and population management team affected the district-level management, oversight. 54. To what extent did the district health and population management team affect the coordination between stakeholders, DoH, PWD, PPHI, and education department? Did it make it much better, better, worse, or did it have no effect? 1 Much better 2 Better 3 No effect...........................GO TO Q 56 4 Worse 5 Don’t know......................GO TO Q 56 55. Please explain how you think the district health and population management team affected the coordination between stakeholders, DoH, PWD, PPHI, and education department. 56. Besides what we discussed above, do you think changes in the district health and population management team have caused any other changes in your district? 1 Yes 2 No 3 Don’t know 57. If so, what are those changes and how did changes in the district health and population management team contribute to the change? (PROBE FOR HOW CHANGES IN THE DHPMT CONTRIBUTED TO THE CHANGE) 58. Did you face any challenges in revitalization of district health and population management team? 1 Yes 2 No .......................GO TO Q 60 Health Systems Strengthening Component: Final Evaluation Report 64 59. IF YES, Please describe the challenges you faced and how you resolved the challenges. SECTION 5: Technical assistance to improve coordination between stakeholders: Department of Health (DoH), Population Welfare Department (PWD), and the People’s Primary Healthcare Initiative (PPHI) (DHPMT) 60. How are the district health and population management team meetings organized PROBE: FOLLOW UP ON ACTION POINTS DISCUSSED IN THE PREVIOUS MEETING? 61. Have you encountered any challenges in organizing district health and population management team meetings? 1 Yes 2 No ...........GO TO Q 63 62. IF YES, Please describe the challenges? Please explain? 63. How could the organization of the district health and population management team meetings be improved? 64. What is the role of the JSI field manager in organizing the district health and population management team meetings? 65. Can the Department of Health fill all of the roles of the JSI field manager that are essential to the functioning of the district health and population management team meetings after the JSI HSSC project ends? 1 Yes 2 No 3 Don’t know..........GO TO Q 67 Health Systems Strengthening Component: Final Evaluation Report 65 66. IF YES, How? IF NO, why not? Please explain? (PROBE FOR HOW THE DOH WILL HANDLE SETTING AGENDAS, ENSURING PARTICIPATION, TAKING MINUTE, LOGISTICS, FOLLOWING UP ON ACTION POINTS, AND OTHER JSI ROLES MENTIONED IN QUESTION 68) 67. In your opinion, will the Department of Health continue the district health and population management team meetings after the JSI HSSC project ends? 1 Yes..........................GO to Q 71 2 No ..........................ASK Q 70 AND THENGO to Q 79 3 Don’t know..........GO to Q 71 68. IF NO, Why not? (PROBE: AFFORDABILITY/FINANCIAL CONSTRAINTS, CHALLENGES, ETC.) I’d like to talk now about if and how you will continue the district health and population management team meetings after HSSC ends. 69. Do you have plans to transition activities essential to continuing the district health and population management team meetings to other donors or organizations or the public sector? 1 Yes 2 No 3 Don’t know..........GO TO Q 71 70. Please explain what plans you have made or why you have not made plans. (PROBE FOR WHETHER PLANS WERE NECESSARY) 71. Have you changed staffing, or the roles and responsibilities of existing staff, to support continuing the activities without JSI HSSC support (e.g., field coordinators)? 1 Yes 2 No 3 Don’t know..........GO TO Q 73 72. Please explain what staffing changes you have made or why you have not made changes. (PROBE FOR WHETHER CHANGES WERE NECESSARY) Health Systems Strengthening Component: Final Evaluation Report 66 73. Have you made plans to ensure that you retain (e.g., refresher training, on-the-job￾training or mentoring, transition plans for transfers) or acquire the skills (e.g., hire new staff) necessary to continue essential activates? 1 Yes 2 No 3 Don’t know..........GO TO Q 77 74. Please explain what plans you have made or why you have not made plans. (PROBE FOR WHETHER PLANS WERE NECESSARY) 75. Has your institution incorporated the district health and population management team meetings as part of your regular practice? 1 Yes 2 No 3 Don’t know..........GO TO Q 77 76. Please explain how you have incorporated the district health and population management team meetings into our regular process. SECTION 6: Implement individual-level capacity building interventions designed to strengthen management skills at the provincial level, such as training a core group of public sector professional health managers 77. Have you participated in JSI HSSC led trainings? 1 Yes 2 No ...........GO TO Q 81 78. IF YES, What are the names of the courses in which you participated? (WRITE IN NAMES OF COURSE/TRAININGS) 1. Short courses: ____________________________________________ 2. Masters degree programs: ____________________________________ 79. Did the training help you improve your job practices? 1 Yes 2 No ..........................GO TO Q 81 80. IF YES, How did it improve your job practices; how did you use what you learned? IF NO, Why did it not improve your practices? (PROBE FOR RELEVANCE OR QUALITY OF TRAINING) Health Systems Strengthening Component: Final Evaluation Report 67 81. Has any of your staff received training through JSI HSSC? 1 Yes 2 No ..........................GO TO Q 83 3 Don’t know..........GO TO Q 83 82. IF YES, What kind of training did they receive? (PROBE FOR TRAINING TOPICS OR TITLES) 83. Did JSIHSSC individual level capacity building affect the work practices of your staff? 1 Yes 2 No ..........................GO TO Q 85 3 Don’t know..........GO TO Q 85 84. IF YES, How did it improve their work practices? (PROBE FOR RELEVANCE OR QUALITY OF CAPACITY BUILDING ACTIVITIES) 85. Did JSI HSSC individual level capacity building affect your staff’s individual skills or knowledge? 1 Yes 2 No ..........................GO TO Q 87 3 Don’t know..........GO TO Q 87 86. IF YES, How did it improve their skills or knowledge? PROBE FOR RELEVANCE OR QUALITY OF CAPACITY BUILDING ACTIVITIES) SECTION 7: Conduct systems-level capacity building through the development of manuals, protocols, checklists and guidelines. (Leadership & MC) 87. Did you contribute to developing manuals, protocols, checklists, or guidelines designed to improve systems? 1 Yes 2 No ............ GO TO Q 89 88. IF YES, Which ones? (WRITE NAME/ TITLE, OR FUNCTION) Health Systems Strengthening Component: Final Evaluation Report 68 89. Have you used these tools in the last 12 months? 1 Yes 2 No 90. IF YES, Which ones have you used? IF NO, Why have you not used them? 91. Were the tools useful? 1 Yes 2 No ............ GO TO Q 93 92. IF YES, Which ones were useful and how? How did they improve the performance of the Department of Health? 93. Did you encounter any challenges implementing the tools? Please explain. 94. Do you think your department will continue using some or all of the tools after the end of the HSSC project? 1 Yes, all of them 2 Yes, some of them 3 No, none of them 4 Don’t know..........GO TO Q 96 95. IF YES, Which ones will you continue to use and why? IF NO, Which ones will you not continue to use and why? SECTION 8: Established structures to operationalize monitoring and evaluation system by: 1) bringing management information systems online, 2) Integrating all MIS, 3) refurbish M&E cells at DGHSS and DHO offices, and 4) integrated dashboard -SHIS. SHIS, M & S 96. Has the office of your M&E cell (district or provincial) been refurbished? 1 Yes 2 No 3 Don’t know..........GO TO Q 98 97. IF YES, Has that helped in performing MIS related activities? IF NO, Why has the cell not been refurbished? Health Systems Strengthening Component: Final Evaluation Report 69 I’d like to ask you some questions about how changes in the health information system have affected different aspects of your department’s work. 98. Did strengthening the health information system affect planning and management in your department? 1 Yes 2 No 3 Don’t know......................GO TO Q 100 99. IF YES, Please explain how it affected planning and management. (PROBE BETTER/WORSE AND WHY); IF NO, Why did it not improve planning and management? PROBE: ACCESS, ABILITY TO USE, PRIORITIZATION OF NEEDS 100. Did strengthening the health information system affect health facility reporting in your department? 1 Yes 2 No 3 Don’t know......................GO TO Q 102 101. IF YES, Please explain how it affected reporting. (PROBE BETTER/WORSE AND WHY; PROBE FOR TIMELINESS, COMPETENESS, ACCURACY). IF NO, Why did it not improve reporting? 102. Did strengthening the health information system affect organizational behavior around reporting in your department? 1 Yes 2 No 3 Don’t know......................GO TO Q 104 103. IF YES, Please explain how it affected reporting behavior. PROBE: BETTER, GOT WORSE, WHY IF NO, Why did it not improve reporting behavior? PROBE: OPENESS TO SHARE INFORMATION, REGULARITY, CONFINDENCE 104. Did strengthening the health information systems affect the ability and capacity of district health officials to monitor and verify health sector data? 1 Yes 2 No 3 Don’t know......................GO TO Q 106 Health Systems Strengthening Component: Final Evaluation Report 70 105. IF YES, Please explain how it affected monitoring and supervision capacity. (PROBE BETTER/WORSE AND WHY) IF NO, Why did it not improve monitoring and supervision capacity? 106. Did strengthening the health information system affect integration of DHIS, LHW-MIS, CMW-MIS, cLIMS, vLMIS? 1 Yes 2 No 3 Don’t know......................GO TO Q 108 107. IF YES, Please explain how it affected integration. (PROBE BETTER/WORSE AND WHY) IF NO, Why did it not improve integration? SECTION 9: Demand creation for use of information for evidence based decision making (SHIS) 108. What is the regular process of making decisions related to health planning in your district? 109. Has JSI HSSC support changed the decision-making process? 1 Yes 2 No 3 Don’t know......................GO TO Q 111 110. IF YES, How? IF NO, Why not? 111. What sources of information do you use for developing district action plans? 112. What sources of information do you use for preparing for a district health and population management team meeting? 113. What sources of information do you use for preparing and conducting monitoring and supervision visits? Health Systems Strengthening Component: Final Evaluation Report 71 114. To what extent do you agree with the following statement: “the use of information for decision making has increased at the district level”? Do you strongly agree, agree, disagree, strongly disagree, or do you have no opinion? 1 Strongly agree 2 Agree 3 No opinion ......................GO TO Q 116 4 Disagree 5 Strongly disagree 115. Can you please explain why you agree or disagree with this statement. SECTION 10: Develop monitoring and supervisory (M&S) tools and checklists (M & S) I’d like to ask you some questions about how changes in the monitoring and supervision system have affected different aspects of your department’s work. 116. Did you or your department use the monitoring and supervisory tools/checklists during your last monitoring and supervision visit? 1 Yes 2 No 3 Don’t know..........GO TO Q 118 117. IF YES, How did you use the tools? IF NO, Why did you not use the tools? 118. IF YES, In your opinion, were the tools useful? How? SECTION 11: Build capacity of DoH provincial and district staff on M&S system for use of information (Leadership/MC, M & S) I’d like to ask you some questions about how capacity building on the Monitoring and Supervision system have affected different aspects of your department’s work. 119. Have you been trained on the monitoring and supervision system and its online reporting? 1 Yes 2 No ............ GO TO 123 120. IF YES, What key skills did you acquire from training? Health Systems Strengthening Component: Final Evaluation Report 72 121. How often do you use the knowledge and skills acquired during the training? Would you say you use the knowledge and skills often, sometimes, or not at all? 1 Frequently 2 Occasionally 3 Not at all 122. Please explain why you use the knowledge and skills frequently/occasionally /not at all. 123. In your opinion, how important was the contribution of JSI HSS to overall health system strengthening? Give examples. (PROBE FOR HOW THINGS WOULD HAVE BEEN DIFFERENT WITHOUT HSS SUPPORT) 124. In your opinion, how effective were each of the following sets of JJSI/HSSC activities in supporting strengthening the health system? Would you say they were very effective, effective, ineffective, or you don’t know? (CIRCLE ONE NUMBER FOR EACH ROW AND WRITE AN EXPLANATION) Type of activity Very effective Effective Ineffective Don’t know What assistance, if any, was an exception to your overall assessment? 1 How effective was technical assistance for the MIS? 1 2 3 4 2 How effective was training for the MIS? 1 2 3 4 3 How effective as technical assistance for developing the district action plan? 1 2 3 4 4 How effective as training for developing the district action plan? 1 2 3 4 5 How effective was technical assistance to develop checklists and manuals? 1 2 3 4 6 How effective was support for DHMPT facilitation? 1 2 3 4 Health Systems Strengthening Component: Final Evaluation Report 73 125. In your opinion, of the JSI HSSC interventions, which would you consider best practices, innovations, or lessons learned for improving implementation of health system strengthening activities in Pakistan? Please describe and explain why they were best practices, innovations, or lessons learned. 126. In your opinion, has JSI HSS support contributed to improved health service delivery? PROBE: ACCESS, UTILIZATION, QUALITY OF RMNCH SERVICES 1 Yes 2 No ..........................GO TO Q 129 3 Don’t know..........GO TO Q 129 127. IF YES, Please explain what has improved and how HSSC contributed to the improvement. THEN GO TO Q 140 128. IF NO, why not? (PROBE FOR BROADERSTRUCTURAL ISSUES, POLITICAL WILL, SUSTAINABILITY, ETC.) 129. In your opinion, how, if at all, has this project benefited women in your district? 130. In your opinion, how, if at all, has this project benefited the poor in your district? Health Systems Strengthening Component: Final Evaluation Report 74 Instrument for DHIS Coordinator Pakistan Perform; Health Systems Strengthening Component; Final Evaluation A1. Language: A2. Date: A3. Venue: A4. City/district: A5. Institution/Organization: A6. Respondent Name: Title: Gender: A7. Start time: End time: A8. Moderator name: A9. Note taker name: Instructions to interviewer: Read the introduction below INTRODUCTION My name is . I work for a research organization based in Islamabad. We are conducting a study to determine the extent to which the USAID-funded Health Systems Strengthening project has been successful in meeting its objectives. The interview will take approximately 40–60 minutes. We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No _______________) If respondent says yes, continue the interview. If no, try to motivate respondent by answering their questions and explaining the importance of recording the interview. If respondent does not agree to the recording, do not record the interview. Health Systems Strengthening Component: Final Evaluation Report 75 SECTION 1: Introduction 1. What are your roles and responsibilities in this office/department? 2. For how long has your office/department worked with HSSC? years months. SECTION 2: Implement individual-level capacity building interventions designed to strengthen management skills at the provincial level, such as training a core group of public sector professional health managers 3. Have you participated in JSI/HSSC led trainings? 1 Yes 2 No ...........GO TO Q-9 4. IF YES, What are the names of the courses in which you participated? (WRITE IN NAMES OF COURSE/TRAININGS) 1. Short courses: ____________________________________________ 2. Master’s degree programs:____________________________________ 5. Did the training help you improve your job practices? 1 Yes 2 No 6. IF YES, How did it improve your job practices; how did you use what you learned? IF NO, Why did it not improve your practices? (PROBE FOR RELEVANCE OR QUALITY OF TRAINING) SECTION 3: Conduct systems-level capacity building through the development of manuals, protocols, checklists and guidelines. (Leadership & MC) 7. Did you contribute to developing manuals, protocols, checklists, or guidelines designed to improve systems? 1 Yes 2 No ............ GO TO Q 9 8. IF YES, Which ones?(WRITE NAME/ TITLE, OR FUNCTION) 9. Have you used these tools in the last 12 months? 1 Yes 2 No ..........................GO TO Q 16 Health Systems Strengthening Component: Final Evaluation Report 76 10. IF YES, Which ones have you used? IF NO, Why have you not used them? 11. Were the tools useful? 1 Yes 2 No ........................ GO TO Q 13 12. IF YES, Which ones were useful and how? How, if at all, did they improve Department of Health performance? 13. Did you encounter any challenges implementing the tools? Please explain. 14. Do you think your department will continue using some or all of the tools after the end of the HSSC project? 1 Yes, all of them 2 Yes, some of them 3 No, none of them 4 Don’t know...............................GO TO Q16 15. IF YES, Which ones will you continue to use and why? IF NO, Why will you not use them? SECTION 4: Established structures to operationalize monitoring and evaluation system by: 1) bringing management information systems online, 2) Integrating all MIS, 3) refurbish M&E cells at DGHSS and DHO offices, and 4) integrated dashboard -SHIS. SHIS, M & S 16. Has the office of the monitoring and evaluation cell at your office been refurbished? 1 Yes 2 No 3 Don’t know......................GO TO Q 18 17. IF YES, Has that helped in performing MIS related activities? IF NO, Why has the cell not been refurbished? Health Systems Strengthening Component: Final Evaluation Report 77 I’d like to ask you some questions about how changes in the health information system have affected different aspects of your department’s work. 18. Did strengthening the health information system affect planning and management in your department? 1 Yes 2 No 3 Don’t know.......GO TO Q 20 19. IF YES, Please explain how it affected planning and management. (PROBE BETTER/WORSE AND WHY); IF NO, Why did it not improve planning and management? PROBE: ACCESS, ABILITY TO USE, PRIORITIZATION OF NEEDS 20. Did strengthening the health information system affect health facility reporting in your department? 1 Yes 2 No 3 Don’t know.......... GO TO Q 22 21. IF YES, Please explain how it affected reporting. (PROBE BETTER/WORSE AND WHY; PROBE FOR TIMELINESS, COMPETENESS, ACCURACY). IF NO, Why did it not improve reporting? 22. Did strengthening the health information system affect organizational behavior around reporting in your department, for example, openness to sharing information, regularity of reporting, and confidence in the data? 1 Yes 2 No 3 Don’t know.......... GO TO Q 24 23. IF YES, Please explain how it affected reporting behavior. PROBE: BETTER, GOT WORSE, WHY IF NO, Why did it not improve reporting behavior? PROBE: OPENESS TO SHARE INFORMATION, REGULARITY, CONFINDENCE 24. Did strengthening the health information systems affect the ability and capacity of district health officials to monitor and verify health sector data? 1 Yes 2 No 3 Don’t know............ GO TO Q 26 Health Systems Strengthening Component: Final Evaluation Report 78 25. IF YES, Please explain how it affected monitoring and supervision capacity. (PROBE BETTER/WORSE AND WHY)IF NO, Why did it not improve monitoring and supervision capacity? 26. Did strengthening the health information system affect integration of DHIS, LHW-MIS, CMW-MIS, cLIMS, vLMIS? 1 Yes 2 No 3 Don’t know GO TO Q 28 27. IF YES, Please explain how it affected integration. (PROBE BETTER/WORSE AND WHY) IF NO, Why did it not improve integration? SECTION 5: Demand creation for use of information for evidence based decision making (SHIS) 28. What is the regular process of making decisions related to health planning in your district? 29. Has JSI/HSSC support changed the decision-making process? 1 Yes 2 No 3 Don’t know......................GO TO Q 31 30. IF YES, How? IF NO, Why not? 31. To what extent do you agree with the following statement: “the use of information for decision making has increased at the district level”? Do you strongly agree, agree, disagree, strongly disagree, or do you have no opinion? 1 Strongly agree 2 Agree 3 No opinion ............................ GO TO Q 33 4 Disagree 5 Strongly disagree 32. Can you please explain why you agree or disagree with this statement? Health Systems Strengthening Component: Final Evaluation Report 79 SECTION 6: Develop monitoring and supervisory (M&S) tools and checklists (M & S) I’d like to ask you some questions about how changes in the monitoring and supervision system have affected different aspects of your department’s work. 33. Did you use the monitoring and supervisory tools/checklists during your last monitoring and supervision visit? 1 Yes 2 No 34. IF YES, How did you use the tools? IF NO, Why did you not use the tools? 35. IF YES, In your opinion, were the tools useful? How? SECTION 7: Build capacity of DoH provincial and district staff on M&S system for use of information (Leadership/MC, M & S) I’d like to ask you some questions about how capacity building on the monitoring and supervision system have affected different aspects of your department’s work. 36. Have you been trained on the monitoring and supervision system and its online reporting? 1 Yes 2 No ........................ GO TO Q 38 37. IF YES, What, if any, key skills did you acquire from training? 38. How often do you use the knowledge and skills acquired during the training? Would you say you use the knowledge and skills often, occasionally, or not at all? 1 Frequently 2 Occasionally 3 Not at all 39. Please explain why you use the knowledge and skills frequently/occasionally /not at all. 40. In your opinion, how important was the contribution of JSI/HSSC to overall health system strengthening? Give examples. (PROBE FOR HOW THINGS WOULD HAVE BEEN DIFFERENT WITHOUT HSS SUPPORT) Health Systems Strengthening Component: Final Evaluation Report 80 41. In your opinion, how effective were each of the following sets of JSI/HSSC activities in supporting strengthening the health system? Would you say they were very effective, effective, ineffective, or you don’t know? (CIRCLE ONE NUMBER FOR EACH ROW AND WRITE AN EXPLANATION) Type of activity Very effective Effective Ineffective Don’t know What assistance, if any, was an exception to your overall assessment? 1 How effective was technical assistance for the MIS? 1 2 3 4 2 How effective was training for the MIS? 1 2 3 4 3 How effective was technical assistance to develop checklists and manuals? 1 2 3 4 4 How effective was support for DHMPT facilitation? 1 2 3 4 42. In your opinion, of the JSI/HSSC interventions, which would you consider best practices, innovations, or lessons learned for improving implementation of health system strengthening activities in Pakistan? Please describe and explain why they were best practices, innovations, or lessons learned. 43. In your opinion, has JSI/HSSC support contributed to improved health service delivery? PROBE: ACCESS, UTILIZATION, QUALITY OF RMNCH SERVICES 1 Yes 2 No 3 Don’t know......................GO TO Q 48 44. IF YES, Please explain what has improved and how JSI/HSSC contributed to the improvement. 45. IF NO, Why not? (PROBE FOR BROADERSTRUCTURAL ISSUES, POLITICAL WILL, SUSTAINABILITY, ETC.) Health Systems Strengthening Component: Final Evaluation Report 81 46. In your opinion, how, if at all, has this project benefited women in your district? 47. In your opinion, how, if at all, has this project benefited the poor in your district? Health Systems Strengthening Component: Final Evaluation Report 82 Instrument for LHW/MNCH Coordinator/Focal person Pakistan Perform; Health Systems Strengthening Component; Final Evaluation A1. Language: A2. Date: A3. Venue: A4. City/district: A5. Institution/Organization: A6. Respondent Name: Title: Gender: A7. Start time: End time: A8. Moderator name: A9. Note taker name: Instructions to interviewer: Read the introduction below INTRODUCTION My name is . I work for a research organization based in Islamabad. We are conducting a study to determine the extent to which the USAID-funded Health Systems Strengthening project has been successful in meeting its objectives. The interview will take approximately 40–60 minutes. We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No _______________) If respondent says yes, continue the interview. If no, try to motivate respondent by answering their questions and explaining the importance of recording the interview. If respondent does not agree to the recording, do not record the interview. Health Systems Strengthening Component: Final Evaluation Report 83 SECTION 1: Introduction 1. What are your roles and responsibilities in this Program? 2. For how long has JSI HSSC worked with your program at district level? years months SECTION 2: Revitalizing District Health & Population Management Teams (DHPMTs) to conduct district-level management, oversight, and integration of RMNCH activities. 3. Has the role of district health and population management teams changed during the past 5 years in your district? 1 Yes 2 No 3 Don’t know..........GO TO Q 5 4. IF YES, How? IF NO, Why not? 5. In your opinion, has the district health and population management team improved the overall performance of the district health services delivery system? (PROBE: QUALITY, ACCESS, UTILISATION OF RMNCH SERVICES) 1 Yes 2 No 3 Don’t know..........GO TO Q 7 6. IF YES, How? IF NO, Why not? Please explain? Health Systems Strengthening Component: Final Evaluation Report 84 Now I’d like to ask about if and how changes in the district health and population management team have affected various aspects of the health system. For each question, please tell me whether the district health and population management team has made things much better, better, worse, or had no effect. 7. To what extent did the district health and population management team affect the district performance review system? Did it make it much better, better, worse, much worse, or did it have no effect? 1 Much better 2 Better 3 No effect...........................GO TO Q 9 4 Worse 5 Much worse 6 Don’t know 8. Please explain how you think the district health and population management team affected the district performance review system. 9. To what extent did the district health and population management team affect the integration of RMNCH activities, LHW, EPI, MNCH, nutrition programs? Did it make it much better, better, worse, much worse, or did it have no effect? 1 Much better 2 Better 3 No effect...........................GO TO Q 11 4 Worse 5 Much worse 6 Don’t know 10. Please explain how you think the district health and population management team affected the integration of RMNCH activities, LHW, EPI, MNCH, nutrition programs. 11. To what extent did the district health and population management team affect the district￾level management, oversight? Did it make it much better, better, worse, much worse, or have no effect? 1 Much better 2 Better 3 No effect...........................GO TO Q 13 4 Worse 5 Much worse 6 Don’t know Health Systems Strengthening Component: Final Evaluation Report 85 12. Please explain how you think the district health and population management team affected the district-level management, oversight. 13. To what extent did the district health and population management team affect the coordination between stakeholders, DoH, PWD, PPHI, and education department? Did it make it much better, better, worse, much worse, or did it have no effect? 1 Much better 2 Better 3 No effect...........................GO TO Q 15 4 Worse 5 Much worse 6 Don’t know 14. Please explain how you think the district health and population management team affected the coordination between stakeholders, DoH, PWD, PPHI, and education department. 15. Besides what we discussed above, do you think changes in the district health and population management team have caused any other changes in the performance of your district’s health services delivery system? 1 Yes 2 No ..................................GO TO Q 17 3 Don’t know..................GO TO Q 17 16. IF YES, what are those changes and how did changes in the district health and population management team contribute to the change?(PROBE FOR HOW CHANGES IN THE DHPMT CONTRIBUTED TO THE CHANGE) 17. Did you face any challenges in resuming the district health and population management team meetings? 1 Yes 2 No .......................GO TO Q 19 18. IF YES, Please describe the challenges you faced and how you resolved the challenges. Health Systems Strengthening Component: Final Evaluation Report 86 SECTION 3: Technical assistance to improve coordination between stakeholders: Department of Health (DoH), Population Welfare Department (PWD), and the People’s Primary Healthcare Initiative (PPHI) (DHPMT) 19. How are the district health and population management team meetings organized? (PROBE: FOLLOW UP ON ACTION POINTS DISCUSSED IN THE PREVIOUS MEETING?) 20. Have you encountered any challenges in organizing district health and population management team meetings? 1 Yes 2 No 21. IF YES, Please describe the challenges? Please explain? 22. IF YES, How could the organization of the district health and population management team meetings be improved? 23. What is the role of the JSI field manager in organizing the district health and population management team meetings? 24. Can the Department of Health fill all of the roles of the JSI HSSC field manager that are essential to the functioning of the district health and population management team meetings after the JSI HSSC project ends? 1 Yes 2 No 3 Don’t know....................... GO TO Q 26 25. IF YES, How? IF NO, why not? Please explain? (PROBE FOR HOW THE DOH WILL HANDLE SETTING AGENDAS, ENSURING PARTICIPATION, TAKING MINUTE, LOGISTICS, FOLLOWING UP ON ACTION POINTS, AND OTHER JSI ROLES) Health Systems Strengthening Component: Final Evaluation Report 87 26. In your opinion, will the Department of Health continue the district health and population management team meetings after the JSI HSSC project ends? 1 Yes 2 No 3 Don’t know......................GO TO Q 28 27. IF NO, Why not? (PROBE: AFFORDABILITY/FINANCIAL CONSTRAINTS, CHALLENGES, ETC.) 28. Has your Program/ incorporated the district health and population management team meetings as part of your regular practice? 1 Yes 2 No 3 Don’t know 29. Please explain how you have incorporated the district health and population management team meetings into our regular process. SECTION 4: Implement individual-level capacity building interventions designed to strengthen management skills at the provincial level, such as training a core group of public sector professional health managers 30. Have you participated in JSI HSSC led trainings? 1 Yes 2 No 31. IF YES, What are the names of the courses in which you participated? (WRITE IN NAMES OF COURSE/TRAININGS) 1 Short courses: _________________________________________ 2 Master’s degree programs:_________________________________ 32. Did the training help you improve your job practices? 1 Yes 2 No 33. IF YES, How did it improve your job practices; how did you use what you learned? IF NO, Why did it not improve your practices? (PROBE FOR RELEVANCE OR QUALITY OF TRAINING) Health Systems Strengthening Component: Final Evaluation Report 88 SECTION 5: Conduct systems-level capacity building through the development of manuals, protocols, checklists and guidelines. (Leadership & MC) 34. Did you contribute to developing manuals, protocols, checklists, or guidelines designed to improve systems? 1 Yes 2 No 35. IF YES, Which ones? (WRITE NAME/ TITLE, OR FUNCTION) 36. Have you used these tools in the last 12 months? 1 Yes 2 No 37. IF YES, Which ones have you used? IF NO, Why have you not used them? 38. Were the tools useful? 1 Yes 2 No 39. IF YES, Which ones were useful and how? How did they improve DOH performance? 40. Did you encounter any challenges implementing the tools? Please explain. 41. Do you think your department will continue using some or all of the tools after the end of the HSSC project? 1 Yes, all of them 2 Yes, some of them 3 No, none of them 4 Don’t know.....................................GO TO Q 43 42. IF YES, Which ones will you continue to use and why? IF NO, Which ones will you not continue to use and why? Health Systems Strengthening Component: Final Evaluation Report 89 SECTION 6: Demand creation for use of information for evidence based decision making (SHIS) 43. What is the regular process of making decisions related to health planning in your district? 44. Has JSI HSSC support changed the decision-making process? 1 Yes 2 No 3 Don’t know......................GO TO Q 46 45. IF YES, How? IF NO, Why not? 46. To what extent do you agree with the following statement: “the use of information for decision making has increased at the district level”? Do you strongly agree, agree, disagree, strongly disagree, or do you have no opinion? 1 Strongly agree 2 Agree 3 No opinion ..................................GO TO Q 48 4 Disagree 5 Strongly disagree 47. Can you please explain why you agree or disagree with this statement. 48. How did your program contribute to the development of the Sindh Health Information System dashboard? 49. How did your program contribute to the integration of Information systems (LHW, EPI, MNCH and Nutrition) ? Health Systems Strengthening Component: Final Evaluation Report 90 SECTION 7: Develop monitoring and supervisory (M&S) tools and checklists (M & S) I’d like to ask you some questions about how changes in the monitoring and supervision system have affected different aspects of your department’s work. 50. Didyou or your department use the monitoring and supervisory tools/checklists during your last monitoring and supervision visit? 1 Yes 2 No 3 Don’t know......................GO TO Q 53 51. IF YES, How did you use the tools? IF NO, Why did you not use the tools? 52. IF YES, In your opinion, were the tools useful? How? SECTION 8: Build capacity of DoH provincial and district staff on M&S system for use of information (Leadership/Management Capacity, M & S) I’d like to ask you some questions about how capacity building on the Monitoring and Supervision system have affected different aspects of your department’s work. 53. Have you been trained on the monitoring and supervision system and its online reporting? 1 Yes 2 No ........................ GO TO Q 55 54. IF YES, What key skills did you acquire from training? 55. How often do you use the knowledge and skills acquired during the training? Would you say you use the knowledge and skills often, sometimes, or not at all? 1 Frequently 2 Occasionally 3 Not at all 56. Please explain why you use the knowledge and skills frequently/occasionally /not at all. Health Systems Strengthening Component: Final Evaluation Report 91 57. In your opinion, how important was the contribution of JSI HSSC to overall health system strengthening? Give examples. (PROBE FOR HOW THINGS WOULD HAVE BEEN DIFFERENT WITHOUT HSS SUPPORT) 58. In your opinion, has JSI HSSC support contributed to improved health service delivery? PROBE: ACCESS, UTILIZATION, QUALITY OF RMNCH SERVICES 1 Yes 2 No 3 Don’t know......................GO TO Q 61 59. IF YES, Please explain what has improved and how JSI/HSSC contributed to the improvement. 60. IF NO, why not? (PROBE FOR BROADERSTRUCTURAL ISSUES, POLITICAL WILL, SUSTAINABILITY, ETC.) 61. In your opinion, how, if at all, has this project benefited women and /children in your district? 62. In your opinion, how, if at all, has this project benefited the poor in your district? Health Systems Strengthening Component: Final Evaluation Report 92 Instrument for District Population Welfare Officer (DPWO) Pakistan Perform; Health Systems Strengthening Component; Final Evaluation A1. Language: A2. Date: A3. Venue: A4. City/district: A5. Institution/Organization: A6. Respondent Name, Title and Gender A7. Start time: End time: A8. Moderator name: A9. Note taker name: Instructions to interviewer: Read the introduction below INTRODUCTION My name is . I work for a research organization based in Islamabad. We are conducting a study to determine the extent to which the USAID-funded Health Systems Strengthening Component has been successful in meeting its objectives. The interview will take approximately 40–60 minutes. We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No _______________) If respondent says yes, continue the interview. If no, try to motivate respondent by answering their questions and explaining the importance of recording the interview. If respondent does not agree to the recording, do not record the interview. Health Systems Strengthening Component: Final Evaluation Report 93 SECTION 1: Introduction 1. What are your roles and responsibilities in this department? 2. For how long has your department worked with JSI HSSC in this district? years months. I’d like to ask you about whether you or your department participated in HSSC interventions or activities related to the five systems HSSC supported; the district action plan and medium-term budgetary framework, the district health and population management teams, the Sindh health information system, leadership and management capacity building, and the monitoring and supervision system. I’ll ask about each of these systems separately. 3. Did you or your district office participate in any JSI HSSC interventions or activities related to district action planning and the medium-term budgetary framework? 1 Yes 2 No 3 Don’t know......................GO TO Q 6 4. IF YES, in which interventions or activities? 5. IF YES, What was your specific role in these interventions or activities? 6. Do you regularly attend district health and population management team quarterly meetings? 1 Yes 2 No 7. IF YES, What was your specific role in these interventions or activities? Health Systems Strengthening Component: Final Evaluation Report 94 8. Did you or your district office participate in any JSI HSSC interventions or activities related to the Sindh health information system? 1 Yes 2 No 3 Don’t know......................GO TO Q 11 9. IF YES, In which interventions or activities? 10. IF YES, What was your specific role in these interventions or activities? 11. Did you or your district office participate in any JSI HSSC interventions or activities related to leadership and management capacity building? 1 Yes 2 No 3 Don’t know......................GO TO Q 14 12. IF YES, In which interventions or activities? 13. IF YES, What was your specific role in these interventions or activities? 14. Did you or your district office participate in any JSI HSSC interventions or activities related to the monitoring and supervision system? 1 Yes 2 No 3 Don’t know......................GO TO Q 17 15. IF YES, In which interventions or activities? 16. IF YES, What was your specific role in these interventions or activities? Health Systems Strengthening Component: Final Evaluation Report 95 SECTION 2: Revitalizing District Health & Population Management Teams (DHPMTs) to conduct district-level management, oversight, and integration of RMNCH activities. 17. Has the role of district health and population management teams changed over the course of the past 5 years? 1 Yes 2 No 3 Don’t know......................GO TO Q 19 18. IF YES, How? IF NO, Why not? 19. Did the resumption of district health and population management team meetings improve the overall performance of the district health/Family Planning service delivery system? 1 Yes 2 No 3 Don’t know......................GO TO Q 21 20. IF YES, How? IF NO, Why not? Please explain? (PROBE: PLANNING, BUDGETING, M&E, COORDINATION, AND DECISION MAKING) 21. What, if anything, did HSSC do to help district health and population management teams resume meeting? 22. What were most effective activities in enabling coordination at district level through district health and population management team? 23. What were the least effective activities in enabling coordination at district level through district health and population management team? 24. Was the district health and population management team successful in improving coordination among DOH, PWD, PPHI, and education? 1 Yes 2 No 3 Don’t know......................GO TO Q 26 Health Systems Strengthening Component: Final Evaluation Report 96 25. Did district health and population management teams improve the integration of LHW, EPI, MNCH, and nutrition programs? 1 Yes 2 No 3 Don’t know......................GO TO Q 28 26. IF YES, How? IF NO, Why not? Please explain? SECTION 3: Technical assistance to improve coordination between stakeholders: Department of Health (DoH), Population Welfare Department (PWD), and the People’s Primary Healthcare Initiative (PPHI) (DHPMT) 27. What is the process of organizing a district health and population management team meeting? ? 28. Have you observed any challenges in organizing district health and population management team meetings? 1 Yes 2 No ......................................GO TO Q 31 3 Don’t know......................GO TO Q 31 29. IF YES, What challenges? Please explain? PROBE: WHAT WAS DONE TO ADDRESS THE CHALLENGES? 30. Do you have any suggestions on how district health and population management team meetings can be better organized? 31. What is the role of the JSI field manager in organizing these meetings? 32. In your opinion, can the Department of Health fill the role of JSI field manager after the end of HSSC project? 1 Yes 2 No 3 Don’t know......................GO TO Q 35 Health Systems Strengthening Component: Final Evaluation Report 97 33. IF YES, How? IF NO, Why not? Please explain. (PROBE: AGENDA SETTING, ENSURING PARTICIPATION, MINUTE TAKING, LOGISTICS, FOLLOW UP OF ACTION POINTS) SECTION 4: Implement individual-level capacity building interventions designed to strengthen management skills at the provincial level, such as training a core group of public sector professional health managers 34. Have you participated in JSI HSSC led trainings? 1 Yes 2 No ......................................GO TO Q 41 3 Don’t know......................GO TO Q 41 35. If yes, which ones? 1. short courses:_____________________________________________ 2. Masters degree programs: ____________________________________ 36. Was the training useful in the performance of your or your job? 1 Yes 2 No 37. IF YES, How? IF NO, Why not? 38. Has your staff received any training sponsored by JSI HSSC? 1 Yes 2 No ......................................GO TO Q 43 3 Don’t know......................GO TO Q 43 39. IF YES, What kind of training? (PROBE FOR TRAINING TOPICS OR TITLES) 40. Do you see any changes in the performance of your staff that you think are as a result of the training from JSI HSSC? 1 Yes 2 No 3 Don’t know......................GO TO Q 43 41. IF YES, How? IF NO, Why not? Health Systems Strengthening Component: Final Evaluation Report 98 42. In your opinion, has HSSC contributed to improved health service delivery? 1 Yes, how?_____________________________________________ 2 No, why not? __________________________________________ 43. In your opinion, how, if at all, has this project benefited women in your district? 44. In your opinion, how, if at all, has this project benefited the poor in your district? Health Systems Strengthening Component: Final Evaluation Report 99 Instrument for District Manager (DM) PPHI Pakistan Perform; Health Systems Strengthening Component; Final Evaluation A1. Language: A2. Date: A3. Venue: A4. City/district: A5. Institution/Organization: A6. Respondent Name: Title: Gender: A7. Start time: End time: A8. Moderator name: A9. Note taker name: Instructions to interviewer: Read the introduction below INTRODUCTION My name is . I work for a research organization based in Islamabad. We are conducting a study to determine the extent to which the USAID-funded Health Systems Strengthening project has been successful in meeting its objectives. The interview will take approximately 40–60 minutes. We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No _______________) If respondent says yes, continue the interview. If no, try to motivate respondent by answering their questions and explaining the importance of recording the interview. If respondent does not agree to the recording, do not record the interview. Health Systems Strengthening Component: Final Evaluation Report 100 SECTION 1: Introduction 1. What are your roles and responsibilities in this department? 2. For how long has your department worked with JSI/HSSC? years months SECTION 2: Technical assistance for the development of district action plans (DAPs) to improve coverage of basic health services, referral linkages among different levels of the health system, based on district needs. (DAP); District level capacity building in costing DAPs and transitioning from traditional budgeting processes to medium term budgetary frameworks. (DAP & MTBF) Now I’m going to ask you some more questions about the district action plans and the planning process. 3. In your opinion, why did JSI/HSSC introduce the district action plans? 4. Did the district action plan or the planning process affect health service delivery in your district? PROBE: NEEDS BASED BUDGETING, HEALTH FACILITY ASSESSMENT, PRIORIZATION OF NEEDS, RMNCH, ALLOCATION OF RESOURCES, REFERRAL LINKAGES AT DIFFERENT LEVELS. 1 Yes 2 No 3 Don’t know......................GO TO Q 6 5. IF YES, How (positively/negatively) did the plan or the planning process affect health service delivery and how did the plan or planning process contribute to the change? IF NO, Why did the plan or the planning process not affect health service delivery? 6. In your opinion, which, if any, JSI/HSSC activities need to be continued for the district action planning process to continue after JSI/HSSC ends? Why are these important? 7. In your opinion, is the district likely to continue to develop plans after JSI/HSSC ends? 1 Yes......................................... GO TO Q 9 2 No 3 Don’t know......................... GO TO Q 9 Health Systems Strengthening Component: Final Evaluation Report 101 8. IF NO, Why not?(PROBE: AFFORDABILITY/FINANCIAL CONSTRAINTS, CHALLENGES) SECTION 3: Revitalizing District Health & Population Management Teams (DHPMTs) to conduct district-level management, oversight, and integration of RMNCH activities. 9. In your opinion, has the district health and population management team improved the overall performance of the district health services delivery system? (PROBE: QUALITY, ACCESS, UTILISATION OF RMNCH SERVICES) 1 Yes 2 No 3 Don’t know......................GO TO Q 11 10. IF YES, How? IF NO, Why not? Please explain? Now I’d like to ask about if and how changes in the district health and population management team have affected various aspects of the health system. For each question, please tell me whether the district health and population management team has made things much better, better, worse, or had no effect. 11. To what extent did the district health and population management team affect the district performance review system? Did it make it much better, better, worse, much worse, or did it have no effect? (Probe: quarterly review meeting to assess health facility reporting, stock￾taking, monitoring and supervision, quality of care, human resource issues etc.) 1 Much better 2 Better 3 No effect...........................GO TO Q 13 4 Worse 5 Much worse 6 Don’t know 12. Please explain how you think the district health and population management team affected the district performance review system. Health Systems Strengthening Component: Final Evaluation Report 102 13. To what extent did the district health and population management team affect the district￾level management, oversight? Did it make it much better, better, worse, much worse, or did it have no effect? 1 Much better 2 Better 3 No effect 4 Worse 5 Much worse 6 Don’t know 14. Please explain how you think the district health and population management team affected the district-level management, oversight. 15. To what extent did the district health and population management team affect the coordination between stakeholders, DoH, PWD, PPHI, and education department? Did it make it much better, better, worse, much worse, or did it have no effect? 1 Much better 2 Better 3 No effect...........................GO TO Q 17 4 Worse 5 Much worse 6 Don’t know 16. Please explain how you think the district health and population management team affected the coordination between stakeholders, DoH, PWD, PPHI, and education department. 17. Besides what we discussed above, do you think changes in the district health and population management team have caused any other changes in your district? 1 Yes 2 No 3 Don’t know......................GO TO Q 21 18. If so, what are those changes and how did changes in the district health and population management team contribute to the change? (PROBE FOR HOW CHANGES IN THE DHPMT CONTRIBUTED TO THE CHANGE) Health Systems Strengthening Component: Final Evaluation Report 103 SECTION 4: Technical assistance to improve coordination between stakeholders: Department of Health (DoH), Population Welfare Department (PWD), and the People’s Primary Healthcare Initiative (PPHI) (DHPMT) 19. How are the district health and population management team meetings organized (PROBE: FOLLOW UP ON ACTION POINTS DISCUSSED IN PREVIOUS MEETING)? 20. What is the role of the JSI field manager in organizing the district health and population management team meetings? 21. Can the Department of Health fill all of the roles of the JSI field manager that are essential to the functioning of the district health and population management team meetings after the JSI/HSSC project ends? 1 Yes 2 No 3 Don’t know......................GO TO Q 23 22. IF YES, How? IF NO, why not? Please explain? (PROBE FOR HOW THE DOH WILL HANDLE SETTING AGENDAS, ENSURING PARTICIPATION, TAKING MINUTE, LOGISTICS, FOLLOWING UP ON ACTION POINTS, AND OTHER JSI ROLES MENTIONED) 23. In your opinion, will the Department of Health continue the district health and population management team meetings after the JSI/HSSC project ends? 1 Yes......................................GO TO Q 25 2 No ......................................GO TO Q 25 3 Don’t know 24. IF NO, Why not? (PROBE: AFFORDABILITY/FINANCIAL CONSTRAINTS, CHALLENGES, ETC.) Health Systems Strengthening Component: Final Evaluation Report 104 25. To what extent do you agree with the following statement: “the use of information for decision making has increased at the district level”? Do you strongly agree, agree, disagree, strongly disagree, or do you have no opinion? 1 Strongly agree 2 Agree 3 No opinion ..................................GO TO Q 27 4 Disagree 5 Strongly disagree 26. Can you please explain why you agree or disagree with this statement. 27. How did your program contribute to the development of the Sindh Health Information System dashboard? 28. How did your program contribute to the integration of Information systems (LHW, EPI, MNCH and Nutrition)? SECTION 5: Implement individual-level capacity building interventions designed to strengthen management skills at the provincial level, such as training a core group of public sector professional health managers 29. Have you participated in JSI/HSSC led trainings? 1 Yes 2 No .......................GO TO Q 31 30. IF YES, What are the names of the courses in which you participated? (WRITE IN NAMES OF COURSE/TRAININGS) 1. Short courses: ____________________________________________ 2. Master’s degree programs:____________________________________ 31. Did the training help you improve your job practices? 1 Yes 2 No 32. IF YES, How did it improve your job practices; how did you use what you learned? IF NO, Why did it not improve your practices? (PROBE FOR RELEVANCE OR QUALITY OF TRAINING) Health Systems Strengthening Component: Final Evaluation Report 105 33. Has any of your staff received training through JSI/HSSC? 1 Yes 2 No ......................................GO TO Q 36 3 Don’t know......................GO TO Q 36 34. IF YES, What kind of training did they receive? (PROBE FOR TRAINING TOPICS OR TITLES and duration) 35. Did JSI/HSSC individual level capacity building affect the work practices of your staff? 1 Yes 2 No ......................................GO TO Q 37 3 Don’t know......................GO TO Q 37 36. IF YES, How did it improve their work practices? (PROBE FOR RELEVANCE OR QUALITY OF CAPACITY BUILDING ACTIVITIES) 37. Did JSI/HSSC individual level capacity building affect your staff’s individual skills or knowledge? 1 Yes 2 No ......................................GO TO Q 40 3 Don’t know......................GO TO Q 45 38. IF YES, How did it improve their skills or knowledge? PROBE FOR RELEVANCE OR QUALITY OF CAPACITY BUILDING ACTIVITIES) 39. IF YES, What key skills did they acquire from training? 40. In your opinion, how important was the contribution of JSI/HSSC to overall health system strengthening? Give examples. (PROBE FOR HOW THINGS WOULD HAVE BEEN DIFFERENT WITHOUT HSS SUPPORT) Health Systems Strengthening Component: Final Evaluation Report 106 41. In your opinion, of the JSI/HSSC interventions, which would you consider best practices, innovations, or lessons learned for improving implementation of health system strengthening activities in Pakistan? Please describe and explain why they were best practices, innovations, or lessons learned. 42. In your opinion, has JSI/HSSC support contributed to improved health service delivery? PROBE: ACCESS, UTILIZATION, QUALITY OF RMNCH SERVICES 1 Yes 2 No 3 Don’t know......................GO TO Q 45 43. IF YES, Please explain what has improved and how HSSC contributed to the improvement. 44. IF NO, why not? (PROBE FOR BROADERSTRUCTURAL ISSUES, POLITICAL WILL, SUSTAINABILITY, ETC.) 45. In your opinion, how, if at all, has this project benefited women and children in your district? 46. In your opinion, how, if at all, has this project benefited the poor in your district? Health Systems Strengthening Component: Final Evaluation Report 107 Instrument for District Education Officer (DEO) Pakistan Perform; Health Systems Strengthening Component; Final Evaluation A1. Language: A2. Date: A3. Venue A4. City/district: A5. Institution/Organization: A6. Respondent Name, Title and Gender A7. Start time: End time: A8. Moderator name: A9. Note taker name: Instructions to interviewer: Read the introduction below INTRODUCTION My name is . I work for a research organization based in Islamabad. We are conducting a study to determine the extent to which the USAID-funded Health Systems Strengthening Component has been successful in meeting its objectives. The interview will take approximately 40–60 minutes. We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No _______________) If respondent says yes, continue the interview. If no, try to motivate respondent by answering their questions and explaining the importance of recording the interview. If respondent does not agree to the recording, do not record the interview. Health Systems Strengthening Component: Final Evaluation Report 108 SECTION 1: Introduction 1. What is the role of your department with regard to JSI/HSSC interventions? 2. For how long has the district education department worked with JSI/HSSC? years months? I’d like to ask you about whether you or your department participated in HSSC interventions or activities related to the five systems HSSC supported; the district action plan and medium-term budgetary framework, the district health and population management teams, the Sindh health information system, leadership and management capacity building, and the monitoring and supervision system. I’ll ask about each of these systems separately. 3. Did you or your department participate in any JSI/HSSC interventions or activities related to district action planning and the medium-term budgetary framework? 1 Yes 2 No ....................................... GO TO Q 6 3 Don’t know....................... GO TO Q 6 4. IF YES, In which interventions or activities? 5. IF YES, What was your specific role in these interventions or activities? 6. Did you or your department participate in any JSI/HSSC interventions or activities related to the district health and population management team? 1 Yes 2 No .....................................GO TO Q 9 3 Don’t know......................GO TO Q 9 7. IF YES, In which interventions or activities? 8. IF YES, What was your specific role in these interventions or activities? Health Systems Strengthening Component: Final Evaluation Report 109 9. Did you or your department participate in any JSI HSSC interventions or activities related to leadership and management capacity building? 1 Yes 2 No ......................................GO TO Q 12 3 Don’t know......................GO TO Q 12 10. IF YES, In which interventions or activities? 11. IF YES, What was your specific role in these interventions or activities? SECTION 2: Revitalizing District Health & Population Management Teams (DHPMTs) to conduct district-level management, oversight, and integration of RMNCH activities. (DHPMTs) 12. What is the role of your department in district health and population management team meetings? 13. Have the district health and population management team meetings changed over the course of 5 years? 1 Yes 2 No 3 Don’t know......................GO TO Q 15 14. IF YES, How? IF NO, why not?(PROBE: SCOPE, ROLE, MEMBERSHIPS/COMPOSITION, FREQUENCY OF MEETINGS, EFFECTIVENESS) 15. What, if anything, did JSI do to help district health and population management teams resume meeting? 16. What, if anything, were the most effective activities in improving coordination at district level through district health and population management team meetings? Health Systems Strengthening Component: Final Evaluation Report 110 17. What, if anything, were the least effective activities in improving coordination at district level through district health and population management team? 18. Was the district health and population management team successful in improving coordination with DoH, PWD, PPHI, and Education Department? 1 Yes 2 No 3 Don’t know......................GO TO Q 20 19. IF YES, How? IF NO, Why not? SECTION 3: Technical assistance to improve coordination between stakeholders: Department of Health (DoH), Population Welfare Department (PWD), the People’s Primary Healthcare Initiative (PPHI) and Education Department (DHPMT) 20. How are the district health and population management team meetings organized? How could this be improved? 21. Have you observed any challenges in organizing the district health and population management team meetings? 1 Yes 2 No ......................................GO TO Q 23. 3 Don’t know......................GO TO Q 23 22. IF YES, what? Please explain? PROBE: WHAT WAS DONE TO ADDRESS THE CHALLENGES? 23. What is the role of the field manager (JSI) in organizing these meetings? 24. In your opinion, will district health and population management team meetings be conducted effectively after the end of the JSI HSSC project? 1 Yes 2 No 3 Don’t know......................GO TO Q 26 Health Systems Strengthening Component: Final Evaluation Report 111 25. IF YES, How? IF NO, Why not? Please explain? 26. Did resuming the district health and population management team meetings improve the overall performance of the district health service delivery system? 1 Yes 2 No 3 Don’t know......................GO TO Q 28 27. IF YES, How? IF NO, Why not? Please explain? (PROBE: PLANNING, BUDGETING, M&E, COORDINATION, AND DECISION MAKING) 28. In your opinion, how, if at all, has this project benefited women and children in your district? 29. In your opinion, how, if at all, has this project benefited the poor in your district? Health Systems Strengthening Component: Final Evaluation Report 112 Instrument for JSI Field Coordinator Pakistan Perform; Health Systems Strengthening Component; Final Evaluation A1. Language: A2. Date: A3. Venue: A4. City/district: A5. Institution/Organization: JSI A6. Respondent Name, Title and Gender A7. Start time: End time: A8. Moderator name: A9. Note taker name: Instructions to interviewer: Read the introduction below INTRODUCTION My name is . I work for a research organization based in Islamabad. We are conducting a study to determine the extent to which the USAID-funded Health Systems Strengthening Component has been successful in meeting its objectives. The interview will take approximately 40–60 minutes. We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No _______________) If respondent says yes, continue the interview. If no, try to motivate respondent by answering their questions and explaining the importance of recording the interview. If respondent does not agree to the recording, do not record the interview. Health Systems Strengthening Component: Final Evaluation Report 113 SECTION 1: INTRODUCTION 1. For how long have you worked with HSSC? years I’d like to ask you about whether you or your department participated in HSSC interventions or activities related to the five systems HSSC supported; the district action plan and medium-term budgetary framework, the district health and population management teams, the Sindh health information system, leadership and management capacity building, and the monitoring and supervision system. I’ll ask about each of these systems separately. 2. Did you participate in any JSI HSSC interventions or activities related to district action planning and the medium-term budgetary framework, DHPMT, Dashboard, SHIS, M&S, etc? 1 Yes 2 No 3 Don’t know......................GO TO Q 6 3. IF YES, in which interventions or activities? 4. IF YES, What was your specific role in these interventions or activities? 5. Did you participate in any JSIHSSC interventions or activities related to the district health and population management team? 1 Yes 2 No 3 Don’t know......................GO TO Q 9 6. IF YES, in which interventions or activities? 7. IF YES, What was your specific role in these interventions or activities? 8. Did you participate in any JSI HSSC interventions or activities related to the Sindh health information system? 1 Yes 2 No 3 Don’t know......................GO TO Q 12 Health Systems Strengthening Component: Final Evaluation Report 114 9. IF YES, in which interventions or activities? 10. IF YES, What was your specific role in these interventions or activities? 11. Did you or your department participate in any JSI HSSC interventions or activities related to leadership and management capacity building? 1 Yes 2 No 3 Don’t know......................GO TO Q 15 12. IF YES, in which interventions or activities? 13. IF YES, What was your specific role in these interventions or activities? 14. Did you or your department participate in any JSI HSSC interventions or activities related to the monitoring and supervision system? 1 Yes 2 No 3 Don’t know......................GO TO Q 18 15. IF YES, In which interventions or activities? 16. IF YES, What was your specific role in these interventions or activities? SECTION 2: Revitalizing District Health & Population Management Teams (DHPMTs) to conduct district-level management, oversight, and integration of RMNCH activities. 17. What were your roles and responsibilities in district health and population management team? Health Systems Strengthening Component: Final Evaluation Report 115 18. Has the role of district health and population management teams changed over the course of 5 years? 1 Yes 2 No 3 Don’t know......................GO TO Q 20 19. IF YES, How? IF NO, Why not? 20. What, if anything, did HSSC do to help district health and population management teams resume meeting? 21. What were most effective activities in enabling coordination at district level through district health and population management team? 22. What were the least effective activities in enabling coordination at district level through district health and population management team? 23. Was the district health and population management team successful in improving coordination between DOHPWD and PPHI and Education department at district level? 1 Yes 2 No 3 Don’t know....................GO TO Q 25 24. IF YES, How? IF NO, Why not? 25. Did district health and population management teams improve the integration of LHW, EPI, MNCH, and nutrition programs? 1 Yes 2 No 3 Don’t know......................GO TO Q 27 26. IF YES, How? IF NO, Why not? Health Systems Strengthening Component: Final Evaluation Report 116 SECTION 3: Technical assistance to improve coordination between stakeholders: Department of Health (DoH), Population Welfare Department (PWD), and the People’s Primary Healthcare Initiative (PPHI) (DHPMT) 27. What is the process of organizing a district health and population management team meeting? 28. Have you observed any challenges in organizing the district health and population management team meetings? 1 Yes 2 No 29. IF YES, Can you describe the challenges you observed? 30. IF YES, How could district health and population management team meetings be improved? 31. In your opinion, will the Department of Health take up the role of JSI field manager after the end of HSSC project? 1 Yes 2 No 3 Don’t know......................GO TO Q 33 32. IF YES, How? IF NO, Why not? Please explain? (PROBE: AGENDA SETTING, ENSURING PARTICIPATION, MINUTE TAKING, LOGISTICS, FOLLOW UP OF ACTION POINTS) 33. Did the resumption of district health and population management teams improve the overall performance of the district health service delivery system? 1 Yes 2 No 3 Don’t know 34. IF YES, How? IF NO, Why not? Please explain? (PROBE: PLANNING, BUDGETING, M&E, COORDINATION, AND DECISION MAKING) Health Systems Strengthening Component: Final Evaluation Report 117 35. If you were to perform your role as JSI Field Coordinator all over again in the JSI/HSSC project, what would you do differently, and why? Health Systems Strengthening Component: Final Evaluation Report 118 Key Informant Interview Instrument for development partners/donors Pakistan Perform; Health Systems Strengthening Component; Final Evaluation A1. Language: A2. Date: A3. Venue: A4. City/district: A5. Institution/Organization: A6. Respondent Name, Title and Gender A7. Start time: End time: A8. Moderator name: A9. Note taker name: Instructions to interviewer: Read the introduction below INTRODUCTION My name is . I work for a research organization based in Islamabad. We are conducting a study to determine the extent to which the USAID-funded Health Systems Strengthening Component has been successful in meeting its objectives. The interview will take approximately 40–60 minutes. We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No _______________) If respondent says yes, continue the interview. If no, try to motivate respondent by answering their questions and explaining the importance of recording the interview. If respondent does not agree to the recording, do not record the interview. Health Systems Strengthening Component: Final Evaluation Report 119 1. What are your roles and responsibilities in this department/office? 2. Are you familiar with JSI project? 1 Yes 2 No ............................ GO TO Q 4 3. IF YES, How did you become familiar with JSI project? 4. Has your organization collaborated with JSI project? 1 Yes 2 No ............................ GO TO Q 17 5. IF YES, How have you collaborated (i.e., on what) and for how long? 6. On what approaches, processes, and activities have you collaborated? 7. What approaches, processes, and activities were the most effective in achieving results? PROBE: WHY WERE THEY EFFECTIVE AND TO WHAT RESULTS DID THEY CONTRIBUTE? 8. What activities were the least effective in achieving results? 9. Which approaches, processes, and activities do you think are likely to be replicated or scaled up and why? 10. Which approaches, processes, activities are less likely to be replicated or scaled up and why? 11. Has your organization scaled up/ planned to scale up any of these activities? 1 Yes 2 No ............................ GO TO Q 13 Health Systems Strengthening Component: Final Evaluation Report 120 12. IF YES, Which ones? 13. Has your organization provided funding (or allocated budget) to continue these activities? PROBE: FEDERAL, PROVINCIAL AND DISTRICT LEVEL 1 Yes 2 No ...............................................GO TO Q 15 3 Don’t know ..............................GO TO Q 15 14. IF YES, How? 15. Did the JSI project coordinate with other donors’ projects in the province? 1 Yes 2 No 3 Don’t know...............................GO TO Q 17 16. IF YES, How? IF NO, Why not? 17. Did the JSI project coordinate or consult with NGOs or the private sector at the province or district level? 1 Yes 2 No 3 Don’t know 18. IF YES, How? IF NO, Why not? Health Systems Strengthening Component: Final Evaluation Report 121 General Key Informant Interview Guide for JSI Pakistan Perform; Health Systems Strengthening Component; Final Evaluation A1. Language: A2. Date: A3. Venue: A4. City/district: A5. Institution/Organization: A6. Respondent Name, Title and Gender: A7. Start time: End time: A8. Moderator name: A9. Note taker name: Instructions to interviewer: Read the introduction below INTRODUCTION My name is . I work for a research organization based in Islamabad. We are conducting a study to determine the extent to which the USAID-funded Health Systems Strengthening Component has been successful in meeting its objectives. The interview will take approximately 40–60 minutes. We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No _____________ ) If respondent says yes, continue the interview. If no, try to motivate respondent by answering their questions and explaining the importance of recording the interview. If respondent does not agree to the recording, do not record the interview. Health Systems Strengthening Component: Final Evaluation Report 122 1. What are your roles and responsibilities in this office in relation to HSSC project? 2. In your opinion, did HSSC strengthen health systems in project areas? 1 Don’t know......................GO TO Q 6 2 None of these 3. IF YES, How? IF NO, why not? PROBE FOR DESIGN/RELEVANCE, IMPLEMENTATION, OR GOVERNMENT CAPACITY OR WILLINGNESS, STRUCTURAL ISSUES, EXTERNAL FACTORS, ETC. 4. In your opinion, what were most the useful activities in achieving results? Please elaborate. PROBE FOR WHY THEY WERE EFFECTIVE ACTIVITIES, DESIGN/RELEVANCE OR IMPLEMENTATION? 5. What were the least useful activities in achieving results? Please elaborate. PROBE FOR WHY THEY WERE NOT EFFECTIVE ACTIVITIES, DESIGN/RELEVANCE OR IMPLEMENTATION? 6. What were key constraints or challenges that may have hindered the achievement of results? Please describe and explain why they were constraints or challenges. 7. What would you change, or do differently, to improve performance in future similar projects, and why? 8. In your opinion, has HSSC project identified any best practices or introduced any innovative approaches? 1 Yes 2 No ......................................GO TO Q 10 3 Don’t know......................GO TO Q 10 9. IF YES, Please describe the best practices or innovations and why they were best practices or innovations? Health Systems Strengthening Component: Final Evaluation Report 123 10. Do you think the Department of Health will continue HSSC-initiated activities on their own after HSSC ends? 1 Yes 2 No ......................................GO TO Q 12 3 Don’t know......................GO TO Q 12 11. IF YES, How, what, if any, plans have they put in place to sustain activities? IF NO, Why not? 12. What are some of the lessons learnt from HSSC that could improve performance of future similar programs in Pakistan or elsewhere? 13. Are you aware of any other development partners that are considering supporting future health system strengthening in Sindh? 1 Yes 2 No ......................................GO TO Q 12 3 Don’t know......................GO TO Q 12 14. IF YES, at what level? Which components? IF NO, Why not? 15. In your opinion, has HSSC contributed to improved health service delivery? 1 Yes, how _________________________________________________________ 2 No, why not? ______________________________________________________ 16. In your opinion, how, if at all, has this project benefited women? 17. In your opinion, how, if at all, has this project benefited the poor? Health Systems Strengthening Component: Final Evaluation Report 124 Provincial Health Department Instrument: DG Health Services, Focal Person M&E Cell (Secretary Health Office), Focal Person M&E Cell (DG Health Office), Provincial Coordinator-DHIS, PDPPHI, DG-PWD, Provincial MNCH Program Coordinator, Provincial LHW Coordinator A1. Language: A2. Date: A3. Venue: A4. City/district: A5. Institution/Organization: DG Office, Hyderabad A6. Respondents Name: Title: Gender: A7. Start time: End time: A8. Moderator name: A9. Note taker name: Instructions to interviewer: Read the introduction below INTRODUCTION My name is . I work for a research organization based in Islamabad. We are conducting an evaluation to determine the extent to which the USAID-funded Health Systems Strengthening project has been successful in meeting its objectives. The interview will take approximately 60 to 90 minutes. We ensure that everything that you say will remain confidential. We will not use your name in our reports or give your name to anyone outside of our research team. We would like to record the conversation, so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No YES ) If respondent says yes, continue the interview. If no, try to motivate respondent by answering their questions and explaining the importance of recording the interview. If respondent does not agree to the recording, do not record the interview. Health Systems Strengthening Component: Final Evaluation Report 125 SECTION 1: Introduction 1. What are your major roles and responsibilities in the department in which you are currently serving? 2. For how long has your department worked with HSSC? years months I’d like to ask you about whether you or your department participated in JSI/HSSC interventions or activities related to the five systems JSI/HSSC supported: the district action plan and medium-term budgetary framework, the district health and population management teams, the Sindh health information system, leadership and management capacity building, and the monitoring and supervision system. I’ll ask about each of these systems separately. 3. Did you or your department participate in any JSI/HSSC interventions or activities related to district action planning and the medium-term budgetary framework? 1 Yes 2 No .......................GO TO Q 6 4. IF YES, in which interventions or activities? 5. IF YES, What was your specific role in these interventions or activities? 6. Did you or your department participate in any JSI/HSSC interventions or activities related to the district health and population management team? 1 Yes 2 No ......................GO TO Q 9 7. IF YES, in which interventions or activities? 8. IF YES, What was your specific role in these interventions or activities? Health Systems Strengthening Component: Final Evaluation Report 126 9. Did you or your department participate in any JSI/HSSC interventions or activities related to the Sindh health information system? 1 Yes 2 No ......................GO TO Q 12 10. IF YES, in which interventions or activities? 11. IF YES, What was your specific role in these interventions or activities? 12. Did you or your department participate in any JSI/HSSC interventions or activities related to leadership and management capacity building? 1 Yes 2 No ......................GO TO Q 15 13. IF YES, in which interventions or activities? 14. IF YES, What was your specific role in these interventions or activities? 15. Did you or your department participate in any JSI/HSSC interventions or activities related to the monitoring and supervision system? 1 Yes 2 No ......................GO TO Q 18 16. IF YES, in which interventions or activities? 17. IF YES, What was your specific role in these interventions or activities? Health Systems Strengthening Component: Final Evaluation Report 127 SECTION 2: Technical assistance for the development of district action plans (DAPs) to improve coverage of basic health services, referral linkages among different levels of the health system, based on district needs. (DAP); District level capacity building in costing DAPs and transitioning from traditional budgeting processes to medium term budgetary frameworks. (DAP & MTBF) Now I’m going to ask you some more detailed questions about the district action plans and the planning process. 18. In your opinion, why did JSI/HSSC introduce the district action plans? 19. Did the district action plan or the planning process affect health service delivery in the districts? PROBE: NEEDS BASED BUDGETING, HEALTH FACILITY ASSESSMENT, PRIORIZATION OF NEEDS, RMNCH, ALLOCATION OF RESOURCES. 1 Yes 2 No 20. IF YES, How (positively/negatively) did the plan or the planning process affect health service delivery and how did the plan or planning process contribute to the change? IF NO, Why did the plan or the planning process not affect health service delivery? 21. In your opinion, did the districts face any challenges transitioning from the traditional budgeting to the medium term budgetary framework? 1 Yes 2 No ......................GO TO Q 23 22. IF YES, Please describe the challenges they faced and how these challenges were resolved. 23. In your opinion, what were two or three most significant benefits of introducing the district action plan? I’d like to talk now about if and how the district action planning process will continue after the JSI/HSSC project ends. 24. In your opinion, which, if any, JSI/HSSC activities need to be continued for the district action planning process to continue after JSI/HSSC ends? Why are these important? Health Systems Strengthening Component: Final Evaluation Report 128 25. In your opinion, is the district likely to continue to develop plans after HSSC ends? 1 Yes 2 No ....................................GO TO Q 27 3 Don’t know....................GO TO Q 27 26. IF NO, Why not?(PROBE: AFFORDABILITY/FINANCIAL CONSTRAINTS, CHALLENGES) 27. Does the provincial office have plans to transition activities associated with the district action planning process to other donors, organizations, or the public sector? 1 Yes 2 No ....................................GO TO Q 29 3 Don’t know....................GO TO Q 29 28. IF YES, Please explain the plans you have made. SECTION 4: Revitalizing District Health & Population Management Teams (DHPMTs) to conduct district-level management, oversight, and integration of RMNCH activities. 29. Has the role of district health and population management teams changed during the past 5 years? 1 Yes 2 No 3 Don’t know......................GO TO Q 31 30. IF YES, How? IF NO, Why not? 31. In your opinion, has the district health and population management team improved the overall performance of the district? (PROBE: QUALITY, ACCESS, UTILISATION OF RMNCH SERVICES) 1 Yes 2 No 3 Don’t know..................GO TO Q 33 32. IF YES, How? IF NO, Why not? Please explain? Health Systems Strengthening Component: Final Evaluation Report 129 Now I’d like to ask about if and how changes in the district health and population management team have affected various aspects of the health system. For each question, please tell me whether the district health and population management team has made things much better, better, worse, or had no effect. 33. To what extent did the district health and population management team affect the district performance review system? Did it make it much better, better, worse, or did it have no effect? 1 Much better 2 Better 3 No effect...........................GO TO Q 35 4 Worse 5 Don’t know......................GO TO Q 35 34. Please explain how you think the district health and population management team affected the district performance review system? 35. To what extent did the district health and population management team affect the integration of RMNCH activities, LHW, EPI, MNCH, nutrition programs? Did it make it much better, better, worse, much worse, or did it have no effect? 1 Much better 2 Better 3 No effect...........................GO TO Q 37 4 Worse 5 Don’t know......................GO TO Q 37 36. Please explain how you think the district health and population management team affected the integration of RMNCH activities, LHW, EPI, MNCH, nutrition programs. 37. To what extent did the district health and population management team affect the district￾level management, oversight? Did it make it much better, better, worse, or did it have no effect? 1 Much better 2 Better 3 No effect...........................GO TO Q 39 4 Worse 5 Don’t know......................GO TO Q 39 38. Please explain how you think the district health and population management team affected the district-level management, oversight? Health Systems Strengthening Component: Final Evaluation Report 130 39. To what extent did the district health and population management team affect the coordination between stakeholders, DoH, PWD, PPHI, and education department? Did it make it much better, better, worse, or did it have no effect? 1 Much better 2 Better 3 No effect...........................GO TO Q 41 4 Worse 5 Don’t know......................GO TO Q 41 40. Please explain how you think the district health and population management team affected the coordination between stakeholders, DoH, PWD, PPHI, and education department. 41. Besides what we discussed above, do you think changes in the district health and population management team have caused any other changes in your district? 1 Yes 2 No ...................................GO TO Q 43 3 Don’t know....................GO TO Q 43 42. IF YES, What are those changes and how did changes in the district health and population management team contribute to the change?(PROBE FOR HOW CHANGES IN THE DHPMT CONTRIBUTED TO THE CHANGE) 43. In your opinion, did the districts face any challenges in revitalization of district health and population management team? 1 Yes 2 No ....................................GO TO Q 45 3 Don’t know....................GO TO Q 45 44. IF YES, Please describe the challenges you faced and how you resolved the challenges. SECTION 5: Technical assistance to improve coordination between stakeholders: Department of Health (DoH), Population Welfare Department (PWD), and the People’s Primary Healthcare Initiative (PPHI) (DHPMT) 45. In your opinion how organization of the district health and population management team meetings at the district level could be improved? Health Systems Strengthening Component: Final Evaluation Report 131 46. In your opinion, will the DoH continue the district health and population management team meetings after the JSI/HSSC ends? 1 Yes 2 No 3 Don’t know......................GO TO Q 48 47. IF NO, Why not? (PROBE: AFFORDABILITY/FINANCIAL CONSTRAINTS, CHALLENGES, ETC.) 48. Has your institution incorporated the district health and population management team process as part of your regular practice? 1 Yes 2 No ......................................GO TO Q 50 3 Don’t know......................GO TO Q 50 49. Please explain how you have incorporated the district health and population management team meetings into your regular process? SECTION 6: Implement individual-level capacity building interventions designed to strengthen management skills at the provincial level, such as training a core group of public sector professional health managers 50. Have you participated in JSI/HSSC led trainings? 1 Yes 2 No .......................GO TO Q 52 51. IF YES, What are the names of the courses in which you participated? (WRITE IN NAMES OF COURSES/TRAININGS) 1 Short courses: _____________________________________________________ 2 Master’s degree programs:_____________________________________________ 52. Did the training help you improve your job practices? 1 Yes 2 No 3 Don’t know......................GO TO Q 54 53. IF YES, How did it improve your job practices; how did you use what you learned? IF NO, Why did it not improve your practices? (PROBE FOR RELEVANCE OR QUALITY OF TRAINING) Health Systems Strengthening Component: Final Evaluation Report 132 54. Has any of your staff received training through JSI/HSSC? 1 Yes 2 No 3 Don’t know......................GO TO Q 56 55. IF YES, What kind of training did they receive? (PROBE FOR TRAINING TOPICS OR TITLES) 56. Did JSI/HSSC individual level capacity building affect the work practices of your staff? 1 Yes 2 No ......................................GO TO Q 58 3 Don’t know......................GO TO Q 58 57. IF YES, How did it improve their work practices? (PROBE FOR RELEVANCE OR QUALITY OF CAPACITY BUILDING ACTIVITIES) 58. Did JSI/HSSC individual level capacity building affect your staff’s individual skills or knowledge? 1 Yes 2 No ......................................GO TO Q 60 3 Don’t know......................GO TO Q 60 59. IF YES, How did it improve their skills or knowledge? PROBE FOR RELEVANCE OR QUALITY OF CAPACITY BUILDING ACTIVITIES) 60. Did JSI/HSSC individual level capacity building affect the number of trained public sector professional health managers? 1 Yes 2 No 3 Don’t know......................GO TO Q 62 61. IF NO, Why did it not increase the number of trained health managers? (PROBE FOR RELEVANCE OR QUALITY OF CAPACITY BUILDING ACTIVITIES) Health Systems Strengthening Component: Final Evaluation Report 133 SECTION 7: Conduct systems-level capacity building through the development of manuals, protocols, checklists and guidelines. (Leadership & MC) 62. Did you contribute to developing manuals, protocols, checklists, or guidelines designed to improve systems? 1 Yes 2 No ......................... GO TO Q 64 63. IF YES, Which ones?(WRITE NAME, TITLE, OR FUNCTION) 64. Have you used these tools in the last 12 months? 1 Yes 2 No ......................... GO TO Q 66 65. IF YES, Which ones have you used? IF NO, Why have you not used them? 66. Were the tools useful? 1 Yes 2 No ........................ GO TO Q 68 67. IF YES, Which ones were useful and how? How did they improve DOH performance? 68. Did you encounter any challenges implementing the tools? Please explain. 69. Do you think your department will continue using some or all of the tools after the end of the HSSC project? 1 Yes, all of them 2 Yes, some of them 3 No, none of them 4 Don’t know.............................GO TO Q 71 70. IF YES, Which ones will you continue to use and why? IF NO, Which ones will you not continue to use and why? Health Systems Strengthening Component: Final Evaluation Report 134 SECTION 8: Established structures to operationalize monitoring and evaluation system by: 1) bringing management information systems online, 2) Integrating all MIS, 3) refurbish M&E cells at DGHSS and DHO offices, and 4) integrated dashboard -SHIS. SHIS, M & S 71. Has the office of the monitoring and evaluation cell at the provincial level been refurbished? 1 Yes 2 No 3 Don’t know.............................GO TO Q 73 72. IF YES, Has that helped in performing MIS-related activities? IF NO, Why has the cell’s office not been refurbished? I’d like to ask you some questions about how changes in the health information system have affected different aspects of your department’s work. 73. Did strengthening the health information system affect planning and management in your department? 1 Yes 2 No 3 Don’t know.............................GO TO Q 75 74. IF YES, please explain how it affected planning and management. (PROBE BETTER/WORSE AND WHY); IF NO, why did it not improve planning and management? PROBE: ACCESS, ABILITY TO USE, PRIORITIZATION OF NEEDS 75. Did strengthening the health information system affect health facility reporting in your department? 1 Yes 2 No 3 Don’t know.............................GO TO Q 77 76. IF YES, please explain how it affected reporting. (PROBE BETTER/WORSE AND WHY; PROBE FOR TIMELINESS, COMPETENESS, ACCURACY). IF NO, why did it not improve reporting? 77. Did strengthening the health information system affect organizational behavior around reporting in your department? 1 Yes 2 No 3 Don’t know.............................GO TO Q 79 Health Systems Strengthening Component: Final Evaluation Report 135 78. IF YES, please explain how it affected reporting behavior. PROBE: BETTER, GOT WORSE, WHY IF NO, why did it not improve reporting behavior? PROBE: OPENESS TO SHARE INFORMATION, REGULARITY, CONFINDENCE 79. Did strengthening the health information systems affect the ability and capacity of provincial health officials to monitor and verify health sector data? 1 Yes 2 No 3 Don’t know.............................GO TO Q 81 80. IF YES, please explain how it affected monitoring and supervision capacity. (PROBE BETTER/WORSE AND WHY) IF NO, why did it not improve monitoring and supervision capacity? 81. Did strengthening the health information system affect integration of DHIS, LHW-MIS, CMW-MIS, cLIMS, vLMIS? 1 Yes 2 No 3 Don’t know.............................GO TO Q 83 82. IF YES, Please explain how it affected integration. (PROBE BETTER/WORSE AND WHY) IF NO, Why did it not improve integration? SECTION 9: Demand creation for use of information for evidence-based decision making(SHIS) 83. What is the regular process of making decisions related to health planning in your department? 84. Has JSI/HSSC support changed the decision-making process? 1 Yes 2 No 3 Don’t know.............................GO TO Q 86 85. IF YES, How? IF NO, Why not? Health Systems Strengthening Component: Final Evaluation Report 136 86. In your opinion, what sources of information do the district health offices use for developing district action plans? 87. What sources of information do you use for preparing and conducting monitoring and supervision visits? 88. To what extent do you agree with the following statement: “the use of information for decision making has increased at the district level”? Do you strongly agree, agree, disagree, strongly disagree, or do you have no opinion? 1 Strongly agree 2 Agree 3 No opinion ............................ GO TO Q 90 4 Disagree 5 Strongly disagree 89. Can you please explain why you agree or disagree with this statement? SECTION 10: Develop monitoring and supervisory (M&S) tools and checklists (M & S) I’d like to ask you some questions about how changes in the monitoring and supervision system have affected different aspects of your department’s work. 90. Did you or your department use the monitoring and supervisory tools/checklists during your last monitoring and supervision visit? 1 Yes 2 No 3 Don’t know.............................GO TO Q 93 91. IF YES, How did you use the tools? IF NO, Why did you not use the tools? 92. IF YES, in your opinion, were the tools useful? How? Health Systems Strengthening Component: Final Evaluation Report 137 SECTION 11: Build capacity of DoH provincial and district staff on M&S system for use of information (Leadership/MC, M & S) I’d like to ask you some questions about how capacity building on the monitoring and supervision system have affected different aspects of your department’s work. 93. Have you been trained on the monitoring and supervision system and its online reporting? 1 Yes 2 No ......................................GO TO Q 97 94. IF YES, What key skills did you acquire from training? 95. How often do you use the knowledge and skills acquired during the training? Would you say you use the knowledge and skills frequently, occasionally, or not at all? 1 Frequently 2 Occasionally 3 Not at all 96. Please explain why you use the knowledge and skills frequently/occasionally /not at all. 97. In your opinion, how important was the contribution of JSI/HSSC to overall health system strengthening? Give examples. (PROBE FOR HOW THINGS WOULD HAVE BEEN DIFFERENT WITHOUT JSI/HSSC SUPPORT) Health Systems Strengthening Component: Final Evaluation Report 138 98. In your opinion, how effective were each of the following sets of JSI/HSSC activities in supporting strengthening the health system? Would you say they were very effective, effective, ineffective, or you don’t know? (CIRCLE ONE NUMBER FOR EACH ROW AND WRITE AN EXPLANATION) Type of activity Very effective Effective Ineffective Don’t know What assistance, if any, was an exceptionto your overall assessment? 1 How effective was technical assistance for the MIS? 1 2 3 4 2 How effective was training for the MIS? 1 2 3 4 3 How effective as technical assistance for developing the district action plan? 1 2 3 4 4 How effective as training for developing the district action plan? 1 2 3 4 5 How effective was technical assistance to develop checklists and manuals? 1 2 3 4 6 How effective was support for DHMPT facilitation? 1 2 3 4 99. In your opinion, of the JSI/HSSC interventions, which would you consider best practices, innovations, or lessons learned for improving implementation of health system strengthening activities in Pakistan or elsewhere? Please describe and explain why they were best practices, innovations, or lessons learned. 100. In your opinion, has JSI/HSSC support contributed to improved health service delivery? PROBE: ACCESS, UTILIZATION, QUALITY OF RMNCH SERVICES 1 Yes 2 No ....................................GO TO Q 102 3 Don’t know....................GO TO Q 103 Health Systems Strengthening Component: Final Evaluation Report 139 101. IF YES, Please explain what has improved and how JSI/ HSSC contributed to the improvement. THEN GO TO Q 103 102. IF NO, why not? (PROBE FOR BROADER STRUCTURAL ISSUES, POLITICAL WILL, SUSTAINABILITY, ETC.) 103. In your opinion, how, if at all, has this project benefited women in the districts? 104. In your opinion, how, if at all, has this project benefited the poor in the districts? Health Systems Strengthening Component: Final Evaluation Report 140 General Key Informant Interview Guide for USAID Pakistan Perform; Health Systems Strengthening Component; Final Evaluation A1. Language: A2. Date: A3. Venue: A4. City/district: A5. Institution/Organization: A6. Respondent Name, Title and Gender A7. Start time: End time: A8. Moderator name: A9. Note taker name: Instructions to interviewer: Read the introduction below INTRODUCTION My name is . I work for a research organization based in Islamabad. We are conducting a study to determine the extent to which Pakistan Perform; Health Systems Strengthening Component has been successful in meeting its objectives. The interview will take approximately 40–60 minutes. We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No) If respondent says yes, continue the interview. If no, try to motivate respondent by answering their questions and explaining the importance of recording the interview. If respondent does not agree to the recording, do not record the interview. Health Systems Strengthening Component: Final Evaluation Report 141 1. What are your roles and responsibilities in this office in relation to HSSC project? 2. In your opinion, did HSSC strengthen health systems in project areas? 1 YES 2 NO 3 Don’t Know 3. If yes, how? If not, why do you think so? 4. What were the most useful activities in achieving results? Please elaborate. 5. What were the least useful activities in achieving results? Please elaborate. 6. What were key constraints or challenges that may have hindered the achievement of greater results? 7. What would you change, or do differently, in future projects, for next time and why? 8. In your opinion has HSSC Project had any best practices and introduced any innovative approaches? 1. YES 2. NO 3. Don’t Know 9. If yes, which ones? 10. Do you think DOH will continue HSSC initiated activities on their own? 1 YES 2 NO 3 Don’t Know 11. If yes, How? If not, why do you think so? Health Systems Strengthening Component: Final Evaluation Report 142 12. What are some of the lessons learnt from HSSC? 13. Is USAID considering supporting future health system strengthening through its core program? 1. YES 2. NO 3. Don’t Know 14. If yes, at what level? Which components? If no, why not? 15. In your opinion, has HSSC contributed to improved health service delivery? 1 Yes, how _________________________________________________________ 2 No, why not? ______________________________________________________ 16. In your opinion, how has this project benefited women? 17. In your opinion, how this project benefited the poor? Health Systems Strengthening Component: Final Evaluation Report 143 Key Informant Interview Instrument for Federal Level Pakistan Perform; Health Systems Strengthening Component; Final Evaluation A1. Language: A2. Date: A3. Venue: A4. City/district: A5. Institution/Organization: A6. Respondent Name, Title and Gender A7. Start time: End time: A8. Moderator name: A9. Note taker name: Instructions to interviewer: Read the introduction below INTRODUCTION My name is . I work for a research organization based in Islamabad. We are conducting a study to determine the extent to which the USAID-funded Health Systems Strengthening Component has been successful in meeting its objectives. The interview will take approximately 40–60 minutes. We will treat everything you say here confidentially. We will not use your name in our reports or give your name to anyone outside of the research team. We would like to record the conversation so we can refer to the recording when we prepare our notes. Do I have your permission to record the interview? (Yes/No _______________) If respondent says yes, continue the interview. If no, try to motivate respondent by answering their questions and explaining the importance of recording the interview. If respondent does not agree to the recording, do not record the interview. Health Systems Strengthening Component: Final Evaluation Report 144 1. What are your roles and responsibilities in this Ministry//office? 2. Are you aware of the JSI/HSSC project? 1 Yes 2 No ......................................GO TO Q4 3 Don’t know......................GO TO Q4 3. IF YES, What was your role, if any, with the JSI/HSSC project? 4. What was the role, if any, of JSI in establishing the health planning systems strengthening and information analysis unit in the ministry? 5. What was the reason for establishing the health planning systems strengthening and information analysis unit at the federal level? PROBE: WHY NOT IN SINDH “Do you see any changes in decision making as a result of implementing the health planning systems strengthening and information analysis unit?” Please explain.(PROBE: WHAT CHANGES DID YOU SEE AND HOW DID THE HEALTH SYSTEMS POLICY AND INFORMATION UNIT CONTRIBUTE) 6. Did you face any challenges in establishing the health planning systems strengthening and information analysis unit at federal level? Please describe the challenges. 7. In your opinion, will the government be able to continue the health planning systems strengthening and information analysis unit after the end of the JSI/HSSC project? PROBE: SYSTEMS AND STRUCTURES, FINANCES, HR, EQUIPMENTS, MATERIALS 1 Yes 2 No 3 Don’t know......................GO TO Q 10 8. IF YES, How? IF NO, Why not? 9. What was JSI/HSSC’s contribution, if any, to the development of the anti-microbial resistance policy? (PROBE: WHAT TYPE OF ASSISTANCE DID THE JSI PROJECT HELP WITH DEVELOPMENT OF THE POLICY?) 10. In your opinion, what was the rationale for developing the anti-microbial resistance policy? 11. Has the anti-microbial resistance policy been implemented? 1 Yes 2 No 3 Don’t know......................GO TO Q 14 12. IF YES, How? IF NO, Why not? PROBE: SYSTEMS AND STRUCTURES Health Systems Strengthening Component: Final Evaluation Report 145 13. What role, if any, did JSI/HSSC play in the development of the National Health Vision? Please explain. 14. What was the rationale for developing this policy? 15. Has it been implemented? 1 Yes 2 No ......................................ASK NEXT QUESTION AND THEN GO TO Q 20 3 Don’t know......................GO TO Q 20 16. IF YES, How? IF NO, Why not? 17. Do you see any changes as a result of implementing the national health vision? 1 Yes 2 No 3 Don’t know......................GO TO Q 20 18. IF YES, How? IF NO, Why not? 19. What systems and structures, if any, are in place to ensure implementation of national health vision? 20. What role, if any, did JSI/HSSC play in introducing Chlorhexidine (CHX) in Pakistan? 21. Do you see any changes in newborn care as a result of introducing CHX in Pakistan? (PROBE: ACCESS, UTILIZATION, QUALITY OF SERVICE, ETC) 1 Yes 2 No 3 Don’t know......................GO TO Q 24 22. IF YES, What has changed? IF NO, Why not? 23. In your opinion, will CHX scale-up continue after the JSI/HSSC project support ends? 1 Yes 2 No 3 Don’t know......................GO TO Q 26 24. IF YES, how will scale-up be continued? 25. Have the trainings of CHX been scaled up at the provincial and district levels? 1 Yes 2 No 3 Don’t know......................GO TO Q 28 26. IF YES, How? IF NO, Why not? Health Systems Strengthening Component: Final Evaluation Report 146 27. In your opinion did JSI/HSSC meet its objective of strengthening transparency and accountability at the federal level? PROBE: ACCESS TO INFORMATION, PARTICIPATION, EVIDENCE BASED DECISION, TAKING RESPONSIBILITY HSPIU 1 Yes 2 No 28. IF YES, How? IF NO, Why not? 29. If you had to implement these activities in the future, how might you implement them differently? PROBE: (WHAT ACTIVITIES SHOULD THE PROJECT DO MORE OF, WHAT ACTIVITIES SHOULD THE PROJECT DO LESS OF? Health Systems Strengthening Component: Final Evaluation Report 147 Annex 4: Effectiveness of District-Level Activities This annex summarizes findings on the effectiveness of HSS district-level activities in supporting and strengthening the health systems. Name of Activity Percentage of Respondents Who Said Activity Was Effective Reasons for Effectiveness Individual-level capacity building: The objective of the individual-level capacity building activity was to increase the capacity of district-level health staff to effectively operate systems that enable effective and equitable service delivery. This involved transfer of knowledge and skills needed by managerial and executive staff. Individual capacity development included performance monitoring and management; coaching and mentoring; participation in planning and implementation of programs; and work assignments that promoted increased skills development. Critical aspects of individual capacity building were on-the-job training, followed by support and accountability in the work unit to ensure application of knowledge on the job. 100 percent reported improved job practices. The HSS training improved knowledge and skills of managerial staff. The program conducted both long- and short-term courses and provided technical assistance for the implementation of the other four subsystems. The technical assistance primarily consisted of on-the-job and hands-on support. The HSS project trained a total of 3,668 health workers, among whom 77 participated in long courses: 70 participated in master’s in public health (MPH) programs, and 7 participated in master’s in public policy (MPP) programs. Other trainees participated in a broad range of short courses, including management (86); development of DAP (105); development of MTBF (580); DHIS (2,350); M&S checklist (200); online dashboard (200); and DAP software (70). Systems-level capacity building: Systems-level capacity building included technical assistance to develop a comprehensive long-term vision for health service delivery that is responsive to the needs of the community and to global commitments; to develop the legislation, policies, and strategies to achieve that vision; to mobilize, deploy, and account for resources; and to provide transparent and accountable governance. Systems-level capacity building also included engaging and guiding DoH to improve service delivery by unifying protocols and standard operating procedures. 100 percent reported that the tools were useful. The HSS project engaged and guided DoH staff in improving service delivery by merging clinical standards, protocols, and standard operating procedures for service delivery, management, and M&S. Health Systems Strengthening Component: Final Evaluation Report 148 Name of Activity Percentage of Respondents Who Said Activity Was Effective Reasons for Effectiveness Technical assistance for DAP: HSS provided technical assistance for DAP to improve coverage of basic health services and provide referral linkages among different levels of the health system based on district needs. HSS assisted with development of district analytical profiles, operational plans, activity-based costing, and needs-based budgeting. 85 percent reported that technical assistance for DAP was effective. Technical assistance strengthened DAP planning processes, created new budget lines, and promoted informed decision-making. Training for DAP and MTBF: HSS provided district-level training for costing DAPs and transitioning from traditional budgeting processes to MTBFs. 80 percent reported that training for DAP and MTBF was effective. Training for DAP enabled DoH to undertake needs-based planning and budgeting. Revitalization of DHPMTs: HSS provided technical assistance to DHPMTs to conduct district-level management, oversight, and integration of RMNCH activities. 100 percent reported that facilitation for DHPMTs was effective. DHPMTs strengthened planning processes to improve service delivery in their respective districts. Technical assistance for improving coordination of DHPMTs: HSS created forums to resolve district-level issues through coordination among district health offices, PWD, district education offices, and PPHI. 91 percent reported that DHPMTs improved coordination between stakeholders. In December 2013, the chief secretary of Sindh notified DHPMTs to resolve operational and managerial issues through coordination with PPHI, PWD, DoE, and municipal corporation authorities, contributing to the integration of RMNCH activities. Coordination between districts and provinces: HSS developed an online system to share meeting minutes and district actions with the DGHS at the provincial office. A small proportion of respondents (10 percent) reflected that the coordination between district and provincial officials improved. District health offices upload DHPMT meeting minutes on a quarterly basis; however, the provincial DoH does not resolve district issues in a timely manner. Operationalization of M&E system: HSS established M&E cells within district health offices to input performance data and in the DGHS office to monitor the overall progress of provincial and district programs. The M&E system was devised to collect, compile, and disseminate up-to-date information for all programs, especially information pertaining to key performance indicators, through the Sindh province M&E Dashboard. The M&E Dashboard reflects the overall ranking of all the districts of Sindh. 86 percent pointed out that M&E cells (district or provincial) have been refurbished. Twenty-one out of 23 districts have established M&E cells. Health facility reporting has improved, largely due to functioning M&E cells and shared accountability and transparency. Access to and availability of data have improved. Demand creation for use of information: The aim of developing the Sindh Health Information System (SHIS) was to facilitate evidence-based decision-making at various levels of the health system. 97 percent agreed that use of information for decision-making at the district level has improved. SHIS enabled leaders at the district and provincial levels to provide oversight and make evidence￾based decisions. Health Systems Strengthening Component: Final Evaluation Report 149 Name of Activity Percentage of Respondents Who Said Activity Was Effective Reasons for Effectiveness Development of M&S checklists and tools: HSS developed online M&S checklists to monitor the quality of health services with continuous supervisory support to service providers. 29 percent reported that they contribute to developing manuals, protocols, checklists, or guidelines designed to improve systems. Online analytical reports of the M&S checklists are available so that supervisors can provide feedback to improve quality of care. M&S tools enabled recipients to identify and resolve problems with limited resources. The M&S system also contributed to changes in people’s mindsets about sharing information and increased confidence in reported data among program coordinators. Capacity building in M&S and use of information: HSS provided training to district- and provincial-level health managers on monitoring and supervisory systems and provided hands-on support for proper use of tools to enable the DoH to develop online M&S plans. 62 percent reported that they have been trained on monitoring and supervisory systems and online reporting. Training improved accuracy, timeliness, completeness, and frequency of health facility reporting, and strengthened management oversight and feedback loops within and between departments. U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523