SSDI-SYSTEMS ACTIVITY PERFORMANCE EVALUATION October 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently for DevTech by Dr. Frank Paulin, Mr. Themba Mhango, and Mr. Willie Kachaka. ACKNOWLEDGEMENTS This evaluation was made possible with financial support provided by the United States Agency for International Development (USAID) mission in Malawi. DevTech and the evaluation team would like to thank the many individuals, organizations and institutions whose support was crucial to the success of this evaluation. Throughout our journey of inquiry, which spanned the capital city of Lilongwe and nine districts in the five health zones, we were met with kindness, openness and willingness to respond to our probing questions. The team would especially like to thank the USAID resident mission staff for their support, in particular Mr. Chimwemwe Chitsulo, Monitoring, Evaluation and Learning Specialist, for guidance throughout the evaluation process. We are grateful to the resident staff of Abt Associates who provided the evaluation team with valuable data and insights into the activity’s dynamics and evolution. Our special thanks go to Mr. Takondwa Mwase, Chief of Party. Finally we would like to recognize all the senior and mid-level officials from the Ministry of Health, who took time out from their busy schedule to meet with us. Our special thanks go to Dr. Robson Kachala, the SSDI project coordinator at the Ministry of Health, for his support. Finally we would like to thank all Ministry of Health staff interviewed during our data collection in the districts visited: The information they provided to us proved invaluable for the report’s findings, conclusions and recommendations. SSDI-Systems Activity Performance Evaluation October 2016 AID-612-TO-16-00003 DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. SSDI-Systems Performance Evaluation i CONTENTS ACRONYMS AND ABBREVIATIONS ......................................................................................III EXECUTIVE SUMMARY.............................................................................................................IV ACTIVITY BACKGROUND ..................................................................................................................................... IV EVALUATION PURPOSE, QUESTIONS AND AUDIENCE ............................................................................ IV EVALUATION DESIGN, METHODOLOGY AND LIMITATIONS ................................................................ IV KEY FINDINGS, CONCLUSIONS AND RECOMMENDATIONS ..................................................................V I. ACTIVITY BACKGROUND ................................................................................................ 1 ACTIVITY CONTEXT ................................................................................................................................................1 EVALUATION PURPOSE, QUESTIONS AND AUDIENCE .............................................................................1 DEVELOPMENT HYPOTHESIS ................................................................................................................................2 II. EVALUATION DESIGN, METHODS AND LIMITATIONS ............................................ 3 EVALUATION DESIGN ..............................................................................................................................................3 DATA SOURCES...............................................................................................................................................................3 SAMPLING METHODOLOGY ..........................................................................................................................................4 FIELD IMPLEMENTATION PLAN ......................................................................................................................................4 DATA ANALYSIS .........................................................................................................................................................5 LIMITATIONS AND CONSTRAINTS ....................................................................................................................5 III. FINDINGS AND RECOMMENDATIONS .......................................................................... 7 EVALUATION QUESTION 1 ....................................................................................................................................7 EVALUATION QUESTION 2 ..................................................................................................................................10 EVALUATION QUESTION 3 ..................................................................................................................................15 EVALUATION QUESTION 4 ..................................................................................................................................18 EVALUATION QUESTION 5 ..................................................................................................................................23 IV. GENDER AND DISABILITY ANALYSIS ......................................................................... 26 ANNEX I: SCOPE OF WORK AND BACKGROUND INFORMATION .............................. 27 ANNEX 2: EVALUATION PLANNING MATRIX .................................................................. 30 ANNEX 3: LIST OF 16 PMP PRIORITY INDICATORS ......................................................... 33 ANNEX 4: LIST OF ZONES, DISTRICTS AND FACILITIES VISITED ............................... 35 ANNEX 5: LIST OF INTERVIEWEES ...................................................................................... 36 ANNEX 6: DATA COLLECTION INSTRUMENTS AND GUIDELINE ............................... 40 SSDI-Systems Performance Evaluation ii ANNEX 7: SUMMARY OF QUALITATIVE ANALYSIS OF TRANSCRIBED KEY INFORMANT INTERVIEWS ..................................................................................................... 46 ANNEX 8: BIBLIOGRAPHY ...................................................................................................... 53 LIST OF FIGURES Figure 1. Total data collected by type of respondents and instrument used ......................................................5 Figure 2. Progress on key policies, strategies and guidelines finalization .............................................................8 Figure 3. Percent of respondents who felt that e-ISS had eliminated parallel supervision ...............ERROR! BOOKMARK NOT DEFINED. Figure 4. Improved regularity of supervision ............................................................................................................11 Figure 5. Staff encouraged to act on supervision recommendations ..................................................................12 Figure 6. Staff reporting identification of new issues through e-ISS ....................................................................13 Figure 7. Staff reporting increased job satisfaction through e-ISS ........................................................................13 Figure 8. Training improved quality of service ..........................................................................................................13 Figure 9. Routine use of HMIS/DHIS data for program planning .........................................................................15 Figure 10. Progress on quarterly and annual performance reviews ....................................................................17 Figure 11. Levels of confidence in using iHRIS data in planning and review ......................................................17 Figure 12. Record-keeping before and after financial management coaching....................................................19 Figure 13. Perceived effectiveness of coaching by long-term advisors ...............................................................20 Figure 14. Perceived effectiveness of coaching by short-term advisors .............................................................20 Figure 15. Staff interviewed that had attended training ..........................................................................................21 Figure 16. Effectiveness of training as best capacity development .......................................................................21 Figure 17. Perceptions of effectiveness of training in improving services ..........................................................22 SSDI-Systems Performance Evaluation iii ACRONYMS AND ABBREVIATIONS CMED Central Monitoring and Evaluation Division DHIS2 District Health Information System version 2 DHO District Health Office DHMT District Health Management Team DIP District Implementation Plan DPPD Department of Policy Planning and Development HSSP Malawi Health Sector Strategic Plan e-ISS Electronic Integrated Supportive Supervision iHRIS Human Resource Information System IT Information technology HIV Human Immunodeficiency Virus HMIS Health management information system MoH Ministry of Health MYP Multiyear Plan NAC National Aids Commission NGO Non-governmental organization NHA National Health Accounts NMCP National Malaria Control Program PBI Performance-based incentive PDU Policy Development Unit PMP Performance monitoring plan SSDI Support for Service Delivery Integration SWAp Sector-wide approach USAID U.S. Agency for International Development ZHO Zonal Health Office SSDI-Systems Performance Evaluation iv EXECUTIVE SUMMARY ACTIVITY BACKGROUND Support for Service Delivery Integration (SSDI) was a five-year, USAID-funded project, implemented through three separate but interrelated activities: SSDI-Systems, SSDI-Services and SSDI￾Communications. The SSDI-Systems activity, implemented by Abt Associates Inc. from September 2011 to September 2016, focused on six areas of results: (1) Increased and coordinated advocacy for and implementation of evidence-based policies; (2) Strengthened strategic management and leadership capacity of the Ministry of Health (MoH); (3) Improved and strengthened MoH zone supervision structures; (4) Improved leadership and management of human resources for health; (5) Improved decentralized management of district health services; and (6) Strengthened health financing mechanisms, fiscal planning and budget execution capability at national, zone and district levels for sustainability. EVALUATION PURPOSE, QUESTIONS AND AUDIENCE The purpose of this performance evaluation was to determine the effectiveness of the approach taken to improve the functionality of the Malawian health system to achieve equitable and sustainable health results in line with current health priorities. The following questions formed the basis of the evaluation, as outlined in Annex 1:  To what extent did SSDI-Systems enable institutionalization of health policy and financing functions, including policy analysis and development and National Health Accounts (NHA), in the MoH Department of Planning and Policy Development (DPPD)?  How successful has the integrated supportive supervision mobile tool been in improving supervision of health facilities, including follow-up of issues identified in supervision?  To what extent did SSDI-Systems improve utilization of data (including the human resource information system (iHRIS), District Health Information System version 2 (DHIS2) and supportive supervision data) for evidence-based planning and decision-making at central and district levels?  Of the capacity-strengthening approaches employed by SSDI-Systems (e.g., secondment of staff to MoH/National AIDS Commission (NAC) positions, provision of technical assistance by external advisors, training and coaching of MoH and district staff, and operational and logistical support), which have most enabled the MoH to address central and district system-level issues?  What are the most significant accomplishments, best practices and lessons learned from the SSDI-Systems activity? Explicitly identify and document the facilitating and inhibiting factors to positive performance for each of the above questions. The findings of this evaluation will inform the MoH, USAID and other development partners in the design and prioritization of future investments in strengthening Malawi’s health system. EVALUATION DESIGN, METHODOLOGY AND LIMITATIONS The evaluation team comprised three consultants and two research assistants, and it was conducted between July 25 and Sept. 30, 2016. Data review and areas of inquiry covered the entire duration of the activity, with additional emphasis on activities conducted after the 2014 midterm performance evaluation. A multistage sampling and mixed-methods approach was chosen to collect data. A total of 86 stakeholders from the MoH and development partners were interviewed. Quantitative data analysis was performed with STATA 14.0 and Excel software, and qualitative analysis with DEDOOSE software. This non-experimental design does not account for influence of non-SSDI-Systems interventions because no control districts were assessed. Baseline-to-endline comparisons could have been influenced by some SSDI-Systems Performance Evaluation v unobserved confounding factors. There is also the possibility of recall bias as the evaluation inquired into changes that occurred over a five-year period. Multiple SSDI evaluations conducted simultaneously may have contributed to respondent fatigue. The evaluators did not question validity of published activity data. KEY FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Evaluation question 1 To what extent did SSDI-Systems enable institutionalization of health policy and financing functions, including policy analysis and development and NHA, in the MoH’s DPPD? Findings Before the project, the RFA highlighted that the MoH faced limited policy development capacity at the central level, both in numbers and skills in policy development and analysis. Between 2012 and 2016, SSDI-Systems significantly strengthened the institutional development of the Policy Development Unit (PDU): Terms of reference were developed, a methodological approach toward policy development introduced through mentoring and staffing levels increased. In 2016, the PDU was leads the MoH on all policy developments. The DPPD’s health financing function was mostly strengthened by establishing the position of an NHA desk officer and forming a Health Financing Task Force in the sector-wide approach (SWAp) secretariat. Significant numbers of health financing outputs were developed and disseminated (National Health Financing Strategy, NHA, expenditure tracking studies, performance-based incentive (PBI) pilot design). No structure comparable to the PDU exists in the MoH to date to provide overall stewardship and oversight over development of health financing options. Conclusion The MoH’s policy-development process has been strengthened with the establishment of the PDU. DPPD now is able to lead in policy development and analysis by adapting to the use of the new guidelines, which were structured and follow an evidenced-based approach unlike before 2012. Despite the lack of a unit to coordinate health financing functions under one roof, institutionalization of health financing in the MoH in collaboration with other partners has been achieved. To date, the PDU is able to successfully coordinate all health financing functions with key partners supporting health reforms, as evidenced by participation of the executive and senior government officials in disseminating policies and health financing options. Piloting of health financing options such as PBI remains a challenge in the absence of a dedicated institutional home and with little coordination among designated focal points. Recommendations  Continue strategic support to the PDU to sustain the institutional capacity gains because up until now, not all key positions are filled, for example the position of a deputy director-level official was vacant during the final year of the project.  Advocate for establishing a dedicated health financing unit within the DPPD responsible for coordinating all health financing functions, which require a specific set of expertise. All health financing reform initiatives should be migrated to this unit within one year.  Repeat the successful PDU capacity and institutional development approach for the health financing unit, with terms of reference and well-qualified staff, to give it mandate and governance structures for all health financing functions. SSDI-Systems Performance Evaluation vi Evaluation question 2 How successful has the integrated supportive supervision mobile tool been in improving supervision of health facilities, including follow-up of issues identified in supervision? Findings The Health Sector Strategic Plan (HSSP) 2012-2016 highlights the lack of effective management tools for improving zonal and District Health Management Team (DHMT) supervision of health care workers to strengthen quality of service as one of the key challenges faced by Malawi’s health systems. In a short period of time (2012-2015), SSDI-Systems has supported the MoH in developing the smartphone-based Electronic Integrated Supportive Supervision (e-ISS) tool, provided the hardware, trained sufficient staff to operate the system and scaled up its use to the 15 SSDI districts. Although reliable pre-2012 data on supervision is not available, most staff interviewed agree that the tool has improved the regularity of supervision of DHMTs by Zonal Health Offices (ZHOs). Supervision quality remains an issue, and the MoH has responded with the creation of a dedicated Quality Management Unit (QMU) within the DPPD. The unit will be the institutional home for e-ISS in the future and oversee its roll-out as a national program. Conclusion The e-ISS tool is a success story because already the MoH feels ready to scale it up to a national program. Remaining implementation challenges are evident in the fact that while ZHOs complete 80 percent of their quarterly supervision schedule, DHMTs’ compliance with supervision guidelines remains an issue that needs to be addressed by the newly established QMU on a priority basis. Districts also need to include in their District Implementation Plans (DIPs) sufficient budget for effective adherence to supervision schedules. On the technical side, it is recommended to migrate the e-ISS tool onto the DHIS2 platform to increase its management relevance. It is important to continue training staff in the tool’s use to counter staff attrition. Recommendations  Institutionalization and capacity development of the QMU should be supported to further strengthen the use of the e-ISS tool as a service quality improvement tool.  The recommendations of the e-ISS guideline on scheduling and resourcing external and internal supervision should be reviewed with all ZHO and DHMT stakeholders to pave the way for harmonized and cost-effective implementation.  It is recommended to link the e-ISS tool with the DHIS2 platform so that its utility as a “real time” management tool is further enhanced. Evaluation question 3 To what extent did SSDI-Systems improve utilization of data (including iHRIS, DHIS and supportive supervision data) for evidence-based planning and decision-making at central and district levels? Findings One of the main weaknesses of Malawi’s health systems, as identified in the HSSP and RFA is the weak data basis for policy development and service planning. SSDI-Systems support enabled the Central Monitoring and Evaluation System (CMED) to improve health management information system (HMIS) “It (the tool) has brought seriousness both at management level, (and) at the zonal level. Because when we go there (to DHMT) we are guided by this tool and we make informed decisions right there and we see how the facility is performing.”–Zone Health Officer SSDI-Systems Performance Evaluation vii reporting completeness from 40 to 80 percent, which represents a good basis for evidence-based planning. SSDI-Systems then supported the Human Resource Management Department in the development and roll-out of iHRIS, which, for the first time, provided the DHMTs with a tool to proactively manage its workforce. Developed with SSDI-Systems support, the new e-ISS tool links supervision findings to HMIS targets so that recommendations are evidence-based. SSDI-Systems further developed improved guidelines and processes for the management of Multiyear Plans (MYP)/DIPs and the periodic performance review processes, both of which are now intrinsically linked to HMIS data. All of these activities were paired with sufficient and effective trainings. All staff interviewed agreed that that the quality and follow-up of district planning and performance reviews has improved with the introduction of the guidelines and tools supported by SSDI-Systems. All human resource officers interviewed liked and used iHRIS for developing reports for District Health Offices (DHOs) and felt that their human resource database was mostly up to date. DHMTs now regularly discuss and follow up on e-ISS recommendations and iHRIS reports at their meetings. Conclusion SSDI-Systems has supported the MoH in a short period of time to successfully address one of its key weaknesses, the collection and use of data for planning and management of district health services. The structured approach–first addressing deficiencies in data reporting completeness, followed by developing and rolling out planning and management tools for DHMTs, all of which are intrinsically linked to HMIS data availability–resulted in significantly improved district plans, performance reviews, service supervision and workforce management. Training in the use of the tool for staff at the DHMT level and above was effective in creating the required competencies at this level. It is uncertain, however, how this training momentum can be maintained in the future. However, as HMIS data quality is mostly determined at health-facility levels, the evaluators’ findings suggest that future support might direct training on data reporting and basic analysis toward this level. Recommendations  While data reporting completeness is high, quality of reported data, especially from health facilities, is still an issue. The QMU should be supported to develop and apply a routine data quality assurance tool. Management of this tool can be institutionalized in the ZHOs, linked to its DHMT supervision mandate.  To maintain the planning and data analysis skills at DHMT levels, the ZHOs should take a stronger role in continuous professional development of DHMT members.  Sufficient budget needs to be secured in approved annual district health budgets for DHMT compliance with mandatory requirements for MYP/DIP planning, e-ISS supervision and periodic performance reviews, or else sustainability of achievements is uncertain. Evaluation question 4 Of the capacity-strengthening approaches employed by SSDI-Systems (e.g., secondment, training and coaching of MoH and district staff, and operational and logistical support), which have most enabled the MoH to address central and district systems-level issues? Findings SSDI-Systems Performance Evaluation viii The mentoring approach by long-term advisors is considered by MoH staff as the most effective in the institutional development of the DPPD. Long-term advisors also very successfully mentored district accountants in financial management. The seconded experts to the NAC and National Malaria Control Program (NMCP) provided temporary relief for important capacity gaps. Respondents considered coaching by a long-term advisor, in conjunction with training, as more effective for building capacity than coaching through a short-term advisor. SSDI-Systems trained 1,404 MoH staff at the central, zone and district levels on technical and managerial topics such as management and leadership, DHIS, e-ISS, iHRIS and financial management. Staff interviewed felt that training had made them confident in the use of the tool they were trained on. They also considered management and leadership training as the most important training received. The high frequency of staff rotation is persistently viewed by DHMT members interviewed as the primary limiting factor to successful institutional development. Conclusion Mentoring by long-term advisors was very effective, as judged by its impact on the PDU’s institutional development from a dormant to a highly productive unit. However, with the exception of the two financial management advisors, mentoring was limited to the MoH level. Coaching and training by long￾term advisors proved successful for capacity development in the use of technical tools at the DHMT level. It was less successful than mentoring in initiating institutional behavior change, as judged by the observed low compliance of DHMTs with supervision scheduling guidelines. Short-term advisors were effective in developing the technology solutions (iHRIS, e-ISS), and while MoH staff opined that it would have been better for them to stay for a longer time, this would not have been cost-effective from the activity’s perspective. Recommendations  Mentoring support through long-term advisors should also be placed at ZHOs so that institutional development, for which mentoring proved the best approach, can complement capacity development or training initiatives for improved impact and sustainability of new technologies introduced (such as e-ISS).  Short-term advisors and CP should remain the same throughout the product development cycle to maintain MoH commitment and ownership of the process.  Training in the use of a tool should be followed up with coaching in a real setting, such as a planning workshop, to ensure that use of the tool becomes routine.  Assess if continuous long-term mentoring support to DHMTs in planning, financial management and supervision would be effective in improving their compliance with MoH guidelines. Evaluation question 5 What are the most significant accomplishments, best practices and lessons learned from the SSDI￾Systems activity? Explicitly identify and document the facilitating and inhibiting factors to positive performance for each of the above questions. Facilitating factors  The MoH was receptive to the introduction of innovative technologies and solutions. The DPPD was an excellent strategic entry point. “Before SSDI [Systems] there were a lot audit queries because there were a lot of missing document due to poor filling system and the vouchers were not accompanied by documents” –DHO SSDI-Systems Performance Evaluation ix  The long- and short-term advisors contracted were of high quality.  A very thorough and methodological approach was used for developing policies, guidelines and tools, accompanied by effective training and coaching. Inhibiting factors  Communication gaps between long-term advisors and ZHOs/DHMTs caused disjointed planning.  There was unclear delineation of responsibilities between SSDI-Services and SSDI-Systems at the district level.  Introduction of e-ISS and DHIS2 caused loss of perceived entitlements for staff to deliver monthly paper-based reports. Unreliable internet connectivity limits use of internet-based tools and frustrates staff. Accomplishments  The DPPD is sustainably strengthened as the institutional home for policy development, MYP/DIP planning, NHA, HMIS/DHIS, performance review and supervision.  High-quality tools and guidelines were developed to support the DPPD in these functions. These guidelines and tools were piloted, rolled out and are now used in SSDI districts; the MoH intends to scale up their use countrywide.  A well-planned and executed training plan ensured that sufficient staff were competent, despite frequent DHMT staff transfers.  All tools developed were perceived as practical, easy to use and supportive in design. Lessons learned  The long-term advisor mentoring approach proved successful in both capacity and institutional development, and in establishing and anchoring new processes and tools within the MoH. At the district level, SSDI-Systems support was limited to training and intermittent coaching, which was effective in developing technical capacity for use of tools but less so in institutionalizing their routine use. Therefore, future SSDI support may consider placement of long-term advisors in ZHOs to help speed up the management change process required for effective institutionalization of the management tools and processes introduced.  Continuity of engagement between short-term advisors and the MoH’s point of contact should be ensured for the entire process of developing an innovation.  The consistency of the evidence-based methodological approach (baseline assessments, product design and pilot, evaluation and scale-up) used by SSDI-Systems must be credited with the successful development, ownership and scale-up of innovations introduced, such as the e-ISS, the iHRIS, PBI, the MYP/DIP, District Health Stakeholder Forum and the improved performance review mechanisms. SSDI-Systems Performance Evaluation 1 I. ACTIVITY BACKGROUND ACTIVITY CONTEXT Support for Service Delivery Integration (SSDI) was USAID/Malawi’s flagship health activity and aimed to achieve the USAID/Malawi Country Development Cooperation Strategy strategic goal to improve the quality of life for Malawians through improving social development, increasing sustainable livelihoods and the exercising of citizen rights and responsibilities. It was designed as one broad health initiative implemented through three separate but thematically interrelated activities: SSDI-Systems, SSDI￾Services, and SSDI Communications. SSDI-Systems defined its mission as “to assist the Ministry of Health (MoH) to improve policies, management and leadership, and fiscal responsibility to advance Malawi’s health system and the sustainable impact of the Essential Health Package.” Abt Associates was contracted and implemented the SSDI-Systems activity between 2012 and 2016, focusing on six areas of results: 1. Increased and coordinated advocacy for and implementation of evidence-based policies 2. Strengthened strategic management and leadership capacity of the MoH 3. Improved and strengthened MoH zone supervision structures 4. Improved leadership and management of human resources for health 5. Improved decentralized management of district health services 6. Strengthened health financing mechanisms, fiscal planning and budget execution capability at national, zone and district levels for sustainability SSDI-Systems was designed at the same time the health sector wide approach (SWAp), the Health Sector Support Program (HSSP), was developed and appraised by the MoH and participating development partners. To align the performance of SSDI-Systems with the HSSP, the activity’s original performance monitoring plan (PMP) selected 56 indicators from the HSSP M&E framework to report on. Throughout its duration, SSDI-Systems received requests from the MoH for additional, unplanned and unbudgeted support. Many of the requests could be accommodated by the activity’s management, with prior approval of USAID, provided the request was within its overall scope of work. In 2012, the scope of SSDI-Systems work was widened to include long-term technical assistance to the National Malaria Control Program (NMCP) and the National AIDS Commission (NAC). The midterm performance evaluation (2014) perceived these multiple additional requests as potentially derailing the activity’s focus and therefore recommended that the activity should:  Prioritize completion and approval of the six policies underway and refrain from developing those in discussion or newly identified.  Where pilot activities are underway, focus on successful completion and documentation of best practices without further scale-up. Three other significant changes were made as a result of midterm performance evaluation recommendations. The Performance Management System and the activity’s budget were reduced, the list of 56 PMP indicators was reduced to 16 priority indicators, and the overall budget was reduced. EVALUATION PURPOSE, QUESTIONS AND AUDIENCE The purpose of this end-of-activity performance evaluation is to determine the effectiveness of the approach taken to improve the functionality of the Malawian health system to achieve equitable and sustainable health results in line with current health priorities. Its findings will inform the MoH, USAID, and other development partners in the design of future development cooperation activities. SSDI-Systems Performance Evaluation 2 The evaluation was conducted to respond to the following questions: 1. To what extent did SSDI-Systems enable institutionalization of health policy and financing functions, including policy analysis and development and National Health Accounts (NHA), in the MoH’s Department of Planning and Policy Development (DPPD)? 2. How successful has the integrated supportive supervision mobile tool been in improving supervision of health facilities, including follow-up of issues identified in supervision? 3. To what extent did SSDI-Systems improve utilization of data (including iHRIS, DHIS2 and supportive supervision data) for evidence-based planning and decision-making at central and district levels? 4. Of the capacity strengthening approaches employed by SSDI-Systems (e.g., secondment of staff to MoH/NAC positions, provision of technical assistance by external advisors, training and coaching of MoH and district staff, and operational and logistical support), which have most enabled the MoH to address central and district systems-level issues? 5. What are the most significant accomplishments, best practices and lessons learned from the SSDI-Systems activity? Explicitly identify and document the facilitating and inhibiting factors to positive performance for each of the above questions. A complete description of the evaluation statement of work can be found in Annex 1. DEVELOPMENT HYPOTHESIS The SSDI project development hypothesis postulates that “Programming health interventions through an integrated platform, consisting of activities in health policy, systems strengthening, support for integrated health service delivery, and social and behavioral change communication, will result in significant expansion of coverage, quality and utilization of priority HP services at community clinics, health centers and district hospitals.” To operationalize this development hypothesis, SSDI was designed and implemented through three separate yet interrelated sector activities, SSDI-Systems, SSDI-Services and SSDI-Communications. The SSDI-Systems mission was defined as “To assist the Ministry of Health to improve policies, management and leadership, and fiscal responsibility to advance Malawi’s health system and the sustainable impact of the Essential Health Package.” SSDI-Systems Performance Evaluation 3 II. EVALUATION DESIGN, METHODS AND LIMITATIONS The evaluation was conducted between July 25 and Sept. 30, 2016 and reviewed the period from SSDI￾Systems inception until June 2016. The core evaluation team consisted of three consultants: Dr. Frank Paulin, Team Leader; Mr. Themba Mhango, Health Systems Specialist; Mr. Willie Kachaka, Research Analyst. Field work was supported by two research assistants, Lucky Namisengo and Tiwonge Mwale, from a local research firm, Evidence for Change Solutions. The research assistants were contracted to support the recording and transcription of key informant interviews. Logistics support was provided by DevTech through a Lilongwe-based logistical consultant. The evaluation findings were presented to SSDI-Systems stakeholders in a findings workshop on Sept. 20, 2016. The final evaluation report was submitted to USAID/Malawi on Sept. 30, 2016. EVALUATION DESIGN After the evaluation methodology was developed, a team planning meeting was held at USAID and the design was agreed upon. Pre-testing and training of research assistants was conducted prior to start of data collection. After finalizing and piloting the key informant interview guide and the self-assessment questionnaire, data collection was done Aug. 3-28, 2016. A multistage sampling and mixed-methods approach was chosen to collect data. This non-experimental design does not account for the influence of non-SSDI-Systems interventions because no control districts were assessed. Baseline-endline comparisons could have been influenced by some unobserved confounding factors. There is also the possibility of recall bias because the evaluation assessed perceived changes over a five-year period. Multiple evaluations of SSDI sector activities conducted simultaneously may have contributed to respondent fatigue. DATA SOURCES The following three key data sources were used to address the evaluation objectives and questions: Documents:  Activity documents (see Annex 11)  Facility-based documents collected during the district-level data collection  Printed policies and guidelines developed with SSDI-Systems support People:  External stakeholders: USAID, Abt, development partners, non-governmental organizations (NGOs)  Internal stakeholders: MoH, ZHO, DHMT, NAC, NMCP Illustrative examples:  Samples of graphs and service information depicted at DHO blackboards Document review The team conducted a review of the project literature obtained from the Abt Associates and other documents identified by USAID/Malawi prior to arriving in the country. A qualitative analysis of these documents was conducted to identify themes and triangulate with other data collected and analyzed as part of this evaluation (Annex 11). SSDI-Systems Performance Evaluation 4 Secondary data analysis The evaluation used the 16 PMP indicators in the project’s results framework to compare progress to date across key service indicators. The evaluation selected key output and outcome indicators from each activity’s PMP and documented progress on each indicator. Indicator selection was based on: (1) relevance in responding to the evaluation questions; (2) relevance to the activity’s main focus area; and (3) ability to capture the value of SSDI-Systems’ investments over the past five years. In addition, the team tracked and reported on selected SSDI-Systems PMP indicators for the trend analysis. Primary data collection The evaluation team used a mixed methodology that comprised both qualitative and quantitative primary data collection:  Quantitative data to examine temporal changes (endline-baseline where available, self￾assessment questionnaires with key informants)  Qualitative data to provide more in-depth understanding of management change dynamics and perspectives of implementers Data collection instruments used included the following:  Semi-structured interviews were conducted with MoH staff at all levels, USAID/Malawi staff, and SSDI (Abt Associate) staff.  Key informant interviews were conducted with MoH staff at all levels, USAID/Malawi staff, SSDI staff, health providers, and decisionmakers and other key stakeholders at the central, zonal and district levels.  Self-assessment questionnaires were administered to all those that completed the key informant interviews. All data collection tools were pre-tested, revised and finalized prior to data collection. SAMPLING METHODOLOGY A purposive mixed-method approach was used to sample nine out of the fifteen SSDI-Systems intervention districts in the five health zones, namely Chitipa (North Zone), Lilongwe (CW Zone); Nkhotakota and Dowa (CE Zone); Nsanje, Mulanje and Balaka (South East). Figure 1 shows the SSDI target districts (in yellow) and the nine districts selected for the evaluation (in orange). The criteria for selection included geographic and population variation, accessibility to zonal offices and timing and type of SSDI-Systems interventions. In each district, the DHMT and staff from one health facility were interviewed. Selection of districts and facilities was finalized in consultation with USAID/Malawi and SSDI-Systems. FIELD IMPLEMENTATION PLAN The evaluation team interviewed national stakeholders, MoH officials, and staff of USAID, Abt Associates and development partner representatives. For field work the team split into two sub-teams, Figure 1. SSDI districts selected for data collection SSDI-Systems Performance Evaluation 5 one covering districts in the North, Central West and Central East zones and the other covering districts in the South East and South West zones. As competency in English language is very good among senior staff and mid-level health workers, the key informant interviews were conducted in English. Figure 1 shows the total numbers of interviews conducted at all levels. Location Semi-structured interview Key informant interview Self￾assessment Lilongwe (MoH, NAC, development partner, USAID, Abt) 19 Zone Offices 14 10 District Health Offices 31 28 Rural health facilities 24 24 Total 19 69 62 Figure 1. Total data collected by type of respondents and instrument used DATA ANALYSIS Quantitative data The self-assessment surveys from key informants were cleaned and analyzed with STATA 14.0 and Excel software, using indicator-specific computations. Frequencies, means and chi-squared tests for overall significance were conducted for all key indicators. The descriptive data were disaggregated by type of stakeholder. Secondary data sources from the PMP database were also analyzed using the same approach but using comparative analysis between baseline and endline data to assess changes in behavior as a result of the communication campaign. Qualitative data Data from semi-structured interviews and key informant interviews were digitally recorded and transcribed. The research analyst then coded each of the FGD transcripts. A preliminary thematic analysis was conducted on the transcribed notes based on the primary research questions and a review of a subset of transcripts. As new issues arose, the “auto-coding” option in DEDOOSE was used to revisit documents that had previously been coded but did not include the newly formed codes. This approach allowed for a certain level of inductive analysis, which is crucial in working with qualitative data, while simultaneously maintaining structures that would allow the analysis to be tailored to specific evaluation questions. The transcripts also included direct quotes by topic that can be used for illustrating trends. The qualitative analysis used two primary approaches: theme analysis and narrative analysis. Theme analysis organized data into categories by identifying recurring themes in the data and creating labels under different categories. Once themes/categories were identified, narrative analysis was used to examine the relationships between codes. LIMITATIONS AND CONSTRAINTS This sampling strategy is designed to capture only areas targeted by SSDI-Systems interventions, in line with the evaluation design. Therefore, all evaluation results presented in this report are statistically representative only of districts benefitting from SSDI-Systems interventions and cannot generate findings that represent the larger population of Malawi. SSDI-Systems Performance Evaluation 6 Secondly, as key informant interviews represent a significant portion of the data collected, potential biases related to personal opinions and recollection cannot be ruled out. The evaluation team attempted to mitigate these risks by using multiple interviewers and facilitators and by triangulating results across multiple data collection methodologies. Thirdly, this non-experimental design does not account for influence of non-SSDI-Systems interventions because no control districts were assessed. Baseline-endline comparisons could have been influenced by some unobserved confounding factors. There is also the possibility of recall bias as the evaluation assessed perceived changes over a five-year period. Conducting multiple SSDI assessments simultaneously may have contributed to respondent fatigue. Fourth, this evaluation only inquired into the effectiveness of the SSDI-Systems approach and not into the combined effectiveness of the three SSDI sector activities. SSDI-Systems Performance Evaluation 7 III. FINDINGS AND RECOMMENDATIONS EVALUATION QUESTION 1 To what extent did SSDI-Systems enable institutionalization of health policy and financing functions, including policy analysis and development and NHA, in the MoH DPPD? Findings The PDU within the DPPD was non-existent five years ago. As indicated in the RFA, the MoH was challenged with limited capacity for policy development at the central level, specifically: Key DPPD positions lacked skills or remained vacant for long periods, and the department had limited mandate in policy development and analysis. With SSDI-Systems’ support, the DPPD successfully advocated to the MoH for creation of the PDU, which now has three staff positions and is headed by a deputy director￾level official. The PDU’s approved terms of reference mandate it as the overall lead in policy development for the MoH. Before the PDU was established, each department or unit within the MoH initiated and developed policies independently and primarily in an uncoordinated and unstructured way, leading to substandard documents, as reported during a key informant interview. Based on these observations, right from its inception, SSDI-Systems supported the PDU in the development of the “Policy Development Framework for the Public Health Sector,” adopted from the OPC set of general guidelines on policy development and analysis. This framework has since been applied to all policies developed. By 2014, development of PDU capacity through mentoring by SSDI-Systems long-term advisors and the return of Malawi professionals who had studied abroad created the momentum of the PDU that today qualifies it as the steward of policy development in the MoH. This does not mean that all policies are developed entirely by the PDU; the technical departments still are responsible for technical content. The PDU ensures that due process is followed by all concerned and that the steps as laid out in the guidelines are followed. The sincerity of the PDU’s commitment is reflected in the fact that for the past three years, a remarkable 100 percent of all planned quarterly meetings indeed took place. The PDU has also rationalized the various policy development initiatives that were underway or newly requested. The unit successfully advocated for a number of policies to be downgraded to guidelines and strategies, thus curbing the proliferation of policies observed before 2012 and strengthening departments’ and units’ focus on implementation. So far, with the project’s technical and financial support, the PDU has coordinated the following policy documents, which are at different stages:  National Medicines Policy–Stage 5  National Health Policy–Stage 3  Medical Equipment Management Policy–Stage 3  Guidelines for Policy Development and Analysis in the Public Health Sector in Malawi–Stage 5  Guidelines for the Management of Task Shifting to Health Surveillance Assistants in Malawi– Stage 5  Guidelines for Management of Community Health Volunteers in Malawi–Stage 5  National Malaria Policy–Stage 1 By 2016, four of the seven policy documents, according to the activity’s PMP, were completed as policies or downgraded as guidelines (Guidelines for Policy Development and Analysis in the Public “Whenever some NGOs came to Malawi they would just involve the health surveillance assistants….. withdrawing them from communities, training them, giving them various aspects of work which were over and above their normal job description so when the assessment was done it was found that there is a need for a policy guidance” –MoH official SSDI-Systems Performance Evaluation 8 Health Sector; Guidelines for the Management Task Shifting to Health Surveillance Assistants in Malawi; Guidelines for the Management of Community Health Volunteers; National Medicines Policy). Two policy documents (National Health Policy; National Health Physical Assets Management Policy) await official endorsement by the MoH and the Cabinet, which is outside the influence of the activity. The National Malaria Policy is put on hold by the MoH. All these policies are developed. Figure 2. Progress on key policies, strategies and guidelines finalization Key: Stage 1: Situation analysis and baseline setting (10 percent complete); Stage 2: Stakeholder consensus building (30 percent complete); Stage 3: Formulation and policy drafting (50 percent complete); Stage 4: Official endorsement (60 percent complete); Stage 5: Operationalization (80 percent compete); and Stage 6: Impacts/Outcomes assessments (100 percent complete) It is fair to conclude that the improvements of the health policy development and analysis in Malawi, as acknowledged by senior MoH officials during key informant interviews, largely are attributable to the SSDI-Systems’ support to the PDU. According to key informant interviews, the SWAp unit was already coordinating all financing issues, and there was limited capacity in health financing in both numbers and skills in the DPPD. Due to the fact that a dedicated health financing unit does not exist, SSDI-Systems supported the formation of the Health Financing Task Force within the Finance and Procurement Technical Working Group, developed its terms of reference and defined its roles and responsibilities in leading the development of the Health Financing Strategy. In collaboration with the Clinton Health Access Initiative, the project supported development of the Health Financing Strategy Concept Note. The Task Force was also responsible for overseeing the piloting of performance based incentives (PBI) and results-based financing and other health financing activities. PBI activities are currently piloted by SSDI-Services in three districts (Chitipa, Nkhotakota and Mangochi). An evaluation is planned but not yet concluded. None of the health financing measures have yet been evaluated or reached maturity to scale up nationally. Throughout this process, a laudably inclusive approach was maintained. MoH directors, senior managers, development partners and other stakeholders were all included in the process. Name of Document S1 S2 S3 S4 S5 S6 National Medicines Policy National Health Policy Medical Equipment Management Policy Guidelines for Management of Task Shifting to Health Surveillance Assistants in Malawi Guidelines for Policy Development & Analysis in Public Health Sector Guidelines for Management of Community Health Volunteers National Malaria Policy SSDI-Systems Performance Evaluation 9 Various development partners support various health financing initiatives in Malawi. Therefore, most measures cannot be attributed to a single actor. After SWAp disbandment in 2013, since the health financing strategy development was also seen as health financing policy development, the MoH felt that it was better to place financing policy/strategy development in the PDU. To date, the PDU continues to successfully coordinate all health financing functions with key partners supporting health financing reforms (World Bank, GIZ, P4H, Clinton Health Access Initiative, USAID, SSDI-Systems, etc.) and all major stakeholders. As such, SSDI-Systems regards this as a major achievement in health financing institutionalization in the MoH. Furthermore, the establishment of the NHA desk, the basis for institutional sustainability, can be attributed solely to SSDI-Systems advocacy and technical support. The project managed to successfully support two rounds of NHA, covering fiscal years 2009/10-2011/12 and 2012/13-2014/15 in collaboration with WHO. The expenditure-tracking studies of district health expenditures can also be attributed to SSDI-Systems. In addition, the activity supported four other priority health reform options: establishment of a health fund, health insurance, reviewing the CHAM–MoH memorandum of understanding, and central hospital reforms. Analysis of the key informant interview and self-assessment respondent findings indicate that SSDI￾Systems’ contributions are known and valued. Staff at all levels value the impact of SSDI-Systems’ support to health policy and planning (66.7 percent MoH, 84 percent ZHO and 85.7 percent of DHO) more than its support to health financing and budgeting (50 percent of MoH, 53 percent ZHO and 71.4 percent of DHO). Conclusions SSDI-Systems has sustainably institutionalized capacity for key health policy and financing functions in the DPPD through its advocacy and support toward establishing the PDU and Health Financing Task Force. With guidelines for policy development and an analysis framework for the public sector in place, the PDU has been widely accepted as a leading unit for developing policy, strategy and guideline documents. As for the health financing function, to date, the PDU continues to successfully coordinate all health financing functions with key partners supporting health financing reforms. However, while a number of health financing reform options were successfully supported by SSDI￾Systems at a technical level, on the institutional development level, only the NHA desk can be judged as successful institutional development in health financing. Recommendations 1. Continue strategic support to the PDU to sustain the institutional capacity gains, because not all key positions are filled, for example, the position of deputy director-level official became vacant the during final year of the project. 2. Advocate for the establishment of a dedicated health financing unit within the DPPD responsible for coordinating all health financing functions, which require specific set of expertise. All health financing reform option initiatives should be migrated to this unit within one year. SSDI-Systems Performance Evaluation 10 3. Repeat successful PDU capacity and institutional development approach for the health financing unit, with its terms of reference and well-qualified staff, to give it mandate and governance structures for all health financing functions. EVALUATION QUESTION 2 How successful has the integrated supportive supervision mobile tool been in improving supervision of health facilities, including follow-up of issues identified in supervision? Findings The 2012-2016 Health Sector Strategic Plan (HSSP) and the RFA both highlight the lack of effective management tools to improve zonal and DHMT supervision of health care workers to strengthen quality of service as one of the key challenges faced by the Malawi health system. Through the provision of dedicated technical assistance between 2012 and 2015, SSDI-Systems supported the MoH, through CMED, to develop and roll out an innovative integrated supervision tool to all SSDI districts. By 2016, the majority of health facilities in the 15 SSDI-districts received regular supervision, recommendations and follow-up, and the MoH intends to scale up the tool as a national program. Optimization of the supervision process and resource use is still sub-optimal and ongoing. The evaluation’s inquiry into the effectiveness of the e-ISS tool extended to the health facility level, knowing that SSDI-Services and not SSDI-Systems provides implementation support to this level. The rational for this decision was that in order to derive recommendations with respect to changes in service quality (the overall purpose of SSDI), impact and bottlenecks in relation to this level needed to be ascertained. Reliable pre-2012 data on DHMTs’ capacity, scope and regularity of supervision do not exist. Therefore, the evaluation team relied on anecdotal evidence collected during key informant interviews and self￾assessments. Evidence thus collected suggests that the old, paper-based supervision methodology was mostly sporadic in nature and perceived as inspection or policing rather than supportive. Feedback recommendations arrived late, if at all, and follow-up on recommendations was rare. Exceptions were supervision of externally funded programs, such as HIV, TB and malaria development partner- and NGO-funded supervision. By 2014, SSDI-Systems had fielded intermittent short-term consultants who, together with MoH experts, developed the e-ISS tool. This was facilitated by the taskforce established by the Planning and Policy Development directorate. During this period, the electronic program and two checklist versions for the e-ISS tool, one for district-level facilities and one for central hospitals, were developed. After a pilot phase in three districts and a pre-test in two central hospitals, the checklist for districts was finalized, and in 2014 the tool was rolled out to the 15 SSDI districts. The checklist for tertiary care hospitals remains to be finalized, pending agreements on checklist inputs from various medical specialty units. A total of 135 smartphones were distributed to the 15 SSDI districts, and 448 staff (MoH 46, ZHO 54, DHMT 348) received training on the use of the e-ISS tool. In addition, a few staff have been trained as technical support focal points or “trouble shooters.” This high level of training has effectively offset the impact of ongoing trained staff attrition through transfers and retirement. Indeed, of all ZHO and DHMT level supervisors interviewed who had received e-ISS training, more than half felt confident in the use of the tool (see Figure 8). Throughout the checklist revision process, CMED maintained an inclusive approach so that the tool is now owned and used by most directorates and SSDI-Systems Performance Evaluation 11 departments in the MoH, such as Clinical, Preventive, Nursing, Reproductive Health, Preventive Health and other technical departments. Only the HIV, Malaria and TB programs maintain a separate supervision structure with funding support from development partners and implementing NGOs. Until recently, CMED was the institutional home for the e-ISS tool, but responsibility for e-ISS has recently been shifted to a newly established Quality Management Unit (QMU), also under the DPPD. This shift reflects the DPPD’s current focus on data quality, away from the previous focus on data quantity or reporting (reporting regularity of health facilities has improved from 40 percent to 80 percent between 2012 and 1016). The e-ISS tool pilot showed that a minimum of two and maximum of four supervisors are adequate for a health center, while a minimum of four and maximum of six is adequate for district hospital supervision. The e-ISS tool has helped DHMTs to divide their eight-member teams into two teams of four, leaving one team to attend to other office commitments. This arrangement has increased the number of facilities visited by DHMTs, but it is not yet uniformly implemented across all districts. Many DHMT members interviewed still report deployment of supervision teams of four to eight staff. The evaluators were also told during key informant interviews that frequently parallel supervision visits at the health facility level are initiated by NGOs and community-based organizations active in the district, not necessarily with consent of the DHO. Figure 3. Percent of respondents who felt that e-ISS had eliminated parallel supervision Planning for and scheduling of external quarterly supervision visits of health facilities are not completely harmonized across the districts. While some districts conclude their (external) supervision visits in a short period of time at the beginning of each quarter, other DHMTs spread their supervision visits over the three months. In health facilities visited, the evaluation team found little evidence for a structured internal supervision follow-up of recommendations as envisaged by the e-ISS guidelines. All DHOs visited had monthly supervision schedules for health facilities prominently displayed on notice boards. Detailed information on supervision completion by DHMTs is not collected routinely; therefore, the evaluation measured supervision regularity as perceived by staff at the three levels (Figure 4). It is interesting to note that while only 33 percent of health staff feel that they are regularly supervised, more than double of DHMT and ZHO staff feel that way. 38 57 17 00 10 20 30 40 50 60 Zonal level staff District staff Health Facility Staff (Percent) SSDI-Systems Performance Evaluation 12 Figure 4. Improved regularity of supervision During supervision, checklist-based observations are entered live into the smartphone. Smartphones are linked to indicators, built into the e-ISS checklist, which have been developed from HMIS and program￾based indicators against national standards of performance. Supervision observations entered are immediately assessed against these indicators and targets through an automated analysis program. This enables the supervisor and staff to visualize and discuss findings at the end of the supervision. Conclusions are shown as traffic lights: red, yellow and green. This feedback system seems to have elevated the supervision act from ritualistic to productive. As mentioned above, supervision with the use of e-ISS is not perceived as policing anymore, and indeed (see Figure 7) the majority of staff feels encouraged to take actions based on supervision feedback they received. Figure 5. Staff encouraged to act on supervision recommendations Another positive outcome is the fact that the e-ISS tool has enabled users to identify new issues (see Figure 6). Issues identified related to areas such as quality of reporting, but also drug management, hygiene and cleanliness deficits. About 40 percent of recommendations at the health facility level were reported by key informant interviews as manageable without extra resources. 85 76 33 00 10 20 30 40 50 60 70 80 90 Zonal level staff District staff Health Facility Staff (Percent) 85 81 50 00 10 20 30 40 50 60 70 80 90 Zonal level staff District staff Health Facility Staff (Percent) SSDI-Systems Performance Evaluation 13 Figure 6. Staff reporting identification of new issues through e-ISS The move from a paper-based, inspection-type supervision approach to this electronic supportive approach has also improved job satisfaction (Figure 7). CHAM facilities were reported, by three DHOs interviewed, to be opposed to the e-ISS tool at first. After a test run in their facility, they quickly became convinced of its utility and approved of its regular use. Figure 7. Staff reporting increased job satisfaction through e-ISS Figure 8. Training improved quality of service 85 71 39 00 10 20 30 40 50 60 70 80 90 Zonal level staff District staff Health Facility Staff (Percent) 38 62 56 00 10 20 30 40 50 60 70 Zonal level staff District staff Health Facility Staff (Percent) 69 69 23 31 57 52 40 38 0 10 20 30 40 50 60 70 80 L&M e-SST Tool iHRIS DHIS2 (Percent) Zonal level staff District staff SSDI-Systems Performance Evaluation 14 Insufficient internet connectivity, lack of fuel, competing priorities and lack of incentives were mentioned by DHMT staff as main reasons why supervision schedules are not followed more rigorously. SSDI￾Systems has supported the MoH to lobby with the CDC (which funds the Baobab electronic medical records system) to allow Baobab to connect iHRIS to its network in all 28 districts. This work is currently in progress–Baobab IT experts are visiting all districts making the connections. Conclusions There is a general and widespread acceptance and even enthusiasm among staff at all levels about the e￾ISS tool. However, this enthusiasm is a bit suffocated by unstable internet connectivity (airtime) and unfavorable incentive structures; both are real issues that need to be addressed. The move from the paper-based to the smartphone-based supervision has revolutionized supervision practice and acceptance. It is now perceived as action-oriented and evidence-based through its linkage to HMIS targets. Recommendations are now regularly used by the supervised staff but also by the DHMTs in planning and review processes. CMED was a good process-champion for the development and roll-out of the tool; SSDI-Systems was an excellent agent for the technology development. The technical working group provided effective oversight and steered the development process. The establishment of the QMU as the dedicated unit to lead on further refinement and expansion of the e-ISS tool to non-SSDI districts further institutionalizes the tool as a core health service quality management tool of the MoH. Regularity of both external and internal supervision remains an issue and is currently insufficiently monitored and enforced. The upcoming project should strengthen the harmonization of external supervision and reinforce internal supervision, as described by the e-ISS guidelines. The internal supervision is and should be done by facility management at two- or four-week intervals to follow up on the implementation of corrective actions recommended. In hindsight, it may have been better to combine zonal and district support into one SSDI sector activity (i.e., SSDI-Services) component and limit the health systems support to the central MoH level. This way, developing supervision capacity might have been more seamless from the ZHO to health facility levels. Sufficient smartphones to operate the system were provided, and sufficient capacity exists at all levels, despite frequent staff rotations in and out of SSDI districts. The evaluation team recommends considering the use of tablets instead of smartphones. Tablet use can be restricted to targeted applications, thus limiting the airtime required to operate the tool. Finally the issues of unfiltered email supervision upstream to directors and mid-level managers’ alerts need to be addressed. A filter should be developed that better targets the appropriate audience to act on the supervision alerts they receive. Recommendations 1. Institutionalizing and developing capacity of the QMU should be supported to further strengthen the use of the e-ISS tool as a service quality improvement tool. 2. The recommendations of the e-ISS guideline on scheduling and resourcing of external and internal supervision should be reconfirmed through a consensus workshop with all ZHO and DHMT stakeholders to pave the way for better harmonized and effective implementation. “After supervision we do management meeting to follow up on the issues raised and give them feedback on our findings and apart from that we have quarterly meetings with the districts and we discuss some of the issues noted in the districts. And when we go back to the districts to make follow ups on the issues that we discussed.”–Zonal officer SSDI-Systems Performance Evaluation 15 3. It is recommended to link the e-ISS tool with the DHIS2 platform so that its utility as a “real￾time” management tool is further enhanced. EVALUATION QUESTION 3 To what extent did SSDI-Systems improve utilization of data (including iHRIS, DHIS2 and supportive supervision data) for evidence-based planning and decision-making at central and district levels? Findings The HSSP and RFA identified the weak data basis for policy development and service planning as one of the main weaknesses of Malawi’s health systems. SSDI-Systems’ support enabled the MoH to improve its planning and performance review mechanisms through the development, improvement and institutionalization of guidelines and data management tools. SSDI-Systems also provided capacity development on these tools through training and coaching. Implementation of the tools at the district level was managed primarily by SSDI-Services, the effectiveness of which is not part of this evaluation. However, as the team was also requested to visit and interview health facility service providers, some of the key informant interview findings at this level are included in this report. At the central level, the project supported both CMED and the Human Resource Management Department to develop and finalize two key documents, the HMIS and human resources for health strategic plans, which now serve as frameworks for HMIS and human resources for health strengthening efforts across the health sector. Completeness of service reporting by districts was only 40 percent in 2014. SSDI-Systems supported CMED in revision of the reporting formats and trained 322 staff (217 DHMT, 15 ZHO and 90 central hospital) in data management and on the use of the revised HMIS tools and registers; this contributed to improved completeness of reporting by SSDI districts to CMED. Staff confidence in the improved HMIS system is reflected in the findings (Figure 9) that 100 percent of MoH, 77 percent of ZHO and 86 percent of DHMT staff interviewed felt that their institution now routinely uses HMIS data for planning. Figure 9. Routine use of HMIS/DHIS data for program planning SSDI-Systems also supported the migration of paper-based HMIS to the DHIS platform for two central hospitals, while SSDI-Services supported the migration of HMIS data at the district level. SSDI-Systems also provided the MoH and DHMT with support to improve the effectiveness of MYP/DIP planning processes through the development and disseminations of the MYP/DIP Guidelines. These link 100 85 81 00 20 40 60 80 100 120 MOH Zonal level staff District staff (Percent) SSDI-Systems Performance Evaluation 16 the planning process to HMIS performance and national targets, thus improving the plans’ allocative and technical efficiency. SSDI-Systems facilitated training, workshops and coaching to strengthen the capacity of staff in the application of these guidelines. During the field visits to the health institutions, 70 percent of ZHO and 71 percent of DHMT staff indicated that they had participated in MYP/DIP development. SSDI-Systems also established the District Health Stakeholder Forum—a forum for integrating NGOs and community-based organizations that provide health services in the districts into the district planning process. This forum has created a platform to share activity plans and budgets and agree to a common program of work—a district SWAp. DHOs interviewed agree that this forum has helped to prevent duplication of efforts and helped DHMTs to better allocate their scarce resources. By 2016, all of the SSDI districts had established such a forum, which is headed by the district commissioner, with the DHO as the secretariat. This forum appears to be rapidly emerging as the supreme decision-making body for planning and financing of district health activities, very much in the spirit of the Malawi Local Government Act on decentralization. SSDI-Systems also developed, in collaboration with the Human Resource Management Department, the internet-based iHRIS platform, which has been rolled out and is functional in all 15 SSDI districts, the five ZHOs and at the Human Resource Management Department. In all the districts that the evaluation team visited, including MoH headquarters, human resource officers demonstrated both practical and analytical skills in using the iHRIS module. They were able to produce up-to-date retirement-planning reports, vacancy analysis and current staffing levels. These reports are highly appreciated by DHMT members for their forward planning use, for example for deployment within the districts and/or lobbying for additional staff from MoH headquarters. Most DHOs were observed to post graphs on the notice board illustrating current establishment and forthcoming retirements. The scope of iHRIS is currently being expanded to include the training module, which will enable districts to better target and allocate training and other capacity-development measures. SSDI-Systems also supported CMED in improving the effectiveness of performance reviews by including HMIS, e-ISS and iHRIS data in the process. Sixty-nine percent of ZHO and 90 percent of DHMT staff interviewed had participated in either a quarterly or annual performance review, and the majority (54 percent of ZHO and 60 percent of DHMT staff) perceived that quality of annual and quarterly performance reviews had improved (Figure 10). In terms of whether the project has strengthened the process and quality of district budget development and execution, 53.8 percent of ZHO and 75 percent of DHMT staff perceived that district budget development and execution has improved. “Training and coaching on the use of multi-year plan and district Implementation plan guidelines resulted in greatly improved district plans and ultimately into the national plan and budget” –MoH official SSDI-Systems Performance Evaluation 17 Figure 10. Progress on quarterly and annual performance reviews Note: Figures for health facilities are included, although SSDI-Systems capacity-development engagement was limited to the DHMT level and above. However, it illustrates that there is a need for further capacity development at this level. The success of SSDI-Systems in improving use of data for decision-making at the district level can be summed up by the findings in Figure 11. Also, as a result of technical support, training and coaching, 77 percent of ZHO, 86 percent of DHMT and 17 percent of health facility staff perceive that their institution is now confident in the use iHRIS, DHIS and e-ISS for review and planning. Figure 11. Levels of confidence in using iHRIS data in planning and review Conclusions Evidence-based planning both at the central MoH, but most importantly at the DHMT level, has improved significantly from 2012 to 2016. The improved availability of up-to-date HMIS data and its linkage to management tools, especially supervision (e-ISS), planning (MYP/DIP) and performance reviews, have revolutionized the entire district management landscape; dialogue among stakeholder forum members is now based on facts rather than “guesstimates.” There was a consensus among staff interviewed that with the increased use of data, district budget preparation and execution is also better prioritized and effective. District planning, supervision and review meetings are increasingly supported by stakeholder forum members. Health facility staff scored low in most assessments on data use. This reflects the fact that SSDI-Systems mostly worked at the levels of the DHMT and above, where data is managed electronically. Data recording and use at the health facility level is still paper-based, with a few exceptions where pilot Progress on Quarterly and Annual Performance reviews ZHO DHMT HF % use DHIS2 data in MYPP/DIP preparation when setting performance standards against targets. 62% 86% 28% % perceived that annual and quarterly performance reviews has improved. 54 % 60% 22% % have participated in MYP/DIP planning 70% 71% 6% % have participated in District Quarterly and Annual Reviews of DIP 69% 90% 28% 83 62 76 00 10 20 30 40 50 60 70 80 90 MOH Zonal level staff District staff (Percent) SSDI-Systems Performance Evaluation 18 initiatives are implemented. However, the team felt that even at the facility level there is a growing recognition of the importance of data and proper reporting. Future support may strengthen health facility data management and analysis at health facility levels to complement capacity development at DHMT levels. Human resource management officers at all levels are enthusiastic about iHRIS. While its utility is significant, it still is too narrowly restricted to administrative human resource management processes. The evaluation team was informed that work was underway to include the training module in the iHRIS package. Timely training for newly transferred human resource officers (common civil service pool) on iHRIS remains a challenge. Internet connectivity is affecting the functionality of DHIS2 and iHRIS, but SSDI-Systems has supported the MoH to lobby with the CDC (which funds Baobab electronic medical records system) to allow Baobab to connect iHRIS to its network in all 28 districts. This work is currently in progress—Baobab IT experts are visiting all districts making the connections. Recommendations  While data reporting completeness is high, quality of reported data, especially from health facility, is still an issue. The QMU should be supported to develop and apply a routine data quality assurance tool. Management of this tool can be institutionalized in the ZHOs and linked to their DHMT supervision mandate.  To maintain DHMTs’ planning and data analysis skills, the ZHOs should take a stronger role in continuous professional development of DHMT members.  Sufficient budget needs to be secured in approved annual district health budgets for DHMT compliance with mandatory requirements for MYP/DIP planning, e-ISS supervision and periodic performance reviews. EVALUATION QUESTION 4 Of the capacity-strengthening approaches employed by SSDI-Systems (e.g., secondment of staff to MoH/NAC positions, provision of technical assistance by external advisors, training and coaching of MoH and district staff, and operational and logistical support), which have most enabled the MoH to address central and district systems-level issues? FINDINGS SSDI-Systems used the following approaches to institutional capacity development at central, zone and district levels: secondment, mentoring, coaching and training. Their definitions are detailed in the glossary. At the central MoH level, two full-term advisers were seconded to the NAC and NMCP where they worked as de facto integrated experts rather than as technical advisers. While their support to the NAC and NMCP delivered on important tasks that helped the MoH to overcome internal capacity bottlenecks, such as complying with the new Global Fund funding mechanism, this technical assistance modality provided for only limited sustainable capacity development. A more sustainable and effective mentoring approach, highly appreciated by the MoH, was applied for developing DPPD capacity through part-time placement of a long-term senior advisor in the PDU. The presence of continuous technical mentoring support at the workplace for 2-4 days per week was considered very helpful. This way routine and non-routine issues could be jointly discussed, analyzed, “He represented MoH at CCM meetings” –NAC official SSDI-Systems Performance Evaluation 19 solutions developed and applied without the advisor “taking over”. Similarly the continuity of mentoring by resident Abt technical staff to other units and departments such as CMED, Human Resource Management Department and CSD was perceived as effective. The mentoring approach used by zone-based long-term advisers to help improve financial management at district level was perceived as successful by accounts officers but less so by other DHMT staff. Indeed, 80 percent of district account offices mentored in financial management reported an increase in their assessed financial management capacity score. The following areas showed the highest improvements: (i) reduction of audit observations, (ii) improved fleet and fuel management, (ii) record keeping. However, as mentioned above, some DHOs felt that financial management mentoring went beyond its mandate and at times acted as auditors (travel expenditure validity observations). Figure 12. Record-keeping before and after financial management coaching In addition to mentoring MoH departments, long-term technical staff based in the contractor’s Lilongwe office provided coaching support to districts. This support was usually provided through active participation in training, planning and review workshops. Overall, 61.5 percent of ZHO, 60 percent of DHMT and 77.8 percent of health facility staff appreciated this approach (see Figure 13). DHMT members mentioned that because of late notification of upcoming coaching visits, staff could not always fully participate, as their time was already booked by “competing priorities.” ZHOs felt sidelined in the planning of such coaching activities, being notified rather than included in the planning and scheduling process of coaching visits. Health facility staff who had the chance to participate in such coaching activities valued the experience highly. SSDI-Systems Performance Evaluation 20 Figure 13. Perceived effectiveness of coaching by long-term advisors Coaching by short-term advisors, engaged to provide ongoing technical expertise for the development and roll-out of innovative management tools (iHRIS, e-ISS, DHIS2), was regarded as slightly less effective than coaching by long-term advisors (Figures 13 and14). Key informant interview findings from the MoH level indicated that this model was by far less appreciated then the coaching from long-term advisors. Especially when short-term expert’s visits were spaced six or more months apart, this was even felt as disrupting rather than supporting the continuity of their internal capacity-development process. The main argument was that MoH staff that had previously worked with the advisor on the development of a tool was transferred or given new assignments by the time the adviser returned. Therefore, the continuity and ownership of the tool-development process was perceived as compromised. Figure 14. Perceived effectiveness of coaching by short-term advisors With the Local Government Act of 1998, the Government of Malawi has effectively decentralized the delivery of health services to the district assemblies. With this act, the role of the MoH gradually shifts from being a service provider to becoming a steward of the health sector. The SSDI-Systems activity supported this paradigm shift with a very effective leadership and management training for senior and mid-level managers at the MoH, ZHO and DHMT levels. By June 2016, the management and leadership program has successfully graduated 135 managers (37 from central level, 13 from zones and 85 DHMT). Senior MoH officials and staff from ZHOs and DHMTs were all very appreciative of the management and leadership program and consistently rated it as most relevant and effective training they had received. The most common reason for this rating given by DHMT staff during interviews was that the training enabled participants to understand the complexity, interrelationship and importance of different management tools, such as reporting, supervision, planning, budgeting and performance management. 61.5 60.0 77.8 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 Zonal level staff District staff Health Facility Staff (Percent) 53.9 57.9 77.8 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 Zonal level staff District staff Health Facility Staff (Percent) SSDI-Systems Performance Evaluation 21 The training’s duration of up to six months was perceived with mixed feelings by DHMT members; most felt this approach was good but too long. Participants mentioned that while the topics of the management and leadership training were not new to them, they nevertheless appreciated it very much, especially as it effectively addressed new health architecture and technology change in Malawi. Figure 15. Staff interviewed that had attended training Training was numerically the predominant capacity-development approach and seamlessly accompanied all guideline- and tool-development processes. During the first four years, a massive 1,404 staff from MoH, ZHO and DSHMT levels received training in management and leadership, e-ISS, DHIS, iHRIS, and financial management. Indeed at the time of interview, nearly half of the DHMT members interviewed had received training in e-ISS, management and leadership and about one third in iHRIS and DHIS. Therefore, it is fair to conclude that all required data management capacities are available for DHMTs. Rating of the trainings received was uniformly high; of those interviewed, 80 percent of DHMT staff and 85 percent of ZHO staff considered training the best capacity-development approach (Figure 16). Figure 16. Effectiveness of training as best capacity development The majority of staff perceived that training had improved the quality of their service (Figure 17). It is indeed commendable that SSDI-Systems managed, through a well-designed training plan, to maintain a pool of trained and competent DHMT staff in all the 15 districts despite high staff transfers. In fact, 73 percent of DHMT and 46 percent of ZHO staff considered high staff turnover to limit sustainable capacity development. 33 17 17 33 69 69 23 31 57 52 40 38 17 6 6 22 0 10 20 30 40 50 60 70 80 L&M e-SST Tool iHRIS DHIS2 (Percent) MOH Zonal level staff District staff Health Facility Staff 85 80 77 78 79 80 81 82 83 84 85 Zonal level staff District staff (Percent) SSDI-Systems Performance Evaluation 22 Figure 17. Perceptions of effectiveness of training in improving services Human resources officers and accounts staff are from the common civil service pool and are rotated across line ministries. Newly appointed accounts officers at the DHMT level reported that improvements in financial management, such as filing, fleet and fuel management and pre-audit preparations, can easily be maintained without extra training. The situation for human resource officers differs. iHRIS is an internet-based tool not used outside the MoH, and it is neither possible nor cost￾effective for SSDI-Systems to offer a new training each time a new human resource officer is transferred into one of the SSDI districts. However, data entry clerks at district level have also been trained in iHRIS, and they help incoming human resource officers to get confident in the use of iHRIS. CONCLUSIONS The mentoring approach through long-term advisors was extremely effective for capacity and institutional development of the PDU and for financial management at DHMT levels. The continuity, the personal touch and the professional qualification of the advisor appear to have been key to the successful institutional development of the PDU as the institutional home for policy analysis and development for all MoH departments. This can be substantiated by the fact that for the last three years, no policy was developed without PDU’s consent, support and leadership. As mentoring requires continuous physical proximity between the mentor and the mentored, this approach could not be extended to zonal or district offices for cost reasons. Even the successful long￾term advisors on financial management had to be withdrawn due to budget reasons. Long-term technical advisors based in Lilongwe, however, supported the districts through intermittent coaching, which proved very effective in enforcing capacity developed through specific training on, for example, MYP/DIP guidelines, use of HMIS data for performance reviews or on the use of e-ISS tool. However, judging from responses received during key informant interviews at the district level, coaching was not as effective in initiating change at the institutional level as it was at the individual level. This can be validated, for example, by the observation that while e-ISS skills are well developed across DHMT members, institutionalized regularity of supervision at this level is limping. The short-term advisor selected to develop technology solutions and tools such as iHRIS and e-ISS through coaching were of high quality, if judged against the wide acceptance and successful roll-out of the tools developed. However, lack of a continuous working relationship between the short-term advisor and MoH point of contact (due to frequent reassignment of CP staff) is perceived by the MoH as negative. In fact, loss of ownership over the development process was mentioned by a MoH staff member in that context. However, this dilemma is difficult to resolve for SSDI and needs, first of all, commitment by the MoH to not transfer staff during the entire development process of a product. 69.2 100.0 50.0 0.0 20.0 40.0 60.0 80.0 100.0 120.0 Zonal level staff District staff Health Facility Staff (Percent) SSDI-Systems Performance Evaluation 23 Short-term advisors for highly technical work, such as DHIS, e-ISS and iHRIS, are expensive and globally in short supply. Long-term deployment for this type of work is therefore not feasible or cost-effective. Another aspect that distinguishes SSDI-Systems’ coaching from its mentoring approach is that the former was reported by staff as excellent in enforcing the application of technical skills acquired during training (such as use of data during DIP, supervision and performance reviews) while the mentoring approach was credited with the development of sustainable institutional capacity through change management, especially if coupled with management and leadership training. Training was very effective, as judged by the perceived and demonstrated skills and competencies of staff interviewed at all levels in the use of tools and guidelines developed with support of SSDI-Systems. The impact of skilled staff attrition caused by frequent staff transfers was cushioned through frequent repeat trainings throughout the project duration. Management and leadership training stands out as it gave participants a holistic overview on how the various management tools developed under SSDI-Systems interact and complement each other to improve health services. The seconded long-term advisers helped the NMCP and the NAC especially to comply with the new funding model of the Global Fund. While the capacity-development impact of this measure appears to be limited, its support to the MoH in its fight against HIV and malaria is commendable. Sufficient logistics and operations support for mentoring, coaching and training operations was provided by SSDI-Systems, the MoH and other development partners. RECOMMENDATIONS 1. Mentoring support through long-term advisors should also be placed at ZHOs so that institutional development, for which mentoring proved the best approach, can complement capacity development or training initiatives for improved impact and sustainability of new technologies introduced (such as e-ISS). 2. The short-term advisor and CP should remain the same throughout the product development cycle so that MoH commitment and ownership of the process is maintained. 3. Training in the use of a tool should be followed by coaching in a real setting, such as a planning workshop, to ensure that use of the tool becomes routine. 4. Assess if continuous long-term mentoring support to DHMT in planning, financial management and supervision would be effective in improving compliance of DHMTs with MoH guidelines. 5. MoH should regularly offer newly appointed mid-level managers and supervisors an induction training that builds competencies in the use of mandatory management tools to ensure ongoing compliance, especially at the DHMT level, with MoH guidelines. EVALUATION QUESTION 5 What are the most significant accomplishments, best practices and lessons learned from the SSDI-Systems activity? Explicitly identify and document the facilitating and inhibiting factors to positive performance for each of the above questions. FACILITATING FACTORS  The MoH was receptive to the introduction of innovative technologies and solutions. Within the MoH, the DPPD was an excellent strategic MoH entry point for the activity.  The long- and short-term advisors contracted were of high quality and were continuously supported by the MoH through technical working groups. The advisors applied a very thorough and methodological approach to developing policies, guidelines and tools. SSDI-Systems Performance Evaluation 24  The key to success for the institutionalization of the tools and the application of guidelines at the district level was mentoring, coaching and continuous training support to senior and mid￾level managers at the MoH, ZHO and DHMT levels. INHIBITING FACTORS  Lack of long-term advisor presence at the zone and district levels resulted at times in disjointed planning and “parachuting” of activities.  At the district level, there was an unclear delineation between SSDI-Services’ and SSDI-Systems’ responsibilities.  Introduction of e-ISS and DHIS caused loss of perceived entitlements (per diem) for staff to deliver regular monthly paper-based reports.  Unreliable internet connectivity limits the use of internet-based tools and frustrates staff.  Lack of sufficient operations budget limits more regular supervision of health facilities by DHMTs. ACCOMPLISHMENTS  The DPPD has been sustainably strengthened as the institutional home for policy development, MYP/DIP planning, NHA, HMIS, performance review and supervision.  Tools and guidelines were developed to support the DPPD in its functions. These guidelines and tools have been piloted and rolled out in SSDI districts, and the MoH intends to scale up its use countrywide.  A well-thought-through training plan ensured that despite frequent DHMT staff transfers out of SSDI districts, skill levels in the use of the tools remained high.  Most staff perceive that the quality of their contribution to health services has improved, especially through the use of the e-ISS tool LESSONS LEARNED  The excellent mentoring relationship between the long-term advisor and the DPPD/PDU enabled not only the successful development of tools (such as e-ISS, policy development and analysis guideline, health surveillance assistants’ task-shifting guideline) and processes (MYP/DIP, performance reviews) but also downstream implementation of these tools and processes at the zonal and district levels for better decision-making.  Coaching though long-term advisors combined with training is the best method for capacity development.  Continuity of engagement between short-term advisors and the MoH’s point of contact should be ensured for the entire process of developing an innovation.  The consistency of the evidence-based methodological approach (baseline assessments, product design and pilot, evaluation and scale-up) used by SSDI-Systems must be credited with the successful development, ownership and scale-up of innovations introduced, such as the e-ISS, the iHRIS, PBI, the MYP/DIP, District Health Stakeholder Forum and the improved performance review mechanisms.  The long-term advisor mentoring approach proved successful in both capacity and institutional development and in establishing and anchoring new processes and tools within the MoH. At the district level, SSDI-Systems’ support was limited to training and intermittent coaching, which was effective in developing technical capacity for use of tools but less so in institutionalizing the routine use of the tool. Therefore, future SSDI support may consider placement of long-term SSDI-Systems Performance Evaluation 25 advisors in ZHOs to help speed up the management change process required for effective institutionalization of the management tools and processes introduced.  Sufficient training needs to be maintained throughout the project’s duration to counter skills attrition through staff transfers.  The introduction of e-ISS and the migration of HMIS onto the DHIS platform have negatively affected DHMT members’ perceived travel entitlement as they are no longer required to travel and hand-deliver monthly HMIS performance reports to ZHOs. Similarly, the e-ISS tool aims to limit the number of staff required to travel for the purpose of supervision. This has caused some resistance within DHMTs to accepting and implementing the new tools. In future support programs, the impact of new technology on established human behavior needs to be better assessed so that mitigating strategies are developed simultaneously with the tool.  In some of the districts visited, more than 20 NGOs or community-based organizations are active in health program interventions. SSDI support toward the development of the District Health Stakeholder Forum has helped to develop a “one-district plan,” but not a “one-district incentive plan.” It is laudable that NGOs and community-based organizations work together with district health staff, but this has also caused a competitive environment where staff tend to prioritize their work according to incentives on offer. This is a sensitive issue, which was usually paraphrased by DHMT staff interviewed as “competing priorities” and mentioned as a primary cause for non-compliance with e-ISS guidelines. This dilemma needs to be discussed and resolved in District Health Stakeholder Forum meetings. SSDI-Systems Performance Evaluation 26 IV. GENDER AND DISABILITY ANALYSIS Potential positive or negative impacts of the SSDI-Systems activity on gender were monitored as an integral topic throughout the inquiries. The overwhelming majority of the support provided by SSDI￾Systems was on institutional and individual capacity development. Gender imbalances at the management level of the health sector exist, but the determinants for this bias were outside the project condition. As in most neighboring countries, the percentage of women is highest at the service-delivery level and gradually decreases toward the level of MoH. Most of the human resource officers were female, while most district health officers are male. Therefore, gender participation in the trainings followed a similar pattern. Gender disaggregation of training opportunities provided was not reported by the activity. The evaluators’ data suggest that selection of trainees was based on the individual’s work profile without a gender bias. HMIS data were disaggregated by gender already before SSDI-Systems support was initiated. With respect to the tools developed under SSDI-Systems support, the evaluators could not find any gender difference regarding ability to handle the internet- and smartphone-based management tools developed. The activity had no impact nor did it intend to have an impact on issues related to disability. Disability services are not yet systematically addressed and mainstreamed into primary health care in Malawi. SSDI-Systems Performance Evaluation 27 ANNEX I: SCOPE OF WORK AND BACKGROUND INFORMATION USAID/Malawi’s Country Development Cooperation Strategy (CDCS), which covers the period 2013- 2018, has as its overarching strategic goal to: Improve the Quality of Life for Malawians. This is supported through three development objectives (DOs): DO 1: Social Development Improved; DO 2: Sustainable Livelihoods Increased; and DO 3: Citizen Rights and Responsibilities Exercised. USAID’s CDCS hypothesizes: If assistance is integrated, then development results will be enhanced, more sustainable, and lead to achievement of the CDCS goal: Malawian’s Quality of Life Improved. The CDCS will promote integration through the concentration of program and financial resources by what the Mission calls a “3-C Approach”: • Co-locating interventions to the extent that it is sensible; • Coordinating better within USAID and with other Development Partners (DPs), and • Collaborating to foster linkages among implementing partners and the DPs to improve results and sustainability. There are 28 districts in Malawi. All three DOs will implement activities in the three CDCS focus districts (Lilongwe, Balaka and Machinga); two DOs will implement activities in ten districts; and one DO will implement in seven districts. The three DOs will also provide limited nation-wide assistance in the remaining seven districts. One way that USAID/Malawi seeks to achieve the CDCS goal is through increased availability and improved quality of essential social services. The Mission is engaged in a range of health system strengthening activities to expand facility- and community-level service delivery and increase the number of people receiving high impact, high quality services. Support for Service Delivery Integration (SSDI) is USAID/Malawi’s flagship health activity. This activity consists of three inter-related sector activities, namely 1) SSDI-Services, 2) SSDI-Communications, and 3) SSDI-Systems. In close collaboration with the Ministry of Health (MoH), SSDI interventions support the increased availability and quality of the Essential Health Package (EHP) services; reinforce health promotion and disease prevention among households; and strengthen elements of the health system to sustain effective EHP delivery. SSDI’s development hypothesis postulates that: Programming health interventions through an integrated platform, consisting of activities in health policy and systems strengthening, support for integrated health service delivery, and social and behavior change communication, will result in significant expansion of coverage, quality and utilization of priority EHP services at community clinics, health centers and district hospitals. SSDI’s interventions align with the CDCS’s DO 1: Social Development Improved; and with crosscutting Sub-Intermediate Results (SIR) 1: Capacity of Institutions Improved), and SIR 2: Positive Behaviors Adopted). SSDI interventions also contribute directly to the Malawi Health Sector Strategic Plan (HSSP) 2011-2016. SSDI-Services’ strategies and interventions are designed to complement and support the social and behavioral change communications and health system strengthening interventions of Sectors II and III of the SSDI overall activity (SSDI-Communications and SSDI-Systems). 1) SSD-I Services Support for Service Delivery (SSDI-Services) provides an integrated service delivery program to improve the health and well-being of Malawians by improving the quality of priority Essential Health Package (EHP) services at the community- and referral- (health centers and District hospitals) levels. SSDI-Systems Performance Evaluation 28 SSDI-Services, implemented by JHPIEGO, is a five year USAID-funded project that runs from November 2011-November 2016. Its primary objectives include:  Increase access and utilization of EHP services for women and children and engage men in health care;  Improve quality of health services at community and facility level in target districts;  Improve health-seeking behavior by individuals, families and communities;  Strengthen health care delivery system via the development, testing, and scaling up of innovative and sustainable community-based service delivery approaches;1 and  Develop coherent and mutually supportive activities between the Government of Malawi (GoM), the three SSDI elements, the Private Sector and Social Marketing (PSSM) partners, and other national stakeholders to ensure integration and leveraging of program inputs to scale up service delivery. 2) SSD-I Communications SSDI-Communications is a five-year (September 2011- September 2016) USAID/Malawi social and behavior change communication (SBCC) activity. SSDI-Communications promotes normative and individual behavior change in several priority health areas, including HIV and AIDS, maternal and child health, malaria, nutrition, water and sanitation and family planning. The activity addresses barriers to behavior change at the structural, service delivery, societal, and personal levels. SSDI-Communication primary objectives include:  Strengthening national and targeted district level planning and coordination on EHP priorities;  Developing and producing evidence-based SBCC packages under multi-level media campaigns;  Building capacity of key national institutional partners and targeted district SSDI partners for effective SBCC strategic planning and delivery; and  Identifying best practices for SBCC implementation through formative research and testing innovative approaches. 3) SSD-I Systems SSDI-Systems is a five-year USAID-funded project awarded to Abt. Associates Inc. running from September 2011 to September 2016, whose mission is to assist the Ministry of Health (MoH) to improve policies, management and leadership, and fiscal responsibility to advance Malawi’s health system and the sustainable impact of the Essential Health Package (EPH). SSDI-Systems provides appropriate, relevant, and coordinated interventions at the national, zonal, district, and local levels. This sector’s primary objectives are to:  Provide the MoH with expert technical assistance in policy development;  Clarify and strengthen management functions at all levels for quality assurance;  Improve the current health management information system (HMIS) to ensure that key staff can carry out rigorous and routine high-quality data collection to support evidence￾based decision making;  Adapt proven tools and metrics to bolster monitoring and evaluation (M&E), financial management, and Human Resources for Health (HRH); and  Execute gender-sensitive programming that takes into account the exponential benefits resulting from activities that advance women’s and girls’ equity and health status. SSDI-Systems Performance Evaluation 29 PURPOSE OF THE PERFORMANCE EVALUATION The Contractor must carry out up to a minimum of two end-of-activity performance evaluations out of the following three. The specific purposes of each evaluation are as listed below. Purpose of Performance Evaluation of SSD-I Services: To determine the effect of SSDI-Services’ interventions on improved service delivery and quality of care at supported community clinics and health facilities; expanded coverage of quality EHP services; and increased uptake of quality integrated EHP. The main objectives are to measure and determine the extent to which SSDI-Services interventions had on quality of and access to care; provide in-depth insights into the facilitating and limiting factors of increased service utilization at each level of service delivery; and document progress made towards building MoH capacity to deliver quality EHP services. The evaluation outputs must provide evidence based recommendations on key actions required of USAID/Malawi and MoH to improve their activity planning. In addition, the evaluation outputs must provide recommendations to USAID/Malawi and the MoH to inform future implementation of integrated health service delivery-focused programming. Recommendations must go beyond general high level recommendations, and be based on a review of what worked well under SSDI-Services, and must articulate specific key approaches for the future. Purpose of Performance Evaluation of SSD-I Communications: To determine the effectiveness of the SSDI-Communications’ multilevel approach to promote normative behavior change and health seeking practices. Findings and recommendations from this evaluation will inform the implementation of USAID/Malawi’s new SBCC and integrated health activities. The main objectives are to determine the extent SSDI-Communications achieved its four primary objectives (see C1 above), with specific focus on extent to which SSDI-Communications was able to reposition the Health Education Section (HES); the degree to which SSDI-Communications’ campaigns resonated with individuals and communities; and an appraisal of the community mobilization implementation model. The evaluation outputs must provide recommendations to USAID/Malawi and the MoH to inform future implementation of SBCC programming in Malawi. Purpose of Performance Evaluation of SSD-I Systems: To determine the effectiveness of the SSDI-Systems approach to support Malawi-led and Malawi-owned efforts to achieve sustainable health results in line with current health priorities. The main objectives of this end-of-activity performance evaluation are to assess SSDI-Systems methodologies and approaches to capacity strengthening, and institutionalization of key MoH functions, including supportive supervision; management structures, responding to stakeholder expressed needs, and mentorship) in relation to the activity’s achievements. The evaluation outputs must provide recommendations to USAID/Malawi and MoH to inform future implementation of health system strengthening programming in Malawi. SSDI-Systems Performance Evaluation 30 ANNEX 2: EVALUATION PLANNING MATRIX SSDI Systems Evaluation Planning Matrix Evaluation questions Measures or indicators Data collection methods Data sources Design strategy/ framework Sampling methodology Data collection instrument Data analysis methodology #1: To what extent did SSDI-Systems enable institutionalization of health policy and financing functions, including policy analysis and development and NHA, in the MoH DPPD? Policy development is institutionalized and operational in DPPD/DPU NHA capacity developed, NHA regularly produced and data utilized in planning at MoH and district levels Health policy is developed on evidence created by NHA, iHRIS and HMIS/DHIS2 Annual budget development, budget execution and audit at central and district level is improved Desk review Semi￾structured interview Key informant interview NHA, SSDI￾Systems staff, MoH, Secretary for Health, Director Planning and Policy Development, Deputy Director SWAP, Head of Policy Unit, civil society, Ministry of Finance, MASM, NGO, HSSP-2 development partner) Compare changes over life cycle of project to judge net contribution of SSDI Systems project. Purposive sampling of up to 20 senior staff for semi￾structured interviews Interview notes Compare baseline with targets and trend over time #2: How successful has the integrated supportive supervision mobile tool been in improving supervision of health facilities, including follow-up of issues identified in supervision? % of health facilities which recorded supervisory visits (that used the tool) within past 3 months that have (i) followed-up on recommendations (ii) that did not follow-up on recommendations % of planned supervisory visits (with the tool) that took place # of supervisory visits that did not use the tool % of supervised staff that have received training in the use of the tool Desk review Semi￾structured interview Key informant interview Self￾assessment Documen￾tation at zonal and district level DPPD, Dept. of Clinical Services, Zonal Supervisors, DHMT members, health facility staff Compare behavior changes over life cycle of project Purposive sampling of up to 20 senior staff for semi￾structured interviews Purposive sampling of up to 60 key informant interviews and self-assessments across 4 zones and 8 districts Interview notes Key informant interview transcripts Self￾assessment data Documen￾tation at zonal and district level Review of trends over time since inception #3: To what extent % of districts that use guidelines Desk review DPPD, CMED, Compare changes Purposive Interview Compare SSDI-Systems Performance Evaluation 31 did SSDI-Systems improve utilization of data (including iHRIS, DHIS2 and supportive supervision data) for evidence-based planning and decision-making at central and district levels? and evidence from HMIS/DHIS2, iHRIS and NHA for annual and three-year strategic implementation planning % of interviewees that received training and are confident in the use of data for decision making tools (HMIS, iHRIS and NHA) % of districts that raised funds to fill funding gaps in their annual DIP Semi￾structured interview Key informant interview Self￾assessment Documen￾tation at zonal and district level HEU, HRMO, Zonal Supervisors/ Managers, health facility staff over life cycle of project to judge net contribution of SSDI Systems project. sampling of 20 key officials, involved in SSDI-Systems from MoH, development partners, Abt Assoc. and mission for SSIs Purposive sampling of up to 60 KII and self-assessments across 4 zones and 8 districts notes Key informant interview transcripts Self￾assessment data Documentation at zonal and district level baseline with targets and trend over time #4: Of the capacity￾strengthening approaches employed by SSDI￾Systems (e.g., secondment of staff to MoH/NAC positions, provision of technical assistance by external advisors, training and coaching of MOH and district staff, and operational and logistical support), which have most enabled the MoH to address systems￾level issues at central and district levels? % of requests by MOH for secondment of staff to MoH/NAC positions at central and district levels, post initial intervention that was supported. Number of requests by MOH management for technical assistance by external advisors at central and district levels, post initial intervention that was supported. % of seconded staff/technical assistance whose expert profile matched requirements by MoH. Nature of requests for operational and logistics support that could not be provided by the project. Average time between request for technical assistance and delivery of technical assistance Average time between request for and placement of seconded staff MoH staff that have been trained Desk review Semi￾structured interview MoH managers at central, zonal and district levels; NAC and MNCP executive officer, SSDI staff; frontline health care providers at facility level, key informant interview Assess readiness of project design to respond to clients expressed capacity￾strengthening needs Purposive sampling of 20 key officials, involved in SSDI-Systems from MoH, development partners, Abt Assoc. and mission for SSIs Purposive sampling of up to 60 key informant interviews and self-assessments across 4 zones and 8 districts Data from desk review Interview notes Key informant interview SSDI-Systems Performance Evaluation 32 within past 4 years in management and leadership that remain in MoH #5: What are the most significant accomplishments, best practices, and lessons learned from the SSDI￾Systems activity? Explicitly identify and document the facilitating and inhibiting factors to positive performance for each of the above questions. Number of positive responses from SSDI-Systems activity beneficiaries on given accomplishment, best practices, or lesson learned; Documented evidence of said accomplishment, best practice or lesson learned appearing in the official and public domain Desk review Semi￾structured interviews Key informant interview Interviews with internal and external stakeholder Synthesize from findings (Evaluation question 1-4) Purposive sampling of 20 key officials, involved in SSDI-Systems from MoH, development partners, Abt Assoc. and mission for SSIs Purposive sampling of up to 60 key informant interviews and self-assessments across 4 zones and 8 districts SWOT analysis SSDI-Systems Performance Evaluation 33 ANNEX 3: LIST OF 16 PMP PRIORITY INDICATORS PERIOD YEAR 1 (2012) YEAR 2 (2013) YEAR 3 (2014) YEAR 4 (2015) YEAR 5 (2016) Indicator Unit of Measure Target Result Target Result Target Result Target Result EOP Target Results to date 1 Number of policies, guidelines, and regulations that have been improved with project support and approved by the Government of Malawi policy/guideline documents 2 0 2 0 3 1 3 2 7 4 2 Percentage of quarterly meetings held by the PDU during the reporting period meetings - 100% 75% 100% 100% 100% 100% 100% 100% 5b Percentage of non-MoH stakeholders (including civil society organizations) that were engaged in the policy￾development process during the reporting period individual representatives n/a - 100% 0 100% 116% 100% 100% 100% 100% 17 Percentage of SSDI-targeted districts that received written feedback on their HMIS reports in a semiannual period districts 6% 0 100% 0 100% 100% 100% 100% 100% 100% 22 Number of health workers that have successfully completed a management and leadership in￾service training with approved tools health workers n/a - 180 252 25 21 18 27 43 51 28a Percentage of target districts receiving the set number of integrated supportive supervision visits district n/a - 7% 0 53% 20% 100% 40% 100% 80% 28b Percentage of target facilities receiving the set number of integrated supportive supervision visits facilities n/a - n/a - 55% 21% 100% 27% 100% 79% SSDI-Systems Performance Evaluation 34 PERIOD YEAR 1 (2012) YEAR 2 (2013) YEAR 3 (2014) YEAR 4 (2015) YEAR 5 (2016) Indicator Unit of Measure Target Result Target Result Target Result Target Result EOP Target Results to date 36b Number of SSDI-supported cost centers that updated iHRIS data during the quarter cost centers n/a - n/a - 19 19 19 19 19 19 39B Number of people that were trained in staff performance management system individuals n/a - n/a - 1125 970 n/a n/a 1,125 1,001 40 Percentage of districts that have a valid annual or multiyear DIP in place districts n/a - n/a - 53% 13% 100% 100% 100% 100% 42A Number of district-level participants that complete the management and leadership capacity-building program using the standard set of training materials participants n/a - n/a - 56 36 24 49 56 85 45E Percentage of expenditure being queried by auditors expenditures n/a - n/a - <5% 1.60% <5% 0.05% <1% 0.05% 45C Percentage of districts that record improved scores on the financial management coaching tool from the previous reporting period districts n/a n/a 75% 100% 100% 80% 100% 80% 48B Percentage of national-level staff that have been coached on Global Fund financial management practices individuals coached n/a n/a 12 10 12 12 12 12 49 National Health Financing Strategy produced and accepted by the Government of Malawi yes/no yes no yes no yes no yes no yes no 51B Number of expenditure analysis reports produced with project support reports n/a n/a 2 2 14 6 16 ANNEX 4: LIST OF ZONES, DISTRICTS AND FACILITIES VISITED Name of Health Facility District GPS coordinates X Y Eastings Northings Nsanje District Hospital Nsanje 35.25989 -16.9177 740712.2 8128332 Mbenje Health Centre Nsanje 35.22086 -16.9948 736456.8 8119844 Mulanje District hospital Mulanje 35.5075 -16.0256 768326.2 8226772 Chonde Health centre Mulanje 35.32317 -15.9973 748626.1 8230126 Ntaja Health Centre Machinga 35.5295 -14.8681 772189.1 8354862 Machinga Hospital Machinga 35.60706 -14.7725 780662.8 8365349 Balaka Hospital Balaka 34.94953 -14.9851 709643 8342542 Mangochi Hospital Machinga 35.26472 -14.4819 744112 8397907 Monkeybay Community Hospital Mangochi 34.91083 -14.085 706314.6 8442173 Mitundu Community Hospital Lilongwe 33.77444 -14.2483 583676.1 8453267 Dowa District Hospital Dowa 33.93689 -13.6551 601330.6 8490358 Mponela Community Hospital Dowa 33.73761 -13.5325 579817.2 8503986 Kapenda Health Centre Chitipa 33.24904 -9.49456 527340.1 8950556 Chitipa District Hospital Chitipa 33.26611 -9.78528 529184.2 8918414 Nkhotakota District Hospital Nkhotakota 34.29239 -12.9289 640201.8 8570496 Nkhunga Health Centre Nkhotakota 34.10822 -12.4802 620431.6 8621329 Lilongwe District Hospital Lilongwe 33.77478 -13.991 583676.1 8453267 SSDI-Systems Performance Evaluation 36 ANNEX 5: LIST OF INTERVIEWEES Name Position Gender District Institution Central level Chimwemwe Mvula Assistant Director of Clinical Services M Lilongwe Ministry of Health Blessings Mbewe Principal HRM M Lilongwe Ministry of Health Rhino Mchenga Deputy Director CMED M Lilongwe Ministry of Health Dr. Stone Kabuluzi Director Preventive Medicine M Lilongwe Ministry of Health Dr. Rabson Kachala Head of Health SWAp Secretariat and Coordinator of SSDI Project Evaluation for the Republic of Malawi Government M Lilongwe Ministry of Health Dr. Dominic Nkhoma Head of Policy Development Unit (PDU) M Lilongwe Ministry of Health Mr. Precious Phiri Deputy Director Preventive Medicine (Primary Health Care) M Lilongwe Ministry of Health Partners level M Hudson Zithane￾Nkunika Technical Lead Result 1 (Policy) M Lilongwe Abt Jakob Kawonga Technical Lead Result 1 (M&E, DHIS2) M Lilongwe Abt Grace Banda Technical Lead Result 3 (Supportive Supervision) F Lilongwe Abt Mrs. Bona Mjojo Technical Lead Result 4 and 2 (iHRIS, PMS, HR Planning) F Lilongwe Abt Amanda Manjolo Technical Lead Result 5 (DIP and reviews, Health Stakeholder Forum) F Lilongwe Abt Rodney Mwaisimba Technical Lead Result 5 (Financial Management) M Lilongwe Abt Ndasowa Chitule Former AOR M Lilongwe USAID Lilly Banda Deputy Head, Health Office F Lilongwe USAID Dr. Andrina Mwansambo Director of Policy and Development F Lilongwe NAC Amy Diallo AOR M Lilongwe USAID Matthews Mviiri PBI Manager M Lilongwe Options Zonal level Dr. Malangizo Mwale Zone Manager M Blantyre SW Zone Leonard Banda Assistant Zone Supervisor M Blantyre SW Zone Noel Kasomekela Zone M & E officer M Blantyre SW Zone Raymond Kawaye Acting M & E M Zomba SE Zone SSDI-Systems Performance Evaluation 37 Dr. Msadala Zone Manager M Zomba SE Zone Alinafe Mangulenje Zone Nursing Officer F Zomba SE Zone Ali Ndelemani Radiology Supervisor M Lilongwe CE Zone Noel Mphasa Zone TB Supervisor M Lilongwe CE Zone Elizabeth Chitsa Banda Zone Manager F Lilongwe CE Zone Dr. Jean Namasasu Zone Manager M Lilongwe CE Zone Christina Mchoma Ag. Technical Officer F Lilongwe CW Zone Dr. Owen Musopole Zone Manager M North North Rose Chisiza Assistant Zone Supervisor F North North Nelson Nkosi Zone Radiography Supervisor M North North District level Pilirani Kabango Senior Nursing Officer (matron) F Lilongwe Mitundu Hospital Kamnager Unthu Statistician (HMIS Staff) M Mitundu Hospital Jean Kaponda Assistant Hospital Services Administrator F Lilongwe Mitundu Hospital Sharif Rajab Clinical Officer (In-charge) M Mitundu Hospital Mayamiko Machika District Nursing Officer M Dowa Dowa District Hospital George Mphatsa Clinical Officer M Dowa Dowa District Hospital Samson Kumphale Environmental Health Officer/EPI Coordinator M Dowa Dowa District Hospital Dave Nuka Health Education Officer/PRO M Dowa Dowa District Hospital Eness Banda Reg. Nurse/Midwife F Dowa Mponela Rural Hospital Yohane Madiwa Statistical Clerk M Dowa Mponela Rural Hospital Frank Linzie Clinical Officer-in-charge M Dowa Mponela Rural Hospital Boyce Nyirongo Senior H.S.A. (PBI Coordinator) M Chitipa Kapenda Health Centre Michael Rhaheya Senior Medical Assistant M Chitipa Kapenda Health Centre Nicholas Simukonda Senior H.S.A. M Chitipa Kapenda Health Centre Ken Kanyika Nurse Midwife Technician M Chitipa Kapenda Health Centre Selemani District Nursing Officer M Chitipa Chitipa District SSDI-Systems Performance Evaluation 38 Kondowe Hospital Mac Donald Nkhonjera Principal Health Services Administrator M Chitipa Chitipa District Hospital Chelewani Human Resource Management Officer M Chitipa Chitipa District Hospital Noel Zandola District Environmental Health Officer M Chitipa Chitipa District Hospital Manda Lab Manager/PBI Coordinator M Nkhotakota Nkhotakota District Hospital Oscar Msutu HMIS Officer M Nkhotakota Nkhotakota District Hospital Maybe Ntambalika Human Resource Management Officer M Nkhotakota Nkhotakota District Hospital Simon Ntopi District Environmental Health Officer M Nkhotakota Nkhotakota District Hospital Mervice Kampheta Data Clerk F Nkhotakota Nkhunga Health Centre Abraham Paul Pharmacy Assistant M Nkhotakota Nkhunga Health Centre Rueben Chiperesa S.H.S.A. M Nkhotakota Nkhunga Health Centre Eness Banda Medical Assistant F Nkhotakota Nkhunga Health Centre Misomali HMIS Officer F Lilongwe Lilongwe District Hospital Paul Chunga District Environmental Health Officer M Lilongwe Lilongwe District Hospital Patricia Kapena District Nursing Officer F Nsanje Nsanje District Hospital Felix Moniya Hospital Administrator M Nsanje Nsanje District Hospital Bernard Mvuwa HMIS Officer M Nsanje Nsanje District Hospital Dr. Yamikani Masitala District Medical Officer M Nsanje Nsanje District Hospital Mary Mainje Medical Assistant F Nsanje Mbenje Health Centre Dr. Khuliena Kabwere District Health Officer M Mulanje Mulanje District Hospital Francis Mapeto Huma Resources Officer M Mulanje Mulanje District Hospital Nyirenda Senior Accounts Assistant M Mulanje Mulanje District Hospital Neffi Matupa Nurse F Mulanje Chonde Health Centre Sam Kamwalila Health Surveillance Assistant-DHIS2 Focal Point Officer M Mulanje Chonde Health Centre Mike Mabauti Senior Medical Officer M Mulanje Chonde Health SSDI-Systems Performance Evaluation 39 Centre Lizzie Mumba Senior Health Surveillance Assistant F Mulanje Chonde Health Centre Akuzike Mkali Medical Technician M Machinga Ntaja Willex Msowoya Statistical Clerk M Machinga Ntaja Rotina Mlombwa Hospital Administrators F Machinga Machinga Hospital Mponda Human Resources Officer M Machinga Machinga Hospital Dr. Mtibu District Health Officer M Machinga Machinga Hospital Kabambe Accountant M Machinga Machinga Hospital Elick Mhango Human Resources Management Officer M Balaka Balaka Hospital Tikiwa Accountant M Balaka Balaka Hospital Dr. Peno District Health Officer M Mangochi Mangochi District Hospital Agness Jenda Human Resources Officer F Mangochi Mangochi District Hospital Benadetta Kambalame IT officer F Mangochi Mangochi District Hospital Macdornad Gongwe Senior Nursing Officer M Mangochi Monkeybay Hospital Ashly Malpass Volunteer F Mangochi Monkeybay Hospital SSDI-Systems Performance Evaluation 40 ANNEX 6: DATA COLLECTION INSTRUMENTS AND GUIDELINE SSDI-Systems Final Performance Evaluation Instrument: Key informant interview guide Organization: Site: Respondent ID: Title: Name: Respondent Gender (M/F) Stakeholder: SSDI-Systems staff (Abt, ……) 1 Development partner (HSSP-2 partner) 2 Zonal-level staff (Trainer of trainers, supervisor) 3 District-level staff (HMIS Staff) 4 Donor (USAID) 5 Ministry of Health 6 Other (Please write): 7 Hello! My name is________________________, I am an interviewer for DevTech, which is doing a study for the Ministry of Health and its partners. We are here to ask questions about the SSDI-Systems activity and hope you can help us. All answers will remain confidential. It should take about an hour. Please be advised that you are not in any way obliged to participate in this interview, and you can discontinue the interview at any time without any penalty. You can also refuse to answer any question and move on to the next one. At the end of this interview we would like to ask you to fill out a short self-assessment questionnaire about SSDI’s performance. Please answer « yes » if you agree to participate. 1. Kindly explain in a few words what you know about how SSDI Systems supports the health sector in Malawi. Probe: Policy analysis and development; health financing and financial management; evidence-based decision-making (DHIS–iHRIS–NHA), supportive supervision, human resources management, leadership & management of HR, decentralization 2. Kindly explain how SSDI Systems has supported the MoH in its role as steward for health policy and planning and health financing. Probe: (i) Provision of LTE; STE, TA and, (ii) TOR and staffing, training, roles and responsibilities of PDU, PU, CMED, DHRMD 3. Kindly explain how the SSDI Systems has supported the decentralization of health planning and management to district level. Probe: DIP, IHRIS, HMIS-DHIS, FM Training, Health Forum 4. In your opinion, has SSDI-Systems supported better evidence-based decision-making within your institution? Probe: If and how the development of the new health policy has benefitted from DHIS2, IHRIS, NHA SSDI-Systems Performance Evaluation 41 5. In your opinion, has the supportive supervision tool significantly improved the performance of the health workforce at the facility level? Probe: ease of use, follow-up support (DHMT), training, improved regularity of supervision, ongoing parallel supervision 6. To your knowledge, has the SSDI-Systems project supported the MoH in the development of health financing innovations? Probe: health fund, health insurance, PBI, PPP CHAM, paying service guidelines 7. In your opinion, which capacity-development approach used by SSDI-Systems do you consider was most effective? Probe: LAM training, specific technical training, external adviser, coaching, integrated expert 8. Would you please describe the SSDI-Services accomplishments/lessons learned that stand out as best practices? What are the main factors that contributed to these successes? Probe: facilitating and inhibiting factors; recommendations for future USAID support Thank you very for your participation, distribute self-assessment SSDI-Systems Final Performance Evaluation Instrument: Self-Assessment Questionnaire (Likert Scale) Stakeholder: SSDI implementers and partners Organization: Site: Respondent ID: Title: Name: Respondent Gender (M/F): PART A: Please circle the number of the stakeholder group that you belong to: SSDI-Systems staff (Abt, ……) 1 Development partner (HSSP-2 partner) 2 Zonal-level staff (Trainer of Trainer, Supervisor) 3 District-level staff 4 Donor (USAID) 5 Ministry of Health 6 Other (Please write): 7 PART B: For each question, please tick the box that best matches your answer (1, 2, 3, 4 or 5 “I don’t know”). Please answer the questions from the perspective of your institution. Topic: Institutional Development 1. Within the last five years, to what extent has SSDI-Systems improved the effectiveness of your institution in the following areas: 1 “not at all” 2 “a little” 3 “moderately” 4 “a lot” 5 “I don’t know” Policy analysis and SSDI-Systems Performance Evaluation 42 development Financial management Supportive supervision at district level Information management (DHIS2) Workforce management (iHRIS) Budget preparation (NHA) Leadership and management (LAM) 2. Within the last five years, to what extent has SSDI-Systems strengthened the process and quality of your institution in the following areas: Note: Only select choices that match your level of work 1 “not at all” 2 “a little” 3 “moderately” 4 “a lot” 5 “I don’t know” MoH: Annual planning MoH: Annual and semiannual performance reviews Zone: Annual and quarterly performance reviews District: District Implementation Plan (DIP) District: 3-year Medium￾Term Plan (MTP) District: quarterly and annual review of DIP District: Budget development and execution 3. Within the last five years, to what extent has SSDI-Systems developed financial management capacity of your institution the following areas: 1 “not at all” 2 “a little” 3 “moderately” 4 “a lot” 5 “I don’t know” Fleet and fuel management Payroll management IFMIS Audit Drug and stores Staffing Procurement Asset management/ inventory Accounting Budget and control SSDI-Systems Performance Evaluation 43 Topic: Integrated supervision 4. At the district level, the use of the SSDI-Systems-supported mobile supportive supervision mobile tool has: 1 “not at all” 2 “a little” 3 “moderately” 4 “a lot” 5 “I don’t know” Eliminated parallel supervision structures Improved regularity of supervision Improved documented feedback provided from supervisor Encouraged staff to take more actions (based on feedback from supervision) Enabled the identification of new issues Effected that supervision findings are now regularly discussed at DHMT meeting Improved job satisfaction of supervised staff Improved quality of service delivery Led to cost savings Topic: Data for Decision-making 5. As a result of SSDI-Systems support, your institutions now routinely use iHRIS data for health workforce development and deployment: 1 “not at all” 2 “a little” 3 “moderately” 4 “a lot” 5 “I don’t know” 6. As a result of SSDI-Systems support, your institution now routinely use DHIS-2 data for program planning: 1 “not at all” 2 “a little” 3 “moderately” 4 “a lot” 5 “I don’t know” 7. As a result of SSDI-Systems support, your institution now routinely use mobile supervision reports for planning and management: 1 “not at all” 2 “a little” 3 “moderately” 4 “a lot” 5 “I don’t know” SSDI-Systems Performance Evaluation 44 8. As a result of SSDI-Systems support all relevant staff of your institution are now confident in the use of iHRIS and DHIS-2 tools for planning and review: 1 “not at all” 2 “a little” 3 “moderately” 4 “a lot” 5 “I don’t know” 9. As a result of SSDI-Systems support, targeted districts now use iHRIS, DHIS-2 and supervision data for developing and reviewing their annual DIP: 1 “not at all” 2 “a little” 3 “moderately” 4 “a lot” 5 “I don’t know” Topic: Capacity development 10. In your opinion, which capacity development modality used by SSDI-Systems has produced the best result: 1 “not at all” 2 “a little” 3 “moderately” 4 “a lot” 5 “I don’t know” Long-term experts (secondment of staff) Short-term experts Coaching Training 11. SSDI-Systems support through capacity development of staff enabled facility to improve quality and effectiveness of their services: 1 “not at all” 2 “a little” 3 “moderately” 4 “a lot” 5 “I don’t know” 12. SSDI-Systems support to sustainable capacity development is limited due to high staff rotation/redeployment within MoH: 1 “not at all” 2 “a little” 3 “moderately” 4 “a lot” 5 “I don’t know” 13. Have you received training under the SSDI-Systems support in the following areas: Area of training Year of training Comment, if any Leadership and management Training in use of supervision tool Training in use of iHRIS Training in use of DHIS2 Other (specify) Other (specify) Other (specify) Other (specify) SSDI-Systems Performance Evaluation 45 PART C: Please answer in your own short words the following questions: 14. What were the success stories of SSDI-Systems activities? ______________________________________________________________________ ______________________________________________________________________ 15. What were the missed opportunities of SSDI-Systems activities? ______________________________________________________________________ _____________________________________________________________________________ _______________________________________________________________ SSDI-Systems Performance Evaluation 46 ANNEX 7: SUMMARY OF QUALITATIVE ANALYSIS OF TRANSCRIBED KEY INFORMANT INTERVIEWS Evaluation Question 1: Health Policy One of the success stories about the support from SSDI-Systems is the establishment of the PDU as a dedicated working place for policy development. In the past, most of the departments initiated their own policy development process; now everything on policy development is channeled through the PDU. This unit is now headed by the deputy director and has three additional staff, including the NHA desk officer. The program also developed the Health Policy Development and Analysis Framework, the National Districts Implementation Guidelines and other priority policies, on which MoH staff were oriented. The OPC has been a facilitating factor in that it developed a general framework on how public sector policy should be developed. Based on the OPC framework that the MoH adapted, the project supported the development of a guideline on how to develop the health policy. Policy workshops were presided over by the minister and participated in by all directors, program managers and the concerned stakeholders. During these policy workshops, the stakeholders had to go through the guidelines by using the national health policy as a case study to find out whether some issues have been left out or wrongly placed. However, some old outstanding policies such as the National Malaria Policy and the Health Promotion Policy were being finalized in their respective departments. These policy guidelines have great impact on the way of conducting business within the ministry. QUOTES “The ministry invites programme managers or DHMT members to a dissemination workshops where new guidelines and policies are communicated to us. The ministry is very kin to disseminate any changes in policy guidelines to the districts.” The Health Surveillance Assistants Task-Shifting Policy The Health Surveillance Assistants policy was downgraded to a guideline. This guideline states how the other stakeholders like NGOs can work with Health Surveillance Assistants at district and community levels. In the past NGOs would just come in and start engaging the Health Surveillance Assistant without looking at their capacity, case workloads and job description, and in some instances withdrawing them from the communities where they were working. The implementation of the HSA task-shifting is accompanied by the M&E plan. When the joint supervision teams go to the districts, they use the framework of indicators (monitoring and evaluation plan) to see whether the implementation of the project is done according to the policy guidelines and to find out whether there is adherence or not in implementation of the guidelines, i.e., are the Health Surveillance Assistants doing what is in the policy guidelines. Through supervision where we (M&E officers) use performance indicators we ask are HSAs doing ABCD. In the process, they will know that if they are still doing activities that the guidelines are saying they shouldn’t, then they will straight away take some corrective measures against non-adherence to the implementation of the guidelines. SSDI-Systems Performance Evaluation 47 QUOTES “In the past whenever NGOs came to the districts.....they would push tasks on Health Surveillance Assistants over and above their normal job descriptions...then an assessment was done and found out that there is a need for a policy guidance” Development of Multiyear Plans and DIPs: The MYPs and DIPs have a set of indicators against which progress on implementation can be assessed. At the district level, the district stakeholder forum is very successful, the DC is the chair, usually a member of a NGO is the co-chair, and the DHO takes on the role of secretariat but usually delegates this function to the environmental health officer. Before developing DIPs, technical advice was sought from the zones’ monitoring and evaluation officer. Health Financing Functions SSDI also worked in areas of health financing, which includes: health insurance, health fund, and it assisted in developing the MoU with CHAM. SSDI also supported the health reforms program. SSDI was the only technical partner at the time the MoH was trying to conceive some reform areas. SSDI have been mentoring the PDU on NHAs. NHA data were used to inform the Health Financing Strategy. NHA data have informed the MoH on how far the MOH can deal with more taxes, and out-of￾pocket payments. NHA informed the MoH on the risks of using the user fees. Previously, resource￾mapping exercises were replacing NHA, but now with the SSDI project, the NHA task team is a stand￾alone with its own methodology to follow. To diversify the resource generation at health facilities, DPPD engaged the Ministry of Finance to allow the facilities (central hospitals) to retain funds from paying services. SSDI helped the PDU to come up with financial management and operational guidelines that the funds hospitals generate through paying services are collected and utilized following laid-down standard financial procedures. Furthermore, although SSDI was not directly involved in the reforms, it advised the ministry on health reforms; many reform areas were already under exploration with SSDI. SSDI provided PDU with documentation that was used to start the reform process. So I would largely credit the health reforms process (hospital autonomy, decentralization, health fund and health insurance) to the SSDI program. Currently the district health budget submitted to the DC contains only the government budget, whereas the plan and expenditure framework that is submitted to MoH covers the entire budget, the total cost estimate, including contribution pledged by members of the Stakeholder Forum in cash or usually on kind. However, the DC is the chair of the stakeholder forum and should therefore also own in the entire DIP planning process. It is fair to say the DHO office is in charge and uses this to leverage with stakeholder additional resources to implement the DIP. The Stakeholder Forum was developed to include non-government actors in the district planning process. Challenges Indecisive decision-making by government: The MoH developed a national malaria policy, but this was put on hold not to proceed at the last minute after engaging a senior technical advisor to help the MOH SSDI-Systems Performance Evaluation 48 to develop the policy. The reason was that the NMCP personnel could not proceed because they were involved in other activities. Staff rotations: Transferring of staff responsible for the project has been quite regular; there is no capacity building because the responsible staff are kept on moving or assigned to other duties. For example, three officers were transferred from PDU and there came new officers who assumed the coordination role; this disrupted the momentum of the establishment of the PDU. This includes assigning staff in an ad hoc manner to develop policy when they have not been trained to do so. Snail pace in approval of policies: There are a lot of procedures before approving policy, as some of the policies have to go through OPC, and the project and/or MoH have very little powers to influence them. The policy formulation goes through three stages: The first one is the committee of the principle secretaries to review, then the Cabinet committee of social services, and it then goes to Cabinet for review as well. The good example is the national health policy, which started way back in 2003, and it was handed over to the project to help in speeding up the process, but up to now it is not finalized; it is waiting for Cabinet approval. Limited financial management at district level: District-level staff have little knowledge in leadership management; most of them do not know financial management, how to solve problems, and they do not have background of team-building. Before SSDI there were a lot audit queries because there were a lot of missing documents due to poor filing systems, and the vouchers were not accompanied by documents. Funding Issues: Low resources provided to the project affect implementation of some projects, i.e. cars. Funds should be given to the zone officer or the DHO, and make them accountable. At the district level, government has one pot account that is used by all the departments. Sometimes you find that your allocation has gone without the knowledge of the head of that sector. There is a need for openness in terms of budgets. QUOTES “NGOs do not share their budgets with the DHO, and when they report on achievements we cannot validate their information on their impact areas they just come and start implementing their projects without consulting the DHOs.” Conclusions Now partners are supporting what the districts have drawn up as a priorities and lined them up in the DIP and not what the partner wants as a priority. The health sector now involves stakeholders in the development of the DIPs, before the priorities are discussed at the zonal level. There is a need for SSDI to assist in the capacity building of the director of finance and accounts office so that all officers are conversant with the disbursements of funds in line with decentralization procedures so that all funds are accounted. SSDI has also helped in development of guidelines for the work that the community volunteers do in the country to rationalize what they do in the community. Evaluation Question 2: Integrated Supportive Supervision SSDI-Systems Performance Evaluation 49 The zonal and district-level officers reported that SSDI trained ZHOs and DHMTs on how to collect information using the integrated supportive supervision and how to measure the districts’ and health centers’ performance. Previously, there used to be a paper-based supportive supervision checklist. Now all SSDI districts are using an electronic supportive supervising tool. The checklist is already loaded into the smartphone and you go through the checklist electronically on the smartphone as you are conducting supervision. The system also calculates the indicators for specific areas, so that you can see whether the district is doing better or not. The integrated supportive supervision tool has made the compiling of reports simple. Quotes “After supervision we do management meeting to follow up on the issues raised and give them feedback on our findings and apart from that we have quarterly meetings with the districts and we discuss some of the issues noted in the districts. And when we go back to the districts to make follow ups on the issues that we discussed.” –Zonal officer “This tool has brought seriousness both at district management level and at the zonal level because when we go for supervision, we are guided by this tool and we make informed decisions right there and make follow ups to see how the facility is performing.” “This tool has really reduced the number of visits because the checklist itself is broad, it covers a lot of programs and it’s an integrated checklist.” Since the introduction of the supervision tool, production of supervisory reports and follow-up system have improved in the districts supported by the SSDI as compared to non-SSDI districts. The project has also supported the zone team to conduct district monitoring visits. Furthermore, district-level informants reported that with the use of the mobile supervision checklist, they are able to see what is happening in their respective districts and other districts. The views at district level also suggest that the integrated supportive supervision tool has made the compiling of reports simple. QUOTES “We have moved from paper based system to electronic system and it has simplified our work load and everybody from the ministry has access to it, and now I’m able to compare the performance-whether I am performing better, i.e. having more greens or not having more yellows, or reds.”–DHO “Integrated supportive supervision is a nice approach because you do supervision today you discuss the gaps, you have a work plan and you give each other time lines. And it is so helpful, to do the follow up of the action points.”–DHO “The use of a smart phone is a very objective way of conducting an assessment because it gives a feedback on the spot on how the facility is performing. And when there is a problem we even ask what could be the solutions and every one see where the problem is which was very had for paper based approach.”–DHO Health center staff are sidelined from the integrated supportive supervision tool training, and it becomes difficult for their supervisors to supervise them using the tool. Conclusions Since the introduction of the tool, the supervision reporting system has improved. The tool is used during integrated supervision, where all the supervisors go together to supervise according to their specialties in their fields as a team. SSDI-Systems Performance Evaluation 50 There are also program-specific supervision on a quarterly basis but not as team. And there is also supervision that has been organized at the ministry headquarters by specific departments. Now we are conducting joint activities with stakeholders because we plan together. Development of DIP review meetings, now you can see the project coordinators understanding the indicators and reporting time. Evaluation Question 3: Use of Data for Decision-making At the central level, SSDI-Systems has supported (member of core team) the development of guidelines for the annual DIP and the multiyear (3-year) District Health Plan (MYP). There is now a gap year: HSSP (2011-2016) is coming to a close, but HSSP2 (2016-2022) not yet started, so as of now only DIP is developed. HMIS/DHIS data are currently analyzed as part of the HSSP2 and the MYP process. At the zonal level, staff were trained in the supervision tool and feel that that ISS is a good system because you receive the feedback right there whether you are doing better or not, and it is cost￾effective in terms of savings. At the district level, SSDI-Systems developed DIP guidelines, and people were trained on how to use them and supported development of DIP/MYP and how to conduct regular monthly, semiannual and annual review mechanisms of these plans. As per the Local Government Act, it is the District Council that approves the DIP budget but not the (technical) plan. The technical part is approved thorough the MoH. SSDI has also helped the MoH in creation of a human resources for health database to assist the ministry to track human resources and deployment of staff in the health sector using iHRIS. With iHRIS it is easy to track staff according to grades, vacancy rate, when someone is going to retire. iHRIS is a very good planning tool; for example, vacancy rate is at 40 percent–this we have come to know because of this tool. Each and every member of the management level of the DHMT has access to it. The iHRIS has helped the human resources office in workforce planning. For example, with the use of the tool we are able to see the possibility of moving some members of staff from the health centers that are having more officers and transferring them to health facilities that had shortage of staff. Quotes “The iHRIS is a good tool because you know which areas are not doing well and areas that are doing well while before iHRIS tool we were paper based supervision. With this tool there is now a possibility of moving staff members from the health centres that are having more officers and transferring them to health facilities that have shortage of staff using iHRIS.”–District human resources officer Most informants at the district level felt that the DIP developed is now based on HMIS/DHIS2 and iHRIS data, whereas before plans were only based on the budget and list of some activities and HMIS data were not used. Quotes “This DHIS2 is a good system but we have problems with internet connectivity and now we are using dongle. We are provided with K6000 to buy airtime from the DHOs office or our partner SSDI office but it should be much better if we were connected to the internet.”–HMIS officer “Now if I want to see how the district is performing, I will not go to the districts looking for their books I will simply go into DHIS2 tool.” SSDI-Systems Performance Evaluation 51 “Now I am working hand in hand with the partners making sure that the DHIS tool is utilized to the maximum and we are encouraging them that when they are coming for the review meetings data should be retrieved from the DHIS tool.”–Zonal officer Challenges Shifting members of staff is a big problem in government, because the person who has been transferred will go with the knowledge, especially when the other members were not learning from him. Lack of internet connectivity Shortage of airtime Few officers trained in iHRIS at the zonal level The health facility staff do not normally participate in the development of the DIPs. The capacity at the health center level is little in terms of IT knowledge and IT technology, human resources. Poor funding from government A lot of senior people go for supervision, leaving the hospital with junior staff. Lack of expertise to use the HMIS Local government guidelines are different from the planning department, and they confuse the system. The system does not provide information for a specific zones, but it provides the information for the whole nation and you have to concentrate on the districts that you supervise. We have the competing priorities within the districts and are handled by the same small team of the DHMT; there is need for the DHMT to stick to their core mandate. Shifting of trained staff members to other health facilities Some of the smartphones are not sending the information. Shortage of transport to conduct supervision Conclusions The DMHT are conversant with the DHIS tools and how they can retrieve the data. The electronic tool introduced by the SSDI has reduced workload to data officers who were using paper-based checklist and transferring it again to the electronic. Planning processes have now been simplified because they can quickly know which indicators are doing well and the indicators that are not doing well. Utilization of data has been simplified any person from the central office can see how districts are performing on the computer. With SSDI training, there is an improvement in financial management filing systems, budget and pre-audit preparation. Evaluation Question #4: Capacity Development At the national level, SSDI has strengthened capacity in systems, leadership management, supervision and financial management in the health sector. SSDI-Systems Performance Evaluation 52 Zonal offices reported that SSDI has helped in capacity building at management levels and district levels. Senior managers were sent to the Malawi Institute of Management to undergo training focusing on leadership management. In addition, SSDI-Systems had seconded financial management staff at the zonal level to support districts, but these had to be withdrawn due to funding problems. It is felt that annual performance appraisals are done but not at the expected standard; they are done mainly at the central level, and there is no dialogue with the zonal, DHO and the central levels on career building. Training and coaching on the use of the MYP/DIP guidelines resulted in greatly improved district plans and ultimately into the national plan and budget. The project also supported the PDU through coaching. The time spent by our staff in the PDU has gradually decreased as we are phasing out. Conclusions SSDI has helped the districts with setting up of health sector clusters within the districts, where all sectors to do with health delivery are coordinated by the health stakeholders forum (committee). The project coordinated the training of DHMT staff in iHRIS and DHIS2. SSDI has provided DHMTs with the technical guidance and inputs on how to develop these DIPs. Trained DHMT leadership and management for six months at MIM and conducted coaching and mentorship to DHMT besides supporting monitoring their work plans. Inhibiting factors Facilitating factors Health policy functions At times indecisiveness by government, long approval procedures, still unclear distinction between policy, strategy, action plan and guidelines PDU is now acknowledged as focal department for policy analysis and development. Health financing functions Supportive supervision Use of data at central level Policy process improved as national development requires data/evidence during the development process Use of data at district level MYP/DIP guidelines necessitate districts to plan with data use Lack of training on MYP/DIP guidelines and access to HMIS/iHRIS data limit operational planning competence Capacity development at central level Capacity developed in PDU, NHA and CMED through coaching and training Staff rotation and uncoordinated technical assistance will negatively impact on sustainability of capacity development. Capacity development at district level Capacity developed in DHMTs through coaching and training Budget cuts required Abt to withdraw staff from zonal offices. SSDI-Systems Performance Evaluation 53 ANNEX 8: BIBLIOGRAPHY 1) SSDI-Systems. Proposal Document, 2010. 2) SSDI-Systems. Performance Monitoring Plan 2014. 3) SSDI-Systems. Revised Monitoring & Evaluation Plan, 2016. 4) SSDI-Systems. Year I quarter I report, 2012. 5) SSDI-Systems. Year I semiannual report, 2012. 6) SSDI-Systems. Year I quarter 3 report, 2012. 7) SSDI-Systems. Year I annual report, 2012. 8) SSDI-Systems. Year 2 quarter I report 2013. 9) SSDI-Systems. Year 2 semiannual report, 2013. 10) SSDI-Systems. Year 2 quarter 3 report, 2013. 11) SSDI-Systems. Year 2 annual report, 2013. 12) SSDI-Systems. Year 3 quarter I report, 2014. 13) SSDI-Systems. Year 3 semiannual report, 2014. 14) SSDI-Systems. Year 3 quarter 3 report, 2014. 15) SSDI-Systems. Year 3 annual report, 2014. 16) SSDI-Systems. Year 4 quarter I report, 2015. 17) SSDI-Systems. Year 4 semiannual report, 2015. 18) SSDI-Systems. Year 4 quarter 3 report, 2015 19) SSDI-Systems. Year 4 annual report, 2015. 20) SSDI-Systems. Year 5 quarter I report, 2016. 21) SSDI-Systems. Year 5 semiannual report, 2016. 22) SSDI-Systems. Program Booklet, 2016. 23) SSDI-Systems. Success Stories 24) SSDI-Systems. Policy Inventory and Assessment, 2013. 25) SSDI-Systems. Integrated Supportive Supervision Assessment Desk Review, 2012. 26) SSDI-Systems. Integrated Supportive Supervision Assessment Report, 2014. 27) SSDI-Systems. Performance Management Systems Assessment Report, 2014. 28) SSDI-Systems. Integrated Human Resources Information System Assessment Report, 2014. 29) SSDI-Systems. Leadership and Management Training Needs Assessment Report, 2013. 30) SSDI-Systems. District Financial Management Assessment Report, 2013. 31) SSDI-Systems. Health Financing Situation Analysis Report, 2012. 32) SSDI-Systems. Health Financing Strategy Technical Evaluation Report, 2013. 33) SSDI-Systems. Performance-Based Incentives Assessment Baseline, 2013. 34) SSDI-Systems. Performance-Based Incentives Feasibility Report, 2012. 35) SSDI-Systems. Draft Health Financing Strategy, 2014. 36) Ministry of Health. Malawi Health Sector Strategic Plan 2011-2016. U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523