October 2014 This publication was produced at the request of the United States Agency for International Development (USAID). It was prepared independently by International Business and Technical Consultants, Inc. (IBTCI). It was authored by Pinar Senlet, Chifundo Kachiza, Jennifer Katekaine, and Jennifer Peters. EVALUATION USAID/Malawi Support for Service Delivery – Integration Performance Evaluation USAID/Malawi Support for Service Delivery – Integration (SSD-I) Performance Evaluation Revised November 5, 2014 AID-RAN-I-00-09-00016 Cover Photo Credit: Jennifer Peters Caption: Women and children from SSD-I sites DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development (USAID) or the United States Government. ACKNOWLEDGMENTS The evaluation team is grateful to the more than 500 individuals who generously gave their time and insights for this report. Throughout this evaluation, the team benefited from the contributions of dozens of Ministry of Health (MOH) directors, managers and health care providers, as well as beneficiaries of the Essential Health Package (EHP) in Malawi. As such this report represents a collective effort and incorporates opinions and observations of a large community of individuals. The team would especially like to thank Chimwemwe Chitsulo, USAID/Malawi’s Monitoring, Evaluation and Learning Specialist and Contracting Officer’s Representative (COR), for his thoughtful guidance throughout the evaluation process. The team is also grateful to the implementing partners of Support for Service Delivery-Integration (SSD-I) project for making their staff and project information so readily available to the team. To all who are committed to improved health service delivery in Malawi, we offer these observations, analysis, and recommendations in the confidence that further progress will be realized through collective efforts. CONTENTS ACKNOWLEDGMENTS ................................................................................................................................................ 2 ACRONYMS ....................................................................................................................................................................... 5 EXECUTIVE SUMMARY .................................................................................................................................................. 1 Evaluation Purpose and Questions ........................................................................................................................... 1 Project Background ...................................................................................................................................................... 1 Evaluation Design, Methods and Limitations .......................................................................................................... 2 Findings, Conclusions and Recommendations........................................................................................................ 2 I. EVALUATION PURPOSE & EVALUATION QUESTIONS ................................................................................. 7 Evaluation Purpose ........................................................................................................................................................ 7 Evaluation Questions .................................................................................................................................................... 7 II. PROJECT BACKGROUND ....................................................................................................................................... 8 III. EVALUATION METHODS & LIMITATIONS ....................................................................................................... 9 IV. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS ...................................................................... 12 SSDI-SERVICES: FINDINGS, CONCLUSIONS AND RECOMMENDATIONS ......................................... 12 Summary Overview ................................................................................................................................................ 12 Accomplishments .................................................................................................................................................... 12 Constraints and Challenges .................................................................................................................................. 15 Conclusions .............................................................................................................................................................. 17 Recommendations .................................................................................................................................................. 17 SSDI-COMMUNICATIONS: FINDINGS, CONCLUSIONS AND RECOMMENDATIONS .................. 18 Summary Overview ................................................................................................................................................ 18 Accomplishments .................................................................................................................................................... 19 Constraints and Challenges .................................................................................................................................. 21 Conclusions .............................................................................................................................................................. 21 Recommendations .................................................................................................................................................. 22 SSDI-SYSTEMS: FINDINGS, CONCLUSIONS AND RECOMMENDATIONS .......................................... 22 Summary Overview ................................................................................................................................................ 22 Accomplishments .................................................................................................................................................... 23 Constraints and Challenges .................................................................................................................................. 25 Conclusions .............................................................................................................................................................. 27 Recommendations .................................................................................................................................................. 27 SSDI-OVERVIEW: FINDINGS, CONCLUSIONS AND RECOMMENDATIONS ..................................... 27 Accomplishments .................................................................................................................................................... 27 Constraints and Challenges .................................................................................................................................. 29 Conclusions .............................................................................................................................................................. 30 Recommendations for the Remaining Life of SSD-I ....................................................................................... 31 Recommendations for the Follow-On Project ................................................................................................ 31 ANNEXES ......................................................................................................................................................................... 32 Annex I: Evaluation Statement of Work ................................................................................................................ 33 Annex II: Evaluation Matrix ....................................................................................................................................... 45 Annex III: List of Documents Reviewed ................................................................................................................ 50 Annex IV: List of Key Informants Interviewed ..................................................................................................... 53 Annex V: Data Collection Instruments .................................................................................................................. 57 Annex VI: Informed Consent Statement ............................................................................................................... 66 Annex VII: SSDI-Services: Progress Against PMP Targets ................................................................................. 67 Annex VIII: Trend Analysis of SSD-I Indicators ................................................................................................... 68 Annex IX: SSDI-Services: List of Technical Interventions................................................................................. 73 Annex X: Findings from the Facility Mini-survey ................................................................................................. 76 Annex XI: Progress Towards PMP Indicators ...................................................................................................... 82 Annex XII: Focus Group Discussion Summary .................................................................................................... 83 Annex XIII: SSDI-Systems: Progress Against PMP Targets ............................................................................... 92 Annex XIV: SSDI-Systems: Zonal and District Implementation Table by Activity ...................................... 93 TABLES, FIGURES AND GRAPHS Table 1: Key enabling and limiting elements of the SSD-I approach......................................................................6 Table 2: Summary of Respondents by Methodology (KIIs, mini-survey, FGDs) ............................................... 10 Figure 1: SSD-I Target Districts and Evaluation Districts ........................................................................................ 9 Graph 1: Percent Coverage of Facility-Based Interventions (291 Facilities) ..................................................... 14 Graph 2: SSDI-Systems Coverage by Activity at the District Level .................................................................... 23 Graph 3: SSDI-Systems Coverage by Activity at the Zonal Level ........................................................................ 24 5 ACRONYMS ANC Antenatal Care AOR Agreement Officer Representative ART Antiretroviral Therapy BCC Behavior Change Communication BEmONC Basic Emergency Obstetrics and Newborn Care CA Cooperative Agreement CAG Community Action Group CBDC Community-based Distribution of Contraceptives CBMNC Community-based Maternal and Neonatal Care CDC Centers for Disease Control and Prevention CDCS Country Development Cooperation Strategy CHAM Christian Health Association of Malawi CMAM Community Management of Acute Malnutrition COP Chief of Party DHIS2 District Health Information System 2 DHMT District Health Management Team DHO District Health Officer DHS Demographic Health Survey DIP District Implementation Plan DQA Data Quality Assessment EGPAF Elizabeth Glazer Pediatric AIDS Foundation EHP Essential Healthcare Package EID Early Infant Diagnosis FANC Focused Antenatal Care FGD Focus Group Discussion FP Family Planning GOM Government of Malawi HBB Helping Babies Breathe HEU Health Education Unit 6 HIV/AIDS Human Immune Virus/Acquired Immune Deficiency Syndrome HMIS Health Management Information System HMS Helping Mothers Survive HPN Health Population Nutrition HRH Human Resources for Health HTC HIV Testing and Counseling iCCM Integrated Community Case Management IPT2 Intermittent Preventive Therapy (2nd dose) iHRIS Human Resource Information System JHU-CCP Johns Hopkins University Center for Communications Program KII Key Informant Interview KMC Kangaroo Mother Care IBTCI International Business & Technical Consultants, Inc. LAPM Long Action and Permanent Methods LB Live Births M&E Monitoring and Evaluation MAG Management Advisory Group MDG Millennium Development Goals MNCH Maternal, Neonatal and Child Health MOH Ministry of Health MSH Management Sciences for Health MVA Manual Vacuum Aspiration NGO Nongovernmental Organization NHA National Health Accounts NMCP National Malaria Control Program PBI Performance-based Incentives PDU Policy Development Unit PMP Performance Monitoring Plan PMS Performance Management System PPH Prevention of Postpartum Hemorrhage PQI Performance Quality Improvement 7 RFA Request for Application SBCC Social and Behavior Change Communication SSD-I Support for Service Delivery – Integration SWAp Sector-wide Approach TBA Traditional Birth Attendant TWG Technical Working Group USAID United States Agency for International Development VSO Voluntary Service Overseas WHO World Health Organization 1 EXECUTIVE SUMMARY EVALUATION PURPOSE AND QUESTIONS The purpose of this performance evaluation is to determine the effectiveness of the Support for Service Delivery – Integration (SSD-I) approach to increase availability and utilization of quality integrated Essential Healthcare Package (EHP) services and its performance in strengthening Malawi’s health system. The specific objectives of this performance evaluation are to measure, document and determine the extent to which SSD-I activities have contributed to: o Increased availability and utilization of EHP services (Sector 1); o Improved health promotion and adoption of normative health behaviors (Sector 2); and, o Improved functionality of the health system to support delivery of integrated health services (Sector 3). The body of the report responds to the specific objectives of the evaluation and concludes with an overview section on the SSD-I model. The following key questions guided this performance evaluation: 1. To what extent has SSD-I’s approach improved the delivery of quality EHP services and health indicators at SSD-I supported service delivery points? 2. To which extent has SSD-I’s health system strengthening and capacity building approach—at central, zonal and district levels—responded to specific bottlenecks that impact service delivery? 3. What effect has joint coordination and implementation of interventions across SSD-I had on the achievement of sector-specific and USAID’s health objectives? 4. To what extent has SSD-I’s approach responded to client needs and supported families in adopting improved preventative and health-seeking behaviors? 5. What elements of SSD-I’s approach have either enabled or limited its ability to improve the capacity of the government in the following areas: health service delivery, fiscal responsibility, management and leadership, behavior change communication (BCC), and use of data for decision-making; and what recommendations can be made for overcoming these bottlenecks in future? The findings, conclusions and recommendations for each are summarized below. PROJECT BACKGROUND Malawi scores very low on major health indicators: maternal mortality (675 maternal deaths per 100,000 live births (LB); under five mortality (112 deaths per 1,000 LB); and infant mortality (66 deaths per 1,000 LB) (Demographic Health Survey [DHS] 2010)1. Maternal morbidity and mortality are exacerbated by high fertility rates and high adolescent pregnancy rates. To reach Millennium Development Goal (MDG) targets, the Malawi government must: i) improve the quality and coverage of EHP services; ii) strengthen the performance of the health system to deliver EHP services; iii) reduce risk factors to health; and iv) improve equity and efficiency in the delivery of free, quality EHP services. To help address these issues, USAID/Malawi is funding three five-year activities that make up the SSD-I project launched in 2011. 1 Malawi 2010 Demographic and Health Survey (DHS). 2 SSD-I is USAID Malawi’s flagship health project and consists of three inter-related sector activities: SSDI￾Services (Sector 1), SSDI-Communications (Sector 2) and SSDI-Systems (Sector 3). In close collaboration with the Ministry of Health (MOH), SSD-I activities support the increased availability, quality and utilization of EHP services; reinforce health promotion and disease prevention among households; and strengthen elements of the health system to sustain effective EHP delivery. The SSD-I activities are designed to reduce fertility, lower the risk of human immuno-deficiency virus (HIV) infection, and reduce maternal, infant and under-five mortality rates over a five-year period (2011 – 2016). The activities are awarded and being implemented through three separate Cooperative Agreements. Although each implementing partner (IP) has a distinct scope of work and mandate, they are expected to collaborate and work together throughout the life of the project. All three SSD-I activities target the same 15 districts and are intended to complement and support each other. EVALUATION DESIGN, METHODS AND LIMITATIONS The evaluation was conducted by a team of five consultants and five research assistants between July 21 and October 10, 2014. It covered the period from project inception from September 2011 to June 2014. The evaluation used a mix of qualitative and quantitative data collection and analysis, including: document review; data review of key health service indicators, service statistics from intervention and control districts and progress towards selected Performance Monitoring Plan (PMP) indicators for the three activities. These included a total of 22 indicators; 93 key information interviews (KIIs); 63 facility mini￾surveys with health care providers; and 48 focus group discussions (FGDs) with 393 community beneficiaries and volunteers. Since non-probability sampling methods were used for selecting districts, communities and participants, the evaluation team was not able to generalize findings to the entire SSD-I Project and target population. The qualitative data was also subject to interviewer and respondent biases. The evaluation team mitigated interviewer bias by working in pairs to compare notes and analyze data as a group. FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Evaluation question 1: To what extent has SSD-I’s approach improved the delivery of quality EHP services and health indicators at SSD-I supported service delivery points? SSDI-Services has 82 routine output and outcome indicators in all program areas. The team selected 10 key outcome indicators and analyzed quarterly trends from project inception through March 20142. The activity has achieved or exceeded five of the 10 targets and is very close to achieving the other five. If these positive trends continue, SSDI-Services is likely to achieve all 10 in Years 4 and 5. SSDI-Services’ mandate was to assist the MOH to strengthen delivery of 15 high-impact interventions across the following six technical areas: maternal and newborn health; child health; family planning (FP) and reproductive health; HIV/AIDS and tuberculosis (TB); malaria; and nutrition. Once implementation started, the list grew to 30 priority interventions. In addition, the activity provided inputs in the following eight cross-cutting areas: capacity building; commodity facilitation; provision of equipment and 2 The evaluation used health service data to compare the progress-to-date across key service indicators. Service data for control sites from the project’s baseline provided comparative data as a proxy for determining project outcomes at midterm. Evaluators selected key output and outcome indicators from each activity’s PMP and documented progress on each indicator, based on: 1) relevance in responding to evaluation questions; 2) relevance of the indicators to the project’s main focus area; and 3) the ability to capture the value of SSD-I investments during the past three years. Health service data from the DHIS2 and data from SSD-I’s baseline survey for control sites were used to compare the progress to date across key service indicators. The team tracked and reported on selected SSD-I PMP indicators for the trend analysis. 3 supplies; infrastructural improvements; transportation support; logistics management; monitoring and evaluation (M&E); and Performance Based Incentives (PBI). Lists of interventions and the cross-cutting areas are found in Annex IX. The technical interventions have been implemented at varying levels of coverage across the 15 districts. SSDI-Services has made progress in training and capacity building of the health system. In total, SSDI￾Services supported the training of more than 18,000 health workers3 in technical areas as well as performance improvement tools. Mini-survey among the health service providers show that about 80% rated them “very useful”. Respondents felt that SSDI-Services support had increased efficiency (50%), access (60%) and quality (80%) of service delivery. Review of PMP data show that most service delivery targets have been achieved, and efficiency, access, quality and utilization of key EHP services has increased. However, design flaws, increasing demands and expansion of coverage has meant the achievement of both PMP and service indicators comes at the expense of MOH and District Health Officer (DHO) ownership, institutionalization and sustainability. The activity cannot feasibly be expected to achieve its entire indicators and hand over to the MOH in five years. It is not realistic to expect SSDI￾Services to effectively and fully implement all interventions across all districts, nor is it prudent to do so without prioritization of high-impact interventions. Key recommendations: 1. USAID should direct and assist the project to refocus on the key premises of the Request for Application (RFA). The team recommends prioritization of 15 high-impact interventions in the final two years of the project 2. SSDI-Services should work to revitalize the bottom-up, supportive nature of the relationship they initially shared with the District Health Management Teams (DHMTs) to increase ownership and likelihood of sustainability of interventions in the longer-term. Evaluation question 2: To what extent has SSD-I’s health system strengthening and capacity building approach—at central, zonal and district levels—responded to specific bottlenecks that impact service delivery? SSDI-Systems tracks 54 routine output and outcome indicators. The evaluation team selected six outcome indicators across the six result areas. (See footnote 2.) Of the six indicators selected, one has been achieved, four are partially achieved and one is unlikely to be achieved at the end of the project. The activity has strengthened the institutional capacity of the MOH to: develop evidence-based health policies; provide strategic leadership and management, and strengthen decentralized health services; revitalize plans and implement performance management system (PMS); track health expenditure and use health financing tools and improve district planning and financial management. SSDI-Systems and Services have worked jointly to upgrade the Health Management Information System (HMIS) to a web￾based District Health Information System 2 (DHIS2) for improved data management and to design and pilot a Performance-Based Incentive (PBI) scheme. SSDI-Systems has made great strides towards achieving its targets. However, as final approval of both policies and systems lies with the MOH and Government of Malawi (GOM), full achievement of many targets is beyond the activity’s or USAID’s control. As this is the beginning of the fourth year of implementation, it is unlikely that the activity will be able to scale-up all interventions in all 15 districts, and build adequate capacity with the MOH to institutionalize and sustain the interventions. 3 Includes health workers who were trained multiple times. 4 Key recommendations: 1. Align all partners with quarterly District Implementation Plan (DIP) reviews and generate district plans for systems interventions in consultation with DHMTs. 2. Prioritize completion and approval of the six policies underway and refrain from developing those in discussion or newly identified. 3. Where pilot activities are underway, focus on successfully completing and documenting best practices without further scale-up. Evaluation question 3: To what extent has SSD-I’s approach responded to client needs and supported families in adopting improved preventative and health-seeking behaviors? SSDI-Communications is on target for the majority of its PMP indicators; where it is not, it is the result of funding constraints4. The SSD-I project has built capacity at the central level (with private sector and MOH partners), district, and community levels (through training and capacity building of SSDI-Services and sub-grantees). Baseline research formed a strong evidence base for SSDI-Communications and the Health Education Unit (HEU) to strategically design and implement mass and community campaigns. The development of various policies, strategies, toolkits and campaign materials has been strategic and is likely to result in increased capacity within the MOH for future social and behavioral change communications (SBCC) activities. SSDI-Communications has been constrained by cuts in funding and the removal of community mobilization from their mandate. This has left SSDI-Services and SSDI-Communications in a complex situation regarding monitoring and evaluation (M&E), revision of messages and activities and subsequently, the attribution of both progress and constraints in their combined BCC efforts. Ultimately, it is only through the end line survey and service statistics that the impact of the combined efforts of SSDI-Communications and SSDI-Services can be measured. FGDs elicited many indications of positive progress: respondents across all group types and geographical locations mentioned increased access to a number of new and improved services. However, respondents also noted that demand had increased where services were not consistently available and requested increased availability of community-based services. Demand for essential health services has long outstripped supply in Malawi. USAID and SSD-I should think strategically and ensure careful targeting and tailoring of BCC efforts to create balanced demand to available health services. Key recommendations: 1. SSD-I project indicators should be revised to reflect the reduction in budget and share attribution of the impact of BCC activities across SSDI-Services and SSDI-Communications activities. 2. Increasing demand for services should align with the 15 high-impact interventions recommended to SSDI-Services and be carefully coordinated and integrated only where services are consistently available. Evaluation question 4: What effect has joint coordination and implementation of interventions across SSD-I had on the achievement of sector-specific and USAID’s health objectives? As a model to ensure close collaboration and coordination among activities with a common goal, it worked well. Per the contractual agreement cited above, the activities are doing their best to 4 As reported in SSDI-Communications Annual Reports for years 2011-12 and 2012-13. 5 collaborate and coordinate among themselves to maximize impact since they are cognizant of the fact that they have the same end goal. The SSD-I project follows the premise of USAID Malawi’s Country Development Cooperation Strategy (CDCS) 3-C Approach: Co-locating, Coordinating and Collaborating. The positive aspects and achievements of this close coordination and collaboration model include: o Integration of key services at the facility level; o Targeting of all SSD-I interventions in the same 15 districts, thereby co-locating services, communications and systems interventions; o Synchronization of the three activities’ start and end dates; o Coordination and “integration” of planning and leveraging of resources in the implementation of activities. The division of SSD-I into three activities at the design phase led to ambiguities about the roles of each activity and constraints to integrated implementation. Coordination and collaboration among SSD-I activities is not as strong as it could be due to overlaps, ambiguity, and gaps in the mandates among the three. Several areas were addressed with USAID during the early years of implementation. Ambiguities persist, including: strengthening of referral systems; improved data management and use; planning and financial support for DIPs; M&E of BCC initiatives; and supportive supervision with the DHMTs. The SSD-I project has the potential to leave lasting changes in the way EHP services are delivered in Malawi. One of the unique aspects of the SSD-I project is that it was designed to support service delivery, behavior change, and an enabling policy and systems environment in one program, implemented by three activities through close collaboration. This model is intended to holistically address the multi￾faceted challenges and achieve improved health outcomes. While the integration of focal EHP areas at the facility level has been successful, greater efforts are needed to ensure synergy and coordination among the three SSD-I activities. Key recommendations: 1. USAID can facilitate better coordination and synergies among the three activities by assigning a single coordinating point person who shares the larger vision of SSDI as a single program. The coordinator should chair the Management Advisory Group (MAG), with the mandate to develop a common vision among the three activities. 2. USAID, in consultation with the three activities, should amend the activities’ mandates to iron out remaining gaps, overlaps and ambiguities. 3. The three activities should have routine meetings among themselves to ensure that: a) activity work plans are complementary; b) resources are leveraged; and c) comparative advantages of each activity and implementing agency are maximized at the district level. Evaluation question 5: What elements of SSD-I’s approach have either enabled or limited its ability to improve the capacity of government in the following areas: health service delivery, fiscal responsibility, management and leadership, BCC, and use of data for decision-making; and what recommendations can be made for overcoming these bottlenecks in future? 6 Table 1. Key enabling and limiting elements of the SSD-I approach. Component Enabling Element Limiting Element Health service delivery Provision of training, best practices, equipment and supplies Lack of prioritization of high-impact interventions; Low MOH ownership of interventions. Fiscal responsibility Zonal Finance Coaches to support districts in financial management and reporting Lack of presence at the district level limits ability to coordinate key partners and stakeholders, such as the Christian Health Association of Malawi (CHAM) to contribute to the DIPs. Management and leadership Integration of mentorship and coaching alongside data for decision￾making into trainings DHMTs stretched by competing demands from three SSD-I projects and other implementing partners Data for decision-making Improved data reporting and management under web-based DHIS2. Supervisors are trained to review HMIS and supervision data for specific health facilities prior to supervision visits to provide context for the visit. Supportive supervision data is used by DHMT Supervisors to verify the monthly HMIS reports forwarded to DHIS2 and to identify ways to address challenges. Lack of integration of supportive supervision in DHIS2 and gaps in follow-up and mentorship to district HMIS officers. This will depend on expected changes to both the content of the supportive checklists and technology used to collate data, as the project integrates lessons learned from the initiative’s pilot stage. Behavior change communication (BCC) Strong evidence base and strategic design of mass and community campaigns Reduced funding and division of BCC component across two projects Key recommendations: 1. The follow-on project should: a. Be one Cooperative Agreement that houses the three components of service delivery, systems strengthening and SBCC. b. Focus on priority, high-impact interventions building on achievements and lessons learned from all three SSD-I activities. c. Within the mandate and technical focus of USAID and SSD-I, embrace a holistic, bottom-up and decentralized approach that is responsive to individualized district needs. 7 I. EVALUATION PURPOSE & EVALUATION QUESTIONS EVALUATION PURPOSE The purpose of this performance evaluation, as stated in the Scope of Work (Annex I: Evaluation Statement of Work), is to determine the effectiveness of the SSD-I approach is to increase availability and utilization of quality integrated Essential Healthcare Package (EHP) services and its performance in strengthening Malawi’s health system. The specific objectives of this performance evaluation are to measure, document and determine the extent to which SSD-I activities have: o Increased availability and utilization of EHP services (Sector 1); o Improved health promotion and adoption of normative health behaviors (Sector 2); and, o Improved functionality of the health system to support delivery of integrated health services (Sector 3). The target audiences for this evaluation include USAID/Malawi, Government of Malawi (GOM) and other stakeholders and the implementing partners; JHPIEGO, Johns Hopkins University Center for Communications Program (JHU-CCP) and Abt. Associates. Findings and recommendations from this evaluation will inform implementation of SSD-I over the remaining life of activity, as well as USAID/Malawi’s design of future investments in health. The results of this evaluation will contribute to the learning agenda on integrated health programming under the Global Health Initiative and the USAID/Malawi Mission’s Country Development Cooperation Strategy (CDCS). EVALUATION QUESTIONS The following key questions guided the performance evaluation, as outlined in the Scope of Work (Annex I): 1. To what extent has SSD-I’s approach improved the delivery of quality EHP services and health indicators at SSD-I supported service delivery points? 2. To what extent has SSD-I’s health system strengthening and capacity building approach—at the central, zonal and district levels—responded to specific bottlenecks that impact service delivery? 3. What effect has joint coordination and implementation of interventions across SSD-I had on the achievement of sector-specific and USAID’s health objectives? 4. To what extent has SSD-I’s approach responded to client needs and supported families in adopting improved preventative and health-seeking behaviors? 5. What elements of SSD-I’s approach have either enabled or limited its ability to improve the capacity of the government in the following areas: health service delivery, fiscal responsibility, management and leadership, behavior change communication (BCC), and use of data for decision-making; and what recommendations can be made for overcoming these bottlenecks in future? 8 II. PROJECT BACKGROUND Malawi health statistics reflect challenges at the household, community and health system levels to protect against HIV/AIDS, malaria, maternal and child morbidity and mortality. Malawi scores very low on major health indicators: maternal mortality (675 maternal deaths per 100,000 live birth (LB)); under five mortality (112 deaths per 1,000 LB); and infant mortality (66 deaths per 1,000 LB) (DHS 2010)5. Maternal morbidity and mortality is exacerbated by high fertility rates and high adolescent pregnancy rates. More than one in five adolescent girls begin bearing children by age 17, a major health concern because of the increased risks of death and disability to both mother and child during pregnancy and childbirth. Though HIV prevalence among adults has stabilized around 10%, it is 30% higher among women and twice as high among urban residents. With 1.1 million people living with HIV, Malawi ranks ninth in the world. Sadly, the average Malawian lives 54 years on average (Population Reference Bureau 2013)6. Together these statistics more than justify USAID’s health investment in Malawi. To reach Millennium Development Goal (MDG) targets, the Malawi government must: i) improve the quality and coverage of EHP services; ii) strengthen the performance of the health system to deliver EHP services; iii) reduce risk factors to health; and iv) improve equity and efficiency in the delivery of free, quality EHP services. The attainment of these objectives will contribute to poverty reduction and the socio-economic development of Malawians. To help address these issues, USAID/Malawi is funding three five-year projects that make up the SSD-I project launched in 2011. SSD-I is USAID Malawi’s flagship health project and consists of three inter-related sector activities: SSDI￾Services (Sector 1), SSDI-Communications (Sector 2) and SSDI-Systems (Sector 3). In close collaboration with the Ministry of Health (MOH), SSD-I activities support the increased availability and quality of the EHP services; reinforce health promotion and disease prevention among households; and strengthen elements of the health system to sustain effective EHP delivery. The SSD-I project was designed to reduce fertility, lower the risk of HIV, and reduce maternal, infant and under-five mortality rates during a five-year period (2011 – 2016). The activities were awarded and are being implemented through three separate Cooperative Agreements (CAs). Although each implementer has a distinct scope of work and mandate, they are expected to collaborate and work together throughout the life of the project. All three SSD-I projects target the same 15 districts and are intended to complement and support each other. 5 Malawi 2010 Demographic and Health Survey. 6 Population Reference Bureau, 2013 World Population Data Sheet. 9 III. EVALUATION METHODS & LIMITATIONS The evaluation was conducted between July 21, 2014 and October 10, 2014 and covered the period from the SSD-I project inception in September 2011 through the end of June 2014. The evaluation team included five consultants: Pinar Senlet, Team Leader; Chifundo Rose Kachiza, Public Health Specialist; Jennifer Peters, Social Behavior Change Specialist; Jennifer Kaahwa Katekaine, Health Systems Specialist; and James Kaphuka, Research Analyst. In addition to the core team, five Research Assistants (RAs) were hired to assist with data collection in the field. A team planning meeting was conducted in Lilongwe to develop the evaluation methodology (Annex II: Evaluation Matrix) and data collection tools, and begin pre-testing and training of RAs prior to start of data collection. The findings of evaluation were presented to the SSD-I stakeholders in a Findings Workshop held on September 26, 2014. The final evaluation report was submitted to USAID/Malawi on November 5, 2014. Study design: The evaluation used a mixed methodology with qualitative and quantitative data collection. Selection of districts, health facilities and communities The evaluation selected eight out of the 15 SSD-I intervention districts from each of the five health zones. Figure 1 shows the SSD-I target districts (in blue) and the eight districts selected for the evaluation (in yellow). District selection was based on the timeframe and coverage of the project activities, geographic and population representation, and the availability of baseline and comparative data. In each district, three facilities (the district hospital, one urban health center and one rural health center) were selected on the following basis: 1) time and scope of SSD-I projects interventions; 2) geographical (urban/rural); and 3) population representation. Two communities, one urban and one rural, were selected within the catchment areas of the health centers for FGDs. Three FGDs were conducted in each area: one with beneficiary women, one with beneficiary men, and one with community volunteers. Selection of districts, health facilities and communities was finalized in consultation with and guidance from USAID/Malawi and the SSD-I activities. Table 1 shows the total KIIs, mini-surveys and FGDs conducted at all levels. Figure 1: SSD-I Target Districts and Evaluation Districts 10 Table 1: Summary of Respondents by Methodology (KIIs, mini-survey, FGDs) Stakeholders KIIs USAID Agreement Officer Representatives (AORs) and Health Population Nutrition (HPN) management 6 MOH staff, central level 23 SSD-I staff, central level 10 Donors and implementing partners (IPs) 4 MOH zonal staff and DHMTs 24 SSD-I zonal and district staff 26 Facility Level Mini-Survey District hospital (8 total) 19 Urban health center (8 total) 21 Rural health center (8 total) 23 Beneficiaries and Volunteers FGDs Urban communities (24 total) 195 Rural communities (24 total) 198 Total, all activities 549 Key data sources The following data sources were used to collect data to address the evaluation objectives and questions: o Background documents. The evaluation team reviewed documents indicated in the Request for Task Order Proposal (RFTOP) and other documents identified by USAID and the SSD-I project prior to arriving in the country. A qualitative analysis of these documents was conducted to identify themes and triangulate with other data collected and analyzed as part of this evaluation (Annex III: List of Documents Reviewed). o Secondary data analysis. First, the evaluation used health service data to compare the progress to date across key service indicators. Service data for control sites from the project’s baseline was used to provide comparative data as a proxy for determining project outcomes at midterm. Second, the evaluation selected key output and outcome indicators from each activity’s Performance Management Plan (PMP) and documented progress on each indicator. The selection of PMP indicators was based on: 1) the relevance in responding to the evaluation questions; 2) the relevance of the indicators to the activity’s main focus area; and 3) the ability to capture the value of SSD-I investments over the past three years. Health service data from the DHIS2 and data from SSD-I’s baseline survey for control sites were used to compare the progress to date across key service indicators. In addition, the team tracked and reported on selected SSD-I PMP indicators for the trend analysis. 11 o Focus group discussions. Information about perceptions of current and past (during the last two years) services and household health practices among targeted community members were assessed through FGDs. Criteria for selecting communities included: i) duration of the SSD-I’s presence in the community; ii) project baseline data; and iii) equal proportion of urban and rural communities. In each district, RAs facilitated three FGDs among selected SSD-I beneficiaries (women of reproductive age, men/partners and community volunteers). Communities in which FGDs were held were mapped using GPS technology. GPS coordinates were collected at each site. The RAs were trained in conducting the FGDs and use of GPS technology prior to the field trips. o Key informant interviews. KIIs were conducted with MOH staff at all levels, USAID/Malawi staff, the SSD-I staff, health providers at the facility and community levels, policy and decision￾makers and other key stakeholders at the central, zonal and district levels. o Facility mini-surveys. A cross-sectional structured facility mini-survey was conducted with service providers and managers at the facilities visited by the team. The purpose of the mini￾survey was to obtain insights into perceptions regarding the availability, access, and use of EHP services. In addition, the aim was to assess changes in the capacity of service providers and facilities in providing quality EHP services during the past three years. All data collection tools were pre-tested, revised and finalized prior to data collection (Annex IV: List of Key Informants Interviewed and Annex V: Data Collection Instruments). Data analysis: The evaluation team conducted thematic analysis of data obtained through KIIs and FGDs. Content analysis of project documents and trend analysis was performed on PMP and service indicator data. Mini-survey data was analyzed using STATA. Ethical considerations Verbal and written consent from the mini-survey participants were obtained according to guidelines outlined in the IBTCI Ethical Standards and Protocols for Field Research standard operations procedure. The information provided in this report is not linked to any specific person, and all data from interviews and discussions is kept confidential and used for evaluation purposes only (Annex VI: Informed Consent Statement). Limitations of evaluation methods Since non-probability sampling methods were used for selecting districts, communities and participants, the evaluation team is not able to generate findings that statistically represent the larger population from which they were drawn. Second, since the key informants constitute the primary source of information, the interview data may be subject to personal biases, opinions and recollection. The evaluation team mitigated interviewer bias by meeting frequently to compare transcripts and analyze the data as a group. 12 IV. FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS SSD-I SERVICES: FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Summary Overview SSDI-Services is a five-year activity (November 2011 – November 2016) with a funding level of $65 million. The purpose of the activity is to implement an integrated service delivery program to expand and improve the quality of selected EHP services at the facility and community levels. The SSDI-Services program description stated that “the Project will be responsible for scaled up and integrated priority EHP service delivery within primary health care clinics as well as referral levels in two to three targeted districts in each of the five zones of Malawi so as to cover a total population of approximately 6,600,000” Initially, SSDI-Services began in 10 districts. The activity was expanded to 15 districts at the end of Year 1. SSDI-Services supports the MOH in six technical areas (maternal and newborn health, child health, family planning and reproductive health, HIV/AIDS and tuberculosis, nutrition and malaria) included in the 13 focal areas of the EHP7: The activity implements a range of interventions at the facility and community levels, in coordination with the central MOH, Zonal Health Offices and DHMTs. The activity is being implemented by a consortium comprising: JHPIEGO, CARE International, Plan International, and Save the Children International. JHPIEGO is the prime and technical lead in maternal and reproductive health, HIV/AIDS and malaria, and works in one district (Lilongwe) in the Central West Zone. JHPIEGO is also the prime and technical lead for PBI implementation in the SSD-I consortium. Save the Children is the technical lead in newborn, child health, nutrition and community￾based services and works in eight districts in the South East and South West Zones. CARE International works in four districts in the Central East Zone, and Plan International is in two districts in the Northern Zone. SSDI-Services is headquartered in Lilongwe with 12 core staff including seven technical team members. The project has five zonal and 15 district-level offices. In the zones, the activity has four staff (Zonal Manager, Technical Specialist, M&E Specialist and Data Management Assistant), while at the district level it has four staff members: the District Team Leader, Clinical Coordinator, Community Coordinator and M&E Coordinator. The District Team Leaders report to the Zonal Manager, who in turn reports to the Deputy Chief of Party. There are no technical specialists on the district teams. District teams are intended to receive technical assistance and support from the central team as required and upon request. Accomplishments Activity indicators and service statistics The activity has a robust PMP and currently reports on 82 routine output and outcome indicators in all program areas. The activity’s PMP started with 49 indicators, which increased to 82 in the second year. The team selected 10 key outcome indicators and analyzed quarterly trends from activity’s inception 7 The MOH lists 13 priority EHP areas in the “Health Sector Strategic Plan, 2011-2016”: page 41. 13 through March 2014, using the criteria defined under footnote #2. Progress was tracked against targets set for September 2013, the end of the activity’s second implementation year. The activity has achieved or exceeded five of the 10 targets, and is very close to achieving the other five. If these positive trends continue, we believe the activity will achieve all 10 in Years 4 and 5. These are considerable achievements (Annex VII: SSDI-Services: Progress Against PMP Targets; Graphs on individual indicator trends are found in Annex VIII: Trends Analysis of SSD-I Indicators). In 2012, the activity conducted a comprehensive situation analysis of selected facilities in the 15 intervention districts and four control (non-SSD-I) districts. A total of 63 facilities in intervention districts and 19 facilities in control districts were sampled. Information collected included: infrastructure and facility conditions, human resources, technical skills, existing interventions, equipment, supplies and available service statistics. Based on the findings of the baseline survey, SSD-I and DHMT members jointly identified gaps and needs and decided upon key areas of support for the activity. The evaluation team compared service data between intervention and control sites from baseline (Oct 2010 – Sept 2011) and midterm (April 2013 – March 2014) for six indicators routinely collected by the DHIS2. The comparative analysis of baseline and midterm data revealed mixed results. The analysis is not very helpful in determining the activity’s achievements, as “control facilities” are not really non￾interventions sites. The CDC has been providing assistance to these districts through the Elizabeth Glazer Pediatric AIDS Foundation (EGPAF) and Management Sciences for Health (MSH) since 2013, with an emphasis on HIV/AIDS. Technical and cross-cutting interventions Initially, the activity’s mandate was to assist the MOH to strengthen the delivery of 15 high-impact interventions across the technical areas, including: Long Acting and Permanent Methods (LAPM), Community-Based Distribution of Contraceptives (CBD), Basic Emergency Obstetric and Newborn Care (BEmONC), Focused Antenatal Care (FANC), Helping Babies Breathe (HBB), Kangaroo Mother Care (KMC), Community-Based Maternal and Neonatal Care (CBMNC), Helping Mothers Survive (HMS), Prevention of Postpartum Hemorrhage (PPH), Initiation of HIV+ Pregnant Women on ART, HIV Testing and Counseling (HTC), Early Infant Diagnosis (EID), Integrated Community Case Management (iCCM), Intermittent Prevention Therapy (IPT2), and Community Management of Acute Malnutrition (CMAM). Once implementation started, both USAID and the MOH requested inclusion of additional interventions and the list grew to 30 interventions. The team identified 30 technical interventions currently being implemented, monitored and reported in the focal areas. In addition, the activity provides inputs in eight crosscutting areas (Annex IX: SSDI-Services: List of Technical Interventions). The technical interventions are implemented at varying levels of coverage across the 15 districts. Graph 1 shows the percent coverage of facility-based interventions (the total number of facilities in the 15 districts is 291)8. It should be noted that BEmONC, a high-priority intervention, is not included in this graph as only selected facilities are designated by the MOH to provide BEmONC services. Of the 291 facilities analyzed, 103 are designated by the MOH to be BEmONC sites and, of these, 90 facilities (87%) have been supported to provide BEmONC functions. Also, iCCM is implemented in village clinics in hard-to￾reach areas. Due to the increased number of technical interventions, key informants from the project, USAID and the DHMTs felt the activity had been stretched too thin. As a result, many interventions are implemented at varying levels of coverage, intensity and depth. In more than half of the districts, DHMTs also felt the activity was left in a difficult situation, as SSD-I district teams were under increasing 8 The evaluation team was not able to conduct a similar analysis for the community-based interventions. 14 pressure to achieve indicators that at times came at the expense of the close collaboration and supportive function these teams had initially developed with the DHMTs. Graph 1: Percent Coverage of Facility-Based Interventions (291 Facilities) Community mobilization activities under SSDI-Services are implemented by 10 local NGO sub-grantees. These NGOs were awarded sub-grants to implement community mobilization components in each of the 15 districts. Initiation of community mobilization activities was delayed until 2013, due to reasons beyond the activity’s control. The sub-grantees are using the Community Action Cycle and Community Scorecard approaches working with Community Action Groups and through existing community structures. The activity believes increased service utilization is the best indicator of success of community mobilization efforts and we agree. (Please see the SSDI-Communications section below for more on activity-wide BCC activities.) In total, SSDI-Services supported the training of more than 18,000 health workers9 in technical areas as well as performance improvement tools. A mini￾survey was used to assess usefulness and effectiveness of training and support. More than 66% of respondents were male with 33.3% being female health workers. These included clinical officers and medical assistants (26.9%, nurse 9 Includes health workers who were trained multiple times. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% EC LAPM PPFP PAC YFS HBB FANC IMCI iCCM HMIS ETAT KMC PPH CCS EID PMTCT ART PITC HTC Percentage Covered Uncovered “Coaching and mentorship has been most effective and is likely to be sustainable. The quality of data has greatly improved through the rolling out the DHIS tool in all districts.” KII with Zonal MOH Officer Family Planning/Reproductive Health Maternal, Neonatal and Child Health HIV/AIDS 15 midwives (39.6%), health surveillance assistants (17%) and other (15.8%). The results of the mini-survey found that 80% of service providers found training and support “very useful”. Examples of the usefulness of trainings included: new skill sets such as manual vacuum aspiration (MVA) and LAPM, multi-skilling10 and increased coverage of iCCM. Half (50%) of the respondents felt that SSDI-Services support had increased efficiency in areas such as reduced waiting times, service integration and multi-skilling. Sixty percent felt access increased due to task-shifting, geographic coverage, types of services, and an increase in outreach services. Quality improvements were noted by 80% of respondents, for improvements in attitudes of health workers, adherence to standards and protocols, and patient care (Annex X: Findings from the Facility Mini-Survey). Nearly all respondents requested more equipment, supplies, infrastructural improvements, assistance with transport, vehicle maintenance, and reported that lack of basic infrastructure and supplies were required to support the delivery of quality service. SSDI-Services also provided significant leadership and contribution to DHIS2 roll-out nationally and in all 15 districts through capacity-building of HMIS Officers, CMED staff, and Program Coordinators, as well as on-going technical support at facility, district and zonal level. The activity also undertook procurement and installation of computer and DHIS hardware as well as furniture for the District HMIS offices. Overall, KIIs and the mini-survey found that SSDI-Services has successfully improved the technical capacity of health workers through training, mentoring, supervision and provision of protocols, guidelines and performance quality improvement (PQI). The functionality of health facilities has also improved, though to a lesser extent, through the provision of equipment and supplies, minor infrastructural improvements and commodity facilitation11. The project has been successful in integrating service delivery at the facility level. All of the KIs interviewed supported the integration of focal areas12. This integration of services aligns with the health sector program goals. Constraints and Challenges Activity design The activity was designed when Sector-Wide Approach (SWAp) financing was in place and assisting the MOH to overcome constraints in their financial and human resources and infrastructure. The loss of SWAp financing, alongside additional request from the MOH and USAID to increase both technical interventions and geographic scale during the first two years of the activity further challenged the activity. While all 30 interventions being implemented are internationally recognized as best practices, it is not realistic to focus on all of them, especially given the well-known shortage of staff, inadequate supplies/commodities, inadequate infrastructure, and budgetary constraints of the MOH. USAID approval for refurbishments and equipment purchase is sometimes slow, and the activity’s funding was deemed insufficient by all DHMTs and health care workers (per the mini-survey results) to support all of the interventions. One hundred percent of KIIs with DHMTs and mini-surveys with health care workers found an imbalance between the increase in technical capacity and training and the inputs 10 Defined as: The training of a single health care worker in multiple skill-sets. 11 “Commodity facilitation” is the term used by SSDI-Services to describe assistance with commodities transport and logistics from district to facility level. 12 The only exception was the National Malaria Control Program (NMCP), which has a vertical program and intends to remain as such. “The DHO is extremely pleased with SSD-I support. The technical areas chosen for interventions were a good fit. Without USAID and SSD-I support, we would be crippled.” KII with District Health Officer “Do fewer things – just to do them fully so we can have results.” KII, DHMT member 16 for infrastructure, supplies, materials and refurbishments. Refurbishments are “patchy”13 and spread across so many centers that they are unable to bring all facilities up to standard. The activity trained a large number of staff with the assumption that if staff capacity were to be high and demand created in communities, this would be a sufficient number of staff to provide quality services. If the infrastructure is not up to standard, and supplies and materials and equipment are inadequate, the improvement of quality, accessibility and utilization of the services will be limited. Implementation challenges Nearly two-thirds of KIIs with DHMTs indicated that while the activity began with a very close collaborative and bottom up approach (e.g. the facility baseline survey and joint identification of gaps to be addressed). As the project was stretched during Years 2 and 3, it became increasingly challenging for them to achieve all that was asked and maintain the same approach. In more than half of the districts visited, DHMTs reported weaker communication and collaboration, and at times, inadequate involvement of the DHMT in the project interventions. Of the eight DHMTs interviewed, two remain highly satisfied with their working relationship with SSD-I. Evaluators received mixed responses from a further three DHMTs, and three expressed discontent with their current relationship with the activity. Based on numerous KIIs in the field, the evaluation team’s assessment of these issues is as follows: SSDI-Services is under pressure to implement a broad range of interventions and show results within a limited timeframe. Given the shortage of staff and resources within the DHMTs at times, in order to speed up implementation and achieve expected results, the activity is forced to plan and implement interventions on its own. In some settings, SSD-I is implementing interventions on behalf of and not alongside the MOH, thus compromising the stakeholders’ ownership of the activity interventions. As a result, MOH ownership in some districts is low, which raises concerns about institutionalization, and ultimately, the sustainability of the interventions once the activity comes to an end. Based on interviews with the project’s field staff and the team’s observations in the districts, the activity has a centralized structure, and with the increase in interventions, has adopted a top-down approach in the planning and implementation of so many programs. This approach does not always match the decentralized policy and nature of the MOH. District staff report that pressure from the central level comes from each of the six technical team members who do not view a district holistically or coordinate as well as they might amongst themselves. Since all activity interventions are implemented in the districts, one would expect that the core headquarters team would work in tandem with the districts, not just to supervise but rather to facilitate implementation. While district teams are tasked with the coordination and integration of SSD-I activities and targets into the DIPs, pressure to implement a number of initiatives and the supply of equipment for districts is often decided upon at headquarters and sent to the districts. The individual needs of the districts and facilities are not always taken into consideration. 13 KII with one DHMT. “If you want to walk fast, go alone. If you want to go far, go together.” African proverb from KII at USAID “There are more people both at the central and zonal levels who are supervising a small team at district level [which] has many more responsibilities.” KII SSD-I district staff “It is a one size fits all approach, it doesn’t always work.” KII DHMT member 17 Per the terms of the RFA14, implementation of district level activities is not apportioned to the partners according to their technical areas of expertise. For example, JHPIEGO, whose strength is in clinical care and service delivery, is only responsible for implementation in one district, while Save the Children, CARE and Plan, whose expertise lies in community mobilization, are implementing the full basket of facility and community-based interventions in the other 14 districts. Thus SSDI-Services implementation design does not seem to draw on the consortium member’s ideal mix of technical strengths to complement each other. Conclusions SSDI-Services has made significant progress in training and capacity building of the health system at district, facility and community levels. They are on target for the majority of their indicators and efficiency, access, quality and utilization of key EHP services has increased. However, design flaws, increasing demands and expansion of coverage (both in terms of number of interventions and geographical coverage) has meant the achievement of both PMP and service indicators comes at the expense of MOH and DHO ownership, institutionalization and sustainability. The activity cannot feasibly be expected to achieve its entire indicators and hand over to the MOH in five years. In its final two years, prioritization of coverage of high-impact interventions is likely to result in improved and sustained health outcomes over the longer–term. Recommendations 1. USAID and SSDI-Services should focus on high-impact priority interventions in the final two years of the activity. The evaluation team proposes that the activity focus on the following best practices and areas in which the project has already made a lot of progress. They are areas repeatedly listed from all cadres of health staff as success stories and are most likely to provide the highest health impact as a result. o Family planning and reproductive health: 1) LAPM, 2) CBD o Maternal, neonatal and child health: 3) BEmONC, 4) FANC, 5) HBB, 6) KMC, 7) CBMNC, 8) HMS 9) PPH o HIV/AIDS: 10) Initiation of HIV pregnant women on ART, 11) HTC, 12) EID o Nutrition: 13) CMAM o Malaria: 14) iCCM 15) IPT2 Uptake USAID should direct and assist the activity to refocus on the key premises of the RFA. According to the RFA, USAID’s Strategic Outcomes and Results included: “Comprehensive delivery of key selected, high-impact interventions – at both community and facility level, and with particular focus on family planning and interventions to reduce maternal and newborn mortality” (Page 11). As for saturation or coverage, the RFA states: “To ensure impact at the district level, [the project should] provide technical and implementation assistance for full coverage (“saturation” or at least 80% of traditional authorities) of target districts, rather than spread activities thinly over many districts.” Furthermore, the RFA states: “The recipient should consider how they will phase implementation so as to achieve coverage and impact in the targeted districts … keep appropriate focus on priority services, and build capacity of MOH staff and local, indigenous partners to assure medium to long term sustainability of community level impact” (page 16). 14 RFA section “SSD Geographic Coverage/Award Structure”: page 17. 18 2. Community mobilization interventions should continue as planned. However, the project should work (through improved M&E structures and in close coordination with SSDI-Communications) to insure that community mobilization efforts are closely aligned with the availability of services. 3. At the end of the activity, SSDI-Services should document the progress, best practices and lessons learned in each of the selected interventions and provide an in-depth assessment on what has worked and what has not worked for improved planning in the future. 4. SSDI-Services should work more closely with all DHMTs to provide a more supportive, responsive and flexible bottom-up approach adapted to the needs on the ground. Work planning should begin at the district and move up to the headquarters, rather than from a top down, “one size fits all” approach. This approach may slow down the implementation of interventions and achievement of some targets. Nonetheless, USAID and the project should prioritize MOH ownership and increased likelihood of sustainability in the long run over unsustainable gains in health service delivery in the short run. 5. SSDI-Services should adopt a similar responsive, facilitative and decentralized approach with its district teams. The activity should institute decentralized systems and structures to support district level planning, technical support, resource allocation and logistics. The central office should be more flexible in supporting the needs of DHOs. Technical advisors should coordinate better to assist districts in achieving all objectives, not just those in their technical area. SSDI-COMMUNICATIONS: FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Summary Overview SSDI-Communications is a five-year (September 2011 – September 2016) SBCC activity with a funding ceiling of $24,200,000. USAID awarded this activity to the Johns Hopkins Bloomberg School of Public Health Center for Communications Program (JHU-CCP) in partnership with Save the Children International and a number of local organizations (including private sector media groups and local non￾profits). JHU-CCP is the prime and takes the lead on formative research, SBCC, mass media and community campaign development. They work closely with SSDI-Services technical heads who review and comment on all messages and materials in development. Materials are distributed through SSDI￾Services community mobilization sub-grantees. Save the Children is responsible for technical input in community mobilization. They led the development of the community mobilization strategy and toolkits for implementing the Community Action Cycle. At district level, Save the Children undertook all training of the SSDI-Services Community Coordinators and sub-grantee staff. They also provided ongoing technical support to SSDI-Services and sub-grantees as well as supervisory visits in the communities (though at a lower level than planned due to funding constraints). At the central level, SSDI-Communications has 12 core program staff including two senior technical officers. They also have two zonal officers in each health zone: one SBCC Coordinator and one Community Mobilization Coordinator. The activity’s geographic focus is two-fold: 1) nationwide coverage through mass media, campaigns and capacity building; and 2) technical assistance and coordination of implementation of community level activities (in partnership with SSDI-Services) in the 15 focal districts of the SSD-I platform. SSDI-Communications aims to support SBCC by delivering a comprehensive package of technical assistance to the GOM and local community mobilization partners of SSDI-Services. The activity has four objectives: 19 o To strengthen the national and targeted district level SBCC planning and coordination; o To develop and produce SBCC packages implemented through a multi-level, multi-media campaign; o To build capacity of key national and district level partners; o To identify best practices for SBCC implementation. SSDI-Communications supports the National Health Sector Strategic Plan’s goal to “improve the quality of life and reduce risk of ill health and occurrence of premature deaths”. Health promotion is a key pillar of the national strategic plan and places particular emphasis on reducing risk factors to health through effective preventive practices and health-seeking behaviors. The PMP includes 35 output and outcome indicators (Annex XI: SSDI-Communications: Progress toward PMP Indicators). Accomplishments Planning, coordination and capacity building SSDI-Communications assisted the Health Education Unit (HEU) of the MOH to finalize the National Communications Policy. This document had been drafted and revised a number of times but never finalized. With help from the activity, this was finally completed. In addition, working closely with the HEU, the activity developed its own internal SSDI-Communications Strategy based on findings from the activity’s baseline assessment and formative research. As SSDI-Communications implements only a portion of the full EHP focal areas, training, coordination, close working collaboration with the HEU provided the department with sufficient skills and capacity to build on the internal strategy to develop an MOH umbrella strategy for all of the EHP areas15. SSDI-Communications also developed a Community Mobilization Strategy to guide the community mobilization activities under SSDI-Services and sub￾grantees in reaching out to households. The activity helped to revitalize the Communications Technical Working Group (TWG), which assisted the HEU to better coordinate all BCC activities of its various implementing partners. The activity also seconded one SBCC Technical Assistant to the HEU, who provided technical assistance on a day-to-day basis. Technical assistance in knowledge management— specifically in the development of M&E tools and a database to house all BCC materials—was also provided. The development process and final strategic document is noteworthy. At the outset of the SSD-I activity, there were myriad health communications strategies without an overarching document covering all of the EHP areas. The HEU did not have the human or other resources required to pull these together into one unifying package. In addition, it was beyond the mandate of SSDI-Communications to undertake this task. To address this challenge, the activity worked closely with the HEU to develop a strategic document for the activity’s six focal areas, and ensured that HEU skill sets and capacity were strengthened. As a result, HEU personnel were able to utilize this experience and the initial strategic document to guide them in developing a national strategic plan covering all of the EHP areas. SSDI-Communications undertook training in a number of areas for a number of different cadres. At national level, they trained staff at the HEU, journalists and private media partners. At the district and community levels, they led a training of 79 trainers for SSDI-Services and sub-grantee staff. Through the partner Save the Children International, SSDI-Communications provided technical training, some oversight and supervision (although limited due to funding constraints), and is also now working with SSDI-Services to devise an M&E plan for community activities. 15 The project assisted with the development of a Malaria Communications Strategy at the bequest of the NMCP, who wanted a standalone document for all their activities. 20 Research and SBCC materials and campaign In 2012, SSDI-Communications conducted baseline research using a quantitative survey and FGDs, with comprehensive findings and recommendations across all six EHP areas. Findings from the baseline research, prior experience in the Bridge HIV program and other national research comprised the evidence base from which all BCC messages and materials were developed. SSDI-Communications developed a “Moyo ndi Mpamba” (Life is Precious) platform from which a variety of positive, preventive health messages were disseminated using both multi-media and interpersonal channels. To date, the campaign has included: radio spots campaigns and two radio programs (drama and magazine), BCC print materials, a billboard campaign, and zonal community media activities (in Year 1); community drama (in Year 3); a marriage counselor initiative (in Years 3 and 4); and road shows. The combined BCC efforts and activities reached two million individuals through 8,124 radio spots; 88,357 individuals through community theater; and one million individuals through print materials, including posters. The activity is also in the process of targeting all households in one traditional authority of each target district with family health booklets, which include a variety of key messages on healthy practices across all six EHPs, and newlyweds with a marriage counseling guide, which offers key messages on family planning, safer sex, pregnancy and childbirth. Community perspectives from FGDs The evaluation team conducted 48 FGDs with beneficiary women, men and community volunteers in both urban and rural settings to assess changes in service availability (at both community and facility levels) as well as changes in individual, family and community behavior in the first half of the activity (Annex XII: Focus Group Discussion Summary). The FGDs elicited many indications of positive progress, as well as constraints and areas where more is needed. Of note, respondents across all group types and geographic locations mentioned increased access to a number of new or improved services including: long-term family planning, ARVs, iCCM, CMAM and nutritional information and services. Overall, FGDs found increased understanding of the importance of many new interventions, and a resulting increase in felt need and demand. Qualitative data from FGDs with both men and women shows that the primary concerns at community level, with 87.5% having reported HIV/AIDS, Family Planning, malaria, and bed nets as their main concern. Men and women in 93.7% of FGDs reported having heard health messages related to family planning, child spacing, food preparation, HIV counseling and testing, nutrition education, and the importance of using bed nets and building pit latrines. Health behavior changes such as couple HTC, husbands accompanying their wives to family planning, and/or ANC and men’s use of condoms in 56.2% of FGDs conducted with men and women. In all FGDs conducted with beneficiaries (100%), the main sources of health messages were HSAs, radio, and health care workers. There were a number of reports, among both men and women that “fewer people were dying of HIV/AIDS”, “gender-based violence had declined, and men were accompanying their wives to antenatal care and family planning.” However, these gains were offset by complaints that demand had increased where services were not consistently available. While respondents recognized and appreciated the increased numbers of trained health care providers in recent years, they felt that training sometimes came at the expense of consistent access to care (as providers were often away and not available when they went to their nearest clinic). This combined with inconsistent availability of supplies (e.g. malaria medicines and both long and short term FP methods) and basic “Demand for family planning services has increased; for instance, my register shows an increase from about 36 last year to almost 80 clients at this time.” FGD with CAG volunteers “As a community we are worried because … We have transport challenges when a woman goes into labor, especially at night. And yet the health facility is far. What plans does the government have to assist in this area?” FGD with female beneficiaries 21 equipment or inadequate infrastructure, resulted in frustration and was counter-productive in increasing confidence in consistent access to quality EHP services. Respondents (of all types) in districts in Central and Southern zones mentioned the new policy regarding facility-based birth attendance as “positive” because of the increase in skill sets to manage emergency maternal and neonatal needs. Yet, there was a unanimous request in all districts for community-based skilled birth centers and/or health workers, due to long distances to existing facilities and waiting period for scheduling visits. This was a barrier for women with large families to care for and feed. Similarly, assistance with transport, improved and larger maternity waiting areas/shelters, beds and facilities with toilets and water supplies were deemed necessary, if women were to give birth at the health center. FGD participants reported that women were penalized by virtue of bylaws that fine families who do not comply by attending ANC with their partners or not giving birth at a facility. This was seen as disproportionately affecting already disadvantaged women and families living in remote and poor communities. In addition to support for improved ANC and skilled birth capacity at health centers, communities across the board requested more community based services, specifically: community birthing centers as well as community-based ANC, ARTs, and increased treatment services within iCCM (such as malaria screening and treatment). A number of positive changes were noted in terms of gender and communications between couples: communities reported more open communication amongst couples and support for women regarding family planning, ANC, facility-based births and autonomous decision-making regarding a woman’s health needs as well as those of her children. Roughly one-third of communities felt gender-based violence had diminished as a result of BCC and mobilization efforts to improve awareness and support for women’s health issues. Some areas still need more work, for example, the preponderance of male nurse/midwives is still a constraint to utilization of maternal and reproductive health services, and more advocacy and promotion of these men as “appropriate” deemed necessary for both women and their partners. To a lesser extent, family planning is still seen as a woman’s responsibility in some communities. Constraints and Challenges Initially, both SSDI-Services and Communications were awarded funds to work with sub-grantees on community mobilization efforts. In Year 2, USAID decided that sub-grantees should remain under SSDI￾Services, leaving SSDI-Communications with only technical assistance, training and limited supervision of community activities. As a result, SSDI-Communications has limited and indirect influence over community mobilization activities, depriving the activity of JHU-CCP’s core technical strength and limiting the extent to which the activity can measure and attribute the impact of its varying BCC activities. This process also delayed the launch of community mobilization activities by sub-grantees for more than a year, presenting a further constraint in achieving impact from BCC activities for both SSDI￾Communications and Services. At present, only limited monitoring and reporting information is shared between these two projects, which will hinder SSDI-Communications from providing ongoing technical support and/or from revising and targeting trainings, messages and materials in the final two years. Attribution of the impact of BCC efforts will be a challenge, and ultimately, SSDI-Communications cannot be held solely responsible for this at district and sub district levels. Conclusions SSDI-Communications is on target for the majority of its PMP indicators. The project has built capacity at the central level with private sector and MOH partners, and at district and community levels through training and capacity building of SSDI-Services and sub-grantees. Baseline research formed a strong evidence base for the activity and HEU to strategically design and implement mass and community 22 campaigns. The development of various policies, strategies, tool kits and campaign materials has been strategic and is likely to result in increased capacity within the MOH for future SBCC activities. SSDI-Communications has been constrained by cuts in funding and the removal of community mobilization from their mandate. This has left SSDI-Services and SSDI-Communications in a complex situation regarding M&E, the revision of messages and activities and subsequently, the attribution of both progress and constraints in their combined BCC efforts. Ultimately, it is only through the end line survey and service statistics that the impact of the combined efforts of SSDI-Communications and SSDI￾Services can be measured. Evaluators caution that demand for essential health services has long outstripped supply in Malawi, and advise careful consideration of the importance of strategic targeting and tailoring BCC efforts to balance demand creation efforts with availability of health services. Recommendations 1. Review and revise SSD-I indicators taking into account the following two factors: o First, funding levels are not sufficient to achieve all output and outcome indicators. o Second, SSDI-Communications cannot be held 100% accountable for the impact of BCC efforts (including changes in knowledge, attitudes and behavior), as they are not responsible for the implementation of community activities. 2. Increase coordination of M&E between SSDI-Services and SSDI-Communications in order to have larger and better gains from their combined BCC efforts. This should be done every six months to tailor and improve messaging. 3. BCC messages and efforts should focus on the same 15 key interventions as recommended to SSDI-Services, such as SUN, which supports CMAM and contributes substantially in improving the nutritional status of children, (or on health preventive behavior that can be undertaken in the home/community not service dependent). 4. Increasing demand for services should be carefully coordinated and “integrated” only where services are accessible and available. Gender and equity issues also need to be carefully considered, to ensure equal access and provision of quality care for all. SSD-I SYSTEMS: FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Summary Overview SSDI-Systems is a five-year project (September 2011 – September 2016) with a funding level of $9,999,741. The purpose of SSDI-Systems is to provide technical assistance to MOH to improve policies, management and leadership, and fiscal responsibility, in a collaborative effort to strengthen Malawi’s health care system. SSDI-Systems is implemented by Abt Associates Inc. The project has 15 core staff of which 12 are technical staff at the main office in Lilongwe and three are Zonal Financial Management staff covering the five zones. The project has no presence at the district level. SSDI-Systems has six result areas that include: increased advocacy for and implementation of evidence￾based policies; strengthened leadership and management capacity of MOH; improved and strengthened zonal supervision structures; improved leadership and management of human resources for health (HRH); improved decentralized management of district health services; and, strengthened health financing mechanism, financial planning and budget execution capability at national, zonal and district levels. 23 Accomplishments The activity conducted a number of situational assessments based on the need in the specific result areas at the start of the activity, the findings of which informed the SSDI-Systems and MOH’s joint selection of interventions within the six focus areas. SSDI-Systems has a comprehensive PMP and tracks 54 routine output and outcome indicators. The evaluation team selected six outcome indicators across the six result areas, which are analyzed from the start of the project to June 2014 (Annex XIII: SSDI-Systems: Progress against PMP Targets, see footnote 2). Of the six, one has already been achieved, four were partially achieved at the end of Year 3, and one is unlikely to be achieved at the end of the activity. At the central level, the activity has provided technical assistance to coordinate, develop and analyze policies including the secondment of a full-time policy advisor, under the Policy and Development Unit. They have also supported National Malaria Control Programme with a seconded technical advisor for M&E. The project has strengthened the capacity of the MOH in health financing, financial planning and budget execution through the development of the health financing strategy and undertaking the National Health Accounts Study. SSDI-Systems developed a capacity development plan and trained senior and middle level managers at central and zonal levels in leadership and management including use of data for decision￾making. Furthermore the activity has provided assistance to finalize, prioritize and cost HRH strategic plan, strengthen the use of HMIS and Human Resource Information System (iHRIS) data at the MOH headquarters in health and workforce planning. SSDI-Systems has also supported the MOH at the central level to revise and integrate its approach to supervision, which is now based on HMIS indicators and employs smartphone technology. The activity trained coordinators to use this new approach in supervising the zonal and district levels of health care. They have also worked in collaboration with SSDI-Services and the MOH to design a pilot performance￾based incentive (PBI) scheme. At the district level, SSDI-Systems has trained, coached and mentored DHMTs in leadership and management, including the use of data for decision-making. The activity has strengthened financial management of district hospitals in budgeting, procurement, accounting, fleet and fuel management and financial reporting. They have also provided technical and financial support for the DIP development and reviews. Furthermore, they have scaled up training on iHRIS to the districts in preparation for establishment of the system. The activity has trained Zonal Health Officers, DHMTs and coordinators in integrated support supervision using smartphone technology. They have also built the capacity of district in performance management system and either established or strengthened stakeholder groups for improved coordination. “The establishment of the Policy Development Unit and policy development, strengthening capacity of the government in leadership and management have been very helpful.” KII central MOH “Performance management has given us a much clearer picture of our work: it doesn’t favor anyone and shows you both the good and the bad” KII, Healthcare worker 24 Graph 2: SSDI-Systems Coverage by Activity at the District Level Graph 3: SSDI-Systems Coverage by Activity at the Zonal Level As is the nature of most health systems strengthening projects, SSDI-Systems has implemented the bulk of its interventions at the central level. The project has interventions in all the 15 SSD-I selected districts. However, implementation of the activity’s interventions in these districts is at varying levels. Most zonal and district level interventions have just concluded the pilot stage and are beginning to scale￾up in Year 4. For example, the PBI pilot began in September 2014 in three districts. The pilot for iHRIS (at three cost centers) ended in March 2014 and the remaining cost centers received training in mid￾2014. Graphs 2 and 3 below show the project’s coverage of activities at the zonal and district levels. While coverage for iHRIS training is 100%, only six districts have updated iHRIS data. In addition, 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Data for Decision-Making Leadership & Management Integrated Supportive Staff Performance Appraisal iHRIS Development of Multi-Year District Stakeholder Forum DIP and ZIP Reviews Financial Management Covered Not Covered 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Data for Decision Making Leadership & Management Capacity Integrated Supportive Supervision Not Covered Covered 25 whereas PMS has 80% coverage for training, the completion rate has declined from 48% to 30%16 (Annex XIV: SSDI-Systems Zonal and District Implementation Summary table by Activity). In summary, SSDI-Systems has strengthened the institutional capacity of the MOH to: o Coordinate and develop evidence based health policies through the establishment of the Policy Development Unit including seconding staff to the unit. By August 2014, six out of 10 policies were half way through the development process with one approved and five awaiting approval. o Provide strategic leadership and management and strengthen decentralized health services through training, integrated supportive supervision and mentorship. All the districts implementing supportive supervision reported receiving useful guidance, technical support and feedback from their supervisors. o Revitalize the plans and implement the GOM Performance Management System (PMS) by training staff at the central level and in 12 districts. District staff who received orientation on PMS reported better communication with their supervisors. PMS is now a requirement and used as a tool for staff promotion. o Track health expenditure and develop health-financing tools for improved resource generation, allocation and utilization through the NHA study and development of the Health Financing Strategy. o Improve district planning and financial management amongst accounting officers in all 15 districts through coaching and mentorship in budgeting, procurement, records management and responding to audit queries. The project has provided technical assistance to DHMTs and the District Councils in multi-year planning and DIP progress reviews. Previously, districts used to develop annual plans but now they are developing three-year strategic plans. District financial reporting has improved, with reduced audit queries. O Upgrade the District Health Information System (DHIS1) to DHIS2 and establish iHRIS in collaboration with MSH and Voluntary Service Overseas (VSO) to improve data management for evidence-based decision-making at district and central level. DHIS2 is a web-based system and is intended to house all data in one database. Constraints and Challenges Challenges with activity design SSDI-Systems has no presence at the district level and financial coaches are only available at the zonal level. There are systems-related bottlenecks at the district and facility levels that require regular and focused mentorship and follow-up, especially with regard to weak data management and use, multi-year planning, DIP reviews and strengthening referral system. While SSDI-Services is trying to fill some of the gaps, these areas are not its specialty and it is already overstretched, so the support to SSDI-Services it is asked to render is seen as additional work to them. As explained in KIIs with USAID and SSD-I, SSDI-Systems delayed in the initiation of policy development activities due to diverging and competing demands from the technical teams and departments at both USAID and the MOH, especially in undertaking HIV/AIDS and malaria policies. Implementation of PBI was delayed by one and a half years due to delayed approval by USAID. USAID has altered the original scheme from provision of both facility and provider incentives to only facility- 16 Ministry of Health Performance Management Systems Pilot Implementation Assessment Report 2013. 26 based improvements without incentives to providers. The initial design of the PBI was strong and would have been a more robust option. The current design does not follow global best practices and lessons learned as found in the activity’s situational analysis17. It is therefore unlikely to achieve its intended objective. In addition, many facilities will not qualify in the areas identified for assessment because current standards in these facilities are still very low. Furthermore, funds awarded to a facility are directly linked to improving these specific areas of weakness according to the set criteria, which may not be a priority nor appropriate for some facilities and in the Malawi context. Challenges associated with the activity While all KIIs with DHMTs trained in smartphone technology reported that this technology is designed to more readily provide results, the tool has several challenges that render it difficult to use. To begin with, the activity did not provide most districts with the phones after the training. Secondly many of the trained staff who were provided with phones are still not using them. This is partly due to inability to troubleshoot technology-related problems. The checklist was reported to be too long, labor-intensive and not practical in staff-constrained facilities. Though the new supervision system has potential for integrated supervision, instead of selected coordinators undertaking all areas of supervision, each coordinator is still undertaking supervision in their sole areas of focus. Only the DHO has access to the results of supportive supervision by use of a password. Due to these constraints, most districts are still using the manually-integrated checklists and analyzing data using Excel. The findings of this evaluation concur with many found in the pilot assessment: o Smartphones were not always available to supervisors who would like to conduct frequent supervisions either because the facilities had not yet received them or because of the difficulty in accessing phones at the facility. o Once supervision results were uploaded to the database, they were no longer stored on the phone. o The majority of the district supervisors for the pilot facilities did not feel sufficiently trained to use the tool. o None of the administrators were aware of the supervision database, nor had they been trained on how to access the results of the supervisory visits. o Seven out of 14 district-level supervisors had a technical problem with the phone.18 DHIS2 is still in its infancy and management and use of data is still weak, especially at district level. Data that is collected is not always entered and submitted on time mainly due to human resource constraints in the HMIS offices of all eight districts visited. Currently SSDI-Services provides technical support to HMIS officers to collect, enter data and produce reports. There is however a missing link at the district level to strengthen the use of data for decision-making and planning such as linking integrated support supervision data to DHIS and HMIS, to strengthen DIP development and reviews, establishing a feedback loop from central and zones back to districts and communities, as well as to address other issues such as supply chain management, logistics management, referrals and registers. Whereas SSDI-Systems has assisted in the establishment of stakeholder groups at the district level, the participation of these groups in the DIP is still minimal. There is still no engagement of key civil society organizations, such as CHAM, in the policy development and analysis process. 17 “Performance-Based Incentives”, USAID Assist Program, 2014, Performance-Based Incentives in Malawi: A Situational Analysis, SSDI-Systems, September 2012: page 16. 18 Supportive Supervision Assessment of the Pilot Phase, February 2014. 27 Conclusions The activity has made great strides towards achieving its targets. However, as final approval of both policies and systems lies with the MOH and GOM, full achievement of many targets is beyond the activity’s or USAID’s control. As this is the beginning of the fourth year of implementation, it is unlikely that the activity will be able to scale-up all programs in all 15 districts and build adequate capacity with the MOH to institutionalize and sustain the interventions. Recommendations 1. Strengthen stakeholder coordination at the central, zonal and district levels to improve collaboration, leverage resources, and increase synergy. Align all partners with the quarterly DIP reviews in which decisions pertaining to the roles of different partners are discussed. 2. Generate district plans for systems interventions in consultation with DHMTs. In six out of the eight sampled districts, the DHMTs felt that the activity made minimal consultations with them and this compromised ownership and institutionalization. 3. Prioritize the drafting and completion of the one policy now in development and refrain from starting the three still in discussion or newly identified. This will require a modification in activity indicators from 10 policies to seven. 4. Provide focused training to the districts that are already using smartphones for integrated supervision and continuously monitor their use but do not scale-up further. 5. The PBI initiative should be put on hold until a mechanism for provision of provider incentives is established. 6. Where pilot activities are underway, SSDI-Systems should not attempt to rollout or scale-up activities. Rather, they should focus on successfully completing and documenting best practices and lessons learned in these pilots by the end of the activity. SSD-I OVERVIEW: FINDINGS, CONCLUSIONS AND RECOMMENDATIONS Accomplishments Defining integration. In recent years, the term “integration” has gained increasing prominence in the global health community to describe service delivery, systems and management, and in some cases all three are implied. Confusion surrounding the definition and use of the word often results in misunderstanding among policy makers and health practitioners, affecting program development and implementation. The term “integrated health services” also has various applications and can refer to a number of health service issues. The World Health Organization (WHO) lists more than six common uses of the term in different settings. Key lessons learned on integration of health services in developing country settings include: o Supporting integration does not always mean that everything has to be integrated into one package. There are many possible permutations, based on the realities of the individual countries. o Integration is not a cure for inadequate resources, although it can lessen the burden. o There are more examples of policies in favor of integrated services than of actual successful implementation. In the context of the SSD-I project, integration means different things to different audiences causing confusion among stakeholders. The evaluation team could not find documentation from USAID that provides clear guidance for integration amongst partners. For the purposes of this evaluation, the team conducted a literature review of WHO documents to define two forms of integration: 1) integration of 28 SSD-I’s six focal EHP areas at the service delivery level; and, 2) programmatic integration of SSDI￾Services, SSDI-Communications, and SSDI-Systems. Integration of EHP focal areas. Through its network of facilities, the MOH provides a range of EHP services under one roof to catchment populations. SSD-I supported the MOH to deliver a wide range of EHP interventions in an integrated manner to increase access, quality and utilization. The integration of EHP focal areas at the facility level is logical and feasible and is working well. All individuals interviewed by the team value SSD-I assistance with the integrated service delivery model. From a management point of view, it increases efficiencies. From the service providers’ perspective, it is empowering. From the clients’ perspective, the model provides the opportunity to receive coordinated care rather than having separate visits for separate services. Integration of SSD-I activities. In the RFA, “integration” is defined as the integration of service delivery at facility level within the six focal EHP areas. It was not defined by the integration of the three SSD-I activities. The RFA states: “The SSD [Services] project is one of the three interrelated solicitations being issued by USAID/Malawi to support the GOM’s Health Sector Strategy. The recipient of this award is expected to implement an integrated service delivery program that will assure significant expansion and improved quality of priority essential health package services at the community and referral levels… Although each Recipient will have its distinct scope of work and roles and responsibilities, they must closely and effectively collaborate throughout the life of these projects. USAID strongly suggests that interested applicants read all sector description as the program as a whole is tightly linked to one another”. By design and by definition, the three activities are interrelated, but not integrated. Though one RFA was issued, three separate Cooperative Agreements (CAs) were awarded. The CAs have different mandates and deliverables and are reporting to three AORs. Thus, KIIs with the MOH (at all levels), USAID and the project revealed that the term “integrated” has caused confusion and is a misnomer for the SSD-I model. However, as a model to ensure close collaboration and coordination among projects with a common goal, it works well. Per the contractual agreement cited above, the activities are doing their best to collaborate and coordinate among themselves to maximize impact since they are cognizant of the fact that they have the same end-goal. SSD-I follows the premise of USAID Malawi’s CDCS 3-C Approach: Co-locating, Coordinating and Collaborating. The positive aspects and achievements of this close coordination and collaboration model include: o Integration of key services at the facility level; o Targeting of all SSD-I interventions in the same 15 districts, thereby co-locating Services, Communications, and Systems interventions; o Synchronization of the three activities’ start and end dates; o Though complex to achieve across three CAs, coordination and “integration” of planning and leveraging of resources in the implementation of activities19. 19 Although this was the initial thinking, projects do not have an integrated work plan, due to the fact that they have distinct mandates and reporting requirements. “The ‘integrated’ model is a challenge because all technical teams here at USAID have their own interests.” KII with USAID 29 Constraints and Challenges When the RFA was developed, KIIs with USAID indicated that there was limited alignment of the allocation of funds from each of the six technical areas with the relative burden of disease or desired outcome and impact expected as a result. KIIs with both USAID and the activities indicated that this has resulted in a vertical “tug-of-war” from the technical departments at USAID, and each, understandably, wanted to see interventions, results and indicators matching their funding commitments. Coordination and collaboration among the SSD-I activities was not as strong as it could be and there are overlaps, ambiguity, and gaps in mandates among the three activities. While several of these were addressed and rectified during the last two years of implementation, there are still overlaps and gaps that need to be ironed out. Unless the mandates of the projects are further clarified and amended, these ambiguities will remain. Examples include: o Although improving referral systems is mentioned in the RFA, it was not included in any of the three activities’ mandates. o There remain gaps and ambiguities between SSDI-Systems and SSDI-Services regarding improvements in data management and use. SSDI-Services and SSDI-Systems agreed between themselves that SSDI-Systems would work at the national level and Services at the districts to improve HMIS and their mandates were modified accordingly. However, to complete their mandates (e.g. for DIPs and multi-year plans), both projects need to work at district level. o Confusion over who was responsible for the planning and financial support of the DIP process led to administrative conflict between SSDI-Systems and SSDI-Services. The two activities had a formal agreement that SSDI-Systems supports the multi-year DIP planning process while SSDI￾Services focuses on supporting the annual DIP with respect to technical coverage of EHP services. However, according to some DHO interviews, implementation is strained around this issue since there wasn’t an agreement between the two projects on who would provide the funding for these activities. o There was overlap between SSDI-Services and SSDI-Communications regarding sub-grantees and community mobilization. Ultimately, this was awarded to SSDI-Services and SSDI-Communications is responsible for technical assistance, training and oversight. Together, Services and Communications developed an M&E system for community level activities, however, it is unclear whether the data and reports from this system will be shared with Communications and if, given their limited ability to work at district and community level, whether this information will be used to guide changes and tweak messages and activities in years 4 and 5. If not, this is a missed opportunity for the program. Support for supervision of DHMT and facility and community-based interventions from SSDI-Systems and SSDI-Services is ambiguous and/or not well coordinated. For example, SSDI-Services claims that assistance with DHMT administrative supervision is supported by SSDI-Systems while supervision of technical services by Coordinators is supported by SSDI￾Services. SSDI-Systems does not agree with this, and DHMT’s in some districts report that supervision is delayed or not done as a result. o Supply chain management is not included in any of the activities’ mandates. SSDI-Services has, on occasion, supported distribution of commodities and supplies to facilities as a stop gap measure to mitigate stock-outs though this is not in their mandate. Lack of presence of SSDI-Communications and SSDI-Systems at the district level undermines SSDI￾Communications’ ability to provide technical assistance to sub-grantees working in communities and SSDI-Systems’ ability to address health system bottlenecks at this level. SSD-I as a whole could have been more effective if all three activities were designed to complement each other and were integrated at the district level. Examples of systems bottlenecks at the district level include: weak data management 30 leading to inadequate support to multi-year planning and irregular DIP reviews, which ultimately lowered the use of data for decision-making. During the first year of project implementation, expansion of geographic coverage from 10 to 15 districts with the same funding level further complicated and constrained implementation and led to thinly spread programming in parts of the 15 districts. The RFA was ambiguous regarding district coverage, stating: “The SSD-I project will be responsible for scaled up and integrated priority EHP service delivery within primary health care clinics as well as referral levels in two to three targeted districts in each of the five zones of Malawi so as to cover a total population of approximately 8,000,000.” While the RFA allowed for 10 up to 15 districts to be included, the expansion to all 15, combined with the increase in interventions and indicators, has been a challenge for the projects, as noted by the MOH, USAID and the projects themselves. Other management and policy issues The Management Advisory Group (MAG) is the coordinating body for SSD-I. Initially, it comprised the senior and technical staff of the three activities and chaired by the SSDI-Services COP, with the objective of strengthening coordination amongst the three activities.20 Once implementation began, the composition of the MAG was expanded to include the USAID Health Office (including the AORs and all technical staff). As such, in its first year the MAG was instrumental in providing guidance as a coordinating body. However, in the last two years and as the activities gained traction, both USAID and SSD-I staff expressed significant coordination challenges as MAG meetings were held less frequently and attendance was inconsistent. Respondents felt the MAG was too big to meet regularly or function as a coordinating body. The issue of per diems for the MOH staff or the “full board policy” constrained the project’s ability to maintain or make further progress. This is an issue brought up by almost all respondents interviewed and USAID is well aware of the problem. This is a high-level policy issue beyond the scope of this evaluation. However, the evaluation team feels strongly that in the near future, the repercussions of the policy will be more visible and the performance of the program may decline. Conclusions The SSD-I program has potential to leave lasting changes in the way EHP service is delivered in Malawi. One of the unique aspects of SSD-I is that it was designed to support service delivery, behavior change, and an enabling policy and systems environment in one program, implemented by three activities through close collaboration. This model is intended to address the multi-faceted challenges holistically and achieve improved health outcomes. While the integration of focal EHP areas at the facility level has been successful, greater efforts are needed to ensure synergy and coordination between the three SSD-I activities. The division of projects at the design phase led to ambiguities about the roles of each project and constraints in integrated implementation. The evaluation team commends all three activities in achieving the majority of their intended targets at midterm. However, the activities have all been stretched doing so, and this comes at the expense of ownership, institutionalization and sustainability of efforts at the MOH. Because human resource constraints and scarce financial resources are overwhelming in Malawi, it is unlikely that any project will reach its full potential to improve the health situation until these issues are addressed. Although the SSD-I project has been addressing HR and financial constraints through 20 SSDI-Services Program Description. 31 technical interventions, these constraints are beyond the mandate and resources of USAID/Malawi and the SSD-I. The team recommends that future projects and goals be designed in light of this situation. Recommendations for the Remaining Life of SSD-I 1. USAID may facilitate better coordination and synergies between the three activities by assigning a single coordinating point person who shares the larger vision of SSDI as a single program. The coordinator should chair the MAG with the mandate to develop a common vision among the three activities. We recommend that USAID: a) Modify the structure of the MAG to include fewer key members from both USAID and the projects, and b) Clarify the mandate of the group to focus on coordination and strategic guidance for the entire SSD-I rather than focusing on individual activities and technical areas. The team also recommends that each of the three AORs are enabled to allocate more time to their respective projects and work together more closely in managing the overall SSD-I program. 2. All three activities should work more closely with the MOH and DHMTs to respond to and support MOH priorities at all levels. SSD-I activities should work together to facilitate MOH implementation of interventions rather than implementing interventions independently. 3. USAID, in consultation with the three activities, should amend the activities’ mandates to iron out remaining gaps, overlaps and ambiguities. 4. The three activities should have routine meetings among themselves at the district level to ensure that: a) project work plans are complementary, b) resources are leveraged, and c) comparative advantages of each project and implementing agency is maximized at the district level. SSD-I activities have established functional coordination/collaboration mechanisms at the central level; however, the efforts should further extend to the district level. Recommendations for the Follow-On project 1. USAID/Malawi should continue pursuing the same goal of increased availability and quality of EHP service delivery, systems strengthening and BCC to improve the health of Malawian families. Having an integrated project will help improve programming and strengthen the health system. 2. The follow-on project should: a) Be one Cooperative Agreement, which houses the three activities and has one AOR to ensure cohesion. b) Focus on priority, high-impact interventions within the focal areas of the EHP, building on achievements and lessons learned from SSD-I. c) Within the mandate of the project and technical focus in the USAID design, embrace a holistic bottom-up, decentralized and systems strengthening approach, which emphasizes district consultations, MOH institutionalization and ownership. d) Again, within the mandate and technical focus of the project, maintain a flexible approach to set targets and plan interventions in response to individual, district-specific needs. e) Provide adequate financial support for infrastructural improvements in tandem with the basic requirements for quality service provision. f) Include implementation of all three activities at the district level to improve synergies, coordination and impact. 32 ANNEXES ANNEX I: EVALUATION STATEMENT OF WORK ANNEX II: EVALUATION MATRIX ANNEX III: LIST OF DOCUMENTS REVIEWED ANNEX IV: LIST OF KEY INFORMANTS INTERVIEWED ANNEX V: DATA COLLECTION INSTRUMENTS ANNEX VI: INFORMED CONSENT STATEMENT ANNEX VII: SSDI-SERVICES: PROGRESS AGAINST PMP TARGETS ANNEX VIII: TREND ANLAYSIS OF SSD-I INDICATORS ANNEX IX: SSDI-SERVICES: LIST OF TECHNICAL INTERVENTIONS ANNEX X: FINDINGS FROM THE FACILITY MINI-SURVEY ANNEX XI: PROGRESS TOWARD PMP INDICATORS ANNEX XII: FOCUS GROUP DISCUSSION SUMMARY ANNEX XIII: SSDI-SYSTEMS: PROGRESS AGAINST PMP TARGETS ANNEX XIV: SSD-I SYSTEMS: ZONAL AND DISTRIC IMPLEMENTATION TABLE BY ACTIVITY 33 ANNEX I: EVALUATION STATEMENT OF WORK SECTION C DESCRIPTION/SPECIFICATION/STATEMENT OF WORK C.1. INTRODUCTION AND BACKGROUND USAID/Malawi’s new Country Development Cooperation Strategy (CDCS) covers the period 2013 through 2018. The overarching strategic goal is to improve the Quality of Life for Malawians. This is supported through three development objectives (DOs) that include: Social Development Improved, Sustainable Livelihoods Increased, and Citizen Rights and Responsibilities Exercised. Along with the DOs, there are also four sub-intermediate results (sub-IRs) that cut across the entire CDCS portfolio. One sub-IR is entitled the Capacity of Institutions Improved. Various assessments highlighted the persistent theme of weak organizational capacity for both government and non￾governmental bodies that comprise the partner and counterpart base for the strategy. In order for the three DOs to be achieved, the capacity to plan, manage, evaluate and account for resources among this partner base has to be supported. USAID/Malawi’s CDCS hypothesizes: If assistance is integrated, then development results will be enhanced, more sustainable and lead to achievement of the CDCS goal: Malawian’s Quality of Life Improved. The CDCS will promote integration through the concentration of program and financial resources through what the Mission is calling a 3-C Approach by:  Co-locating interventions to the extent that it is sensible;  Coordinating better within USAID and with other Development Partners (DPs), and  Collaborating to foster linkages among implementing partners and the DPs to improve results, and sustainability. There are 27 districts in Malawi. All three DOs will work in three focus districts; two DOs will work in ten districts; one DO will work in seven districts; and the remaining seven districts will receive limited nation-wide assistance. One way that USAID/Malawi seeks to improve the quality of life for all Malawians is through increased availability and quality of essential social services. The Mission is engaged in a range of health system strengthening activities to expand facility and community level service delivery and increase the number of people who receive high impact and high quality services. The Support for Service Delivery Integration (SSD-I) is USAID/Malawi’s flagship health project. This project consists of three inter-related sector activities, namely SSDI-Services, SSDI-Communications and SSDI-Systems. In close collaboration with the Ministry of Health, SSD-I activities support the increased availability and quality of the Essential Health Package (EHP) services; reinforce health promotion and disease prevention among households; and strengthen elements of the health system to sustain effective EHP delivery. SSD-I’s activities align with USAID/Malawi’s Country Development Cooperation Strategy (CDCS) Development Objective (DO) 1 (Social Development Improved) and with cross-cutting Sub￾Intermediate Results (SIR) 1 (Capacity of institutions improved) and SIR 2 (Positive behaviors adopted). SSD-I activities contribute directly to the Malawi Health Sector Strategic Plan (HSSP) 2011-2016. SECTOR I: Support for Service Delivery (SSDI-Services) provides an integrated service delivery program to improve the health and well-being of Malawians by improving the quality of priority Essential Health Package (EHP) services at the community and referral (health centers and District hospitals) levels. SSDI￾Services’ primary objectives include: 34  Increase access and utilization of EHP services for women and children and engage men in health care;  Improve quality of health services at community and facility level in target districts;  Improve health-seeking behavior by individuals, families and communities;  Strengthen health care delivery system via the development, testing, and scaling up of innovative and sustainable community-based service delivery approaches;21 and  Develop coherent, and mutually supportive activities between the Government of Malawi (GOM), the three SSD-I project elements, the Private Sector and Social Marketing (PSSM) partners, and other national stakeholders to ensure integration and leveraging of program inputs to scale up service delivery. SECTOR II: Social and Behavior Change Communication (SSD-I Communication) is the Social and Behavior Change Communication (SBCC) component that promotes normative and behavior change in several health areas. SSDI Communication addresses barriers to behavior change at the structural, service delivery, societal and personal levels to enable the design of effective interventions at all levels. SSD-I Communication primary objectives include:  Strengthen national and targeted district level planning and coordination on EHP priorities;  Develop and produce evidence based SBCC packages under multi-level media campaign;  Build capacity of key national institutional partners and targeted district SSD-I partners for effective SBCC strategic planning and delivery; and  Identify best practices for SBCC implementation through formative research and testing innovative approaches. SECTOR III: Health Policy and Systems Strengthening (SSDI-Systems) assists Malawi’s Ministry of Health (MOH) to improve policies, management and leadership, and fiscal responsibility to advance Malawi’s health system and the sustainable impact of the Essential Health package. SSD-I systems provides appropriate, relevant, and coordinated interventions at the national, zonal, district, and local levels. SSDI-Systems primary objectives include:  Provide the MOH with expert technical assistance in policy development;  Clarify and strengthen management functions at all levels for quality assurance;  Improve the current health management information system (HMIS) to ensure that key staff can carry out rigorous and routine high-quality data collection to support evidence-based decision making;  Adapt proven tools and metrics to bolster monitoring and evaluation (M&E), financial management, and Human Resources for Health (HRH); and  Execute gender-sensitive programming that takes into account the exponential benefits resulting from activities that advance women’s and girls’ equity and health status. 21 Note these build on experiences in Malawi to date and considering GOM medium-and long-term budgetary constraints. 35 SSDI-Systems strategies and activities are designed to complement and support the service delivery and social/behavioral change interventions of Sectors I and II (SSDI-Services and SSDI-Communication). SSD-I’s development hypothesis postulates that: Programming health interventions through an integrated platform, consisting of activities in health policy and systems strengthening, support for integrated health service delivery, and social and behavior change communication, will result in significant expansion of coverage, quality and utilization of priority Essential Health Package services at community clinics, health centers and district hospitals. C.2. PURPOSE OF THE PERFORMANCE EVALUATION The Contractor will carry out a performance evaluation to determine the effectiveness of the SSD-I approach in supporting increased availability and utilization of quality integrated EHP services; and its performance in strengthening Malawi’s health system. Findings and recommendations from this evaluation will inform implementation of SSD-I over the remaining life of activity, as well as USAID/Malawi’s design of future investments in health. This evaluation will contribute to the learning agenda on integrated health programming under the Global Health Initiative and the Mission’s CDCS. C.2.1. TARGET AREA AND GROUPS The Contractor must implement the evaluation at the central, zone, district and community levels. SSD-I interventions are implemented in 15 target districts (see map below). The SSDI-Services activity supports delivery of integrated EHP services at district and community levels; and provides health systems support and strengthening, primarily at zonal and district levels. SSD-I support service delivery at approximately 270 sites, which include district hospitals, health centers and community/village clinic. The SSD-I Communication activity provides nationwide coverage, through mass media campaigns and capacity building; and implements intensive Behavior Change Communication interventions at community level. The SSDI-System activity supports policy analysis and development at MOH, and direct technical assistance to program managers at central, zonal and district levels in Management and Leadership, Fiscal Responsibility and Monitoring and Evaluation. C.3. OBJECTIVES SSDI Districts include: Mangochi, Machinga, Balaka, Zomba, Mulanje, Phalombe, Nkhotakota Nsanje, Chikhwawa, Kasungu, Dowa, Salima, Lilongwe, Chitipa and Karonga 36 The main objectives of this performance evaluation are to measure, document and determine the extent to which SSD-I activities have contributed to: 1. Increased availability and utilization of EHP services (sector 1); 2. Improved health promotion and adoption of normative health behaviors (sector 2); and 3. Improved functionality of the health system22 to support delivery of integrated health services (sector 3). The Contractor must determine the extent to which SSD-I is achieving its objectives in integrated programming, as well as sector-specific objectives. The Contractor must identify the key factors that have either enabled or limited SSD-I’s achievement of its objectives and the key bottlenecks that are limiting MOH’s capacity to adopt the successful components of SSD-I’s approach to integrated EHP service delivery. The Contractor must make recommendations on key actions required of SSD-I, USAID/Malawi and MOH to enhance achievement of SSD-I objectives and enhance sustainable delivery of integrated EHP services. C.3.1. EVALUATION QUESTIONS At a minimum the Contractor must address the following questions: 1) To what extent has SSD-I’s approach improved the delivery of quality EHP services and health indicators at SSD-I supported service delivery points? 2) To which extent has SSD-I’s health system strengthening and capacity building approach—at central, zonal and district levels—responded to specific bottlenecks23 that impact service delivery? 3) What effect has joint coordination and implementation of interventions across SSD-I had on the achievement of sector-specific and USAID’s health objectives24? 4) To what extent has SSD-I’s approach responded to client needs and supported families in adopting improved preventative and health-seeking behaviors? 5) What elements of SSD-I’s approach have either enabled or limited its ability to improve the capacity of government in the following areas: health service delivery, fiscal responsibility, management and leadership, behavior change communication, use of data for decision-making; and what recommendations can be made for overcoming these bottlenecks in future? 22 Improved functionality refers to MOH’s improved policy environment to support effective delivery of integrated EHP services; and improved capacity of MOH in the areas of fiscal responsibility, management and leadership. 23 Upstream bottlenecks include: enabling policy environment to support effective delivery of integrated EHP services; capacity of MOH in the areas of fiscal responsibility, management and leadership and Behavior Change Communication; and management information systems. Please refer to the activity Request for Applications for detailed descriptions. 24A change in nomenclature of USAID objectives has been effected since the inception of SSD-I. At design, SSD-I contributed to achievement of USAID Health Office’s Strategic Objective “Increased availability and utilization of quality EHP services.” With the introduction of the Malawi CDCS in 2013, SSD-I activities fall under Development Objective “Social Development Improved.” 37 C.3.2. TASKS The Contractor must perform the following tasks as part of this scope of work: 1. Draft Inception report; 2. Develop evaluation model; 3. Test and verify the evaluation model; 4. Deploy a field team; 5. Collect the relevant data to inform the evaluation; 6. Conduct bi-weekly oral debriefing meetings to update USAID of evaluation progress and preliminary findings; 7. Draft Final Report; 8. Presentation to USAID/Malawi on report findings; 9. Conduct a stakeholder’s workshop to present the draft evaluation findings to relevant stakeholders for validation of findings; 10. Upload Final Report (USAID-Approved) to the USAID Development Experience Clearance House (https://dec.usaid.gov); 11. Produce and deliver 20 copies (hard copy) of the Final Report to USAID; and 12. Produce and deliver 1 memory stick containing electronic copies of all evaluation products (data, tools, presentations and reports). C.4. RESULTS: DELIVERABLES AND OUTPUTS Inception Report The inception report must describe the conceptual framework the evaluator will use to undertake the evaluation. It must detail the evaluation methodology (i.e. how each question will be answered by way of data collection methods, data sources, sampling and indicators) and address all technical requirements. At a minimum the inception report must contain the following: a. A work plan, which indicates the phases in the evaluation with key deliverables and milestones and key personnel responsibility. USAID evaluation point(s) of contact will review and approve this report before the contractor begins implementing the evaluation plan. b. Complete set of evaluation questions, elaborated on as necessary. Any questions added during the contract negotiations must be clearly indicated and any deleted questions must be mentioned with a reason as to their exclusion. c. Discussion of the overall approach of the evaluation, highlighting the conceptual model(s) adopted. This must incorporate an analysis of the intervention logic of the program. d. Discussion of risks and limitations that may undermine the reliability and validity of the evaluation results. 38 e. Specification of indicator, index, or indicators that must be used as a guide in answering each question. f. Discussion of the data collection and data analysis methods that will be used for each question. State the limitations for each method. Include the level of precision required for quantitative and qualitative methods and value scales or coding used for qualitative methods. Standard data collection methods for USAID evaluations are: surveys, questionnaires, interviews, focus groups, document review and observations. g. Detail key data sources that will be selected to answer each evaluation question. h. Explanation of how existing data will be incorporated and used to answer the evaluation questions. i. Discussion of the sampling methods and details. Include area and population to be represented, rationale for selection, mechanics of selection, sample size (for each unit of analysis), sample precision and confidence and limitations. j. Summarized evaluation methodology in an evaluation planning matrix that must contain the following column headings: evaluation question, measure (s) or indicator(s), data collection method(s), data source, design strategy / framework for each question, sampling methodology, data collection instrument(s) for each question and data analysis methodology on each evaluation question. k. Timeline showing the evaluation phases (data collection, data analysis and reporting) with their key deliverables and milestones. l. Specific responsibilities of each team member for each evaluation phase. Include any changes in the evaluation team. m. Discussion of logistics for carrying out the evaluation. Include specific assistance that will be required from USAID, such as providing arrangements for key contacts within the Mission or Government. n. Discussion on the use of spatial data collection methods and formats to ensure locations included in the evaluation sampling frame are captured for integration into the Mission’s geographic information system and to permit spatial analysis of evaluation data at the facility level. The Contractor must provide geo-referenced data sets to the USAID/Malawi COR. At a minimum, data must be provided in an MS Excel sheet that includes a unique identifier for each data record, with latitude and longitude locations in decimal degree format to the fifth place (e.g., 34.45673 and -13.36712). During the inception plan the COR will work closely with the Contractor to determine other applicable evaluation data that will be included into the spatial data table for each evaluation location. USAID/Malawi will provide a generic EXCEL template for the Contractor to use that will facilitate this process. o. Appended draft instruments for data collection specific to questions and indicators in the evaluation. The inception report must clearly document and discuss how gender and disability analysis will be integrated into the design of the evaluation. 39 Evaluation Methodology The evaluation model must be characterized by a mixed methods design and include appropriate sample sizes required to ensure scientific rigor. Data on service delivery is available; and the mission will share (in MS Excel) the data sets with the Contractor. Experiential data must be collected from sample populations of program managers, health care providers and beneficiaries. The model must describe and document the methodological approach used as well as all analytical aspects. The model must be clearly developed and documented and follow USAID Evaluation and performance evaluation best practices. The model must include an evaluation framework and assessment tools for each evaluation question and highlight the conceptual model(s); and specify the measurement criteria to be used to respond to each question. It must discuss any risks and limitations that may undermine the reliability and validity of the evaluation results. The model must outline data collection processes for each question. At a minimum, the evaluation model must include the following elements: 1. Secondary data analysis of the baseline and routine health service delivery records, collected by SSDI-Services from November 2011 to December 2013, to determine the trends in utilization of EHP services and health outcomes in the 15 SSD-I districts. 2. Key Informant Interviews or Focus Group Discussions with stakeholders to determine the effect of SSD-I implementation on improved functioning of the health system. Identify key factors that have enabled or limited SSD-I’s achievement of its objectives. Key informants should include MOH managers at central, zonal and district levels; SSD-I staff; frontline health care providers at facility and community service delivery points; and USAID health office staff. 3. Key informant interviews or focus group discussions with stakeholders to determine key enabling factors for strengthening integrated EHP health service delivery, functioning of the health system at central, zonal and district levels, and adoption of healthy behaviors in the home. Key informants should include MOH managers at central, zonal and district levels; SSD-I staff; frontline health care providers at facility and community service delivery points; and members of the Health Donor Group. 4. Focus group discussions with community members within the SSD-I districts to determine their knowledge, attitudes towards preventive health care interventions and health-seeking behaviors; and their perceptions of factors that enable or limit their use of integrated EHP services. The sample size and number of sites to be visited will be determined by the methodology of the evaluation design. The Contractor is not required to visit all 15 districts. Constraints to Data Collection and Analysis 40 A number of factors could constrain the ability to collect or analyze data. 1. Language: Though English is the official language for professional communication, some of the stakeholders, in particular frontline health workers, may be more comfortable communicating their ideas in Chichewa. The Contractor should include individuals fluent in Chichewa on the evaluation team, particularly for the key informant interviews and focus group discussions. 2. Geography and infrastructure: Even with sampling, the evaluation will require considerable travel throughout the country to reach project implementation sites and beneficiaries. The road infrastructure may render some sites difficult to access or inaccessible, particularly during the rainy season. Gender Analysis USAID requires evaluations to consider differences in the ways that women and men participate in or benefit from projects. As heads of most households, men have great influence in the overall welfare of the family; this includes household food security as well as the ability of the family to seek health care and adopt healthy behaviors. The evaluation findings, conclusion and recommendations must include an analysis of any differences in how the SSD-I model has affected the participation of men and women in use of integrated EHP services and adoption of healthy behaviors. Debriefing Meetings The Contractor must provide bi-weekly debriefing meetings to USAID. These meetings must include a discussion on progress to date, provide a summary of any analytical results, discuss challenges, successes and planned work over the remaining duration of the evaluation. The team leader of the evaluation team will be required to routinely communicate updates to the Contracting Officer’s Representative. USAID/Malawi Presentation The Contractor must deliver an oral presentation of the evaluation findings, conclusions and recommendations for each question to USAID, prior to finalizing the draft evaluation report. Findings Workshop After incorporating USAID’s comments into the draft final evaluation report, the contractor is required to present the key findings, conclusions and recommendations at a half-day workshop. The workshop must be held in Lilongwe and is anticipated to be attended by between 35 to 40 key stakeholders. The Contractor is responsible for costs, logistics and managing invitations to this workshop. The Contractor must produce a summary/briefer (max. 3 pages) of key findings, conclusions and recommendations to be distributed to stakeholders during the workshop. Final Evaluation Report The Contractor must submit an evidence-based final evaluation report that answers, in full, each evaluation question. The report must be no longer than 30 pages in length (excluding annexes) and comply with the Checklist for Assessing USAID Evaluation Reports and the technical requirements listed below. The Contractor must submit 20 hard-bound copies to USAID/Malawi and an electronic copy in a memory stick. The memory stick must include electronic versions of all tools and products of the evaluation, including instruments and data in data formats suitable for reanalysis. The Contractor must ensure that 41 Appendix I of the USAID Evaluation Policy – Criteria to Ensure the Quality of the Evaluation Report is followed. This includes:  The evaluation report must represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why;  Evaluation reports must address all evaluation questions included in the scope of work;  The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by either the Contracting Officer’s Representative or Contracting Officer as required;  Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report;  Evaluation findings must assess outcomes and impact on males and females;  Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.);  Evaluation findings must be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings must be specific, concise and supported by strong quantitative or qualitative evidence;  Sources of information must be properly identified and listed in an annex;  Recommendations must be supported by a specific set of findings; and  Recommendations must be action-oriented, practical and specific, with defined responsibility for the action. The body of the report must be no more than 30 pages in length (excluding any annexes) and include the following sections:  Executive Summary: This section must precisely provide the project background, including evaluation design and methodologies, key findings, main conclusions and recommendations from the evaluation.  Background: This section must provide a brief description of the project that highlights the scope of the project, project development hypothesis, activities undertaken in the project, key impact indicators of the project and impact areas of the project. Other activities that complemented the project activities directly or indirectly in the intervention districts must also be highlighted. In addition to this, the section must detail the methodology and related research protocols undertaken in conducting the evaluation and related limitations encountered during the project implementation and evaluation. 42  Findings and empirical facts collected during the evaluation: This section must present findings from the evaluation. The evaluation findings must be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. The evaluation findings must assess key outcomes and impacts as structured around the organizational framework of the evaluation questions. The findings must be specific, concise and supported by strong quantitative and qualitative evidence analyzed through scientifically plausible methodologies. Sources of information used in arriving at the findings must be properly acknowledged and listed in an annex.  Conclusions (interpretations and judgments based on the findings): Evaluation conclusions must be presented for each key finding. The conclusions must logically follow from the gathered data and findings. These must be explicitly justified. If and when necessary, the evaluator must state his/her assumptions, judgments and value premises so that readers can better understand and assess them.  Recommendations (proposed actions for management): This section must precisely and clearly present recommendations that must be drawn from specific findings. The recommendations must be stated in an action-oriented fashion, must be practical, specific, and with defined responsibility for the requisite action. The recommendations presented in this section must follow the evaluation questions as the organizational framework. The Contractor must build into the allowable time schedules for debriefing with USAID/Malawi. The Contractor must submit 20 hard bound copies to USAID/Malawi and an electronic copy in an appropriate media including all instruments and data in formats suitable for reanalysis. C.5. TEAM COMPOSITION The Contractor must use a team structure that will cost effectively complete the evaluation. In order to mitigate the perception or reality of biased measurement or reporting due to conflict of interest the evaluation team must comprise personnel external to management or implementation of the SSD-I project. In accordance with USAID Forward principles, and as a means of fostering local capacity/expertise, USAID/Malawi requires that at least one member of the key personnel be a Malawian in order to provide context and linkages to key stakeholders in Malawi’s health sector. Curriculum Vitae for all key personnel that reflect the individual’s expertise relevant to the evaluation must be submitted for review. The staffing plan should identify the proposed Team Leader and demonstrate the team’s balance between evaluation expertise, subject matter expertise, and research specialization. The staffing plan must also articulate the roles and responsibilities for each key personnel with regards to responding the evaluation questions. It is anticipated that the key personnel will be supported by an administrative or logistics assistant. 1. Team Leader (TL) (key person): The TL will provide overall oversight and management of the evaluation team. S/he will ultimately be responsible for the quality of evaluation outputs and ensure timely submission of each deliverable. S/he will provide team leadership, plan and coordinate meetings and site visits, and be responsible for payments of local logistical needs and local staff working with the team. S/he will lead the preparation and presentation of the key evaluation findings and recommendations to the USAID/Malawi team and other key stakeholders. 43 The TL must have a postgraduate degree and at least 10 years of experience in the requisite technical background to provide state-of-the-art technical leadership and hands-on abilities experience in managing health programs in developing countries, including activities aimed at strengthening health systems for improved maternal and child health outcomes. S/he should have direct experience and solid understanding of best practices in integrated health service delivery and technical knowledge representing the full range of programming under SSD-I (i.e. integrated health service delivery, health systems strengthening and social behavior change communication). S/he should have at least 5 years of experience in conducting public health evaluations. 2. Public Health Specialist (key person): This team member must have a post-graduate degree, at least 8 years of experience in the requisite technical background to provide state-of-the-art technical leadership and hands-on abilities in: family planning and reproductive health; maternal health, newborn health, child health, malaria, HIV/AIDS, and nutrition. USAID/Malawi expects that the specialist will have experience with both facility-based and community-based health programming. The Public Health Specialist should have complementary skills and knowledge in the technical areas required for the evaluation. S/he should also have experience conducting public health evaluations. 3. Social Behavior Change Specialist (key person): This specialist must have a post-graduate degree, at least 8 years of experience in the requisite technical background to provide state-of￾the-art technical leadership and hands-on abilities in the design and management of community￾level mobilization activities, and community-appropriate SBCC activities and packages; advising and supporting capacity building for improved coordination and use of SBCC activities and materials. S/he should have extensive experience in identifying, adapting and guiding the use of SBCC tools and approaches that facilitate effective community-level SBCC activities, developing and delivering targeted training and technical assistance to improve SBCC and community mobilization for priority EHP services. S/he should also have experience conducting public health evaluations. 4. Health Systems Specialist (key person): This specialist must have a postgraduate degree, at least 8 years of experience in the requisite technical background to provide state-of-the-art technical leadership and hands-on abilities in: management of health systems in developing countries; specifically activities aimed at implementing solutions to address constraints to human resources issues; planning and management in decentralized systems; and health financing. S/he should have experience in policy analysis, adapting and guiding the use of tools and approaches that facilitate health systems operations, developing and delivering targeted training and technical assistance to improve management and problem-solving and other skills of health systems leaders and managers. S/he should also have experience conducting public health evaluations. 5. Research Analyst: This analyst must have a postgraduate degree, at least 3 years of experience and knowledge of data analysis and data management techniques of both quantitative and qualitative data using SPSS, STATA or NVivo. The Research Analyst will support the evaluation team by managing and analyzing performance data from SSD-I sector partners and assist with the interpretation and communication of results. This person should have experience with geographic information systems and be able to assist the team to ensure that the evaluation is spatially referenced for inclusion into the Mission’s GIS. 44 6. Logistics/Administrative Assistant: The Assistant should have at a minimum a diploma in a relevant field. S/he should have experience in organizing events. S/he will assist the Team Leader in the organization of meetings, arranging field visits, organizing local travel, arranging local accommodation, and scheduling of appointments with stakeholders. USAID/Malawi M&E Specialists may work with the evaluation team as part of the Agency’s efforts to strengthen the Agency's learning. They will work under the direction of the Evaluation Team Leader in: 1) instrument development and piloting; 2) data collection; 3) data analysis; and 4) synthesis of results. The USAID staff will not have authority to provide technical direction to the Contractor. The Contracting Officer (CO) and Contracting Officer’s Representative (COR) are the only individuals within USAID who can provide technical direction on the evaluation to the Contractor. Curriculum Vitas for the USAID/Malawi M&E Specialists will be available upon request. Key personnel from the SSD-I sector activities will work with the Contractor by providing key documents and insights. The Contractor should not contact the SSD-I project partners independently but coordinate all requests for information through USAID/Malawi. In addition, USAID/Malawi encourages the involvement of representatives from the MOH in the evaluation process. C.6. PLACE OF PERFORMANCE The place of performance is Malawi with travel to health facilities and rural communities in the 15 SSD-I districts. The extent of the travel will be determined by the evaluation model (sample size and data collection methods). The Health Office M&E Specialist will serve as the primary Point of Contact and Contracting Officer’s Representative (COR) for the SSD-I Performance Evaluation. C.7. LOGISTICS The Contractor is responsible for all logistics including coordinating all travel around the country, lodging, printing, office space, equipment and car rentals, financing from the contract award and managing dissemination of results. The USAID/Malawi Health Office will work with the Contractor to set up initial meetings with key government officials and stakeholders. C.8. EXISTING INFORMATION The following documents will be made available to the Contractor for reference and use for the performance evaluation. The documents listed are applicable for each SSD-I sector activity (SSDI-Services, SSD-I Communication, SSDI-Systems), unless otherwise indicated: 1) SSD-I Program Descriptions; 2) SSDI-Communications Baseline Assessment Report; 3) SSDI-Services Baseline Assessment Report; 4) Quarterly Reports (FY 2011 – FY 2014 Q1); 5) Annual Reports (FY 2011 – FY 2014); 6) Annual Work Plans (FY 2011 – FY 2014); 7) Performance Monitoring Plans (FY 2011 – FY 2014); and 8) Checklist for Assessing USAID Evaluation Reports. [END OF SECTION C] 45 ANNEX II: EVALUATION MATRIX Evaluation Question Indicator/Measure Data Source/ Collection Method Sampling Methodology Data Analysis Methodology #1. To what extent has SSD-I’s approach improved the delivery of quality EHP services and health indicators at the SSD-I supported service delivery points? Changes in 10 key performance outcome and output indicators at SO and IR levels at 2011 vs. 201425 KII: Key USAID management, implementing partner, and Government of Malawi (GOM) Ministry of Health (MOH) staff, other health donors Data review: All USAID KPIs, including outcome and output objectives, results, and activities Data extraction from secondary sources: DHIS, DHS, MICS, GIS Document review: Annual Work Plans and Reports, Performance Monitoring Plans (PMPs), Policy Guidelines All intervention health zones and selection of one or more districts in each health zone Secondary data sources: District Health Information Systems 2 (DHIS2), HMIS (2011-2014), DHS (2004, and 2010), MICS (2006) USAID/Malawi health projects’ performance monitoring systems data (2011-2014) Quantitative analysis of KPI trends overtime, comparison with baseline, if available or obtained from secondary data sources and performance targets Quantitative analysis of key intervention output and outcome data disaggregated by age group, gender, location (urban/rural) Qualitative analysis; content analysis, themes, depth to support Data triangulation 25 SSDI-Services Indicators: 1) Family Planning: # of service delivery sites offering LAPM; couple years of protection; 2) Maternal, Neonatal and Child Health: # of sites/district providing BEmONC; % of births by skilled attendant; % of newborns not breathing at birth successfully resuscitated; % pregnant women attending ANC tested for HIV; 3) Nutrition: # of facilities with capacity to manage acute under-nutrition; 4) Malaria: % of pregnant women at ANC who receive 2nd IPT; 5) HIV/AIDS: % of HIV positive pregnant women initiated on ART; 6) Mentoring and PQI: % of facilities receiving at least 1 supervisory visit per quarter. 46 Evaluation Question Indicator/Measure Data Source/ Collection Method Sampling Methodology Data Analysis Methodology #2. To what extent has SSD-I’s health system strengthening and capacity building approach—at the central, zonal and district levels— responded to specific bottlenecks that impact service delivery? Changes in key performance outcome and output indicators at SO and IR levels at 2011 vs. 2014 # of policies, guidelines, regulations improved and adopted by the GOM # of CSOs receiving USG assistance engaged in advocacy intervention % of target districts analyzing and reporting data in line with national HMIS data quality standards % of target districts receiving at least 75% of the set number of integrated annual supportive supervision visits per guidelines % of districts that implemented follow-up actions based on results of integrated support supervision % of districts using financial data in District Implementation Plans Document review: Annual Work Plans, M&E Plans, Quality Improvement Plans, Policy Guidelines, Data review: USAID all KPIs- outcome and output at SO, IR, and activity levels KIIs: USAID, program partners, GOM/MOH staff, health providers and managers at the central, zone district and providers at facility and community levels. Cross-sectional mini￾survey of health facilities; structured questionnaire Program data review, minutes of meetings All intervention health zones and selection of one or more districts in each health zone Time frame for data review and data collection: 2011-2014 Selection of KII participants: representation of central, zonal and district level stakeholders Selection of participants for survey (equal representation of district/community/ facility rural/urban providers) Equal representation of gender PMP data (2011-2014) Qualitative analysis: content analysis, themes, depth to support Quantitative analysis of KPIs trends overtime; Comparison with baseline data obtained from secondary data sources and performance targets Use of a three-point rating system (positive change, negative change, unchanged) for outcomes and (achieved, not achieved, partially achieved) for output indicators Disaggregation of gender data Data triangulation and correlation with quantitative program data: input indicators per intermediate result ,Correlations with implemented activities and budget inputs 47 Evaluation Question Indicator/Measure Data Source/Collection Method Sampling Methodology Data Analysis Methodology #3. What effect has joint coordination and implementation of interventions across the SSD-I had on the achievement of sector-specific and USAID’s health objectives? # of instances where respondents reported: positive/negative/no effect; emerging themes for each type of response Comparative review of:  MOH Departmental and Program Planning;  DIPs; and  SSDI-Services, SSD-I Communication and SSDI￾Systems planning Comparative review of:  Output/results of implementation of these plans and the synergies and challenges that result KIIs: Key USAID, implementing partner, and GOM/MOH staff, health providers and managers at the central, zone, district and community levels, local implementing partners, other health donors Document review: Meeting Reports, Portfolio Management Documents All intervention health zones and selected districts in each health zone Selection of KII participants: representation of central, zonal and district level stakeholders Qualitative analysis: content analysis, themes, depth to support analysis of three dimensions: 1) Overall portfolio management; 2) Management and coordination of the inter-related SSD-I programs; and 3) The inter-SSD-I partner coordination 48 Evaluation Question Indicator/Measure Data Source/Collection Method Sampling Methodology Data Analysis Methodology #4. To what extent has the SSD-I approach responded to client needs and aided families in adopting improved preventative and health-seeking behaviors? Changes in key performance outcome and output indicators at SO and IR levels at 2011 vs. 201426 The evaluation will also review:  The use of evidence (both existing data and project￾implemented research) to develop campaigns, messages and develop strategies and mode for reaching beneficiaries;  The M&E mechanisms in place (for both SSD-I-Services and SSDI-Communications) to monitor impact of SBCC messages and activities and revise these accordingly and as needed;  Responsiveness of SSDI￾Communications to capacity building and needs of the HEU Data review: SSDI￾Communications Baseline survey, behavioral research and BCC guides KIIs with implementing partners, including MOH HEU Data extraction from secondary sources: DHIS, DHS, MICS, GIS Document review: Annual Work Plans, Annual Reports, PMPs, Policy Guidelines Cross-sectional mini-survey of health providers at the district and community levels using a structured questionnaire FGDs with beneficiaries at the community level: men and women of reproductive age (in separate groups); pregnant women (multiparous with living infants and children under five years) All intervention health zones and selected districts in each health zone Selection of KII participants: representation of central, zonal and district level stakeholders Selection of participants for survey (equal representation of district/community and village level providers) Selection of participants for FGD per community (eight to10 participants, equal gender representation) Qualitative analysis: content analysis, themes, depth to support Data triangulation and correlation with quantitative survey data 26 SSDI-Communications Indicators: 1) Planning/Coordination: # of BCC policies and guidelines; # of DIPs which include SBCC; 2) Evidence-based SBCC Package Development: # of strategies, toolkits, campaigns developed; # of people reached with mass media and community mobilization; 3) Capacity Building: Capacity building tool refined; # of organizational assessments conducted; 4) Best Practices: # of formative research pieces undertaken and utilized. 49 Evaluation Question Indicator/ Measure Data Source/Collection Method Sampling Methodology Data Analysis Methodology #5. What elements of the SSD-I approach have either enabled or limited its ability to improve the capacity of the government in the following areas: health service delivery, fiscal responsibility, management and leadership, behavior change communication, and the use of data for decision-making? What recommendations can be made for overcoming these bottlenecks in the future? Changes in key performance outcome and output indicators at SO and IR levels at 2011 vs. 2014 Output indictors related to capacity building for:  Health service delivery;  Fiscal responsibility;  Management and leadership;  Behavior change communications;  Use of data for decision￾making. KIIs: Key USAID, implementing partner, and GOM/MOH staff, health providers and managers at central, zone, district and community levels, local implementing partners Document review: Review of Quarterly and Annual Reports, Budget Allocations, Meeting Reports, Portfolio Management Documents All intervention health zones and selection of one or more districts in each health zone Selection of KII participants: representation of central, zonal and district level stakeholders Qualitative analysis: content analysis, themes, depth to support 50 ANNEX III: LIST OF DOCUMENTS REVIEWED SSD-I – COMMUNICATIONS Annual Reports FY 12 Annual Report: September 2011 – September 2012 FY 13 Annual Report: September 2012 – September 2013 Quarterly Reports FY 2011, Quarterly report covering, Sept 2011 – Dec 2011 FY 2012, Quarterly report covering, Jan 2012 – Mar 2012 FY 2012, Quarterly report covering, April 2012 – June 2012 FY 2012, Quarterly report covering, Jul 2012 – Sept 2012 FY 2012, Quarterly report covering, Oct 2012 – Dec 2012 FY 2013, Quarterly report covering, Jan 2013 – Mar 2013 FY 2013, Quarterly report covering, Apr 2013 – June 2013 FY 2013, Quarterly report covering, Jul 2013 – Sept 2013 FY 2013, Quarterly report covering, Oct 2013 – Dec 2013 Annual Work plans FY12 Annual Work plan, Oct 2011 – Sept 2012 FY 13 Annual Work plan Oct 2012 – Sept 2013 FY14 Annual Work plan, Oct 2013 – Sept 2014 Projection Description, John Hopkins Bloomberg School of Public Health Performance Management Plan Performance Monitoring Plan, October 2011 – September 2016 Other Documents Findings from the 2012 Baseline Survey of 15 Districts in Malawi Community Mobilization Strategy for Health Support for Service Delivery – Communications – Project Cycle Support for Service Delivery – Communications – Program Description SSD-I SERVICES Annual Reports FY 2012 Annual Progress Report (October 2011 – September 2012) FY2013 Annual Progress Report (October 2012 – September 2013). Final year 2 Annual Report _Appendix A summary of year 2 achievements Quarterly Reports FY 2011, Quarterly report covering, Sept 2011 – Dec 2011 FY 2012, Quarterly report covering, Jan 2012 – Mar 2012 FY 2012, Quarterly report covering, April 2012 – June 2012 FY 2012, Quarterly report covering, Jul 2012 – Sept 2012 FY 2012, Quarterly report covering, Oct 2012 – Dec 2012 FY 2013, Quarterly report covering, Jan 2013 – Mar 2013 FY 2013, Quarterly report covering, Apr 2013 – June 2013 51 FY 2013, Quarterly report covering, Jul 2013 – Sept 2013 FY 2013, Quarterly report covering, Oct 2013 – Dec 2013 Annual Work plans Year 1 Work Plan, October 2011 – September 2012, Jhpiego in collaboration with Save the Children Federation, Inc. Plan International. Year 2 Work Plan, October 2012 – September 2013, Jhpiego in collaboration with Save the Children Federation, Inc. Plan International Year 3 Work Plan, October 2013 – September 2014, Jhpiego in collaboration with Save the Children Federation, Inc. Plan International Performance Monitoring Plan PMP November 2011 to September 2016 Routine Output and Outcome Indicators FY 12 (October 2011 – September 2012) Other Documents Martin Fischer Support for Service Delivery-Excellence (SSD-E) Program, Cooperative Agreement # AID-612- A-11-00003, November 8, 2011 Baseline Assessment and Rapid Situation Analysis Report, June 2012 Clinical Management of HIV in Children and Adults, Malawi Integrated Guidelines for providing HIV Services, 2014 SSD-I HEALTH SYSTEMS Annual Reports Year 1 Quarter 4 (Jul- Sept) and Annual Report October 2011 – September 2012 Year 2 Annual Progress Report October 2012 – September 2013 Year 2 Quarter 2 and Semi-Annual Report Oct 2012 – Mar 2013 Year 3 Semi-Annual Report Oct 2013 – Mar 2014 Quarterly Reports FY 2011, Quarterly report covering Sept 2011 – Dec 2011 FY 2012, Quarterly report covering Jan 2012 – Mar 2012 FY 2012, Quarterly report covering April 2012 – June 2012 FY 2012, Quarterly report covering Jul 2012 – Sept 2012 FY 2012, Quarterly report covering Oct 2012 – Dec 2012 FY 2013, Quarterly report covering Jan 2013 – Mar 2013 FY 2013, Quarterly report covering Apr 2013 – June 2013 FY 2013, Quarterly report covering Jul 2013 – Sept 2013 FY 2013, Quarterly report covering Oct 2013 – Dec 2013 FY 2014, Quarterly report covering April 2014 – June 2014 Performance Monitoring Plans Performance Monitoring Plan, October 2011 – September 2012 52 Baselines and Other Documents Malawi Support for Service Delivery, Sector III: Health Policy and Systems Strengthening (HPSS) RFA No 674- 10-0051 Technical Application, September 13, 2010 Support for Service Delivery - Systems; Program Description Guidelines for Policy Development and Analysis for the Public Health Sector in Malawi, February 2014, GOM/MOH Findings of the Supportive Supervision Assessment of the Pilot Phase, March 2014 Integrated Supportive Supervision in Malawi- Concept Note; Recommendations for MOH SS Task Force, August 2012. Ministry of Health Performance Management System Pilot Implementation Assessment Report, December 2013 Ministry of Health iHRIS Pilot Implementation Assessment Report, February 2014 SSDI-Systems Project Mid-Term Progress Report Developing and reviewing public sector health policies and guidelines to enhance evidence-based decision￾making, A brief presentation to Senior Management Committee; GOM/MOH, December 2012. Consolidated Financial Management Capacity Assessment for Fifteen Districts Health Offices, August 2013 Malawi Health Finance Situation Analysis Report GOM/MOH, September 2012 Malawi Health Financing Strategy; Technical Evaluation of Options May 2013 Performance Based Incentives in Malawi: A Situational Analysis, September 2012, Health Systems 20/20 General Documents Integrated Health Services - What and Why? Making health systems work, Technical Brief No1, 2008, World Health Organization. Malawi Ministry of Health Sector Strategic Plan 2011 – 2016, GOM/MOH 53 ANNEX IV: LIST OF KEY INFORMANTS INTERVIEWED USAID/Malawi Lilly Banda, Deputy Director Health Office Ruth Madison, Alternative AOR SSDI Services and Family Health Team Leader Chimwemwe Chitsulo, Monitoring, Evaluation and Learning Specialist and COR for the evaluation contract Gomezgani Jenda, AOR SSDI-Communications Violet Orchardson, AOR SSDI-Services Ndasowa Chitule, AOR SSDI-Systems Central Ministry of Health Chris Kang’ombe, Secretary for Health Samuel Chembe, Deputy Secretary to the Policy and Planning Commission Dominic Nkhoma, Health Economist, Policy and Planning Unit Salim Sumaisi, Director of Human Resources Department Doreen Ali, Director of National Malaria Program Frank Chimbwandira, Director of HIV/AIDS Unit Twambirire Phiri, Acting Director, Reproductive Health Unit Rabson Kachala, Deputy Director, Health Sector Wide Approach (SWAp) H. Kamkwamba, Director, Health Education Unit Austin Makwakwa, Senior Officer, Health Education Unit Sangwani Phiri, Senior Officer, Health Education Unit Gillian Nkhalamba, Focal person HRH management Shira Bandazi, Director of Nursing Melia Maganga, Director of Finance Precious Mtotha, Deputy Director of Finance Malumbo Kausi, Principal Accountant Blessings Chipendo, Chief Accountant Washington Kayimvi, Coordinator of Donor Funds Felix Pensulo, Deputy Director, Department of Nutrition and HIV/AIDS Diane Khonje, Chief, Department of Reproductive Health Humphreys Nsona, IMCI Unit Manager 54 Janet Guta, Deputy Director, Directorate of Clinical Services Zonal and District-Level MOH Mc Donald Msadala, Zonal Health Officer, Southeast Zone Evelyn Chitsa Banda, Central West Zone Office Director Owen Musopole, Zonal Supervisor, Northern Zone Malangizo Mbewe, Zonal Supervisor, South West Zone Mwawi Mwale, Lilongwe District Health Officer Amber Majidu, Chikwawa District Health Officer Florence Bwanali, Machinga District Health Officer Leonard Mchombo, Machinga District Hospital Administrator Macleod Piringu, Machinga District HIV/AIDS Coordinator Jeffrey Jooma, Mangoshi District Medical Officer William Peno, Mangoshi District Health Officer Chilopa Kondwani, Mangoshi District HMIS Officer Kondwani Mamba, Mangoshi District Environmental Health Officer Dr. Mbamba, Salima District Health Officer K.C. Kondowe, Kasungu District Environmental Health Officer J.Z. Nkhambule, Kasungu District Health Officer Prosper Mbemba, Kasungu District DHIS Officer Tiyezge Dhaamini, Kasungu District Community Nurse Coordinator Serra Chanachi, Kasungu District Hospital Matron Betha Chikuse, Balaka District Health Officer Chifundo Mambulu, Balaka District Nursing Officer Moses Nyirenda, Financial Management Coach, Southwest Zone Tionge Mhango, Hospital Administrator, Kaseye Hospital, Chitipa District Dr. Eugene Katenga Kaunda, Chitipa District Health Officer SSDI-Services Dan Wendo, Chief of Party Premilla Bartlett, Deputy Chief of Party Lolade Oseni, Senior Monitoring and Evaluation Advisor 55 Thoko Lusinje, Manager of Health Programs, Plan Malawi Joby George, the former Representative of Save the Children in Malawi Augustine Chikoko, Malawi Care International Representative Mathews Chavunya, Machinga District Community Coordinator Wezi Banda, Zonal Technical Specialist, South West Zone Jane Banda, Lilongwe District Team Leader Thokozani Bema, South East Zone Zonal Manager Joviter Mwaulemu, Salima District Team Leader Mutiphalile Bundula, Salima District M&E Coordinator Joseph Ntwana, Salima District Clinical Coordinator Chimwemwe Limani, Salima District Community Coordinator Martha Chiwaya, Machinga District Community Coordinator Christabel Namomde, Balaka District Team Leader Chikondi Makawa, Balaka District Community Coordinator Noel Kasomekera, Mangochi District Community Coordinator Grevasio Chamatambe, Mangoshi District Team Leader Simplex Chithyola, Community Mobilization Coordinator Balaka District Vennie Arcado, District Team Leader, Chikwawa Janet Nkhoma, Kasungu District Team Leader Anacklet Lupiya, Kasungu District Community Coordinator Abraham Malumba, Kasungu District Clinical Coordinator Felix Phuka, Kasungu District M&E Coordinator Amos Kachulu, Chitipa District Team Leader Chikosa Ngwira, Northern Zone Manager SSDI-Communications Fayyaz Khan, Chief of Party Alinafe Kasiye, Deputy Chief of Party Angela Chitsime, Southwest Zone SBCC Coordinator Johnathan Ziba, SBCC Coordinator Alexander Maganga, Zone Community Mobilization Officer 56 SSDI-Systems Takondwa Mwase, Chief of Party Jacob Kawonga, Senior M&E Advisor George Ngwira, Northern Zone Finance Management Coach Moses Nyirenda, Southwest Zone Finance Management Coach FICA Nicholas Bosscher, Attaché for Development Cooperation, Embassy of Belgian Delegation of Flanders (Formerly FICA) Norway Embassy Hildegunn Tobiassen, First Secretary, Norwegian Embassy KfW Patrick Rudolph, KfW Malawi Office Director CHAM Dr. Mwayi Makoka, Executive Director 57 ANNEX V: DATA COLLECTION INSTRUMENTS MALAWI SSD-I PROJECT PERFORMANCE EVALUATION Key Informant Interview Guide 1. How did the SSDI-Services activity interventions improve the delivery of EHP services and health indicators at the activity sites? What more could be done? Please give examples. 2. Which approaches of the SSDI-Services activity were successful in improving the capacity of the government in health service delivery? Please give examples. 3. How did the SSDI-Communications activity interventions respond to client needs and support families in adopting improved preventative and health seeking behaviors? What more could be done? Please give examples. Probe for gender and disability inclusion. 4. Which approaches of the SSDI-Communications activity were successful in improving the capacity of the government in behavior change and communication? Please give examples. 5. How did the SSDI-Systems activity interventions strengthen health systems at the central, zonal and district levels and respond to specific bottlenecks that impacted service delivery? What more could be done? Please give examples. 6. Which approaches of the SSDI-Systems activity were successful in improving the capacity of the government in fiscal responsibility, management and leadership and use of data for decision-making? Please give examples. 7. What can be done to overcome bottlenecks for improving the government’s capacity in the areas of health service delivery, health systems strengthening and behavior change communications in the future? 8. How has the SSD-I model (programmatic and service integration of the three sectors) affected the achievement of MOH and USAID’s health objectives? Has the SSD-I model been successful? Why or why not? Probe: any effect of joint planning and coordinated implementation. 58 MALAWI SSD-I PROJECT PERFORMANCE EVALUATION Focus Group Discussion Guide FOR: Women of Reproductive Age OR Men of Reproductive Age (Separate FGDs) We are from IBTCI and are conducting a survey to help the MOH and partners improve the services and care provided to you. We are here today to talk about your health and that of your children. This group discussion should not take more than 30 minutes and all answers will remain confidential; however you may choose not to answer any questions if you are not comfortable. We hope you will, as your responses will assist the MOH and partners to improve health services provided to you. May be begin? Interviewer: ________________ Date: ___________________ District: ___________________ Community: ___________________ GPS coordinates (latitude/longitude): ___________________ 1. What are the primary health concerns/diseases that affect you and your family and this community? Prompt to reach: malaria, nutrition, WASH, FP, MNCH, HIV. 2. What information have you received/heard in the last two to three months about these, and from whom/where? From where/whom do you get most health information and advice? No prompt. List all examples given – for any/all diseases: radio, HAS, community volunteer, community based activity. 3. What services can/do you access at your closest health facility? How has this changed/improved in the last year? Prompt as needed: use of MNCH services, use of FP services; HIV services; malaria, WASH and nutrition services. 4. What about in your community? What new/different information or services are available now compared to a year ago? Prompt as needed: MNCH, FP, HIV, malaria, WASH and nutrition services. 5. Has anything changed in your family and community health behavior in the last year as a result of the messages and information you have heard? Prompt as needed: use of MNCH services, use of FP services; HIV services; malaria, WASH and nutrition services. 6. How has your communication with and decision-making with your husband/wife changed regarding the health of your family? No prompt: List any/all examples and suggestions given. 7. What information and/or services could still be improved in your community/facility? No prompt: List any/all examples and suggestions given. 59 MALAWI SSD-I PROJECT PERFORMANCE EVALUATION Focus Group Discussion Guide FOR: Community Volunteers We are from IBTCI and are conducting a survey to help the MOH and partners improve the services and care provided to you. We are here today to talk about health in your community. This group discussion should not take more than 30 minutes and all answers will remain confidential; however you may choose not to answer any questions if you are not comfortable. We hope you will, as your responses will assist the MOH and partners to improve health services provided to you. May be begin? Interviewer: ________________ Date: ___________________ District: ___________________ Community: ___________________ GPS coordinates (latitude/longitude): ___________________ 1. What are the primary health concerns/diseases that affect your community? Prompt to reach: malaria, nutrition, WASH, FP, MNCH, HIV. 2. What training and assistance have you received in the last one to two years to assist you in your community health work? No prompt. List all examples given – for any/all diseases: training, supervision, BCC materials, health supplies, other. 3. What changes in behavior amongst families have you noticed as a result of your work and the assistance given you and your local health facility? Prompt as needed: improved knowledge and awareness; support amongst couples; use of MNCH services, use of FP services; HIV services; malaria, WASH and nutrition services. 4. Is there increased demand for and use of community and facility based services in your community compared to a year ago? Which and why? Prompt as needed: MNCH, FP, HIV, malaria, WASH and nutrition services. 5. What is your relationship with the HSA and facility now, and is it better or worse than one year ago? How? Prompt as needed: Improved coordination training, supervision, reporting, feedback, supplies, other. 6. What additional support would you like to help you in your work and/or to improve health services in your community/facility? No prompt: List any/all examples and suggestions given. 60 Mini-survey Service Delivery Questionnaire 14 AUGUST VERSION International Business & Technical Consultants, Inc. (IBTCI) AUGUST 2014 FACILITY IDENTIFICATION 1 FACILITY NUMBER (LEAVE THE BOXES BLANK) . . . . . . . . . . . . . . . . . . . . . . 2 NAME OF FACILITY 3 LOCATION OF FACILITY (NAME OF DISTRICT) 4 TYPE OF FACILITY HOSPITAL ............................................................................................................................................................................................................... 1 RURAL HEALTH CENTRE ................................................................................................................................................................................. 2 URBAN HEALTH CENTRE ................................................................................................................................................................................ 3 61 INTERVIEWER VISITS 1 2 3 FINAL VISIT 005 DATE DAY MONTH YEAR 006 INTERVIEWER NAME INT. NUMBER 007 RESULT RESULT CODE RESULT CODES (LAST VISIT): 1 = COMPLETED 2 = POSTPONED 3 = PARTIALLY COMPLETED 4= REFUSED 6 = OTHER (SPECIFY) 62 2 0 1 4 CONSENT FIND THE MANAGER/THE PERSON IN-CHARGE OF THE FACILITY, OR THE MOST SENIOR HEALTH WORKER WHO IS PRESENT AT THE FACILITY READ THE FOLLOWING GREETING: Hello. My name is , and I work with IBTCI. We are conducting a survey, and we would like you to participate in it. I would like to ask you some questions about the SSD-I project and the assistance provided to your health facility in the last two years. This information will help the MOH and project improve its support to you in future. The survey usually takes about 20 minutes. Regardless of the information that you provide to us your answers will remain completely confidential and will not be revealed to anyone. Participation in this survey is voluntary, and you may decide not to answer any questions you choose. However we hope that you will participate in this survey since your answers are very useful to us. Do you have any questions about for me about the survey? May we begin? YES, PERMISSION IS GIVEN GO TO Q101 TO RECORD THE TIME AND BEGIN THE INTERVIEW NO, PERMISSION IS NOT GIVEN WRITE 4 (RESULT CODE) IN Q007 . STOP THE INTERVIEW INTERVIEWER'S SIGNATURE INDICATING CONSENT OBTAINED DAY MONTH YEAR 101 RECORD INTERVIEW START TIME . . HOURS MINUTES 102 NAME OF THE HEALTH CARE PROVIDER COMPLETING THE SURVEY 103 SEX OF THE RESPONDENT MALE..........................................................................................................................1 FEMALE......................................................................................................................2 104 What is your current occupation category or qualification? For example, are you a registered nurse or a medical doctor? GENERALIST MEDICAL DOCTOR...................................................................01 SPECIALIST MEDICAL DOCTOR......................................................................02 CLINICAL OFFICER. .............................................................................................03 MEDICAL ASSISTANT..........................................................................................04 REGISTERED NURSE 05 ENROLLED NURSE 06 COMMUNITY HEALTH NURSE 07 ENROLLED NURSE MIDWIFE............................................................................08 LABORATORY TECHNICIAN. ..........................................................................09 ENVIRONMENTAL HEALTH OFFICER. ..........................................................10 HEALTH SURVEILLANCE ASSISTANT. . . . . . . . . . . . . . . 11 HIV TESTING AND COUNSELING (HTC) COUNSELORS.......................12 GENERAL ADMINISTRATIVE STAFF................................................................13 OTHER 96 63 201 What type of health facility services are offered at your facility? What types of services are available in this area today? CIRCLE ALL THAT APPLY. PROBE TO ARRIVE AT APPROPRIATE RESPONSE FAMILY PLANNING . . . . . . . . . . . . . . . . . . . . . . A MATERNAL CARE. . . . . . . . . . . . . . . . . . . . . . . B POSTPARTUM AND NEWBORN CARE . . . . . C CHILD HEALTH . . . . . . . . . . . . . . . . . . . . . . . . . D NUTRITION . . . . . . . . . . . . . . . . . . . . . . . . . . . . E MALARIA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F HIV/AIDS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . G 202 01 02 03 04 05 06 07 08 09 10 11 Please tell me if your facility has received any of the following assistance from SSD-I technical training IF ASSISTANCE IS RECEIVED, rate usefulness on scale 1-3 1 = NOT AT ALL 2= USEFUL 3= VERY USEFUL (A) RECEIVED IF ASSISTANCE RECEIVED (B) USEFULNESS YES NO NOT AT ALL USEFUL VERY USEFUL FAMILY PLANNING 1 2 1 2 3 Basic Emergency Management Obstetric and Newborn Care (BEmONC) & Maternal Neonatal and Child Health (MNCH) 1 2 1 2 3 Emergency Triage and Treatment (ETAT) 1 2 1 2 3 INTEGRATED MANAGEMENT OF CHILDHOOD ILLNESS 1 2 1 2 3 PERFORMANCE QUALITY IMPROVEMENT FOR INFECTION PREVENTION 1 2 1 2 3 HIV/AIDS (HTC, PMTCT, ART, TB) 1 2 1 2 3 Water and Sanitation Hygiene (WASH) 1 2 1 2 3 MALARIA 1 2 1 2 3 Community Case Management (CCM) 1 2 1 2 3 COMMUNITY MOBILIZATION 1 2 1 2 3 OTHER (SPECIFY) 1 2 1 2 3 203 01 02 03 04 05 06 07 08 09 Please tell me if your facility has received any of the following assistance from SSD-I Other Support/Training IF ASSISTANCE IS RECEIVED, rate usefulness on scale 1-3 1 = NOT AT ALL 2= USEFUL 3= VERY USEFUL (A) RECEIVED IF ASSISTANCE RECEIVED (B) USEFULNESS YES NO NOT AT ALL USEFUL VERY USEFUL SUPPORT SUPERVISION 1 2 1 2 3 COACHING AND MENTORSHIP 1 2 1 2 3 PERFOMANCE MANAGEMENT 1 2 1 2 3 FINANCIAL MANAGEMENT 1 2 1 2 3 PROVISION OF EQUIPMENT, PHARMACEUTICAL SUPPLIES 1 2 1 2 3 IEC MATERIALS 1 2 1 2 3 HEALTH MANAGEMENT INFORMATION SYSTEMS 1 2 1 2 3 SUPPLY CHAIN MANAGEMENT 1 2 1 2 3 OTHER (SPECIFY) 1 2 1 2 3 64 204 In what ways has the assistance given in the above areas improved efficiency in service delivery? CIRCLE ALL THAT APPLY. PROBE TO ARRIVE AT APPROPRIATE RESPONSE REDUCED WAITING TIME . . . . . . . . . . . . . . . . A SERVICE INTEGRATION. . . . . . . . . . . . . . . . . B AVAILABILITY OF MEDICINES, SUPPLIES AND MATERIALS . . . . . . . . . . . . . . . . . . . . . . . . . . . C ACCOUNTABILITY AND FINANCIAL REPORTI D USE OF DATA FOR DECISION MAKING . . . . . E IMPROVED LMIS . . . . . . . . . . . . . . . . . . . . . . . F MULTI-SKILLING. . . . . . . . . . . . . . . . . . . . . . . . G NONE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H OTHER X (SPECIFY) 205 How has the assistance received improved the quality of services? CIRCLE ALL THAT APPLY. PROBE TO ARRIVE AT APPROPRIATE RESPONSE PATIENT CARE . . . . . . . . . . . . . . . . . . . . . . . . A INFECTION PREVENTION AND CONTROL. . . B ADHERENCE TO STANDARD OPERATING PROCEDURES. . . . . . . . . . . . . . . . . . . . . . . . . C ATTITUDES/MOTIVATION OF HEALTH CARE WORKERS. . . . . . . . . . . . . . . . . . . . . . . . . . . . D CLIENT SATISFACTION. . . . . . . . . . . . . . . . . . E NONE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . F OTHER X (SPECIFY) 206 Has the assistance increased utilization of the facilities? YES............................................................................................. 1 NO............................................................................................. 2 208 207 How has the assistance increased utilization of the facilities? CIRCLE ALL THAT APPLY. PROBE TO ARRIVE AT APPROPRIATE RESPONSE REDUCED DROP-OUTS . . . . . . . . . . . . . . . . . A IMPROVED ADHERENCE TO TREATMENT. . . B INCREASED ULTILIZATION BY WOMEN . . . . C INCREASED ULTILIZATION BY MEN . . . . . . . D OTHER X (SPECIFY) 208 In what ways did the assistance received improved access to services? CIRCLE ALL THAT APPLY. PROBE TO ARRIVE AT APPROPRIATE RESPONSE INCREASED NUMBER OF OUTREACH SERVICES. . . . . . . . . . . . . . . . . . . . . . . . . . . . . A TASK-SHIFTING . . . . . . . . . . . . . . . . . . . . . . . . B INCREASED POPULATION/GEOGRAPHIC COVERAGE. . . . . . . . . . . . . . . . . . . . . . . . . . . . C INCREASED TYPE OF SERVICES . . . . . . . . . . D OTHER X (SPECIFY) 209 01 02 03 04 05 06 Did the project support integration any of the services in this facility? YES NO PARTIALLY Antenatal Care /Prevention of Mother to Child Transmission(ANC/PMTCT) 1 2 3 HIV/TB 1 2 3 FAMILYPLANNING/TB 1 2 3 NUTRITION / HIV /TB 1 2 3 FAMILYPLANNING/HIV 1 2 3 OTHER (SPECIFY) 1 2 3 210 What other inputs from the project would be useful to resolve health services delivery gaps? 211 RECORD INTERVIEW FINISH TIME HOUR . . . . . . . . . . . . . . . . . . . . MINUTES . . . . . . . . . . . . . . . . . . 65 INTERVIEWER'S OBSERVATIONS TO BE FILLED IN AFTER COMPLETING INTERVIEW COMMENTS ABOUT RESPONDENT: COMMENTS ON SPECIFIC QUESTIONS: ANY OTHER COMMENTS: 66 ANNEX VI: INFORMED CONSENT STATEMENT Hello. My name is ______________________________ and I work with IBTCI. We are conducting a survey and we would like you to participate in it. I would like to ask you some questions about the SSD￾I project and the assistance provided to your health facility in the last two years. This information will help the MOH and project improve its support to you in future. The survey usually takes 20 minutes. Regardless of the information that you provide to us, your answers will remain completely confidential and will not be revealed to anyone. Participation in this survey is voluntary and you may decide not to answer any of the questions, if you choose. However, we hope that you will participate in this survey since your answers are very useful to us. Do you have any questions for me about the survey? May we begin? Interviewer's signature: _________________________ Date: ________________ 67 ANNEX VII: SSDI-SERVICES: PROGRESS AGAINST PMP TARGETS Services PMP Indicators: Baseline FY12 qtr3 FY12 qtr4 FY13 qtr1 FY13 qtr2 FY13 qtr3 FY13 qtr4 14 target FY14 qtr1 FY14 qtr2 Family Planning # of sites that offer LAPM 77 129 129 192 203 203 215 229 223 236 Couple Years of Protection 60,548 19,222 22,161 45,080 63,629 90,259 94,970 352,726 101,383 110,823 Maternal Health # of sites provide BEmONC 43 37 50 65 65 70 74 103 82 90 % of skilled births in the facility 71% 94% 94% 94% 93% 95% 95% 96% 94% 93% Neonatal and Child Health % of newborns resuscitated 57% 47% 75% 76% 73% 76% 87% 90% 83% 85% Nutrition # of facilities with CMAM 98 140 145 206 217 226 228 268 266 266 Malaria % pregnant women receiving IPT2 14% 14% 20% 46% 55% 58% 62% 60% 61% 58% HIV/AIDS % pregnant women tested for HIV 63% 67% 72% 62% 76% 72% 81% 85% 81% 78% # (%) HIV+ PW initiated on ART 2208 3,332 (89%) 3,670 (97%) 2,958 (89%) 4,556 (88%) 4,038 (86%) 4,475 (90%) 26,414 (100%) 5,727 (96%) 5,623 (94%) Mentoring and PQI # facilities w/ 1 supervision quarter 48 - - 119 190 194 212 274 242 233 68 ANNEX VIII: TREND ANALYSIS OF SSD-I INDICATORS Quarterly Trends of Select SSD-I Indicators 77 129 129 192 203 203 215 223 236 0 50 100 150 200 250 Apr-12 Jun-12 Sep- 12 Dec -13 Mar-13 Jun -13 Sep-13 Dec-13 Mar-14 Number of Sites Month Number of service delivery sites that offer LAPM Target September 2013 (229) 61 80 102 147 211 301 396 497 608 0 100 200 300 400 500 600 700 Apr-12 Jun-12 Sep- 12 Dec -13 Mar-13 Jun -13 Sep-13 Dec-13 Mar-14 Number of Interviewees (in thousands) Month Couple Years of Protection Target September 2013 (352,726) 69 43 37 50 65 65 70 74 82 90 0 20 40 60 80 100 Apr-12 Jun-12 Sep- 12 Dec -13 Mar-13 Jun -13 Sep-13 Dec-13 Mar-14 Number of Sites Month Number of Sites that Provide BEmONC 71 94 94 94 93 95 95 94 93 0 10 20 30 40 50 60 70 80 90 100 Apr-12 Jun-12 Sep- 12 Dec -13 Mar-13 Jun -13 Sep-13 Dec-13 Mar-14 Percentage Month Percentage of Births Attended by a skilled birth attendant in the facility Target September 2013 (96%) Target September 2013 (103) 70 57 47 75 76 73 76 87 83 85 0 10 20 30 40 50 60 70 80 90 100 Apr-12 Jun-12 Sep- 12 Dec -13 Mar-13 Jun -13 Sep-13 Dec-13 Mar-14 Percentage Month Percentage of newborns not breathing at birth that were resuscitated successfully Target September 2013 (90%) 98 140 145 206 217 226 228 266 266 0 50 100 150 200 250 300 Apr-12 Jun-12 Sep- 12 Dec -13 Mar-13 Jun -13 Sep-13 Dec-13 Mar-14 Number of Facilities Month Number of health facilities with established capacity to manage acute malnutrition (CMAM) Target September 2013 (268) 71 14 14 20 46 55 58 62 61 58 0 10 20 30 40 50 60 70 80 90 100 Apr-12 Jun-12 Sep- 12 Dec -13 Mar-13 Jun -13 Sep-13 Dec-13 Mar-14 Percentage Month Percentage of pregnant women attending antenatal care who receive IPT2 63 67 72 62 76 72 81 81 78 0 10 20 30 40 50 60 70 80 90 100 Apr-12 Jun-12 Sep- 12 Dec -13 Mar-13 Jun -13 Sep-13 Dec-13 Mar-14 Percentage Month Percentage of pregnant women attending ANC that are tested for HIV Target September 2013 (60%) Target September 2013 (85%) 72 2.208 5.54 9.21 12.168 16.724 20.762 25.237 30.964 36.587 0 5 10 15 20 25 30 35 40 Apr-12 Jun-12 Sep- 12 Dec -13 Mar-13 Jun -13 Sep-13 Dec-13 Mar-14 Number of Women (in thousands) Month Number of HIV+ pregnant women initiated on ART 48 119 190 194 212 242 233 0 50 100 150 200 250 300 Apr-12 Jun-12 Sep- 12 Dec -13 Mar-13 Jun -13 Sep-13 Dec-13 Mar-14 Number of Facilities Month Number of health facilities in SSD-I districts receiving one or more supervision visits per quarter Target September 2013 (26,414) Target September 2013 (274) 73 ANNEX IX: SSDI-SERVICES: LISTS OF INTERVENTIONS LIST OF TECHNICAL INTERVENTIONS Family Planning and Reproductive Health Long Acting and Permanent Methods (LAPM) Emergency Contraception (EC) Post-partum Family Planning (PPFP) Community Based Distribution (CBD) Youth Friendly Health Services (YFHS) HIV and TB Elimination of Maternal to Child Transmission (EMTCT) Initiation of HIV+ Pregnant Women on ART Provider Initiated HIV Testing and Counseling (PITC) Improving HIV Testing and Counseling Skills (HTC) Early Infant Diagnosis (EID) Support for Improving Capacity for TB Management Nutrition Scaling up Nutrition (SUN) Community Management of Acute Malnutrition (CMAM) Malaria Support for Malaria Case Management Uptake of IPT2 for Pregnant Women Maternal and Newborn Health Basic Emergency Management of Obstetrics & Newborn Care Focused Antenatal Care (FANC) Post Abortion Care (PAC) Cervical Cancer Screening Helping Babies Breathe (HBB) Kangaroo Mother Care (KMC) Community Based Management of Maternal and Neonatal Care (CBMNC) 74 Helping Mothers Survives (HMS) Emergency Triage and Treatment (ETAT) Maternal and Neonatal Death Audits Community Based use of Misoprostol to Prevent Postpartum Hemorrhage (PPH) Child Health Integrated Management of Childhood Illnesses (IMCI) Integrated Community Case Management of Childhood Illnesses (iCCM) Child Health Days and EPI + Integrated Family Outreach Days LIST OF CROSS-CUTTING INTERVENTIONS Capacity Building Training Mentoring and Supervision Clinical Guidelines and Training Manuals Performance Quality Improvement (PQI) Standard-Based Management and Recognition (SBM-R) Commodity Facilitation Assisting Facilities to Receive Commodities Equipment and Supplies Provision of Equipment and Supplies Infrastructure Improvements Solar Suitcases Minor Refurbishments Transportation Support Motorcycle Ambulances Fuel Logistics Management LMIS Improving Monitoring and Evaluation Data Quality Assessment Tool (DQA) Data Management Support (DHIS2) Performance-Based Incentives (PBI) 75 LIST OF RECOMMENDED HIGH-IMPACT PRIORITY INTERVENTIONS Family Planning and Reproductive Health LAPM, CBD HIV and TB Initiation of HIV + Pregnant Women on ART, HTC, EID Nutrition CHAM Malaria iCCM, IPT2 Uptake Maternal, Newborn and Child Health BEmONC, FANC, HBB, KMC, HMS, PPH 76 ANNEX X: FINDINGS FROM THE FACILITY MINI-SURVEY Purpose and Methodology: The mini-survey was conducted to assess the usefulness and effectiveness of training and support provided by SSD￾I amongst 63 health workers in 24 health facilities (eight district hospitals, eight urban and eight rural health centers) in eight districts across five zones. Four cadres of service providers (Clinical Officers and Medical Assistants; Nurse Midwives, and Health Surveillance Assistants or HSAs) participated in the survey of which 42 were males and 21 were females. All facilities were supported by SSD-I and offered health services across the project’s six focal areas. SSD-I technical training assistance provided: Table 2.2a summarizes the various types of technical trainings provided to different cadres of health care workers interviewed. Table 2.2a. Percentage of health care providers who received SSD-I technical training by type. Type of Assistance Clinical Officer/ Medical Assistant Nurse Midwife H.S.A. Other Total Family Planning 82.4 72.0 90.9 60.0 76.2 BEmONC& MNCH 82.4 76.0 54.6 50.0 69.4 Emergency Triage and Treatment (ETAT) 82.4 56.0 45.5 50.0 60.3 Integrated Management of Childhood Illness (IMCI) 82.4 60.0 90.9 60.0 71.4 PQI, Infection Prevention 94.1 84.0 54.6 60.0 77.8 HIV/AIDS (HTC, PMTCT, ART, TB) 76.5 88.0 54.6 60.0 74.6 Water and Sanitation Hygiene (WASH) 35.3 28.0 27.3 50.0 33.3 Malaria 58.8 60.0 45.5 60.0 57.1 Integrated Community Case Management (iCCM) 76.5 64.0 81.8 70.0 71.4 Community Mobilization 52.9 56.0 63.6 70.0 58.7 Other 17.7 12.0 27.3 10.0 15.9 Total 17 25 11 10 63 77 Usefulness of technical training: Alongside a high volume of personnel trained in initial or refresher courses by SSD-I, all respondents felt that trainings were “useful”, the majority rating them “very useful”. Examples of the usefulness of trainings included: new skill sets such as MVA and LAPM, multi-skilling and increased coverage of iCCM. Nonetheless, a large proportion of respondents felt more targeted training was needed, either for themselves or their colleagues, across all technical areas. Table 2.2b. Percent and distribution of "usefulness" of SSD-I technical training amongst health care providers. Type of Assistance Clinical Officer/ Medical Assistant Nurse Midwife Health Surveillance Assistant Other Not at All Useful Very Useful Not at All Useful Very Useful Not at All Useful Very Useful Not at All Useful Very Useful Family Planning 0.0 7.1 92.9 0.0 16.7 83.3 0.0 0.0 100.0 0.0 0.0 100.0 BEmONC & MNCH 0 14.3 85.7 0 10.5 89.5 0.0 0.0 100.0 0.0 20.0 80.0 ETAT 0 14.3 85.7 0 14.3 85.7 0.0 0.0 100.0 0.0 40.0 60.0 IMCI 0 14.3 85.7 0 13.3 86.7 0.0 0.0 100.0 0.0 16.7 83.3 PQI, Infection Prevention 0 6.3 93.8 0 4.8 95.2 0.0 0.0 100.0 0.0 16.7 83.3 HIV/AIDS 0.0 7.7 92.3 0.0 18.2 81.8 0.0 0.0 100.0 0.0 10.6 89.4 WASH 0.0 0.0 100.0 0.0 57.1 42.9 0.0 0.0 100.0 0.0 60.0 40.0 Malaria 0.0 0.0 100.0 0.0 46.7 53.3 0.0 20.0 80.0 0.0 16.7 83.3 iCCM 0.0 30.8 69.2 0.0 25.0 75.0 0.0 0.0 100.0 0.0 14.3 85.7 Community Mobilization 0.0 22.2 77.8 0.0 21.4 78.6 0.0 14.3 85.7 0.0 28.6 71.4 Other 0.0 0.0 100.0 0.0 33.3 66.7 0.0 33.3 66.7 0.0 0.0 100.0 78 SSD-I management and other support provided: In addition to technical training, SSDI-Services provided management and other support at facility level. These activities included: support supervision; coaching and mentorship; provision of equipment, supplies and IEC materials and strengthening of HMIS. Table 2.3: Percentage of health care providers who received management and other support by type of assistance. Type of Assistance Health Care Provider Category Clinical Officer/ Medical Assistant Nurse Midwife Health Surveillance Assistant Other Total Support Supervision 82.4 88.0 72.7 50.0 77.8 Coaching and Mentorship 88.2 88.0 72.7 80.0 84.1 Performance Management 52.9 60.0 45.5 50.0 54.0 Provision of Equipment and Supplies 82.4 96.0 81.8 70.0 85.7 IEC Materials 76.5 96.0 90.9 80.0 87.3 Health Management Information Systems 76.5 64.0 63.6 70.0 68.3 Other 0.0 8.0 0.0 0.0 3.7 Total 17 25 11 10 63 79 Usefulness of management and other support: All respondents indicated that management and other support from SSD-I was either “useful” or “very useful”. Of note, support supervision and provision of equipment and supplies were two of the most highly valued interventions in this category. Still, respondents indicated some gaps or constraints in these support areas. Notably, nearly 100% of respondents requested more equipment, supplies, infrastructural improvements, assistance with transport and the maintenance of vehicles. Overall, respondents felt the usefulness of the technical trainings was compromised by the lack of basic infrastructure and supplies required to support the delivery of quality service. Respondents were aware that the project had limited funds and resources for these functions, but requested that in future, more support be provided in these areas. Similarly, limited or low levels of allowances and per diem alongside MOH human resource constraints, particularly in rural areas, were mentioned as areas for improvement and assistance in future programming. Table 2.3b. Percent and distribution of "usefulness" of SSD-I management and other support amongst health care providers. Type of Assistance Clinical Officer/Medical Assistant Nurse Midwife Health Surveillance Assistant Other Not at All Useful Very Useful Not at All Useful Very Useful Not At All Useful Very Useful Not At All Useful Very Useful Support Supervision 0 0 100 0 18.18 81.82 0 0.0 100.0 0 0.0 100.0 Coaching and Mentorship 0 20 80 0 27.27 72.73 0 12.5 87.5 0 37.5 62.5 Performance Management 0 44.4 55.6 0 20 80 0 0.0 100.0 0 40.0 60.0 Provision of Equipment & Supplies 0 7.1 92.9 0 12.5 87.5 0 0.0 100.0 0 0.0 100.0 IEC Materials 0.0 7.7 92.3 0.0 25.0 75.0 0.0 10.0 90.0 0.0 37.5 62.5 HMIS 0.0 15.4 84.6 0.0 12.5 87.5 0.0 28.6 71.4 0.0 14.3 85.7 Other 0.0 0.0 0.0 0.0 0.0 100.0 0.0 0.0 0.0 0.0 0.0 0.0 80 Improved efficiency from SSD-I support: The majority of respondents felt that SSDI-Services support had increased efficiency of service delivery. Approximately 50% of all respondents named reduced waiting time; service integration; availability of resources and use of data for decision-making as efficiency improvements. Table 2.4. Number and percent distribution of improved efficiency of services. Categories Number Percent Reduced Waiting Time 49 79.0 Service Integration 49 79.0 Availability of Medicines, Supplies and Materials 45 72.6 Use of Data for Decision Making 50 80.7 Multi-Skilling 54 87.1 None 2 3.2 Other 4 6.5 Total Respondents 62 Improved quality of services from SSD-I support: The majority of respondents felt that SSDI-Services support had increased quality of service delivery. Over 80% of respondents listed improved quality in patient care, infection prevention, adherence to standards and protocols, attitudes of health workers and client satisfaction. Table 2.5: Number and percent distribution of improved quality of services. Categories Number Percent Patient Care 52 88.1 Infection Prevention and Control 51 86.4 Adherence to Standard Procedures/Protocols 52 88.1 Attitudes/Motivation of Health Care Workers 55 93.2 Client Satisfaction 55 93.2 None 1 1.7 Other 7 11.9 Total Respondents 59 81 Improved access to services from SSD-I support: Over 60% of respondents named increases in outreach services, task-shifting, geographic coverage, and types of services offered as the reasons access had increased. Table 2.6: Number and percent distribution of improved access to services. Categories Number Percent Increased Number of Outreach Services 48 80.0 Task-Shifting 51 85.0 Increased Population/Geographic Coverage 44 73.3 Increased Type of Services 42 70.0 Other 5 8.3 Total Respondents 60 Improved utilization of services from SSD-I support: All respondents felt that SSDI-Services support had increased service utilization. Respondents felt that utilization had increased more amongst women than men, while 62% mentioned a reduction in dropouts and 78% improved adherence to treatment. Table 2.7: Number and percent distribution of increased utilization of services. Categories Number Percent Reduced Drop-Outs 37 62.7 Improved Adherence to Treatment 46 78.0 Increased Utilization by Women 57 96.6 Increased Utilization by Men 41 69.5 Other 9 15.3 Total Respondents 59 82 ANNEX XI: PROGRESS TOWARDS PMP INDICATORS SSDI-Communications indicators and progress through midterm Baseline FY 12 Results FY 13 Target FY 13 Results Comments Objective 1: Planning and Coordination Number of policies and strategies finalized 0 0 2 2 National Health Promotion Policy, SBCC Strategy Objective 2: Evidence based SBCC Package Development Number of people that have seen/heard an MNCH, FP/RH message 1.2 million 713,566 500,000 1,274,54 3 Target exceeded Percent of target population reached with malaria messages 37% 35% 66% Target exceeded Objective 3: Capacity Building Number of district trainers trained in community mobilization 0 27 42 49 76 TOT total Capacity building plan for HEU in place 0 0 1 1 HEU received training, BCC library, recording studio, M&E systems Objective 4: Best Practices Identified Number of formative research pieces undertaken and utilized 0 1 N/A N/A Baseline includes quantitative and FGD research 83 ANNEX XII: FOCUS GROUP DISCUSSION SUMMARY Focus Group Discussion Summary: Community Action Groups Total Districts: 8 (Lilongwe, Salima, Kasungu, Mangoche, Machinga, Balaka, Chitipa, Chikwawa) Total FDGs conducted: 16 Total number of participants: 137 1. Primary health concerns/diseases that affect the community The most common diseases listed included:  Malaria; waterborne diseases (diarrhea, vomiting, dysentery); HIV/AIDS; TB; cough; bilharzia; skin infections; anemia; eye infections and trachoma; and cancer. Other barriers to good health mentioned include:  Inadequate access to mosquito nets; poor knowledge/barriers to use of mosquito nets; unprotected/unsafe sex; barriers to HTC; myths around family planning; resistance from men in limiting family size; lack of consistent supply of FP commodities; preference for home births with traditional birth attendants (TBAs); complicated deliveries in transit; insufficient messages on MNCH issues; strain (e.g. malnutrition) of frequent childbirth on families; gender issues around male nurses; unavailability of health care workers due to capacity building and training; and shortage of CBD workers. 2. Training and assistance received in the last one to two years Community Action Group Volunteers received trainings on:  Identifying and prioritizing health issues in the community; preparing work plans to address health problems; implementing FP, nutrition, breastfeeding and malaria activities; promoting the use of NMCP and facility based childbirth; conducting under five clinics; and implementing HTC and FP methods;  The primary constraint given was the short timeframe for trainings and the amount of covered in a day. However, the picture cards given out assisted in overcoming this barrier and acted as a reminder while they work. Other assistance received included:  Picture guide books to use in educational activities; shoes, bags, bicycles, umbrellas, t-shirts, flipcharts, condoms, gloves and chlorine solution; and notebooks for writing up reports. 3. Changes in behavior amongst families in the last one to two years Health-seeking behaviors:  Increased understanding of health problems, leading to uptake of HTC, male involvement in family health care;  Health-seeking behavior has increased, preference for modern medicine over traditional. Malaria:  Increased use of services when sick, and compliance with treatment regime;  Increased demand for and use of nets and removal of stagnant water. 84 HIV/AIDS:  Better couple communication, couples testing and disclosure of status;  Increased ARV adherence and decreased stigma and discrimination;  Better understanding and use of HTC and condoms, particularly amongst youth. Family planning:  Mass sensitization leading to majority of women using FP in communities;  HIV positive women encouraged to and do use FP to prevent pregnancies;  Gender based violence around access to and use of FP decreased. “Demand for family planning services has increased; for instance, my register shows an increase from about 36 last year to almost 80 clients at this time.” CAG volunteer The following FP constraints were mentioned:  Demand for family planning has increased and is often higher than supply;  Condoms are more sought after but are not always available;  Men still consider FP to be a woman’s responsibility. MNCH:  Community mobilization, sometimes through and with TBAs, has increased facility based births, though some women still deliver at home;  More men accompany their wives to ANC, PMTCT and to some extent, to deliver at the health facility, though men still see this as expensive and cost-ineffective use of time;  Most pregnant women from the community go for ANC in first trimester because they now know the importance of PMTCT.  Poor attitudes of health workers discourage women from using MNCH services more often;  Increased number of pregnant women attending antenatal services and hospital deliveries; more families with transport schemes and emergency fund savings;  More mothers are breastfeeding exclusively and taking their babies for immunizations. Nutrition:  Community mobilization has increased awareness; families now eat more nutritional food and have vegetable gardens;  Parents understand the importance of giving nutritious foods to their children;  Poverty is still a major constraint in these efforts;  HSAs provision of supplements for malnourished children is important. WASH:  Improved hygiene and sanitation in homes and communities (including toilets, rubbish pits, safe water treatment and storage and hand-washing) as a result of community outreach work;  Increased awareness about dangers of open defecation (near lake, etc.). 85 4. Demand for and use of community and facility based services  Demand for services in general has increased because of community outreach and mobilization efforts;  However, constraints exist: Long distances and impassable roads/routes and low/no consistent stocks of essential medicines and supplies. Nutrition:  Increased demand for nutritional education (six food groups and preparing balanced meals;  Constraints exist around availability of nutritional supplements and time taken to travel back and forth to facilities to get these;  They should be available in communities through iCCM, and particularly for the very poor. Family planning:  Increased demand for FP, especially LAPM;  Low availability of LAPM in community health posts and lack of stock is still a problem and major constraint. WASH:  Safe water sources are very scarce: need more piped water, protected wells and boreholes;  More households practicing good hand-washing;  Though more households have pit latrines, more assistance in constructing these is needed. Malaria:  Increased demand for nets, currently only given to pregnant women;  Low access to/availability of malaria medicines;  Need for ambulance services for sick pregnant women, and children;  Village health posts should be able to test for malaria;  Low/no referral systems to hospitals when needed. MNCH:  Increased provision of services at community level is needed: ANC, PNC, under-five, etc. Also, if women are required to delivery in a health facility, that health facility should to be closer to the community and/or waiting areas bigger to provide for women and their caretakers;  Accessing health services is a challenge due to low/no emergency transport;  iCCM should include treatment of illnesses for children six to 12 years;  More work on acceptance of male nurses, particularly in maternity. HIV/AIDS:  Access is an issue: low/irregular condom supplies, nutritional supplements for PLWHA, availability of HTC and ARTs;  Much of this could and should be provided at community level. 86 5. Community volunteer’s relationship with the HSA and health facility HSA:  Coordination and planning has improved in the last one to two years: technical support is provided by the HSA through bi-monthly meetings in communities;  HSAs are the link between CAGs and health centers, collecting data together, offering feedback and referral mechanism and the provision of supplies;  Most volunteers reported a strong relationship and closer links with their health center, working alongside HSAs at community and family health outreach events;  However, they complain of lack of sufficient resources, feedback, supervision and mentoring. Health Facility:  When the CAGs have issues, they report these to the HSA who takes them to the facility. Mostly, they receive support as a result;  However, there is no direct link between them and the centers, so they have to rely on the HSAs, which doesn’t always work;  They are not recognized or acknowledged for their help by health workers at the center; these one-sided relationships demoralize them. 6. Additional support required by community volunteers  Volunteers would like: bicycles; more training and refresher courses; better counseling skills; some form of “identity” (uniform, badge, bags) as official volunteers; resource materials in local languages; some form of incentive (soap, strong shoes); incentives for meetings and workshops; megaphones to use for community mobilization; more HSAs particularly in hard to reach areas; more LLIN availability; working materials for latrines and improved water sources; more IEC materials; increased availability of basic medicines in communities, especially family planning and first aid boxes; and bicycle ambulances. Focus Group Discussion Summary, Male Beneficiaries Total Districts: 8 (Lilongwe, Salima, Kasungu, Mangoche, Machinga, Balaka, Chitipa, Chikwawa) Total FDGs conducted: 16 Total number of participants: 124 1. Primary health concerns/diseases that affect families and the community The most common diseases listed included:  Malaria; waterborne diseases (diarrhea, vomiting, dysentery); HIV/AIDS and STDs; TB; cough; bilharzia; malnutrition; skin infections; anemia; eye infections and trachoma; and cancer. “Today we have male midwives, to make matters worse, they are young and unmarried. Some members of the community do not understand why this is so. Consequently, some women decide to give birth at home with traditional birth attendants rather than at the health facility with health personnel. They feel ashamed to give birth with support of a male midwife”. Male beneficiary 87 Other barriers to good health mentioned included:  Access to safe water and sanitation; inadequate human and other resources at health centers (including drugs); poor supplies of medicines for iCCM; inadequate/poor emergency transport systems; poverty and problems with nutrition; misconceptions and low levels of knowledge about safety of FP; teen pregnancies; inadequate skills of TBAs and community birth resources; and reluctance for women to see male nurse/midwives. 2. Information received and heard in the last two to three months Health information received and heard in the last two to three months include:  Importance of consistent LLIN use;  WASH for Diarrhea prevention;  Seeking treatment fast, especially for malaria;  Importance of male circumcision and prevention of STDs and HIV (condoms, be faithful);  Need to reduce stigma and discrimination of HIV in communities;  Nutrition and the six food groups;  Advantages of family planning for women, families and communities;  Importance of facility based births and danger signs in pregnancy;  Attendance of ANC with your wife, in first trimester and receiving HTC together;  Importance of FP for families, and not just a woman’s issue;  The “Moyo Ndi Mpamba” (Life is capital, without good health one cannot live out one’s plans);  Information was mostly received by: radio, health workers and HSAs, community and CAG volunteers, community dramas, the “Moyo Ndi Mpamba” booklet and songs. 3. Increases or improvements in services available at the closest health facility in the last year Better access and quality care in the following:  ARVs and HIV and STD counseling and testing;  Community health: condoms, door to door HTC;  Malaria screening and improved drug treatment, including Fansidar for pregnant women;  FP and a wide choice of methods;  Skilled birth attendance and better MNCH services in health centers, (coupled with fines for home births);  Immunizations and under-five clinics for children;  Nutritional information and treatment and supplements for malnourished children;  WASH practices – sanitation, hand washing and safe water. Less access due to:  LLINs only for pregnant women, which used to be distributed widely in communities;  Sales of FP by health staff meaning some can’t afford it and poor availability of condoms;  Distance to facilities and lack of community-level skilled birth options;  Distribution/use of nutritional supplements by health personnel and not to rightful patients;  Inadequate staffing and low specialization of services at health facilities;  Lack of consistent supply of drugs and medications apart from Panadol, necessitating a trip to the pharmacy to pay for these. 88 4. New information or services now available in the community The following information or services are newly available in communities:  Counseling and testing for HIV;  New and improved ART medication;  Male circumcision and HIV prevention (male and female condoms);  Fines and punishment for home and in transit deliveries;  Information on malaria and exclusive breastfeeding;  “Moyo ndi mpamba” booklets provided to households and education from CAG members on healthy behaviors to improve on family health;  Family planning for young people;  Encouraging expectant couples to go for ANC together;  WASH campaigns: use of latrines, hand-washing, sanitation. 5. Changes in family and community health behavior  More couples and pregnant women going for HTC;  Consistent use of LLINs, amongst those that have them;  Seeking treatment fast and completing the course of treatment for malaria;  Increased facility based deliveries and planning for emergency transport;  Attendance at ANC with wives;  Improved nutritional knowledge and eating in homes;  Improved WASH at home and in communities;  Increased demand for and better communications around family planning (reducing risky sexual behaviors and domestic violence). No change was noted in the following areas:  Risky sexual practices, despite messages about condom use and safer sex, men and particularly migratory populations continue to engage in high-risk sex. 6. Changes in communication and decision-making between husbands and wives  More open communications and support for about HIV testing, PMTCT, use of services by pregnant women and women’s knowledge and opinions about health and family health needs;  More male involvement and support for family planning and decision-making;  Consultations on use of household resources, WASH and nutrition. 7. Information and/or services that could be improved in the community/facility  Prompt reporting to work by health staff, late openings and early closing of facilities hamper access, weekend hours also;  Provision of ANC and more under-five days at community level  A community level facility to facilitate ANC and facility based births; “Village heads spoke strongly against households that did not have a toilet. Village heads said they would mobilize people to construct a toilet for such households, and the culprit will have to pay these people for constructing the toilet. …this is probably the most famous piece of information in this community this year’’ Male beneficiary 89  Improved water sources and pit latrines;  More nutrition programs and more supplies at community level;  More ARV services at community level;  More information and provision of male circumcision;  More trained health personnel at facilities;  Reliable transport means, especially in emergencies;  Increased awareness raising to overcome barriers to using male midwives;  More training of HSAs to increase skill sets in communities;  Use of other methods for BCC besides print materials;  More information about the safety of FP (to address concerns about cancer);  Adequate and consistent drug stocks;  Better patient care and counseling skills of health care workers. Focus Group Discussion Summary, Female Beneficiaries Total Districts: 8 (Lilongwe, Salima, Kasungu, Mangoche, Machinga, Balaka, Chitipa, Chikwawa) Total FDGs conducted: 16 Total number of participants: 132 1. Primary health concerns/diseases that affect families and the community The most common diseases listed included:  Malaria; waterborne diseases (diarrhea, cholera, vomiting, dysentery); HIV/AIDS; TB; cough; bilharzia; skin infections; anemia; eye infections and trachoma; hypertension, stroke, diabetes and cancer. Other barriers to good health mentioned included:  Malnutrition and low availability of nutritious foods; unsafe water; access to LLINs; home deliveries and births in transit; no/low supplies of FP methods and their side effects (Depo-Provera); FP as a woman’s issue; stigma and discrimination around HIV/AIDS; shame and culture of silence increases spread of HIV; long distances to health center; inadequate staffing at health centers; poverty. 2. Information received and heard in the last two to three months Health information received and heard in the last two to three months include:  Vitamin A, supplemental feeding and growth monitoring for children under 5;  WASH, safe water storage and hand-washing information;  Consistent use of LLINs for malaria prevention and seeking treatment early;  Locally available nutritious foods and preparation;  HIV prevention, including abstinence, condoms, HTC and signs and symptoms of HIV;  FP and LAPM, especially for single and HIV+ women;  Nevaripine for HIV+ pregnant women;  Importance of health facility based births;  Adherence to ARTs and reduction of stigma and discrimination;  “Moyo Ndi Mpamba,” a booklet for households with messages on malaria, hygiene, safe water, family planning, nutrition, breast feeding, HIV prevention and testing, use of LLINs; 90  Information is received mostly through: HSAs, health center personnel, CAGs and volunteers, radio (“Moyo Ndi Mpamba” programme), community leaders and churches. 3. Increases or improvements in services available at the closest health facility in the last year Better access to the following:  Treatment for malaria, TB and other diseases; HTC services and information as well as ART services; anti-malarials, PMTCT, family planning, LLINs, supplements for pregnant women;  Community health: condoms, nutritional information, WASH, iCCM and immunizations. More consistent and higher quality services were also reported:  Amongst MNCH services: skilled (clean and safe) birth attendance, treatment for malnourished children, provision of supplements, Waterguard, LLINs, management of obstetric and neonatal complications;  Integration of services, including HTC for all OPC and ANC clients. 4. Information or services available now in the community The following information or services are newly available in community:  Provision of counseling and supplements from HSAs on healthy practices including nutrition and WASH;  Family planning for youth and unmarried people;  Mobilization of pregnant women to go to ANC in first trimester, for PMTCT and to the health center at eight months for skilled birth attendance;  New ARV regime which are more effective with less side effects;  The “Moyo Ndi Mpamba” radio program includes special and in depth communication messages – including WASH, FP, nutrition, healthy mothers and babies - aimed at encouraging individuals to view their lives as an investment and to do all they can to preserve it. Household hygiene, family planning for families, nutrition education for families and pregnant women for health of mother and baby, exclusive breast feeding for baby during the first six months. 5. Changes in family and community health behavior  Cleaner environment and better hygiene practices, and a reduction in diarrhea;  More family planning, smaller and better spaced families;  Better nutritional habits utilizing locally available nutritious foods;  More consistent use of LLINs and seeking treatment faster with malaria symptoms;  Community-based HSAs improve access to iCCM and other key services;  More women are receiving ANC early, delivering at centers and going for PNC;  More people are getting HIV tested and more men access condoms at centers;  Families are more openly discussing HIV prevention and condom use;  They believe men engage in less risky behavior; they are more faithful than before;  More women are exclusively breastfeeding babies until they are six months. 6. Changes in communication and decision-making between husbands and wives:  Increased support for safe delivery, ANC and PNC and family planning; 91  Better communication about WASH, nutrition and malaria prevention;  More open discussions about condoms, safe sex and more couples going for HTC;  More respect for women’s opinions and right to decide about their health;  Improved sex lives as a result of more open communication and the decision to adopt FP. 7. Information and/or services that could be improved in the community/facility The following are some of the needs of different communities:  Increased numbers of HSAs, midwives, ambulances and emergency transport;  Increased access to blood transfusions and other life-saving services at health centers;  Increased access to safe water and sanitation measures;  Sufficient supplies of blankets, running water, clean toilets, placenta pits, beds and privacy in the maternity ward, especially if women are “forced” to delivery there;  Maternity services closer to communities so it’s easier to deliver where they are supposed to;  More consistent supplies of family planning and other drugs at facility and community level;  More health staff quarters to improve staffing and reduce delays during emergencies;  More nurse/midwives to reduce workload, improve attitudes and reduce maternal and neonatal deaths;  Increased access to LLINs for all community members;  Provision of nutritional supplements at community level;  Better patient care and skills from HSAs, especially when handling under five;  Introduction of ANC, HCT, malaria testing and other adult services at community level;  Building of community centers: the community would provide bricks for this if the MOH would agree;  A place at health centers to store dead bodies until they can be removed. “As a community we are worried because … the government stopped TBA services in the community, but we have transport challenges when a woman goes into labor at night and yet the health center is far. What plans does government have to assist in this area?” -Female beneficiary 92 ANNEX XIII: SSDI-SYSTEMS: PROGRESS AGAINST PMP TARGETS SSDI-Systems Indicators Baseline Value 2011-2012 Results 2012–2013 Results 2013 –2014 Results EOP Target Comments Policies, guidelines, regulations 0 0 0 1 10 Out of 10: 1 approved, 5 awaiting approval, 1 situational analysis stage; 2 newly identified; 1 under discussion % quarterly meetings held by the PDU 0 75% 100% 100% 100% This indicator is a proxy for functionality of the revitalized PDU # non-MOH stakeholders engaged in the policy development 0 - - - 6 This is a new indicator included during the 2014 PMP revision hence no result has been recorded % districts receiving target supervision visits 0 13% 13% 40% 100% # of cost centers that updated iHRIS data 0 - - 9 19 This indicator is a proxy for measuring IHRIS use. Results in 2014 include 3 pilot cost centers and 6 additional. National Health Financing Strategy approved by GOM No No No No Yes Draft National Health Financing Strategy is in place awaiting government approval 93 ANNEX XIV: SSDI-SYSTEMS: ZONAL AND DISTRICT IMPLEMENTATION BY ACTIVITY MANGOCHI MACHINGA BALAKA ZOMBA MULANJE PHALOMBE NKHOTAKOT ANSANJE CHIKHWAWA KASUNGU DOWA SALIMA LILONGWE CHITIPA KARONGA NORTH ZONE CE ZONE CW ZONE SE ZONE SW ZONE MOH HQ TOTAL 1 Strengthen capacity for evidence-based policies X 1 2 In-depth policy analysis X 1 3 Strengthen data for decision-making X X X X X X X X X X X 11 4 Support to NMCP X 1 5 Leadership & management capacity X X X X X X X X X X X 11 6 Integrated supportive supervision X X X X X X X X X X X X X X X X X 17 7 Staff performance appraisal X X X X X X X X X X X X 12 8 Implementation and use of iHRIS X X X X X X X X X X X X X X X X 19+ 9 Development of multi￾year DIP X X X X X X X X X X X X 12 10 District stakeholder forum formation and strengthening X X X X X X X X X X X 11 11 DIP reviews X X X X X X X 7 12 District financial management coaching X X X X X X X X X X X X X X X X 15+ 13 Strengthen health financing mechanisms X 1 Explanatory notes on coverage of some activities: 1. Revitalization of PDU and part-time technical assistance on policy at MOH HQ and working on five health policies and two guidelines 3. Development of training materials on data use and training of seven district-, three zone- and one national-level team (mid-level management) through leadership and management development program 5. Development of training materials on leadership and management at national level and training of seven district-, three zone-, and one national-level team (mid-level management). All central region SSDI-supported districts – Lilongwe, Salima, Dowa, Kasungu, and Nkhotakota – are covered by SSDI Systems as they have been taken by University of Cape Town 6. Twelve districts, four zones, and MOH HQ have been trained on the use of the new, integrated, supportive supervision system – started with a pilot in two districts of Salima and Balaka districts and Zomba central hospital – and only rolled out to other districts in May 2014 94 8. Available in all 19 SSDI-targeted cost centers and SSDI-targeted districts. Through collaboration with MSH, this activity has been rolled-out to all 13 non-SSDI districts across the country. 12. Financial management coaching has covered all the 15 SSDI-targeted districts, MOH HQ and MOLGRD. Financial management covers the following systems issues: planning and budgeting, accounting and reporting, internal controls, procurement, drugs management, and stores management. This activity involves technical assistance at the national level, consisting of NHA exercises, health￾financing situation analysis, health financing options, and technical evaluation. 13. Health financing strategy development, health financing advocacy through national-level financing summits, and development of health financing technical briefs (e.g.: health insurance) for consideration by government; supporting MOH staff working with the Global Fund staff on financial management improvement. 95 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov