MCHIP EVALUATION REPORT 1 END OF PROJECT EVALUATION OF THE MATERNAL AND CHILD HEALTH INTEGRATED PROGRAM (MCHIP) ASSOCIATE AWARD IN ZIMBABWE Task Order: AID-613-TO-16-00005 Contract No: AID-OAA-I-15-00028 FINAL EVALUATION REPORT January 2017 This publication was produced for review by the United States Agency for International Development. It was prepared for The Mitchell Group, Inc. independently by Sean Drysdale, Evaluation Team Leader. MCHIP EVALUATION REPORT 1 END OF PROJECT EVALUATION OF THE MATERNAL AND CHILD HEALTH INTEGRATED PROGRAM (MCHIP) ASSOCIATE AWARD IN ZIMBABWE PERFORMED UNDER TASK ORDER NO. AID-613-TO-16-00005 PPL/LER MONITORING & EVALUATION IDIQ CONTRACT NO. AID-OAA-I-15-00028 Final Evaluation Report January 2017 MCHIP EVALUATION REPORT i DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. MCHIP EVALUATION REPORT ii ACKNOWLEDGEMENTS The Evaluation Team would like to thank all the staff at the MoHCC (Ministry of Health and Child Care) in Harare and Manicaland who took time out of their busy schedules to talk to us about the MCHIP program; the dedicated USAID Zimbabwe staff who provided on-going feedback and support throughout the evaluation; all the health workers who patiently and kindly answered our questions and filled out a lengthy survey, despite long lines of patients waiting for their attention; and the Village Health Workers who traveled long distances and often waited hours to speak to us so enthusiastically about their role in improving health in their communities and then sang us songs in extraordinary harmonies about their work. The team is also immensely grateful for being allowed access to and use of the various records from which data was extracted; and for the clarifications and explanations given as and when required. MCHIP EVALUATION REPORT iii CONTENTS Contents ....................................................................................................................................................... iii List of Tables................................................................................................................................................iv List of Figures ..............................................................................................................................................iv Acronyms ....................................................................................................................................................... v EXECUTIVE SUMMARY ..............................................................................................................................1 Introduction and Background ..............................................................................................................................................1 Evaluation Purpose, Team, Methodology ........................................................................................................................1 Findings and Conclusions .......................................................................................................................................................2 Overall Conclusions...................................................................................................................................................................6 Recommendations .....................................................................................................................................................................7 Evaluation Purpose & Questions ...........................................................................................................8 Evaluation Purpose ...................................................................................................................................................................8 Evaluation Questions................................................................................................................................................................9 Project Background ................................................................................................................................ 10 Context ........................................................................................................................................................................................ 10 Project Overview ..................................................................................................................................................................... 10 MCHIP Project Goal and Objectives ................................................................................................................................. 11 Evaluation DESIGN, Methods & Limitations ................................................................................... 13 District and Sample Selection ............................................................................................................................................ 13 Document Review ................................................................................................................................................................... 14 Qualitative Data ....................................................................................................................................................................... 14 Quantitative Data .................................................................................................................................................................... 16 Quality Assurance and Limitations of Data Collection ............................................................................................ 17 Findings, Conclusions & Recommendations .................................................................................. 19 Conclusions................................................................................................................................................................................ 43 Recommendations .................................................................................................................................................................. 44 MCHIP EVALUATION REPORT iv LIST OF TABLES Table 1. Selected MNCH Indicators 10 Table 2. Qualitative Data 15 Table 3. Quantitative Data 16 Table 4 MCHIP Training Expenditure in USD in Manicaland 2014-2016 22 Table 5 Total MCHIP Expenditure on MCCM and cPQI Activities in Manicaland 2014-2016 23 Table 6 Village Health Worker Knowledge Scores 24 Table 7 Inter-District Comparison of VHWs Knowledge Scores 25 Table 8 Results of Observed VHWs skills around ANC/PNC and Managing Sick Child 28 Table 9 Comparison of VHW Skill Levels: Pre- and Post-Training and 2016 Evaluation 28 Table 10 Distribution of Focus Group Discussion Participants by Location and Gender 34 Table 11 Survey Respondent Characteristics 34 LIST OF FIGURES Figure 1. Annual Malaria Incidence Map, Zimbabwe 2014 14 Figure 2. Training Received by VHWs in 2 Years Prior to the Survey 19 Figure 3. Equipment Availability in Surveyed VHWs 21 Figure 4. Availability of Commodities Reported by Surveyed VHWs 22 Figure 5. Village Health Worker Scores per Subject Area and Overall 25 Figure 6. Percentage of Malaria Workload Undertaken by VHWs 31 Figure 7. Percent of Diarrhea Cases with Dehydration in <5s 32 Figure 8. Percent of Pneumonia Cases Classified as Severe in <5s 32 Figure 9. Percent of Malaria Case Fatality Rate 33 Figure 10. Trend in Early ANC Booking 34 Figure 11. Trend in Early Neonatal Deaths 34 MCHIP EVALUATION REPORT v ACRONYMS AA Associate Award ANC Antenatal Care ARK Absolute Return for Kids Project cHMIS Community Health Management Information System cMNCH Community Maternal Neonatal & Child Health cPQI Community Performance Quality Improvement DHE District Health Executive DFID Department for International Development (United Kingdom) DNO District Nursing Officer EHO Environmental Health Officer ENND Early Neonatal Death EPI Expanded Program for Immunization EU European Union FP Family Planning HBB Helping Babies Breathe HMIS Health Management Information System HTF Health Transition Fund HW Health Worker iCCM Integrated Community Case Management (Of Childhood Illness) IMNCI Integrated Management of Newborn and Child Illnesses IP Infection Prevention IPTp Intermittent Preventive Treatment of Malaria in Pregnant Women IYCF Infant and Young Child Feeding KI Key Informant LBW Low Birth Weight MCCM Malaria Community Case Management MNCH Maternal Neonatal Child Health MoHCC Ministry of Health and Child Care NIHFA National Integrated Health Facility Assessment NMCP National Malaria Control Program PCV Pneumococcal Vaccine PPH Post-Partum Hemorrhage PQI Performance Quality Improvement MCHIP EVALUATION REPORT vi QI Quality Improvement RDT Rapid Diagnostic Test (for Malaria) RED Reaching Every District SND Standard Normal Deviation SS Supportive Supervision TMG The Mitchell Group TWG Technical Working Group USAID United States Agency for International Development USG United States Government VHW Village Health Worker MCHIP EVALUATION REPORT 1 EXECUTIVE SUMMARY Introduction and Background MCHIP was the USAID global flagship maternal and child health program implemented by a consortium of US￾Based NGOs. The Maternal Child Survival Program (MCSP) replaced MCHIP in 2014. The MCHIP/Zimbabwe project was launched in 2010. The first phase was completed in September 2013 and subsequently extended to February 2014. Following a successful external evaluation, MCHIP was granted a follow-on Associate Award (AA) in 2014 to expand its activities from 17 to 36 health facilities and from 1 to all 7 districts in Manicaland Province. The goal of MCHIP project in Zimbabwe is to contribute to accelerated and sustainable improvement in MNCH through the scaling up of evidence-based, high impact, and integrated interventions. The project aims to contribute to reductions in maternal and child mortality and thus support Zimbabwe’s progress towards Millennium Development Goals four and five on reducing child mortality and improving maternal health. MCHIP works with the MOHCC and a number of strategic donor and NGO partners in Zimbabwe at the national level, and in Manicaland Province, which in 2008 still had, the worst MNCH indicators nationally. In Manicaland, MCHIP is active in the Provincial Hospital, district hospitals, rural health centers and communities and at the national level, the project assists the development of national policies, strategies and guidelines. Evaluation Purpose, Team, Methodology The end of project evaluation of the MCHIP-AA was carried out in November and December of 2016. It was limited to the community component of the project with the purpose of: • Informing decision-making and guiding the design and implementation of the new MNCH activity being designed by USAID. • Informing the design and implementation of future PMI-supported malaria community case management (MCCM) activities that are currently part of the MCHIP AA community approach. • Highlighting areas where VHWs need additional training or support to deliver quality community health services • Helping to identify successful community health service delivery and quality improvement approaches. The evaluation addressed the following key questions: • To what extent have VHWs in MCHIP-supported districts effectively delivered quality RMNCH services? • To what extent are VHWs from MCHIP-supported districts similar or different in the performance of duties when compared to VHWs in non MCHIP-supported districts? • What factors have facilitated or hindered village health workers in performing their duties in MCHIP-supported and non-MCHIP supported districts? • What is the acceptability of the USAID-supported cPQI approach to national stakeholders, including key departments within the MoHCC? The Evaluation Team (hereinafter the team) was headed by an international public health physician with extensive expertise in MNCH, health systems and health financing, including many years in Zimbabwe; with two local consultants, one with broad experience in reproductive health and the second a health information and data MCHIP EVALUATION REPORT 2 management expert. The team was supplemented during the fieldwork by 3 focus group discussion leaders and transcribers and supported by VHW trainers recruited from the evaluation districts. The evaluation methodology used a combination of quantitative and qualitative techniques, including: a focused document review; VHW knowledge survey; a practical skills assessment; separate focus group discussions with VHWs, their clients and health center committee members; and key informant interviews in Harare and Manicaland province. Secondary data was examined using the national HMIS, the cHMIS, and VHW registers and program reports. Focusing on the key questions, the evaluation compared VHW performance and attitudes in Chimanimani, the district where MCHIP is implementing the full cPQI1 intervention alongside the MCCM training, with Nyanga, a district where it is supporting MCCM activities only and with a district, Centenary, where MCHIP has no presence to all. Direct indicators of VHWs outputs and outcomes were not available, so the team focused on proxy indicators. The evaluators conducted briefings with MCHIP and the USAID mission at the beginning and end of the fieldwork and provided the latter with weekly written updates. Limitations of the evaluation included the small number of health facilities visited; the differences in data recording tools used by VHWs making direct comparisons difficult; missing or incomplete data; logistical and time constraints to conduct the skills assessments; and the absence of many of the key informants. Findings and Conclusions To what extent have VHWs in MCHIP-supported districts effectively delivered quality RMNCH services? The main inputs from the program in the intervention district were: development of training materials and job aids; support to training on MCCM, cPQI and peer supervision; support to post training follow up (PTFU); support to supervision by both health facility staff and peer supervisors; and provision of limited equipment and supplies. All VHWs had the appropriate job aids – registers, report forms and health education flip charts. Training coverage was high in Chimanimani, with most subject areas reaching 90%+ of VHWs. Supervision was reported to be regular in the majority of sites and was deemed to be most welcome and very helpful in ensuring VHW maintained high standards. Peer supervision was seen in an especially positive light. The availability of equipment was generally high (80-90%) and there was a low incidence of stock outs of essential commodities (<30% for most essential items). The average cost of training a CBHW in one subject area was $ 259.25, with MCHIP costs accounting for 22% of this. The estimated cost of a supportive visit, either for PTFU or supervision, was $ 2,222.00 per trip, of which 77% were direct or indirect MCHIP costs. 56% of training costs and 34% of support costs were accounted for by staff per diems, which was the largest item of expenditure. Each VHW was supplied with a limited scale of equipment costing approximately $ 61.78. Collectively these elements were instrumental in ensuring that VHWs retained the knowledge and skills they were given during training. Their knowledge levels were high with an average score on the questionnaire of 79% and their scores on the skills assessment even more impressive with an average score above 90%. This latter score was an improvement over the levels previously recorded during PTFU. An assessment of the registers revealed a high level of use and good completion. Recording the process of a client interaction was well done 1 cPQI is an approach to improving the quality of care in the community comprising of: in-service training in MCCM and integrated MNCH, post training follow up, peer to peer supportive supervision; supervisory VHW checklists; pilot register and reporting tools that are also job aides; and review meetings at HFs. MCHIP EVALUATION REPORT 3 with 94% of ANC attendees and 78% of sick children having the presence or absence of danger signs recorded. 61% of patients referred to a health facility had a referral note. Monthly report forms were regularly completed and returned to the health facility and these were consistently and accurately collated. The main reported impact of VHW work was a reduction in health facility workload and a reduction in both the morbidity and mortality from malaria. The former was not consistent across the district and those facilities that were better managed seemed to have a greater reduction in workload. Malaria case fatality was low, although variable over time. Both the number of severe diarrhea and pneumonia cases decreased over the period of the program, which could be attributed to early identification and treatment, or referral, by VHWs. Over the same period, the proportion of 1st ANC visits before 16 weeks increased and the rate of early neonatal death – a somewhat remote indicator – went down. The referral system appeared to be working, although it was difficult to assess accurately due to inadequacies in the data available. Patients were being referred, in the majority of cases appropriately, and those arriving at the health facility with a referral note were given priority. It was not possible to assess what proportion of referred patients actually went to the facility. The main identified shortcoming in the system was the lack of formal feedback. This only occurred when VHWs followed up patients personally, or the patient returned to let the VHW know the outcome of their visit to the facility. The VHWs had a very positive attitude towards their work, and it was clear that they were highly committed and motivated. They saw their role as improving community health though education, by improving access and reducing cost. Many complaints were heard regarding the inadequacy of the monthly allowance and the irregularity of its payment, but the VHWs appear to be motivated as much by non-monetary incentives, such as community appreciation, higher standing in the community, and a sense of having contributed and made a difference to the health and well-being of the community. The allowance issue aside, the main criticism made by VHWs was the lack of adequate equipment and supplies, which threatened their ability to function effectively, and reduced community faith in the program. The low allowances also meant they had to undertake other income generating activities, which meant they were not always available when required, a situation the community acknowledged and seemed to accept, albeit unwillingly. Throughout Chimanimani community satisfaction was high. Health facility staff noted a drop in workload, reductions in severity of cases seen and mortality, and improved health seeking behavior. Clients were unanimously happy with the service they received, citing more convenient and hence cheaper care which they deemed appropriate and effective. Community leaders valued the service and facilitated access by repairing roads to the VHW’s house or exempted them from community labor without loss of privileges. Community members unanimously supported any suggestions of increasing allowances. To what extent are VHWs from MCHIP-supported districts similar or different in the performance of duties when compared to VHWs in non MCHIP-supported districts? In Nyanga, the non-intervention district in Manicaland, MCHIP provided training in MCCM with limited refresher training in assessing and managing a sick child. On request to from the DHE, MCHIP assisted with supportive supervision, mainly by provision of transport. In Centenary, MCHIP supplied nothing; support to training on MCCM was provided by ZAPIM, another USAID funded PMI initiative. However unlike MCHIP, ZAPIM has no presence on the ground. The costs of MCCM training were the same as noted above. No job aids were available in either comparison district. VHWs used counter books to record their activities, with little consistency in the layout of the books. Coartem registers were in use in Nyanga, but not in Centenary. Training coverage was lower than in Chimanimani, with coverage in the 50-60% range in Nyanga and 30-40% range in Centenary. Especially in Centenary, comments were made frequently regarding the apparent MCHIP EVALUATION REPORT 4 arbitrariness of selection for training, which did not seem to be based on need. In Chimanimani, MCHIP maintained a training database which may account for some of the differences and eh more equitable coverage. Supervision in Nyanga was reported to be frequent and welcome whereas in Centenary, some VHWs reported no supervisory visits in the previous 2 years. Equipment availability was frequently reported to be below 50% in both districts and stock outs were significantly higher than Chimanimani, frequently above 50% in Nyanga and above 80% in Centenary. VHW knowledge levels were significantly lower in both district than in Chimanimani, with Nyanga, at 66%, being slightly better than Centenary at 63%. The differences were least for malaria and highest in management of a sick child. Skills were not assessed in Centenary due to a lack of trainers, but in Nyanga the scores were much lower than Chimanimani – 43% vs 90% for ANC; 32% vs 91% for PNC and 78 vs 93% for child health. It was not possible to compare recording and reporting directly with Chimanimani as the two comparison districts used a different system. However, the evaluation team’s assessment is that Chimanimani was better than Nyanga, which in turn was better than Centenary. This conclusion is based on the more systematic approach used in Chimanimani, which made it easier for both the VHWs and the supervisors to collect and collate the data. It also made for clearer and easier tracking of patients, although this can be considerably improved in all sites. The proportion of malaria workload undertaken by VHWs was most consistent in Nyanga, reflecting, the team believes, the stronger district management there. The differences between facilities were greatest in Centenary, which had both the highest and the lowest levels. Malaria case fatality was acceptably low in both districts. The downward trend in severe diarrhea cases was steeper in Centenary than Chimanimani, whilst in Nyanga no clear trend was seen. In the case of severe pneumonia, Nyanga showed a marked upward trend, while there was no clear trend in Centenary. The trend for early booking was upwards in both districts, with some facilities in Centenary recording, a dubious rate of almost 100%. ENND rates were heading downwards in Nyanga and upwards in Centenary. As in Chimanimani, VHWs had a very positive attitude towards their work, and it was clear that they were highly committed and motivated. Their understanding of their role was comparable. However they seemed to have more complaints, although the complaints were remarkably similar to those in Chimanimani. One finding was that relations between health facility staff and VHWs are noticeably more strained in Centenary that elsewhere. Community satisfaction was high in both comparison districts. Some health facility staff in Centenary were not as committed to the VHW program as in the other districts; indeed some there complained that workload had increased thanks to the VHW efforts. Whether the lack of commitment was a fundamental objection or a lack of understanding was not clear. In Centenary, dissatisfaction was more evident than in other districts, and it appeared that a lack of good management structures was preventing appropriate remedial action. In both comparison districts, there was, again, universal support for increasing VHW numbers raising their allowances and improving equipment and commodity availability. What factors have facilitated or hindered village health workers in performing their duties in MCHIP￾supported and non-MCHIP supported districts? The biggest factor influencing the performance of the VHWs appears to be the attitude of the DHE and, more importantly, of the health facility staff. Where this was positive and supportive the VHWs were seen as an integral part of the health care system and made the greatest contribution. MCHIP EVALUATION REPORT 5 The improved support and supervision provided in Chimanimani, especially by peer supervisors, was undoubtedly a major factor in the improved performance there. The PTFU and ongoing on-the-job training provided ensured the maintenance of high knowledge and skill levels. This was reinforced by the availability of custom-made registers and other job aids. The biggest obstacle to better performance was a lack of strong management of the wider VHW program. The number of VHW is enormous – more in some districts than formal health staff – yet there are no specific program managers and no clear national strategy. The number of VHWs was often cited as an obstacle, and while this may be true in some areas, the improved performance in Chimanimani was achieved by a relatively lower number of VHWs. The other major obstacles included inadequate and, more importantly, irregular allowances meaning VHWs could not be available on a full time basis; lack of transport to visit patients and the health facility; lack of essential equipment – scales were most frequently mentioned – and inadequate commodity supplies; and poor relations with health facility staff. What is the acceptability of the USAID-supported cPQI approach to national stakeholders, including key departments within the MoHCC? The number of key informants interviewed was unfortunately low, especially at MoHCC, due to competing priorities. Of greatest concern was that the individual responsible for the national VHW program professed total ignorance of the MCHIP program; MCHIP refuted this, producing evidence that they had tried to make contact to no avail. Other key informants were knowledgeable to the extent that one would expect given their direct involvement or proximity to the program. All who volunteered an opinion felt the program was a success. MCHIP’s approach, which was highly praised in that it was integrated fully into the provincial and district structures and processes, was thought to be key in this respect. The sense of ownership was strong; there was no sense that this was “MCHIP’s program” as is often the case in such settings. MCHIP was seen as a supportive, trusted and reliable resource. The most effective parts of the intervention were the peer supervisors and the job aids. The main adaptations proposed were not really within the programs control. Increasing the number of VHWs was the most frequent suggestion followed by improvements in allowances equipment availability and commodity supply. With the exception of some KIs within the province and districts, the issue of strengthened management was not mentioned directly, but was alluded to. This most often referred to better support and supervision. Everyone agreed that the program should be rolled out to other districts. Cost was cited as an obstacle and, even with the additional funds available in Chimanimani, MCHIP were not able to meet all demands. If scale up is to be successful, national support will be essential and this is perhaps an area where MCHIP could direct their advocacy efforts. MCHIP EVALUATION REPORT 6 Overall Conclusions The overall conclusion is that MCHIP has designed and successfully implemented a program to strengthen and improve the quality of community-based care in a rural setting. The presence of knowledgeable, skilled, highly motivated and committed cadre of health workers in the community has increased access to and reduced the cost of health care to a population in an area where travel can be extremely difficult. The observed reductions in morbidity and mortality due to the targeted diseases could readily be attributed to the intervention, whilst the increase in disease prevention activities and the improvement in health seeking behavior are considered to be clear evidence of its success. Knowledge, skills and performance were all higher in the intervention district. All components contributed to the success, but peer supervision and availability of job aids perhaps made the greatest contribution. The MCHIP program exists within the wider VHW program and any efforts to roll out or scale up the intervention would need to take into account the current status of this program. MCHIP EVALUATION REPORT 7 Recommendations 1. Continue program in Chimanimani: The program in Chimanimani is considered a success and should be continued in Chimanimani. When possible the resources available should be increased to allow full and equitable distribution of equipment and where possible to support management strengthening. 2. Roll out the intervention throughout Manicaland: As soon as resources permit, the program should be extended to the other districts in Manicaland. This should be accompanied by strengthened and standardized district management and supervision. Success at this level will provide much stronger evidence as to the effectiveness of the approach and allow increased advocacy at national level. 3. Engage MoHCC through PMD on strengthening the wider VHW program management structures Without a stronger national VHW program with a clear strategy, implementation plan and management structure, interventions such as MCHIP’s will always struggle. A stronger management of the program is essential to protect the gains already made and prevent VHW from being over burdened by the demands of many different programs. 4. Strengthen coordination with other programs and alignment of guidelines and protocols The inconsistencies regarding the definition of early antenatal booking and the timing of the first dose of SP in pregnancy, whilst not critical in its own right, is symptomatic of a lack of coordination between programs. Each program approaches implementation and training independently and this can lead to conflicting information being presented to trainees and lead to inconsistency and possible mistakes. Responsibility for this coordination / harmonisation lies at national level, and the post of National VHW Program Manager, for which interviews have recently been held, should be filled as quickly as possible. 5. Rationalise and align the cHMI systems. The cHMIS introduced by MCHIP has several advantages over the existing national HMIS community component. However the two do not collect the same data, the latter collecting a wider range. The wider range reflects the role the VHWs play in programs other than MNCH and the demands of these programs for data. The HMIS instruments used to collect this data are not user-friendly and undoubtedly lead to data of doubtful quality. It is recommended therefore that MCHIP work with Provincial and District HIOs to develop a tool, similar to the cHMIS to cover the full range of data needs. MCHIP EVALUATION REPORT 8 EVALUATION PURPOSE & QUESTIONS Evaluation Purpose The end of project evaluation of the USAID Zimbabwe Maternal and Child Health Integrated Program (MCHIP) Associate Award was carried out between November and December 2016. The three-person team comprised Dr Sean F Drysdale, a physician with broad expertise in MNCH, health systems and health financing; Monica Mandiki, a nurse midwife with extensive experience in Reproductive Health; and Munjira Mutambwa, an expert in health information, data management and statistics. The team was supplemented, for the fieldwork, by 3 focus group discussion (FGD) rapporteurs and 2 village health worker trainers provided by the District Health Executives (DHE). The SOW for the evaluation is at Annex A. The purpose of the evaluation was to: • Inform decision-making and guide the design and implementation of the USAID/Zimbabwe’s new MNCH activity. • Inform the design and implementation of future PMI-supported malaria community case management (MCCM) activities that are currently part of the MCHIP AA community approach. • Highlight areas where VHWs need additional training or support to deliver quality community health services • Help to identify successful community health service delivery and quality improvement approaches. The evaluation report will be shared within USAID/Zimbabwe and with USAID/Washington. The final report will be posted onto the USAID Development Experience Clearinghouse (DEC) website USAID’s MNCH activities are designed in line with Government of Zimbabwe national health priorities and implemented through public health facilities to enhance current government efforts to improve service delivery. As such, the MoHCC, particularly the National Malaria Control Program (NMCP), is a key stakeholder in this evaluation. The NMCP maintains that testing and treating malaria in communities increases access and improves outcomes of community members who become ill with malaria and VHWs are an important cadre for preventing and treating malaria. USAID activities are also designed to complement and leverage support from other development partners. These development partners have a keen interest in the outcome of this evaluation. The following is a list of partners to whom the evaluation report is likely to be useful:  Key MoHCC departments including: NMCP, Reproductive Health, Expanded Program on Immunization (EPI), Child Health, Nutrition, Quality of Care/Improvement; HIV and TB, and M&E units;  Development and technical partners including: UNICEF; UNFPA; World Health Organization (WHO); World Bank; United Kingdom Department for International Development (DFID);  NGOs and CBOs including: Absolute Return for Kids (ARK); Plan International; Elizabeth Glaser Pediatric AIDS Foundation (EGPAF); Organization for Public Health Interventions and Development (OPHID); Population Services International (PSI); CORDAID; Abt. Associates, Inc.; The UNION and  Academic partners like the Liverpool School of Tropical Medicine and the University of Zimbabwe The evaluation findings will, therefore, be shared with the host government, implementing partner organizations, development partners, and other relevant national stakeholders. The report will be submitted to the Development Exchange Clearing House (DEC). MCHIP EVALUATION REPORT 9 Evaluation Questions The final evaluation of MCHIP AA addressed the following key and sub questions: 1. To what extent have VHWs in MCHIP-supported districts effectively delivered quality RMNCH services? a. Describe the key capacity building support provided to VHWs in MCHIP intervention districts that were most effective in improving quality of care and the associated costs. b. What are the VHWs’ MNCH knowledge levels? c. How skilled are they in MNCH service delivery? d. How well do they record and report on services delivered? e. What are their service delivery outputs? f. How well is the referral system working? g. What are their attitudes toward their duties? h. What are community attitudes and perceptions towards VHWs performance? 2. To what extent are VHWs from MCHIP-supported districts similar or different in the performance of duties when compared to VHWs in non MCHIP-supported districts, on issues covered in question 1, above? 3. What factors have facilitated or hindered village health workers in performing their duties in MCHIP￾supported and non-MCHIP supported districts? 4. What is the acceptability of the USAID-supported cPQI approach to national stakeholders, including key departments within the MoHCC? a. How well do they know about the intervention? b. To what extent do they think the approach has been effective? Which components have been most effective? What are their views towards its success? c. How should the approach be adapted if at all? d. Should it be scaled up to other districts? MCHIP EVALUATION REPORT 10 PROJECT BACKGROUND Context In 2008, USAID identified maternal and child health as a critical area of need in Zimbabwe and, in 2009, awarded MCHIP a 3-year co-operative agreement to provide technical assistance in maternal, newborn and child health, including immunization and post-partum family planning. The approach was to address core challenges in the sector including the human resource crisis and the poor state of health infrastructure. The purpose of the award was to support the Ministry of Health and Child Care (MoHCC) to formulate national policies strategies and programs that increase the populations access to affordable evidence-based, high-impact maternal, neonatal and child health interventions. An independent external performance evaluation in 2013 reported that MCHIP activities had contributed significantly to learning and innovation and that facilities it supported had recorded positive progress in reduction of maternal and early neonatal mortality. Key partner, stakeholder and community perceptions were all positive and the report recommended a continuation and expansion of the activities. MCHIP was consequently granted a follow-on Associate Award (AA) in 2014 to expand its activities from 17 to 36 high volume health facilities and from 2 to all 7 districts in Manicaland Province. Table 1. Selected MNCH Indicators Indicator Rate Source Maternal Mortality Ratio (Deaths /100,000 live births) 960 / 651 ZDHS 2010 / 2015 Children Under 5 year Mortality Rate (Deaths /1,000 live births) 84 / 69 ZDHS 2010 / 2015 Infant Mortality Rate (Deaths/1,000 live births) 57 / 50 ZDHS 2010 / 2015 Neonatal Mortality Rate (Deaths /1,000 live births) 31 / 29 ZDHS 2010 / 2015 As noted in Table 1 above, although considerable progress has been made over the duration of the program, mortality rates in Zimbabwe remain high, especially for mothers and neonates. In Manicaland mortality rates were reported to be the highest in the country prior to the first intervention2 and at the start of the second they remained in that position. Although it had risen from the bottom of the table as far as neonatal mortality was concerned – to 2nd worst – on all other measures of childhood mortality – neonatal, infant and less than 5 years – Manicaland still had the highest death rates.3 Project Overview The MCHIP/Zimbabwe project was launched in 2010, and from 2011 MCHIP supported two districts to improve performance and quality of MNCH services at facility level. In 2012, MCHIP expanded the support to include VHWs working at community level through a community based performance and quality improvement approach. The first implementation phase was completed in September 2013; the project was extended to February 2014, and subsequently expanded and continued with new funding until 2016. 2 ZDHS 2010 and PNMS 2007 3 ZDHS 2015 MCHIP EVALUATION REPORT 11 The MCHIP program is a centralized Leader with Associates (LWA) cooperative agreement and was the USAID global flagship maternal and child health program from 2009-2014. The Maternal Child Survival Program (MCSP) has since replaced it. Currently MCHIP Zimbabwe is working in partnership primarily with the MoHCC, Jhpiego, John Snow Inc. (JSI) and Save the Children. In Zimbabwe, JSI is the lead organization in Zimbabwe. MCHIP Project Goal and Objectives The goal of MCHIP AA project in Zimbabwe was to increase access to high-quality MNCH services and strengthen health services in Zimbabwe by supporting the MoHCC and contributing to the scale-up and rollout of evidence-based, high-impact interventions that will reduce maternal, newborn and child morbidity and mortality, reduce malnutrition levels and support progress towards the attainment of MDGs 4 and 5. The project objectives were: 1. To strengthen the capacity of the MoHCC at national level to formulate evidence-based national health policies, strategies and programs to enhance scale-up of high-impact MNCH health interventions; 2. To strengthen the capacity of the MoHCC at provincial and district level to improve the quality of integrated MNCH services at HFs and in the community to support national-level scale-up plans; and 3. To strengthen the capacity of CSOs to implement MNCH activities and manage U.S. Government (USG) funding. Under the AA, MCHIP planned to build on its experience over the previous 3 years to strengthen the MoHCC’s capacity to deliver high-quality MNCH services at scale. It sought to expand the skilled workforce and leverage resources (e.g., HTF, RBF) to ensure the MoHCC has strong institutional systems at the national level for long￾term sustainability. Using a collaborative approach, MCHIP sought to expand key MNCH interventions by: increasing the pool of human resources with knowledge and skills necessary to deliver quality services; building the capacity of district and provincial staff to plan, implement and monitor services and uptake of healthy behaviors; evaluating progress towards and identifying barriers to improved outcomes; deepening linkages with the MoHCC and partners at national, provincial, district and community levels; and using a QI approach in targeted facilities to ensure health workers acquire and maintain competencies to deliver best practices and high￾quality MNCH services to every client, every time. MCHIP works at the national, provincial and district levels in all 7 districts of Manicaland Province to support implementation of activities in the Provincial Hospital, district hospitals, rural health centers and communities. At the national level, the project assists the development and implementation of national policies, strategies and guidelines. The project provides direct technical support at the Provincial, district and health facility level, to implement MNCH interventions. The Standards Based management and Recognition (SBM-R) QI approach is being implemented in 36 facilities – 5 per district plus the Provincial Hospital. Activities focus on training, supervision and mentorship of health workers in order to improve the quality of care offered to women and children. At community level, MCHIP supports improvements in the coverage and quality of high-impact MNCH interventions provided by Village Health Workers (VHWs). This is done through provision of both formal and on-the-job training and supervision in the integrated management of maternal, newborn and childhood illnesses. The project developed integrated training materials and supervisory tools to carry out close supervision by the MCHIP district coordinators. Regular update and joint review meetings for health workers were held to encourage greater peer-to-peer support and utilization of data for decision-making. An important aspect of this MCHIP EVALUATION REPORT 12 work was support to VHWs and promotion of linkages between the community and the formal public health system. Support was directed towards ensuring that visits were made to pregnant women during the antenatal period and to mothers and newborns in the immediate postnatal period on or before Day 3 and on Day 7. Health education activities focused on ensuring women and families know the relevant danger signs in the antenatal and postnatal periods, and were aware of and implementing other health promoting and disease prevention activities. MCHIP also worked with VHWs to strengthen their capacity to engage with local governance structures like Ward Health Teams and Village/Ward Development Committees and influence the priority level accorded to MNCH in local planning. The cPQI was implemented fully in Chimanimani district, focusing on training in MCCM and cMNCH of childhood illness, supportive and peer supervision, a community health information system, procurement support, and development of tools to assist and support VHWs in their work. MCHIP EVALUATION REPORT 13 EVALUATION DESIGN, METHODS & LIMITATIONS The evaluation used a combination of quantitative and qualitative techniques. These included: a focused document review; a village health worker survey using a self-administered questionnaire; an assessment of practical skills; key informant interviews; and focus group discussions with Health Centre Committees (HCC), village health workers and clients of VHWs. MCHIP’s community intervention takes place on two levels: in Chimanimani support includes support on MCCM plus the full cPQI package1 whilst in Nyanga the support is to MCCM and limited MNCH, but without the other cPQI elements. Furthermore, ongoing regular support is provided in Chimanimani, whilst in Nyanga this is provided only on request from the DHE. To take account of this, the evaluation compared a district where MCHIP was implementing the full package, with a district where it was providing support excluding the cPQI and lastly with a district where MCHIP was not active at all. The VHW, or community based, intervention by MCHIP is but a small part of the whole MCHIP project, and the VHWs supported by MCHIP work within the wider framework of the national VHW program. It is important to remember this because it means that while the contribution of the VHWs to changes in health related behavior may be substantial, indeed critical, their contribution to improvements in health outcomes or status may be overshadowed by interventions that have a much greater impact. So for example, improving the ENND rate requires intervention at a number of levels, from community, through rural clinic and district hospital to referral hospital. Arguably, the final outcome depends much more on improvements in intra-partum care and care of the newborn than on VHW interventions. However, the work done by VHWs – encouraging early booking, education about and identification of danger signs, advice on nutrition, encouraging planning for delivery, assisting with transport, etc. – undoubtedly makes a contribution to improved outcomes, although this is difficult, if not impossible to quantify. Similarly, the policies governing the national VHW program, can limit what MCHIP-type interventions may or may not do; hence the importance of MCHIP’s activities at national and provincial level to motivate positive change. The evaluation team was conscious of these limitations and tried to take account of them in reaching their conclusions and making recommendations. The detailed work-plan of the evaluation is at Annex B. District and Sample Selection In selecting the comparison districts, the evaluation team was guided by MoHCC, MCHIP and USAID. The cPQI approach is being implemented in two districts: Chimanimani and Mutasa. However, since activity had only recently commenced in Mutasa, it was agreed, with USAID and MCHIP, that Chimanimani was the only choice for the “intensive” district. Nyanga was chosen as the “less intensive” district. As seen in Figure 1 below, both Nyanga and Chimanimani have a high incidence of malaria, and this characteristic was considered to be the main criterion on which selection of the third district should be based. Combining this with a desire to have similar terrain and local customs, this reduced the choice to districts in Mashonaland Central and to some in Mashonaland East. The choice was discussed with the NMCP and it was agreed that Centenary, in Mashonaland Central would be the comparison district. It was agreed that a sample of VHWs should be selected from across the districts. The most efficient way to access such a dispersed sample in a short time would have been to bring them all to a central location. This would have been logistically difficult and, so it was decided to cluster facilities and bring VHWs to the site most convenient for them. Thus 4 sites were purposively chosen in each district, based on ease of access for VHWs. MCHIP EVALUATION REPORT 14 Based on the results of the MCHIP baseline assessment report4 it was determined that a sample size of 80 VHWs would be sufficient for the main knowledge survey to give an estimate with 5-10% precision and a confidence level of 95%. The number of VHWs provided from each facility was decided on the basis of proportional allocation. The final selection of individuals was left to district and health facility staff. Figure 1. Annual Malaria Incidence Map, Zimbabwe 2014 Official permission to carry out the evaluation and collect data was obtained from the MoHCC prior to the start of fieldwork. This was passed down to lower levels of the system in writing and local clearance was subsequently obtained for the team to operate. All respondents were informed that they were free not to answer any of the questions if they chose not to and that participation in the FGD was entirely voluntary. Document Review The documents reviewed before and during the evaluation included those listed in Annex C. Qualitative Data Table 2 below summarizes the primary qualitative data collected. 4 A Performance And Quality Improvement Approach For Village Health Workers In Community Maternal, Newborn And Child Health – Baseline Assessment Report; Mutare And Chimanimani Districts, Zimbabwe: MCHIP Zimbabwe in Collaboration with the MoHCC, Manicaland Province. January 2013. MCHIP EVALUATION REPORT 15 Table 2. Qualitative Data Key Informant Interviews Focus Group Discussions • Semi Structured to explore attitudes on: • Knowledge of the programme • Appropriateness and acceptability • Performance - effectiveness • Challenges and barriers to achievement of results • Adaptability • Scale-up • 10-12 VHWs self-selected from those who completed the knowledge survey • If sufficient number was available, a separate FGD was held with peer supervisors • Available members of the HCC • Available clients of VHWs who were attending the facility and agreed to participate Key Informant Interviews A list of key informants was compiled in discussions with USAID and MCHIP. The list comprised a number of staff at Development Partners and UN Agencies, Country Directors at partner NGOs and of in key departments at the MoHCC Head Office, key managers at the Provincial Medical Director’s (PMD) Office, members of the District Health Executives (DHE), Nurses in Charge of facilities, health course at MCHIP. A list of targeted informants with the outcome of the team’s efforts is at Annex D. Copies of the interview guides are at Appendix 5 and Appendix 6 to Annex E. Of the 44 potential informants, only 25 were interviewed. Response was particularly poor at MoHCC head office and at UN Agencies and Development Partners. The main reason for this overall lack of success was duty travel and conflicting priorities. Respondents frequently commented on the “epidemic” of workshops in the Zimbabwe health sector, which takes key staff away from their formal duties repeatedly and for considerable periods. KIIs were semi-structured and provided qualitative information on how MCHIP and the VHWs element of their program is perceived, where and how it fits in the national VHWs program, what it has contributed to better MNCH care at district levels, and the potential for rolling it out and scaling it up. Focus Group Discussions The team conducted FGDs with VHWs and, if sufficient numbers were available, conducted a separate discussion with Peer supervisors; with clients of VHWs if available; and with available HCC members. Three local experienced data note takers were recruited to manage the discussions and, during the majority of discussions, a team member with extensive technical knowledge in MNCH was present during the VHWs FGDs to ensure key responses to priority questions were consistently elicited. Village Health Workers The evaluation team conducted 12 focus group discussions (FGDs) with 159 village health workers – 132 female and 27 male – at the visited health facilities to gain an in-depth understanding of their work experiences, including successes and challenges, their views on the value of the MCHIP intervention and their suggestions to improve the program. At 3 facilities, there were a sufficient number of Peer Supervisors available to make a separate FGD possible; 27 peer supervisors – 25 female and 2 male – took part. Thus a total of 15 FGDs were conducted with VHWs. A table showing the gender and locations of VHWs is included at Annex F and a copy of the discussion guide is at Appendix 2 to Annex E. Village Health Workers’ Clients MCHIP EVALUATION REPORT 16 12 FGDs were conducted with 103 – 94 female and 9 male – clients attending the facility on the day of the team’s visit who had previously been clients of VHWs. Findings from these FGDs supplemented the information provided by the VHWs on the acceptability of the program in the community and its value in timely and effective management of illness and reducing their costs. A copy of the discussion guide is at Appendix 4 to Annex E. Health Centre Committees 40 members – 30 male and 10 female – of Health Center Committees took part in FGDs in 10 facilities. In some facilities VHWs were represented on the committee, demonstrating close ties between the facility and the community. These discussions provided information on the perceived added value of the VHWs, the impact of their activities on health center workload and the challenges faced by both facilities and VHWs. A copy of the discussion guide is at Appendix 3 to Annex E. Quantitative Data The evaluation team collected and analyzed primary quantitative data using a survey administered to VHWs, a training record tool, an equipment availability tool and project documents. The team also examined data from a number of secondary sources as listed in the table below. Table 3. Quantitative Data • The VHWs knowledge questionnaire: o Was self-administered o Contained 26 closed and open￾ended questions o Was based on existing pre- and post-test tools • The self-completed training tool sought to quantify the training received in a number of areas during the previous 2 years • The self-completed equipment and supply tool sought to quantify equipment availability and serviceability and to identify any shortcomings in commodity supply • Had 222 respondents from all health facilities in the three districts • Secondary data sources include: o MCHIP quarterly and annual reports o Zimbabwe DHS reports o Facility level data reported through the National Health Management Systems (HMIS) o VHWs data accessed at health facilities visited (cHMIS) o VHWs registers made available at interview sites (cHMIS) • The skills survey assessed a limited number (11) VHWs in Nyanga and Chimanimani examining skills in ANC and child health MCHIP EVALUATION REPORT 17 VHW Knowledge Survey In order to assess the knowledge levels, the evaluation team developed a self-administered questionnaire based on existing pre- and post-test questions administered to VHWs. MCHIP was consulted in the development of the, survey tool, which was then approved by USAID. The tool was translated into Shona and back translated, before being piloted in a non-evaluation district prior to its use. A copy of the questionnaire is at Appendix 1 to Annex E. The method used to select participants in the survey is outlined above in the section on “District and Sample Selection”. Of the 228 surveys completed by VHWs, 83 were from Chimanimani, 75 from Nyanga and 67 from Centenary. 84 percent of respondents were female and the average age was 44 years. Of those responding to the question, 29% reported that they had finished primary education; 22% had some secondary education; 46% had completed secondary education; and 3% had a higher-level qualification. Demographic characteristics of the respondents are shown in the tables in Annex F. VHWs Skills Two VHWs trainers were recruited from each district to conduct assessments of VHWs’ skills. The performance standards were drawn from checklists developed by the MOHCW and MCHIP5. VHWs were randomly selected from those who participated in the FGDs and their competency was assessed in ANC, PNC and management of a sick child. Because of the logistical challenges in trying to observe VHWs at home dealing with actual clients, simulations were used in health facilities. A total of 35 observations were carried out. A copy of the assessment checklist is at Appendix 7 to Annex E. VHWs Training Record To complement the assessment of knowledge levels among the VHWs, the evaluation team developed a training checklist to elicit information on the training received by VHWs in the preceding two years. Thirteen areas were targeted, based on the combination of training programs offered and facilitated by MCHIP. 224 VHWs completed the checklist: 82 from Chimanimani district; 75 Nyanga; and 67 from Centenary. A copy of the data collection tool is at Appendix 8 to Annex E. VHWs equipment and supplies survey Without the pre-requisite equipment and the necessary basic supplies, the VHW’s ability to provide care is highly compromised. With this in mind, VHWs were asked to complete a 9-item equipment checklist, and a 9-item supplies and provisions checklist. A copy of the data collection tool is at Appendix 9 to Annex E. Quality Assurance and Limitations of Data Collection To ensure the collection of the highest-possible-quality data, the team collaborated with NMCP, USAID and MCHIP project staff in the development of the tools and selection of sites and sampling of VHWs. Collected data was organized, entered and crosschecked by the team throughout the collection period. Recordings of FGDs were transcribed as soon as possible – usually the same evening – after recording. Weekly updates were provided to USAID and close links were maintained with MCHIP throughout the process. MCHIP project staff in Harare and Manicaland provided data and facilitated logistic arrangements as needed and requested by the team. The team had good control over this data and therefore quality assurance was good. The quality of secondary data was not in the hands of the evaluation team. Data sources included: 5 Sources included:  Community Based Care for Mothers and Newborns: A Manual for Village Health Workers  Guidelines for Management of Malaria in Zimbabwe. MCHIP EVALUATION REPORT 18  VHWs registers (MNCH register, counter books, Coartem register)  VHWs monthly summary reports (C5s, Ministry summary form)  Health facility T5 forms  DHIS for selected variables from 2010 to present  MCHIP programmatic data Limitations in the data collected included:  The small number of facilities visited, meaning the availability of collated VHW data was limited;  The tools used by VHWs to record and report their activities differed making comparison between Chimanimani and the other districts difficult  The data recorded by VHWs was not tailored to the evaluation question; determining the actual output or workload of VHWs was therefore problematic, especially in the comparison districts  Logistic constraints meaning simulations had to replace observation of actual VHWs/client consultation when assessing skills;  Non-availability of VHWs trainers to conduct skills assessment in Centenary;  Recall bias when completing the training record;  The non-availability of many of the proposed key informants - particularly the MoHCC;  Some collated facility level reports were missing – this was more of a problem with VHW data than T5 data: This being a performance evaluation, the focus was on outputs and outcomes. The evaluation question related to the VHW outputs, however, and not to program outputs. This presented an unforeseen difficulty since the data collected by the VHWs and facilities did not directly relate to outputs – certainly in Chimanimani. Thinking of what might reasonably constitute a VHW output, one could include, the number of women counseled on danger signs in pregnancy or in a newborn; or in the number of children properly treated for diarrhea. These data were not collected routinely and trying to collate them from existing VHW records would have been excessively time consuming and incomplete The evaluation team therefore looked for proxy measures of VHW output. MCHIP EVALUATION REPORT 19 FINDINGS, CONCLUSIONS & RECOMMENDATIONS6 1. To what extent have VHWs in MCHIP-supported districts effectively delivered quality RMNCH services and to what extent are VHWs from MCHIP-supported districts similar or different in the performance of duties when compared to VHWs in non MCHIP-supported districts (i) Describe the key capacity building support provided to VHWs in MCHIP intervention districts that were most effective in improving quality of care and the associated costs. FINDINGS The inputs from MCHIP in Chimanimani included:  Support to training, including the development of training manuals and job aids  Support to post training follow-up  Limited equipment and commodity supply including stationery for referring, recording and reporting  Support to DHE and VHWs to participate in bi-quarterly supervisory visits by DHE staff to facilities MCHIP supported the training of VHWs and associated staff, including VHWs trainers, peer supervisors and school health masters. The training was provided using national and, where these were not available, specifically developed, training materials, including facilitator and participant manuals. Once training was completed the VHWs were provided with job aids and guidelines to assist them; these included the VHWs register for Community Maternal Newborn and Child Health and flip charts for use when conducting health education. The chart below shows the percentage of VHWs who reported receiving training in the identified topics in 2015 and 2016 and shows considerable variation between districts and, in the comparison districts, between topics. Figure 2. Training Received by VHWs in 2 Years Prior to the Survey 6 All quotes from VHWs and informants can be found in Annex J. 0% 20% 40% 60% 80% 100% 120% Nyanga Chimanimani Centenary MCHIP EVALUATION REPORT 20 MCHIP maintain a training database, although the team did not see this. In Centenary there appeared to be no systematic attempt to maintain a record of who had been trained in what. The impact was evident in disgruntled VHWs who said that some VHWs received more training than others. Nurses from the health facilities in Centenary also commented that lack of a system of selection resulted in apparent favoritism and a failure to provide needs based capacity building. Inadequate resources to provide comprehensive training to everyone exacerbated this. One VHW said she had been trained in 2004 and had not received any training on key MNCH issues since then. i The biggest differences in training between districts, rather self-evidently, occurred in those topics that were peculiar to the MCHIP intervention, particularly training of peer supervisors. This cadre was mentioned on numerous occasions in Chimanimani as being of great support and help to individual VHWs. In Centenary and Nyanga, training was provided in the same topics, although, as can be seen in Figure 2, the proportion of VHWs receiving training recently was considerably lower than in Chimanimani. The only exception was in MCCM, on which a greater proportion of respondents in Centenary were trained. The Global Fund, ZAPIM and World Vision have provided support to training activities in Centenary, which were carried out by District and Provincial staff. A full statistical comparison of the results is shown in Annex G, but overall, for the difference between Chimanimani and Nyanga, Υ2=3.02*10-41 (p<0.00001), and between Chimanimani and Centenary, Υ2=8.08*10-37 (p<0.00001). The level and degree of supportive supervision differed between districts. MCHIP provided support to facilitate supportive supervisory activities in Chimanimani. Shortly after training, VHWs received a post-training follow￾up (PTFU) visit to confirm their grasp and understanding of the materials presented during training and to assist in solving any particular issues they were facing. Peer supervisors work with their colleagues to ensure on-going readily available support and monthly meetings are held with VHWs at their parent health facility to discuss any relevant issues and provide supplies. In addition to providing support to these activities on demand, MCHIP, in collaboration with the DHE, conducted bi-quarterly supportive supervision visits to facilities in Chimanimani. During the visits VHWs were invited to attend the parent facility, some VHW homesteads were visited; for these particular activities, VHWs were assisted with refreshments and a small allowance. In the comparison districts, there was no post-training follow up and all supervision was conducted by the PHE/DHE; EHTs played a major role in this due to the fact that they are the only cadre, at facility level, with transport. In Nyanga, MCHIP assisted with transport, but only when requested, and in Centenary, ZAPIM may provide some ongoing support in the near future. Supervision was, however, not systematic and was usually conducted at the same time as monitoring and supervision of other programs, e.g. RBF and EPI. MCHIP provided some items of equipment to VHWs on completion of training, although it was noted that not everyone received everything due to resource constraints. Differences were found in the availability of serviceable equipment and other job aids. As seen from Figure 3 the only items for which availability was comparable throughout were weighing scales and MUAC tapes; for the former, though, there were numerous complaints that scales had been handed in for repair and never returned. This, and the lack of adequate scales for initial distribution already mentioned, resulted in VHWs having to borrow scales from neighboring colleagues. A full statistical comparison of the results is shown in Annex G, but overall, for the difference between Chimanimani and Nyanga, Υ2=2.39*10-35 (p<0.00001), and between Chimanimani and Centenary, Υ2=4.15*10- 49 (p<0.00001). Although not formally assessed, VHWs frequently complained regarding the absence of stationery for recording and referrals. Some VHWs said they just write a referral note on any piece of paper they can find. MCHIP’s provision of the community register, including referral sheets made a big impact here. It was much easier for VHWs to record and report on their activities than in other districts.ii MCHIP EVALUATION REPORT 21 Figure 3. Equipment Availability in Surveyed VHWs7 In Chimanimani, as illustrated in Figure 4 VHWs experienced considerably fewer instances of stock out of essential commodities than in the other districts. It was not clear, in many cases, whether the stock out was due to non-availability of the item, a failure to distribute it to VHWs. However in at least one instance facility staff agreed that they did not distribute RDTs and ACTs to VHWs because they feared they would not have sufficient for their own use; this in spite of this situation never having arisen, and readily accessible restocking from DHE or neighboring facilities.iii A full statistical comparison of the results is shown in Annex G, but overall, for the difference between Chimanimani and Nyanga, Υ2=4.21*10-15 (p<0.00001), and between Chimanimani and Centenary, Υ2=4.79*10-38 (p<0.00001). 7 Statistical comparison is provided in Annex G 0% 20% 40% 60% 80% 100% 120% Nyanga Chimanimani Centenary MCHIP EVALUATION REPORT 22 Figure 4. Availability of Commodities Reported by Surveyed VHWs7 Based on the available figures, the training costs throughout the program are shown in Table 4. This shows the average cost for training a CBHW in any one area was $ 259.25. Table 4. MCHIP Training Expenditure in USD in Manicaland 2014-2016 Training Activity Unit Cost ($) Frequency Expenditure ($) Peer Supervisors 268.56 111 29,810.16 VHW- MCCM 242.99 3163 768,577.37 SHM - MCCM 358.17 555 198,784.35 VHW- cPQI 233.42 399 93,134.58 TOT 361.40 57 20,599.80 TOTAL 4285 1,110,906.26 AVERAGE COST/TRAINING 259.25 Some data was not available so the following assumptions were made: a. A district visit for supportive supervision cost he same as a PTFU trip b. There were an estimated 50 PTFU visits throughout the program c. There were 32 supportive supervision visits – 2 / quarter for 3 years for Chimanimani and 2 / quarter for 1 year for Mutasa Based on these assumptions, the total expenditure on the community program in Manicaland, excluding any costs for development of training materials and job aids, but including all expenditure on MCCM training in districts other than Chimanimani and Mutasa, was $ 1,315,722 as shown in Table 5. The total MCHIP program budget over 3 years is $ 15,000,000. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Nyanga Chimanimani Centenary MCHIP EVALUATION REPORT 23 Table 5. Total MCHIP Expenditure on MCCM and cPQI Activities in Manicaland 2014-20168 Activity Avg Unit Cost ($) Frequency Expenditure ($) Training 259.25 4,285 1,110,906.26 PTFU 2,222.00 50 111,100.00 Equipment 61.78 366 22,612.00 SS Visits 2,222.00 32 71,104.00 TOTAL 1,315,722.26 The cost of fully equipping a VHW – with an initial stock of consumables, a bicycle, a scale, a thermometer and a timer – was estimated to be $ 327.29. However, the available budget would not meet the requirement in full so a smaller scale of equipment was provided to 366 VHWs (196 in Chimanimani and 170 in Mutasa) costing $22,612. Each VHW was given equipment worth approximately $ 61.78. MCHIP supported supervisory visits to Chimanimani on a regular basis and helped out Nyanga when requested. The estimated cost per visit / trip was not available so it was assumed that the cost per trip was the same as for a PTFU trip. The costs of supporting Nyanga are not included. The average cost of training a CBHW, including training of trainers, was $ 259.25, of which, on average, 22% was direct and indirect MCHIP costs. Each PTFU visit cost $ 2,222 per trip per district, of which 77% was direct or indirect MCHIP costs. In all cases 50% of the MCHIP costs related to staff costs, the remainder to allowances, accommodation and fuel. For all activities, training, follow-up and supervision, the biggest cost item was per diems, excluding accommodation costs, for participants and facilitators, including MCHIP staff. 56% of training costs were for per diems, while for PTFU visits, per diems accounted for 34% of expenditure. CONCLUSIONS MCHIP’s training database, although it has recognized shortcomings, seems to have made a difference in ensuring that, in Chimanimani, most VHWs were trained on most topics – or at least it appeared systematic and equitable. Peer supervisors were widely accepted in Chimanimani and their presence was supported by all other VHWs. They appear to provide consistent, regular and accessible support, which undoubtedly has made a difference to the performance of their colleagues. MCHIP’s support to DHEs facilitated regular scheduled supervision in Chimanimani, which was clearly absent from other districts. This improved performance was also facilitated by the availability of job aids – registers and flip charts – and the necessary stationery for referring patients and recording activities, and enhanced by the regular availability of supplies and commodities. It was clear, mainly from the difference in stock outs, that facility staff in the two comparison districts, and more so in Centenary, are either not familiar with, or have not bought into, the concept of community management of diseases. The relative costs of the MCHIP input may seem high, but it is considered that they are worth it. MCHIP has developed a relatively simple model, which is somewhat self-evident. They took the people they were given, 8 The figures for cost of supportive supervisory trip and the number of PTFU trips are estimates. MCHIP EVALUATION REPORT 24 trained them, equipped them with the tools to do the job and to record and report on their activities and then followed up to check that things were going according to plan and to support the workforce. It is the team’s view that the district management in Chimanimani resembles that in Centenary so the difference in performance of VHWs can reasonably be attributed to MCHIP support. That Nyanga VHWs perform so well is more a reflection on the DHE there than on MCHIP support, though that undoubtedly makes a substantial contribution. What is more of a concern is the high percentage of training costs spent on per diems. Due to the poor salaries paid by Government, although in much of this case the recipients are non-salaried VHWs, per diems have become something of a problem. The rates are set by Government and health staff now rely on them to supplement their salaries. The result is that too many individuals are away from their posts for too long and too frequently, attending workshops and other activities, which may or may not be of strong relevance to their work.. (ii) What are the VHWs’ MNCH knowledge levels? Knowledge Levels A questionnaire was administered to all VHWs who attended the facilities visited. Findings The overall scores and the scores for the individual sections of the questionnaire are shown in Table 6 for each district. Chimanimani had the highest score at 78.7%, with Nyanga scoring 66%, and Centenary 63%. This is shown graphically in Figure 5. Table 6. Village Health Worker Knowledge Scores District CH NY CE % % % Malaria 83.6% 77.0% 72.8% ANC/P NC 72.4% 42.4% 41.2% CH 68.8% 56.9% 54.4% Gen 85.0% 71.7% 70.8% Total 78.7% 66.0% 63.0% SND MCHIP EVALUATION REPORT 25 Figure 5. Village Health Worker Scores per Subject Area and Overall The overall score in Chimanimani was significantly better than in the other two districts as shown in Table 7. The difference between Nyanga and Centenary was just significant at the 5% level. The differences in malaria scores were similar to the overall differences, reflecting the fact that malaria questions made up the bulk of the questionnaire. The difference between Chimanimani and Nyanga was not quite as marked, but still highly significant. Table 7. Inter-District Comparison of VHWs Knowledge Scores Overall Scores Malaria Scores Comparison t p Comparison t p CH/NY 7.7932 <0.0001 CH/NY 3.7288 0.000134 CH/CE 9.6229 <.00001 CH/CE 5.9965 < 0.00001 NY/CE 1.99134 0.04839 NY/CE 2.0677 0.04051 ANC/PNC Scores Child Health Scores Comparison t p Comparison t p CH/NY 11.6517 < 0.00001 CH/NY 4.2443 0.000019 CH/CE 10.9258 <0.00001 CH/CE 5.1027 < 0.00001 NY/CE 0.3988 0.3453 NY/CE 1.3023 0.0975 On the ANC/PNC questions, Chimanimani, again, scored significantly better that the other districts. There was no significant difference between Nyanga and Centenary. The picture was similar for child health questions, with Chimanimani scoring significantly better. Some individual questions caused difficulty in all districts. The timing of the 1st dose of sulphadoxime / pyrimethamine (SP) to prevent malaria in pregnancy was not well known. Only 4.8%, 16% and 29.9% of respondents gave the correct answer in Chimanimani, Nyanga and Centenary respectively. The low score in Chimanimani prompted a review, revealing differences between districts in recommendations made during 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% 80.0% 90.0% Malaria ANC/PNC CH Gen Overall CH % NY % CE % MCHIP EVALUATION REPORT 26 training (see below). Taking this into account raised the score to 55.4%; considerably better, but still one of the lowest scores for individual questions in Chimanimani. The correct definition of early booking in pregnancy was known by only 31.3%, 44% and 43.3% in Chimanimani, Nyanga and Centenary respectively. (See below for different definitions) In Nyanga and Centenary questions on danger signs in pregnancy, in the post-natal period and in the newborn were not well answered. The respective scores were 29%, 33% and 48% in Nyanga and 28.7%, 26.9% and 35.2% in Centenary. In Chimanimani the scores were 84.8%, 82.2% and 63.6% respectively. One question on management of a child with diarrhea required a two-part answer – ORS and zinc. In Chimanimani, only 43 respondents (51.8%) included zinc in the answer. In Nyanga the response was poorer at 16 respondents (21.3%) and extremely worrying in Centenary, where only 1 respondent (1.5%) mentioned zinc9 . 9 See section on supplies for more on zinc. MCHIP EVALUATION REPORT 27 Conclusions Based on these results, it is clear that VHWs in Chimanimani are more knowledgeable than their counterparts in other districts. Those in Nyanga did better than those in Centenary, except on ANC/PNC, but the difference was not so marked as with Chimanimani. The poor response regarding the correct definition of early booking is not surprising. The VHWs manual defines it as before 12 weeks; the PMTCT guidelines in 2010 said 14 weeks, although this has now changed since Option B+ recommends starting ART immediately in pregnant women, irrespective of the stage of pregnancy; and the T5 records an early booking as one occurring before 16 weeks. In its focused ANC program, WHO recommends that the first ANC visit should take place in the first trimester, i.e. before 13 weeks (or the end of the 12th week).10 Uncertainty regarding the timing of the 1st dose of SP in pregnancy while a concern is probably not too serious since this will be prescribed and administered by HWs at facility level. However VHWs should know the correct date to allow them to advise women and ensure that they have been given the prophylaxis correctly. The extremely poor response in Chimanimani puzzled the team. Further enquiries with the CNO in each of the 3 districts revealed that in Chimanimani the VHWs were trained that early booking was before 12 weeks and that SP should be given at booking provided it was after 12 weeks. In the other 2 districts, the message was that early booking was before 16 weeks and that SP should be given at 16 weeks or post-quickening. The timing of quickening is variable; it can occur as early as 13 weeks or as late as 25 weeks. WHO recommends giving the first dose as early as possible in the 2nd trimester, which starts at 13 weeks. It is recommended that to prevent confusion, the WHO definition be used and any reference to quickening be removed from the guidelines. The fact that knowledge and use of zinc, in the management of childhood diarrhea, is so low should be of much greater concern. This is an essential part of the treatment protocol and is most definitely an evidence-based, high-impact intervention that can save lives. The reasons for its low use must be explored and remediable action taken. Discussion with the VHW Trainer in Centenary revealed that the circular authorizing VHWs to prescribe zinc had not been received at the DHE so they had not made any arrangements to make it available. This, however, does not explain the low performance in other districts. (iii) How skilled are they in MNCH service delivery? FINDINGS Because the necessary VHW trainers were not available in Centenary, practical sessions were conducted on the two Manicaland districts only. The tasks given included management of a child presenting with fever; thus carrying out a RDT for malaria was necessary. The VHWs in Chimanimani scored better than those in Nyanga district. Table 8 shows a mean score of 90% in Chimanimani, with a range of 86-100%, against a mean of 43% in Nyanga, with a range of 36-50% for managing a pregnant woman. In the provision of post-natal care, the mean score was 91% (Range 85-100%) in Chimanimani against 32% in Nyanga (Range 5-68%). Mean scores for managing a sick child were 93% (Range 93-95%) in Chimanimani and 78% in Nyanga (Range 74-81%). 10 WHO FANC Guidelines MCHIP EVALUATION REPORT 28 Table 8. Results of Observed VHWs skills around ANC/PNC and Managing Sick Child Area of Service Delivery % Scores Chimanimani % Scores Nyanga ANC 90 43 PNC 91 32 Managing a sick child 93 78 The difference between districts was least in management of a sick child – i.e. MCCM, which would be consistent with the fact that both districts received full training in this subject area, but only in Chimanimani was refresher training provided in ANC and PNC. The PTFU provided by MCHIP was repeatedly credited with accounting for at least part of the difference. One additional difference was that VHWs in Chimanimani mentioned that they had been trained in, and encouraged, Kangaroo Mother Care. Table 9. Comparison of VHW Skill Levels: Pre- and Post-Training and 2016 Evaluation 2012 Baseline 2013 2nd Assessment 2016 Evaluation Assessing mother during ANC home visit 35.2% 90.8% 90% Managing mother during ANC home visit 30.2% 80.4% Assessing and managing mother during PNC visit 33.0% 66.6% 93.8% Assessing sick child 31.5% 67.8% 93% Managing sick child 25.6% 76.9% In order to assess retention of skills, the team reviewed the scores obtained by the Chimanimani VHWs with those recorded previously by MCHIP11 . Table 9, above shows the scores. A direct comparison was not exactly possible as the previous assessments for ANC and PNC had been sub-divided. However it is clear that scores in 2013 were better than in 2012, and the improved level of skills had been retained or improved by 2016. Recording and reporting on activities was a further skill imparted by MCHIP as part of the introduction of the cHMIS. This is dealt with below. CONCLUSIONS VHWs from Chimanimani demonstrated greater skill in addressing clinical MNCH issues than those in Nyanga. Based on feedback from FGDs, this is attributed to a combination of PTFU, peer supervision and regular supportive supervision by the DHE, supported by MCHIP. 11 Improving the Quality of Community-Based Maternal, Newborn, and Child Case Management. Experience for Chimanimani District, Zimbabwe. Technical Brief. MCHIP Zimbabwe, April 2016. MCHIP EVALUATION REPORT 29 The study confirms the MoHCC and MCHIP finding that VHWs The evaluation confirms the finding, by MoHCC and MCHIP, that performance of VHWs receiving the cPQI intervention improved in areas of patient assessment, case management and data recording and reporting when compared to VHWs not receiving the intervention. (iv) How well do they record and report on services delivered? FINDINGS A major complaint often heard as one goes down the health system is the number of registers front-line health workers are required to complete and the number of reports demanded of them. It is no surprise then to learn that MCHIP’s proposed introduction of a new register – the Community MNCH Register, comprising 4 forms and a referral slip – in Chimanimani was greeted with some dismay and some resistance. The register, however, functions not only as a record, it also acts as a job-aid and a repository for the data required to complete the monthly report form. This simplified the life of VHW in Chimanimani and consequently it was rapidly accepted and assimilated into the wider district HMIS. The register has 2 forms; the first (C1) has 3 sections devoted to: an ANC consultation; a PNC visit for the mother; and one for the newborn. The second form (C2) is used when managing an ill child. The form is designed in such a way that the VHW ticks boxes as she moves through the consultation. This acts as a guide or protocol for the VHW herself and also allows for easy monitoring of quality and content of consultations by supervisors. The register also has a number of duplicated, tear-out referral slips. At the end of each month, the VHW completes a report – the C5 – extracting data from the register using built￾in guidance notes on the form. A copy of the register can be seen at ANNEX H. This register was only in use in Chimanimani, although some Nyanga VHWs, who had been trained in managing a sick child, used the child health section. In the comparison districts the register’s place was taken by ruled counter-books in which the layout and content was determined locally. These counter-books did not always collect the data required to complete the MoHCC monthly report form. There was a distinct contrast between the counter-books maintained in Nyanga and those in Centenary. The former were better and more consistently laid out and completed. In Nyanga, malaria cases were recorded in both registers and it was possible to track case from one register to the other. The counter-books in Centenary were used mostly to record growth monitoring and some form of village register. The MoHCC reporting form and the C5 differ in the data they collect. The latter is concerned only with MNCH and malaria, whilst the former covers a number of other program areas including PMTCT, EPI, OI/ART, condom distribution, growth monitoring and health promotion activities. Copies of the forms are attached as Appendices 1 and 2 to Annex I. The Chimanimani VHWs do not use the MoHCC form, and since the cHMIS collects less data, this must leave a gap in the DHIS records. The NMCP also requires that VHWs using RDTs and dispensing ACTs maintain a register – the Coartem Register – in order to record cases and monitor stocks. This was in use in Chimanimani and in Nyanga, but not in Centenary. The reason for its absence in Centenary was not clear. In terms of VHWs recording their activities, it was possible to assess this from records in Chimanimani only. The team extracted a number of records from sample of registers brought to the FGDs. Initially the team tried to track referrals to the clinic, determine how many referred individuals had actually attended the clinic and what the outcome had been. This proved not to be possible, so completion of the registers became the focus of attention.  It was difficult to assess completeness of patient identity data, as different fields were required depending on the nature of the consultation, yet there was no clear indication as to what was required and when. MCHIP EVALUATION REPORT 30  Of the ANC records examined, 23% of clients had been seen before 14 weeks and 67% were referred. 75% of women had their estimated date of delivery correctly calculated. In 94% of cases, danger signs in pregnancy had been sought and 12% of women were recorded as having one or more danger sign. However, only 13% of these latter women were recorded as having been referred.  78% of sick children were checked for the six danger signs; 9% had one or more present; and, in contrast to ANC cases, 82% of those with danger signs were referred.  Overall 64% of VHW clients were referred to a health facility, and 61% of these had a referral note. 47% of such patients had a follow-up visit recorded by the VHW. The responsibility for collecting and collating VHWs monthly data returns varied between district and facilities. In some it was the N-i-C, in other it was the EHT, while in one facility a specifically recruited volunteer was responsible. Our sample of clinics was, admittedly, small, but it was clearly noticeable that the filing and availability of monthly reports, and indeed the returns used to prepare these, varied greatly. Chimanimani and Nyanga were, however, generally better than Centenary. One final observation on this topic was that it was often unclear how many VHWs were “on the books” of each facility. Recruitment seemed to be an ongoing process driven by communities, rather than a DHE plan. Returns were not 100% and it was unclear what happened if a VHW(s) did not put in a report for one or more months. CONCLUSIONS The team members were unanimous in their impression – given the small sample size it would be unfair to use a word stronger than impression – that recording and reporting in Chimanimani was better than Nyanga, which in turn was better than Centenary. This conclusion is based on the more systematic approach used in Chimanimani, which made it easier for both the VHWs and the supervisors to collect and collate the data. It also made for clearer and easier tracking of patients, although this can be considerably improved in all sites. The cMNCH was of great help in both data management and in clinical management of patients and is to be strongly recommended. The differences between the data collected on the MoHCC form and on the C5 reflect the lack of clear management of the wider VHW program. Clearly MCHIP’s focus is not the same as the Ministry’s, but the number of activities on which the VHWs are required to report on the latter’s form suggests that greater control may be necessary if the VHWs are not to be overburdened and overwhelmed by excessive demands of different programs. The difference between rates of referral of women with danger signs and infants with danger signs in Chimanimani is notable. Further investigation revealed no clear reason for this. It was felt that perhaps with women there was more of a negotiation regarding the seriousness of a symptom or sign – “everybody gets swollen legs in pregnancy” with women that with children. In the latter case the VHW advice was heeded more readily. (v) What are their service delivery outputs? Findings Quantifying VHW outputs was problematic, both because of the sampling approach used and due to lack of complete data. The rationale behind the VHWs program is to bring health services closer to communities. This should result in a shift of the burden of care from facilities, thus reducing their workload. To test this hypothesis we examined the number of cases of malaria tested and treated by VHWs as a percentage of the total clinic workload in the facilities we visited to administer the VHWs knowledge questionnaire. It would have been informative to look at the situation prior to the intervention, but data for this was not available. MCHIP EVALUATION REPORT 31 Figure 6. Percentage of Malaria Workload Undertaken by VHWs From Figure 6 above it can be seen that the contribution of VHWs to a reduction on facility workload varies, not only between districts, but also within districts. The variation is least in Nyanga, where on average 51% of cases of malaria are managed in the community, and greatest in Centenary, where the average is 28%. In Chimanimani, the percentage is close to that in Nyanga at 47%. Interestingly, the facility where VHWs make the highest contribution (82% of cases) is St Albert’s Mission Hospital, in Centenary, where the overall proportion is lowest. The differences between districts are significant. VHWs in Nyanga see significantly more cases than in Chimanimani (z=2.336, p<0.01) and when compared to Centenary, VHWs in both Manicaland districts see significantly more cases (CH vs CE (z=15.551, p<0.01); NY vs CE (z=18.065, p<0.01)) than those in Centenary. In Chimanimani, however, the VHWs seem to be better at identifying malaria clinically than facility staff; the proportion of suspects testing positive in the community is 33.6% compared to 19.7% in facilities (z=14.01, p<0.01). The difference was also significant in the other districts, but in the opposite direction; facility staff had a better “hit rate”. (Nyanga 28.9% vs 32.1% (z=-2.78, p<0.01) and Centenary 30.5% vs 34.5% (z=-4.46, p<0.01)). The reasons for, and the relevance of, this finding are unclear. Apart from bringing services closer to communities and thus improving access to care at the same time as reducing costs, VHWs are tasked with identifying illness in the community and either providing treatment, for a limited number of diseases, or referring the patient to a health facility. If they were successful in doing this, one would expect, as a result of earlier treatment or referral, the number of deaths and severe cases of illness recorded at the facilities to be reduced. The number of deaths and percentage of cases classed as severe, for a limited number of diseases, were compared using data from the national HMIS. As illustrated in Figure 7 below, the trends for severe pneumonia and diarrhea-with-dehydration in Chimanimani were clearly downwards. The differences between 2012 and 2016 were significant for both diarrhea, (z=6.93, p<0.01) and pneumonia (z=4.27, p<0.01). 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% % of Malaria Workload Undertaken by VHWs by Facility in 2016 VHW Malaria Suspect % VHW Malaria Case % Chimanimani Nyanga Centenary MCHIP EVALUATION REPORT 32 Figure 7. Percent of Diarrhea Cases with Dehydration in <5s In Nyanga the case was considerably different; there was no significant change in the percentage of cases of diarrhea presenting with dehydration (z=0.695, p=0.484), while for pneumonia, as shown in Figure 8, the situation appeared to be significantly worse, (z=-10.83, p<0.01). Figure 8. Percent of Pneumonia Cases Classified as Severe in <5s In Manicaland as a whole, the findings were mixed. The situation for diarrhea improved (z=7.59, p<0.01), while the situation for pneumonia worsened (z=-3.10, p=0.0019) In comparison, the situation in Centenary was also mixed. The diarrhea situation improved (z=7.59, p<0.01) while the situation regarding pneumonia deteriorated slightly (z=-1.46, p=0.144). Malaria cases and deaths in <5s were too low to make comparisons meaningful. Instead case fatality in all age groups was compared as illustrated in Figure 9. 0.0% 1.0% 2.0% 3.0% 4.0% 5.0% 6.0% 7.0% 8.0% 9.0% 2012 2013 2014 2015 2016 % Pneumonia Cases Classed as Severe in <5s Pneum - CH Pneum - NY Pneum - Mn Pneum - CE 0.0% 2.0% 4.0% 6.0% 8.0% 10.0% 12.0% 2012 2013 2014 2015 2016 % Diarrhea Cases with Dehydration in <5s Diarr - CH Diarr - NY Diarr - Mn Diarr - CE MCHIP EVALUATION REPORT 33 Across the board the CFR was acceptably low, never rising above 2.5 deaths /1,000 cases. In spite of the other positive findings in this evaluation, Chimanimani consistently recorded the highest death rates. Figure 9. Percent of Malaria Case Fatality Rate Another area where the activities of VHWs might be expected to improve health seeking behavior, and hence pregnancy outcomes, is in early booking for ANC. There is clearly some confusion over what constitutes “early booking”. The VHWs training manual defines it as before 12 weeks, while PMTCT guidelines define it as before 14 weeks. The early-booking indicator on the T5, however defines it as booking before 16 weeks. Since out data was extracted from the DHIS, this is the definition used here. Figure 10 below clearly shows that the trend in all districts, and in Manicaland is upwards. Nyanga is doing slightly better than the Provincial average and Chimanimani slightly worse. The performance in Centenary and the trend is considerably better that in Manicaland. However some data observed during the fieldwork suggests this change may be related to RBF payments. 0.00 0.50 1.00 1.50 2.00 2.50 2014 2015 2016 Annual Malaria Case Fatality per 1,000 Cases 2014 - 2016 Ch Ny Ce MCHIP EVALUATION REPORT 34 Figure 10. Trend in Early ANC Booking Whilst early neonatal death (ENND) rate is a good indicator of the quality of antenatal and intra-partum care, it relates more to what happens at the facility level that at community level. However, VHWs contribute to reductions in ENND rate by encouraging early AN booking; by identifying danger signs early and between formal AN visits; by encouraging women to be prepared for the birth; and by assisting them, when necessary, to get to the health facility. Since we noted a correlation between how well health facilities and how well VHWs were functioning, we compared ENND rate in the three districts in Figure 11. Simply looking at the trend, Chimanimani and Nyanga are decreasing – a sign of improving quality of care – while Centenary is going in the wrong direction. The trend in Manicaland overall is improving. Figure 11. Trend in Early Neonatal Deaths 0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 Trend in Proportion of First ANC Visits Before 16 Weeks 4 per. Mov. Avg. (Chimanimani) 4 per. Mov. Avg. (Nyanga) 4 per. Mov. Avg. (Centenary) 0.00 5.00 10.00 15.00 20.00 25.00 30.00 Trend in ENND / 1,000 Deliveries by District 2012 - 2016 Linear (Chimanimani) Linear (Nyanga) Linear (Centenary) MCHIP EVALUATION REPORT 35 CONCLUSIONS There was little to choose between Chimanimani and Nyanga in terms of the proportion of malaria cases managed in the community. However this proportion was more consistent across clinics in Nyanga. The evaluation team’s impression was that performance on this score related more to differences in facility and district management than to the VHWs. The finding that the best performing facility is in the worst performing district reinforced this impression. Similarly, there are no real differences in the malaria case fatality rate between the districts. The rate is low across the board – an indicator that cases are being recognized, diagnosed and treated early and appropriately – and this could be, at least partially, attributed to the success of the VHWs and MCCM. It is tempting to attribute the reduction in severe diarrhea and pneumonia cases in Chimanimani to the activities of the VHWs, and indeed this would not be an unreasonable conclusion. Prompt treatment and early referral would undoubtedly improve outcomes and it is clear that VHWs in Chimanimani have the knowledge to do this, and their knowledge levels are greater than the other districts’ VHWs. The difference in the changes in diarrhea incidence – down in Chimanimani and level or upwards in the other areas – could well be due to the higher use of zinc by VHWs in Chimanimani. VHWs are well placed to have an impact on the number of women booking early for ANC. Whilst the trend is upwards everywhere, there is still room for further improvement. The RBF program pays an incentive for each woman recorded as booking early, and this has incentivized one facility to achieve a, scarcely believable, early booking level of 99%. Care is clearly required with such an intervention as it may lead to erroneous conclusions and conceal areas where attention and improvement is still required. Perhaps the indicator most removed from the VHWs is the ENND rate. Their activities can surely contribute to this moving in the right direction, even if the contribution is modest. That the ENND in Manicaland is going down, in contrast to the trend in Centenary, suggests improving quality of intra-partum and post-natal care in that Province, which may be largely due to MCHIP’s wider program interventions. (vi) How well is the referral system working? FINDINGS It was clearly evident that patients were consulting VHWs and were being referred to health facilities when necessary.iv This system was being reinforced by health staff in some locations, more so than in others. It was also clear that, in most facilities, patients with a referral note from a VHW received priority when they arrived at the facility.v The above quotations indicate that many, if not most, referrals are appropriate. Although health facility staff agreed with this assessment, it was not possible, however, to quantify the finding.vi Nor, for the same reason, was it possible to conclude how many referred patients actually went to the facility. As noted above only 61% of patients referred in Chimanimani had a referral slip. This figure was not available in the other districts. In Chimanimani, 47% of those referred had a VHW follow-up visit, but this did not actually record whether the patient had attended or not. Feedback from facilities to VHWs on referred patients was poor. Notwithstanding the acceptability of the system to communities, shortages of equipment and supplies are undermining the efforts of the VHWs. Respondents commented on the time wasted if the VHW was not able to test for malaria or provide simple treatments. Another criticism was that repeating a test when a patient was referred to the clinic, e.g. for malaria, was a waste of both time and resources. MCHIP EVALUATION REPORT 36 CONCLUSIONS Although there is insufficient data to make firm quantifiable conclusions regarding the efficiency and effectiveness of the referral system, anecdotal evidence indicated that it is indeed in place, is acceptable to communities and appears to be working well. In a number of instances, communities were reported as having established local transport systems to assist with emergency referrals indicating their buy-in. There is no feedback loop in the system however, which would be valuable to facilitate assessments of VHW performance and for recording patient compliance. If the system is to retain the support of communities, efforts must be made to ensure adequate supplies of commodities to VHWs. (vii)What are their attitudes toward their duties? The approach and attitude of VHWs towards their work were explored during FGDs and KIIs. The discussions sought to answer the question under the headings: appropriateness; acceptability; affordability; reliability; adequacy of support; and sustainability. The same questions were addressed with VHWs, community members and health staff and other key informants. FINDINGS - VHWs Appropriate: Within virtually all focus groups, VHWs articulated their roles and responsibilities similarly and in much detail demonstrating interest and understanding of what is expected of them. They all appeared to feel that what they were asked to do, and doing, was appropriate in improving access and reducing costs.vii Acceptable: VHWs felt that they were generally accepted by the community with “challenges here and there”.viii Some of the challenges cited in Chimanimani included an inability to approach the more affluent community members, some religious objectors, and sometimes those women perceived to be in early stages of pregnancy. Some areas, however, now have VHWs drawn from the Johanne Marange sect considered to be the main religious objector.ix In contrast to the spirit in Chimanimani and Nyanga some VHWs in Centenary felt the relations between them and nurses at Health Facilities were strained. Some VHWs said that health workers did not appreciate them and their work. One nurse in charge confirmed that they were aware that in some instances negative attitudes were displayed by nurses.x Affordable: The services provided by the VHWs are free. While locally available care reduces costs to patients, some VHWs in Chimanimani reported using their own money to provide the service. VHWs believed that they had both improved access and reduced costs.xi Paradoxically, in Nyanga VHWs felt that they do not always reduce costs as the client’s time is wasted when they (the VHWs) do not have stocks. Again VHWs reported that sometimes they end up using own resources (scotch carts and cattle) to enable clients to access urgently required services. MCHIP EVALUATION REPORT 37 Reliable Universally VHWs felt that while they were a good resource for the community their services could not always be depended on. Service delivery appeared to be more consistent in Chimanimani, but these VHWs also reported that their reliability was compromised by shortages here and there, lack of equipment e.g. weighing scales, and their own non-availability at times as they pursue livelihood activities.xii This situation was more so in Centenary where VHWs stated that they were usually short of supplies and frequently could not test and treat malaria as they lacked testing kits and medicines. This was confirmed by nurses who supplied them mostly during peak seasons or when the felt they had excess. Supportive In Chimanimani VHWs felt they had very strong support as MCHIP participated in the post-training follow up. This boosted their self-esteem and their social standing. The VHWs also appreciate Peer supervision.xiii The support visits by health staff were reported to be more frequent in Chimanimani than anywhere else. Planned regularity differed amongst facilities from biannually, and quarterly, to monthly, with one clinic reporting it had not conducted any for the year, citing shortage of staff. In some instances VHWs reported few or even no support.xiv VHWs felt that HF could be more supportive by providing incentives on time and adding items such as soap and communication allowances and lighting to make their job much easier. On the same lines, in Centenary, some felt that allocation of refresher training and other courses, by District and facility staff, was not always based on need.xv VHWs in Chimanimani applauded MCHIP for the thorough training and follow up, saying it helped them provide necessary and quality services. While now accepted as resource persons in the community, they acknowledged that they cannot manage everything and sometimes refer difficult to the Health Facility. Sustainable Village health workers across all the districts were committed to their work. Some echoed that they had been trained as VHWs more than 20 years ago and operate despite challenges. Village health workers, even those in Chimanimani, felt that their operations are threatened mostly by lack of supplies than other important challenges including inadequate incentives. This compromises safety and motivation. Many continue their work purely because of internal motivation. xvi CONCLUSIONS It is clear that VHWs are motivated by more than monetary and financial gain. They have a degree of self￾motivation that is reinforced by the positive response of their communities and by their improved social standing. It seems clear that the appreciation of the communities is expressed in different ways, and that the VHWs appreciate this. However failings in the program with regard to equipment and supplies, and this applies less to Chimanimani than the other districts, frustrate them and they feel they are letting themselves and the community down by not being able to provide the promised services. This is evidenced by their use of their own resources to assist patients. MCHIP contribution through the cPQI has contributed to higher level of motivation and more positive attitudes for VHWs in Chimanimani than in the two comparison districts. The Chimanimani VHWs reported higher levels of satisfaction with their training, their equipment and, especially, the support they were provided with and they attributed this, largely, to MCHIP’s support.xvii (viii) What are community attitudes and perceptions towards VHWs performance? MCHIP EVALUATION REPORT 38 The approach used with VHWs was repeated in seeking out community views on the same set of questions as above. FINDINGS Appropriate Client satisfaction was recorded throughout all the discussions held with clients – somewhat more so in Chimanimani – and Health Centre Committees also rated the work of VHWs very highly.xviii Only in Centenary were sentiments expressed by HCC that Health Facility workload was increasing because VHWs were referring too many clients.xix Acceptable There were many testimonies demonstrating acceptance of VHWs. In Chimanimani it was common practice that for minor ailments the VHWs is the first point of call. The clients were also happy with the quality of services VHWs in Chimanimani offered. Many positive statements were obtained. HCC members and clients said the HFs encouraged the practice. According to one Councillor member of a HCC there was resistance at the beginning, but this seems to have lessened over time.xx Indeed some communities have identified shortcomings in the system and developed their own solutions. . Delays in attending facilities were noted in all districts because of distances and costs and in a number of instances, communities reported having established local transport arrangement to assist with emergency referrals indicating their buy-in.xxi The community malaria case management had now been fully embraced. In Centenary VHWs were accepted and known to play the key role of health promotion.xxii Affordable Clients appreciated services offered stating that it was good to receive free life-saving services within close range. This was much appreciated in Chimanimani where the terrain is difficult to manoeuvre. In Nyanga a VHW, while reporting that she had tested 60 clients in a day during a malaria outbreak, commented.xxiii Reliable There were several testimonies of quality care having been offered by VHWs. Generally these were good, but occasionally critical. In Centenary, RDT kits and ACT are not generally distributed to VHWs. And some of the most critical comments came from there.xxiv Recommendations from across districts by HCC and clients included increasing the numbers to shorten distances and improve communication and information sharing. In Chimanimani clients mentioned that younger VHWs were sometimes more likely to be unavailable. Some reports referred to the lack of supplies for VHWs as undermining confidence in the program. Supportive There were more clients that testified to follow ups by VHWs in Chimanimani than Nyanga and Centenary. Some clients in Chimanimani commented that elderly VHWs could not ride bicycles and therefore are limited in their ability to follow up clients.xxv Sustainable Both HCC members and clients felt that the program was sustainable, but with some enhancements. Many recommended an increase in the number of VHWs to cut distances travelled and allow the VHWs time to attend to livelihood issues.xxvi MCHIP EVALUATION REPORT 39 Clients were generally happy with the service and both they and HCC members lobbied for greater incentives for VHWs to keep them committed. Use of health transition funds to boost supplies for VHWs was also mentioned by HCC members from Chimanimani.xxvii CONCLUSIONS The VHW program in general is well accepted in most places. It was clear however, that this acceptance was greater in Manicaland than in Centenary. The community in Chimanimani embraced the VHWs program to the extent of introducing incentives at community level, which can only improve the sustainability of the program. The greater acceptance of VHWs at health facilities, probably due to MCHIP support, was felt to play a considerable part in this. In all districts the issue of stock-outs was raised. If the program is to be fully accepted and successful, then it must be reliable. Clients appear to be more accepting of a VHW being absent in order to make a living that of a VHW who cannot provide a service because she doesn’t have the necessary equipment or supplies. Community members are aware of the pressure on VHWs and consistently lobbied for increases in the number of VHWs as well as their allowances / incentives. 2. What factors have facilitated or hindered village health workers in performing their duties in MCHIP-supported and non-MCHIP supported districts? FINDINGS The biggest factor influencing the performance of the VHWs appears to be the attitude of the DHE and, more importantly, of the health facility staff. Where this was positive and supportive the VHWs were seen as an integral part of the health care system. Otherwise they were seen, at best as an appendage, or at worst a nuisance that increased HF workload. At some facilities in Centenary, it was clear that the concept of VHWs as a part of the system was either not understood, or not accepted. HF staff would not distribute RDTs and ACT to VHWs in case they, the HF, ran out. The team attempted to get the staff to understand that the same number of patients would be treated with the available supplies, and that if the VHW treated the patients, HF workload would fall. Furthermore, the facility was better placed to get urgent resupply than a VHW. Clearly this message needs to be more widely and strongly disseminated. The contributions of MCHIP and their high standing in Manicaland, thanks largely to the success of the wider SBM-R program at facility level, definitely contributes to the acceptance of the VHW program in both Chimanimani and Nyanga. MCHIP’s ongoing involvement in supportive supervision ensured that HF staff remained regularly in contact with VHWs, and these visits were greatly appreciated as they were seen to enhance VHW standing in the community. Peer supervision was greatly appreciated in Chimanimani. These supervisors were considered “colleagues”, “one of us” and the fact that they had first-hand knowledge of the challenges made them an invaluable support. They also ensured that support and supervision took place more frequently, although it was noted that they were not considered an alternative to formal health worker support. That said, one KI noted the “Peer supervision is working better than management”. The knowledge and skill levels in Manicaland were significantly higher than in Centenary, and Chimanimani performed better than Nyanga. This would suggest that the MCHIP approach to training, which is competency based, gives better results that the approach used in Centenary. However the contribution to this improved performance by PTFU and stronger ongoing supportive supervision should not be underestimated. MCHIP EVALUATION REPORT 40 The job aids provided by MCHIP, as well as being greatly appreciated, assisted VHWs in assessing and treating patients, and also in providing health education. One DMO commented: ““VHWs are not that educated; the flip charts are very helpful. I have been there, witnessed it and it is very good” The Register for Community Maternal Newborn and Child Health also contributed greatly to the success of the program. Not only were VHWs guided during each assessment of individual patients, but record keeping and reporting in Chimanimani were better than in the other districts. Linked to the comments earlier regarding DHE support, the biggest obstacles to better performance, to scaling up and to sustainability is the lack of a clear strategy and adequate management structures for the wider national VHW program. The VHWs are numerous; one DMO commented that there were more VHW than formal health staff, and no one to manage them properly. The lack or formality also puts the VHWs at some personal risk if they are injured in the course of their duties as one VHW reported.xxviii Although the number of available VHWs was frequently cited as an obstacle, this is not supported on closer examination. The number of VHW’s per head of population is highest in Centenary (2.00/1,000) and lowest in Chimanimani (1.04/1,000) – Nyanga falling almost exactly in the middle (1.58/1,000). It is clear that factors other than absolute numbers are more important. Availability of transport to visit clients was stated to be a problem. Some VHWs had bicycles, but these were not always suitable, either due to the terrain or the age of the VHW. The provision of motorcycles was suggested by a few, but it is unlikely that the older VHWs would be able to handle these. Communication between facilities and VHWs was problematic. Some VHWs didn’t have mobile phones and many of those who did, had no airtime. Whilst it was suggested that provision of phones and/or airtime would facilitate reporting, the team felt it could be more useful in getting advice and support in difficult cases or to advise facilities of incoming patients. Lack of equipment and inadequate supply of commodities are obvious hindrances to VHWs in their efforts and at the same time undermine community confidence in the program. . In particular shortages of paracetamol and zinc were mentioned. Except in some clinics in Centenary RDT and ACT supply appeared adequate. Related to this was a lack of storage space for their equipment. A secure and safe place to store medicines was repeatedly noted in FGDs as being required.xxix Some communities supported VHWs by exempting them from working in food-for-work schemes while still providing them with food, while others repaired roads to VHW houses in order to improve access. However perhaps the biggest obstacle is the low and irregular allowances paid to VHWs. The demands placed on them are considerable and expectations are high. Yet these are, largely, poor rural women who have to maintain a family. Their work as VHWs takes them away from so-called “livelihood activities” making an already difficult situation worse. Communities recognize this and understand when a VHW is not available for this reason, but the risk to sustainability of the program is very real. CONCLUSIONS The strongest facilitator for the VHW program is buy-in and support of the DHE and health facility staff. Most other things fall from this; good supportive supervision enhanced standing in the community and regular and adequate supplies. The biggest hindrances, apart from absence of the above, include poor availability of equipment and commodities, and the harm to motivation caused by irregular, more so than inadequate, allowances. MCHIP EVALUATION REPORT 41 3. What is the acceptability of the USAID-supported cQI approach to national stakeholders, including key departments within the MoHCC? (i) How well do they know about the intervention? FINDINGS As noted earlier, most KIs were in the provinces or districts and thus closely involved with the program. Obviously informants in Nyanga and Centenary were not as knowledgeable about the VHWs aspects of the program as those at provincial level and in Chimanimani, but were, nonetheless, well informed about the wider MNCH program. At Provincial level and at both DHEs in Manicaland, the informants noted MCCM and all 4 elements of the cPQI approach – refresher training in cMNCH, iCCM of childhood illness, cHMIS and peer supervision. They all mentioned the job aids, registers, reporting forms and referral notes introduced by MCHIP. The only MoHCC officer interviewed, the Director of the NMCP, was equally well informed. By contrast, the Director Nursing Services, who is responsible for the VHWs program nationally, declined to be interviewed claiming to know nothing at all about MCHIPs program with VHWs. MCHIP was able to demonstrate that efforts had been made to contact the office of the individual concerned, but to no avail. At all levels, comments were made regarding the support of MCHIP in planning and prioritization. It was noted that this was very collaborative, even down to community level, which was much appreciated. The staff members interviewed at ZAPIM, a program taking on similar responsibility for support to the NMCP in neighboring districts, were familiar with MCHIP’s approach, but not perhaps as familiar as one might have expected. ZAPIM appears to have its own approach, which does not involve deployment of provincial or district level staff, although the Director NMCP stated that he was keen for ZAPIM to adopt the same approach as MCHIP, especially with respect to improving data collection. CONCLUSIONS Those involved with the program are knowledgeable about its distinguishing characteristics but, as one would expect, the further removed from actual implementation the less detail was known. MCHIP has made efforts to ensure all relevant stakeholders are informed about the program and the response from the Director Nursing Services was surprising. (ii) To what extent do they think the approach has been effective? Which components have been most effective? What are their views towards its success? FINDINGS All respondents were positive regarding the effectiveness of the program and in fact all elements were considered successful. The value of improved data collection was very much appreciated in demonstrating the success.xxx VHWs and communities were reported to have been empowered to take responsibility for their own health and health care. VHWs were considered to play a major role in changing health-seeking behaviors particularly amongst groups that traditionally reject formal health interventions.xxxi The most frequently cited impact of the program was a reduction in case severity and mortality from malaria. This has resulted in a reduction in workload at facilities, which has in turn facilitated an improvement if quality of care.xxxii Increases in ANC attendances and institutional deliveries were also partially attributed to the efforts of VHWs; as were reductions in severity of pneumonia and diarrhea. It was noted by one respondent that although ANC was improving, PNC was still struggling somewhat. MCHIP EVALUATION REPORT 42 There was considerable support for the peer supervisors, who were seen to be important in filling recognized gaps in routing support and supervision. The registers, guidelines and job aids were seem to be very important in improving and maintaining quality. The job aids especially were seen to be successful.xxxiii CONCLUSIONS The program was generally highly thought of and has contributed substantially to positive changes in health￾seeking behavior; reduced costs and improved access to care for patients, reductions in mortality and morbidity; reduction in pressure of health facility staff; and consequently improvements in quality of care. All who volunteered an opinion felt the program was a success. MCHIP’s approach, which was highly praised in that it was integrated fully into the provincial and district structures and processes, was thought to be key in this respect. The sense of ownership was strong; there was no sense that this was “MCHIP’s program” as is often the case in such settings. MCHIP was seen as a supportive, trusted and reliable resource. The most effective parts of the intervention were the peer supervisors and the job aids. (iii) How should the approach be adapted if at all? FINDINGS Few adaptations to the MCHIP program were suggested; most changes referred to improvements in the wider VHWs program, or one to a policy change that again would affect the wider program. The most common comments related to:  Increasing the number of VHWs so that none had to cover two villages.  Managing expectations of health managers and not overloading VHWs  Strengthening management support and developing a management structure for the program.  Improving VHWs activity monitoring and supportive supervision – related to the above.  Improving equipment availability and commodity supply  Increasing the monthly allowance and ensuring it is paid regularly and on time  Improving incentives – a little can make a big difference CONCLUSIONS MCHIP’s intervention seems to have been well designed and covers all the areas considered to be important by all stakeholders. The improvements that were suggested are largely outside MCHIP’s control, if not their sphere of influence. However as noted above the general approach of full engagement with provincial and district structures is critical. (iv) Should it be scaled up to other districts? FINDINGS The overall opinion was that the program should indeed be rolled out to other districts. The costs of the program were considered to be a potential obstacle as was the lack of proper management of the national VHWs program. CONCLUSIONS By trying to strengthen the areas suggested and perhaps focusing on these areas, MCHIP may be able to strengthen their advocacy for scaling-up and rolling out the program, which is considered appropriate and successful. MCHIP EVALUATION REPORT 43 Conclusions The overall conclusion is that MCHIP have designed and successfully implemented a program to strengthen and improve the quality of community-based care in a rural setting. The presence of knowledgeable, skilled, highly motivated and committed cadre of health workers in the community has increased access to and reduced the cost of health care to a population in an area where travel can be extremely difficult. The resultant reductions in morbidity and mortality, the increases in disease prevention activities and the improvements in health seeking behavior are all evidence of this success. The differences between Chimanimani and the comparison districts is further evidence of the appropriateness of the approach and MCHIP’s success. In this regard it should be appreciated that Nyanga is, in most respects, the best performing district in Manicaland, so the outcome of the comparison was, by no means, a foregone conclusion. That Chimanimani outperformed Nyanga on most measures is testimony to the success of the intervention. The components of the intervention – training with PTFU, supportive supervision by peers; use of the cHMIS and other job aids; and procurement of the necessary tools of the trade – are equally important. This any plan to scale up or roll out the intervention must include all four. Although the evaluation was of MCHIP’s intervention, it was not possible to do this in isolation from the wider VHW program. This program has been in existence for 30+ years with varying degrees of support during that period. Unfortunately the team was not able to interview the officer responsible for the national program. However other key informants were clear that the lack of a management structure and dedicated managers at district level was a severe hindrance to the performance of the program. “If support at the national level is there, it will go a long way” “It can be expanded, but needs a proper management structure” This management structure is required, not only to set policy and give guidance, but also to protect VHWs from being overburdened by different programs putting demands on their time. “The harvest is plentiful but the laborers are few. Are we ready to ask for help? Then we must be ready to compensate them. We must manage our own expectations, and be realistic” At the same time strengthened management will contribute to ensuring VHWs have the tools to do the job since, in the words of a KI: “VHWs are an extended army. They are really relevant provided they have the tools they need. Without the tools, they become irrelevant.” The evident confusion amongst VHWs in some areas could be resolved by having a defined management structure with clear lines of communication and accountability. The range of activities undertaken by the VHW, means that they receive instructions and guidelines from a number of other programs – EPI, HIV, PMTCT to name a few. Identifying where these conflict and resolving that conflict would be a role for the national management. MCHIP EVALUATION REPORT 44 Recommendations 1. Continue program in Chimanimani: The program in Chimanimani is considered a success and should be continued in Chimanimani. When possible the resources available should be increased to allow full and equitable distribution of equipment and where possible to support management strengthening. 2. Roll the intervention out throughout Manicaland: As soon as resources permit, the program should be extended to the other districts in Manicaland. This should be accompanied by strengthened and standardized district management and supervision. Success at this level will provide much stronger evidence as to the effectiveness of the approach and allow increased advocacy at national level. 3. Engage MoHCC through PMD on strengthening the wider VHW program management structures Without a stronger national VHW program with a clear strategy, implementation plan and management structure, interventions such as MCHIP’s will always struggle. A stronger management of the program is essential to protect the gains already made and prevent VHW from being over burdened by the demands of many different programs. 4. Strengthen coordination with other programs and alignment of guidelines and protocols The confusion regarding the definition of early antenatal booking and the timing of the first dose of SP in pregnancy, whilst not critical in its own right, is symptomatic of a lack of coordination between programs. Each program approaches implementation and training independently and this can lead to conflicting information being presented to trainees and lead to confusion and mistakes. Responsibility for this coordination must lie with the national program manager, a post that does not currently exist. This should be rectified without delay. 5. Rationalise and align the cHMI systems. The cHMIS introduced by MCHIP has several advantages over the existing national HMIS community component. However the two do not collect the same data, the latter collecting a wider range. The wider range reflects the role the VHWs play in programs other than MNCH and the demands of these programs for data. The instruments used to collect this data are not user-friendly and undoubtedly lead to data of doubtful quality. It is recommended therefore that MCHIP work with Provincial and District HIOs to develop a tool, similar to the cHMIS to cover the full range of data needs. MCHIP EVALUATION REPORT 45 Quotes from VHWs and Other Informants i “We are not consulted. We just receive names of who should attend planned courses and therefore selection is not based on needs that we are aware of” “I was recruited in 2004 and since then have not received any refresher training yet others have attended so many trainings”. ii “If you can see from what records we handed to you we lack stationary. For referral we sometimes write on whatever paper we can lay our hands on”. “The record books also aid us when we are stuck. They provide good guidance” iii We only provide RDT kits and malaria treatment in peak seasons and provided we have enough”. iv “Referrals are being managed well and in many cases make a difference.” v “My temperature was too high so the health worker referred me to Biriiri hospital where I produced the referral letter that helped me get served faster. I went back to VHW and told about the result which I was happy about” “When I took my child to the VHW, the child had convulsions so I was referred to the clinic. When I visited the facility I did not have to join the queue. When I got back home I told the VHW of my experience.” “The VHW tested me for malaria and I came out as positive. She did not have medication therefore she referred me to the clinic. She actually escorted me to the facility!” vi “The danger signs were well understood however timeliness in some areas was affected by distances and terrain. Pre referral treatment was being given appropriately. Referral forms were being completed appropriately” “Most cases referred were appropriate but need more training in timeliness and provision of appropriate pre-referral treatment and management” vii “Now I can stand in front of people with confidence; they always look for me and I am satisfied.” viii “We are no longer required to work in “food-for-work” activities yet still receive our allocations.” “The roads to our homesteads have been cleared to allow easy access for the community and vehicles.” The fact that I am consulted at all gatherings makes me happy. ix “We feel less confident when dealing with higher class people in the community.” “The Johanne Marange apostolic sect does not welcome fumigation of their houses and seeking medical help.” “Husbands are less participatory in registering pregnancies.” x “Sometimes we come for meetings and they tell us to go back” “The nurses here have attitudes; “You VHWs and your clients give us too much work”.” “Clients referred are not always attended. Sometimes they are sent back three times before receiving service.” MCHIP EVALUATION REPORT 46 “I know some nurses have attitudes and we are trying to deal with that” xi “It brought convenience to the community as we have been trained to help the people near us.” “We provide free services to communities and save them transport costs.” “The very difficult terrain in Chimanimani is a challenge to VHWs as we sometimes cannot even ride bicycles to offer appropriate service or end up using own money to follow up clients” xii “At times we aren’t available because we need to have income to survive so we go away at times and when the people need help we won’t be around to assist.” “We lack equipment like thermometers.” “We do not have paracetamol to give our patients at home.” xiii “Community see that we are part of the health system. Makes us feel very good” “Peer supervision is very key as we do the same thing and where I am not sure my peer will show me the way.” “We are helped as we consult each other when we both don’t have any idea we go to the nurse for guidance.” “ … helps to support each other” “I can ask if I have forgotten something” xiv “We need more home support visits by health workers” “We are visited quarterly for supervision.” “We are supposed to be supervised quarter but I think they are at times too busy to visit us, but the MCHIP came in July this year.” “Twice a year by the nurse who checked on my books, check the rubbish pit and the toilet. They last came in 2015.” “I have never had a supervisor who visited me in 2 years except once when an EHT came to visit me.” xv “If they could give us some soap on quarterly basis and cell phones to enable us to send statistics of the patients we treat.” “There is need for an allowance to enable us to buy paraffin to have a light to use in assisting patient.” “We do not know the criteria for selection. Some of us attend many courses others nothing”. xvi “We continue to work. We admit this is voluntary work but sometimes it is very difficult” “I was trained as VHW 20 years ago but beside information from monthly meeting I have not been selected for any major training in mother and child health.” “We are treating without protective clothing, like gloves.” “MCHIP raised our expectations which are now threatened” “I’m satisfied with my job because there are no longer people dying of malaria” “Death rates have reduced in our area because treatment is free and we are close by.” MCHIP EVALUATION REPORT 47 xvii “MCHIP has boosted us by providing raincoats, umbrellas, bags, T-shirts.” “Those that do not have MCHIP are missing out” xviii “We reach the hospital after receiving help from them” “Ward 13 is very happy with VHWs” “They are our health resource persons.” “Hygiene has improved because of them (VHWs)” “They played a pivotal role in encouraging people to use the HF when they are ill. As a leader in my community I can see that the people now value health unlike before. The VHWs encouraged sanitation through building toilets and erecting good stalls where plates are washed and left to dry. There is a huge improvement in that regard. They enlightened the people in the community and they now know that some of the things they used to overlook are harmful to health. That consciousness is a result of the efforts by VHWs. Its very helpful”. xix “You VHWs and your clients give us too much work” xx “I went to the VHWs with my husband who was sick. He tested positive for malaria and we were given medication and told to report to hospital if he did not improve. He healed and we were very happy” “There was resistance from the people and they did not want to listen to VHWs and I had to chip in and threaten the people to chuck them out of my ward if they don’t listen to them. Some of the women resisted to have their children’s growth monitored by VHWs as they looked down upon them. This was solved by consulting the local leadership, like me as the councilor. We would then introduce the VHWs to the people. I would urge the people to take them seriously.” “We also encourage community incentives such as exemptions from work on development programmes and even contributions” xxi “Some clients are referred but do not report to clinic on time due to cost of transport and distances from “When ill one goes to the Health facility as there is not much VHWs can do” xxii “We no longer have people dying of malaria.” “We promote malaria preventive measures and it is working. Malaria has reduced”. xxiii “VHWs have lowered costs considerably for clients and more decentralised services are recommended” “Our clinics are far and with VHW close to us, lives are saved free of charge”. “VHW’s reduce the costs of treatment because our catchment area does not have any clinics but a hospital only. For example if you have a headache you come here to the hospital and pay $5 consultation fee and $2 for paracetamol. If you go to the VHWs all that is for free so one would have reduced costs” “The clients were saved travel and hospital costs.” xxiv “We were very happy” “We were helped” “We always go there” ‘Going to VHWs when sick only delays treatment as many times they do not have the treatments”. MCHIP EVALUATION REPORT 48 “Malaria is only tested here at the clinic because all the equipment is available.” “Our village health worker only weighs children but does not escort pregnant women to the HF; she does not visit people in the villages at all. She does not tell us why she does not offer help. The last time we convened for a meeting to discuss the issue she was absent”. “Our community is not happy at all because she does not deliver. She tells you that she is busy with her work. She does not visit people at all”. xxv “It is easier for the younger VHW to ride bicycles and do their work however sometimes when needed they will not be available as some travel to buy and sell goods to take care of their families”. xxvi “Make sure each village has a VHWs” “Some villages are very big” “They are doing very good work. We encourage the local leadership to exempt them from community work” xxvii “We are trying to use some of the Health Transition Funds (sic) to boost supplies for VHWs” xxviii “In Nyanga there are now 400+ VHWs. That’s more than health staff. We need an officer at district level to manage them properly” “I broke my arm on my way to treat a malaria patient. I could not afford to have an x-ray done but as a health worker volunteer I expected to be helped as well as I was affected in the course of my duties.” xxix “Community also have expectations of receiving services as they are encouraged to report to the VHW first for minor ailments. In this regard failure to offer help wastes time and resources for clients.” xxx “Volunteers are very committed and they do a lot” “The program has made a big impact and now we have data to prove impact” xxxi “Community has ownership of its own health – the VHWs have taken charge” “They (VHWs) organize transport to the clinic either through the clinic or traditional structures” “The VHWs is a friend to the patient. They can easily reach them and negotiate with them. (to take the desired action)” xxxii “We can now afford to take lunch, and tea-breaks” “Quality of care has improved because of less pressure of work” “Malaria has gone down due to prevention and we have fewer deaths – 20 last year and only 7 so far this year” xxxiii “VHWs are not that educated; the flip charts are very helpful. I have been there, witnessed it and it is very good” “Peer supervision is working much better than management” “The peer supervision aspect is important – they need more training and support however.” MCHIP EVALUATION REPORT 49 51 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523 Tel: (202) 712-0000 Fax: (202) 216-3524 www.usaid.gov U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523