ADVANCING PARTNERS & COMMUNITIES Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program December, 2014 Advancing Partners & Communities Advancing Partners & Communities (APC) is a five-year cooperative agreement funded by the U.S. Agency for International Development under Agreement No. AID-OAA-A-12-00047, beginning October 1, 2012. APC is implemented by JSI Research & Training Institute, Inc., in collaboration with FHI 360. The project focuses on advancing and supporting community programs that seek to improve the overall health of communities and achieve other health-related impacts, especially in relationship to family planning. APC provides global leadership for community-based programming, executes and manages small- and medium-sized sub-awards, supports procurement reform by preparing awards for execution by USAID, and builds technical capacity of organizations to implement effective programs. Recommended Citation Adamchak S., B. Randrianasolo., O. Rabozakandraina, K. Katz., J. Bratt., A. Brinson, S. Mercer, M. Brown, and A. Hernandez. Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program. 2014, FHI 360: Durham, NC. Acknowledgment This mid-term evaluation of the Malagasy Heniky ny Fahasalamana program (Malagasy Healthy Families, or MAHEFA) program could not have been completed without the collaboration and cooperation of literally hundreds of people. Citizens, community health workers, technical agents, NGO grantees and partners, staff of the Ministries of Health and Water as well as MAHEFA staff themselves gave generously of their time and their opinions to participate in focus group discussions, in-depth interviews, and local observations and site visits. The MAHEFA team was extraordinarily responsive to requests for information, and did everything possible to facilitate the evaluation team’s ease of movement during field work in three regions. USAID/Madagascar provided guidance and encouragement for this endeavor. The members of evaluation team gratefully acknowledges the support of all. This publication was produced by Advancing Partners & Communities (APC), a five-year cooperative agreement funded by the U.S. Agency for International Development under Agreement No. AID-OAA-A-12-00047, beginning October 1, 2012. The authors' views expressed in this publication do not necessarily reflect the views of the U.S. Agency for International Development or the United States Government. ADVANCING PARTNERS & COMMUNITIES Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program December, 2014 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program i Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program TABLE OF CONTENTS ACRONYMS ..................................................................................................................................II LIST OF TABLES .........................................................................................................................IV LIST OF FIGURES .......................................................................................................................IV EXECUTIVE SUMMARY.............................................................................................................. V I. INTRODUCTION .............................................................................................................. 1 Evaluation Objectives and Expected Outcomes ........................................................................................................ 3 II. METHODOLOGY .............................................................................................................. 4 Geographic Scope and Sample ....................................................................................................................................... 4 Key Stakeholders and Beneficiaries ............................................................................................................................... 5 Fieldwork ............................................................................................................................................................................. 6 Data Transcription, Translation and Analysis ............................................................................................................. 7 III. FINDINGS........................................................................................................................... 7 PART 1: INTERVIEW FINDINGS ................................................................................................................................ 7 1. Beneficiaries ........................................................................................................................................................... 7 2. Community Health Workers (CHWs) ........................................................................................................ 12 3. Community leaders ........................................................................................................................................... 17 4. Technical Accompaniers (Technical Agents) (TA).....................................................................................21 5. NGO Grantees ................................................................................................................................................... 24 6. Regional and National Levels .......................................................................................................................... 26 Discussion of Interview Findings .................................................................................................................................. 33 PART 2: RESULTS OF ECONOMIC ANALYSES .................................................................................................. 39 1. Cost-efficiency analysis ..................................................................................................................................... 39 2. Earned Value Management (EVM) .................................................................................................................. 44 IV. LIMITATIONS AND CONCLUSION ............................................................................ 49 Appendix 1: Map of Intervention Zones for USAID/MAHEFA ............................................................................ 50 Appendix 2: Summary of site visits ............................................................................................................................ 51 Appendix 3: Summary of visits to water points ...................................................................................................... 53 ii Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program ACRONYMS ACT Artemisinin-based combination therapy BCE Behavior change empowerment BENM Beneficiaries of activities aimed at mothers BENTU Beneficiaries of emergency transport BENW Beneficiaries of WASH activities CBIHP Community-based Integrated Health Program CHW Community health worker CIMCI Community Integrated management of childhood illness (PECIMEc) CLTS Community-led total sanitation CoSan Commité de Santé (Community Health Committee) CSB Centre de Santé de Base (Primary health care center) CU5 Children under age 5 DRSP Direction Régionale de la Santé Publique (Regional Director of Public Health) DIREAU Direction Régionale en Eau (Regional Water Director) EMMP Environmental Monitoring and Mitigation Plan EV Earned value EVM Earned Value Management FGD Focus group discussion FP Family planning FTE Full-time equivalent GOM Government of Madagascar IDI In-depth interview MAHEFA Malagasy Heniky ny Fahasalamana (Malagasy Healthy Families) M&E Monitoring and evaluation MI Medical Inspector MNCH Maternal, newborn and child health MOH Ministry of Health MOU Memorandum of Understanding MSIM Marie Stopes International, Madagascar NGO Non-governmental organization ORS Oral rehydration solution PA Point d’Approvisionnement (Provision point) PSI Population Services International RMA Rapport mensuelle d’activités (Monthly Activity Report) RSE Regional M & E Officers iii Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program STI Sexually transmitted infection TA United States Agency for International Development USAID United States Agency for International Development USD U.S. dollar WASH Water, sanitation and hygiene iv Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program LIST OF TABLES Table 1: MAHEFA Framework ....................................................................................................................................... 3 Table 2: Locations of site visits by region, district, commune and Fokontany ................................................... 5 Table 3: Distribution of Focus Group Discussions and In-Depth Interviews by Subject ................................. 6 Table 4: Cumulative expenditures by region (in US$) ............................................................................................ 40 Table 5: Cumulative outputs by region ...................................................................................................................... 41 Table 6: Productivity and cost-efficiency measures by region .............................................................................. 42 Table 7: MAHEFA activities percent complete by project year ........................................................................... 46 Table 8: Amount spent by program area, June 2014 .............................................................................................. 46 LIST OF FIGURES Figure 1: Illustration of MAHEFA Structure ................................................................................................................ 2 v Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program EXECUTIVE SUMMARY Evaluation Objectives and Expected Outcomes At the request of the United States Agency for International Development (USAID) Mission in Madagascar (USAID), FHI 360 undertook a qualitative mid-term evaluation of the Malagasy Heniky ny Fahasalamana program (Malagasy Healthy Families, or MAHEFA, CA number 687-A-00-11-00013-00). MAHEFA provides improved services in maternal and child health, reproductive health, nutrition, water and sanitation, hygiene and malaria treatment in difficult-to-access and underserved areas in six regions of Madagascar. It is a five-year program (May 23, 2011 to May 22, 2016) implemented by JSI Research & Training Institute, Inc., The Manoff Group and Transaid. The evaluation was undertaken to produce a strategic review of project performance to date, in order to achieve the following objectives:  To help USAID Mission and the project implementer identify and understand successes to date, problems and challenges that need to be addressed; and provide actionable short and medium-term recommendations.  To assess the cost-efficiency of different interventions across the project’s zones of intervention.  To ascertain to what extent the cost incurred is adequate to the work completed, given the project's goal related to equity and geographic coverage. Will the remaining funds cover the life of project planned activities? Will the tasks be done given the remaining time and money? Methodology A five person team composed of a social scientist, two economists and two doctors with research experience conducted the evaluation during May through August, 2014. Qualitative methods were used to respond to objective 1, including in-depth interviews, focus group discussions, document reviews, and observations. Relevant program budgets, financial reports, annual work plans and annual reports were used to collect the cost inputs needed to conduct a cost-efficiency analysis and an earned value management analysis. Three regions (Diana, Sofia, and Menabe), and within them, 8 districts, were purposively selected for field visits. Seventeen different categories of stakeholders were included in the data collection, in addition to MAHEFA staff. In total, the team conducted more than 33 in-depth interviews (IDIs), 28 focus group discussions (FGDs), and 11 courtesy visits. The team also visited 10 community health worker sites, and seven water points supported by MAHEFA. The sample districts are representative of program beneficiaries and program activities, but it must be noted that they were selected in part due to their relative accessibility. The MAHEFA program functions in some of the most disadvantaged areas of Madagascar, and the successes and challenges encountered in more distant locations may differ from those observed by the team. Similarly, while a large volume of data were collected, in aggregate it represents a small fraction of the actors touched by the MAHEFA program. All in-depth interviews and FGDs were digitally recorded, transcribed, and translated; transcriptions were coded using N-Vivo 10.0 with codebooks continually revised as new data became available. During analysis, key themes and relevant quotations were summarized from the coded data, and subsequently synthesized for this report. vi Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Findings Beneficiaries Virtually all participants in FGD with beneficiaries, mothers of children under 5 and WASH (Water, sanitation and hygiene) accurately cited the range of topics community health workers (CHWs) are trained to promote in their communities. Nevertheless, not all participants reported putting what they learned into practice, indicating a need for continuous reinforcement of healthy behaviors. In every FGD, at least one participant indicated they had sought care for a child with general symptoms of illness. Adults generally sought care for family planning. Proximity and ease of access were cited as the main reasons for seeking care from a CHW. However, many participants reported going directly to a health center (CSB) or hospital, rather than visiting the CHW, viewing the CHW more as a “sensitizer” than a provider. This may have been in part a reflection of proximity to CSB in some of the locations visited, and also reflecting a program strategy, that CHW residing within 5 kilometers of a CSB should devote most of their effort to community sensitization. Respondents participating in FGD about emergency transport were generally positive about the service, but expressed reservation about safety, speed, and the medical or first aid skills of the riders. In every FGD with beneficiaries, at least one benefit to individuals or the community was mentioned, indicating a sustained appreciation overall of the work done by CHWs. The most significant challenge to this is seen in changing WASH behaviors, in particular in building latrines, promoting consistent hand washing practices, and eliminating open defecation. CHW CHWs were appreciative of their training, with several across different FGD commenting that they were challenged in completing registers and preparing reports. There is a significant burden in that the program utilizes different registers for each core area of focus, consistent with Government of Madagascar guidelines. CHW receive direct support and supervision from TAs associated with MAHEFA NGO grantees, and are also meant to be supervised by their local CSB. Due to resource constraints, the latter happens far less frequently, and often only is manifested in the monthly reporting meeting held at the CSB. Overall CHWs found the TAs to be friendly and helpful during their regular supervisory visits. Relations with the public sector CSB were less favorable; participants in two of six FGD believed the CSB were not cordial in receiving CHW-referred clients, and that they were strict and not helpful. In more than half of the CHW FGD, participants mentioned problems with stock outs of their supplies, and noted that they needed to replenish at pharmacies, causing them to increase their prices. This situation was not explored in depth in the FGD. In all but one FGD, CHWs noted positive health outcomes as a result of the sensitizations they conducted in their communities. These included fewer childhood illness, uptake of family planning, more vaccinations, and more people practicing good hygiene. Messages regarding maternal and child health and family planning resonated more strongly with the community, while WASH topics were more difficult to convey in a manner that effected change in the population. CHW mentioned three major challenges in carrying out their work: transport, safety and compensation. Nevertheless, most CHWs reported professional and personal satisfaction with their engagement with MAHEFA. vii Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Community leaders FGD and IDI were conducted with several cadres of community leaders: members of Community Health Committees (CoSan), organizers of mutuelles de santé (health mutual aid societies), mayors of communes, and directors of CSB. Members of the CoSan reported that the efforts of the CHW have resulted in an increase in health care seeking in their communities, with people making more visits both to the CSB and to CHWs. CoSan members reported increases in vaccinations, family planning use, prenatal care, and institutional deliveries. Within the FGD with CoSan members, challenges were mentioned such as the competing priorities for CHW between minding their fields and carrying out their health tasks; surmounting the obstacles of working with illiterate populations; and finding appropriate locations to construct toilets. MAHEFA has introduced several pilot tests of mutuelles de santé, seemingly to mixed results. It appears that recruitment of members has been slow, and among those who have joined there is concern that they have paid into the system, but have not yet reaped benefits. That said, it is acknowledged that these community insurance schemes are an innovative approach that requires multiple steps to develop awareness of community health needs, establishment of management systems, and extensive community engagement in order to succeed. Mayors of communes were very positive about the impact of the MAHEFA program, and noted many positive outcomes, including the proximity of CHWs for people seeking care, affordable drugs and medications, a move away from traditional medicine, better data on health events, better sanitation, and decreased morbidity. Interviews with CSB directors revealed a broad appreciation for the sensitization work carried out by the CHWs. All reported changes in basic health and hygiene behaviors, and specifically pointed to the increased use of family planning in their communities, and higher rates of vaccination. The directors also pointed to greater use of CSB and CHW health and medical care and a move away from traditional healers, but it is not possible to verify if this perception reflects a real change. The interviews with CSB directors also underscored the challenges of working with the public sector due to the recent political crisis in Madagascar and the ensuing United States government sanctions against working with the government of Madagascar. For example, the program could not provide training to CSB staff and, as a result, the CSB directors did not take part in CHW supervision. Some were reluctant to validate their practical training and some complained about the CHW report quality. Technical Agents All of the TAs interviewed expressed appreciation for their collaboration with MAHEFA and community actors such as CoSan and mayors. Six of eight believed that the CHWs were effective in communicating health information and promoting behavior change. There was a perception that the training in different topics provided by MAHEFA motivated the CHW and made them feel more responsible in carrying out their tasks. According to the TA, the proximity of CHW to the community also promoted more frequent and earlier care seeking. Particular improvements were cited in WASH habits, and in the uptake of family planning. The TAs mentioned several challenges to CHW performance, consistent with those noted by other respondents. Specifically they mentioned stock out of some socially marketed products, low allowances for the CHW, delays in certifications due to resistance on the part of CSB directors, and poor performance in completing reports and forms correctly. Virtually all the TAs mentioned the deeply entrenched social norms regarding open defecation as a major hurdle confronted by the CHW in their efforts to promote better hygiene and sanitation. viii Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Half of the TAs reported that they struggled to complete their work, primarily due to large distances that some must travel in order to supervise CHW located in distant communities, and the associated problems with available transport. Occasionally they will make a long journey, only to find that the CHW is not present at the time they arrive. NGO Grantees NGO grantee representatives expressed satisfaction with the planning, implementation, technical support and impact of the MAHEFA program. They were appreciative of training, technical support and equipment that they had received. Nevertheless, as might be expected, there were institutional conflicts reported by some of the NGOs in their relationship with MAHEFA. Generally these concerned reporting processes, which were deemed to be strict and very detailed. MAHEFA, for its part, has noted that it conforms to the procedures required by both JSI and USAID, and that it has been a time￾consuming process to provide the technical support needed by some of the local NGOs to reach international reporting standards. NGO grantees also reported some problems in finding appropriately qualified staff in the regions, coping with inaccurate data reported by CHWs, and adequate cash flow. Regional and National Experts The Medical Inspectors (MIs) who were interviewed reported collaborative relationships with the MAHEFA Regional Coordinators. While valuing the work of MAHEFA, they raised concerns about the capacity of many of the NGOs to support integrated health services, considering that several had historically focused on a single health or social issue. This has been addressed by MAHEFA in its support of the NGOs Grantees. The MIs also commented that some community members found it hard to accept CHWs, and that there was also some tension between CSB and CHWs. They also expressed concern about the quality of the data reported by CHWs and TAs. During courtesy visits to the three Regional Directors of Public Health, favorable observations were offered regarding the contributions of MAHEFA in implementing complementary community health interventions. The challenges mentioned by the Direction Régionale de la Santé Publique (DRSP) reiterate those mentioned by other respondents: entrenched social norms regarding sanitation, CSB reluctance to be engaged with CHW training and supervision, and inadequate communication among all actors. The three Regional Directors of Water were unanimous in acknowledging MAHEFA’s success in contributing to an improved water infrastructure and supporting dissemination of WASH sensitization activities. Much of the day-to-day coordination with NGO Grantees falls to the six MAHEFA Regional Coordinators. According to the Regional Coordinators this has presented numerous challenges, in large part due to the relative inexperience of local NGOs in community health. Other constraints, such as limited NGO resources for transportation were mentioned, as were security concerns in some areas. Finally, Regional Coordinators mentioned that the capacity to manage stock varies among the CHWs, and occasional stock outs occur. These do not appear to be serious problems. Two MAHEFA staff members engaged in Monitoring and Evaluation (M&E) were interviewed. They are focused on continually improving data quality received from the NGO grantees. They cited several issues that they confront in this task: turnover in TAs, varying quality of data from the CHWs and challenges in identifying CHW who need assistance and support, due to data aggregation and the difficulty some CHWs face in having their monthly report validated by CSB Directors. ix Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program The six regional WASH Coordinators were enthusiastic about their work, and determined to advance activities that have been delayed from the inception of the program, well documented elsewhere. The process of water point construction or rehabilitation is a lengthy one, requiring multiple consultations with the Ministry of Water, engagement of local consulting firms to evaluate community needs and identify construction sites, contracting construction firms, carrying out multiple Environmental Monitoring and Mitigation Plans (EMMP) and inspections, all within the context of limited time to undertake construction in order to avoid the rainy season. PSI and Marie Stopes International, Madagascar While MAHEFA has memoranda of understanding (MOU) with several partners to enhance project implementation, the evaluation looked at only two: PSI and Marie Stopes International, Madagascar (MSIM). PSI, through its network of social marketing distribution teams, provides products for the CHWs. PSI had not previously had many distribution points in the six regions in which MAHEFA works, and through the MOU the two organizations established a mutually beneficial relationship that expanded PSI’s network while ensuring MAHEFA-supported CHW had access to heath supplies. A similar, complementary relationship exists with MSIM, in which MSIM is able to provide access to long-acting and permanent methods of contraception to meet the needs of CHW clients in MAHEFA program regions. Summary of Interview Findings In summary, the findings from the many interviews and focus groups, as well as observations and review of MAHEFA documents, point to a number of over-arching challenges that the program has confronted, and continues to address as it moves forward. These include:  Entrenched social norms that are resistant to change, particularly related to ending open defecation, constructing latrines, and making hand washing routine  Tensions with CSB directors and Medical Inspectors, grounded in limitations for direct interaction due to political and policy constraints imposed by the US Government  Limited NGO Grantee capacity in regions that have been historically under-resourced and are geographically unappealing for professionals  Dissatisfaction with salaries, per diems and travel allowances on the part of CHW and other community volunteers  Questionable data quality emanating from the lowest level up through the MAHEFA system  Commodity stock outs due to unavailability of particular products, geographic or weather constraints, or poor management on the part of CHWs  Turnover of CHWs and TAs, leading to consideration of how to efficiently train new staff  Mal-distribution of TA workload  Condom use remains low, and there is reported resistance to use by males  Promising results with the introduction of emergency transport, while needing further community mobilization and sensitization its use  Equipment maintenance, including bicycles, wells and pumps  Security concerns both due to increases of organized crime in some locations, and the fears on the part of CHW and TA of being at risk because of their engagement with the program and their perceived wealth x Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Cost efficiency analysis Although the relative rankings vary slightly across the two output-related measures (all outputs per CHW, treatment outputs per CHW) a consistent pattern emerges. Sofia and Boeny rank first or second in terms of both measures of cost-efficiency, Menabe and Melaky are situated in the middle, and Sava and Diana rank last or second-to-last on both measures. Main drivers of expenses in the regions include the number of full-time MAHEFA staff, the number of NGO grantees, and the amount of international partner involvement. Expenditures are driven by the number of CHW in the regions, factors such as remoteness, and the number of activities in sites. Drivers of outputs are include the number of CHW supported and the productivity of the CHW. Earned Value Analysis The results of the earned value analysis (EVA) show that the MAHEFA program is slightly behind schedule (5%) as well as slightly over-spent (USD 878,546) 37 months into the program life-cycle. However, both situations are understandable in the context of the delays experienced in a number of start-up activities in regions not previously supported by USAID health interventions, the challenges of finding appropriate and adequately staffed NGO grantees in the regions, delays incurred during 2013 while a remediation plan was developed and put into place, and the larger investments typically made during the initial phases of such a large and complex program. It is anticipated that spending will slow during Years 4 and 5 of the program as fewer new activities are initiated and MAHEFA is on track to complete most of its planned activities. Recommendations: short term 1. Continue efforts to promote latrine use and routine hand washing practices. Develop illustrative sets of building plans using local materials. Provide estimates of cost of construction using local materials so people will have accurate perceptions of financial requirements. Collaborate with local leaders to conduct “village walks” in which suitable locations for latrines are identified that are mutually agreeable to neighborhoods. 2. MAHEFA headquarters and regional staff should continue to monitor and support NGOs to complete procedures and meet deadlines in order to ensure that the program meets targets in a timely and cost-effective manner. Following the final round of grant making, relevant NGO partners should be convened and the final year activities and objectives clearly delineated, with appropriate milestones established. 3. Increase the frequency with which condoms are part of sensitization efforts in order to reduce the entrenched opposition to their use reported by CHWs. Take advantage of home visits to conduct couple counseling on contraceptives and underscore the contributions of condoms in preventing both pregnancy and sexually transmitted infections (STIs). Recommendations: mid- and long term 4. Together with USAID/Madagascar and the Ministries of Public Health and Water, MAHEFA must determine what level of engagement with CSB directors and MI is feasible during the remaining duration of the program. Expectations must be realistic, and should focus on engaging CSB directors to certify CHWs who are in the training pipeline but have not yet been approved to initiate activities. The CSB directors and MI should also be engaged in each region to clarify expectations about data quality, format and use. 5. Increase the focus on training NGO M&E staff and TAs in data quality management, skills that will contribute to local capacity building for future activities. Hands-on sessions should include xi Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program interpretation and analysis of actual data, using active CHW registers and reports. Similarly, TAs can organize sessions with CHW to review data and exchange experiences in a more in-depth and focused manner, rather than simply collating data. 6. MAHEFA should work with the NGO grantees to establish reasonable numbers of CHWs to be supervised by each TA. Using a geographic unit (commune) for allocation has resulted in some TAs being responsible for up to 60 CHW, while others handle only one-fourth that number. Coupled with the time and distance challenges of supervising such a large number, there is a disproportionate burden shouldered by some of the TAs. 7. CHWs, TAs and MAHEFA regional staff need to manage expectations concerning emergency transport. MAHEFA can strengthen community-level support for emergency transport during the preparation stage, when workshops and technical review are conducted. MAHEFA should reinforce messages provided at this preparation stage about the benefits of the service rather than allowing community members to imagine what calamities might befall people using the transport.” 8. All relevant actors, including members of CoSans, Mutuelles de santé, CHWs and other community partners should be encouraged to promote the benefits of community savings programs for health care, maintenance of wells and water points, and upkeep of emergency transport. If possible, MAHEFA should identify communities in which revolving funds have been successfully introduced, and share the results in other locations, perhaps using testimonials, a promotional video, or portable story board. Considerations for a follow-on program MAHEFA has made important strides in its pilot work in developing mutuelles de santé, in its introduction of emergency transport, and its collaboration with PSI and MSIM, among others, to leverage the impact of program intervention. It is suggested that the lessons learned from these experiences be carried forward and be built upon in a follow-on community health program. MAHEFA is on course to make up for many of the delays experienced in construction and rehabilitation efforts, but there remains a great demand for clean water sources in the program regions. Again recognizing that some important lessons were learned, ranging from the specific language that must be included in a Cooperative Agreement to a realistic appreciation of the time required to identify sites, obtain permits, undertake and fully complete construction and conduct final inspections, such efforts should be included in a follow-on program. Similarly, MAHEFA has faced an uphill battle in promotion of latrine use. This very fundamental element of good public health and sanitation practices must be reinforced. It may be worth considering a study tour to India, which is currently waging a very public, nationwide effort to also reduce open defecation, to learn from the experiences there in promoting social change. This may be an opportunity to engage Government of Madagascar (GOM) counterparts from the Ministries of Health (MOH) or Water, so as to spur greater focus on this issue. Alternatively, MAHEFA could convene a workshop of regional MOH and MOW to share the experience of India (perhaps inviting an Indian expert to travel to Madagascar), to explore what best practices may be adapted to the context of Madagascar. From the standpoint of sustainability and contributing to improved quality and utility of national health data, future projects should consciously invest in training and promotion of lower level health cadres such as CHW on good data management practices. This need not be a complex exercise, but it should warrant specific and sustained attention during initial training and during on-going supervisions. This focus should be maintained upward through the data reporting system. xii Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Finally, in a complex community health program such as MAHEFA, implemented on such a large scale and decentralized manner, it is essential that a follow-on project be realistic in its estimation of the capacity of regional and local partners, be they NGOs or public sector. MAHEFA invested an unexpectedly large amount of effort in supporting its NGO grantees. This has likely contributed to stronger capacity within many of the organizations, but there will continue to be additional development needs in the future. Should a future program collaborate more directly with local health authorities, it is very likely that some similar reinforcement of skills in technical updates, data management, supervisory practices, and materials support may also be necessary. 1 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program I. INTRODUCTION The USAID/MAHEFA program, known as “Malagasy Heniky ny Fahasalamana” (MAHEFA, CA number 687-A-00-11-00013-00), and translated as “Malagasy Healthy Families” aims to provide improved services in maternal and child health, reproductive health, nutrition, water and sanitation, hygiene and malaria treatment in difficult-to-access and underserved areas. Interventions are taking place in 24 districts within 6 regions of Madagascar: Menabe, Melaky, Boeny, Sava, Sofia, and Diana (see Appendix 1 for map of intervention regions). This Community-Based Integrated Health Program is a five-year program (May 23, 2011 to May 22, 2016) implemented by JSI Research & Training Institute, Inc. and two international partners, The Manoff Group and Transaid. The objective of the program is to increase the use of proven community-based interventions and essential products among underserved populations in six northern and western regions of Madagascar. Achievement of this objective is contingent on reaching three intermediate results: 1. Increase demand for high quality services and products 2. Increase availability of high impact services and products 3. Improve the quality of care delivered by community-based health practitioners. By improving the health status in focus districts and regions of Madagascar, the program will touch close to a quarter of the Malagasy population or 3.9 million people. Among the results expected are:  Public health impact at scale  Strengthened capacity of information and monitoring, logistics, training and supervisory systems  Increased coverage among the most vulnerable and marginalized  Increased community participation in decision-making concerning management and financing of local health and water/sanitation services  Advanced global best practices in family planning and maternal, newborn and child health MAHEFA is a decentralized program that functions under the guidelines of the Madagascar National Policy for Community Health and the National Water Code (see Figure 1). All activities undertaken by the program are aligned with national priorities, and the program strives to contribute to national public health goals, as well as USAID/Madagascar priorities. MAHEFA partners with an array of local, international and government partners at both the national and regional levels. There are five regional offices (activities in Boeny are managed from headquarters, with a small satellite office in Boeny). Staff in the regional offices are direct liaisons with the local non-governmental organization (NGO) partners responsible for implementing specific activities of the MAHEFA program. Regional offices are staffed by: a Regional Director; technical coordinators for health; water, sanitation and hygiene (WASH); monitoring and evaluation (M&E) officer; finance officer; and relevant support staff. 2 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Figure 1: Illustration of MAHEFA Structure Madagascar National Policy for Community Health National Water Code Through a series of annually-awarded grants, MAHEFA works through local NGOs to recruit, train, and deploy technical accompaniers or agents (TA) who are responsible for supervising community health workers (CHW). The CHW are a cadre authorized in the National Policy for Community Health. Whereas some CHW were previously trained only in community integrated management of childhood illness (C-IMCI, referred to locally as PCIMEc), a unique feature of the MAHEFA program is that CHW are trained in diverse public health interventions, including: maternal, newborn and child health (MNCH); family planning (FP) and prevention of sexually transmitted infections; prevention and treatment of malaria; nutrition counseling; and WASH, including water treatment and storage, hand washing, and latrine building. CHW are trained to provide counseling and treatment of basic illnesses, generate demand through community sensitization and home visits, and when necessary, refer clients to health facilities for care. They also sell a range of products including medicines, water purification solution, and family planning methods provided through the USAID-funded social marketing program. MAHEFA PARTNERS: PSI, MSM, PC, JHPIEGO, UNICEF, Government of Madagascar 3 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program The work of the CHWs takes place in an environment of mutually-supportive program activities (see Table 1) that:  Fosters engaged communities to demand and utilize services  Provides technical support and training  Ensures products and infrastructure are available to support healthy communities Table 1: MAHEFA Framework Engaged communities to demand and utilize services Technical assistance and training Products and infrastructure to support healthy communities  Champion communities (KMSm)  Behavior change empowerment (BCE)  Community-led total sanitation (CLTS)  Community health insurance (mutuelles de santé)  Youth engagement  Technical training in maternal, newborn, and child health; water, sanitation and hygiene, nutrition, integrated community case management, family planning/reproductive health, malaria  Technical training in BCE  Follow-up supervision  Connecting to the referral system  Refresher trainings  Improved access to potable water and improved sanitation facilities through construction of wells  Logistical support for materials and tools  Mobility of CHWs  Emergency transport Source: Madagascar Community-Based Integrated Health Program (CBIHP) 2014Purpose of Evaluation The evaluation findings will be primarily used for improving the effectiveness and efficiency of the MAHEFA program, and to some extent used by USAID for the design of follow-on community health care activities. EVALUATION OBJECTIVES AND EXPECTED OUTCOMES According to the evaluation scope of work, the expected outcome of the USAID/MAHEFA mid-term evaluation is a strategic review of project performance to date, in order to achieve the following objectives: 1. To help USAID Mission and the project implementer identify and understand successes to date, problems and challenges that need to be addressed; and provide actionable short and medium-term recommendations. 2. To assess the cost-efficiency of different interventions across the project’s zones of intervention. 3. To ascertain to what extent the cost incurred is adequate to the work completed, given the project's goal related to equity and geographic coverage. Will the remaining funds cover the life of project planned activities? Will the tasks be done given the remaining time and money? 4 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program II. METHODOLOGY A five-person evaluation team was formed by FHI 360. FHI 360 team members included Dr. Susan Adamchak, team leader and responsible for analysis of Objective 1; John Bratt, an economist tasked with conducting the cost-efficiency analysis included in Objective 2; and Angela Brinson, who carried out the earned value management analysis. Two local professionals were also included on the team, Dr. Bodo Randrianasolo served as Researcher, and Dr. Oliva Rabozakandraina served as Research Associate. Qualitative research methods were used to collect data necessary to respond to the first objective. To identify challenges encountered and successes achieved by the project, the team conducted in-depth interviews (IDIs) with key informants, carried out a series of focus group discussions (FGDs) with program stakeholders, conducted field observations, and implemented document reviews. This part of the evaluation focused on activities related to reproductive health, maternal and child health, water and sanitation, and infectious and other diseases. Youth peer education and m-Health, bicycle distribution and use, and a comprehensive review of data quality were not included. Relevant program budgets, financial reports, annual work plans and annual reports were used to collect the cost inputs needed to conduct the financial analyses. Cost-efficiency considers the resources being used to produce output(s). Efficiency means either that 1) the most output possible is produced for the resources used, or 2) minimal resources used to produce a given level of output. An Earned Value Management (EVM) methodology is used to analyze program performance relative to budget and time. By comparing the budgets for work scheduled to be completed by a certain time to the budget of work actually completed by that time and the costs incurred for the work completed, variances in schedule (behind or ahead of schedule) and cost (under or over budget) can be identified. Further details on the methods used for each economic analysis can be found with the presentation of the results later in the report. GEOGRAPHIC SCOPE AND SAMPLE Three regions were purposively selected for inclusion in the field visits (Table 2). Menabe was included because it is the first region within which MAHEFA began to work, and is also the site of several pilot projects, including emergency transport, distribution of Chlorhexidine and development of health mutuelles societies. Sofia is the largest region in which MAHEFA works, accounting for nearly forty percent of the CHW trained under the auspices of the program. It is also characterized by a great number of communities that are inaccessible by road for significant periods throughout the year. Finally, Diana is the region in which activities were most recently initiated and will serve as a reference for the scale up of activities. Within each region, two (Sofia) or three (Diana, Menabe) districts were selected for field visits, based on distance from regional capitals or market towns, road conditions, and program activities. The sample districts are not statistically representative of the program beneficiaries or program activities in the regions, and it must be kept in mind that while a large volume of data were collected, in aggregate it represents a small fraction of the actors touched by the MAHEFA program. The sample districts are not representative of the full array of program beneficiaries and program activities, but they were selected in part due to their relative accessibility. The MAHEFA program 5 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program functions in some of the most disadvantaged areas of Madagascar, and the successes and challenges encountered in more distant locations may differ from those observed by the team. Table 2: Locations of site visits by region, district, commune and Fokontany Region District Commune Fokontany Diana Diego I Antsiranana II Ambanja Diego Suarez Sakaramy Djangoa Ambodimanga Ramena Antranokarany Cap Diego Anamakia Ambodimanary Ankarongana Ambolodimaka Androhibe Sakalava Sofia Antsohihy Boriziny Ambodimadiro Antsahabe Anjiamangirana Tsarahasina Boriziny II Antsahabe Anjiamangirana II Matsaborifanjava Ampobibitika Menabe Morondava Mahabo Belo/Tsiribihina Bemanonga Analaiva Mahabo Tsimafana Beroboka Tanambao Marofototra Tsinjorano Analaiva Tanandava Tsimafana Beroboka Centre KEY STAKEHOLDERS AND BENEFICIARIES The evaluation team collected data from MAHEFA program staff at headquarters and regional levels; staff from the Ministry of Public Health at national, regional, district and commune levels; local NGO grantees; technical agents; health committee members and directors of primary health care centers at the commune level, and community leaders such as mayors of fokontany; community health workers; and direct beneficiaries such as caretakers of children under age 5, well or water point users and users of emergency transport. The total number of FGDs and IDIs for each subject category is summarized in Table 3. 6 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Table 3: Distribution of Focus Group Discussions and In-Depth Interviews by Subject Name Type of transcript Number of transcripts Beneficiaries – Mothers (BENM) FGD 6 Beneficiaries – Emergency Transport (BENTU) FGD 2 Beneficiaries – Water (BENW) FGD 4 Community Health Workers (CHWs) FGD IDI 8 1 Community Health Committee members (CoSan) FGD 5 Director of Primary Health Care Center (CSB) IDI 7 Mayor (Maire) IDI 4 Technical agent (TA) IDI 8 NGO IDI 4 Health Mutuelles (Insurance) (MS) FGD IDI 1 1 Medical Inspector (MI) IDI 3 Regional Coordinator FGD 1 RSE (M & E officers) IDI 2 Marie Stopes IDI 1 WASH Regional Coordinators FGD 1 PSI IDI 1 TU- Emergency Transport IDI 1 Total 61 (FGD=28 / IDI=33) In addition, courtesy visits were made to two Medical Inspectors, and staff from the regional offices of the Ministry of Public Health and the Ministry of Water. Finally, observational visits were made to 10 sites where CHWs work. These visits used a checklist to assess CHW training, if there was a toilet and tippy tap, and if they had certain registers, tools, medicines, and other products. FIELDWORK Work began in May 2014 with a review of documents provided by USAID/Madagascar and MAHEFA. These included annual work plans, annual reports, financial reports, monitoring and evaluation tools, research summaries, the cooperative agreement and amendments, and a remedial plan developed in 2013. Fieldwork began June 2, 2014. The in-country team of Adamchak, Randrianasolo and Rabozakandraina spent one week in Antananarivo, meeting with the USAID/Madagascar evaluation team, briefing with MAHEFA staff, conducting interviews with key technical personnel, representatives of NGO partners, and finalizing fieldwork plans and interview guides. From Antananarivo, the field team traveled to the Diego Suarez, the regional capital of Diana on June 9, 2014. During the following two weeks the team traveled by road through Diana and Sofia, conducting interviews and focus group discussions with a wide array of stakeholders. Subsequently, the two local researchers traveled to a third region, Menabe, July 6 to 13, 2014. 7 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program DATA TRANSCRIPTION, TRANSLATION AND ANALYSIS All in-depth interviews and focus group discussions were digitally recorded. Recordings were transcribed by a team of experienced transcriptionists into either French or Malagasy. French transcriptions were immediately forwarded to FHI 360, while Malagasy transcriptions were sent to a second team of qualified translators. All transcriptions were verified by the local researcher and research associate prior to being forwarded to the translators, and translations were again verified before the final transcripts were sent to FHI 360. Upon receipt at FHI 360, transcripts were logged into a central data base, and were then assigned to one of several analysts for coding. The majority of transcripts were coded using N-Vivo version 10.0, using a codebook that had been initially developed based on the interview and focus group discussion guides. As qualitative data analysis is typically an iterative process, the code books were revised and updated as new themes emerged in the coding process. In a small number of cases, primarily when only a single person with unique input was interviewed, the transcripts were summarized in Microsoft Word. During analysis, key themes and relevant quotations were summarized from the coded data, and subsequently synthesized for the report. While the results include descriptions of MAHEFA activities we are only providing a very broad-brush description of the full MAHEFA program, given that the intended readers will largely be very familiar with the program structure and primary activities, all of which are well documented in MAHEFA's annual reports and planning documents. III. FINDINGS The results are organized into two parts. The first part presents the findings of the interviews, and is organized according to the MAHEFA structure as presented in Figure 1. We start by discussing the findings at the beneficiary level, then move up the pyramid to the CHWs, the level of the community leaders ((members of the health committees (CoSan), mayors and directors of primary health care centers (CSBs)), technical agents (TAs), NGO partners, and finally regional and national level MAHEFA staff. Each section presents the main successes and challenges noted at each level. Part two presents the results on cost efficiency and earned value management analyses. The summary of the site visits to CHW can be found in Appendix 2, and a summary of site visits to water points is included in Appendix 3. PART 1: INTERVIEW FINDINGS 1. Beneficiaries Community members, the primary beneficiaries of the MAHEFA program, receive health education or sensitization through public meetings and home visits, as well as health services through the CHWs. Both are intended to improve health seeking behavior leading to a positive impact on community health. Beneficiary perceptions of sensitization activities, and the influence of these activities on their health seeking behavior and on the health of their communities are described below. The findings come from FGDs with the beneficiaries of activities aimed at mothers with children under the age of 5 (BENM), beneficiaries of WASH activities (BENW), and beneficiaries of emergency transport services (BENTU). 8 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Sensitization In all but one of the FGDs with BENM and BENW participants talked about the CHWs activities in sensitizing communities through public meetings on a range of topics focused on improving maternal and child health and water, sanitation and hygiene (WASH). Topics discussed during community events included child nutrition and feeding practices (including breastfeeding through 6 months), childhood vaccinations, awareness of seeking care for children from CHWs, hand washing, bathing children, latrine building, and family planning. In addition to these topics, all FGDs discussed education received during visits to the households and beneficiaries reported receiving specific education on the following topics during household visits: smoking while pregnant, nutrition, treatment for a child with fever, malaria prevention practices, prenatal care, latrine building and stopping open defecation, hand washing practices (using a tippy tap, when to wash hands), water purification methods (sun method, Sur’Eau), condom use, and diarrhea treatment. The frequency of visits from CHWs appears to vary across the districts, communes, and fokontany and mentions of visits from CHWs were not consistent across FGD. Some participants said the CHW never visited their house or had not come by in years, whereas others said that CHWs came monthly or even weekly. Two participants said the following regarding what they had learned from sensitization activities:  “They sensitize us on building latrines…. Water shouldn’t be stored in the open but should be covered to prevent children from having diarrhea…for my case, since they had said it, diarrhea stopped”  “What I think of the Sur'Eau? It’s really useful, as we never knew nothing about it before! We had diarrhea before they were making sensitization about Sur'Eau.” While the education was generally believed to be helpful, some acknowledged that they did not always put what they learned into practice. Participants in all four BENW FGD mentioned the education they received regarding hand washing techniques and water purification, but only participants from two FGDs said they consistently used these techniques and believed others in their community do so as well, and participants in a third group said that these behaviors were practiced inconsistently. Participants in the final group had only received training recently, and had not yet had time to adopt their new behaviors routinely. Similarly, two FGDs with BENM reported that they received education on condom use but said that they couldn’t use them because their partners do not want to. As expressed by one participant, “We want to do it but men don’t.” Health seeking care Participants in all of the BENM FGDs mentioned seeking care from the CHWs for their children and three FGDs mentioned seeking care for themselves. The discussions on where they go for their health care were somewhat contradictory. On the one hand, in every FGD, at least one participant described a time when she sought care from a CHW. These participants said they sought care from CHWs when the child had general symptoms of illness such as a fever, swelling, diarrhea, stomach aches, malaria, coughing, and trouble breathing. Adults sought care from CHWs mainly for family planning. The reasons they sought care from the CHW first over the CSB were proximity and ease of access. If the CHW could not treat the child, they then made the trip to the CSB. However, many participants in each FGD said they either go directly to the hospital or to the CSB instead. CHWs were perceived by most BENM participants as “sensitizers”, whereas medical care and treatment should be taken care of at the CSB or hospital instead. In addition community members also go to the CHWs to purchase medicine. Three BENM FGDs discussed purchasing medicine from the CHW, and five of six FGDs said the medicines were reasonably 9 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program priced compared to other places where they can purchase medication. When asked what she thought about the price of the medicine sole by the CHW, one participant replied “It’s cheap! 500 ar is enough to save our children’s’ life! We’re happy!” Stock outs, however, were mentioned as an issue in all six FGDs with mothers. As expressed by another participant, “There should always be medicines available! Sometimes there are no medicines and we have trouble especially at night… We have to go to Diego when there are no medicines…We should only go to Diego for serious illness!” In the FGDs with the beneficiaries of emergency transport (BENTU) the rider respondents explained that when the emergency transportation services began, their initial understanding was that they were strictly for pregnant women and children under 5, not realizing that any member of the community was eligible, while prioritizing women and children under 5. This misperception was corrected during subsequent follow-up and supervision and people realized that the services were available to anyone within the community. During the BENTU interviews community members expressed their positive thoughts about the emergency transportation services. One interviewee said she felt more at ease in a rickshaw than a car and was happy it was cheaper than a car. She also stated it was beneficial to use the rickshaw because it would depart as soon as the patient needed to leave, while other forms of transportation wait for other passengers. The knowledge about the emergency services was perceived to have become widespread throughout the community. Community members were aware of the availability of emergency transport, respondents explained. A total of seven individuals were transported within a three month time period. The interviewees expressed gratitude and appreciation for the emergency services:  “We appreciate it because it helps us. If any problem occurs at night, as my fellow participant here said, people may get sick at night, this equipment can transport the person since we don't have any car here. So we count on this equipment to transport the patient to the doctor…We appreciate it and are thankful for it.”  “We feel relieved and thankful for the presence of this equipment in our village, because without it, it would have been difficult for us to bring a woman in labor to the independent doctor. It also helps us to inform the women here if something happens during the night.” The BENTU respondents also shared challenges related to emergency transport. These challenges include:  Patient issues. The BENTU respondents explained they are not sure how to handle the hypothetical situation of a patient dying during transport, given that the driver is not medically trained. The interviewees shared their concern related to speed of the bike ambulance and the possibility that women might give birth while traveling. In addition, the rickshaw can only carry one passenger which some perceived to be a problem. Finally, community members assume that the emergency bike riders will provide first aid but they do not have that capability.  Driver challenges. Drivers explained that they are volunteers and are unpaid for their work. They expressed how exhausting it is to transport patients and described their job fatigue and lack of pay.  Safety concerns. People are afraid of falling from the bike. (At the time this evaluation was underway, MAHEFA made changes to the bike design to improve stability and address this concern.)  Equipment and road conditions. The bikes have no lights on them and the drivers do not have raincoats. The drivers said they have to take care of the bikes themselves and do not receive money for maintaining the equipment. In one community, the roads are sandy and bikes have 10 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program trouble working in those conditions. (MAHEFA reports that riders and supervisors are trained to carry out routine maintenance, but they are not expected to pay for maintenance; there is a management system in place to do so.)  Security concerns. Because of security issues at night, a patient must be accompanied by someone else and cannot travel alone. The interviewees explained that there are thieves who may attack the bikes. It should be noted that the TU drivers receive training on how to transport a pregnant woman, for example, avoiding positions that might be dangerous for the mother or the baby. MAHEFA is considering whether to offer Red Cross training in first aid to the drivers, bearing in mind the need to make sure that the drivers are not perceived as doctors who are able to take care of the patients “en route”. The MAHEFA program also acknowledges that the rickshaw and bicycle-ambulance have their limitations, two of them being the fact that they can carry only one person and that it doesn’t go as fast as a motorized vehicle. However, these are also practical solutions in many low-resource settings where vehicle ownership and maintenance is impractical and demand for emergency transport comparatively low. Some of these challenges are described by participants as follows:  “The rickshaw is helpful but the problem is it doesn’t run fast. For example it’s terrible for a woman in labor who is going to deliver within 15 minutes and yet they are still at the crossroads, at 5 km from Bemanonga. So, if they travel by rickshaw it won’t be possible. She will give birth before they reach the bridge.”  “I am a driver, not a doctor! What can I do if the woman gives birth during the travel? It’s really a problem.”  “I encourage people to volunteer to ride this bike, but I know that as human beings it’s not obvious to wake up at night…so when this occurs, some actually are quite reluctant. But since they had already committed themselves to do it, they went for the sake of the patient…We don’t get paid indeed, but since the bike was a donation offered to us, we can’t just let it down, we need to make efforts. The riders live really far from each other. So it’s the ones that live nearby that are often called upon and they are starting to be fed up. Their sleep is interrupted…” Before introducing the emergency transport interventions, Transaid did a thorough transport and logistics needs assessment to understand the context in MAHEFA regions; subsequently an analysis of costs and options was conducted to choose the best solutions for community-managed emergency transport with a focus on sustainability. Lower cost and lower maintenance solutions, which include bicycle-ambulances, stretchers and non￾motorized canoes, were chosen, considering adaptation to local geographic context, cost of production and ability to maintain at local level. Experience has shown that motorized emergency transport, such as motorcycle ambulances (which could be appropriate for some of the areas where MAHEFA works and where fuel is available) is more expensive to run and harder/more expensive to maintain. Such type of motorized transport would be best to place at CSB level, to benefit from government support; however due to restrictions to work with the government at time of implementation, this option was not feasible. Options such as investing in a classical ambulance fleet need to be explored in full collaboration with the government (therefore not possible) and also they do not address the issue of having a mode of transport readily available at the community level in areas where distances are long and communication is a challenge. 11 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Impact on community/ health behavior There was a widespread belief expressed in the beneficiary FGDs that community members had made changes as a result of CHW sensitization activities and that these resulted in positive impacts on the community. Every FGD mentioned at least one perceived improvement in health either for an individual or in the community as a result of the CHW work. These improvements included:  Increased use of mosquito nets in the community  Lower rates of diarrhea  Improved hand washing practices (including hand washing before meals, after using the bathroom, etc.)  Increased use of family planning/improved birth spacing  Lower childhood mortality from malaria and diarrhea  Increase use of water purification methods (water boiling, sun method)  Increase in childhood vaccination  Breastfeeding for infants<6 months and following nutrition guidelines after 6 months  Increased building and use of latrines Beneficiaries of WASH activities in three FGDs specifically talked about perceived improvements in their communities including the reduction of open defecation, greater use of Sur’Eau water purification solution, lower incidence of diarrhea, and improved hand washing practices. The other FGD had just received WASH education and believed they could not speak as to whether or not changes had occurred. The education led to other activities and one FGD mentioned that they had established a village association to manage the use of water and infrastructure maintenance, one other group was in the process of forming one. Two villages had established mechanisms for co-payments to use water and two had established a system for well maintenance. One beneficiary described the benefits of WASH as follows: “Since they sensitize the population the percentage of feverish and diarrheic people decreased...Before it was 80% people who caught diarrhea. But now it is about 30%...” In the words of another, “It [cases of diarrhea] is less than in the past. It’s not the same as before. Since the well was rehabilitated … people’s health is getting better.” Respondents, however, mentioned a number of environmental and structural challenges in implementing WASH in their community. Only one of the four FGD mentioned having at least one well in their commune that functions year-round; this was a pre-existing well that MAHEFA rehabilitated. One other FGD said their well was completed but was not open for people to use until it was approved; others in the commune do not function during the dry season. Barriers to building or repairing wells varied but were most often the result of not having needed resources, delays due to regulations or other bureaucratic issues, or difficulty with getting contractors to complete the work. In addition, several participants in FGDs mentioned that pumps or pipes were broken, preventing community members from accessing a functioning well. All four FGDs mentioned trouble with maintaining enough water for people to use during the dry season. There problems were summed up as follows: “I think it will never be sufficient. We already send a request to JIRAMA about this situation. We’d like to ask for another well, to add up with the 2 existing wells…If we content ourselves with this only well, the kids are suffering mainly during dry season. We would appreciate if you could give us a public standpipe.” With regards to toilets and open defecation, all four FGD mentioned having been sensitized by the CHW to avoid open defecation and use latrines, but only two FGD said that the number of latrines in the community had increased. Perceptions and social norms concerning locating latrines in and the home seemed to be the biggest barrier to increasing their number and use. Some mentioned wanting to build a latrine but were not able to because of finances or lack of sufficient privacy. 12 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program  “It is difficult, we even destroy ours because of our neighbor’s kitchen, his house was near. So we cannot have peace in mind thinking about that. That is the main reason why people do not build toilet...It does not mean that they are not motivated...”  “I want to add something about the toilet, we are motivated to build toilet but we have financial problem. If we build it ourselves it will not conform to the standards and may cause epidemics.” Summary Virtually all participants in FGD with beneficiaries (mothers of children under 5 and WASH) accurately cited the range of topics CHWs are trained to promote in their communities. Nevertheless, not all participants reported putting what they learned into practice, indicating a need for continuous reinforcement of healthy behaviors. In every FGD, at least one participant indicated they had sought care for a child with general symptoms of illness. Adults generally sought care for family planning. Proximity and ease of access were cited as the main reasons for seeking care from a CHW. However, many participants reported going directly to a health center (CSB) or hospital, rather than visiting the CHW, viewing the CHW more as a “sensitizer” than a provider. This may have been in part a reflection of proximity to CSB in some of the locations visited, and also reflecting a program strategy, that CHW residing within 5 kilometers of a CSB should devote most of their effort to community sensitization. Respondents participating in FGD about emergency transport were generally positive about the service, but expressed reservation about safety, speed, and the medical or first aid skills of the riders. In every FGD with beneficiaries, at least one benefit to individuals or the community was mentioned, indicating a sustained appreciation overall of the work done by CHWs. The most significant challenge to this is seen in changing WASH behaviors, in particular in building latrines, promoting consistent hand washing practices, and eliminating open defecation. 2. Community Health Workers (CHWs) This section explores CHWs observations on the training they received, their working relationships with CSBs and TAs, stock management, the impact they believe they have had on their communities, logistical and administrative challenges, and job satisfaction. Overall CHWs reported being satisfied with the work they are doing for MAHEFA, feel they were sufficiently trained, and believe their work is benefitting communities and in some cases themselves and their families. Nonetheless, there are also difficulties they have encountered, some which have been resolved and some which still exist. Training In general, most CHWs in FGDs reported that the trainings went very well with little need for improvement. Most CHWs found the training to be straightforward and clear. They liked how the trainers used the boards and provided them with manuals to reference when they return to their communities. One person found the family planning to be “the most clear”. A few noted that the mock sensitizations during training helped make things easier to understand. They felt that the trainers were able to answer all their questions clearly and concretely. A few expressed a need for additional training such as refresher trainings. In addition, some CHWs wanted to have training on injecting Depo Provera in the thigh rather than the arm. A few mentioned wanting training on how to do blood pressure checks. The following are some thoughts CHWs had about the training they received:  “As for me, I think their methods are OK. We could understand the teaching very well and we can, in turn, we can teach people in our village. We really understand the message and everything is clear for us.” 13 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program  “Ok, as for me I was satisfied with the training on sensitization because they ask questions and I can tell my opinions. So, if there is something wrong, they can correct me. They give concrete examples… It was easy for me to understand thanks to their techniques. Sensitization seems to be easier. They always try to find ways so that we can grasp it.”  “It is difficult to say that this is sufficient but it needs to be improved. That means, we need to make some review frequently since it is new and it is required to implement the activities. So if I say it is sufficient, it will prevent from new thing to happen. It is not my intention, it’d better to improve and strengthen what has been done so far.” A few CHWs in different FGDs reported some difficulties with preparing reports due to a lack of training. In one FGD, a few said they initially had difficulty in preparing the reports but then said they were later taught by the TAs how to do them and now they are able to do them correctly. In another FGD, some reported that the CHWs had trouble filling in their registers and they all filled them out in different ways due to the lack of standards or training on how to complete them. There was also a feeling that there are too many books to fill in at one time. Some of these difficulties were expressed by CHWs as follows:  “As for me I find that it is difficult to use them since we don't master them yet mainly with reports. But as our TA did his best on explaining it to us, so it is getting clearer now.”  “Besides, we haven’t received training about this notebook for children. It was just distributed like that. And now we have trouble when we have to weigh the kids; where should we put the weight: in the yellow and red part or where… we didn’t learn about it.” Relationships with TAs and CSBs CHWs are supervised by TAs employed through the MAHEFA program NGO grantees and are also under the jurisdiction of the Directors of the CSBs. In general, for issues beyond their capacity the CHWs go to or refer patients to the CSB. For issues related to filling out the monthly reports, the majority stated that they turn to the TA for help. For the most part, CHWs indicated that they have positive relationships with the TAs and the local CSBs with which they work but two FGDs reported challenges in working with CSBs. Most CHWs report that the TAs visit them once a month. During these visits they speak about their activities and look over the registers. The meetings provide the CHWs with a chance to ask questions and obtain explanations for things that are not clear to them. The TAs also explain how to fill out the registers and check their stock and other materials. A few mentioned that they feel that the application of the training becomes clearer when the TAs visit. In general, the CHWs find the TAs to be friendly. Reports on relationships with CSBs were more mixed. Overall, in four FGDs, the CHWs reported a good relationship with the CSB. They report a good collaboration, and in the words of one CHW, “they are eager to help when we come and ask advices…” In contrast, in two FGDs there were discussions about poor relationships with CSBs. They thought that the CSBs did not treat their referred clients well and that they were very strict with the CHWs. Several CHWs in one FGD specifically mentioned that they felt that the CSBs were not always helpful or available. Stock management CHWs have a variety of adult and child medications for sale to treat symptoms such as fever, cough, and diarrhea. They also sell family planning products including condoms, oral contraceptives (e.g. PilPlan), injectable contraceptives (e.g. Depo Provera) and Cycle Beads. Once a CHW is trained and certified, 14 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program s/he is given a start-up kit stocked with socially marketed products. As these are sold, the profit is used to re-stock the kit, with a small amount remaining for the CHW. Highest demand appears to be for medicines such as Paracetamol and Pneumostop®, and for family planning methods such and Depo or PilPlan, A few CHW mentioned that condoms were not in demand—one noted, “They don’t like it.” Several CHW noted that community members find the prices of the medications sold by CHWs to be inexpensive and the same medicines are more costly at groceries, pharmacies, and doctors’ offices. However, one CHW noted a misconception that some community members think the CHWs get the medication from the government for free and therefore expect the medication to be given to them for free. The CHW stock comes from provision points (PA) stocked by PSI (discussed further below). The CHWs discussed how they manage their supplies and record their stocks and sales, and place orders when they are running out. The challenge mentioned by most CHWs was the problem of stock outs at their level, though some also discussed stock outs at the level of the PA. In over half of the CHW FGDs, they reported that they replenish medications at the pharmacy when they run out, while others noted that they refer patients to the CSB, hospital or health center. One problem with buying medicines from the pharmacy is that the CHW then has to sell it at the pharmacy price.  “Moreover, when we face shortage and as we go to the PA and find no medicine. We don’t resign. Sometimes, we go to pharmacy to buy some since we don’t want the patient to suffer until the PSI comes over there. That’s how I proceed to solve my problems.”  “Concerning the medicines, about the stock at our PA. For a month, there is no medicine within the PA since all the CHW use it all over the district. So the supply shortage is from the PA center. We wait for a week and it has not arrived yet. We are from the countryside and that’s the main difficulty.”  “In general, we’d like to maintain the collaboration with MAHEFA. But the problem is about medicines... we can’t foresee when there is an outage and yet we don’t any transportation means to search rapidly for medicines. We must walk for kilometers, at least for 10km.” Community impact In all but one FGD, CHW participants commented that they had noticed positive health outcomes due to the sensitizations they conduct in the community. The outcomes they observed include fewer child illnesses (diarrhea, malaria, fever, respiratory, malnutrition), more uptake in family planning methods (pilplan and injectables), fewer unwanted pregnancy, fewer maternal and child deaths, more vaccinations, more people using Sur’Eau and practicing good hygiene, clean water, use of the wells built by MAHEFA, and increased use of antenatal care. Many are noticing changes in health seeking behaviors with more community members coming to them for advice and medication rather than traveling far to the CSB. One reported that community members sometimes do not even go to the doctor to get treated because of distance but now they are getting treatment because they can easily visit the CHW. They are reporting that the community members are thankful for the CHWs activities because now they do not need to walk many kilometers to see a doctor, they can go directly to the CHW. CHWs mentioned that they felt that messages about maternal and child health and family planning were the easiest to deliver. One difficulty they faced, however, was that women say they want to receive the injectable shot in the thighs, but the CHWs are only trained to administer it in the arms. The women end up going to the CSB because they want the shot in the thighs. Women also want their blood pressure and weight to be checked while visiting the CHWs but they do not provide those services. 15 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program The topics that were reported to be the most difficult to sensitize on were hand washing, latrines, Sur’Eau, and clean water storage, though a few mentioned difficulties sensitizing on malaria, malnutrition, diarrhea, and condoms and pearl. Many believed they had the least success in promoting latrine construction and use. While some CHWs report that people are starting to use them, it is, however, a slow process. Some explained that the reason progress is slow is because it is “difficult to change old habits.” One CHW described success after fines were implemented for not using a latrine. Difficulties in promoting latrine use included:  People don’t want to smell excrement  Lack of space due to close neighbors or constraints to choosing a location  Some cannot afford a toilet  Elders have a very difficult time accepting defecating in the house  Certain social groups have taboos (Royals, people from the south)  It offends ancestors They also believed it was difficult to convince people that latrine use helps to prevent disease because some of the CHWs, while aware of microbial transmission by flies from feces to food, did not feel comfortable in describing disease transmission and illnesses caused by open defecation. There were further difficulties in promoting the “Tippy Tap” for hand washing using a 1.5 liter water bottle. Community members evidently resist constructing Tippy Taps for a variety of reasons: they cannot afford to buy the bottle, bottles are more valuable for storing milk or oil, children take the tippy tap to play with, they may be stolen, and the plastic bottle can be broken easily. Finally, some CHWs in one FGD noted that adults were asking them why they were only providing care to children and not to them also. Some also noted a problem in referrals since mothers do not always follow through on the referrals from the CHW to the CSB for their children. The following quotes illustrate the thoughts of two CHWs on the impact they believe their work as had on their communities.  “The CHWs activities are successful in a way that they facilitate the doctor’s job. When there were no CHW yet, the doctors worked until 8 PM. Their tasks are reduced since the existence of the CHW. Besides, the people do not have to walk for kilometers because they go directly to the CHW. It is easier for them to bring their children at the site.”  “It concerns family planning, many women are doing it now. No maternal mortality anymore. It is the same with vaccination, they are almost vaccinated. Mortality of children under 5 caused by malaria, diarrhea and respiratory infections is slowing down. There is no more incidence about it. As for malnutrition, it begins to disappear. Before we counted 4 children with malnutrition, now there is one left. No new case has been noticed and those having issues become to be normal.” Logistical and administrative challenges Three challenges mentioned by CHWs related to transport, safety and compensation. Some CHWs live far and need a method of transportation, such as a bicycle, to attend trainings, pick up medications, deliver their reports, and carry materials. In one FGD, a few complained that they thought they were going to receive bicycles but they had not yet arrived. There were also comments that although some had received a bicycle and had been trained to do maintenance on them, it was difficult to find the spare parts that were needed for some of the repairs. Without additional probing, there is no way to know if these CHWs live far from market centers where spare parts are widely available, but many participants traveled as much as 30 kilometers to participate in the FGD, and other CHWs reside farther still from large towns. 16 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Some also expressed concerns about their safety and said they were afraid to travel with their materials in the countryside. They were worried that their materials would be stolen and that they would be robbed because people would think that they have a lot of money. Finally, there were a few complaints about finances, specifically, that the payment and per diem given is too low. Two people complained that the fees given to the CHW are not equal, with one arguing that CHWs receive less than what others receive to attend trainings. Work satisfaction Most CHWs reported being satisfied with the collaboration with MAHEFA because they were provided with materials and knowledge. Several reported being satisfied because now they can care for their community and family locally. One reported they appreciated that MAHEFA linked them with other CHWs and they can now exchange ideas. They also appreciated the monitoring of their work. One person reported that MAHEFA made changes to the register and report to make them simpler and easier to fill out. They would like to maintain collaboration with MAHEFA. In the words of a few of the CHWs:  “We’re satisfied with MAHEFA because we received knowledge, materials; we can care for our family. Our families don’t need to go to Sakaramy anymore but they can receive treatment locally…I am satisfied!  “Yes, I agree with them. I am satisfied because I learned good things. I could care for the community and for my family…There are real improvements since the CHW have started their work… Everything is ok! “  “Also, the monitoring they are doing in our work also worked! It works because they really take a deep look at our work, plus they gathered us every month. They also look at things that didn’t work and need to be changed. These are the things that worked in our cooperation with them so far.” Some CHWs reported that their work has also had personal benefits. A few reported being better able to handle illnesses that arise in their own families because they have the medications to treat and care for them. Two CHWs reported that they can now provide family planning to their wives and don't have to travel long distances to the CSB. A few CHWs built their own latrines. Summary CHWs were appreciative of their training, with several across different FGD commenting that they were challenged in completing registers and preparing reports. There is a significant burden in that the program utilizes different registers for each core area of focus, consistent with Government of Madagascar guidelines. CHW receive direct support and supervision from TAs associated with MAHEFA NGO grantees, and are also meant to be supervised by their local CSB. Due to resource constraints, the latter happens far less frequently, and often only is manifested in the monthly reporting meeting held at the CSB. Overall CHWs found the TAs to be friendly and helpful during their regular supervisory visits. Relations with the public sector CSB were less favorable; participants in two of six FGD believed the CSB were not cordial in receiving CHW-referred clients, and that they were strict and not helpful. In more than half of the CHW FGD, participants mentioned problems with stock outs of their supplies, and noted that they needed to replenish at pharmacies, causing them to increase their prices. This situation was not explored in depth in the FGD, but it raises two concerns, that either there are limitations to the current supply system, or that CHW may be colluding with pharmacy owners to purchase stock and sell at higher prices. 17 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program In all but one FGD, CHWs noted positive health outcomes as a result of the sensitizations they conducted in their communities. These included fewer childhood illness, uptake of family planning, more vaccinations, and more people practicing good hygiene. Messages regarding maternal and child health and family planning resonated more strongly with the community, while WASH topics were more difficult to convey in a manner that effected change in the population. Most CHWs reported professional and personal satisfaction with their engagement with MAHEFA. 3. Community leaders FGDs with members of the community health committee (CoSan) and staff working with the mutual aid societies (mutuelles de santé), and IDIs with commune mayors and directors of CSBs support the impressions of the beneficiaries and the CHWs that MAHEFA has contributed to improvements in their communities yet also highlight challenges in program implementation. The comments by CoSan and Mayors focused more on changes and challenges at the community level, while the CSB directors talked more about MAHEFA’s influence on health care seeking behaviors. CoSan CoSan respondents reported increases in health care seeking behaviors and changes in community perceptions regarding health and sanitation. Participants shared that people have more trust in the CSB personnel since the start of the MAHEFA program and had begun visiting the CSBs more since the CHWs disseminated health care information. Furthermore, parents who previously avoided seeking medical treatment for their children have changed their perspectives and now bring their children to the CHW for care. One CoSan participant shared how having a CHW as an additional medical provider positively impacted how children received medical treatment:  “As for children... We have just one doctor and when there are times he has to go to a meeting, he refused to receive some patients. But since the collaboration with CHW, even feverish children can be treated here. That is the advantage!” Many CoSan participants also stated that people are less fearful and more cooperative with obtaining children’s vaccinations. As voiced by two participants, “You know, in the past people were afraid to get vaccinated because they thought it made them sick… Now we see there are changes.”, and, “In our fokontany, children used to run away in the forest during vaccination period. But now people are motivated to get their children vaccinated…” Maternal health was another topic discussed by CoSan focus group members. Some shared that family planning methods were being used. A few said that they perceived that more women were getting prenatal care and that women were more likely to give birth in the hospitals and less likely to do so in the villages. One member said that MAHEFA ambulance bicycles helped pregnant women in need of care at night. Some respondents said that they thought there was a lower birth rate. A number of CoSan FGD participants reported that they perceived decreases in illness, specifically, in children’s cough, diarrhea, fever, malaria, and malnutrition cases. Several also perceived reductions in children’s deaths due to diarrhea and fever. Several respondents shared their thoughts about the construction of latrines. Many reported decreased open defecation as a result of the MAHEFA program. One of the positive impacts of the program was in change in perceptions as expressed by this participant: 18 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program  “Sensitizing is the most noticeable…It’s good to see that people’s way of thinking changed… I remember very well that in 2012 when I first came here, there are a lot of people who did open defecation in the forest… but now thanks to the sensitization made by the CHW, a lot of people have their own toilets… That’s the change in their mentality.” CoSan members also described a variety of challenges related to the CHWs, community members, the community itself, emergency transport, and others.  CHWs. Some CHWs resigned leading to a gap in services for that area. Some CHWs were not validated. Additionally, the CHWs had competing priorities between seeing patients and attending to their work in the fields.  Community Members. Some CoSan members explained that community members were under the false impression that they would get a pump since another community received one from MAHEFA. One reported it was hard to convince illiterate people to build a toilet. Patients were also confused about the role of CHWs and the role of CoSan, and some patients chose traditional healing instead. Some people responded in a negative way to the outreach efforts. When taught information, people requested items like corn powder or soap to be given to them during the sensitization. On another note, they were reportedly using Sur’Eau for laundry, one participant reported.  Infrastructure. CoSan members said there were difficulties with finding space to build a health site and several of them said their site had not yet been built. Some respondents described issues with toilet construction. People were prevented from building a toilet by their neighbors because of complaints from them. As one noted, “And they even prevent their neighbor from building toilet near to their house. So it creates problem within the society.” Clean drinking water was reported as an ongoing issue as was availability of public fountains.  Emergency Transportation Challenges. One respondent said they had difficulty getting transport at night and was of the opinion that patients go by car. Another respondent said people chose to go by bus because it’s faster. Yet another CoSan member said women had been giving birth at home because they lacked the knowledge that the emergency transport was available at night. Finally, one mentioned that there was a belief among the people that a stretcher was only for dead people, so they did not want to use it.  Other. Lack of equipment and lack of medications were mentioned by some of the CoSan respondents. Mutual Aid Societies (Mutuelles de santé) MAHEFA has experimented with introducing health Mutual Aid Societies (“Mutuelles de santé”) in several of the regions in which it works. Mutuelles de santé function as a community-supported, pre￾paid insurance fund, in which members deposit a fixed amount each month, and from which they can withdraw payments for medical care and procedures. Members have to pay fees for 3 to 6 months before they can benefit from the program (in Sofia and Menabe, respectively). In one district in Menabe, Bemanonga, approximately one-third of the monthly fee is applied to a transportation maintenance cost which allows members access to transportation. Certain illnesses are covered like malaria and fever. Blood tests, ultrasounds, and surgeries are part of the covered services. Payments are made directly by the management of the mutuelles de santé and the local hospital. Individuals with other illnesses such as cancer, diabetes, or leprosy are not accepted into the association. The mutuelle de santé in Sofia provides an illustration of how the system works.  We only work with the State hospitals, not in private ones. We don’t give money directly to the member as each doctor has his own account there. Doctors also have a partnership with the pharmacy. State hospitals have pharmacies and we also have our villagers’ pharmacy here; the doctors have an account there and money is deposited only at a village where the doctor has an account. Money from the 19 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program mutuelle is transferred to the doctor’s account; with the medicine’s price. For example, if we give a patient ten thousand ariary, he or she may not spend all this amount, only the medicines price is paid. For example, if the cost of the treatment for simple malaria is about two thousand ariary, only two thousand ariary is drawn out. The communities in which the mutuelles de santé function are not well-to-do, and some of the mutuelles de santé report having difficulty recruiting members. For example, during a focus group with leaders of a mutuelles de santé, one participant noted that there are about twenty thousand people in its catchment area, yet they had signed up only about 60 members. Nevertheless, the leaders believe they are doing well in informing people about the benefits of the mutuelles de santé and that between the CHWs and the mutuelles de santé more people can get care in their community and have increased access to the hospital through less costly transport. As one interviewee noted:  We, who are designed to be the leaders of this health mutuelle, have already informed people and educated them about it. We’ve noticed that they really know about it and that they recognize our last awareness because they know now that this mutuelle is not only meant for the wealthiest. Even poor people can benefit from it.” One director of a mutuelle de santé who was interviewed mentioned that he believed that the CoSan did not fulfill their responsibilities to inform people about the mutuelle se santé, and were not concerned about it. He implied that the CHWs were also responsible for the lack of awareness about mutuelles de santé in the community. It was also noted that some people are discouraged because they have paid a year of contributions but had not needed medical care (a good thing!), and hence they had not yet benefited from their contribution. Mayors The commune mayors were overall very positive about the MAHEFA program and noted many beneficial contributions of the program to their communities. Specifically, three of the four mayors mentioned that MAHEFA facilitated collaboration among different stakeholders to improve health and they recognized MAHEFA for providing sensitization, and implementation of the entire project. All four acknowledged that MAHEFA has provided training and they noted that their communities are appreciative of MAHEFA’s efforts. In three IDIs, mayors noted that people come to their office to thank them for improved health. One mentioned his communities’ ability to work together and the collaboration with MAHEFA as the reason for winning the Healthy Commune award. Other positive outcomes include:  Community members seek out the CHW for care so they don’t have to go far away  Medicine is affordable  Fewer people rely on traditional medicine  Increased birth reports from midwives  Better data collection of child births and reduced childhood mortality  Increased number of people with latrines in their homes and decreased open defecation  Increased use of tippy tap and proper hand washing technique  Increased number of wells among fokontany  Decreased morbidity overall (particularly from malaria) The challenges or suggested improvements noted by the mayors were all specific to their community without much overlap between participants. Three out of four mayors said they thought community members were changing their health behaviors based on CHW sensitization but in two IDIs they said community members were slow to make changes, and that fewer toilets were built than they would have liked as a result. Other challenges included bicycles breaking down, the political crisis, an 20 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program insufficient number of doctors for the community, and CHWs arriving late or not showing up to public meetings they have called. Suggested improvements from the mayors were: give each CHW their own equipment so that they do not have to share; increase the types and number of training given to CHWs; improve supervision of CHWs; and increase the number of wells per fokontany. The mayors have the following things to say about their communities:  “There are many changes. Our Commune is becoming clean. We understand what cleanliness is. Regarding to health, child diseases have been reduced and children death rate has really decreased, there is no more child who dies from disease. There was a time when children suffer from strong diarrhea. The minute they feel pain, they died. Fortunately, it does not exist anymore. Before, children died from strong diarrhea. Now, they don’t. We can notice real changes since MAHEFA has worked in our Commune. We really do as they tell us to.”  “We get gradual results because people don’t have enough place where to build it. That’s their problem. But there is a group of people in the surrounding here who decided to build a toilet for them all. It’s so nice to see such solidarity!”  “Changes aren’t the same in each fokontany. Some are rapidly changing whereas others are slower, especially in the remote areas. Nevertheless people are eager to join Health site.”  “Development and improvement are extremely important for us. In my opinion, what is lacking … often … there is not enough supervision. It may not be enough, but we think it’s better if we go to the sites together…because we’re not meeting so often...it's only them (MAHEFA) who go there alone.”  “It’s a kind of medical secret …they say they cannot keep excrement inside a house. Some people still maintain this belief! They prefer to do it far away in the forest but not in their own house…Another reason is they don’t have the means…They cannot afford building toilets… Life is hard; people don’t even have the means to build a hen house! They aren’t going to waste money for toilets.” CSB Directors The CSB directors reported changes in hand washing, latrine use, decreases in children under 5 with fever, diarrhea, and pneumonia, an increase in the number of births at the CSB, and an increase in number of vaccinations and family planning use as a result of MAHEFA and the CHW sensitization work. However, some acknowledge that is it difficult to change peoples’ behaviors or mindsets since people are reluctant to change their habits. They also noted changes in health seeking behavior. Before MAHEFA, caregivers of children under 5 would go to traditional healers for care but due to the sensitization now they either go directly to the CHW or to the hospital. However, two CSB directors said that it is still a challenge to change people’s behavior from seeking care from a traditional healer to a CSB. Two expressed the thought that behavior change in seeking care at the hospital is going slowly. Four noted that they had experienced a reduction in the number of patients at the CSB as people were now seeking care with the CHW instead, although generally speaking the CSBs felt more people were seeking out health care. As one CSB noted:  “That’s why I said that they are more approachable than…you see, they are close to the population, so it’s easier for people to join them than coming to the hospital. That’s the reality. At the CSB, it has decreased but in CHWs’, it is really increasing, New users and regular users.” Some issues that were expressed may reflect structural challenges the MAHEFA program faced in adhering to U.S. Government restrictions on working with the public sector during the recent political crisis in Madagascar. For instance, some CSB directors mentioned that they were not able to benefit from some of the training that the CHWs received and as a result the CHW know more about certain topics, such as cycle beads as a method of family planning. One CSB expressed concerns that the CHW 21 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program and TAs do not have adequate supervision from the CSB directors and NGOs but that they did not have the means to make field supervision visits. Four CSB expressed concerns about CHW reports (some are late or not filled in correctly), though several mentioned they have worked with CHWs and TA to resolve these issues. Summary FGD and IDI were conducted with several cadres of community leaders: members of Community Health Committees, organizers of mutuelles de santé (health mutual aid societies), mayors of communes, and directors of CSB. Members of the CoSan reported that the efforts of the CHW have resulted in an increase in health care seeking in their communities, with people making more visits both to the CSB and to CHWs. CoSan members reported increases in vaccinations, family planning use, prenatal care, and institutional deliveries. Within the FGD with CoSan members, challenges were mentioned such as the competing priorities for CHW between minding their fields and carrying out their health tasks; surmounting the obstacles of working with illiterate populations; and finding appropriate locations to construct toilets. MAHEFA has introduced several pilot tests of mutuelles de santé, seemingly to mixed results. It appears that recruitment of members has been slow, and among those who have joined there is concern that they have paid into the system, but have not yet reaped benefits. That said, it is acknowledged that these community insurance schemes are an innovative approach that requires multiple steps to develop awareness of community health needs, establishment of management systems, and extensive community engagement in order to succeed. Mayors of communes were very positive about the impact of the MAHEFA program, and noted many positive outcomes, including the proximity of CHWs for people seeking care, affordable drugs and medications, a move away from traditional medicine, better data on health events, better sanitation, and decreased morbidity. Interviews with CSB directors revealed a broad appreciation for the sensitization work carried out by the CHWs. All reported changes in basic health and hygiene behaviors, and specifically pointed to the increased use of family planning in their communities, and higher rates of vaccination. The directors also pointed to greater use of CSB and CHW health and medical care and a move away from traditional healers, but it is not possible to verify if this perception reflects a real change. The interviews with CSB directors also underscored the challenges of working with the public MAHEFA has documented well in its annual reports, due to the recent political crisis in Madagascar: CSB directors were not permitted to participate in training, did not supervise CHW, did not validate their practical family planning training, and complained about the quality of data reported. Without deeper investigation of the entire data reporting process, it is difficult to determine whether the data quality was truly poor, and if so, what caused it: inaccurate recordkeeping by the CHWs, inadequate compilation by the TAs, or lack of supervision by the CSB. Nevertheless, the team heard frequent complaints about data quality from CSB and regional MOH staff, and observed numerous errors in CHW service registers. 4. Technical Agents (TA) The most prominent success cited in the IDIs with TAs was their ability to collaborate with MAHEFA and the other partners working together on this project. Seven of the eight TAs interviewed said they felt they were able to successfully collaborate with everyone involved. The one TA who reported difficultly in collaborating with the fokontany chief and mayor, did, however, have a positive relationship with MAHEFA. TA feelings about collaboration were expressed in the following quotes: 22 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program  “Yes! We work closely with them and have no problems. Either it is with CSB, CoSan, Chief CoSan at the city hall. The Mayor really appreciates CHWs works!”  “In general, we can say that the cooperation is good because it is not me as MAHEFA representative who work alone for the project implementation but all the community takes part in it. They even lead it and I find it wonderful. They are always available day and night. Even from A [not written out] we can call without hesitation the Mayor and the Chief of the CSB telling them that we should do such and such thing. It’s a nice cooperation.”  “Well, for instance, I collaborate with the Mayor when I need to broadcast radio spots. He owns a radio station in xxx. So whenever I broadcast spots on his radio station, I don’t have to pay like the other customers. He told me: ‘Since our partnership aims to improve the project and the welfare of the commune, you don’t have to pay for the spots and I will broadcast them for free.’ In addition to that, he is also among those who have helped set up CHW sites. He encourages the fokontany to create sites for the CHWs. Among the fokontany in this commune, only two of them don’t have any site.” All eight TAs had positive things to say about the CHWs and had noticed positive changes in health seeking behavior in the community since the start of the MAHEFA project:  Six TAs believed that CHWs were effective in communicating health information and changing health seeking behaviors in their communities. Two thought the work of the CHWs had improved since the beginning of MAHEFA. They felt the increased training on a wider variety of topics made them more motivated to sensitize people and made them more responsible in filling out their forms and paperwork.  Two TAs noted that community members mentioned that the price of medications from the CHWs was much more affordable compared to the cost at the pharmacy, making medications more accessible to the community.  Improvements in children’s health and decreasing rates of child mortality were cited by three TAs. Additionally, two noted a perceived decrease in the number of children suffering from diarrhea and malaria in their community. o “As per the diseases, I can mention diarrhea and malaria. We have noticed that these diseases have decreased because during the KMSm evaluation, the doctors who came here for the evaluation said that the rates of malaria and diarrhea have decreased. I also felt that way. When I first arrived here, a lot of people, a lot of children had malaria. But now, I see that diseases have decreased.”  Seven TAs noted an improvement in community members’ willingness to seek care. They said that prior to MAHEFA, people would not seek care until they were extremely ill due to a mistrust of doctors, inability to pay, or distance required to travel to obtain medical care. As a result of MAHEFA, members of the community readily seek out assistance from CHWs before they are very ill because of their proximity in the community and their expertise, improving the likelihood of positive health outcomes.  Half of the TAs said that they had noticed a reduction in stigma regarding family planning use among both women and men, and an increase in use of family planning methods among women. Half also noted an increase in the number of women seeking out antenatal care. o “If you come to xxx` on Wednesday, you’ll see how motivated people are; they are queuing to have Depo. They are really convinced about family planning. Even men are convinced, most of the time they used to prevent their wife from making family planning and they even say it’s God who will care for their children. From now on they urge their wife to make family planning to be able to earn their livings”  Six TAs said they have seen an improvement in WASH habits, including using Sur’Eau, boiling water, using wells as a primary water source, using tippy taps, and using proper hand washing techniques at appropriate times. Half said they have noticed a decrease in open defecation and 23 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program an increased use of or interest in building latrines near homes, reflecting the CHWs ability to effectively change social norms concerning open defecation and latrine use.  Half of the TAs said the current systems and processes worked well, including the improved reporting forms that better match the education level of CHWs, the ability of CHWs to obtain certifications and become approved to provide certain types of education and care, and completing trainings. Challenges mentioned by the TAs were similar to those mentioned by the CHWs, CoSan, CSB directors, and Mayors, and include obstacles the CHWs face, the difficulties in changing social norms, and their own trials in doing their jobs.  CHW. Almost all of the TAs said that stocks outs of medications were a challenge for CHWs. One mentioned that PSI had a lengthy stock out of Viasur, and two blamed the stock outs on the PA’s unwillingness to stock enough products for the CHWs.  “We really thank the MI for that as he was sensitizing us all about hygiene and cleaning up. We can say that people are no more drinking unsafe water but are now used to pouring Sur’Eau in water, as CHWs have convinced them to. Before, only few people did use Sur’Eau. But the trouble is that sometimes Sur’Eau is not available at the PA, they have no more stocks of medicines and products, and the CHW are complaining at me.” o TAs also mentioned that CHWs often complain about the lack of pay or low per diem rates for travel to the CSB or other locations, with three IDI participants mentioning this.  “… It is only with the CHW that we are having a small issue. They complain about the small amount of money received during the training. They get discouraged sometimes.” o Certification and validation of CHWs on certain topics was also mentioned by six TAs. Oftentimes CHWs are held up on the validation of certificates due to lack of trust by the director of the CSB. Two TAs mentioned that they believed the chief of the fokontany or the Mayor do not trust the CHWs because they believe the CHWs are earning money for their work. o All TAs mentioned difficulty in getting CHWs to complete necessary forms and in a timely manner. Reasons for this include low education level of some CHWs, the volume of forms and registers the CHWs must complete, and postponing paperwork because the CHWs are not paid to complete it. Three TAs noted that CHWs also work in the field, diverting their attention from their CHW responsibilities. Half of the TAs also mentioned needing to take time to correct numerous mistakes on CHW completed forms. The stock form was specifically mentioned by two TAs as a form that was particularly difficult. One TA explained a change in one form to make it easier to complete.  “But the new requirements are for example: how many sensitization campaigns were carried out? When? How many men and women? It is really simple. For example, about women: they should mention the number of women who came there, the number of those who are pregnant, the date of the first prenatal visit. It’s the same with family planning: the number of new users should be mentioned, also the number of regular user and the number of the loss of follow-up. And they inscribe the total number. This one is simpler for them and is adapted to their level of education compared to the former method.”  Social norms. Seven TAs said changing social norms around open defecation and building latrines near homes was a major challenge. Reasons given ranged from cultural taboos and that even the CHWs themselves choosing to openly defecate instead of setting an example for the community. 24 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Larger issues such as famine and political uprisings make changing social norms even more difficult as was mentioned by two TAs. o “As for the toilets, people are used to defecation in open space. So they replied that even their ancestors did the same, and they lived for a long time. So why to change now?”  TA responsibilities. Half of the TAs said they struggle to meet the demands of the job in the time they have available, particularly due to the necessary supervisory visits they must make on a regular basis. Following implementation of the Remedial Plan in 2013, MAHEFA worked with the NGO grantees to reduce the number of communes for which each TA is responsible from 3 to 5, to only one. While in theory this was a reasonable approach, it did not address all disparities as the number of CHW working in a commune can vary from fewer than 20 to more than 60. Further reduction is necessary for those TAs assigned to communes with a large number of fokontany/CHWs. Because fokontany can be far apart and take a significant amount of time to reach, particularly when problems arise (such as supply stock out or CHWs are not present when TAs arrive at the fokontany), they struggle to visit everyone and complete other duties. The most commonly reported consequence of this is the inability to submit paperwork on time, as this task is put aside to prioritize supervisory visits and trainings. Summary All of the TAs interviewed expressed appreciation for their collaboration with MAHEFA and community actors such as CoSan and mayors. Six of eight believed that the CHWs were effective in communicating health information and promoting behavior change. There was a perception that the training in different topics provided by MAHEFA motivated the CHW and made them feel more responsible in carrying out their tasks. According to the TA, the proximity of CHW to the community also promoted more frequent and earlier care seeking. Particular improvements were cited in WASH habits, and in the uptake of family planning. The TAs mentioned several challenges to CHW performance, consistent with those noted by other respondents. Specifically they mentioned stock out of some socially marketed products, low allowances for the CHW, delays in certifications due to resistance on the part of CSB directors, and poor performance in completing reports and forms correctly. Virtually all the TAs mentioned the deeply entrenched social norms regarding open defecation as a major hurdle confronted by the CHW in their efforts to promote better hygiene and sanitation. Half of the TAs reported that they struggled to complete their work, primarily due to large distances that some must travel in order to supervise CHW located in distant communities, and the associated problems with available transport. Occasionally they will make a long journey, only to find that the CHW is not present at the time they arrive. 5. NGO Grantees The NGO Grantee respondents reported successes regarding the timing of the implementation of the MAHEFA program, the impact of the program itself, and the technical support provided. An interviewee stated the timing and roll out of the program was well planned. Another respondent described general satisfaction with the work done by the MAHEFA program and good collaboration was noted. The interviewee believed that the health of the community had improved due to the interventions provided by the MAHEFA staff and believed that the sensitization campaigns were successful. It was reported that the MAHEFA program provided motivation for the TAs and CHWs. One NGO interviewee explained how the startup kits and equipment given to the CHWs influenced the success of the program: 25 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program  “The startup kits, it is part of the factors of this success. These CHW were given a startup kit regarding FP and malaria. There are also some equipment like bag, register, pen, and materials such as child balance…All of them are the factors of the success for the activities funded by MAHEFA and USAID.” MAHEFA provided technical and material support to field staff and two stated that all staff (including CHWs, TAs, the technical manager, and the monitoring and evaluation manager) have benefited and received the necessary training to enable them to carry out their activities. Three of the NGO respondents reported satisfaction with the technical support and three reported satisfaction with the training the CHWs and TAs received. Two interviewees said equipment had also been provided such as computers. The NGO respondents also reported they had some difficulties working with MAHEFA. One stated it was difficult to make recommendations or have discussions with MAHEFA because of their organizational procedures. There was a complaint that MAHEFA’s procedures did not take into account those of the NGOs and that MAHEFA made decisions without discussion with the NGOs. One example cited was the requirement of needing to travel at least10 kilometers in order to qualify to receive a per diem. This was perceived as too far and had an effect on the motivation of the TAs. Another NGO participant shared that there was a disconnect between the central level of the NGO and technical details in the field, making it hard to meet deadlines or requests from MAHEFA. That is, the central level is not always aware of specific details of field activities, and must obtain the information from regional offices in order to be responsive to MAHEFA requests, and it is not always possible to do this in a timely way. It was noted that MAHEFA was not very flexible with the implementation process by the NGO, which contributed to the reasons why they did not meet their goals. However, it must be reiterated that MAHEFA follows the targets and timelines established in its cooperative agreement, and tries to ensure that all NGO partners are well aware of expectations regarding their respective roles in supporting program achievements. The quote below describes the lack of discussion in relation to the distance the TAs travelled and received reimbursement:  “This is the same approach that didn’t work. When there is a decision to make, there should be a discussion and inform us that there will be a review… We can search for a common ground with NGO’s, instead of dictating that 5km becomes 10 km. The NGO’s also have their procedures, this is the MAHEFA approach which doesn’t work.” Other issues that the NGO respondents discussed were as follows:  Staffing and logistical difficulties. One respondent found it difficult to complete the financial reports required by MAHEFA and had to work closely with the TAs in order to prepare them. Another interviewee said the workload was too high. Issues with transportation were also mentioned. For example, there were problems with delivering materials to fokontany once they arrived at regional office and then need to go into the field. Once materials were at the site, the respondents explained, the CHW could not carry the large number of materials being supplied. In some cases, there were delays in collection of documents and reports due to the inaccessibility of some of the sites.  Challenges in data accuracy. Two of the NGO respondents reported challenges with data accuracy in CHW reports. To address this, they identified CHWs who have difficulty reporting and had the TAs supervise them daily in filling in the RMA forms. They also implemented monthly supervision meetings of the TAs as well as the completion of data quality assurance tests to avoid data inconsistences. One of the interviewees stated the staff were not trained in how to calculate the indicators and they struggled with that task. 26 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program  Challenges in funding. NGO respondents explained some challenges in funding. Two interviewees said delays in funding from MAHEFA created delays in funding to TAs and one further stated that the funds were not sufficient and contributed to delays in delivery of materials to CHWs. One interviewee noted that delays in funding were a result of not meeting reporting deadlines and shared feelings of frustration related to the funding process. o “And that creates problems because we cannot meet the deadline, we cannot meet the deadline set by MAHEFA. Yet, I here at the central level, I cannot do anything because sometimes MAHEFA requests: give us a list of the CHWs that have not yet done a certain task. Me, I don’t not have that information. Sometimes he sends us an email at 8 in the morning and the deadline is at 17h. There is always a hurry… We tell them to send us the request a little in advance. We have not yet asked this of MAHEFA. They are the ones that require that as long as you don’t send the reports we will not send you the funds. So it’s always us who has to respect their requirement.” Summary NGO grantee representatives expressed satisfaction with the planning, implementation, technical support and impact of the MAHEFA program. They were appreciative of training, technical support and equipment that they had received. Nevertheless, some NGOs expressed discontent with reporting processes, which were deemed to be strict and very detailed. MAHEFA, for its part, has noted that it conforms to the procedures required by both JSI and USAID, and that it has been a time-consuming process to provide the technical support needed by some of the local NGOs to reach international reporting standards. NGO grantees also reported some problems in finding appropriately qualified staff in the regions, coping with inaccurate data reported by CHWs, and adequate cash flow. 6. Regional and National Levels Interviews were conducted with a range of MAHEFA staff, partners, and stakeholders at the regional and national levels who have responsibility for or are affected by the MAHEFA project. These include Medical Inspectors as well as MAHEFA regional coordinators, Monitoring and Evaluation regional officers, and WASH coordinators. Interviews were also conducted with representatives of PSI and Marie Stopes International, Madagascar (MSIM); both organizations have Memoranda of Understanding (MOU) with MAHEFA to provide supportive roles in project implementation. These interviews provide further information on how the project operates across levels and the relationships between the various stakeholders. Also, the interviews specifically address issues of data collection and quality and product supply. In addition, courtesy visits were made with staff from the Ministry of Public Health and Ministry of Water in the three regions, and notes from those visits are included here. MEDICAL INSPECTORS (MI) The three MIs interviewed indicated that they have established collaborative relationships with the Regional Coordinators and that they contact each other when an issue arises. They reported that the majority of CHWs had validated trainings in PCIMEc and family planning but some had yet to be validated. The MIs discussed challenges at all levels of the project. At the NGO level, the MIs believed that the NGOs needed to have experience in community health as a requirement for being selected to collaborate with MAHEFA. Either they had expertise in health but specifically HIV/AIDS or NGOs were geared towards another social concern rather than integrated health. For example, FISA in Diego Suarez was a pioneer in family planning in Madagascar. When MAHEFA began, they had to integrate other components such as nutrition and neonatal health. The NGO had to have experience in all those fields, which was a huge challenge in terms of technical capacity to find resources or staff who can assume this implementation. In addition, a few respondents reported that staff turnover at the NGO level has been 27 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program a problem because of the time needed to train new TAs. They similarly noted that turnover among the CHWs was a problem and one interviewee said the training that the new CHWs received was not adequate. Several issues on the role of the CHWs and their relation to the CSBs were discussed. The MIs noted that community members found it difficult to adjust to the new role of the CHWs, and their expanded capacity. “The CHWs are members of the community and the villagers know them but it can be difficult to suddenly see them as doctors, offering FP services...” One MI said the CSB viewed the CHWs in a negative manner, “There is a tension between the public sector and the CHWs, as they are volunteers. People in the CSB see them as adversaries, in competition rather than as partners in developing health communities. They are worried to lose their clients.” In addition, there was an indication from two MIs that CSBs felt excluded from the process of training the CHWs. “There was a problem with CSB Chiefs, they felt neglected because they weren’t asked to train the CHWs [on PCIMEc and FP]. It was a really big problem that people from outside gave the theoretical training to the CHWs but the CSB Chiefs had to validate and provide practical training.” Data quality at the TA and CHW levels were also discussed by the MIs. During one interview the participant was concerned that newly hired TAs might not receive adequate training before beginning work, which would affect data quality. One MI reported that the TAs were uncomfortable analyzing data. The MIs also noted that there were differences in the skill levels of CHW in completing their reports, which would also affect data quality. Overall problems include transportation and geographic distances. Many NGOs did not have enough motorcycles or vehicles to travel to areas that needed health care services. In some cases, NGOs had cars funded by other projects but they needed to share their use across several projects. This made traveling long distances difficult for staff. Political issues related to restrictions in conducting training, validation of courses, and involvement of personnel were reported to impede the program. Lastly, the problem of stock outs of medicines was mentioned. Finally one MI interviewee described some negative feedback related to the start of the MAHEFA program: “When a public institution is working for the population and then a private organization comes from elsewhere and wants to disturb what is already established, it’s a challenge. When the hierarchy is respected together we follow steps and we have a good relationship.” DRSP Courtesy Visits Three visits were conducted with Ministry of Health Regional Directors of Public Health (DRSP) from Diana, Sofia and Menabe. They acknowledged the contributions of MAHEFA in working with the DRSP to implement the community health system, working with the CHWs in providing sensitization concerning WASH and health of children under 5, speeding up the implementation of the community health systems, and improving consultation and service delivery rates at the local CSB. For instance, in Diana, family planning use has increased to 27% since the start of MAHEFA activities, close to their objective of 33%. Challenges mentioned include lack of compensation for CHWs, difficulty changing social norms regarding use of latrines, difficulty mobilizing local authorities to build CHW sites, insufficient communication between CHWs and CSB workers, CSB reluctance at becoming involved in CHW training, MIs who feel they do not see the TAs frequently enough, CSB workers who do not know their role in the National Policy for Community Health and associated guidelines, incoherence of data, and lack of oversight of CHWs resulting in misinformation being communicated to the community. 28 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Two of the three DRSP expressed concerns about the project’s sustainability past 2016; they doubt the program will be able to continue without the funding and motivational support of MAHEFA. One DRSP suggested that the CHW should be assessed periodically to ascertain their competence, and incompetent CHW should be replaced. A similar sentiment was expressed by another DRSP, who feared that some CHW may become “out of control” after they have gained some knowledge, and thus care should be taken in setting parameters for the CHW activities. The same DRSP recommended that the TA should make courtesy visits to the DRSP to introduce themselves when they begin working. DIREAU Courtesy Visits Visits were also conducted with the Regional Directors of the Ministry of Water (DIREAU) in the same three regions: Diana, Sofia and Menabe. MAHEFA contributions and successes included constructing and repairing numerous water points, working in remote areas (which other programs are unwilling to do), supporting the formation of formal water associations in communities, being willing to replace community facilitators who did not complete their work, and supporting the dissemination of WASH activities and sensitization. One DIREAU mentioned that efforts to motivate communities to use the tippy tap hand washing system were finally successful as a result of MAHEFA support. The only challenge mentioned by a single DIREAU was that MAHEFA did not submit reports on time, limiting the Ministry of Water’s ability to do follow-up and coordinate WASH activities. Recommendations from DIREAU include requesting MAHEFA to submit reports on time, to communicate disruptions in construction, submit results of the physical, chemical, and biological tests of the water, and continue sensitization on a more frequent basis with CHWs who are from the community they are sensitizing. Regional Coordinators The MAHEFA Regional Coordinators manage the project on-the-ground. They are responsible for planning activities in the region, monitoring and supervising the activities of the NGOs, and providing coaching for NGO staff involved in the project. NGO partners meet weekly with MAHEFA to update each other on progress toward achieving goals, and to share successes and challenges. If required, MAHEFA technical staff from headquarters or the regional office provide direct support to individual NGOs. In turn, the NGOs meet monthly with their own teams to review and coordinate activities; MAHEFA staff often try to participate in these. In addition to the weekly monitoring meetings, MAHEFA and the NGO partners meet quarterly to review objectives, approaches and new strategies, and to share good practices. MAHEFA regional staff also make quarterly supervision visits, accompanying NGO staff to visit CHWs in the community. In October-November 2013, regional meetings were held with the local participating NGOs (and their national staff, if appropriate), to review the new work plan approved by USAID. One Regional Coordinator suggested that it would be helpful to continue doing this on an annual basis at the beginning of each new fiscal year, which MAHEFA intends to do. Collaboration with NGO partners has been a more demanding challenge for MAHEFA than initially anticipated. While it was anticipated that experience in community health would be a qualification for selection for the NGOs, such expertise did not always exist in the regions. As no large-scale community health programs had ever been implemented in these regions, the expertise of the local NGOs was not always an ideal fit with the aims of MAHEFA. That is, some of the NGOs had been formed to respond to HIV/AIDS activities, or environment, or family planning, but few encompassed the full range of technical expertise required by the project. It was also a challenge for the NGO to find 29 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program resources or personnel with the appropriate technical capacity to assure project implementation. As one Regional Coordinator noted:  “We have six NGO which each has its own performance, each has its own background, and therefore it is a great challenge to manage all these NGO. Second, regarding the activities on the ground also, there are territories…I would not truly say virgin, but the MAHEFA program…it is MAHEFA which began, which works and covers the entire region. And it is a grand challenge for the NGO, for us and for the NGO to cover a district, all of a district and all the fokontany, it’s very hard. And I think that monitoring these NGO, in the implementation of their activities on the ground… it is a big challenge which continues to be relevant even now.” In addition, the NGOs were expected to contribute some of their resources to the program, which was not always possible. The team did not determine whether NGOs proposed these resources in their bids to MAHEFA, or if MAHEFA did not communicate these expectations clearly at the beginning of their relationship with the NGOs, and there was an assumption that certain resources would be available. For example, many of the NGOs have vehicles or motorcycles which were anticipated to be available on occasion for MAHEFA activities. However, NGOs may have support from different donors or funding agencies, and vehicle use is prioritized for other projects. MAHEFA was able to provide motorcycles to some NGOs, but challenges remain in reaching distant communities. Security concerns were also cited by the Regional Coordinators as an obstacle to some of their work, as well as the political restrictions against working with public institutions which made it difficult to manage training and to validate practicum. Product Supply Finally, one of the regional coordinators described the product supply system to the CHWs. There are two systems of supply for the CHW, the CSB for three basic health products, and PSI for socially marketed commodities. When CHW first begin working, they receive a startup kit with stock free of charge. As they sell the products, the earnings re used to replenish stock, with a small amount of profit. During their monthly review meetings, officials from the restocking site and the CHW calculate quarterly consumption estimate re-stocking needs. Through an MOU with PSI, MAHEFA enabled expansion of PSI supply points into communes where they previously did not have a presence. Provincial supply centers support provision points (PA). In addition, there are some “PA relais” who move stock to more remote areas, particularly during the rainy season which can last three to four months. There are also safety stocks at community level and at CHW sites. In some cases there are temporary PAs established that function only during the rainy season so that they are closer to the CHW sites. Capacity to manage stock varies among the CHWs, and stock outs occasionally occur. When this happens, CHW are meant to refer clients either to the CSB, where most health products are available free of charge, or to local pharmacies, where commodities must be purchased, typically at a cost higher than that charged by the CHW. MAHEFA is in the process of training the NGO partners on stock management then in turn they will train the CHW. There is not usually a shortage of all products, but specific products like Sur’Eau®, zinc, Viasur®, and Pneumostop® have been out of supply on occasion. Generally stock outs last about 2 weeks to a month, before the CHW resupplies him or herself. In the case of Viasur®, and Pneumostop® the PSI supplier was out for three months. 30 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Monitoring and Evaluation Officers In-depth interviews were conducted with one central and one regional staff member responsible for monitoring and evaluation (RSE). They described the process for collecting data as follows: They (CHWs) had to collect data on a daily basis with reports. There are tools for daily data collection and monthly reports. A TA from the NGO is responsible for a commune, some had two or three communes at first. It was changed later to one TA per commune. The TA sends his report to the NGO then the NGO compiles a report by district and sends a monthly report at the regional level of MAHEFA. At the CHW level, he validates the report himself then the TA. At the NGO level, their technical officer is responsible. At the regional level, the regional coordinator is responsible. The role of the MAHEFA Regional RSE is to ensure the quality of the data that are conveyed to headquarters. They participate in training the CHWs, the TA, and NGO partners, including follow-up assessments and activity reports. They receive reports from each of the NGOs collaborating with MAHEFA in the region, validate the data, and compile the data into a regional report, and forward them to headquarters. The RSEs noted the following number of challenges in ensuring good quality data for the program:  Staff retention. One RSE noted that the level of performance from each NGO varies due to TA turnover. If the TAs change the new TA may not have received appropriate training and therefore may not function as well resulting in poor data quality. They need to identify which TAs are weak and/or new in order to retrain them.  Data quality. Both RSEs reported receiving poor quality data from the CHWs and mentioned that the reports vary from one CHW to another. “I really want to insist on the fact that reports are standard because CHWs are using the same tools. All the report tools are the same. They complete it in the same way as there is already a standard guide in the register. They only have to master the techniques and make some efforts.”  It can be difficult to pinpoint which CHW needs additional support to improve the data because the data are merged for all CHWs for which an individual TA is responsible. Additionally, there are times when the CHWs don’t record their activities. As one noted, “They do outreach but do not record. Sometimes they forget their register at home and continue with their activities without logging them.”  Poor relationship with CSB Directors. The Directors of the CSBs often report that the data are not useable, but the RSE are not sure if the reason they are saying this is because of the political situation and strained relations as a result of the restrictions the program faced in working with the public sector. Though the national policies call for CSB Directors to receive and validate individual CHW reports before compiling them, some Directors do not perceive this as their responsibility and prefer to receive compiled data, contributing to their perception that the reports are not useable. The RSEs acknowledged that MAHEFA is well aware of the problems with data quality, and within the past year focused on capacity building of the NGO and the TAs with regards to data quality. As a result, a monthly report verification form was created for the TAs to help ensure that the data received from the CHW is validated and of good quality. Nevertheless, there is a need to improve TA capacity to critically review the data they see in the CHW reports and registers. MAHEFA Regional WASH Coordinators An important component of the MAHEFA program is its investment in Water, Sanitation and Hygiene (WASH). The teams refer to “hard” water activities, including construction and rehabilitation of water points, wells, and latrine foundations, and “soft” activities that promote behavior change related to 31 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program water purification and safe storage practices, regular hand washing at appropriate occasions, latrine use and reduction of open defecation. According to the WASH coordinators, in many ways this has been the most challenging activity to implement. To start, activities were delayed by nearly 18 months due to administrative demands necessary to amend the Cooperative Agreement with USAID, which through an oversight did not specifically include the word “Construction”. Once approval was finally obtained to advance the project, the team needed to consult with the Ministry of Water to determine where to locate wells, according to national criteria. Subsequently, contracts were entered into with eight consulting firms to evaluate community needs and possible sites for the wells to be dug. Next, enterprises were identified to do the actual construction or repairs. Further slowing the process is the requirement that Environmental Monitoring and Mitigation Plans (EMMP) must be done during each phase of the process, i.e., by the consulting firm, enterprise, and upon transfer to the community. The EMMPs specify safety and security measures that need to be in place in order to maintain the well and respect the environment. Finally, construction and rehabilitation is affected by the rainy season, with drilling prohibited during this time. Many communities have water associations, which will be engaged to develop community savings or microfinance efforts to generate funds for maintenance of the wells. Activities related to this had only just begun at the time of the evaluation in some communities, as the creation of Water User Associations is linked to the progress of well construction. Finally, MAHEFA WASH coordinators work with the TAs to monitor their activities, and participate in local events to promote WASH behaviors. They also collaborate with other NGOs in their regions to promote harmonized efforts to support Committees for Total Sanitation (CLTS). And, they work with local partners and institutions such as churches to promote latrine construction and use. In a notable program success, authorities in Diana agreed not to tax latrine construction. Finally, one coordinator reiterated the challenges of working in low resource communities: “We work with very deprived communities, where people are more concerned about getting enough to eat, and less so with what comes out.” Summary The Medical Inspectors who were interviewed reported collaborative relationships with the MAHEFA Regional Coordinators. While valuing the work of MAHEFA, they raised concerns about the capacity of many of the NGOs to support integrated health services, considering that several had historically focused on a single health or social issue. This has been addressed by MAHEFA in its support of the NGOs Grantees. The MIs also commented that some community members found it hard to accept CHWs, and that there was also some tension between CSB and CHWs. They also expressed concern about the quality of the data reported by CHWs and TAs. During courtesy visits to the three Regional Directors of Public Health, favorable observations were offered regarding the contributions of MAHEFA in implementing complementary community health interventions. The challenges mentioned by the DRSP reiterate those mentioned by other respondents: entrenched social norms regarding sanitation, CSB reluctance to be engaged with CHW training and supervision, and inadequate communication among all actors. The three Regional Directors of Water were unanimous in acknowledging MAHEFA’s success in contributing to an improved water infrastructure and supporting dissemination of WASH sensitization activities. Much of the day-to-day coordination with NGO Grantees falls to the six MAHEFA Regional Coordinators. This has presented numerous challenges, in large part due to the relative inexperience of local NGOs in community health. Other constraints, such as limited NGO resources for transportation 32 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program were mentioned, as were security concerns in some areas. Finally, Regional Coordinators mentioned that the capacity to manage stock varies among the CHWs, and occasional stock outs occur. These do not appear to be serious problems. Two MAHEFA staff members engaged in Monitoring and Evaluation were interviewed. They are focused on continually improving data quality received from the NGO grantees. They cited several issues that they confront in this task: turnover in TAs, varying quality of data from the CHWs and challenges in identifying CHW who need assistance and support, due to data aggregation, and the uncooperative relationships with CSB Directors. The six regional WASH Coordinators were enthusiastic about their work, and determined to advance activities that have been delayed from the inception of the program, well documented elsewhere. The process of water point construction or rehabilitation is a lengthy one, requiring multiple consultations with the Ministry of Water, engagement of local consulting firms to evaluate community needs and identify construction sites, contracting construction firms, carrying out multiple EMMP and inspections, all within the context of limited time to undertake construction in order to avoid the rainy season. Population Services International (PSI) MAHEFA has a Memorandum of Understanding (MOU) with PSI, which implements a nation-wide social marketing program of health and sanitation products in Madagascar. PSI fields four distribution teams in the MAHEFA program regions, in Diana, Melaky, Menabe and Sofia; the team in Diana also covers Sava. In addition to supply depots in the regional capitals, PSI also supports provision points (PA) in communes. PAs are usually located in the primary city of the commune, and are complemented by a “relay” supplier who supports more distant and inaccessible zones. Normally PSI stops its distribution before reaching the inaccessible areas, due both to poor road conditions and security concerns; the relay points were developed to address the needs of CHWs supported by MAHEFA to have access to stock even in remote areas. MAHEFA has undertaken several efforts to complement PSI activities on supplying very hard-to-reach areas. An outstanding example is MAHEFA’s work in partnership with HoverAid, initially in Boeny, and expanding to sites in Menabe. Through this effort, hovercraft are used to ferry supplies to remote areas accessible by river. In addition, MAHEFA made the connection with helicopter services in Mandritsara to allow PSI to broaden their transport options for health commodity delivery. PSI acknowledges that there are challenges in maintaining consistent supplies at all the relay points, for several reasons. First, the people responsible for maintaining the relay point are meant to send their stock data by mobile phone, but it seems that not all are able to master this. Second, the cellphone network coverage itself presents difficulties, as some areas are so remote that signals are inconsistent. Finally, those maintaining the relay posts are not adequately compensated for their work. Typically the relay stocks commodities sufficient to match monthly consumption by the CHWs in its catchment area, plus one month of security stock. However, if CHWs do not purchase their re-supply, the PA does not have the means to buy replacement stock. Further, if there is a surge in disease incidence, there may be a run on stock. Provisions are based on data received from the CHWs, and that information is necessary to plan for appropriate supply, but there are still gaps in communication about data and its quality. Of all the products supplied by PSI, only Sur’Eau is produced in Madagascar; all other commodities are imported. Thus, if there are any problems with importation or availability of stock, it ripples through the entire system. 33 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Some products that are available are not appealing to consumers. In particular, PSI makes condoms available throughout the country in order to meet U.S. Government requirements that all people wishing to use contraception have a full range of methods available. Marie Stopes International Madagascar The MOU with Marie Stopes International Madagascar (MSIM) establishes an agreement by which MSIM provides long-acting and permanent methods of contraception to women and men in MAHEFA￾supported regions. It also provides for free office space at the regional office in Menabe and it is planned that in the future MSIM will cost-share use of a commercial hovercraft in Melaky to reach inaccessible areas. MSIM notifies the local NGO partners when their mobile team is going to be in the area. If CHWs have a client who wishes to use one of these methods (implants, tubal ligation, vasectomy), they refer them to the mobile teams. According to MSIM, collaboration has been particularly successful in Menabe. While the respondent did not provide any data to support her observation, she reported that “CHWs have a great added value in FP training” and “Four in 10 [clients] say CHW” persuaded them to use FP. Summary While MAHEFA has memoranda of understanding with several partners to enhance project implementation, the evaluation looked at only two: PSI and Marie Stopes International, Madagascar. PSI, through its network of social marketing distribution teams, provides products for the CHWs. PSI had not previously had many distribution points in the six regions in which MAHEFA works, and through the MOU the two organizations established a mutually beneficial relationship that expanded PSI’s network while ensuring MAHEFA-supported CHW had access to heath supplies. A similar, complementary relationship exists with MSIM, in which MSIM is able to provide access to long-acting and permanent methods of contraception to meet the needs of CHW clients in MAHEFA program regions. DISCUSSION OF INTERVIEW FINDINGS In the more than three years since MAHEFA has been implemented, the results from interviews from all levels of the project structure suggest that MAHEFA is making strides to improve the health of communities in under-served and hard to reach areas of Madagascar. Multiple challenges exist and remain to be resolved, nonetheless, there is a sense of appreciation of the work that MAHEFA has accomplished to date and a feeling that the program is on the path to success in meeting its main objectives. MAHEFA was applauded for its roll out of the program. It was viewed as well planned and coordinated. The program seems to be particularly successful in bringing together various stakeholders from a range of sectors and regions to work together for a common goal. The overall feeling is that they have provided good technical support and training. At almost all levels, respondents point to positive impacts in the communities served by MAHEFA. Interviewees spoke about of improvements in vaccination rates, use of antenatal care, increased use of mosquito nets, and increases in family planning use. They also point to resulting decreases in childhood diarrhea, malaria, and respiratory infections. People spoke of increased awareness of the benefits of hand washing, use of Sur’Eau and solar water purification techniques, and in many communities there was an appreciation for the construction or rehabilitation of wells and water points. Furthermore, people believe that health seeking behaviors have improved, especially with regard to seeking care for 34 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program children, and there is less reliance on traditional healers and more on CHWs and CSBs. Beneficiaries are very appreciative that the CHWs sell medicines and other products at prices that are less expensive than at other places where they would purchase them. Many challenges were noted at all levels, especially difficulties in changing behaviors deeply rooted in social norms. In particular, respondents referred to resistance to building latrines, eliminating open defecation, and adopting condom use. Social norms. Social norms and cultural beliefs are strong forces to overcome in the promotion of public health. In a country like Madagascar, typified by insular communities with impoverished populations possessing limited education and lacking of awareness of modes of disease transmission, it is a formidable challenge to promote a change as fundamental as latrine use. People cite ancestral beliefs, cultural taboos, and old habits to rationalize their resistance to change. For some, the financial burden of acquiring building materials is an obstacle to adopting a new behavior. Nevertheless, in most of the communities the team visited, at least a few recently constructed latrines were observed. As community members observe these innovators, it is likely that social change will occur over time, as improved sanitation habits start to become the “new normal”. Tension with CSB Directors and Medical Inspectors. The evaluation team encountered a range of attitudes regarding the interaction of MAHEFA staff with the Directors of CSBs and Medical Inspectors. For some, MAHEFA-supported services were seen as a welcome addition to the public health infrastructure in their communities. Respondents were aware of their own limited resources, and were pleased with the contribution MAHEFA makes in terms of delivering basic services more conveniently in the community and with promoting health seeking behaviors that brought clients to the CSB. Other respondents expressed frustration with the program, primarily linked to exclusion from training opportunities and to poor quality data submitted by CHWs and TAs. It is difficult, in the context of a formal and cordial interview, to ascertain to what extent these are legitimate concerns, and not simply unhappiness with limited opportunities to collect per diem and transport allowances. With the lifting of the ban on working with the public sector, there may be opportunities to engage the CSB Directors and MIs more directly in supervising CHW and identifying data weaknesses and improving data quality, especially given that this is one role for the Directors explicitly identified in the National Community Health Guidelines. That noted, the MAHEFA budget was not constructed to include extensive engagement with regional and district level MOH staff, so mutually beneficial activities and budget reallocations will need to be carefully considered. NGO selection and relations. When the original proposal for the MAHEFA project was developed, it drew on the experience of two prior community health projects implemented in other regions of the country, Santénet I and Santénet II. Unfortunately, conditions in the MAHEFA regions were significantly different from those benefiting from Santénet interventions. One deficit in particular was the limited number of NGO partners working in the six regions with community health experience. While this was originally to be one of the selection criteria for NGO partners, MAHEFA had to adapt to local constraints and consider NGOs that had any development experience. As a result, many of the partner organizations had previously specialized in related, but not exact, community activities. These included family planning promotion, education programs for at-risk girls, HIV/AIDS support, and environmental action groups. This lack of expertise in critical program areas consequently meant that MAHEFA needed to invest far more technical support in its NGO partners than had been envisioned from the outset. The regional 35 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program NGOs needed to hire appropriately skilled staff, which occasionally was in limited supply in the regions. These staff needed to be trained both on the technical content of program activities, as well as project management skills that would permit them to comprehend and adhere to MAHEFA reporting requirements and USAID-mandated activities, such as environmental mitigation evaluation audits. NGO partners apply annually for grants from MAHEFA, and are selected on a competitive basis. The NGOs selected possess a range of experience, from large, national organizations with multiple offices around the country and funding support from multiple donor agencies, to smaller, more localized groups with limited resources. This diversity, and MAHEFA’s stringent guidelines for the submission of financial and project activity reports, has resulted in a mix of responses from NGO partners. Some express appreciation to MAHEFA for upgrading technical and administrative skills, while others focus on the burden of reporting, expectations of short turn-around time to responses for information, lack of transparency in decision making, and the perceived superseding NGO policies and procedures by imposing those of MAHEFA. In October 2014 MAHEFA will make a final round of grants, limiting competition to NGOs with which it currently works. Salaries, per diem, travel payments. As was noted in the introduction, MAHEFA is a highly decentralized program relying on increasingly distant actors to contribute to its success. Many of these key individuals are volunteers or are appointed by their local communities—emergency transport drivers, water association members, organizers of the mutuelles de santé, indeed the CHW themselves. As volunteers, they are provided with training and information in order to carry out their roles, but few receive any allowance or “motivation” for time spent on their tasks. Nevertheless, many individuals mentioned that their neighbors perceived them to receive financial benefit from the program, and subsequently assumed that they had become “wealthier” as a result. CHWs remarked that some of their clients believed that the CHW received their stock at no cost, and hence should not charge for products they sell. Data quality. Many respondents expressed concern with data quality manifested as information moves through the system. CHWs reported challenges in completing all the required documentation for client visits and sensitization activities, TAs were often unable to reconcile CHW reports or to detect aberrant patterns in data, some CSB directors say that they include CHW data in their monthly reports while others complain it is not useable. Interviewees noted that MAHEFA is already trying to address at least some of these challenges. For instance it was realized that CHWs vary in their ability to complete the forms correctly due to educational limitations or carelessness in completing all required registers and forms relevant to particular client visits. The forms have already changed several times during the course of the program to make them more manageable. However, a number of the registers used by the CHWs are the standard Ministry of Public Health registers, and it is not possible to modify them. While MAHEFA is promoting integrated health services, each individual service must be recorded in a unique register. Reviewing data patterns and trends and being able to identify inconsistencies is a skill that is not inherent in everyone. It can be taught, and MAHEFA has developed an M&E validation manual for TAs and NGO M&E staff, and has begun training on its use. The manual is very comprehensive and carefully thought through, but would benefit from some additional content and editing. Suggested additions include:  an acronym list  a flow chart of the verification process that systematically indicates the high-level steps included in the process 36 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program  placement of all general instructions at the beginning of the document (for example, those included at the end of step 2 regarding use of a red pen, conducting the review in the presence of the CHW, etc.)  more specific detail about anomalous patterns in data that might be observed: all clients listed in the same order, month after month, use of the same pen, with identical handwriting implying the CHW has filled out the register at one time large deviations in monthly reports either month￾to-month or year to year; limited data for some services, implying that CHW may favor some services or activities over others  more explicit instructions or examples about actions to take to redress weaknesses or errors. Commodity stock outs. Short of conducting a systematic analysis of the entire logistic system, it is difficult to estimate the scope and magnitude of commodity stock outs. When virtually all the products available to the CHW are sourced outside Madagascar, the effect of any delay in delivery or release from the port will ripple through the system. Even with the establishment of provision points and relay depots, the distance between the fokontany in which the CHW resides and the location of the PA may be great. High water and poor roads render many communities completely inaccessible during the rainy season, and demand forward planning to ensure adequate stocks are delivered well in advance of these conditions. Improved record keeping, forecasting and stock management on the part of both CHWs and NGOs will contribute to better estimates of need and more timely delivery of stock. Turnover of CHW and TAs. Several respondents mentioned that there has been turnover in TAs as well as loss of CHWs trained by the project, due to taking other employment, relocation, death, or disinterest. MAHEFA has worked with local officials to identify replacements for CHW, but has limited control over this as the individuals must be selected by the community and endorsed by the local CSB. Following the evaluation debriefing in which TA turnover was mentioned, MAHEFA initiated a survey of the NGO partners to determine if this is in fact a serious problem or whether it is more a perception than reality. The principle concern with turnover is how to efficiently provide training to newly recruited staff. The evaluation team encountered three TAs who had not been fully trained in either the program’s technical topics nor in supervision skills. The team was also not able to obtain a clearly articulated plan for re￾training; it seems to be done on as needed in each region for each NGO. Mal-distribution of TA workload. MAHEFA has already been responsive to the heavy workload borne by TAs by reducing the number of communes for which each is responsible. Nevertheless, the number of CHW working in a commune may vary from fewer than 20 to more than 60, resulting in an inequitable distribution of responsibility for some TAs. Condom use. As is the case in many other countries, there are some indications of resistance to using condoms among men in the MAHEFA program areas. While Madagascar has a comparatively low level of HIV prevalence, the incidence of other sexually transmitted infections remains a concern. Furthermore, while use of other contraceptive methods is slowly increasing, many women remain with an unmet need for family planning for spacing or limiting future births. PSI is helping to maintain a consistent stock of condoms with CHWs so that a full range of contraceptive methods are available either on site or through referrals to CSB or MSIM. Condom promotion should continue to be part of the sensitization efforts of CHWs. Emergency transport. MAHEFA’s efforts to provide emergency transport in several districts has met with mixed results. On one hand, people are pleased to have an alternative means to reach a health care provider, using a locally available bike-powered ambulance. On the other hand, they would prefer 37 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program motorized transport that would provide a smoother ride, more rapid travel time, and greater security. While no tragedies have yet occurred, respondents were plagued by “what if” worries: “What if the patient dies en route?” “What if a woman gives birth on the road?” “What if we are attacked by thieves?” “What if the driver is not strong enough to pedal?” Equipment maintenance. Participants in several FGD worried about maintenance of bikes, wells and water points provided through MAHEFA. In theory, MAHEFA aims to promote the sustainability of these resources by providing tools and training for bike repair and well maintenance to community members. In some cases, the program has also supported the formation of water associations or community support groups for the emergency transport, with the idea that members would contribute a small amount monthly that would be banked for the maintenance of the equipment. In fact, such groups have been slow to form, and there are misconceptions about how their contributions would be spent. Well construction has been delayed for many reasons, and people often start to use wells before they have been inspected and certified; pumps and gates have been broken and covers removed. Security concerns. A number of community members and CHWs expressed concerns about security, citing worries about appearing “too wealthy” due to possession of a bicycle, MAHEFA products, or the perception of being paid a salary. People also worried about traveling long distances or at night. There is little that MAHEFA can do to address this, beyond acknowledging peoples’ fears, and promoting caution when carrying out program activities. Awareness raising among communities that the CHWs and other actors are unpaid volunteers may also be of some value. Recommendations for immediate action 1. Continue efforts to promote latrine use and routine hand washing practices. Develop a few sets of building plans using local materials (corrugated tin, flattened steel drums, woven raffia, bamboo, wattle and daub) that are appropriate for the different locations, to satisfy people’s concerns that their construction may not meet standards. Provide estimates of cost of construction using local materials so people will have accurate perceptions of financial requirements. Collaborate with local leaders to conduct “village walks” in which suitable locations for latrines are identified that are mutually agreeable to neighborhoods. 2. MAHEFA HQ and regional staff continue to monitor and support NGOs to complete procedures and meet deadlines in order to ensure that the program meets targets in a timely and cost-effective manner. Following the final round of grant making, relevant NGO partners should be convened and the final year activities and objectives clearly delineated, with appropriate milestones established. MAHEFA should foster even more transparent communication with NGO grantees to ensure the full understanding of USAID provisions and regulations that govern grant agreements, and continue its efforts to provide assistance so that NGO grantees understand and follow these regulations. 3. Increase the frequency with which condoms are part of sensitization efforts. Take advantage of home visits to conduct couple counseling on contraceptives and underscore the contributions of condoms in preventing both pregnancy and STIs. Use more entertaining sensitization techniques during contraceptive demonstrations (for example, blowing up condom contests) to de-stigmatize their use. Recommendations for mid-term action 4. Together with USAID/Madagascar and the Ministries of Public Health and Water, MAHEFA must determine what level of engagement with CSB directors and MI is feasible during the remaining duration of the program. Expectations must be realistic, and should focus on engaging CSB directors 38 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program to certify CHWs who are in the training pipeline but have not yet been approved to initiate activities. The CSB directors and MI should also be engaged in each region to clarify expectations about data quality, format and use. Criticisms should be explicit and informed by examples. 5. During the final project years, MAHEFA should increase its focus on training NGO M&E staff and TAs in data quality management, skills that will contribute to local capacity building for future activities. Hands-on sessions should include interpretation and analysis of actual data, using active CHW registers and reports. Similarly, TAs can organize sessions with several CHW from adjacent fokontany to review data and exchange experiences. Alternatively, sample sessions can be conducted on specific forms or registers during the monthly meeting at which CHWs submit their reports. In view of its importance for stock management, attention should be paid to correct use of the commodity use form. The recently-developed M&E manual is a useful tool to guide this training, but it is suggested that even more basic exercises be developed. For example, these might compare monthly trends in reported data, to see if service utilization remains stable over time, or increases as communities become more comfortable with accessing care from CHW. What patterns can be detected in monthly reports of contraceptive use? Are the data reported consistent with stock supplies? Are there seasonal variations that might be expected, and are the data consistent with anticipated increases or decreases in illness frequency, such as malaria or respiratory infections? How can the data be used to plan community outreach activities? In other words, what story do the data tell, and what is the logic that underlies it? 6. MAHEFA should work with the NGO grantees to establish reasonable numbers of CHWs to be supervised by each TA. Using a geographic unit (commune) for allocation has resulted in some TAs being responsible for up to 60 CHW, while others handle only one-fourth that number. Coupled with the time and distance challenges of supervising such a large number, there is a disproportionate burden shouldered by some of the TAs. 7. CHWs, TAs and MAHEFA regional staff need to manage expectations concerning emergency transport. Communities must be persuaded that having some transport is better than none. Rather than imagining what calamities might befall people using the transport, community members need to see the benefits of the service, perhaps in the form of a happy and healthy mother who return home safely after delivery. 8. All relevant actors, including members of CoSans, Mutuelles de santé, CHWs and other community partners should be encouraged to promote the benefits of community revolving funds for health care, maintenance of wells and water points, and upkeep of emergency transport. If possible, MAHEFA should identify communities in which revolving funds have been successfully introduced, and share the results in other locations, perhaps using testimonials, a promotional video, or portable story board. Considerations for follow-on activities MAHEFA has made important strides in its pilot work in developing mutuelles de santé, in its introduction of emergency transport, and its collaboration with PSI and MSIM, among others, to leverage the impact of program intervention. It is suggested that the lessons learned from these experiences be carried forward and be built upon in a follow-on community health program. For a host of reasons, documented here and in MAHEFA project documents, the WASH activities proposed for the program have been challenging to implement. MAHEFA is on course to make up for many of the delays experienced in construction and rehabilitation efforts, but there remains a great 39 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program demand for clean water sources in the program regions. Again recognizing that some important lessons were learned, ranging from the specific language that must be included in a Cooperative Agreement to a realistic appreciation of the time required to identify sites, obtain permits, undertake and fully complete construction and conduct final inspections, such efforts should be included in a follow-on program. Similarly, MAHEFA has faced an uphill battle in promotion of latrine use. This very fundamental element of good public health and sanitation practices must be reinforced. It may be worth considering a study tour to India, which is currently waging a very public, nationwide effort to also reduce open defecation, to learn from the experiences there in promoting social change. From the standpoint of sustainability and contributing to improved quality and utility of national health data, future projects should consciously invest in training and promotion of lower level health cadres such as CHW on good data management practices. This need not be a complex exercise, but it should warrant specific and sustained attention during initial training and during on-going supervisions. This focus should be maintained upward through the data reporting system. Finally, in a complex community health program such as MAHEFA, implemented on such a large scale and decentralized manner, it is essential that a follow-on project be realistic in its estimation of the capacity of regional and local partners, be they NGOs or public sector. MAHEFA invested an unexpectedly large amount of effort in supporting its NGO grantees. This has likely contributed to stronger capacity within many of the organizations, but there will continue to be additional development needs in the future. Should a future program collaborate more directly with local health authorities, it is very likely that some similar reinforcement of skills in technical updates, data management, supervisory practices, and materials support may also be necessary. PART 2: RESULTS OF ECONOMIC ANALYSES 1. Cost-efficiency analysis Methods Cost-efficiency analysis examines productivity of resource use by relating costs of inputs to the outputs produced over a defined time period. The MAHEFA cost-efficiency analysis combines information from two sources: cumulative MAHEFA expenditures covering the time period from the beginning of the project through 31 March 2014, and cumulative M&E data (for the same time period) on numbers of fully functional CHWs and outputs reported by these CHWs. MAHEFA expenditure categories include main office and regional office expenditures, subcontracts to local NGOs working in each region, and subcontracts to two international NGOs, Transaid and the Manoff Group. MAHEFA senior managers provided information on regional office expenditures and NGO expenditures (local and international) within each of the six project regions. MAHEFA main office expenditures were allocated to regions based on the number of MAHEFA full-time equivalent staff (FTEs) working in each regional office. This allocation method was used for two reasons: first, salaries of regional staff are paid from the main office budget; and second, the level of main office effort expended in each region would be expected to vary in proportion with staff level of effort. 40 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program MAHEFA senior managers also provided information on the number of Districts and Communes with MAHEFA-sponsored activities in each region, the number of Technical Agents trained, the number of CHWs receiving support from MAHEFA, the number of CHWs reporting activities (a subset of the previous number) and the number of beneficiaries seeking care from CHWs. MAHEFA also provided cumulative data on five key activities reported by CHWs in each zone: (1) Number of cases of child diarrhea treated with oral rehydration solution (ORS) by community health workers; (2) Number of children under five years old (CU5) presenting with pneumonia and given appropriate care; (3) Number of regular users of family planning (4) Number of malaria cases treated with Artemisinin-based combination therapy (ACT) within 24 hours, and (5) Number of children reached by USG-supported nutrition programs. These five indicators were aggregated to produce two summary indicators for each NGO and region: all five indicators, and the four “non-nutrition” indicators, which we term “treatment outputs”. We then calculated the following indicators for each region: (1) expenditure per CHW reporting activities (to indicate the average MAHEFA investment in each CHW); (2) expenditure per CHW output (using the two different measures of CHW output), and (3) the number of outputs per CHW reporting activities, to show the average productivity of CHWs across local NGOs and regions. Quality of service provided by CHWs was not assessed in this analysis. Results Table 4 presents cumulative expenditures through 31 March 2014 (i.e., since the beginning of the MAHEFA program) by region. Expenditures show substantial variation across regions, with Sofia having accounted for 27% of the total cumulative expenditure of US$18,129,240, and Boeny having accounted for 5% of the total. Main office expenditure is the largest component of spending in all regions and accounted for 51% of overall MAHEFA expenditure, but it should be pointed out that MAHEFA regional staff salaries are included in this expenditure category. Local NGO subcontracts comprised approximately 20% of overall expenditure, and international subcontracts accounted for approximately 4% of expenditure. Table 4: Cumulative expenditures by region (in US$) Menabe Melaky Boeny Sava Diana Sofia Total Regional Office 1,087,754 725,170 181,292 362,585 906,462 1,269,047 4,532,310 Local NGO Sub-awards 719,398 441,552 204,000 141,815 680,788 1,466,888 3,654,440 International Sub-awards 209,882 61,888 15,472 147,994 77,360 225,354 737,949 Main Office 1,796,008 1,571,507 449,002 1,347,006 2,020,509 2,020,509 9,204,541 Total 3,813,042 2,800,116 849,766 1,999,400 3,685,118 4,981,798 18,129,239 Percent 21% 15% 5% 11% 20% 27% 41 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Table 5 presents information on MAHEFA program outputs, as well as key outputs reported by the subset of MAHEFA-supported CHWs who reported activities. As was the case with expenditures, cumulative MAHEFA program outputs were highest in Sofia where 98 TAs had been trained and 2,138 CHWs were reporting activities as of 31 March 2014; and lowest in Boeny, where 10 TAs had been trained and 221 CHWs were reporting activities. The picture is slightly different with MAHEFA￾supported CHW outputs, where Sofia once again led the six regions, but Diana and Sava both reported lower totals than Boeny, mainly due to the higher numbers of children reached by nutrition programs in Boeny. Table 5: Cumulative outputs by region Menabe Melaky Boeny Sava Diana Sofia Total MAHEFA Outputs TAs trained 46 40 10 19 65 98 278 % 17% 14% 4% 7% 23% 35% CHWs receiving support 1,029 689 221 306 1,122 2,446 5,813 % 18% 12% 4% 5% 19% 42% CHW reporting activities 934 594 221 251 855 2,138 4,993 % 19% 12% 4% 5% 17% 43% CHW Outputs Child diarrhea treated 8,716 10,543 5,621 3,633 6,813 34,302 69,628 % 13% 15% 8% 5% 10% 49% Child pneumonia treated 6,229 9,213 3,540 2,729 7,935 19,229 48,875 % 13% 19% 7% 6% 16% 39% Regular FP users 16,340 9,869 6,576 5,632 13,409 36,911 88,737 % 18% 11% 7% 6% 15% 42% Malaria treated 14,223 20,622 14,674 2,879 4,625 39,943 96,966 % 15% 21% 15% 3% 5% 41% Child nutrition 1,151,443 420,780 350,951 276,522 190,754 2,416,770 4,807,220 % 24% 9% 7% 6% 4% 50% Total CHW Outputs 1,196,951 471,027 381,362 291,395 223,536 2,547,155 5,111,430 % 23% 9% 7% 6% 4% 50% Total Treatment Outputs1 45,508 50,247 30,411 14,873 32,782 130,385 304,210 % 15% 17% 10% 5% 11% 43% 1 Treatment outputs are defined as total outputs minus the nutrition output. 42 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Table 6 shows selected measures of productivity and cost-efficiency by region. Boeny had the highest CHW productivity on both measures, while Diana had the lowest CHW productivity on both measures. CHWs in Sofia and Menabe ranked higher on productivity when all outputs were considered, while CHWs in Melaky produced relatively more treatment outputs than other regions. The “expenditure per CHW” measure shows the level of investment of MAHEFA resources across regions, and indicates that relatively more resources per CHW were invested in Sava, while relatively fewer resources per CHW were invested in Sofia. Next, the “expenditure per CHW output” is one of two measures of cost-efficiency taking into consideration the services delivered to CHW clients. CHWs in Sofia deliver services to clients at the lowest expenditure per unit of output, reflecting the lower unit expenditure per CHW as well as the high productivity per CHW. Expenditure per unit of output is highest in Diana, driven mainly by lower CHW productivity in that region relative to others regions. Finally, using treatment outputs as the denominator, Boeny and Sofia are the most cost-efficient regions under this scenario, while Sava and Diana are the least cost-efficient. Table 6: Productivity and cost-efficiency measures by region Menabe Melaky Boeny Sava Diana Sofia Productivity measures All outputs per CHW 1,282 793 1,726 1,161 261 1,657 Treatment outputs per CHW 49 85 138 59 38 61 Cost-efficiency measures Expenditure (US$): per CHW $ 4,082 $ 4,714 $ 3,845 $ 7,966 $ 4,310 $ 2,330 per CHW output $ 3.19 $ 5.94 $ 2.23 $ 6.86 $ 16.49 $ 1.96 per treatment output $83.79 $55.73 $27.94 $134.43 $112.41 $38.21 Observations on the cost-efficiency analysis Consistency of cost-efficiency measures – are the relative rankings consistent across regions? Although the relative rankings vary slightly across the two output-related measures, a consistent pattern does emerge. Sofia and Boeny rank first or second in terms of both measures of cost-efficiency, Menabe and Melaky are situated in the middle, and Sava and Diana rank last or second-to-last on both measures. 43 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Drivers of differences in cost-efficiency across regions To assess differences in cost-efficiency across regions, it is helpful to think about the main drivers of total expenditure and the main drivers of total outputs. Main drivers of expenditure include the number of full-time MAHEFA staff in each region, the number of NGO partners in each region, and the amount of international partner involvement in a region. The amount of MAHEFA expenditure in a region is based on a number of factors. First, GOM policy specifies the number of people to be served per CHW and the area where they will serve (two CHW per fokontany). Thus, regions with more fokontany like Sofia have larger numbers of CHW and therefore receive more program resources than less densely￾populated regions like Boeny. Also, the number of sub-grantees is linked to number of districts, which explains why MAHEFA works with nine local NGOs in Sofia, and one in Boeny. Additionally, MAHEFA allocates program resources based on other factors such as remoteness (higher travel costs for supervision and for CHW trips to the CSB for monthly meetings, etc.), and number of activities at sites (all sites have integrated activities but some sites have extra activities). Main drivers of outputs in a region are associated with factors such as the number of CHWs supported and the productivity of these CHWs. Internally, MAHEFA uses the same approach to support CHWs in all regions, with regional NGO teams being the primary implementer of activities. The regional NGO teams have different levels of efficiency and this affects the support received by the CHWs, which in turn affects their performance. NGOs have had varying levels of difficulty recruiting and retaining qualified staff, managing their funding to keep activities going, and providing quality management in MAHEFA areas. Externally, support from GOM health directors and medical inspectors in each region varies greatly. GOM staff in some regions provided quicker approvals and support for CHWs to receive clinical training and certification. Timing of training and certification affects the length of service time and the number of patients treated by CHWs, all of which contribute to variability in productivity between regions. Delays in certification likely explain the low productivity and cost-efficiency in Diana, which has had a high number of untrained and uncertified CHWs during the period under study. Other external factors may include differences in: intensity and duration of MAHEFA programming since initiation, support from international partners, support from district/regional leadership of GOM; supervision and support from their CSB supervisors; support from the community (providing a work space or health hut, leaders support and give importance to CHW work); and accessibility since it helps determine supervision of and interactions with CHWs by GOM staff, NGO and MAHEFA staff and access to commodities. It would be ideal if cost-efficiency measures reflected underlying efficiency of production, allowing for identification of sub-grantees and regions where staff were using effective approaches for converting inputs into outputs, ultimately enabling translation of these practices from more efficient sub￾grantees/regions to less efficient ones. But the reality of the MAHEFA program is considerably more complex, and the cost-efficiency measure reflect a host of factors whose influence is difficult to quantify. Without a way to control for these factors, it is extremely difficult to disentangle efficiency-related factors from all other possible reasons for differences in expenditure per unit of output across regions. Selection of denominators for cost-output calculations a. Outputs versus outcomes The SOW explicitly requested a “cost-efficiency” analysis, which by definition examines relationships between inputs and outputs, thus focusing attention on production of output and comparing different entities in terms of how efficiently they transform inputs into outputs. When outcome-level data are available, the possibility of conducting a “cost-effectiveness” analysis arises, which compares different entities in terms of the cost per unit of health effect produced. MAHEFA currently has data on a range of outputs, but will not have outcome-level data until the survey is completed later in 2014. 44 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program b. Selection of output measures The SOW did not provide explicit guidance on which outputs to use in the analysis. The evaluation team worked with MAHEFA senior managers to prioritize activities and output definitions based on MAHEFA’s and USAID’s definitions of core activities and key outputs. Each of the five output measures was designated since the start of the program to represent the core health activities with the highest potential to reduce morbidity and mortality. MAHEFA emphasized that these five outputs are core activities of the program, with several discussions with USAID taking place emphasizing the importance of these activities over pilots and innovations (e.g., transport2, CHX, mutuelles de santé). While it is true that much of MAHEFA’s effort focuses on preparing TAs and managing NGO efforts, ultimately the success of these activities should be reflected in the outputs produced and reported by CHWs. c. Aggregation of output measures One of the inherent problems with evaluating integrated programs like MAHEFA is that these programs produce disparate outputs that are not easily combined or aggregated to represent program output. An apparent solution could be to apply some weighting criteria to reflect the relative “health value” or by the time needed for the CHW to complete, but there is presently no methodological consensus on how such weighting would be done. Given these challenges and a lack of a methodology for weighting outputs, we chose to compute two different cost-output ratios (with and without the nutrition output) and determine whether the ordering of regions according to cost-output was consistent To weight these differently would require development of a new metric that could combine impacts such as lives saved, illness alleviated, and unwanted pregnancies averted, which is beyond the scope of this evaluation and would require special research efforts to produce. The price of equity The analysis measures cost-efficiency in the context of the program that MAHEFA was operating during the period ending on March 31, 2014. Estimates of costs of extending the MAHEFA program to more remote communities were not calculated. It would be feasible, however, to work with MAHEFA to develop a set of assumptions about how such an expansion would be structured. For example, would this effort build out from an existing platform (such as Boeny) into more districts in the same or nearby region, or would it entail establishing new platforms in different provinces? Would it work with existing NGOs or recruit new ones? Once the assumptions were spelled out, we could work with MAHEFA senior managers to generate pro-forma activity budgets to estimate costs of extending services to remote communities (this would be a separate activity requiring its own resources). 2. Earned Value Management (EVM) Introduction The purpose of this analysis using Earned Value Management (EVM) is to compare the work plan activities performed against the activities intended to be performed. This evaluation how much of the schedule has been completed against how much was expected to be completed. It also illustrates how much of the project budget has been spent to date compared to the amount that was expected to be 2 In the specific case of emergency transport, MAHEFA does not collect expenditure and output information down to this level of granularity, and it attempting to estimate expenditure per unit of output with available MAHEFA data would require so many assumptions as to call into question the validity of the estimate. Such information would be useful for assessing “value for money” of emergency transport, but would better be built in as a prospective data collection effort. 45 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program spent, based on the amount of work performed. In the following sections we present information pertaining to the project schedule of activities and their completion status, and the program budget data, both of which are the essential elements of calculating the EVM. The results of the EVM follow, and explanations for some of the delayed activities conclude this part of the report. Schedule The lifespan for this project is May 23, 2011 through May 22, 2016, a total of 60 months. We used months-to-date (37 months as of June 30, 2014) to determine the planned percent complete to date, which is 61%. The goal of the analysis is to compare the actual budget to the tasks undertaken and then to project the budget for tasks remaining. Ideally it would have been preferred to have a project schedule that covered the life of the project and each task associated with a budget code. Since the yearly plans did not have this information, we used the Environmental Mitigation and Monitoring Reports (EMMR) to categorize activity status. The EMMRs consistently provided a subjective status of planned activities. Based on the statuses, we assigned percentages to each activity. The analysis consist of the actual budget for year-to-date compared to percentage of activities complete (based on percentages assigned to each status). We derived the actual percent completion using the EMMR in each year’s annual plans. Although this is a 5-year project, the schedules are created on an annual basis. There was no way to create a 5-year baseline for EVM. The most consistent source of information was annual reports that assigned statuses to planned activities, such as “completed”, “ongoing”, and “postponed”. Since EVM is a formula that requires calculation of percent of activities complete, we assigned percentage values to each status as follows:  Completed= 100%  Ongoing as planned = 100%  Ongoing with delays (appropriate) = 75%  Ongoing with delays (not appropriate)= 50%  Postponed (appropriate)=75%  Postponed (not appropriate)=50% “Ongoing” activities occur over the life of the project.  Activities that were ongoing and on schedule are weighted 100%.  Activities that were naturally ongoing with delays are weighted at 75%  Activities that were delayed longer than expectations are weighted at 50%. “Postponed” activities may be deliberate or beyond the control of the project.  If an item was postponed for deliberate reasons (for example, to ensure a prerequisite activity is complete) it is weighted at 75%.  If an item was postponed due to circumstances beyond the project, it is weighted at 50%. If an activity was postponed, but was within the control of the project staff to get back on schedule, for example by working additional hours, performing multiple tasks concurrently, etc., it was weighted higher (75%) than activities delayed at least two weeks and beyond the staff’s control (50%). 46 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program The percentage assignments are arbitrary and are meant to show if the project is ‘on schedule’ or ‘behind schedule’. For ease of calculation, the values were assigned in increments of 25 percentage points. We could have chosen any percentage amounts for the statuses as long as they were appropriately weighted relative to each other. For example, ‘completed’ must have a higher weight than ‘ongoing’. One could assign completed activities 100% and ongoing activities 98%. The numerical values would change; however, the overall result of the analysis would remain the same. It would still show the project is slightly behind schedule. Table 7 shows the total completion activity status for project years one, two and three. Table 7: MAHEFA activities percent complete by project year Year Percent complete Yr1 97% Yr2 84% Yr3 96% This information allows us to derive the following measures: a) Current percent of activities complete for the project’s first 37 months b) Total percent of activities complete based on 60 months c) Whether the project is ahead of or behind schedule Budget For budget data, we used SF425 financial documents with expenditures to-date through the end of June 2014. We then added accruals for end of June 2014 to determine the total amount spent to date. The original ceiling amount of the award is based on initial project documentation. Original Ceiling 34,999,935 Spending as of June 30, 2014 Expenditure/Accruals 20,478,510 Below is the current amount spent for each of the five program areas designated by USAID/Madagascar. The amount spent is allocated to each of the areas based on the percentage assigned. Table 8: Amount spent by program area, June 2014 Program Area Amount Spent Percent Water 4,095,702 20% MNCH 7,372,264 36% FP & RH 7,167,478 35% MALARIA 1,228,711 6% NUTRITION 614,355 3% TOTAL 20,478,510 100% 47 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program This budget data together with the schedule information allows us to derive the following: a) earned value b) projected earned value c) over-budget or under-budget status for the activities conducted to date. Earned Value Management (EVM) SCHEDULE A) Current percent complete: .97 + .84 + .96 / 3 = 92% The current percentage complete of years 1-3 is the average of Years 1, 2 and 3. B) Total percent complete= .61 x.92= 56% The life of the project is 60 months. The information provided-to-date is 37 months. We divide 60 months (project life) by 37 months (time incurred) to derive 61%. At 37 months, the completion rate is 92%. Therefore, the project is 92% of 61% complete, or 56% complete. If the activities of the first 3 years had been 100% complete instead of 92% complete, then the project would be on track: 61% complete at 37 months instead of 56%. C) Behind schedule: (Expected completion) 61%-(Actual completion) 56%=5% Subtracting the actual percent of activities completed from the expected percent completed for this stage of the project yields a difference of 5%, indicating the project is slightly behind schedule. COSTS D) EV= 34,999,935 x .56 = USD 19,599,964 The Earned Value is calculated by multiplying the project funding ceiling by the percent of activities completed: EV= Ceiling x Total % complete. At 56% complete, the planned value is USD 19,599,964; however, the actual spent as of June 30, 2014 is USD 20,478,510. E) Projected EV = 34,999,935 x .44= USD 15,339,971 The projected earned value calculates the amount of spending if the spending rate remains the same for the remainder of the project. Since the project is 56% complete, keeping a constant spending rate will be 44% or the project time remaining multiplied by the original budget. The actual amount spent (20,478,510) + projected EV (15,339,971) indicates that the total project amount spent at the current rate will be $35,818,481. F) The project is over budget: USD 878,546 The amount the project is over- or under budget is calculated by subtracting the EV (USD 19,599,964) from the actual amount spent (USD 20,478,510). However, it should be noted that spending for this project will not continue at the same rate in Y4 and Y5. Much of the initial spending was start-up costs 48 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program that will not repeat in subsequent years. The Y5 budget is estimated at 2 million and considerably lower than previous years, as Y5 is closing the project where costs are much lower than start up. Behind Schedule Delays Below are some examples of project delays. Details on items that were delayed are listed in a separate document that is available upon request.  Start-up: Overall delay in starting up field activities related to remoteness and lack of prior program assistance in MAHEFA zones resulting in difficulties identifying local qualified human resources and establishing and furnishing regional offices. Another area of difficulty is with regards to identifying and contracting local NGOs since they either did not exist or were not qualified which led to the contracted NGOs often having slow start-ups.  NGO capacity: Low technical and managerial capacities among the NGOs necessitated an increased focus on capacity building, which, while essential to help build up and strengthen local partners as an investment towards long term sustainability, slowed implementation of activities. Partnerships with some NGOs who were not able to perform had to be terminated which further delayed implementation in some areas.  Commodities, procurement and supply points: The process of supplying commodities and responsibilities for MAHEFA, USAID and USAID’s social marketing partner were not clear from the beginning of the program, and MAHEFA also experienced delays in identifying appropriate staff and developing systems for handling procurement. For CHW re-supply, the existing system of Supply Point Agents (Points d’Approvisionnement, PAs) needed much reinforcement at the time that MAHEFA started. At the beginning of MAHEFA, 277 new supply points had to be established by PSI to serve the districts and communes of MAHEFA; by early in 2013, less than half of these were functional. These needs had to be met by PSI, not MAHEFA but had a significant impact of MAHEFA’s ability to support CHWs. By FY2014, the majority were functional.  Restrictions on interacting with GOM: Although MAHEFA was always aware of and respected U.S. government restrictions on interacting with the GOM, the restrictions also caused delays in some areas in achieving results due to instances where GOM activities received preferential treatment and where CSB staff have not been available or able to help CHWs complete their stages pratiques.  Water infrastructure: Initial delays for building the infrastructure for wells due to waiting for documentation for well construction to be added to the award (a long waiting period for the Cooperative Agreement to be amended to enable well construction to be undertaken under the award).  Seasonal changes in access: Progress is slower during Madagascar’s rainy season, generally from November-March. The CHWs’ ability to get supplies, receive supervision, attend meetings, and report data during these months is limited, especially in areas most affected by flooding.  Training: MAHEFA experienced delays in meeting the high training needs of all CHWs in a timely way, especially in areas that experienced delays in completing stages pratiques. Significant level of effort was expended during PY 2-3 to complete activities and procurement to support more than 3000 CHWs that were supposed to receive appropriate training and start-up kits from the NSA Project. 49 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program  Miscellaneous: The PY1-PY2 work plans were approved with a scale-up model that would enable MAHEFA to complete basic research for program design before scaling up to all regions of MAHEFA. In other words, this was not a delay but a design feature. Discussion According to the EVM, MAHEFA has completed 56 percent of its planned activities during 61 percent of its project duration, implying 5 percent of planned activities delayed or behind schedule. If current spending rates are maintained, the program is projected to be USD 818,546 over budget; however, spending is expected to decline significantly during the final two project years, as most training and infrastructure improvements will have been completed. IV. LIMITATIONS AND CONCLUSION Limitations This evaluation had several limitations. While data were collected from numerous individuals, it is just a small percent of those reached by the program and these findings cannot be considered representative of the entire population of those served by MAHEFA. In addition, this is a qualitative evaluation and all findings are the perceptions or thoughts of those interviewed. While many spoke about improved health outcomes in their communities, these are self-reported and are not verified by quantitative data, nor can we assume all changes are entirely attributable to MAHEFA in the absence of a study designed to answer the question of the impact of the program. Finally, some of the findings, particularly at the higher levels (NGO, regional and national levels) are based on interviews with a small number of individuals, so these results in particular should be interpreted with caution. Conclusion The MAHEFA program has been developed on a strong foundation. The program has been guided by a theoretical and practical orientation to activities grounded in ethnographic research, barrier analysis, annotated bibliographies of best practices and pilot studies of new interventions. It has implemented transparent processes in bidding, staff recruiting, NGO contracting, and its MOUs with PSI and MSIM, as well as other development partners who were not included in this evaluation. The program has been adaptive and responsive to changing contexts, quickly recognizing the need to reduce the TA to CHW ratio by hiring additional TAs, providing more intensive technical support to NGO partners, and reducing the number of regions in which it works in response to its Remedial Plan. More than one respondent spoke to the challenges of introducing a program on such a large scale in “virgin territory”, i.e., parts of Madagascar that had never before experienced an integrated public health program supported by USAID. Political and technical challenges were many, including establishing relationships with the public sector during a period of U.S. Government non-engagement policies, identifying NGO partners with technical and administrative capacity to support the program, finding and recruiting staff with appropriate skills and knowledge of the various regional contexts at all levels, and reliance on an increasingly decentralized structure relying on the performance of volunteers at the most very basic levels. Overlay these considerations with a poor transport system, distant communities rendered inaccessible for several months of the year, stark poverty, and deeply engrained social norms in opposition to project interventions, and it is remarkable that the program has achieved as much as it has. Thousands of CHWs have been trained and certified, hundreds of TAs now offer supervision, NGO partners have been strengthened, and community members have benefited from improved access to health information and services. 50 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program APPENDIX 1: MAP OF INTERVENTION ZONES FOR USAID/MAHEFA 51 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program APPENDIX 2: SUMMARY OF SITE VISITS Totals/Averages (totals are given as number of yes responses) Dates of visit 6/10-7/12 Independent site (Y/N) 7/10 Avg. Distance from the CSB 5.4 Avg. Distance from the commune 6.9 Avg. Distance from the district town 18.9 Avg. Distance from national road 3.2 Transport means from the district car: 10/10; Motorboat: 2/10; Charrette 1/10; Canoe: 1/10 Transport means from the commune car: 10/10; on foot: 1/10; Motorboat 2/10; cart 1/10; Canoe: 1/10 CHW #1 (Man /woman) 6 men, 4 women Received trainings PCIMEC: 9/9; PF4 9/9; DEPO 7/9; Nutrition 9/9; WASH 9/9 (1 absent) Outstanding training DEPO 2/9 CHW #2 (Man /woman) 2 men, 6 women (1 absent) Received trainings PCIMEC: 6/7; PF4 7/7; DEPO 4/7; Nutrition 7/7; WASH 7/7 (2 absent) Outstanding training Depo 3/7, PCIMEC 1/7 Schedule for receiving patients, posted outside (Y/N) 10/10 Tippy-tap (Y/N) 10/10 Toilet (Y/N) 8/10 Registers and forms Register for children under 5 (Y/N) 10/10 Checklist form for children (Y/N) 10/10 Register for women (Y/N) 10/10 Register for FP (Y/N) 10/10 Checklist form for PF (Y/N) 10/10 Register for sensitization (Y/N) 10/10 Additional report register (Y/N) 10/10 Site report register (Y/N) 10/10 Other tools Timer (Y/N) 10/10 Scale for children (Y/N) 10/10 Isothermal bag (Y/N) 10/10 52 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program Medicines and others Paracetamol (Y/N) 10/10 Cotrim (Y/N) 10/10 Pneumostop® (Y/N) 10/10 Actipal (Y/N) 10/10 ViaSur (Y/N) 10/10 RDT (Y/N) 10/10 Gloves (Y/N) 10/10 Safety box (Y/N) 10/10 PILPLAN (Y/N) 10/10 CONFIANCE (Y/N) 10/10 Condoms (Y/N) 9/10 Cyclebeads 10/10 Sur’Eau (Y/N) 10/10 53 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program APPENDIX 3: SUMMARY OF VISITS TO WATER POINTS Date Fokontany Commune District Region Notes 10/06/2014 Ambalavola Diego Suarez Diego Suarez Diana - First new Kiosk :  finished on February 2014  Member of AUE: a president, a president deputy, a secretary, chief of Fokontany and a keeper  Open : 5:00 am￾9:00 am, 11:00 am-2:30pm, 4:30pm-5:30 pm  For 50 household  200l/day for 4000ar per household per month - Second kiosk: not yet functional , waiting for JIRAMA decision 11/06/2014 Ambodimanary Sakaramy Antsiranana II Diana Rehabilitated well - Functional - Refer to FGD_006 13/06/2014 Ambolodimaka Ambodimanga Ramena Ambanja Diana 3 new wells under construction  Refer to FGD_010 19/06/2014 Ampombibitika Boriziny II Boriziny Sofia Rehabilitated well  Refer to FGD_019  The pump does not work 07/07/2014 Antsakoameloka Morondava Menabe New Kiosk  Built in March or April 2014 54 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program  Only 6 household use this kiosk (it is supposed to be used by 44 household)  100 Ariary for 20l  The president of AUE was absent  This kiosk is implemented inside someone’s courtyard  People prefer to use wells 07/07/2014 Andabatoara Morondava Menabe New kiosk  Built on 30 April 2014  50 ariary / 20l and 30 /15l: to pay JIRAMA invoice  Open: 6:00am￾12:00 and 2:00pm￾6:00pm  36 household use this kiosk  People use wells and some people 08/07/2014 Tanambao Marofototra Mahasoa Bemanonga Morondava Menabe New well  No problem  People are satisfied  600 Ariary per month/ household 55 Mid-Term Evaluation of Malagasy Heniky ny Fahasalamana (MAHEFA) Program