Catholic Relief Services - USCCB Child Survival and Health Grant Project Implemented by: CRS-Nicaragua, Caritas Matagalpa Diocese and the Ministry of Health-Nicaragua Cooperative Agreement Number: GHN-A-00-08-00007-00 Grant Period: October 1, 2008 to September 30, 2012 Project Location: Municipalities of Matiguás and Río Blanco in the Province of Matagalpa; Municipality of Waslala in the North Atlantic Autonomous Region and Municipality of Bocana de Paiwas in the South Atlantic Autonomous Region Final Evaluation September 2012 Principal Author: Sandra Wilcox, Consultant ii Table of Contents Acronyms ............................................................................................................. iii A. Executive Summary.......................................................................................... 1 B. Overview of the Project .................................................................................... 7 C. Evaluation Assessment Methology and Limitations..................................... 11 D. Data Quality and Use..................................................................................... 11 E. Presentation of Progress toward Achieving Project Results .........................15 F. Discussion of the Progress toward Achievement of Results .........................20 G. Assessing the Operations Research Study.....................................................36 H. Potential for Scale-up....................................................................................38 I. Conclusions and Recommendations...............................................................39 ANNEXES Annex 1: Learning Briefs/Results Highlight Annex 2: Publications and Presentations Related to the Project Annex 3: Project Management Evaluation Annex 4: Work plan Table Annex 5: Rapid CATCH Table Annex 6: Final KPC Report Annex 7: CHW Training Matrix Annex 8: Evaluation Team Members Annex 9: Evaluation Assessment Methodology Annex 10: Persons Interviewed and Contacted during the MTE Annex 11: Final OR Report Annex 12: Project Data Form Annex 13: Grantee Plans to Address Final Evaluation iii ACRONYMS AINMA Integral Care for Women, Children and Adolescents AMTSL Active Management of the Third Stage of Labor ANC Antenatal Care BCA Behavior Change Agent (Promotor de Cambio de Comportamiento) BCC Behavior Change Communication BF Breast Feeding BFHI Baby Friendly Health Initiative BL Baseline Caritas Social Services Arm of the Catholic Church CHA Community Health Agent (Salubrista) CHW Community Health Worker (Brigadista) CIES Center for Health Research and Studies CSP Child Survival Project CRS Catholic Relief Services DIP Detailed Implementation Plan EBF Exclusive Breastfeeding EOC Emergency Obstetric Care ETB Emergency Transport Brigade (BTEO) FGD Focus Group Discussion HCI Health Care Improvement Project HF Health Facility (Center and Post) HFA Health Facility Assessment HIS Health Information System HQ Headquarters (of CRS in Baltimore, Maryland) KPC Knowledge, Practice and Coverage LQAS Lot Quality Assurance Sampling M&E Monitoring and Evaluation MINSA Ministry of Health of Nicaragua MNC Maternal – Neonatal Care MOH Ministry of Health MTE Mid Term Evaluation MWH Maternity Waiting Home (Casa Materna) N-IMCI Neonatal Integrated Management of Childhood Illnesses NGO Non-Governmental Organization NICASALUD Nicaraguan NGO Health Network OR Operational Research PAHO Pan-American Health Organization SICO Community Information System (MINSA & Project) SILAIS Departmental level of MINSA TBA Traditional Birth Attendant (Partera) UNAN Autonomous University of Nicaragua UNICEF United Nations Children’s Fund USAID United States Agency for International Development WHO World Health Organization 1 A. Executive Summary Catholic Relief Services (CRS) received funding from USAID for the implementation of a four year Child Survival Project in Nicaragua (October 2008-September 2012). Goal: Contribute to the reduction of maternal and neonatal morbidity and mortality in the municipalities of Matiguas, Río Blanco, Paiwas and Waslala of the Matagalpa SILAIS by 2012. There are 125 target communities and 13 Ministry of Health (MINSA) facilities. The project was implemented by three principal partners. CRS led efforts in institutional strengthening for MINSA facilities under Objective 2: Increased families’ access to quality maternal and neonatal services and Caritas Matagalpa Diocese implemented community activities in coordination with MINSA under Objective 1: Improved knowledge and behaviors for maternal and neonatal health among families and communities. The overall project plan was to implement with MINSA a complete “package” of household, community and health facility (HF) activities to strengthen maternal neonatal care. These activities included: neonatal IMCI at both clinical and community levels, birth planning, lifesaving skills, nutrition during pregnancy, breastfeeding, institutional strengthening for MINSA (training, equipment, and quality improvement). The main implementation strategy for working at the community level was the formation, training and support given to a network of volunteers in activities specific to improving maternal newborn care (70% level of effort) and nutrition (30% level of effort). At the community level, volunteers counseled families, provided group education, and monitored and referred women for required services. The CSP included one innovation: Engaging Men to Improve Care-Seeking in 20 communities where 61 behavior change volunteers have been trained. Qualitative research was conducted to understand the situation and opinions of local men, to identify barriers and facilitators, and to negotiate new behaviors. The formative research methodology was developed by NICASALUD, and the Center for Health Research and Studies measured impact through a series of quantitative surveys. The project also focused on another activity: increasing access to emergency health services in 25 isolated rural communities. This activity was implemented through the use of higher level volunteers trained in lifesaving skills, emergency care, neonatal IMCI, and birth planning. The volunteers have been equipped with essential drugs and supplies for primary care and emergencies and are monitored monthly by the MOH health facility staff. KPC and FE results demonstrated that the project achieved its planned results. All planned activities have been completed. Although the project was behind schedule at the mid-term, they were able to catch up and conclude planned activities. Positive findings during the FE include:  Key design factors that contributed to program success: good coordination with the MOH, updating volunteer skills, MINSA and volunteer ownership of the CSP strategy, and community acceptance of the project strategy.  The KPC and FE interviews revealed: increased use of health services, institutional births (88- 90%), use of MWHs, and ANC (76%).  Two especially positive interventions were cultural adaptation of births and involving men in maternal health, including accompanying the woman to antenatal care and during childbirth.  The network of volunteers is active and strengthened and has improved links with MINSA.  The KPC found that the percent of women with birth plan cards rose from 0 to 82% in Matiguas and 0 to 69% in Rio Blanco and Paiwas. 2  The KPC demonstrates that mothers’ knowledge of pregnancy and delivery danger signs rose between 50 to 70 percentage points while signs during postpartum rose 20-40 points. Fathers’ knowledge of these danger signs rose at the same levels, with the exception of knowledge of postpartum signs, which was higher among fathers (60 and 40%).  The emergency response system has been strengthened through increased community awareness of danger signs, training of 5 Emergency Transport Brigadistas per community, referrals, community planning for emergencies and improved emergency response at health facilities. All 100 communities have emergency plans.  Sustainability: The project has made a major effort to classify all communities, of which 85 were certified by MINSA and considered A–level, as being able to sustain themselves, plan, and meet community health needs. All 100 communities have mutual support agreements signed with MINSA.  Improved supervision of volunteer activities by MINSA, particularly CHWs, TBAs, and CHAs  All 100 communities have funds for obstetric and other emergencies, which they have raised through community savings funds and raffles. Over the life of the project these communities have maintained funds amounting to more than 150,000 Córdobas.  Basic medicines and equipment are available in health centers for MNC Conclusions A lesson learned was that the CSP’s work to involve men in health care coincided nicely with the government’s focus on its policies to humanize and culturally adapt birthing practices at the MOH service facilities. Thus the CSP activity assured that the MOH norms were implemented. One of the questions voiced by staff, MINSA, SILAIS and the communities was why the innovation projects were implemented without available resources to further expand them. One of the insights gained by the staff while conducting the innovation was the importance of the educational method used to approach men. They found that as men became more supportive of their family’s health needs and helpful at home, the levels of family violence diminished. Also, the use of religious leaders as BCAs was important as they were influential and also good family role models. Although the quantitative CIES evaluation of the innovation activity did not show a difference between case and control communities in terms of ANC visits, overall there was a significant increase in the number of women attending ANC during pregnancy. Additionally, there were significant differences and improvements in terms of joint decision-making and women who sought care for themselves and their children together with their husbands. Recommendations Given that the project ended in September 2012, it may not be possible to conduct further activities. The project implemented its plans for certifying communities and closing the project as of September 30th. However, if resources become available for some additional steps, the following are recommended: Disseminate results  The project has much to contribute to the Child Survival knowledge base, most notable being the experience involving men in care seeking and shared decision making. Also, the project’s 3 experience with the humanization and cultural adaptation of the birthing process and use of MWH experience should be documented and disseminated.  USAID/Nicaragua recommended that the project’s results be presented to central MINSA and also to the medical schools, which are lacking in strategies for community involvement in health care services.  The CSP’s successful CHA experience needs to be presented to central MINSA as a possible strategy for their implementation of the new “model for community family health” initiative (MOSAFC).  CRS should continue to present results of the CSP experience at CORE Group meetings and other forums.  CRS should consider including the CSP experience as one of its “Most Significant Change” case studies using its methodology to capture qualitative results from the project, particularly the men’s innovation activity. Other recommendations  If the innovation activity were to be replicated and taken to scale, the staff would have to drastically reduce the formative research phase during which the methodology was extensively tested and materials validated over a period of 18 months. They would also have to address whether they would use so many incentives, which add cost and hamper sustainability.  It would be helpful to have a qualitative evaluation of the attitudes and practices of families participating in the men’s innovation intervention that could explain more clearly why the values and perceptions have changed among women and men and provide more information regarding case and control group differences.  USAID should review some of the KPC indicators. For example, the question that asks mothers if their newborn received proper cord care at birth is difficult for mothers to answer, particularly if they had hospital deliveries, since it is likely they may not have been paying attention to their newborns during that time. Questions about the third stage of labor are also difficult for mothers to answer. 4 Table 1: Summary of Major Project Accomplishments SO1: Improved knowledge and behaviors for maternal and neonatal health among families and communities Project Inputs Activities Outputs Outcomes • Radio Time, IEC and BCC Materials • Community volunteers (CHW, TBA, BCA, ETB) MINSA and project staff • Birth Plan and SICO forms • Equipment for ETB • Partners: CIES, NICASALUD • Contracting of NICASALUD and CIES to implement and measure innovation for improved involvement of men in MNC • Implementation of four MINSA strategies for MNC (Birth Planning, Community Neonatal IMCI, Life Saving Skills, Nutrition during Pregnancy Safe Motherhood & breastfeeding) • Educational activities through home visits, counseling, and group educational sessions • Monitoring and supervision • Implemented community classification and certification system for sustainability. • Communities with functioning ETBs with equipment and emergency savings plans • Communities with functioning SICO • Trained volunteers referring, educating and monitoring pregnant women and newborns • Increased knowledge of danger signs and improved practices • Involvement of men in MNC and care seeking • Referral/counter-referral system • Accredited all 1,400 volunteers and certified all 100 communities signed by MINSA • Increase in use of services: ANC, institutional births, MWH, referrals • Anecdotal reporting of decreased maternal and neonatal mortality • Increased MNC knowledge among all target groups; strengthened all community volunteer networks; strengthened community￾MINSA relationships; upgraded community management capacity. • Anecdotal reporting of decreased domestic violence SO2: Increased families’ access to quality maternal and neonatal services Project Inputs Activities Outputs Outcomes • IEC and BCC Materials • Trainers • MINSA staff in 3 health centers and 10 health posts and project staff • MWH staff and facilities • Community volunteers (CHAs and TBAs) • MINSA protocols and quality standards tools • Partners: UNAN, HCI • Essential equipment and supplies in HFs and MWHs for MNC • Implementation of innovation for improved community management of MNC by use of highly trained CHAs • Cultural adaptation of birthing process including training in HFs and MWH • Training for HF staff in emergency obstetric and neonatal care, IMCI, ANC, natural FP, EPI other project strategies • Monitoring of quality standards and supervision • Refresher meetings • Equipping HF with essential supplies for MNC and some remodeling of HFs • Establishment of Center for Training of health staff in Matiguas • Trained MINSA staff providing improved maternal and neonatal services • HFs and MWHs offering improved cultural services; traditional diet, position during delivery, presence of family at birth • Increased emphasis on recognition of danger signs; pregnancy, delivery, postpartum, neonatal • Improved quality of care for MNC, including counseling during ANC • CHAs providing services in isolated communities • Increase in use of services: ANC, institutional births, MWH, referrals • Anecdotal reporting of decreased maternal and neonatal mortality • Prevention of the first delay (improved knowledge of danger signs and prompt treatment-seeking behavior). Also 2nd delay (establishment of functional ET teams), and 3rd delay by improving HW EONC skills. 5 • Basic supplies and equipment for CHAs • Reactivation of QI teams Table 1A: Summary of Major Accomplishments of Innovation Project for Men’s Behavior Change SO1: Identify key factors, perceived benefits and obstacles that influenced men’s behaviors regarding joint decision making for care seeking and participation in care of their partners and children and determine their willingness to adopt new behaviors and put them into practice (formative research). Inputs Activities Results Outcomes • Project staff and Ministry of Health staff. • Training • Forms for data processing • Development of qualitative research methodology • Training of Caritas technical staff and MOH staff. • Collection and processing of data from 14 focus groups and 32 individual interviews. • Analysis and application of data findings • Worked with men in probing phase • Selection of key behaviors • 15 technical staff trained in innovation methodology • Identification and listing of key change behaviors. • Elaboration of research protocol • Identification of resistances, barriers, enablers and motivating factors for changing men’s behaviors SO2: Development of an approach for application and supervision of behavior change interventions that promote change and facilitates development of BCC strategies, tools and materials. Inputs Activities Results Outcomes • Forms • Project staff • Methodology for organizing community workshops • BCC materials • Development of intervention strategy • Development of forms and check-lists • Pre-testing of forms and checklists • Regular review meetings with technical staff • Review of materials and consultation with communities and volunteers • Training of leaders through 8 workshops • Selection of BCC volunteers (3 per community) • Training of BCC volunteers • Development and pre-testing of BCC materials • Community intervention strategies • Definition of key activities • Approval and agreements on intervention reached with communities and municipal authorities. • BCC promoters selected in 20 communities • Promoters trained • BCC materials developed • Broad community interest and acceptance of the strategy • Active community participation in selection of promoters and preparation of materials • Interest in project expressed by municipal authorities, MOH, MOE and national police. 6 • SO3. Conduct small scale pilot program to test the feasibility of behavior change activities with men Inputs Activities Results Outcomes • Community BCC promoters • BCC materials • Forms for HIS • Supervision Guide • Community and municipal launch activities • Elaboration of community census • Mapping of communities, utilization of laminated posters to identify which behaviors the men’s role includes at home, in care￾seeking, acting on danger signs, joint decision making, etc. • Negotiating behaviors to be changed • BCC promoter follow-up w/ targeted men • Meetings, education talks, assemblies with communities • Coordination with MOH and MWHs • Project monitoring of intervention activities • Census and maps of 20 communities • Active participation of 303 men • 651 behavior changes negotiated and implemented with men in target communities • Launch of BCC strategy in 20 communities and 1 municipality • Functioning information system • 155 men accompanied their wives for prenatal visits • 51 men accompanied their wives during delivery • The MOH permits men to participate in prenatal visits and deliveries. 7 B. Overview of the Project Catholic Relief Services received funding from USAID for the implementation of a four year Child Survival Project in Nicaragua. Funding was provided from October 2008 until September 2012. The project originally had two interventions; maternal and neonatal care (MNC) with 70% level of effort and nutrition with 30% level of effort. However, after the mid-term evaluation, the nutrition component was eliminated, per MTE recommendations. The project was implemented by three principal partners. CRS is leading efforts in institutional strengthening for Ministry of Health (MINSA) facilities and staff under Strategic Objective 2 and Caritas Matagalpa Diocese is implementing community activities in coordination with MINSA under Strategic Objective 1: Results Framework Goal: Contribute to the reduction of maternal and neonatal morbidity and mortality in the municipalities of Matiguas, Río Blanco, Paiwas and Waslala of the Matagalpa SILAIS by 2012. SO1: Improved knowledge and behaviors for maternal and neonatal health among families and communities SO2: Increased families’ access to quality maternal and neonatal services. R1.1: Improved practices of positive health behaviors among pregnant women and mothers R1.2: Increased uptake of immediate emergency care among pregnant women and mothers with sick newborns Strategies/Activities: IRs 1.1.1., 1.2.1 and 1.3.1 Community Mobilization: organize community health workers and TBAs, birth plan. Organize and equip Emergency Transport, establish a referral system. Revitalize the Municipal Health Commissions; engage the municipal government in emergency transport plans IRs 1.1.2 and 1.3.2 Behavior Change Communication: counseling during home visits, broadcast radio messages, work with men, and conduct operational research with male partners of pregnant women. Promote MELA and IRs.1.1.3 and 1.2.2 Quality Assurance: train volunteers in birth plan, counseling, danger signs for pregnant women and newborns; essential newborn care, monitoring of community volunteers, train staff from maternity waiting homes. R2.1: Improved MOH staff skills during ANC obstetric/neonatal care. R2.2: Improved Community Health Workers and Traditional Birth Attendant skills in using birth plan, home-based life saving skills and Neonatal IMCI to provide care to pregnant women and newborns Strategies/Activities: IRs 2.2.1 Community Mobilization: Organize TBAs and CHWs, monitor community volunteers. Linkages among CHWs, TBAs and health posts/centers IRs 2.1.1 and 2.2.2 Quality Assurance: train MINSA staff in essential obstetric and neonatal care, counseling on danger signs during pregnancy, delivery, postpartum period, newborns and maternal nutrition. Train TBAs and CHWs in home-based life saving skills IRs 2.1.2 and 2.2.3 Establish a referral and counter-referral system. R1.3: Improved nutritional practices among pregnant, lactating women and newborns. 8 Project location The project was implemented in the municipalities of Matiguás and Río Blanco in the Province of Matagalpa; Municipality of Waslala in the North Atlantic Autonomous Region and Municipality of Bocana de Paiwas in the South Atlantic Autonomous Region. There are 125 target communities: all activities were implemented in 100 communities and 25 received the Community Health Agents (CHAs) activity only (see Section E). The CSP worked with three MINSA Health Centers and 10 Health Posts in Matiguas, Paiwas and Rio Blanco. The CSP also provided limited support to the Health Center and Maternal Waiting Home (MWH) in Waslala. Project beneficiaries MatiguasPaiwas y Rio BlancoTotal Beneficiaries Infants < 12 months: 1,150 1,993 3,543 Children 12-23 months: 1,175 2,056 3,231 Children 0-23 months: 2,325 4,049 6,374 Children 24-59 months: 3,633 6,342 9,975 Children 0-59 months: 5,958 10,391 16,349 Women 15-49 years: 11,124 16,646 27,770 Population of Target Area: 44,971 68,589 113,560 Communities 62 26 P & 37 RB 125 Implementation strategy The main implementation strategy for working at the community level was formation, training and support to a network of volunteers: Community Health Workers (CHW), Traditional Birth Attendants (TBA), Community Health Agents (CHA) and Behavior Change Agents (BCA) in activities specific to improving maternal newborn care (MNC) and nutrition. Other strategies included: community organization and mobilization; household, community and MINSA behavioral changes; and strengthening MINSA capacities. The MNC component addressed the three delays that contribute to maternal and neonatal morbidity and mortality: • the delay in recognizing danger signs and decision making through the behavioral change strategy and the principal innovation of Engaging Men to Improve Care-Seeking; • the delay in seeking help through community organization and the formation of Emergency Transport Brigades (ETB); • and the delay in diagnosing and providing timely care through the strengthening of MINSA capacity and by improving community response to maternal and neonatal complications. The program implemented the nutrition component through continuously improving the quality of antenatal care (ANC), focusing on improving the capacity of health care personnel to provide counseling to pregnant women regarding the frequency and quality of foods in the diet, counseling mothers on breastfeeding and complementary feeding during ANC and providing iron and folic acid tablets and Neonatal Integrated Management of Childhood Illnesses (N-IMCI) care. At the community level, TBAs and CHWs counsel women during home visits and provide group education to mothers and monitor and refer pregnant and postpartum women for required services. The overall CSP plan was to implement with MINSA a complete “package” of household, community and health facility (HF) activities to strengthen maternal neonatal care. These included: • Neonatal IMCI at both clinical and community levels 9 • Birth Planning (Plan de Parto) including support for use of MWH and cultural adaptation of birth (includes men’s or other person’s accompaniment at delivery, flexibility of delivery positions, etc.) • Life Saving Skills • Nutrition During Pregnancy • Breastfeeding • Institutional Strengthening for MINSA (training, equipment, Quality Improvement (QI) process) The project provided training to MINSA staff in N-IMCI and Emergency Obstetric Care (EOC) while lending technical assistance to implement systems to monitor and improve the quality of maternal and newborn care. After the MTE, the project took the documented experience and procedures of the formative research activities for the innovation: Engaging Men to Improve Care-Seeking and used it to implement the innovation in 20 communities of Matiguas, Rio Blanco and Paiwas as a test for the potential for scaling up. During the final year, the project has made attempts to share this strategy with MINSA at the SILAIS (departmental level of MINSA) level to discuss experiences, lessons learned and results with the hope of encouraging replication by other municipalities and agencies. A key element of the CSP was a behavior change strategy to increase pregnant women’s and their partners’ knowledge of danger signs and healthy practices. The strategy included several different methodologies. At the household level, trained TBAs use birth plan formats to counsel and negotiate with pregnant women and their partners about the application of new practices related to how to plan for delivery and the importance of seeking ANC, institutional deliveries, and postpartum and newborn care. The TBAs gave mothers a birth planning pamphlet early in pregnancy and encouraged them to do all of the activities listed in the pamphlet as well as save money for their deliveries and expenses for their new baby. These messages were also reinforced by local radio station programs. These birth planning activities were reinforced by the MOH norm to “culturally adapt the birthing process” to the techniques and traditions of community delivery practices. Although this is an official MOH policy, the project implemented it at the municipal level by training personnel, and in some areas providing supplies and equipment. Women can now choose who they want to accompany them to delivery. For example, it can be a family member, husband, or TBA. In addition, she can choose her delivery position, food she eats, whether or not to be shaved, etc. The project has supported the implementation of this practice in all target facilities through staff training and supplying MNC supplies and equipment, including furniture and ropes for delivery in traditional sitting positions. Another activity, Improving Community Response to Maternal and Neonatal Complications, was originally proposed as an innovation, but was considered by USAID to be a program activity. The objective of this activity was to increase access to emergency MNC in 25 rural communities that are more than three hours travel from the closest HF. The project identified 25 CHAs and trained them on lifesaving skills, emergency care to stabilize pregnant women and newborns before transferring them, neonatal IMCI and other skills outside the scope for typical CHWs. The CHAs offered MNC and attended other emergencies in two to four assigned communities near their homes. The project equipped them with essential supplies, drugs and materials for emergencies and they were monitored directly by the MOH staff. To ensure sustainability, CHAs did not work directly with the Caritas staff, but worked closely with MINSA health posts and centers. The CHAs actively sought direct support from MINSA by asking them to supply small medical kits and provide further training in skills such as suturing. A few CHAs have dropped out of the project, often migrating for work. By the end of the project there were 16 active CHAs, though some were expected to return this past fall when the work season is over. 10 The CSP included one innovation: Innovation: Engaging Men to Improve Care-Seeking in 20 communities of the municipalities of Matiguas, Rio Blanco and Paiwas. Initially, a series of focus groups and interviews were conducted to better understand the situation and local men’s opinions, to identify barriers and facilitators from their perspectives, and to negotiate with them regarding new behaviors that would contribute to the health of their families. The formative research stage lasted for 18 months and consisted of developing behavioral techniques and tools and then testing them in the communities to gauge their acceptability. The formative research methodology was developed by NICASALUD, and CIES (Center for Health Research and Studies) measured impact through a series of surveys (see annex 11 for final report). After the research stage, and for another 18 months, the field staff and BCAs continued implementing the BCC strategies and working with the men in their respective communities. Behaviors include joint decision making on where to go for ANC, delivery, and postpartum/newborn care; distribution of household tasks; and male participation during ANC and delivery. Approximately 60 BCAs have been trained to work with 10 families each. Additionally, the BCAs used sports events such as baseball and soccer as opportunities to promote BCC messages. Work plan Annex 4 includes the work plan table as presented in the DIP with an explanation of what was achieved or changed during the CSP. Integrating outreach visits with MINSA and supporting national vaccination campaigns were not specified in the work plan, but are activities which were implemented. Sustainability planning, annual meetings with partners, and USAID activities were added to the work plan after the MTE evaluation. Collaboration with USAID Mission CRS sees USAID Nicaragua as a critical partner. The two organizations share similar objectives in terms of strengthening MINSA’s efforts through complementary programming to maximize impact in Nicaragua. USAID provides an important opportunity to improve the scale of project interventions and a forum for exploring ways to scale up. Unfortunately, the USAID mission is closing its health office in 2013, so funding for scale up will not be available. During the FE debriefing in September with the mission, Dr. Valverde made several suggestions for disseminating project results to the central MINSA offices as well as the departmental MOH–SILAIS. She also suggested presenting the results to medical students who could benefit from knowledge of community approaches since their training is mainly clinical. She stressed that USAID was trying to do this with all their health projects in the hope that MINSA will continue the work. Unfortunately, there was not much time to do this as the project closed on September 30th 2012. According to the USAID representative, the project contributed to the Mission’s overall health objectives as related to improved health and decreased mortality for women and newborns. Dr. Boza from CRS was the representative to the networking meetings with USAID sponsored NGOs. USAID and CRS have had regular meetings, exchange of information and other collaborations such as USAID’s role in the approval of materials developed for the innovation. CRS has invited USAID to participate in and visit project activities. In mid-2012, USAID could not attend the community categorization and certification event. USAID had planned to attend the accreditation of volunteers event, however suspended that visit due to heavy rains. 11 Visits and Communication with USAID- Nicaragua Date Objectives Participants USAID Participants CRS Dec 2008 Discuss the pending project proposal for Child Survival Iván Tercero José Mendieta, Oscar Boza, Elena McEwan and Conor Walsh Jan 2009 Meet the new Project Development Specialist for the Health and Education Office USAID￾Nicaragua Mr. Callagan, US Ambassador to Nicaragua, Iván Tercero, Marianéla Corriols and other NGOs José Mendieta, Oscar Boza Feb 2009 Present results from baselines and discuss the DIP Marianela Corriols José Mendieta, Oscar Boza April 2009 NGO meeting to present advances of projects financed by USAID Marianela Corriols, Dr Valverde José Mendieta, Oscar Boza March 2010 To present results, objectives and planned activities of the project Marianela Corriols, Dr Valverde Hugh Aprile, Kristin Rosenow, Jose Mendieta, Elena McEwan, Oscar Boza March￾Nov 2010 Frequent email communications with Dr. Corriols to provide updates and to discuss ideas and suggestions related to the educational materials the project developed Jan 2011 Presented project advances at meeting for USAID health and education NGOs hosted by NICASALUD Dr. Klelia Valverde Jose Mendieta, Julio Valerio Feb 2012 Presented the Behavior Change project objectives and results to USAID Dr. Klelia Valverde, Alicia Julio Valerio, Jose. Mendieta C. Evaluation Assessment Methodology and Limitations The final evaluation (FE) was conducted during August and September 2012 by a multi-disciplinary team comprised of 16 members (see Annex 8 for team members). The team utilized a participatory methodology (see Annex 9 for a summary of MTE methodology) to elicit findings and conclusions regarding activities conducted during the project period. Additional interviews were conducted by an external evaluator and principal author of this report (see Annex 10 for a list of people interviewed). Final evaluation team conclusions and recommendations are included in this report and summarized in Section G Conclusions and Recommendations. There were no substantial changes in the CSP since approval of the DIP that required a modification to the Cooperative Agreement. The corrected M&E plan from the DIP is included in this report. Due to recommended changes to some indicators, and several errors in the original M&E matrix, a number of changes were made to the indicators at the time of the MTE. However, in accord with recommendations from MCHIP, the final KPC measured original indicators as well as the corrected ones. D. Data Quality and Use System effectiveness The CSP has developed several systems for Monitoring and Evaluation (M&E) that are effective in measuring progress towards project objectives. There are two sets of indicators that the project is tracking: the M&E Plan as presented in the DIP and indicators from the work plan. Some of the positive findings are that the CSP is using and supporting the MINSA community information system (SICO) and using the standardized Knowledge, Practice and Coverage (KPC) survey tool for evaluation. The main weakness of the system prior to the MTE was that it collected too much information and was burdensome for volunteers and field staff. According to the MTE, use of data for decision making was not a focus. However, the project staff claims that they did use process indicators 12 to prioritize activities for each FY. They also said that they made decisions regarding MOH staff training (analysis of charts by QI teams) and CHA training based on analysis of field data. The other problem was that there was not adequate staff for monitoring and supervision. This was remedied in July 2010 when Caritas contracted an M&E Supervisor (who was previously a field technician). CRS’ M&E coordinator began working with the CSP in September of 2009 and left in August of 2010. The CSP decided that they did not need to replace him given the strength of the Caritas staff. Evaluation activities A KPC survey using 30 cluster sampling was conducted at baseline and was repeated at the final evaluation in June/July 2012. The sampling frame was 100 communities. The MTE KPC survey report clearly described the indicator definitions, but there were some problems with the quality of the report. For example, according to the MTE survey results, more women were receiving “quality” ANC than those who actually received ANC. There were also some inconsistencies in that the expected KPC answers did not completely correspond to the educational messages being promoted by MINSA and the CSP. These were addressed after the MTE. The FE KPC survey revealed that some KPC questions may not have been appropriate to ask the target audience. For example, mothers were asked if their infant received appropriate clean cord care at delivery (indicator 2.2.2). This target (70%) was not met, and it is assumed by the KPC External Consultant, project and MINSA staff that the mothers who were asked this question did not remember whether they received appropriate cord care due to different factors. Mothers may not have been paying attention to what happened immediately after they gave birth, since they were preoccupied with the delivery. At the same time, a health facility assessment (HFA) conducted by the project in 2009 (baseline) and during the final HFA in July 2012, found that 100% of facilities in Matiguas and 90% in Rio Blanco and Paiwas provided appropriate care to newborns in accordance with protocols (including clean cord care). The same issue might be raised for indicator 2.1.9, active management of the third stage of labor, because the mothers might not be have been paying attention to the three actions during delivery and while recovering from the birth, but in the KPC these targets were met. In addition to the KPC survey, one of the main sources of information listed in the M&E Plan was the Health Facility Assessment (HFA). The CSP chose to not use a standardized HFA, but instead developed a combination of tools, which included observation of ANC, exit interviews and a quality checklist for chart reviews. The project also used MOH guidelines to assess MNC quality as part of the QA process. The results of this assessment were not available at the MTE, except where narrative information was included on ANC observations as part of the KPC Survey report. At the time of the FE, the final HFA had been completed and some findings are included in the M&E table 2. In follow￾up to MTE recommendations, some indicators were modified from the M&E Plan to align with MINSA quality indicators and were monitored through the QI tools used by the project. In the baseline KPC, three indicators were left out (2.1.1A, 2.1.2A, and 2.1.3A) of the survey so to secure this information, the Caritas staff included the questions in their internal monitoring and HFA assessment. The results are included in Table 3. There is also an issue with indicator 2.1.6, which measures the percent of children 0-23 months that received a postnatal visit by a health worker within 3 days of birth. Results indicate that the project fell short of this target by about 10 points in each site. This seems suspicious because the next indicator (2.1.7) measures the number of mothers who received postnatal visits within 3 days of delivery and here the indicators were surpassed in Matiguas and almost met in Rio Blanco/Paiwas. Since the visits would have been made to the HFs or by HF staff, it seems likely that both mother and infant would have been seen at the same time, so it is unclear why the neonatal visits were not done. Since all newborns were checked before being discharged from the HF 13 and 90% of deliveries were institutional, it is possible the gaps are a result of mothers not remembering whether they received a postnatal visit. Additionally, the FE team believes that mothers may remember better if they were checked than if their babies were checked. Also to be noted in the KPC is that the breastfeeding education and performance targets were not met and the “baby-friendly hospital” activities were suspended as recommended by the MTE team. This is because the MOH decided not to continue with the baby friendly hospital initiative. There were too many activities and requirements for the CSP to complete with their limited staff. For these reasons, the MTE team recommended that the BFHI component and the formation of breastfeeding support groups be dropped from the CSP. However, the MTE did recommend that the nutrition component be strengthened and include breastfeeding promotion. Training on breastfeeding and LAM was also given to MINSA staff and the volunteers continued to provide nutrition education messages in talks, counseling and on the radio. Monitoring activities SICO — MINSA’s national community information system — was supported by the CSP. The basic instruments for monitoring the Birth Plan were from MINSA and these tools will continue to be used in the future. Also, CHAs are using MINSA reporting forms which they deliver directly to the HF staff and give copies to the CSP. The HF staff tabulates results and sends the information on to the SILAIS. Additional tools were added by the CSP to measure project targets, such as community birth registries and registers of men participating in the behavior change innovation activity which was filled out by BCAs. In addition, with private funds, CRS tested the use of cell phones in 10 communities in Matiguas for reporting on project activities. This has proved to be an efficient reporting mechanism. There are also forms used for monitoring community home visits with target groups, which indicate when these were done with MINSA staff. The project’s main monitoring tool is the Annual Operating Plan which is based on the DIP, as well as the supervision guide and the aforementioned MINSA forms and registries. The project has also developed a number of checklists and manuals for the volunteers including a manual for CHWs to use for organizing group activities and education sessions. The project is also writing the curriculum it used for training the CHAs, which is currently being reviewed by the SILAIS in Matagalpa. The MTE noted that the project monitoring system was overburdened with too many complicated forms asking for excessive information. The quality of the data at the community level was questionable because many of the volunteers do not read and write and the tools were not well adapted for people with low literacy levels. As a result of MTE recommendations, the project HIS was simplified. The number of forms and indicators were reduced to align with M&E indicators and the volunteers with low literacy skills were able to complete them. The BCC materials in particular use lots of visuals and are easily used by low literate volunteers. Also in accord with the MTE recommendations, the project refocused on indicators in the M&E plan as opposed to the 100 indicators in the work plan. An MTE recommendation to improve monitoring was for the project to use Lot Quality Assurance Sampling (LQAS) annually as part of its supervisory system, to monitor progress. The project did use LQAS in year 3. The project developed a database containing information about community volunteers which includes contact information, personal information and training received. This database is shared with MINSA and per MTE recommendations has been used by MINSA to provide accreditation to CHWs. Through project support, the accreditation process was completed with MINSA in 2012. 14 Caritas staff has developed information files for each community which includes a community diagnosis, map, information about each volunteer, reports on field visits, monitoring information, information on the ETB (statutes for the brigade and use of emergency credit and a transportation plan), reports on outreach visits and other information. Health facilities have found this information to be helpful and some have started similar files. Use of information One of the weaknesses of the CSP identified by the MTE team was the flow of information. The CSP field staff plays a pivotal role in collecting information in the field and at the time the information was not always shared appropriately. The CSP participates in monthly MINSA Technical Council meetings to report on project activities. One of the strengths of SICO is having a feedback mechanism for communities to use the information collected for decision making. Per MTE recommendations, the data collected in the field by CSP staff has been shared with MINSA and the communities monthly and with SILIAS quarterly so that all partners were informed of project advances and weaknesses and participate in decision making. The CSP has come a long way in using project data for decisions since the MTE. At the time of the MTE, data was collected by volunteers for reporting the numbers of pregnant women and referrals made (using the MINSA system), but not necessarily used for decision making. Now, the monthly data is carefully analyzed with MINSA HF staff. Pregnant women or newborns that need to be followed are visited by volunteers and if necessary, MINSA staff and provided appropriate care. Volunteers go out of their way to track down and document referrals and counter referrals at health centers and health posts to make sure their patients have sought care and been treated appropriately. There is a good working relationship between volunteers and MINSA staff, so that if a patient will not go to the clinic, often the nurse at the health post can be relied on to help convince the patient of what needs to be done. 15 E. Presentation of Progress toward Achieving Project Results Presentation of Progress toward Objectives Table 2: M&E Matrix—Progress at Project End Project Indicators Data Source Matiguas Río Blanco/Paiwas BL Final Target BL Final Tar get S.O 1. Improved knowledge and behaviors for maternal and neonatal health among families and communities R1.1 Improved practices of positive health behaviors among pregnant women 1.1.1: Children age 0-23 months that were dried and wrapped with warm clothes or sheets immediately following delivery. KPC BL/ final survey 95.7 98 100 92.6 94 100 1.1.2: Mothers with child age 0-23m reported taking iron and folic acid supplements during the first trimester of pregnancy. * KPC BL/ final survey 47.4 85.9 60 44.3 61 60 1.1.3: Mothers with child age 0-23m who can name two or more danger signs during pregnancy. KPC BL/ final survey 6.3 79 50 10 60 50 1.1.4: Mothers with child age 0-23m who can name two or more danger signs during delivery. KPC BL/ final survey 13 71 50 11.1 51 50 1.1.5: Mothers with child age 0-23m who can name two or more danger signs during the postpartum period. KPC BL/ final survey 27.5 74 50 30.5 30.5 50 1.1.6: Men that live in the house of the mother interviewed who can name two or more danger signs during pregnancy. KPC BL/ final survey 16 81 40 15.4 66 40 1.1.7: Men that live in the house of the mother interviewed who can name two KPC BL/final survey 12.6 72 40 9.7 49 40 16 Project Indicators Data Source Matiguas Río Blanco/Paiwas BL Final Target BL Final Tar get or more danger signs during delivery. 1.1.8: Men that live in the house of the mother interviewed who can name 2 or more danger signs during postpartum KPC BL/ final survey 10.3 74 40 11.7 53 40 1.1.9: Mothers with child age 0-23m who completed the birth plan card. * KPC BL/ final 0 81.6 70 0 68.8 70 1.1.10: Mothers with child age 0- 23m who had savings during the pregnancy with their youngest child. KPC BL/ final survey 78.5 98.4 89 68.8 84 80 1.1.11: Newborns who received an examination within 48 hours of delivery* KPC BL/ final survey 50.3 82.7 66 44.6 67.9 61 R.1.2: Increased uptake of immediate emergency care among pregnant women and mothers with sick newborns 1.2.1: Mothers with child age 0-23ms who sought care within 24 hours in the health center when their newborns were sick.* KPC BL/ final survey 55.6 89.6 66 58.3 79 69 1.2.2: Mothers with child age 0-23m who can name two or more danger signs in newborns. KPC BL/ final survey 16.5 78 40 18.4 57 40 1.2.4: Fathers with child age 0-23m who can name two or more danger signs in newborns. KPC BL/ final survey 15.6 78 40 10 57 40 R.1.3: Improved nutritional practices among pregnant women and newborns 1.3.1: Newborns who were put to the breast within 1 hour of delivery and did not receive any pre￾KPC BL/ final survey 66.5 56.9 82 58.4 57 74 17 Project Indicators Data Source Matiguas Río Blanco/Paiwas BL Final Target BL Final Tar get breastfeeding foods.* 1.3.2 Children age 0-5m who were exclusively breastfed during the last 24 hours. KPC BL/ final survey 34.5 53.8 56 26.6 36 47 1.3.3 % Mothers with child 0 – 23m report eating more than normal during the last pregnancy and lactation.* KPC BL/ final survey Not collected 51.7 50 Not collecte d 38.3 50 Strategic Objective 2: Increased families’ access to quality maternal and neonatal services Result 2.1: Improved MOH staff skills during ANC obstetric/neonatal care 2.1.1: % Neonatal consultations provided according to the neonatal IMCI guides. BL/Final HFA/ IMCI checklist 0 100 30 0 83 30 2.1.2: % Mothers with child 0-23m receive counseling by health workers on breastfeeding. HFA/ IMCI checklist 4.5 33 70 32.5 30 70 2.1.3: % MOH personnel who have received supervision using the neonatal IMCI supervision guides.* BL/Final HFA/ IMCI checklist 0 66 50 0 66.6 50 2.1.1A: % pregnant women who filled out the 13 item in the HCBP card correctly BL/Final HFA checklist 55 70 90 50 53 90 2.1.2A: % women with obstetric emergencies who received treatment in accord with MINSA protocols BL/Final HFA/ IMCI checklist 70 100 90 60 80 90 2.1.3A: % of newborns who adequately receive immediate routine care according to protocol BL/Final HFA/ IMCI checklist 75 100 90 63 90 90 2.1.4: % Women with child 0-23m with 4 or more prenatal visits when pregnant with BL/Final Project HIS 41 76.9 70 53.4 56 70 18 Project Indicators Data Source Matiguas Río Blanco/Paiwas BL Final Target BL Final Tar get youngest child. 2.1.5: Mothers with child age 0-23m who delivered their youngest child in a health unit. * KPC BL/ final Project HIS 66.6 90.6 77 76.5 88 85 2.1.6: Children 0- 23m received a post-natal visit by a properly trained health worker within 3 days of delivery. * KPC BL/ final Project HIS 50.3 57.3 66 44.6 49 60 2.1.8: Mothers with child age 0-23m who received a postpartum visit from a properly trained health worker within three days of the delivery of their youngest child. * KPC BL/ final survey Project HIS 56 80 72 48.6 61 65 2.1.9: Mothers with child age 0-23m who received active management of the third stage of labor during the delivery of their youngest child in a health center. KPC baseline and final survey 30 58 47 28.5 44 44 2.1.10: Public health facilities in the 4 project municipalities certified as child friendly. * Certification by the MOH 0 Suspended by MTE recommendatio n 100 0 Suspended by MTE recommendatio n 100 Result 2.2 Improved Community Health Workers and Traditional Birth Attendant skills in using birth plan, home-based life saving skills and Neonatal IMCI to provide care to pregnant women and newborns 2.2.1: Communities who have benefited from a transport plan for emergency obstetric running. * KPC BL/ final Project HIS 0 100 80 0 100 80 2.2.2: % Children age 0-23m who received clean cord care at the time of delivery. KPC BL/ final survey 30.5 55 70 25.8 41 70 2.2.3: % CHWs and TBAs from remote KPC BL/ final survey 0 100 50 0 100 50 19 Project Indicators Data Source Matiguas Río Blanco/Paiwas BL Final Target BL Final Tar get communities with skills in neonatal resuscitation and/or emergency obstetrical first aid.* 2.2.4: % Communities who have a functioning SICO Project HIS 0 100 50 0 100 50 2.2.5: % CHWs and TBAs that promote birth plan. Project HIS 0 100 50 0 100 50 2.2.6: % CHWs who received a supportive supervision visit at least one in the past 3 months. Project HIS 0 100 100 0 100 100 2.2.7: % of CHWs who are trained in neonatal IMCI. Project HIS 0 100 100 0 100 100 2.2.8: % of CHWs who have been trained in behavioral change and in adult education methods.* Project HIS 0 100 100 0 100 100 2.2.9: % Communities who have a support group for breastfeeding and safe motherhood.* KPC BL/ final survey 0 Suspended by MTE recommendatio n 100 0 Suspended by MTE recommendatio n 100 2.2.10: Number of Male leaders who are trained in education for dialogue. * Project HIS 0 34 30 0 30 30 * A change in this indicator was made during the MTE. 20 F. Discussion of the Progress toward Achievement of Results The majority of planned activities were completed and targets met (with the exception of education and training on breastfeeding). There are a number of positive finding during the FE:  The team found that the key design factors contributing to program success include: - Good coordination with the MOH - The reorganization of the volunteer network and coordination among different volunteers - That MINSA and the volunteers took ownership of the CSP strategy - The involvement of some state agencies (MOE, MOF) and the mayoral offices in the strategies - The organization and availability of refurbished Maternity Waiting Homes (MWF). - Community acceptance of project and strategies  The KPC and majority of respondents interviewed cited examples of perceived impact; increased use of health services, institutional births (88-90%), use of MWHs, and ANC (76% in Matiguas had 4 ANC visits, though lower in Rio Blanco and Paiwas). A number of people mentioned a decline in maternal mortality though this is difficult to measure. However, according to FE interviews, HWs and community workers believe that the numbers of maternal deaths have dropped in project communities.  The two most frequently mentioned positive interventions were cultural adaptation of births and birth planning. Cultural adaptation of births includes men’s involvement and accompaniment of woman to ANC visits and during childbirth. Birth planning includes figuring out transportation, saving money for the newborn and birth, and planning prenatal visits and who will accompany the pregnant woman to the birth, etc. The KPC found that the percent of women with birth plan cards rose from 0 to 82% in Matiguas and 69% in Rio Blanco and Paiwas.  There has been a great deal of emphasis placed on knowledge of danger signs during pregnancy, delivery, puerperal, and for the newborn. The KPC demonstrates that knowledge of pregnancy and delivery danger signs rose between 50 to 70 percentage points while mothers’ knowledge of signs during postpartum rose 20-40 points. Interestingly, men’s knowledge of these danger signs rose at the same levels except for postpartum signs, which were higher among the men (60% and 40%).  Use of Quality Improvement (QI) activities have helped to improve the quality of services in Health Facilities (HFs). Themes addressed have included management of obstetrical complications, clinical and neonatal IMCI, humanization and cultural adaptation of birthing process. Quality norms and lists of pending deliveries were displayed on the walls of all delivery rooms at HFs visited. During interviews it was discovered that most HFs no longer meet in groups. Instead, the QI officer reviews service records and addresses issues individually with the indicated staff member. This may be partly due to the highly politicized environment at MINSA.  The network of volunteers is active and strengthened, provides good mutual support within each community and has improved links with HFs including regular monthly meetings to compare reports, make follow-up visits, gather supplies, etc.  The emergency response system has been strengthened through recognition of danger signs, training of 5 ETBs per community, referrals, community planning for emergencies and improved emergency response at HFs. All 100 communities have emergency plans.  Basic medicines and equipment are available for MNC at the health centers facilities.  Sustainability: There has been a strong effort to classify all communities, of which 85 were certified y MINSA and considered A–level, as being able to sustain themselves, plan, and meet 21 community needs. All 100 communities have mutual support agreements signed by them and MINSA to continue working together on health priorities.  Improved supervision of volunteer activities by MINSA, particularly CHWs, TBAs, and CHAs  All 100 communities have funds for obstetric and other emergencies, which they have raised through community savings funds and raffles. Over the life of the project these communities have maintained funds amounting to more than 150,000 Córdoba.  The project also provided supplies and equipment to the MHWs in the 3 municipalities. Some of the identified weaknesses include: • Minimal coordination between MINSA and the municipalities due to political divisions • Although the communities are more equipped to sustain their activities now, there is concern on their part and MINSAs that it will be difficult to maintain this effort without a project to support them. This is mainly because MINSA’s priority has always been at the service facility level and not the community level. • High level of staff turnover at MINSA presents obstacles for continued community and health facility relationships and support. The project is implementing various MINSA national strategies for improving the health of women and newborns. The strength of this design is the synergy between various activities at different levels (i.e., community, HF and municipality levels). The staff and team noted several times that the success of the project had much to do with the confluence of activities occurring within the project and also at MINSA. For example, the government’s norms and laws which encourage birth planning and the cultural adaptation of birthing practices worked well with the project’s strategy to encourage men to participate in their spouse’s pregnancy, delivery and child care. Many MINSA staff commented on the fact that allowing men or TBAs into delivery was a big change for them, and according to interviews, in some remote areas this practice remains a challenge. As part of this effort, the project encouraged focus groups with mothers, TBAs and MINSA staff so that staff could hear about the kinds of changes mothers wanted in institutional delivery practices. As a result, birthing chairs, ropes to pull during birth, and other practices and furniture were added to delivery rooms. The MTE commented on the vertical nature of task division between different groups of volunteers and between the CRS and Caritas partners. However, by the time of the FE, these strategies appeared to be more integrated and supportive of one another. For example, interviews indicated that the volunteers often worked together when conducting home visits. TBAs often went with CHWs to visit new mothers. If there were difficulties, the volunteers sought out the HF staff to help with a patient. If there was an emergency, the TBA asked the ETB volunteers to help transport the patient to an ambulance. Likewise, the Caritas technical team worked well with the CRS staff in coordinating HF support for community activities, training and other needs. Contribution toward Objectives/Results SO1: Improved knowledge and behaviors for maternal and neonatal health among families and communities Role of Community Health Workers A principal implementation strategy was the use of community volunteers as a mechanism to increase use of health services, identify and monitor pregnant and postpartum women, and recognize danger signs and actions to take to solve problems identified. Some of the types of volunteers were those who have traditionally existed in Nicaragua (CHWs and TBAs) and some were new (BCAs, ETBs and 22 CHAs). All of the volunteers were community based and were not government cadres. None were paid. The CHW matrix in Annex 7 provides details on the training that volunteers have received. One of the project innovations was the formation of CHAs, which is discussed under Objective 2. The project identified, strengthened and supported a total of 1,416 volunteers (1,600 trained): • 60 BCAs, including religious leaders • 99 TBAs (and counselors who are younger women that assume some of the roles of the TBA, without the expectation that they will assist at home births) • 196 CHWs • 483 Community Health Committee Members • 528 ETB Members • 16 CHAs (trained 25) • 34 Religious leaders (pastors and deacons) Most of the CHWs and TBAs involved in the CSP have been working as volunteers for many years and most interviewed had worked more than five years. The CHAs are newly formed and have worked 3 years or less. As noted above, one of the strengths of the project has been the mutual support afforded by the network of volunteers. There were many reports of strong coordination among the various volunteers during the FE. The ETB works closely with the TBAs for emergencies, and the CHWs are seen as a valuable resource in the community and often conduct visits with the TBAs. This support network helped motivate volunteers to continue working and has also helped compensate for volunteers that have certain limitations (e.g., migration during certain periods or illiteracy). During the FE the volunteers reported that they work closely with Catholic and Protestant churches. Many volunteers mentioned conducting educational meetings after church services. This was seen as an opportune time to meet since community members were already gathered in one place. Also, according to volunteers and CRS/Caritas, the bishop of Matagalpa has taken an active interest in this project — particularly the innovation — and has instructed his priests to support community health activities. Volunteers also frequently worked with other NGOs and linked with HF staff through MINSA monthly meetings. Training/education The project employed a methodology of cascade training whereby Caritas and MINSA staff were trained, who then provided training to the community volunteers who then educated community members through group sessions and counseling during home visits. The MTE identified some weaknesses in the training. MINSA staff were not as involved in training as needed and Caritas staff lacked skills in working with low literate audiences. However, after the MTE the CSP responded by providing adult education training to Caritas staff who then worked with volunteers to improve their facilitation skills for non-literate communities. Per MTE recommendations, refresher training was provided to volunteers during the second half of the project. The project also adopted a strategy of holding monthly meetings at the health facilities instead of asking everyone to come to a central location in Matiguas or Matagalpa. During the meetings, volunteers could go over their reports with MINSA staff and discuss problems. MTE recommendations were taken into account, such as reducing the use of technical language and 23 providing mentoring at the sites. The meetings were also an opportunity for technical clarifications and short updates and helped strengthen the working relationships between volunteers and MINSA staff. The MTE stated that although communication skills had been taught to all volunteers, training on these skills had been more extensive and emphasized (negotiation, dialogue for change, etc.) in the BCA training and the BCC training given to Caritas staff. The MTE identified the need to provide more participative training and counseling skills to the other volunteers who also needed these skills given the requirements of their work. There was much discussion during FE interviews about the participative education and counseling activities conducted, but the training chart did not indicate that there had been any additional training provided to volunteers. Project staff said that this was done informally by field technicians during community visits. It did not appear that there had been exchange visits by different actors to different sites per another MTE recommendation. Behavior Change Communication (BCC) The project’s main BCC focus has been its innovation activity with men. Despite this, the rest of the project focused on changing behavior through its training of MINSA staff and volunteers and the community education activities conducted by the volunteers. All of the activities that promoted birth planning, community neonatal IMCI, life saving skills, nutrition during pregnancy, development of community emergency plans, etc., have used BCC approaches in the educational activities through home visits, counseling and group talks. The CSP has obtained materials to support the communication process, including: 120 flipcharts on danger signs for CHWs developed by MINSA; 113 sets of four maternal nutrition counseling cards for TBAs (out of the total MINSA nutrition package); 700 calendars with danger signs were distributed to some CHWs, leaders and HFs (5-7 per community); 2,000 Birth Plan cards were distributed; and a Manual for Birth Planning from MINSA was given to volunteers trained in Birth Planning. The project made good use of the MINSA developed materials. The CSP developed some materials to cover gaps where no materials were available. Radio has been an important medium for transmission of project related messages. According to FE interviews, MWH staff felt that radio messages had contributed to an increase in in the use of the homes, and HF staff felt that radio had helped increase the use of the facilities. Men and women interviewed mentioned radio as an effective method for transmitting health messages. Effectiveness of radio messages has not been evaluated. There are many materials available to support the development of counseling skills for volunteers. FANTA, NICASALUD and other partners developed a quality checklist that is used as a tool for routine monitoring of the quality of volunteer counseling, for providing coaching and feedback and as an outline for trainers. Caritas has implemented another project in the same geographic area as the CSP and this project has been an additional resource for counseling materials. Supervision Project supervision targeted two separate levels: the supervision by Caritas management staff of the field technicians and the supervision of volunteers by MINSA and CSP staff. Both systems have been strengthened by the project. Per MTE recommendations, a supervision checklist was developed and used by Caritas staff. There is also a separate function for monitoring the community plans developed by the CHWs as part of the N-IMCI strategy. 24 After the MTE, the project strengthened the joint monitoring activities with MINSA staff. It reorganized monthly meetings so that volunteers and Caritas staff met with MINSA staff in the closest health facility to the volunteers instead of meeting as a group of volunteers in Matiguas. This has helped strengthen the relationship between HF staff and volunteers as well as monitoring of activities, reporting and referral systems. This is a good system for supervision and on-going training of volunteers. Referral system The referral system is a crosscutting mechanism for both maternal and child health care seeking. This is a national MINSA system that the project has been supporting and improving. Of the six HFs visited during the FE, all had referral and counter-referral forms and the majority of volunteers are actively referring patients. The numbers of referrals are increasing. For example, at the MTE there were 85 referrals between April to June 2010 and in 2012, there were 398 during the same quarter. Likewise, counter referrals have increased, though not as dramatically as the referrals. In the first quarter of 2011 there were 3 and in the second quarter of 2012 there were 88. There were some issues with the system, as most people expected that if they received a referral, they could go to the head of the line, even if it was not an emergency. The MINSA format that was used does not differentiate between routine health care and emergencies. Volunteers reported that the biggest problem with referrals was that the HF staff did not place any importance on the referral. To resolve this issue, the project worked with MINSA to come up with a list of what MINSA deems an emergency and volunteers held meetings with communities to explain the system. The project is still working to improve the system, trying to convince health staff that when they accept referrals and send counter-referrals back, it motivates the volunteers. The other issue has been counter-referrals, which were frequently not completed by HF staff and returned to the person referring. In an effort to keep the system working, some health workers from health posts would look for referrals when they were at the health center facility, look up what treatment the patient received, and then bring the information back to their home facility. Sometimes the counter-referrals were given to the patient, but often the patient forgets to bring them back to the volunteer. The project has worked to improve the system, but it is slow. As noted above, the numbers of counter-referrals were increasing, but as can be seen from the above numbers, there were only 88 counter-referrals out of 393 referrals for the April-June quarter. It is hoped that the system will continue to improve after the project ends with the strengthened relationships between volunteers and HF staff. Emergency transport Emergency transport brigades (ETB) existed in many communities before the CSP began, but most were inactive. The ETB system is part of the national Birth Planning strategy. It is designed for obstetrical emergencies but is used for any type of emergency. The members of the brigade have been trained in Birth Planning and they reported during the evaluation that they were making home visits to encourage institutional births, use of the MWH, and recognition of danger signs. Each community has a team that is composed of five members (most were men but some communities have incorporated women). All have received basic equipment including a hammock, flashlight and water bottle. All communities have a savings fund for medical emergencies with statutes to guide their use. During FE interviews most communities could relate how the transport system had been used in an emergency situation and how funds had been allocated. The ETBs were active in collecting funds for the emergency fund; linked with private vehicle owners, not just the MINSA ambulance; and 25 referred people to the HFs and MWHs. Funds were loaned to most families, but donated to very poor families. The fund is not necessarily sustainable with the original money collected, but follows the tradition of collecting money when a family is in need so that money will need to be continuously replenished. At the time of the FE, the ETBs had transported 129 patients for obstetrical and neonatal emergencies. Birth Planning (Plan de Parto) This is the most firmly established of the MINSA strategies and was the first to be introduced by the CSP. The Birth Plan is designed to overcome delays in recognizing and seeking care in childbirth and includes filling out a Birth Plan format during a counseling session with the TBA or HF staff, and the pregnant woman and her family (preferably with the husband). The format includes messages and actions on danger signs, ANC, saving money for expenses, where the birth will take place, use of the MWH, transport in case of an emergency, who will accompany the woman during delivery, and other preparations. The birth planning leaflet was designed for low literate audiences and was easy to follow once TBAs were trained. The TBAs also maintained a list of pregnant women and a map showing where they were located. They were supposed to make home visits to monitor the woman throughout her pregnancy and postpartum and provide counseling. The TBAs were trained to use flipcharts (MOH) as an aid for teaching mothers and explaining danger signs. In case of a problem, the TBA could refer patients to the HFs and MWH and worked closely with the ETB in case of an emergency. Many of the TBAs are illiterate, making the filling out of pregnancy registers and other SICO formats difficult. To improve this, the CSP arranged for registers to be reviewed during monthly meetings at the HF and the staff help complete the paperwork. Per MTE recommendations, training of all TBAs in the birth planning process was completed. HF staff was also given more training on cultural adaptation of the birth process. In addition, CHWs and CHAs were also included in the birth planning training. Maternity Waiting Home (MWH) The MWH is an excellent strategy for increasing institutional births and many of the homes have been functioning for many years. During interviews the women in the MWH said the majority of their previous births had been at home. The women had traveled about 2-8 hours from their homes to the MWH. Most women brought some food item as a donation, and most came with either a partner or family member. There were only a few women available postpartum, but they all said they had someone accompany them during delivery (TBA and mother), were given a choice in their birthing position, and had been allowed adequate privacy. Some women stated that they were a bit bored at the facility and when questioned about what they did during the day, they said they slept, cooked and talked with friends. One of the doctors was pleased to note that if the women stayed for the recommended 2 weeks before birth, they often gained weight at the end of their pregnancy and were more rested. All cadres of volunteers and HF staff actively support and refer women to the MWH. Volunteers reported that they convince women to use the MWH by soliciting support from their partners and families, and by sharing positive experiences of other women who have used the MWH. However, some women refuse to go to the MWH in which case the health facility personnel and community leaders visit them. The increased use of the MWH caused an increase in expenses which were not covered by their current budgets. Different strategies were used by the MWH to continue operations, including: • Sought funds from various organizations, including the municipality, MINSA, and NGOs • Asked each woman to voluntarily bring some food with her 26 • The MWH in one district received help from NGOs to establish income generating projects including table and chair rentals and the establishment of a small store. • After the mid-term, many communities were encouraged to support the MHWs by donating food. In one district, there was an organized plan for each community to donate food on different months. The other districts also encouraged communities to donate food to the MHWs. Neonatal-IMCI Beginning in June 2010, the CSP trained all 200 CHWs in N-IMCI. The project also trained 25 CHAs in community N-IMCI. PAHO co-facilitated the training with Caritas and MINSA. MINSA materials were not available, so the project developed their own materials and will introduce the MINSA materials at a later date. Live saving skills CRS Nicaragua received technical assistance from CRS Honduras on developing the life savings skills strategy, but the Honduras program focuses on improving home births as opposed to encouraging institutional births. Only Caritas field staff, MINSA and staff from the MWHs were trained at the time of the MTE but CHAs received this training during the second half of the project. The MTE report mentions that the project planned to develop a flipchart which focused on prevention and management of complications during pregnancy, birth, postpartum and for the newborn. TBAs were to be trained on use of the flipchart. However, there was no evidence of this flipchart at the FE. Additionally, health personnel were reluctant to train TBAs too extensively in management of birth complications since their role had been reduced to “birth counselors.” Nutrition during pregnancy Nutrition represents 30% of the CSP level of effort. At the time of the MTE, this component had not yet been introduced, however, breastfeeding training did begin in 2011. The CSP trained 100 TBAs, 200 CHWs and 25 CHAs. In addition, the TBAs and CHAs received training on nutrition during pregnancy focusing on appropriate diet and iron and folic acid supplements. They received part of the MINSA set of nutrition counseling cards pertaining to nutrition in pregnancy and were trained on how to use them. Key nutrition messages include: • Early initiation of breastfeeding without giving any pre-breastfeeding food • Exclusive breastfeeding until about six months old • Exclusive breastfeeding as a means of birth spacing • Continued breastfeeding for 24 months with the introduction of complementary foods around 6 months • Increase energy intake during pregnancy and reducing the workload • Dealing with side effects of iron tablets and folic acid provided by the MINSA during pregnancy • Increase intake of foods, particularly fruits and vegetables, during pregnancy and lactation • Provision of postpartum Vitamin A supplementation at health facility Breastfeeding As noted above, the training for this strategy was conducted after the MTE. The breastfeeding strategy initially included a number of activities that were no longer feasible at the time of the MTE and were not continued. The plan to implement the Baby Friendly Health Initiative (BFHI) was based on the assumption that it would support a national initiative. However, the national committee for BFHI was 27 disbanded due to shifting government priorities. Therefore, the MTE recommended that the project suspend this initiative given the large number of activities the project was doing and the length of time required to implement the 11 steps and do BFHI well. Another strategy was to form breastfeeding support groups in each community. Given that volunteers were already implementing a number of national initiatives using counseling and group education, the MTE recommended that these two communication methodologies be strengthened and used for all topics, rather than adding a third methodology, support groups, that would dilute the quality of community education. The original focus was to promote a number of natural family planning methods, but the MTE recommended that the project only emphasize Lactational Amenorrhea which would strengthen the use of exclusive breastfeeding. Despite the increased breastfeeding and nutrition training, the levels of exclusive breastfeeding within the previous 24 hours among lactating mothers and the percentage of newborns breastfed within one hour of delivery did not meet their KPC targets. When asked about this, project staff indicated that with the discontinuation of the BFHI initiative, breastfeeding was de￾emphasized by the health system. However, the KPC does show that consumption of iron and folic acid supplements during the first trimester of pregnancy did increase, and in Matiguas this percentage significantly surpassed the target. It is recommended that the low breastfeeding rates be brought to the attention of the MOH, stressing that not only were the percentages low, but the rate of breastfeeding immediately after delivery appears to have dropped from baseline levels four years ago. SO2: Increased families’ access to quality maternal and neonatal services. There is substantial overlap between the two objectives and some points were previously described. This section will focus on improvement of the quality of MNC services in MINSA HFs and the new strategy of introducing CHAs. A number of activities have been carried out during the CSP, including: • Training of HF staff • Supply of essential materials and equipment for MNC • Reactivation of QI activities including the monitoring of quality indicators • Humanization and cultural adaptation of childbirth in coordination with Health Care Improvement (HCI) project of USAID • Alliance with Autonomous University of Nicaragua (UNAN) in León to develop a training plan using specialists from the university and the creation of a training center in Matiguas • Coordination with, and referral to the MWH, particularly for women from isolated communities • Annual planning in each municipality • Use of national protocols for standardized routine treatment and in case of emergencies Most people felt that services had improved in HFs, although there were some concerns about what would happen when the project ended and whether it would be possible to maintain relationships with HF staff. According to FE interviews with HF staff, they felt the quality of services had been improved through project activities in the following ways:  Better coordination between volunteers and HFs 28  MINSA and the volunteers took ownership of the project strategy  Involving TBAs in institutional childbirth had improved client-doctor relations  Access to essential supplies for quality care  Better communication between the community volunteers and MINSA through monthly meetings.  Monitoring and follow-up on quality standards for MNC  Having flexibility in the birthing process, according to the preferences of the woman  More counseling given and attention paid to pregnant women  Improvement in the way people are treated at the HFs  Improved patient charting (clinical history)  Has helped MINSA reach their national indicators  Improved referral system to HFs In general, health staff thought that quality of care had improved during the time of the CS project. However, HF staff felt that further work is needed to continue improving some patient-doctor relations (and this was also reflected in interviews with volunteers and community members who felt that relationships had improved with many providers, but some were still difficult to deal with). Men and women interviewed had some complaints about poor treatment by some personnel, lack of triage and long wait times, but all agreed that quality had improved. Training of HF staff Training was carried out as planned and is summarized below. Topic Number trained Natural family planning methods 40 National vaccination campaign refresher 436 EOC 71 Neonatal IMCI 74 Workshops on cultural adaptation 248 Supply of essential materials and equipment for MNC Basic equipment and supplies have been provided to the 13 HFs and in general, basic needs were met. An inventory conducted during the FE found that all basic medications were available in the six HFs visited by the team. Five of the 6 HFs had a dopler ultrasound provided by the project. The project also conducted some minor remodeling of HFs to make them more culturally appropriate for birthing. They provided birthing stools and ropes for women who preferred traditional birth positions. The project provided showers and added items to existing delivery room equipment in two of the delivery rooms. In addition, the project provided supplies and minor equipment to the MWHs. These included mosquito nets, stoves, refrigerators, utensils, cookware, water filters and fans. Quality Improvement (QI) Quality improvement activities have been introduced throughout Nicaragua by the Quality Assurance Project and HCI in the past. Over time, many of the teams have ceased to function and during the first part of the project the CSP reactivated the QI teams supporting activities including the monitoring of national quality indicators in the 13 HFs in the project area. National checklists were used to ensure that correct protocols were being used. HCI has provided support to the project on QI but there is only minimal involvement at the SILAIS level QI team. 29 One of the most important QI activities is the monitoring of the correct application of national standards and protocols in MNC. HF staff reported a number of ways this was done including file reviews, case studies, reporting at Technical Council meetings on compliance, exit interviews for client satisfaction, continuing education and quality circles. There is also evidence of monitoring support in the delivery rooms where norms and protocols were placed on the walls. Each of these health centers received training on the norms and protocols by the project. The majority of HFs visited conduct self-evaluations either once or twice a year that they use to provide feedback to staff, identify topics for continuing education, and to elicit agreements for making improvements. A number of improvements in MNC care were reported, including the routine use of partographs, increased use of urine exams for proteinuria during ANC, and improved privacy at HFs. In most of the facilities visited during the FE, the QI teams continue to be weak and their sustainability is questionable. One issue is that quality improvement often focuses on the weaknesses of one person, rather than on the system. Health facility staff explained that they found it too difficult to work with teams to review quality issues. There were too many items to be reviewed and it was difficult for a team to find time to meet. Therefore, the procedure is for the designated QI officer to review charts and findings and speak to the staff individually about any problems encountered. Thus, the QI teams were reduced to one person, rather than making group decisions that address hospital systems and procedures. Quality improvement becomes more an issue of individual performance and it is seen as more of an authoritative rather than participatory process. There has also been a change in MINSA leadership and staff commented that this has had an impact on QI. Before, the circles would meet to review activities and address problems, but now the problems have not been addressed. Now that the CS project has ended, the responsibility falls to MINSA which needs to take a more active role in institutionalizing QI activities and taking ownership of the process in order to maintain a sustainable improvement in the quality of MNC services. Community Health Agents (CHAs) The project’s objective was to improve Community Response to Maternal and Neonatal Complications through the use of specially trained CHAs in isolated communities. At the request of SILAIS, this activity was added during DIP development as a pilot for the introduction of a new level of volunteer health worker for isolated communities that will be given equipment and training beyond what a CHW receives. Initially, the project planned this as an innovation activity, but upon review USAID decided that it fit better as a component of the regular project activity supporting MINSA’s MNC activity. The DIP contains some conflicting information about this innovative activity. It was decided when the DIP was finalized that OR would only be conducted for the primary innovation and not this one, but the protocol for conducting OR for the CHA innovation was erroneously included in the DIP. One activity in the DIP work plan specified that OR would be conducted for the CHA innovation, but this was included in error. CRS has developed a curriculum for CHA training and is waiting for approval by SILAIS/MINSA with the hope of establishing CHAs as a national program. The manual was given to SILAIS 3 years ago and although they have not approved it yet, they did approve the National Autonomous University (UNAN) to train the CHAs and signed the certificates for the training. The project staff feels that their work with UNAN was an important lesson learned because they learned new methodologies to train CHAs. The UNAN staff provided theoretical training for a short period and then gave the CHAs practical tasks to carry out in their communities. After a few weeks, the CHAs came back to review experiences and then proceeded to the next topic. The CHAs learned how to run community meetings 30 using agendas and written agreements and apply new behaviors in addition to providing basic health services. Each CHA is responsible for 2-3 communities in an effort to expand coverage for the most isolated communities. They received training in leadership and community mobilization, IMCI, SICO, nutrition during pregnancy, Birth Plan strategy, N-IMCI, first aid and Life Saving Skills. Recognizing the value of the CHAs, MINSA then asked that they be given more training and equipment to take blood pressure, pulse, respiration rates, etc. After this training, the CHAs asked MINSA to supply them with small drug kits that they can resupply at the HFs. The project was not involved in this acquisition and views this as a testament to the self sufficiency of the CHAs. The project supplied basic equipment such as BP cuffs, stethoscopes, alcohol, antiseptic, and thermometers. MINSA, however, s supplies essential drugs, gauze, cotton, syringes, suture kits, etc. As their role has evolved, the CHAs have developed a direct relationship with MINSA and their local HFs. MINSA sees them as a welcome partner. As a result, the project considers them volunteers that they support indirectly. The CHAs send reports directly to MINSA using SICO forms and coordinate patients through their local HF. An issue is that although 25 CHAs were trained, several have migrated or found other jobs. As a result, there were only 16 still operating in the project area. 2. Contextual Factors The project got off to a slow start as it took six months for Caritas to produce an agreement between the two bishops that have responsibility for the project area. The project area encompasses two different dioceses. After 6 months, a decision was finally made for the project to be managed out of the Matagalpa Diocese. Caritas staff were not hired until April of 2009; seven months after funding began, so they were not involved in baseline studies or in the formulation of the DIP. Due to problems in obtaining exoneration of taxes, motorcycles were not available for the first four months after staff were hired, limiting the field staff’s ability to travel to the communities. Other factors that influenced project implementation, and efforts to mitigate those factors are summarized below. Contextual Factor Efforts to Mitigate Factors High level of violence and political division in the zone (especially in Paiwas where there are two seats of power) CSP tried to work through municipal councils, but this may not be the most adequate solution due to political divisions. The innovation has helped to mitigate problems of intra-familial violence through social pressure. Political tensions continued throughout the project due to elections and divisiveness of political parties. Divisive political environment Staff and HWs complained that since the last election, politics is interfering more than before. With MINSA in a different party from the municipalities of the 3 districts, it has been difficult to get cooperation among the entities. In 2011, there were presidential elections and in 2012 there will be municipal elections. CSP tried to work with this, but it was difficult and the divisiveness contributed to much of the MOH staff turnover. Many of the individuals and directors that the project has relationships with have changed. The FE team had to search for HWs who had experience with the project. Epidemics: H1N1 in 2010 and leptospirosis in 2011 caused HWs CSP worked to support existing staff during these epidemics when much attention was diverted to dealing with disease control. The project supported 31 to be unavailable for project activities vaccine efforts and other MOH needs so that personnel could also focus on MNC. High levels of illiteracy Although staff were trained to work with low literate audiences, low literacy makes the learning process slower. Poor access and isolated communities The new CHAs helped to provide services to some isolated communities. All field staff were provided with motorcycles. Seasonal migration of volunteers and community members limits activities during certain times of the year The CSP developed a network of volunteers that provided mutual support. By working as a team, they can continue to provide services when some volunteers are absent. Having large numbers of volunteers in each community also helped mitigate this issue. Continuous rotation of MINSA staff, for example, the municipal director in Rio Blanco has changed three times since the project began. The CSP advocated with SILAIS to maintain staff that have been trained within the project area. In some cases they had been successful, but with the worsening political environment this proved more difficult. Weak appropriation of activities by MINSA management team and lack of institutionalization of strategies such as QI. This had been the most challenging factor and the project tried to address it through its sustainability plan and activity. CRS and Caritas are Catholic organizations but need to include Evangelical Protestants The CSP added credibility because it is supported by Catholic organizations within a primarily Catholic country. The CSP took numerous steps to involve other religious leaders which embrace a wider segment of the population. The innovation actively sought out religious leaders to be involved as BCAs in counseling men and families and this has strengthened the strategy considerably MINSA lacks sufficient budget to cover their planned activities and there is insufficient staff, which limits supervision and follow-up The CSP provided some financial support to HFs by supplying basic equipment and materials, and covered some expenses for outreach visits and the MWH. This support was obviously not sustainable, but CSP had been advocating for support through SILAIS. Also, the decentralization of supervision to HFs in the districts during monthly meetings has helped. 3. Role of key partners Partners Role in Project Results of Collaboration Caritas, Matagalpa Diocese Principal implementer of community activities in coordination with MINSA at the municipal level Implementation of community activities in 100 communities, together with MINSA Implementation of the innovations of male involvement with MNC in 20 communities and CHAs in 25 communities UNAN Leon Institutional strengthening for MINSA staff in MNC Training for CHAs and development of a training center in Matiguas MINSA staff training in Neonatal IMCI and EOC Formation of a specialized team of trainers in pediatrics and gynecology/obstetrics Training of 25 CHAs NICASALUD Provided technical assistance in the formative research and implementation of the innovation for male involvement in MNC Qualitative studies with men and their partners completed Community strategies for advocating behavior change in men developed Development of IEC/BCC materials Training of 66 BCAs and 14 Caritas staff in behavior change methodologies CIES UNAN Managua Operations research on the innovation for male involvement in MNC, including a baseline, midterm Baseline completed, midterm completed in 2011 and final completed in 2012 (see report Annex 12) 32 Partners Role in Project Results of Collaboration and final survey MINSA Provide overall technical guidance and follow-up for the implementation of strategies at the community and institutional levels in coordination with CSP staff Joint proposal design and development of the DIP Follow-up on strategies with the volunteer network Implementation of a referral/counter-referral system Development of a joint annual training plan in the 3 municipalities Implementation of the strategy of humanization and cultural adaptation of birthing Tripartite alliance with UNAN and CRS for the accreditation of a training center in the HF in Matiguas Joint development of a community accreditation process and certification of successful communities by MINSA HCI Provide technical assistance in the implementation of QI activities and cultural adaptation of childbirth Workshops to share experiences and develop implementation plans for cultural adaptation Sharing of QI methodology and tools CRS/Caritas effectiveness The MTE report noted some issues between Caritas and CRS. However, by the time of the FE, most of these had been resolved. Part of the problem had been the slow start on the part of Caritas due to the different diocese management described above. This meant that CRS had to initially take charge of many activities and this lead to confusion later on the part of volunteers and HF staff. After the MTE, the project worked hard to bring all staff up to speed and also had team building activities to strengthen the partnership between the two organizations. The FE team did not hear any concerns about the working relationships between the two organizations from HWs, project staff or volunteers. MINSA involvement and sustainability Despite the political issues with MINSA and the lack of resources, the project has worked hard to link project activities with HWs in all project facilities and strengthen support for volunteers. Health workers meet regularly with volunteers and CSP staff during monthly meetings at the HFs to review progress, address issues and plan. After the MTE, the project began working on the sustainability plan to evaluate communities based on criteria for sustainability. The CSP developed specific steps and criteria for categorizing communities in terms of readiness to take on management of their own health problems and issues, working closely with MINSA. The criteria were listed in a Diagnostic Guide for Categorizing Communities that was administered by the volunteers. Communities were classified as A, B, or C according to the criteria. An initial assessment was done in early 2012 and communities ranked B or C were given a list of criteria that they needed to improve to be ranked A. These communities were then re-evaluated in mid-2012. In the end, 85 of the 100 communities were classified as A, 12 as B and 3 as C. Those classified as A were given a certificate signed by MINSA, CRS and Caritas. All the communities have an agreement signed by MINSA that stipulates how the communities and MINSA will continue to work together after the project ends. Some of the improvements in HW activity include the following:  Improving referral and counter referral system in all 100 communities  96 monthly community monitoring visits by MINSA, HP and HC personnel  MINSA support of CHAs by providing additional training and small drug kits and supplies  In the 3 municipalities HWs are using humane culturally appropriate birthing practices  Delivery rooms are adequately equipped and norms are pasted on walls 33  In year 3 the CSP had 8 meetings with MINSA to share MTE results and sustainability plans There is concern as to whether all these improvements will continue after the CSP ends, but most of these activities have been standardized by MINSA. Now that the systems are established, the volunteers trained, HWs trained and the community agreements in place, the project is hopeful that implemented activities and strengthened systems will continue. Municipal involvement The Municipal Health Councils (Consejo Municipal de Salud) are chaired by the mayor of each municipality and attended by MINSA, Ministry of Education, Ministry of Family Services, Army, Police, and NGOs. The meetings are often conflictive due to the involvement of members from different political parties and are often unable to move forward in decision making. These councils were formed years ago and gradually disappeared. They were reactivated by the project, disappeared, and were reactivated again. The MTE questioned whether it was worth trying to work with them. However, the project did continue to hold meetings with the 3 beneficiary municipalities as well as the SILAIS informing them of project progress and challenges. The municipalities worked with the project to address the difficulties. The municipalities did provide support to MWHs, and participating municipal groups (police, Ministry of Family Welfare and Ministry of Education) were particularly supportive of the BCA activities particularly in Rio Blanco. Additionally, the mayor of Ubu Norte provided an ambulance and covered costs for emergency transport. The involvement of municipal authorities will be critical for the sustainability of MNC activities. 4. Overall Design Factors The FE team believed that a key factor in the project design was the coordinated working relationship with MINSA. The partnership has been critical for strengthening services in the 3 municipalities and now with strengthened systems, the continued provision of services. Training and upgrading the HW skills concurrently with upgrading volunteer skills had been key to ensuring all were on the same page and can work in a coordinated fashion. Decentralizing the monthly volunteer meetings has also improved relationships and sustainability. Now each group of volunteers meets with MINSA staff in their relative HFs instead of with Caritas staff in Matiguas. This model has directly encouraged local monitoring of volunteer activity and strengthened relationships. Another design factor influencing the project was the decision to focus on men as an innovation strategy in a project area known for its high level of violence and “machismo” both within and outside the family. While the innovation only focused on 20 communities, the effects appear to have been transmitted beyond these communities, according to the CIES final report. This could be because the volunteers from all communities were aware of the innovation strategy and may have encouraged men to become involved. Also, radio messages and other community education activities/events were designed for the OR/innovation, but these messages were transmitted all over the three municipalities and may have influenced men’s behaviors. Other factors in the design that influenced project efforts include:  Re-organization and re-training of the community volunteer network  MINSA and the network took ownership of the project strategy  Involvement of some state agencies and mayor’s offices  Strengthening of the Maternity Waiting Homes  Community approach for involvement in the project 34  Developing yearly training plans with MINSA staff during the first 2 years when MINSA was open to working with the project  Alliance with the University of Leon allowed a quality training program to evolve and encouraged development of a training center in Matiguas.  Work with HCI system to monitor and implement QI standards related to project objectives of addressing obstetrical complications, neonatal IMCI, resuscitation of the newborn and cultural adaptation of birth process.  Face to face work with men was the best method for convincing them to change behavior  The project’s activity took place at a time when a confluence of factors was occurring in the country that contributed to its success (i.e., the institutionalization of the birth plan strategy and the cultural adaptation of birthing practices). These factors allowed the MOH to be more flexible and accepting of traditional birthing practices, welcoming of men at the health facilities, and this in turn, encouraged individuals to seek institutional deliveries.  The BCC innovation was completely accepted by all parties: central and regional MINSA, churches, municipalities, police, etc. Dissemination The project has made an effort to share its results with MINSA and the municipalities. Over 12 management meetings have been held with MINSA to review project activities, plans, MTE results and to share and gain support for the sustainability plan. In addition, 3 meetings per year in years 3 and 4 were held with SILAIS Matagalpa as well as the management teams from the 3 project municipalities. The project also met with USAID and USAID funded NGOs to share project results during the last year. During the debriefing meeting the USAID/Managua officials encouraged the project to disseminate its results, particularly the men’s innovation study results, with MINSA. However, the project feels that this pilot effort needs another expanded pilot phase before scaling up given that so far it has only been tested in 20 communities for 1.5 years. Ideally, if the SILAIS in Matagalpa could expand this effort and show results, then it could be scaled up nationally. The project does believe that the CHA model of using more highly trained CHAs to cover basic services in more isolated areas could be expanded and scaled up by MINSA and that this activity does fall within current MUFOC plans to decentralize services to small local areas. CRS has presented this model widely to officials in the Matagalpa region. However, they had not yet disseminated the information at the national level by the end of the project in September 2012. The innovation has been presented by the project backstop at the following global meetings: 1. The Gender Equity session at the CORE Group SPRING MEETING “Getting to the Heart of the Matter: Communities and Health Systems Strengthening”, 2010 in Baltimore, MD 2. Revisiting Trials of Improved Practices Methodology session at the CORE Group SPRING MEETING “ Equity in Health: Ensuring Access, Increasing Use”, 2011 in Baltimore, MD 3. At the Saving Lives at Birth: A Grand Challenge for Development event organized by The Bill & Melinda Gates Foundation, from July 12-14, 2012 in Seattle In addition, a video was produced and shared at the CoreGroup Spring meeting in 2012 and copies were shared with the local mission, NicaSalud Federation, the CSHGP and MCHIP teams and CRS country offices and regions. 35 As noted above in the discussion of MINSA Involvement and Sustainability in section F3, SO2, after the MTE the project developed a sustainability plan that included all 100 communities it was working with as it phased out its support. This section describes how the communities were evaluated and classified according to established criteria and the 85 communities classified as A level were given graduation certificates as being a Healthy Community” signed by MINSA, CRS and Caritas. A “Healthy Community” is defined as “one that has reached a pre-determined level of dominance over knowledge and skills for community action and maternal and neonatal health so that, collectively, the families can provide for their own care and continue to take action to maintain their health, with the help of community volunteers and connections with MINSA, municipal governments, local organizations, and other stakeholders.” All communities signed agreements with MINSA to continue supporting health activities. This plan was completed on schedule and most communities were judged competent to sustain their local health systems. The main contribution for financial sustainability has been the project’s assistance in establishing savings funds in the communities for emergencies. Over the life of the project, savings have amounted to 150,000 Cordobas and the funds have been used to help 384 families (see section F, SO1 for more detail). Thus with the reactivated and strengthened volunteer capacity, better links with MINSA services and the savings funds for emergency transport, the communities are now able to address health problems and access services. However, there are no cost recovery schemes in place since drugs are supposed to be free and though there are consultation fees, the bulk of MCH services are free. Families are encouraged to save money as part of the “Birth Plan” for any birth related expenses. There are no corporate sponsorships. Sections E and F of this report describe how the project worked with 7 cadres of volunteers to educate communities and build demand for services. Additionally, the CSP strengthened MNC skills of HWs in the project area (see section F, Objective 2 above). As can be evidenced from the KPC data 90% of mothers (an increase from 2008 BL 55%) had institutional deliveries, 77% of mothers in Matiguas attended 4 or more prenatal visits and an average of 85% (BL 56%) of mothers took their children to a health facility within 24 hours of onset of symptoms, demonstrating increased demand for and accessing of services. In accord with the sustainability planning effort described above, the CSP expects that with the certified communities able to recognize danger signs and emergencies, refer patients to HFs, and have ETB volunteers and savings funds in place to manage emergencies, the MOH HFs will be able to respond effectively to any problems. As described in section F above (Objective 2), the project has also worked to strengthen HW skills through training as well as improve quality of care through support of QI. Now with the joint certification of communities by MINSA, it is expected that the strengthened health system will be able to adequately meet projected health needs. The project also worked to strengthen the health information system as described in section D of this report. Now that the volunteers are coordinating with HF workers monthly to review SICO forms and report data, it is expected that the improved reporting system will continue. As mentioned in other sections of this report, USAID/Nicaragua will be closing its health office in 2013, so it will not be able to provide any further support for this CSP activity (see concluding remarks in this regard). At this point, there have not been any policy changes to allow additional funding to sustain these activities. It is expected that the retrained volunteers and health workers will be able to keep the CSP strengthened system working. However, some communities and HWs expressed concern 36 during the FE regarding how long the existing and improved health system can continue to operate without any outside support, particularly given the constant turnover of personnel in the current MOH. Much will depend on MINSA’s ability to continue providing services and overseeing outreach activities in the foreseeable future. G. Assessing the Operations Research Study Innovation - Engaging Men to Improve Care-Seeking was carried out in 20 communities with technical assistance from NICASALUD. The innovation followed a “Trials in Improved Practices” design where behaviors were identified, practiced, and then an analysis carried out by the men to determine which were the most feasible. The approach included 7 behavior modification strategies. Using a cascade training model, first the field agents from Caritas were trained and they then trained the Behavior Change Agent (BCA) volunteers. Three BCAs were trained in each community to work with men to negotiate behavior change for improved MNC. They used venues such as sporting events as educational opportunities, as well as face to face counseling methods to negotiate behavior change. CIES developed an Operations Research (OR) design based on 20 target communities and 20 control communities (within the 100 communities where the CSP was implemented). CIES conducted a baseline and a final survey to measure changes in the population. A summary of the NICASALUD report on the innovation and CIES final report is included in Annex 11. Some highlights from the CIES evaluation report demonstrate that:  In the innovation communities, there was an increase in the percentage of women completing 4 prenatal visits or more (Bl-64, Fl-77%) but there was no difference between case and control communities on this behavioral improvement.  In innovation communities, the percentage of women who stated that they had decided jointly with their husbands regarding the number of children they would have and the spacing between pregnancies increased (BL-55%, FL-71%). However, this gain was also equal to that of the control communities.  In innovation communities, the percentage of women who sought ANC care with their husbands increased (Bl-43%, FL 64%) and this was higher than that of the control communities (Bl-47%, Fl-39%).  In innovation communities, the percent of men who accompanied their wives to HFs for newborn care increased (Bl-55%, Fl-78%) and this was a much greater increase than that in the control communities (Bl-35%, Fl-46%).  In innovation communities, there was a significant increase in the percentage of men who participated in the delivery of their child (Bl-49%, Fl-60%) and this difference was not evident in control communities (Bl-47%, Fl-45%). The project staff believed the reason there was no difference between control and case communities in terms of women completing 4 or more prenatal visits was related to the fact that both BCC and non￾BCC communities were promoting messages about prenatal visits during home visits, group education talks and on the radio. Although specific radio spots were developed for the innovation activity, the radio programming was still heard by all communities in the project. It was concluded that there may have been contamination of this indicator. The similar levels of improvement related to the indicator on joint decision making about child spacing and numbers of children may be due to other factors or programs since this was not a major project focus. However, the increased levels of men’s accompaniment to ANC visits, newborn care and deliveries are definitely attributable to intervention activity. 37 The way the innovation was conducted will make it difficult to replicate for two reasons: 1) There were too many inputs included as incentives for the innovation to be realistically introduced on a larger scale by MINSA or other NGOs. The CSP needs to identify ways to lower the cost before it can be scaled-up. During the MTE, a discussion was held with the team to analyze the possibility of making changes to the innovation. The field staff strongly felt that community expectations had been raised concerning the incentives they would receive and that it would put the field staff in a difficult situation if they were to make any changes. 2) A second issue is since this was a new innovation, there were no previous projects on which to base time and cost estimates. As a result, the project underestimated the amount of time and funds that needed to be dedicated to the innovation and OR. A contract for $35,000 was awarded to NICASALUD and CRS. Caritas staff estimate that implementation of the innovation required approximately 50% of their time. Staff received 400 hours of training on the innovation and BCAs received 40 hours. Materials were developed for education, but due to a lengthy process of validation and approval, initially the BCAs only received a photocopy of the materials, which include one guide for diagnosis and another guide for counseling. Eventually the materials were approved and printed, but the formative research and testing of materials and strategies took over 18 months. An important step in getting people to change behavior is getting them to articulate their beliefs. During the FE, a group interview was conducted with men participating in the innovation. Most of the men were able to clearly articulate their modified views on shared responsibilities and decision making. They talked about the importance of helping their wives at home with tasks such as housework, particularly tasks requiring heavy lifting such as carrying water and wood, in addition to accompanying their wives to clinic visits, MWHs and deliveries, though many said that they could not wait with their wives due to work and asked relatives or TBAs to accompany them. The MTE expressed concern that so many men talked more about supporting housework and child care activities as opposed to support for clinic visits. However, the project staff felt these activities were just as important, particularly helping mothers with heavy household tasks during the later stages of pregnancy. By the time of the FE, many men who were interviewed talked about their experiences with institutional deliveries as well as ANC visits and about saving money for births. Some were quite concerned about what would happen when CRS left and they had to deal with changing personnel at health facilities. It was interesting that when the wives were interviewed, they talked about how supportive their husbands had become with pregnancy related, child care and household activities. They were particularly vocal about how their relationships had changed with their husbands, saying the men were more attentive and caring, and for them this was more interesting than the support they were receiving during pregnancy and birth (although this was also happening). Another issue with scaling up this activity is that it would require that the project work with government at the national level so it can be incorporated into the system. Currently, the national government’s approach to men’s violent behavior is coercive. They have laws that punish violent behavior, but they do not have programs to educate men and women about building healthy relationships. For this reason, particularly in Paiwas and Rio Blanco, the CS project has caught the attention of the police, the Ministry of Education, and the Ministry of Family, as they see the innovation as a successful method to educate communities about healthy family behaviors that tend to reduce violence. These organizations would like to see the innovation expanded and scaled up, but unfortunately with the project ending and no possibility of further CS funding either centrally or from 38 the mission, this is unlikely. Also since this was a community-based initiative, it is unlikely to be picked up by the MOH. One of the interesting lessons learned regarding the BCC activity was that when staff and participants were asked why they thought this intervention had worked, given the challenges of changing rooted male behavior and related traditions often involving violence, they said they thought that the involvement of the religious leaders had played a significant role. The project worked hard to select BCA leaders for this activity that were already models for desired male behavior. The project encouraged Catholic deacons and Protestant pastors to become BCAs, recognizing their level of influence in the communities. Also, many of these men commented during the FE that they used the time after religious services and gatherings as a forum to talk to people about project activities. In addition, the Bishop of Matagalpa was very interested in this activity and encouraged all his priests to support it, hoping to improve men’s behaviors and family involvement. H. Potential for Scale-up The opinion voiced by project staff and partners during the FE is that in order to scale up a project, you need support from the government. As noted above when discussing the men’s behavior change innovation, right now the government’s approach to dealing with men’s violence is coercive. It does not take into account the kind of strategy that relies on educating men to take care of their families’ health and as a result improves the family environment. Also, since this is an activity based at community level and not one that directly interacts with the health system, it is unlikely that it could be scaled up through the health system. In addition as noted above, this activity has only been piloted in 20 communities and the FE team believed it would need to be tested in a broader area before it would be ready to be scaled up nationally. Despite these obstacles for scaling the project, there was a lot of interest in the activity expressed by HWs, municipalities, religious groups, police and other social sectors. But at this point there is no funding or structure to do it. On the other hand, the CHA activity fits well within the MINSA national strategy. There is a new model for community family health being promoted by MINSA called MOSAFC. This model would require having health providers and services at local levels for a small number of communities. Project staff believe this is a good model but that MINSA does not have enough staff to manage it. They suggest that the CHAs would be ideal for operationalizing this model and supplying the needed services. To do this, CRS would have to present it to the national MINSA and so far they have only presented it at the regional level. No guidelines or manuals have been produced for these activities. However, CRS, using its own funds, has elaborated a generic guide for conducting behavior change activities based on the project experience. This guide has not yet been approved by CRS. They are thinking about using this approach in Africa with the new CSPs in Ghana and Benin and also with HIV projects. CRS is also contemplating an adult education guide for BCC activities. When the project began working with the volunteers, they received authorization from MINSA to work with them in the 3 project municipalities. This was done for two reasons: 1) to orient MINSA to the project’s purpose and objectives and 2) so that when men began accompanying their wives to clinic visits and deliveries, MINSA would support this and not turn them away. As it turned out, at the time the project started MINSA was beginning to enforce its norm to humanize the birthing process which includes being able to have anyone you want accompany you to services and deliveries. 39 Two of the project activities were reported through the national HIS. These include the pregnancy registers filled out by volunteers and then summarized by field staff each month and turned into the local HF for inclusion in their monthly data collection and reporting to the SILAIS. In addition, the CHAs use the SICO forms (national HIS) to report their treatment and referral activities monthly to their local HF in accord with MINSA standards and norms. The research organization CIES that was contracted to conduct the baseline, midterm and final assessment of the men’s behavior change operations research activity also has an MPH program. CIES has incorporated these OR evaluations and reports as part of its MPH curriculum. Although NICASALUD was contracted to develop the BCC strategy, there were lessons learned in how to apply it at the community level through Caritas staff’s field testing of various methods and strategies with men. Unfortunately, this process has not been written down as a formal document, though they do have the teaching sessions and laminated tools used for the teaching and counseling activities. Because of their interest in the activity, the project also trained the police chiefs in each of the 3 municipalities on the strategy. The police would like to adapt the methodology and apply it to their work with drugs, alcohol and family relationships. However, they lack the resources to do this and with the project closing there will no longer be technical staff available to help them. I. Conclusions and Recommendations The majority of planned activities were completed and targets met (with the exception of education and training on breastfeeding). There were a number of positive findings during the FE:  The team found that the key design factors contributing to program success include: - Good coordination with the MOH - The reorganization of the volunteer network and coordination among different volunteers - That MINSA and the volunteers took ownership of the CSP strategy - The involvement of some state agencies (MOE, MOF) and the mayoral offices in the strategies - The organization and availability of refurbished Maternity Waiting Homes (MWF). - Community acceptance of project and strategies  The KPC and majority of respondents interviewed cited examples of perceived impact; increased use of health services, institutional births (88-90%), use of MWHs, and ANC (76% in Matiguas had 4 ANC visits, though lower in Rio Blanco and Paiwas). A number of people mentioned a decline in maternal mortality though this is difficult to measure. However, it appears that in project communities the numbers of maternal deaths have dropped.  The two most frequently mentioned positive interventions were cultural adaptation of births including men’s involvement and accompanying the woman to ANC visits and during childbirth. The other is the birth planning which includes figuring out transportation, saving money for the newborn and birth, planning prenatal visits and who will accompany to the birth, etc. The KPC found that the percent of women with birth plan cards rose from 0 to 82% in Matiguas and 69% in Rio Blanco and Paiwas.  There has been a great deal of emphasis placed on knowledge of danger signs during pregnancy, delivery, puerperal, and for the newborn. The KPC demonstrates that knowledge of pregnancy and delivery danger signs rose between 50 to 70 percentage points while mothers knowledge of signs during postpartum rose 20-40 points. Interestingly, men’s knowledge of these danger signs rose at the same levels except for postpartum signs, which were higher among the men (60 and 40%). 40  Use of QI activities has helped to improve the quality of services in HFs. Themes addressed included management of obstetrical complications, clinical and neonatal IMCI, humanization and cultural adaptation of birthing process. Quality norms and lists of pending deliveries were displayed on the walls of all delivery rooms at HFs visited. During interviews it was discovered that most HFs no longer meet in groups instead the QI officer reviews service records and addresses issues individually with the indicated staff member. For this reason, larger issues that require management level decisions go unaddressed. This may be partly due to the highly politicized environment at MINSA.  The network of volunteers is active and strengthened and provides good mutual support within each community and has improved links with HFs through regular monthly meetings to compare reports, make follow-up visits, gather supplies, etc.  The emergency response system has been strengthened through recognition of danger signs, training of 5 ETBs per community, referrals, community planning for emergencies and improved emergency response at HFs. All 100 communities have emergency plans.  Basic medicines and equipment are available for MNC  Sustainability: Big effort to classify all communities of which 85 were certified y MINSA and considered A –level being able to sustain themselves and plan and meet community health needs. All 100 communities have an agreement of mutual support signed by them and MINSA to continue working together on health priorities.  Improved supervision of volunteer activities by MINSA, particularly CHWs, TBAs, and CHAs  All 100 communities have funds for obstetric and other emergencies, which they have raised through donations and raffles. Over the life of the project these communities have maintained funds amounting to more than 150,000 Córdoba. Conclusions The various MINSA strategies that the CSP is implementing began to show their potential as an effective means of diminishing maternal/neonatal morbidity and mortality from the synergy among interventions. A lesson learned is that the CSP’s work to involve men in health care coincided nicely with the government’s emphasis on humanization and cultural adaptation of birthing practices and use of birth plans. These MOH directives encouraged health facilities to allow husbands and other family members to accompany women for ANC visits and deliveries. This and the community level education activities with men combined synergistically to improve women’s access to health care. One of the questions voiced by staff, MINSA, SILAIS and the communities is why the innovation projects were put out there if there aren’t resources available to further expand them. They see this innovation as a pilot phase that once proven successful should be taken to scale. Given that Nicaragua is no longer included on the CSHGP eligible country list and that the USAID mission is closing its health office (as well as several other donors), there do not appear to be opportunities to fund such an activity and these individuals wondered why provisions weren’t made for further development or expansion of such innovative projects. The project staff also concluded that the innovation activity should have been a separate project given the tremendous amount of time and effort needed to complete the activity indicating that 50 percent of staff time and significant resources were devoted to this effort. One of the insights gained by the staff in conducting the innovation is that the educational method used to approach men was very important. They found that as men became more supportive of their family’s health needs, joint decision making and helping at home, the levels of family violence 41 diminished. Also, the use of religious leaders as BCAs was important as they were influential and also good family role models. In sum, staff believed that if the CSP had used a more aggressive approach to confront men’s behavior, the men would have become defensive and not been receptive. However, by focusing on specific health and household tasks in a non-threatening way, the project was able to convince them to support healthy family behaviors. Although the quantitative CIES evaluation of the innovation activity did not show a difference between case and control communities in terms of ANC visits, overall there was a significant increase in the numbers of women attending ANC during pregnancy. Also, there were significant improvements in terms of joint decision making and women who sought care for themselves and their children together with their husbands. Recommendations Given that the project ended in September 2012, conducting further activities was not possible. The project implemented its plans for certifying communities and closed the project as of September 30th. However, if resources become available for some additional steps, the following are recommended: Disseminate results  The project has much to contribute to the Child Survival knowledge base, most notable being the experience involving men in care seeking and shared decision making. Also the experience with the humanization and cultural adaptation of the birthing process and use of MWH experience should be documented and disseminated.  USAID/Nicaragua recommended that the project’s results be presented to central MINSA and also to the medical schools, which are lacking in strategies for community involvement in health care services.  The CSP’s successful CHA experience needs to be presented to central MINSA as a possible strategy for their implementation of the new “model for community family health” initiative (MOSAFC).  CRS should continue to present results of the CSP experience at CORE Group meetings and other forums.  CRS should consider including the CSP experience as one of its “Most Significant Change” case studies using its methodology to capture qualitative results from the project, particularly the men’s innovation activity. Other recommendations  If the innovation activity were to be replicated and taken to scale, the staff would have to drastically reduce the formative research phase during which the methodology was extensively tested and materials validated over a period of 18 months. They would also have to address whether they would use so many incentives, which add cost and hamper sustainability. CRS did develop a more streamlined model to scale up the innovation and submitted this proposal to the Grand Challenges competition and although it was not funded, the staff do believe the methodology can be streamlined and still be effective. But this streamlined model may need further testing before it is implemented on a large scale.  It would be helpful to have a qualitative evaluation of the attitudes and practices of families participating in the men’s innovation intervention that could explain more clearly why the values and perceptions have changed among women and men and provide more information regarding case and control group differences. 42  Recommend that MINSA evaluate and address the reduced effectiveness of the QI system and the resulting lack of team decision-making to improve services.  USAID should review some of the KPC indicators. For example, the question that asks mothers if their newborn received proper cord care at birth is difficult for mothers to answer, particularly if they had hospital deliveries, since it is likely they may not have been paying attention to their newborns during that time. Questions about the third stage of labor are also difficult for mothers to answer.