MIDTERM EVALUATION OF THE REPROSALUD PROJECT EXECUTIVE SUMMARY OF RESULTS By: Bonnie Shepard Delicia Ferrando Arlette Beltran Submitted by: LTG Associates, Inc. and TvT Associates, Inc. Submitted to: The United States Agency for International Development in Peru Under Contract No. HRN-I-00-99-00002-00, Task Order Number 812 March 2002 Midterm Evaluation of the Reprosalud Project, was prepared under the auspices of the U.S. Agency for International Development (USAID)/Peru under the terms of the Monitoring, Evaluation and Design Support (MEDS) project, Contract No. HRN-I-00-99-00002-00, Task Order No. 812. The opinions expressed herein are those of the authors and do not necessarily reflect the views of LTG Associates, TvT Associates, or USAID. Information about this and other MEDS publications may be obtained from: Monitoring, Evaluation and Design Support (MEDS) Project 1101 Vermont Avenue, N.W., Suite 900 Washington, DC 20005 Phone: (202) 898-0980 Fax: (202) 898-9397 cbillingsley@medsproject.com www.medsproject.com ACRONYMS AND FOREIGN TERMS CBO Community-based organization MOH Ministry of Health NGO Nongovernmental organizations RTI Reproductive tract infection STI Sexually-transmitted infection USAID United States Agency for International Development CONTENTS ACRONYMS AND FOREIGN TERMS EXECUTIVE SUMMARY .................................................................................................1 A. Background Information....................................................................................1 B. Summary of Main Conclusions..........................................................................3 Conclusions on Impact.................................................................................3 Organizational Performance and Costs........................................................4 Barriers and Facilitating Factors in the Advocacy Program........................5 C. Recommendations ..............................................................................................6 Replication of Phase One .............................................................................6 Strengthening the Advocacy Program.........................................................7 Maximizing and Sustaining Impact in Current Districts .............................7 Evaluation and Monitoring ..........................................................................7 D. Final Comments.................................................................................................8 ANNEXES A: Objectives and Methodology of the Midterm Evaluation............................................11 B: Results Framework and Impact Study Tables.............................................................15 C: Impact Study Graphs....................................................................................................25 D: Detailed Summary of Impact Findings and Recommendations ..................................31 E: Process Evaluation Tables ...........................................................................................41 F: Cost Analysis Data.......................................................................................................49 TABLES Table 1: Overview of Results and Indicators..................................................................16 Table 2: Summary of Impact of Interventions.................................................................19 Table 3: Comparison of Baseline and Midterm Evaluation Results Framework Indicators: Intervention Communities and Control Communities ......................................21 Table 4: Reproductive Health Indicators.........................................................................26 Table 5: Gender Issues Indicators....................................................................................27 Table 6: Reproductive Health and Gender Issues Indicators...........................................28 Table 7: Synopsis of General Recommendations to Reprosalud.....................................44 Table 8: MOH Limitations and Financial Issues.............................................................45 Table 9: Impact on Women.............................................................................................46 Table 10: Impact on Men...................................................................................................46 Table 11: Did Reprosalud Have an Impact on Use of Services?.......................................47 Table 12: Changes Implemented in Services to Improve Quality.....................................47 Table 13: Combined Effectiveness Indicators Constructed for the Cost Analysis............50 Table 14: RH Effectiveness Ratios at National Level.......................................................53 Table 15: Cost per Beneficiary for the Main Activities of the Reprosalud Program........53 FIGURES Figure 1: Four Prenatal Visits in Health Services Last Pregnancy.................................26 Figure 2: Health Professional Assisted Last Childbirth..................................................26 Figure 3: Percent of Women Who Decide What the Money They Earn Will Be Spent on............................................................................27 Figure 4: Percent of Women Who State Their Unwillingness to Be Forced or Convinced to Have Sex..............................................................27 Figure 5: Percent of Women Who Have Attended a Health Service for Reproductive Health or Family Planning Services............28 Figure 6: Percent of All Women Who Would Go to a Healthcare Facility in the Event of Symptoms that Are Warning Signs of Risk ............................28 Figure 7: Percent of Women and Men Who Believe that Housework Is As Important As the Husband’s Work.........................................................29 Figure 8: Shared and Unshared Priorities........................................................................42 Figure 9: Essential Communications Flows in Advocacy Program, Phase Two ............43 1 EXECUTIVE SUMMARY A. BACKGROUND INFORMATION The Reprosalud program is a unique example of a reproductive health and gender equity program whose beneficiaries are in hard-to-reach communities. The program is a 10-year cooperative agreement1 (1995–2005) between the United States Agency for International Development (USAID) and Movimiento Manuela Ramos, a Peruvian feminist organization, the purpose of which is to improve the reproductive health of low-income women in rural and peri-urban zones in Peru. The principal expected result of Reprosalud is that women will increase their use of specific interventions that could protect their reproductive health, including a range from individual health-protective behaviors to increased use of formal health services. Reprosalud also aims to have a positive impact on socio-cultural factors that affect women’s health, especially gender issues. Reprosalud represents a unique learning opportunity for the reproductive health field. There are very few examples worldwide of participatory community-based reproductive health/gender equity programs that have been implemented on such a massive scale in hard-to-reach communities of linguistic or ethnic minorities. Phase One of the project (1996–2000) involved subgrants to more than 2,500 women’s community-based organizations (CBOs) in low-income communities in 8 departments,2 including many in which the main language is Quechua or Aymara. Seventy percent of the communities involved in Phase One were rural. As of December 2001, 231 subgrantee CBOs have completed 2 subprojects and conducted participatory educational interventions on 2–3 reproductive health problems identified by the community.3 Over 190,000 individuals 1 The agreement was signed in August 1995 and the project began in 1996. It has had two 5-year authorizations, with a current end date of September 30, 2005. 2 The department is the main geo-political division in Peru. In the Ministry of Health, the next level of authority below the national is departmental. 3 One hundred and sixty two subgrantees have completed 3 themes, and 69 have completed 2. Each subproject involves several neighboring “associated” CBOs as well, so that the total number of women’s organizations involved was 2,568. Basic Principles of the Reprosalud Program · Commitment to gender equity and women’s empowerment, · Commitment to participatory processes that put community members in charge, · Promotion of sexual and reproductive health and rights, and · Respect for indigenous cultures, integrating modern health knowledge with traditional knowledge and practices that are not harmful to health. MIDTERM EVALUATION OF THE REPROSALUD PROJECT 2 (65 percent women, 35 percent men) participated in the Phase One subprojects, and up to one million family members have benefited indirectly.4 The Reprosalud workshops use a participatory methodology that encourages participants to reflect on both the physiological and social causes of health problems, as well as possible solutions. The basic module addresses anatomy, physiology of reproduction, and gender roles. Based on the priorities of the great majority of CBOs, Reprosalud developed four modules: family planning, pregnancy and childbirth, reproductive tract infections (RTIs), and violence, with separate editions for men and women, and for the populations in the highlands and jungle areas. Phase One included an additional component to empower women also supported income￾generation through microcredit (“community banks”) and product development— producing and marketing handicraft items for purchasers with bulk orders. However, this component is no longer part of the Reprosalud Program, since the microcredit program became self-sufficient in 1999 and product development received funding from another source in 2000.5 In Phase Two, which started in 2001, the planned focus for Reprosalud’s subgrant program was “Promotion and Defense,” i.e., advocacy conducted by the CBOs with the health sector. The objective of the advocacy program is to establish a mutually beneficial and sustainable relationship between the health referral center6 for each area and the elected presidents and trained promoters of local CBOs in 78 defined catchment areas. Negotiated agreements form the basis of subgrants, and are expected to result in greater numbers of women using the public health services, and services that are more acceptable to community women and responsive to their needs. In Phase Two subprojects, the CBO promoters will run community educational workshops, refer women to the health services, and collect and provide feedback on the quality of services. In turn, the health professionals are expected to agree to do whatever they can to make the services more acceptable and responsive to the women from the communities in the catchment area. To determine the content of subprojects, the CBO promoters carry out a new diagnosis of reproductive health needs and users’ views of service quality. Thus, subprojects might include increased education on a particular topic, training of CBO promoters by Ministry of Health (MOH) staff on key topics, training of service providers in quality, and provide adequate funds to better equip a health post or center. 4 Database reports, Reprosalud, December 2001. Project data indicate that each CBO member has an average of 5.3 family members. The districts in which Reprosalud is active represent 10% of the population of Peru. 5 This evaluation concentrates on the reproductive health component of the program. The cost analysis analyzes the cost per beneficiary of the income-generation component. The product development program is now called “MERCOMUJER,” and receives support from the Small Enterprise Department Unit of USAID/Washington. 6 Health centers are secondary-level facilities that serve as referral centers for a catchment area, with basic surgical, obstetrical, and hospital facilities. While hospitals are tertiary-level facilities, often they serve as the basic referral center, especially in urban areas. EXECUTIVE SUMMARY 3 B. SUMMARY OF MAIN CONCLUSIONS Conclusions on Impact The data from the three components of the evaluation7 support the findings presented below. ß In the impact study, Reprosalud communities showed significant gains in all but 3 of the 39 indicators in the results framework.8 The comparison communities showed significant gains in all but nine indicators. However, in 17 of the 39 indicators, the relative level of the gains in Reprosalud communities ranged from 10 percent to 92 percent higher than the gains in the comparison communities.9 Significant Increases in the Strategic Objective of Women’s Use of Reproductive Health Interventions Are Attributable to Reprosalud ß Compared to women in the comparison communities, women in Reprosalud communities were 27 percent more likely to have their last childbirth attended by a health professional, 15 percent more likely to have had 4 prenatal visits in health services, and 18 percent more likely to seek treatment for RTIs from both community and formal sector sources. ß Women in Reprosalud communities were 15 percent more likely to use a family planning method, and 18 percent less likely to have unmet need for family planning.10 Using composite indicators, the cost effectiveness study showed greater impact in Reprosalud communities on use of childbirth and prenatal services than on family planning use. ß More than 75 percent of health professionals interviewed attributed increases in service use wholly or partially to Reprosalud’s efforts. The Gains Attributable to the Project in the Intermediate Results Are Mixed ß Reprosalud has significantly increased womens’ knowledge of modern contraceptive methods, RTIs, and the fertile days of the menstrual cycle. 7 This midterm evaluation has three components: a quantitative impact study, a process evaluation, and a cost analysis. This summary draws on the main findings and conclusions of the three studies. The quantitative impact study uses a quasi-experimental design that compares results between 25 sub-grantee communities and 25 comparison communities. The objectives and methodology are detailed in annex A. 8 Refer to Results Framework in annex B. See also graphs in annex C. 9 The gains attributable to the program were calculated through use of the odds ratio. See annex B for Tables from the Impact Study. 10 This indicator includes users of rhythm method who do not correctly identify the fertile period. MIDTERM EVALUATION OF THE REPROSALUD PROJECT 4 ß Reprosalud has had a positive, but less significant effect on achieving more equitable gender relations between women and their spouses and families, with more impact on women than on men. · There is strong support among all stakeholders for continuing to work on gender issues and with men, especially at the community level, with respondents stating that violence and other negative attitudes continue in their communities. ß For many intermediate indicators for both reproductive health and gender, both intervention and comparison communities increased significantly between baseline and midterm, but there is no significant gain attributable to the project. The Reprosalud project period coincided with a time of heavy investments by MOH, donor agencies11 and nongovernmental organizations (NGOs) in improving access and quality of care in reproductive health services and in increasing gender equity. ß The process evaluation gave evidence of increased communication within the family and with providers, of women becoming more capable end users of health services, of increased civic participation by women, and of lives saved through actions by the CBO promoters. Organizational Performance and Costs ß Movimiento Manuela Ramos has met Reprosalud’s multiple organizational and programmatic challenges with competence and flexibility. This Peruvian women’s NGO has implemented a large-scale program that surpassed its original objectives for coverage, using a highly participatory methodology that defies standardization, in hard-to-reach communities that posed numerous logistical and cultural barriers. Reprosalud directors demonstrated flexibility by acceding to community women’s requests to work with men. The ensuing men’s educational program took place simultaneously with the women’s program, and incurred a major unplanned investment in staff hiring, training, and materials development. · Most external stakeholders recognize that Reprosalud’s methodology and guiding principles have gained them a uniquely high level of acceptance in the communities with a long history of mistrust of outsiders. They highlight Reprosalud’s impact on increases in knowledge and use of services. ß The cost per beneficiary for the 5-year span of Phase One (1996–2000) was US$48.51, which is roughly comparable to the cost to the MOH in Peru of US$47.22 of attending one pregnant woman through prenatal care and 11 See a complete listing in footnote 30 in the Impact Study. EXECUTIVE SUMMARY 5 childbirth. Given this comparison, the intensive nature of the educational intervention and the cost of working in rural areas, Reprosalud’s costs seem reasonable. However, the evaluators lacked appropriate comparisons. Cost analyses from comparable community-based, health education programs were not available to the evaluation team. ß A cost-benefit analysis should take into account that the program is expected to generate benefits for years to come, both within the districts where it intervened, and through replications by other agencies using the Reprosalud educational materials and methodology. The estimated cost per beneficiary of replication—after subtracting certain startup and research costs and the cost of developing the educational materials—was 92 percent of the actual cost.12 ß Program managers have showed concern for cost effectiveness in their decisions. Due to concerns for cost in Ucayali, and potential impact in San Martín and Lima,13 in 2000 the project withdrew from these three departments. They also amalgamated the two Puno offices. Barriers and Facilitating Factors in the Advocacy Program ß The opinions of both internal and external stakeholders who will be important to the success of Phase Two are mainly favorable, laying a sound foundation for the next four years of the program. These findings from the process evaluation related to health authorities and health providers, CBO promoters and women, local authorities, and other NGOs. ß The MOH’s perennial emphasis on increasing service use, their investment in improving quality of care as key to achieving this goal, and their increasing use of user feedback as the key criteria to evaluate quality, are all facilitating factors for the advocacy program. ß Most of the demands of the CBOs coincide well with MOH priorities, which focus on all interventions that reduce maternal mortality, including family planning. MOH and CBO priorities do not coincide in the following areas: low-cost treatment of RTIs and sexually transmitted infections (STIs), professional attendance at home births, and some aspects of culturally appropriate childbirth practices.14 ß The MOH’s budgetary limitations will pose a barrier to some of the CBO demands for reduced waiting time, provision of low-cost medicines for curing 12 For cost of replication, we assumed that the replicating agency was already operating in the area, with the offices and vehicles needed to reach rural communities, and that they would only need to reprint the existing educational materials. 13 Jungle communities in Ucayali that fit the profile for program intervention were mainly accessible by boat. In San Martin and Lima, the1997-98 baseline studies showed much higher rates of reproductive health knowledge and service use than in the remaining five departments in the highlands. 14 See graph on shared and unshared priorities in annex E. MIDTERM EVALUATION OF THE REPROSALUD PROJECT 6 RTIs and STIs, and capacity to resolve other health problems at the local (health post) level.15 C. RECOMMENDATIONS Following are recommendations for replication and extension of coverage of the educational model, the Advocacy Program, and the evaluation design. These recommendations are described more fully in annex D and in the Process Evaluation Report (see pages 49–54). Replication of Phase One To maximize the benefits from the investment in Reprosalud, it is recommended that program managers give high priority to encouraging replication through other NGOs, educational programs, and international agencies during the next four years. Therefore, staff should do active outreach to other institutions, and develop and provide a full kit of all the tools, manuals, and materials developed for Phase One. Replication by the MOH or by the current CBO promoters is included in the plans for a few Phase Two subprojects, but these are not the main focus of Phase Two, and there are no funds earmarked in the budget for this purpose. We recommend that Reprosalud involve male promoters in any replications in new communities that express interest in workshops for men. An important challenge in replication efforts is to assist other organizations to incorporate the guiding principles of the program in the replication. Additional intervention and training may be necessary to ensure that these principles are uniformly applied (see box on page 1). Suggested modifications in the Phase One model for replication efforts by other agencies include the following: ß Experiment with introducing male and female educational components simultaneously by working with both male and female CBOs. ß Strengthen the focus on gender issues through use of radio programs and videos. These educational materials will be developed by Reprosalud in the communications program in Phase Two. ß Train health providers to be responsive to community feedback and to monitor quality of care where no complementary systems exist. 15 A key demand in many rural and periurban areas is to increase the capacity of MOH facilities to resolve problems at the local, primary-level, so that women do not have to travel to the health center in order to resolve their health problem. In some cases, the health center cannot resolve it either. EXECUTIVE SUMMARY 7 Strengthening the Advocacy Program The process evaluation provided data from which several suggestions for improving the advocacy program were developed, including close monitoring of the communications to and from the community level, and of the level of effort of both CBO promoters and presidents. Maximizing and Sustaining Impact in Current Districts Reprosalud’s educational activities should continue in current communities and expand to new ones in the same district. These activities should focus on areas of weakness identified in the evaluation. As the main vehicle to maximize impact in current districts, Reprosalud should implement a communications program to extend coverage of the program’s educational messages, to reinforce knowledge, attitude and behavior changes, and to reach youth and men with messages tailored to them. Reprosalud should involve trained male and youth promoters to facilitate outreach to their peers—male and youth involvement is important to achieve the goal of creating a sustainable change in the culture of the community regarding reproductive health and gender issues. Furthermore, data from the process evaluation suggests that the support from local authorities is stronger when they are involved in educational activities for men. Reprosalud can involve male and youth promoters by ß enlisting their assistance for the development of products and messages for the communications program, ß inviting them to education and training activities for promoters taking place in Phase Two, and ß negotiating recognition for the male promoters as MOH community agents. Evaluation and Monitoring To strengthen the evaluation and monitoring component of the program, see all of Section G in the process evaluation report for a full discussion. ß Reprosalud needs to improve oversight by hiring an evaluation and monitoring director. ß Reprosalud should increase the resources devoted to collection and analysis of qualitative process and impact data, to answer important questions about Phase Two. Additional secondary analysis of the quantitative data already collected would also enable the program to compare performance by department, and by other variables such as the presence of a male, youth or income-generation component. MIDTERM EVALUATION OF THE REPROSALUD PROJECT 8 D. FINAL COMMENTS The data indeed suggest that the program has had an important positive impact on its beneficiaries and has brought to them significant benefits over and above those realized in most communities in these departments, where other MOH and NGO programs have been operating with similar goals. Reprosalud is distinct because of its investment in community-level, participatory educational strategies in hard-to-reach communities, and its guiding principles. The most interesting aspect of the results on Reprosalud’s impact is that its gains as compared to nonintervention communities are so significant in indicators of use of reproductive health interventions, while mixed in many intermediate indicators that supposedly lead to these health-protective behaviors. These results suggest that such intensive community-based strategies may not be necessary to attain some desired increases in knowledge, opinions, and attitudes. However, in these communities that have historically been hard to reach, the Reprosalud model gave the complementary input needed to achieve significant gains in behavior—in use of reproductive health interventions. Some recommendations in this report arise from a systems approach to the dynamics of cultural change, whether in communities, in family, or in health services. This approach assumes that change happens faster and with less resistance when working with two or more parts of a system than when working in just one. For example, in efforts to achieve gender equity, it is probably more efficient to work with both men and women than to work with women only. In the same vein, the findings from this evaluation suggest that when aiming to improve the use of services among members of these hard-to-reach communities, “supply-side” changes in the quality of care and in access often are not enough to achieve coverage goals. In these communities, high levels of mistrust and lack of knowledge inhibit change until culturally affirming and community-run educational efforts help to overcome these obstacles. ANNEXES A: Objectives and Methodology of the Midterm Evaluation............................................11 B: Results Framework and Impact Study Tables.............................................................15 C: Impact Study Graphs....................................................................................................25 D: Detailed Summary of Impact Findings and Recommendations ..................................31 E: Process Evaluation Tables ...........................................................................................41 F: Cost Analysis Data.......................................................................................................49 ANNEX A OBJECTIVES AND METHODOLOGY OF THE MIDTERM EVALUATION MIDTERM EVALUATION OF THE REPROSALUD PROJECT 12 Planning for this midterm evaluation began in June 2001 and data collection took place in October and November 2001, at a point in the life of the project when the educational activities for Phase One had concluded, and the Advocacy Program for the second five￾year phase was just gearing up. All plans to extend coverage and increase impact were on hold, awaiting the results of this evaluation. The general objectives of the evaluation are: 1. To determine the impact of the educational program in the first 5 years (Phase One) of the Reprosalud project on the participating women and their communities, according to the results framework. 2. To collect lessons learned on Phase One for purposes of replication, and 3. For Phase Two, 2001-2005: a. To recommend improvements and an evaluation design for the Advocacy Program, and b. To recommend next steps that would extend the coverage and maximize the impact of the Phase One educational program. The evaluation has three components: a quantitative impact study, a process evaluation, and a cost analysis. This executive summary summarizes the main conclusions of the three components with regard to the general objectives. The quantitative impact study uses a quasi-experimental design that compares results between 25 subgrantee communities and 25 comparison communities.1 Separate questionnaires for both women and men have close-ended questions based on the Results Framework and indicators of the project. Reprosalud’s Monitoring and Evaluation Unit collected the baseline and midterm data. Fifty percent of the households in the study were CBO members, and 50% community members; in larger communities. For a variety of reasons related to the logistics of data collection, the unit of analysis is the social network of the CBO members, and not the individuals surveyed. Because there were significant differences on many indicators between the baseline values of the intervention and comparison communities, the evaluation measures differences in the amount of increase or decrease, using the Odds Ratio to compare the relative change from the baseline in Reprosalud and comparison communities. The Odds Ratio allows the evaluator to attribute changes to Reprosalud with more certainty. The two main sources of information for the midterm process evaluation were a round of primary data collection in five departments and at headquarters through semi-structured interviews by B. Shepard and D. Ferrando, and review of project documents. The 1 The quantitative data was problematic, because the program has suffered from under-investment in evaluation since its inception. See a full explanation of the problems with the data in section I.B of the Cost Benefit Analysis. ANNEXES 13 evaluation team collected information from health authorities and providers, local authorities, CBO promoters and members, other NGOs involved in reproductive health, Reprosalud staff, and USAID staff. The evaluation team conducted 168 semi-structured individual interviews and 5 group interviews with the informants in 2-3 districts in five departments and in Lima.2 Based on field notes, the results were loaded into a database, and analyzed by subject codes attached to the questions. The objectives of the cost analysis were: 1) to determine the cost per beneficiary at the national and departmental levels for each of the components of the program: reproductive health, micro-credit, and product development; 2) to analyze the cost-efficiency of each program nationally and by department3 ; and 3) to determine the cost of replicating the reproductive health component of the program. To determine the cost per beneficiary, the evaluator assigned headquarters and non￾program costs to the departments and to a program. She divided the total departmental and national costs by their corresponding number of beneficiaries. To analyze cost￾efficiency, she used the same 25 comparison and intervention communities as the impact study, and compared the levels of change on four composite indicators of reproductive health and gender equity. To determine cost per beneficiary of replication, she subtracted some costs related to research and materials development, and other startup costs that would not apply to an existing program in similar areas. 2 See Methodology Annex I in the process evaluation report for a table describing the numbers and types of interviews in the evaluation. Three ex-staff of USAID and Reprosalud who were closely involved in designing and implementing Reprosalud were interviewed as well. 3 Unfortunately, the size and representativity of the sample in each region does not allow valid comparisons between departments. MIDTERM EVALUATION OF THE REPROSALUD PROJECT 14 ANNEX B RESULTS FRAMEWORK AND IMPACT STUDY TABLES MIDTERM EVALUATION OF THE REPROSALUD PROJECT 16 Table 1: Overview of Results and Indicators Results Indicators SO: Increased use by women of interventions in reproductive health 1. % of women who had at least 4 prenatal visits to a healthcare professional during a pregnancy occurring within the last two years (since the last study) 2. % of women whose last delivery, occurring within the last two years, was performed by a healthcare professional 3. % of women who use some method of birth control 4. % of women with an unmet need for family planning (including those who use the rhythm method and are unsure of their own fertile days) * % of women with an unmet fertile-period need (ENDES 96 rhythm￾method users who do not know when their own fertile days are have been added to this group) 24. % of women who have gone to a healthcare facility for consultation regarding reproductive health or family planning 30a. % of women who have gone to a healthcare professional due to discomfort related to RTIs (or vaginal discharge) RI 1: More equitable gender relations between women and their partners/families 5. % of women whose partner helps care for the children if they become ill 6. % of women who decide how to spend the money they earn 7. % of women who make joint decisions with their partner about sexual relations, birth control methods, and number of children 9. % of women who share one or more household chore with their partner 11. % of women who make a joint decision with their partner about the educational level their children should reach 11ª. % Of women who have spoken with their partner about the number of children they want to have. RI 1.1: Strengthened ability of women to achieve equality in gender relations 12 % of women who agree that a woman should be able to decide to use family planning even if her partner objects (This is also an indicator for RI2) 12.1 % of women who state their unwillingness to be forced or convinced to have sex 13. % of women who have spoken with their partner about family planning more than two times in the last 12 months % of women who have spoken with their partner about STD’s more than two times in the last 12 months 14. % of women who would go to the police/authorities if their partner abused them 15. % of women who have spoken with their partner about the risks of pregnancy and postpartum in the last 12 months 16. % of women who have spoken with their children age 12 years and older about male-female relationships and family planning more than two times in the last 12 months 16a. % of women who know where to go for help or advice if they are abused RI 1.2: Increase in positive attitudes of men towards equitable relationships with women and family 17. % of men who believe that it is never right to hit a woman 18. % of men who believe that a man does not have the right to force an unwilling women to have sex 18.1 % of men who believe that both parents should take care of the children when they become ill ANNEXES 17 Results Indicators RI 1.3: Increase in women's knowledge about gender equality 19. % of women who believe that the work they do outside the home is as important as the work done outside the home by their partner 19.1 % of men who believe that the work they do outside the home is as important as the work done outside the home by their partner 20. % of women who believe that housework is as important as the work done outside the home by their partner 20.1 % of men who believe that the housework done by their partner is as important as his own work done outside the home 21. % of women who believe that their sons and daughters should reach the same level of education 21.1 % of men who believe that their sons and daughters should reach the same level of education 23. % of women who got a higher score on the final (comprehension level) exam for the gender chapter RI 2: Increased capacity of women in use of reproductive health services 12. % of women who agree that a woman should be able to decide use family planning even if her partner objects (This is also an indicator for RI 1.1) 33. % of women who have incurred a health expense in the last 12 months 33.a. % of women who use health services even though they cannot pay the fees. RI 2.1: Increased capacity of women as end users of formal health services % of women who sought help from any source for discomfort related to RTIs. 32. % of women who take care of their health in order to feel well RI 2.2: Increase in women's health as a priority within the home and community 29. % of women who would go to a healthcare facility in the event of symptoms that are warning signs of risk. % of providers who believe that communication with women users has improved % of providers who note that users are more assertive, i.e. more apt to give feedback or to make demands. RI 2.21: Women have greater willingness to use health services 29. % of women who would go to a healthcare facility in the event of symptoms that are warning signs of risk. (also RI2.2) 31. % of women who would advise someone with RTIs to go to a healthcare facility RI 2.3: Women have increased access to income-generating sources, credit, and markets 34. 35. 36. 37. RI 2.4: CBO’s increase their abilities to organize reproductive health services in communities 35. 36. 37. 38. Service sub-projects Microcredits and product development MIDTERM EVALUATION OF THE REPROSALUD PROJECT 18 Results Indicators RI 2.5: Women participate more in identifying and performing activities according to their priorities 34. 35. 36. 37. RI 2.6: Increase in women's knowledge about their reproductive health needs 38. % of women who know how at least one modern contraceptive method works 39. % of women using the rhythm method who know the fertile days of their cycle 40. % of women who can recognize some symptom of pregnancy or postpartum warning signs that indicate risk 43. % of women who know how RTIs are spread 43a. % of women who have heard about Pap and breast exams % of women who know to use a condom to protect against AIDS/STD’s % of women who got a higher score on the final (comprehension level) exam for the basic and specific module RI 3: Women in CBO’s actively participate in the proposal formulation, adaptation, and supervision process for reproductive health programs RI 3.1: Authorities are sensitized and prepared to make changes that include sexual and reproductive rights from a woman’s perspective RI 3.2: Women in CBO’s have an increased ability to represent, defend, and negotiate the interests and sexual and reproductive rights of women Reproductive Health sub￾projects and others ANNEXES 19 Table 2: Summary of Impact of Interventions Net increase, in percentage points, between the baseline and the midterm evaluation Indicators InterventionControl Odds Ratio Odds Ratio from 1.199 to 1.999 VERY SIGNIFICANT 38. % of women who know how at least one modern contraceptive method works 42.0 17.0 1.920 43. % of women who know how RTIs are spread 14.1 6.7 1.599 32. % of women who take care of their health in order to feel well 8.3 -8.8 1.325 2. % of women whose last delivery, occurring within the last two years, was performed by a healthcare professional 11.9 1.8 1.274 39. % of women using the rhythm method who know the fertile days of their cycle 5.8 0.1 1.253 6. % of women who decide how to spend the money they earn 4.5 -2.0 1.225 4.1 % of women with an unmet need for family planning -9.2 -5.0 0.817 4. % of women with an unmet need for family planning (including those who use the rhythm method and are unsure of their own fertile days) -11.2 -3.2 0.823 Odds Ratio from 1.100 to 1.198 SOMEWHAT HIGH 12.1 % of women who state their unwillingness to be forced or convinced to have sex 13.4 3.3 1.194 30. % of women who have sought treatment for symptoms of RTIs (or vaginal discharge) 10.2 -1.0 1.178 3. % of women who use some method of birth control 13.4 4.0 1.153 1.a % of women who had at least 4 prenatal visits to a healthcare professional during a pregnancy occurring within the last two years 27.5 18.8 1.148 14. % of women would go to the police/authorities if their partner abused them 24.1 21.5 1.133 7. % of women who make joint decisions with their partner about sexual relations, birth control methods, and number of children 9.2 5.2 1.129 21. % of women who believe that their sons and daughters should reach the same level of education 5.2 -4.9 1.118 11. % of women who make a joint decision with their partner about the educational level their children should reach 6.6 0.2 1.111 Odds Ratio from 1.050 to 1.099 MODERATE 19.1 % of men who believe that the work they do outside the home is as important as the work their partner does outside the home 17.7 12.4 1.097 43.b % of women who believe that while a women is breastfeeding it is dificult for her to become pregnant 22.3 16.0 1.091 16.a % of women who know where to go for help or advice if they are abused 20.0 14.7 1.083 27. % of women who have gone to a healthcare facility for prenatal and postnatal care, and who believe that the services provided by the nearest facility are good (or very good) 5.8 1.5 1.081 MIDTERM EVALUATION OF THE REPROSALUD PROJECT 20 Net increase, in percentage points, between the baseline and the midterm evaluation Indicators InterventionControl Odds Ratio 13. % of women who have spoken with their partner about family planning more than 2 times in the last 12 months 5.1 2.7 1.080 43.a % of women who have heard about Pap and breast exams 18.2 12.2 1.069 30.a % of women who have gone to a healthcare professional due to discomfort related to RTIs (or vaginal discharge) 10.8 7.0 1.068 Odds Ratio from 1 to 1.049 LOW 18. % of men who believe that a man does not have the right to force an unwilling woman to have sex 6.9 3.0 1.046 17. % of men who believe that it is never right to hit a woman 6.6 4.2 1.041 21.1 % of men who believe that their sons and daughters should reach the same level of education 2.3 -0.7 1.034 33. % of women who incurred a health expense in the last 12 months 0.5 -0.5 1.021 12. % of women who agree that a woman should be able to decide to take care of herself even if her partner objects -4.5 -4.6 1.006 11.a % of women who have spoken with their partner about the number of children they want to have 4.8 4.7 1.003 Odds Ratio less than 1 PROGRAM HAD NO EFFECT 29. % of women who would go to a healthcare facility in the event of symptoms that are warning signs of risk 29.0 25.4 0.994 9. % of women who share one or more household chore with their partner 23.5 19.3 0.988 19. % of women who believe that the work they do outside the home is as important as the work done outside the home by their partner 6.4 6.8 0.986 5. % of women whose partner helps care for the children if they become ill 6.0 5.3 0.982 26. % of women who believe that the services provided by the nearest healthcare facility are good (or very good) 10.7 16.9 0.936 20. % of women who believe that housework is as important as the work done outside the home by their partner 10.4 12.5 0.926 20.1 % of men who believe that the housework done by their partner is as important as his own work done outside the home 11.2 14.6 0.910 24. % of women who have gone to a healthcare facility for consultation regarding reproductive health or family planning 15.9 17.2 0.909 16. % of women who have spoken with their children age 12 years and older about male-female relationships and family planning more than two times in the last 12 months 8.9 6.5 0.845 40. % of women who can recognize some symptom of pregnancy or postpartum warning signs that indicate risk 26.6 26.8 0.782 ANNEXES 21 Table 3 COMPARISON OF THE BASELINE (BL) AND THE MIDTERM EVALUATION (IE) RESULTS FRAMEWORK INDICATORS: INTERVENTION COMMUNITIES AND CONTROL COMMUNITIES Intervention Community Control Community Indicators BL IE Dif. n BL n IE z Signif BL IE Dif. n BL n IE z Signif Odds Ratio OR SO: Increased use by women of interventions in reproductive health 1.a % of women who had at least 4 prenatal visits to a healthcare professional during a pregnancy occuring within the last two years 55.6 83.1 27.5 372 261 -7.240 Sig (0.05) 62.2 81.0 18.8 312.0 294.0 -5.114 Sig (0.05) 1.148 2. % of women whose last delivery, occurring within the last two years, was performed by a healthcare professional 36.1 48.0 11.9 465 279 -3.200 Sig (0.05) 40.9 42.7 1.8 425 321 -0.494 1.274 3. % of women who use some method of birth control 58.4 71.8 13.4 1046 855 -6.071 Sig (0.05) 60.6 64.6 4.0 1076 823 -1.783 Sig (0.05) 1.153 4. % of women with an unmet need for family planning (including those who use the rhythm method and are unsure of their own fertile days) 48.4 37.3 -11.1 1034 848 4.834 Sig (0.05) 50.1 46.9 -3.2 1071 818 1.379 Sig (0.10) 0.823 4.1 % of women with an unmet need for family planning 27.1 17.9 -9.2 1042 855 4.741 Sig (0.05) 26.1 21.1 -5.0 1069 821 2.525 Sig (0.05) 0.817 RI 1: More equitable gender relations between women and their partners/families 5. % of women whose partner helps care for the children if they become ill 38.7 44.7 6.0 955 783 -2.527 Sig (0.05) 30.0 35.3 5.3 979 753 -2.338 Sig (0.05) 0.982 6. % of women who decide how to spend the money they earn 31.6 36.1 4.5 1599 1389 -2.596 Sig (0.05) 29.8 27.8 -2.0 1540 1246 1.158 1.225 7. % of women who make joint decisions with their partner about sexual relations, birth control methods, and number of children 23.7 32.9 9.2 1035 848 -4.431 Sig (0.05) 22.6 27.8 5.2 1067 817 -2.588 Sig (0.05) 1.129 9. % of women who share one or more household chore with their partner 23.2 46.7 23.5 1045 855 -10.77 Sig (0.05) 18.6 37.9 19.3 1075 824 -9.392 Sig (0.05) 0.998 11. % of women who make a joint decision with their partner about the educational level their children should reach 57.4 64.0 6.6 965 776 -2.798 Sig (0.05) 57.8 58.0 0.2 971 767 -0.084 1.111 11.a % of women who have spoken with their partner about the number of children they want to have 67.8 72.6 4.8 1054 853 -2.273 Sig (0.05) 69.3 74.0 4.7 1074.0 820.0 -2.241 Sig (0.05) 1.003 RI 1.1: Strengthened ability of women to achieve equality in gender relations 12. % of women who agree that a woman should be able to decide use family planning even if her partner objects 68.9 64.4 -4.5 1039 855 2.071 Sig (0.05) 65.0 60.4 -4.6 1073 823 2.056 Sig (0.05) 1.006 12.1 % of women who state their unwillingness to be forced or convinced to have sex 50.6 64.0 13.4 1040 852 -5.852 Sig (0.05) 55.4 58.7 3.3 1067 820 -1.435 Sig (0.10) 1.194 13. % of women who have spoken with their partner about family planning more than 2 times in the last 12 months 15.2 20.3 5.1 1044 852 -2.908 Sig (0.05) 11.4 14.1 2.7 1074 822 -1.757 Sig (0.05) 1.080 14. % of women would go to the police/authorities if their partner abused them 14.3 38.4 24.1 1037 854 -12.01 Sig (0.05) 15.7 37.2 21.5 1067 822 -10.691 Sig (0.05) 1.133 MIDTERM EVALUATION OF THE REPROSALUD PROJECT 22 Intervention Control Indicators BL IE Dif. n BL n IE z Signif Bl IE Dif. n BL n IE z Signif Odds Ratio OR 16. % of women who have spoken with their children age 12 years and older about male-female relationships and family planning more than two times in the last 12 months 7.1 16.0 8.9 396 381 -3.893 Sig (0.05) 3.9 10.4 6.5 388 289 -3.352 Sig (0.05) 0.845 16.a % of women who know where to go for help or advice if they are abused 45.6 65.6 20.0 1607 1395 -10.98 Sig (0.05) 44.8 59.5 14.7 1549.0 1251.0 -7.737 Sig (0.05) 1.083 RI 1.2: Increase in positive attitudes of men towards equitable relationships with women and family 17. % of men who believe that it is never right to hit a woman 57.4 64.0 6.6 707 697 -2.531 Sig (0.05) 58.8 63.0 4.2 679 633 -1.557 Sig (0.10) 1.041 18. % of men who believe that a man does not have the right to force an unwilling woman to have sex 84.6 91.5 6.9 706 697 -3.981 Sig (0.05) 88.8 91.8 3.0 681 633 -1.831 Sig (0.05) 1.046 RI 1.3: Increase in women's knowledge about gender equality 19. % of women who believe that the work they do outside the home is as important as the work done outside the home by their partner 51.4 57.8 6.4 730 813 -2.522 Sig (0.05) 48.4 55.2 6.8 997 786 -2.852 Sig (0.05) 0.986 19.1 % of men who believe that the work they do outside the home is as important as the work done outside the home by their partner 52.1 69.8 17.7 476 620 -5.987 Sig (0.05) 55.9 68.3 12.4 612 571 -4.388 Sig (0.05) 1.097 20. % of women who believe that housework is as important as the work done outside the home by their partner 33.7 44.1 10.4 804 855 -4.339 Sig (0.05) 30.2 42.7 12.5 1073 822 -5.631 Sig (0.05) 0.926 20.1 % of men who believe that the housework done by their partner is as important as his own work done outside the home 36.2 47.4 11.2 561 698 -3.997 Sig (0.05) 33.3 47.9 14.6 682 633 -5.393 Sig (0.05) 0.910 21. % of women who believe that their sons and daughters should reach the same level of education 86.7 91.9 5.2 610 492 -2.745 Sig (0.05) 94.5 89.6 -4.9 602 442 2.957 Sig (0.05) 1.118 21.1 % of males who believe that their sons and daughters should reach the same level of education 89.1 91.4 2.3 385 386 -1.077 91.6 90.9 -0.7 370 320 0.325 1.034 RI 2: Increased capacity of women in use of reproductive health services 24. % of women who have gone to a healthcare facility for consultation regarding reproductive health or family planning 26.3 42.2 15.9 1606 1392 -9.187 Sig (0.05) 22.5 39.7 17.2 1549 1248 -9.851 Sig (0.05) 0.909 RI 2.1: Improved capacity of women as end users of formal health services 26. % of women who believe that the services provided by the nearest healthcare facility are good (or very good) 45.5 56.2 10.7 1260 1039 -5.107 Sig (0.05) 52.8 69.7 16.9 1550 968 -8.397 Sig (0.05) 0.936 27. % of women who have gone to a healthcare facility for prenatal and postnatal care, and who believe that the services provided by the nearest facilityt are good (or very good) 55.4 61.2 5.8 271 344 -1.450 Sig (0.10) 67.8 69.3 1.5 314 271 -0.389 1.081 RI 2.2: Increase in women's health as a priority within the home and community 29. % of women who would go to a healthcare facility in the event of symptoms that are warning signs of risk 42.6 71.6 29.0 655 783 -11.11 Sig (0.05) 36.8 62.2 25.4 1005 786 -10.678 Sig (0.05) 0.994 30. % of women who have sought treatment for symptoms of RTIs (or vaginal discharge) 63.2 73.4 10.2 392 247 -2.673 Sig (0.05) 71.5 70.5 -1.0 358 219 0.257 1.178 ANNEXES 23 Intervention Control Indicators BL IE Dif. n BL n IE z Signif BL IE Dif. n BL n IE z Signif Odds Ratio OR 30a. % of women who have gone to a healthcare professional due to discomfort related to RTIs 50.6 61.4 10.8 393 246 -2.669 Sig (0.05) 51.4 58.4 7.0 358 219 -1.637 Sig (0.10) 1.068 32. % of women who take care of their health in order to feel well 56.8 65.1 8.3 1246 1393 -4.368 Sig (0.05) 65.2 56.4 -8.8 1535 1250 4.741 Sig (0.05) 1.325 33. % of women who have incurred a health expense in the last 12 months 49.5 50.0 0.5 1608 1395 -0.273 45.7 45.2 -0.5 1549 921 0.241 1.021 RI 2.6: Increase in women's knowledge about their reproductive health needs 38. % of women who know how at least one modern contraceptive method works 13.5 55.5 42.0 1608 1395 -24.41 Sig (0.05) 14.9 31.9 17.0 1550 1251 -10.713 Sig (0.05) 1.920 39. % of women using the rhythm method who know the fertile days of their cycle 22.5 28.3 5.8 239 214 -1.419 Sig (0.10) 24.0 24.1 0.1 306 233 -0.027 0.782 40. % of women who can recognize some symptom of pregnancy or postpartum warning signs that indicate risk 19.4 46.0 26.6 1044 855 -12.43 Sig (0.05) 13.2 40.0 26.8 1076 823 -13.399 Sig (0.05) 0.782 43. % of women who know how RTIs are spread 4.9 19.0 14.1 1193 1150 -10.57 Sig (0.05) 4.7 11.4 6.7 1007 921 -5.448 Sig (0.05) 1.599 43.a % of women who have heard about Pap and breast exams 63.2 81.4 18.2 1608 1395 -11.04 Sig (0.05) 59.6 71.8 12.2 1550.0 1251.0 -6.732 Sig (0.05) 1.069 43.b % of women who believe that while a women is breastfeeding it is dificult for her to become pregnant 34.0 56.3 22.3 1193 1001 -10.48 Sig (0.05) 30.9 46.9 16.0 1200.0 951.0 -7.594 Sig (0.05) 1.091 MIDTERM EVALUATION OF THE REPROSALUD PROJECT 24 ANNEX C IMPACT STUDY GRAPHS MIDTERM EVALUATION OF THE REPROSALUD PROJECT 26 Table 4: Reproductive Health Indicators Net increase, in percentage points, between the baseline and the midterm evaluation Reproductive Health Indicators Invervention Control Odds Ratio 1.a % of women who had at least 4 prenatal visits to a healthcare professional during a pregnancy occuring within the last two years 27.5 18.8 1.148 2. % of women whose last delivery, occurring within the last two years, was performed by a healthcare professional 11.9 1.8 1.274 Figure 1: Four Prenatal Visits in Health Services During Last Pregnancy 50 55 60 65 70 75 80 85 Baseline Midterm evaluation Percentage Intervention Control Net Increase 27.5 18.8 Intervention Control Figure 2: Health Professional Assisted Last Childbirth 25 30 35 40 45 50 55 60 Baseline Midterm Evaluation Percentage Intervention Control Net Increase 11.9 1.8 Intervention Control ANNEXES 27 Table 5: Gender Issues Indicators Net increase, in percentage points, between the baseline and the midterm evaluation Indicators on Gender Issues Invervention Control Odds Ratio 6. % of women who decide how to spend the money they earn 4.5 -2.0 1.225 12.1 % of women who state their unwillingness to be forced or convinced to have sex 13.4 3.3 1.194 Figure 3: Percent of Women Who Decide How to Spend the Money They Earn 25 27.5 30 32.5 35 37.5 40 Baseline Midterm Evaluation Percentage e Intervention Control Net Increase 4.5 -2.0 Intervention Control Figure 4: Percent of Women Who State Their Unwillingness to Be Forced or Convinced to Have Sex 45 50 55 60 65 Baseline Midterm Evaluation Percentage Intervention Control Net Increase 13.4 3.3 Intervention Control MIDTERM EVALUATION OF THE REPROSALUD PROJECT 28 Table 6: Reproductive Health and Gender Issues Indicators Net increase, in percentage points, between the baseline and the midterm evaluation Indicators on Reproductive Health and Gender Issues that showed significant increases in both intervention and control commmunities, but no gains attributable solely to Reprosalud Invervention Control Odds Ratio 29. % of women who would go to a healthcare facility in the event of symptoms that are warning signs of risk 24. % of women who have gone to a healthcare facility for consultation regarding reproductive health or family planning 20 & 20.1 % of women and men who believe that housework is just as important as the husband’s work outside the home Figure 5: Percent of Women Who Have Gone to a Healthcare Facility for Reproductive Health or Family Planning Services 20 25 30 35 40 45 Baseline Midterm Evaluation Percentage Intervention Control Net Increase 15.9 17.2 Intervention Control Figure 6: Percent of All Women Who Would Go to a Healthcare Facility in the Event of Symptoms That Are Warning Signs of Risk. 30 40 50 60 70 80 Baseline Midterm Evaluation Percentage Intervention Control Net Increase 25.4 29.0 Intervention Control ANNEXES 29 Figure 7: Percent of Women and Men Who Believe that Housework Is As Important As the Husband’s Work Outside the Home 25 30 35 40 45 50 Baseline Midterm Evaluation Percentage Women Intervention Women Control Men Intervention Men Control Men Women Net Increase 10.4 12.5 11.2 14.6 Intervention Control Intervention Control Women Men MIDTERM EVALUATION OF THE REPROSALUD PROJECT 30 ANNEX D DETAILED FINDINGS OF IMPACT STUDY AND RECOMMENDATIONS MIDTERM EVALUATION OF THE REPROSALUD PROJECT 32 In comparison to similar communities where Reprosalud did not intervene, the program has shown substantially greater increases on key reproductive health indicators related to knowledge and to the Strategic Objective of use of reproductive health intervention.4 : · Knowledge: 92% more likely to know how at least one modern contraceptive method functions, 60% more likely to know how RTIs are contracted, and rhythm users are 25% more likely to know the fertile days in the menstrual cycle. · 27% more likely to have their last childbirth attended by a health professional, 15% more likely to have had four pre-natal visits in health services, and 18% more likely to seek treatment for RTIs from both community and formal sector sources (see graphs from impact study in annex C). · 15% more likely to use a family planning method, and 18% less likely to have unmet need for family planning.5 Using composite indicators, the cost effectiveness study showed greater impact on use of childbirth and prenatal services than on family planning use. · In the process evaluation, more than 75% of health professionals interviewed attributed increases in service use wholly or partially to Reprosalud’s efforts. The process evaluation yielded instances of dramatic positive effects on health, including lives saved. See the “Achievements” annex in the Process Evaluation. While the indicator “Number of lives saved” falls outside of the results framework, the CBO promoters in some instances clearly were the deciding factor in saving a woman’s life. Examples include persistence in helping a woman to detect and then treat an advanced case of cervical cancer, and many instances of overcoming many obstacles in order to get a woman in childbirth with danger signs to a health facility. The process evaluation gave evidence of women’s increased communication within the family and with providers, of women becoming more capable end users of health services, of increased civic participation by women, and of unplanned effects (at this stage) on quality of care in services. · Sixty percent of health professionals said that Reprosalud has played a significant role in increases in quality of services, mainly through its effect on the provider-user relationship in which the women became more capable end users. Both they and the CBO promoters and members made the following two points: o Users are more apt to give feedback and make demands, thus contributing to efforts to improve quality o Users are more educated and communicative, making the providers’ job easier and more efficient. 4 The percent of greater likelihood refers to women in Reprosalud communities, and the percent of change corresponds to the “odds ratio” in Tables 24 and 25 of the Impact Study annexed to this summary. 5 This indicator includes users of rhythm method who do not correctly identify the fertile period. ANNEXES 33 · There was much evidence of increased civic participation in the process evaluation, including election of CBO promoters as local authorities.6 Reprosalud has had a positive but less significant effect on the Intermediate Result of achieving more equitable gender relations between women and their spouses and families. There are several possible explanations for these uneven results: the program’s greater emphasis on health issues than on gender issues, more cultural resistance to changes in gender roles than to increasing usage of health services, a lower investment in working with men than with women, and the fledgling state of the art in constructing indicators and survey instruments on gender issues.7 The following indicators show increases for women that are attributable to the program. See illustrative graphs from the impact study, Annex 4. · 22.5% more likely to decide how they will spend money they earn, · 19% more likely to report not submitting to forced or pressured sex, · 13% more likely to go to the authorities if their spouses beat them · 13% more likely to decide jointly with their spouses on matters related to children’s education, sexual relations, contraception and number of children. · 12% of women (but only 3.4% of men) more likely to agree that boy and girl children should have the same educational level. There is strong support among all stakeholders for continuing to work on gender issues and with men, especially at the community level, with respondents stating that violence and other negative attitudes continue in their communities. This finding is understandable given that not all CBOs worked with men, and that the work with them received less emphasis. · Among all respondents, when asked how the project might expand, more work with men and on violence was in the top three most frequent responses, and in the top five when respondents were asked a general question about their recommendations to Reprosalud for the next period. Reprosalud’s educational activities and communications program in Phase Two should emphasize the following indicators for which Reprosalud communities enjoyed significant gains, but the actual percentage is still much less than ideal. · Women whose last childbirth was attended by a health professional rose from 36.1% to 48%. Women who know the danger signs for childbirth and post￾natal rose from 19.4% to 46%. · Users of rhythm who know their fertile days rose from 22.5% to 28.3% 6 See Annex on Achievements in Process Evaluation Report. 7 The state of the art in investigating reproductive health matters in quantitative surveys is much more advanced in Peru and worldwide than in investigating gender issues. Therefore, we would advise caution in interpreting these findings on gender issues because of some methodological problems in the survey, and because of the other factors named which do not indicate program failure, but rather the need to adjust the model to intensify attention to gender issues and to men. The ethnographic study should shed more light on the dynamics of change in gender issues in the program. MIDTERM EVALUATION OF THE REPROSALUD PROJECT 34 · Women who know how at least one modern contraceptive method works rose from 13.5% to 55.5%. · Women who know how vaginal discharge is contracted rose from 4.9% to 19% · Women in union with unmet need for family planning fell from 48.4% to 37.3% · Women who decide how they will spend the money they earn rose from 31.6% to 36.1% · Women who decide jointly with their spouses on sexual relations, contraception, and number of children rose from 23.7% to 32.9%. The number of women who had spoken “often” to partners about family planning rose from 15 to 20 percent, and “often” to children over 12 about this and relations in a couple rose from 7.1% to 16%. · Women who would go to the authorities if beaten rose from 14.3% to 38.4% . For other indicators, results were disappointing and education in the next period needs to reinforce these issues: · Women who agree that women should use contraception even when her partner is opposed fell significantly in both Reprosalud and comparison communities. In Reprosalud communities, the decrease was from 68.9% to 64.4%.8 · Both men (36.2% to 47.4%) and women ( 33.7% to 44.1%) rose in giving equal value to women’s work in the home, but to low levels, and with no gain in comparison to the non-intervention communities. For some intermediate indicators for both reproductive health and gender, both intervention and comparison communities increased significantly between baseline and midterm, but there is no significant difference between the levels of increase. For these indicators, the Reprosalud program did not add any significant gain in the intervention community to what would have taken place anyway. The Reprosalud project period coincided with a time of intense social change in Peru. The MOH and many donor agencies9 invested heavily in improving access and quality of care in family planning and maternal-child health services in order to improve health statistics and service use. NGOs and some government agencies undertook ambitious grassroots and communications projects to increase gender equity. The following selection is from a longer list of indicators where no significant additional impact from Reprosalud was found (see illustrative graphs from the impact study in annex C). · For the percent of women to go to a healthcare facility in the event of warning signs of risk, the gain was 29% for Reprosalud communities and 25.4% for comparison communities. 8 This is an idea that contravenes the culture of these communities, according to the baseline of the ethnographic study. 9 See a complete listing on page 95, footnote 30, in the Impact Study. ANNEXES 35 · For the percent of women who attended a health service for reproductive health or family planning services in the last year, the gain was 16% in Reprosalud communities and 17.2% in comparison communities. · For the percent of women to recognize pregnancy or postpartum warning signs indicating the presence of a risk, the gain was 26.6% in Reprosalud communities and 26.8% in comparison communities. · For the percent of women to share one or more household chores with their partner, the gain was 23.5% in Reprosalud communities and 19.3% in comparison communities. · For the percent of women who believe that housework is just as important as their husband’s work, the gain was 10.4% in Reprosalud communities and 12.5% in comparison communities. A. MORE DETAILS ON RECOMMENDATIONS Following are recommendations for replication and extension of coverage of the educational model, for the Advocacy Program, and for evaluation design. These recommendations are described more fully in the Process Evaluation Report Replication And Extension Of Coverage To maximize the benefits from the investment in Reprosalud, we recommend that program managers give high priority to encouraging replication through other NGOs, educational programs, and international agencies during the next four years. Replication by the MOH, or by the current CBO promoters, is both possible and included in the plans for some sub-projects, but there are many logistical and financial obstacles in both cases. Staff should do active outreach to other institutions. Reprosalud’s substantial investment in investigating key reproductive health and gender problems, and in producing culturally appropriate training manuals and educational materials, make replication feasible in other hard-to-reach communities in the Andean jungle and highlands areas, both in Peru and in other Andean countries. Their methodology and tools could be adapted for other settings. Through such replication, this investment can continue to produce benefits for years to come. An important challenge in replication efforts is to assist other organizations to incorporate the guiding principles of the program in the replication (see box page 1). Commitment to gender equity, respect for indigenous cultures and for low-income communities, true community participation, and rights-based, comprehensive approaches to reproductive and sexual health, imply comprehensive cultural changes in some organizations. Additional intervention and training may be necessary in some or all of these principles. Suggested modifications in the Phase One educational program for replication efforts o To introduce male and female educational components simultaneously by working with both male and female CBOs. The resistance to the program MIDTERM EVALUATION OF THE REPROSALUD PROJECT 36 in its early stages might have been lessened if both male and female CBOs were involved. Also, this modification might increase the program’s impact on gender issues, and the level of support from local authorities. o To strengthen the focus on gender issues in replications through use of radio programs and videos, which Reprosalud will develop in the communications program in Phase Two. This suggestion is in response to the mixed results on gender issues. o To train health providers in responsiveness to community feedback and quality of care in contexts where no complementary program exists to do so. Reprosalud could count on many complementary MOH-led programs that improved access through insurance, and quality partly by emphasizing user feedback and provider/user relations. Programs in other contexts may not be able to count on this complementary and necessary input. o To weigh opportunity costs to the community participants vs. the sustainability of the program in setting policy on financial incentives and non-cash benefits for promoters. The level of commitment of Reprosalud￾trained promoters in Phase Two will demonstrate whether the financial incentives to promoters provided by Reprosalud lead to high dropout rates of promoters once these incentives are withdrawn, and other non-cash incentives replace them. To facilitate replication of the Phase One educational program o Develop and provide a full kit of all the tools, manuals, and materials developed for Phase One. o Respond positively to all MOH requests to replicate the program through the sub-projects, or through agreements at the departmental level. o Increase outreach to other NGOs, educational institutions and international agencies working in Peru and in the Andean region to publicize the availability of training for replication o Develop diagnostic procedures to determine whether agencies need reinforcement in some or all of the underlying principles of the program. Identify additional tools and curricula for filling these gaps. To strengthen the advocacy program o Closely supervise and evaluate whether communication is flowing from the community level to the negotiating team, and vice-versa. Identify ways to build these communications into routine interactions and sub￾project events. o Monitor level of promoter involvement in community-level educational activities closely, and devise low-cost motivation strategies if necessary. o Devise a back-up plan for cases in which the CBO President is not participating as planned in the Defenders’ Committees. Reliance on CBO Presidents for the Defenders Committees is a potential weakness. They rotate frequently and may not have sufficient commitment to the program. ANNEXES 37 Maximizing Impact in Current Districts The process and impact evaluations identified the following needs for reinforcing impact that the strategies in Phase Two must address: 1) The need for refresher training for promoters and CBO members (see the section on remaining needs on page 45–46 of the Process Evaluation Report). 2) The need to reach more men, and more youth of both sexes (ages 15-24). 3) The need to strengthen educational interventions on gender issues. 4) The importance of taking advantage of the trained male and youth promoters in those communities that worked with them, to keep these promoters engaged with the program. Discussion All sub-projects in Phase Two include educational activities at the community level, determined by a diagnosis of reproductive health needs conducted just before the negotiation with the health center. Current plans only call for involvement of the female CBO promoters. The content of these activities has been shaped by needs expressed by both CBO members and the health sector. They will not necessarily include more work with men and youth, more focus on gender issues, or even reinforcing the reproductive health knowledge needs identified in this evaluation. The following recommendations are designed to meet these needs. o To implement a communications program with the following goals: ß Develop messages that support efforts to expand coverage on the part of the CBO promoters and others replicating the program, by lessening resistance to the program’s educational messages and creating demand for the program in new communities. ß Reinforce health-protective knowledge, attitude and behavior changes on gender and reproductive health issues. Besides providing both CBO promoters and women with educational inputs, this program would also focus on youth and to men, for whom Phase One activities had less impact. ß Involve CBO promoters, and male and youth promoters, in competitions as a mechanism to develop radio programs and educational videos. o To involve male and youth promoters trained by Reprosalud as much as possible in Phase Two education and training activities, including the community-level workshops, and negotiate recognition for them as MOH community agents. o To continue to work with men. Community-level respondents universally recommended continuing work with men. Men’s involvement is key to achieve the goal of creating a sustainable change in the culture of the community regarding reproductive health and gender issues. Furthermore, data from the process evaluation suggests that the support from local MIDTERM EVALUATION OF THE REPROSALUD PROJECT 38 authorities is stronger when they are involved in educational activities for men. Increasing the level of support from local authorities is extremely important to the advocacy program during the next four years, and a key factor in the sustainability of the program after Reprosalud ends. Evaluation and monitoring · To strengthen the evaluation and monitoring component of the program: See all of Section G in the process evaluation report for a full discussion. o Improve oversight by hiring an evaluation and monitoring director o Analyze the whole database of 70 communities to compare results by department. The findings on departmental differences from the cost effectiveness study were not reliable because of the small sample size. o Devote increased resources to collection of qualitative process and impact data, adding evaluation questions pertinent to the advocacy program and to replications. Monitor unplanned benefits from the program such as lives saved, and women elected to local offices. o Improve the results and indicators framework as suggested in the attached annex. Develop improved Intermediate Results and indicators for the advocacy program. Additional Research · To conduct studies that would advance the state of knowledge in the reproductive health field and inform replication efforts. Reprosalud—with its current budget and staff—could not undertake these studies. If USAID or other stakeholders decide to gather the resources to carry out any of them, the efforts should be organized in a way as not to create added burdens on the staff. o Compare the costs of Reprosalud for each percentage gain in usage of services above the gains of comparison communities with those of a program that improves access and quality of care in health services. o Carry out a cost-benefit analysis that takes into account the sustainable effects of the program both in the intervention communities, and in communities where replication takes place. o Compare the results in communities with and without autodiagnósticos. o Evaluate the extent and quality of MOH replication of the program in Ucayali and San Martín, where MOH was trained to replicate the program and supervise the CBO promoters after Reprosalud’s withdrawal. o Compare the results of Reprosalud with communities where the program works with both men’s and women’s CBOs simultaneously, and strengthens the component on gender issues. o The findings from the impact study on the differences between communities with both income-generation and reproductive health components, and communities with only reproductive health, were ambiguous because the sample size was too small. It would be useful to ANNEXES 39 manipulate the larger database of 70 communities to see whether more reliable findings could be generated. MIDTERM EVALUATION OF THE REPROSALUD PROJECT 40 ANNEX E PROCESS EVALUATION TABLES MIDTERM EVALUATION OF THE REPROSALUD PROJECT 42 SHARED AND UNSHARED PRIORITIES IN THE ADVOCACY PROGRAM The following chart reveals the main areas of shared and unshared priorities expressed by health authorities, health providers, and promoters. Key: ____ _ _ ____ boxes on the left: MOH priorities not shared by Reprosalud. __ __ __ __ __ __ boxes in the middle: shared priorities. ________ boxes on the right: Reprosalud priorities not shared by MOH. - - - - - - - - - - boxes: partially shared category “culturally appropriate institutional births.” Figure 8: Shared and Unshared Priorities MOH PRIORITIES Many other maternal￾child health and primary health care issues, e.g. measles vaccination campaign at time of study. REPROSALUD/CBO PRIORITIES SHARED PRIORITIES MATERNAL MORTALITY Professionals attend home births Culturally appropriate institutional births Treatment of RTIs & STIs * (not life threat￾ening) Treatment of life￾threatening STIs (e.g. syphilis) and cervical cancer Full coverage for pre-natal care Access to family planning methods Free or reduced cost treatment and medi￾cines for cervical cancer, STIs, & RTIs Achieving 80% coverage in institutional childbirth ANNEXES 43 Figure 9: Essential Communications Flows in Advocacy Program, Phase Two Health Center or Hospital Sub-project steering committee Steering Committee of the Defenders’ Committee (directiva) Steering Committee of the Promoters’ Network Defenders’ Committee Promoters’ Network CBO Presidents (15- 20 CBOs in each sub￾project) CBO Promoters (from 15 to 20 CBOs in each subproject) Local authorities CBO Members & spouses Other community women & spouses Zonal coordinator Zonal coordinator MIDTERM EVALUATION OF THE REPROSALUD PROJECT 44 Table 7: Synopsis of General Recommendations to Reprosalud i.e. response to a general, open-ended question Recommendation Total Who Expansion of Coverage 39 Promoters; Local Authorities Continue with Phase One 39 CBO Members; Local Authorities Coordination w/ DISA / MINSA 32 Health Authorities; Providers More Supervision & Training of CBOs Promoters 30 Promoters; Health Authorities; Providers More Work w/ Men, Couples, and Family 27 Promoters; Local Authorities More Work w/ Youth 18 Promoters; Local Authorities; CBO Members More Coordination between RS, Health Authorities, & Local Authorities 15 Promoters; Local Authorities Sharing RS Methodology w/ MINSA &/or NGOs (Training Staff & Promoters) 14 Health Authorities; Other RH Programs ANNEXES 45 Table 8: MOH Limitations and Financial Issues Barriers Related to Under-funding (Total = 25) Financial Limitations, Budget (Includes Infrastructure & Equipment) Deficient Supply of Medications and FP Methods Limited Personnel (General and Specialized) Limited Community Outreach Cultural Sensitivity (Childbirth Customs) Employees' lack of interest and motivation Continuity of Personnel in positions (Administrative Problems) Service Problems and Management Language Barriers (Need of Bilingual Personnel) Ancash Health Authorities 4 1 1 Providers Ayacucho* Health Authorities 3 1 1 Providers 1 1 1 Huancavelica Health Authorities 4 1 Providers 1 1 1 La Libertad Health Authorities 3 1 Providers 2 Puno Health Authorities 1 1 1 1 Providers 1 1 1 TOTAL 17 4 3 1 3 2 2 2 1 MIDTERM EVALUATION OF THE REPROSALUD PROJECT 46 Table 9: Impact on Women More knowledge about RH "Takes care of her health" (Includes Better hygiene, RTIs and ETS) More use of services Use of Family Planning Can communica te w/ Provider / makes Demands & Give Feedback Better communi￾cation & relationship with spouse More political participation No or very few changes Health Authorities 10 5 6 3 3 1 1 1 Providers 8 4 7 2 1 1 1 1 Local Authorities 2 5 3 3 1 5 2 4 CBO Members 19 18 11 15 7 2 2 TOTAL 39 32 27 23 12 9 6 6 Table 10: Impact on Men Insufficient Impact on Men Positive Attitudes in Gender Relations (General Statements) Positive Change in Family Planning Use& Attitudes Increased RH Health Care Knowledge & Education Decrease of Domestic Violence & Rape Improved Self-care & Hygiene (prevention of STDs) Increased Use of Services by Men & Women Men’s Domestic Help Improved Dialogue & Communica￾tion with Women (& Family) Health Authorities 2 6 5 5 6 3 4 Providers 7 4 3 4 2 1 3 Local Authorities 5 5 5 2 1 6 3 2 2 CBO Members 11 11 7 4 6 1 1 6 2 TOTAL 25 26 20 15 15 10 9 8 7 ANNEXES 47 Table 11: Did Reprosalud Have an Impact on Use of Services? Table 12: Changes Implemented in Services to Improve Quality *provider dismissed staff offenders HEALTH AUTHORITIES & PROVIDERS Due to Reprosalud Partially Due to Reprosalud Unclear No/ Other Reason Ancash 3 1 3 Ayacucho 5 3 1 Huancavelica 2 6 3 La Libertad 5 1 1 Puno 3 3 TOTAL 18 13 5 4 Provider attends home births or home prenatal Provider adapts to cultural preferences Free childbirth services Better treatment (more "friendly", more responsive) More Quechua speakers Increase access through hours, more personnel, etc. New services, RTIs, violence Less waiting time, more efficiency Ancash 1 4 3 4* 1 Ayacucho 3 2 1 Huancavelica 2 5 4 2 2 1 La Libertad 1 4 1 5 3 2 Puno 2 1 1 2 1 TOTALS 4 18 8 10 3 7 2 2 CHILDBIRTH TOPICS According to Health Professionals MIDTERM EVALUATION OF THE REPROSALUD PROJECT 48 ANNEX F COST ANALYSIS DATA MIDTERM EVALUATION OF THE REPROSALUD PROJECT 50 Table 13: Combined Effectiveness Indicators Constructed for the Cost Analysis Nr. Results grouping Indicators Description of combined indicator I SO: “women make greater use of reproductive health services” a) Women receive prenatal and birth care from trained personnel. (Indicators 1 and 2) b) Women in a relationship who use contraceptive methods when they really need them (excluding those who do not use contraceptive methods for reasons relating to values 02 to 08, 20 and 21 in question 214); new indicator: based on questions 211 and 215. "Integral" indicator, which gives a weighting of 50% to each of the two following combined indicators: a) one ranging from 1 (% of women with children under the age of 3 receiving one of the health services in a) of the preceding column) to 2 (% of women with children under the age of 3 receiving both of the services in a) of the preceding column. b) one ranging from 1 (% of women in a relationship who need to use contraceptive methods and replied 'no' to question 211 but 'yes' to 215) to 2 (% of women in a relationship who need to use contraceptive methods and replied 'yes' to question 211). II IR 1, IR 1.1 and IR 1.3 (a combination of the three for women): “women have more equitable gender relationships with their partners and families", "women strengthen their capacity to bring about changes in their gender relationships”; “women and men have greater knowledge of gender equity” Women who take key family decisions with their partners (indicators 7, 9 and 11), who have good communication on sexual issues with their partner (indicator 13) and who believe that daughters and sons should be educated to the same level (indicator 21). Combined indicator ranging from 1 (% of women in a relationship who fulfill only one of these assertions) to 5 (% of women in a relationship who meet all five)10 10 Initially, a combined indicator ranging from 1 to 7 was proposed, and included two additional indicators: (i) women who believe that their work outside or within the home is as important as the work done by their husband (a combination of indicators 19 and 20); and (ii) women who have frequently talked with their partner about STDs in the last 12 months (indicator 13.1). However, these indicators were considered for developing an alternative indicator, given that the number of void replies to the questions which made them up resulted in the number of cases dropping significantly. ANNEXES 51 11 Initially, a combined indicator ranging from 1 to 5 was proposed and included two additional indicators: (i) men who believe that both should look after the children when they are ill (indicator 18.1); and (ii) men who believe that their work out of or in the home is as important or less important than that of their wives (a combination of indicators 19.1 and 20.1). However, these indicators were considered for developing a separate alternative indicator, given that the number of void replies to the questions that made them up resulted in a significant reduction in the number of cases. Nr. Results grouping Indicators Description of combined indicator I IR 1.2 and IR 1.3: (a combination of the two for men): “increase in positive attitudes and practices of men in their relationship with women and with their families", "women and men increase their knowledge on gender equity" Men who believe that women should not be physically assaulted (indicator 17), who believe that they cannot demand sexual relations if the woman is unwilling (indicator 18); and who believe that daughters and sons should be educated to the same level (indicator 21.1) Combined indicator ranging from 1 (% of men in relationships who fulfill only one of these assertions) to 3 (% of men in relationships who meet all 3).11 II IR 2, IR 2.1, IR 2.2: (a combination of the three): "women have a greater capacity to access RH services", "women improve their capacity as end consumers of formal health services", "increase in the positive appreciation of women's health within the home and within the community”. Women who do not forego access to health services through lack of income (new indicator: question 502a, reply 'no'), women who look after their health for a feeling of well-being (indicator 32) and women who have invested in health in the last 12 months. Combined indicator: ranging from 1 (% of women who fulfill only one of these assertions) to 3 (% of women who fulfill all three). MIDTERM EVALUATION OF THE REPROSALUD PROJECT 52 Nbr. Results grouping Indicators Description of combined indicator III IR 1.2 and IR 1.3: (a combination of the two for men): “increase in positive attitudes and practices of men in their relationship with women and with their families", "women and men increase their knowledge on gender equity" Men who believe that women should not be physically assaulted (indicator 17), who believe that they cannot demand sexual relations if the woman is unwilling (indicator 18); and who believe that daughters and sons should be educated to the same level (indicator 21.1) Combined indicator ranging from 1 (% of men in relationships who fulfill only one of these assertions) to 3 (% of men in relationships who meet all 3). IV IR 2, IR 2.1, IR 2.2: (a combination of the three): "women have a greater capacity to access RH services", "women improve their capacity as end consumers of formal health services", "increase in priority given to women's health within the home and within the community”. Women who do not forego access to health services through lack of income (new indicator: question 502a, reply 'no'), women who look after their health for a feeling of well-being (indicator 32) and women who have invested in health in the last 12 months. Combined indicator: ranging from 1 (% of women who fulfill only one of these assertions) to 3 (% of women who fulfill all three). ANNEXES 53 Table 14: RH Effectiveness Ratios at National Level BASE LINE MIDTERM EVAL. Improvement % Comp. S + A Comp. S + A Comp.c/ S+A c/ Difference d/ Ia (SO) 1.04 1.04 1.30 1.43 25.41% 37.75% 12.34% Ib (SO)12 1.88 1.86 1.91 1.92 1.85% 3.21% 1.37% I (SO) a/ 1.46 1.45 1.61 1.67 10.23% 15.58% 5.35% II (IR 1., IR 1.1, IR 1.3) 1.50 1.57 1.63 1.97 8.84% 25.34% 16.50% III (IR 1.2, IR 1.3) 2.09 2.06 2.08 2.20 -0.35% 6.77% 7.12% IV (IR 2., IR 2.1, IR 2.2) 1.60 1.54 1.51 1.71 -5.46% 10.80% 16.26% Global Indicator b/ 1.66 1.66 1.71 1.89 2.83% 14.04% 11.22% a/ I (SO) = [ Ia (SO) + Ib (SO) ] / 2 b/ Global Indicator = [ I + II + III + IV ] / 4 c/ Improvement % = [ (BASE LINE) Indicator / (MIDTERM EVAL.) Indicator ] – 1, for each CBO group. d/ Difference = Improvement (S + A) – Improvement (Comp.). Table 15: Cost Per Beneficiary for the Main Activities of The Reprosalud Program (Soles at January 1995 value)13 Total Cost Total Benef iciarie Cost per Beneficiar y Total Cost Total Benef iciarie Cost per Beneficia ries Total Cost Total Benef iciarie Cost per Beneficiary Ayacuc ho 2,116,278 .46 1823 0 116.0 9 62,327. 47 17 8 350.1 5 Huar az 2,061,320 .08 1535 5 134.2 4 82,966. 51 10 1 821.4 5 Huancavel ica 2,071,072 .14 2282 9 90.7 2 Trujill o 1,498,411 .20 1640 7 91.3 3 1,568,045 .41 316 8 494.9 6 48,978. 56 40 1,224. 46 Lima Este 667,861. 46 382 6 174.5 6 Pun o 1,783,724 .94 2036 7 87.5 8 1,581,409 .95 302 4 522.9 5 102,789. 88 13 4 767.0 9 Juliac a 1,735,201 .71 1844 2 94.0 9 59,783. 88 12 7 470.7 3 Tarapo to 1,042,694 .15 1243 2 83.8 7 927,620. 01 253 9 365.3 5 142,917. 37 61 2,342. 91 Pucall pa 1,410,905 .52 816 8 172.7 4 1,874,846 .23 411 2 455.9 5 82,162. 41 52 1,580. 05 Tota l 14,387,469 .66 13605 6 105.7 5 5,951,921 .60 1284 3 463.4 4 581,926. 08 69 3 839.7 1 REPRODUCTIVE HEALTH MICROCREDITS OS PRODUCT DEVELOPMENT REGIO N 12 Because of the way this indicator on family planning use was constructed, it gives very different results from the impact study, which shows a 13% rise in use of a contraceptive method in Reprosalud communities and a 4% rise in comparison communities, and a 15% greater likelihood, according to the odds ratio, that women from Reprosalud communities would use a contraceptive method. The impact study is more precise. 13 Exchange rate for 1995 soles is 2.18 soles.