MIDTERM EVALUATION OF REPROSALUD PROCESS EVALUATION REPORT By: Delicia Ferrando Bonnie Shepard Submitted by: LTG Associates, Inc. and TvT Associates, Inc. Submitted to: The United States Agency for International Development Under Contract No. HRN-I-00-99-00002-00, Task Order Number 812 January 2002 WORKING PAPER Midterm Evaluation of ReproSalud, Process Evaluation Report was prepared under the auspices of the U.S. Agency for International Development (USAID) under the terms of the Monitoring, Evaluation, and Design Support (MEDS) project, Contract No. HRN-I-00-99-00002-00, Task Order No. 812. The opinions expressed herein are those of the authors and do not necessarily reflect the views of LTG Associates, TvT Associates, or USAID. Information about this and other MEDS publications may be obtained from: Monitoring, Evaluation, and Design Support (MEDS) Project 1101 Vermont Avenue, N.W., Suite 900 Washington, DC 20005 Phone: (202) 898-0980 Fax: (202) 898-9397 jklement@medsproject.com www.medsproject.com Preface to the Report The general objectives of the evaluation are: 1. To determine the impact of the educational program in the first five years (Phase One) of the ReproSalud Project on the participating women and their communities, according to the results framework. 2. To collect lessons learned on Phase One (1996-2000) for purposes of replication, and 3. For Phase Two, (2001-2005): To recommend improvements and an evaluation design for the Advocacy Program, and To recommend next steps that would extend the coverage and maximize the impact of the Phase One educational program. The midterm evaluation has three components: the process evaluation, the analysis of the results of the quantitative impact study, and the cost analysis. This report describes the findings and recommendations from the process evaluation. Delicia Ferrando is the author of the quantitative impact study. Arlette Beltrán of the Universidad del Pacífico wrote the cost analysis. Bonnie Shepard is also the author of the final executive summary paper with recommendations based on the results of all three components of the evaluation. The planning for this evaluation began in June 2001. The data collection for the process evaluation took place in October and November 2001, at a point in the life of the project when the activities for the first five-year phase had concluded, and those for the second five-year phase were just gearing up. Therefore, the process evaluation could only comment on the plans for the next phase, and on barriers and facilitating factors for those plans. See Annex I for a full description of the methodology and for the research instruments. Acknowledgements We owe a major debt of gratitude to the staff of ReproSalud for all of their time, patience, logistical assistance, hospitality, and for their insights into the complexities of this project. We should especially highlight the efforts of Susana Moscoso, Sandra Vallenas, and Carmen Yon in Lima; the regional coordinators Janet Espinoza in La Libertad; Celina Salcedo Martines in Ayacucho; Fernanda Loayza Condori in Huancavelica; Emilia Calisaya Medina in Ancash; and Luz Estrada Alarcón in Puno. The ReproSalud regional promoters provided valuable insights into the community level work. Christine Adamczyck of USAID in Lima provided invaluable technical guidance and logistical support for this evaluation, and asked all the hard questions. Our thanks to Christie Billingsley and Daniele Avila of the MEDS project in Washington for cheerfully keeping the administrative wheels rolling. We owe a special thank you to researchers Bertha Silva and Ada Baldeón for assistance with data collection, to Claudia Ordoñez and David Holmstrom for their work on the database, and to Tania Hannan of NSight for a careful editing job. Finally, the women and men of the communities we visited in five departments were unfailing in their generosity with their time, insights and hospitality. They are a source of inspiration to all of us. GLOSSARY Autodiagnostico. CBO. Community-based organization. In Peru, there are two widespread networks of women’s community-based organizations. A “Vaso de Leche” (Glass of Milk) Committee needs to exist for a community to receive nutritional supplements from the government’s food program for low-income families. Mothers Clubs also exist nationwide and function as channels for governmental aid. The membership in the two organizations overlaps, at least partially. Sub grantee and associated CBOs. Sub grantees (contrapartes or ganadoras) were the CBOs that actually administered the sub grants from ReproSalud and organized the activities under the sub grant. They were chosen via competition in the districts involved in the project. Associated CBOs in the same district were then offered the same opportunities to have promoters trained in reproductive health and to hold educational activities with the support of the subproject. The main difference between the sub grantee and associated CBOs was that the sub grantees gained valuable experience in administering and leading projects, and received the needed training. CBO promoters. Members of the CBO, trained by ReproSalud to give community level educational workshops during Phase One, and also serving as a key referral link between community women and the health services. Those interviewed were from both sub grantee CBOs and associated CBOs. CBO promoters’ network. These networks have been organized in Phase Two in order to help negotiate and carry out agreements in a defined catchments area between the CBOs and the health service that is the referral center for that area. CBO members. The process evaluation interviewed members, all of whom are women, from both sub grantee and associated CBOs. They participated in the educational workshops run by the promoters. CBO president. The elected official representative of the CBO. Community health agents (CHA): This is the official name of the MOH-sponsored health promoters. Informally, they are also referred to as promoters, but, to avoid confusion, this report uses the term “community agents” when referring to the MOH promoters, and “CBO promoters” when referring to those trained by ReproSalud and active in the project. Currently, most of the community health agents are men, mainly involved in primary health care and sanitation. Generally, they are not trained to deal with reproductive health issues. Department. At present, the largest political and geographic division in Peru. When REPROSALUD started, the largest unit was the “region,” thus, the project selected eight regions for intervention. For purposes of the MOH structure, the departmental level was then, and still is, the next administrative/political level under the national level. District. The smallest geographic division after the “department” and “province” levels. It usually coincides with the catchments area of a health center. Hospitals are located only in districts that are capitals of provinces. Health centers. In Peru, these are secondary level facilities that serve as referral centers for a catchments area, with basic surgical, obstetrical, and hospital facilities. Hospitals. While hospitals are tertiary-level facilities, the maternity hospitals often serve as the basic referral centers in districts that are capitals of provinces as well. Health posts. The primary level facilities; in rural areas they are sometimes staffed by "technicians," the equivalent in training to a nurse's aide. Health sector authorities. Those in decision-making roles at the department level (DISA) and directors of health centers or hospitals at the district level. These respondents may or may not have had interactions with the ReproSalud, due to high personnel turnover in the MOH at these levels. Health care providers. In the process evaluation, those interviewed were mainly directors of rural or periurban health posts, or midwives in health centers or hospitals. One criterion for the interviews was that the provider had to have interacted directly with the ReproSalud promoters and the CBOs. Local and communal authorities. Mainly mayors in urban areas, “governors” of neighborhoods in periurban communities, and communal authorities in rural areas. All of those interviewed were male. Other reproductive health project (RH) directors. Coordinators of other major RH projects or NGOs operating in the same department were interviewed. Not all of those interviewed had a close working relationship with ReproSalud. Phase One. From 1996-2000, the first phase in the ReproSalud Project. This phase concentrated on CBO-led activities to identify priority reproductive health problems and carry out educational workshops and other interventions to address these problems. Phase Two. From 2001-2005, the second phase in the ReproSalud Project. This phase concentrates on advocacy led by the CBO presidents and promoters. The sub grants support negotiated agreements with the health service referral center to improve quality of care, to increase usage of the services, and to promote health in the communities. Refrigerio. Refreshments served at a meeting or a talk. The refrigerio seemed to vary from site to site. It was often as simple as crackers and soda, but we heard some anecdotes of people cooking for the refrigerio. Reproductive tract infection (RTI). There are three kinds of RTIs: 1) sexually transmitted infections (STIs) such as gonorrhea, HIV/AIDS, chlamydia; 2) infections caused by overgrowth of bacteria naturally present in the vagina, such as bacterial vaginosis and candidiasis; and 3) iatrogenic infections, caused by medical procedures such as induced abortion, IUD insertion, or childbirth. This can happen if surgical instruments used in a procedure are not properly sterilized, or if an infection already present in the lower reproductive tract is pushed through the cervix into the upper reproductive tract. For a complete explanation of RTIs, see http://www.rho.org/html/rtis_overview.htm. In the ReproSalud Project, CBO promoters and members gave high priority to the reproductive health problems of "descensos" (discharge) or "regla blanca” (white menstrual period), which are the most common symptoms of RTIs such as candiasis, bacterial vaginosis, and trichomonas. Probably, there are several types of infections causing these problems in the project area. ReproSalud coordinators. The directors of the ReproSalud offices in each department. They supervise all staff, and direct the implementation of the project in their departments. ReproSalud regional promoters. The main ReproSalud staff working at the community level. Each regional promoter trains the CBO promoters and the steering committee and supervises the subprojects in one or more districts. Sexual and Reproductive Rights Defender’s Committees. The committees set up under Phase Two of the project to conduct negotiations with health authorities at the district or departmental level. The Defenders’ Committees are composed of the presidents of the CBOs participating in ReproSalud. Sexually transmitted infection. (STI). See explanation under "reproductive tract infection” Subproject or sub grant. The official mechanism for providing financial support to a plan of activities developed by CBOs in both Phases One and Two of the ReproSalud Project. Subproject steering committee. In Phase One, these committees were composed of elected coordinator and treasurer and all of the promoters. The CBO president supervises the steering committee. ACRONYMS AND FOREIGN TERMS CBO Community-based organization CLAS Comité Local de Administración de Salud (Local Committee for Health Administration) DISA The Authority at the Departmental level of the Ministry of Health MINSA MMR Movimiento Manuela Ramos (Manuela Ra mos Movement) MOH Ministry of Health NGO Nongovernmental organization RH Reproductive health RS ReproSalud RTI Reproductive tract infection STI Sexually transmitted infection USAID United States Agency for International Development USAID United States Agency for International Development i CONTENTS PREFACE TO THE REPORT ACKNOWLEDGMENTS GLOSSARY ACRONYMS AND FOREIGN TERMS I. Description of the ReproSalud Project............................................................................1 A. Description of Phase One ...................................................................................2 B. Description of Phase Two Advocacy Program..................................................4 II. Background: Movimiento Manuela Ramos and the Origins of the ReproSalud Project .......................................................................................................7 III. Observations on Phase One ...........................................................................................9 A. Experiences and Opinions of Key Stakeholders on Phase One .........................9 ReproSalud Staff Opinions on Factors in the Success of Phase One ..........9 Relations with the Health Sector................................................................10 Relations with Local Authorities ...............................................................12 Relations with other NGOs and Multisectoral Committees......................13 B. Perceived Impact of Phase One Program.........................................................15 General Perception of Impact on Women..................................................15 Impact on Women of Income Generation Component ..............................18 Impact on Men’s Attitudes and Behavior, and Impact on Relationships Between Men and Women.........................................................................19 Impact on Youth.........................................................................................20 Impact on the Use of Services and on Service Quality..............................21 Women’s Views on Quality of Care and “Intercutural” Treatment...........24 C. Stakeholders’ Recommendations to ReproSalud .............................................25 D. Comment on Phase One ...................................................................................26 IV. Observations on Phase Two........................................................................................27 A. Background Information..................................................................................27 B. Findings: Potential Barriers and Facilitating Factors in the Implementation of Phase Two..............................................................................................................28 Issues to be Negotiated: Shared and Unshared Priorities .........................29 C. Recommendations for Advocacy and Health Promotion for RTIs ..................34 D. Essential Copmmunications Flows in Advocacy Program, Phase Two ..........35 MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT ii Key Assumptions and Evaluation Questions in the Design of Advocacy Program......................................................................................................36 E. Concerns Raised by the Phase Two Advocacy Model and Recommendations37 V. Other Recommendations for Phase Two .....................................................................39 A. Maximizing and Sustaining Impact in Current Districts.................................39 Discussion and Recommendations ............................................................40 ReproSalud Should Involve Trained Male Promoters to Reach Their Peers .................................................................................................43 B. Expanding Coverage Through Replication......................................................44 C. Final Thought on Sustainability of Impact.......................................................50 VI. Evaluating Design and Recommendations .................................................................51 A. General Comment on Scaling Up and Evaluation...........................................51 B. Comments on Evaluation Concerns in ReproSalud .........................................52 C. General Recommendations on Evaluation.......................................................54 Methods of Data Collection.......................................................................54 Evaluation Questions for Phase Two Advocacy Component ....................55 Evaluation of Replication in New Communities or Districts ....................57 Evaluation Questions Related to the Communications Program...............58 Evaluation Questions or Studies Related to the Communities Currently Involved in the Project ...............................................................................56 VII. Concluding Remarks.................................................................................................59 VIII. References................................................................................................................61 1 I. DESCRIPTION OF THE REPROSALUD PROJECT The ReproSalud project is a unique case of a comprehensive scaled-up reproductive health and gender equity project whose beneficiaries are in hard-to-reach communities, where it is difficult to bring any intervention to scale. The project is a 10-year cooperative agreement1 (1995-2005) between USAID and Movimiento Manuela Ramos, a Peruvian feminist non-governmental organization (NGO), to improve the reproductive health of low-income women in rural and periurban zones in Peru. ReproSalud is a cooperative agreement between USAID and Movimiento Manuela Ramos signed in 1995, and beginning implementation in 1996. Its main goal is to improve the reproductive health of low-income women from rural and periurban zones in Perú. Its principal expected result is that women increase their use of all types of interventions that could protect their reproductive health, from individual health￾protective behaviors (such as improving iron intake in their diet) to increased use of formal health services. The project is currently active in the six departments of Ancash, Ayacucho, Huancavelica, La Libertad, Puno, and Lima. In many of the communities currently involved in the project, the principal language is either Quechua or Aymara. The principal expected result of ReproSalud is that women will increase their use of all types of interventions that could protect their reproductive health, from individual health-protective behaviors (such as improved hygiene to prevent RTIs2 and increased use of family planning methods) to increased use of formal health services. The project is also expected to have a positive impact on social and cultural factors that affect women’s health, mainly on traditional patterns of gender relations that subject women to violence and limit their autonomy, their participation in community affairs, and their ability to access health care. Phase One of the project (1996-2000) involved 2-3 cycles of sub grants to women’s community-based organizations (CBOs) in low-income communities in eight departments. Each sub grant supported participatory educational and community-based interventions on one or two specific reproductive-health problems identified by the community. In an additional component to empower women, during Phase One ReproSalud also supported income-generation through micro-credit (“community banks”) and “product development” (producing and marketing handicraft items for purchasers with bulk orders). These components are no longer part of the ReproSalud Project, since the micro- 1 The agreement was signed in August 1995, and the project began in 1996. It has had two five-year authorizations, with a current end-date of September 30, 2005. 2 Reproductive tract infections. See glossary for a full description. The community women refer to RTIs as descensos or regla blanca, i.e. vaginal discharge. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 2 credit program became self-sufficient in 1999, and product development received funding from another source in 2000.3 In Phase Two, which started in 2001, ReproSalud’s strategic objective is to establish a sustainable negotiated relationship within 78 defined catchments areas,4 between the health referral center5 for each area and the elected presidents and trained promoters of the CBOs in that area. The main results expected are greater numbers of women using the public health services, and services that are more acceptable to community women and responsive to their needs. A. DESCRIPTION OF PHASE ONE In Phase One, the project initiated activities in eight departments in Peru: the five highland departments of Ancash, Ayacucho, Huancavelica, La Libertad, Puno; two jungle departments – Ucayali and San Martin; and Lima. In 2000, ReproSalud and USAID staff decided to halt activities in Ucayali because of their high cost per beneficiary, and to halt the project in San Martín and Lima after the baseline survey revealed that the potential impact of the project in these two departments would be very limited. At the time of this mid-term evaluation, the project is active in the five highland departments.6 3 This evaluation concentrates on the reproductive health component of the program. The cost analysis analyzes the cost per beneficiary of the income-generation component. The product development program is now called “MERCOMUJER,” and receives support from the Small Enterprise Department Unit of USAID/Washington. 4 In Phase Two the number of districts involved is lower than the number of health centers/hospitals involved (71 versus 78) because some districts have more than one health center or hospital. Therefore, this report refers to “catchment areas” as opposed to “districts” when discussing Phase Two plans. 5 Health centers are secondary level facilities that serve as referral centers for a catchment area, with basic surgical, obstetrical, and hospital facilities. While hospitals are tertiary-level facilities, often they serve as the basic referral center, especially in urban areas. 6 Some results from Ucayali, San Martin, and Lima are in the cost analysis study, for comparative purposes. Since the communities in San Martín and Ucayali had completed the requisite two subprojects, they are included in the quantitative impact study. At the time of this evaluation study, two separate ReproSalud offices in the Aymara and Quechua-speaking areas of Puno had just been consolidated into one. Basic Principles of the ReproSalud Program · commitment to gender equity and women’s empowerment · commitment to participatory processes that put community members in charge, · promotion of sexual and reproductive health and rights; · respect for indigenous cultures, integrating modern health knowledge with traditional knowledge and practices that are not harmful to health. I. DESCRIPTION OF THE REPROSALUD PROJECT 3 So far in Phase One, 231 sub grantee CBOs have completed two subprojects.7 Each subproject involves several neighboring “associated” CBOs as well, bringing the total number of women’s organizations completing the project to 2,568. As of December 2001, 123,917 women and 66,370 men in eight departments — a total of 190,287 individuals — have directly benefited from the community-based educational subprojects in Phase One.8 Project data indicate that each CBO member has an average of 5.3 family members, so that one could estimate that up to one million people benefited directly or indirectly from the activities of the project in the communities where ReproSalud held educational activities.9 Seventy-seven percent of the beneficiaries live in rural, mainly mountainous, areas, and 23% live in periurban areas. However, the periurban sub grants often involved associated rural communities. To summarize briefly, the process in Phase One was the following:10 Selection of the subgrantee CBO11 (contraparte) was carried out first by selecting districts, or smaller geographical areas within districts, with high levels of unsatisfied basic needs, considering reproductive health statistics as well.12 Then all women’s CBOs in the district or area were invited to join a competition. First, they had to present written proof that they were a legitimate operating organization, in the form of by-laws, minutes of meetings, and letters of support from local authorities. CBOs that passed the first screening developed a “sociodrama” (a skit) about the reproductive health problem that they would like to address. A panel of judges, which included local officials, evaluated the skits. The winning CBO was awarded the task of administering the subproject. Administration involved convening and organizing all project meetings and events, managing the project’s finances, and taking the main responsibility for reporting. The other competitors were invited to participate in the subproject as “associated CBOs.” Their promoters were trained along with those of the winning CBO, and they gave the same educational workshops in their communities. The main difference between the subgrantee and associated CBOs was that the subgrantees gained legitimacy and experience as administrators of externally-funded projects. Identification of health problems. The autodiagnóstico is a participatory community health diagnosis. Over the course of four or five sessions, participants used a 7 162 have completed all three reproductive health themes, and 69 have completed two, the latter including 28 communities from the two jungle departments. 8 Database reports, ReproSalud, December 2001. 9 Average number of family members for those interviewed in the quantitative impact study. The exact number would be 1,008,521, but there is some duplication because some male and female beneficiaries are spouses. 10 The following description of Phase One is based on published work by Anna-Britt Coe, and borrows her language extensively. Coe, 2000, p. 12. 11 Also called the “winners” (ganadoras) by the project. The CBO administering the subgrant was called the “contraparte” in Spanish; in this report they are referred to as the “subgrantee.” 12 In 2000, 51 highlands CBOs were chosen on two additional criteria: expansion of the project’s rural coverage, and strategic placement in the catchment area of major health centers or hospitals. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 4 methodology called a “tree of problems” to identify and reflect on health problems and their root causes. At the end of the process, participants prioritized the problems and voted on the most important one to address in their subproject. The full cycle of Phase One subprojects was to include attention to three reproductive health problems. Design of the subproject. ReproSalud staff worked with each group of CBOs (i.e., the subgrantee and the associated CBOs in the same district or area) to design the intervention. Each subproject was about 6-10 months in length, and cost from $2,500- $6,500. The CBO contributed 10% of the costs. Under the supervision of the CBO president, a steering committee (Núcleo Responsable) was formed by an elected coordinator, a treasurer, and the group of trained promoters. The president, coordinator and treasurer were trained in project administration and accounting. Promoter selection. CBO members who wished to become promoters received training in reproductive health. Those with the highest test scores became CBO health promoters. They received a small financial incentive 13 in return for replicating their educational workshops within their communities. Educational workshops. These workshops used a participatory methodology that encouraged participants to reflect on both the physiological and social causes of health problems, as well as on possible solutions. The basic module addresses anatomy, physiology of reproduction, and gender roles. Other modules, based on the majority of CBOs’ choices of high-priority problems, include Too Many Children, Sicknesses of the Sexual Organs, and Difficulties in Childbirth. ReproSalud staff added a module on Violence, considering that addressing this problem was essential to the goal of achieving greater gender equity. Subproject evaluation. The CBO steering committee and all of the promoters evaluated each subproject along with ReproSalud staff, following a standard set of questions and a point system. They then reviewed their autodiagnósticos to confirm or change the topics for their second and third educational subprojects. B. DESCRIPTION OF PHASE TWO ADVOCACY PROGRAM For the purpose of negotiating agreements with the 78 referral centers, ReproSalud has organized the CBOs involved in ReproSalud in Phase One into 111 Sexual and Reproductive Rights Defender’s Committees—whose members are the presidents of the CBOs—and 111 corresponding promoters’ networks.14 The presidents, as the elected representatives of the women’s organizations, lend legitimacy to the negotiating team, while the promoters are in charge of the community-based health promotion. The negotiated agreements will form the basis of the 78 subprojects that ReproSalud will support in Phase Two. The defenders’ committees and promoters’ 13 ReproSalud pegged the amount of the financial incentive to the opportunity costs for the promoters, that is, what they would have earned by their usual productive activities. 14 Some of the 78 catchment areas involved in the project have a large enough geographical area to necessitate that two or more committee/network pairs be involved in the negotiations. I. DESCRIPTION OF THE REPROSALUD PROJECT 5 networks in each catchment area elect a five-person steering committee to conduct the negotiations, resulting in a 10-person Central Steering Committee. This committee also decides how to administer the subproject that will finance the activities agreed on in the negotiations. In Phase Two subprojects, the CBO promoters offer their commitment to run community educational workshops, to refer women to the health services, and to collect and provide feedback on the quality of services. In turn, the health professionals are expected to agree to do whatever they can to make the services more acceptable and responsive to the women from the communities in the catchment area. Thus, subprojects might include increased education on a particular topic, training of CBO promoters by MOH staff on key topics (thus replacing the function of the ReproSalud staff), training of providers in qua lity, and even funds to better equip a health post or center.15 Because the basic unit targeted for advocacy is a health center or hospital with its network of health posts, there are many fewer subgrants in this Phase, making the project less complicated administratively for ReproSalud, but possibly more complicated for the CBOs, which now have to organize themselves into catchment area–wide entities. The process for Phase Two is as follows: ß In order to prepare for the negotiations, all CBO promoters and presidents carry out a diagnosis (based on household interviews) of the main reproductive health problems and feedback about the health service. ß ReproSalud organizes training workshops for both the promoters and presidents in quality of care, advocacy (negotiations, etc.), leadership, and in some of the particulars of the MOH system. The training is meant to address concerns described in the section below on Phase Two, p. 47. ß The steering committees of the promoters’ and defenders’ networks meet to prepare their agenda. In order for this to work as planned, the representatives of the promoters at the community level must have communicated well enough with the leadership and the membership so that all communities’ views and demands are represented. ß One key negotiation goal for ReproSalud at this stage is that all of the CBO promoters receive official recognition from MOH as “community agents,” that is, they would continue to function as community-level health promoters under MOH’s supervision. Community agents receive certain benefits, such 15 The timing of this evaluation makes it impossible to comment on the content or strategies in the subprojects. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 6 as free heath services for themselves and their family, but do not receive any financial incentive or reimbursement for travel expenses.16 ß During the Phase Two subprojects, each CBO promoter will be in charge of 20–30 families, conducting house-to-house visits to encourage the use of health services when needed and to educate about preventive self-care practices and reproductive health. Promoters are also encouraged to continue community educational workshops, but this time without the small financial incentives offered in Phase One. Depending on the budget of the subproject, there may be a support for refreshments during workshops. ß Most negotiating teams expect that MOH will not be able to meet all of their demands in the negotiated agreements. The defenders’ networks will then work with ReproSalud staff to determine which unmet demands have some chance of being resolved at a higher level in the health system. Other demands might be adapted to make them more feasible. 16 See glossary. This is the official name of the MOH-sponsored health promoters. Informally, they are also referred to as promoters, but to avoid confusion, this report uses the term “community agents” when referring to the MOH promoters, and “CBO promoters” when referring to the community level promoters trained by ReproSalud. 7 II. BACKGROUND: MOVIMIENTO MANUELA RAMOS AND THE ORIGINS OF THE REPROSALUD PROJECT Movimiento Manuela Ramos (MMR) was faced with a formidable organizational challenge when they started ReproSalud in 1995-96. They had transform themselves into an organization with national scope, and with the infrastructure and staff needed to work in indigenous communities in rural areas. All of the regional staff in eight new offices in seven departments were new to the organization and had to be trained. Staff emphasized that the first two years involved much trial and error as they learned how best to achieve their goals. Although MMR staff had experience in implementing similar projects on a small scale, ReproSalud was a new scaled-up venture that aimed to conserve the essential elements of smaller pilot projects while designing an easily replicable model. The main organizational challenges in the ReproSalud project were as follows: ß Starting and equipping eight new regional offices ß Hiring and training staff who were fluent in Quechua or Aymara as needed, and sympathetic to the basic principles of the project. ß Developing the administrative, financial, monitoring, and supervision systems needed for such a large-scale project ß Dealing with the complexities of procuring vehicles and equipment and of reporting in their first experience as a USAID grantee ß Developing all of the educational and training materials used in the project, an investment and activity, which was not in the original proposal. Designers had assumed that other institutions would have developed appropriate materials. ß Starting on a large scale both a reproductive health program and two programs for income generation. ß Coping with under-investment in evaluation and lack of an evaluation and monitoring director for most of the life of the project. This lack led to conflicting opinions and mandates about how to evaluate the project, how involved MMR should be in the evaluation, and what basic research should be conducted. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 8 ß Responding to the sometimes-conflicting concerns of different stakeholders: USAID, consultants, MOH, the CBOs, local authorities, their colleagues in other feminist organizations, and other reproductive health NGOs. ß Persuading skeptics within MOH and other organizations that rural indigenous women with low educational levels have the capacity to diagnose reproductive health problems, set appropriate priorities, administer subprojects responsibly, and identify the key improvements in services that will make them more acceptable and responsive to rural indigenous communities. ß Meeting unforeseen demands arising from the initial community diagnostic workshops. When significant numbers of women demanded that the project include education for men, ReproSalud faced the additional challenge of adding educational activities that were appropriate for men to the project. This entailed hiring male regional staff, developing new educational curricula and materials, and training community-level male promoters—all of which incurred costs that were not included in the original budget.17 In summary, MMR has implemented a large-scale nationwide project that surpassed its original objectives for coverage, using a highly participatory methodology that defied standardization, in hard-to-reach communities that posed numerous logistical and cultural barriers to their work. The project managers evidenced flexibility in acceding to the community women’s requests to work with men as well as women, necessitating a major unplanned investment in staff hiring, training, and materials development. ReproSalud has enjoyed a measure of impact and acceptance in these communities that external stakeholders18 recognize as an important achievement. 17 At the end of Phase One, the regional male promoters (ReproSalud staff) were laid off. Given that the Phase Two training focused on negotiations between the women’s CBOs and the health centers, the male promoters were not invited to the training sessions. 18 Respondents in this category were health authorities, health providers, local authorities, and other NGOs working in the same regions. 9 III. OBSERVATIONS ON PHASE ONE A. EXPERIENCES AND OPINIONS OF KEY STAKEHOLDERS ON PHASE ONE The opinions of both internal and external stakeholders who will be key to the success of Phase Two are mainly favorable, laying a sound foundation for the next four years of the project. Health authorities and health providers appreciate the “bridging” role between health services and low-income communities played by the CBO promoters trained by ReproSalud, and tend to credit these promoters wholly or partially for notable increases in service use. Likewise, CBO promoters had a mainly favorable view of their relationship with the health institutions that they refer to. Health professionals’ main request was for closer and more regular cooperation; the design of Phase Two should guarantee that this request is met. Local authorities also tended to have a favorable opinion of the project and provided crucial support during Phase One. Increased cooperation with them in Phase Two should help the project become sustainable. A small sample of other NGOs mainly had favorable opinions of their collaboration with ReproSalud and/or of the project model. Criticisms pertained to their desire for greater cooperation, and access to replication of the project. At the community level, the promoter training sessions and the subsequent educational workshops in the communities were extremely popular with CBO promoters and members, as well as with local authorities. The popularity of this component has created a demand for permanent education and training that the project cannot fully respond to in this next Phase. Another indicator of community satisfaction is the low dropout rate from the project. Only 16 out of 247 CBOs (6.5%) dropped out of the project, for a variety of reasons. ReproSalud staff expressed that MMR has perfected the process for Phase One as it went along. Therefore, they had few substantive suggestions for changes in the model. ReproSalud Staff Opinions on Factors in the Success of Phase One ß ReproSalud staff and several other respondents highlighted the autodiagnóstico as being a key factor in their ability to work in these highland communities. First, the process helped them understand the women’s views about their health problems, including why they prioritized these problems. Second, the process brought the CBO members on board, involved them in identifying problems, put them in charge of basic decisions about the educational focus of the subproject, and helped secure their commitment for the interventions that followed. ß ReproSalud staff believes that the process of choosing the communities in which they would intervene was effective. Screening first by level of need, and then by MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 10 the organizational capacity of the CBO, generally resulted in subprojects that both met their goals and had a measurable impact. The use of socio-drama competitions as an indicator of organizational capacity was effective in most cases; we only heard scattered anecdotes of sub grantees that did not perform their responsibilities. However, a few MOH and ReproSalud staff believe that proximity to a MOH service and the opinions of MOH providers should have been taken into account in the selection of the communities or districts. Providers often know which communities or districts have the most pressing reproductive health needs in their catchments area. ß ReproSalud’s policy of flexibility to respond to needs identified by the CBOs was extremely important to the results. In theory, this responsiveness is the rule in a participatory project, but in fact, the principle is rarely put into practice when the participatory process turns up priorities that the NGO or agency had not foreseen or budgeted for, and may not share. A major turning point in the project was its inclusion of education for men, once a clear demand for this arose from a sizable group of CBOs during subproject design. Relations with the Health Sector Gaining the support of health sector authorities and providers was a secondary goal, and not a central focus, in all of the Phase One projects. This aspect of the design of Phase One is due in part to historical circumstances and in part to USAID’s (Project 200019) and the MOH’s complementary investments in improving the quality of care in the departments where ReproSalud was active. During the first three years of the project, relations between the health sector and the feminist and reproductive health NGOs were often strained because of the government’s sterilization campaigns.20 Nevertheless, all of the subproject evaluations include a section on progress in contacting health personnel, which was a task of the promoters. In late 2001, four years after the campaigns ended, the relationship between ReproSalud staff and CBO promoters with the health sector seems well established and still seems to be improving, as evidenced by the analysis of data below from both health sector professionals and CBO promoters.21 The following tables analyze interviewees’ responses to questions about relations between ReproSalud and the health sector. 19 Project 2000 is a 9-year bilateral agreement between USAID and MOH, with technical assistance from Pathfinder International, that focuses on improving the quality of care in maternal-child health in MOH secondary and tertiary level facilities, in order to increase their usage. The program conducted intensive training of administrators and providers in quality principles, and evaluated progress closely, granting the institution "certification" as high quality once the facility had achieved certain quality benchmarks. 20 During 1994-1997, the Fujimori government heavily promoted widespread sterilization campaigns, with providers under pressure to fulfill monthly and yearly targets. In some instances, this pressure generated a coercive mode of interaction with users that made official cooperation extremely difficult with projects such as ReproSalud that operated within a reproductive rights framework. See Tamayo, 1999, and Shepard, 2002. 21 One exception to this positive trend was among some health authorities in Huancavelica, where ReproSalud publicly denounced a Ministry staff person for coercing women to insert IUDs. 11 PHASE ONE EXPERIENCES HEALTH AUTHORITIES ON LEVEL OF COORDINATION WITH: REPROSALUD STAFF CBO PROMOTERS HIGH MEDIUM LOW HIGH MEDIUM LOW Ayacucho 3 3 1 1 6 Ancash 4 1 2 1 Huancavelica 3 1 2 1 1 4 La Libertad 3 1 1 3 Puno 1 1 1 3 Subtotal 14 6 4 3 7 14 HEALTH PROVIDERS ON LEVEL OF COORDINATION WITH: REPROSALUD STAFF CBO PROMOTERS HIGH MEDIUM LOW HIGH MEDIUM LOW Ayacucho 1 2 1 2 Ancash 2 1 3 Huancavelica 2 2 2 2 La Libertad 2 1 3 Puno 2 1 3 Subtotal 9 7 0 12 4 0 Fourteen out of 24 health authorities ranked their level of coordination with ReproSalud staff as high. At the local level, health providers divided their ranks between high (9) and medium (7). Not surprisingly, the health authorities had little contact with CBO promoters, with the exception of the Coordinators of Community Participation in the DISA (the departmental health authority). While health authorities tended to give neutral factual descriptions when asked about their experiences with ReproSalud, 12 out of 14 health providers — who have had the closest contact to date — spontaneously made positive comments. Many CBO promoters are very actively referring women to health services. Health providers greatly appreciate this, because it helps them with their coverage targets. There are accounts of promoters heroically exerting themselves to overcome opposition from a family or husband in order to get a woman to the health post for childbirth or treatment of cervical cancer. (See Annex III.1. on “Achievements”) The promoters’ view of their relationship with the health providers also tended to be positive, and focused on their role as a source of referrals to the health center. (See Annex II, Table 2a.) MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 12 PROMOTERS QUESTION 12 Relationship with MINSA Positive Mixed Negative N/A Total Ayacucho 7 1 2 Ancash 6 1 1 Huancavelica 6 2 1 La Libertad 6 1 1 Puno 5 4 TOTALS 30 5 6 3 44 When asked how ReproSalud could improve its relationship with the health sector and about general recommendations for ReproSalud, the clear majority of health sector respondents requested more coordination. They expressed a general goal of working jointly on shared priorities, and sometimes said they would appreciate regularly programmed meetings. Some respondents had specific ideas, expressed in the table below. 22 Phase Two incorporates greater coordination for mutual benefit in the negotiated agreements, so there is a high level of congruence between ReproSalud’s priorities and those of health authorities and providers. How to Improve Relationship with MOH? Health Authorities & Health Providers Better Coordination Continue education & training (Phase I) Improve supervision of promoters & strengthen referrals More understandi ng of MINSA norms, limits, working conditions Joint training of MINSA and RS promoters Phase II plans, negotiations with CBO members Train MINSA staff in quality & RS model. Ancash 3 3 3 Ayacucho 4 1 2 1 1 Huancavelica 3 2 1 2 La Libertad 3 1 1 1 Puno 4 1 1 TOTALS 17 6 7 2 1 2 3 Relations with Local Authorities23 In general, the level of support expressed by local authorities seems very high. Out of the 20 local authorities interviewed, 16 expressed positive opinions, 2 were neutral, and 2 (in Ancash and Ayacucho) were openly hostile. Seventeen out of 20 reported that 22 Unfortunately, this sample is too small and the specific ideas too varied to provide useful guidance to the program. 23 See glossary. All authorities interviewed were male and were elected officials. 13 ReproSalud has had a positive impact in the community, with emphasis on family planning use and a lower incidence of domestic violence. In their Phase One subprojects, the CBOs were required to provide a local contribution that amounted to about 10% of the value of the project. In many cases, this contribution took the form of a locale or other resources provided by local authorities. More than 17 out of 33 promoters mentioned that local authorities provided the locale used by the CBO; in some cases land was donated to build a locale. Other types of logistical support included transportation, equipment and materials, and refreshments. According to several respondents, local elected authorities play a key social and political role in these highland communities. It is therefore very important to have their support to lend legitimacy to new and somewhat controversial projects such as ReproSalud. Fifteen CBO promoters spoke of the importance of having local authorities help convene community members to the promoters’ educational workshops. Especially in outreach to men, the approval or participation of local authorities seems to have made a crucial difference. In many cases, promoters mentioned that the men would not have let their wives attend the workshops, or would not have attended themselves, without this seal of legitimacy from the local authority. In the subprojects that worked with men, some supportive local authorities went house-to-house, talking with the men and convincing them to attend. Many local authorities either attended the workshops for men, or lent their official stamp of approval by inaugurating the project. Finally, in a small number of cases the local authorities helped with official support when the CBO needed it, whether for legal recognition or negotiating an issue with the health sector. In Phase Two advocacy efforts, this type of support will be extremely important. Relations with Other NGOs and Multisectoral Committees Only 10 other NGOs were interviewed, two per department. The size of the sample makes it difficult to draw any firm conclusions, except for a general impression from the interviews that other NGOs have a high opinion of the project model, and that there is unmet demand among these NGOs for access to the materials and training to replicate the project wholly or partially. Six NGOs described a very positive relationship with ReproSalud regional staff. Two admired the model but faulted ReproSalud for poor coordination, and two were critical, saying that ReproSalud does not coordinate sufficiently with them or with the health sector. One NGO criticized ReproSalud for its supposed over-emphasis of natural family-planning methods.24 Most of the departments have some kind of multisectoral health committees, either at the departmental, provincial, or district level. 25 Generally, these are convened by a Ministry 24 Staff said that this impression is erroneous, arising from misinterpretation of a brainstorming list of issues during a training workshop. The list included a statement by one of the participants to the effect that natural family planning is the only safe method. This view is common among women in highland communities, and it is not surprising that it surfaced at the workshop. 25 Also called “mesas de concertación,” (coordination committees) MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 14 and include NGOs, municipal authorities, and, depending on the topic, other government officials. When convened by the Ministry of Health, these committees generally have the purpose of bringing all of the resources in the region to bear, in a coordinated way, on the achievement of shared objectives. During the team’s visit, for example, there was a national measles vaccination campaign. These committees did not seem to have a significant presence in Ancash and Huancavelica. In Puno there were committees at the provincial level, but few at the departmental level. ReproSalud was involved in all operational committees,26 with the exception of two committees in La Libertad. Only one of the committees described by informants involved CBOs. When these committees operate as planned, they hold great promise for providing an institutional space for coordination between the Ministry and civil society groups. However, they do not hold high potential as a venue for negotiating demands. Their level of operation is so variable that ReproSalud staff and CBOs would have to evaluate them on a case-by-case basis to determine whether it is feasible to use them to advance dialogue between CBOs and the health sector. Interviews with ReproSalud staff and other NGOs yielded information about relationships with NGOs that are replicating the project. The issue of financial incentives to promoters ReproSalud’s policy of paying CBO promoters was the aspect of Phase One that caused the greatest controversy — and often hostility — among both health sector respondents and other NGOs engaged in reproductive health. There are valid arguments for giving financial incentives, and for not giving them. MOH is not sufficiently funded to be able to give any financial incentives to their community agents; their agents are not even reimbursed for travel costs. However, they do receive training, exemption from service fees for themselves and their families, and in some cases goods from government programs. International NGOs follow the same policy, so as not to create problems with MOH, and also because the policy of giving financial incentives is not “sustainable.” The main concern that MOH personnel and other NGOs voice is that if promoters become used to incentives, they will not work without payment when the project that funded the incentives ends—and thus, the investment will be lost. ReproSalud staff, on the other hand, believes that payment is a matter of social justice and gender equity, since low￾income women are always asked to contribute their time for free, while people assume that men need to be paid. ReproSalud tried to set the amount of the financial incentive at the level of “opportunity costs,” that is, what women would earn in a day working on their land. The evaluation of Phase Two will determine whether ReproSalud’s policy results in lack of sustainability in the CBO promoters’ involvement. 26 Health professionals reported several that exist in name only, meet very rarely, or involve NGOs only tangentially. 15 B. PERCEIVED IMPACT OF PHASE ONE PROGRAM All respondents were asked about their opinions on the impact of the program on women, men, relationships between the couple, and relations between community members or CBOs and the health sector. Local authorities and promoters were asked a general question about the impact on women and men. CBO members were asked about the effect on them personally and about their relationship with health providers. Health authorities and providers were asked about improvements in the use of services and the quality of services, the user/provider relationship, and their views on the role of ReproSalud in any changes they perceive. General Perception of Impact on Women The process evaluation results confirm that most stakeholders, including women themselves, perceive that the project has had a significant, positive effect on women’s lives and health. The most often-mentioned impact was an increase in knowledge about reproductive and sexual health topics. Another frequently mentioned impact was an increase in assertiveness in general, especially with providers, and within the family. The changes falling into this category that were most often mentioned include a loss of shame and the ability to speak in public, to speak on taboo topics, to engage in dialogue with spouses and health providers, and to stand up for one’s rights. ReproSalud staff, health professionals, and other NGOs all speculated on the features of the project that helped it to achieve this impact on women. Several mentioned that the project is unique in its foundation of respect for and in-depth knowledge of Andean cultures, in a society that many Peruvians characterize as racist and discriminatory. Others highlighted the project’s focus on gender issues and the highly participatory methodology. The following points explain in more depth the aspects of the project that empowered women, according to several informants: ß The highly participatory methodology of the autodiagnóstico, and then the design of subprojects. Women who were not used to having their opinions heard and respected were in charge of assessing the community’s needs and designing a project to answer those ne eds. They needed to learn to speak out in a group and give value to their opinions. ß The leadership of the sub grantee CBOs were in charge of organizing the activities and managing the finances for the subproject. They learned to report on accomplishments using monitoring forms and to carry out a self-evaluation. While they were accountable to ReproSalud, as any grantee is to any donor, they were in charge, and thus gained legitimacy in the eyes of the community. ß The project had a policy of hiring Quechua- and Aymara-speaking regional promoters and conducting training in these languages, counteracting the usual MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 16 discrimination faced by women who are most comfortable speaking those languages. ß The project had a policy of respecting the women’s traditional health beliefs and integrating them in the educational modules with modern health knowledge, whenever traditional beliefs were not damaging to health. This is very evident in the projects’ work on childbirth and maternal mortality. ReproSalud staff has helped the CBOs to negotiate with some health centers and hospitals to incorporate a complex variety of traditional practices associated with childbirth, while at the same time emphasizing the recognition of the warning signs for a high-risk birth or obstetrical emergency and the importance of having all births attended by a trained professional. ß The project explicitly addressed gender issues, promoting more equity in relationships with men, authorities, and health providers; informing women of their rights; and encouraging them to give greater importance to their health, their well-being, girls’ education, and their voice within the family and community. Although women did not choose the issue of domestic violence as their highest priority topic for Phase One workshops, ReproSalud added a module on violence. They believe that women’s autonomy in general, and their ability to safeguard their reproductive health in particular, are both severely compromised when violence against women is not directly challenged as a norm. ß The project explicitly broached taboo subjects, not only with words but also with pictures, which many women initially found embarrassing. Eventually, these women were able to call body parts by their name and talk explicitly about sexual and reproductive health issues, thus communicating more effectively with health providers. The themes most frequently mentioned by respondents when reflecting on the projects’ impact were as follows: ß The type of impact most mentioned was an increase in knowledge about sexual and reproductive health topics. All those attending workshops took a pre-test and post-test, the results of which were reported in the monitoring system, and showed impressive short-term gains in knowledge. The quantitative study also showed significant long-term gains in knowledge.27 ß Giving greater importance to self-care -- such as changes in hygienic practices to prevent reproductive tract infections (RTIs) -- and to their own health, including overcoming family’s opposition to taking women to health centers when in labor. ß Increased willingness to use health services for Pap smears, family planning, RTI treatment, or childbirth. 27 See Annex III.2. for a detailed discussion of the findings on knowledge of the fertile period, and Annex 5 to the Executive Summary on Detailed Findings on Impact. 17 ß Increased willingness to use modern family planning methods ß Ability to be assertive with health providers about the need for information, for respectful treatment, for waivers of service fees, and for culturally appropriate childbirth services. ß Other themes that emerged had much to do with the increased ability to communicate in general 1. Loss of an overall feeling of shame and timidity, resulting in the ability to speak in a group. Women in these communities are often ashamed to speak in public. 2. Loss of embarrassment about formerly taboo topics, resulting in ability to speak about them not just with each other, but also with their spouses, with health providers, and in public meetings. 3. Better communication in the couple, and greater ability to negotiate needs and wishes within the couple. Some mentioned examples of women negotiating greater autonomy and freedom of movement than they had before. ß ReproSalud staff cited many instances of increased political participation of women and increased respect for women’s abilities among local authorities. Theoretically, this would especially be true in the sub grantee CBOs, where the women developed plans and budgets, organized complex activities, and kept track of finances. There are numerous accounts of CBO members who lost their fear of public speaking because of ReproSalud training and, as a result, became more active in their communities and were elected as local authorities within the last three years. The following table shows the opinions of most stakeholders (all except the CBO promoters)28 about the project’s impact on women. 28 The responses of the CBO promoters to a general question posed to them on impact are not included in this analysis because they were overly generalized and positive. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 18 IMPACT ON WOMEN More knowledge about RH "Takes care of her health" (Includes Better hygiene, RTIs and ETS) More use of services Use of Family Planning Can communicate w/ Provider / makes Demands & Give Feedback Health Authorities 10 5 6 3 3 Providers 8 4 7 2 1 Local Authorities 2 5 3 3 1 CBO Members 19 18 11 15 7 TOTAL 39 32 27 23 12 IMPACT ON WOMEN Better communi￾cation & relationship with spouse More political participation No or very few changes Health Authorities 1 1 1 Providers 1 1 1 Local Authorities 5 2 4 CBO Members 2 2 TOTAL 9 6 6 It is interesting to note that the CBO members were no more likely than any other respondent to highlight changes related to gender relations or gender equity. When women were asked about the workshops given by promoters, 35 mentioned health-related topics and only five mentioned the content related to gender issues. This is a first clue that the content on gender issues need to be strengthened in the replication of Phase One training. However, one could argue that the types of impact described above related to communications are gender-related. These changes are evidence that women have overcome some of the limitations of the traditional female role, which restricts women to private spaces and passive acceptance of what others dictate. Quotes from both providers and CBO members (see section p. 30 on quality of care) reaffirm this perception of fundamental change in the communications between female users and providers of either sex. Impact on Women of Income Generation Component The aspect of the project that was most controversial within USAID and international reproductive health NGOs — its use of population funds to support income-generation for women in the form of micro credit programs and product development — was 19 designed to enhance the effects of reproductive health and gender education. Besides rendering obvious benefits to these low-income families, increasing women’s income was designed to help promote the changes in gender relations and increased used of services that the project aspired to. Unfortunately, the quantitative study sample is too small and the conclusions too mixed to draw any definitive conclusions about whether this synergy occurred. In this study, none of the ReproSalud staff noticed any significant differences in impact between the communities that had both the reproductive health program and the income-generation program, and those that had reproductive health alone. However, several CBO members and local authorities requested such an income￾generation program in the next phase, arguing that interventions that increase family income would reduce resistance to the program among the more skeptical men and local authorities. Impact on Men’s Attitudes and Behavior, and Impact on Relationships Between and Women and Men The results for male participants are encouraging, but reflect the fact that only about 50% of the subprojects worked with men. Phase One workshops were held for men only when the women’s CBO requested it in their subproject proposal. This may partially explain the high number of respondents who said that the work with men was insufficient. The table above refers to the themes of the responses on impact; many gave mixed responses, pointing out positive changes, and at the same time, explaining why what had been done was insufficient. Male promoters were not connected to any community-based organization, in part because the men’s entry point to the project was the women’s CBO. The men who REPORT 8 - IMPACT ON MEN Insuffi￾cient Impact on Men Positive Attitudes in Gender Relations (General Statements) Positive Change in Family Planning Use & Attitudes Increased RH Health Care Knowledge & Education Decrease of Domestic Violence & rape Improved Self-care & Hygiene ( Prevention of STDs) Increased Use of Services by Men & Women Men's Domestic Help Improved Dialogue and Communic ation with Women (& family) HEALTH AUTHORITIES 2 6 5 5 6 3 4 LOCAL AUTHORITIES 5 5 5 2 1 6 3 2 2 PROVIDERS 7 4 3 4 2 1 3 CBO MEMBERS 11 11 7 4 6 1 1 6 2 TOTAL 25 26 20 15 15 10 9 8 7 MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 20 participated in the educational activity were convened haphazardly, probably through friendships and family networks. Some of the male promoters were husbands of female promoters. From a comparison of the interviews of female and male promoters, it is clear that male promoters had more trouble recruiting a stable group for their workshops. They encountered greater resistance among the men in their communities than the female promoters did among women. This difficulty suggests that replication of this program among men should involve a CBO with male membership, such as a sports club or a group convened to organize the community for development or agricultural purposes. Lacking the convening mechanism of a CBO, the local authority often helped convene the men, sometimes going house to house. Several respondents who gave suggestions on how to entice resistant men to workshops mentioned the house-to-house strategy as the most appropriate. At the time of the data collection, Phase One subprojects had ended, and only female CBO promoters were invited for the advocacy-related training of Phase Two. This led to a perception at the community level that ReproSalud’s work with men had ended. Most ReproSalud staff and CBO female promoters felt that this was not a problem, that the men understood that the women should be in charge of negotiating with the health center since they are the main users. The small sample (12) of male promoters was generally enthusiastic about their work and about the ReproSalud project; three male promoters were clearly disgruntled about not being included in the new ReproSalud training. Although some educational workshops will be part of the subprojects in Phase Two, many of the respondents were not sure whether men would participate in them. Some of the male promoters clearly expected to continue to give workshops. Continuance and strengthening of ReproSalud’s educational activities for men enjoys broad-based support among the stakeholders of the project.29 100% of CBO promoters and members said “yes” when asked whether work with men should continue, stating that violence and other negative attitudes continue in their communities. Among all respondents, when asked how the project might expand, more work with men and on violence was in the top three most frequent responses, and in the top five when respondents were asked a general question about their suggestion to ReproSalud for the future. Impact on Youth Work with unmarried adolescents was not planned in the project, yet 232 of the 435 subprojects in Phase One involved working with youth. The age group from 15 to 24 is underrepresented in the CBOs, which tend to include mainly married women with children; therefore, this group is also under-represented in the quantitative impact study. Not enough data emerged from this study to give any meaningful opinions on impact, and the quantitative study did not analyze results by age. A secondary analysis would help determine whether projects that included your promoter training had a higher impact on youth. 29 See tables 5b, 6, and 11a in Annex II. 21 Impact on the Use of Services and on Service Quality ReproSalud was originally designed to serve as a “demand-side” complement, at the community level, to other USAID-supported “supply-side” projects, especially Project 2000. The two projects were designed to work simultaneously; Project 200030 would ensure that the quality of services was improved while ReproSalud and CBO promoters worked at the community level to increase people’s reproductive health knowledge and their willingness to use the services. The CBO promoters were to refer women to health centers for childbirth, Pap smears, RTIs, family planning, and pre-natal care. Therefore, while impact on the quality of services was not an explicit goal for ReproSalud in Phase One, an increase in service use is one of the principal expected long-term results from the project. Indeed, data from both this study and the quantitative study indicate that Phase One had a significant impact on service use. Unexpectedly, there is some evidence that the project has had a direct, albeit lesser, effect on the quality of services as well. Impact on Use of Services In this study, health professionals had highly favorable opinions about ReproSalud’s impact on the use of services. Out of 40 health-sector professionals interviewed, 100% reported a significant increase in the number of users of their services. More than 75% (31 out of 40) attributed this increase wholly (18 out of 40) or partially (13 out of 40) to the efforts of ReproSalud.31 The other most-cited factors in this increase were related to MOH efforts — increased financial access through maternal and schoolchild health insurance and increased quality of care. The issue of increasing coverage, especially for any services related to reducing maternal mortality, is highly important to these professionals, so that they are likely to have a favorable view of any project that helps them to achieve this goal. 32 30 See glossary, and footnote 19. 31 See Table below. ReproSalud did a study of use of services in a sample of the CBOs, published in September 2000. The conclusions only say that there were increases in family planning use in the majority of establishments after the end of the first subproject; there is not a complete analysis of the results. The study design cannot link these positive results with any service-related or ReproSalud-related factors. 32 MOH’s priorities and how they interact with the goals of the project will be discussed in depth below in the section on the Advocacy Program. pp. 35-41. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 22 DID REPROSALUD HAVE AN IMPACT ON USE OF SERVICES ? HEALTH AUTHORITIES & PROVIDERS Due to Reprosalud Partially Due to Reprosalud Unclear No/ Other Reason Ancash 3 1 3 Ayacucho 5 3 1 Huancavelica 2 6 3 La Libertad 5 1 1 Puno 3 3 TOTAL 18 13 5 4 Quality of Care Issues in the Health Sector The health professionals’ discussions of their goals for increase in coverage and in service quality were interwoven; most understand that increases in quality are an essential part of strategies to increase coverage, especially in these “hard-to-reach” communities that distrust the modern health sector and modern medicines. All health sector respondents demonstrated a high level of attention to quality concerns, and MOH evaluation criteria fully incorporate quality principles. However, as is frequently the case, reality lags behind rhetoric. The team heard about many current problems, including a lack of essential supplies, verbal abuse, and failure to deliver results of Pap smears.33 Still, the table below shows multiple examples of concrete changes and improvements. Moreover, the health sector’s high level of attention to quality should facilitate the CBOs’ negotiations with the health sector in Phase Two. Provider attends home births or home prenatal Provider adapts to cultural preference s Free childbirth services Better treatment (more "friendly", more responsive) More Quechua speakers Increase access through hours, more personnel, etc. New services, RTIs, violence Less waiting time, more efficienc y Ancash 1 4 3 4* 1 Ayacucho 3 2 1 Huancavelica 2 5 4 2 2 1 La Libertad 1 4 1 5 3 2 Puno 2 1 1 2 1 TOTALS 4 18 8 10 3 7 2 2 CHILDBIRTH TOPICS Changes Implemented in Services to Improve Quality, according to health professionals 33 There were some terrible accounts of verbal abuse from users of one hospital in Puno that was certified as high quality by Project 2000. See table 3d, Annex II on CBO members’ complaints. 23 Three-quarters of the health sector respondents mentioned changes designed to increase their quality of care and/or their coverage. These changes are summarized above. Given MOH’s priority of reducing maternal mortality, topics related to obstetrical services received the bulk of the attention; the other most frequently mentioned improvement was in the manner in which service users were treated. Despite the frequently cited budgetary limitations in MOH, seven facilities were able to increase hours and personnel. Although reduction in waiting time -- women’s other major complaint —— was rarely mentioned, increases in hours and personnel should have a significant effect on waiting time.34 At the time of this report, the influence of Project 2000 was evident in interviews with health authorities and providers. Twenty-four out of 40 respondents mentioned user satisfaction and feedback as their main evaluation criteria.35 No other topic received more than seven mentions. In addition, in some departments the Population Council helped authorities conduct user-satisfaction surveys. The strong emphasis on user feedback is a key facilitating factor for the negotiations between the CBOs and the health sector. Given this emphasis within the health sector, one of the main benefits that the CBO promoters can offer is to serve as a channel for user feedback, and to stimulate users to voice their concerns. ReproSalud’s Impact on Quality of Care36 There is evidence that ReproSalud had an indirect, albeit unplanned, effect on quality of services as well. As subprojects addressed women’s prioritized reproductive health problems, the CBO promoters began to refer women to services, and as a result, entered into active dialogue with service providers. Feedback from community women on quality issues and on other barriers to the use of services— channeled through the CBO promoters—has been part of this dialogue. Forty percent of MOH professionals attributed improvements in quality to their own efforts or to those of projects such as Project 2000. However, 60% (18 out of 30) of MOH professionals believed that ReproSalud has played a significant role in observed increases in the quality of services. Fourteen respondents mentioned changes in the service users’ knowledge and attitudes as a contributing factor to quality improvements. They made the following main points: ß Users that give feedback and make demands contribute to efforts to improve quality (7) 34 While the timing of this evaluation does not allow for a thorough analysis of Phase Two demands in the negotiation process and of the responses of the health sector, in some of the first negotiations, promoters reported that their requests for expansion of hours of attention were granted, or are under serious consideration. 35 See Table 3a in Annex II. 36 See Tables 3b, 3c, and 3f in Annex II. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 24 ß More educated and communicative users make the providers’ job much easier and more efficient (7) When asked a related question, whether ReproSalud has affected the provider/user relationship, 72% (33/46) of health professionals and 80 percent (30/37) of the CBO members made these same two points. (See Annex II, table 3c.) Women’s Views on Quality of Care and “Intercultural” Treatment37 According to ReproSalud staff and some CBO members, when community women consider quality of care issues, they place a high priority on the capacity of a health service to resolve their health problem. Local health posts often lack essential medicines, supplies, and adequately trained personnel, necessitating a long and costly trip to a health center in order to resolve the ir health problem. In the case of medicines for RTIs, the health center cannot provide free or low-cost medicines, so that the women have not been able tot resolve this important health problem. In this study, and several others in Peru, another of women’s most frequent complaints is that of “mistreatment,” referring usually to verbal abuse. Indigenous women complain that they are discriminated against, yelled at, scolded, called promiscuous, called cochina (filthy, like a pig), told that they smell, and are not given information. Three CBO members in Puno describe the situation: “I want all women to be treated alike, even when they come in poleras [traditional woolen skirt]. Sometimes those who come from rural areas are smelly, so providers don’t want to serve them or touch them; they yell at them.” In the hospital, providers despise women who come in poleras, and they say, “You are filthy.” “They don’t treat us well. They just scold us, they yell, ‘you’re dirty.’” “Regarding women’s hygiene, we know that it is customary for women in the sierras to not wash their private parts. They say that they can go 8 or 10 days without washing because they believe that the urine cleans the vagina: It is warm, sterile, and a disinfectant that kills microbes on the genitals. They think, “If animals don’t wash themselves, why should they?” The odor does not bother them. For them it smells normal, not dirty. They dry themselves with the last polera (traditional woolen skirt) that they have (they use 9 to 13 poleras). They dry themselves with the one closest to the skin. They change it once a week. When they visit the clinic they are asked to change the polerita, and they get washed with cold water, and they say that this makes them catch cold. This makes them not want to go to the clinic. They also do not like to expose their genitals. Now things are changing: They are given soap and a bowl of warm water and can wash themselves. This works better. In our meeting, we told them that they had to do this every day, and especially when they were going to have sexual relations, and that the men had to wash themselves as well [in order to avoid RTIs]. Little by little they are beginning to understand.”38 The quote above shows how one health professional explains and deals with this situation—with an attitude of respect for the women’s behavior, which she interprets as due to women’s customs and belief systems. The infrequent washing may not be solely a matter of beliefs and customs, however. Several ReproSalud staff pointed out that the 37 See Table 3d Annex II for a summary of women’s complaints, in response to a question about whether they felt ashamed or uncomfortable for any reason during their visits to the health provider. 38 DISA offical, La Libertad. 25 official fails to mention that many rural highlands women do not have access to warm water at home. Lack of education is also an issue; the ReproSalud educational module “Sicknesses of the Sexual Organs” has a section on self-care and hygiene that made a big impression among the respondents who attended the workshops. They cited learning the importance of washing the genital area and proper washing after defecation as one of the most important benefits from Phase One. C. STAKEHOLDERS’ RECOMMENDATIONS TO REPROSALUD There is a clear majority opinion among stakeholders interviewed in the process evaluation that educational activities should continue in current communities, and expand to new communities. Evaluators asked all respondents to make general recommendations to ReproSalud, and some were asked about whether and how the project should expand. Expanding coverage (39) and continuing with Phase One activities (39) were the two most frequent responses for general recommendations. Local authorities and CBO members were strongly represented in those recommending continuation of Phase One activities. Out of 84 suggestions about whether and how ReproSalud should expand,39 39 gave responses that indicate support for expanding and wholly or partially replicating the educational activities of Phase One.40 The next most popular response was more work with men and/or more on violence (14) — also a suggestion related to continuing educational activities. While “more training for promoters” was only mentioned seven times in the individual interviews, group meetings of promoters in Ayacucho and Puno strongly emphasized their need for more training. Again, this suggestion is closely linked to the promoters’ educational activities. Although post-tests show high gains in knowledge, several of those interviewed said they needed refresher courses. The subproject budgets for Phase Two give much less emphasis to educational workshops, but all of the subprojects are “integrated,” that is, they include community￾level educational workshops. Probably, there will be many fewer workshops during this coming period, both because the promoters no longer receive financial incentives to do so, and because fewer are planned. During the field visits, which occurred during the interval between Phase One and Phase Two, the sense that project activities were at a low ebb was palpable among both community-level respondents and health providers, and as noted above, the demand for continuing activities was high. Since ReproSalud does not have the resources to continue a full-fledged Phase One Program in the same communities, this evaluation recommends that any demands for educational reinforcement that are not met by Phase Two “integrated” sub grants be met by a new mass communications program. 39 Some people made more than one suggestion. Health authorities, health providers, and ReproSalud promoters were also consulted on this topic. 40 See Tables 6 and 11 in Annex II. The responses that lead to this conclusion in Table 11 fell into 4 types: more education (including suggestions for new topics), more of the same (i.e. Phase One), suggestions of geographical areas into which the program should expand, and replication of the whole current model. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 26 D. COMMENT ON FINDINGS FROM PHASE ONE All categories of respondents in this study clearly value the guiding principles of the ReproSalud project and its model of intervention. Women in the communities involved are grateful to have found their voices, and to know more about reproductive health. Many men – both promoters and local authorities – appreciate the importance of the information and messages in the educational workshops. Even before the service-focused interventions of Phase Two’s advocacy program, ReproSalud seems to have had a significant impact on the strategic objective – on use of reproductive health interventions. Health professionals appreciate the CBO promoters’ bridging role between the health services and these hard-to-reach communities. Local authorities provided essential support to the CBOs – a support that the project hopes to increase as it enters this next phase and aims for sustainability once the project has ended. Thinking of health services and the communities they serve as two parts of an interdependent system, working toward the same goals in both parts of the system should be more efficient than working in just one part. When aiming to improve the use of services in these hard-to-reach communities, “supply-side” changes in the quality of care often are not enough to achieve higher coverage. High levels of mistrust, lack of knowledge, and fear in these communities inhibit change until culturally affirming educational efforts help overcome these obstacles. In the health services, the nearly universal acceptance of the importance of user feedback and of improving the user/provider relationships seems to have made most providers receptive to more knowledgeable and assertive users. ReproSalud provided a necessary complement to MOH programs in these communities to achieve an important impact on the use of reproductive health interventions. 27 IV. OBSERVATIONS ON PHASE TWO At the time of the data collection, Phase Two activities were just beginning,41 so the findings from the evaluation mostly pertain to facilitating factors and barriers, respondents’ perceptions of the current plans, and to concerns about the model. A. BACKGROUND INFORMATION As is often the case in Peru, at the time of this report the health sector is in a considerable state of flux. This more or less permanent condition — often due to rotation in top-level political appointees — has been aggravated recently by upheavals in the government during the last two years. Most of the departmental authorities are political appointees, not civil service personnel. Ex-president Fujimori’s government had initiated an ambitious process of health sector reform involving decentralization, the devolution of authority to the departmental level, greater self-financing, new insurance schemes, and experiments in community oversight.42 The transitional government after Fujimori lasted only nine months but installed new health authorities in all of the departments. At the time of data collection, in October 2000, Alejandro Toledo’s newly elected government was still in the process of replacing the authorities from the transitional government. Many health authorities that were interviewed had been in their posts for a year or less and expected to be leaving by December 2001. To further complicate the political context within which Phase II will operate, Toledo’s first Health Minister, Dr. Solari,43 merged the reproductive health program in the Ministry of Health with the Maternal-Child Health Program as an efficiency measure. Many respondents feared this was a signal of weak support for family planning. Further evidence suggests that this fear may be justified; a few health professionals and ReproSalud staff reported a shortage of certain contraceptive methods in the health centers and posts. Reducing maternal mortality is a top priority in the health system at this time; priorities of ReproSalud unrelated to maternal mortality or to deaths in general are not priorities for the Ministry. Within this framework of priorities, family planning’s importance derives from its role in reducing maternal mortality. The discourse on this subject among health professionals in this study was surprisingly uniform and has changed radically in the short space of two years. Through 2000, family planning was considered a high priority in and of itself, not simply as a means to preventing maternal mortality. 41 The first subprojects had just arrived at headquarters in December. 42 The CLAS (Comité Local de Administración en Salud) stands for the Local Health Administration Committee, which runs a self-sustaining and autonomous health center. The governing committee includes community representatives appointed by the Director of the Center. The current leaders of the MOH intend to continue and expand these experiments initiated under Fujimori. 43 At time of this writing, Dr. Carboni had replaced Dr. Solari as the Minister. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 28 For many years, the Peruvian health system has strongly emphasized coverage in its system of incentives and disincentives for health personnel. Whether the intervention whose coverage to be increased is measles vaccinations, pap smears, or sterilizations, the system is the same. Targets are set — annually or monthly — for each institution, and providers who do not meet their targets often suffer negative consequences, especially if they are under contracts and are not permanent civil service (nombrado) employees. Providers or institutions that exceed their targets are rewarded in some way—for example, with promotions or granting of budget requests. Since increasing coverage is such a high priority in the Peruvian health system, ReproSalud’s Phase II subprojects – in which CBO promoters help the services to meet their coverage goals -- are extremely attractive to health professionals. On the other hand, the existence of such targets can lead to human rights abuses, as the following discussion on issues related to childbirth will illustrate. B. FINDINGS: POTENTIAL BARRIERS AND FACILITATING FACTORS IN THE IMPLEMENTATION OF PHASE TWO In the cases where some negotiations had already taken place, it was clear that the agreements were structured for mutual benefit, thus laying a sound basis for future collaboration. In subprojects negotiations, CBOs ask MOH to respond as well as it can to the demands from the Defender’s Committees. Most of the current negotiations also aim to have the CBO promoters receive official recognition as MOH “community agents.” For their part, the CBO promoters pledge to refer community women to reproductive health services, to conduct educational activities, and to stimulate participation in general health campaigns, such as measles vaccinations. MOH recognizes that CBO promoters and staff have a level of acceptance in and access to indigenous low-income communities that MOH does not have. Those health professionals who have participated in Phase Two conversations or negotiations appreciated this reciprocal framework, and had mainly favorable attitudes toward ReproSalud’s Phase Two plans. Another factor in favor of Phase Two negotiations is the Peruvian health sector’s growing emphasis on user satisfaction as an essential criterion for evaluating service quality. An essential element of the Phase Two advocacy model is that CBO promoters will relay users’ feedback on quality of care to the health providers. (See previous discussion in section on quality of care.) In the design phase of the advocacy program, ReproSalud staff identified three potential problems in the CBOs that could pose barriers to success. Their main strategy for addressing these problems was providing the initial leadership and advocacy training to the members of defenders’ committees and promoters’ networks. The problems that staff identified are as follows: ß CBO members’ tendency to choose presidents based on political connections that determine how much patronage or goods she can secure for the members. Especially during electoral periods, this tendency politicizes the CBOs, so that they lose their autonomy to a political party. The leadership course deals with V. OTHER RECOMMENDATIONS FOR PHASE TWO 29 questions related to organiza tional autonomy and criteria for choosing leaders. The training aims to raise awareness among members that they should choose presidents who can unite their members around non-partisan initiatives that benefit all community members. ß Most negotiating team members do not understand the MOH system. They need to understand which types of problems can be resolved at which levels so that, for example, they don’t come to the health center with demands that can be resolved only by the DISA. ß Staff spoke of CBOs’ tendency to present long lists of complaints, without clear priorities for MOH staff to respond to, in some of the initial meetings. The advocacy course is meant to prepare the negotiation team to strategically present their demands and the commitments that they can offer in return. Evaluation of the training must focus on whether these potential problems were averted and, if not, what additional inputs are needed. An important barrier to the health sector’s capacity to respond to the demands from the CBOs is the lack of sufficient funding. Most interviewees within the sector felt that the basic limitation on their ability to respond to the demands from the CBOs in current or future negotiations had to do with insufficient financing.44 Twenty-five out of 35 responses pointed to financial limitations or other issues stemming from financing: limited supply of personnel, medications and contraceptives; and lack of ability to do community outreach. Issues to be Negotiated: Shared and Unshared Priorities The other significant barrier to the Phase Two negotiations has to do with the areas in which MOH priorities and the CBO priorities do not overlap. The primary message from health authorities is that their priority is maternal mortality, above all else. Fortunately, some priorities expressed by the CBOs, such as family planning (“too many children”), Quechua-speaking providers, hospital attention for obstetrical emergencies, and culturally appropriate attention for childbirth, are accepted by MOH explicitly as a means to this end. Prevention and treatment of cervical cancer and life-threatening STIs (sexually transmitted infections), such as syphilis, are a shared priority because they are a cause of infant and/or maternal mortality. However, other issues prioritized by the CBOs and MOH personnel reveal partially shared priorities and significant areas of potential conflict. (See “Do priorities coincide?” Annex II, table 10 and graph on following page). Most MOH facilities could not accept the full range of demands related to the CBO members’ preferences for childbirth. Most respondents pointed out that treatment of RTIs is not a shared priority. MOH respondents said that they are not willing to place high priority on treating RTIs because they are not life threatening, and the medicines are expensive. The data from the ReproSalud 44 See Annex II, Table Titled, MOH Limitations and Financial Issues. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 30 autodiagnósticos (Yon, 2000) clearly show that women in Andean highland communities do not agree. RTI symptoms affect almost every aspect of their lives, including their relationship with their spouse, and in some cases is associated with conditions leading to death, such as the association of bacterial vaginosis with complications of pregnancy and childbirth. Women in the ReproSalud communities place a high priority on receiving treatment. The following chart reveals the main areas of shared and unshared priorities expressed by health authorities, health providers, and promoters. · ____ _ _ ____ Boxes on the left = MOH priorities not shared by ReproSalud. · __ __ __ __ __ __ Boxes in the middle = shared priorities. · ________ Boxes on the right = ReproSalud priorities not shared by MOH. · : - - - - - - - - - - Dotted box = partially shared category is green “culturally appropriate institutional births.” MOH PRIORITIES Many other maternal￾child health and primary health care issues, e.g. easles vaccination campaign at time of study. REPROSALUD/CBO PRIORITIES SHARED PRIORITIES MATERNAL MORTALITY Professionals attend home births Culturally appropriate institutional births Treatment of RTIs & STIs * (not life threat￾ening) Treatment of life￾threatening STIs (e.g. syphilis) and cervical cancer Full coverage for pre-natal care Access to family planning methods Free or reduced cost treatment and medi￾cines for cervical cancer, STIs, & RTIs Achieving 80% coverage in institutional childbirth V. OTHER RECOMMENDATIONS FOR PHASE TWO 31 Low-Cost Treatments and Medicines Given the severe financial constraints faced by the public health sector, this is a difficult negotiating point, yet it is one of the main demands of the CBOs. In general, health professionals said that they could easily waive the cost of the medical visit, but that their budget for supplies and medicines rarely allows them to provide these at no cost. There is evidence that the fees charged by health institutions vary widely, even within departments; research into the official guidelines on fees might give the CBOs negotiating leverage on this point. In some settings, institutions charge users for the supplies used during childbirth because they ran out of stock, and had no budget to buy more. Any negotiating points that involve non-budgeted expenses might need to be resolved at the departmental level, and not at the level of the health center or health post; however, the providers at the more local levels can serve as key strategic allies for these demands. Issues related to Childbirth There are clear signals that the same system of provider incentives and disincentives that got the Peruvian health system into trouble during the sterilization campaigns is now being applied vigorously to the goal of increasing the percentage of births in institutions and increasing compliance with prenatal care visits. Because the providers feel pressure, they in turn apply pressure to users in order to meet their coverage targets. In some districts, women who have home births are “fined” by having to pay higher-than-usual fee for birth certificates. In Puno, more than one anecdote surfaced of women’s voter cards being retained at the health center at the time of their first prenatal visit, to ensure that they completed their required number of controls. After detecting this problem in the first two departments visited, the evaluation team asked health authorities and providers in three departments whether they had targets (metas) for institutional childbirth. Out of 29 responses, 24 said that there were targets, but only nine said that they might suffer some negative consequences if they did not meet their targets.45 These negative consequences include receiving an unfavorable evaluation, being overlooked for promotions, being moved laterally, and even being dismissed from the post. There is evidence that the target of institutional childbirth is inconsistently applied. In some districts, authorities have made the goal more flexible — that is, to increase the number of births attended by a health professional, without specifying where the birth takes place. In 4 out of the 15 districts visited by the team, providers go to homes to attend births. However, other health providers voiced opposition to this idea, saying that the distances and communications difficulties meant that the woman might die before they even reach her. Others pointed out that their budget for transportation is very limited, not sufficient to regularly travel to isolated rural areas. In summary, the CBOs that give high priority to having home births attended by health professionals will probably meet with much resistance. 45 Due to social acceptability bias, the number who feared negative consequences was probably higher. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 32 One chance remark by a security guard in Ayacucho to an evaluation team member illuminates the pressure that health professionals are under to reduce maternal mortality, and how this pressure can undermine the goal. He said that a midwife at the health center had just been fired because a maternal death occurred on her watch earlier that week; other informants confirmed this practice. Such pressure leads to a high level of anger at users who arrive in critical condition at maternity hospitals, and probably to their mistreatment and even denial of access. The team heard one anecdote of a woman in labor in Puno being turned away when she arrived in critical condition, under the excuse that she could not pay. (Maternal insurance provides free childbirth services.) The provider in charge thus avoided losing his/her job. Several providers in the interview mentioned that one of the benefits of the CBO promoters’ work is that women are arriving in a more timely fashion to the health services, before they are in critical condition. The subtext to these remarks is that this improvement in how women use services does not just save the woman’s life; it also saves the provider’s job. The positive side of the undue pressure on health personnel to meet their targets for institutional childbirth is that they are finally willing to experiment with adaptations to their services that will make them more acceptable to Andean women. Eighteen of the 40 respondents reported a wide variety of adaptations to their services46 — from full-scale traditional birthing rooms to less resource-intensive concessions, such as allowing the presence of the spouse and the traditional herbal beverages used by women in labor. The ability to give birth in a squatting position is a frequent demand from the women; some providers were not willing to concede on this point because they perceived this practice as either more inconvenient or more risky. In fact, the health institutions tended to concede only those points that were most acceptable to Western-trained personnel and/or didn’t involve significant outlays of funds. CBOs should be able to gain further concessions in the negotiation process. Interviews revealed other possible points for negotiation. Where services are charging for childbirth, because of the maternal-health insurance scheme, a demand to make these services free should yield results. In cases where services levy extra fines for home childbirth and use other coercive measures to increase prenatal and childbirth coverage, these are clear violations of users’ rights, and the Defender’s Committees should bring these cases to the attention of the DISA, and the Defensoria del Pueblo. 47 Finally, some promoters and health post “technicians” (equivalent to male and female nurses’ aides) expressed a desire to be trained as lay midwives, in order to increase access to trained assistance during labor for rural women in remote areas.48 Other common measures to increase access to emergency obstetrical care could be explored. 46 Refer to table on page 47 An ombudsperson’s office that investigates human rights violations by public agencies. 48 This demand might be controversial in Peru, but attendance by trained lay midwives is better than nothing in instances where professional assistance is not available. V. OTHER RECOMMENDATIONS FOR PHASE TWO 33 Cervical cancer, RTIs and STIs49 Team members wondered why, after two or more years of education on RTIs, more than 75% of the women still named RTIs as one of their top health problems. The answer turned out to be complex. First, as the tables above show, the diagnosis and treatment of RTIs and STIs is not a high priority for MOH. When women come for Pap smears, reading the samples for infections incurs additional costs, and most providers probably diagnose RTIs symptomatically. Theoretically, treatment is available through MOH’s PROCETTS program, which is the arm in charge of preventing and treating HIV/AIDS and other STIs, but in fact, STI treatments are only available free for commercial sex workers, a “high risk group” for contracting HIV. In addition, none of the five departments reported that these medicines are available in the public sector. The cost of the medicines in pharmacies is prohibitive for most families from these communities. In the negotiations that had taken place at time of the study, free or low-cost medicines to cure RTIs were one of the subprojects’ most frequent demands.50 Another barrier is that both women and their spouses in highland communities are particularly resistant to any medical procedure that involves a pelvic exam for the woman. This resistance increases greatly if the provider is a man, as is the case in a large number of rural health posts. ReproSalud educational efforts have worked hard on overcoming this resistance, and persuading both women and men that for a variety of situations — prenatal care, childbirth, Pap smears, diagnosis of RTIs and STIs — a pelvic exam is necessary. Evidence from all three components of this evaluation shows that both ReproSalud and MOH efforts have succeeded in increasing the use of childbirth and pre￾natal services, but even after these gains, the usage is still much lower than ideal. The situation concerning cervical cancer prevention is also complex. In recent years, MOH has put a high priority on Pap smear campaigns, in response to high cervical cancer rates in Peru. Out of 40 CBO members responding to the question, 32 had gone for their Pap test, many of them for the first time. However, some interviews indicated that promoters have to negotiate Pap tests for large groups of CBO members ahead of time. Furthermore, logistical failures in the health system mean that in some departments or districts, those tested do not reliably receive their results.51 In cases where cancer is detected, some of the CBO promoters have made heroic efforts to help a CBO member receive treatment, which often involves negotiating a reduction or waiver of fees for surgery and medicines. (See Appendix III on Achievements.) Respondents knew that for prevention and treatment of sexually transmitted RTIs, both members of the couple must be involved. However, ReproSalud’s inclusion of men in the subprojects was not universal, and in communities where there were workshops for men, 49 For excellent technical overview of both RTIs and cervical cancer, see http://rho.org/and http://www.who.int/HIV_AIDS/figures/global_report.html - Chlamydial_facts. 50 A personal communication from Carmen Yon, March 2002, suggests that providers can be persuaded to put a higher priority on helping women solve this problem. Initial negotiations from some districts in La Libertad and Ayacucho led to access to medicines for RTIs. 51 In this small sample, only 3 out of the 40 CBO members did not receive their results, but we heard about other cases from promoters. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 34 not all of them attended. All of the socio-economic and gender factors noted in the social science literature on the HIV/AIDS epidemic apply to sexually transmitted RTIs in these communities, including forced sex, multiple partners for men, and women’s inability to negotiate condom use with their partners or to persuade their partner to be treated. It is evident that there is a wide span of solutions to this problem: self-care practices, medical treatment, and advocacy with the health sector, as well as more complex interventions to address social and cultural factors that contribute to the disease. The solutions differ depending on the type of infection. C. RECOMMENDATIONS FOR ADVOCACY AND HEALTH PROMOTION FOR RTIS 1. Focus on prevention and treatment of easily diagnosed and cured RTIs and STIs, such as yeast infections, syphilis and trichomoniasis, at the local level. ReproSalud should concentrate on reinforcing prevention information, such as hygienic practices and condom use, and work to overcome men’s resistance to being treated when necessary. Secure greater access to medicines either through advocacy with the health services or through group fund-raising activities such as raffles. 2. Advocate for testing and treatment for less easily diagnosed RTIs in cases of pelvic inflammatory disease (PID) and infertility, when other conditions have been ruled out. Chlamydia and gonorrhea are not as easily diagnosed, and often go undetected until women seek treatment for infertility or PID. 3. Advocate at the departmental or national level to change the MOH policy of providing reduced-cost or free treatment for STIs only to sex workers. 4. Educational efforts in Phase II on gender issues and RTIs must intensify, through both community workshops and a mass communications program, with outreach to women, men and youth. Education should address traditional gendered sexual norms such as sexual coercion and multiple partners, as well as condom use and treatment of both men and women for STIs. 5. Make a policy brief of bio-medical information on the different RTIs and STIs that are related to vaginal discharge, including which conditions could lead to mortality and under which circumstances. This information can be used for negotiations with health authorities or providers who do not agree to CBO demands on accessibility of treatment. V. OTHER RECOMMENDATIONS FOR PHASE TWO 35 D. ESSENTIAL COMMUNICATIONS FLOWS IN ADVOCACY PROGRAM, PHASE TWO Subproject steering committee (when more than one zone) Steering Committee of the Defenders’ Committee (directiva) Steering Committee of the Promoters’ Network Promoters’ Network CBO Presidents (15- 20 CBOs in each subproject) CBO Promoters (from 15 to 20 CBOs in each subproject) Local authorities CBO Members & spouses Other community women & spouses Defenders’ Committee Community representative Health Center or Hospital MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 36 Key Assumptions and Evaluation Questions in the Design of Advocacy Program ß These assumptions are what designers assume will happen, and what needs to happen for the model to function. They need to be evaluated during the implementation of Phase Two to ascertain why the project did or did not achieve its goals. ß Health authorities will deal respectfully with the CBO leaders and will take all feasible and necessary actions to respond to women’s demands. ß The negotiations between the CBOs and the providers will lead to changes in the health services that improve their quality and make them more acceptable. ß The budgetary constraints in the health sector will not form an insuperable barrier to its ability to address the CBOs’ demands, or to handling increases in numbers of users. ß Differences in the priorities set by the CBO leaders in the defender’s committees and promoter networks and MOH priorities will be dealt with constructively in the negotiation process, so that agreements on shared goals can be forged and maintained. ß Negotiated agreements between the CBO leaders and the health authorities will be perceived as beneficial and fair by their respective constituencies – the CBO promoters and members, and the health care providers in the health center’s network. ß The flow of information up and down the levels shown on page 40 will result in demands that reflect community members’ needs, and in awareness at the community level of the status of negotiations. ß This two-way flow of information will occur throughout the subproject period, not just at the time of negotiations, so that leaders are aware of users’ perceptions regarding improvement or lack of improvement in the services, and also communicate news on improvements or lack thereof to the community. ß Promoters will continue to actively encourage women to use reproductive health interventions and services. ß Community women will then increase their use of reproductive health interventions and services, and as a result, community-wide reproductive health indicators will improve. V. OTHER RECOMMENDATIONS FOR PHASE TWO 37 ß The promoters will carry out the responsibilities assumed by the CBOs in the negotiations process. They will form a relationship with 15-20 families to promote their health, and will give educational workshops to the wider community. ß Promoters’ responsibilities will give them personal satisfaction, community recognition, and incentives from MOH. These will provide sufficient motivation to carry out these activities. ß The knowledge gains, attitude changes and behavioral changes promoted during Phase One will be maintained and expanded within the current ReproSalud communities and in neighboring communities by the community workshops in Phase Two, and by the Communications program. E. CONCERNS RAISED BY THE PHASE TWO ADVOCACY MODEL AND RECOMMENDATIONS ß The project has designed a complex structure for communications flows. Communications must travel up and down through several levels from the base to the subproject steering committee if the model is to work as planned. The structure is reminiscent of a political party, which is not surprising, given that this is a constituency-based advocacy project. Communication can break down at any level, in either direction. Recommendations: Pay close attention to communication flows in supervision and evaluation. Solicit ideas for how to build these communications into routine interactions and regular subproject events. ß The advocacy and health promotion zonal and district structure may not be sustainable. The ReproSalud budget and the subproject budgets foot the cost of meetings and supervision. Recommendation: Evaluate possible paths to sustainability for this structure, and lay the basis for this during Phase Two. ß In the advocacy activities, the CBO Presidents and the CBO promoters are the protagonists, and the “action” is no longer at the community level. Only five of the CBO Presidents and five promoters are actually negotiating with Health Centers. All others are not directly involved in the advocacy, except during the network or defenders’ committee meetings, and ReproSalud will be much less visible at the community level. During this phase, there are apt to be fewer community workshops; the promoters’ main activity is going house to house to the families that they are working with. Recommendation: Involve CBO promoters and members in the communications strategy. ß The CBO promoters might lack sufficient motivation to be actively engaged. In most cases, their commitment will involve actively encouraging women to use the health services, and serving the “bridging” role that MOH respondents commented on so favorably. In many cases, they will still give community workshops, but they might not be sufficiently motivated to do so without financial MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 38 incentives. Recommendation: Monitor level of promoter involvement closely, and devise low-cost motivation strategies if necessary. ß ReproSalud staff pointed out the dangers of relying on the CBO Presidents for the Defenders Network. They rotate frequently, so that several of the current presidents have had little exposure to the ReproSalud project. As noted above, often they are chosen not for the ir leadership qualities but for their ability to secure resources for the members, often through political connections. Close monitoring will be important in determining whether the leadership course helps members use different criteria and whether CBO Presidents are motivated and informed enough to carry out their expected role. Recommendation: Devise a back-up plan for cases in which the CBO President is not participating in the Defenders’ Committees. ß Male promoters have been marginalized, and there is no mechanism to re-involve them in community-level activities. This marginalization of male promoters would make it hard to make further progress on gender issues, and might cost the project some of its current support among local authorities. Recommendation: The negotiated agreements should include the male promoters as well as the female promoters as MOH community agents. The male promoters should be involved in the educational activities in Phase Two, and in the communications project. ß Multisectoral Committees do not usually include CBOs as participants. Recommendation: If participation in one of these multisectoral committees seems worthwhile, legitimizing the CBOs so that they get a place at the table is an important goal, since these venues tend to be highly professionalized. 39 V. OTHER RECOMMENDATIONS FOR PHASE TWO Since ReproSalud is a ten-year project with a beginning and an end, the central strategic challenge is how to maximize the benefits of the project during its lifetime, and then make them sustainable over time. In business language — how to obtain the greatest current and long-term yield from this investment? The discussions with staff and other stakeholders during this evaluation transformed this challenge into two major issues: ß How to maximize and sustain positive community-level changes in the districts currently involved in ReproSalud? ß How best to expand coverage by replicating the project in new communities and districts. ReproSalud has only a limited budget for activities implemented directly by ReproSalud to address these two issues during the next four years, and a sizeable proportion of the budget is devoted to the Advocacy Program. This evaluation proposes two centerpieces for achieving these goals: ß A mass communications program centered on video and radio production and dissemination, and ß Active encouragement of replication by NGOs and other institutions. A. MAXIMIZING AND SUSTAINING IMPACT IN CURRENT DISTRICTS The process and impact evaluations identified the following related needs in communities involved in ReproSalud that the strategies in Phase Two must address: 1. The need for refresher training for CBO promoters. 2. The need to maximize and sustain health-protective knowledge, attitudes and behavior among CBO members and community women and men. 3. The need to reinforce the sustainability of the project’s impact on the culture of the community, by reaching more men and more youth of both sexes (ages 15- 24).52 52 Young women in this age group are under-represented in the membership of the CBOs. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 40 4. The need to increase the project’s impact on gender equity, with a special focus on violence against women, women’s reproductive rights, shared decision-making and communications on sexual and reproductive matters, and valuing and sharing housework. 5. The need to engage community-level participants in Phase One (female CBO members, male promoters, and youth promoters) with the program. They play a passive or non-existent role in the advocacy program. The changes ReproSalud promotes could be more easily subverted if the project becomes less visible at the community level. Discussion & Recommendations The original designers of ReproSalud expected the main sustainable effects to be positive changes in health-protective knowledge, attitudes, and behaviors at the community level, once a “critical ma ss” of community members was affected by the project.53 Therefore, a crucial question related to the sustainability of the effects of the project is how impact spreads through the CBO’s social network, whether the gains for the CBO members and their social network – especially with regard to any changes in gender relations – will be eroded over time if they are not further reinforced during Phase II. The evaluation during Phase II should make a first attempt to answer these questions. This study provides some anecdotal evidence that female participants spoke to and influenced family members, including husbands, daughters and sons. Women who were illiterate shared the educational materials with their sons and daughters, asking them to read it out loud. In fact, when ReproSalud staff asked the women whether they wanted the educational materials in their native languages or in Spanish, the women requested that the materials be in Spanish, so that their children could read them. With regard to men, however, several respondents indicated that ReproSalud had a positive impact only on those who actually attended the educational workshops. One Ancash promoter said that men listen more to outsiders than to the women in their communities. One hundred percent of the community-level respondents (n=72) who were asked whether work with men should continue said “yes,” citing a variety of reasons that had to do with insufficient impact of the project on men in their communities. These observations suggest that the expected diffusion of health-protective changes in the social network of participants might not take place among men. The ethnographic study should examine this question. While several respondents recommended more work with youth, other efforts at convening stable groups of youth in the rural highlands have had great difficulties due to constant mobility in this age group.54 Given the high cost and difficulty of convening 53 Interview, Susan Brems, and Project Paper. 54 Private communication from Carmen Yon, referring to an experience of the PRIME project in Huancavelica. V. OTHER RECOMMENDATIONS FOR PHASE TWO 41 stable groups of youth, the communications program should be the main vehicle for education of rural and periurban youth in Phase Two.55 ReproSalud’s educational activities and communications program in Phase Two should emphasize several indicators for which ReproSalud communities enjoyed significant gains, but the actual percentage is still much less than ideal. ß Women whose last childbirth was attended by a health professional = 48%. ß Women who know the danger signs for childbirth and post-natal period = 46%. ß Users of rhythm who know their fertile days = 28.3%56 ß Women who know how at least one modern contraceptive method functions = 55.5%. ß Women who know how vaginal discharge is contracted = 19% ß Women in union with unmet need for family planning = 37.3% ß Women who decide how they will spend the money they earn =36.1% ß Women who decide jointly with their spouses on sexual relations, contraception, and number of children = 32.9%. ß Women who had spoken more than twice to spouses about family planning = 20 percent, and to children over 12 about this and relations in a couple = 16%. ß Women who would go to the authorities if beaten = 38.4%. ß Women who agree that women should use contraception even when her partner is opposed = 64.4%.57 This value fell significantly in both ReproSalud and comparison communities. ß Men (47.4%) and women (44.1%) who give equal value to women’s work in the home. ReproSalud should implement a radio- and video-based communications program within the current districts. This program would help meet many of the needs named above. Furthermore, radio has the advantage of helping the program’s messages saturate a whole district, since ReproSalud’s coverage per district is quite low, varying from a high of 55 Some communities may have decided to give priority to working with youth in their subproject, but there is no protram-wide focus on youth in the current plans for Phase Two. 56 See Annex III. On fertile days. All women and men, not just rhythm users, should know these facts. 57 This is an idea that contravenes the culture of these communities, according to the baseline of the ethnographic study. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 42 17.6% in Ancash to a low of 9% in Puno Aymara.58 Radio can also reach very remote communities in which the program ha s not intervened. Videos fulfill a different function, mainly as an educational tool for stimulating group discussions. Having access to these videos– and in some instances helping to create them -- would serve as a powerful attraction to the educational workshops to be given by the CBO promoters during this phase. ß The main objective of the communications program would be to maximize and sustain health-protective knowledge, attitude and behavior changes to achieve greater gender equity and reproductive health in ReproSalud communities. ß One short-term objective should be to produce a minimum of three radio shows and videos (one each for women, men, and youth) for each of the five modules in the ReproSalud educational program, of high enough quality to be given or sold to those replicating the program. ß The program should set other short-term objectives for numbers of educational workshops that use the videos and radio shows in each community. Proposed elements of the communications program (Some of these elements modify Gamucio’s proposal.) 58 See Graph 2 of Impact Study. Communications Program, as proposed by Alfonso Gamucio 1. Conduct a survey to determine the communications resources in the districts involved in ReproSalud, and select priority districts based on this information. 2. Use a selection process (possibly a competition) to select and then train promoters who will receive intensive training in communications. 3. Conduct and evaluate a pilot radio program series on reproductive health, involving these trained promoters, the regional ReproSalud offices, a feminist NGO specializing in radio, and community radios in the zone. Create a “Manuela Ramos” who listens to problems from local women (recruited from those involved in Phase One), offers solutions, and gives referral information to local CBOs and the health sector. 4. Conduct a community video project with the trained promoters in charge. Give them additional training in video. Start in a small number of pilot zones first. Format of 20-25-minute interviews with rural women recounting their experiences, their problems, and solutions found through ReproSalud program. V. OTHER RECOMMENDATIONS FOR PHASE TWO 43 ß Educational radio programs along the lines suggested by Gamucio, but with four personages answering questions: a woman (Manuela Ramos), a man (Manuel?), and a couple of male and female youth, in order to address the perspectives and health needs of men, women, and youth separately. In some programs, the woman and man would appear together in dialogue, and in others they could appear separately, addressing either a male or female audience. ß In addition to the formats for the radio shows and videos proposed by Gamucio, add the alternative of radio and video socio-dramas. ReproSalud could repeat the highly successful strategy of “sociodrama” (skit) competitions among CBOs, but this time for radio shows and videos on specific topics, and when appropriate, targeting women, men, and youth separately. These should be competitions for “best script,” either in writing or on tape, in Spanish, Quechua, or Aymara. The competitions would help remobilize the CBO membership, male promoters, female CBO promoters, and youth promoters. ß Develop promotional radio spots immediately before or after the programs that would invite new communities to solicit an educational program from the CBO or other institutions replicating the ReproSalud project in the same district. ReproSalud would need to develop a mechanism for clearing and referring such requests, probably through the health center. ß This new project should begin simultaneously in all five departments, without a pilot phase. The promoters’ networks should be involved in designing the strategy for getting the project going. In order to give promoters from all 78 subprojects the chance to participate in the competitions, communications training will take place at the departmental level for one promoter chosen by competition from each subproject. The prolonged video and communications training envisioned by Gamucio should be shortened, with each trainees attending only one workshop. ß A survey of communications outlets is not necessary; the CBOs are capable of doing this with little effort. The radio programs should broadcast on local stations that are most listened to in the current communities involved in ReproSalud and in other new communities nearby. ß A communications expert should carry out the program. ReproSalud already plans to hire such a person to oversee, train, and design the evaluation of this component. ß Widespread dissemination among all community members of the information leaflets given to those attending workshops in Phase One This is already in the plans for Phase Two. ReproSalud should involve trained male promoters to reach their peers Male involvement is key to creating a sustainable change in the culture of the community regarding reproductive health and gender issues. Furthermore, data from the process MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 44 evaluation suggests that the support from local authorities is stronger when they are involved in educational activities for men. ReproSalud can involve male promoters in the following ways: ß Recruit them to help deve lop products and messages for the communications program, as mentioned above. ß Invite them to any education and training activities for promoters taking place in Phase Two. ß Negotiate recognition for the male promoters as MOH community agents. B. EXPANDING COVERAGE THROUGH REPLICATION The benefits and impact of Phase One —demonstrated by all three components of this evaluation — provide a sound foundation for recommending that ReproSalud’s educational model be wholly or partially replicated in new communities, with the focus on gender issues strengthened, and with other recommended modifications in the model. Replicability is a daunting challenge in project models like ReproSalud that are highly participatory and complex, and that operate in communities that are “hard to reach” both culturally and geographically. ReproSalud’s substantial investment in investigating key reproductive health and gender problems, and in producing culturally appropriate training manuals and educational materials, make replication feasible in other such communities in the Andean jungle and highlands areas, both in Peru and in other Andean countries. The methodology and tools could be adapted for other settings outside this region. To maximize the benefits from the investment in ReproSalud, we recommend that project managers give highest priority to encouraging replication through other NGOs, educational programs, and international agencies during the next four years. The following table shows the plans that were in effect or under consideration at the time of the evaluation. DEPARTMENT REPLICATING AGENCY & coverage DETAILS La Libertad MOH, and numerous district-level requests. MOH received guides and has requested training in their use. ReproSalud plans to provide training Ancash Various agencies and institutes have requested training, including MOH, CARE, CEDEP and other NGOs outside of current districts. No concrete plans yet. One NGO expressed strong interest in receiving training, but did not receive a positive response from ReproSalud Ayacucho MOH, CEDAP, VECINOS PERU, Salud sin Limites ReproSalud office plans to respond to all of these requests during Phase Two Huancavelica NGO Anccara - 270 promoters and 15 ReproSalud office plans to respond to V. OTHER RECOMMENDATIONS FOR PHASE TWO 45 specialists in literacy program, Nursing School in University, MOH in districts where ReproSalud is absent. all the requests. Puno GTZ – trained staff reach 1000 women Youth program staff – reached 500 youth Promujer, a credit program with 6,000 beneficiaries Instituto Superior Ayaviri – trained staff reach 500 beneficiaries Agreements already in operation Ucayali MOH – 70 staff trained San Martín MOH – 60 staff trained The original designers did not envision having MOH or the other major reproductive health NGOs replicate the project activities. The context has changed since 1994-95, however, and the potential for collaboration with the health sector and replication by other NGOs is much greater than it was at that time. There are three basic recommendations to ReproSalud to enhance replication opportunities: 1. Develop diagnostic tools to determine whether replicating agencies need reinforcement in some or all of the underlying principles of the project, and identify additional training tools and curricula for filling these gaps. “Transfer of methodology” is a misnomer; true replication would be transfer of guiding principles and methodology. An important challenge in replication efforts is to assist other organizations to incorporate the guiding principles of the project in the replication. All imply comprehensive cultural changes in some organizations. 2. Develop and provide a full kit of all the tools, manuals, and materials developed for Phase One. During the next year, MMR should gather all final comments on basic tools and resources for replicating the project and collect them into one set of materials. ReproSalud has invested a lot in creating these well-tested tools that are culturally appropriate for both men and women in the Andean highland and jungle areas. There are now final versions of guidelines and manuals for each step of Phase One: district needs assessment, CBO selection process, a booklet describing how to run the competition, a detailed guide for running the autodiagnóstico, a guide for subproject design, guidelines for implementation, guidelines for running every type of meeting in both Phases, and a guide for subproject evaluation. All final evaluation instruments should be part of the kit, so that some replications might generate additional data to complement those collected by ReproSalud. 3. Increase outreach to other NGOs, educational institutions, rural and women’s development programs, and international agencies working in Peru and in the Andean MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 46 region to publicize the availability of training for replication. To facilitate this replication, ReproSalud should: ß Develop communications products to lessen resistance to the project’s educational messages and create demand for the project in new communities. Promotion of the ReproSalud project and messages through one-to-two-minute radio “spots” that are either stand-alone or placed before or after longer educational programs. ß Develop an organized training program. Tasks would include a developing a mailing list of likely agencies, a brochure advertising the availability of training in the model, and a training schedule both for Lima and for each department. 4. Encourage, but give lower priority to, replications by MOH The main argument in favor of relying on MOH to replicate the project both during Phase II and in the long term is that “MOH will always be there,” while NGOs come and go depending on their funding. This argument of permanence needs to be qualified, since MOH experiences a continual rotation of staff, especially at the higher levels. This causes major disruption and even many respondents within the health sector recognize this as a key organizational weakness. Additional barriers to relying on MOH for replication are as follows: 1. MOH personnel’s low level of commitment and unwillingness to undergo hardship to conduct activities in far-flung rural communities. The level of commitment demonstrated by the ReproSalud regional staff exists only in exceptional cases among MOH personnel, who are low-paid and rarely willing to spend the night in rural communities (as the regional staff sometimes had to do during the autodiagnósticos and the CBO promoter training). 2. Health post staff cannot be absent from the post for long periods of time; therefore, any courses they gave would have to be within the health post itself, limiting their ability to reach surrounding communities. 3. According to most informants, most MOH staff would need much more intensive training in the guiding principles mentioned above: gender issues, interculturality, participatory approaches, etc. The current training modules might not be intensive enough on these issues to meet their needs. 4. Most MOH professional staff does not speak Quechua or Aymara. 5. The current MOH community agents are mostly male and have not been trained in reproductive health. In spite of these barriers, in all of the departments except Puno, MOH has been trained or has requested training. Furthermore, MOH staff could give the ReproSalud educational V. OTHER RECOMMENDATIONS FOR PHASE TWO 47 workshops in communities that are too distant for the CBO promoters. Therefore, this evaluation recommends that ReproSalud should: ß Respond positively to all MOH requests to replicate the project through the subprojects, or through agreements at the departmental level. In theory, this is already happening in Ucayali and San Martín. In order to better understand and overcome the barriers mentioned above, it would be important to evaluate the experience in Ucayali and San Martín to understand how to structure training for MOH personnel in other departments. ß Evaluate whether or not it would be beneficial to involve the MOH community agents who have not been trained by ReproSalud. Often, these agents are the only MOH-trained individuals present in communities that a) are distant from health posts, and b) have not participated in ReproSalud’s Phase One. 5. Recommendations for subproject-funded replications in nearby communities: Replication by the current CBO promoters is both possible and included in the plans for some subprojects. However, in most cases CBO promoters can only travel to nearby communities, and face significant barriers to doing so. ß Involve trained male promoters as well as the female CBO promoters in replications supported through the subprojects. ReproSalud should contact the local authorities and CBOs in these communities to see whether they would also like to have educational workshops for men. When the answer is “yes,” send trained male promoters to carry out workshops for men in these communities. ß ReproSalud staff needs to evaluate whether the Phase II subprojects sufficiently reimburse the CBO promoters for their opportunity costs in outreach to new communities; if not, the planned visits might not take place. Opportunity costs might include time a way from productive activities and household chores, need to arrange childcare, and for women, opposition from spouses for her travel outside the community. ß Provide refresher training for CBO promoters in reproductive health and gender issues, through the communications program, MOH-led training, and by having them attend training of MOH personnel. Both the female and male promoters should be involved in this training. ß Help CBO promoters obtain the often-invaluable support of local authorities. Local governors/mayors can provide transportation, provide locales, help to convene community members, lend legitimacy to the project, and reduce resistance from men. They also often provide important additional support, such as food and snacks, sound equipment, and lighting. If the local authorities in new communities could provide transportation and some reimbursement, the opportunity costs for CBO promoters would be less. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 48 6. Suggested modifications in the model of the Phase One educational program: ß To introduce male and female educational components simultaneously. The resistance to the project in its early stages might have been lessened if both men and women were involved. In addition, this modification might facilitate support from local authorities. Discussion: ReproSalud staff believes that the women should be involved first and that men should be involved only at the women’s request. Proceeding otherwise, staff contends, would disempower women and take the project out of their hands. However, this evaluation recommends otherwise. Interviews with local authorities and CBO members and promoters repeatedly refer to the importance of involving men, in many cases so that women can get permission to attend the ReproSalud meetings. These comments lead one to wonder whether women with the least autonomy in their marital relationships may have been under-represented in the autodiagnóstico and in the CBO membership. In replications, involving men from the beginning might also avoid the resistance and hostility from male community members (some of them husbands of CBO members) that many CBO promoters reported in the early stages of Phase One. Many communities would prefer to educate men and women separately on personal topics such as sexuality and reproduction. Simultaneous but separate involvement of men and women in autodiagnósticos and educational workshops should not undermine women’s participation or inhibit their reflections in the group. After workshops expose both sexes to opportunities for reflection and clear messages that promote gender equity, it would be beneficial to organize some events for both sexes that allow dialogue. Intensifying the intervention with men should enhance efforts to promote gender equity and women’s empowerment, make these efforts more sustainable, and help garner the support of mostly-male local authorities. ß Strengthen the focus on gender issues. The results of the quantitative survey show uneven gains on gender issues. Gender-related attitudes and behaviors are very deeply rooted, and there is not as much support in the community and in the health sector for changing them as there is for changing ideas and practices directly related to reproductive health. For this reason, those replicating this project must plan for a longer-term effort targeted at both sexes to promote gender equity. Replications could achieve better results on gender issues in three ways: ß Use of radio programs and videos on gender issues, which ReproSalud will develop in the communications program in Phase Two. ß Identify complementary educational materials and exercises, developed by other projects, to use in the replications. ß Consider various strategies for overcoming men’s resistance to participating in a ReproSalud replication. The project could work through a CBO that involves men, such as a sports club or agricultural committee, so that there is an V. OTHER RECOMMENDATIONS FOR PHASE TWO 49 organization to give legitimacy and sustainability to the effort and provide a built￾in audience for the educational program. In the absence of such a CBO, the most recommended strategy for convening men is to involve local authorities in convening the meetings and going house to house to talk to men about the project. ß To train health providers in responsiveness to community feedback and quality of care in contexts where no complementary project exists to do so. ReproSalud communities also benefited from complementary MOH-led programs that helped achieve the objective of increased use of services. These programs improved access through insurance, and improved quality partly by emphasizing user feedback and provider/user relations. Programs in other contexts may need to provide this complementary and necessary input simultaneously with community education. ß To weigh opportunity costs to the community participants vs. the sustainability of the project in setting policy on financial incentives and non-cash benefits for promoters. In Phase Two, as part of the negotiated agreements, the CBO promoters will receive the non-cash incentives offered by MOH and other intangible benefits such as legitimation of their role. The level of commitment of these promoters under these conditions will demonstrate whether the withdrawal of the financial incentives to promoters provided by ReproSalud leads to high dropout rates of promoters, as some respondents predict. In this study, all of the promoters except some of the men said that they were willing to keep on working as MOH community agents.59 ß Provide non-cash incentives to attend community activities, especially food. 60 In highland communities, where food is scarce and there is a fair amount of malnutrition, the provision of the “refrigerio” (refreshments served at a meeting or workshop)61 assumes a higher importance for the participants than most city￾based project managers would understand. CBO promoters’ and members’ lamented the loss of the funds for the refrigerio in the period between the end of Phase One and the beginning of Phase Two. ß Consider whether to implement the entire program or just some topics. The full program includes four modules, with five 3-4 hour sessions in each module. In some of the Phase One communities, these sessions had to take place on Saturday, so the full program spanned 20 weeks. Others were able to take advantage of the slow season in agricultural communities, and hold an intensive program over a much shorter period. For maximum benefit, a replication should include all five modules, and possibly some additional topics. Given the weak areas identified in 59 See Table 8 in Annex II. 60 This is not a modification in the model, but respondents’ frequent mention of this issue highlighted its importance. 61 The refrigerio seemed to vary from site to site. It was often as simple as crackers and soda, though we heard some anecdotes of people cooking for the refrigerio. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 50 the evaluation, it is probably not advisable to shorten the training. However, many programs have to pick only high priority topics, because of limited budgets. C. FINAL THOUGHT ON SUSTAINABILITY OF IMPACT This section has outlined many suggestions for nourishing the main sources of sustainability of the project, and for maximizing the benefits of this investment – the methodological tools and educational materials that facilitate replication, strengthening the actual model used in replications, using a communications program and other educational interventions to help create and sustain the critical mass of people at the community level committed to change. However, in fact, after ReproSalud ends, the principal source of sustainability is the people involved—the community members, the CBO leaders and promoters, and also the professionals hired and trained by Movimiento Manuela Ramos. The regional coordinators and regional promoters are an important resource for any institution wishing to replicate the project. Many of the regional promoters are young women, often with university-level training, who have just started their professional careers. Young men worked in the departmental offices, albeit for a shorter period. Both will bring the intensive, formative experience they have gained in ReproSalud to all their future employers. These young women and men represent one of the main sustainable assets produced by the ReproSalud project. At the community level, not just women, but men and youth have been trained. In this next period, we recommend that the project take advantage of their investment in all three groups to expand and sustain the impact of the project at the community level. 51 VI. EVALUATION DESIGN AND RECOMMENDATIONS A. GENERAL COMMENT ON SCALING UP AND EVALUATION In our opinion, ReproSalud is one of the most innovative reproductive health projects funded by USAID or any other international cooperation agency. The real innovation in the case of ReproSalud is scaling up. There are similar small pilot projects, but none we know of have reached this scale or developed the tools for large-scale replication. Both Movimiento Manuela Ramos and other organizations in Peru and other countries have conducted similar pilot projects with intensive community-level participation and a focus on reproductive health, gender equity, and women’s empowerment. A much smaller sub￾set of these have worked with indigenous rural linguistic minorities.62 Even fewer organizations, to our knowledge, have incorporated a quasi-experimental design in an impact eva luation. Because of its relative uniqueness and the wealth of data available, the results of all ReproSalud evaluations are extremely important for the sexual and reproductive health field. It is widely accepted that scaling up an innovative small pilot project is not worthwhile unless the pilot has been well-evaluated. However, it is also important to evaluate thoroughly the scaled-up version, for two reasons: ß To understand how best to implement the scaled-up version of a project model. When scaling up innovative pilot projects, things change. Extending the geographical reach entails more complexity. The barriers and facilitating factors in several sites may be significantly different from that in the pilot site. The training of staff has to be standardized, whereas the pilot project could tailor training to the staff and the circumstances. Bringing to scale entails more problems with “quality control,” with more attention to training and supervision. Logistical and administrative problems can pose obstacles to proper implementation. All of these problems need to be solved, but also studied to explain the results of impact studies, especially if the scaled-up version is shown to have less impact than the pilot. ß To determine whether or not the scaled-up version of the project has enough impact to warrant further replication and extension of coverage. Because of the 62 The examples in Peru that come to mind are some of Centro de la Mujer Peruana Flora Tristan’s work with rural women, two of the subprojects of the Consorcio Mujer program (1993-98), and the “bi￾alfabetización” (bilingual Quechua-Spanish literacy program focusing on RH and gender issues) program in Huancavelica, first supported by UNFPA and now run out of CEPAL. Worldwide, similar small-scale programs are documented in the literature, especially post-Cairo (ICPD); most are funded by private foundations and European donors. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 52 differences names above, there could be less or more impact, or a different kind of impact, than that found in the pilot evaluation. B. COMMENTS ON EVALUATION CONCERNS IN REPROSALUD From the beginning, ReproSalud’s evaluation has suffered from a lack of investment and attention. When project leaders and USAID staff used a variety of consultants to make up for the lack of an Evaluation Director, lack of continuity and conflicting opinions caused a chaotic and difficult situation for the staff, with lack of clarity about the basic questions that need to be answered, how to answer them, and who should be responsible for collecting and analyzing the data. Only one of the symptoms of this lack of continuity is that the current results framework and evaluation plan cannot produce a significant portion of the information on impact promised in the Project Paper. Following are more details about the problems noted. ß Lack of planning and under-investment in impact evaluation in the original design of the project. There is no evaluation plan in the project paper. According to a former USAID staff person in charge of technical assistance, the project needed at least another full-time person, and evaluation was one of the areas that suffered as a result. According to ReproSalud staff, MMR and USAID modified by mutual agreement an initial plan to place evaluation with an external organization. These changes in plans delayed the formation of the evaluation unit and the hiring of qualified social scientists. The Pop Tech technical assistance to redirect the research toward impact evaluation did not arrive until 1998, after the project had already been functioning for two years. Some of the “baseline” data of projects started in the first half of 1997 was collected after initial interventions had begun, and was discarded. ß Continuing lack of continuity and oversight of the evaluation of the project. There have been a bewildering variety of staff and consultants with differing opinions. The impact study could not include some of the indicators in the results framework in her analysis because there have been 17 versions of the questionnaire. It is symptomatic that no one has updated the draft evaluation plan written by a former ReproSalud Evaluation Director to correspond to the current plans. At present, USAID/Lima no longer has personnel solely dedicated to coordinating technical support for this project, and the ReproSalud project has no full-time Evaluation Director. Staff and consultants have carried out some excellent and useful research on quality of care and on reproductive health intentions, but the essential function of overall project evaluation — both process and impact — is still not covered. ß Much of the data generated by the quantitative study had to be discarded at mid￾term because of design flaws, and several indicators had insufficient data because of the multiple versions of the questionnaire. In the end, only 25 out of the 70 sub grantee communities in which data were collected, and none of the 17 associated VI. EVALUATION DESIGN AND RECOMMENDATIONS 53 communities, were included in the analysis of the data. The unit of analysis had to be the whole community, actually the social network of the CBO members, because it was a study of households at specific addresses, and not individuals. The impact data collection included interviews with service providers. These data were not analyzed at mid-term, but would be important to analyze (in all three iterations of data collection) at the end of the project because the analysis at that point should reveal the effect of Phase Two activities on questions related to the services. ß The ethnographic impact study produces some important insights, but is not suited to answer all of the important evaluation questions pertinent to Phase Two. 63 It is an ethnographic study in six communities; three sub grantee and three associated. While these communities were not included in the quantitative study, they are nearby and similar to the communities that were included. The study is very intensive, involving two-month stays in each community by two researchers, and then the need to transcribe, translate and code a large amount of data. As currently designed, the study can produce in-depth data on certain key issues in the results framework for which the quantitative study is not suited: 1. Processes of change in gender relations; 2. How the collective capacity of the CBO is increased to organize at the community level; 3. Socio-cultural barriers and facilitating factors for the expected results; 4. The social networks through which the influence of the project spreads; and 5. Whether the educational impact of Phase One has been sustained or strengthened by the lesser level of activities during Phase Two. ß The monitoring system is excellent, but it gathers more information than ReproSalud uses. At the same time, some of the more exciting achievements of specific subprojects go unrecorded, such as concrete examples of lives saved or increases in women’s civic participation. The subproject evaluations, in particular, are rich unexploited sources of qualitative data on lessons learned. ß The design for evaluating the advocacy component of Phase Two is inadequate. The baseline impact study concentrates almost solely on the issue of women’s participation. Its questions do not take into account the impact that Phase Two activities should have on the use of services and on service-quality improvements made as a result of negotiations and feedback. Furthermore, only the opinions of health authorities and local authorities are gathered. The CBO presidents’ and 63 Unfortunately, timing made it impossible to use the results of the qualitative study to explain those of the quantitative study. Because the study was designed so late in the life of the project, the baseline information was gathered in 2000 and has only just been analyzed. The mid-term data will be collected in April 2002. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 54 promoters’ viewpoints on the impact on themselves as individuals, and on their organizations and communities, will not be represented. C. GENERAL RECOMMENDATIONS ON EVALUATION In the next period of the project, ReproSalud needs clear and consistent evaluation leadership so that both monitoring and evaluation will answer key questions about the implementation of Phase Two, and about the long-term impact of both Phases One and Two. Either ReproSalud or USAID should hire a highly qualified full-time staff member to develop the evaluation design, and to coordinate the collection of monitoring, qualitative and quantitative data. Methods of Data Collection The following methods of data collection are recommended for Phase Two of the project.64 Collection of data on the first four items can continue to be in the hands of ReproSalud staff. ß Monitoring and subproject evaluation by ReproSalud staff. Bimonthly monitoring and subproject evaluation forms should be analyzed and redesigned in early 2002, so that they cover some of the essential questions below. The recommendations involve considerable expansion of effort, especially in analysis. The Monitoring and Evaluation Unit should set up a plan to analyze the results. ß A scaled down ethnographic study in April and in 2004 could answer questions pertinent to Phase Two and to the long-term effects of Phase One. A reduced effort to collect mid-term and final data would yield valuable information about cultural and community-level factors that facilitated or hindered the processes of change expected to result from Phase One and Phase Two activities. The reduction should reduce the cost by 30%. The following reductions could be made, and the study would still yield valuable information: - At mid-term, include the original six communities, to conserve the comparison between sub grantee and associated communities and the three regions. If budget constraints mandate a reduced effort in 2004, reduce the final data collection to three communities only if no significant differences emerge between the sub grantee and associated communities. - Researchers should stay for one month in each community instead of two. - Shorten the individual interviews by 50%, concentrating on the key indicators in the results framework and on the questions listed below. 64 The latter two components should be contracted to an outside organization and/or managed by USAID. The outside consultants hired to do the analysis of the qualitative and quantitative studies should be living in Peru. VI. EVALUATION DESIGN AND RECOMMENDATIONS 55 - Conserve the collection of data on paradigmatic reproductive health events (casos ampliados) ß The final quantitative impact study will shed some light on the long-term effects of both Phases and the short-term effects of Phase Two. Finally, the version of the questionnaire should be as close to that used at mid-term as possible, so that the full range of indicators could be included in the final study. This study should include analysis of all of the data collected from service providers during the three iterations of data collection. The data from the advocacy baseline mainly serve s to answer questions about women’s increased civic participation, either in their communities or in the health system. ß Regular collection of data by process evaluators. There should be one researcher and a research assistant assigned to each department; every six months they should spend two weeks visiting a sample of districts (number to depend on budget). The researcher would use semi-structured interview guides for respondents at all levels, similar to the methodology described in this evaluation. To cut costs and time, field notes, rather than transcripts, would be used as the primary data. Whenever possible, visits should coincide with significant events in the advocacy program, such as negotiation meetings with the health sector, or meetings of the Defenders Committee, to allow structured observation. Evaluation Questions for Phase Two Advocacy Component The following questions correspond to monitoring, process, and impact questions that could be answered by the above combination of methods. One could argue that process evaluation is not strictly necessary, since in all cases there is an internal mechanism for gathering the data, given an expansion of subproject monitoring & evaluation. However, because of the sub grantee relationship between Movimiento Manuela Ramos and the CBOs, the answers to certain questions would not be reliable. Furthermore, systematic data collection would involve a significant extra time commitment for departmental staff. External evaluation methods could best answer the questions that have to do with level of fulfillment of responsibilities under sub grants, or views on the ReproSalud project. However, if the process evaluation proves to be unduly costly, the scale of it, and the number of questions included, could be significantly reduced or cut altogether. Looking at the proposed triangulation of methods in the following table, for some questions cutting the process evaluation out altogether would greatly reduce the reliability of the evidence, but there would still be some evidence. ReproSalud staff should consider the following table a first draft, on which they should improve. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 56 Questions Source of data 1) Have users increased in the centers of reference and health posts because of the agreements? IS, SE 2) What aspects of services have providers and health authorities changed in response to the agreements? SE 3) What are providers’ views of the relationship with the Defenders Committee and Promoters Network? In their views, what elements of the subproject have and have not functioned well? SE, PE 4) Do users feel that the services’ capacity to resolve their health problems has improved? Do providers treat them more respectfully? IS, PE, ES 5) How well are the promoters fulfilling their commitments to engage in referrals? To conduct educational activities? How well are they fulfilling other commitments in the subprojects? SE, M, PE, ES (2004) 6) When nearby communities request the replication of the project, does the reimbursement offered sufficiently offset the opportunity costs for the CBO promoters? SE, PE 7) Are the demands and observations of CBO members on the quality of health services being communicated to the Steering Committees in the advocacy program? Conversely, is information on the progress of the negotiations flowing from the Steering Committee through the network members? M, PE, ES 8) How much dialogue or discussion happens at the community level on how to handle impasses in negotiations, or on health centers’ failure to meet key demands of CBOs? Does the Defenders’ Network consult the CBO members? M, PE, SE, ES 9) What factors in the Defenders Network’s strategies might explain the success or failure in having their demands met by the health center or DISA authorities? What factors within MOH explain success or failure? SE, PE, ES 10) How does the regular rotation in the leadership of the CBOs affect the operation of the Defenders’ Network? What are the best strategies for bringing new CBO Presidents on board and securing their active involvement in advocacy? M, PE 11) How does the Defenders’ Network communicate with the Promoters’ Network above the level of each CBO and subproject? What departmental￾level issues do they need to communicate about, and why? PE, M 12) Does the lack of financial incentives affect the CBO Presidents’ and promoters’ motivation to participate in the process? What other factors affect motivation and participation? PE, ES, M 13) Do CBO members and other community stakeholders view Phase II advocacy as beneficial? How does their distance from the “action” in advocacy affect their views of the ReproSalud project? Are other educational interventions by ReproSalud in Phase Two visible, and how are they viewed? PE, SE 14) Has the advocacy training effort averted the problems that the training course was designed to avert? What additional input could help to address these problems? PE, SE VI. EVALUATION DESIGN AND RECOMMENDATIONS 57 Evaluation of Replication in New Communities or Districts The following items state the important evaluation questions that could be answered with an additional investment in evaluation research. All items would demand a round of data collection in 2002. Within the subprojects, compare the results between communities with and without autodiagnósticos, and those receiving the full set of topics vs. those who receive only selected topics. 65 Current plans for replication in neighboring communities do not include an autodiagnóstico, but some communities might request to implement the full model, including the autodiagnóstico. Similarly, some communities might request only one or two of the topics. It would be important to observe the difference in the effects on women’s empowerment, on the functioning of the CBO, and on health-protective attitudes and behavior between the full model of Phase One and the more limited versions. Data collection method: pre and post design with a shortened version of the impact study questionnaire, and questions added about CBO functioning. Within the subprojects and in replications by other institutions (if possible), compare the results between communities that worked with women only, and communities that work with men and women simultaneously. It is likely that the demands from local communities will naturally divide into these two processes. This comparison would help refute or support ReproSalud staff’s hypothesis that introducing education to men and women at once would reduce the effect on women's empowerment. Data collection method: pre and post design with a shortened version of the impact study questionnaire. Evaluate the post-training experience in Ucayali and San Martín to understand how best to structure training for MOH personnel in other departments. ReproSalud should evaluate the quality and viability of replications by MOH in San Martín and Ucayali. ReproSalud needs to evaluate what additional educational modules would be necessary to train MOH personnel in both the principles and the methodology of the project, and whether the training strategies overcome the barriers mentioned. The evaluation should help to determine which MOH personnel (local, district, or departmental levels) are best suited to carry out replications. Data collection method: post-intervention interviews with MOH personnel and CBO promoters. If possible, observation of educational workshops given by MOH personnel or community agents. 65 The results of the test would be more reliable if the communities were randomly assigned to one model or the other. However, in order to increase acceptance of the program it is important to follow the community's lead, especially if the request comes from local authorities. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 58 Evaluation Questions Related to the Communications Program The person hired to run this program will develop a full evaluation plan. The questions derive from the objectives of the program described previously. Men and Youth: The age range from 15-24 is under-represented in the quantitative survey, and men are asked a much more limited set of questions. It would be important to evaluate thoroughly whatever component of the communications program is directed to youth and men. . For example, the communications evaluation could measure changes in group norms among youth and men through focus groups or semi-structured interviews in selected communities before and after the intervention. Evaluation Questions or Studies Related to the Communities Currently Involved in the Project Secondary analyses of the database from the impact study would yield important information at low cost: ß Analyze the whole database of 70 communities to compare results by department. The findings on departmental differences from the cost effectiveness study were not reliable because of the small sample size. ß Compare the results of the 70 sub grantee communities with those of the 17 associated communities, to see if there are any significant differences. ß In the 70 sub grantee communities, compare results between those who held educational workshops for men, and those that didn’t. Perform a similar comparison for those who worked with youth and those that didn’t, using only the 18-24-year-old respondents in the database. ß Compare the results for the whole database of those communities with income￾generating activities, and those without. Use the ethnographic study to complement information from the impact study on the following questions: ß Do some of the positive health-protective gains fade if they are not reinforced by further educational activities? Which gains are sustainable over time, and which fade without additional reinforcement from community-level activities? From messages in the media? Are gender-related gains more susceptible to erosion than gains related to reproductive health? ß Does the influence of the project expand to the CBO members’ social network — to neighbors, family members, and the next generation? Does influence spread differently among women than among men? 59 VII. CONCLUDING REMARKS Movimiento Manuela Ramos succeeded beyond most reasonable expectations in meeting their multiple organizational and programmatic challenges. They have implemented a large-scale nationwide project that surpassed its original objectives, using a highly participatory methodology that defied standardization, in hard-to-reach communities that posed numerous logistical and cultural barriers to their work. For many years, both MOH and other NGOs have tried different strategies to improve reproductive health in rural and/or indigenous communities. In spite of these efforts, distrust of health services and distrust of modern medicine and contraceptives have remained high in these communities. This divide between the communities and the health sector -- when coupled with poverty and physical/cultural isolation – has resulted in severe continuing reproductive health problems. ReproSalud has enjoyed a measure of success and acceptance that is remarkable given this history. As evaluators, we have recommendations for improvement or adjustment of plans in the following areas: ß To evaluate the advocacy program closely, given concerns with the model, and prepare in advance for potential problems. ß In the communities involved in Phase Two, to provide educational reinforcement in issues that need improvement, through continued training of promoters and through the communications program. ß To involve male promoters in training and outreach efforts to new communities, as MOH community agents, and in the communications program ß To implement a mass communications program in the current districts to give greater saturation and coverage for the educational messages of the program. Implement, with adjustments to that proposed by A. Gamucio that promote greater involvement at the community level, targeted approaches to men and youth, and support to efforts to replicate the ReproSalud project. ß Increased cooperation with other NGOs, in order to maximize the potential for replication by other agencies. ß Expansion and restructuring of the evaluation and monitoring component of the project, with increased resources devoted to it, a person in charge, redesign of the qualitative impact study, carrying out process and impact evaluation for Phase II, improvements to the results framework and indicators, and expanding the monitoring data that are collected and analyzed. MIDTERM EVALUATION OF REPROSALUD: PROCESS EVALUATION REPORT 60 ß Other measures to enhance replication, including intensifying attention to guiding principles in the training, and producing a unified set of training and educational materials. All sexual and reproductive health programs in the Andean region – both jungle and highlands – could benefit from replicating this project when addressing a sharp divide between the health sector and hard-to-reach low-income communities. The project makes a unique contribution through its guiding principles: its recognition of the central role of gender equity in achieving health goals, its participatory and intercultural approach, its focus on human and users’ rights, and its comprehensive approach to health issues. Some recommendations in this report arise from a systems approach to the dynamics of cultural change, whether in communities, in family, or in health services. This approach assumes that change happens faster and with less resistance when working with two or more parts of a system than when working in just one. For example, in efforts to achieve gender equity, it is probably more efficient to work with both men and women than to work with just women. In the same vein, the findings from this evaluation suggest that when aiming to improve the use of services among members of these hard-to-reach communities, “supply-side” changes in the quality of care and in access often are not enough to achieve coverage goals. In these communities, high levels of mistrust and lack of knowledge inhibit change until culturally affirming and community-run educational efforts such as ReproSalud help to overcome these obstacles. 61 VIII. REFERENCES Alberti, A., Caro, D., Posner, J., Schuler, S., Coe, A. July, 1998. “Strengthening ReproSalud’s Monitoring and Evaluation Strategies.” POPTECH Report Nº 97-124-066. Arlington, VA: Population Technical Assistance Project. Anderson, Jeanine, 2001. Tendiendo Puentes. Lima, Peru: Movimiento Manuela Ramos. Feringa, Barbara, 1999. “Two Years Off the Ground,” Lima, Peru: USAID. Coe, Anna-Britt. 2001. Health, Rights and Realities: An Analysis of the ReproSalud Project in Peru. Takoma Park, MD: Center for Health and Gender Equity. Gumucio Dagrón, Alfonso, 2001. “La Comunicación en el Proceso Participativo de ReproSalud” (Communications in ReproSalud’s Participation Process). Unpublished report. Rogow, Debbie, 2000. Alone You Are Nobody, Together We Float: The Manuela Ramos Movement, Nº 10 in the Quality/Calidad/Qualité Series. New York: Population Council. Shepard, Bonnie, 2002. “ ‘Let’s Be Citizens, Not Patients!’: Women’s Groups in Peru Assert Their Right to High-Quality Reproductive Health Care” in Responding to Cairo: Case Studies of Changing Practice in Reproductive Health and Family Planning. New York: Population Council. Tamayo, Giulia, 1999. Nada Personal. Reporte de Derechos Humanos sobre la Aplicación de la Anticoncepción Quirúrgica en el Perú 1996-1998. Lima: CLADEM. Yon, Carmen, 2000. Hablan Las Mujeres Andinas. Lima, Peru: Movimiento Manuela Ramos