JUNE 2005 This publication was produced for review by the United States Agency for International Development. It was prepared by the Quality Assurance Project and authored by Maria Amelia Viteri. QUALITY ASSURANCE PROJECT EVALUATION REPORT EVALUATION OF THE LATIN AMERICAN AND CARIBBEAN MAXIMIZING ACCESS AND QUALITY EXCHANGE EVALUATION REPORT EVALUATION OF THE LATIN AMERICAN AND CARIBBEAN MAXIMIZING ACCESS AND QUALITY EXCHANGE June 2005 Maria Amelia Viteri DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. The Quality Assurance Project (QAP) is funded by the United States Agency for International Development (USAID) under Contract Number GPH-C-00-02-00004-00. The project serves developing countries eligible for USAID assistance, USAID Missions and Bureaus, and other agencies and nongovernmental organizations that cooperate with USAID. QAP offers technical assistance in the management of quality assurance and workforce development in healthcare, helping develop feasible, affordable approaches to comprehensive change in health service delivery. The project includes prime contractor University Research Co., LLC (URC), Initiatives Inc., and Joint Commission Resources, Inc. Recommended citation: Viteri MA. 2005. Evaluation of the Latin American and Caribbean Maximizing Access and Quality Exchange. Evaluation Report. Published for the U.S. Agency for International Development (USAID) by the Quality Assurance Project. About this series: The Evaluation Report series presents the findings, recommendations, and lessons learned of quality assurance program evaluations and other evaluation studies conducted by QAP. Please visit http://www.qaproject.org/pubs/pubsreports.html for information on other QAP evaluation reports. Acknowledgements: The Quality Assurance Project would like to thank Kellie Stewart of USAID and Tita Oronoz of Family Health International for their helpful comments on draft versions of this report. LAC MAQ Exchange Evaluation · i TABLE OF CONTENTS EXECUTIVE SUMMARY...................................................................................................................II ABBREVIATIONS .............................................................................................................................IV I. BACKGROUND............................................................................................................................1 II. EVALUATION OF THE LAC MAQ EXCHANGE.................................................................2 A. OBJECTIVES .............................................................................................................................2 B. METHODOLOGY .......................................................................................................................2 C. STUDY CONSTRAINTS...............................................................................................................2 III. FINDINGS .................................................................................................................................3 A. DOMINICAN REPUBLIC .............................................................................................................4 B. EL SALVADOR..........................................................................................................................5 C. GUATEMALA ............................................................................................................................6 D. HONDURAS ..............................................................................................................................6 E. NICARAGUA.................................................................................................................................7 IV. CONCLUSIONS........................................................................................................................9 V. RECOMMENDATIONS FOR FUTURE EXCHANGES ...................................................... 10 APPENDICES ..................................................................................................................................... 12 APPENDIX A: LAC MAQ EXCHANGE PARTICIPANTS BY COUNTRY.................................................... 13 APPENDIX B: DATA COLLECTION INSTRUMENTS................................................................................ 15 APPENDIX C: AUTHOR’S NOTES ON DATA COLLECTION BY COUNTRY ............................................... 21 ii · LAC MAQ Exchange Evaluation EXECUTIVE SUMMARY Representatives of five Latin American and Caribbean (LAC) countries assembled in Tegucigalpa, Honduras, in April 2002 to attend the first regional Maximizing Access and Quality (MAQ) Exchange, which was jointly funded by the United States Agency for International Development (USAID) LAC Bureau and the Global Health Bureau through the MAQ Initiative1 . A MAQ Exchange is a means of engaging USAID Missions, their country counterparts, USAID/W and collaborating agencies in a dialogue aimed at developing or improving programs that reflect MAQ principles and practices. It is a product of the MAQ Initiative, which for several years has joined the efforts of USAID, cooperating agencies, and host country partners to collect field expertise and to apply state-of-the-art methods to maximize access to high quality family planning and other selected reproductive health services through a client-centered approach. In keeping with the purpose of a MAQ Exchange, the first LAC MAQ Exchange was a five-day event designed to provide information on a range of evidence-based best practices and stimulate ongoing actions to improve both access to and the quality of family planning and reproductive healthcare services in the five countries. The 65 participants from Ministries of Health, Health Secretariats, USAID Missions, nongovernmental organizations, and cooperating agencies concluded the meeting by developing action plans that they would implement after returning home. The teams could select any topic within the scope of family planning and reproductive health and seek $15,000 seed money from USAID to cover costs related to implementing their action plans. In early 2004, USAID asked the Quality Assurance Project to evaluate the results of the Exchange and the seed funds disbursed. As reported here, the evaluation found mixed results. There was a clear consensus among the country participants about the high value of the themes and content of the Exchange and the high quality of the presentations at the meeting in Honduras. The conceptual frameworks and methodological tools and supporting materials presented at the Exchange were considered by virtually all respondents to be very useful, and many informants reported to have used the materials in their own activities, in many cases, independently from the action plans. With respect to implementation of the action plans developed, the El Salvador team coalesced and largely achieved its action plan; good progress was achieved in the Dominican Republic and Nicaragua; and only limited progress was made in Guatemala and Honduras. The El Salvador team “was well organized from the start.” It defined roles for team members, kept and distributed meeting minutes, and exemplified effective team process. We report that while the El Salvador action plan was near completion at the time of the evaluation, the team expected to continue working together thereafter. This indicates that the concept and content of the Exchange can indeed facilitate sustained teamwork and the implementation of desired changes in clinical services. The reasons for the differential impact among the five countries are many and relate in large part to the political, organizational, and cultural context unique to each country. First, it is difficult to separate the impact of the Exchange from that of pre-existing working relationships and coordination mechanisms and organizational culture in each country. One of the factors that seemed to constrain the participation of the full panel of team members in subsequent action plan implementation was the selection of a topic that was more narrowly focused on the interests of one institution. Also, many of the MAQ participants 1 The Maximizing Access and Quality (MAQ) Initiative is an ongoing forum that brings together staff from USAID/Washington, USAID Missions, the cooperating agency (CA) community, and other partners to identify and implement practical, cost-effective, and evidence-based interventions to improve both the access to and quality of family planning and reproductive health services. The MAQ Initiative is sponsored by USAID's Office of Population and Reproductive Health. For more information on the MAQ Initiative, visit the MAQ website at http://www.maqweb.org. LAC MAQ Exchange Evaluation · iii simply had an already very full work agenda and very limited time to take on new responsibilities. Staff turnover and changes in assignments proved to be serious impediments to the work of at least two of the country teams. In some cases, it was unclear to the participants who was in charge of providing follow-up technical assistance to the process—the Quality Assurance Project, Family Health International, or USAID. The majority of key informants expressed a need to have an “official” person designated to monitor and accompany the MAQ team in terms of proposal implementation, calls for meetings, and similar issues. The evaluation found that the Exchange had stimulated communications and the development of professional networks to varying degrees in all five countries. This process was most clearly visible in El Salvador. Centralization of decision-making and control over the seed grant funds seemed to pose an obstacle for the interaction of the groups, particularly in the Dominican Republic and Nicaragua, since it tied the information as well as the decisions regarding implementation and development of the action plan to one person or institution. The fact that grant funding was issued to a single institution in each country led to unequal participation of MAQ Exchange participants in the follow-on activities. The report concludes with recommendations to strengthen future Exchanges in the areas of participant selection, technical content of the Exchange, selection of action plan topics, monitoring of action plan implementation, and management of funds. iv · LAC MAQ Exchange Evaluation ABBREVIATIONS ADOPLAFAM Dominican Family Planning Association ADS Salvadoran Demographic Association AGOG Association of Gynecology and Obstetrics of Guatemala AMS Association for the Auto-determination and Development of Salvadoran Women APROFAM Family Planning Association of Guatemala ASHONPLAFA Honduras Family Planning Association ASOGOES Association of Gynecology and Obstetrics of El Salvador CA Cooperating Agency CEMUJER Institute of Women’s Studies (El Salvador) DPS Provincial Directorate of Health (Dominican Republic) FHI Family Health International FP/RH Family Planning/Reproductive Health IEC Information, Education, and Communication IEPROES Specialized Institute for Health Professionals (El Salvador) IHSS Honduran Social Security Institute ISDEMU Salvadoran Institute for Women’s Development ISSS Salvadoran Social Security Institute JHPIEGO Johns Hopkins Program in International Reproductive Health Education KAP Knowledge, Attitudes and Practices LAC Latin America and Caribbean MAQ Maximizing Access and Quality MCH Maternal and Child Health MINSA Ministry of Health (Nicaragua) MOH Ministry of Health MNH Maternal and Neonatal Health MSPAS Ministry of Public Health and Social Assistance NGO Non-Governmental Organization PROFAMILIA Family Planning Association of Nicaragua PVO Private Voluntary Organization QA Quality Assurance QAP Quality Assurance Project QI Quality Improvement SDOG Dominican Society of Obstetrics and Gynecology SESPAS State Secretariat of Public Health (Dominican Republic) SILAIS Local Integrated Health Care System (Nicaragua) UES University of El Salvador UNASA University of Santa Ana (El Salvador) URC University Research Co., LLC LAC MAQ Exchange Evaluation · 1 I. BACKGROUND Instituted by the U.S. Agency for International Development (USAID) in 1994, the Maximizing Access and Quality (MAQ) Initiative is a collaborative partnership of organizations working to improve reproductive health and family planning services in developing countries. With USAID funding, MAQ provides opportunities for healthcare providers and program managers to collaborate and pool their knowledge and field experience to identify and promote state-of-the-art tools and concepts in family planning and reproductive health (FP/RH). One of the most significant MAQ dissemination efforts are MAQ exchanges, which bring together staff from USAID/Washington, USAID Missions, cooperating agencies (CAs), and program managers to identify and implement practical, cost-effective interventions to improve both access to and the quality of family planning and reproductive healthcare services. MAQ Exchanges offer a forum where experts can bring information on a range of evidence-based best practices in FP/RH service delivery to teams of policymakers and healthcare providers in a given country. The exchange format incorporates both technical presentations and discussions where participants share their own programmatic experiences. Content selected for presentation at an exchange is tailored to the needs of participants, based on preparatory assessments conducted in-country. Exchanges vary in length from three to five days and include workshops where participant teams create action plans to be implemented when team members return to their work place. USAID sponsored its first regional MAQ Exchange in Tegucigalpa, Honduras, April 22-26, 2002. In addition to providing quality assurance training for FP/RH clinical settings, the Latin American and Caribbean (LAC) Exchange was oriented to the development of quality programs in the participant countries: the Dominican Republic, El Salvador, Guatemala, Honduras, and Nicaragua. The goals of the LAC MAQ Exchange were to: • Sensitize staff from the five countries, USAID Missions, and partner organizations about issues of quality and access in FP/RH programs; • Share family planning challenges and best practices, particularly with Ministry of Health (MOH) counterparts; • Exchange and update knowledge of FP/RH practices for practitioners and managers from Ministries of Health, nongovernmental organizations (NGOs), and USAID; • Introduce participants to concepts of organizational development, leadership, and sustaining quality; and • Develop realistic country action plans to implement specific activities to improve the quality of FP/RH services. The five-day Honduras conference had three parts: a mini-university made up of concurrent sessions where participants attended required and elective courses on best practices in family planning; plenary presentations on quality assurance, leadership, and sustainability; and working groups during which each country group developed a single action plan for the initiation or development of quality improvement activities related to reproductive health. Some 65 representatives of Ministries of Health/Health Secretariats, USAID Washington and Missions, CAs, and NGOs attended; a list of the LAC Exchange participants by country is found in Appendix A. The LAC MAQ Exchange was organized and facilitated by USAID, Family Health International (FHI), and the Quality Assurance Project (QAP). Management Sciences for Health and the Population Council also provided speakers and some logistical support for the Exchange. 2 · LAC MAQ Exchange Evaluation Following the Exchange, country teams were expected to finalize and then implement their action plans. Some technical assistance to carry out the action plans was provided by various USAID cooperating agencies as part of their USAID-supported work programs. Up to $15,000 in seed grant funds was later made available to each country team, based on approved proposals, to carry out new initiatives related to improving access and quality of family planning and reproductive health services. FHI was asked to coordinate the projects that followed from the Exchange and administer the grant funds upon approval of proposals by a core technical review committee. II. EVALUATION OF THE LAC MAQ EXCHANGE In follow-up to the LAC MAQ Exchange, QAP was asked by USAID to conduct an evaluation to determine the effectiveness of the MAQ Exchange format and gauge the progress made by country teams in implementing the action plans developed at the meeting in Honduras. The evaluation was conducted in February–March 2004, approximately 21 months after the Exchange took place and some 6-8 months after the disbursement of seed grant funds. A social scientist was hired by QAP to visit each of the five countries involved in the Exchange to gather data through interviews with members of the MAQ teams. A. OBJECTIVES The evaluation was intended to inform the design of future exchanges by documenting the impact of the LAC MAQ Exchange on the spread of MAQ concepts and approaches in the participating countries. The specific objectives of the evaluation were to: • Determine the effectiveness of the LAC MAQ Exchange design in spreading best practices in each country through application of concepts and materials presented in the Honduras meeting, • Assess progress made by teams in implementing their action plans and seed grant projects, • Determine what impact the LAC MAQ Exchange had on networking and coordination among MOH, USAID partners, and NGOs, and • Provide recommendations to optimize the field-level impact of future exchanges. B. METHODOLOGY The data collection methods used in the research were group and in-depth interviews, supplemented by a written questionnaire given to respondents who could not be interviewed. (The three data collection instruments are found in Appendix B.) In-depth interviews elicit informative interests (spontaneous memories), beliefs (expectations and value orientations on received information), and wishes (internal motivations—conscious or unconscious). The three instruments tools allowed for the collection of information in a limited time frame despite the large number of informants and the complexities of their schedules. C. STUDY CONSTRAINTS In interpreting the findings from participant interviews, it is important to recognize that it is impossible to completely separate the impact of the LAC MAQ Exchange from the effects of other programs and initiatives which may have affected the quality and accessibility of family planning and reproductive health services in the five countries. It should also be recognized that government policies and priorities related to FP/RH differ among the five countries, such that the same intervention (LAC MAQ Exchange) may be expected to have a different impact in the social, cultural, and political climate of each country. While the evaluation attempted to discern the Exchange’s impact, readers should recognize the difficulty in doing so precisely. Lastly, both an informant’s willingness to be forthcoming and the order of interviews necessarily affect the quality of the information collected. LAC MAQ Exchange Evaluation · 3 III. FINDINGS This section provides information on the teams’ experiences after the Exchange, examining both their progress in implementing their action plans and the degree of inter-agency coordination achieved in furthering the quality and access of family planning and reproductive health services. While the Exchange participants came from different organizations, it was expected that they would constitute a MAQ “network” in the country, continuing to share experiences and to work collaboratively toward larger goals of family planning service expansion and quality improvement. Table 1 summarizes the problem addressed and proposed interventions for each of the five country action plans. Table 1: Country Action Plans Developed at LAC MAQ Exchange Country Problem Causes Proposed Interventions Dominican Republic Norms and standards for delivery and post￾partum care are not being put in practice, especially in public sector hospitals Lack of incentives, staff attitudes, weak communication of standards, lack of client awareness, no supervision or ongoing monitoring Develop and disseminate national standards for delivery and post-partum care; training related to the standards; introduction of a system for measuring compliance with standards; development of IEC materials for clients El Salvador Inadequate preparation of new service providers in sexual and reproductive health Faculty are not up-to-date on new standards and methods Policy level support for curriculum revision; creation of training teams to update skills of medical and nursing faculty Guatemala Limited availability and delivery of family planning for women in postpartum and post-abortion care in 2 hospitals Lack of training and supervision of health providers in FP; lack of recognition of importance of reproductive health services; lack of coordination between MOH and NGOs; limited infrastructure and equipment Coordinated planning between hospital and the Health Area; training and supervision in competency-based FP, post￾partum and post-abortion care; strengthen logistics and supplies; and supervision Honduras High maternal mortality from post￾partum hemorrhage in Mario Rivas Hospital Failure to comply with norms; poor performance by health personnel; lack of triage; poor communication; lack of motivation; lack of lab personnel; lack of blood and oxytocin; weak referral links; Training in application of norms; use of quality improvement methods with hospital personnel Nicaragua Unmet demand for FP among postpartum clients Establish facility-based teams to define, monitor and evaluate quality standards for FP services provided to post-partum clients; strengthen FP counseling and referral of post-partum clients; conduct KAP surveys to assess IEC effectiveness 4 · LAC MAQ Exchange Evaluation A. DOMINICAN REPUBLIC The influence of the LAC MAQ Exchange in the Dominican Republic was observed in terms of improvements in maternal care protocols and replication of training in MAQ concepts. The Exchange succeeded in creating greater awareness among personnel about the importance of documenting, communicating, and implementing protocols for partum and post-partum procedures Before the Exchange, each institution had its own maternal care protocols, implicit or explicit, and these often did not reflect the latest evidence-based practices. For example, before the Exchange, Los Mina Hospital in Santo Domingo used numbers to identify and refer to patients. After the team obtained information on best practices from the Exchange, the hospital changed its policy to now use names instead of numbers to identify and address patients. Francisco Gonzalvo Hospital in La Romana has since January 2004 fully integrated updated protocols for delivery and post-abortion care in its obstetrics services, and patients are clearly benefiting from the best practices promoted at the Exchange. Patients in labor may now be accompanied by relatives and can have liquids, neither of which happened before. While these changes cannot be exclusively credited to the Exchange and its follow-up activities, they do suggest that the Exchange influenced participating institutions. At the time of the evaluation, baseline assessments had been completed at the two Secretariat of Health (SESPAS) hospitals and the ADOPLAFAM clinic. Updated clinical guidelines were being developed and implemented at Francisco Gonzalvo Hospital in La Romana, and the Dominican Society of Gynecology and Obstetrics (SDOG) was negotiating implementation at La Vega Hospital, where training has been done. It is interesting to note that Francisco Gonzalvo Hospital had just started an adolescent ward. Though not fully attributable to the Exchange, the ward provides information and counseling and has an exclusive room for adolescents and their newborns. One Exchange participant, the director of one of the biggest hospitals in Santo Domingo (Los Mina), commented on the value and utility of the mini-university. He had replicated it with colleagues both at the hospital and the university. Participation in the Exchange also appeared to have strengthened linkages between public and private providers. The ADOPLAFAM director commented on the Exchange’s contribution to strengthening referral linkages between private FP clinics and public referral hospitals. He said that when he formerly sent women with high-risk pregnancies to the main public hospitals (Villa Mella or Los Mina), they were not received but rather, sent back. Since the Exchange opened up a dialogue with public sector facilities and increased understanding of how the family planning clinic operates, referrals from ADOPLAFAM are now accepted at the public hospitals. The ADOPLAFAM director said he now refers women without fear that they will be sent back. He also said that the ADOPLAFAM clinic was able to apply Exchange concepts in the area of epidemiology, introducing new techniques to evaluate the services provided. Team success in terms of implementing the action plan can be credited to the centralization of the management of MAQ-supported activities, including funds management, and to the supportive involvement of the USAID Mission. FHI staff and the staff of the FHI-CONECTA Project played a crucial role, especially in following up and supporting the MAQ team activities. Furthermore, the team members’ roles were clear from the start: the group worked together to develop a chart assigning each member a certain function to contribute to completion of the action plan. At the same time, some respondents voiced the concern that funding and management decision-making was too concentrated in one organization, the SDOG. Historical rivalries among the different MAQ team institutions and staff may have contributed to the fact that, by the time of the evaluation, the number of Exchange participants still active on the team was less than half of what it was at the time of the meeting in Honduras. One issue that presented itself more in the island nation of the Dominican Republic was the fact that in the process of implementing the plan, team leaders who had been students at various teaching institutions were put in the position of needing to direct their former teachers. The former students commented that they had struggled with the resulting dynamics. LAC MAQ Exchange Evaluation · 5 B. EL SALVADOR El Salvador was the leader among the country teams that participated in the Exchange in terms of broad dissemination of the concepts promoted at the Exchange and progress in implementing its action plan and in achieving meaningful, ongoing collaboration between the MOH, NGOs, and other institutions on MAQ-related issues. At the time of the evaluation, the team had nearly achieved all of the action plan’s objectives and had good possibilities for follow-on activities. The El Salvador team demonstrated how distinct agencies can effectively cooperate with each other and exhibited team dynamics that engendered success. Ongoing programming and inter-agency coordination has taken place through regular meetings and exchanges between the 10 key national institutions working in reproductive health, including the Ministry of Public Health (MSPAS), Salvadoran Social Security Institute (ISSS), the Gynecology and Obstetrics Association (ASOGOES), NGOs, universities, and nursing schools. Coordination has also been actively facilitated by the USAID Mission in El Salvador. Of the original 12 members who attended the Exchange, eight continue to participate in the action plan activities. A key result of the MAQ country team’s activities was the development of a full pre-service training curriculum in reproductive health for medical and nursing schools, including a Training Handbook on Sexual and Reproductive Health. The handbook covers 47 topics and references the Exchange’s best practices framework. It includes text and illustrations and explains in a clear and concise way, how to provide client-friendly, high-quality services. The illustrations can be used as audiovisual aids (universities and other medical training institutions are more likely to have an overhead projector than a computer) and are also available on CD-ROM. A training of trainers course for facilitators and medical and nursing faculty was also implemented. By the time of the evaluation, the team had replicated the content of the MAQ Exchange for 113 people at the country level, including pathologists, gynecologists, and others involved in FP/RH. The team has engaged 30 fellows (two for each university or nursing school) to monitor and replicate the action plan objectives: 18 are constantly moving from place to place, and 12 are in charge of monitoring. Universities that were participating at the time of the evaluation were Andrés Bello, San Miguel, and Santa Ana. The action plan coordinator, who works at the University of Santa Ana (UNASA), praised the interest in reproductive health that the Exchange generated. The university developed an awareness-raising workshop for students on sexuality and reproductive health. She also said that the number of unwanted pregnancies at UNASA has decreased since the Exchange. Informants indicated that USAID’s ability to bring these institutions together and its ongoing influence, exercised by Maricarmen Estrada, were key in stimulating Exchange participants to work together following the meeting in Honduras. All the original Exchange participant institutions cooperated, creating the example wherein progress was made and results shared. Informants also recognized the value of the team’s diversity, noting “the richness of variety of knowledge”. Additionally, relations between USAID and the public institutions (ISSS, ASOGOES, MSPAS) have been friendly, facilitating joint work for common aims. Similarly, PRIME has played an important role in coordinating funds and working closely with USAID to establish common goals. The Institute of Woman Studies (CEMUJER) and the Association for Self-determination and Development of Salvadoran Woman (AMS) contributed to the gender focus in the curriculum proposal. Another factor was that the team was well organized from the start, writing meeting minutes and sending them to those who did not attend, ensuring regular and fluid communication. The group even organized a three-day workshop to work with tools from the Exchange that relate to the action plan. Turnover of team members was a problem, but the team process achieved in El Salvador served to mitigate its impact. The ISSS representative noted, however, that “people make the processes,” suggesting that the success of the 6 · LAC MAQ Exchange Evaluation El Salvador team may also be due to the unique characteristics of the individuals involved. The strong support of the Dean of the Medical School at UNASA and the availability of medical and other faculty to help carry out the action plan also likely contributed to successful implementation. Finally, another enabling factor in El Salvador may have been the fact that additional resources were available to fund the project identified at the MAQ Exchange. The amount available from MAQ for the El Salvador project ($15,000) actually represented only a portion of its total budget (approximately $85,000). C. GUATEMALA The Guatemala country team was not able to implement its initial action plan, confronting multiple constraints related principally to national level political instability. Outcomes had not matured when the evaluation took place. The original Guatemala plan sought to strengthen both access to and systematization of FP services for post-partum patients in two rural maternity clinics. Additionally, the team wanted to create two training centers on vasectomies and intrauterine device insertion in those facilities. The Guatemala team failed to decide which organization would be in charge of funds management during the Exchange. This responsibility was later assigned to the Guatemalan Association of Obstetrics and Gynecology (AGOG). Turnover and lack of commitment by either individuals who attended the Exchange or organizations represented there combined to undermine the likelihood of success. Time constraints resulted from busy agendas of those involved, due to their multiple professional responsibilities. Only one team member— Dr. Mirna Patricia Barahona of the MSPAS—made consistent efforts to implement the action plan. Support from the USAID Mission was hindered when the staff member who had been involved in the Exchange was moved to a different position. The fact that the topic had been suggested by representatives of the MSPAS meant that other team members needed MSPAS support to implement the project. Repeated MSPAS staff turnover required repeated negotiations, inhibiting implementation. One key informant, who joined the team after the Exchange and is participating in its reactivation, said the action plan needs some sort of official backing from the MSPAS to ensure project execution. The group interview highlighted the innovative and practical character of the information provided in the Exchange, specifically mentioning the possibility of working on different parallel activities by alternating topics. The group also stressed the need to define roles and have participants make a commitment prior to participating in the Exchange. At the time of this evaluation, the team had recently developed a new action plan and was bringing in new team members. A mini-workshop was being considered to inform a broader range of healthcare professionals on the MAQ Exchange purpose and content. Informants credited this progress to follow-up from FHI. D. HONDURAS All informants mentioned the value of the Exchange methodology and contents, noting in particular the organization of topics and the materials provided. While the action plan was only partially implemented by the time of the evaluation, the MAQ Exchange did have several positive results. Theoretical concepts and methodologies from the Contraceptive Technology Update presented at the MAQ Exchange were well received by participating Honduran organizations. For example, the MAQ Checklist and the FHI Pregnancy Checklist tools are frequently used. Some of the MAQ team institutions had been using prenatal care checklists and other tools before the Exchange, and afterwards, they made them mandatory, LAC MAQ Exchange Evaluation · 7 especially for complications. It should also be noted that prior to the Exchange, a number of efforts to improve the quality of healthcare services in Honduras were underway by JHPIEGO, QAP, and the Secretariat of Health—efforts which supported the same concepts and tools as promoted in the LAC MAQ Exchange. The information provided at the MAQ Exchange motivated ASHONPLAFA staff, assuring them that they were following quality control principles in providing sexual and reproductive healthcare. The director of ASHONPLAFA uses the MAQ CD for training diverse actors: institutional personnel, healthcare providers, and adolescents. Initially, the group worked well together in designing the implementation plan, which was sent to FHI and shared with the core technical review committee, comprised of USAID, FHI, and QAP staff. FHI communicated the technical review committee’s assessment that the plan was too ambitious and would be setting the team up for failure. Once some members of the group received the technical review feedback, the group lost cohesion and momentum. The loss was exacerbated and accelerated by initial participants who delegated responsibilities to other people who had not attended the Exchange. A second action plan was submitted to the technical committee and comments returned to the team; the committee required further changes to the action plan before funding would be released. Reports on whether the final plan was actually returned to FHI differed. The team held meetings regularly during the first three months but dissolved soon thereafter, leaving only a few people aware of the plan’s status and next necessary actions. Reports on the disbursement of funds also differed. The project objective and topic were neither relevant to all team members participants nor a priority for reducing maternal mortality. Some team members attempted to develop another proposal and started working on it. Additionally, neither hospital identified as a site for the intervention was informed of the action plan’s intention and objectives. The team had selected the two sites for implementation of the plan when none of the hospitals’ representatives were present. Furthermore, most of the MCH Department staff who represented the Secretariat of Health at the Exchange did not have the necessary authority to commit the Secretariat of Health to the plan of action as developed. Turnover was a major problem in Honduras. (Exchange participant turnover was 80%.) Some Exchange participants were facing a transition while they attended the workshop, contributing to their lack of involvement. Also, the roles and responsibilities of the team members were never clearly defined; informants gave contradictory opinions regarding who was in charge of monitoring, notifying the group of meetings, funding, etc. Geographical distances also impeded the team’s progress: Three of its highly motivated members lived three to four hours from Tegucigalpa, precluding attendance at meetings and other participation. The range and number of restraints and difficulties faced by the Honduran team precluded its success; many are beyond the control of those who would plan future exchanges, but such planners should be aware of the potential obstacles. E. NICARAGUA The impact of the Exchange has been considerable in Nicaragua, with implementation of the action plan well advanced at the time of the evaluation. The action plan sought to expand post-partum family planning services in five facilities, including both Ministry of Health (MINSA) and PROFAMILIA facilities in the Integrated Local Health Systems (SILAIS) of Jinotega and Matagalpa. PROFAMILIA carried out baseline contraceptive technology knowledge, attitude, and practice (KAP) surveys of health staff in each of the five facilities: the Jinotega and Matagalpa SILAIS hospitals, Hospital Bertha Calderón in Managua, and the PROFAMILIA clinics in Jinotega and Matagalpa. Family planning content presented 8 · LAC MAQ Exchange Evaluation at the Exchange has been replicated for other clinical staff at both the PROFAMILIA and Matagalpa and Jinotega SILAIS facilities. The Matagalpa SILAIS hospital has implemented the Medical Eligibility Criteria and Pregnancy Checklist, opened a family planning clinic and adolescents’ area (not fully as a result of the action plan), and presented different training workshops for health providers. All hospital staff were trained in the importance of referring women to FP counseling; two training sessions on post-obstetric event counseling had been presented at the time of the evaluation, and a third was planned for the end of March. A physician was hired to manage the FP clinic and provide counseling, and a nurse trained in MAQ concepts/methodology with an emphasis on FP/RH is in charge of monitoring/supervision. Similar workshops have been presented at PROFAMILIA clinics in Managua and Matagalpa. PROFAMILIA has been working with the Jinotega SILAIS hospital to establish a similar system. The family planning clinic at the Jinotega hospital was ready to open at the time of the evaluation. All five clients interviewed at Matagalpa hospital had received counseling on FP and expressed their discontent with the shortage of beds, crowded rooms, and shortage of hospital personnel. Since the clinic has one nurse per 30 women in labor, meeting all their needs is impossible, especially since this nurse is responsible for the high-risk delivery ward as well. The Bertha Calderón Hospital director said that a year before the evaluation, two MAQ team members had visited the hospital and made a presentation of the project, encouraging the hospital to join. The hospital sent a proposal as agreed, but had not received any other communication from the team. On their own initiative, hospital staff used MAQ materials to implement some activities (e.g., a family planning survey) that required no additional funds. Another example of spread of MAQ concepts “on their own” was found in the Chinandega SILAIS Hospital, where the physician who attended the Exchange has applied MAQ ideas in monitoring sexual and reproductive health services. Similarly, the Director of Ipas reports that she uses the MAQ materials in trainings she conducts and that she has replicated information from the Exchange for her colleagues. The MINSA staff member responsible for Integral Care for Women and Adolescents in Managua has joined the MAQ team to facilitate Ministry-based activities. He also expressed interest in replicating the Matagalpa/Jinotega experience at other hospitals. Informant interviews indicated that about 65% of Exchange participants did not know the status of the action plan and aired concerns, noting particularly their lack of awareness of meetings and the baseline results. Two of these respondents were located in rural areas and were not able to attend any meetings, which were all held in Managua. The Director of the Matagalpa Hospital provided some details regarding the use of the seed funds. He was delegated by the Medical Director of PROFAMILIA to oversee MAQ action plan implementation. He reported that some funds had been used to advertise cradles (as opposed to the current situation of mothers and infants sharing a single twin bed) in order to attract clients to the hospital’s FP/RH services. PROFAMILIA had agreed to purchase the cradles, but at the time of the evaluation, they had not been received. More than 80% of the Nicaraguan Exchange participants stressed the value of the MAQ Exchange in terms of knowing “what has been done by whom in each organization working on provision of reproductive health services.” At the same time, some Exchange informants were not satisfied with how the funds were managed or with the activities contemplated under the project. Implementation of the action plan was highly centralized through the team coordinator, who is also the Director of PROFAMILIA and the person who managed the seed funds. Centralization by one institution or individual can undermine attainment of the Exchange’s objective of enabling key actors from key institutions to work cooperatively, improving overall FP/RH programming by sharing expertise, knowledge, and experience. Team members’ roles seem to have been poorly defined, although one LAC MAQ Exchange Evaluation · 9 informant at a meeting with the MAQ group said that the “roles were clearly defined during the Exchange.” Delegation of MAQ functions to other staff who had not attended the Exchange was also a problem in Nicaragua. The role of QAP in supporting MAQ implementation was not very clear as well; the PROFAMILIA Medical Director said that QAP’s role had varied from little to great involvement in different action plan activities. IV. CONCLUSIONS Returning to the objectives of the evaluation, this section addresses the evidence found by the evaluator for impact of the LAC MAQ Exchange. With respect to the effectiveness of the LAC MAQ Exchange design in spreading family planning and reproductive health information and best practices in the five countries, there was a clear consensus among the country participants about the high value of the themes and content of the Exchange and the high quality of the presentations at the meeting in Honduras. Some 90% of the participants reported that both the conceptual frameworks and the methodological tools and supporting materials presented at the Exchange were very useful and could be readily adapted for replication at the country level. Some teams (notably in El Salvador, Honduras, and Nicaragua) replicated the Exchange content in their home organizations, and many informants used the materials in their own activities independent of the teamwork. Many informants mentioned the possibility of “speaking the same language” as one of the benefits of the LAC MAQ Exchange. This reference is important since it makes it possible to establish communication networks that create opportunities for cooperation and collaboration. With respect to progress made by teams in implementing their action plans and seed grant projects, the evaluation found excellent progress and tangible results in El Salvador, good progress in the Dominican Republic and Nicaragua, and limited progress in Guatemala and Honduras. The reasons for this differential result are many and relate in large part to the political, organizational, and cultural context unique to each country. In the cases of Honduras and Guatemala, the Ministry of Health representatives left their positions or changed functions. This radically affected the implementation of the action plan proposed, since it was understood that those representatives would act as champions and facilitators of the proposal within the MOH. One of the factors that seemed to constrain the participation of all country team members was the selection of a topic that was more narrowly focused on the interests of one institution. Also, many of the MAQ participants simply had an already very full work agenda and very limited time to take on new responsibilities. El Salvador was basically the only team that devoted a large number of hours (i.e., time away from their other work and personal activities) to MAQ-related activities. While turnover and time constraints were problematical for all five teams, the strong team process instituted in El Salvador served as a countervailing force. The El Salvador team as a whole demonstrated greater internal motivation to pursue the implementation of their action plan than did the other teams. The additional resources that were available in El Salvador to fund the action plan coming out of the Exchange may also have been important in facilitating the team’s success. Other important obstacles to progress in action plan implementation were the short time (12 months) designated for this activity and the impossibility of having all members of the group meet regularly due to geographical distance. It must be considered that mobilization for MAQ team meetings represents an extra cost that not all people can afford or want to spend. 10 · LAC MAQ Exchange Evaluation Technical assistance was also unclear to the participants, as well as who was in charge of supervising the process (USAID, QAP, or FHI).2 The majority of key informants expressed a need to have an “official” person designated to monitor and accompany the MAQ team in terms of proposal implementation, calls for meetings, and similar issues. This was the role played by a USAID staff member in El Salvador, and to a lesser extent by FHI-CONECTA staff in the Dominican Republic. Finally, one of the key objectives of the MAQ Exchange was to act as a promoter of inter-agency coordination among the key actors in the area of reproductive health in each country. The evaluation found that the Exchange had stimulated communications and the development of professional networks to varying degrees in all five countries. This process was most clearly visible in El Salvador, where each Exchange participant contributed his/her knowledge and specialization area to the collaborative development of the training manual. The fact that gender was included as a cross-cutting issue in the manual is another example that highlights the transfer of ideas facilitated by the Exchange in El Salvador. Centralization of decision-making and control did constitute an obstacle for the interaction of the groups, particularly in the Dominican Republic and Nicaragua, since it tied the information as well as the decisions regarding implementation and development of the action plan to one person or institution. With respect to fostering dialogue and collaborative action among distinct actors, it is difficult to separate the impact of the Exchange from that of pre-existing working relationships and coordination mechanisms and organizational culture in each country. Organizational culture can be a restraining (or enabling) factor and is something USAID cannot control but should certainly be aware of and try to influence. The evaluation found numerous instances of coordination weakened by over-centralization, both by Ministries of Health and NGOs. Working through trusted cooperating agencies/NGOs with similar agendas and that have been directed to share information and power is one approach. Some informants also suggested that USAID play a more active role, in conjunction with FHI, throughout the implementation of the country plans.3 V. RECOMMENDATIONS FOR FUTURE EXCHANGES This section extrapolates from the findings the strengths and weaknesses of the exchange format in terms of its ability to foster both project implementation and interagency cooperation. Recommendations to strengthen future exchanges are grouped to address the following aspects of an Exchange: 1) selection of participants; 2) technical content of the Exchange; 3) selection of action plan topics; 4) monitoring of action plan implementation; and 5) management of funds. Participant selection Planners of future exchanges seeking to avoid turnover-related weaknesses could consider inviting several staff members from lead organizations so that if someone leaves, others can take over action plan responsibilities. Content of the Exchange USAID should consider adding a session to the Exchange to strengthen team building and to clarify the role of team members in implementing any action plan developed. In future Exchanges, the El Salvador experience could be related as a model for those who are selecting and implementing action plans. Of particular import were that the plan addressed an issue within the scope of the participants and their organizations. Secondly, the team’s behavior was exemplary in terms of both organizing itself and proceeding with the work plan despite whatever disagreements it encountered. 2 QAP was given additional MAQ funding in FY03 specifically to provide support and follow-up to the country teams. 3 FHI was tasked by USAID to take the lead in managing the small grants. LAC MAQ Exchange Evaluation · 11 Team members’ roles also need to be clearly defined during the Exchange. Should time run short, members should at least commit to perform this step at the earliest possible opportunity. Who will monitor progress and manage finances are particularly important issues to address in this step. The evaluation did not examine the extent to which sufficient time was allotted to teams to define members’ roles, but those teams that did so had better results: Planners of future exchanges should try to build in enough time for this step. Another topic to consider adding is an exercise that would help people internalize the value of diversity. While the El Salvador team appreciated and benefited from diversity among its members, other country teams did not, suggesting that the Exchange format might gain by highlighting the value of diversity. Identification of Action Plan Topics It is necessary to consider the internal political structure of the country and its dynamics when selecting topics for action plans or grant proposals, since these may hinder the possibilities of carrying out particular proposals. The public health sector decides on overall policies and interventions in each country and works with international donors and NGOs; thus, public sector representatives tend to exert strong influence over the selection of action plan topics. But to maintain team member commitment to action plan implementation, action plans need to reflect both the country’s and team members’ priorities, and there should be some overlap between action plan activities and the participants’ professional responsibilities. If the topic of the proposal is not directly linked with the activities of the participants involved, it will be difficult for them to find extra time to devote to that particular project, considering their already overburdened work schedules. If the Ministry of Health is not leading the action plan topic, the team needs to take steps early in the planning process to assure MOH buy-in and support for the action plan. Additional steps include maintaining professional relations with the MOH authority with oversight for the action plan topic and keeping that office well informed of all progress. Facilities that are selected to serve as implementation sites for any action plan must be informed and involved in the planning. Monitoring of Implementation After implementation of action plans has begun, a monitoring mechanism needs to be built into the funding/resource allocation process to ensure the continuation of teamwork. For instance, Exchange participants could be surveyed by email to ask whether they are still participating on the team and whether they feel their involvement is worthwhile. Tying survey results to funds release would prevent centralization of information, funding, and decision-making. Respondents suggested that monitoring should be carried out by a specific actor with sufficient time available for that purpose. Funds Management Transparency with regard to the management of funds engenders more favorable working relationships. The processes of how the funds will be released, which organization will administer the funds, how participating organizations can access funds, etc. should be part of the Exchange learning sessions. USAID or its designee could continuously monitor fund disbursement and share information with all participants. 12 · LAC MAQ Exchange Evaluation APPENDICES APPENDIX A: LAC MAQ EXCHANGE PARTICIPANTS BY COUNTRY APPENDIX B: DATA COLLECTION INSTRUMENTS APPENDIX C: AUTHOR’S NOTES ON DATA COLLECTION BY COUNTRY LAC MAQ Exchange Evaluation · 13 APPENDIX A: LAC MAQ EXCHANGE PARTICIPANTS BY COUNTRY COUNTRY PARTICIPANT/ORGANIZATION Dominican Republic José Gregorio Aponte Romero, ADOPLAFAM Victor Calderón, SESPAS Carmela Cordero, EngenderHealth Milton Cordero, Sociedad Dominicana de Ginecología y Obstetricia Manuel Costa Polanco, DPS, La Vega José Figueroa, Sociedad Dominicana de Ginecología y Obstetricia Timoteo Fulgencio, DPS La Romana Amaury Guillén, Hospital Maternidad Altagracia Hilario Reyes Pérez, Hospital Maternidad Los Mina Dolores Rodríguez Lappot, PROFAMILIA Milady Román, Instituto Dominicano de Seguridad Social David Losk, USAID Elba Mercado, USAID El Salvador Mirna Elizabeth Argueta Martínez, AMS Samuel Castor Gonzales, Asociación Demográfica Salvadoreña Ena Cordón, Universidad de El Salvador Eduardo Córdova Macías, Asociación de Ginecología y Obstetricia de El Salvador Douglas Jarquín, PRIME Guillermo Mata, Colegio Médico Doris Aída Montenegro Soria, CEMUJER Consuelo Olano de Elías, MSPAS Haydée Padilla de Escobar, MSPAS Marina Padilla de Gil, Instituto Salvadoreño del Seguro Social Claudia Elizabeth Rodríguez Brito, Instituto Salvadoreño para el Desarrollo de la Mujer Maricarmen Estrada, USAID Guatemala Zonia Aguilar, APROFAM Mirna Patricia Barahona Villatoro, MSPAS Yadira Villaseñor de Cross, JHPIEGO Telma Duarte, APROFAM Ervin Herman Kiesling Alvarado, MSPAS María Eugenia de León de Monroy, MSPAS Carlos Fernando Marroquín Vásquez, AGOG Lucia Merino, Proyecto Policy II Loyda Brown Montenegro, Asociación Guatemalteca de Mujeres Médicas Edwin Montúfar, Calidad en Salud Velia Lorena Oliva Herrera, MSPAS, Escuintla Lucrecia Castillo, USAID Lucrecia Peinado, USAID 14 · LAC MAQ Exchange Evaluation Honduras Elida Rosa Aguilar Norma Aly, Quality Assurance Project María de los Angeles Barahona, ASHONPLAFA Sadith Cáceres, PRODIM Pablo Dominguez, ASHONPLAFA Gustavo Flores, PHRplus Project Ivo Flores, Secretariat of Health Rosa Marlen Flores, EngenderHealth María Elena Guevara Ana Ruth Gutiérrez, Secretariat of Health Carmen Lobo de Lanza Laura Martínez, Secretariat of Health Cecilia Maurente Erwin Ochoa, JHPIEGO María Elena de Pérez, ASHONPLAFA Olga Portillo, Secretariat of Health Alba Lidia Sánchez, Secretariat of Health Angel Coca, USAID María del Carmen Miranda, USAID Ernesto A. Pinto, USAID Nicaragua Ovidio Samuel Blanco Castillo, PRIME Marta María Blandón, IPAS María Esther Estrada, SILAIS Chinandega Diony Javier Fuentes Cortés, Management Sciences for Health/PROSALUD Ximena Gutiérrez Gómez, Johns Hopkins University Carlos José Jarquín González, PROFAMILIA María Lourdes Martínez Morales, MINSA Yadira Tinoco Martínez, SILAIS Estelí Tomasita Medina, Universidad Autónoma, Managua Luis Manuel Urbina Tellez, Quality Assurance Project Roberto Pao Kraudy, Red NICASALUD Luis Yescas, MINSA Alonzo Wind, USAID LAC MAQ Exchange Evaluation · 15 APPENDIX B: DATA COLLECTION INSTRUMENTS Instrument #1: Individual Interview Guide Background 1. Name: Institution: Brief description of responsibilities in case your work/activity has changed due to sub￾regional Exchange: 2. Did you participate in the sub-regional MAQ Exchange that took place in April 2002 in Honduras? (If yes, go on with question 3-, if not, question 4) 3. (If your answer was affirmative) How did you get involved in MAQ Exchange as a participant? 4. (If your answer is negative) How did you get involved in the MAQ Exchange Follow-up activities? MAQ Exchange Impact 5. Once the team was formed, were roles allocated allowing to know clearly who would be in charge of the notification, funding and related matters? 6. How was the proposal topic decided? 7. Were you involved in the proposal’s preparation or action plan? Did you agree on the topic? 8. Otherwise, have you read that proposal? Did you participate by providing ideas and comments during its revision? 9. Describe the aims and activities of the country proposal. 10. Which is the organization that is leading the proposal implementation? Why? 11. What was your involvement in the follow-up activities after the Exchange? 12. Can you give us your opinion about MAQ contents, the Exchange, the approach as it is? 13. MAQ Information that you consider most useful in your work as part of the organization you are representing to. 14. Do you consider that the collected information has been helpful for you and your organization? If your answer is affirmative, Could you give more information? If your answer is negative, Could you give more information? 15. Could you provide some examples about how you used MAQ information in your work? 16. Could you mention some experiences, methodologies, etc, learned from colleagues that participated in MAQ (if it is the case)? Were you be able to adapt that knowledge to your own work? How? 16 · LAC MAQ Exchange Evaluation 17. Can you mention some activities, changes, improvements or related issues that have occurred in your organization because of the Exchange, out of the proposal’s results? 18. What was your involvement in these activities? Proposal, Implementation, Content 19. Is the proposal linked to your work area so that time investment in this activity can be useful to your activities within the organization? 20. Which activities from the proposal have been implemented? 21. How do the implemented activities differ from the proposal? Why do they vary? 22. What were the obstacles in the implementation of the proposal? 23. What actions, activities and related issues were easy to implement in the proposal? Why do you think, they were easier to implement? 24. Could you point out actions, activities or similar performances that were difficult to carry out in the implementation of the proposal? Why? 25. According to your opinion, does the proposal match the work team priorities? If your answer is negative, why? 26. Do you consider that the proposal complies with the priorities of your organization? 27. Do you consider that the proposal complies with the priorities of your country? 28. According to your point of view, is there any change in the quality or quantity of service as a proposal result? If your answer is affirmative, which are those changes? 29. If your answer is affirmative, How are these changes reflected? 30. Do you consider that the concepts that MAQ manages are applied to the socio-cultural context of the country? LAC MAQ Exchange Evaluation · 17 Instrument #2: Group (Country Team) Interview Guide Background 1. Name: Institution: Brief description of responsibilities in case your work/activity has changed due to sub￾regional Exchange 2. Did you participate in the sub-regional MAQ Exchange that took place on May 2002 in Honduras? (If yes, go on with question 3-, if not, question 4) 3. (If your answer was affirmative) How did you get involved in MAQ Exchange as a participant? How was your involvement in the follow-up activities to the Exchange? 4. (If your answer is negative) How did you get involved in the MAQ Exchange Follow-up activities? 5. Had some of you worked conjointly before to MAQ Exchange? If your answer is affirmative, which way? Proposal and Implementation 6. Did the work team have meetings during last year to modify, develop and monitor the implementation of the work plan? If yes, how many times did you get together? Who participated. If not, can you give more information? 7. Was there any instance (for example, during a group meeting) when the team felt “deadlocked” or it did not know clearly how to go on? If yes, how did you solve the problem? 8. How and how much did the country coordinator support the implementation of the plan? 9. Does your work group have the participation of a USAID representative at any time? 10. Is the work team implementing the proposal? Describe public institutions and NGOs that have had an active role in the implementation of the action plan. Include technical assistance, financial contributions, staff, resources, etc. 11. What activities have been implemented up to now? Which ones were easy to implement? 12. What do you consider was the most difficult to implement? Why? How did, or would, you solve these difficulties? 13. Have the proposal’s plans changed as a result of the implementation? Which way? 14. In terms of inter-agency work, which factors do you consider have been successful? Which ones rewarding? Which ones were challenging? 15. Up to the moment, what has the impact of the proposal implementation been? Which changes have taken place? Is there collected information to document changes? If your answer is affirmative, which are the results? 18 · LAC MAQ Exchange Evaluation Exchange Impact 16. Have you started to work conjointly besides the implementation of the proposal? What were these exchanges, meetings, etc. about? 17. Has the Inter-agency communication changed some way due to the cooperation, result of the proposal implementation? 18. Are there initiatives on policies or leadership that could be directly or indirectly attributed to MAQ Exchange? 19. Have there been replications of MAQ Exchange in a whole or a part in your country since April 2002? 20. Have MAQ materials been used or/and adapted? If yes, which way? By whom? In which context? 21. Have you had communication with other team members since MAQ Exchange? If yes, which was the purpose or content of that cooperation? Content 22. Within MAQ content, which concepts have been useful for the proposal within the socio-cultural context of your country? 23. According to your opinion, were there concepts within MAQ that have hinder its implementation? LAC MAQ Exchange Evaluation · 19 Instrument #3: Written Questionnaire (for Individuals Who Could Not Be Interviewed) Background 1. Name: Institution: Brief description of responsibilities in case your work/activity has changed due to sub￾regional Exchange 2. Did you participate in the sub-regional MAQ Exchange that took place on May 2002 in Honduras? (If yes, go on with question 3; if not, go to question 4) 3. (If your answer was affirmative) How did you get involved in MAQ Exchange as a participant? 4. (If your answer is negative) How did you get involved in the MAQ Exchange Follow-up activities? Proposal and Implementation 5. Were you involved in the preparation of the action plan/proposal? If not, did you have input into its revision? 6. Otherwise, have you read that proposal? Did you provide comments, ideas during its revision? 7. Briefly describe the aims activities of your country proposal? 8. Which is the organization that is leading the proposal’s implementation? Why? 9. Which activities have been implemented from the proposal? 10. How do the implemented activities differ from the proposal? Why do they vary? 11. Can you point out some actions, activities or similar performances that were easy to implement in the proposal? Why do you think they were easier to implement? 12. Could you point out actions, activities or similar performances that were difficult to carry out in the implementation of the proposal? Why? 13. According to your opinion, does the proposal match the work team priorities? If your answer is negative, why? 14. Do you consider that the proposal complies with the priorities of your organization? Which way? 15. According to your point of view, is there any change in the quality or quantity of service as a result of the proposal? If your answer is affirmative, which are those changes? If your answer is affirmative, proceed to question 16. If it is negative, proceed to question 17. 16. If your answer is affirmative, explain why you consider these changes positive. 17. If your answer is negative, to what do you attribute this lack of effect? 20 · LAC MAQ Exchange Evaluation Exchange Impact 18. What is your general impression of the Exchange? Which were its outstanding, less positive, and/or negative aspects? 19. How have you used MAQ Exchange information in your own work? Have you or your organization shared formal or informally this information with other people or organizations? 20. Have you used the tool set from the quality improvement framework to identify other actions that led to other improvement activities? If your answer is affirmative, what was the result? 21. (If you did not participate in MAQ Exchange) what did you learn from your colleagues that participated in the workshop? How have you used this information in your own work? 22. Which activities have you started in your organization out the proposal itself? LAC MAQ Exchange Evaluation · 21 APPENDIX C: AUTHOR’S NOTES ON DATA COLLECTION BY COUNTRY El Salvador The field work in El Salvador included several interviews with key informants as well as visits to beneficiary population and field coordinators who are actively implementing the initial proposal in Santa Ana and San Miguel. The group interview had the participation of five out of twelve team members, who contributed their team experience and knowledge from the MAQ Exchange. Additionally, ASOGOES had prepared a presentation that described the different meetings and workshops that the group has had to develop and follow-up the training curriculum and job aids that it prepared for different medical and nursing institutions and universities. In-depth interviews with team members helped me to formulate some hypotheses related to key questions of the evaluation. I also had the opportunity to visit and talk to a variety of key informants and beneficiary population of the project. This allowed more direct contact with the population, to gather information on how they understand and apply the different activities that the project includes. The visit to the Santa Ana University, located in the rural zone of El Salvador, showed the motivation, interest, and commitment of the actors of that institution, whose efforts linked the director level to the community. Santa Ana University (UNASA) As a direct result of the MAQ Exchange, the head of the School of Medicine UNASA arranged a workshop of faculty to identify problems in the delivery of sexual and reproductive health services and determine the appropriate profile of healthcare professionals in this practice area. Afterwards, a plan was designed to introduce the MAQ team’s curriculum. A second meeting formed work teams in different areas, such as a hospital team, a former student team, university teaching staff, and teaching staff from other training universities. Another activity resulting from the MAQ Exchange was the training of youth educators who could transmit the MAQ content. With regard to the need to have authorization from the Ministry of Education to change the curriculum, the director of the School of Medicine at UNASA mentioned the plan of the Ministry of Education to carry out a curricular reform in 2005. This could be the starting point for new objectives within the current proposal. The MAQ team handbook was found at both the school library and the general library for use by teaching staff, students, and others. The UNASA professors interviewed, both women, are using the handbook in their classes, stimulating student discussion and interest. Both referred to the emphasis they place on women as regulators of their pregnancy and on the need for the inclusion of men in ensuring sexual and reproductive health. San Pedro de Usulután Hospital While the director of San Pedro de Usulután Hospital did not participate in the MAQ Exchange, the team invited him to participate in action plan activities and provided training on MAQ content. In visiting the hospital, I witnessed significant changes for teens in labor. Formerly, the signs that hung over their beds had only numbers and symptomatic records. Now, the signs include the patients’ names, and names are used in speaking to patients, fostering a more respectful, cordial, and affectionate relationship between doctor and patient. 22 · LAC MAQ Exchange Evaluation Specialized Institute for Health Professionals (IEPROES), San Miguel I had the opportunity to attend a “Maximizing Access and Quality” workshop in San Miguel. This conference was organized by the Instituto Especializado de Educación Superior de Profesionales de Salud de El Salvador – IEPROES (Specialized Institute for Higher Education of Health Professionals from El Salvador) and the PRIME II Project. Informants, a nurse and an IEPROES teacher, discussed the first sensitization workshop they had organized: It sought to incorporate MAQ concepts into the Institute’s practices and to replicate them for students and educational partners. The training curriculum and the materials provided by the MAQ team are being used by teachers within their weekly subject program. San Miguel is highly motivated to pursue a sensitization approach in treating patients and to implement teen counseling. I observed that several of the Exchange topics have been incorporated into the IEPROES training curriculum. The San Miguel MAQ coordinator said that the transparencies and audiovisual aids distributed at the MAQ Exchange motivated the students to learn. She also noted the materials’ value, since teachers no longer had to prepare aids themselves. She noted that the materials were particularly useful in addressing gender issues, such as including men in sexual and reproductive health. She also credited the MAQ team with a change whereby women are accompanied during labor pain, when no support had been provided before. Soon after the training, a team was formed to monitor both the mother and newborn baby with monitoring charts. Additionally, the director did a presentation about the received training, its impact at different levels of the institution, and multiple ideas arose from this training. Honduras The field work in Honduras included a variety of interviews with key informants and a visit to the Secretariat of Health (Secretaría de Salud). The QAP office was the main meeting place. The group survey had the participation of six out of 16 members who contributed their knowledge and experience as a MAQ team. Additionally, I was able to carry out in-depth interviews with five key informants, four of them having participated in the Exchange and having had decisive roles in the follow￾up of the initial proposal. The fifth person interviewed was the head of Maternal and Child Health division of the Secretariat of Health, who serves as coordinator of the action plan although he did not participate in the Exchange. There were people whom I did not have the opportunity to interview due to their location or their having retired. One of the proposed interviewees had even completely changed her work area and organizational affiliation. Dominican Republic The field work in the Dominican Republic included a variety of interviews with key informants and visits to Los Mina Maternity Hospital, Francisco Gonzalvo Hospital in La Romana and ADOPLAFAM. The CONECTA office was the main meeting place. The group survey had the participation of 8 out of 11 original members as well as the CONECTA coordinator and the director and the head of the Department of Gynecology and Obstetrics of the Francisco Gonzalvo Hospital. The latter three informants, though not participants in the MAQ Exchange, were later trained in the MAQ methodology and joined the team. Additionally, I had the opportunity to have in-depth interviews with services providers at ADOPLAFAM. LAC MAQ Exchange Evaluation · 23 Nicaragua The field work in Nicaragua included a variety of interviews with key informants, with the QAP office as the main meeting place. The group survey had the participation of 3 out of 15 original members, as well as participants from MINSA, NICASALUD, USAID, and Profamilia who were delegated to participate in place of the original representatives, either as a result of change of duties or moves to other organizations. Additionally, I interviewed the Director of PROFAMILIA, the organization in charge of handling the funds. I also visited a PROFAMILIA Clinic in Matagalpa, where I had the opportunity to interview the director and the person in charge of counseling. We also visited Matagalpa Hospital, where we had meetings with the director. We were able to carry out a group survey with the users of sexual and reproductive health services. Also, we had the opportunity to conduct interviews with the chief of the Family Planning/Puerperium/Post-partum ward, the MAQ coordinator, and the chief of nursing. Finally, we were also able to visit the national referral hospital, Hospital Bertha Calderón. The opportunity to have had access to information on the part of these diverse actors provided better information for the analysis. Guatemala The field work in Guatemala included interviews with key informants of the original MAQ group as well as participants that are currently working in the new MAQ group. The group survey had the participation of 3 out of 11 original members. Another MSPAS staff member that had been trained in MAQ methodology but not attended the Exchange was also present. Additionally, I had the opportunity to interview a technical staff member of the Maternal and Neonatal Health component of the Reproductive Health Program Implementing Unit, who is actively involved in the re-activation of the proposal. I also conducted a phone interview with one MAQ Exchange participant from USAID. QUALITY ASSURANCE PROJECT University Research Co., LLC 7200 Wisconsin Avenue, Suite 600 Bethesda, MD 20814 USA Tel: (301) 654-8338 Fax: (301) 941-8427 www.qaproject.org