EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE July 2007 This publication was produced for review by the United States Agency for International Development. It was prepared by Stanley Terrell and Pola Brenner. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE DISCLAIMER The views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 07-001-007) is available in printed or online versions. Online documents can be located in the GH Tech web site library at www.ghtechproject.com/resources.aspx. Documents are also made available through the Development Experience Clearing House (www.dec.org). Additional information can be obtained from The Global Health Technical Assistance Project 1250 Eye St., NW, Suite 1100 Washington, DC 20005 Tel: (202) 521-1900 Fax: (202) 521-1901 info@ghtechproject.com This document, GH Tech Report Number (07-001-007), was submitted by The QED Group, LLC, with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. GHS-I-00-05-00005-00 ACKNOWLEDGMENTS The evaluation team acknowledges and thanks the many people and organizations that contributed, directly and indirectly, to this assignment. All involved with the AMI and SAIDI Initiatives were most cooperative, enthusiastic, and willing to dedicate whatever time was requested of them to respond to queries and requests for information. It is therefore difficult to single out any individual or organization. However, very special thanks are offered to: • Peg Marshall, USAID/LAC/RSD, and Jaime Chang, USAID/Lima, for their continuous guidance and feedback, and Trenton Ruebush, USAID/GH/HIDN/ID and Susan Bacheller, USAID/GH/HIDN/ID for providing important background context. • Maria Miralles, MSH RPM Plus, for her general support, and Melissa Thumm, AMI, and Beth Yeager, SAIDI. • Keith Carter, Lourdes Barrios, and Rainier Escalada, PAHO/Washington, Roberto Montoya, PAHO/Brazil, and Jose Pablo Escobar, PAHO/Colombia for their help with AMI, and Gabriel Schmunis and Maria de la Paz Ade, PAHO/Washington, and Marco Fidel Suarez, PAHO/Bolivia, for their help with SAIDI. • Victor Pribluda and Adrian Barojas, USP/DQI, for their assistance with both AMI and SAIDI. • Alexandre de Macedo and Ray Beach, CDC, for support with AMI and William Mackenzie, CDC for SAIDI. • Marisabel Sanchez, Links Media, for support and information on SAIDI. • Anibal Sosa, APUA, for information related to SAIDI, • Jan Karlo Zavalaga, DIGEMID Peru, and Rossana Geng, Proyecto Vigia Peru, for coordinating the interviews and the evaluation process and providing information about activities in Peru. • Graciela Gamarra, MoH/Paraguay, for coordinating the interviews and the evaluation process and providing information about activities in Paraguay. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE iv ACRONYMS AND ABBREVIATIONS ACT Artemisinin-based combination therapy AIDS Acquired immunodeficiency syndrome AIS Acción Internacional para la Salud (International Action for Health) AMB Antimicrobial AMI Amazon Malaria Initiative AMR Antimicrobial resistance APUA Alliance for the Prudent Use of Antibiotics CDC U.S. Centers for Disease Control and Prevention CEMIT Centro Multidisciplinario de Investigaciones Tecnológicas (Multidisciplinary Center of Technological Research) CIM Centro de Información de Medicamentos (Drug Information Center) CONCAMYT Control de Calidad de Medicamentos y Toxicología (Quality Control of Medicines and Toxicology) CNCC National Center for Quality Control, National Health Institute, Peru DIGEMID Dirección General de Medicamentos, Insumos y Drogas (Directorate of Medicines, Supplies, and Drugs) DISA Dirección de Salud (Directorate of Health) DRA Drug Regulatory Authority GF/GFATM Global Fund to Fight AIDS, Tuberculosis, and Malaria GH TECH Global Health Technical Assistance Project IC Infection control IR Intermediate result IT Information technology LAC Latin America Caribbean LM Links Media MDR Multi–drug-resistant MoH Ministry of Health MSH Management Sciences for Health NGO Nongovernmental organization NI Nosocomial infections NMP/NMCP National Malaria Control Program OTCA Organización del Tratado de Cooperación Amazónica PAHO Pan American Health Organization PAMAFRO Global Fund Amazon Basin Malaria Project PCR Polymerase Chain Reaction PEPFAR President’s Emergency Plan for AIDS Relief PHN Population, Health and Nutrition PMM Pharmaceutical Management of Malaria QA Quality assurance EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE v QC Quality control RAVREDA Amazon Network for the Surveillance of Anti-Malarial Drug Resistance RBM Roll Back Malaria RDT Rapid Diagnostic Test RPM Plus Rational Pharmaceutical Management Plus RSD Office of Regional Sustainable Development SAIDI South American Infectious Disease Initiative SC Steering committee SIAMED Model System for Computer-assisted Drug Registration SOP Standard operating procedures S-P Sulfadoxine-Pyrimethamine TA Technical assistance TB Tuberculosis TDR UNICEF/UNDP/ World Bank/WHO Special Programme for Research and Training in Tropical Diseases UNIMED Unidad de Medicamentos (Medicine Unit) USAID United States Agency for International Development USP/DQI United States Pharmacopeia / Drug Quality Information Program WHO World Health Organization EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE vi CONTENTS ACKNOWLEDGMENTS ................................................................................................................ iv ACRONYMS AND ABBREVIATIONS............................................................................................ v EXECUTIVE SUMMARY................................................................................................................. 1 INTRODUCTION ............................................................................................................................. 4 METHODOLOGY............................................................................................................................ 5 AMAZON MALARIA INITIATIVE.................................................................................................... 7 Background: Malaria Control and Prevention in the Amazon Basin................................... 7 The Amazon Malaria Initiative (AMI)........................................................................................ 8 AMI Findings............................................................................................................................ 11 AMI Conclusions/Lessons Learned ...................................................................................... 50 AMI Recommendations .......................................................................................................... 50 THE SOUTH AMERICAN INFECTIOUS DISEASE INITIATIVE (SAIDI)..................................... 53 Background ............................................................................................................................. 53 Findings ................................................................................................................................... 56 SAIDI CONCLUSIONS AND LESSONS LEARNED.................................................................... 63 The Model................................................................................................................................. 63 SAIDI Recommendations ....................................................................................................... 65 ANNEX 1: SCOPE OF WORK...................................................................................................... 66 ANNEX 2: REFERENCES—DOCUMENTS ................................................................................. 75 ANNEX 3 DISCUSSION GUIDES................................................................................................. 84 ANNEX 4 PERSONS CONTACTED............................................................................................. 93 Persons Interviewed by Country—AMI................................................................................. 93 Persons Interviewed by Country—SAIDI.............................................................................. 96 INTERNATIONAL PARTNERS .................................................................................................... 99 USAID MISSIONS......................................................................................................................... 99 FIGURES Figure 1. AMI Management Structure........................................................................................ 20 Figure 2. Conceptual Model of Drug Use Interventions .......................................................... 55 EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE vii TABLES Table 1. AMI Interviews by Country............................................................................................. 5 Table 2. SAIDI Interviews by Country.......................................................................................... 6 Table 3. Results Frameworks for Amazon Malaria Initiative..................................................... 9 Table 4. AMI Partner Agencies and Their Roles....................................................................... 10 Table 5. AMI Counterpart Contributions, 2002–2007 ............................................................... 16 Table 6. Outcomes by Work Area .............................................................................................. 22 Table 7. Amazon Malaria Initiative: Tools and Protocols Implemented................................. 24 Table 8. Outcomes by Intermediate Result............................................................................... 43 Table 9. SAIDI Focus by Country............................................................................................... 56 EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE viii EXECUTIVE SUMMARY The U.S. Agency for International Development (USAID) Latin American and Caribbean Bureau (USAID/LAC) requested that the USAID/Washington GH TECH Project conduct an external evaluation of the Amazon Malaria Initiative (AMI) and the South American Infectious Disease Initiative (SAIDI). USAID/LAC will use the results of this evaluation to 1. Determine whether funding for AMI and SAIDI should be extended or a new regional infectious disease initiative should be designed. 2. Assess the progress of AMI and SAIDI toward achieving their expected results. 3. Document lessons learned from implementing the management model used for coordinating both projects. The GH TECH Project contracted with a two-person evaluation team: a team leader/malaria program specialist to evaluate AMI and a specialist in antimicrobial resistance (AMR) to evaluate SAIDI. The evaluation was conducted from March through June 2007. Along with the scope of work (Annex 1), the GH TECH Evaluation Team received a considerable bibliography of work plans, project reports, trip reports, and presentations on a CD ROM, which they supplemented during the evaluation (Annex 2). The team clarified points about the methodology in teleconferences and direct communications with USAID officials and met with or had telephone communications with all the AMI and SAIDI partners. The team drafted discussion guides that were vetted by USAID personnel and revised after the first team planning meeting and a round of field interviews in Peru (Annex 3). The AMI evaluator made visits to three of the partners in Washington (the Pan American Health Organization, PAHO; Management Sciences for Health, MSH; and U.S. Pharmocopeia, USP) and contacted the other partner, the Centers for Disease Control (CDC), by telephone. He conducted interviews in Peru, Colombia, Brazil, and Suriname, and interviewed informants by telephone in Guyana, Bolivia, Ecuador, and Venezuela. The AMI contacts were coordinated through PAHO except for Peru, which was coordinated by the USAID/Lima Mission (Annex 4). In addition to telephone interviews with US-based partners, the SAIDI evaluator visited Peru, Bolivia, and Paraguay. MSH coordinated the visits in Peru and Paraguay and PAHO coordinated the Bolivia visit. The SAIDI evaluator interviewed 62 people, most in person (Annex 4). USAID/LAC has supported the Amazon Malaria Initiative since 2001. Target countries for the initiative were Bolivia, Brazil, Colombia, Ecuador, Guyana, Peru, Suriname, and Venezuela. Coordinated by the PAHO, AMI has helped countries to analyze drug resistance study findings and has implemented new treatment guidelines, provided technical assistance (TA) and equipment for monitoring drug quality, assisted in managing supply chains from acquisition through distribution and use, trained program managers, and adapted vector control interventions to the malaria situation in the Amazon region. The SAIDI was formed in 2003 to focus efforts on slowing AMR by improving availability and appropriate use of good-quality antibiotics. Coordinated by MSH, SAIDI is piloting community￾based AMR programs to help participating countries find local approaches to contain AMR, tailored to each country’s unique needs. Target countries for the initiative are Bolivia, Paraguay, and Peru. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 1 AMI and SAIDI present interesting comparisons. They both operate in numerous countries in partnership with different agencies blending complementary skills to a common end. Both projects demonstrated considerable flexibility in their development, capitalizing on the considerable expertise and experience of partners to help shape the initiatives rather than presenting them with a series of closed-end tasks. While this approach requires more time, it clearly enriched both initiatives and allowed them to take an integrated approach that avoided duplication of efforts. Both initiatives also displayed flexibility and responsiveness to individual country needs and to the extent possible adopted a bottom-up approach. AMI built its efforts around a subregional network that had emerged to deal with the problem of resistance to antimalarial drugs; considerable, though not coordinated, research had already been done before the initiative. AMI had natural counterparts within the malaria control programs of each country. SAIDI, dealing with a more generalized problem, had to engage in awareness and coalition-building exercises before getting to the planning stage. In Bolivia it did not really have a national counterpart. The need to go through these exercises, which was the correct approach, extended the length of time SAIDI needed before it could enter the planning and implementation phases. SAIDI takes an interdisciplinary and holistic ecological approach in which the community is considered as an ecosystem and the human, institutional, and systemic factors that influence AMR are taken into account. Box 1 Value of a Subregional Approach The vector, host, and agent do not respect national boundaries. Exchanges and participation in regional meetings motivate nationals to perform well in order to stand out among their peers (“a healthy competitive effect”). Subregional training, TA, and the development of guidelines and protocols provide economies of scale. Replication of research and studies in multiple sites using common protocols allows for country comparability and provides a critical mass of useful information. Comparable epidemiological and entomological information increases the knowledge base and makes possible better decisions. The approach makes it possible to attack cross-border problems in a coordinated fashion (e.g., gold miners in Brazil, Suriname, and Guyana). It provides a platform for resolving cross-border issues and supports smaller countries as they make their case to larger neighbors. Situational Determinants Favoring a Subregional Approach Countries clearly recognize that they have a common problem and there is a need for and advantages to a subregional approach. A regional entity that has the capacity to convene the relevant actors greatly facilitates the process. Participating countries have the capacity to benefit from the technical assistance. There is a previous history of subregional collaboration and experience in the thematic area to build on. There are few or no single-country programs in the subregion that have the critical mass for reaching similar outcomes. Countries understand both their needs and the value they bring and are prepared to respect those of their neighbors. AMI was a true subregional initiative with subregional training, workshops, meetings, coordination in joint protocol development, information sharing, and a considerable amount of South￾South cooperation. SAIDI evolved more as parallel country programs but has some elements that could be used as the basis for a subregional approach. The AMI subregional model is replicable elsewhere, taking into account facilitating EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 2 factors identified earlier (Box 1). The partnership approach employed by both initiatives (Box 2) is replicable whenever there is a problem that is best addressed through a multidisciplinary approach—which in public health is most problems. Users of this approach should be aware that extra development time and flexibility are needed to take full advantage of partner experiences and capabilities. Box 2 Partnership Approach Advantages: Multiple experiences, from both within and outside the subregion, have clear benefits. Complementary skill sets contribute to a systems approach to problem-solving. A healthy tension between different viewpoints (e.g., scientific rigor versus practical applicability) leads to better products. Qualifiers: A steering committee is essential to balance different points of view, maintain transparency, and keep activities on track. It is also essential to have a single partner act as interlocutor with countries. Seeking consensus (“shared vision”) lengthens but enriches the planning and implementation process. The roles for the different agencies must be carefully defined. Programming and scheduling can be complicated, particularly if personnel have multiple other commitments outside as well as within the region. AMI has largely resolved the problem that it was originally designed to address: the need for comparable information to support evidence-based policies for effective therapeutic treatment of uncomplicated P. falciparum malaria. Support for improving the management and quality assurance of antimalarials to effectively implement treatment policies continues, as does monitoring their efficacy. However, AMI has also branched out into the control area and through its strategies for planning local control interventions appears to be moving toward the SAIDI “ecological” model. AMI deserves continued support, not only for what it has so far achieved but for its potential to help further reduce malaria in the Amazon area. SAIDI is an important attempt to apply a comprehensive and innovative systems model to an emerging problem. The SAIDI partners have taken the correct approach in gathering information and building coalitions before planning and implementation proceed. SAIDI does need further follow-up to measure changed behaviors in the target populations (prescribers, dispensers, and consumers) to demonstrate the more appropriate use of AMBs (less use when not necessary, more use as required, and improved selection and quality of the AMBs). SAIDI also merits continued support, at least until all evidence of the results of the pilot community interventions is in, before any decisions are made on next steps in dissemination and replication. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 3 INTRODUCTION The USAID Latin American and Caribbean Bureau (USAID/LAC) requested that the USAID/Washington GH TECH Project conduct an external evaluation of the Amazon Malaria Initiative (AMI) and the South American Infectious Disease Initiative (SAIDI). The objectives of this evaluation are to 1. Determine the effectiveness of the approach used and the outcomes achieved by the initiatives to synthesize programming efforts in numerous countries. 2. Identify documents that need to be packaged for wider dissemination. The results of this evaluation will be used as the basis for USAID/LAC to 1. Determine whether to extend funding for AMI and SAIDI or design a new regional infectious disease initiative. 2. Assess the progress of AMI and SAIDI toward their expected results. 3. Document lessons learned from the management model used for coordinating AMI and SAIDI. The evaluation, conducted between March and June 2007, assessed progress to date on achieving the agreed objectives and reviewed the programmatic, technical, and managerial strengths and weaknesses of all AMI and SAIDI components. In this report the team presents results achieved, lessons learned, and recommendations for future activities. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 4 METHODOLOGY The GH TECH Project contracted with a two-person evaluation team: a team leader/malaria program specialist to evaluate the AMI, and an AMR specialist to evaluate SAIDI. Along with the scope of work (SOW; Annex 1), the GH TECH team received a considerable bibliography of work plans, project reports, trip reports, and presentations on a CD ROM, to which they added during the course of the evaluation (Annex 2). The team clarified points about the methodology in teleconferences and direct communications with USAID officials, and met or talked by telephone with all the AMI and SAIDI partners. Based on the SOW and these communications, the team drafted a discussion guide. The first draft, in English, covered both AMI and SAIDI because the two initiatives had some evaluation objectives in common. The guide was translated into Spanish and then separate versions were drawn up for AMI and SAIDI. The discussion guides were vetted with USAID personnel, whose comments were incorporated into the final version, which was further revised after the first team planning meeting and a round of field interviews in Peru (Annex 3). A version of the AMI discussion guide was prepared in English for use in Guyana and Suriname. The AMI evaluator visited three of the partners (PAHO, MSH, and USP) in Washington and contacted the other partner, CDC, by telephone. He visited and conducted interviews in Peru, Colombia, Brazil, and Suriname and interviewed informants by telephone in Guyana, Bolivia, Ecuador, and Venezuela (Table 1 and Annex 4). The AMI contacts were coordinated through PAHO except for Peru, which was coordinated by the USAID/Lima Mission. Table 1. AMI Interviews by Country Number of Number of Country Interviewees Interviews Persons Peru 4 USAID and PAMAFRO officials, PAHO liaison and national counterparts 10 Colombia 15 Host country counterparts from the MoH and National Health Institute, university researchers, PAHO liaison 15 Brazil 9 Host country counterparts from MoH, university researchers, PAHO liaisons 11 Suriname 5 Host country counterparts from MoH, university researchers, PAHO liaison 6 Bolivia 3 PAHO adviser, NMP coordinator, USAID/La Paz 3 Guyana 2 PAHO adviser, NMP director 2 Ecuador 2 PAHO transmittable diseases adviser, AMI project coordinator, NMP personnel 2 Venezuela 1 PAHO liaison 1 Staff of USAID/LAC, USAID/GH, CDC, MSH/RPM Plus, USP/ DQI, PAHO United States 8 12 Geneva 1 WHO staff 1 Total 50 63 EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 5 In addition to telephone interviews with US-based partners, the SAIDI evaluator visited Peru, Bolivia, and Paraguay. MSH coordinated the visits in Peru and Paraguay and PAHO the Bolivia visit. The SAIDI evaluator interviewed 63 people, mostly in person (Table 2 and Annex 4). Table 2. SAIDI Interviews by Country Country Number of Interviews Number of Interviewees Persons International partners and USAID 16 Chile (by phone) 8 Peru 6 Key contacts and USAID personnel 16 Bolivia 8 Key contacts and PAHO personnel 10 Paraguay 8 Key contacts and USAID personnel 21 Total 30 63 The evaluation team, with support from GH TECH staff, drafted a report that was distributed for comments on May 17 and briefed USAID project staff on May 23. The team then incorporated into the report comments received at the briefing and in response to the draft. The team made a final presentation to the larger USAID team on June 6. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 6 AMAZON MALARIA INITIATIVE 1 BACKGROUND: MALARIA CONTROL AND PREVENTION IN THE AMAZON BASIN In the early and mid-1990s, malaria was pervasive in the Amazon Basin region (Bolivia, Brazil, Colombia, Ecuador, French Guyana, Peru, Suriname, and Venezuela). Though things began to improve in the late 1990s and early 2000s, Plasmodium falciparum, the parasite responsible for the severe form of the disease, then appeared; it is resistant to inexpensive first-line antimalarial drugs. Resistance to a second-line antimalarial drug, sulfadoxine pyrimethamine, has been reported in Colombia, Peru, and Venezuela. Furthermore, there is little control over population migration between countries in the region, and malaria vector mosquitoes do not respect borders. Thus, ineffective or incomplete control and treatment in one country of the region affects the prevalence of malaria in neighboring counties. In the Americas, 21 countries continue to report malaria; an estimated 41 million people—one of three inhabitants—currently are at moderate to high risk of infection. Since 1987 cases reported in the region reached more than 1 million a year, peaking in 1998 at about 1.3 million. That same year, 85 percent of reported malaria cases (1,097,570 of 1,288,648 cases) were from the nine countries in the Amazon basin. P. falciparum accounted for 32 percent of these cases and at least 72 percent of all malaria-attributed deaths. Between 1990 and 1998 several Amazon countries carried out studies to assess antimalarial drug resistance. Despite the importance of these initiatives, the efforts were not part of any systematic approach to map the geographic distribution or intensity of drug resistance. Results could not be compared or used to obtain in-depth information. Recognizing the need to address the re-emerging challenge of malaria in the Amazon in concordance with the new World Health Organization (WHO) multipronged, evidence-based Roll Back Malaria (RBM) strategy; the Pan American Health Organization /WHO Regional Office for the Americas (PAHO /WHO-AMRO) convened a group of experts in Manaus, Brazil, in 1998 to review a protocol for assessing the effectiveness of antimalarial drugs. Following the example of the East African Network for Monitoring Antimalarial Treatment, the group drafted a standardized method for evaluating the therapeutic effectiveness of antimalarial drugs used in managing P. falciparum cases in the Americas. The group also prepared preliminary standards for evaluating the effectiveness of chloroquine against P. vivax that were reviewed in 2000 by a group of experts convened by PAHO. Between 1999 and 2002 the WHO Special Program for Research and Training in Tropical Diseases, RBM, and PAHO supported some efficacy studies using this protocol in Bolivia, Colombia, Ecuador, Peru, and Venezuela. USAID and the U.S. Naval Medical Research Center laboratory in Lima, Peru, cofunded and provided TA to the studies using the new PAHO protocol in Peru, Bolivia, Ecuador, and Suriname. The results provided very useful information on levels of therapeutic failure of chloroquine (Colombia, Ecuador, and Venezuela); sulfadoxine pyrimethamine (Bolivia, Colombia, Ecuador, and Peru); quinine (Venezuela); and combinations of amodiaquine and sulfadoxine pyrimethamine (Colombia) and quinine and tetracycline (Bolivia). Based on the results of the efficacy studies, Peru and Bolivia both adopted a new malaria treatment policy using artemisinin-based combination therapy (ACT) in 2001. 1 This section was adapted from a draft PAHO document, “Improving Malaria Control in the Amazon.” EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 7 In March 2001, during the Third Meeting of the Network for the Surveillance of Emerging Infectious Diseases in the Amazon Countries in Salvador, Brazil, the ministries of health of participating nations—Bolivia, Brazil, Colombia, Ecuador, Peru, Suriname, and Venezuela— made a commitment to work together to monitor antimalarial drug resistance and use the information acquired as the basis for rational malaria treatment policies; they formed RAVREDA, the Amazon Network for Surveillance of Anti-malarial Drug Resistance. USAID and PAHO /WHO-AMRO committed to supporting the efforts. THE AMAZON MALARIA INITIATIVE (AMI) In October 2001 the USAID Latin America and Caribbean Bureau, Office of Regional Sustainable Development (LAC/RSD) launched the AMI. Using a common conceptual framework to select and coordinate activities in priority countries, the initiative is intended to improve malaria control at the subregional level and help decrease national morbidity and mortality. The objective of AMI is that “malaria control programs in the Amazon Basin subregion substantially incorporate selected best practices.” The anticipated results are that: • Reliable and standardized surveillance information on malaria drug resistance will be used to monitor trends and more effectively target disease control efforts; • Laboratory diagnosis of malaria will be improved; • Tools and approaches like rapid diagnostics and bed nets will be adapted, tested in local settings, and disseminated; and • Vector control, especially insecticide resistance, will be studied. The eight AMI target countries are Bolivia, Brazil, Colombia, Ecuador, Guyana, Peru, Suriname, and Venezuela. TA is being provided by the PAHO, the CDC, the MSH Rational Pharmaceutical Management Plus (RPM Plus) program and the U.S. Pharmacopeia/Drug Quality Information (USP/DQI) program. USAID mission programs in Peru and Bolivia are helping coordinate activities in those countries. The activities supported by AMI contribute to the USAID LAC/RSD Strategic Objective: “PHN Policies and Programs Developed and Advanced in LAC.” The achievement of this expected result is supported by three intermediate results (IRs): IR1) Evidence base increased; IR2) evidence base communicated and used; and IR3) more inclusive and better informed policy process promoted. Thematic areas or lines of work that have evolved under AMI are • Surveillance of antimalarial drug resistance • Access to diagnosis and treatment • Access to and use of drugs (including adherence) • Monitoring of drug quality • Entomology • Stratification and information use for control measures EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 8 USAID launched AMI as the mechanism for focusing its financial and technical resources in support of the RBM partnership in Latin America and to promote coordination of efforts among all partners in the region through RAVREDA. An initial technical group met in Santa Cruz, Bolivia, in March 2002 that included representatives from RAVREDA (Bolivia, Brazil, Colombia, Ecuador, Guyana, Peru, Suriname, and Venezuela), CDC, PAHO, USAID, and the WHO Headquarters in Geneva. Later the same year AMI incorporated two USAID partners, the MSH/RPM Plus program and the USP/DQI program, into the initiative. In September 2002 partners began to implement their work plans within a common set of objectives and strategies. The initial aim of the project was to support participating countries in revising antimalarial drug treatment policies based on scientific evidence obtained through drug efficacy trials. In collaboration of project partners, countries also undertook activities on drug quality assurance, adherence to treatment, and supply chain management. The AMI objectives were modified in 2004 when the activity was extended to incorporate entomology with the aim of promoting integrated vector management. The objectives are outlined in Table 3 using the USAID LAC/PHN Results Framework: Table 3. Results Frameworks for Amazon Malaria Initiative 2002–2005 2005–2007 Strategic Objective: More effective Strategic Objective: PHN policies and delivery of selected health services and programs developed and advanced in LAC policy interventions IR 3.6.1; Amazon Malaria Results IR 01: Evidence base for LAC PHN Package: Reliable and standardized malaria priorities for malaria increased; drug resistance information available Outcome 01: Antimalarial drug resistance assessed, drug policies defined, use of efficacious antimalarials promoted, and entomological information available to guide control activities and promote integrated vector management IR 02: Evidence base for LAC PHN IR 3.6.2: Tools and approaches developed or priorities for malaria communicated adapted, tested, and disseminated and used; Outcome 02: Health care workers, policy decision makers, professional societies and, vulnerable groups informed of appropriate strategies and interventions to be implemented IR 03: More inclusive and better informed IR 3.6.3: Partnerships to improve malaria policy process promoted; control enhanced Outcome 03: Health policy decision-makers and other stakeholders using information to ensure implementation of revised policy EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 9 Each partner brings unique expertise to the initiative that is critical to the design and implementation of sustainable interventions to control malaria (Table 4). Table 4. AMI Partner Agencies and Their Roles Partner Role PAHO Provides coordination and secretariat role for preparing work plans and reports and conducting regional meetings. Provides TA to countries and helps them draft work plans; reports and monitor implementation. CDC Provides TA and training in surveillance, study protocols, laboratory skills (in vitro studies, molecular markers, and blood level measurement), rapid diagnostic tests, and entomology at both subregional and country levels. MSH/RPM Plus Provides TA and training in pharmaceutical management, with a focus on drug availability, prescribing and dispensing practices, and patient adherence; and management of the supply chain, including quantifying needs and identifying and correcting weaknesses in the system for supplying malaria medicines and supplies. USP/DQI Provides TA, training, and equipment for monitoring drug quality at the central level to drug regulatory authorities and official drug control labs and at sentinel sites through minilabs. A steering committee (SC) consisting of representatives from each AMI partner directs the initiative and facilitates consensus on issues. PAHO as coordinator organizes two annual meetings of the SC. In September, at the beginning of the project’s fiscal year, the SC and two to three RAVREDA members from national malaria programs meet in Washington, DC, to analyze progress and difficulties in implementing the project over the previous six months and to review each country’s work plan for the next half-year. The September meetings are to refine lines of work and activities for the new period, and approve the budget for activities supported by counterpart funds that complement USAID funding. The second meeting, held in March, six months into the fiscal year, takes place during the RAVREDA Annual Meeting, which is held in a different country each year; it brings together representatives of Ministries of Health and the AMI country coordinators along with institutions and partners that support the initiative at the local and regional levels. In the days leading up to the SC meeting, as part of the agenda of the Annual Meeting, the partners hold detailed discussions on the results of activities carried out during the preceding year. The SC then formulates recommendations based on the results presented; revises the agenda for the current period; and defines the content, locations, and dates for joint activities among the countries. Every August the RAVREDA team in each country, led by the Ministry of Health (MoH) and with support from the PAHO project coordinating team, prepares a work plan for the next period EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 10 following the general guidelines developed during the Annual Meeting and the lines of work that were defined and agreed based on regional activities being carried out. The RAVREDA coordinating team within PAHO and the other AMI technical partners also prepare work plans for the new period at this time. These plans are reviewed at the September meeting of the SC so that the proposals for work by the various USAID partners can be coordinated with country interests and opportunities for South-South cooperation can be identified. The AMI technical leads provide comments to each country on its activity plan to ensure that proposals respond to regional priorities. Progress on the plans is reviewed with the countries at the March RAVREDA and SC meetings, which call attention to any deviations from protocols and work plans. AMI FINDINGS General Impressions The general view of AMI as reflected in the responses to the questions in the discussion guide was almost universally and enthusiastically positive. One said that AMI has been a “blessing for my country.” Comments, criticisms, and suggestions generally referred to particular issues and tended to suggest minor refinements rather than fundamental changes to the program. Asked what was the most valuable contribution the initiative had made to their country, most respondents refused to be limited to just one and recited a variety of work areas, including treatment, diagnosis, drug management and quality, and entomology. Several respondents considered the leading AMI contribution to their country to be the creation of a culture of information-based decision-making that had permitted a change to more rational and effective science-based treatment regimens. The response to the question “Should USAID continue supporting the initiative?” was unanimous: USAID should continue support for this initiative rather than alternative uses of funding (although it should be pointed out that the sample was limited to parties with an interest in AMI continuing). • The quantity, variety, and quality of the activities and products encountered during the evaluation were most impressive. However, the evaluation process also uncovered several AMI shortcomings: • The initiative has not systematically documented its outcomes and success stories, of which there are many. • Nor has there been sufficient publication of study results in technical journals. • Dissemination of information has not been as proactive and current as it should be. • There has not been enough focus on policy dialogue and sustainability. RAVREDA has been crucial for coordinating substantial contributions of its country members as partners in the project’s success, but the network has been confused with the AMI project. It will be important to clarify the relationship of AMI and RAVREDA, and the role of USAID and PAHO. Although it is clearly a key player in any future activities and in the ultimate sustainability of AMI interventions, it is not yet clear how RAVREDA’s identity, role, and function will evolve. Also, although the countries are now deploying a formidable number of surveillance tools (in vitro studies, molecular markers, and blood drug level measurements) along with the in vivo efficacy and adherence studies, there does not appear to be a comprehensive second-generation surveillance model that would allow the countries to determine which mix of tools to use, and when and where. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 11 Multi-Agency Partnership Approach AMI has from the beginning used a multi-agency partnership approach. USAID/LAC and the missions in Bolivia and Peru did the initial project design work in consultation with CDC, PAHO, and the pilot countries. Agreements for the initiative were in place in fiscal year 2001, and USAID partners MSH/RPM Plus and the USP/DQI were incorporated in fiscal year 2002. The agencies coordinate their work plans with country work plans using a common template through a consultative process. PAHO is the interlocutor between the agencies and the countries through resident focal points in each country. The participants found that the value added by this approach greatly outweighed the limitations in terms of complexity and a slightly longer planning process (see Box 2). Agency representatives and country counterparts thought the partner agencies had well-defined complementary roles that led to a systems approach for resolving public health problems and avoided fragmentation of efforts. Country respondents greatly appreciated the quality and range of technical services available to them that built on experiences from within and without the region (one example given was the adaptation of PEPFAR tools by MSH/RPM Plus to monitor malaria drug supply chain management). Other tools for monitoring the malaria drug situation were adapted from the RPM Plus assessment guide for the Pharmaceutical Management for Malaria (PMM), which was developed mainly for RPM’s work in Africa. A number of respondents from both partner agencies and countries mentioned what they generally thought to be a “healthy tension” between agencies, primarily PAHO and CDC, over the difference between a “scientific approach” and the practicality of tools and products. The respondents believed that even though this dynamic tension made for frank and open discussions and sometimes slowed the protocol approval and implementation process, it resulted in better and more useful tools and results. As one interviewee said, “The best argument usually wins.” Essential to the success of this approach were • the well-defined roles of the partner agencies; • the function of the SC; and • the fact that one agency, in this case PAHO, acted as interlocutor between countries and partners, keeping things simple and manageable from the country point of view. Subregional Approach and South-South Cooperation The respondents were very clear about the benefits derived from the subregional approach due to both the nature of a problem that crosses frontiers and also to the advantages of interactions with their peers and sharing information collected through common protocols. The subregional meetings of countries with the participation of international experts have been tremendously stimulating and beneficial. A very frequent response was that it helped countries find solutions to common problems. As one researcher put it, “I can get the science from the Internet, but I get practical working solutions from my colleagues.” There was also near-unanimous agreement that subregional meetings where each country presents its work made for a “healthy competitive” effect: country representatives were motivated to perform because they wanted to look good or stand out in front of their peers from other countries. The harmonization of efforts through the ability to make informed decisions based on comparable country information cannot be stressed enough. It was frequently recognized as a one of the initiative’s greatest contributions. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 12 A number of respondents from smaller countries mentioned that AMI gave them a forum to make their needs heard. They also mentioned that in concert with their peers and with support from the international experts, they were able to resolve issues with larger neighbors, something they might not have been able to achieve on their own. Respondents from Brazil, the largest country, recognized that the subregional initiative helped move them toward a more effective and rational treatment protocol. The subregional initiative also established an effective platform for South￾South cooperation (see Box 3). Box 3 Some Examples of South-South Cooperation Under AMI Provider Recipient Topic Brazil Guyana Information system; in vivo Brazil Suriname, Guyana Entomology Brazil Guyana, Colombia In vitro Peru Guyana Diagnostic quality Peru Ecuador, Bolivia Molecular markers Colombia Bolivia In vitro Colombia Guyana Diagnostic quality Venezuela Guyana Diagnostic quality Venezuela Suriname Entomology Guyana Suriname Drug quality Suriname Guyana Microscopy South-South Cooperation with Non-AMI Countries Provider Recipient Topic Peru Nicaragua Efficacy studies Guyana Haiti Diagnosis Potential South-South Cooperation Providers Provider Recipient Topic Peru Measuring blood drug levels (efficacy studies) Brazil In vitro Colombia In vitro Information Dissemination Information was disseminated throughout the subregion through a quarterly bulletin, the RAVREDA–AMI page on the PAHO Internet site, and electronically by the PAHO AMI coordinator in Brazil (whose efforts were widely recognized and appreciated), as well as by PAHO regional advisers in Suriname and Washington and the USAID/Peru technical officer, who all took an active personal interest in getting the information out. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 13 However, much more could be done to disseminate information, particularly by using information technology (IT) for more proactive and interactive experiences. While most interviewees were familiar with and appreciative of the PAHO web site, it is not sufficiently current and gets limited use. There was very strong interest and support for more current and interactive web-based dissemination, e.g., forums for a limited period on specific topics and portals where certain people could have access to draft documents for comment. There was also support for access to an annotated bibliography of project-related studies and documents. What is most obviously lacking is dissemination of comprehensive information (see Box 4) tied to an advocacy strategy that identifies target audiences, products, and results that can be incorporated into country and subregional work plans. Also, there was support for updating previous analyses (Brazil and Peru) and carrying out new economic impact and cost/benefit analyses to inform policy dialogue and help convince decision-makers outside the health sector to view malaria control as an investment rather than a cost. Information dissemination within countries is highly variable, depending upon the physical and human infrastructure. Some of the larger countries (Brazil, Colombia, and Peru) have annual meetings of professional associations where study results are presented and discussed, though some informants mentioned that the participants selected to attend subregional events were not always the most appropriate ones and did not always disseminate information or replicate knowledge and skills upon their return. Country bulletins using a standardized format could help disseminate AMI innovations and practices and also elevate the visibility of RAVREDA. The initiative should place more emphasis on in-country coordination and information dissemination to broaden participation and enhance country buy-in, including forums for researchers, program managers, and if possible policymakers to discuss research findings and their programmatic implications. Box 4 Information Dissemination Strengths: Quarterly Bulletin is well-received. The web page is recognized. The regional coordinator is extremely active and committed to sending out information, mostly via email Annual regional meeting and regional and national trainings and workshops are good for distributing information. Weaknesses: The Web page is not updated often enough. There is no interactive portal. Electronic distribution needs to be more systematically targeted. Electronic dissemination does not always trickle down within countries Country participants at regional workshops/training do not always disseminate and replicate information, training, and skills. Rotation of national personnel causes problems. There is not enough publication in scientific journals. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 14 EXTERNAL EVA THE SOUTH AM Conceptual Model eptual Model USAID’s conceptual s conceptual model of AModel of AMI has the following ele I has the following elements: • Vector and treatment control m Vector and treatment control measures in neig asures in neighboring countries need to be countries need to be harmonized. • Targeting resources to selected activities in priorityin priority countries through a common countries through a common framework could im work could improve malaria control thro prove malaria control throughout the subregio the subregion. • Establishing Establishing a surveillance ne a surveillance network using standardized techniques twork using standardized techniques could permcould permit analyses and comparisons that woul arisons that would in turd in turn lead to more effective and coor re effective and coordinated response dinated response measures. • Fostering part Fostering partnerships would prom nerships would promote learniote learning between countries and leverage technical and financial resources for and financial resources for better malaria control. control. Generally Generally the respondents the respondents agreed firmagreed firmly with this model, particularly in t del, particularly in the areas of e areas of harmonization and standar onization and standardized techniqized techniques for cross-countr es for cross-country comparisons. The m arisons. The most common suggestions f suggestions for what could r what could be added to t be added to the model were to include s odel were to include social mobilization and obilization and community participation, and to reinf participation, and to reinforce management of malaria control laria control programs (Box 5). s (Box 5). There was There was also a suggestion to incorporate a a suggestion to incorporate a containment goal with targets, as RBM does. Institutionaliz Institutionalization and Sustainabilitstainability The pilot countries have adopted not only many of the AMI-support AMI-supported policies, tools, and methods but also the philosophy of data-driven decision￾making. Human resources have been strengthened in a number of technical areas, including research, laboratory skills, drug management, and entomology. Although Although there is no formal requirement for counterpart contributi contribution, according to the lim according to the limited data available (Table 5) the countries have been major contributors to AMI activities in terms of personnel and in-kind contr d contributions. Many respondents thought that AMI had that AMI had helped leverage national resources. The countries also are purchasing newer and more costly ACT antimalaria treatments. However, except for Peru it was hard to find specific examples of a government now paying for specific budget items previously paid for with AMI funds. Furthermore, the support for AMI outside of the National Malaria Program and health sector is highly variable. Some of the countries, particularly Bolivia, Suriname, and Colombia, seem to be particularly to be particularlyvulnerable to the withdrawal of external support. Box 5 USAID AMI Conceptual Model Key components: Suggested additional areas: Standardization of protocols Harmonized treatment and control practices, particularly in border areas Training for management of malaria programs, particularly in light of national decentralization policies Community mobilization and participation Containment of malaria (with specific objectives) More attention to P. vivax now that P. falciparum is being effectively treated LUATION OF THE AMAZON MALARIA INITIATIVE AND ERICA INFECTIOUS DISEASE INITIATIVE 15 Table 5. AMI Counterpart Contributions, 2002–2007 2002 2003 2004 2005 2006 2007 USAID Countries USAID Countries USAID Countries USAID Countries USAID Countries USAID Countries TOTAL BOL 0 0 0 0 329,710 127,610 0 0 200,000 397,000 110,620 0 1,164,940 IR1 0 0 0 0 246,510 70,860 0 0 101,000 188,000 62,000 0 IR2 0 0 0 0 36,969 50,150 0 0 93,500 202,000 31,620 0 IR3 0 0 0 0 46,231 6,600 0 0 5,500 7,000 17,000 0 BRA 206,000 0 300,000 292,000 190,000 274,258 200,000 0 200,000 175,000 95,000 0 1,932,258 IR1 155,000 0 200,000 255,000 170,000 219,258 127,000 0 73,000 20,000 60,000 0 IR2 15,000 0 35,000 37,000 15,000 28,000 46,000 0 62,000 9,000 18,000 0 IR3 36,000 0 65,000 0 5,000 27,000 27,000 0 65,000 146,000 17,000 0 COL 170,000 0 200,000 0 155,000 208,840 180,000 0 180,000 0 120,000 0 1,213,840 IR1 125,000 0 140,000 0 152,000 157,800 111,600 0 95,000 0 35,000 0 IR2 15,000 0 26,000 0 0 33,840 44,000 0 46,000 0 53,000 0 IR3 30,000 0 34,000 0 3,000 17,200 24,400 0 39,000 0 32,000 0 ECU 122,000 0 225,000 0 110,000 43,000 180,000 0 180,000 75,000 100,000 0 1,035,000 IR1 82,000 0 113,000 0 100,000 31,000 80,000 0 80,000 41,000 49,500 0 IR2 10,000 0 34,000 0 10,000 10,000 58,500 0 65,000 14,000 28,500 0 IR3 30,000 0 78,000 0 0 2,000 41,500 0 35,000 20,000 22,000 0 GUY 68,500 0 120,000 0 63,550 25,300 150,000 0 150,000 0 115,000 0 692,350 IR1 43,000 0 75,000 0 53,000 21,500 65,000 0 57,000 0 40,000 0 IR2 5,500 0 10,000 0 8,000 3,800 43,500 0 43,000 0 38,000 0 IR3 20,000 0 35,000 0 2,550 0 41,500 0 50,000 0 37,000 0 PER 0 0 0 0 200,000 0 0 0 864,600 262,000 132,743 0 1,459,343 IR1 0 0 0 0 147,385 0 0 0 283,800 86,000 56,743 0 IR2 0 0 0 0 35,315 0 0 0 445,500 135,000 67,000 0 IR3 0 0 0 0 17,300 0 0 0 135,300 41,000 9,000 0 SUR 90,000 0 170,000 0 79,650 9,000 170,000 0 150,000 0 60,000 0 728,650 IR1 50,000 0 90,000 0 73,000 7,000 98,000 0 102,000 0 36,000 0 IR2 15,000 0 35,000 0 6,650 2,000 47,000 0 36,000 0 12,000 0 IR3 25,000 0 45,000 0 0 0 25,000 0 12,000 0 12,000 0 EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 16 Table 5. AMI Counterpart Contributions, 2002–2007 2002 2003 2004 2005 2006 2007 USAID Countries USAID Countries USAID Countries USAID Countries USAID Countries USAID Countries TOTAL VEN 122,000 0 236,000 0 116,000 168,000 180,000 0 180,000 121,000 0 0 1,123,000 IR1 77,000 0 119,000 0 96,000 157,100 75,000 0 80,800 64,000 0 0 IR2 15,000 0 47,000 0 10,000 2,400 55,500 0 56,000 48,000 0 0 IR3 30,000 0 70,000 0 10,000 8,500 49,500 0 43,200 9,000 0 0 REG 164,000 415,500 203,867 505,500 398,819 415,000 360,354 0 375,929 315,000 855,133 390,000 4,399,102 IR1 115,000 150,000 138,867 240,000 324,919 150,000 29,000 0 239,750 315,000 726,221 0 IR2 14,000 265,500 15,000 265,500 33,900 265,000 175,500 0 57,929 0 77,698 390,000 IR3 35,000 0 50,000 0 40,000 0 155,854 0 78,250 0 51,214 0 TOTAL 942,500 415,500 1,454,867 797,500 1,642,729 1,271,008 1,420,354 0 2,480,529 1,345,000 1,588,496 390,000 13,748,483 Source: PAHO. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 17 Sustainability operates on many levels and has geographic, technical, and financial components. At the national level, it requires political support, from both within and outside the health sector. Ecuador has a policy instrument that commits the country to assuming responsibility for AMI￾supported interventions; hopefully, budgetary allotments will follow. Colombia is formulating a similar policy. AMI should incorporate into future country work plans the development of specific policy instruments that will enhance the sustainability of its achievements. At the local level, many respondents were clear that sustainability will come when municipalities accept their responsibility and incorporate malaria (and other vector control measures) into their work plans. AMI should consider introducing in each country one pilot intervention with community mobilization and municipal participation using control guidelines that can be documented and replicated. One limitation that repeatedly came up is the high turnover of both political and technical personnel. While this problem will never go away, its impact can be mitigated through a three￾pronged strategy of 1. formal ratification of policy instruments; 2. intersectoral and interinstitutional groups or committees (including the participation of international cooperation agencies), with some kind of formal recognition, committed to the continuity, maintenance, and implementation of the policy instruments and plans; and 3. installed capacity to provide technical training and updating. Maintaining skill sets in the face of turnover of country technical personnel requires that adequate training programs be in place along with guidelines for supervising programs. Box 6 Criteria for Graduating a Country from Support for Specific Activities • There is professional capacity with trained personnel occupying staff positions. This capacity should include the ability to replicate their skills through pre- and in-service training and supervision in activity-related skill areas. Coming up with equitable criteria for graduating countries from external subsidies for specific activities was a challenge for countries so diverse in terms of size and human and physical resources. Some minimal criteria for graduation are listed in Box 6. Graduation will have to be negotiated country by country, with milestones specified in work plans. • Information-based policy instruments commit the country to support specific activities. • National resources are currently assigned to relevant activities or are potentially available. Management, Decision-making, and Prioritization Although an eight-country subregional program built on multi-agency implementation would seem to have numerous potential management pitfalls, the initiative generally received kudos for its management and for the decision-making and prioritization process. Critical factors for this success were • constructing work plans, both country and agency, on a common template; EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 18 • having one agency with representation in each country function as the interlocutor between the agencies and the countries; and • using the SC to keep plans and activities on track. The planning process is largely viewed as country-driven, within the limits of AMI guidelines (although at least one researcher felt that the research agenda was predetermined). In general, opinion was split between those who wanted more flexibility to adapt to the realities and priorities of individual countries and those who thought that the initiative was flexible enough. While no respondent failed to appreciate the value of standardized protocols, there was variation about how much flexibility there should be for adapting them to the country situation. A common observation was that the relatively low incidence of P. falciparum in the subregion and the mobility of the target populations meant that individual studies had to stay open a prohibitively long (and expensive) time to meet sample requirements. There were also requests for more flexibility in the number of samples that need to be collected for quality control of drugs and the blood slide validation process. The SC seems to have been effective in calling country attention to deviations from work plans and protocols during the annual meetings. With few exceptions, respondents viewed the decision￾making and prioritization process to be fair and transparent. A few researchers in one country felt that the results of their research were not considered in decision-making and that there occasionally was a problem of conflicts between their institutional norms and those of PAHO (including what one respondent perceived as a limitation of project grants to $15,000), which subcontracted them under the project. PAHO was generally respected by the country programs and considered knowledgeable about local conditions and actors. Its resident advisers and focal points, particularly the AMI regional coordinator, generally received high marks for their performance. Likewise, the technical capacity and responsiveness of the other partner agencies generally elicited very favorable comments. Countries recognized that AMI gave them a platform that was open to having their interests and concerns taken into account, even if these were not always accepted. The quality and relevance of the TA provided by the partner agencies was generally lauded, although there was occasional reference to some CDC-proposed protocols (e.g., adherence) as being overly complex. Suggested improvements included allowing more time in planning sessions to reach a “shared vision.” It was also suggested that partner agency planned budget allocations by country be shared with the countries to give them a better idea of what support to expect. The only examples of unplanned agency activities mentioned were a few CDC studies that were considered negative examples; the general feeling was that partner agencies should not deviate from the SC-brokered planning process. USAID project management is shared between one LAC/RSD technical person, who focused on relations with the partner agencies, and a Foreign Service National (FSN) technical officer from USAID/Peru (approximately one-third time) in the field. They appear to have been effective in managing relations with the agencies and keeping activities on track (Figure 1). The USAID management burden for an eight-country program would have to be considered light. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 19 Figure 1. AMI Management Structure USAID LAC USAID Peru Bol Ec Col Ven Sur Guy Bra Per CDC MSH USP/DQI PAHO AMI Management Structure 2006 - 2007 Respondent Interest in Future Activities There was complete and emphatic consensus among the respondents in favor of continuing USAID support to AMI. The best reasons for doing so are quoted in Box 7. In terms of future activities, respondents generally wanted more of the same support in the same manner. Mentioned most often was the need to move more aggressively into entomology (insecticide susceptibility, vector behavior and incrimination, and stratification and focalization of control measures), which is at a relatively early stage under AMI. Many respondents thought the project should incorporate information, education, and communication to mobilize community participation (as recommended by RBM), with evaluation of impact, to improve patient adherence. There was considerable interest in further support for management of the medical supply chain, including quantification of needs, and for monitoring drug quality. There was also support for more reinforcement of national malaria programs (NMPs), including reengineering to better adapt to the decentralization and “horizontalization” of vector control programs. Box 7 Reasons for Continuing USAID Support to AMI “For the excellent results obtained to date; for the necessity to advance further in institutionalization and sustainability; and for the possibility to achieve medium-term results in reducing the morbidity/mortality of a problem that continues to be a priority for the development and improvement of the quality of life for the inhabitants of the region.” There were many expressions of interest in training and human resource development to strengthen programs and sustainability. One respondent suggested a master’s degree in “malariology,” but after discussion what a number EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 20 of respondents thought to be more appropriate and effective was a diploma-level degree course made up of distinct modules that could be taught as stand-alone courses. The target population would be country, district, and state malaria program managers. These modules could be drafted as distance-learning modules, which would limit the need for “concentrations.” The Colombian national education authority, SENA, is about to embark on a national malaria training program and could possibly be a partner in producing the modules. Several respondents expressed interest in incorporating malaria in pregnancy into AMI. However, while this would be of undeniable benefit to some individuals, it is not likely to have much impact on the malaria burden in the region. AMI has helped build an effective subregional mechanism for generating and applying information for generating evidence-based policy changes, initially for therapeutic efficacy of uncomplicated P. falciparum and then for drug quality monitoring; drug supply chain management, access, use and adherence; and, most recently entomology. Within resource limitations, it would be desirable to continue applying this capacity to other malaria-related problems for a more integrated ecological approach to the problem. However, great care should be taken to guard against overstretching absorptive capacities of countries and to protect their ability to continue performing subregional drug resistance surveillance using standard protocols. AMI and the Global Fund In theory AMI and the Global Fund to Fight AIDS, Tuberculosis, and Malaria are a perfect complement. AMI works to formulate, document, and disseminate best practices. The Global Fund supports large-scale interventions and the purchase of commodities (120,000 bed nets in the Amazon region). There is a subregional Amazon Basin Global Fund Project (PAMAFRO) covering the border areas of four Andean countries: Venezuela, Peru, Colombia, and Ecuador. Suriname and Guyana both have their own malaria projects and Bolivia had a grant that was not approved for the second phase. Bolivia, Brazil, Colombia, and Ecuador are all preparing Global Fund malaria grant applications for this coming round. The degree of coordination between Global Fund projects and AMI depends on the country and whom you talk to. However, at a minimum Global Fund projects are complying with national treatment policies drafted with support from AMI. The limited U.S. Government investment in AMI has helped make the larger U.S. Government contribution to Global Fund investment more effective. AMI coordinators should continue to coordinate with the Global Fund by, e.g., inserting explicit references to specific AMI tools and procedures, such as stratification/focalization of interventions and drug management, into current proposals. AMI should also explore providing direct TA to Global Fund projects in monitoring and evaluating intervention impact using epidemiological and entomological data. Major AMI Results The most important and perhaps the most long-lasting result that AMI has achieved is a collateral, unintended consequence: The initiative has helped create a “culture of information-based decision-making” in the subregion. This response was repeatedly confirmed. Respondents added that now, when confronted by a problem, they first ask what information is available. Another respondent elaborated by saying that the initiative has “added intelligence to the program.” A further important result is that it has created an effective and widely accepted mechanism that has cemented a subregional approach to using standardized protocols and procedures for solving common problems. Ecuador has signed agreements with Peru and Bolivia for cross-border collaboration; Peru has resolved a treatment issue in border areas with Brazil; and Brazil, Guyana, EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 21 EXTERNAL EVA THE SOUTH AM LUATION OF THE AMAZON MALARIA INITIATIVE AND ERICA INFECTIOUS DISEASE INITIATIVE 22 Suriname, and Venezuela have collaborated on addressing the mobile “garimpo” (gold-miner) problem. Although hard to measure, another major result has been the considerable technical strengthening of local counterparts through training and direct participation in research and other activities that has built their competence and self-confidence. Although AMI investments have generally been small, they have often been strategic, have leveraged local resources, and, as one researcher in a smaller country put it, “had a profound impact on my Institute.” Last but not least, the initiative has fulfilled its initial mission in that: All eight countries have adopted effective ACT for treatment of uncomplicated P. falciparum malaria based on research information. Table 6 summarizes outcomes by work area and Table 7 reviews the status by country of tools introduced or adapted through AMI. Table 6. Outcomes by Work Area Work Area Result 1. Surveillance of anti-malarial drug resistance —In vivo 82 efficacy studies (61 P. falciparum and 21 P. vivax) using standardized protocols Studies underway in Bolivia, Brazil, Colombia, —In vitro Peru, and Venezuela using standardized protocols —Molecular markers Studies underway on resistance in Ecuador, Peru, and Venezuela 2. Improved access and quality of diagnosis and treatment Country detection networks expanded to —Access to diagnosis reduce delay (Ecuador: from 70 to 253 diagnostic sites) Rapid Diagnostic Test (RDT) trials (including cost-effectiveness) completed in 6 countries; of limited use when microscopy not available WHO guidelines adapted and measures —Quality of diagnosis underway in most countries to improve slide validation and introduce competency evaluation; some countries producing panels for competency testing Quantification improved in four countries (Brazil, Colombia, Ecuador, and Guyana); drug —Drug availability and use availability and use studies in Ecuador, Suriname, Colombia, Venezuela (pilot), and Bolivia using MSH methodology; community- EXTERNAL EVA THE SOUTH AM LUATION OF THE AMAZON MALARIA INITIATIVE AND ERICA INFECTIOUS DISEASE INITIATIVE 23 Table 6. Outcomes by Work Area Work Area Result —Adherence level assessments in Guyana (as part of the mining community study) and Peru (as part of a study of childhood illnesses); study in Peru before MSH training Guideline and protocols for adherence studies; preliminary adherence studies (Bolivia, Colombia, Ecuador, Venezuela, Suriname, and Brazil); improvements in prescription practices derived from adherence studies (Ecuador); strategy for improving P. vivax packaging being developed (Venezuela) 3. Monitoring drug quality —Peripheral level —Central level All countries using minilabs to monitor drug quality; most countries doing confirmatory testing; quality problems detected in Brazil, Bolivia, Colombia, Ecuador, Suriname, and Venezuela Improved quality control (QC) testing procedures and laboratory practices (Bolivia, Brazil, Ecuador, Guyana, Peru, Suriname, and Venezuela) 4. Improved information for vector control Subregional agreement on vector control surveillance, including insecticide susceptibility using bottle method for screening in 6 countries 5. Stratification and analysis of information for control Training in stratification and preliminary exercises with some countries; malaria information systems developed in Ecuador and Guyana with design and tools for stratification at local and national levels; local experience in Brazil promoting malaria control approach based on stratification at local level; development of data base of five years morbidity and analysis in a high-risk area in Colombia Table 7. Amazon Malaria Initiative: Tools and Protocols Implemented Area Tool— Subregional Summary Bolivia Brazil Colombia Ecuador Guyana Peru Suriname Venezuela Efficacy studies: Standardized protocols adapted and personnel trained in all countries; 82 efficacy studies carried out (61 P. falc. and 21 P. vivax) and incorporated into data base. Completed studies for P. falciparum; ACT evaluated; completed studies for P. vivax; P. vivax studies in process Completed studies for P. falciparum; ACT evaluated completed studies for P. vivax; P. vivax studies in process Completed studies for P. falciparum; ACT evaluated; completed studies for P. vivax Completed studies for P. falciparum; ACT evaluated Completed studies for P. falciparum; ACT evaluated; completed studies for P. vivax Completed studies for P. falciparum; ACT evaluated; P. vivax studies in process Completed studies for P. falciparum; ACT evaluated; P. vivax studies in process Completed studies for P. falciparum; ACT evaluated; P. vivax studies in progress In vitro studies:- Standardized protocols defined; support being provided for training and reagents Training in Colombia; starting second phase (in field) First phase completed; starting second phase (in field) First phase completed; starting second phase (in field) Training in Brazil Using Polymerase Chain Reaction (PCR) Starting second phase (in field) Surveillance of antimalarial drug resistance Molecular markers: Priorities for molecular markers defined: recrudescence and Sulfadoxine￾Pyrimehamine (SP) resistance Patterns for SP resistance Patterns for SP resistance In beginning phase ACT adopted in all countries Effectiveness studies Epidemiological impact study underway Epidemiological impact study underway. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 24 Table 7. Amazon Malaria Initiative: Tools and Protocols Implemented Area Tool— Subregional Summary Bolivia Brazil Colombia Ecuador Guyana Peru Suriname Venezuela Diagnostic QC:- Regional guidelines for competency and slide evaluation drafted based on WHO guidelines; in general, networks have been improved in coverage and quality New proposal for slide validation reviewed; two experiences developing panels for competency evaluation New proposal for slide validation reviewed; competency evaluation promoted but not implemented; certification process for microscopists Changes in slide validation implemented; gradual implementation of methodology for evaluating competency using panels Community￾level study Experience developing panels for competency evaluation No changes implemented. Diagnosis Rapid test studies: Currently awaiting new WHO recom￾mendations for global RDT evaluation study. Specificity and sensitivity evaluation; use by health workers evaluation Specificity and sensitivity evaluation; stability study Specificity and sensitivity evaluation Cost￾effectivenes s study; use by health workers evaluation; effectivenes s study Specificity and sensitivity evaluation; use by health workers in remote areas evaluation Specificity and sensitivity evaluation Adherence studies: Standardized protocol adopted. P. vivax P. vivax and P. falciparum, changes in prescription and dispensation P. vivax and P. falciparum P. vivax P. falciparum (own protocol) Drug use and access Pharmaceutical Management for Malaria (PMM) Assessment Manual produced and adopted for studies of access and use Drug access and use study using PMM Improving the warehouse system; replicated PMM course in the states Drug access and use study using PMM; replicated PMM course in the states Drug access and use study using PMM Drug access and use study using adapted methodology, addressing weaknesses in supply chain Drug access and use study using PMM Drug access and use study using PMM EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 25 Table 7. Amazon Malaria Initiative: Tools and Protocols Implemented Area Tool— Subregional Summary Bolivia Brazil Colombia Ecuador Guyana Peru Suriname Venezuela Drug use and access Monitoring drug availability and diagnostic norms: Rapid assessment methodology developed Tool adapted; pilots being planned Pilots being planned Pilot being planned Tool adapted; pilot conducted Drug quality Drug quality monitoring Use of minilabs; confirmatory tests at reference laboratory; problems uncovered with expired medicines and packaging; QCl of all lots of meds before distribution Use of minilabs; confirmatory tests at reference laboratory; problems uncovered with doxycycline and quinine. Use of minilabs Use of minilabs; confirmatory tests at reference laboratory; problems uncovered with unregistered medicines Use of minilabs; confirmatory tests at reference laboratory; problems uncovered with expired medicines; QC of all lots of meds before distribution Minilabs study being organized Use of minilabs; confirmatory tests by local staff Use of minilabs; problems with packaging encountered; QC of all lots of meds before distribution Bottle susceptibility studies: Methodology validated and adopted On course: training activities for starting in 2007 On course and producing mortality curves In progress In progress Ongoing In progress Vector control Strategy for improving vector control decisions Implementation being promoted with CDC support Disseminated within technical staff and control managers; implementation being promoted Disseminated within technical staff and control managers; implementation being promoted Disseminated within technical staff and control managers; implementation being promoted Pilot with entomological component Intermittent irrigation of rice fields Trained in basic entomology Being disseminated within technical staff and control managers; QC of all lots of meds before distribution EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 26 LUATION OF THE AMAZON MALARIA INITIATIVE AND ERICA INFECTIOUS DISEASE INITIATIVE 27 Table 7. Amazon Malaria Initiative: Tools and Protocols Implemented Area Tool— Subregional Summary Bolivia Brazil Colombia Ecuador Guyana Peru Suriname Venezuela Stratification / focalization Improvements in malaria information systems: Tools developed Application of stratification tools in process Activities to improve analysis and stratification on course Improvements in information systems New information system for malaria now being implemented New information system for malaria now being implemented One out of three products finalized and approved; process just beginning Mapping of villages and houses Beginning the process in certain areas EXTERNAL EVA THE SOUTH AM Outcomes by Intermediate Result The outcomes listed in Table 8 are intermediate results for the extension period, 2005–2007. There was some confusion among the partners about IRs 2 and 3, and they have been generally working and recording results more by line of work than by IR. If there is an extension period, it would be desirable to revisit the IRs and tailor them more specifically to AMI’s work and desired results. Table 8. Outcomes by Intermediate Result IR1: Evidence Base for LAC PHN Priorities Increased Line of Work Outcome/Results Significance I. Drug Resistance A. Efficacy All countries have executed efficacy studies for P. falciparum (61) and most countries for P. vivax (210) using a standardized methodology. Information base exists for making decisions (see IR 2) along with the capacity for updating the base as necessary. B. Drug quality 1. Identification of quality assurance (QA) deficiency in the following: •Registration (Peru 2003, 2005; Ecuador 2003; Brazil 2006; Bolivia 2005) •Storage and distribution (Ecuador 2003; Brazil 2006) •Quality of medication at dispensing site (Ecuador 2003; Brazil 2006) •Post-marketing surveillance (Brazil 2006; Ecuador 2003; Peru 2005) •Implementation of policies for free access to antimalarial medication (Ecuador 2003) 2. Need identified for a decentralized and rapid QC assessment tool at endemic areas in all countries (Minilabs) 1. Official institutions responsible for QA took steps to address system deficiencies by requesting TA or revised policies and regulations. 2a). Countries have information on the quality of medicines at peripheral and remote dispensing sites. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 43 Table 8. Outcomes by Intermediate Result IR1: Evidence Base for LAC PHN Priorities Increased Line of Work Outcome/Results Significance 2b) Illegal sources of malaria medication at remote dispensing sites identified C. Adherence, availability and use studies 1. Generic study protocol for measuring adherence drafted 2. Pharmaceutical Management for Malaria (PMM) assessments of availability and use1 conducted in 5 countries (Bolivia, Colombia, Ecuador, Suriname, and Venezuela) 3. Community-level assessments of availability and use of medicines for malaria (combined with other assessments) conducted in Guyana2 and Peru.3 1. Malaria programs in AMI countries have a standard methodology for conducting studies of adherence, which allows them to identify the scope of the problem and use the results for evidence-based planning. 2. Five countries in AMI have evidence base for identifying and prioritizing problems, planning and implementing activities, and serving as a baseline for future evaluations of availability and use. 3. Peru has evidence on care-seeking behavior and the availability and use of malaria medicines for children under 5 at the community level. Guyana has evidence on the availability and use of malaria medicines in mining communities. D. Commodities 1. Procurement 1. See PMM assessments of availability, which include examination of procurement practices. 1. See above. 2. Supply chain management 1. See PMM assessments of availability, which examine impact of supply chain on availability and use. 1. See above. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 44 Table 8. Outcomes by Intermediate Result IR1: Evidence Base for LAC PHN Priorities Increased Line of Work Outcome/Results Significance 2. Supply chain management capacity assessed4 in 3 areas in Guyana 2. Weaknesses in supply chain management at the facility level addressed in Guyana E. Diagnosis and treatment See PMM assessments of use, which include a component on prescribing and dispensing See above. II. Surveillance A. Case diagnosis (including rapid diagnostic tests) Diagnostic QC: •Regional guidelines for competency and slide evaluation developed based on WHO guidelines •In general, networks improved in coverage and quality Rapid test studies: •Studies (including cost￾effectiveness) studied; currently awaiting new WHO recommendations for global RDT evaluation study. Countries have guidelines to improve QC. Countries have an information base for extending diagnostic services to remote areas. B. Reporting systems, analysis, and data usage Systems improved in Brazil, Colombia, Guyana, Ecuador, and Suriname Countries have tools for more rapid and detailed analyses. III. Entomology A. Insecticide susceptibility “Bottle – method” studies adopted in most countries Study results on susceptibility are available. B. Integrated evidence-based vector control strategies and tools created and adopted Guidelines and tools developed Countries have tools to make rational decisions. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 45 Table 8. Outcomes by Intermediate Result IR2: Evidence Base for LAC PHN Priorities Communicated and Used Line of Work Outcome/Results Significance I. Drug Resistance A. Efficacy All countries have formally instituted ACT for P. falciparum. Countries have effective regimes for treating P. falciparum that seem to be lessening the burden of the disease. B. Drug quality 1. Central level: Improved QC testing procedures and laboratory practices (Bolivia, Brazil, Ecuador, Guyana, Peru, Suriname, and Venezuela) 2. Peripheral level: All countries adhere to good practices in monitoring quality (minilabs). 1. Countries are better able to ensure the quality of medications. 2. All countries have a reliable tool for rapid and continuous assessment of antimalarial quality at sentinel sites. C. Adherence, availability, and use studies 1. Availability and use assessment results disseminated to stakeholders in Bolivia, Colombia, Ecuador, Suriname, and Venezuela. 1. Appropriate interventions identified based on evidence in five countries. 2. Action plans developed by national counterparts; implementation underway in Bolivia. D. Commodities 1. Procurement 2. Equipment 1. NMCP officials from 4 countries (Brazil, Colombia, Ecuador, and Guyana) can make more accurate needs estimates after attending a workshop on quantification for malaria medicines. 2. Laboratory better equipped for performing QC analysis (Peru, Bolivia, Guyana, and Ecuador). 3. All AMI countries received tools for quick QC of 1. More accurate needs estimates are informing country procurement requests. 2 & 3. See above (Drug quality). EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 46 Table 8. Outcomes by Intermediate Result IR2: Evidence Base for LAC PHN Priorities Communicated and Used Line of Work Outcome/Results Significance antimalarial medication at the periphery 2. Supply chain management 1. Assessment results shared on poor compliance with established standard operating procedures (SOPs) and discussed at all levels of the supply system in Guyana. 2. A manual on good storage practices for malaria medicines at health facilities in Ecuador was distributed to health facilities treating malaria. 1. In Guyana, facilities at the regional and community level follow the SOPs. 2. There is a standard against which practices can be assessed and that can be used for training. E. Diagnosis and Treatment See C above. See C above. II. Surveillance A. Case diagnosis (including rapid diagnostic tests) Guidelines for competency evaluation and slide validation being implemented Rapid tests being used in remote areas The coverage and quality of diagnosis has improved in most countries, which improves control and reduce the impact of the disease. Access to diagnosis in remote areas is better. B. Reporting systems, analysis, and data usage Brazil, Colombia, Guyana, Ecuador, and Suriname using improved information systems Analyses are now being disaggregated to the municipal level. III. Entomology A. Insecticide susceptibility B. Integrated evidence-based vector control strategies and tools created and adopted Most countries now applying stratification/focalization manuals in one or more municipalities Control measures more rationally applied should lead be cost-effective and have more impact. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 47 Table 8. Outcomes by Intermediate Result IR3: More inclusive and better informed policy process promoted Line of Work Outcome/Results Significance I. Drug Resistance A. Efficacy B. Drug quality 1. Health authorities are aware of quality problems identified by the monitoring system at peripheral sites 2. In Bolivia, Guyana and Venezuela all lots of malarial medication undergo QC analysis before entering the distribution chain. 1. Using evidence-based data, Health authorities can take appropriate measures to: a) sequester poor-quality products, thus ensuring patient safety; and b) identify and remedy poor manufacturing practices 2. Quality of medications utilized by malaria programs ensured C. Adherence, availability, and use studies 1. In Guyana, a local group of multisectoral stakeholders was organized to meet with the NMCP to discuss and coordinate activities. 2. Operational status of ACT implementation assessed in Ecuador. 1. The NMCP coordinates regular meetings to ensure stakeholder participation in policy decisions and program implementation. 2. Ecuador NMCP taking next steps to finish implementing a comprehensive ACT policy. D. Commodities 1. Procurement 2. Supply chain management Brazil and Colombia replicated the PMM course for national stakeholders In Brazil and Colombia, national stakeholders made aware of role of pharmaceutical management in effective malaria program implementation. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 48 Table 8. Outcomes by Intermediate Result IR3: More inclusive and better informed policy process promoted Line of Work Outcome/Results Significance Ecuador replicated the quantification course for heads of province-level programs. In Ecuador, program managers in the provinces better understand the importance of good quantification and have more capacity to quantify medicine and supply needs for malaria diagnosis and treatment. National programs demonstrated ability to conduct the course and communicate critical messages. E. Diagnosis and Treatment II. Surveillance A. Case diagnosis (including Rapid diagnostic tests B. Reporting systems, analysis, and data usage Available to discuss at municipal level Should result in more municipal participation III. Entomology A. Insecticide susceptibility B. Integrated evidence-based vector control strategies and tools created and adopted Strategies being applied at the municipal level Participation of local municipalities should result in more local buy-in and sustainability. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 49 AMI CONCLUSIONS/LESSONS LEARNED 1. Adaptation and standardization of protocols for a multicountry program is a slow process that requires time, field trials, and evaluation and discussion. 2. A multiagency approach can add considerable value but requires more time for planning, common work plan formats, and coordination mechanisms. This approach could be replicated by both national and subregional programs. 3. The AMI subregional approach added considerable value in terms of economies of scale in TA and training; motivating countries to perform well and adopt reforms; accumulating knowledge, South-South cooperation, and the transfer of lessons learned to solve common problems; and creating a useful platform for resolving intercountry and cross-border problems. This approach could be replicated in other subregions that meet the situational determinants. 4. Now that providers and researchers are responding that they are encountering very few treatment failures for uncomplicated P. falciparum, a second-generation model for surveillance of antimalarial drug resistance is needed to rationalize information gathering. 5. Maintaining the integrity and comparability of the applications of standardized protocols requires continued on-site TA for quality assurance. 6. Achievements have been significant but are fragile; they need continued external support and accompaniment, particularly in regard to surveillance of drug efficacy. More policy instruments committing governments to supporting specific activities are needed. 7. AMI has helped create a respected regional platform for malaria control that can be expanded to incorporate other elements to promote more integrated control for increased impact. However, care must be taken not to lose sight of the initiative’s original mission and capacity, which is to address antimalarial drug resistance at the subregional level. AMI RECOMMENDATIONS 1. Above all else, maintain the capability to monitor drug resistance during the transition to full implementation of ACT. 2. Design a comprehensive second-generation surveillance model appropriate for this new phase of malaria control with situational criteria and protocols (similar to the vector control manuals) to incorporate, as needed, monitoring of treatment failures; sentinel sites for efficacy studies (focusing on quality rather than quantity); active case detection; access, use, and adherence studies; blood-level studies; in vitro studies (again focusing on quality, not quantity); and molecular markers. Detection and follow-up of treatment failures should be emphasized since some methodologies do not seem to have been fully validated. 3. Provide direct monitoring and accompaniment to maintain the capacity to continue performing resistance studies as needed, and the integrity of subregional standardized protocols as they are applied in-country. Consider pooled (meta) analyses of subregional research data. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 50 4. Revise all work plans and report formats to focus on milestones, outcomes, and results. Put less emphasis on activities in project reports and set up an instrument to monitor specific policy instrument outcomes (see the Central American HIV/AIDS Policy Matrix). If possible, take a more strategic view (3 years) with long-range objectives (by region and country) that have verifiable annual milestones. Pay more attention to institutionalization and sustainability by incorporating sustainability milestones into country work plans, including graduation from external support for specific activities. 5. Strengthen ties with PAMAFRO and other Global Fund Projects, present and proposed, and Organización del Tratado de Cooperación Amazónica (OTCA), particularly with regard to policy dialogue and impact evaluation of interventions, such as bed nets and community mobilization. Draft and disseminate model policy instruments. 6. Clarify RAVREDA’s identity as distinct from AMI (no longer use the term RAVREDA￾AMI). Elevate RAVREDA’s status and its work by seeking formal recognition of the network through an inter-ministerial agreement. If done in collaboration with OTCA, it could have foreign affairs chancelleries and ministries of health as cosponsors. Formalize national multisectoral committees and regional or departmental equivalents. Further strengthen RAVREDA’s sustainability by helping it to draft proposals in conjunction with local research institutions. Consider adding one or two non-NMP representatives per country to the regional RAVREDA technical committee. 7. Strengthen use of IT for information dissemination including video and teleconferences, interactive web-based forums on specific topics, and distance training. Make the RAVREDA web page more accessible (e.g., rename it “RAVREDA.org”). Develop distance training modules that could add up to a diploma-level course in malaria control. 8. Continue to support acquisition and supply chain management (including pharmaceuticals, lab supplies, and insecticides) to make programs more cost-effective. 9. If resources permit, consider other types of program management training and reinforcement that could increase program efficiency, effectiveness, and sustainability. 10. Incorporate a communications component, similar to the one in SAIDI, to better understand patient and provider behaviors. Design targeted communications strategies for different audiences including advocacy with policy-makers and other stakeholders, the technical and scientific community, and providers and patients. Consider producing economic impact studies. 11. Continue support for development and application of stratification and focalization of control efforts and evaluation of intervention impact by strengthening the epidemiological-entomological surveillance information system to produce more baseline and follow-up data. Pay particular attention to the quality of malaria case detection surveillance data and reporting. Consider doing a pilot intervention of an approach, including community participation and mobilization, that integrates treatment and vector control for a more ecological approach that could also make the program more sustainable. 12. Insist that participants in regional workshops have a clear obligation to disseminate results or replicate training upon their return home. 13. Replicate the AMI subregional and partnership approaches for malaria control in Central America and possibly other subregions after reviewing the situational determinants. In Central America, this would go beyond just having the AMI countries providing TA by EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 51 organizing work areas, e.g. insecticide susceptibility and other studies, under a common template. 14. Ensure proper QC of antimalarial medicines at peripheral sites by training personnel and expanding utilization of minilabs to cover all risk areas. If resources for tuberculosis become available, extend the use of minilabs for monitoring drug quality of anti-TB drugs—taking care not to degrade country capacity to continue monitoring antimalarial drug quality. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 52 THE SOUTH AMERICAN INFECTIOUS DISEASE INITIATIVE (SAIDI) 2 BACKGROUND Building upon the success of the AMI approach, the SAIDI was formed in 2003 to help slow the development of AMR by improving the availability and the use of quality antibiotics. AMR is threatening to undermine the advances achieved through priority health programs for TB, malaria, and HIV/AIDS by rendering current treatments ineffective. AMR is the result of an increased exposure of microorganisms to antimicrobial medicines and the subsequent evolution of survival mechanisms in these microorganisms. The consequences of AMR include increases in mortality and morbidity and higher costs for health care worldwide. Among the many factors that influence the development of AMR, the major contributors from a public health perspective are the unnecessary use of AMBs for common conditions, inappropriate dosages of antimicrobials when they are required, and the proliferation of poor quality medicines. Health systems contribute to this situation when the laws do not ensure the appropriate use of quality antimicrobials, and when managerial mechanisms for selection, procurement, distribution, and use of these valuable medicines are inadequate. Physicians, pharmacists, and drug vendors contribute to the unnecessary use of these drugs by prescribing and selling inappropriate treatments. Moreover, patients who have previously benefited from antimicrobials tend to self￾medicate inappropriately. New strategies and more resources for second-line medicines may soon be needed for highly prevalent diseases as conventional treatments fail. Improvements in control of infectious diseases worldwide are increasingly threatened by the ability of microbes to build resistance to the effect of AMBs. AMR begins with genetic mutations that enable the microbes to survive even when the drugs are present. In response to this situation, USAID/LAC/RSD designed SAIDI as a subregional strategy for Bolivia, Peru, and Paraguay. The general goal of the initiative was to contain AMR by improving the availability and evidence-based use of good-quality AMBs. Working with national and local counterparts, USAID convened organizations working on rational drug use and AMR-related activities to tailor activities to each country’s unique needs. The international partners contributing to SAIDI activities are the RPM Plus program of MSH, USP/DQI, CDC, the PAHO Communicable Diseases Division, the Alliance for Prudent Use of Antibiotics (APUA), and Links Media. Box 8 Factors that Influence AMR Unnecessar 2 This section is adapted from project reports and presentations provided by MSH/RPM Plus. • y use of antimicrobials • Inappropriate dosages of antimicrobials • Proliferation of poor quality or substandard drugs • Physicians, pharmacists, and drug vendors overprescribing antimicrobials • Patient self-medication • Inadequate hospital infection control programs • Overuse of antimicrobials in animals and agriculture Courtesy of MSH/RPM Plus EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 53 The initiative was co-managed by USAID/Washington and the Mission in Lima, Peru. The partnership model was based on the one used in the AMI, in which a number of SAIDI partners participate. MSH has the secretariat and interagency coordinating role for SAIDI. SAIDI is piloting community-based AMR programs to help participating countries find local approaches to contain AMR through phased steps done in collaboration with local partners (Box 9). For SAIDI purposes, “community” refers to all actors and sectors whose conduct in some way can affect AMR within a given locality. The partner agencies were able to follow this stepwise approach in Peru and Paraguay, but in Bolivia, due to political changes after the pre￾assessment phase, they had to adapt their approach and worked separately with local partners with whom they already had contact. In Paraguay and Peru, international and local partners are now in the early stages of implementation. In helping countries to analyze factors that contribute to AMR and strengthen their capacities to intervene to contain AMR, SAIDI is working toward the following IRs: IR1: Increase the evidence base on factors that contribute to the emergence of AMR in LAC. IR2: Improve local capacity to develop and implement appropriate interventions to contain AMR (focusing on the use of AMBs). IR3: Disseminate information and lessons learned from community-level initiatives. It is expected that countries participating in the initiative will establish national policies and technical guidelines for prevention, diagnosis, and treatment of infectious diseases; management of medicines; quality assurance of AMBs (including TB drugs) and their appropriate use; a permanent surveillance AMR system; and design of interventions and ultimately communication strategies for effecting permanent behavior changes. The SAIDI community approach is an interdisciplinary and holistic ecological approach in which the community is considered as an ecosystem that takes into account human, institutional, and systemic factors that influence AMR. Figure 2 provides a model for drug use interventions that identifies proximal factors as behaviors of prescribers, dispensers, and consumers. However, the model is overly simplistic in terms of the SAIDI ecological approach because it does not adequately take into account the political and regulatory environment, quality of manufacturing, and other factors, such as widespread use of AMBs by veterinarians, that contribute to resistance. The SAIDI rationale is that an effective program must deal with the underlying causes of AMR, which requires local understanding of the problem and the dynamics of the wide range of stakeholders, and tailoring responses to that situation. Box 9 SAIDI Phases • Preparation • Situation analysis • Coalition building • Plan formulation • Implementing and evaluating planned activities Courtesy of MSH/RPM Plus EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 54 Figure 2. Conceptual Model of Drug Use Interventions Framework for Drug Use Interventions AMR Prescribing Consumption Dispensing Lack of knowledge Financial interest Beliefs, norms and values Inadequate information Poor training Financial interest Inadequate information Beliefs and values Lack of information Over-/under￾dosing Wrong drug Drug not needed Drug availability Wrong diagnosis Wrong drug Too much / too little Poor storage/ packing Over- / Under￾use Multiple prescribers Irrational combinations Drug not needed Wrong drug Drug availability Financial constraints Clinics Household/ Community Hospitals Drug shops/vendors Source: MSH/RPM Plus The SAIDI objective is to help countries assess factors contributing to AMR and strengthen their capacity to design interventions to contain it, including multi-drug-resistant (MDR) TB, by gathering evidence on the management and use of antimicrobials to improved decision-making; collaborating with other countries on effective approaches; and enhancing information sharing and dissemination of lessons learned (Box 10). Box 10 SAIDI Objectives General: • Help countries assess factors contributing to AMR and strengthen their capacity to design interventions to contain AMR (including MDR-TB). Specific: • Gather evidence on management and use of AMBs in humans to improve decision making. • Collaborate with other countries to develop effective approaches to address AMR, focusing on AMB use. • Enhance information sharing and dissemination of lessons learned Courtesy of MSH/RPM Plus EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 55 FINDINGS Coordination Local involvement in project implementation and leadership varied by country. In Peru and Paraguay, national stakeholders have been involved since the pre-assessment phase, although some partner training activities in Peru did not seem to have been coordinated with local authorities. In Bolivia, a national group was not formed after the pre-assessment phase due to a change in local authorities, so PAHO has been coordinating activities with local partners there. The initiative in Peru is being piloted in DISA Callao, a small geographic area that is a local administrative unit of the national health system. Most of the activities in this area focused on AMB use by the “community” in a broad sense of the word that takes into account all public and private actors. Local authorities were very accepting of the initiative, which was coordinated by DIGEMID, the Division of the MoH that is in charge of the registration and use of medicines. The Vigia Project, a USAID/Peru Project with the MoH, was also instrumental in launching the initiative. The country made the plan and set priorities based on the results of the assessment through a workshop with national and international partners to define basic work areas. The national partners then developed a logical framework for the work areas. In Bolivia, due to the political situation no national partner took leadership of the project after the assessment phase, so SAIDI is not country-driven there. PAHO and the other partners made different strategies and activities available to local organizations with which they already had relationships, keeping in mind the SAIDI objectives. The local organizations accepted these strategies based on technical leadership by PAHO. In Paraguay, the MoH Health Surveillance Unit took the lead and designated a medical doctor to conduct local activities under the supervision of the Director of the Health Surveillance Unit (which includes infectious diseases). The doctor chosen had project management experience, having been in charge of the antismoking program. The country made plans and set priorities through a process similar to that used in Peru. A summary of the project focus by country is shown in Table 9. Table 9. SAIDI Focus by Country Local Main Interventional Country Focus Approach Coordinator Area Peru DIGEMID/MoH DISA Callao Community: dispensers, prescribers, and consumers in health centers, hospitals, and pharmacies Respiratory tract infections in children under 5 Bolivia PAHO Selected health facilities Hospitals and primary care Quality assurance of drugs, including TB drugs, and general interventions to prevent infection and prevent and contain AMR; training in infection control and microbiology EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 56 Paraguay MoH Division of Health Surveillance National (starting with Gran Asuncion and Central) Hospitals and community Respiratory tract infections in children under 5 in four hospitals; general interventions nationwide, including a surveillance system for AMR and TB Effectiveness and Outcomes International Partner Perspective During the evaluation, the international partners gave their perspective on the project, their contribution to the initiative, and their principal results. All of them recognized the importance of taking a coordinated approach to the AMR problem. They recognized that this approach added value to the initiative because it permitted participants to face the problem from different perspectives and brought different expertise to bear on a common objective, even though it extended the time needed for planning. The weaknesses most often mentioned were the short time for implementation and continual changes in national authorities, which delayed decision making and wasted the efforts invested in people who did not remain with the program. They also recognized that results to date have been limited and more activities have been carried out in Peru and Paraguay than in Bolivia. Regarding the sustainability of the initiative, even though Peru and Paraguay had already been allocating some resources for AMR-related activities, it is not clear that they will allocate enough to continue the SAIDI activities after the project ends due to political contingencies and in the case of Bolivia the lack of a national partner. How local partnerships and coordination will be sustained without external inputs and stimulation also needs to be worked out. Local Partner Perspective There was considerable concurrence of local partners with the views expressed by international partners. The SAIDI approach was seen favorably as a way to address the complex AMR problem with an integrated vision covering all the areas relevant to the problem. Moreover, it is participatory and allows the countries to set their own priorities. Respondents recognized that this approach takes a long time to initiate and accomplish results. However, they also expressed concerns about the uncertainty of financing continuing, and about changes of authority and responsibilities that produced delays in decision making and can reverse previous agreements. Only two respondents suggested alternate models. One suggestion was to allow more time to achieve more ambitious goals. The other was to approach the problem of AMR from a damage or consequences perspective rather than a risk-based approach (i.e., good infection management programs prevent AMR). The program components identified as key to success in Peru and Paraguay were (a) coordination by recognized local authorities, such as the MoH; (b) the fact that the international partners work directly with local counterparts; (c) the technical excellence and support of international partners; and (d) clear definition of the objectives. In Bolivia, the SAIDI community approach has not yet been implemented, so although a number of valuable AMR capacity-strengthening activities have been conducted there, Bolivia cannot be considered a successful application of the community model. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 57 Most of those interviewed believed that the SAIDI model is replicable elsewhere because it is very participatory; it takes into account the local situation and problems: it provides useful technical tools based on a local assessment: and it is flexible and adaptable. However, very few people knew that the initiative was intended to be subregional. Some respondents in Bolivia though that the subregional approach was a good intention of the international partners, but SAIDI did not work out that way. SAIDI really became a multicountry rather than a subregional program. The priorities and general approach differed by country. In Peru prioritization and decisions were made in terms of both technical issues and pragmatic priorities (DISA Callao is a small administrative unit of the MoH and may not be a priority in terms of health problems, but local partners believed that results could be achieved there quickly). In Paraguay practical aspects also were considered, but decisions were oriented more to the technical aspects. In both countries priorities were set by the national partners taking into account the initial assessment and responding to their own problems. In Bolivia, due to lack of a national counterpart, PAHO took the initiative to coordinate activities based on the results of the country pre-assessment and assessment phases in which the partners participated. The international partners proposed activities in Bolivia independently of each other based on previous relationships with local organizations, and these were then approved by the SC. In Paraguay and Peru most of those interviewed considered the prioritization process to have been appropriate because it was based on what they perceived as real problems. Two respondents thought the process did not take enough account of the civil community and was based only on a technical perspective. In Bolivia, where there was no local prioritization process, local actors participating in the initiative interpreted SAIDI more as a PAHO-led initiative than as a true prioritization exercise. Even though the prioritization process was considered “appropriate,” more than 50 percent of the respondents did not find it fully participatory because many important actors—some universities, nongovernmental organizations, veterinarians, and civil society—were not active partners. In Bolivia important SAIDI activities were carried out by CONCAMYT, which is part of the MoH but acted as an independent unit and did not coordinate with MoH central management. There was little other involvement for the MoH other than some coordination with UNIMED, the drug regulatory authority. Many respondents were unaware of the reviews of both the scientific and popular literature that the international partners conducted before designing strategies. During two interviews in Peru, people considered the evidence incomplete because it did not take into account important aspects of AMR, such as the rational use of AMBs in hospitals. A suggestion for improving the prioritization process was to include more actors, particularly universities, the MoH, and the community. In fact, many of these other organizations had participated in the pre-assessment and assessment phases, but apparently a number of them dropped out, even though invited to continue. In any case, the strategies adopted to meet country needs were considered of high quality and relevance, mainly because of the recognized technical excellence of the international partners. The laboratory equipment of CONCAMYT in Bolivia, the Drug Information Center in Callao, and SIAMED (a computerized information system for drug quality assurance) in Paraguay were singled out as positive examples. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 58 Since SAIDI is still in an early stage, there has not yet been widespread dissemination of information in-country, and its activities are still known only to a very small group of people. To better disseminate the information, respondents recommended exchanges of experiences and lessons learned with other countries in meetings with local interest groups, replication of activities like courses or other training programs, and the exchange of expertise among countries. Also suggested was the need to distribute and publish studies and documents and support information systems in health care facilities and in community interest groups through the methods recommended by Links Media, which offers the necessary training. Respondents mentioned very few activities not originally planned that were carried out in connection with or as a consequence of planned activities and presence in the field. One such activity was a study of medicine costs in Bolivia that was initiated by SAIDI and continued by the country. What the respondents thought worked well within the initiative were the awareness of the national partners of the problem of AMR and the motivation to do something about it, and the technical support and the relationship with international partners. However, all those interviewed mentioned problems and obstacles that require modification or adjustment. The main obstacles were lack of commitment by national authorities; limited participation of civil society; lack of human resources; continued changes in the authorities and people in charge of the programs; and the fact that in no country has the initiative been formally incorporated into national policy or law. The respondents also mentioned as a constraint lack of resources for follow-up and evaluation of strategies implemented. Box 11 Respondent Suggestions for SAIDI Dissemination • Exchanges of experiences and lessons learned between countries • National meetings with local interest groups to share lessons learned • Replication of activities and training • South-South cooperation • Publish and distribute studies and documents Although no formal policy changes have yet been institutionalized as a consequence of the project, the countries have adopted valuable local guidelines and normative procedures. CONCAMYT Bolivia, which received equipment and training from SAIDI, updated its internal guidelines and protocols. In Peru the National Center for Quality Control (CNCC) improved its Quality Manual and standard operating procedures (SOPs) with international technical assistance. In Paraguay SAIDI is implementing guidelines for therapeutic use of antimicrobials and helped generate a national nosocomial infection (NI) prevention program. The chief of the National TB Program took training courses offered by CDC (TB infection control [IC]) and MSH (management of TB drugs and individual patient treatment kits) and has publicized the use of individual patient treatment kits promoted by the training to improve management of TB drug inventory in health facilities. The respondents considered the tools used in the initiative to be useful, especially those related to drug quality and management, such as USP and MSH tools adapted to the local context, but their application will depend on local capacity. Many of the tools are still in development so they are not yet widely used, though there are a few examples of tools in use in all the participating countries. Communication between countries has been informal and generally sporadic, but as examples, Paraguay used an assessment tool for hand washing developed by Peru, and CONCAMYT Bolivia is processing antibiotic and TB medicines from Paraguay. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 59 SAIDI has not yet developed subregional training programs. The minilabs strategy used in the AMI project for monitoring drug quality has not been used in SAIDI. USP proposed to DIGIMED Peru that minilabs be used to strengthen capacity for surveillance and post-marketing control but had not yet received a response at the time of the evaluation. The countries are now designating some resources for SAIDI activities for logistics (documents, transport, etc.) and staff in charge of coordinating the initiative. Paraguay and especially Peru had previously designated some resources for AMR-related activities, but it is not clear that the countries are planning to budget for full sustainability of the initiative. In Bolivia the Director of CONCAMYT mentioned that the only way to get resources is to design new projects because it is not easy for the country to invest in AMR. Respondents in all three countries see graduation from external assistance to be far away. Graduation criteria mentioned most often were independence in health decisions, model implemented in the country, compliance with guidelines and regulations, quality assurance for medicines, a permanent budget for AMR, a national AMR policy, AMR in the pregraduate curriculum, and appropriate staffing (see Box 6 in the AMI section). All the informants felt that USAID should continue to support this initiative because the time has been too short to see results; many strategies have not been fully implemented; this is a global problem; and USAID has an ethical responsibility to continue the initiative. Respondents mentioned a number of aspects of the initiative that should be continued: training in communications, regulatory issues, community participation, and NIs; evaluation of interventions in the community, quality assurance for medicines, technical assistance, AMR surveillance; publication of technical documents, and monitoring of interventions specific to MDR TB agents. Respondents also suggested that USAID should support complementary infectious disease initiatives. In all three countries NIs were mentioned as a major problem that needs to be approached independently. Also mentioned were TB, respiratory and intestinal infections in children, and bartonelosis. Those interviewed thought the main achievement of SAIDI has been a team approach to the AMR problem using different perspectives in an integrated (“comprehensive”) manner. In effect, it appears that SAIDI has helped promote a new approach that avoids the fragmentation of previous efforts. In Bolivia, there was not the same coordination as with AMI. In SAIDI the dialogue is with each partner independently; in AMI there are annual meetings with all international and local partners. The SC for SAIDI did not have a role similar to the AMI SC. In AMI, there is a permanent national coordinator; in SAIDI national coordinators are not fully dedicated to the project and have changed several times. The USAID Mission that was most involved in the project was Peru, which participated actively as facilitator and in the formulation and evaluation of objectives, outcomes, and activities. The Missions in Bolivia and Paraguay responded to requests, sometimes played a facilitating role, and were kept informed of project activities. Activities coordinated by the CDC, PAHO, and MSH in the three countries were oriented to the prevention of respiratory NIs in health care institutions, with a focus on TB control and prevention. As a result of these activities, countries drew up local plans that are now being evaluated by the CDC. Professionals believe that approaching MDR TB from the perspective of preventing infections is innovative and necessary. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 60 Activities and Products to achieve SAIDI Objectives International and local partners together planned strategies to reach SAIDI objectives by achieving three intermediate results (IR). Many activities created under IR1 and IR2 have increased the evidence base of factors contributing to the emergence of AMR and improved local capacity to develop and implement interventions to contain it that are focused on AMB use. For IR3, which refers to dissemination of lessons learned, the activities are still in an early stage so there has not yet been much opportunity for dissemination. For both IR1 and IR2 SAIDI international and national partners drafted strategies that were implemented in the three countries, but most of the strategies were planned in each country to respond to local needs. As the SAIDI model intended, most of the strategies were oriented to the community level and primary care and focused on AMB use and drug QA. Fewer activities were oriented to the hospital setting. The pre-assessment and assessment phases sought to establish the baseline status of NI surveillance and control. The next phase, as part of coalition building, included training in control of transmission, prevention, and surveillance of respiratory infections in hospitals, with emphasis on TB transmission. National plans were approved for Peru and Paraguay based on the assessment results. Most of the products generated by the initiative have not been completed and widely disseminated except for laboratory drug QA, the new PAHO Guidelines for Infectious Diseases, and establishment of drug information centers to provide technical information on TB and other drugs to health professionals. Boxes 13 and 14 summarize activities to date. SAIDI has clearly promoted multiple strategies for diminishing AMR in the three countries using a partnership approach. The initiative, which was designed in five phases (Box 9), is currently in the implementation phase. Box 13 Products for IR1: Increase the evidence base on factors contributing to the emergence of AMR in LAC. • Review unpublished (grey) and popular literature on AMB trends (3 countries). • Identify drug QA/QC deficiencies (3 countries). • Assess quality of AMBs for TB and other diseases (3 countries). • Assess AMB availability and use in health facilities (Peru, Paraguay) and presence of TB drugs in pharmacies (Peru). • Assess quality of AMR surveillance (Bolivia and Paraguay). • Do qualitative studies of consumers, providers, and dispensers (Paraguay and Peru). • Do quality assessments of AMR surveillance and evaluate hospital NI programs, cost of AMB prophylaxis and treatment, QC by the drug regulatory authority, and veterinary use of AMBs (Paraguay). EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 61 The assessments revealed extensive and inappropriate use of AMBs and TB drugs in the community and in hospitals, poor quality AMBs in the market, and poor IC practices. Drug control and the market are not monitored extensively in any of the three countries; and because the requirements for registration or renewal are not stringent, there are serious QA deficiencies. Accordingly, the SAIDI strategies were oriented to improve the selection and management of AMBs and TB drugs, establish mechanisms for assuring the quality of AMBs, decrease use of AMBs when they are unnecessary, and monitor AMR and IC programs. International Partners The international partners differed in terms of interests, involvement, and outcomes, which may explain why some partners felt there was a lack of clarity in initial guidance and role definition. MSH and Links Media have been very involved in implementing projects responding to local priorities in Peru and Paraguay. USP has conducted more uniform and continuous activities in all three countries. PAHO and CDC seem to be continuing with previous activities that are in harmony with the SAIDI model; CDC was slow to begin work due to an internal change in the unit responsible for the project. The participation of APUA, which works through country chapters and their volunteers, has so far been somewhat limited. Box 14 Products for IR2: Improve local capacity to develop and implement appropriate interventions to contain AMR (focusing on AMB use). • TB infection control (IC) training of trainers, revision of the IC manual, and updating of the MDR-TB manual (3 countries) • Drug quality laboratories equipped and personnel monitoring TB and other drugs (3 countries) • Drug information centers in operation (Peru and Paraguay) • Pharmaceutical management of MDR TB using SOPs (Peru) • Pilot for improving supply of TB drugs (Santa Cruz, Bolivia) • QA for lab monitoring of AMB use (Bolivia) • Communications plans prepared, personnel trained, and materials in production (Peru and Paraguay); spots on AMB use aired (Bolivia) • Individualized patient kit for TB introduced (Paraguay) • Personnel trained in quantification, storage, distribution, QA, and monitoring (Paraguay) Local coordination was a problem. It was difficult to assign accountability due to political changes and different normative cultures and processes for getting things done. In Peru and Paraguay those responsible locally changed several times and in Bolivia it was not possible to establish any local coordination. In general those in charge of local coordination are not decision makers; they do not have the authority or the mandate to enforce adoption and continuation of SAIDI strategies or to ensure their sustainability. In Peru many of those interviewed saw the project as more of an opportunity to add resources to a local health unit. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 62 SAIDI CONCLUSIONS AND LESSONS LEARNED THE MODEL Though conceived as a subregional initiative based on the AMI model, in responding to local conditions and realities, SAIDI formulated strategies for diminishing AMR in three countries taking more a multicountry rather than a subregional approach. This was appropriate given the lack of a very specific focus, unlike AMI at the outset, and the fact that the three countries do not constitute a subregion (Peru and Bolivia are Andean countries; Paraguay is part of MERCOSUR). The subregional approach was not recognized by local partners and the exchange of experiences, technology, and structure among participating countries has been minimal. Nevertheless, international partners formulated major strategies similarly, so there is a platform (Box 15) for future joint work and exchanges as a natural evolution to a subregional approach. The AMI subregional approach added considerable value in terms of economy, motivation, dissemination of knowledge, cooperation, and exchanges between countries. This experience should be used as a basis for incorporating subregional elements into SAIDI. Box 15 Common SAIDI Work Areas and Tools Across Countries • The training courses offered by LM, CDC, and USP (including those on TB) • QA for drugs (including those for TB) • PAHO external evaluation of hospitals • Guidelines and plans generated by the project • Publications on infectious diseases • Curricula (e.g., IC for health sciences students working in hospitals) The community approach (in the broad sense of the term community) employed by SAIDI is a potentially valuable mechanism for controlling AMR related to TB and other infectious diseases. Use of AMB is widespread, and failures that contribute to AMR have been observed at all levels of the system (consumers, prescribers, dispensers, and QA). In an ecological approach all the actors are considered relevant; a global strategy to control AMR and MDR TB should incorporate IC programs and rational use of AMB in hospitals, especially tertiary facilities where patients are more severely infected and drug- resistant and where the use of wide spectrum AMB and poor IC practices constitute a permanent risk factor for AMR. The successful AMI experience may have contributed to the partner model (four of the six SAIDI partners also participate in AMI), and the approach of facilitating relationships with local counterparts, and similar management measures, such as SCs, were used. Also, the AMI experience was highly relevant to activities to help improve drug QA. SAIDI partnerships between agencies, between numerous sectors within countries, and between agencies and host country partners seem to have been effective in building and maintaining the community and ecological models, except in Bolivia, where the agencies had to adopt other strategies to keep activities going. The start-up and assessment phases took more time than expected. From the first partnership start-up meeting to completion of the assessment in each country was one to two years. Intervention and implementation is just beginning. Most of the products generated or adapted by EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 63 the project (training manuals, guidelines, studies, and investigations) are thus still being validated or have not been published, so their dissemination and use are still very limited. Among finished products, the project has contributed strongly to QA, enhancing QC in laboratories, and local capacity for registration and renewal of drug licenses. Countries were provided with essential tools and equipment to ensure the continuity of these changes. In all three countries personnel were trained to the highest current standards. Although not a direct result of SAIDI, PAHO has sponsored six individuals, three each from Bolivia and Paraguay, as candidates for a master’s degree in control of hospital infections at the School of Medicine of the University of Valparaíso, Chile. This will help build local capacity. CONCAMYT/Bolivia and CNCC/Peru, which received intensive training, have been judged in external evaluations to perform very well. The SAIDI focus to date has appropriately been on the technical counterparts, but to fully implement the ecological model the partners will need to pay more attention to the regulatory area and the policy environment to identify instruments and reforms that will enable the replication, dissemination, and sustainability of best practices and lessons learned from the pilot activities. SAIDI emerged from what could be described as a “design and implement” approach whereby USAID brought together partners that had been working separately in areas related to AMR to design an integrated holistic approach to an emerging and multidimensional health problem that threatens gains made by previous primary health care programs. This approach was highly appropriate given the lack of previous experience with a comprehensive response to the AMR problem and the need for strategies to be adapted to local situations. This approach, incorporating bottom-up planning at the country level, requires more coordination and therefore more time to achieve local ownership of the program. The SAIDI model seems to have been successful in two of the three countries and has carried out important anti-AMR activities in the third. Now that SAIDI is actively underway, it needs to consider two further phases that have not yet been adequately delineated: evaluation, and replication or scaling up. In terms of scaling up, SAIDI will need full documentation not only of products but also of processes used to localize responses; even within a country, the dynamics are likely to vary depending upon local characteristics. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 64 SAIDI RECOMMENDATIONS 1. Carry the innovative community-based plans in Peru and Paraguay through the final evaluation stage. Draft a comprehensive evaluation model that defines indicators of success. 2. Limit activities in Bolivia to MDR TB management, drug QC,, and integrated supply chain management in an integrated manner until a committed national counterpart can take responsibility for coordinating a community approach. 3. Continue with the Links Media information dissemination plan, especially activities targeted to policy makers. 4. Promote elements of a subregional approach and South-South cooperation by • Facilitating the exchange of tools and expertise among countries (training materials, manuals, guidelines, etc.); • Convening subregional meetings with decision makers from each country to share experiences, products, and tools generated by the initiative; and • Setting up subregional trainings with participants who commit to replicate and disseminate SAIDI tools and information in their countries. 5. Enhance the institutionalization of the lessons learned and SAIDI strategies developed through the pilots by • Identifying regulatory instruments needed to control AMR at all levels; • Formalizing agreements and assigning responsibilities for specific commitments and targets (e.g., regulations or policies) in the final phase of the project; and • Establishing national and subregional strategic alliances among groups such as APUA affiliates to undertake AMR awareness, advocacy, and policy dialogue activities. 6. Plan for preparing documentation and guides for scaling up community-based interventions to contain AMR. 7. Consider hiring liaisons in each country to coordinate local activities and monitor achievement against recommendations in coordination with international partners. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 65 ANNEX 1: SCOPE OF WORK Statement of Work Amazon Malaria Initiative and South American Infectious Disease Initiative External Evaluation Team I. Purpose This request is for an external evaluation of the Amazon Malaria Initiative (AMI) and South American Infectious Disease Initiative (SAIDI) programs. Specifically, the results of this evaluation will be used to provide the basis for USAID/LAC to 1. 1. Determine whether funding for AMI and SAIDI should be extended or a new regional infectious disease initiative should be designed. 2. 2 Assess the progress of AMI and SAIDI toward achieving their expected results. 3. 3. Document lessons learned from implementing the management model used for coordinating AMI and SAIDI. The objectives of this evaluation are to 1. Determine the effectiveness of the approach used and outcomes achieved by the initiatives so as to coordinate and synthesize programming efforts across multiple countries. 2. Identify documents that need to be finalized and packaged for wider dissemination for the benefit of USAID-supported efforts. It is expected that this evaluation will begin on or about late February and will be completed by mid-June 2007. II. Background This Statement of Work sets forth guidelines for an evaluation of USAID/LAC’s Amazon Malaria Initiative and South American Infectious Disease Initiative. USAID/LAC has supported the Amazon Malaria Initiative since 2001. Target countries for the initiative are Bolivia, Brazil, Colombia, Ecuador, Guyana, Peru, Suriname, and Venezuela. The focus of the program, coordinated by PAHO, was to achieve the following objectives and expected results: 1. Reliable and standardized malaria drug efficacy information available. 2. Reliable entomological information available for areas of high transmission risk in each country. 3. Tools and approaches developed, adapted, tested, and disseminated. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 66 4. Partnerships to improve malaria control in the subregion enhanced. Specifically, AMI has assisted countries to analyze drug resistance study findings, and has implemented new treatment guidelines, trained program managers, and adapted vector control interventions to the malaria situation in the Amazon region. The AMI program has achieved a number of key outcomes including: • All eight Amazon Basin countries have changed their malaria treatment policies based on the results of the efficacy studies supported by AMI. In all countries first and second line standard treatment regimens have been brought in line with the epidemiology of the disease in that country. • All eight Amazon Basin countries have accepted the WHO drug efficacy monitoring protocol, ensuring that data are comparable across countries. • All eight Amazon Basin countries have established a continuous drug quality control of antimalarial medicines dispensed at risk areas. • The project’s biannual meetings facilitated the sharing of progress and stimulated collegial discussions through annual meetings. These discussions had a very positive influence on the implementation of the program. Building upon the success of the AMI approach, the South American Infectious Disease Initiative was formed in 2003 to focus efforts on slowing the development of antimicrobial resistance by improving the availability and the use of good quality antibiotics. Coordinated by Management Sciences for Health, SAIDI is implementing pilots of community-based anti-microbial-resistance (AMR) programs to help participating countries find local approaches to contain AMR, tailored to meet each country’s specific needs. Target countries for the initiative are Bolivia, Paraguay, and Peru. To assist countries in the analysis of factors that contribute to the development of AMR and strengthen their capacities to develop interventions appropriate for containing AMR, SAIDI is working toward the following objectives: 1. Gather evidence regarding management and use of AMR in humans for improved decision making. 2. Support countries to develop effective approaches to address AMR focusing on antimicrobial use. 3. Enhance information sharing and dissemination of lessons learned. During the first year of SAIDI, partners focused on collecting information to provide a profile of the problem of AMR in each country. Pre-assessment visits were made and national working groups were formed. In all three countries assessment activities were initiated and this year these assessments were completed. Based on the information gathered through these activities, SAIDI international partners will work with national counterparts to develop effective, multifaceted approaches to address AMR containment at the local level. SAIDI is also working with national partners to identify opportunities for in-country dissemination of information on all phases of SAIDI activities and to promote the sustainability of connections established through the initiative. As information from the assessments becomes available, SAIDI will work with national partners in the preparation of publications, presentations, or other appropriate materials to share this information with stakeholders. Funding for both the AMI and SAIDI initiatives is programmed through FY 2007. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 67 III. Evaluation Objectives The evaluation team will assess the progress made to date in achieving the specific objectives in the agreements and review the programmatic, technical, and managerial strengths and weaknesses of all AMI and SAIDI components by addressing the following evaluation objectives. Based on the findings, the team will present results achieved to date, document lessons learned, and present recommendations for future activities. Information provided in regard to both programs shall be disaggregated under intermediate results (e.g., under AMI, information must be disaggregated to include drug efficacy testing, rapid test, quality control of antimalarials, etc.). Objective 1: Determine the effectiveness of the approach used and outcomes achieved by the initiatives to coordinate and synthesize programming efforts across multiple countries. 1. Assess the suitability and success of the AMI and SAIDI partnership approaches in increasing the efficiency and efficacy of USAID funds in the fight against infectious diseases. a. Is the model effective? b. What alternative models are there? c. Is the model replicable? d. Are the conceptual frameworks used by the initiatives suitable for the containment of infectious diseases in participating countries? 2. Assess the rationale, quality, and outcomes/results of AMI and SAIDI activities planned and implemented to date (i.e., evidence gathered/not gathered/otherwise; approach (es) developed; information shared among USAID partners, between countries, etc.). 3. Assess the rationale, quality, and outcomes/results of activities not originally planned that technical assistance partners have carried out in connection with or as a consequence of planned activities and work/presence in the field 4. Discuss the current situation and future needs of the initiatives. a. What has worked well within the initiative? b. Who else will be supporting the initiative? c. What problems and/or obstacles within the initiative require further tuning? d. What alternate models or options are there? 5. Discuss the sustainability of the initiatives. a. Are changes in policies institutionalized and sustainable? b. Are new community programs sustainable? c. Provide recommendations regarding criteria that should be used for graduation from country assistance. Objective 2: Identify key documents that need to be finalized and packaged for wider dissemination for the benefit of USAID supported efforts. 1. Discuss the applicability of the tools developed under these initiatives. a. Are these tools applied for both a research and a public health approach? b. How have participating countries used the common tools? EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 68 2. Provide recommendations regarding dissemination of key documents to benefit other USAID supported efforts. IV. Methodology The evaluation team will propose a detailed methodology for collecting the necessary information and data and a detailed work plan, including travel schedules. The proposal should include a description of how the methodology responds to the above tasks and questions; and from whom and how the data will be collected and analyzed. The methodology should be collaborative and participatory, including plans for conducting interviews with implementing partners and stakeholders (both in Washington, DC, and in the field). The plan should also include a full review of background materials provided, such as annual reports and important protocols developed. A sample of participating countries will be chosen for each initiative included in this evaluation. SAIDI programs in Peru, Paraguay, and Bolivia will be evaluated; AMI programs in Brazil, Colombia, Peru, and Suriname have been selected for this evaluation. A list of key informants will be identified by USAID and partners. During the two-day startup of the evaluation, initial interviews with USAID and implementing partners will be conducted by the Team Leader through face-to face meetings and/or telephone, including conference calls and computer-based conference calls (for example, SKYPE). During this time the Team Leader, with input from the AMR specialist, will plan the field visit trips and the country interviews. The Team Leader with also prepare a draft report outline for USAID review. The Team Leader, with input from the AMR specialist, will prepare interview instruments. Selected country visits will be determined by USAID in collaboration with the team based on a set of criteria (jointly developed by USAID/LAC and the team) which will help to identify useful initiative program advances and unanticipated outcomes. A team planning meeting will be held in Peru (facilitated by the GH Tech Team Leader) followed by visits to the other selected countries. An illustrative time table can be seen in Section IX below. Following the field visits, the team will travel to DC for follow-up interviews, continue work on drafting sections of the report, and debrief with USAID and implementing partners. The Team Leader will then be responsible for completing the draft of the evaluation report. The Team Leader will revise the draft report based on comments received from USAID/LAC and will travel to Washington for a final debriefing. V. Deliverables Work Plan: The team will prepare a detailed work plan which shall include the methodology to be used in this assessment and a timeline for work. The work plan shall be sent to USAID/LAC for approval no later than the third day of work on this evaluation. Preliminary Report: The team will submit a preliminary report including findings and recommendations upon completion of the mid-evaluation meeting. This report should not exceed 30 pages in length (not including appendices, lists of contacts, etc.). This draft will include findings and recommendations for Bureau and Mission review. USAID will have one week to provide comments and suggestions to the evaluation team, which shall be addressed in the final report. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 69 Final Report: The team will submit a final report no more than two weeks following the Washington, DC debrief meeting. This report should not exceed 50 pages in length (not including appendices, lists of contacts, etc.). The report will be disseminated within USAID and the initiative partners by USAID/LAC Bureau. Debrief: The team will present the major findings to a USAID/Washington audience and partners through a PowerPoint presentation at the conclusion of the evaluation. VI. Reference Materials USAID/Washington will give guidance and provide the evaluators with the reference materials (hard copy and/or electronic links) required for development of the evaluation instruments. Annex 1 provides a selected list of reports, studies, protocols, and other documents that the team should review and take into consideration when preparing for and conducting the evaluation. GH Tech will provide copies of background materials to the team. The evaluation team is expected to collect and annotate additional documents and materials as available. VII. Team Composition and Qualifications The evaluation team shall consist of a Team Leader/Malaria Programs Specialist, an AMR Specialist, and an Assignment Manager. 1. Team Leader/ Malaria Program Specialist will have a dual role as both the team leader and as the Malaria Program Specialist. This consultant should have at least 10 years experience designing, implementing, and evaluating public health programs, with expertise in institutional development especially in the USAID/ LAC region. He/she should have extensive experience conducting qualitative research and carrying out cost￾effectiveness studies. Familiarity with malaria and AMR programming issues is desirable. The Team Leader should also have experience leading evaluation teams and preparing high-quality project reports. He/she should also have a post graduate degree in public health or an applicable social sciences field. Excellent oral and written skills and fluency in Spanish are required. As the Team Leader, he/she will provide leadership for the team, finalize the evaluation methodology design, coordinate activities, arrange periodic team meetings, consolidate individual input from team members, and coordinate the process of assembling the final findings and recommendations into a high-quality document. He/she will be responsible for writing the final report and leading the preparation and presentation of key findings and recommendations to USAID/Washington, implementing partners, stakeholders, and others. As the Malaria Program Specialist, the qualifications are at least 10 years of experience with malaria/infectious disease program analysis. He/she should have experience in program assessment and evaluation methodologies. Familiarity with planning, implementation, and evaluation of USAID activities is desirable. He/she should have a post graduate degree in public health or medicine, with extensive experience in public health aspects of malaria control. Excellent oral and written skills are required and fluency in Spanish or Portuguese is preferred. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 70 The Malaria Specialist will participate in the design of the evaluation methodology and all team meetings, conduct interviews with AMI implementing partners and stakeholders, and provide key findings and recommendations. 2. AMR Specialist with at least 10 years of experience in anti-microbial-resistance programs, including drug selection, procurement, distribution, quality assurance, and use. Familiarity with behavior change communications and community mobilization approaches is required. He/she should also have experience in program assessment and evaluation methodologies. Familiarity with planning, implementation. and evaluation of USAID activities is desirable. He/she should have a post graduate degree in public health or medicine, with extensive experience in public health aspects of anti-microbial resistance. Excellent oral and written skills are required and fluency in Spanish is preferred. The AMR Specialist will participate in the design of the evaluation methodology and all team meetings, conduct interviews with SAIDI implementing partners and stakeholders, and provide key findings and recommendations to the Team Leader for the final report. He/she will participate in the presentation of key findings and recommendations to USAID/Washington, implementing partners, stakeholders, and others. 3. Assignment Manager - Based on the SOW and understanding of the methods and procedures to be applied in the AMI-SAIDI Evaluation, we believe that the third team member should play the role of Assignment Coordinator, providing a cost-effective option for gathering materials and coordinating the efforts of potentially remote team members, while allowing the team leader and AMR specialist to focus on the technical aspects of the task. The Assignment Coordinator would have responsibility for gathering critical information, coordination, and facilitation aspects (especially the complicated travel logistics – travel schedules, Mission travel concurrence cables, ticket purchases) of this task, ensuring that the work moves forward swiftly and smoothly. This would include coordinating meetings and interviews, obtaining documents, supporting the development of tools, performing critical follow-up, and supporting the preparation of the final report and briefings/debriefings with USAID. The Assignment Coordinator would be charged with managing many of the tasks related to bringing the team and information together. We have successfully used this approach in past assignments and found the methodology to be sound and cost-effective and keeps the assignment on schedule. It is estimated that the Level of Effort (LOE) for this assessment is approximately as follows: • Team Leader/Public Health Programs Specialist: 61 person days (10 days fieldwork) • AMR Specialist: 24 person days (7 days fieldwork) • Assignment Manager: 14 person days VIII. Logistics USAID/Washington (LAC/RSD) will provide overall direction to the evaluation team (including country selection), identify and provide key documents, and assist in facilitating implementation of the agreed-upon work plan, including interviews with key personnel. USAID field missions in participating countries will be asked to provide input on key personnel and programs, as well as provide logistics assistance to the team whenever possible. This assistance could include arranging in-country travel and transportation (including airport pickup) and lodging. USAID/LAC and field mission personnel shall be available to the team for consultation regarding EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 71 sources and technical issues before and during the evaluation process. All team country travel will occur only after Mission travel concurrences are received. USAID/LAC will arrange, at a minimum, the following meetings: 1. Team Planning Meeting to review the scope of work, determine the evaluation methodology, finalize the key research questions, and review the evaluation schedule. 2. Mid-evaluation Meeting upon the evaluation team’s return from the fieldwork portion of the evaluation. This meeting will allow the team and USAID/LAC to discuss findings to date and troubleshoot possible obstacles to completing the evaluation as planned. The preliminary report will be presented and discussed at this meeting. 3. Debrief Meeting to be held at the conclusion of the evaluation for USAID/LAC, implementing partners, stakeholders, and others as appropriate. In this meeting the evaluation team will present the findings and recommendations of the final report through a PowerPoint presentation. The evaluation team is responsible for identifying and organizing other appointments and meetings as required during the course of the evaluation. Where necessary, especially with regard to meetings with government officials and stakeholders, USAID/LAC and USAID missions may assist in arranging and/or participate in some of these meetings. IX. Period of Performance and Level of Effort USAID anticipates that the evaluation will begin in late February/early March and will be completed by mid-June, including preparation days, field work, and report writing and finalization. The illustrative timetable of events is as follows: LOE Activity Who Deadline (days) Pre-fieldwork activities Team Leader/AMI Evaluator (NC) Background reading 6 AMR Specialist 6 Travel to DC Team Leader/ AMI Evaluator 1 USAID/LAC Health Team Activity startup interviews Pan American Health Organization Finalize fieldwork schedule Management Sciences for Health Draft report outline 3 March 8,2007 United States Pharmacopoeia Work plan (DC) Team Leader/ AMI Evaluator Travel to NC Prepare data collection instrument(s) Team Leader/ AMI Evaluator 5 March 16, 2007 Fieldwork Travel to Brazil Team Leader/ AMI Evaluator 1 EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 72 LOE Activity Who Deadline (days) Brazil - AMI Team Leader/ AMI Evaluator 4 - Travel to Peru Team Leader/ AMI Evaluator (from Brazil) 1 1 AMR Specialist (from Chile) USAID/LAC Health Team Team planning meeting 1 Team Leader/ AMI Evaluator March 26, 2007 (in Peru) 1 AMR Specialist Peru: Team Leader/ AMI Evaluator 2 - SAIDI AMR Specialist 2 - AMI Travel home to NC Team Leader/ AMI Evaluator 1 Team leader travels back to NC: unavailable period March 30 – April 6 Travel to Bolivia AMR Specialist 1 Bolivia AMR Specialist 2 − SAIDI evaluation Travel to Chile for unavailable period March 30 – April 6 AMR Specialist Travel to Paraguay AMR Specialist 1 Paraguay AMR Specialist 2 --- SAIDI evaluation Travel to Chile AMR Specialist 1 Travel to Suriname Team Leader/ AMI Evaluator 1 Suriname Team Leader/ AMI Evaluator 1 - AMI evaluation Travel to Colombia Team Leader/ AMI Evaluator 1 Colombia Team Leader/ AMI Evaluator 2 − AMI evaluation Team Leader/ AMI Evaluator (from Colombia) 1 Travel to DC Post-fieldwork activities Mid-evaluation Meeting (including preparation for meeting and finalization of preliminary report) Meeting and Preliminary report due March 20 USAID/LAC Team Leader/ AMI Evaluator 5 AMR Specialist (Conference) 1 Implementing partner USAID/LAC Health Team EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 73 LOE Activity Who Deadline (days) interviews Pan American Health Organization Management Sciences for Health United States Pharmacopoeia Team Leader/ AMI Evaluator 1 Team Leader/ AMI Evaluator 20 Draft for editing: May 18 Report writing AMR Specialist 6 Editing Editor 2 To DC (from NC) Team Leader/ AMI Evaluator 1 Debrief Meeting USAID/LAC o/a May 22 Team Leader/ AMI Evaluator 2 Travel to NC Team Leader/ AMI Evaluator 1 Final report o/a June 6 Annex 1 — Selected list of background materials 1. AMI AAD 2. AMI Annual Report Format 3. Malaria—Plans for the Future Presentation 4. AMI Annual Report, 1 October 2003–30 September 2004 5. AMI Final Report for Extension Period, 1 October 2004–31 March 2005 6. SAIDI Biannual Report, October 2004–March 2005 7. Others Annex 2—List of Key Stakeholders and Partners • USAID • Pan American Health Organization (PAHO) • Management Sciences for Health (MSH) • United States Pharmacopoeia (USP) • Ministries of Health of the visited countries and local partners. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 74 ANNEX 2: REFERENCES—DOCUMENTS 1. AMI Annual Reports Comments 1.1 PAHO Annual Report Oct 02 Sep 03.doc The document consolidates information from PAHO and partners. 1.2 PAHO Annual Report 03-04.doc PAHO Annual Report 03-04 has two parts, one in Word and the other in Excel. 1.2.a PAHO Annual Report WP 03 04.xls Final Report of the Grant 2001-2004 (for extension 1.3 PAHO Final Report Oct04-Mar05.doc period Oct 04–Mar 05) 1.4 PAHO Annual Report Oct 04 Sep 05 New Grant.doc New Grant 2004–2007 1.6. BOL Annual Report Oct 04–Sep 05.doc 1.7 CDC Annual Report Oct 04–Sep05.doc 1.8 PERU Annual Report Oct 02–Sep 03.doc 1.9 PERU Annual Report Oct 03–Sep 04.doc PERU Annual Report 03-04 has two parts, one in Word and the other in Excel. 1.9.a PERU Annual Report WP 03 04.xls 1.10 PERU Annual Report Oct 04–Sep 05.doc 1:11 PAHO Annual Report Oct 05–Sep 05 1.12 PAHO Progress Report Oct 06–Mar 07 1.13 CDC Mid-Year Report FY 07 EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 75 Calendars and Newsletters Comments 2.1 Calendar 2004-2005 2.2 Calendar 2005-2006 2.3 No. 1 Oct-Dec04 2.4 No. 2 Jan-Mar05 2.5 No. 3 Apr-Jun05 2.6 No. 4 Jul-Sep05 2.7 No. 5 Oct-Dec05 2.8 No. 6 Jan-Mar 06 2.9 No. 7 Apr-Jun 06 Workshops and meetings 2006 3.1 Minutes of Annual and SC meeting Apr 06.doc Quito, Ecuador – 25-27 abril 2006 3.1.a Agenda Quito English.doc 3.2 Reunión Panamá.Entomología.doc Ciudad de Panamá, Panamá - 1-4 agosto 2006 3.2.a guía para el control vectorial.doc 3.3 Informe en Inglés Taller sobre medición sérica Belém, Brasil, 29 mayo – 2 junio 2 2006 3.4 Quantification Agenda Preliminar_English.doc Santa Cruz, Bolivia - 22-25 agosto 2006 3.4.a Data for the estimation of needs_English.doc 3.5 Agenda Taxonomia Sep12, 06.doc Bogotá, Colombia, 25-28 de septiembre 2006 (a realizarse) 3.6 CD-ROM with presentations from annual meeting Campo de Jordao, Brazil Mar 07 3.7 Investigación operacional sobre la implementación del uso de pruebas rápidas de diagnóstico de malaria Guayaquil – mayo 23 – 25 de 2005 2.10 No. 8 Jul-Sep06 2.10 No. 9 Oct-Dec 06 (draft) EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 76 EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 77 WORKPLANS 4.1 BOL WP4.xls 4.2 BRA WP4.DOC 4.3 CDC WP4.xls 4.4 COL WP4.DOC 4.5 ECU WP4.DOC 4.6 GUY WP4.DOC 4.7 REGIONAL WP4.DOC 4.8 SUR WP4.DOC 4.9 USP WP4.doc 4.10 VEN WP4.DOC WP4 stands for the period 2004-2005; these were sent to USAID as separate documents on either Excel or Word. 4.11 BOL WP5.doc 4. 12 CDC WP5.xls 4.13 PAHO WP5 06.DOC 4.14 USP WP5.doc WP5 stands for the period 2005-2006. PAHO WP5 consolidates workplans of BRA-COL-ECU-GUY￾SUR-VEN-REG 4.15 Work plans (all) for FY 2007 In Excel format PUBLICATIONS Economic impact of malaria in Peru USAID-MOH Lima, 2000 Modifying national malaria treatment policies in Peru Trenton K Ruebush II, Daniel Neyra, César Cabezas, Journal of Public Health Policy, 2004, 25 (3/4), Health Module pg. 328 El proceso de adecuación y cambio en la política del tratamiento de la malaria por Plasmodium falciparum en el Perú, 1990-2001 Rev. Peru med exp salud publica 2003; 20 (3) Costo efectividad del cambio de los esquemas de tratamiento para malaria en el Perú (1999-2003) Rev. Peru Med Exp Salud Publica. 2004, 21(4) Selection of an Economic Evaluation Methodology for the Main Interventions for Prevention and Control of Malaria Used by the Ministry of Health VIGIA Project, Lima, June 12, 2001 OTHER AMI AAD Draft 8-3-01 Access and Use Draft MSH, Mar. 2, 2007 2. SAIDI FILE NAME DOCUMENT TITLE DESCRIPTION Documento de Presentacion SAIDI, Jan 2005.doc Iniciativa de Enfermedades Infecciosas en América del Sur (SAIDI): Apoyando el Desarrollo de Estrategias Locales para Contener la Resistencia Antimicrobiana en Países de la Región Andina y Paraguay Spanish. Introductory document distributed primarily in pre-assessment visits that describes basis for the initiative, objectives, initial plan of action, and information on each partner. SAIDI Biannual Report to USAID April 29 05.doc South American Infectious Disease Initiative Biannual English. Report to USAID, Joint report prepared by RPM Plus with input from other partners. Describes main accomplishments and next steps from October 2004 to March 2005. October 2004 – March 2005 SAIDI Biannual Report to USAID April–Oct 05.doc South American Infectious Disease Initiative, SAIDI English. Biannual Report to USAID, Joint report prepared by RPM Plus with input from other partners. Describes main accomplishments and next steps from April to October 2005. April 2005 – October 2005 SAIDI Annual Report to USAID April 05 –March 06.doc South American Infectious Disease Initiative, SAIDI English. Biannual Report to USAID, Joint report prepared by RPM Plus with input from other partners. Describes main accomplishments and next steps from Apr 2005 to March 2006. April 2005 – March 2006 SAIDI Products Indicators and Outcomes Nov 05.xls USAID South American Infectious Disease Initiative English. Products, Monitoring Indicators and Outcomes for FY05 Activities Chart of products, indicators, and outcomes for each activity included in SAIDI partners’ FY05 work plans for review during partners meeting in October 2006. SAIDI Goals PP.ppt SAIDI Objectives (PowerPoint slide) SAIDI objectives as agreed by partners in a December 2004 meeting. LAC INF (SAIDI) FINAL.doc RPM Plus work plan for the South American Infectious Disease Initiative (SAIDI) Proposed RPM Plus SAIDI activities for Oct 04– Sept 05 October 2004–September 2005 EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 78 FILE NAME DOCUMENT TITLE DESCRIPTION SAIDI FY05 Workplan LAC.doc RPM Plus work plan for the South American Infectious Disease Initiative (SAIDI) Proposed RPM Plus SAIDI activities for Oct 05– Sept 06 October 2005–September 2006 RPM Plus LAC SAIDI FY06 Workplan Final.doc RPM Plus work plan for the South American Infectious Disease Initiative (SAIDI) Proposed RPM Plus SAIDI activities for Oct 06– Sept 07 October 2006–September 2007 Yeager_Paraguay_Feb2005_Trip Rep SAIDI Pre-Assessment Visit to Paraguay Trip report from pre-assessment visit to Paraguay February 14–18, 2005 Trip Report Yeager_Paraguay_July2005_Tri pRep SAIDI Assessment Visit to Paraguay Trip report from assessment trip to Paraguay June 12–23, 2005,Trip Report Yeager_Paraguay_SAIDI_Aug 2005_TripRep Follow-up Visit to Paraguay for SAIDI Assessment Trip report to follow up assessment activities August 8 – 13, 2005: Trip Report Barillas_Paraguay_December20 05_Trip Rep Rational Pharmaceutical Management Plus Trip report on TB TA provided Informe de Viaje a Paraguay: Diciembre de 2005 Barillas_Paraguay_Spanish_Trip Rep_03 06 Taller de Distribución de Medicamentos e Insumos Farmacéuticos para la Tuberculosis Follow-up to TB TA provided and training Asunción, Paraguay 27 – 29 de Marzo de 2006 Yeager_Barillas_Sosa_Barojas_ Paraguay_June2006_TripRep Workshop with SAIDI national and international partners to prioritize the objectives and activities of a plan to contain and prevent antimicrobial resistance in Paraguay and SAIDI Steering Committee Meeting, June 20–June 30, 2006 Presentation of assessment results and planning of focus activities SAIDI_Bolivia_May2005_ TripRep-2 SAIDI Pre-Assessment Visit to Bolivia Trip report from initial visit to Bolivia May 9–13, 2005, Trip Report Yeager_Peru_Feb2005_TripRep SAIDI Pre-Assessment Visit to Peru Trip report from initial visit to Peru February 28–March 4, 2005 Yeager_SAIDI_Peru_July2005_ TripRep Meeting with SAIDI national partners Planning meeting with national partners in Peru Lima, Peru, June 24–27, 2005 Yeager_Sanchez_Sosa_Peru_S Meeting with SAIDI national partners Planning meeting with national partners in Peru EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 79 FILE NAME DOCUMENT TITLE DESCRIPTION ept2005_TripRep Lima, Peru, September 21–23, 2005 Yeager_Peru_January2006_Trip Rep SAIDI Assessment Visit to Peru: Preparation and initiation of assessment activities Preparation for AMR assessment activities in Callao in Peru January 23 – 27, 2006, Trip Report Yeager_Smine_Sosa_Sanchez_ Peru_April2006_TripRep Workshop with SAIDI national and international partners to prioritize the objectives and activities of a plan to contain and prevent antimicrobial resistance in Callao and SAIDI Steering Committee Meeting, April 17–21, 2006: Trip Report Workshop entailed presentation of assessment results and planning of focus activities. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 80 FILE NAME DOCUMENT TITLE SAIDI WP FY06.doc USAID South American Infectious Disease Initiative Proposed Work Plan for the Period October 2005 Through September 2006, USP DQI Work Plan SAIDI WP FY05.doc South American Infectious Diseases Initiative, USP DQI Proposed Work Plan for the Period of October 2004 through September 2005 SAIDI Working Plan Draft FY 07.doc USAID South American Infectious Disease Initiative Proposed Work Plan for the Period October 2006 through September 2007, USP DQI Work Plan Peru-2006, sent 3-9-06.doc USP DQI Trip Report- PERU, January 16–22, 2006 Peru-2006, sent 2-27-06.doc USP DQI Trip Report Training Workshop on Drug Registration Using the World Health Organization SIAMED Software, PERU, January 16–22, 2006 Peru-2006, sent 2-16-06.doc Trip Report–A. Smine, N. Davydova, A. Barojas USP Drug Quality and Information Program Peru—January 16–22, 2006 Peru-2005, sent 5-23-05.doc Trip Report—Drs. Dat Tran and Edwin Toledo USP Drug Quality and Information Program Lima, Peru – February 18–March 4, 2005 Peru-2005, sent 12-1-05.doc Trip Report – Abdelkrim Smine USP Drug Quality and Information Program Peru, October 24–25, 2005 Paraguay2005, sent 10-28-05.doc USP DQI Trip Report–Paraguay September 19–23, 2005 Paraguay2005, sent 3-14.doc Trip Report–Nancy Blum USP Drug Quality and Information Program Paraguay–February 12–19, 2005 Paraguay2005sent 8-4-05.doc Trip Report–Dat Tran and Adrian Barojas USP Drug Quality and Information Program Paraguay–June 14–17, 2005 Bolivia-2005- sent 8-22-05.doc Trip Report–Abdelkrim Smine USP Drug Quality and Information Program Bolivia, 9–13 May 2005 EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 81 EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 82 FILE NAME DOCUMENT TITLE Bolivia-2006- sent5-9-06.doc Trip Report–N. Davydova, Ph.D. USP Drug Quality and Information Program Bolivia—March 13–17, 2006 Bolivia-2005- sent 1-20-06.doc USP DQI Trip Report–Bolivia November 28–December 2, 2005 355 IIH (PAR) Evaluación del sistema de vigilancia de enfermedades infecciosas emergentes en en Paragraph, Septiembre–Octubre 2005 349 IIH (PER) LA SITUACION DE LA PREVENCION Y CONTROL DE LA INFECCION HOSPITALARIA EN PERU Septiembre 2005 326 Eval resist antib (NIC) Evaluación del sistema de vigilancia de enfermedades infecciosas emergentes Vigilancia de la resistencia a los antibióticos Managua, Nicaragua Octubre, 2005 365 Eval IIH y Resist Antib (ELS-Final) Evaluación del sistema de vigilancia de enfermedades infecciosas emergences Vigilancia de la resistencia a los antibióticos y control de la infección hospitalaria El Salvador Octubre 2005 PAR-ARG(jun-jul) Informe de Viaje–Gabriel Schmunis Asunción, Paraguay y Buenos Aires, Argentina 26 mayo–4 julio, 2006 PER -SAIDI(Abr) Informe de Viaje–Gabriel Schmunis Lima, Perú 19–20 abril, 2006 PAR(Abr) Informe de Viaje–Gabriel Schmunis Asunción, Paraguay 29 Marzo–7 Abril, 2006 BOL(Mar) Informe de Viaje–Gabriel Schmunis Santa Cruz, Bolivia 7–12 marzo, 2006 OTHER FILES SUBMITED FOR SAIDI EVALUATION DOCUMENT TITLE AUTHOR DESCRIPTION Guia Práctica para la Interpretación Clínica del Antibiograma Pocket publication supported by pharmaceutical industry in Bolivia during SAIDI period to orient general physicians to analyze and interpret antibiograms Dr. Christian Trigoso Dra. Esther Damián APUA and INLASA Bolivia 2006 Boletines Informativos sobre uso de medicamentos UNIMED Bolivia 2006 Series of bulletins supported by PAHO giving recommendations of rational use of medicines Boletines Informativos sobre uso de medicamentos AIS Bolivia 2006 Series of bulletins supported by NOVIB (Holland) and MISEREOR (German) giving recommendations for rational use of medicines Petitorio nacional de medicamentos esenciales MoH Peru 2005 Pocket publication supported by Vigia Project and USAID that includes current national law, and description of essential drugs with active principles and doses for use Manual de buenas prácticas de prescripción MoH Peru 2005 and 2007 Publication supported by Vigia Project and USAID giving recommendations on good prescription of medicines Estudio sobre los factores determinantes de la prescripción y dispensación de antibióticos por médicos, químicos farmacéuticos y vendedores de farmacias/boticas de El Callao, Perú Links Media and APUA Peru 2006 Results of a study about factors that determine prescription and dispensing of antimicrobials by physicians and pharmacists in Callao, Peru, supported by SAIDI Estudio sobre los factores determinantes del uso de antibióticos en los consumidores de El Callao, Perú Links Media and APUA Peru 2006 Results of a study about factors that determine use of antimicrobials by consumers in Callao (districts of Bellavista, La Perla, and Carmen de la Legua), supported by SAIDI Evaluación de la situación de los antimicrobianos en la red BEPECA de la Dirección de Salud Callao DIGEMID, MSH, DISA CALLAO, Peru 2006 Preliminary results of transversal analysis of access, use, and availability of antimicrobials in public and private pharmacies of the BEPECA network in Callao, supported by SAIDI EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 83 ANNEX 3 DISCUSSION GUIDES March 29, 2007 AMI Evaluation Discussion Guide AMI Evaluation Date: __________ Country: __________ Name: _______________ Position: ____________ Location of interview: ___________________ Introduction: (To be read to the interviewee) The USAID Latin American and Caribbean Bureau (USAID/LAC) has requested that the USAID/Washington GH TECH Project conduct an external evaluation of the Amazon Malaria Initiative (AMI). The results of this evaluation will be used to provide the basis for USAID/LAC to 1. Determine whether funding for AMI should be extended or a new regional infectious disease initiative should be designed. 2. Assess the progress of AMI toward achieving its expected results. 3. Document lessons learned from implementing the management model used for coordinating AMI. The objectives of this evaluation are to 1. Determine the effectiveness of the approach used and outcomes achieved by the initiatives to coordinate and synthesize programming efforts across multiple countries. 2. Identify key documents that need to be finalized and packaged for wider dissemination. The evaluation team will assess the progress made to date in achieving the specific objectives in the agreements and review the programmatic, technical, and managerial strengths and weaknesses of all AMI components. Based on the findings, the team will present results achieved to date, document lessons learned, and present recommendations for future activities. USAID/LAC has supported AMI since 2001. Target countries for the initiative include Bolivia, Brazil, Colombia, Ecuador, Guyana, Peru, Suriname, and Venezuela. The overall result for AMI is: “Los programas de control de la malaria en la Sub-Región Amazónica han incorporado buenas prácticas de manera substancial The achievement of this final expected result was supported by three Intermediate Results (IR): IR1) Evidence base increased; IR2) Evidence base communicated and used; and IR3) More inclusive and better informed policy process promoted. The focus of the program was to achieve the following objectives and expected results: 1. Reliable and standardized malaria drug efficacy information available. 2. Reliable entomological information available for high-transmission-risk areas in each country. 3. Tools and approaches developed, adapted, tested, and/or disseminated. 4. Partnerships to improve malaria control in the subregion enhanced. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 84 Specific Questions: We are now going to ask you some specific questions about the program. All information that you provide will be strictly confidential. Also, with your permission, as a memory aide, I would like to record this interview. The recording will not be shared with anyone else. Please take a minute to explain what your participation in or relationship has been to the AMI initiative, in what capacity, and how long you have been associated with it. Objective 1: Determine the effectiveness of the approach used and outcomes achieved by the initiatives to coordinate and synthesize programming efforts across multiple countries. In order to design, implement, and oversee the project, USAID promoted a subregional partnership approach for joint priority setting among the implementing partner agencies, including PAHO, MSH/RPM Plus, CDC, and USP DQI in addition to USAID. These implementing partners collaborated on all levels of decision making, including priority setting, development and approval of the work plans, monitoring of implementation, and assignment of resources. 1.1 In this section of the interview we will discuss what has been your experience with this sub-regional partnership approach. a. Is the AMI partnership approach effective in achieving the project results? What would you say were the strengths and weaknesses of this approach? b. Can you suggest alternative models or approaches for a subregional initiative? If so, what do you see as the advantages of these alternate approaches? c. What has been the value-added this subregional approach brings to the initiative as opposed to a country by country approach? d. Is the AMI subregional model replicable in other subregions, and what would be the criteria or situational determinants that would make it an applicable model for replication in a given subregion? AMI applied a conceptual model based on the hypotheses that • Vector and treatment control measures in neighboring countries need to be harmonized • Targeting resources to selected activities in priority countries through a common framework could improve malaria control at the subregional level. • Establishing a surveillance network using standardized techniques would permit analyzes and comparisons that would in turn lead to more effective and coordinated response measures. • Fostering partnerships would promote learning across countries and the leveraging of technical and financial resources for better malaria control. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 85 1.2 Now we are going to discuss this conceptual model: a. Was the conceptual model used by AMI suitable for containing malaria in participating countries? Please explain. b. What could be done to improve this conceptual model? What components are essential to its success? What are not so important? Are there other important components that could contribute to its success? 1.3. In relation to the AMI decision-making and priority-setting process: a. How would you assess the decision-making process that determined what activities were to be carried out and the rationale behind those decisions? b. Did this process reach an appropriate balance between the need for methodological rigor and standardization, and the need for practical public health applications? If not, how could a more appropriate balance have been achieved? c. In your opinion, were the activities carried out and evidence gathered the most appropriate ones for containing malaria, or were there other important ones that were not included in the initiative? d. How could this priority setting process be improved? 1.4 Tools and approaches developed, adapted, tested and/or disseminated under the initiative included drug efficacy testing; rapid tests; quality control of antimalarials; and improved entomological information and strategies. a. What outcomes/results of AMI activities planned and implemented to date (i.e., evidence gathered; approaches developed; regulations, guidelines, research completed and disseminated; new programs, etc.) in your country are you familiar with? b. How would you assess their quality and practical relevance to your country program’s needs? Please be specific in referring to particular tools and products. c. How did you keep informed about relevant AMI activities, information, and approaches? Was the initiative sufficiently proactive and effective in disseminating information, lessons learned, and state of the art? d. How could information dissemination be improved at the subregional level? e. How were these findings, products, protocols, norms, and guides disseminated to the appropriate levels within your country? f. How could the information dissemination process within your country be improved? EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 86 1.5 Are you familiar with any results of activities not originally planned or contemplated that technical assistance partners have carried out in connection with or as a consequence of planned activities and work/presence in the field? How would you assess their rationale, quality, and outcomes? Please cite specific examples. 1.6 Now we will discuss the current situation and future needs of the initiative. a. What has worked well within the initiative? b. What other stakeholders (individuals or institutions) in your country should be supporting the initiative through their participation as partners (can prompt with examples: research institutes, medical faculties, other branches of the MOH at the central or other levels, private sector partners, local governments)? c. What problems and/or obstacles are there within the initiative that require further modification or adjustments? Objective 2: Identify key documents that need to be finalized and packaged for wider dissemination for the benefit of USAID-supported efforts. 2.1 This initiative has promoted changes through the development and application of a number of tools (research protocols, norms and guidelines, etc.). a. What changes in policies or practices have occurred in your country as a result of this initiative? b. What has been the applicability of the tools developed under these initiatives? Please cite specific examples. If not applicable, why not? c. Are these tools applied for both a research and a public health approach? Please be specific. d. How have participating countries used the common tools? Please be specific in mentioning which tools have been used and what has been the experience of using these tools. 2.2 What key documents and tools developed by this initiative merit broader dissemination and why? 2.3 Now we will discuss the institutionalization and sustainability of the initiative: a. Are changes in policies adopted institutionalized and sustainable? What have been the obstacles to their adoption and implementation, and have they been overcome? If not, what could be done to overcome them? b. Are new programs at the different levels (national, provincial, and local) sustainable? What have been the obstacles to sustainability at the various EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 87 levels and have they been overcome? If not, what could be done to overcome them? c. Is the country now allocating or planning to allocate budgetary resources for activities supported under the initiative? If so, which ones? Which other critical ones should it also be planning to cover? d. Provide recommendations regarding criteria that should be used for graduation of country assistance from direct project support for specific activities. 3. Should USAID continue to support this initiative (yes/no)? a. Why? b. If yes, what aspects of the initiative should be carried forward and how could it be improved? Should anything be eliminated? c. Should USAID be supporting some other malaria/infectious disease initiative instead? If so, why and what shape or form should the support take? d. What is has been the single most valuable achievement/outcome of AMI for your country? Finally, is there anything else you would like to share in terms of the initiative’s results, products, implementation, and management/coordination; or any recommendations for the future? EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 88 March 16, 2007 Draft Interview Instrument SAIDI Evaluation Date: __________ Country: __________ Name: _______________ Position: ____________ Location of interview: ___________________ Introduction: (To be read to the interviewee) The USAID Latin American and Caribbean Bureau (USAID/LAC) has requested that the USAID/Washington GH TECH Project conduct an external evaluation of the South American Infectious Disease Initiative (SAIDI). The results of this evaluation will be used to provide the basis for USAID/LAC to 1. Determine whether funding for SAIDI should be extended or a new regional infectious disease initiative should be designed. 2. Assess the progress of SAIDI toward achieving its expected results. 3. Document lessons earned from implementing the management model used for coordinating SAIDI. The objectives of this evaluation are to: 1. Determine the effectiveness of the approach used and outcomes achieved by the initiatives to coordinate and synthesize programming efforts across multiple countries. 2. Identify key documents that need to be finalized and packaged for wider dissemination. The evaluation team will assess the progress made to date in achieving the specific objectives in the agreements and review the programmatic, technical, and managerial strengths and weaknesses of all SAIDI components. Based on the findings, the team will present results achieved to date, document lessons learned, and present recommendations for future activities. Building upon the success of the AMI approach, the South American Infectious Disease Initiative was formed in 2003 to focus efforts on slowing the development of antimicrobial resistance by improving the availability and the use of good quality antibiotics. SAIDI is implementing pilots of community-based anti-microbial-resistance (AMR) programs to help participating countries find local approaches to contain AMR, tailored to meet each country’s specific needs. Implementing partners include MSH/RPM Plus, APUA, Links Media, USP/DQI, PAHO, and CDC. Target countries for the initiative include Bolivia, Paraguay, and Peru. To assist countries in the analysis of factors that contribute to the development of AMR and strengthen their capacities to develop interventions appropriate for containing AMR, SAIDI is working toward these Intermediate Results: IR1: Increase the evidence base of factors contributing to the emergence of AMR in LAC; IR2: Improve local capacity to develop and implement appropriate interventions to contain AMR (focusing on antimicrobial use); and IR3: Disseminate information and lessons learned from community-level initiatives. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 89 Specific Questions: We are now going to ask you some specific questions about the program. All information that you provide will be strictly confidential. Also, with your permission, as a memory aide, I would like to record this interview. The recording will not be shared with anyone else. Please take a minute to explain what your participation in or relationship has been to the AIM/SAIDI initiative, in what capacity, and how long you have been associated with it. Objective 1: In order to design, implement, and oversee the project, USAID promoted a regional partnership approach for joint priority setting among the implementing partner agencies. PAHO, MSH/RPM Plus, and other contributing partners, including the RPM Plus Project, CDC, USP DQI, Links, and AUPA) in addition to USAID. These implementing partners collaborated on all levels of decision making including development and approval of work plans, monitoring of implementation, and assignment of resources. 3. In this section of the interview we will discuss what has been your experience with this regional partnership approach. a. Is the SAIDI partnership model effective in achieving the project results? What would you say were the strengths and weaknesses of this approach? b. Can you suggest alternative models for a regional initiative? If so, what do you see as the advantages of these alternate approaches? c. What components of the model are essential for its success? What are other important facilitating factors that could contribute to its success? d. Is the regional model replicable, and what would be the criteria or situational determinants that would make it an applicable model for replication in a given subregion? e. Were the conceptual frameworks used by the regional initiative suitable for containing infectious diseases in participating countries? Please explain. What could be done to improve these frameworks? f. What is the value-added this regional approach brings to the initiative? 4. In relation to the decision-making and priority-setting process: a. How would you assess the rationale and decision-making process that determined what activities were to be carried out? b. Was this process sufficiently participative in striking a balance between the need for technical and scientific rigor and the need for practical public health applications? c. How could this priority-setting process be improved? EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 90 5. Tools and approaches developed, adapted, tested, and/or disseminated under the project included, but are not limited to, drug efficacy testing; rapid tests; quality control of antimalarials; and improved entomological information and strategies. a. What was the AMR situation in the country before the SAIDI project? Had there been a previous diagnosis? b. What outcomes/results of SAIDI activities planned and implemented to date (i.e., evidence gathered; approaches developed; regulations, guidelines, research completed and disseminated; new programs, etc.) in your country are you familiar with? c. How would you assess their quality and relevance to your country program’s needs? Please be specific in referring to particular tools and products. d. How did you keep informed about SAIDI activities and AMR information and approaches? Was the initiative sufficiently proactive and effective in disseminating information, lessons learned, tools, and state of the art approaches? e. How could the information dissemination be improved? 6. Are you familiar with any results of activities not originally planned or contemplated that technical assistance partners have carried out in connection with or as a consequence of planned activities and work/presence in the field? How would you assess their rationale, quality, and outcomes? Please cite specific examples. 7. Now we will discuss the current situation and future needs of the initiative. a. What has worked well within the initiative? b. What other potential sources of support are there for the initiative? c. What problems and/or obstacles are there within the initiative that require further modification or adjustments? Objective 2: Identify key documents that need to be finalized and packaged for wider dissemination for the benefit of USAID-supported efforts. 1. This initiative has promoted changes through the development and application of a number of tools (research protocols, norms and guidelines, etc.). a. What changes in policies or practices have occurred in your country as a result of this initiative? b. What has been the applicability of the tools developed under these initiatives? Please cite specific examples. c. Are these tools applied for both a research and a public health approach? Please be specific. EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 91 d. How have participating countries used the common tools? Please be specific in mentioning which tools have been used and what has been the experience of using these tools. 2. What key documents and tools developed by this initiative merit broader dissemination and why? 3. Now we will discuss the sustainability of the initiative: a. Are changes in policies adopted institutionalized and sustainable? What have been the obstacles to their adoption and implementation, and have they been overcome? If not, what could be done to overcome them? b. Are new community programs sustainable? What have been the obstacles to sustainability at this level and have they been overcome? If not, what could be done to overcome them? c. Is the country now allocating or planning to allocate budgetary resources for activities supported under the initiative? If so, which ones? Which other critical ones should it also be planning to cover? d. Provide recommendations regarding criteria that should be used for graduation of country assistance from direct project support for specific activities. 3. Should USAID continue to support this initiative (yes/no)? a. Why? b. If yes, what aspects of the initiative should be carried forward and how could it be improved? Should anything be eliminated? c. Should USAID be supporting some other malaria/infectious diseases initiative (not including avian virus and HIV/AIDS) instead? If so, why and what shape or form should the support take? Finally, is there anything else you would like to share in terms of the initiative’s results, products, implementation, and management/coordination; or recommendations for the future? EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 92 ANNEX 4 PERSONS CONTACTED Persons Interviewed by Country—AMI Country Person Organization/Responsibility Peru Dr. Jaime Chang USAID Technical Officer Dr. Angel Rosas PAMAFRO Dr. Alejandro Llanos PAMAFRO Dr. Mario Valcarcel PAHO, Adviser Luz Esther Vazquez Director, VIGA Vazquez Wilder Carpio Montenegro Technical Adviser/VIGA Elena Ogusuka Asato Coordinator, DIGESA Leoni Herrera Hurtado MoH, Technical Team Adviser, National Health Institute Cesar Cabezas Coordinator, ESNP y CENFMETAX, MoH Luis Miguel León Garcia PAHO Adviser, Malaria Control Program Colombia Dr. Jose Pablo Escobar Dr. Gilberto Alvarez Asesor Viceministerio de Salud, Director General de Salud Pública Coordinador Nacional Programa de Prevención y Control de Malaria y otras enfermedades transmitidas por vectores (ETV) Dr. Padilla Médico Epidemiólogo, Asesor Internacional Enfermedades Transmisibles, OPS/OMS Dr. Roberto Sempertegui, Colombia OPS/OMS Profesora Investigadora, Universidad de Antioquia, Dra. Silvia Blair (Tel) Medellín Investigadora Centro Internacional de Entrenamiento e Investigación Medicas (CIDEIM), Cali Dra. Lyda Osório (Tel) Consultor por producto Proyecto IAM – RAVREDA Departamento del Choco, Dr. Fredy Córdoba (Tel) OPS/OMS Coordinador Laboratorio Parasitología, Instituto Nacional e Salud (INS) Coordinadora Laboratorio de Entomología, INS Dr. Santiago Nichols (Tel) Coordinadora Programa EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 93 Country Person Organization/Responsibility Prevención y Control de Malaria y otras ETV Dra. Ligia Lugo (Tel) Profesora Investigadora, Facultad de Medicina, Universidad Nacional de Colombia Dra. Ligia Pérez (Tel) Dra. Martha Quiñónez (Tel) Coordinador Programa de Prevención y Control de la malaria y otras ETV, Secretaria Dr. Humberto Escobar (Tel) de Salud del Valle Coordinadora Operativa Programa de Prevención y Control de Malaria y otras ETV, Instituto Departamental de Salud de Nariño Dra. Pilar Pérez (Tel) Quimica Farmaceutica, Laboratorio Departamental de Salud Pública de Antioquia Olga Lucia Muñoz (Tel) PAHO–AMI Regional Coordinator Brazil Dr. Roberto Montoya Paola Marchesini PAHO–AMI Country Adviser Country AMI Coordinator, Dra. Rosali La Corte Entomology NMP Director Dr. Ladislau Medical Director NMP Dr. Magabiera Drug management Maria de Paz Luna Efficacy studies Rui Moreira Braz Efficacy studies, ITM/Manuas Franklin Simoes Reference Laboratory, Chagas Elza Pereira Inst. In vitro studies and medicine quality control, Chagas Inst. Marinete Póvoa Giselle Viana Suriname Dr. Stephan Vreden RAVREDA Coordinator Ms. Helene Hiwat Global Fund/Entomology Dr. Glenn Lavenberg Global Fund Dr. Marthele Eersel Director General for Health Dr. Dayanand Panchoe Director of Entomology, MoH Medical Research Institute Dr. Adhin Bolivia Dr. Marco Fidel Suarez PAHO Adviser Dr. Juan Carlos Arraya NMP Coordinator EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 94 Country Person Organization/Responsibility Stanley Blanco USAID/La Paz Guyana Dra Tamara Mancero PAHO Adviser Karancha Krishnallal NMP Director Ecuador Dr. Delmin Cury PAHO Transmittable Diseases Adviser Dr. Raul Veloz AMI Project Coordinator, NMP Venezuela Dr. Soledad Perez PAHO Adviser United States Peg Marshall (Tel) USAID LAC Trenton Ruebush (Tel) USAID GH Susan Bacheller USAID GH Ray Beach (Tel) CDC Alex Macedo (Tel) CDC Maria Miralles MSH/RPM Plus Melissa Thumm (Tel) MSH/RPM Plus Victor Pribluda USP DQI Adrian Barojas USP DQI Keith Carter PAHO Lourdes Barrios PAHO Rainier Escalada PAHO Geneva Dr. Pascal Ringwald WHO EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 95 Persons Interviewed by Country —SAIDI Peru Name Institution Contact information Relation with SAIDI Dr. Jan Karlo Zavalaga DIGEMID 011-511-422-9200, ext. 412 Coordinator of SAIDI rational use of activities medicines jkzavalagam@hotmail.com jzavalaga@digemid.minsa.g ob.pe Dr. Susana Vasquez DIGEMID 011-511-422-9200, ext 411 Technical support in charge of rational use of medicines svasquez@digemid.minsa.g ob.pe Dr. Victor Dongo DIGEMID 011-511-422-9200 Technical support General Director vdongo@digemid.minsa.go b.pe Dr. Rossana Geng Proyecto VIGIA 011-511-3303643 Coordination of SAIDI Consultant in use of activities medicines rgengolaechea@yahoo.es Dr. Luz Esther Vasquez Proyecto VIGIA 011-511-330-3643 Technical support Director Vasquez23@telefonica.net. pe Angie Caballero Communications 51-1-4456147 consultant hired by Links Media for SAIDI activities In charge of communication acaballero@linksmedia.net strategy Dr. Cesar Sangay President of Peru APUA Chapter cesarsangayc@yahoo.com Review of guidelines Dr. Jorge Velasquez DISA Callao 511-963-73071 (cell) Technical support Director of Medicines 511-465-5279 or 511-429- 1424 (DISA) dmcallao@minsa.gob.pe Dr. Marisela Mallqui DISA Callao 511-649-953 (cell) Coordinator of SAIDI Director of Health activities of the People 511-465-5279 or 511-429- 1424 (DISA) gmmallqui@yahoo.es Dr. Edson Mesa AIS NGO 511-346-2325 National partner emezacor@yahoo.es ais@aislac.org Dr. David Vivar PROVIDA NGO National partner Dr. Javier Yamoza Municipality of Lima National partner EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 96 Paraguay Name Institution Contact information Relation with SAIDI Dr. Graciela Gamarra MoH 011-595-21-222013 (work) CVoordinator for SAIDI chronic diseases 011-0983-496074 (cell) Paraguay activities maria.cairo@gmail.com Dr. Margarita Villafañe Science Faculty, University of Asuncion Coordination of studies in diagnosis phase Nancy Holt RN MoH National Nosocomial Infection Program Dr. Estela Quiñones MoH National Nosocomial Infection Program Studies of medicine quality Dr. Mirta Emeri Dirección Nacional de Vigilancia Sanitaria (National Direction of Sanitary surveillance) Dr. Wilma Basualdo Infectious disease specialist, 011-595-0971-252748 (cell) Coordination of studies in wdb@rieder.net.py diagnosis phase; Pediatric Hospital review of guidelines Dr. Ana Campuzano de Rolon Pediatric infectious disease specialist, rogger@conexion.com.py Hospital Materno￾Infantil Coordination of studies in diagnosis phase; review of guidelines Dr. Zully Vera de Molinas Chief of Medicine Information Center 011-595-21-0971-738940 (cell) SAIDI assessment activities and coordcim@qui.una.py implementation of CIM Dr. Mercedes Carrillo Director, National Public Health Institute 011 595-21 294999 First coordinator of SAIDI mechicarrillo@hotmail.com initiative Dr. Juan Jara In charge of National TB Program, MoH Coordinator of SAIDI TB activities Liliana Espinola Carmen Buzarquis CEMIT University of Asuncion Collaboration with CIM communications platform Rosa Ramirez Julia Zelaya Evaluation of medicine QA Gladys Lugo Edmundo Granada EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 97 Bolivia Name Institution Contact information Relation with SAIDI Dr. Marco Fidel Suarez PAHO / Paraguay 011-591-2-241-2465 or 241- 2313 ext. 651 Accompanied SAIDI from Infectious Disease the start Consultant masuarez61@gmail.com msuarez@bol.ops-oms.org Dr. Christian Trigoso Director, INLASA 011-591-2-725-30175 (cell) Accompanied SAIDI partners on the pre￾assessment visit in May 2005 President, APUA Bolivia chtrigoso@latinmail.com 591-2-2200489 Participation and coordination of SAIDI TB strategies Dra. Mirtha Camacho Chief, National TB Program tbcos@hotmail.com Dra. Olga Fujita Director, 591-2-2440122 Technical support UNIMED ofujita@sns.gov.bo Dra. Susana 591-2-2440122 Sanjinez In charge of Supply Unit, UNIMED Participation and coordination of SAIDI strategies in drug use susysanjines@hotmail.com ssanjines@sns.gov.bo 591-2-2 441479 / 2440378 Has not had any relationship so far Dra. Ruth Calderon Chief, MoH Quality Department somaroca@yahoo.com Dr. Ramiro 591-2-2441479 Asturizaga Quality of Services, MoH Has not had any relationship so far ramiro@colmedlp.zzn.com Dr. Raúl 591-2-2734880 Villanueva Professor of Infectology, University Mayor Assessment studies; NI villeteraul@gmail.com manual Dr. Oscar Lanza Director, AIS 591-2-2222987 Support of activities using SAIDI resources through PAHO aisbol@entelnet.bo Director, CONCAMYT Dra. Cecilia 591-2-2226670 Garnica Coordinator of laboratory equipment and training courses in this area garnicalopez@yahoo.es EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 98 INTERNATIONAL PARTNERS Name Organization William Mac Kenzie CDC Stephen Benoit Anibal Soza APUA Beth Yaegger MSH RPM Plus Maria Miralles Edgard Barillas Marisabel Sanchez Links Media Gabriel Schmunis PAHO Maria Paz Ade Adrian Barojas USP DQI Victor Pribluda USAID MISSIONS Name USAID Mission Jaime Chang Peru Stanley Blanco Bolivia Graciela Avila Paraguay EXTERNAL EVALUATION OF THE AMAZON MALARIA INITIATIVE AND THE SOUTH AMERICA INFECTIOUS DISEASE INITIATIVE 99