August 2014 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Irene Agyepong, Lynne Franco, Judith Fullerton, Gray Handley, Denis Prager (Team Leader), Ashley Strahley, and Mary Taylor. Health Research Program (HaRP) Evaluation Final Report HEALTH RESEARCH PROGRAM (HARP) EVALUATION FINAL REPORT AID-OAA-C-13-00113 August 2014 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. (Cover photo credit: USAID/Tash McCarroll) CONTENTS Acronyms ....................................................................................................................................................... i Executive Summary .................................................................................................................................. ii 1. Introduction ........................................................................................................................................ 1 1.1. Purpose of the evaluation ......................................................................................................... 1 1.2. Background: HaRP Mission, strategy, and activities ................................................................ 1 2. Evaluation Design ............................................................................................................................. 5 2.1. Evaluation Questions ................................................................................................................ 5 2.2. Evaluation Approach ................................................................................................................. 5 2.3. Sources of Data......................................................................................................................... 6 2.4. Data Analysis ............................................................................................................................ 6 2.5. Ethics ......................................................................................................................................... 7 2.6. limitations .................................................................................................................................. 7 3. Evaluation Findings ........................................................................................................................... 8 3.1. HaRP achievements ................................................................................................................. 8 3.2. HaRP Conceptual Framework ................................................................................................12 3.3. HaRP Partnerships..................................................................................................................14 3.4. HaRP Process: Assessment/Priority Setting ..........................................................................16 3.5. HaRP Process: Development and introduction .......................................................................17 3.6. HaRP process: Field implementation and scale-up................................................................20 3.7. HaRP mechanisms and structures ..........................................................................................22 3.8. Implementation Research/delivery science ............................................................................24 4. Evaluation conclusions and recommendations ...............................................................................25 4.1. HaRP’s Approach ....................................................................................................................25 4.2. HaRP’s research-to-use strategy ............................................................................................26 4.3. HaRP’s priority-setting process ...............................................................................................26 4.4. HaRP’s development and introduction processes ..................................................................27 4.5. HaRP’s field implementation and scale-up processes ............................................................27 4.6. How HaRP works with partners ..............................................................................................28 4.7. HaRP structures and mechanisms ..........................................................................................28 4.8. Implementation research/delivery science ..............................................................................29 Annexes ......................................................................................................................................................30 Annex 1. Evaluation Statement of Work .............................................................................................30 Annex 2. Case Study- Chlorhexidine for Newborn Cord Care ...........................................................38 Annex 3. Case Study- Respectful Maternity Care ..............................................................................47 Annex 4. Table of interviews conducted .............................................................................................53 Annex 5. Interview Guide-Case Studies .............................................................................................54 Annex 6. Interview Guide-Partners ....................................................................................................57 Annex 7. Biographical Sketches of EvaluationTeam Members .........................................................61 Annex 8. Project Budgets by USAID source since 2009 ....................................................................65 Annex 9. Disclosure of Conflict of Interest .........................................................................................66 i ACRONYMS AAD Activity Approval Document CHX Chlorhexidine COI Consultant Conflict of Interest CRA The Country Research Activity GRA Global Research Activity HaRP Health Research Program HIDN Office of Health, Infectious Diseases, and Nutrition HRCI The Health Research Challenge for Impact MCHIP Maternal and Child Health Integrated Program MCA Maternal, Newborn, Child and Adolescent Health MNCH Maternal, Newborn, and Child Health ORT Oral Rehydration Therapy PATH Program in Appropriate Technology for Health PMTCT Prevention of Mother-to-Child Transmission PQM Promoting the Quality of Medicines RMC Respectful Maternity Care SNL Saving Newborn Lives SOW Scope of Work TRAction The Translating Research into Action Project UNCOLSC UN Commission on Life-Saving Commodities URC-CHS University Research Co.-Center for Health Services USAID U.S. Agency for International Development WHO World Health Organization WRA White Ribbon Alliance ii EXECUTIVE SUMMARY Background: The U.S. Agency for International Development’s Health Research Program (HaRP) commissioned an external evaluation to: (i) assess the effectiveness and impact of its managed research￾to-use strategy; and (ii) identify issues for consideration in designing future efforts focused on accelerated research, research utilization, and scale-up. To address these questions, the Evaluation Team: (i) conducted case studies on the development and introduction of chlorhexidine (CHX) for the prevention of neonatal infection and mortality, and on the promotion of respectful maternity care in addressing disrespect and abuse in childbirth; (ii) obtained a range of perspectives on the degree to which the HaRP strategy and process contribute to the development of new interventions/products and to accelerating the translation of those interventions into use and scale-up, and (iii) reviewed documents relative to HaRP. Structured interviews were conducted in January-February 2014 with 65 individuals: 37 representing U.S.-based organizations, 11 representing global organizations, and 17 representing country level organizations. Data from the interviews were coded and analyzed using a cross-platform application for qualitative data analysis that searched for and grouped interview responses by key words and phrases. The Evaluation Team conducted thematic and content analyses, then, as a group, developed a set of findings, draft conclusions and recommendations. What is HaRP? The Health Research Program and strategy were initiated in FY 2003 by the Office of Health, Infectious Disease, and Nutrition, and are designed to accelerate the development and translation of research products into the effective implementation of USAID and partner country health programs. The HaRP program identifies, develops, tests, and facilitates the introduction of new or refined tools, technologies, approaches, policies, and interventions intended to improve the health status of infants, children, mothers, and families in developing countries. Activities are guided by a pathway of intervention development from priority setting to introduction to use at scale. The HaRP strategy guides collaborative efforts of USAID staff, grantees, and other partners. Presently, five major activities are directly managed by core USAID research staff. These include The Health Research Challenge for Impact (HRCI), a cooperative agreement with Johns Hopkins University; The Translating Research into Action Project (TRAction), a cooperative agreement with University Research Co. LLC; support to the World Health Organization (WHO)/Maternal, Newborn, Child and Adolescent Health (MCA) department; HealthTech V, a cooperative agreement with Path; and Accelovate, a cooperative agreement with Jhpiego. HaRP Achievements: The research conducted under the HaRP strategy has delivered important contributions in interventions, approaches, and programs that improve newborn survival, case management of child illness, maternal health, and the systems that support them. HaRP funded and facilitated the transfer of knowledge for essential newborn care, prevention and treatment of newborn infections/sepsis, tools for management of birth asphyxia, zinc/oral rehydration therapy (ORT) treatment of child diarrhea, and CHW treatment of pneumonia. Many of these programs have begun to go to scale in focus countries. Building on what they learned from the introduction of Zinc/ORT, HaRP helped accelerate the introduction and adoption of CHX and continues to facilitate support for the 15 countries that have committed to national programs. Where HaRP has been most successful was in their engagement of researchers, technologies, developers, and implementers. HaRP has effectively used researcher groups and meetings to help harmonize the demonstration of simplified antibiotic treatment iii for newborn sepsis and to support WHO in its rapid move to guidelines and country support. Their work also helped develop or adapt tools such as uterotonics to prevent post-partum hemorrhage and ensure supply of magnesium sulfate for pre-eclampsia and eclampsia. As more products and interventions proved successful and challenges to their widespread use became pressing, HaRP focused its support to system topics such as Respectful Maternity Care (RMC) and the integration of Prevention of Mother to Child Transmission (PMTCT) - Maternal, Newborn and Child Health (MNCH). The development of new mechanisms, such as the TRAction project that seeks to reduce the ‘know-do’ gap more quickly, have added new angles to HaRP’s activities. Throughout its technical work, HaRP has made effective use of partnerships -- internally and externally -- with global MNCH stakeholders. Respondents noted the utility of HaRP-sponsored meetings and technical advisory groups, and were appreciative of the technical expertise and facilitative style of the HaRP team. Key Conclusions and Recommendations: The following represent the highest priority conclusions and recommendations. A full set of conclusions and recommendations can be found in the body of the report. PRIORITY #1 – Increasing local engagement throughout the research-to-use process Conclusion: To be effective, HaRP’s support to development and introduction of new interventions must be accompanied by input from local partners who will ultimately be responsible for implementing those interventions in the field. Recommendation: Early in the research-to-use process, HaRP should focus more on ensuring that its partners are creating or enabling others to create local environments conducive to acceptance, adoption, and use of new interventions. This may require new approaches or more staff resources or both in order to close the global-local divide. Recommendation: HaRP should provide incentives for its implementing partners to work with country-level stakeholders as equal partners in the research-to-use process, and prioritize, conduct, and share research in ways that fully engage those who will ultimately be responsible for implementing and scaling up these new interventions and approaches. PRIORITY #2 – Examining ways to better leverage USAID structures Conclusion: HaRP’s efforts to create an effective, rapid and synergistically supported research-to-use process are challenged by a number of USAID-related structures and dynamics, including the relationships between USAID/Washington and Missions, and among USAID programs, as well by as USAID funding structures and cycles. Recommendation: HaRP should increase its efforts to engage USAID Missions in the research-to-use process early and consistently, anticipating their significant value in facilitating the introduction, field implementation, and scale-up of new interventions. PRIORITY #3 – Placing a greater focus on implementation research Conclusion: With other donors focusing on discovery science and development, and the pipeline filled with new interventions and products, there is a need for increased attention to more rapidly moving interventions/products through the research-to-use process to introduction, use, and scale-up. Recommendation: HaRP should place greater emphasis on implementation research and contribute to thinking, from day one, about what will be needed for effective field implementation and scale-up. iv This focus should not be to the exclusion of funding effectiveness research in situations where HaRP’s investment will add value and fill key gaps. PRIORITY #4 – Facilitating processes and capacities for more effective interaction between implementers and researchers Conclusion: Implementation research/delivery science will be more effective if it is a two-way process with strong local input. This will require development of stronger in-country research capacities. Recommendation: HaRP should lead, support, and leverage the development of a new paradigm and mind-set for a two-way and bottom-up implementation research/delivery science, valuing the local perspectives and insights critical to understanding the environment for introduction and scale-up of new interventions, and the usefulness of implementation research findings. Local capacity for implementation of research/delivery science is critical and HaRP should promote and facilitate its rapid development. PRIORITY #5 – Strengthening the role of implementation research/delivery science in the research-to-use process Conclusion: Implementation research/delivery science is designed to understand what, why, and how interventions work in “real world” settings and to test approaches for their improvement. HaRP is recognized for leadership and progress in this space (TRAction, meetings), but more needs to be done. Nearly all stakeholders called for a priority focus on implementation research moving forward (although not necessarily an exclusive focus). Recommendation: HaRP should continue to expand its role as the leader in directly supporting, and influencing others to support, implementation research and delivery science. Recommendation: HaRP should advocate and leverage partners/resources to document lessons learned from implementation and ultimate results: (i) surveillance systems that monitor the use of new interventions and their impact on health; and (ii) knowledge management systems to capture key elements of successful and unsuccessful implementation and scale-up. PRIORITY #6 – Strengthening understanding of changing contexts to ensure relevant research results Conclusion: HaRP’s priorities do not always appear to be adequately revised and aligned with changes in context or associated learning. Recommendation: HaRP should formalize a scanning function designed to identify and document changes in global and local context for HaRP funding of research, and refine priorities as needed. For example, with CHX, it would have been important to recognize the increasing rate of facility births and ensure that research is conducted in those settings, too. PRIORITY #7 – Addressing realities (and perceptions) of HaRP research priority changes Conclusion: Partners perceive that HaRP changes its priorities, thus losing programmatic momentum, confusing partners, and jeopardizing success. Recommendation: HaRP should stick with priorities long enough to see them through to an appropriate completion of the research-to-use process, building on positive experiences such as with CHX or TRAction. Concurrently, to prevent misperceptions, they should strengthen communication with partners on follow-through on these priorities over time. 1 1. INTRODUCTION 1.1.PURPOSE OF THE EVALUATION The U.S. Agency for International Development’s Health Research Program (HaRP) commissioned an external evaluation to: (i) assess the effectiveness and impact of HaRP’s managed research-to-use strategy; (ii) identify issues for consideration in designing future efforts focused on accelerated research, research utilization, and scale-up; and (iii) identify issues and questions for further investigation. The results of this evaluation of HaRP strategy and processes are intended to inform future investments and programming by USAID staff and Missions employing research-to-use strategies to advance their global health activities, with special reference to maternal and child health. This evaluation is also intended to guide future directions for a broader international public health community interested in accelerated research-to-use strategies and in advancing the growing field of implementation research. 1.2.BACKGROUND: HaRP MISSION, STRATEGY, AND ACTIVITIES 1.2.1. HaRP MISSION The mission of the Health Research Program (HaRP) is to help USAID identify, develop, and test new and refined tools, technologies, approaches, policies, and/or interventions to improve the health status of infants, children, mothers, and families in developing countries. In pursuing this mission, HaRP conducts strategic planning, problem identification and priority setting, and monitoring of investments in research and translation of that research into use. 1.2.2. HaRP STRATEGY The HaRP conceptual framework or pathway of intervention development from priority setting to introduction to use at scale (Exhibit 1) was designed in 2003 to guide research and research introduction activities financed by the Office of Health, Infectious Diseases, and Nutrition (HIDN) with the goal of accelerating the use of research results and introduction efforts. 1.2.3. EVOLUTION OF HARP Since its inception in 2003, HaRP, with its collaborating partners, has worked to identify and overcome challenges to maternal, newborn, and child health (MNCH) in developing countries. These challenges include infectious diseases, malnutrition, and insufficient or ineffective disease management by health workers. HaRP has supported the conduct of applied research in eight targeted areas: Maternal Health; Infant and Newborn Health; Child Health; Infectious Diseases; Nutrition; and others. During one decade of support, HaRP has funded several streams of research activity and funding, as illustrated in Error! Reference source not found. and presented in more detail in Annex -1857289366. 2 Exhibit 1: Health Research Program’s Conceptual Framework Exhibit 2: HaRP Funding Streams – 2003 to the present 3 In HaRP’s first phase (2003-2009), the Global Research Activity (GRA) focused on establishing an evidence-base for global health programs by conducting multi-country studies and evaluation, while strengthening research capacity in countries through its work with developing country institutions, and mentoring relationships. The Country Research Activity (CRA) conducted country-specific research to address local health priorities and was directly responsible for strengthening national research capacity and engaging new partners in country. HealthTech IV (and later V) identified and advanced new health technologies for application and scale-up. In its second phase (2009-2016), HaRP supported and continues to refine activities to reduce maternal mortality and to advance newborn survival. Priorities for maternal health include ensuring respectful, quality care during pregnancy and birth, early prevention of maternal complications, and access to high￾quality drugs and services for pre-eclampsia, eclampsia and prevention of post-partum hemorrhage. HaRP is also supporting efforts to advance the quality of management of neonatal infections and newborn asphyxia, and to increase access to care. Current activities and mechanisms for HaRP include: The Health Research Challenge for Impact (HRCI) – a cooperative agreement with Johns Hopkins University to support coordinated research studies to accelerate the process of conducting and translating new research into use in field programs. The Translating Research into Action Project (TRAction) – a cooperative agreement with University Research Corporation to address the “know-do” gap: i.e., the translation of research into use so that research findings affect improved public health. This cooperative agreement includes a specific provision for significant USAID involvement. World Health Organization – HaRP supports the WHO maternal, child, and adolescent health and development department to identify, sustain, and increase the effectiveness of strategies/technologies that advance child survival in developing countries.1 HealthTech V – a cooperative agreement with the Program in Appropriate Technology for Health (PATH) that develops, adapts, evaluates and/or facilitates the introduction of affordable and appropriate technology solutions for safe, effective, and more equitable distribution of health care services in low￾resource countries. Accelovate – a cooperative agreement with Jhpiego, is developing, introducing, and supporting the scale-up of new health tools and technologies that are appropriate, affordable and acceptable for distribution and use in low-resource settings with the aim to accelerate reductions in mortality and morbidity in low-resource settings. The execution of the HaRP strategy also involves direct collaborations with USAID- and partner-funded activities such as: the Maternal and Child Health Integrated Program (MCHIP), a cooperative agreement led by Jhpiego; Promoting the Quality of Medicines (PQM), a cooperative agreement with US 1 HaRP works closely with the WHO/Reproductive Health and Research (RHR) department, as a driver of strategic vision and collaboration, but the official USAID contact for this relationship sits outside of HaRP. 4 Pharmacopeia; and the Save the Children Saving Newborn Lives (SNL) program funded by the Bill & Melinda Gates Foundation. Exhibit 3 presents an overview of the specific research topics, timelines and contractual mechanisms HaRP has invested in since 2003. Exhibit 3 highlights a shift in focus over time, as well as HaRP’s frequent use of multiple contractual mechanisms to address a similar topic, and push it along the research-to-use continuum. Exhibit 3: Overview of HaRP’s Specific Research Topics, Timelines and Mechanisms Used 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 GRA CRA HealthTech IV WHO HRCI Accelovate HealthTech V TRAction MNCH Neonatal Sepsis Prevention (CHX) X X X X Community Management of Newborn Sepsis X X X X Essential newborn care X X X X Treatment of newborn sepsis X Outpatient and CHW management of severe and non-severe pneumonia X X CCM for Malaria X Birth Asphyxia (technologies) X Post-partum hemorrhage (tech) X Care at Birth (RMC) X X Injectible Antibiotics for newborn sepsis X Zinc/ORT for Diarrhea X Intergrated community case management X Birth Asphyxia (technologies) X Illness Recognition and Referral (CCM) X X Maternal and Newborn Care (PMTCT-MNCH) X PreEclampsia / Eclampsia management and treatment (MgSO4) X PreEclampsia / Eclampsia (testing of proteinuria X Post-partum hemorrhage (misoprostol and QI for OT) X Barriers to Healthcare access (equity targeting) X Health service delivery-task shifting X Incentives for quality (Performance based incentives and PBF) X Neonatal Sepsis Treatment (gentamicin--Uniject) X Newborn Health (clean delivery kits) X Vaccine, Waste Management (disponsible syringe jet injectors) X Waste Management (safe injection) X Birth Asphyxia (resusitators) X Nutrition (screening for Vitamin A deficiency) X Immunization (reduce misuse of unsterile syringes) X Immunization (reduce need for cold chain) X Immunization (reduce vaccine wastage) X Multiplle (rapid diagnostic test website) X STI Diagnostic (HSV-2) X STI Diagnostic (HPV) X Water Safety (contaimination tests) X Nutrition (screening for micronutrient ) X Anemia (point of collection screening) X Nutrition (anemia screening) X Environmental Health and Safety (cookstsoves) X Environmental Health and Safety (sanitation) X Disease Control (LLITN) X Disease Control (IRS) X Program, Policy and Advocacy specific to MNCH, nutrition, population, family planning, reproductive health, and tuberculosis X Anemia (social marketing) X HIV Prevention (male circumcision uptake) X Health Systems (eHealth) X X OTHER 5 2. EVALUATION DESIGN 2.1. EVALUATION QUESTIONS The Statement of Work for the HaRP Evaluation (Evaluation Statement of Work) lists the following overarching questions to help guide the evaluation process: “What was the fit for purpose of the HaRP-managed accelerated research-to-use strategy for both (i) design and (ii) execution of: a) Intervention research and research introduction/utilization and planning for scale-up of new/refined interventions, focusing on one or more of the following: zinc/reduced osmolatity ORS, management of severe pneumonia at community level, chlorhexidine newborn cord care, and simplified switch therapy for presumptive newborn sepsis. b) Implementation research/health services research focusing on planning for scale or addressing systemic health systems challenges that impede health system functionality and effective implementation of evidence-based approaches.” It was understood that these questions would be modified by the Evaluation Team as the process unfolded. Indeed, as a result of a few evaluation design interviews and other evaluation processes, the Evaluation Team developed the following set of questions to guide the analysis, interpretation, and reporting of the data collected: 1. What has worked well or needs improvement in HaRP’s managed process, funding mechanisms, activities, and partnerships? 2. In what ways has HaRP contributed to: (i) developing important interventions/products; (ii) influencing policy; (iii) getting interventions/products/approaches and policies introduced; and (iv) facilitating scale-up? 3. In order to amplify and accelerate the impact of its future programs, how can HaRP revise its research-to-use strategy/model and strengthen the mechanisms it uses and roles it plays in implementing that strategy? 2.2. EVALUATION APPROACH To address these questions, the Evaluation Team developed a two-pronged approach: 1. Case Studies – Studies of the implementation of the research-to-use process as it applies to: (i) the development and introduction of CHX for prevention of neonatal infection and mortality; and (ii) the promotion of respectful maternity care in addressing disrespect and abuse in childbirth. These case studies offer the opportunity to more fully explore how HaRP contributed to the research-to-use process. Fuller documentation of these two case studies is provided in Annex 2 and Annex 3, respectfully. 2. Assessment of the HaRP Strategy and Process – Collection of information intended to provide perspectives on the degree to which the HaRP strategy and process contribute to the 6 development of new interventions/products and accelerating translation of those interventions into use and scale-up. 2.3. SOURCES OF DATA Data for both of these prongs of the HaRP evaluation were collected through key informant interviews and review of relevant documents, during January-February 2014. Interviews. Structured interviews were conducted with 65 individuals: 37 representing U.S.-based organizations, 11 representing global organizations, and 17 representing country level organizations (See Annex 4). Interviewees generally fell within two categories: 1. HaRP Implementing Partners – organizations funded by HaRP to help carry out its research and research-to-use projects. 2. Other Stakeholders – individuals from U.S., foreign, and international organizations thought to be knowledgeable about the HaRP strategy and process, and capable of commenting on their effectiveness and impacts. Special effort was made to identify and interview individuals representing global, regional, country- and community-level perspectives on HaRP and its programs. Attention was paid to identifying informants representing a variety of perspectives on HaRP and other programs pursuing research-to-use processes. Interviews focused on effectiveness of the HaRP strategy, activities, funding mechanisms, and partnerships, and on the future role of HaRP and USAID. Interviews were conducted using two interview guides – one for HaRP partners, and one for case studies of chlorhexidine and respectful maternity care (See Annex 5 and Annex 6). Document Reviews. Reviews were conducted on documents: (i) related directly to the HaRP strategy, process, and programs with emphasis on chlorhexidine for cord care and Respectful Maternity Care; (ii) describing evaluations of various HaRP activities and programs conducted or funded by other organizations; and (iii) describing models used by other organizations in organizing research programs. Given the time limitations for the evaluation, only the most pertinent subset of documents was reviewed. Deeper document review may have added nuance to the complex nature of HaRP’s work. However, the results of these reviews served as background for the evaluation. 2.4. DATA ANALYSIS Qualitative data from the interviews were coded and analyzed in Dedoose, a cross-platform application for qualitative and quantitative analysis. This platform searched for and grouped interview responses by key words and phrases. The Evaluation Team divided the data and conducted thematic and content analyses. At a three-day team meeting, individual team members presented the findings for discussion and triangulation with other findings (including the case studies). The team developed a coherent set of findings, draft conclusions and recommendations that were presented to and discussed with the HaRP team. Members of the Evaluation Team then wrote various sections of the report that follows. The 7 Team Leader then compiled these sections into what is intended to be a coherent reflection of what was discovered through the evaluation.2 2.5. ETHICS The evaluation process incorporated steps to avoid the actuality or appearance of conflicts of interest on the part of the Evaluation Team and to ensure the anonymity of respondents. Conflicts of Interest. Members of the Evaluation Team recused themselves from interviews with individuals representing organizations with which they had a current relationship as employee, advisor, or consultant. Anonymity of Respondents. The anonymity of respondents was ensured through the following steps: 1. Assurance – each respondent was informed that “information or examples we discuss during this interview will not be attributed to any specific person or institution. Quotes used in the report will be attributed to a general stakeholder group (e.g., research partner, in-country stakeholder, etc.), and all identifying information will be removed.” 2. Removal of Identifying Information – each interview transcript was cleaned of any identifying information and assigned an interview code for the purpose of analysis. Original transcripts with identifying information were destroyed. 2.6. LIMITATIONS The thoroughness and depth of the evaluation summarized in this report were constrained by the following limitations:  This evaluation was conducted under an extremely short timeframe, which placed severe constrictions on time for design, data collection and analysis, and allowed for only two case studies. As a result, it is possible that situations for CHX and RMC are over-represented in the findings.  There was little or no information available for use in assessing the overall effectiveness of the HaRP strategy. This was in part a reflection of HaRP’s mandate to focus on reporting of progress and results, rather than impact.  HaRP is a diverse and broad-ranging program that works fairly organically, making it hard for respondents and evaluators to trace all the places where it is present.  Many respondents in this evaluation were not familiar with HaRP per se and were not able to distinguish it from other USAID programs. They may have been familiar with a particular funding mechanism (e.g., TRAction), but did not link it to HaRP’s overarching strategy. 2 Due to issues with the evaluation’s contracting vehicle availability, the report remained in draft form from March￾May 2014, and was revised June-July and finalized in August 2014. No new data was collected or analyzed after February 2014. 8  The number of contracting mechanisms used (and modified) over time made it difficult to sort out which HaRP activity was contributing to which programmatic priority.  Contacts with individuals working at field level were complicated by the fact that many did not have regular Internet access which created delays in the notification and arrangement of interview dates and times. Some, while interested in being interviewed, were unable to be accommodated in the limited timeframe. 3. EVALUATION FINDINGS The following sections present the findings of the evaluation process. After summarizing key achievements over the life of HaRP and particularly in the last five years, this section will discuss the appropriateness of the overall HaRP conceptual framework, as shown in Exhibit 1, and the way HaRP works with its partners. It will then present findings organized according to the elements of the strategy, from priority-setting to field implementation. The final sections deal with HaRP mechanisms and structures, and with Implementation Research/Delivery Science. Representative quotes are included to capture themes heard in the interviews. 3.1. HaRP ACHIEVEMENTS From 2003 to the present, HaRP and its internal and external partners have made important contributions to the development or adaptation of technologies and interventions for mothers and children, as well as building the knowledge base for measurement, service delivery, and scale-up. HaRP was particularly active in:  Newborn survival: demonstrating the effectiveness of essential newborn care (ENC); testing low cost resuscitation approaches for communities; accelerating the testing, introduction, and adoption of CHX for cord care and short course switch therapy for presumed newborn sepsis at scale, and for supporting studies that demonstrated the effectiveness of community based, simplified treatment for severe pneumonia  Community case management of child illness: demonstrating the effectiveness of outpatient and community health worker (CHW) management of pneumonia; developing, introducing and scaling Zinc/ORT for diarrhea; testing the use of rapid diagnostic tests by CHWs for malaria management; and more recently, integrating the spectrum of case management services from policy, financing and evaluation perspectives  Targeted maternal health tools and approaches: improving local management of pre￾eclampsia/eclampsia with magnesium sulfate; and evaluating the integration of prevention of mother to child transmission of HIV with maternal, newborn, and child health services  Developing solutions such as equity targeting; task shifting; and performance-based incentives to address service delivery challenges for MNCH. HaRP is also recognized for commissioning research into more sensitive new topics such as Respectful Maternity Care (RMC). While this work is at an early stage in the research-to-use transition, it is viewed as a timely example of how to strengthen demand for and use of services. 9 HaRP’s overarching strategy and mechanisms have included MNCH-related funding and/or management of activities by other USAID teams. These funds have supported research into reducing indoor air pollution from cook stoves; health effects of sanitation interventions; comparison of approaches to indoor residual spraying and long lasting insecticide treated nets; vaccine delivery related technologies; and anemia tests. As HaRP has increased the scope of its focus from developing interventions to field implementation and scale-up, it has created new types of mechanisms to better address the ‘know-do’ gap. The TRAction project addresses barriers to the reach and high quality use of life saving interventions. The evaluation included two case studies to elucidate HaRP’s contributions to rapid development and scale-up of interventions/approaches. Exhibit 4 and Exhibit 5 summarize case study findings, and full details are available in Annex 2 and Annex 3. Throughout the interviews conducted for this evaluation, HaRP was recognized for effective partnership and coordination. HaRP has contributed to acceleration of the development and introduction phases for some interventions by strategically using funds to convene and advocate among MNC experts, and by taking a facilitative rather than directive approach. Exhibit 4: Summary of Chlorhexidine for Newborn Cord Care Case Study The 10 years of HaRP history in supporting CHX for newborn cord care is summarized below, illustrating the research, partners, meetings, and WHO authorization. Along with others, HaRP invested in and was credited with leading an accelerated path from research-to-use. This has resulted in the initiation of CHX programming in 16 countries, including Nepal where it is available to two-thirds of the population, a third of whom are supported under a USAID Grand Challenge award. Key Findings from the CHX Case Study: Research Priorities  From a newborn health perspective, HaRP’s choice of research priorities was on target for MNCH impact, however more diversity and greater field input is now needed. 10  More rapid adoption and scale-up of CHX might have been achieved if addressing intervention characteristics, fit with delivery platforms, and policy requirements was done at the priority setting stage.  Nearly all stakeholders recommended that USAID funds more implementation research for MNCH interventions, especially in introduction and scale-up phases. Development & Introduction  Research-to-use of CHX was accelerated by supporting product development and operational studies in parallel with effectiveness studies.  Engagement of the commercial sector was critical to country adoption, but approaches need to better address value for companies, potential markets, and the trust needed between public and private sectors.  HaRP’s leadership in convening and funding support for activities that harmonized research and spread learning, led to faster country uptake of CHX and simplified antibiotic regimens for treating newborn sepsis.  USAID’s changing priorities, internal structure and working culture sometimes complicated HaRP’s progress, most often when country missions and headquarters differed on priorities. Field Implementation & Scale-up  As the 16 target countries move to adopt and scale up CHX, gaps remain in planning and support for sustained delivery at national scale. Lack of useful knowledge management systems result in loss of learning on scale-up of CHX, which is likely to delay successful use by populations in need.  The population impact of CHX for cord care is not and may not be known in the future because robust systems to measure, track and report scale-up and sustained outcomes are not in place. Research-to-Use: What is HaRP’s role? The development of CHX for cord care highlights HaRP’s role as a technical leader, convener, coordinator, funder and advocate in the research-to-use process. Most of HaRP’s effort has been focused on the priority setting, development and introduction phases both globally and in the ‘early adopter’ countries. As national scale-up becomes the target, country activities will predominate and HaRP’s direct role will diminish with respect to other stakeholders, including governments. HaRP might consider more active leveraging of external and internal partners and more explicit use of USAID mechanisms to strengthen feed-forward and feedback loops to more tightly couple field implementation and results within the research-to-use strategy. Exhibit 5: Summary of Respectful Maternity Care Case Study Respectful maternity care is the construct of provider/client (patient) interpersonal relationships that occur across the continuum of the perinatal care timeframe. The topic was delegated to the TRAction project at the Center for Health Services (URC-CHS) that used a small grants mechanism to advance work on the topic. These ranged from consultative meetings and a landscape analysis to pilot learning activities conducted in Kenya and Tanzania. 11 Key Findings from the RMC Case Study: Research Priorities  With the award of HRCI, TRAction, and the new Technologies for Health, HaRP increased its focus on health system challenges. HaRP assessed the priority and comparative advantage for work in RMC and is applauded for the important role they played in bringing attention to the topic.  Extensive consultative meetings and an important landscape analysis yielded a framework that defines disrespect and abuse in childbirth.  Two pilot projects tested promising intervention approaches, and highlighted remaining gaps in knowledge about best intervention practices.  It was acknowledged, in terms of RMC, that everyone is aware that the problem exists, but no one wants to address it. HaRP underestimated the impact that this constraint would have on the pace of progress for the funded projects. Product/Strategy Development  The fact that field research had joint objectives, core indicators, and complementary research and measurement methodologies was recognized as a strength drawing on relative skills of implementing partners.  One of the two field projects was stalled in its early phases because of perceptions that the research might uncover or document personal behaviors that could lead to professional retribution. This was successfully addressed by open engagement of health system providers, but took time, and it reflects the sensitivity of the topic. Introduction  Activities included fora for change, information, standards, local activities, and promotion of champions. Evaluation findings will soon become available.  Very valuable definitions, tools and measurement indicators for RMC have been developed and have a broad utility. They have been widely disseminated in diverse local, regional and global fora.  Formal advocacy activities that achieved wide reach at the global level were undertaken by the White Ribbon Alliance in parallel with the field pilots. Leaders of the projects intend to align the implementation findings with future targeted advocacy initiatives that build on their lessons learned. HaRP’s support of RMC is in the early stages of the research-to-use process. Next steps will follow on reporting of implementation findings. 12 3.2. HaRP CONCEPTUAL FRAMEWORK This section focuses on the HaRP conceptual framework for accelerated research-to-use process, as its “theory of action” and as articulated by the four cylinders in its framework diagram ( Exhibit 1) comprising: (i) assessment – identification of research areas and priorities; (ii) development – applied research, development, and testing of new interventions; (iii) introduction – catalytic activity to accelerate translation of key research findings and their incorporation into regular use in worldwide field programs; and (iv) implementation – use of research results and products in countrywide and global health programs/policies. Through its interviews, the Evaluation Team heard a number of views concerning the appropriateness, relevance, and effectiveness of the HaRP conceptual framework. It should be clearly noted that the actual strategy, as implemented by HaRP, was increasingly more flexible and iterative over time than reflected in this framework. Finding: The HaRP Conceptual Framework, as depicted in Exhibit 1 includes key components of a research-to-use process, but does not match the more non-linear process in use, and does not recognize the various actors that are, or need to be, involved. None of the respondents who knew of the HaRP conceptual framework questioned the appropriateness of the four cylinders (priority setting, development, introduction, field implementation), believing that they encompass the principal elements of the research-to-use process that drives HaRP’s programmatic activities. However, the framework implies a neat, linear progression of sequential steps from discovery to introduction to scale-up, and does not adequately represent either the messiness of the process in the real world or the need for downstream results to influence upstream strategies and tactics. Even the HaRP team’s approaches over the past decade reflect a shift into anticipatory design, feedback loops, and parallel activities. This is seen in their work in CHX and treatment for neonatal sepsis, as well as in design of the TRAction project. However, the conceptual framework has not yet been realigned. “A limitation of the HaRP …. model is that it implies that each step proceeds sequentially from discovery to scale-up, when we are learning that the process is more effective when many of the steps are done in parallel.” (Implementing Partner) “The HaRP [conceptual framework] is missing the feedback loops which we know exist in the real world. Without them, how does what happens with field implementation and scale-up feedback affect priorities and influence development and implementation strategies?” (Implementing Partner) The linearity in the conceptual framework and the depiction of components as cylinders reinforced for some respondents their impression of compartmentalization of the research process reflecting the more compartmentalized nature of USAID as an agency, which inhibits acceleration and effectiveness of the research-to-use process. “One thing that jumps out at me, is that USAID [as an agency] continues to work in a siloed fashion, by technical area. This is reflected in the HaRP … model. As a result, for example, intervention research often does not overlap with health systems research. Yet, we know that you can’t achieve implementation of new interventions without understanding how the health system works in a country.” (International Organization) 13 Finding: One limitation of the HaRP conceptual framework is that it does not indicate who is responsible for carrying out various elements of the strategy across the research-to-use spectrum. Acceleration of the research-to-use process requires effective engagement and hand offs. The HaRP conceptual framework is a comprehensive picture of many of the steps involved in the research-to-use process. Yet, for budget and capacity reasons, HaRP cannot and probably should not undertake all of the steps itself. For example, it doesn’t identify national scale-up as a part of its programmatic purview. Respondents felt that the framework could be more effectively applied if it did, and there was a consensus around those parts that HaRP carries out itself and those that are carried out by others. In some cases, it is organizations directly supported by HaRP; in others, HaRP leverages or facilitates the actions of other players. This facilitation role is increasingly dominant at the field implementation end of the spectrum where progress depends on changes in local policies, practices, and actions. “HaRP’s vision is ultimate scale-up, but it is not clear what role it does or can play in driving interventions toward use and scale-up on the ground in target countries. When action is beyond HaRP’s purview, how effective is the hand-off to those who are positioned to accomplish implementation and scale-up? ” (International Organization) More recently, there has been a greater clarity of roles for HaRP and others in specific interventions such as testing and introduction of simplified antibiotic treatment of neonatal sepsis. There have also been efforts within USAID to define roles and therefore handoffs at the front end (in RFAs) and in work plans such as between HaRP and MCHIP. However, from a strategic point of view, these efforts have been piecemeal and not always well operationalized. The biggest gap in identifying and negotiating roles remains between HaRP at the global level, and country- or field-based actors. One of the challenges facing HaRP and others striving to move interventions along the continuum from discovery to field implementation is turf, i.e., ownership of various elements of the process that may inhibit the two-way communication and collaboration required in order for the process to be successful. “As a new product/intervention moves from development to introduction and field implementation, the ownership of the research-to-use process moves from global/central to local/country levels. That shift in ownership, which should be understood by all involved from the beginning, is neither clear nor explicit in the current HaRP model.” (Implementing Partner) Finding: With other donors focusing on discovery science and development, and the pipeline filled with new interventions and products, there is a need for increased attention to introduction, use, and scale-up. At the same time, implementation perspective must be more deeply incorporated into discovery and development. Achieving a balance among various elements of the research-to-use process is a continuing challenge for HaRP. There is, on one hand, the need to be sure that the pipeline of promising new interventions is sufficient to meet the needs of child and maternal health programs on the ground in target countries. On the other hand, continuing to fill that pipeline with new interventions is futile if the strategies and mechanisms for introduction and scale-up are inadequate or ineffective. Given the growing role of the Bill & Melinda Gates Foundation and others in the discovery and development of new interventions, respondents expressed the view that HaRP can play a more active role in pursuing the elements of the research-to-use process involved in field implementation, use, and scale-up. 14 “I’m not aware of HaRP’s overall strategy beyond its support of specific studies and programs. However, I am aware of TRAction and believe that HaRP should be doing more of this kind of introduction and scale-up. In my view, this is where HaRP can have the greatest impact.” (Implementing Partner) “There are many interventions out there that need to be put to use, but there is no one now willing and capable of bringing together all of the pieces that need to be aligned if new interventions are going to make the transition from concept to product, uptake, and scale- up. Many of the organizations involved are loath to take on the messy business of implementation; too complex, too many moving parts. HaRP has the opportunity to play a critical role in facilitating cooperative efforts and overcoming built-in turf barriers.” (Implementing Partner) While there is an urgent need to apply research to implementation, it is equally important to bring implementation perspective early into discovery science and development. When basic scientists and efficacy researchers are communicating better with users and are informed by user perspectives, delivery system realities, and contextual requirements, interventions and products are more likely to be adopted and to change behavior. Therefore, even with a greater focus on implementation research, it will be important for HaRP to further develop effective mechanisms to ensure that implementers participate and effectively communicate upstream. 3.3. HaRP PARTNERSHIPS The research-to-use process is founded on the concept that research will generate evidence that will be examined and applied in real-world programs to achieve high levels of effective coverage and, therefore, reductions in mortality and morbidity. In order for this process to be effective, many players need to be involved in supporting and conducting the research, but also in making use of the research. Finding: HaRP has done an excellent job of fostering alliances engaging a broad range of stakeholders (particularly at the global level) and creating a common sense of purpose among them. Many respondents, across all stakeholder groups, spoke of HaRP working directly and indirectly to effectively build relationships and spheres of influence. HaRP works directly through the actions of its implementing partners, but also indirectly through its influence to bring people together and foster action among those it is not directly funding. “Most of the contribution from HaRP has been not direct – a little bit of direct. Most of the ways it contributed was indirect and has had to do with relationships facilitated by HaRP.” (Collaborating Organization) Combining a cautious and active approach (the combination being seen as a particular strength of the AOR/management team), the HaRP program has been able to build non-competitive relationships, and has gone a long way in engaging other players and accomplishing what is achieved to date. As one respondent from a collaborating organization expressed: “Fostering partnerships and alliances is a key success. USAID [HaRP] is very good at that. They are also very responsive in outreach when another group is brought to their attention. This approach is fundamental to any progress at global level, and is the model for progress at country, regional, local levels.” (Collaborating Organization) 15 HaRP has effectively made use of its power to convene to push harmonization of research efforts for both sepsis and respectful maternity care. Finding: There is an increasing awareness that broad and early participation of country￾level stakeholders, including professional associations, local academics, and policymakers and implementers, in all phases of the research-to-use process is of key importance. While the experience with zinc indicated the difficulty of moving evidence from research into changes in policies and implementation at country level, HaRP’s investments related to CHX show a more active approach to ensuring correct partnerships all along the continuum – including, as appropriate, other donors, local academic partners, Ministries of Health, and professional associations. The omission of this step within the RMC project in Tanzania was a root cause of delays in the timelines of the research-to￾use process there. “Some products can use the diffusion model, but mostly it is inadequate. You need long-term engagement with research partners with in-country settings, with governments that change a lot, and with local partners, too. We need complexity models for scale and sustainability with upfront development and negotiation.” (Implementing Partner) Introduction and scale-up efforts will be greatly facilitated by the involvement of trusted country-level champions and allies, and this is of critical importance for implementation research, as seen in the example of RMC in Tanzania where local implementing partners in the area were not engaged early on, and local partners that were engaged did not have the trust of those being studied. As a global research mechanism, HaRP has limited opportunities to influence this kind of engagement of country level actors. However, it has influenced country-level participation through the CHX working group and regional meetings it organized, and by what it specifies within Requests for Applications. “The chlorhexidine working group has done better by ensuring that U.S. people engage with their in-country partners – and country people show up by phone or in person to the chlorhexidine working group. So this is “top performers” for country engagement.” (Donor) Finding: The commercial sector is an important potential partner in activities designed to introduce and scale up new interventions, and needs to be engaged early, but carefully in the research-to-use process. Many respondents spoke of the need to engage the commercial sector early on and throughout the research-to-use process because this sector provides a very important perspective and expertise, and has a key role to play in scale-up. There were examples, such as zinc, where the commercial sector did not develop adequate interest in the product, hampering scale-up. In order for introduction and scale-up to be effective, the commercial sector needs to be engaged (at least in discussions) at the product development stage. “Need to be engaging the private sector earlier on in the process… what they can and can’t do. As early as possible, get them involved and localize or regionalize manufacturing because adequate supply is very important. USAID has had mixed success in working with the private sector. How you work with them matters. USAID expects private sector to do something, as opposed to working with them in partnership and building a trusting relationship.” (Implementing Partner) 16 “If HaRP is committed to the introduction process, it has to increase the salience of commercial partners who are realistic about price points, value proposition, feasibility of manufacturing, distribution, and marketing. Have to make the case to industry of risk-sharing.” (Implementing Partner) Engagement of the commercial sector was (and continues to be) critical to country adoption of CHX. In the example of Nepal, the relationship was clearly defined, accountabilities were specified, and expectations were communicated, including future procurement rules. There was value to the pharmaceutical company to improve quality, develop a new product, and more recently, to supply that product and/or technical assistance to companies in other countries such as Nigeria and Ethiopia. More recently, the UN Commission on Life-Saving Commodities (UNCoLSC) CHX Working Group has provided a useful global venue to more closely and effectively engage the commercial sector. (HaRP’s provision of secretariat support has contributed substantially to this success.) However, several respondents spoke about the need to involve them in a sensitive manner and manage any real or perceived conflicts of interest. In Nepal, the commercial sector was purposefully not included in meetings of research findings, whereas doing so in Bangladesh created impressions of undue influence and set back the progress on scale-up. “Both chlorhexidine and Helping Babies Breathe have shown that private sector can play a role. Better to have them engaged – but not too far too fast.” [International Organization] “A manufacturer [was there during dissemination of study findings and] ended up confusing people. It inadvertently became all about promoting interests of a local company. The way they managed participation by private sector partner was a problem. There were key professional leaders that walked away with this is a marketing thing. That set Bangladesh back.” (Collaborating Organization) 3.4. HaRP PROCESS: ASSESSMENT/PRIORITY SETTING The assessment role of HaRP in identifying research areas and priorities, as delineated in the HaRP strategy, was strongly affirmed by all respondents familiar with HaRP itself. Each of the clinical topics in the portfolio can be traced and linked either directly or indirectly to a cause of maternal, newborn or child mortality or morbidity, and the systems research supported addresses service delivery platforms for new interventions. HaRP is also recognized for its role in identifying important, but neglected topics such as respectful maternity care, and for leading the efforts to address them. However, there is also confusion about what drives HaRP priorities over time and a sense that the resulting programs are diffused and poorly coordinated. Finding: HaRP is admired for playing valuable roles in identifying and advocating for attention for neglected topics. Respondents were particularly supportive of HaRP’s instrumental involvement in bringing “orphan” topics to the agenda. The two case studies selected for this review were exemplary of this support. Disrespect and abuse in maternity care was clearly identified as an issue that could adversely affect women’s willingness to seek facility-based health care services. “If HaRP had not funded this RMC focus…the entire research focus would likely have been lost. No one else was interested in placing a priority on this topic.” (Implementing Partner) 17 The operational research piece for CHX for cord care, which forged ahead on the basis of the evidence of its safety and efficacy, despite the then-current WHO guidelines and decades of country-level advocacy for dry cord care practices, was acknowledged as a forward-thinking and bold approach. “…USAID’s priority – accelerate adoption, do a little bit of learning, but then drive it into implementation and programs – they saw that as their role and they did play a leadership role around CHX.” (Donor) Finding: The vision that drives HaRP priorities and programs is not always clear to those directly or indirectly engaged with HaRP, giving the impression of a fragmented program that misses out on potential synergies. The HaRP portfolio is relatively large, comprised of numerous programs, projects and activities intended to advance the HaRP mission. The core HaRP team has a clear vision for the overall program and for how its various elements relate to each other. However, to outsiders, the collection of individual projects or partners seems to lack a central driving vision and, therefore, programmatic glue. This lack of clarity relates more to the overall portfolio than to specific research areas. However, even within specific areas, partners were not always clear about how they were being used to contribute to the bigger effort. “There are times when the boundaries between the work of various partners gets complicated, making it hard to know who is doing what. ….USAID didn’t do a good enough job in communicating with all of us about everything that was going on and by whom, and what they expected us to do.” (Implementing Partner) “An obstacle to successful research-to-use is fragmentation among the various components of HaRP – there seems to be no real overarching coordinating mechanism. HaRP is trying to fit the pieces together, but the continuum is not even close to seamless, especially with the downstream parts involving implementation. Zinc is a good example. It was promising but the pieces were being carried out by different agencies without adequate coordination.” (Implementing Partner) Examples of when HaRP has successfully exercised such leadership do exist around sub-themes, such as simplified antibiotics for treatment of newborn sepsis. In these cases, HaRP has clearly communicated a focused goal, shepherded partner roles by comparative advantage, and actively enabled close coordination. HaRP’s approach with sepsis was shaped by learning from earlier efforts such as zinc for child diarrhea. A related issue is the perception among some of those interviewed that, like the rest of USAID, HaRP changes its priorities often, moving to new priorities and projects before previous ones were seen through to completion. While HaRP has reduced the number of priorities it focuses on, the perception of changing course still lingers. “HaRP’s priorities seem to change often, confusing its partners and allies in the field and interfering with the momentum created for successful introduction, use, and scale-up. Often, HaRP doesn’t seem to stick with a priority long enough to see it through to completion.” (Implementing Partner) 3.5. HaRP PROCESS: DEVELOPMENT AND INTRODUCTION HaRP’s support for applied research on products and interventions, and on introduction of new products, is credited with accelerating the time from research to use of research results. However, respondents emphasized the need to think more deeply and earlier about product registration and 18 generation of demand, and about understanding the existing health systems and context in which new products and interventions are to be introduced. Finding: The research-to-use process for CHX was accelerated by HaRP’s support for implementation research and product development in parallel with effectiveness research. The combination of organizations, projects, and individuals involved with HaRP’s research-to-use process in Nepal enabled the results from the first CHX study to be used immediately in product development and implementation research, at the same time that effectiveness trials were taking place in Bangladesh and Pakistan. HaRP’s simultaneous support during the different stages of research, especially convening experts and financing hard-to-fund activities, clearly shortened the time from research-to-use for CHX. Although stakeholders may not have observed HaRP’s preparation for moving forward even before the proof-of-principle research had started, they were appreciative of the rapid progress into the operational aspects. “Pretty much as soon as results…were coming out, USAID was pushing immediately into OR and thinking about user preferences around formulations, about local manufacturing, about MOH engagements in scaling up” (Donor) “During the time of that study [gel vs. liquid] they were engaging in operations work to understand how to support pilot testing of programs. Was sort of a vision behind it – recognized that there were questions that were going to be asked after efficacy data. Putting the whole community of researchers, policy makers, donors in terms of being ready to answer those questions. …. Shortened the time line.” (Implementing Partner) Finding: Failure to adequately anticipate and address the challenges of registering and obtaining approval for new products has been an obstacle in their development and introduction. The time and procedures required to register new products and interventions in each country has emerged as a significant bottleneck to introduction and scale-up. Examples include: (i) zinc, which was shifted from a mineral supplement to a treatment for sick children and, therefore, had to meet more stringent requirements; and (ii) importation of CHX for cord care into Ethiopia which could have required 18 months of processing. Respondents recognize that this is an area where the U.S. government has influence, and call for greater participation in the future. As noted in the “HaRP Partnership” section above, early engagement of the commercial sector in identifying possible bottlenecks and strategies for overcoming them is also perceived as a way of facilitating introduction. “ [There is an] assumption [that] because evidence is there and policy change that’s all we need – but whole product/introduction is just not that – especially the registration side. It’s so weak within national governments. We are also weak as a collective on this.” (International Organization) “Where we made mistakes was in not understanding the product side of it. What were the implications of packaging it?... Disconnect of research side with manufacturing and regulation. That is where USAID kind of reconnected with having USP involved…Before [zinc was] part of a treatment – registered as a micronutrient – less stringent QA. Had a health manufacture base as micronutrient but making it a treatment – as a community we didn’t think through the implications.” (International Organization) “When it comes to product registration, that is a hurdle where USAID could help. The USG is not always so popular, but there are countries where they have some influence – and they could be less politically correct 19 and be more daring, and help with registration – our FDA could help with this. They could [be] pushing WHO to making the pre-qualification process more efficient.” (Implementing Partner) Finding: The tendency of researchers to focus attention on new products/interventions at the expense of understanding the context in which they are to be introduced has been an obstacle to introduction and scale-up. During priority-setting and product development, the research community, including HaRP, needs to pay more attention to the feasibility of introducing interventions into existing health systems, reinforcing the need for engaging implementers in framing priority-setting and early effectiveness research. HaRP has made efforts in this area, but more needs to be done. Despite its relative success, CHX has faced a number of challenges (including adoption of the intervention required reversing the widely known and accepted WHO guidelines related to dry cord care), and considerable resistance to uptake by professional bodies in Bangladesh. Furthermore, because most trials for cord care took place in community settings, there has been confusion over CHX guidelines for facility births. In Nepal, where female community health volunteers are the vehicle for service delivery, challenges with reach and quality of that platform have hindered the successful scale-up of CHX for cord care. “If you integrate a good [intervention] into a bad program, even good intervention will be bad. This is a challenge for CHX – could have achieved better if there were a better platform.” (Implementing Partner) “If implementers of applied research portfolio were more engaged at that level to help guide their thinking as they develop specific activities, there is more likelihood they will hit the nail on the head, addressing the right questions or issues. Even at this stage identify down the road where we are going to integrate this into a service delivery package. Keep in mind what is practical use of whatever it is that we are testing.” (Collaborating Partner) The progress of the RMC research in Tanzania was delayed by a failure to engage a broad range of community stakeholders at the onset of the project in a timely and sensitive manner. “At country level, [there is] not enough ground work done with professional associations and women’s advocacy groups to inform them of intention of the work – lost too much time making reassurances.” (Collaborating Partner) “The study would have been more effective if the team was more engaged earlier and had not lost time trying to decide on an intervention that was manageable, and if it had taken on board some civil society actors and broadened beyond their comfort zone (health care workers and managers). I think more rapid progress would have been noted if external/ community pressure was engaged.” (USAID) Finding: Too little attention is paid to generation of demand for new interventions, especially early in the research-to-use process. There is a need to solicit commercial sector expertise and country-level engagement earlier in the research-to-use process to understand what is needed for ensuring effective demand for products and interventions when they become available. Respondents commented on the critical insight commercial sector partners can provide about manufacturing and marketing, and emphasized the need to create local demand by engaging communities and providers through education and/or mass media campaigns. Whether or not the commercial sector is involved, creating demand deserves greater attention among upstream researchers and donors. At worst, all fails if users do not want and will not use the 20 intervention and at best, adoption time and scale-up will be longer. HaRP has encountered resistance to work in this space by researchers who fear contamination of studies and implementers who cannot afford to access what are often commercial sector resources. “If you want someone to bring research to practice, they [commercial sector] have the expertise. For example, if we have an idea to develop something, I will ask someone in industry whether it is worth it to pursue, and they can give advice or feedback because they have maybe already had experience or explored this idea.” (Implementing Partner) “We continue to struggle on the whole behavior change side – USAID has an important role to play there. Could probably do more and would be a huge service to increase uptake, put behavior change on the same plane as other aspects of public health – discipline/science we need to do better. Super important.” (Implementing Partner) Finding: Success of introduction efforts requires incorporating strong local input in identifying the barriers to use and scale-up, including attitudes and behaviors, and strategies for overcoming them. Until recently, many discussions around research-to-use focused on how to get policy makers to use research, rather than on how to prioritize, conduct, and share research in ways that engage those who will ultimately be responsible for implementing and scaling up new interventions and approaches. Flexible research strategies are important for successful introduction and scale-up because they allow researchers to adapt when they recognize that interventions must be modified as they encounter real￾world needs, and will require strong in-country capacity for implementation and delivery science. HaRP’s early experience with HRCI’s evaluation research role in the PMTCT-MCH program in Tanzania and TRAction’s focus on incorporating capacity building into local studies are two newer and potentially better approaches. Several respondents across stakeholder groups expressed a view that the unidirectional partnership frame – researcher to implementer — is no longer adequate, and a new paradigm that is two-way and more holistic is needed. “When WHO did the Global Health Research Strategy, it was done by academics and some of the language was really off-putting – “Policy makers should make use of research….” People are recognizing that IR means engaging implementers in the research…” (International Organization) “There have not been enough innovations on the behavior side, not enough local level learning. The current research-to-use paradigm is a one-way paradigm. The future paradigm is two-way -- learning from practice. To understand what works in terms of scale-up and sustainability, lots of important practice and social/organizational changes need to come from below. We need to develop a culture of that kind of learning…” (Implementing Partner) 3.6. HaRP PROCESS: FIELD IMPLEMENTATION AND SCALE-UP As new interventions and approaches move beyond introduction to larger scale implementation, the contributions of other USAID-funded projects – as well as those of other donors – become increasingly important. These include USAID-funded bilateral projects, centrally funded projects such as MCHIP, and global entities that receive USAID and HaRP support such as the UN Commission on Lifesaving Commodities Chlorhexidine Working Group. So, HaRP’s goal of rapid, wide-scale use of interventions 21 relies not only on leveraging these efforts at earlier stages in the research-to-use process, as described earlier, but also on effective collaboration with those supporting the front line of implementation. Finding: HaRP has made contributions to prioritizing and positioning for MNCH scale-up, but there is room for improvement. During the development of CHX and, more recently, simplified antibiotic regimens for treatment of neonatal sepsis, the role of HaRP was recognized as critical in ensuring an early focus on what would be needed for introduction and rapid scale-up. HaRP’s leadership, in turn, led, facilitated, and persuaded stakeholders to move more rapidly toward scale. This worked more effectively in some places due to well-organized local resources (Nepal), and sometimes did not work effectively due to lack of understanding of politics and decision-making (Bangladesh). In addition, HaRP’s ability to systematically engage other relevant USAID resources and to effectively ‘hand off’ for implementation was often limited (MCHIP, Missions). “People anticipated that people will ask questions about scale-up … what can we do during the interim … how to get a population engaged – how do you think about the options – what will people use on the cord, how would they incorporate new advice into understanding of previous messages – HOW it might be done but perhaps not as much focused on the politics and funding requirements for scale-up.” (Implementing Partner) “USAID/HaRP is not well positioned to deal with scale-up – its resources too limited. Need to engage others and work with their own networks (like MCHIP) to take the science forward.” (USAID) Finding: HaRP’s activities have focused more on the development and introduction of interventions, but have not adequately addressed some critical gaps in actions for scale-up and sustained delivery. HaRP has done increasingly well in paying more attention to activities that would build program design information and buy-in, simultaneously with the development and early introduction work. However, at these stages and in planning forward, HaRP and its implementing partners tended to pay too little attention to system requirements that can become significant bottlenecks to large-scale use in most countries. These impediments may vary by country, but they have included administrative and legal mandates, the strength and resilience of the underlying health system or its components, and the requirements for mobilizing partners who are critical to roll out. As noted earlier, the most visible examples have come from the need for regulatory and/or licensing approval in which every country has its own, usually complicated and lengthy, procedures. A major lesson learned from the introduction of zinc for child diarrhea was to better anticipate these needs. The case of a simple product such as CHX (and with the resources of UN Commission on Life Saving Commodities) has not demonstrated forthcoming rapid regulatory approvals in most cases. This indicates that more preparatory work is needed during the development phases. Effective scale-up requires deliberate actions over time. While HaRP’s role in this phase is often facilitative and indirect, it cannot be assumed that introduction will lead to further scale-up. For example, CHX is considered a scale-up success. However, out of 15 countries in some stage of introduction, only one (Nepal) is covering a significant proportion of its population. 22 “Introduction is introduction, needs something next – formalized planning process, multiple partners, not separate. Evaluations. Sharing information to course correct. Have to invest in systematic processes at the downstream level of implementation.” (Collaborating Partner) Finding: HaRP has used its convening capacity to facilitate sharing among experts and South-to-South conversations among country policymakers and practitioners. Well-run local and global expert meetings were very useful and, in some cases, accelerated the research-to-use process by harmonizing research and ensuring collaboration rather than competition. Global and regional technical meetings were effective in sharing experience and convincing policy makers to adopt effective interventions. “In 2008, there was a meeting in London jointly organized by SNL, USAID, WHO – how do we now take what we have learned about sepsis forward …. That was a great process. Harmonization and developed common sense of purpose. Usually agencies do their own thing. I always give this as an example of how to do things in partnership – USAID played a good and critical role in that.” (International Organization) Finding: Inadequacy of current knowledge management systems for USAID and HaRP is leading to loss of learning on key elements of successful implementation and scale-up. While HaRP addresses implementation questions and activities earlier in the research-to-use process, its principal focus of action is at the development and introduction stages. As countries and projects implement these interventions, they are country-focused, tend to be funded by multiple sources and in some ways must be ‘let go’ by HaRP. The sharing of learning that HaRP has established among researchers and during development often does not carry over into implementation and scale-up, and vice versa. “Need funding for evaluation! So much learning in MCHIP, but I fear it will be lost because not properly harvested, pulled back around and used to improve programs – even in same country with same partners. Especially across partners and countries. Knowledge capture and management is one of the weakest areas in the whole newborn movement in which USAID has been a leader.” (Collaborating Partner) Finding: We all lack knowledge of the ultimate impact of new interventions because scale￾up and sustained outcomes are not well measured, tracked, or shared. In general, monitoring and evaluation of large scale delivery programs is weak, inconsistent, and poorly supported. Definitions, standards, and efficient methods for program data collection are lacking and resources are insufficient to better understand what happens at a population level when interventions scale. In at least one country where they had access to this type of resource through HaRP, they were able to more systematically learn from early experience, thus accelerating progress. “If the end of the process isn’t impact – if you have adoption and roll-out but all you’ve accomplished is low coverage – not a worthwhile process. The last third base to home plate – assuring high coverage. Not sufficient attention. Need local capacity for this – more robust in-country capacity.” (Collaborating Partner) 3.7. HaRP MECHANISMS AND STRUCTURES HaRP is challenged by a number of bureaucratic issues, including the relationships between USAID/Washington and Missions and between programs, as well as USAID funding structures and 23 cycles. Many of these are outside of HaRP’s control, but respondents called for more effort to engage Missions in the research-to-use process. Finding: Differences in structure and culture between USAID/Washington and in-country Missions have impeded HaRP’s ability to achieve local introductions, field implementation, use, and scale-up of new interventions. Missions usually do not actively participate in setting global research agendas, and there have been issues with the “handoff” to Missions during introduction and scale-up. Missions are perceived as not understanding or valuing research, or not giving it sufficient priority in the context of many interventions, and more policy and service oriented activities. Missions have played varying roles - in a few cases making significant contributions to intervention introduction and scale-up, such as for newborn programs in Nepal and Bangladesh. However, roles appear to be dependent on individual interest and competition for scarce time. “The Mission didn’t fund the original study because we felt and continue to feel that funding that is allocated to countries really needs to focus more on field implementation, feasibility studies, capacity building, TA and less on fundamental research proof of principle type questions that USAID/Washington should fund. In some cases, Mission funding can help to augment slightly, but the purpose of the funding that goes to countries [is] really for taking proven things and applying it and training on its use, helping the country integrate into a service delivery model.” (USAID-Country) “There is a question mark about how we approach these problems agency-wide and the view of research in Missions. We need to systematically engage field staff in research process/questions.” (USAID) There is concern that without adequate clarity of roles in the research-to-use process, there is a danger that effective interventions will not get introduced and scaled, even by USAID missions. “Don’t visualize it as a hand-off – We don’t have direct connection, so a lot of hand waving.” (USAID) Finding: There is a felt need to reduce bureaucratic budgetary and temporal barriers to program progress. Short funding cycles and approval lag times present obstacles for implementing partners and impede research progress overall. Respondents engaged with the RMC work felt that the nature of this research, including the need to define RMC, sensitize stakeholders and open the discussion, required more time than was allocated under current USAID funding cycles. “The process of evaluating programs and making funding decisions is not clear to me. I hear several times, when we submit proposals or the work-plan, that they will come back after they have talked to the advisory group. We never have a work-plan approved before October 1st . This year, we started in May but by October, we only had provisional approval.” (Implementing Partner) “A big problem is short funding cycles for problems that do not fit within the 2 year or other contractual timeframe. Projects get started – stopped – then have to gear up under perhaps another PI/NGO – no continuity possible.” (Collaborating Partner) 24 Other respondents commented on the lack of synergy and perceived competition between programs within USAID, which they see as wasting resources and/or overshadowing the progress of HaRP and other programs, with specific mention of the Grand Challenges and Survive and Thrive. “The Grand Challenges running parallel to HaRP has been a challenge for HaRP – because some incredible things come out of the HaRP program, but it is not as sexy and not as “WOW” as the Grand Challenges. This creates tensions and challenges.” (USAID) 3.8.IMPLEMENTATION RESEARCH/DELIVERY SCIENCE As a means of facilitating the introduction and ultimate use and scale-up of new interventions, HaRP is supporting and promoting implementing research/delivery science designed to increase knowledge of local contexts where new interventions are to be introduced, including the nature of health systems, policy environments, provider practices, and cultural barriers. To date, TRAction has been HaRP’s most systematic effort and they are beginning work with WHO to build local researcher capacity. Finding: HaRP is playing a valuable leadership role in growing and advancing the field of implementation/delivery science aimed at increasing knowledge of the health system practice and policy environments in which new interventions are introduced. In general, those interviewed applaud HaRP for providing leadership in this area. As one respondent put it, “HaRP should fund more implementation research to learn how to accelerate the rate at which interventions benefit the women and children for whom they were intended. Once you have evidence that an intervention works, you need to go the next step: Is it feasible in the real world? Will clients come for the treatment? Will the providers implement it? How much will it cost? These are critical questions that are not addressed by the more traditional research strategies.” (Implementing Partner) In promoting more implementation research, HaRP is finding it necessary to overcome a prevailing belief that only controlled trials can provide the quality of evidence necessary to justify introduction. “In promoting research strategies that produce evidence more applicable in real-world settings, HaRP is swimming against the prevailing tide of biomedical-type research that stresses randomized trials. However, HaRP is gaining traction with other agencies that are sympathetic to its approach to research and implementation, including WHO and the Bank.” (USAID) “Randomized trials are artificial, not real world. We need more implementation research, both qualitative and quantitative, that focuses on the right end of the research-to-use continuum i.e., effectiveness research that goes beyond RCTs. An example is Gates’ support of implementation of community-based interventions; ministries want to know not only whether a new intervention works, but how much implementing it will cost.” (Implementing Partner) In addition, implementation research is hampered by the differences in perspective between researchers in the North who prefer the more glamorous work of inventing new interventions, and those in the South whose focus is on getting interventions to those who need them most. “Lots of people are working on intervention development and innovation. However, fewer are supporting implementation research. It seems that people in the North are more interested in the development of new interventions as it is more sexy than the messy business of implementation. People in developing countries, however, are more interested in implementation, asking how they can improve the delivery system and get solutions to the people who need them most.” (Implementing Partner) 25 “Implementation/delivery science is hampered by a top-down, one-way mentality that tends to exclude local perspectives and insights. As a result, it is difficult for researchers and local implementers to find common cause in understanding the environment for introduction of new interventions and in getting those interventions introduced and scaled.” (Implementing Partner) HaRP is also challenged with the lack of capacity for conducting implementation research in many of the countries in which HaRP and others would like to promote such research. “The increasing recognition of the need for and emphasis on implementation/delivery science is highlighting the lack of local capacity among researchers and implementers in target countries to conduct implementation research of the quality necessary to facilitate introduction, adoption, and scale-up.” (Implementing Partner) “A significant weakness in the research-to-use paradigm is the lack of capacity to measure actual use, coverage, and scale-up of interventions introduced in target countries. Without data on these indicators, it is difficult to demonstrate the effectiveness of a new intervention, to identify facilitators of and barriers to implementation, and to adjust implementation strategies accordingly.” (International Organization) 4. EVALUATION CONCLUSIONS AND RECOMMENDATIONS Based on the findings of the HaRP evaluation delineated above, the Evaluation Team has developed a series of recommendations that, if adopted, will strengthen HaRP’s strategy and process, and enhance its impact in the countries where it works. Conclusions and recommendations are presented here related to main sections of the findings, and those considered to be highest priority by the Evaluation Team are shown in bold italics. 4.1. HaRP’s APPROACH Conclusion: HaRP’s low-key, understated approach, while admired by many, may lead to lower visibility both within and outside the agency, and hinder its ability to achieve its mission, guide its implementing partners, and coordinate their activities. Recommendation: HaRP should be more assertive in describing its vision, process, and achievements both within and outside of USAID so that it can: (i) be recognized for what it is achieving; (ii) be able to influence the priorities of its implementing partners and in-country allies; (iii) receive more attention from USAID Missions when trying to implement programs in their countries; and (iv) better compete for attention and budgets within the Agency. Conclusion: Those interacting with HaRP are not able to articulate clearly what central vision drives the HaRP portfolio and how projects supported by HaRP relate to each other. As a result, some perceive the portfolio as fragmented and lacking coherence. Recommendation: HaRP should devote greater attention to articulating the vision that drives its research-to-use strategy, and to increasing coordination, collaboration, and synergy among the programs and activities that constitute its portfolio so that the whole is greater than the sum of its parts and individual actors see their contribution to the overall vision. 26 4.2. HaRP’s RESEARCH-TO-USE STRATEGY Conclusion: With other donors focusing on discovery science and development, and the pipeline filled with new interventions and products, there is a need for increased attention to moving those interventions/products through the research-to-use process to introduction, use, and scale￾up. Recommendation: Considering what is perceived by others to be HaRP’s comparative advantage, HaRP should place greater emphasis on implementation research and on contributing to thinking from day one about what will be needed for effective field implementation and scale￾up. This focus on implementation research should not be to the exclusion of funding effectiveness research in situations where HaRP’s investment will add value and fill key gaps. Conclusion: HaRP’s conceptual framework, an idealized depiction of the process for advancing new interventions from discovery to introduction and field implementation, is not an accurate reflection of what happens on the ground in countries where wide-scale use and scale-up are goals, or of the non￾linear process HaRP is often using. Recommendation: HaRP should replace the current conceptual framework with one that: i) reflects the non-linear, iterative and sometimes parallel avenues in the research-to-use process; ii) is more explicit about the specific roles that HaRP plays and those that other stakeholders play; and iii) more clearly lays out the ultimate goals of use (effective coverage with key interventions that will produce impact at scale). Recommendation: In considering how to revise its current strategy, HaRP should define its comparative advantages in global health vis-à-vis other major donors, multilateral and bilateral agencies, and NGOs so that its programs are designed for maximum impact and are complementary to and synergistic with programs of these other organizations. 4.3. HaRP’s PRIORITY-SETTING PROCESS Conclusion: While the process through which HaRP conducts assessments and establishes priorities is rigorous and produces clear goals and expectations, HaRP’s priorities do not always appear to be adequately revised and aligned with changes in context or associated learning. Recommendation: HaRP should formalize a scanning function designed to identify and document changes in the contexts where HaRP is funding research globally and locally, and refine priorities as needed to anticipate and adapt to environmental shifts that have the potential to facilitate or impede its research-to-use efforts. For example, with CHX, it would have been important to recognize the increasing rate of facility births and ensure that research was conducted in those settings, too. Conclusion: Partners perceive that HaRP changes its priorities thus losing programmatic momentum, confusing partners, and jeopardizing success. Recommendation: HaRP should stick with priorities long enough to see them through to an appropriate completion of the research-to-use process, building on positive experiences such as with CHX or TRAction. Concurrently, to prevent misperceptions, they should strengthen communication with partners on their follow-through on these priorities over time. 27 4.4. HaRP’s DEVELOPMENT AND INTRODUCTION PROCESSES Conclusion: To be effective, the processes through which HaRP supports development and introduction of new interventions must be accompanied by input from local partners who will ultimately be responsible for implementing interventions in the field. Recommendation: Early in the research-to-use process, HaRP should focus more on ensuring its partners are creating or enabling others to create local environments conducive to acceptance, adoption, and use of new interventions, by including this goal directly in the scope of work of implementing partners. This may require new approaches and/or more staff resources to close the global/local divide. Recommendation: HaRP should intensify its efforts to identify, enlist, recognize and support country￾level champions (including USAID Missions) to drive the introduction process and overcome local barriers, because local champions who understand the landscape of politics and public health in their country can have much greater influence on policy and practices in their countries than outsiders can. Recommendation: HaRP should anticipate and address, early in the development and introduction processes, the often seemingly intractable challenges of registering and obtaining approval of new interventions in target countries. Conclusion: More attention needs to be paid, earlier in the research-to-use process, to creating demand and changing behavior in the communities where new interventions are to be introduced. Recommendation: HaRP should intensify its efforts and support, early in the research-to-use process, to: (i) create local demand for new interventions in target countries, by engaging communities, providers, and the commercial sector; and (ii) change the behaviors of providers and communities expected to be involved in the introduction and field implementation of new interventions. 4.5. HaRP’s FIELD IMPLEMENTATION AND SCALE-UP PROCESSES Conclusion: As new interventions move beyond introduction to field implementation and scale-up, the contributions of other players –global and local – become increasingly significant. HaRP’s goal of rapid, wide-scale use of interventions relies not only on leveraging these efforts at earlier stages in the research-to-use process, but also on effective collaboration with those who support the front line so that they can pick up where HaRP’s direct involvement drops off. Despite the critical importance of these phases of the process, most current support is for the pre-scale-up phases, leaving critical gaps in planning for scale-up Recommendation: To increase the degree to which interventions in its portfolio are implemented and scaled-up, HaRP should enable its implementing partners to devote greater attention to engaging, as early in the research-to-use process as possible, knowledgeable in-country stakeholders who will be leading field implementation and scale-up efforts. Recommendation: To increase the probability of success of field implementation and scale-up efforts in local settings, in addition to TRAction, HaRP should use more of its own staff resources and have its implementing partners devote more attention to factors capable of either facilitating or impeding the efforts earlier in the research-to-use process, not only in the implementation research phase, but even during the development phase. These factors include strength and resilience of the health systems 28 platform where an intervention is to be integrated, mobilization of partners needed to support roll-out, and relevant regulatory requirements. Conclusion: HaRP has used its convening power to organize meetings of local, regional, and global experts that in some cases accelerated the research-to-use process by harmonizing research and ensuring collaboration rather than competition. Recommendation: HaRP should continue to use its convening capacity to facilitate sharing and learning among global and local experts, and promote south-to-south conversations among country policymakers and practitioners whose decisions determine the success of field implementation and scale-up of new interventions. 4.6. HOW HaRP WORKS WITH PARTNERS Conclusion: Successful engagement of local partners in target countries involves a two-way interaction related to development and introduction of new interventions/products, but the relationships between global and local partners still remain characterized as one-way and/or top-down. Recommendation: HaRP should provide incentives for its implementing partners to work with country-level stakeholders as equal partners in the research-to-use process, and prioritize, conduct, and share research in ways that fully engage those who will ultimately be responsible for implementing and scaling up these new interventions and approaches. Conclusion: Success of HaRP’s research-to-use process requires effective partnerships with a range of organizations –global and local – across the full spectrum from research-to-use and scale-up. Recommendation: HaRP should devote significant attention to achieving broad and early participation of country-level stakeholders (e.g., professional associations, local academics, Ministries of Health and other policymakers, implementers, and the commercial sector), and to identifying trusted champions and allies who will lead field implementation and scale-up efforts on the ground, both through its own actions and by providing incentives for its implementing partners. Conclusion: One group of potentially valuable stakeholders is the commercial sector. They are often only brought into the process at the very end, although they provide unique knowledge and valuable methods that could be applied early on. While conflicts of interest need management, there are successful models of engagement. Recommendation: HaRP should increase its own efforts and those of its implementing partners to engage the commercial sector early in the research-to-use process in order to enhance the likelihood that the interventions that emerge from the development process are appropriate for the contexts into which they are to be introduced, and will be adopted, used, and scaled up. 4.7. HaRP STRUCTURES AND MECHANISMS Conclusion: HaRP’s efforts to create an effective, rapid, and synergistically supported research￾to-use process are challenged by a number of USAID-related structures and dynamics, including the relationship between USAID/Washington and Missions, and among USAID programs, as well as by USAID funding structures and cycles. Many of these are outside of HaRP’s control, but respondents called for more effort to ameliorate them to the extent possible. 29 Recommendation: HaRP should increase its efforts and those of others in the Agency to engage USAID Missions early and consistently in the research-to-use process, anticipating their significant value in laying the groundwork for and in facilitating the introduction of field implementation, and scale-up of new interventions. Recommendation: While USAID funding structure and systems are a contextual challenge beyond HaRP’s control, HaRP should seek creative ways to mitigate the effects of short funding cycles, approval lag times, and other bureaucratic barriers to program progress. This may become more important if a decision is made to prioritize implementation research. 4.8. IMPLEMENTATION RESEARCH/DELIVERY SCIENCE Conclusions: The success of the research-to-use process depends on knowledge of the local contexts into which new interventions are to be introduced, including the nature of health systems, policy environments, provider practices, cultural barriers, etc. Implementation/delivery science is designed to understand what, why, and how interventions work in “real world” settings and to test approaches to improve them. HaRP is recognized for moving forward in this space (TRAction, meetings) but more needs to be done. Nearly all stakeholders view moving MNCH forward as one of the priorities. Recommendation: HaRP should expand its role as the leader in directly supporting and influencing others to support implementation of research/delivery science designed to elucidate the features of the context into which a new intervention is to be introduced, including the country’s health system, policy environments, manufacturing capacities, regulatory requirements, provider practices, and consumer preferences. Recommendation: HaRP should advocate with and leverage other development partners to document implementation learning and ultimate results through: i) surveillance systems that monitor the use of new interventions and their impact on health; and ii) knowledge management systems designed to capture key elements of successful and unsuccessful implementation and scale-up. Conclusion: Implementation research/delivery science will be more effective if it becomes a two￾way process with strong local input, which will require development of stronger in-country research capacities. Recommendation: HaRP should lead, support, and leverage the development of a new two-way and bottom-up paradigm and mind-set for implementation research/delivery science valuing the local perspectives and insights that are critical to understanding the environment for introduction and scale-up of new interventions, and ultimately the usefulness of implementing research findings. Recommendation: Local capacity for implementation research/delivery science is critical and HaRP should promote and facilitate its rapid development. 30 ANNEXES ANNEX 1. EVALUATION STATEMENT OF WORK TITLE: Health Research Program (HaRP) Evaluation Contract: Global Health Technical Assistance Bridge IV Project (GH Tech) PERFORMANCE PERIOD The evaluation is anticipated to involve work over a 3-month period starting early December 2013 and ending in late February 2014. FUNDING SOURCE GH/HIDN PURPOSE OF ASSIGNMENT The purpose of this external process evaluation is to assess the fit for purpose of the managed HaRP research-to-use strategy focusing on planning for scale of managed intervention, implementation/health services research and research introduction/utilization efforts. The primary product of the evaluation will be to develop a list of key considerations, challenges and recommendations for future similar efforts. The HaRP strategy was designed in 2003 as a guiding principle for research and research introduction activities financed by the Office of Health, Infectious Diseases, and Nutrition (HIDN). It was specifically designed as a strategic framework to guide Maternal and Child Health research-to-use activities with a goal of accelerated research and introduction/research utilization. BACKGROUND The USAID Health Research Program (HaRP) and strategy were initiated in FY 2003 by the Office of Health, Infectious Disease, and Nutrition, and are designed to accelerate the development and introduction/translation of research products into the effective implementation of USAID and partner country health programs. The HaRP program identifies, develops, tests, and facilitates the introduction of new/refined tools, technologies, approaches, policies, and/or interventions intended to improve the health status of infants, children, mothers, and families in developing countries. Activities are guided by a Pathway from Research to Field Implementation and Use framework, which outlines a USAID, managed process which is analogous to the value chain the private sector uses for moving research products into use:  Assessment - Strategic planning, consultation, problem identification, and priority setting.  Development - Applied research to create tools, technologies, approaches, and interventions.  Introduction - Catalytic activities including health services research and implementation research as well as other activities to facilitate research translation, adoption, and uptake of intervention and/or product.  Field Implementation - Country level program/policy roll out/diffusion into routine use The HaRP strategy guides a coordinated often concurrent multi-activity effort involving collaboration and coordination of USAID staff, grantees, and other partners during various entry points and stages of 31 the Pathway. . By necessity USAID missions and partners such as WHO and Saving Newborn Lives are key collaborators. Five activities are directly managed by a core USAID staff, but the team also works closely with others. The work of the HaRP program includes the spectrum of traditional research and development inclusive of product development, applied research, health services/implementation research, and other activities to facilitate uptake and planning for scale. The work undertaken under the HaRP strategy is intended to focus on research and development areas which would not advance without USAID technical leadership and investment. Core managed activities include: The Health Research Challenge for Impact (HRCI), a cooperative agreement with Johns Hopkins University is undertaking coordinated research studies to accelerate the process of conducting and translating new research into use in field programs. HRCI conducts health research and evaluations for the development, testing, and refinement of new and improved tools, technologies, approaches, interventions, and policies in developing countries. HRCI is conducting a small number of multi-year research studies and one large-scale evaluation related to newborn, child, maternal, and integrated MNCH health. The Translating Research into Action Project (TRAction), a cooperative agreement with University Research Corporation, is addressing the “know-do” gap; namely the translation of research into use so that research findings impact on improving public health. TRAction is addressing research and introduction gaps in effectively delivering and scaling-up evidence-based interventions and MNCH programmatic approaches. The project supports rigorous and practical health services research, evaluative research, and implementation research to develop effective application or delivery approaches for MNCH tools, interventions, and policies that are under-addressed. It also has a mandate to work on introduction activities to facilitate the translation of research and research findings into use. TRAction solicits, supports, and manages sub-award research, evaluation, and introduction activities. WHO/MCA and WHO/RHR HaRP supports the WHO maternal, child, and adolescent health and development departments at WHO/MCA and WHO/RHR to identify, sustain, and increase the effectiveness of strategies/technologies that advance child survival in developing countries. WHO/MCA and WHO/RHR refine existing strategies/technologies and develop new and cost-effective interventions to reduce mortality and morbidity associated with major childhood illnesses. WHO is responsible for establishing guidelines, standards and policies used by practitioners in hospitals, clinics and community settings. HealthTech V, a cooperative agreement with Path, develops, adapts, evaluates and/or facilitates the introduction of affordable and appropriate technology solutions for the safe, effective, and more equitable distribution of health care services in low-resource countries. This project is addressing implementation barriers (e.g., issues with technical design, supply chain management, and policy) that typically prevent innovative technologies from reaching the most vulnerable populations. Accelovate, a cooperative agreement with Jhpiego, is developing, introducing, and supporting the scale￾up of new health tools and technologies that are appropriate, affordable and acceptable for distribution and use in low-resource settings to accelerate reductions in mortality and morbidity in low-resource settings. Accelovate is helping overcome technical, supply, and policy hurdles to adaptation and advancement of effective technologies through innovations in the value chain and promotion of mainstream use with an emphasis on field introduction and scale-up. 32 The execution of the HaRP strategy also involves direct collaboration with USAID and partner funded activities, as well so-called snowball effect of others advancing work initiated under the HaRP strategy. Illustrative direct partners include flagship projects such as the MCHIP project, a cooperative agreement led by Jhpiego http://www.mchip.net/, PQM, a cooperative agreement with US Pharmacopeia http://www.usp.org/global-health-impact-programs/promoting-quality-medicines-pqmusaid, as well as the Gates Foundation/Save the Children Saving Newborn Lives program http://www.savethechildren.org/site/c.8rKLIXMGIpI4E/b.6234299/. SCOPE OF WORK (SOW) – SOW TO BE REVISED FOLLOWING CONSULTATIONS OF USAID AND EVALUATION TEAM The evaluation will focus an assessment of the fit for purpose of the HaRP strategy. The evaluation should use a comparative case study approach as the basis to focus on forward looking analyses to serve as a foundation to develop recommendations on future research-to-use processes to advance intervention development and implementation/health services research conducted in conjunction with research introduction/utilization, and other activities undertaken in conjunction of support of achievement of USAID development objectives. As appropriate, the evaluation team may draw on past HaRP health research investments including projects such as the Global Research Activity [2003-2009] and HealthTech IV [2011-2015] that link to the existing portfolio and approach. These may include, but are not limited to, interventions and/or products for community management of non-severe and severe pneumonia, simplified antibiotic treatments of sepsis in newborns, chlorhexidine for prevention of infection in newborns, and community case management. Additionally, the evaluation team may wish to review other relevant evaluations and alternative models to conduct similar work inclusive of other bilateral donors, product development partnerships, other USG research efforts, as well as programs funding health services research and implementation research such as the WHO implementation research platform. The 5-year Leader with Associate Cooperative Agreements for HRCI and TRAction is currently scheduled to end on September 30, 2014. A new project design and approval process is anticipated in early FY 2014. The recommendations resulting from this evaluation will inform the development of a new program concept, project appraisal document and any resulting future procurements under HaRP. The final approved project appraisal document will replace the existing HaRP Activity Approval Document (AAD) Audience: USAID staff and Missions to inform future investments and programming of research-to-use strategies with special reference to maternal and child health, but also, more generally, for other related global health activities. This evaluation is also intended to guide discussions on future directions for a broader international public health community interested in accelerated research-to-use strategies, the emerging field of implementation research. The report is anticipated to be available publicly in the Development Experience Clearinghouse (DEC). Intended Use: The information from this evaluation will help (1) develop lessons learned on the fitness for purpose of the HaRP program for USAID, (2) inform broader research and implementation on strengthening research-to-use processes. EVALUATION QUESTIONS – EVALUATION QUESTIONS TO BE REVISED FOLLOWING MEETINGS OF USAID AND THE EVALUATION TEAM 33 The following are an initial list of focus questions for the evaluation. The final list of questions will be developed or refined or both at the first meeting between the evaluation team and USAID. 1. What was the fit for purpose of the HaRP managed accelerated research-to-use strategy for both (i) design and (ii) execution of (a) intervention research and research introduction/utilization and planning for scale-up of new/refined interventions, focusing on one or more of the following: zinc/reduced osmolarity ORS, management of severe pneumonia at community level, chlorhexidine newborn cord care, and simplified switch therapy for presumptive newborn sepsis (b) implementation research/health services research focusing on planning for scale or addressing systemic health systems challenges that impede health system functionality and effective implementation of evidence-based approaches. 2. The primary focus of the evaluation should be to identify design issues and considerations for future efforts focusing on accelerated research, research utilization, and design for scale. Additionally the evaluation team should identify issues and questions that merit further investigation Methods The evaluation team will be encouraged to review the overall strategy design and its existing core programs and partner. They should ideally select 2-4 case studies to review work conducted under HaRP on product development/research and/or health service delivery/implementation research activities. If feasible they are encouraged to compare the HaRP activities with other research and research translation supported by USAID as well as other organizations that support this type of work. For example, the consultants may choose to examine the process and outcomes of HaRP supported research on development, testing, and roll out of chlorhexidine for prevention of newborn mortality compared with the Gates and MacArthur funded research on development, testing, and roll out of anti￾shock garments for postpartum hemorrhage. Sources for data include, but are not limited to, document review and key informant interviews. EVALUATION PROCESS – FINAL METHODOLOGY WILL BE DEVELOPED BY THE EVALUATION TEAM IN COLLABORATION WITH USAID The evaluation team will have to propose an appropriate evaluation approach that will be reviewed and agreed on by USAID before conducting the evaluation. The evaluation team will follow sound accounting procedures and be prudent in using the resources of the evaluation. The evaluation team will also follow a participatory and consultative approach ensuring close involvement of the Government, relevant program partners, and beneficiaries. The evaluation team will use evaluation tools, as well as develop and present, for USAID review and approval as part of the work plan, an analysis plan that details, but is not limited to, how interviews will be transcribed and analyzed; what procedures will be used to analyze qualitative data from key informant and other stakeholder interviews; and how the evaluation will weigh and integrate qualitative data from these sources with project performing monitoring records to reach conclusions about the HaRP projects and program. The information collected will be analyzed by the Evaluation Team and determine the major issues. Interviews 34 The Evaluation Team will conduct in-depth interviews including USAID staff, collaborating agency partners, researchers, policy-makers, other funders of researchers, and relevant implementation partners. All interviews will be in person or on the telephone – no international travel or international site visits are anticipated. TEAM COMPOSITION, SKILLS AND LEVEL OF EFFORT (LOE) Summary of Qualifications required for the Consultant team:  Strong knowledge, skills, and experience in program evaluation and research translation including applied and health services research/implementation, research/operations research, as well as product development.  Knowledge and experience with USAID contracting and reporting requirements, policies and initiatives, tools, and strategic frameworks, preferred.  Experience in public health with technical knowledge and experience with interventions, policies, and programs relevant to contemporary maternal, newborn and child health in developing countries.  Strong qualitative and quantitative analytical skills, and a mixed method orientation.  Advanced written and oral communication skills in English. Composition of Review team Ideally, two to four individuals with expertise in:  Strategic planning and knowledge of research and development  Service delivery implementation challenges related to maternal and child health, as well as familiarity with applied and health services research/implementation research approaches.  Applied, implementation, and health services research expertise  Research translation and dissemination expertise Level of Effort An illustrative table of the LOE is found below. Dates may be modified based on availability of consultants and key stakeholders, and amount of time needed for field work. Activity Team Leader (TBD) Team Member 1 Team Member 2 Team Member 2 POP (illustrative depending on start date) Review of background materials (to be provided by USAID) 5 5 5 3 Dates TPM 3 3 3 3 Dates Travel 3 3 3 - Dates Meetings in D.C. 5 5 5 5 Dates Interviews 5 5 5 5 Dates 35 Analysis of interviews and drafting of preliminary findings and PPT 10 10 10 10 Dates Debrief Presentation of Findings to USAID 1 1 1 1 Dates Evaluation Team incorporates Comments in the Final Draft 5 5 5 - Dates USAID Signs off on the Final Draft for Editing N/A N/A N/A N/A Dates GH Tech Edits/Formats/508 report N/A N/A N/A N/A Three Weeks Total LOE 30 30 30 20 October-January *A six-day work week is approved only for periods of international travel to accommodate travel/work days. LOGISTICS GH Tech will be responsible for all domestic and international travel and consultant logistics. DELIVERABLES AND PRODUCTS – DELIVERABLES WILL BE FURTHER DEFINED FOLLOWING MEETINGS BETWEEN USAID AND THE EVALUATION TEAM The team will prepare the following deliverables; all deliverables will require final approval by USAID/Washington.  Evaluation protocol including work plan, analysis plan, and outline of final report;  Data collection tools;  PowerPoint or initial report of key findings and questions;  Draft report if time permits DATA SETS AND REPORTING GUIDELINES All data instruments, data sets, if appropriate, presentations, meeting notes and report for this evaluation will be presented to USAID on three (3) flash drives to the Evaluation Program Manager. All data on the flash drive will be in an unlocked, editable format. Reporting Guidelines: The draft report should be a comprehensive, analytical, evidence-based evaluation report:  Detail and describe results, effects, constraints, and lessons learned  Provide recommendations and identify key questions for future consideration The report shall follow USAID branding procedures. An acceptable report will meet the following requirements, as per USAID policy (please see: the USAID Evaluation Policy):  The evaluation report should represent a thoughtful, well-researched and well-organized effort to objectively evaluate what worked in the project, what did not and why. 36  The evaluation report should address all evaluation questions included in the scope of work.  The evaluation report should include the scope of work as an Annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline, shall be agreed upon in writing.  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex to the final report.  Evaluation findings will assess outcomes and impacts using gender disaggregated data., if appropriate  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).  Evaluation findings should be presented as analyzed facts, evidence and data, and not based on anecdotes, hearsay or the compilation of people’s opinions.  Findings should be specific, concise and supported by strong quantitative or qualitative evidence.  Sources of information need to be properly identified and listed in an Annex, including a list of all individuals interviewed.  Recommendations need to be supported by findings. Recommendations should be action￾oriented, practical and specific, with defined responsibility for the action. The annexes to the report shall include:  The Evaluation Scope of Work  Any “statements of differences” regarding significant unresolved difference of opinion by funders, implementers, and/or members of the evaluation team  All tools used in conducting the evaluation, such as questionnaires, checklists, survey instruments, and discussion guides  Sources of information, properly identified and listed  Disclosure of conflicts of interest forms for all evaluation team members, either attesting to a lack of conflict of interest or describing existing conflict of interest. Data Quality Standards To be useful for performance management and credible for reporting, USAID Mission/Offices and Missions should ensure that the performance data in the PMP for each DO meet five data quality standards (abbreviated VIPRT). When this is not the case, the known data limitations and plans to address them should be documented in the indicator reference sheet in the PMP. Note that the same data quality standards apply to quantitative and qualitative performance data. (a) Validity. Data should clearly and adequately represent the intended result. Another key issue is whether data reflect a bias such as interviewer bias, unrepresentative sampling, or transcription bias. (b) Integrity. Data that are collected, analyzed, and reported should have established mechanisms in place to reduce the possibility that they are intentionally manipulated for political or personal reasons. (c) Precision. Data should be sufficiently precise to present a fair picture of performance (d) Timeliness. Data should be timely enough to influence management decision-making at the appropriate levels. One key issue is whether the data are available frequently enough to influence the appropriate level of management decisions. A second key issue is whether data are current enough when they become available. 37 For further discussion, see USAID Information Quality Guidelines and related material on the Information Quality Act in ADS 578 and at http://www.usaid.gov/about_usaid/. RELATIONSHIPS AND RESPONSIBILITIES GH Tech will coordinate and manage the evaluation team and will undertake the following specific responsibilities throughout the assignment:  Recruit and hire the evaluation team  Make logistical arrangements for the consultants, including travel and transportation, country travel clearance, lodging, and communications. USAID will provide overall technical leadership and direction for the evaluation team throughout the assignment and will provide assistance with the following tasks:  SOW. Respond to queries about the SOW and/or the assignment at large.  Consultant Conflict of Interest (COI). To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CVs for proposed consultants and provide additional information regarding potential COI with the project contractors evaluated/assessed, and information regarding their affiliates.  Documents. Identify and prioritize background materials for the consultants and provide them to GH Tech, preferably in electronic form, at least one week prior to the inception of the assignment. CONTACT PERSON Neal Brandes  Health Specialist  GH/HIDN  nbrandes@usaid.gov  202-712-4122 Collene Lawhorn  AAAS Fellow  Gh/HIDN  clawhorn@usaid.gov Allysin Moran  Evaluation Technical Advisor  GH/HIDN  amoran@usaid.gov COST ESTIMATE GH Tech will provide a cost estimate for this activity. 38 ANNEX 2. CASE STUDY- CHLORHEXIDINE FOR NEWBORN CORD CARE Each year 3 million newborns die globally, and infection causes approximately 13% of these deaths. (Liu et al. 2012). In the last decade, research done under HaRP, other USAID projects, and partners has provided evidence of community-based effectiveness of the case management of newborn sepsis and for the use of chlorhexidine for cord care to prevent infection (Nepal, 2005; Bangladesh, 2010; Pakistan, 2012). Outcome and programmatic implications for these interventions are well described in the Technical Brief prepared for the Chlorhexidine Working Group. (PATH January 2014) HaRP’s history of CHX for newborn cord care is summarized in Exhibit 6 and began in 2001 with a randomized efficacy trial done by JHU/NNIPS in Nepal. In 2005, this trial demonstrated a 75% decrease in cord infections and a 34% decrease in NMR, at which time USAID/HaRP began convening the Chlorhexidine Working Group (CHX WG) to identify opportunities for coordinated research on product development and introduction. This was followed by a period of concurrent research streams from 2005 to 2010 that included effectiveness trial replication in Bangladesh and Pakistan, operational studies in Nepal and Bangladesh, and product development in Nepal. Nepal set national policy and standards in 2011, and the UNCoLSC selected CHX as 1 of 13 commodities for support, and their CHX Working Group was initiated in 2012. In 2013, WHO placed CHX for cord care on the Essential Medicines List and promulgated use guidelines in early 2014. Exhibit 6: HaRP Conceptual Framework Applied to Chlorhexidine This case study explores how HaRP’s strategy of research-to-use was applied to accelerate and scale use of CHX to save newborn lives. (See Exhibit 6: HaRP Conceptual Framework Applied to Chlorhexidine) HaRP has made investments and provided technical leadership along all four levels of ‘research-to-use,’ including priority setting, product development, introduction, and field implementation. CHX for cord care is one of four interventions that HaRP has helped bring all the way to the scale-up phase. 39 Exhibit 7: Research-to-Use Timeline for Chlorhexidine for Cord Care RESEARCH TO USE: WHAT END RESULTS HAVE BEEN ACHIEVED? As of 2014, CHX for cord care is available to populations in Nepal (45 out of 75 districts), Madagascar (1 to 2 out of 112 districts), Nigeria (Sokoto, Bauchi States), and Liberia (facilities in 6 counties). Three other countries are planning for pilot introduction, 6 are working on aligning policies and guidelines, and 3 are conducting initial stakeholder meetings for a total of 16 countries with intent to scale use. This was accomplished over a period of 10 years after efficacy was established. HaRP was credited with directly contributing to concurrent feasibility and operations testing during effectiveness trials and to attracting global interest in supporting country adoption of CHX. “I was impressed with how quickly the CHX, and sepsis work now, have gone, in terms of the research findings and going to actual usage in countries.... it is about bringing the right people together, and having the publications, and the technical consultations and WHO-type of meetings to move it along and get it established as policy. … CHX could have just as easily NOT been one of the “life-saving commodities,” and the uptake would have been different in countries.” [Donor-Global] PRIORITY SETTING: THE CHOICE OF CHLORHEXIDINE FOR CORD CARE From a newborn health perspective, HaRP’s choice of research priorities was viewed as being on target for MNCH impact. However, the process could be more effective with greater field input and more diverse participation. Prevention and management of newborn infections was universally identified as a critical area for reducing NMR. HaRP’s support to community-based trials of case management of sepsis and CHX directly fit with disease burden and prevailing understanding of interventions that would save lives. 40 However, country stakeholders clearly stated that ideas for research came from the top down via projects or funding mechanisms or both. To be more effective and further accelerate timelines, country researchers and implementers need to have a bigger voice in priority setting. In addition, some respondents believe that cutting-edge and ‘fresh’ ideas will need different approaches than the consultations used most frequently by HaRP. Uptake and scale-up of CHX might have been accelerated if intervention characteristics, fit with delivery platforms and need for policy/guidelines change were more specifically considered at the priority setting stage. CHX for cord care is often called a simple intervention, especially compared to case management of newborn sepsis. However, adoption of the intervention required reversing the widely known, WHO￾recommended guideline of dry cord care. Consequently, resistance by professional bodies, governments and practitioners to adopting CHX was underestimated at the onset and in some countries, such as Bangladesh, set back progress by as much as two years. Effectiveness trials for CHX for cord care have been done mainly in community settings with high levels of home births where infection and neonatal deaths are common. As there have been rapid shifts from home to facility births in some countries, the lack of evidence of effectiveness for facility settings, reflected in the current WHO guideline, has made policy choices messy. There was lack of foresight or scanning for contextual changes or both on the part of researchers and HaRP that might have allowed more adaptation of intervention studies so that findings could remain relevant. The safety of CHX is well accepted and this made it easier to move toward rapid adoption as opposed to antibiotic treatment for newborns that can have adverse effects and requires more complex regulatory approvals and system changes such as authorizing frontline health workers to diagnose and treat illness. During priority setting, more attention needs to be paid to the complexity and requirements of introducing and integrating these interventions into existing systems at scale. “There we made the comparison between sepsis ... never ever any doubt, baby got sick, baby had to be treated. If get treatment close it worked. Never a debate about treating a sick child. CHX is very different – dry cord care vs. something on the cord. Part of the mindset was leave cord alone!” [Implementing Partner - Country] “Even at that stage identify, down the road, where we are going to integrate this into a service delivery package. Keep in mind what is practical use of whatever it is that we are testing.” [International Organization] Most stakeholders recommend that USAID funds more implementation or delivery research for MNCH, especially in introduction and scale-up phases. Most stakeholders overwhelmingly recommend more emphasis on implementation science and applied research to address system barriers and constraints to large scale, sustained use of CHX and other interventions. Some highlight the need to ensure meaningful linkages between efficacy/effectiveness trials and implementation research. “More important to support programs to do a better job in terms of implementation and integration. [Work in] Tanzania is an important example. Think RCTs are easier [while this is] more complex, more real life. You know you are going to get a result in RCT – in implementation -- many more partners involved, MOH’s don’t move. Much more difficult to show the results.” [Implementing Partner-Global] 41 “Don’t delink the two! When people talk about implementation science they lose linkage to effectiveness trials. In a very broad way, we need it but not by itself.” [International Organization] DEVELOPMENT AND INTRODUCTION OF CHX FOR CORD CARE: WHAT WAS LEARNED? Support of product development and operational studies for CHX done in parallel with effectiveness studies accelerated research-to-use, even though it might have gone faster in some countries. In Nepal, a fortuitous combination of organizations, projects, and individuals with a vision acted immediately on results from the first CHX study and initiated product development and implementation design work. This occurred in parallel with CHX trials in Bangladesh and Pakistan starting in 2007, and resulted in CHX for cord care being implemented in 45 of Nepal’s 75 districts today. (In contrast, efficacy to implementation at national scale for vitamin A supplementation in Nepal took 11 years.) The key operational activities that were funded in Nepal included: formative study of cord care/behaviors; consumer product preference studies; non-inferiority trial of gel vs. aqueous solution; gel product specifications; USP-provided technical assistance for manufacturing; and district pilot of product/service delivery. While funding for the Nepal development work came from multiple sources, HaRP championed the efforts, smoothed USAID/Washington support, convened experts, and provided for some key, harder to fund activities. The USAID Mission provided strong support and was a respected working partner in the country process. HaRP’s support to the CHX effectiveness trial in Bangladesh was critical to replicating the evidence and the trial was designed to be more rapidly adapted into the existing health system by including arms that tested both one- and seven-day use in newborns. In 2009, after years of policies for dry cord care, confusing results from these arms (one day-use reduced NMR, seven-day use didn’t) and missteps in engaging the commercial sector, a strong resistance to CHX adoption was created among professional bodies and the MOHFW. It was only very recently (2013) that CHX was adopted into policy, guidelines, and operational plans as one of four key newborn interventions in Bangladesh. This was achieved by more nuanced local advocacy and debate, and south-to-south exchange in technical conferences. The exchange activities were most often supported by projects other than HaRP, including MCHIP and Saving Newborn Lives. “[In Nepal] pretty much as soon as results … were coming out, USAID was pushing immediately into OR and thinking about user preferences around formulations, about local manufacturing, about MOH engagement in scaling-up. Made it very clear that was USAID’s priority – accelerate adoption and do a little bit of learning but then to drive it into implementation and programs and that was good.” [Implementing Partner-Global] “And during that time of that study [gel vs. liquid] they were engaging in operations work to understand how to support pilot testing of programs... Putting the whole community of researchers, policy makers, and donors in terms of being ready to answer those questions. …. shortened the time line.” [Collaborating Partner￾Global] “[In Bangladesh], recognition from HaRP team and from SNL was very helpful to rapidly move from a single study into the next big replicate trial. Neal was very committed to making sure that the questions they were asking were supplemented – (efficacy questions) with more operational questions.” [Collaborating Partner￾Global] 42 “But equipoise which is needed to maintain research was undermined by the effort with product development. At the stakeholder meeting – presence of a pharmaceutical company…mistake to bring it in at that pivotal point. Set back effort by 2 years.” [Collaborating Partner-Global] Engagement of the commercial sector was critical to country adoption of CHX for cord care, but approaches need to be based on greater understanding of the value for companies, markets, and the importance of trust between sectors. Too little engagement of the commercial sector was a common observation in interviews for newborn interventions. Respondents identified the need for a marketing perspective from the outset, deeper understanding of the requirements for ensuring product availability and successful use (including regulation), and a more hard-nosed understanding of changing provider and consumer behaviors at scale. At the same time, the commercial sector is viewed with suspicion because of real and potential conflicts of interest. For CHX, the first gel product was developed in Nepal building on existing USAID-funded projects and Lomus (Nepali company) was treated as a valued partner in the development process. The relationship with the nascent newborn program was clearly defined, accountabilities were specified, and expectations communicated, including future procurement requirements. The process also excluded the company from activities that might be perceived as conflict of interest, such as in early policy discussions with the government. At the same time, there was value brought to the company to develop a new product, to improve quality production and, more recently, to supply product and/or technical assistance to companies in other countries (Nigeria, Ethiopia). In Nepal, success was also based on trust between public and private actors, something difficult to leverage without existing local relationships (as happened in Bangladesh). Globally, USAID’s expectations of the role and attitude of the commercial sector may have made it difficult to work more collaboratively locally. More recently, the UNCoLSC CHX Working Group has provided a useful global venue to engage the commercial sector. “[In Nepal], involvement of private sector was very important. They were engaged from beginning – what to expect, not to expect. If you are willing to provide CHX at no cost or at a cost –…they agreed to do it.” [USAID-Country] “He had already built a lot of trust with the company and they were interested in partnership – products development, manufacturing capacity….Very nice combination of building on existing work and working on private sector side.” [Implementing Partner-Country] “Lomus was never involved in official discussions with the GON…We didn’t want there to be a perception of influence.” [Implementing Partner-Country] HaRP’s leadership, convening power, and funding support of activities that helped harmonize research and disseminate learning have contributed to faster country uptake of CHX and simplified antibiotic regimens for treating newborn sepsis. HaRP, particularly Neal Brandes, is respected for its persistent and modest advocacy for operational research and for its follow-through on activities and decisions regarding newborn health. This has been especially true for CHX and simplified antibiotic treatment of newborn sepsis (SAT). 43 HaRP made good use of invitational meetings to build consensus on evidence and an understanding of implementation implications for these interventions. This was very successful for SAT where researchers harmonized studies early and worked closely together over time. Initial meetings for CHX were not as inclusive and were less practical for coordination, but they did enable sharing of experience. HaRP also supported regional and global technical conferences that enabled south to south exchange of learning and prodded policy and program uptake. “Overall approach was reflected in the success of the 2007 London newborn sepsis consultation – co￾convened by USAID, WHO, SNL…. USAID took a strong position of trying to move the process as quickly as possible…not to just do it as a complete linear sequence. They asked, to make this an expedited process, - what could really be done so that certain things were in parallel? Look at some of the steps that might be truncated or built into others so there is overlap. That was ultimately a good approach. We saw it work.” [Collaborating Partner-Global] “Same strategy that worked well for sepsis didn’t work as well for CHX. … with sepsis priority setting, problem identification was this London meeting with WHO playing a critical leading role. Researchers were on the same page - so we were not having three different studies in three different countries with similar, not same protocols, conclusions show this on one hand, not on the other. That’s when you have policy makers fumbling around trying to make sense of evidence. Researchers will then say they need another study to make a policy recommendation – this takes even more time.” [Implementing Partner-Global] “At every meeting for SAT (including USAID) – [they asked] how will this process culminate in impact? They were better prepared – what information is required – is the information correctly collected, they have moved very fast. At the same time are the papers published – within 3 months a WHO guideline. Partly it will be because we’ve taken initiative.” [International Organization] USAID’s changing priorities, internal structure and working culture have sometimes complicated HaRP’s progress toward objectives. For newborn health, this is seen most acutely between country missions and central offices. The extent to which HaRP can influence country uptake of interventions is helped or hindered by USAID Missions. Missions own working relationships with governments and Ministries of Health, manage more funding, and collaborate more closely with country-based donors and implementing partners. In Nepal and Bangladesh, they made significant investments in moving newborn care to scale, but looked to HaRP to fund effectiveness trials. Much less was done in other countries especially in Sub-Saharan Africa where Missions had different priorities. Several respondents noted that Missions do not effectively participate in setting research agendas with centrally convened groups, yet there are expectations that they will ‘pick up’ activities where HaRP and others leave off. Missions must be more meaningfully involved earlier on if the aim is to achieve more seamless handoffs. Other USAID centrally funded initiatives or projects work at various stages of the research-to-use process but there is lack of clarity on their relative roles and few practical means to work together. For newborn health, the most important of these are the Center for Accelerating Innovation and Impact (CAII) that appears to enjoy higher leadership visibility and autonomy, and the MCHIP Project that is intended to spread successful MNH interventions, but is driven more by Mission requests and funding. Finally, while maternal and newborn offices collaborate well centrally within USAID, respondents report a growing disconnect between maternal and newborn work in the field. This gap will increasingly impede progress to reduce newborn mortality because complications from pre-term birth and stillbirth emerge as the most important causes of death. 44 “So they would need to have a much broader range of technical expertise that sees itself as working in an integrative way in a team that’s problem solving and not just each person working towards his or her area. For instance, MCHIP [staff]…. occupy same office, but not a team. CTOS in MCHIP so many. Pieces within USAID aren’t talking to themselves so they are only able to give a complex answer to a complex question.” [USAID-Global] “To do that you need real ownership by Missions and you need money – were substantially missing for zinc and CHX. Missions – take it or leave it – other priorities. So if USAID is going to be a motor for research to implementation, the way USAID fragments its programming is an obstacle. In a sense this is a problem – USAID programs by project not by program – that’s the limiting factor.” [USAID-Global] “Don’t visualize it as a hand off – more like a hand waving… The way you get a handoff – is like PMI. Have a bucket brigade but organized to move things along – different than here you take it.” [USAID-Global] FIELD IMPLEMENTATION AND SCALE-UP OF CHX: WHAT WAS LEARNED? As CHX has moved into larger scale implementation, the contributions of other USAID-funded projects are not easily distinguished from HaRP. Other USAID funding included country bilateral projects, MCHIP, and global entities that receive some USAID support such as the UNCoLSC CWG. Therefore, HaRP’s ultimate goal of national use of interventions relies partly on the parallel work described earlier but also on effective leveraging and collaboration with those supporting service delivery. “USAID/HaRP is not well positioned to deal with scale-up – resources too limited. Need to engage others and work with their own networks (like MCHIP) to take the science forward.” (USAID-Global) As the 16 target countries for CHX move toward reaching and sustaining national scale coverage, gaps in scale-up plans and support remain. It is not very clear how they will be addressed. In HaRP, most attention was focused on conducting activities that would build program design information and buy-in, in parallel with development and early introduction work. At these early stages, HaRP tended to pay too little attention to regulatory requirements that are now significant bottlenecks in many countries, to the strength and resilience of the platform into which the intervention is to be integrated, and to the mobilization of partners that would be needed to support roll out. While HaRP might have done more to enable earlier planning, this partly comes from the confusion over where HaRP’s direct role ends and how and to whom the implementation baton is to be passed. “Introduction is introduction, needs something next – needs formalized planned process, multiple partners, not separate. Evaluations. Sharing information to course correct. Have to invest in systematic processes at the downstream level of implementation.” [Collaborating Partner-Global] “People anticipated people will ask questions about scale-up … what can we do during the interim … how to get a population engaged – how do you think about the options – what will people use on the cord, how would they incorporate new advice into understanding of previous messages – HOW it might be done but perhaps not as much focused on the politics and funding requirements for scale-up.” (Implementing Partner, Country) 45 “One of the things we missed and didn’t do very well is managing scale-up of program. It went too fast. Focus should be to make sure that program which is currently being done (geographically) – should be fully functional.” [Implementing Partner-Country] Lack of useful knowledge management systems is leading to a loss of learning about the key elements of successful implementation and scale-up. As countries and projects develop these interventions, they are country-focused, tend to be funded by multiple sources and in some ways are ‘let go’ by HaRP. The sharing of learning that HaRP has established among researchers and during development does not carry over into implementation and scale-up and vice versa. With CHX, there has been some exchange in the context of the UNCoLSC CHX Working Group and, intermittently, in global conferences, but systems are not in place for rapid, practical learning around scaling-up, maintaining quality, or integrating it within primary care programs. If HaRP chooses to move more into implementation research, this will be a critical area for improvement. “Need funding for evaluation! So much learning in MCHIP but I fear it will be lost because not properly harvested, pulled back around and used to improve programs – even in same country with same partners. Especially across partners and countries. Knowledge capture and management is one of the weakest areas in the whole newborn movement in which USAID has been a leader.” (Collaborating Partner, Global) We lack knowledge of the ultimate impact of CHX for cord care because robust systems to measure, track and report scale-up and sustained outcomes are not in place. In general, monitoring and evaluation of large-scale delivery programs is weak, inconsistent, and poorly supported. Definitions, standards, and efficient methods for program data collection are lacking and resources are insufficient to better understand what happens at a population level when an intervention such as CHX is scaled. In Nepal where they had access to this type of resource through HaRP during the transition from efficacy to piloting, they were able to more systematically learn from early experience, thus accelerating progress. “If the end of the process isn’t impact – if you have adoption and roll-out but all you’ve accomplished is low coverage – not a worthwhile process. The last third base to home plate – assuring high coverage. Not sufficient attention. Need local capacity for this – more robust in-country capacity.” [Collaborating partner￾Global] RESEARCH TO USE: WHAT IS HARP’S ROLE? The development of CHX for cord care highlights HaRP’s role as a convener, coordinator, funder and advocate in the research-to-use process. Most of HaRP’s effort has focused on Priority Setting, Development and Introduction phases in ‘early adopter’ countries. As Introduction spreads across countries and especially with field implementation and scale-up, country activities become more prominent and HaRP’s direct role diminishes with respect to other stakeholders, including governments. It is clear that HaRP does not have the resources or mandate to provide for national scale-up of interventions such as CHX. There is an understanding that there will be a ‘hand off’ to others, better positioned to address those challenges. However, respondents felt that HaRP’s ultimate aim is sustained coverage and mortality reduction. HaRP might consider more active leveraging of external and internal partners and more explicit use of USAID mechanisms to strengthen feed forward and feedback loops to more tightly couple field implementation and results within the research-to-use strategy. 46 REFERENCES Arifeen et al. 2012. The effect of cord cleansing with chlorhexidine on neonatal mortality in rural Bangladesh: A community-based, cluster-randomised trial. Lancet 379(9820):1022–1028. Chlorhexidine Working Group January 2014. Country Guidance for Umbilical Cord Care. Liu et al. 2012. Child Health Epidemiology Reference Group of WHO and UNICEF. Global, regional, and national causes of child mortality: An updated systematic analysis for 2010 with time trends since 2000. Lancet 379(9832):2151–2161. Mullany et al. 2006. Topical applications of chlorhexidine to the umbilical cord for prevention of omphalitis and neonatal mortality in southern Nepal: A community-based, cluster-randomised trial. Lancet 367(9514):910–918. Mullany, Darmstadt, Tielsch. 2006. Safety and impact of chlorhexidine antisepsis interventions for improving neonatal health in developing countries. Pediatric Infectious Disease Journal. 25(8):665–675. PATH for the Chlorhexidine Working Group January 2014. Technical Brief: Chlorhexidine for Umbilical Cord Care. Segre et al. February 2012. Chlorhexidine for Umbilical Cord Care. A case study prepare for the UN Commission on Life -Saving Commodities for Women and Children. Soofi et al. 2012. Topical application of chlorhexidine to neonatal umbilical cords for prevention of omphalitis and neonatal mortality in a rural district of Pakistan: a community-based, cluster-randomised trial. Lancet. 379(9820):1029–1036. World Health Organization 2013. WHO Recommendations on Postnatal Care of the Mother and Newborn. 47 ANNEX 3. CASE STUDY- RESPECTFUL MATERNITY CARE Respectful maternity care (RMC) is a favored terminology for the construct of provider/client (patient) interpersonal relationships that occur across the continuum of the perinatal care timeframe. The construct emphasizes the importance of underlying professional ethics and psycho/social/cultural aspects of health care delivery as essential elements of the care. The contemporary emphasis emerged from several predecessor larger-scale movements, such as the humanization of birth focus in Latin/South America and the Caribbean (Page 2001), and is demonstrated in a number of smaller-scale activities, such as the Model Maternity Initiative in Mozambique (Jhpiego 2009). The terminology itself reflects an amalgam of constructs, including disrespect and abuse (D&A) in childbirth (USAID 2010), respectful and dignified care, and respectful maternal care (implying care across the reproductive lifetime i.e., beyond the boundaries of the childbearing period). RMC, while fundamentally grounded on human rights principles, has received particular attention because of wider-scale inquiries about global strategies to reduce maternity morbidity and mortality, which have included initiatives to increase the proportion of facility-based births, and births that are supported by skilled birth attendants. The high number of maternal deaths in large part reflects inequities in access to health services. However, evidence shows that women who do have access, may choose not to use those services because of their perceptions of the environment and quality of care received in facilities. Browser and Hill (2010) described 7 categories of D&A3 in childbirth. PRIORITY SETTING USAID HaRP conducted internal priority review activities, and consulted partners to determine niche and comparative advantage in line with the global focus on strengthening health systems and the promotion of interventions that focus on policies and strategies that work to improve maternal health in pregnancy and childbirth; and also in line with the emphasis on maternal mortality as a human rights and equity issue. HaRP subsequently selected RMC for inclusion in its research portfolio, although it might have been perceived as an outlier. “USAID paid a very important role in bringing this topic up for attention. These are system issues that have common cores across health systems and countries.” [International Organization]. “Chlorhexidine [CHX] and RMC are quite different issues because of the nature of the intervention. RMC was based on early work…but very little published literature; needed to start at earliest level; contrary to CHX where [there was] well publicized evidence of an emerging body of [research].” [Collaborating Partner] The topic was taken up by the Translating Research Into Action (TRAction) project, a 5-year cooperative agreement awarded in 2009 by USAID to University Research Co, LLC; Center for Health Services (URC-CHS). URC and the TRAction project use a small-grants mechanism to fund implementation research to develop, test, and compare approaches to more effectively deliver health interventions, increase utilization, achieve coverage, and scale-up evidence-based interventions for priority health problems. The timeline for activities related to RMC is depicted in the exhibit that accompanies this case study. 3 Physical abuse; non-consented care; non-dignified care; discrimination; abandonment/neglect; detention; non￾confidential care. 48 A consultative meeting of researchers and implementing agencies was convened to discuss new activities in RMC, and to assess program needs. This was followed by discussions with a wider group of topic stakeholders, including civil society, advocacy, research and community representatives, during the 2010 Women Deliver meeting (Washington, D.C.). Participants affirmed the need for attention to the topic of RMC. TRAction responded by commissioning the landscape analysis that reviewed the evidence found in published and gray literature with regard to the definition, scope, contributors, and impact of disrespect and abuse in childbirth,…promising intervention approaches…and gaps in the evidence.4 A first RFA was issued in September 2010 and subsequently amended, proposals were accepted in October 2010, and funded in January 2011. HaRP is acknowledged and applauded for its role in bringing attention to this topic. “If HaRP had not funded this RMC focus then…entire research focus would likely have been lost. No one else was interested in placing a priority on this topic.” [Implementing Partner] PRODUCT/STRATEGY DEVELOPMENT Columbia University’s Averting Maternal Death and Disability Program received funding for The Staha Project in Tanzania. Population Council received funding for the Heshima Project in Kenya. Both projects received $600,000 for a two-year project timeline, with an institutional co-funding requirement. An element common to both proposals was the plan for a phase in which the parameters of D&A would be defined, the prevalence documented, and the community consulted about the design and selection of interventions that would be implemented and tested. A post-award meeting was held in March 2011 to advance the vision articulated in the RFA and to promote the successful launch of the studies. Objectives of the meeting included the effort to harmonize research designs, and to identify opportunities for coordination and sharing between the two studies. These discussions identified the need for the two projects to have joint objectives, joint core indicators, and complementary research methodologies. A corollary effort was to generate working definitions of disrespect and abuse in childbirth. The Heshima Project (Population Council/Kenya) The Heshima project was designed as a quasi￾experimental, pre-post research project, located at 12 health facilities in three districts and one large maternity hospital in Nairobi (the 4th district) of Kenya. Project partners were the Kenya Federation of Women Lawyers (FIDA) and the National Nurses Association of Kenya – Midwives’ Chapter (NNAK￾MC), in collaboration with the Kenya Ministry of Health. Project partners conducted focus group discussions with community members (women who had facility based birth and those who delivered at home; family members; community health committees) and with local women’s and civil rights groups to gather their ideas about activities that could be conducted to promote women’s rights during facility-based childbirth. Project partners also conducted in-depth interviews and held informal discussions with senior health managers about this topic. Health facility assessments and direct observations of client-provider interactions during labor and childbirth were conducted (Warren et al. 2013). The interventions designed after these consultation activities were implemented at three levels. Policy level interventions included adapting clarification of values and 4 All citations are drawn from USAID internal documents provided to the USAID HaRP evaluation team, or from interviews with a member of the designated stakeholder group. 49 attitude training modules, promoting dignified and respectful facility-based childbirth, and advocacy for improving governance and accountability in implementation of guidelines and legal and health-related policies. The facility level intervention was developing of a training manual with three modules addressing policy, health facility and community. The policy and facility modules focused on clarification of values and attitude change, using a health rights approach. The community module focused on health rights and law, and the use of mediation as a mechanism for empowering communities to claim their rights for respectful and dignified childbirth. The Heshima project was stalled in its early phases, primarily because of the perception that the research might uncover or document personal behaviors that could lead to professional retribution. This perception was largely fueled by Population Council’s choice of partners, who, by intention, represented advocacy for women’s and provider’s rights. However, they were perceived by some to represent the potential for legal sanction (the lawyers) or license revocation (the regulatory council). The project was also constrained by the unwillingness of many providers to openly acknowledge the fact that D&A was a real issue in their facility. “Project principals failed to anticipate how complex the issue is and how to figure out what to do.” [Implementing Partner-Global]. “They failed to anticipate how long it would take to open the discussion at DHMT and facility manager levels; because that would be having to admit to a problem.” [Implementing Partner-Country] “One respondent noted that the topic made me uncomfortable because it might get the midwives into trouble; because the lawyers were also involved.” [Implementing Partner-Country] Project principals and their counterparts had to spend a good deal of time introducing the project and its intentional outcomes to the district health management team, health facility mangers, and the providers; and in engaging the support of professional associations of doctors, nurses, and midwives. “Principal Investigators did not really appreciate the sensitivity of the topic, and tried to introduce it into districts where there was not a familiar presence of already trusted implementers.” [Collaborating Partner] “Not enough work done with professional associations and women’s advocacy groups to inform them of the intention of the work. Lost too much time making reassurances.” [Collaborating Partner] “Failed to anticipate the challenge of the topic – the benefit that might have come from imbedding this research into MCHIP networks (or similar) where implementation work is already well-known and respected.” [Collaborating partner] The Staha project (AMDD, Tanzania) partnered with the Ifakara Health Institute and located its project in the Tanga region of Tanzania. The project was designed as an exploratory study of the types and prevalence of D&A among women giving birth in public facilities, an exploration of the root causes of disrespectful treatment, and testing of approaches to reduce these behaviors. Project partners conducted exit interviews in one hospital and three health centers in each of two districts, speaking with all women discharged after delivery and conducting focus group discussions among women delivered in facilities and those who chose to deliver at home, to identify root causes and to brainstorm solutions. In-depth discussions were also conducted among community leaders and facility managers. The project interventions and exemplary activities include: A) fora for change (facility-based morning meetings; establishing of village committees; meetings between District and facility management 50 to discuss shortages of tangible and human resources); B) information (a patient reporting call line, exit interviews with clients); C) standards (developing of a client charter for patient and health worker expectations and rights): D) activities (such as health worker recognition; shift change planning meetings): and E) promotion of champions to keep this topic open for discussion. However, these interventions came late in the project timeline (see Exhibit 8), and were implemented only recently. Project principals “seemed unable to move forward with an intervention and to move this process to the next step effectively…Their results and approaches seemed less than adequate to give us a clear way forward in country and did not include the larger audience – including MOHSW adequately.” [USAID] Exhibit 8: Timeline for Respectful Maternity Care “Globally and even nationally it is incomplete and did not seem to have moved forward fast enough to provide us with our next steps. Results are very provider-oriented while we are trying to address community needs.” [USAID] “A key issue in Tanzania is that this is an incredibly sensitive topic, open to lots of policy resistance and political backlash…There is very little civil society infrastructure…can’t say there is outright “naysayer”, but a lot of work goes into getting folk open to talking about the problem.” [Implementing Partner] INTRODUCTION OF INTERVENTION Progress to Date. The projects are currently analyzing follow-up data and have not generated a report of final findings. However, the leadership of both projects is already assessing what future adaptation or scale-up or both might be done in the respective countries, and the potential for wider-scale global adoption. One respondent commented that he had heard “that other NGOs and reps from other countries (Ethiopia, Nigeria) have expressed an interest in taking up this work because they heard about it in so many global meetings.” [Implementing Partner] “The intervention is so very limited [in Kenya] (17 facilities) … that any scale-up in country will take a good deal of time. The research process was very meticulous in getting community involved, and that takes a good deal of time.” [Implementing Partner] 51 “Kenya’s efforts seemed more comprehensive and strategic to address [D&A].The Tanzania program did not really go beyond working within health facilities and immediate providers/managers.” [USAID-Country]. At the same time, valuable definitions, tools and measurement indicators for RMC have broad utility and potential for use in other projects. They have been widely disseminated in diverse local, regional and global forums. A dissemination workshop in Fall 2013 initiated “next step” efforts for research on the topic based on lessons learned about prevalence, definition and measurement challenges, and effective intervention approaches. An Advisory Council for the RMC topic was established under the leadership of White Ribbon Alliance (WRA)5. Initial meetings of the council included a general discussion about an effective framework for advocacy on respectful care at birth. WRA’s advocacy movement over time included numerous activities related to definition of the topic and strategies to address it. Examples of advocacy work include development of an RMC charter: Respectful Maternity care: The Universal Rights of Childbearing Women (2011). The charter delineates seven rights of childbearing women that directly correspond to the seven categories of disrespect and abuse defined in the landscape analysis. “Break the Silence”, a video about the issue of D&A was completed and launched in February 2012. Presentations of findings about the prevalence of the problem of D&A, and preliminary data about effectiveness of intervention strategies have been presented at numerous regional meetings and country/global conferences. Principle investigators of the research projects and USAID regularly attended Advisory Council meetings along the course of the project timeline. Leaders of the TRAction-funded projects, the WRA and USAID are partners in current activities intended to align the findings of the implementation research activities with targeted advocacy initiatives that build on lessons learned. However, WRA reports that it is presently “not aware of any scale-up – still awaiting first results from both research projects, so we don’t know our next steps. Still advocacy doesn’t wait for final data.” REFERENCES Browser, D, and Hill, K. 2010. Exploring evidence and Action for Respectful Care at Birth. Report of a landscape analysis. USAID TRAction Project. Available at: http://tractionproject.org/sites/default/files/Respectful_Care_at_Birth_9-20-101_Final.pdf Jhpiego. 2009. 40 Greatest Hits: Advancing a global movement for respectful maternity care. Available at: http://www.jhpiego.org/content/40-greatest-hits-advancing-global-movement-respectful-maternity-care McConville, B. 2013. Respectful maternity care – How the UK is learning from the developing world. Midwifery. In press DOI dx.doi.org/10.1016/j.midw.2013.12.002 Page, L. 2001. The humanization of birth. Int J Gynecology & Obstetrics. 75;S55-S58. Reis et al. 2012. Respectful maternity care. Country experiences. Available at: http://www.k4health.org/sites/default/files/RMC%20Survey%20Report_0.pdf 5 USAID provided funds to WRA, the Health Policy Project (Futures Group) and to Harvard University for various aspects of RMC advocacy work. The Gates-funded Maternal Health Thematic Fund also contributes to this work. 52 Warren et al. 2013. Study protocol for promoting respectful maternity care initiative to assess, measure and design interventions to reduce disrespect and abuse during childbirth in Kenya. BMC Pregnancy & Childbirth. 13:21 53 ANNEX 4. TABLE OF INTERVIEWS CONDUCTED Type of Respondent US Based International TOTAL Global level Country level Collaborating Partner 6 3 6 15 Donor 2 1 -- 3 Implementing Partner 19 7 7 33 MOH -- -- 2 2 USAID/USG 10 -- 2 12 TOTAL 37 11 17 65 54 ANNEX 5. INTERVIEW GUIDE-CASE STUDIES Highlighted numbers represent relevant evaluation questions Date: Number/Affiliation of Respondent: Interviewer: Introduction Thank you very much for setting aside time to talk with me/us today. I am part of a team evaluating the USAID Health Research Program (HaRP)’s research-to-use process, focused on maternal and child health. We understand that you and/or your organization have played a role in developing, testing or spreading interventions or approaches relevant to this program. In particular, I would like to talk with you about newborn health broadly inclusive of prevention and treatment, but with special emphasis on chlorhexidine cord care or Respectful Maternity Care/Abusive maternal care. I would be grateful if you could spare about 45 minutes to one hour to assist with this evaluation by candidly answering a few questions that will help us understand the effectiveness of USAID’s research-to-use activity. This is not an evaluation of your organization or your work. Before we begin, I/we want to let you know that any information or examples we discuss during this interview will not be attributed to any specific person or institution. Any quotes used in the report will be attributed to a general stakeholder group (e.g., research partner, in-country stakeholder, etc.), and all identifying information will be removed. Please feel free to decline to respond to any of our questions or to stop the interview at any time. Before I/we begin, do you have any questions? BACKGROUND 1. What has been your organization’s and your role and experience with work on [NEWBORN HEALTH/CHX, RMC]? With the USAID research-to-use program? (II.A.3) 2. From a high level (20,000 feet), what has been achieved along the continuum of research-to-use for [NEWBORN HEALTH/CHX, RMC]? From a global perspective? At country level? (I.A.) 3. What has been USAID’s contribution to these achievements? (I.B.) Probe HaRP USAID HaRP’s Activities related to priority setting, intervention research, introduction, and field implementation Considering stages along the research-to-use continuum in more detail: 4. What can you tell me about the key activities or processes that were supported for [newborn health-CHX/RMC]? In what ways have USAID-supported activities enabled or impeded progress or results for [NEWBORN HEALTH/CHX, RMC]? (II.A.1; II.A.2) Probe: priority setting, product development, introduction, and field implementation. (see chart) Probe: How and why did they work? 55 5. Did USAID’s research-to-use process anticipate and support introduction and scale-up of [newborn sepsis/CHX, RMC]? How? a. At country levels? Probe: Country examples – Bangladesh, Nepal, Nigeria for NEWBORN HEALTH/CHX; Probe: Tanzania and more broadly about the increase in country-level introduction efforts, Kenya and global dialogue and attention for RMC. b. At the global level? Probe: Country introduction process and timeframe for CHX Probe: Structures c. What key issues did they fail to anticipate, if any? (II.A.3) USAID HaRP’s structures and partnerships supporting research-to-use 6. USAID HaRP structures its support and work for [NEWBORN HEALTH/CHX, RMC] through projects, partnerships, technical and policy working groups. In what ways do you feel this structure has facilitated accelerating the research-to-use process? In what ways do you feel it needs to be improved to accelerate research-to-use? (II.A.1; II.A.2) Probe for Why or why not? Probe: Can you suggest examples of more successful structures? RMC Probe: RMC compared with Humanization movement and/or other efforts targeted at improving quality and access to maternity services. 7. In line with the ultimate goal of accelerating uptake and scale-up of new interventions, to what degree have key partners and stakeholders been engaged along the process at country level and globally? How were they engaged? (II.A.6) Probe: What other partners or stakeholders should have been engaged? 8. In what ways have the structures or processes for [newborn health/CHX, RMC] enabled or hindered engagement of the key partners in accelerating the research-to￾use process? (II.A.5; II.A.6) Probe for right partners, right types of engagement, right times, why? 9. What other partners or collaborators should have been engaged to be more effective? (II.A.5; II.A.6) Looking Forward 10. If the USAID research-to-use program’s ultimate aim is to increase its effectiveness and accelerate large scale use of products, interventions, or approaches like [NEWBORN HEALTH/CHX,RMC] what would you suggest they do differently now? (II.B.2; II.B.3) Probe for differences in activities, in structures, in partners 11. In thinking about further accelerating the research-to-use process, what are key components/actions/structures needed (regardless of who is supporting them)? (II.B.2) Probe for omissions that should be considered when planning forward What would be the relative importance of: 56 a. Increasing the degree to which timely, program relevant effectiveness data are available to accelerate the research-to-use process? b. Developing more programmatically useful evidence on how to implement evidence-based interventions [implementation science]? c. Embedding or moving research closer to real time program and policy decision￾making? TRAction-specific probes on a) planned expansion and/or amendment of intervention; b) new partners and/or settings (replication); c) dissemination & publication; d) new funding d. Addressing more complex, difficult systems interventions/approaches such as respectful maternity care? 12. Summing up, if you were to reflect back overall, how effective was USAID’s program and why?: (I.A.1; I.A.2; I.A.3) Probes: Developing evidence Informing policy dialogue and evidence-based decision making at country and global levels Fostering the introduction of interventions into populations 13. Within the MNCH arena, what would have been lost if the HaRP program did not exist? Why? 14. Looking forward, what are the three most important things that USAID’s research-to￾use program should do differently to have greater impact on mothers and children? (II.B.2) 15. Would you advise who else would be an important respondent or interviewee for this exercise? Conclusion 16. Is there anything else you would like to say or add? 17. What questions do you have for me/us? THANK YOU VERY MUCH FOR YOUR TIME AND INSIGHTS. 57 ANNEX 6. INTERVIEW GUIDE-PARTNERS Highlighted numbers represent relevant evaluation questions Date: Number /Affiliation of Respondent: Interviewer: Introduction Thank you very much for setting aside time to talk with me/us today. I am part of a team evaluating the USAID Health Research Program (HaRP)’s research-to-use process, which is focused mainly on maternal, newborn, and child health. We understand that you and/or your organization have engaged with this program, somewhere along the research-to-use continuum, i.e. at one of several points from research priority setting through research and introduction into programs. I would be grateful if you could spare about an hour to assist with this evaluation by candidly answering a few questions that will help us understand the effectiveness of USAID’s support to the research-to-use process. This is not an evaluation of your organization or your work, but your comments will inform an assessment of this strategy. Before we begin, I want to let you know that any information or examples we discuss during this interview will not be attributed to any specific person or institution. Any quotes used in the report will be attributed to a general stakeholder group (e.g., research partner, in-country stakeholder, etc.), and all identifying information will be removed. Please feel free to decline to respond to any of our questions or to stop the interview at any time. Before I/we begin, do you have any questions? BACKGROUND 1. In what ways has your organization been engaged with USAID’s program to support the research-to-use process? (II.A.3) Probe for work in relationship to HaRP or engagement with one or more of the collaborating agency partners including- JHU/HRCI, URC/TRAction and its sub-awardees (we can provide a complete list), Path/Healthtech, JPHIEGO/Acceovate, WHO/MCA(CAH), as well UN Commodity Commission work related to maternal and newborn health, efforts involving PQI/USP, MCHIP, and Saving Newborn Lives that also involves the previously mentioned partners or alternatively mention, USAID related work including zinc, newborn sepsis/chlorhexidine, ARI, respectful maternity care implementation research, or other activities or the names of key staff at partner organizations or USAID on Zinc, CHX, iCCM, respectful maternity care, etc., 2. What has been your own role and experience with the USAID program to support the research-to-use process in MNCH? (II.A.3) Overall HaRP Strategy 3. How would you describe USAID’s research-to-use strategy? (background) Probe: What is its overall objective? What kinds of activities did it include? What is its main focus? 58 NOTE: if they are not able to articulate it, tell them that USAID is supporting a managed research-to-use process, with priority setting, product development or intervention research, product introduction or implementation research, and support in the transition to scale-up. USAID HaRP’s activities related to priority setting, intervention research, implementation research, and field implementation With what you know about the program and your own engagements with it, I would like to ask where you think HaRP has been strong, and where it could be improved. You can speak in general, or related to a specific intervention or service delivery approach, citing examples from your personal experience. 4. In what ways have USAID-supported activities [processes] enabled progress or results along the research-to-use continuum? (II.A.1; II.A.2) Probe: why did they work? 5. In what ways have USAID-supported activities [processes] along the research￾to-use continuum ever impeded or slowed down progress or were formulated improperly to lead to results? Alternatively, are there other actions you feel USAID HaRP could have taken, that might have better supported the research￾to-use process? (II.A.1; II.A.2) Probe: How? Why? USAID HaRP’s structures and partnerships to support research-to-use 6. Now thinking about how USAID’s research program structures its support and work, (II.A.1; II.A.2) a) In what ways do you feel this structure has facilitated acceleration of the research-to-use process? b) What have been the strengths and areas for improvement for the research-to-use process? Probe: Projects: HCDR, HCDI/TRAction (and sub-awardees), Accelovate/ JHPEIGO; WHO/MCA and RHR partnerships, technical/policy groups. Note same broad list from above 7. In line with the ultimate goal of accelerating uptake and scale-up of new interventions and approaches, who are the most critical partners in advancing the research-to-use process? (II.A.4) Probe for in-country end users, USAID internal or USG, research partners, and others… 8. Please tell me about one or more experiences [where you observed or were part of a process] of engagement of partners and stakeholders related to USAID HaRP’s efforts. Were the key partners and stakeholders engaged along the process? How were they engaged? What other partners or stakeholders should have been engaged? (II.A.4) Probe for right partners, and right types of engagement, why and how? In what ways has it helped or hindered progress? 9. In what ways have the structures, mechanisms, or processes utilized by USAID enabled or impeded engagement of key partners in accelerating the research-to￾use process? (II.A.5; II.A.6) 59 10.How would you describe the role of (1) USAID staff and (2) collaborating agency partners in advancing the research-to-use process? (II.B.2.d) Probe: Any details, differences with USAID central and mission roles, as well as co-funders/technical partners such as SAVING NEWBORN LIVES HaRP Effectiveness 11.From your perspective, how effective has USAID’s HaRP been in accelerating the research-to-use process, specifically related to: (I.A.1,2,3) a) Developing evidence of effectiveness of specific tools or interventions b) Informing policy development at international or country level. 12.What do you feel have been USAID HaRP’s main contributions to improving MNCH? How did these contributions come about? (I.B) Probe: Remind them of the various partners and initiatives if needed 13.In what ways do you feel that USAID’s research-to-use structure or its supported activities have fostered the introduction and scale-up of interventions or approaches at the country level? What else should they have done? (II.A.3) Probe: Country examples – Bangladesh, Nepal, Nigeria for CHX; Tanzania, Kenya for RMC, or others for other interventions… Looking Forward 14. In thinking about further accelerating the research-to-use process, what are key components/actions/structures needed (regardless of who is supporting them)? (II.B.2) Probe for omissions that should be considered when planning forward a) What would be the relative importance of increasing the degree to which timely, program relevant effectiveness data are available to accelerate the research-to-use process? b) What would be the relative importance of developing more programmatically useful evidence on how to implement evidence-based interventions [implementation science]? TRAaction-specific probes on a) planned expansion and/or amendment of intervention; b) new partners and/or settings (replication); c) dissemination and publication; d) new funding c) How much weight should be placed on addressing more complex, difficult systems interventions/approaches such as respectful maternity care? Please explain (Probe: using [CHX, RMC] type as an example. 15. If the USAID research-to-use program’s ultimate aim is to increase its effectiveness and accelerate large-scale use of products, interventions, or approaches like CHX, RMC or others, what are the three most important things they should do differently to have greater impact on mothers and children? (II.B.2; II.B.3) Probe for differences in activities, in structures, in partners 60 16.Within the MNCH arena, given various funders and implementers, if the USAID HaRP program did not exist, which organizations would undertake similar work and what research-to-use activities would have occurred? (II.B.2.e) Conclusion 17.Having nearly completed this interview, could you advise me on who else would be an important respondent or interviewee and why? 18. Is there anything else you would like to say or add? 19. What questions do you have for me/us? THANK YOU VERY MUCH FOR YOUR TIME AND INSIGHTS. 61 ANNEX 7. BIOGRAPHICAL SKETCHES OF EVALUATIONTEAM MEMBERS IRENE AKUA AGYEPONG University of Ghana School of Public Health, Department of Health Policy Planning and Management, P.O. Box LG 13 Legon, Accra, Ghana Current Position Seconded to the University of Ghana School of Public Health full time by the Ghana Health Service – October 1, 2012 to present Positions Held (Past)  Regional Director of Health Services, Ghana Health Service, Greater Accra Region, April 2004 – October 2012  Professor to the Prince Claus Chair in Development and Equity 2008 – 2010. University of Utrecht, The Netherlands  Part time faculty, University of Ghana School of Public Health  Field Supervisor, MPH program, University of Ghana School of Public Health  District Director of Health Services, Ghana Health Service, Dangme West district, February 1989 – March 2004  Head of Dangme West Health Research Centre, Ghana Health Service, 1992 – 2004  Medical Officer, Children’s block, Korle-Bu Teaching Hospital, August 1988 – February 1989  Medical Officer, Cape Coast Central Hospital, (Obstetrics and Gynaecology), August 1987 – July 1988  House Officer, Korle-Bu Teaching Hospital, Dept. of Surgery, February 1987 – July 1987  House Officer, Korle-Bu Teaching Hospital, Dept. of Internal Medicine, August 1986 – January 1987 Education  MBChB (1986) Bachelor of Medicine, Bachelor of Surgery. University of Ghana Medical School  MCommH (1991) Master of Community Health. Liverpool School of Tropical Medicine, University of Liverpool  Part I West African College of Physicians – Public Health (1992) West African College of Physicians  DRPH (2000) Doctor of Public Health. University of North Carolina at Chapel Hill, School of Public Health.  FGCP: Foundation Fellow Ghana College of Physicians and Surgeons 62 LYNNE MILLER FRANCO, MHS SCD Vice President, Technical Assistance and Evaluation, EnCompass, LLC Dr. Lynne Miller Franco, EnCompass Vice President of Technical Assistance and Evaluation, is an expert in research and evaluation of quality improvement, policy, organizational performance, and sustainability. Dr. Franco has led teams for evaluating USAID Bureau of Policy Planning and Learning’s Program Cycle, PEPFAR’s Caribbean Regional Program, Save the Children’s Saving Newborn Lives, the Gates Foundation’s Maternal Health Task Force, the African Tobacco Control Consortium, and the WHO Centre for Tobacco Control in Africa. In her 30-year career, she has held long-term positions in Benin, Malawi, and Mali, and worked throughout Africa, the Middle East, Eastern Europe and the Caucuses, and Latin America. Dr. Franco has authored peer review publications on quality improvement, institutionalization of quality assurance, health worker motivation and health reform, impact of community-based health insurance, evidence for programming for children affected by HIV and AIDS, and methods of quality assessment. In 2011, one of the evaluation reports she co-authored received the USAID Award for Excellence in Evaluation. Dr. Franco has a BA in Development Studies from University of California Berkeley, and a MHS in Health Planning and a ScD in International Health Systems from Johns Hopkins University School of Hygiene and Public Health. DR. JUDITH T. FULLERTON, PhD Judith Fullerton is a PhD-prepared nurse-midwife, presently working as an independent consultant in the fields of women’s reproductive health, outcomes assessment and evaluation research. Dr. Fullerton is qualified in psychometrics, with applications in test development, validation, and standard setting for health professions credentialing. She has substantial experience in the field of monitoring and evaluation in addition to over four decades of experience as a classroom and clinical educator of nursing, midwifery and medical students. She has published extensively in her areas of expertise. Dr. Fullerton is retired from the rank of Professor, University of California, San Diego, School of Medicine, from the rank of Professor with tenure, University of Texas at El Paso, College of Health Sciences, and from the position of Senior Technical Advisor for Monitoring & Evaluation for the non-profit organization PCI (previously: Project Concern International). 2008 – Current – Independent Consultant: Women’s reproductive health and program evaluation 2004 – 2008 – PCI- Senior Technical Advisor for Monitoring and Evaluation 1998 – 2003 – University of Texas at El Paso Professor (tenured) 1995 – 1999 – University of Texas Health Science Center, San Antonio, School of Nursing: Lillie Cranz Cullen Professor (tenured) and Associate Dean for the Graduate Nursing Program 1981 – 1995 – University of California, San Diego, Department of Family and Preventive Medicine: Director: Nursing Graduate Studies (progressive academic and administrative titles) 1966 – 1981 – Various Registered Nurse and nurse-midwifery clinical staff and education positions: Michigan, New Jersey, New York, Pennsylvania. F. GRAY HANDLEY, MSPH Associate Director for International Research Affairs; National Institute of Allergy and Infectious Diseases (NIAID); National Institutes of Health, Department of Health and Human Services 63 Mr. Handley coordinates and facilitates international research activities for NIAID, the NIH Institute with the largest international engagement. He has previously served as Health Attaché and HHS Regional Representative in southern Africa, at U.S. Embassy in Pretoria, South Africa; and as U.S. Science Attaché and HHS Representative in South Asia at U.S. Embassy in New Delhi, India. At other times during his career, he served as: Associate Director for Prevention Research and International Programs at the NIH Eunice Kennedy Shriver National Institute of Child Health and Human Development; Associate Director for International Relations at the NIH Fogarty International Center; and Global Public Health Advisor for the U.S. Department of State, Bureau for International Organizations, the World Health Organization, the U.S. Department of Defense, and the U.S. Office of Management and Budget. He received his master of science in public health degree at the University of North Carolina, Chapel Hill. DENIS J. PRAGER, PhD Denis J. Prager is president of Strategic Consulting Services, a private consulting practice established in 1994. From 1983 to 1994, he was Director of the Health Program at the John D. and Catherine T. MacArthur Foundation, responsible for the development and implementation of programs in mental health and human development, and in tropical disease research. From 1978 to 1983, Dr. Prager was Associate Director of the White House Office of Science and Technology Policy, responsible for the formulation and implementation of national science and technology policies in the areas of health, agricultural, and environmental sciences. Dr. Prager began his career in 1960 as a research scientist at the National Institutes of Health (NIH); from 1965 1968 he was a U.S. Public Health Service Fellow at Stanford University; in 1969 he was named chief of the Contraceptive Development Branch at NIH. From there he moved to the Battelle Memorial Institute in Seattle where he was the Director of the Battelle Population Research Center. Dr. Prager received his bachelor's degree in electrical engineering from the University of Cincinnati and his PhD in physiology from Stanford University. MARY ELIZABETH TAYLOR Professional Experience Senior Program Officer, Community Health Solutions (2006-2013), Bill & Melinda Gates Foundation (Seattle). Lead foundation officer for community-based maternal and newborn health grants, and country approach lead for Ethiopia, for the world’s largest philanthropy. International Health Systems Consultant (2002-2006). Continuous quality improvement and micro￾systems development specialist for the Gjilan-Dartmouth Primary Health Care and Family Medicine Partnership. Evaluation leader for MSH’s SEAM Project, an international effort directed at private sector strategies to improve access to medicines and rational drug use at the community level. Principal Program Associate and Director of Research and Evaluation (2000-2002), Center for Health Services, Management Sciences for Health (Boston). Developed program activities in child health and HIV/AIDS, including the provision of strategic planning, assessment, and rapid scale-up capacity building. 64 Long Term Consultant (1997-1999), Center for Development and Population Activities (Washington, D.C.). Evaluation and documentation of family planning programs, and facilitation of program design emphasizing systems thinking and continuous improvement. Launched field activities in reproductive health, maternal and child health, and democracy and governance in Nigeria and Nepal. Program Coordinator and Instructor in Community and Family Medicine (1981-1997), Dartmouth Medical School (Hanover, NH). Designed a peer-learning network for Baldridge-style self-assessment within health care organizations through the Center for Health Care Improvement Leadership Development, in partnership with the American Hospital Association, and facilitated self-assessment for the Southern Region of the Hitchcock Clinic. Public Health Field Management Advisor (1985-1990), JSI, Save the Children and Nepal Ministry of Health (Kathmandu, Nepal). Developed and supported the management of community health worker and female community health volunteer programs aimed at improving women’s and children’s health, immunization programs, and local and regional management of health services. Project Associate, Projet de Santé Rurale, (1979-1980), Harvard Institute for International Development (Mali, West Africa). Field management and logistics advisor, responsible for the planning, development, and evaluation of a community-based village health worker project in the rural Sahel. Education PhD and Master of Health Sciences, Johns Hopkins School of Hygiene and Public Health; BA with distinction, Cornell University 65 ANNEX 8. PROJECT BUDGETS BY USAID SOURCE SINCE 2009 Project Partner Mechanism Start date End date Award Ceiling MCH Malaria Nutrition Family Planning Reproductive Health HIV/AIDs Other Total Element funds to date TRAction Translating Research into Action URC Cooperative Agreement 30-Sep-09 30-Sep-14 $47,999,457 $15,011,897 $4,291,000 $1,638,423 $3,442,000 $5,400,000 $29,783,320 HRCI Health Research Challenge for Impact Johns Hopkins Bloomberg School of Public Health Cooperative Agreement 1-Oct-09 30-Sep-14 $17,000,000 $11,429,068 $300,000 $90,000 $1,049,000 $800,000 $1,231,000 $14,899,068 Accelovate Accelrating Innovation Jhpiego Cooperative Agreement 1-Oct-11 30-Sep-16 $24,999,917.00 $3,264,000 $2,150,000 $2,018,866 $7,432,866 HealthTech V Health Technologies PATH Cooperative Agreement 1-Oct-11 30-Sep-16 $24,410,411 $3,057,750 $50,000 $471,000 $3,010,000 $3,530,000 $50,000 $10,168,750 HealthTECH IV Health Technologies PATH Cooperative Agreement 2006 2011 $16,144,000 $3,366,000 $12,459,000 $2,794,000 $34,763,000 WHO WHO USAID WHO Umbrella Agreement 2009 2013 $3,392,785 $3,392,785 Non-Harp managed, but related WRA MCHIP WHO/RHR 66 ANNEX 9. DISCLOSURE OF CONFLICT OF INTEREST 67 Name Deni.!i' J_ Prager Tl'.l:le Pres:ide11t, Strate~ c: Conruhing ServiiCe:s Orpniizldiol:i Glfil Tedi! Birfdg,e 4 Conmbncy IJl'o5i:tion Team ll.eader Awvd NllltiJ.er (Contract or a!Jr"" Contra rt Number. AJD-OAA-C-B-00113 im:tn.mimd llliAID n~ r.:tfsJ Elt'alllllltedl (7;1d!ida fflelllt'h h IPrqs,n;m [HlllRP] fnillllltion p;aJGrt..nam, {s), fm;wumm:llar r.a ) i;-nd award nwnbi;f(s), if c~D.l.i=.h,'Q J II have r:e.BI 1ar poentiid D:!llfids Df O ves XX ND intJ:e1t to disdose.. Efy=-ered di~ ~ d~the foOowins; h;ds: "-' aroxmr!:Ja/~qj bt..-'""1' -.di; ~ ...... ~l"mftdh J. a.--~~ """111 .... "' ""• 'SAD.::or.~lld_,_ no~ _11......,_..-m.~in .. arg-...J""""'"praJilr!tr.J .... Ji.L, ...t- ? . p;l'l11c!ahin. 11111- t.u:\~ M fl ....,;ianf -·-~ arg,..._!.i:,1 -~· .. ""'~""~ .... bioi:v mor:tir4wb rlN outc::-id'b'I• .-.don l!. o.~-.or~~ .... ~~~ b:n.:f_.,_ na~ h>nkv._~ln ""'~l'lr_.a ~ ~wcP~bo ':llb. .. ri/IH~ Cu-.nl "'~'""'""''*-=- W"IBil'lg ............. D'l lw! LMD pr """"'iell ":'law:r;l_'l&' tk~ar •~ftd'_.. :.Ill~ -...-.~1J211 i..tll n 111hdoa. !I. Cu·- =< pr'o'(:urMC ---wlflt.., .::'"ll"""'-.fhr:f.._-,.o!io_m..,~ ""'"'Jri--11'>1 .!'4 --.iirt(l.I-~ ~- .-..-r..i. :I: ~:lnlrmobimdirn."'ct..i..- arg..-U, ~an~-~~rtnarm;. J, imp.kim,;;ntar fiutlriti:on re-se,arcih to;r(s).. if ll'~p.lka!; ~ J I tuwe re.al or potemial Cill!l\,filcls of x ves D NO intee:st to discfose_ If yes ~ed abave,. ~ disdase·the I am u11cfer a 100-day contract to Jh piego/ MCHIP fGllowi~ facts: lma'OJ ~~llf •lmnt""'tlod.dr rh ou~ end of Ja nuary, 2014_ I work unde r a task orrder .Ou! ~ - Lnilc' m.r. J ~/um#.!~ ~ .. -.......mo.--11., _ cnW ___ ..... tnov - w In 6'to ...n:..-a!""' nab:ooltm. .r aan..,,,.,. .............. rr._..,.m...,~ -..--..-.111¥ h¥1rmmlin.~ __ ..., _pq'm 't,I-~ ... - pn:tJrdli ~lilm>oll::tliDdh..~- ~ an:'.,_~m .... aJt,.o:m.n., Di!l!lll'IQ .... ~ ~ fll=f .,..;t:ij~~..-- I ceJ"li.1' (1l tn11t I tllilve ::i~d v ,i5 di~ure rum tu1t; and t:Dtne illl!st al mi; a:oifty !!lll 4ZI tlll!t I wiP ~toe this: djgjjo:...1re '°!!rm JXUl~flt:'fit' e~.o11~ cir=im:iices: m-...nc;~ tr I bllin 5= to f=""0:11ifil"f n~:irrurt oo of ~r a:mp11ni&S", tnen I ~cru to prote:t tlleirinl'ccrr">!:lian 17dt.-rk pro}«r nam~(1J, lmr;..'mni:nt~r t'lfllf!f?/j} 0110' e ~....,,~ ~--•:t:i...-l."' ~ ... ri1-.-or.,._r'fti!i ~..,. tf J/tif ,fli1Ur.«n. J C:.,.....,.., _..., .. llin:l w 1t;-\""'1l lh."'lll .~.,.,_•'Pl!'">''"' "'11 rltl> ~ trlng ..........tn'. ll.:lff'l1iil ., ... .....,,.y., llTpr'!ftl 1'W"j~ fll~•'INI ~11-iMI (\ftt-1" ~ .. Gl'•n1 · WY\IDU "'vtf C.!W"'"H""IU' JI S(l~h\il ifliT; , ...... , ... m ·v i.XAO aplll"41._ ... ll! R'Vr'D;1'1) 111t'nuWOQ~ L.11" Oit'r1•,:i.._,.,'tf:M.J '"'l-••1 .. 11-,m11a:qiJ.":1TDAPf .......... ..,, .s. Gl.1T1c r 11· ~ Mll't•l'WTV'KT wWt"" "1!J•iKN• rNI r<~~ _.,.,s.:.• •.-/11'1''¥ ~rm..' .m tH .. ,.._rnuwrq 0r01o;•....,.,W .. ,~,,.....,.~ftf:i lrl ;6 ,.,.._(}IY••·~ ~ -ril·"d ...... <111 ~· ~,.,., .. .,, ""~rrin L'(.J.~ l.V"loao .... (r.'\.-rta.fU!oi: tft.ltn.111.ffi"IQ 1'W1\l::rlftl I-ta' ... 111 c-.. ~ ... ~'""' Al D-OAA-C-13-00113: GHN·A·OO·OB-000040·00 Bureau for Pore~. f lemng Ard Ll!'.Vlllng- E::v.nlu11Uon or Prngrarn Cydl! lmi:fomc:nr.iibln (2013)- J!.f[).0AA-ft.l-t2-0J!l21; (;hajj &!NIVi!I Rr.;! H'*l'.tl Or-a'lls PTO!lruiiJ~ J use ea. ~klsure- tor u Ion&: ;is ~ retr'illl'Jproprle121rei ~"d ~~'.':m '"'' .. '"'••::•:/ 14 mh'"""h" " """'" - '"""'"' 71 72 U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523