PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION JUNE 2012 This publication was produced for review by the United States Agency for International Development. It was prepared by Ricardo Vernon and Tabitha Sripipatana through the Global Health Technical Assistance Bridge Project. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document (Report No. 12-01-047) is available in printed or online versions. Online documents can be located in the GH Tech website at www.ghtechproject.com. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: GH Tech Bridge Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com This document was submitted by Development and Training Services, Inc., with CAMRIS International and Social & Scientific Systems, Inc., to the United States Agency for International Development under USAID Contract No. AID-OAA-C-12-00004. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION i ACKNOWLEDGMENTS The evaluation team would like to acknowledge the hard work and collegiality of the PROGRESS team, who were very candid sharing project experiences. A special thanks goes to Lucy Wilson and Colleen Macko for their responsiveness to a range of questions and requests and for their exceptional work organizing materials and interviews. And, of course, we are grateful for all the time USAID, Ministries of Health, FHI 360 staff, and partner organizations gave for interviews and for the valuable insights and experiences they shared. ii PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION iii CONTENTS ACRONYM LIST ................................................................................................................................................. v EXECUTIVE SUMMARY ................................................................................................................................. vii Introduction ................................................................................................................................................ vii PROGRESS’s Structure and Achievements......................................................................................... vii Partnerships ..................................................................................................................................................x Support to the Field ...................................................................................................................................xi Evaluation Recommendations ..................................................................................................................xi I. INTRODUCTION ......................................................................................................................................1 Background ....................................................................................................................................................1 Evaluation Objectives and Organization of this Report .....................................................................2 II. METHODOLOGY ......................................................................................................................................3 Composition of the Evaluation Team .....................................................................................................4 III. PROJECT STRUCTURE AND PROGRESS TO MEET OBJECTIVES .............................................5 Project Structure .........................................................................................................................................5 Technical Content Structure ....................................................................................................................8 Country Focus and Factors Affecting It .............................................................................................. 17 Length of Award ....................................................................................................................................... 18 Progress to Meet Objectives and Factors Affecting It ..................................................................... 19 IV. PARTNERSHIPS ........................................................................................................................................ 27 Country, Regional, Global ...................................................................................................................... 27 V. SUPPORT TO THE FIELD ..................................................................................................................... 37 Where Does PROGRESS Work? ......................................................................................................... 37 VI. EVALUATION RECOMMENDATION OVERVIEW ....................................................................... 45 What Questions Does FHI 360 Wish It Could Have Pursued and What Are Potential Ideas for Future Research? ..................................................................................................................... 45 What Should USAID Do to Help Project Implementation? ........................................................... 45 What Should PROGRESS Do During the Last Year of Project Implementation?..................... 46 What Is the Best Structure and Focus for a New Research Award? ........................................... 47 iv PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION ANNEXES ANNEX A. SCOPE OF WORK ................................................................................................................... 49 ANNEX B. PERSONS CONTACTED ........................................................................................................ 57 ANNEX C. REFERENCES .............................................................................................................................. 63 ANNEX D. MONITORING AND EVALUATION FRAMEWORK .................................................... 67 ANNEX E. FHI 360’S CONSIDERATION OF FUTURE THEMATIC AREAS.................................. 69 ANNEX F. INTERVIEW GUIDE .................................................................................................................. 73 ANNEX G. E-MAIL QUESTIONNAIRE ..................................................................................................... 77 FIGURES Figure 1. Four Legacy Areas Organized by Thematic Areas.................................................................. viii Figure 2. PROGRESS Management Structure ...............................................................................................5 Figure 3. Progress Areas by Portfolio Activity .......................................................................................... 37 TABLES Table 1. Selected Characteristics of Countries where PROGRESS Has Conducted Activities.* ....1 Table 2: Number of Projects Conducted by PROGRESS by Region, Country, Legacy Area, Intermediate Result, and Type of Funding (As of April 2012) ............................................... 21 Table 3. Main Achievements and Expected Achievements by Legacy Area and Intermediate Results ........................................................................................................................ 22 Table 4. Example Partnership Country Profile: Rwanda ......................................................................... 29 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION v ACRONYM LIST AIDS Acquired Immunodeficiency Syndrome AOR Agreement officer’s representative CA USAID collaborating agency CB Capacity building CBA2I Community-based access to injectables CBFP Community-based family planning CHW Community health worker CIP Costed Implementation Plan CPR Contraceptive prevalence rate CRTU Contraceptive and Reproductive Health Technologies Research and Utilization Cooperative Agreement CTPH Conservation Through Public Health (Uganda) DMPA, Depo Depot medroxyprogesterone acetate ECSA East, Central. and Southern African Health Community FHI 360 formerly Family Health International FMOH Federal Ministry of Health (Ethiopia) FP Family planning FPTWG Family planning technical working group FS Field support GBM Green Belt Movement (Kenya) GH Bureau for Global Health GH Tech Bridge Global Health Technical Assistance Bridge Project GTL Global technical leadership HC Hormonal contraception HIP High-impact practices HIV Human immunodeficiency virus HQ Headquarters IBP Implementing Best Practices IR Intermediate result IRB Institutional Review Board IPPF International Planned Parenthood Federation IUD, IUCD Intrauterine (contraceptive) device LAC Latin America and the Caribbean vi PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION LAPM Long acting and permanent methods LOL Land O’Lakes, Inc. m4Rh Mobile for reproductive health mHealth Mobile technology for health M&E Monitoring and evaluation MCH Maternal and child health MOH Ministry of Health NIMR National Institute of Medical Research (Tanzania) NGO Non-governmental organizations NSV No-scalpel vasectomy NURSPH National University of Rwanda School of Public Health OPRH Office of Population and Reproductive Health OR Operations research PA Program assistant PATH Program for Appropriate Technology in Health PEC Policy, Evaluation and Communication Division PEPFAR U.S. President's Emergency Plan for AIDS Relief PI Principal investigator PMP Performance Monitoring Plan PPFP Postpartum family planning PPIUCD Postpartum intrauterine contraceptive device PRH Office of Population and Reproductive Health PROGRESS Program Research for Strengthening Services R Research RCQHC Regional Center for Quality of Health Care RH Reproductive health RTU Research, Technology and Utilization Division RU Research utilization SDI Service Delivery Improvement Division SOW Scope of work SubQ Subcutaneous TA Technical assistance US United States of America USAID United States Agency for International Development WHO World Health Organization PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION vii EXECUTIVE SUMMARY INTRODUCTION This report presents an assessment of the Program Research for Strengthening Services (PROGRESS) project, conducted by a team of independent consultants through the GH Tech Bridge Project. PROGRESS is a five-year USAID cooperative agreement that was awarded to Family Health International 360 (FHI 360) in June 2008, ending in June 2013. PROGRESS is a key component in the United States Agency for International Development (USAID) Bureau for Global Health/Office of Population and Reproductive Health/Research, Technology and Utilization Division (GH/PRH/RTU) portfolio seeking to improve the effectiveness of family planning (FP) programs for underserved populations in sub-Saharan Africa and Southeast Asia through innovation and scale-up of evidence-based practices. To achieve this, PROGRESS conducts operations research (OR) studies, research utilization (RU) activities, and capacity building (CB) support and works in collaboration with USAID Missions, Ministries of Health (MOHs), USAID implementing partners, and other stakeholders by means of a core team at headquarters (HQ) and staff on the ground in the FHI 360 country offices. The evaluation was based on a desk review of documents provided by PROGRESS (see Annex C for a list) and face-to-face, telephone, Skype, and e-mail interviews conducted with staff of USAID’s Office of Population and Reproductive Health (OPRH) and USAID Missions, PROGRESS and FHI 360 staff members, and collaborating and implementation partners both in the field and in the United States. Since there is still a year left for PROGRESS, the evaluation team was charged not with analyzing the project’s achievements but with capturing successes and opportunities to influence the last year of project implementation and the lessons learned regarding program structure, management, and partnerships that should be applied to future projects. Particular emphasis was given to capturing the lessons learned for working well with USAID Missions and meeting their future needs. PROGRESS’S STRUCTURE AND ACHIEVEMENTS The PROGRESS team leadership is located at FHI 360’s offices in Research Triangle, North Carolina. With the exception of the Dominican Republic and Pakistan, FHI 360 has field offices in all of the 13 countries in Asia and Africa where PROGRESS has conducted activities. In countries where several studies have been or are being conducted (called major portfolio countries), there is a dedicated PROGRESS lead or point person who manages all activities. PROGRESS is able to draw on the technical expertise across the FHI 360 organization, including biomedical researchers, program researchers, and other staff in field offices, which has been important to conducting global leadership, method mix, and crosscutting activities. PROGRESS was designed around four intermediate results (IRs): research, RU, CB, and improved contraceptive technology/method mix. To provide focus to launch its work, PROGRESS, in collaboration with USAID, narrowed the objective and described four “legacy” areas (project themes). The respondents found that reducing the general objective to improving access to methods and that establishing the four legacy areas addressing key opportunities and needs was an effective way to concentrate efforts in a few key achievable objectives and to avoid diluting scarce resources. Having four legacy themes also helps communicate the project’s objectives, activities, and results. viii PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION The project’s objective is improving access to family planning services among underserved populations through research, research utilization, and capacity building. Four legacy areas bound the general objective and activities, and accomplishments are best organized according to seven thematic areas. Figure 1. Four Legacy Areas Organized by Thematic Areas In addition, the project included global technical leadership (GTL) goals in FP and reproductive health (RH), which in practice requires interacting and influencing multilateral and other international and national reproductive health organizations. GTL activities include coordinating global technical meetings, synthesizing information from research findings, sharing innovative approaches to improving access to FP and south-to-south learning, and facilitating the use of best practices. The project timeline of five years for research and RU activities shaped the structural and programmatic content of PROGRESS. Country selection and research topics were often built from existing platforms or relationships that could be implemented quickly. PROGRESS estimates that it is meeting or exceeding 21 of 27 objectives included in the Performance Monitoring Plan (PMP) as of year four, and that it will meet five of the remaining six objectives in year five. The current workplan is tight, but PROGRESS anticipates wrapping up its work on schedule. However, there are a few activities, namely the Depo-SubQ in Uniject acceptability studies, which experienced significant partner-caused delays, for which any further shift in schedule may impede completion by the end of PROGRESS. As noted above, PROGRESS has another year to complete all activities. However, important work has already been completed. Below is a short summary of the key current successes and areas expected to show the largest impact in the future: PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION ix Community-based family planning (CBFP): Evidence about the feasibility of delivering depot medroxyprogesterone acetate (DMPA) injections through community health workers (CHWs) in four countries supports statements for this practice by normative organizations and arrangements to facilitate the scale-up of the practice in the future, including training toolkits, training CB of regional organizations, and the first stages of scaling-up in four countries. PROGRESS has also conducted operations research (OR) to help assess the capacity of CHWs to absorb this task in the range of services they provide given their workload and the capacity of women to self-screen their contraindications for use. Capacity building activities in Ethiopia: This has provided quality assurance of long-acting and permanent methods (LAPM) provided by medical and non-medical staff, data to the MOH to plan and organize the National Family Planning Program’s activities, data to insure that training achieves appropriate performance of providers, management information systems, and monitoring and evaluation (M&E) centers of excellence where data are collected and analyzed and actions are organized to better respond to program needs. The variety and scope of work (SOW) are Rwanda is probably achieving the same result. Mobile technology for health (mHealth): Using mobile telephones to provide information and facilitate the choice of methods of potential clients and to strengthen the service delivery skills of providers in collaboration with partners. The idea of taking advantage of the rapid increase in the availability of cell phones is very exciting and could have a large impact. Capacity building activities in Tanzania and Kenya: The costed implementation plans (CIP) in Tanzania and Kenya facilitated the collaboration of different service delivery activities, created a single plan of different evidence-based, high-impact practices, and helped secure funding from different donors and local governments. Expanding the method mix: The work conducted under the method-mix legacy area is increasing access to long-acting and permanent methods (LAPM) in different countries and for different user segments. These methods include Multiload IUD, DepoProvera in Uniject, a levonorgestrel￾releasing IUD system, an advanced vasectomy technique, and implants for youth. Postpartum family planning: Five research activities looked at provision of postpartum FP of which one focused on delivery of postpartum IUD services in district hospitals; others looked at integrating FP with immunization services and return to fertility postpartum. These studies will help determine how to better adapt this strategy to sub-Saharan Africa’s conditions and give local partners detailed experience in a controlled environment. Family planning integration with the non-health sector: The increasing number and density of FP service delivery points in rural areas in sub-Saharan Africa is of great importance. If proven effective under PROGRESS, the main question future awards should respond to, regarding work with agricultural and other type of organizations, is to what extent can these interventions be institutionalized and for how long will they remain delivering methods or information. Future studies should also explore the feasibility of using other “beyond health” sites (in particular, primary schools) for service delivery and assess if integration beyond the health sector would have greater impact than adding FP methods and services in organizations already offering some type of health service and/or products (for example, child survival services, drugstores, etc.). x PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION PARTNERSHIPS Partnerships are an essential component of the project structure, and the work of PROGRESS links to institutions working at the global, regional, and country levels. The original design of the project did not include a static consortium of partners. Instead, PROGRESS engaged in flexible and opportunistic partnerships to respond to evolving thematic areas and country needs. The global and regional partnerships, such as with the World Health Organization (WHO) and the East, Central, and Southern African Health Community (ECSA), were built on longstanding relationships with FHI 360. PROGRESS provided a platform and funding to execute mutually desired work that fit the goals of both PROGRESS and the partner institutions. The country-level partnerships were built out of new and existing FHI 360 office relationships according to needs, available staff, and the expertise of potential partners. Many partnerships were built on existing relationships under previous awards (e.g., Contraceptive and Reproductive Health Technologies Research and Utilization Cooperative Agreement [CRTU] and AIDS, Population and Health Integrated Assistance [APHIA] in Kenya). PROGRESS’s major portfolio countries include a range of 6 to 21 partners to execute country activities. Important elements in successful partnerships that PROGRESS utilized that should be retained and/or expanded in the future were (1) the implementing partner bringing funds to the partnership; (2) identifying a common client and adding value to what the partners are doing for this common client; (3) for non-financial partnerships, identification of common interests and incentives; and (4) flexibility, as the priorities of partner organizations and leadership change over time and the SOW needs to adapt. Satisfaction with PROGRESS as a useful partner was almost universal, including other U.S. government agencies, global institutions, and country-specific organizations. USAID Missions found that FHI 360 was a strong and supportive partner to other USAID collaborating agencies (CAs) and to the MOH, a key element that facilitated the research approval process, discussion and consideration of policy changes, and ultimate utilization of research results. Key activities for building partnerships with the MOH included participation in technical working groups, designing and implementing research with MOH staff as co-investigators, and executing CB activities for MOH staff. Partnerships with other CAs providing technical and financial assistance were often built upon requests from the Mission and/or the MOH to work on an activity with specific partners. Obstacles to building partnerships included (1) conflicting agendas and interests of potential CB partners; (2) Mission’s interest in obtaining access to general FHI 360 expertise and not having PROGRESS work on a project theme; (3) external partners’ concern about sharing data showing organizational weaknesses; (4) non-governmental organization (NGO) concern that partnering with FHI 360 would dull the CA’s competitive edge against PROGRESS in future bids; (5) cumbersome and slow administrative processes; and (6) lack of visibility and distinction of PROGRESS separate from FHI 360 in general. The PROGRESS “hand-off” strategy includes planning “the end” from the very beginning of any activity for any level of partner—local, regional, or international. FHI 360 described a very thoughtful strategy acknowledging that hand-off is a process, not an event, which is a goal from the beginning of the activity and influences and shapes the strategy from the beginning. Successful RU and hand-off by the research organization was described by Mission staff and service delivery partners as working directly with MOH, in particular with FP technical working PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION xi groups (FPTWGs) and other partners to bring a successful policy change and inform the scale￾up design with the operational experience of executing the study. The “real” scale-up into services should be the work of both international and national service delivery partners. New project design should incorporate and consider additional resources and support for the process of hand-off. SUPPORT TO THE FIELD Support to the field is a key function of the PROGRESS project as the centrally funded operations research flagship. The investment of core funds into countries led to increasing Mission funding through PROGRESS, demonstrating an important “multiplier” effect on core funds. PROGRESS has been able to leverage funds from 10 Missions and the Africa Bureau for a current total of $12.3 million. The main reason for buying into PROGRESS was the need for the services it could provide. However, other factors mediated buy-in, including (1) previous satisfactory work experience or relationship with FHI 360, particularly with the CRTU project; (2) existing relationships between Mission and OPRH staff members, or between Mission and PROGRESS/FHI 360 staff members which helped “get the door open”; (3) the Mission’s need for a particular set of skills not available in the country at a level of work not justifying a separate agreement; and (4) request of the MOH for assistance in a given area that did not have suitable local providers. Key attributes of PROGRESS in facilitating the buy-ins from different Missions were (1) FHI 360 in-country presence; (2) excellent reputation; (3) timing (the notice or visit came when the Missions were looking for the service); and (4) having a central mechanism that gave access to expertise in an area that did not warrant having a separate in-country program. In all cases, Mission staff reported high satisfaction with the services provided by PROGRESS. Key attributes of the services provided included (1) quality work; (2) flexibility of the program to take on a broad range of activities; (3) easy access, including good and frequent communication with Mission, MOH, and other partners; (4) responsiveness to Mission needs and to MOH requests; (6) ease of mechanism for field support; (7) good relationships with all partners in general and with the MOH in particular; (8) credibility, perceived competence; (9) playing as a member of a team for the greater cause in the country; and (10) good reporting. Among things that could improve, respondents noted (1) need for more staff at the country and local level, in both financial and program management; (2) late start-up of activities; and (3) need for more in-country dissemination and advocacy for scaling-up. One recommendation coming from FHI 360 and external partners for USAID was working with Mission staff to build their understanding of the research process, including ethics approvals. Some frequent obstacles to Mission buy-in are (1) Mission staff turnover; (2) lack of understanding of Mission staff regarding data, evidence, research, RU, and research processes; (4) lack of shared research and utilization models across all projects within the OPRH; and (5) exclusive Mission interest in local (and not global) needs. EVALUATION RECOMMENDATIONS PROGRESS will have an extremely busy last project year completing ongoing activities, and the evaluation team thinks the workplan includes priority activities and that there is no time to start any new activities. PROGRESS has planned to utilize the available timeline well. The evaluation xii PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION team’s recommendations for PROGRESS focus on the planned dissemination and utilization of activities. PROGRESS already has activities planned for Washington DC and the local level. Considering the reported obstacles and opportunities, the evaluation team presents this list of highlighted recommendations for future research projects: There is still a need for an FP operations research flagship project funded and functioning independent of a service delivery project. The project timeline should be 7 to 10 years. Investigate synchronization of award mechanism with when field support decisions are made so that research CA does not lose out on one year of potential field support. Retain the collaborative approach PROGRESS and RTU utilized at the beginning of the project to focus the work into legacy areas. With a longer timeline, the project should conduct more studies testing innovations, things that have not been done before, exciting new and original ideas. A future flagship OR project should continue supporting the strong global/core technical experts to address arising issues such as the hormonal contraception (HC)/human immunodeficiency virus (HIV) research, participate in Cochrane reviews, and prioritize global research agendas, etc. With a longer project timeline, take a more “disciplined” approach to initiating activities and select a small number of core focus countries for research and RU activities. While there is a call to have a core set of focus countries based on an OPRH level strategy, there is also a plea to retain the flexibility of the mechanism to respond to Mission needs. Retain the use of core funding as “seed” funding to introduce OR activities at the country level with an explicit commitment from the Mission that field support will co-fund. One interesting idea was to promote the launch of a new OR activity as a competition to the Missions, that if Missions applied with good ideas for the OR activity, core funds would match their commitments. Move away from the current concept and include CB activities as continuous technical assistance to install and strengthen systems and operating units and, only occasionally, teaching courses and workshops and strengthening the capacity of universities and research institutions. Capacity building in FP program M&E should continue to be a central component of the project, with MOH and FPTWG the main target of CB activities. Develop a specific branding strategy that distinguishes the project from the implementing partner, with resources set aside to participate in more FP research forums and meetings in Washington DC. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 1 I. INTRODUCTION BACKGROUND Despite advances in recent years, countries in Africa and Southeast Asia continue having high fertility rates and low contraceptive prevalence rates. Both supply- and demand-side factors influence this. On the demand side, poverty, low levels of education, and high mortality rates (see Table 1) translate into a large number of children desired by couples and a fragile demand for fertility regulation services and products; whereas on the supply side, Ministries of Health (MOHs) are characterized by small budgets, insufficient infrastructure, a lack of skilled workers, and weak administrative and service delivery systems. A requirement for effective international assistance to help couples achieve their reproductive ideals and obtain the proven benefits of having smaller families is to help family planning (FP) programs in these regions identify and use strategies and service delivery mechanisms that actually work, practices of proven effectiveness and cost-effectiveness in helping couples use contraception. Table 1. Selected Characteristics of Countries where PROGRESS Has Conducted Activities.* REGION/ Country POPULATION (millions) TOTAL FERTILITY RATE (children per woman) INFANT MORTALITY RATE (per 1000) CPR—ALL METHODS CPR— MODERN METHODS GNI PPP PER CÁPITA (US $) % POP < $ 2 US PER DAY AFRICA Ethiopia 87.1 5.3 77 15 14 930 78 Ghana 25.0 4.1 45 24 17 1,530 54 Kenya 41.6 4.7 59 46 39 1,570 40 Malawi 15.9 5.7 84 46 42 990 90 Nigeria 162.3 5.7 89 15 10 2,070 84 Rwanda 10.9 4.6 50 - 45 1,130 90 Senegal 12.8 4.7 51 12 10 1,810 60 Tanzania 46.2 5.4 51 34 26 1,360 88 Uganda 34.5 6.4 74 24 18 1,190 65 Zambia 13.5 6.3 84 41 27 1,280 82 ASIA India 1,241.3 2.6 50 54 47 3,280 76 Pakistan 176.9 3.6 53 27 19 2,680 61 LAC Dom. Rep. 10.0 2.6 23 73 70 8,110 14 *Population Reference Bureau 2011 Population Data Sheet 2 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION This report presents an assessment of the Program Research for Strengthening Services (PROGRESS) project conducted by request from the Research, Technology and Utilization Division (RTU) of the Office of Population and Reproductive Health (OPRH), which has been undertaken by two independent evaluators under contract with the Global Health Technical Assistance Bridge Project (GH Tech Bridge). PROGRESS is a five-year USAID cooperative agreement that was awarded to FHI (Family Health International, later renamed FHI 360) in June 2008 and will end in June 2013. PROGRESS is the key component in the USAID (GH/PRH/RTU) portfolio seeking to improve the effectiveness of FP programs for underserved populations in sub-Saharan Africa and Southeast Asia through innovation and scale-up of evidence-based practices. To achieve this, PROGRESS conducts operations research (OR) studies, research utilization (RU) activities, and activities to strengthen the capacity of local organizations and researchers to conduct research and evaluation. PROGRESS works in collaboration with USAID Missions, Ministries of Health (MOH), USAID implementing partners, and other stakeholders by means of a core team at HQ and staff on the ground in the FHI 360 country offices. EVALUATION OBJECTIVES AND ORGANIZATION OF THIS REPORT The methodology used to conduct this assessment is presented in Section II. In brief, the evaluation team conducted interviews with stakeholders and a desk review of key programmatic documents. In this report, one section is dedicated to each of the following sets of questions: 1. How has PROGRESS evolved and progressed toward achievement of its objectives? What factors motivated the changes? What factors facilitated achievement of project objectives? How has the organizational structure at FHI 360 and USAID accelerated or slowed the process? These questions are addressed in Section III. 2. How have partnerships at the global, regional, and country level contributed to achievement of project objectives, particularly with regard to the implementation of operations research (OR), research utilization (RU), and capacity building (CB)? What type of partnerships were more effective and why? What strategy does PROGRESS have to successfully facilitate the hand-off strategy? These questions are addressed in Section IV. 3. As a centrally funded activity that is intended to support the field, was the design and implementation appropriate to answer field questions? Did PROGRESS find that there is substantial interest and demand for OP, RU, and CB from Missions? – What were the factors that led Mission staff to buy into PROGRESS, both with field support (FS) and supporting core funded activities? What was the range of topics of interest? Were field questions adequately answered? – How did the PROGRESS management team interact with the Missions? What activities were successful in obtaining buy-in, and what hindered activities in country? – How did USAID/Washington interact with the Missions? How were global technical leadership (GTL) priorities balanced with Mission agendas? – These questions are addressed in Section V. Finally, section VI presents recommendations for the way forward. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 3 II. METHODOLOGY The assignment commenced with a two-hour planning telephone conference in which USAID RTU Division staff briefed the two-member consultant team on the PROGRESS cooperative agreement and its activities. USAID discussed the purpose, expectations, and agenda of the assignment with the evaluation team. Since there is still a year left for PROGRESS, the team was charged not with analyzing the project’s achievements but with capturing the lessons learned regarding program structure, management, and partnerships that could be applied to future projects. Particular emphasis was given to capturing the lessons learned for working well with Missions and addressing their future needs, to help ensure that future projects retain what is working and change what is not. In the following days, the team members developed final evaluation questions, data collection instruments, and guidelines, which were reviewed and authorized by RTU staff members. The sources of information for this evaluation included the following: 1. Interviews with key stakeholders, which consisted of – Face-to-face interviews with 13 USAID PRH staff members from three of its four divisions (RTU; Service Delivery Improvement [SDI]; and Policy, Evaluation and Communication [PEC]) and one Africa Bureau staff member on April 10 and 11 in Washington, DC. – Face-to-face interviews and meetings with 39 FHI 360 staff members on April 12 and 13 at FHI 360 HQ in Research Triangle, North Carolina, and telephone interviews with 9 FHI 360 staff members in HQ and field offices. – Telephone interviews with 21 staff members of external partners, 8 staff members of USAID Missions, and 6 MOH staff during a three-week period (April 17–May 4). – Ten written questionnaires completed by U.S. and field-based stakeholders (April 17– May 4). – Annex B lists all persons interviewed as part of this assessment. 2. Desk review of documents (see Annex C) that detail the activities of the project and describe issues related to implementation and resolution, including the following: – PROGRESS award document – Performance Monitoring Plan (PMP) and Gap Analysis – Evolution of the project from the policy determination (PD)—“Crosswalk” – Post-Dakar 2011 management review and technical updates – Annual and interim reports and workplans – Annual results reviews – Publications and related documentation from activities – Management review meeting minutes 4 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION – Project summaries by theme and country – Additional documentation developed by PROGRESS specifically for this evaluation – Approved research protocols – All documents were provided by PROGRESS staff members in electronic form. The contents of the interviews and documents were categorized and summarized, according to the questions presented in section 1.2, to achieve conclusions. COMPOSITION OF THE EVALUATION TEAM The evaluation team was composed by two external consultants identified and recruited through GH Tech. Ricardo Vernon served as team leader. Since July 2008, he has been a consulting partner at Investigación en Salud y Demografía S.C. , a Mexican research, evaluation, and consultancy firm specialized in health, nutrition, education, science and culture, and antipoverty programs. Before July 2008, he worked for 22 years in USAID-funded FP/RH OR programs conducted by the Population Council in Latin America and the Caribbean (LAC) and around the world. In addition, he has worked for the FP/RH program of the MOH of Mexico; for TELEVISA, a Mexican network; and for Opinión Professional, a Mexican polling organization. He holds a Ph.D. in Sociology from the University of Chicago. Tabitha Sripipatana is an independent consultant. She holds a master’s degree in public health from UCLA and has 13 years of professional experience in research and program implementation and evaluation in the RH field. She spent more than 9 years working for the Elizabeth Glaser Pediatric AIDS Foundation and has experience working with USAID, John Snow’s MotherCare project, and the Pacific Institute for Women’s Health. Both have a large number of programmatic and scientific publications, as well as extensive experience with and a thorough understanding of FP/RH programs in less developed countries. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 5 III. PROJECT STRUCTURE AND PROGRESS TO MEET OBJECTIVES This section discusses PROGRESS’s structure in five dimensions: management, technical content (themes), geography (country focus), time (project length), and funding, setting out in each area PROGRESS’s evolution and the factors motivating the changes and affecting the results. The section ends with a discussion of the progress made by the project in meeting its objectives. PROJECT STRUCTURE Management Structure The PROGRESS team leadership is located at FHI 360’s offices in Research Triangle, North Carolina, and has not changed since the beginning of the project. Dr. Maggwa Ndugga is project director, Rose De Buysscher is deputy director for management and administration, and Dr. John Stanback is deputy director for research. Ms. De Buysscher is assisted by technical officer Karin Ganter and Dr. Stanback by technical officer Elena Lebetkin. There are also four program officers—one each for monitoring and evaluation (M&E) (Lucy Wilson); RU(Bill Finger); innovative technologies (Laneta Dorflinger); and CB (Rick Homan)—and one administrative assistant/budget lead (Colleen Macko). Figure 2. PROGRESS Management Structure Both within and outside the PROGRESS team, the competence, professionalism, and collegiality of the team members were praised. Described as an able manager, Dr Maggwa is a respected professional with decades of experience in research programs in Africa with different organizations and a very extensive network of professional contacts who have facilitated achieving the program’s objectives. Dr. Stanback is a much- appreciated researcher with decades of experience and with contributions to the field that have translated into greater access to FP services for large numbers of women in Africa and with a body of published work in the most 6 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION important scientific journals in the field. Ms. De Buysscher also has decades of experience in the administration of USAID-funded programs conducted by FHI 360, which includes field experience. In general, the team was praised for its advanced skill set and for its seriousness in seeking “to produce science.” Regarding the rest of the team, the respondents thought it was especially important that FHI 360 decided to have one PROGRESS staff member devoted to RU. In addition, FHI 360 has a team working on RU, and it was recommended that PROGRESS remember to budget and include in workplans the work that this team undertakes. With the exception of the Dominican Republic and Pakistan, FHI 360 has field offices in all of the countries in Asia and Africa where PROGRESS has conducted activities. In countries where several activities have been or are being conducted (called major portfolio countries), there is a dedicated PROGRESS lead or point person who manages all activities and is a senior staff member within the country office. In major portfolio countries, the project has at least three part- or full-time staff in FHI 360 field offices, as required by the activities in each country. Thus, by billing only for the time used by the project, PROGRESS is able to bring in staff and skills from the larger FHI 360 organization as needed, something which was perceived as an advantage over the usual practices in other organizations of having full-time dedicated project staff members: “They bill like a law firm, very smart way to work.” Finally, PROGRESS is able to draw on the technical expertise across the FHI 360 organization, including biomedical researchers, program researchers, and other staff in field offices, which has been important to conducting global leadership, method mix, and crosscutting activities. Regarding communication between PROGRESS staff, field staff highlighted the importance of the Dakar Global Management Review meeting to share perspectives, methodologies, and results. From these comments, it seems that it would be convenient to hold at least one cross-country project meeting in the final PROGRESS year and in subsequent projects to hold frequent (for example, yearly) meetings with at least some staff of the different offices FHI 360 has administrative systems and methods that allow for monitoring the progress of individual activities and of the project overall. The approval of research activities requires that a formal protocol is submitted, peer-reviewed, and approved by USAID. For other research activities, a concept idea is presented and approved. The progress of each activity is reported through FHI 360’s electronic information system (EIS) on a semiannual basis, which provides the input to prepare annual and semiannual reports and workplans. The project also uses Microsoft Project software to track financial and substantive progress on individual research studies. Each protocol is assigned a number and followed through the full cycle. The system is fed from the reports of the implementing partners. One respondent described the financial reporting system as “complicated” and in need of simplification. While monthly financial reporting was considered appropriate, this partner considered that monthly technical reporting was “too frequent,” because often there would be little to report. However, this gives FHI 360 confidence in planning. Beyond the individual tracking system, the project PMP has two main components for monitoring progress—a logic model with key indicators and the M&E framework, which defines specific objectives for each legacy area. Activities are mapped to these objectives via the PROGRESS Gap Analysis, which indicates the activities that address each of the objectives as laid out in the M&E framework. It serves as a signal for identifying potential gaps in programming. The interviewees did not make any noteworthy comments about these systems, so it appears PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 7 that they are strengths of the project. It is recommended, however, that there be more explicit summaries of the characteristics and achievements of the studies listed in the annual and semiannual reports because currently they do not give a clear picture of the design, scope, progress, and results of individual activities. The USAID management team is located in the RTU Division in the Office of Population and Reproductive Health (GH/PRH/RTU) in Washington, DC. Dr. Mihira Karra is the agreement officer’s representative (AOR), Megan Matthews is the technical advisor, and Matthew Phelps is the program assistant (PA). Until one year ago, Patricia Stephenson acted as the project´s AOR, Judith Manning was the biomedical technical advisor, and Megan Mathews was the PA. Shawn Malarcher joined two years ago as a technical advisor dedicated to RU. Both Ms. Stephenson and Dr. Karra are highly experienced professionals with decades of experience both within the OPRH and in the field. Ms. Matthews has been the most consistent presence and has been on top of the project for four years. Interviewees commented on two related issues: individual management styles and the complexities involved in coordinating actions with the different divisions in the OPRH and between OPHR and USAID Missions in the field. Regarding individual management styles, the differences between the first and the second AOR were highlighted. The main differences mentioned were in proximity and focus. In terms of proximity, the respondents reported easier access with one than the other, who has a busier schedule and relies more on the technical advisor for communication with PROGRESS staff. On the other hand, this reliance on the technical advisor for communication with the FHI 360 team seems to have helped resolve confusing communication situations observed at the beginning of the project, when agreements were made but not documented or communicated to the rest of the stakeholders. Regarding focus, at the beginning of the project there was a strong effort to have proposals comply with the project´s framework, whereas greater flexibility was reported at later stages. This may have been somewhat unavoidable; one of the main advantages of PROGRESS has been its focused approach, and thus the efforts to ensure compliance with the project design in the beginning were to be expected and supported, especially because of the limited availability of funds. On the other hand, there is always the pressure to produce and spend, and this pressure increases with time, so a certain degree of flexibility is needed. Further, USAID Mission staff praised PROGRESS’s flexibility, an attribute that helps attract field support buy-in and facilitates the implementation of activities. Finally, concentrating activities in four countries, as initially proposed, would not have been practical given the core/field support mechanisms, and so greater flexibility was appropriate to help expand the number of countries participating in the project. The recommendations are for continued access and communication as feasible and for consistency in lines of communication with FHI 360 and in the guidance given to the project. PROGRESS will have to continue balancing the demands of the field and Missions with the requests from RTU when reviewing concept submissions and assessing whether proposed activities adhere to the spirit of the project. In terms of USAID methods and systems to manage PROGRESS, as of February 2012, all subcontracts require approval from the AOR and agreement officer, while subagreements up to $2,500,000 require AOR approval only. Considering the workload of the Office of Acquisitions and Assistance, and the historic blackout periods where no approvals are provided, the recommendation is to raise the approval threshold to at least $25,000 for all subcontracts to help the program be responsive to Mission requests and emerging global issues. 8 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION The funding cycle timing created periods with no field support funding at the project start-up and periods of uncertainty to plan next fiscal year activities because the program year vs. fiscal year and the core vs. field support funding are out of sync. This requires different tracking systems with multiyear budgets. It is recommended to have one combined workplan year and fiscal year. For example, for the next program start-up, consider having a new award made in January, with advocacy and buy-in with the Missions in March–May and research/activity launch in September. This funding cycle timing between core vs. field support is beyond the control of individual USAID divisions to change, but the challenge must be marked as it impacts the systems and funding available for project initiation. Regarding communication between the PROGRESS USAID and FHI 360 teams, the core FHI 360 team in North Carolina has bimonthly calls with USAID. The administrative management team holds monthly conference calls with USAID. Both PROGRESS and USAID/Washington report picking up the phone frequently to check in or respond to emerging information needs. Yet, team members on both sides expressed their wish for more frequent communication. Being in different cities forces the teams to have more structured or planned interactions than in awards where both teams are in DC. TECHNICAL CONTENT STRUCTURE FHI’s Original Technical Proposal and Events Subsequently Affecting It In order to understand the evolution of the PROGRESS project, the evaluation team first reviewed the contents of the technical proposal that FHI (now FHI 360) presented to respond to USAID’s request for proposals. In it, FHI proposed a focused approach centered in three separate dimensions: a general objective, four IRs, and four focus countries: The goal was improving access and quality. The four intermediate results were to identify knowledge gaps and prioritize promising solutions (R); prioritize effective practices ready to implement (RU); improve the capacity of developing-country public- and private-sector organizations to produce and use program research results (CB); and prioritize options to improve technologies (new and improved methods, later to become method mix). Four focus countries were to be selected from a proposed preliminary list of eight countries: Ghana, Mali, Nigeria, Rwanda, and Zambia in sub-Saharan Africa, Cambodia and India in Asia, and Guatemala in Latin America. At the same time that PROGRESS was beginning activities, other mechanisms that USAID had used to provide technical and financial assistance in research, RU, introduction of contraceptive technology, and CB to international RH and FP programs and organizations were coming to an end, most importantly the Contraceptive and Reproductive Health Technology, Research and Utilization cooperative agreement (CRTU). FHI’s CRTU project was the means by which several initiatives had been funded, including the Implementing Best Practices (IBP) Initiative and the Cochrane reviews. This is relevant for this evaluation, because several of the activities conducted by these programs involved mid- and long-term valuable commitments by USAID, and PROGRESS became the mechanism that allowed continued work. This highlights the importance of continued funding of research and knowledge management activities of global relevance through centrally funded RTU projects, both in the last year of PROGRESS and in PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 9 future awards, if no alternative mechanisms are established. In addition, a few projects initiated under CRTU had not been completed by the end of the cooperative agreement and were picked up by PROGRESS to avoid losing the investments already expended. Technical Content Structure and Factors Affecting It As mentioned before, PROGRESS was originally structured around four IRs: R, RU, CB, and improved contraceptive technology/method mix.1 However, during the first year of the project, it was clarified that these IRs were too broad to help focus activities. The USAID AOR challenged FHI 360 staff members with questions about what they would want to be remembered for once the project ended and how the world would be a better place as a consequence of having implemented the project. As a result of the extended discussions that ensued, USAID and FHI 360 agreed that the project’s general objective would be “improving access (and not access and quality) to FP services among underserved populations through research, RU, and CB,” a statement that preserves the initial IRs. They also agreed that this general objective would be bounded by the four legacy areas (project themes or areas) that were selected: 1. Task-shifting: Maximize human resources through task-shifting and addressing medical barriers. The focus themes in this legacy area have been community-based family planning (CBFP), in general, and community-based access to injectables (CBA2I), in particular, mostly through community health workers (CHWs) and through drug shops. The main factor facilitating this line of work was an existing body of studies conducted under CRTU. In addition to this line of work, PROGRESS also helped evaluate the delivery of implants and intrauterine devices (IUDs) in Ethiopia by non-medical staff. This legacy theme is of critical importance for countries in Africa, where the small number of service delivery points and physicians and nurses limit access to contraception in general, and to long-term (implants and the IUD) and permanent (female sterilization and vasectomy) methods in particular. 2. Integration: Expand service delivery options within and beyond the health sector. In practice, this has involved activities related to postpartum contraception (integration with post-delivery and immunization services); delivery of information and contraceptive services in communities through non-health (dairy, conservation, environmental, agricultural) organizations; and provision of information to service providers, potential clients, and clients through cell phones (labeled m4Rh for mobile for reproductive health). Delivery of injectable contraceptives through drugstores is also included in this integration area. 3. Method mix: Expand the FP method mix for home, community, and lower-level provider use. In practice, this legacy area has involved helping programs introduce new or existing underutilized contraceptive technology, such as depot medroxyprogesterone acetate (DMPA) through Uniject implants, Multiload IUDs, and vasectomy with cauterization and facial interposition. Another important line of work (especially considering the limited access to female sterilization services in sub-Saharan Africa) is increasing access to Chinese￾manufactured implants (Sino-Implant II) in countries where these are registered; these cost US$8, one fourth of the cost of Jadelle, the equivalent product manufactured under license 1 Research involves providing technical and financial assistance for conducting operations and programmatic studies in collaboration with local institutions; research utilization involves helping programs and institutions identify and use evidence-based practices and strategies that can help them improve their service delivery operations; capacity building involves helping strengthen the capacity of local institutions and researchers to conduct applied program research and evaluation activities; and method mix involves all aspects related to the development and introduction of new contraceptive technology or scaling-up of underutilized contraceptive technology. 10 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION by the Population Council. Finally, of particular importance in this legacy area is the collaboration of PROGRESS with the Program for Appropriate Technology in Health (PATH), the Gates Foundation, and Pfizer, which is in itself an experiment in a new paradigm of public-private partnerships for international assistance.2 4. Capacity building: Increase in-country capacity for R, RU, and M&E. This has involved helping institutions, researchers, and programs strengthen their research and M&E capabilities through courses and workshops, hands-on participation in activities, observation visits, and a variety of other mechanisms; and build functional evaluation units, centers of excellence for M&E, and service delivery information systems. This category also includes assistance in the development of key planning instruments, such as the costed implementation plans (CIP) in Tanzania and Kenya. In addition, the project included global technical leadership (GTL) goals in FP and RH, which in practice requires interacting and influencing multilateral and other international and national RH organizations. GTL activities include coordinating global technical meetings; synthesizing information from research findings (for example, Cochrane reviews); sharing innovative approaches to improving access to FP and south-to-south learning; and facilitating the use of best practices. To a large degree, these crosscutting technical activities were medium￾and long-term activities that had previously been funded through CRTU and previous USAID￾funded OR programs. Partners have included the World Health Organization (WHO), the International Planned Parenthood Federation (IPPF), and the International Federation of Gynecology and Obstetrics. Even though PROGRESS’s Scope of Work (SOW) could be seen as encompassing a matrix with legacy areas in lines and IRs in columns, neither the IRs nor the legacy areas or the crosscutting GTL themes and activities involve mutually exclusive categories. For example, research studies have CB and RU components, and one activity may involve both task-shifting and expanding service delivery options, as in the case of provision of injectable contraceptives through drug shops. Nonetheless, it is possible to judge the degree to which PROGRESS was able to focus activities by using as boundaries the general objective of increasing access, the three IRs of R, RU, and CB and the four legacy areas of task-shifting, integration, method mix and, again, CB. Conclusions and Recommendations for Future Projects and for RTU R, RU, and CB are the defining traits that justify the existence of a stand-alone, core-supported project like PROGRESS. This section presents conclusions and recommendations for future awards related to these IRs based on the findings of this assessment. The following section presents a few recommendations for PROGRESS that the project can consider implementing in the very busy final year of activities. General Objective There was universal agreement that the general objective of “increasing access to FP services” allows for a much more focused project than “increasing access and quality.” Given conditions in Africa, this should remain the general objective of a future award. However, this 2 At the beginning of the project, close to US$1 million were awarded for Depo-SubQ in Uniject work. These funds remained parked until late in the fourth year and had a significant impact on the PROGRESS pipeline for several years, making it more difficult for the AOR to argue for annual funding for new and continuing projects each year. The delays for activity start-up were the unexpected need to complete a pharmacokinetic study of Depo-SubQ, waiting for Pfizer to get Depo-SubQ in Uniject approved in Europe, and, subsequently, the demand by Pfizer that the rather straightforward acceptability study that had been planned had to be run as a clinical trial meeting extremely strict “good clinical practice” criteria. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 11 recommendation should be assessed in light of the extremely rapid increase in contraceptive prevalence observed in countries like Ethiopia and Rwanda, where quality may become a pressing need very early in a follow-on project. In these countries, particular attention should be given to information for clients and informed consent procedures. Research Research conducted by PROGRESS has focused on both intervention and non-intervention. Within the former are a range of studies running from randomized control trials to evaluation of activities conducted by other implementing partners. Recommendations for future awards are to: Conduct a larger proportion of studies testing innovations that can help set a strategic agenda for SDI and PEC projects in the future; examples from PROGRESS research are the impact of offering free pregnancy tests on demand for contraceptive services or adapting the balanced counseling strategy to m4RH. RTU might consider organizing a list of explicit ideas, including criteria for determining their relevance, and circulating these within the OPRH and USAID Missions to obtain feedback and identify opportunities and tentative interest in implementing these in projects and countries. If possible, RTU could conduct this research in more than one country or have replications of interventions showing impact. Expand the documentation and evaluation of task-shifting and integration interventions. In addition to the valuable feasibility and quality assurance studies on injectables, implants, and IUDs that PROGRESS has conducted, respondents mentioned other interventions that CAs and other service delivery organizations are conducting in Africa (for example, tubal ligation by medical officers, the large number of FP-immunization integration projects that PROGRESS has mapped around the world, and Implanon in Ethiopia) that, if properly documented, could add to the evidence on these two legacy areas in Africa. In addition, USAID Missions seem to be interested in buying into these rigorous documentation/evaluation/quality assurance/demonstration projects, and they could help create a larger sense of ownership and credibility among CAs and thus increase the probability of replication and scaling-up. One lesson from PROGRESS, to help avoid rejections from implementing NGOs and CAs, is to have the MOH in each country request such quality assurance and documentation exercises. Continue conducting task-shifting demonstration projects in order to allow countries to become comfortable with the service delivery mode until policy changes occur or other partners pick up an activity. To determine how long each project should continue to be supported, project proposals must identify the decision-making points and key stakeholders for policy change and plan for guidance and technical transition of service delivery experience and materials to the MOH and local service delivery organizations. Develop a framework to make decisions on when to use randomized control trials and systematically use true experimental designs in those cases. Utilization/Dissemination Between OPRH Projects RU and/or hand-off of research results to country-level stakeholders are discussed in the “Partnerships” section. However, several respondents brought up non-PROGRESS-specific concerns regarding limited RU between OPRH projects. The evaluation team offers some 12 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION suggestions regarding the broader need to share research results.3 From the interviews, the team understands that the limited utilization of research results between projects is a consequence of the lack of a mutually shared concept between divisions about the prioritization of practices/methods at the project level and of difficulties in implementing viable coordinating mechanisms to facilitate the use of research results across the office. It is recommended that RTU develop an evidence-based utilization-of-research-results model that defines and characterizes the concept, identifies good practices, can help assess the performance of OR projects along this dimension, and includes explicit utilization goals (for example, in terms of policy changes or scaling-up of activities) set at the beginning of the new project.4 Regarding coordination between RTU, SDI, and PEC, the OPRH has implemented different strategies, among them (a) the existence of internal “champions” who serve as technical resources and seek to promote specific issues for projects, activities, publications, etc.; (b) the dissemination and promotion of evidence-based high-impact practices (HIP) that an RTU working group recommends be included in contracts, agreements, projects, and activities conducted by the other divisions and by Missions56 (perhaps these HIP messages would have greater uptake if they referred to specific countries and projects); and (c) the Africa Bureau seeks to facilitate the implementation of activities that will benefit several countries. The respondents identified participation in country teams as a key opportunity to promote use of research results. To achieve policy goals PROGRESS has partnered with, among others, the Advance FP project in Uganda and Kenya, with ChildFund in Zambia and Senegal, and with the bilateral Targeted States High Impact Project (TSHIP) with John Snow, Inc. in Nigeria. However, it is not clear what mechanisms are used within the OPRH to insure that use of evidence-based research results are included as goals or objectives in policy and service delivery projects or to promote greater coordination between the different agreements in using these results; it is recommended that this be made explicit. It should also be determined whose responsibility it is in each stage to use evidence-based research results pertinent to the project´s objectives. Finally, it is 3 By research results we refer to products that OR projects yield, such as data and tools, practices, strategies, program systems or subsystems, models, etc., which can help programs offer improved services in terms of quality, effectiveness, efficiency, or any other dimension. 4 Research has shown that research results are an input among many others that feed long-term decision-making processes leading to policy changes and that tools, strategies, and models are adapted to the characteristics and needs of different systems. Further, research results feed dynamic processes, and the decisions made on the basis of a research result are afterwards changed by other decisions. The diffusion of innovations literature can also provide valuable insights into the construction of this model. 5The themes promoted and the champions are the following: Contraceptive Security—Alan Bornbush; Poverty and Health Equity—Shawn Malarcher (acting); Youth—Cate Lane; Community-Based Distribution Programs—Victoria Graham; FP and HIV Prevention Integration—Mary Anne Abeyta-Behnke; Postabortion Care—Carolyn Curtis; and Birth Spacing—Maureen Norton. 6 High-impact practices are divided into three categories: (1) Proven: Train, equip, and support CHWs to provide a wide range of FP methods, offer FP counseling and methods when women receive postabortion care services; support distribution of a wide range of FP methods through social marketing; and disseminate locally designed and tested FP messages through multiple channels, including the media and community networks to promote social and behavioral norms; (2) Recommended: Offer a wide range of FP methods through mobile outreach services; train and support pharmacists and drugstore shopkeepers to provide a wide range of FP methods; offer FP services to postpartum women (up to 12 months after birth), such as when screening women during routine child immunization contacts; and (3) Emerging: Support public-private partnerships through NGO contracting, franchising, and vouchers; support provision of FP services and information dissemination through mobile phone technologies. It should be noted that even though they are supposed to be evidence-based recommendations, these include practices for which there seems to be insufficient evidence. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 13 recommended that mutually overlapping planned results between projects be identified routinely and that projects be requested to present a collaboration plan for achieving the overlapping desired results in specific countries. Some partners thought there had been little follow-up of ideas of promising practices from previous OR projects. They thought OPRH should ensure continuity. Capacity Building One general conclusion of this assessment is that CB activities should involve continuous technical assistance (TA) to install and strengthen systems and operating units and, only occasionally, teaching courses and workshops and strengthening the capacity of universities and research institutions. Efforts to build research in general and OR capacity in particular by means of courses and workshops was perceived by some as an activity that universities and public health schools could do better and with more cost efficiency, not requiring the assistance of the project. It was felt that a research project could have more impact in providing hands-on experience and one-to￾one mentorship; helping managers become consumers of research results; and helping researchers write proposals (an activity that presupposes the existence of funds, which should be confirmed before the course or mentorship). PROGRESS’s experience seeking to strengthen the capacities of universities and research institutions has been mixed at best. Originally, they proposed to identify local research institutions, build their capacities to conduct research, and link them to the MOH. The project was able to establish agreements with only two—the National Institute of Medical Research (NIMR) in Tanzania and the National University of Rwanda School of Public Health (NURSPH). In the first case, NIMR is conducting a series of workshops and is mentoring three junior associates in OR. These associates participate in FHI 360 research activities and attend the FHI 360 office two days per week. They also participate in the FPTWG and are encouraged to respond to calls for proposals with the TA of FHI 360 staff. NIMR seems to be very satisfied with the experience; however, other in-country respondents felt that NIMR already had a strong research capacity. The MOH was also concerned that CB focused on an external agency and not the MOH. In the second case, the activities have not been conducted because NURSPH and its staff have many research contracts and consultancies that pay them better than the project (a fact that also shows the lack of need for the strengthening project itself). An important component to strengthen the universities’ capacity to conduct research would be making available scholarships for graduate studies in top universities or recruiting graduates of these universities to teach at the university or to have FHI 360 local researchers teach in the universities, but that is beyond PROGRESS’s SOW. The general experience in the field has also shown that CB of academic and research institutions is more easily achieved when the TA is broad and not tied to a given topic like FP and when the activity is conducted as a dedicated project and not a small activity within a general project. Academic institutions are complex organizations, not easy to influence. A more productive form of CB is to use the project mechanism as a way to mentor professionals through hands-on experience. 14 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION CB in FP program M&E should continue to be a central component of the project, with MOH and FPTWG as the main targets of these activities. An important proportion of field support has been motivated by this task, and some local partners interviewed underlined the importance of receiving this type of assistance given their current lack of capacity to measure and evaluate their own program activities. Other respondents recommended that, in the future, USAID should engage in agreements with regional and local institutions and have them conduct OR, even though it was also mentioned that there were no African institutions that had the capacity to conduct research with the quality of U.S.-based institutions. Perhaps an alternative would be to structure future projects as an alliance between U.S.-based and African regional and local institutions, with the African institution housing a core team of international researchers to conduct project activities with local staff that would remain in the institution at the end of the USAID-funded project. Differentials in salary scales would have to be carefully considered for such a structure. A line of work that future projects should consider is offering local researchers and local private research firms an opportunity to compete for R, RU, and CB opportunities through solicitations under the prime cooperative agreement and include supportive supervision/monitoring as a way to strengthen their capacities. In several countries, for example in India, this type of firm provides services to different related development sectors, including the health sector. Improving their capacities through monitoring and supportive supervision has sector-wide implications. Individual researchers from the academic or public health sectors could also be linked to these activities, having a multiplier effect. There are strong institutions emerging in Africa that are currently very focused on human immunodeficiency virus/acquired immune deficiency syndrome (HIV/AIDS) and supported via PEPFAR. Some respondents felt these emerging institutions should be engaged systematically in FP and contraceptive-related program research so that they can contribute to the wider issues surrounding RH. Finally, one of the main achievements in this area has been the CIPs in Tanzania and Kenya, which are really a form of CB in planning, not in evaluation. However, continuation of this TA in planning should become an integral part of a future project because the process and final plan allow for the opportunity to introduce and utilize evidence-based practices. The CIPS seem to have had a large impact in terms of coordination and of funding for the FP project. Reportedly, the plan flagged critical needs, such as the commodity budget, and moved the Gates Foundation to consider Tanzania a priority country for advanced FP. Britain’s Department for International Development was also convinced to donate money. Thus, the development and application of a single tool focused attention, helped buy-in, and in general seems to have played an important role in repositioning FP in the countries where the activity was conducted. Recommendations for PROGRESS Regarding Intermediate Results As will be seen later in this section, PROGRESS will have an extremely busy last project year completing ongoing activities, so it is unlikely that it can start any new activities. Thus, the recommendations focus on the planned dissemination and utilization activities. The main recommendation is to emphasize local, targeted dissemination. PROGRESS already has local and Washington, DC-based dissemination plans in place and is working with PRH PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 15 champions to communicate to the field. Below are some additional suggestions for PROGRESS to prepare and implement in each country for a tailored dissemination/utilization plan: Finalize key policies and service delivery-related decisions that the project has been trying to influence. Clarify the process that needs to occur to arrive at the decision-making moment. Identify the individuals, working groups, and institutions who would make these decisions, as well as the issues that have prevented them from making the decisions. Identify like-minded, respected professional colleagues who could influence these decision makers. Based on this, a plan to influence the decision makers and those who influence them should be made, and strategies, messages, and materials to achieve this should be developed. As much as possible, PROGRESS should identify other individuals and organizations who have an interest in the same decisions and engage their help in conducting the dissemination plan, for example, other CAs and the FPTWG. The plan should also include specific messages or recommendations that can be discussed with each USAID Mission and with bilateral programs. PROGRESS could strengthen this targeted dissemination by contracting highly influential professionals during year five (for example, key members of medical and nursing associations, key researchers and university professors, heads of medical schools and institutions) to present extensive evidence regarding a given practice (for example, delivery of injectables by CHWs) to a variety of carefully selected (academic, policy, service delivery-related) audiences around the capital and other main cities. PROGRESS reports having already done this in countries such as Kenya, Uganda, and Zambia. Likewise, a tailored document of recommendations for each PEC- and SDI-managed activity could be developed, specifying in each case how the results of PROGRESS could help the project reach its own goals. These should be prepared in coordination with the OPRH project managers and champions. Since general dissemination materials will be needed to inform the different audiences, one dissemination material should be produced and disseminated for each legacy area and each IR. PROGRESS is well advanced in this task. PROGRESS is also planning Washington DC-based project dissemination meetings throughout the next year. One was already held in March 2012. It is recommended that messaging to partner organizations be tailored to how the body of results of this project can help their institutions reach their own goals. Too many of these types of end-of-project meetings are couched as reviews and only attract institutions already involved. The invitation list and meeting space should maximize engaged participation. PROGRESS is already well on its way in planning the next DC event. Conclusions and Recommendations for Future Projects Regarding PROGRESS’s Legacy Areas Structure Respondents found that reducing the general objective to improving access to methods and establishing the four legacy areas addressing key opportunities and needs was an effective way to concentrate efforts in a few key achievable objectives and avoid diluting scarce resources. 16 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION Having four legacy themes also helped communicate the project’s objectives, activities, and results. Even if in a new project the four legacy areas do not remain the same, PROGRESS and USAID staff recommended that the process to select the new objective and legacy areas be repeated. Regarding task-shifting, in some countries like Uganda, there is a clear recognition of the lack of human resources to achieve health goals and of task-shifting as a solution. This raises the question of whether USAID health projects in general should partner with other donors and local institutions to communicate the importance of task-shifting as a solution to apply to a larger set of health service delivery problems than FP. (This could be promoted at the country level by USAID Missions or by the Africa Bureau at the regional level.) A second conclusion is that while PROGRESS has conducted a few studies on how to strengthen CBFP, the actual work of strengthening the community-based health programs that can sustain task-shifting efforts has not been within the project´s mandate. In countries with weak CBFP programs, a new project could take one step back and, in alliance with service delivery projects, help programs achieve an operational performance that can sustain greater task-shifting efforts.7 Finally, in addition to documentation efforts to integrate the delivery LAPM to other methods, future work should pursue improving access to LAPM (especially permanent methods) as a project legacy theme that interacts with the four existing themes. LAPM are crucial for increasing sustainable contraceptive prevalence in Africa and decreasing unmet need, and several strategies that were used in Asia and Latin America could prove effective in Africa. Effective referral systems from CHW and health posts to LAPM, the acceptability of DMPA home self-injections, and the cost effectiveness of LAPM mobile strategies should be among the issues addressed by future projects. Regarding integration, the main question concerns the conditions under which integration beyond the health sector is sustainable in the long term. Provision of methods implies the existence of contraceptive/drug logistical systems or long-term partnerships with FP service providers who can sustain the effort, something that is not easily achievable. The task would seem to be understanding when and how it is convenient to provide on-site services (and which types of services) and when and how referrals should be made. Because of their ubiquity and continuous contact with mothers of children, the consultants believe future projects should assess if primary schools, in coordination with service delivery organizations, can help identify, counsel, and refer women interested in LAPM. A second line of work would be to continue to introduce contraception to health providers, including the current work on drugstores and pharmacies, as well as NGOs that provide health services [such as Conservation through Public Health [CTPH] in Uganda]. The research goal would be to provide guidance on which type of site would be more likely to help increase the number and density of sustainable, cost-effective service delivery outlets. Supply chain and contraceptive logistics should be among the systems studied. Even though method mix activities have been delayed, our understanding is that these delays are not a direct result of PROGRESS management and systems. PROGRESS is conducting these activities in partnership with the Gates Foundation and Pfizer, which has advantages but which 7 It may be the role of the new USAID Health System Strengthening Office to work on improving general health systems in the countries where PROGRESS has worked. A new project could expand its mandate to strengthen service statistics and test/adapt viable counseling, training, and monitoring and supervision systems. This work can be related to FP but also include other tasks. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 17 also tends to stop work whenever one of the partners has concerns about one aspect of the contraceptive introduction work. Depo-SubQ projects are finally beginning in Senegal and Uganda and will have to comply with a very tight schedule to finish by June 2013. These delays should be considered the cost of learning how to set up and work in public-private partnerships, which should be maintained in future projects because they can have a large impact on funding and can help increase the effectiveness of international aid by coordinating the actions of different donors. Finally, regarding crosscutting global leadership, it is essential for USAID worldwide interests to have a mechanism that can help respond to technical requests from USAID/Washington, WHO, and other multilateral and international organizations. To a large degree, these activities are medium- and long-term commitments that had previously been funded through CRTU and the previous USAID-funded OR programs, with additional financial support from The Eunice Kennedy Shriver National Institute of Child Health & Development through an interagency agreement. Given that U. S. government appropriations are providing more resources to the field and fewer allotments for core, it is essential for U.S. international assistance that centrally funded RTU projects are able to fund knowledge generation, knowledge management, and knowledge synthesis activities that have global significance. COUNTRY FOCUS AND FACTORS AFFECTING IT In contrast to the original proposal to focus activities in four countries, PROGRESS has conducted activities in 13 countries: 10 in sub-Saharan Africa (Ethiopia, Ghana, Kenya, Malawi, Nigeria, Rwanda, Senegal, Tanzania, Uganda, and Zambia); two in Asia (India and Pakistan); and one in the Caribbean (the Dominican Republic). In five of these 13 countries, PROGRESS has conducted only one activity (Ghana, Malawi, Nigeria, Pakistan and the Dominican Republic, which was incorporated into the project to help complete a research project begun under CRTU to assess differences in use-continuity rates of pills depending on the type of package used [21- vs. 28-pill package]). In these single-activity countries, investments through year four have been of US$250,000 or less, and in three of these cases, core funds were used to conduct the activity. The remaining eight countries are considered “major portfolio countries.”8 The main factor driving the selection of countries has been the interest of Missions in supporting the global research agenda and having infrastructure at the country level to build on and partners to work with. Focusing activities in a limited set of four countries to increase the probability of achieving lasting impact was an interesting concept but unfeasible in practice, given the core/field support funding mechanisms. Centrally managed FP OR projects are designed to advance the FP technical field and to provide the support needed by USAID Missions to design and implement country-specific strategies. Thus, the projects need to respond to Mission requests. Further, because Missions have the prerogative of accepting or rejecting the project’s presence in the country, and because the project’s presence adds to their management burden, the activities that the project conducts with core funds need to complement and add value to a Mission’s country strategy and relationship-building efforts. Finally, the project´s implementing 8 PROGRESS considers only seven to be major portfolio countries. Despite similarities in the level of funding and number of activities, Zambia is not counted as one because it does not have the same level of staff within the FHI 360 office working on PROGRESS and the work has been conducted mostly through consultants, ChildFund and FHI 360/Zambia staff. PROGRESS has not had the type of flexible funding to support active engagement in the country on a broader range of issues (beyond CBFP and PPFP) as in the seven other countries. However, for the purposes of our discussion and in the interest of simplification, the evaluation team considered eight countries to be this group. 18 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION partner has a financial interest in achieving the project’s ceiling, because the support to their organization is tied to the total amount of funding received from the different sources. Regarding interest from the Missions, Table 2 shows that 11 Missions have supported PROGRESS with an average of $1.1 million dollars over the four-year period (equivalent to $1.375 million dollars over a five-year period). Three countries have provided more than $1 million (Ethiopia, $3.8 million; India, $2.25 million; and Kenya, 1.3 $million); three other countries have provided more than $900,000 during the four-year period. Nevertheless, the idea of seeking to achieve a lasting result by concentrating efforts in a few countries continues to be of tremendous appeal. Perhaps in a new project, an alternative would be to select focus countries, as opportunities arise, for specific IRs or legacy areas. To a certain degree, PROGRESS is already implementing this approach. For example, most of the work in Ethiopia has evolved around CB for evaluation (quality assurance of task-shifting in the delivery of LAPM), a set of skills that were not present in the country. In a similar manner, future OR programs could focus, for example, on achieving greater access to CBFP and CBA2I in a reduced number of countries, while others could focus on task-shifting in IUD and implant provision, integration work, or results related to two legacy areas. This focused approach on one or a few legacy areas in each country could also help the Mission buy in and facilitate making agreements with service delivery CAs. Other factors influencing the selection of countries have been the legacy-related work conducted previously by the CRTU project (for example, injectables by CHWs in Uganda); the relationships that both USAID/Washington and FHI 360 staff had with USAID Missions and potential local partners that could influence the Missions or the MOH; and the opportunities to collaborate in countries where the governments were giving decisive support to FP. As it will be seen in the following section, the work of previous FHI awards has also been fundamental in determining the partnerships that PROGRESS has built. LENGTH OF AWARD As mentioned, PROGRESS is a five-year cooperative agreement. There is almost universal agreement that five-year cycles are insufficient for research projects because (1) the research￾to-utilization cycle is usually longer than five years; (2) research is limited to questions that can be answered in three years at most; (3) funding cycles make it very difficult to obtain field support for the first year and to obtain and spend it in the last year of the agreement; (4) lead time to identify and develop partnerships, obtain buy-in and support, and receive approvals is limited; and (5) there is an inefficiency in building a structure and activities in the first year in order to become fully operational for just three years and then closing everything for the fifth year, resulting in four years lost over a ten-year period, given two separate projects. 9 The recommendation is to plan for a seven-year agreement with a three-year extension on the basis of an assessment to review if the strategic concerns that motivated the agreement remain valid and if the project’s performance warrants the extension. This is evidenced by the difficulty in getting the Depo-SubQ in Uniject acceptability studies planned and negotiated with a 9 Limits research on new technologies, all demand-side questions that cannot be addressed, or questions relating to more complex service delivery strategies, including, for example, mobile sterilization services and other key concerns and opportunities to address population and RH challenges in Africa. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 19 spectrum of partners, the product available, approvals received, and the study fully implemented in the five-year cycle. PROGRESS TO MEET OBJECTIVES AND FACTORS AFFECTING IT Performance Monitoring Plan PROGRESS uses a PMP to track progress toward objectives. The PMP has two main components for tracking progress—a logic model with key indicators and the M&E framework, which defines specific objectives for each legacy area. Activities are mapped to these objectives via the PROGRESS Gap Analysis. The Gap Analysis indicates which activities address each of the objectives as laid out in the M&E framework. It serves as a signal for identifying potential gaps in programming. Currently, PROGRESS estimates that it is meeting or exceeding 21 of its 27 objectives as of year four and it plans to meet five of the remaining six objectives in year five. The one objective that is unlikely to be fully met is “follow-on research and evaluation conducted to support the introduction of tested models of expanding service delivery options,” since the time was not sufficient to both establish an evidence-based model for expanding service delivery and conduct further research and evaluation. Annex D shows the 27 objectives and identifies the six that have not been fully met to date according to PROGRESS’s self-analysis. According to PROGRESS, as of May 28, 2012, 15 of 37 research and evaluation activities were completed or in the final stages of paper-writing, and 11 are scheduled to finish data collection by December 2012 and will have the remaining six months of the award to conduct analysis and dissemination. Only a few (particularly the Depo-SubQ acceptability studies) have a tight schedule, with data collection ending in March 2013, leaving three months for data analysis and report writing. PROGRESS staff is confident that all activities will be completed by June 17, 2013. However, if a no-cost extension is administratively possible, it should be considered around four months before the scheduled date for the end of the project, since it is likely that the method mix research studies will be completed by then as well as the basic dissemination/utilization activities that would allow time to capitalize on the investment made in the research. Table 2 presents a more detailed view of the advancement and characteristics of the work that PROGRESS has conducted in the different countries. In this table, PROGRESS identifies 95 activities. Activities were classified according to the primary legacy area, but as explained before, some activities meet objectives under multiple legacy areas (for example, studies on drug shops). In the case of R, RU, and CB, the number of activities is greater than 95 because PROGRESS perceives several activities to fall under two or all three of these categories. For example, although there are 37 research and evaluation activities with a protocol, other activities contributing new knowledge, such as the Cochrane reviews, are included as research. R and RU activities being implemented with a specific CB partner were also counted as R and/or RU and/or CB. For the funding column, expenditures plus budgeted funds through the current workplan year were included. Only 16 of the 95 activities have been completed; however, those that are primarily RU or CB are designed to continue through the end of the project in June 2013. The activities are spread more or less evenly in the four legacy areas, with CB having the largest number of activities (29) and task-shifting the smallest number (17). In terms of IRs, the largest number fall in R, followed 20 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION by RU and CB. However, if, only those activities with a protocol (which also include evaluation activities) were considered research, the largest number of activities would fall under the RU IR. In terms of funding, about $33.3 million has been obligated; field support (FS) represents 36.6% of this amount. Kenya and Rwanda have investments of more than $4 million, with core funds representing roughly 75%; Ethiopia is close to $4 million, nearly all from FS. Factors explaining the large number of research studies to be completed in the following year include the initial delay because of the out of sync core vs. field support funding, the initial search for opportunities in different countries involving visits and discussions with a large number of potential partners, and the lengthy protocol approval processes, some beyond the control of the project or of USAID, such as local Institutional Review Board (IRB) reviews. Delays in method-mix activities have already been explained as a consequence of lack of actions or of restrictions set by partners. Other than this, PROGRESS staff attributes delays to a careful process, seeking to identify local needs and engage stakeholders from the beginning to generate strong ownership and facilitate greater utilization of results. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 21 Table 2: Number of Projects Conducted by PROGRESS By Region, Country, Legacy Area, Intermediate Result, and Type of Funding (As of April 2012) REGION/ COUNTRY NUMBER OF PROJECTS NUMBER OF PROJECTS BY LEGACY AREA NUMBER OF PROJECTS BY INTERMEDIATE RESULT AMOUNT OF FUNDING BY SOURCE Total Completed On￾going Task￾shifting Expanding SD Options Expanding Method Mix Capacity Building R & RU Cross Cutting Research Research Utilization Capacity Building Core Field Support WH BH WORLD￾WIDE 22 10 12 2 2 2 5 6 5 11 12 3 5,785,283 n/a USA 4 0 4 0 0 0 4 0 0 4 0 0 1,536,689 n/a AFRICA 4 1 3 1 0 0 0 3 0 0 4 2 803,860 350,000 Ethiopia 2 0 2 0 0 0 1 1 0 1 0 2 76,154 3,800,000 Ghana 1 0 1 0 1 0 0 0 0 1 0 0 206,495 0 Kenya 16 1 15 1 0 5 4 6 0 9 9 6 3,136,267 1,300,000 Malawi 1 1 0 1 0 0 0 0 0 1 0 0 178,543 207,000 Nigeria 1 0 1 1 0 0 0 0 0 0 1 0 250,000 Rwanda 10 1 9 3 2 0 2 3 0 7 3 3 3,580,926 870,000 Senegal 3 0 3 1 0 0 1 1 0 2 1 1 591,346 900,000 Tanzania 8 0 7 2 1 2 0 3 0 4 4 5 1,728,779 900,000 Uganda 6 0 6 2 0 1 1 2 0 2 4 2 1,254,083 950,000 Zambia 3 1 2 2 1 0 0 0 0 2 1 1 985,371 400,000 ASIA 0 0 0 0 0 0 0 0 0 0 0 0 n/a n/a India 12 1 11 1 2 1 4 4 0 7 5 3 840,709 2,250,000 Pakistan 1 0 1 0 0 0 1 0 0 1 0 0 194,535 0 LAC 0 0 0 0 0 0 0 0 0 0 0 0 n/a n/a Dominican Republic 1 0 1 0 0 0 1 0 0 1 0 0 194,106 0 TOTAL 95 16 79 17 9 11 24 29 5 53 44 28 21,093,146 12,177,000 22 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION Table 3. Main Achievements and Expected Achievements by Legacy Area and Intermediate Results LEGACY AREA INTERMEDIATE RESULT IR 1—Research IR 2—Research Utilization IR 3—Capacity Building Task￾shifting 1-Four research activities strengthened evidence for CBA2I in Rwanda, Senegal, Malawi, & Zambia. 2-Two research studies (ongoing) will contribute to sustainability of CBFP programs. 3-Research study in Tanzania found that women at Accredited Drug Dispensing Outlets can self-screen for contraindications to Combined Oral Contraceptives using a poster about as well as nurses can. 4-Provided TA for a regional CBFP assessment, led by ECSA, which identified 18 recommendations & has led to a resolution by Health Ministers to develop a standard practice package. 1-Led a global consultation with WHO & USAID; seven other international organizations endorsed the consultation findings: CHWs can safely and effectively administer injectable. 2-CBFP/CBA2I scale-up is underway in Nigeria, Rwanda, Uganda, & Zambia; progress towards CBA2I endorsement is underway in Kenya, Tanzania, & Senegal, including accomplished policy change in Uganda. 3-Developed the CBA2I Toolkit, available via K4Health, and the Invest-FP Calculator, a planning tool for expanded CBFP access. 4-Work towards making CBA2I a global standard of practice with RCQHC, IPPF, & MSI; for example, helped IPPF adapt their training materials to include CBA2I. 1-Building capacity of RCQHC as a regional master training for CBA2I/CBFP. 2-Building capacity of ChildFund to implement and monitor CBA2I in Zambia and Senegal. 3-With the Uganda MOH, building capacity at the district level to strengthen their CBFP program (Village Health Teams) and scale-up CBA2I. Integra￾tion 1-Testing m4Health interventions, including mobile phone-based job aids for CHWs & text message service for clients. 2-Development of an innovative FP text messaging service: m4RH, which was named one of Women Deliver 50 in 2012. 3-Three studies on integrating FP with non-health, including one with positive results on adding a health component to agricultural field days. 4-Five studies will yield results on postpartum FP, including showing the feasibility of postpartum intrauterine contraceptive device (PPIUCD) provision within district-level hospitals. 1-Synthesizing evidence & program experience on FP/immunization integration through a new GTL Working Group, briefs, & map, working with Maternal and Child Health Integration Program (MCHIP)f. 2-India MOH adopting recommendations from FP/immunization integration assessment & implementing them with continued PROGRESS support. 3-M4RH partners providing support by printing promotional materials, incorporating it into mass￾media FP campaigns, and/or funding Text To Change. 4-Non-health partners [(Land o’Lakes (L)& Green Belt Movement (GBM)] are exploring options to scale-up integration of FP within their programming. 1-CB partner CTPH has evolved to become the advocacy partner for the largest Population, Health and Environment (PHE) project in the Africa region (Lake Victoria Basin Initiative). 2-Building MOH capacity to manage m4Health programs by helping establish m4Health Community of Practice in Tanzania & Task Force in Kenya. 3-Building capacity of the GBM through partnerships with the BALANCED Project and training on M&E and data collection. 4-Building capacity of Rwanda MOH to expand PPIUCD in health facilities, through training of master trainers. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 23 Table 3. Main Achievements and Expected Achievements by Legacy Area and Intermediate Results LEGACY AREA INTERMEDIATE RESULT IR 1—Research IR 2—Research Utilization IR 3—Capacity Building Method Mix 1-Post-marketing research of Sino-Implant (II) underway in Kenya and Pakistan. 2-Depo-SubQ in Uniject acceptability studies, including within CBFP programs, underway in Uganda and Senegal. 3-Study on LNG-IUS in Kenya is thought to be the first offering of the product by public sector workers in Africa. 4-Significant funds have been leveraged from the Bill & Melinda Gates Foundation to support PROGRESS research on Sino-Implant (II) and Depo-SubQ in Uniject. 1-Rwanda MOH is supporting scale-up of no-scalpel vasectomy (NSV) with facial interposition & cautery; scale-up is underway in 20 of 30 districts. 2-India Ministry of Health and Family Welfare is scaling up Multiload 375 in the public sector, using recommendations from PROGRESS research. 3-IUD client card developed for Multiload 375 research has been adapted for nationwide routine use within the India public sector. 4-LAPM trainings in Kenya have included a focus on implants for youth, using PROGRESS findings. 1-Capacity building with Ethiopian Federal Ministry of Health (FMOH) has led to an increased demand for data on expansion of the method mix. 2-A national LAPM training plan for Kenya has been developed and is being implemented by PROGRESS and partners. 3-Building capacity of Rwandan MOH to scale-up & monitor NSV. Cross￾cutting 1-Conducting systematic reviews on FP for the Cochrane Collaboration. 2-Global technical leadership provided on High Impact Practices, monitoring scale￾up, and responding to hormonal contraception (HC)/HIV controversy. 3-Developed & analyzed a modified wealth index that allows us to demonstrate programmatic impact to vulnerable populations. 1-Development of FP CIPs supported in Tanzania and Kenya; CIP has become major advocacy tool & led to increased funding for FP in Tanzania; PROGRESS building capacity of MOH to monitor CIP. 2-FP Guidelines in Senegal, Tanzania, and Ethiopia were updated with evidence-based practices. 3-Supporting south-to-south learning with USAID/Africa Bureau, through regional meetings, case studies, and study tours. 1-Establishing regional Centers of Excellence for M&E in Ethiopia. 2-Through secondment of a staff person to the Senegal MOH, building capacity to integrate evidence-based practices into policies and programs. 3-NIMR won UNC MEASURE PRH grant, one of 5 proposals submitted to conduct FP research. 4-NIMR has become a member of the Tanzania FPTWG & working with FPTWG on developing a FP research agenda. 5-Activities & resolutions from the 2010 National Coordinating Agency for Population and Development Leaders’ Conference in Kenya, held with CB from PROGRESS, were included in a new Population Policy, 2011-2030. Based on Table 3, the evaluation team identified what it considered to be the top eight PROGRESS successes. However, it is difficult to highlight a few projects more than others 24 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION because the most important achievement of PROGRESS has been its focused design on a few basic strategies to achieve the general objective of increasing access to services and methods. 1. Community-based family planning: Evidence about the feasibility of delivering DMPA through CHWs in four countries supports statements for this practice by normative organizations, and arrangements to facilitate the scale-up of the practice in the future, including training toolkits, training CB of regional organizations, and the first stages of scaling-up in four countries. PROGRESS has also conducted OR to help assess the capacity of CHWs to absorb this task in the range of services they provide, given their workload and the capacity of women to self-screen for their own contraindications for use. Regarding this legacy area and these particular results, the only recommendation is that, given the variety of CHWs found around the world and in Africa, PROGRESS present the results according to the levels of education, training, and skills of the CHWs to which the results refer. Other regional work to identify needs and strengthen CBFP should also be commended and expanded. 2. Capacity building activities in Ethiopia: This work has provided quality assurance of LAPM methods provided by medical and non-medical staff; data to the MOH to plan and organize the National Family Planning Program’s activities, data to insure that training achieves appropriate performance of providers, management information systems, and M&E centers of excellence where data are collected and analysed; and actions organized to better respond to program needs. (The variety and scope of work in Rwanda is probably achieving the same result.) This set of activities is ingraining in the FMOH a problem-solving, data-base mentality, which is the essence of not only M&E and OR but also, more importantly, of good management. 3. Technology for health (mHealth): This project uses mobile telephones to provide information and facilitate the choice of methods of potential clients and to strengthen the service delivery skills of providers in collaboration with partners. The idea of taking advantage of the rapid increase in the availability of cell phones is very exciting and could have a large impact. Unfortunately, it has not been easy to assess which dependent variables are being measured or to what extent. Nor has it been easy to assess which preliminary results have been observed. In general, it is recommended that the annual and semiannual PROGRESS reports provide this type of information to allow interested professionals to gain a better sense of the activities and the extent of their impact without having to wait until the end of the project, when dissemination materials present the results. 5. Capacity building activities in Tanzania and Kenya: The CIPs in Tanzania and Kenya facilitated the collaboration of different service delivery activities; created a single plan of different evidence-based, high-impact practices; and helped secure funding from different donors and from local governments. PROGRESS should continue to remove barriers to their extended use in Kenya and Tanzania during this last year of activities, and if possible future projects should consider repeating the experience in other countries. 6. Expanding the method mix: The work conducted in the method-mix legacy area is increasing access to LAPM in different countries and among different user segments. These include Multiload IUD, DepoProvera in Uniject, a levonorgestrel-releasing IUD system, an advanced vasectomy technique, and implants for youth. Future projects should continue this work and add to it the documentation, careful evaluation, and quality assurance of task-shifting efforts in the delivery of new and underutilized LAPM that other service delivery organizations and projects are conducting in Africa, with emphasis on permanent methods. 7. Postpartum family planning: Five studies tested the delivery of postpartum IUD services in district hospitals. These studies will help determine how to better adapt this strategy to sub- PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 25 Saharan Africa’s conditions and give local partners detailed experience under controlled conditions. In general, future projects should continue testing and adapting to sub-Saharan conditions the service delivery strategies that have proven effective in other regions of the world, particularly of LAPM, and providing hands-on experience and the opportunity to closely observe the set-up, functioning, and results of these programs. It should be remembered that in other regions, institutionalization of such programs in some countries took up to 20 years. 8. Family planning integration with the non-health sector: The increasing number and density of FP service delivery points in rural areas in sub-Saharan Africa is of great importance. If proven effective under PROGRESS, the main question future awards should address, regarding work with agricultural and other types of organizations, is to what extent these interventions can be institutionalized and how long will they remain delivering methods and information. Future projects should also explore the feasibility of using other “beyond health” sites (in particular, primary schools) for service delivery and should assess if integration beyond the health sector would have greater impact than adding FP methods and services in organizations already offering some type of health services or products (for example, child survival services, drugstores, etc.). 26 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 27 IV. PARTNERSHIPS COUNTRY, REGIONAL, GLOBAL PROGRESS has engaged in partnerships with a wide range of organizations to conduct R, RU, and CB activities. Partnerships are an essential component of the project structure, and the work of PROGRESS links to institutions working at the global, regional, and country levels. In general terms, it is possible to distinguish between implementation partners, which have been the recipients of PROGRESS technical and financial assistance and which conduct interventions to provide services to local populations, and collaborating agencies, which partner with PROGRESS to strengthen the capacities of the local implementing partners. This section analyzes and synthesizes what respondents had to say about partnerships in the large number of interviews done for the evaluation with in-country and local staff members of national and international agencies as well as with staff members of local implementing partners. These respondents spoke about their experiences with FHI 360 and the PROGRESS project. The original design of the project did not include a static consortium of partners. Instead, FHI 360’s response to the request for applications presented letters of support from various institutions and proposed flexible and opportunistic engagement of partners to respond to evolving thematic areas and country needs. The global and regional partnerships, such as those with WHO and ECSA, were built on their longstanding relationships with FHI 360. PROGRESS provided a platform and funding to execute mutually desirable work that fit the goals of both PROGRESS and the partner institutions. The country-level partnerships were built from new and existing FHI 360 office relationships, according to the needs of the proposed activity and to the available staff and expertise of potential partners in a given country. These partnerships have ranged from formal contractual and agreement arrangements with local and international service delivery partners and local research organizations to more informal partnerships with national working team members and other interested institutions. PROGRESS’s major portfolio countries demonstrate a range of 6 to 21 partners to execute country activities. These partnerships have been key to designing and executing interventions, undertaking research, changing policy, and moving to scale-up. National Implementation Partnerships National implementation partners: MOHs have been the most important implementation partners of the program, bringing research questions and RU ideas to PROGRESS and moving results to regional and national practice (for example, policy/guideline changes in Uganda and Rwanda). Other national-level partners include such organizations as the Network of Entrepreneurship & Economic Development (NEED), an Indian microfinance project that is working with PROGRESS to train village health guides to provide FP messages and referrals, and the Green Belt Movement (GBM), a well-known conservation group in Kenya that uses volunteers to provide population, health, and environment (PHE) messages and FP referrals. PROGRESS has engaged a diverse group of partners, in particular in the non-health sector. 28 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION Partnerships with Collaborating Agencies PROGRESS has collaborated with local, national, regional, and international organizations to assist service delivery organizations in providing better and more effective services. Below are a few highlighted examples of the types of partnerships in which PROGRESS has engaged: Local activities: Partners have made contributions to all PROGRESS activities. These have included training and TA (for example, JHIPIEGO and the Institute for Reproductive Health); advocacy and RU (Marie Stopes International); research (TNS Research in India); and implementation of interventions (Text for Change and Pathfinder). Networks: PROGRESS has built up networks for provision of TA, dissemination of results, and recommendations for policy changes in several countries by establishing partnerships with regional organizations such as ECSA, the Regional Center for Quality of Health Care (RCQHC), and the Millennium Villages Project. International partnerships with USAID, the Gates Foundation, PATH, and Pfizer have been essential for contraceptive technology/method mix work. Each partner has brought complementary financial and technical elements to share the costs and work for acceptability research and moving subcutaneous DMPA in Uniject. Global Technical Leadership (GTL): PROGRESS support of a technical consultation in partnership with WHO led to WHO’s endorsement of the provision of injectable contraceptives through CHWs, which later gained the support of IPPF, ECSA, RCQHC, and Management Systems International (MSI) to make CBA2I a global standard of practice recommended by all key normative bodies. Each country portfolio engaged a spectrum of partners as needed to accomplish R, RU, and CB activities. Table 4 profiles Rwanda and showcases the detailed involvement of partner organizations and institutions to achieve PROGRESS objectives. These partnerships are essential to complete PROGRESS’s work, leveraging technical expertise, intervention experience, and additional resources and providing a platform for country ownership of the research results. Because the work of partners is essential, their motivation and timely completion of roles and responsibilities are also essential, though mostly beyond PROGRESS’s control. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 29 Table 4. Example Partnership Country Profile: Rwanda Partner Contribution Comments MOH CBFP: Collaborated on phased scale-up plan for CHWs to provide pills, injectables, and Standard Days Method with a focus on training and M&E systems. Scale-up of Vasectomy: MOH requested TA from FHI 360 as it scales up the availability of vasectomy services across the country. An MOH staff member, a dynamic champion for NSV, is leading this effort. PROGRESS works with the MOH to develop and implement a quality assurance plan. Also working collaboratively to identify indicators to measure quality of care and to develop and implement a monitoring plan, as well as identify and document barriers to and facilitators for choosing vasectomy at structural as well as personal levels. PROGRESS originally funded the adaptation/development of training materials for NSV with cautery and facial interposition, based on EngenderHealth materials and those from an expert consultant who conducted the first training of trainers. Barriers to Expanded Contraceptive Use in Rwanda: The study was requested by the MOH. MOH staff served as co-principal investigators (PIs) and were involved in key activities, including recruitment of data collectors, feedback during site selection, interpretation of results, and a dissemination workshop. Supply-Side Barriers to Expanded Use of Contraception in Rwanda: Study done at the request of MOH. MOH staff are co-PIs and have contributed to study objectives and protocol development. Workload of CHWs and FP uptake study: MOH staff are co-PIs and have contributed to study objectives, protocol development, site selection, etc. Examining the Feasibility and Acceptability of Postpartum IUCD (PPIUCD) Services: The study has a co-PI from the Ministry and various MOH personnel have been involved in key activities, including an initial study tour to Kenya, visits to study sites and mid-study stakeholder workshop; and study updates have been provided at FPTWG meetings. The study has also supported the maternity unit at Muhima Hospital in becoming a national training center for PPIUCD services. Postpartum FP Integration with Immunization Study: MOH has been a partner in developing and implementing the intervention strategy, training health care providers and providing follow-up for the intervention activities. Dr. Fidèle Ngabo of the MOH is co-PI for the study. Ministries of Health have been the most important implementation partners of the program, bringing research questions and RU ideas to PROGRESS and moving results to regional and national practice. 30 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION Table 4. Example Partnership Country Profile: Rwanda Partner Contribution Comments MOH is a key target for capacity building activities such as TA in RU and study tours. PROGRESS provided support to the MOH for adolescent RH assessment, messaging on return to fertility for clients and providers, assessment of implant and IUCD removals and developed case study with the MOH as a global success story. USAID Rwanda The Mission promotes PROGRESS in country as a resource for evidence-based policies. PROGRESS kept the Mission informed and coordinated its activities well with other CAs. USAID Rwanda has invested $870,000 in field support. The USAID Missions appreciated the strong relationships that FHI 360 established with the national MOH and regarded them as a key element that facilitated the research approval process, discussion and consideration of policy changes, and ultimate utilization of research results. Family Planning Technical Working Group (FPTWG) PROGRESS participated in this MOH-led working group. PROGRESS study findings were presented to the FPTWG for consideration and utilization in the development of the national FP workplan. The FPTWG received CB through TA, such as the Technical Update on postpartum FP. Engaging with the FPTWG helps create a sense of ownership of results among all stakeholders. It also provides opportunities for identifying TA and CB needs of partners in the country and for pinpointing specific activities to provide them. For all these reasons, we believe that the next project should promote a functioning FPTWG in all countries with a project portfolio. School of Public Health Social and Cultural Barriers to Expanded Contraceptive Use in Rwanda: subagreement to collect data (execution of study and CB). CB recipient, two-year plan including training and equipment. Highlighted some of the challenges of CB for an independent institution, as the School of Public Health had other priorities than the goals of PROGRESS. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 31 Table 4. Example Partnership Country Profile: Rwanda Partner Contribution Comments Jhpiego Examining the Feasibility and Acceptability of PPIUCD Services: This study was built on the work and experience of Jhpiego in PPIUCD. PROGRESS gave Jhpiego a subcontract to provide clinical training for PPIUCD. Jhpiego oversaw the trainers of service providers at the facility level, provided clinical support supervision, co￾convened a technical update meeting for the MOH to share current information on postpartum FP, adapted the training curriculum, and developed client brochures on FP methods, including PPIUCD. Technical Assistance for Research Utilization: Jhpiego staff participated in a study tour for MOH to observe the successful Jhpiego ACCESS program in Kenya. Strong partnership with an organization that has experience with the intervention. IntraHealth Scale-up of Vasectomy: This intervention is based on IntraHealth’s work. IntraHealth organized vasectomy trainings and produced tools for national scale-up, which were built upon for the NSV with cautery and facial interposition work. The vasectomy training involves a team in each district composed of a medical doctor, anesthesiologist, and nurse. IntraHealth provided leveraged funding for provider training and laboratory technician training for spermagrams. Social and Cultural Barriers to Expanded Contraceptive Use in Rwanda: Collaborated to disseminate key study findings to Rwandan journalists and Parliamentarians. IntraHealth (and UNFPA) contributed financially to the study and its dissemination. PPIUCD: IntraHealth was involved in design of the questionnaire. Strong partnership with an organization that has experience with the intervention. 32 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION Attributes of the Most Successful Partnerships Global and regional partners: Successful partnerships were built on the longstanding reputation and technical expertise FHI 360 brings to any activity, plus the resources that the PROGRESS project was able to provide for common interests. Country-level partners: – Organizations that had a subcontract or subagreement relationship felt, overall, that the experience was positive. – The implementing partner bringing funds to the partnership is an important incentive for partners to participate and keep to deadlines. Partners need to see return on investments and share costs to not strain resources. “You cannot come empty-handed” to research or other types of partnerships. – Partners need to identify a common client, and involvement in the partnership needs to add value to what the partners are doing for this client. Doing a favor for each other should not be the basis of the partnership. While partners have good intentions to carry out interventions/activities, they will be the first items eliminated if the partner organization must choose between competing priorities. – Non-financial partnerships require that common interests are identified and non￾financial incentives exist. In Tanzania, multiple stakeholders partnered to do a universal review of gaps and how to leverage partner resources. The PROGRESS-led stakeholders took a systems-level approach to identifying gaps, which involved the work of all participants. – Flexibility is needed. Priorities of partner organizations change over time, leadership changes, and nothing remains static. Doing activities outside the strict program SOW can be essential to build and sustain the engagement of key partners, such as the Mission and MOH. Satisfaction with PROGRESS’s Partnerships, Attributes Related to Satisfaction, and Components that Facilitated Implementation U.S. government and global-level partners, including NIH and WHO, expressed satisfaction with the work of PROGRESS. There were longstanding relationships between FHI 360 and these institutions with common goals. PROGRESS brought important technical and financial resources for mutually beneficial work. Missions: The Missions in most countries found that FHI 360 was a strong supportive partner to other CAs and to the MOH. The USAID Missions appreciated the strong relationships that FHI 360 established with the national MOHs and regarded them as a key element that facilitated the research approval process, discussion and consideration of policy changes, and ultimate utilization of research results. MOHs: Key activities for building partnerships with the MOHs included participation in technical working groups, designing and implementing research with MOH staff as co-investigators, and executing CB activities for MOH staff. The MOHs noted the most satisfaction with the CB activities (Ethiopia, Senegal) and were concerned in countries where CB focused on external institutions and not the MOH (Tanzania). PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 33 Country-level collaborating agencies: Most of the external partners noted being satisfied, overall, with the partnership and found FHI 360 technical support and capacity sound. Partnerships were often built upon requests from the Mission or the MOH to work on an activity with specific partners (for example in Ethiopia). The connection was successful because of the authority of the Mission and MOH over the partner institution. To build partnerships and capacity, PROGRESS engaged local research institutions and made them co-investigators on research protocols. The success of this strategy was to build the capacity of local investigators while simultaneously establishing local ownership of the data and local incentives for seeing the results being used to improve services. Buy-In and Implementation Obstacles Agendas and interests of potential capacity building partners: Capacity building for local research institutions was a key area in the PROGRESS SOW and one of the most challenging. Most of the original partnerships proposed did not move forward due to their organizational priorities and interests, workloads, alternative opportunities, perceived funding needs, and other reasons. New organizations were engaged, but they were few in number (NIMR, School of Public Health Rwanda) and limited in scope because of each institution’s own goals and competing interests. Missions’ interest in obtaining access to general FHI 360 expertise: When USAID Missions suggested PROGRESS partnerships as a mechanism to access FHI 360 expertise as a whole and not to work on a project theme, the partnerships often did not work out. Often the bilateral services partner had needs that didn’t match PROGRESS’s mandate, for example, commodity logistics and HIV. As far as the project knows, there were no hard feelings about its inability to respond to requests outside of the thematic areas. Local collaborating partner concerns: Sharing operational practices, data, etc., with FHI 360 for research and M&E purposes opens local organizations to scrutiny and showcases weaknesses to the MOH and donors. One Mission had to request that the MOH notify partners and field offices to work with FHI 360 and share data, information, etc. Once the MOH directed field partners and FHI 360 staff assured field offices that the research was not intended as a punitive exercise, the partnership worked well. NGO competition: PROGRESS country partners are often organizations FHI 360 competes with for local service delivery bids. Sharing intervention details and organization information and helping foster relationships with local stakeholders and the MOH through a research activity is perceived as problematic by service delivery organizations. Although there is an acknowledgement that PROGRESS can help answer important questions and bring needed resources to disseminate their work, NGO partners noted having to hold back to protect their organization from future competition with FHI 360 and having to justify the partnership with their own senior leadership. Feelings that implementers do not receive due credit for innovations. Four partner respondents complained that FHI staff often described innovations as their own, when in fact it was the partner’s, which creates bitterness. In PROGRESS program documents, credit for the intervention and innovation is often lacking. 34 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION Administrative barriers. Partners expressed concern about their perception of FHI 360’s lengthy contracts and grants review and approval processes. All said, it was worth it in the end to stick to the agreement, but many noted that it wasted time and relationship goodwill was compromised. Bringing on new partners. There were a few USAID/Washington and NGO partners who expressed a sense that PROGRESS’s strong and smart staff could achieve greater levels of leadership by bringing forward more novel areas of research and doing more to spearhead the global technical agenda. Four respondents noted that PROGRESS was not as visible as past research programs and that the vision was not clearly branded and seen in FP research forums. This lack of visibility was viewed as a barrier for new partners to work with PROGRESS and for current partners to invest in more work. Comments on or “Nested” in Service Delivery Projects During the course of the evaluation, the team considered the value of a separate OR agreement engaging service delivery partners, like PROGRESS, versus a more comprehensive agreement with research nested within a service delivery project, such as EngenderHealth’s RESPOND project. The researchers who commented on the comprehensive design were not in favor for the following reasons: 1. If funds are limited in a service delivery project, research is the first to be eliminated. Research is seen as very expensive relative to the ongoing program-strengthening activities. 2. It is often difficult to engage with the service delivery team on the requirements of the research. 3. The service delivery team does not retain objectivity about the findings. Successful Elements of the PROGRESS Hand-Off Strategy Hand-off for sustainable outcomes of the R, RU, and CB activities is different depending on the activity/model that is targeted to continue and the capacity of the partners involved. Successful hand-off for a new method/service is clearer. When introducing CHWs to providing injectables, the definition of a successful hand-off is for this service to be available in-country after the project ends. For other work, the success is less clearly measurable but just as important to capture; for example, for a study on CHW workload, successful hand-off includes the data successfully influencing regional and national plans for CHW budgets, management, and mentorship. Continuation requires the project and, at some point, other partners to continue doggedly pushing until the service and process are routine in the country. FHI 360’s strategy is to include planning the end from the very beginning of any activity for any level of partner—local, regional, or international. Hand-off in most partnerships was described as including partners from the very beginning of the planning process. FHI 360 described a very thoughtful strategy acknowledging that hand-off is a process, not an event, a goal that influences and shapes the strategy from the beginning of the activity. Successful hand-off included institutionalization of practices by global normative bodies, global and regional TA mechanisms, and country/program ownership. It happens at various PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 35 levels, must be a mutually beneficial relationship, and takes resources to thoughtfully and successfully execute. It is clear that the hand-off process was considered for the PROGRESS activities and carried out to various degrees. The most obviously successful activity was the technical guidance and endorsement that WHO, with FHI 360 and partners, delivered on community-based distribution of injectables. The successful engagement and endorsement from a normative body allows MOHs politically to move this intervention into their country programs. Some hand-off strategies were planned but there was variability in completion. For example, respondents around the Land O’ Lakes intervention and the work in Malawi noted that there was no additional funding to continue follow-up and what one respondent termed “doggedly push” for the advancement of the intervention. Additional funding to work with Land O’ Lakes scale-up is under discussion. FHI 360 is clearly aware of what successful hand-off should look like but did not always have the resources to accomplish it. Successful RU and hand-off by the research organization were described by Mission staff and service delivery partners as working directly with MOH and other partners to bring a successful policy change and inform the scale-up design with the operational experience of executing the study. The “real” scale-up into services should be the work of service delivery partners. New project design must incorporate resources and support for the process of hand-off. “Research institutions do need to go beyond evidence generation and need to have the funding to do this, at least to first stage.” A key component of PROGRESS’s research coordination, utilization, and dissemination in Ethiopia, Kenya, Rwanda, and Tanzania has been the FPTWGs, where all stakeholders meet regularly to discuss needs, opportunities, advances, and results under the leadership of the MOH and, in some cases, with the support of PROGRESS. Respondents in all countries constantly mentioned that it was through these FPTWGs that they were able to influence the type of R, RU and CB activities that were done and learned about the results of the projects and the way they had been utilized. The FPTWGs facilitate responding to the needs perceived by the MOH and help create a sense of ownership of results among all stakeholders. They also provides opportunities for identifying technical assistance and CB needs of partners in the country and for pinpointing specific activities to provide them. For all these reasons, USAID should continue to support functioning FPTWGs in all countries with a project portfolio. International technical exchange meetings and study tours to countries where a given intervention is implemented were also reported to be a very effective means for achieving policy change and creating opportunities for pilot research. We recommend continuing and expanding this practice in future awards. As noted in section III under “Research Utilization,” additional facilitation and encouragement by USAID for service delivery CAs to pick up PROGRESS results would help final utilization and integration into health systems and maximize the U.S. government 36 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION investment in research. For example, CBA2I could be made a required standard of practice within all USAID-funded FP bilaterals. Partnership Recommendations PROGRESS attempted both institutional and individual research CB activities; In future, institutional CB activities should focus on MOH and service delivery organizations and not academic and research institutions, which require broader support than OR projects can afford. Organizational CB can be affected through individual CB of the organization’s staff, making them co-principal investigators in studies, and mentoring them throughout the research process. Service delivery CA contracts and agreements need to include policy change and service delivery innovation goals to promote utilization of research results. These goals should be flexible and revised in annual and midterm reviews according to circumstances. Research projects should help document activities and strategies conducted by CAs. The next flagship research project should develop a specific branding strategy with resources set aside to participate in more FP research forums and meetings in Washington DC. While competition between CAs will always exist, particularly if the institution doing the research also has a service delivery arm, some of the reluctance to partner might be overcome if there were more distinction in being involved with the flagship OR project. The next project should produce “authorship” guidelines, highlighting the need to give due credit to partners in publications and presentations and to discuss the guidelines with partners from the beginning of the collaborative relationship. Future research projects should be designed as separate agreements that are not nested in a service delivery project. The PROGRESS team recommends that a minimum amount of funds should be budgeted for partner organizations to consistently engage and stick to timelines and other agreed-upon parameters. Future project designs should continue to incorporate resources and support for the process of hand-off. One partner recommends signing a memorandum of understanding to improve work and communication at the beginning of a new project, especially for relationships without financial ties. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 37 V. SUPPORT TO THE FIELD The key objectives of centrally managed FP OR programs are to (1) advance the FP global technical field and (2) ensure that USAID field Missions have the support needed to execute country-specific FP strategies. Thus, a key objective of this evaluation is to assess the adequacy of the PROGRESS mechanism to meet the needs of USAID Missions. To do so, the evaluation team interviewed by phone USAID Mission staff members from six countries with a significant PROGRESS portfolio (Ethiopia, India, Kenya, Rwanda, Tanzania, and Uganda) and two from countries with fewer projects (Malawi and Zambia) by means of a questionnaire sent and returned via e-mail. In addition, the staff of FHI 360 and USAID OPRH commented on the buy￾in process, implementation, and needs. WHERE DOES PROGRESS WORK? Figure 3. Progress Areas by Portfolio Activity These inputs reflect the respondents’ perceptions of the Missions’ motives and expectations, as well as the bias resulting from including in the sample mostly staff members from Missions that have given PROGRESS significant support. Also, some USAID Mission staff respondents were new to their positions supporting the PROGRESS activities (e.g., Uganda, Ethiopia, and Rwanda) and did not have the complete historical picture, while some of the USAID activity managers with a historical perspective who were interviewed had moved on to other roles inside and outside USAID and did not have a strong sense of the future direction of the current Mission program. 38 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION Main Reasons Perceived to Motivate Buy-In and Attributes Sought by Missions In all cases, the main reason for buying into PROGRESS was the need for the services it could provide. However, the existence of this need was mediated mostly by interpersonal factors. As expressed by the interviewees, the main reasons for buy-in were the following: 1. Past work experience or relationship with FHI 360, particularly with the CRTU project (e.g., Kenya and Uganda). In this case, FHI 360 was already a member of the team of providers, the Mission had a good opinion of its work, and PROGRESS provided a mechanism to help continue the line of work that FHI 360 had already begun under CRTU. These Missions were also very supportive of CRTU and PROGRESS research and its practical applications. FHI 360 staff, such as Dr. Solomon, Dr. Maggwa, and Dr. Akol (with their entire back-up teams) made themselves very available to these Missions for any requests. 2. Existing relationships between Mission and OPRH staff members (e.g., India and Rwanda). OPRH staff brokered the introduction of PROGRESS, particularly with Dr. Maggwa, the project director. In both cases, the relationship served to obtain meetings in which PROGRESS staff were able to pitch their services and secure funding based on the Missions’ assessed research and evaluation needs and the skills and services offered by PROGRESS. Facilitation by OPHR staff was particularly helpful to pave the path when the proposed work was lacking precedence in the organization or country. “Get that door open”. 3. The Mission’s country program needs for a particular set of skills was not available in the country and the level of work did not justify a separate agreement. The two main cases of this were Ethiopia and Senegal. In both cases, the Missions wanted an independent quality assurance service of the delivery of Implanon and DMPA (respectively) by CHWs, which was also part of the legacy area. In the case of Ethiopia, this opportunity was later expanded to include other CB needs of the MOH in M&E. 4. The government or the MOH asked for assistance in a given area and the Mission searched for the provider of the service and no suitable local providers were found. (For example, in India, local research and evaluation organizations were perceived as having their own agenda that did not include contraceptive introduction.) In several cases, the Mission said the MOH drove the agenda in-country and the Mission followed the MOH, because it would facilitate use of the results for policy making. Key attributes of PROGRESS that seem to have been determinant in facilitating the buy-ins from different Missions were the following: FHI 360 in-country presence: All countries included in our sample already had an FHI 360 office with a base staff that could start providing a level of effort (LOE) for the PROGRESS project, and some interviewees mentioned this as a requirement for effective assistance. However, the drawback of this is that work has to be built on existing platforms and include countries that most likely have already engaged in some of this work in the past. Excellent reputation: The Missions had already had some success with FHI 360’s past work in the current country or in a previous Mission placement and had established a level of trust. Timing: The notice or visit came when the Missions were looking for the service. This was often described as an ad hoc request from the MOH that the Mission was looking to support. Central mechanism: PROGRESS offered a high level of expertise in an area not covered by local organizations but needed by the Mission. In Ethiopia, for example, the Mission felt it PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 39 needed quality assurance follow-up in the delivery of new contraceptive technology, independent from the CAs supporting the interventions, and there were no local organizations with the needed skills; in Rwanda, PROGRESS was assigned the task of promoting evidence-based practices. The main themes that have motivated funding by Missions into PROGRESS have been Supply-side and demand-side diagnostic studies to set the FP agenda (e.g., the study on reasons of non-use in Rwanda) and formative research to inform decision-making and implementation (“it is the way to go”) in Zambia. Mission’s need for quality assurance of delivery of a new or sensitive method (e.g., Implanon in Ethiopia or DMPA in Senegal by CHWs). Surveillance of new method in the country (e.g., Multiload 375 IUD in India). MOH need for capacity building in FP/reproductive health M&E (Ethiopia). Satisfaction with PROGRESS’s Services, Key Attributes Related to Satisfaction, and Key Components that Facilitated Implementation In all cases, Mission staff reported high satisfaction with the services provided by PROGRESS. Key attributes of the services provided include: Quality work completed: the CIP was described as ”the watermark of FP work for the Mission in that year”; study for reasons for non-use in Rwanda were perceived to have set the agenda of the FP program; CBA2I work by FHI 360 in Uganda was influential in helping set policy there; “have provided information on evidence-based practices”; “have identified needs”; ”help understand how new technology is utilized,” “how new contraceptive technology is utilized”; “did path-breaking work with government”; the Ethiopia and India Missions said PROGRESS was directly influencing the design of country bilaterals. Flexibility of the program to take on a broad range of activities. Easy access, good and frequent communication of FHI 360 field offices with Mission, MOH, and other partners. Responsiveness to Mission needs and to MOH requests. Ease of mechanism for field support. Excellent relationships with all partners in general and with the MOH in particular, a factor which was perceived as being as important as data. Credibility, perceived competence. Playing as a member of a team for the greater cause in the country. Good reporting. Among things that could have been better or could improve, respondents noted the following: 40 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION Staff too lean/need more staff at the country and local level, both in financial and program management. Need staff “stationed at least part of the time in the field, not in . . . [the capital city].” Slow initial start-up of activities; late start; core and field support funding decision timing not in sync; lower funding than expected made FHI 360 lose one year of field support. Should do more in-country dissemination and advocacy for scaling-up. Advocacy with donor stakeholder around ethics approvals and the problem with some local IRBs that are requiring high fees. Lack of local researchers, need for CB of local research institutions. This finding conflicts with the evaluation recommendation that CB of institutions should be separate from a research project. We note that there is a need for CB of research institutions, but we believe that the experience shows that FP OR projects are not the best mechanism to help academic institutions acquire greater capacity. The capacity of individual researchers is achieved by providing hands-on experience and providing continuous TA. Buy-In and Implementation Obstacles Staff turnover within the Mission is common, so intense advocacy is required to build support for a centrally managed program for the new activity manager. FHI 360 field offices were reported as the most intensely engaged in rebuilding the support and approvals to continue (e.g., India). Staff turnover creates uncertainty about future funding from the Mission. RTU team members need to be aware of these country transitions to bring up to speed new activity managers and persuade to prioritize funding. There is a perception that PROGRESS project branding is limited. PROGRESS was described by Mission staff as a funding mechanism offered by FHI 360 with no image of its own. This becomes a problem when attracting partners as project involvement does not bring its own distinction. Mission staff lacks proper training and appreciation for (a) what constitutes evidence in the social sciences; (b) what a research program does for FP programs; (c) the importance of data quality; and (d) the time needed to achieve quality documentation and research. RTU should consider direct CB of Mission staff on the research process, imparting a deep understanding of the time required to execute a research protocol and the strict approval and documentation requirements. This would further empower the Mission to plan internally and communicate realistic expectations with the MOH on the timeline/approvals, etc., that are mandatory when executing research. There is a lack of guidance from OPRH on what constitutes evidence and evidence-based programs and how to make decisions on this. This is all the more pressing given the existence of Pathfinder´s Evidence to Action Program, which has as one of its goals scaling￾up of evidence-based practices. There is a perception that the PROGRESS project is scattered and without focus. One respondent called it a “scattershot” feel to the project, with no easily identified central theme being communicated. Four respondents remarked on the lack of a central PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 41 PROGRESS identity, and while the legacy areas were useful for PROGRESS planning, this focus was perceived as not well communicated to the field at large. Partners and FHI 360 offices asked that USAID RTU staff to be more engaged and communicate more with Missions. Field and Core Support Funding The investment of core funds into countries has led to increasing Mission funding through PROGRESS, demonstrating an important “multiplier” effect on core funds. PROGRESS has been able to leverage funds from 10 Missions (an 11th one pending) and the Africa Bureau for a current total of $12.3 million. Several issues affect this process, including the timing of a centrally funded award with Mission funding cycles and the ability to provide funds for priority activities or to advance fund activities of interest to a Mission. The PROGRESS project was awarded in sync with the HQ budget cycle and out of sync with the budgeting cycle of USAID Missions, thereby making it difficult for them to invest in the project in the first year of operation. Missions visited during the assessment/introductory visits by the PROGRESS management team identified priority activities that they would be interested in implementing and receiving support from PROGRESS. Some Missions requested core funds to fully fund activities of local importance, with a promise of future allocation of field support for expansion or follow-on activities. In other instances, the request was for PROGRESS to provide some “seed money” or to forward fund activities, while the Mission looked for its own resources to take full responsibility for funding in future funding cycles. The strategic decision of investing initial core funds to support areas of interest of Missions helped to demonstrate to the Missions that PROGRESS was ready and able to assist in responding to their needs and enabled the project to build relationships as well as recognition by the Missions of the capacity that existed within the PROGRESS project. Mission buy-in has varied greatly. There is continued need for core funds and the flexibility to use these funds to leverage additional resources. Points that Need Discussion or Clarification in Terms of Future Projects There is need for a clear statement on appropriate tasks for the program. The current HIP gives no advantage to R/RU projects over service delivery projects. The PROGRESS traits are not stated or highlighted in a way to interest Mission staff or relate to their needs or interests. Communication points need clarification and targeted messaging for Missions. The strategy or guidance for using core funds is not clear. Should it be used to reward field support grants or to induce them? Or should it be basically used to try to develop new and exciting ideas? Or do all of these? Should use of core funds be tied to field support grants? For example, the Ethiopia Mission stated it would like to see more core funding; in other countries, like Rwanda, one sees one-third FS, two-thirds core. 42 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION Whose research questions should be addressed? Are the Mission staff and partners trained to frame research questions? Where should research questions come from? Respondents note that in-country research questions are coming from the MOH. Where should core funds be used: in exciting countries with a lot of activities and many confounding factors, or in countries with few activities but without a strong commitment? Recommendations for Future Research Programs The following lists all the responses on important themes for the future. The responses from Missions are disaggregated from those of other partners. Most respondents answered that the current thematic areas were still very important and that there was still work to be done that will depend on the final outcome of the current research. In Ethiopia, for example, all respondents in the country perceived the need to continue the current work for several years to address existing challenges and FP program goals. In Rwanda, concern was expressed about the potential end of PROGRESS given the need for an organization focusing on evidence, the role assigned to the project. This list below is ideas of other unanswered questions/thematic areas that are rising in importance (additional detail from FHI 360 in Annex D): Mission Response: Continue fulfilling the same role/doing the same work. Integrate HIV and FP work. Improve FP access for youth. Integrate FP with immunization services. Expand the mandate to include FP and aternal and child health (MCH), allowing Missions access to FHI 360 resources on R, RU, and CB support on FP and general MCH challenges. Other Respondents: Add FP for HIV-positive clients Target youth and use LAPM to help make the case (e.g., Pathfinder project in Ethiopia on implants for youth). Target youth using information communication technology (ICT). Link FP and circumcision, particularly since PEPFAR/Jhpiego have significant funds for circumcision. In general, there should be a clear mandate to establish links between FP and areas of high resources for systems strengthening and greater access (“follow the money”). Expand access to permanent methods and the IUD. This is a critical point since building sustainable high-contraceptive prevalence levels without access to these methods is difficult and expensive. Synthesize research on FP community-based distribution and utilization. This is similar to the 1999 paper Philips did on research over the decade. Do more on how to utilize and make investments more effectively—more on cost￾effectiveness at the country level. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 43 Explore PPP service delivery systems. There is a rapid increase in private sector involvement. Are there ways to get closer collaboration and broad systems integration between the sectors? Cross analysis is needed, as there is a dearth of evidence on the cost￾effectiveness of different systems and obtaining marginal cost differences. Urban populations need more access. Urban poor and rural poor lead different lives and have different levels of access for different reasons. Socioeconomic status and cultural settings are different and affect access to FP. Assess reasons for non-use. Strengthen health systems, and explore the impact of different financial mechanisms, such as a health insurance system. Explore additional areas of task sharing beyond CHWs, such as midwives and clinical officers performing surgical methods. 44 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 45 VI. EVALUATION RECOMMENDATION OVERVIEW The PROGRESS Project has been a well-managed and implemented OR program that has already seen significant success (see Tables 2 and 3). The project management made sound structural decisions considering the timeline constraint of five years and the initial broad IRs and expected results. There is still much to be done, and the program’s impact will not be known until after the final closeout in 2013 and beyond. The evaluation team was charged not with analyzing the project achievements but with capturing the lessons learned that should be applied to this and to future OR projects. In looking to the future, we hope the details and recommendations for the following questions are helpful as USAID considers the future of FP operations research. WHAT QUESTIONS DOES FHI 360 WISH IT COULD HAVE PURSUED AND WHAT ARE POTENTIAL IDEAS FOR FUTURE RESEARCH? Priority research topics FHI 360 would like to pursue further: Research on effective referrals systems for FP, including from CHWs to clinics for LAPMs Research on scale-up of evidence-based practices, including evaluation of the scale-up process and continued impact of the intervention at scale Additional research on providing injectable and oral contraception through drug shops and pharmacies Testing the acceptability and feasibility of self-injection of DMPA OR on supply chain management and contraceptive logistics Additional research on costing and the comparative cost-effectiveness of interventions The evaluation team considers that beyond the current legacy areas, but consistent with the general objective of increasing access, the main research question in sub-Saharan Africa is how to increase access to LAPM, with greater emphasis on permanent methods. This question is being partially answered in legacy areas 2 and 3, but work on this can be substantially increased. WHAT SHOULD USAID DO TO HELP PROJECT IMPLEMENTATION? Develop an evidence-based documented utilization model that serves as a guide for assessing its OR projects, or create a shared utilization concept with the staff of other divisions and partners that incorporates the lessons from the diffusion of innovations literature. Establish good practices for promoting utilization of OR results and make every OR CA accountable for following these good practices. Retain a reserve budget in the mechanism to be used opportunistically to respond quickly to exceptional opportunities linked to project goals (e.g., responding to HC/HIV research). Include policy change and service delivery innovation goals in service delivery CA contracts and agreements in order to promote utilization of research results. These goals should be 46 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION flexible and revised in annual and midterm reviews according to circumstances. Research projects should help document activities and strategies conducted by CAs. Take a more proactive approach, with a longer project timeframe, so that countries address key gaps in research based on objective decision-making and the OPRH strategy of priority countries instead of based on where the implementing partner has strong existing relationships. Hire more research staff on USAID country teams to communicate with the field. Advocate for the Missions to appoint a flagship OR point person who takes a more active role in managing the OR portfolio and is responsible for RU from the project. Educate Mission staff on the research process, evidence-based practices, and what constitutes evidence in the social and biomedical sciences. Undertake country advocacy around ethics approvals and the problem with some local IRBs that require high fees. WHAT SHOULD PROGRESS DO DURING ITS FINAL YEAR? PROGRESS will have an extremely busy last year completing current activities. The evaluation team thinks the workplan includes priority activities, and there is no time to start any new activities. PROGRESS has planned to utilize its available timeline well. The evaluation team’s recommendations for PROGRESS focus on the planned dissemination and utilization activities. PROGRESS has already begun synthesizing and disseminating findings with key stakeholders in all its priority countries and within aspects of each major thematic area. These activities are carefully tailored as needed to specific stakeholder groups, as well as the MOH and FPTWG in many countries. The main recommendations for what PROGRESS should do are to: Create plans to influence decision makers and include development of strategies, messages, and materials targeted for this audience. As far as possible, identify other individuals and organizations that have an interest in the same decisions and engage their help with dissemination, for example, other CAs and the FPTWG. The plan should also include specific messages or recommendations that can be discussed with USAID Missions and bilateral programs. Draft tailored recommendations for each PEC- and SDI-managed project, specifying in each case how the results of PROGRESS could help the project reach its own goals. These should be prepared in coordination with OPRH project managers and champions. Since general dissemination materials will be needed to inform different audiences, one item should be produced and disseminated for each legacy area and each IR. PROGRESS is well ahead of schedule in this task. PROGRESS is also planning Washington DC-based project dissemination meetings throughout the next year, having already held one in March 2012. Messaging to partner organizations should be tailored to how the body of results of this project can help these institutions reach their own goals. Too many end-of-project meetings are couched as reviews and attract only institutions PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 47 already involved. The invitation list and meeting space should maximize engaged participation. PROGRESS is already planning a participatory meeting. Four to six months before the project’s end, if administratively feasible, the project should analyze whether there is a need for a no-cost extension to complete current research studies and appropriately disseminate results and practices. In particular, because of extensive delays due to partner demands with the Depo-SubQ in Uniject, the analysis should be planned for the final months of the project. Because of the complicated partnership for completing this work and the potential for this method to be used for self-administration in the future, it is critical that the final data analysis be completed. FINDINGS AND RECOMMENDATIONS FOR NEW RESEARCH AWARDS Considering the reported obstacles and opportunities, the evaluation team presents this list of recommendations for future research projects: There is still a need for FP OR flagship projects to be funded and function independent of service delivery projects. A project timeline should be 7–10 years. Award mechanisms should be synchronized with the timing for field support decisions so that research CA does not lose out on one year of potential field support. A new project should retain the collaborative approach PROGRESS and RTU utilized at the beginning to focus the work into legacy areas. With a longer timeline, projects should conduct more studies testing things that have not been done before and exciting new ideas. USAID should continue supporting the strong global/core technical experts to, e.g., address emerging issues, such as HC/HIV research, participate in Cochrane reviews, and prioritize global research agendas. With a longer timeline, projects should take a more disciplined approach to initiating projects and select a small number of core-focus countries for R and RU activities. While there is a call for a core set of focus countries based on OPRH strategy, there is also a plea to retain the flexibility of the mechanism to respond to Mission needs. Use of core funding as seed funding to introduce OR activities at the country level should be retained, with an explicit commitment from the Mission that field support will co-fund. One interesting idea was to promote the launch of a new OR project as a competition, so that if Missions applied with good ideas for the OR project, core funds would match their commitment. A new project should move away from the current concept and include CB activities as continuous TA to install and strengthen systems and operating units and, only exceptionally, teaching courses and workshops and strengthening capacity in universities and research institutions. CB in FP program M&E should continue to be supported by USAID, with MOH and FPTWG the main target of CB activities. 48 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION Develop specific branding strategies that distinguish a project from an implementing partner with resources set aside to participate in more FP research forums and meetings in Washington DC. The minimum amount of funds budgeted for partner organizations to consistently engage and stick to timelines and other agreed-upon parameters should be increased/. Additional resources and support for the process of hand-off are warranted Any country research portfolio requires a presence on the ground by the research CA with sufficient LOE to maintain a relationship with national MOH, Mission, and partner CAs. Increased local staff is necessary, especially if a country-focused strategy is applied in the future. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 49 ANNEX A. SCOPE OF WORK The PROGRESS Project End of Project (EOP) Performance Evaluation Scope of Work Estimated Start date: March 2012 6 weeks—No International Travel BACKGROUND The Program Research for Strengthening Services (PROGRESS) cooperative agreement was awarded to FHI (formerly Family Health International, later renamed FHI 360) in June 2008 and will end in June 2013. The USAID management team is located in the Research, Technology and Utilization Division in the Office of Population and Reproductive Health (GH/PRH/RTU) in Washington, DC. Dr. Mihira Karra is the agreement officer’s technical representative (AOR), Ms. Megan Matthews is the technical advisor (TA), and Matthew Phelps is the program assistant (PA). The PROGRESS team leadership is located at FHI 360’s offices in Research Triangle, NC. Dr. Maggwa Ndugga is the project director. Rose De Buysscher and John Stanback are deputy directors. The project’s ceiling is set at $50 million, and the project can accept core funds, field support funds, and interagency agreement pass-through resources. PROGRESS is the flagship Operations Research (OR) and Technical Assistance (TA) mechanism for GH/PRH/RTU. It aspires to improve access to FP among underserved populations in developing countries by conducting research, research utilization, and capacity building. This goal is focused through the following four Legacy Areas: Maximize human resources through task-shifting and addressing medical barriers; Expand Service Delivery Options (SDO) within and beyond the health sector; Expand the FP method mix for home, community, and lower-level provider use; Increase in-country capacity for research, research utilization, and monitoring and evaluation (M&E). The project has developed a detailed Performance Monitoring Plan (PMP) framework to monitor progress towards the overall objective and specific results (See attachment 1). FHI 360 also has a document showing the evolution from the Intermediate Results described in the initial award. While the core project management team for this research project is “lean” and efficient, the PROGRESS project is able to utilize the range of technical expertise across the FHI 360 organization, including biomedical researchers, program researchers, and staff in field offices. This innovative program management approach permits research questions to trickle up from the field, facilitates a high level global learning agenda, and promotes Research to Practice knowledge translation. PROGRESS global technical leadership (GTL) activities include coordinating global technical meetings, synthesizing information from research findings, sharing innovative approaches to improving access to FP with south-to-south learning, and facilitating the use of best practices. As 50 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION the crosscutting operations research project, PROGRESS is directly responsive to requests from USAID Washington and other partners, including the World Health Organization (WHO). PROGRESS works in collaboration with USAID Missions, Ministries of Health (MOH), USAID implementing partners, and other stakeholders by means of staff on the ground in the FHI 360 country offices. Activities include, but are not limited to, conducting operations research studies in collaboration with local stakeholders; building capacity of research institutions, Ministries of Health, and other partners; and providing TA to key stakeholders to advance evidence-based practices. PROGRESS has significant portfolios of work funded by both core and field support resources in Ethiopia, India, Kenya, Rwanda, Senegal, Tanzania, and Uganda. Field support funds have also been received for work in Malawi, Zambia, and Nigeria. Management reviews, existing performance reports, and an extensive PMP sufficiently address the progress toward completion of activities in 2013. Thus the evaluation questions are more broadly applied to the project design and how the midterm adjustments have added value to GTL and Mission research agendas. PURPOSE OF THE EVALUATION The evaluation team will answer the following questions: 1. (30% LOE) How has PROGRESS evolved and progressed towards achievement of its objectives? What factors motivated the changes? What factors facilitated achievement of project objectives? How has the organizational structure at FHI 360 and USAID accelerated or slowed the process? 2. (30% LOE) How have partnerships contributed to achievement of project objectives, particularly with regard to the implementation of research, research utilization and capacity building? What type of partnerships were more effective and why? What strategy does PROGRESS have to facilitate successful “hand-off”? – Partners at global level – Partners at regional level – Partners at country level 3. (40% LOE) As a centrally funded activity that is intended to support the field, was the design and implementation appropriate to answer field questions? Was the experience of PROGRESS that there is substantial interest and demand for Operations Research, Research Utilization, and Capacity Building from Missions? – What were the factors that led Mission staff to buy in to PROGRESS, both with FS and supporting core funded activities? What was the range of topics of interest? Were field questions adequately answered? – How did the PROGRESS Management team interact with the Missions? What activities were successful in obtaining buy-in and what hindered activities in country? – How did USAID Washington interact with the Missions? How were GTL priorities balanced with Mission agendas? PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 51 METHODOLOGY The assignment work will commence with a two-day team-planning meeting (TPM), which can be facilitated remotely. This meeting will allow the team to meet with the USAID staff (RTU Division) to be briefed on the PROGRESS Cooperative Agreement and its activities. USAID will also discuss the purpose, expectations, and agenda of the assignment with the evaluation team. In addition, the team will have time at the TPM to internally: a. clarify team members’ roles and responsibilities, b. review and develop final evaluation questions, c. review and finalize the assignment timeline and share with USAID, d. develop data collection methods, instruments, tools, guidelines and analysis, e. review and clarify any logistical and administrative procedures for the assignment, f. establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion, g. develop a preliminary draft outline of the team’s report, and h. assign drafting responsibilities for the final report. The questionnaire should be developed during the TPM with the knowledge that consultants may be conducting interviews independently, and thus ensuring comparability and consistency. For this evaluation, the sources of information on the performance of the PROGRESS Project include: interviews with key stakeholders and a desk review of documents, which detail the activities of the project and describe issues related to implementation and their resolution. Additional perspectives will be acquired by the evaluation team through interviews. The evaluation team may request additional background information from the AOR and/or from the PROGRESS project core team. 1. Background Documents/Materials: The following documents will be provided to the Evaluation Team. Other documents may be added or requested as needed. – PROGRESS Award Document – Performance Monitoring Plan (PMP), and Gap Analysis – Evolution of the project from the PD—“Cross-walk” – Post Dakar 2011 Management Review and Technical Updates – Annual and interim reports and workplans – Annual results reviews – Publications and related documentation from activities – Other management review meeting minutes – Project summaries by theme and country 52 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 2. Group Meetings and Interviews with PROGRESS Staff: The evaluation team shall travel to FHI 360 Headquarters in Research Triangle, NC, to hold a two-day meeting with the core PROGRESS staff. 3. Key Informant Interviews: Prior to initiation of the evaluation, FHI 360 and GH/PRH/RTU will create a master list of interviewees. FHI 360 will schedule between 75-100 group and individual, in-person and telephone interviews, working with the consultants’ schedules. Suggestions include FHI 360 staff in the field, USAID RTU staff, relevant PRH Global Leadership Priority Champions, USAID PRH Front Office, USAID Mission staff from a selection of countries, USAID Regional Bureau staff, a selection of partners with subagreements with PROGRESS, Ministry of Health delegates, peers from other donors, including the Gates Foundation, the World Health Organization, and the National Institutes of Health (NIH) (which has IAA funding into PROGRESS). This evaluation will not include international travel, but will include domestic travel to Washington DC, and Research Triangle, NC. The team will make the utmost effort to conduct as many in-person interviews as possible, especially with USAID and FHI 360 headquarters key informants. Interviews with individuals who are not local will probably be conducted by telephone or videoconference. The evaluation timeline capitalizes on prescheduled events that bring senior FHI 360 field staff who oversee PROGRESS work in their countries to Washington, DC, on March 12. Additionally, a technical meeting is being planned for March 13 that will include key highlights of PROGRESS country work. DELIVERABLES 1. Preparation of Evaluation Workplan Including Data Collection/Analysis Plan: During the team-planning meeting, the Evaluation Team develops a workplan for approval by USAID, including a data collection and analysis plan and the responsibilities of individual team members. The general methodology to be used will be reviewed and discussed, a key informant interview questionnaire will be created and tested, a schedule will be finalized, and all other operational and logistical issues will be addressed as needed. 2. Final Evaluation Report: Following completion of interviews and prior to the analysis and report writing timeline, the evaluation team should schedule a telephone conference with USAID GH/PRH/RTU contacts Mihira Karra (AOR) and Megan Matthews (TA) to discuss the report writing and table of contents/template. USAID will provide comments and direction at that time to enable a speedy review of the final report. The final evaluation report should include the following: executive summary; scope and methodology used; important findings (empirical facts collected by evaluators); conclusions on key evaluation questions; (evaluators’ interpretations and judgments based on the findings); and conclusions and lessons learned that may have implications for future designs and for others to incorporate into similar programs. The report should be no longer than 30 pages, excluding annexes. The report’s attachments should include, but are not limited to, a copy of this evaluation, the scope of work, and a list of the interviews conducted. The evaluation team will submit the final report, watermarked as a draft, to USAID contacts Mihira Karra (AOR) and Megan Matthews (TA) in electronic form, preferably as a Microsoft Word document. USAID will respond with questions and suggested edits within five days. FHI PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 53 360/PROGRESS will simultaneously be provided with the final report draft and will have the opportunity to review and fact check. The evaluation team then has 3 days for revisions, unless additional time is requested. The final document should be submitted in electronic form to Mihira Karra, Megan Matthews, and Dr. Maggwa Ndugga. The final public report must also be submitted to the Development Experience Clearinghouse (DEC). As this report is an evaluation of the project and makes technical recommendations that are not specific to future USAID procurements, this report will not be classified as procurement sensitive. 3. Debriefings: The Evaluation Team will provide debriefings to USAID and FHI 360 in Washington DC. This will be informal and may be facilitated by videoconference. TEAM COMPOSITION Three external consultants will be identified and recruited through the GH Tech mechanism to serve on this Evaluation Team, together with one USAID operations research specialist. One of the consultants shall be identified as the Evaluation Team Leader. The team should have expertise in the following areas: 1. An advanced degree in the social sciences, an MPH or commensurate experience. 2. Knowledge of, and interest in, FP and related issues, including operations research and research utilization/knowledge translation for the benefits and improvement of FP programs. 3. Experience in the management of FP and other reproductive health services in developing countries. 4. Knowledge of operations and program research and service delivery issues related to reproductive health technologies in developing countries. 5. Knowledge of issues related to information dissemination and utilization of research for program improvement. 6. Developing country experience. 7. Previous experience evaluating complex programs. Potential candidates for this team may include: senior and possibly retired persons with careers related to contraceptive research and development and/or reproductive health care in developing countries. The candidates must be able to work as team members, evaluate and synthesize information quickly, make clear and well-founded recommendations, and contribute to the written report and debriefings. SCHEDULING AND LOGISTICS Once the consultants are identified and recruited, the process for document review and interviews with key informants can begin in order to conduct interviews in March of 2012. The Evaluation Team should adhere to the agreed-upon timeline below and this final report must be completed no later than April 31, 2012. FHI 360, at both the Research Triangle and Washington, DC, locations, will arrange for limited office space, access to computers with Internet, telephones for distance and international interview calls, Skype/Videocon if possible, printing, and other administrative equipment. All resources may not be required by the consultants. Draft Timeline: to be finalized with Consultants. 6 weeks is estimated with partial time/non-traditional work hours and 54 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION remote access acceptable to USAID. The LOE can be fewer days/weeks when consultants work full time. Date Activities On/Around March 1 Submission of materials for desk review. Monday March 5 TPM – discussion of SOW, Team, Methodology, Schedule. Skype Tuesday March 6 TPM – discussion of SOW, Team, Methodology, Schedule. Skype Wednesday March 7 Questionnaire finalization. Remote Thursday March 8 Methodology and desk review. Remote Friday March 9 Methodology and desk review. Remote Sunday March 11 Travel to DC. Travel Monday March 12 9-12 Team Planning 1-5 Face-to-face interview with PROGRESS country office point persons. Phone interviews with those country office point persons not in DC can also be scheduled for this day. FHI 360 DC Tuesday March 13 Half-day technical meeting in Washington, DC, which will include highlights of PROGRESS country work. Evening: Travel to NC. FHI 360 DC Wednesday March 14 Meetings with PROGRESS core staff in NC. FHI 360 NC Thursday March 15 Meetings with PROGRESS core staff in NC. FHI 360 NC Friday March 16 Travel to home location. Monday March 19 Desk review and interviews. Remote Tuesday March 20 Desk review and interviews. Remote Wednesday March 21 Desk review and interviews. Remote Thursday March 22 Desk review and interviews. Remote Friday March 23 Desk review and interviews. Remote PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 55 Date Activities Monday March 26 Information Synthesis and catch-up. Remote Tuesday March 27 Information Synthesis and catch-up. Remote Wednesday March 28 Information Synthesis. Remote Thursday March 29 Information Synthesis. Remote Friday March 30 Information Synthesis. Remote Monday April 2 Report Writing. Remote Tuesday April 3 Report Writing. Remote Wednesday April 4 Report Writing. Remote Thursday April 5 Report Writing. Remote Friday April 6 Report Writing. Remote Monday April 9 OOB Submission to USAID and FHI 360 (Electronic). Tuesday April 10 USAID review. Wednesday April 11 USAID review. Thursday April 12 USAID review. Friday April 13 USAID comments to Consultants. Monday April 16 Report Revision. Remote Tuesday April 17 Report Revision. Remote Wednesday April 18 Report Revision and Travel to DC. Travel Status? Thursday April 19 Report out to USAID and FHI 360. Washington DC or videoconference Friday April 20 Contract close-out, finalization of terms with GHTech Bridge. Travel Home. 56 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION FHI 360 RESPONSIBILITIES Prepare first draft of master list of interviewees, provide copies of all background materials, contact interviewees to prepare interview schedule (with input from evaluation team on preferred timing). USAID EVALUATION TEAM LEAD RESPONSIBILITIES (MEGAN AND MATTHEW) Identification of consultants, confirm master list of interviewees, send introduction letter to interviewees to prepare for scheduling, liaison with GHTech Bridge hiring mechanism, lead TPM meeting, lead draft review procedure at USAID, schedule report out meeting. USAID commits to being available to consultants during the duration of the evaluation for support and guidance. The USAID reviewers will also provide written input on the draft report in a timely fashion. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 57 ANNEX B. PERSONS CONTACTED “T.S.” for interviewed by Tabitha Sripipatana, “R.V.” for interviewed by Ricardo Vernon, “Both” for interviews done in person by both consultants, or “written response” for responses that came by the e-mail questionnaire. ETHIOPIA Ethiopia Federal Ministry of Health Dereje Mamo, Director, Policy and Planning Directorate (R.V.) Sintayehu Abebe, Case Team Leader, Dire Dawa, Urban Health Promotion and Disease Prevention (R.V.) Dr. Mengistu Hailemariam, RH Advisor (R.V.) FHI 360/Ethiopia Francis Okello, Chief of Party, PROGRESS Project-Ethiopia Integrated Family Health Program Dr. Mengistu Asnake, Chief of Party/Deputy Country Representative (R.V.) USAID/Ethiopia Jeanne Rideout, Team Leader for Health (T.S.) Premila Bartlett (Formerly USAID activity manager, currently working for Save the Children in Malawi) (T.S.) INDIA FHI 360/India Bitra George, Country Director (T.S.) Institute of Reproductive Health/India Priya Jha, Country Representative (T.S.) Network of Entrepreneurship and Economic Development (NEED) Mr. Anil K. Singh, CEO (R.V.) USAID/India Vijay Paulrig, Program Management Specialist (activity manager) (T.S.) Lovleen Johri (now with U.S. Embassy, New Delhi, India) (T.S.) KENYA APHIAplus Eastern & Central Dr. Kenneth Chebet, Director (T.S.) FHI 360/Kenya Marsden Solomon, Associate Director (R.V.) 58 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION Kenya Division of Reproductive Health (DRH) Dr. Bashir M. Issak, Head (R.V.) Land o’ Lakes, Ibnc. International/Kenya Mary Munene, Dairy Value Chain Development Coordinator (T.S.) Population Council/Kenya Chi Chi Undie (written response) Dr. Ian Askew, Director (T.S.) USAID/Kenya Jerusha Karuthiru, Program Specialist (T.S.) MALAWI Malawi Ministry of Health Chisale Mhango, Director, Reproductive Health Unit (written response) USAID/Malawi Lilly Banda, Deputy Health Team Leader (written response) NETHERLANDS Text to Change Hajo van Beijma, President (written response) NIGERIA FHI 360/Nigeria Hadiza Khamofu, Director, Medical Services (written response) RWANDA FHI 360/Rwanda Jennifer Wesson, Research Director (R.V.) IntraHealth/Rwanda Ms. Suzanne Mukakabanda, RH/FP Program Manager (R.V.) Rwanda Ministry of Health Dr. Leonard Kagabo, In-charge of Permanent FP Methods (written response) USAID/Rwanda Dr. Eric Kagame, Maternal & Child Health Specialist (T.S.) SENEGAL FHI 360/Senegal Barbara Sow, Country Director (T.S.) PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 59 SWITZERLAND World Health Organization (WHO) Iqbal Shah, Senior Social Scientist, Department of Reproductive Health and Research (T.S.) TANZANIA East, Central & Southern African Health Community (ECSA) Dr. Odongo Odiyo Nyagudi, Manager, Family and Reproductive Health (R.V.) EngenderHealth Joyce Ishengoma, Sr. Policy/ Advocacy Officer (R.V.) Richard Killian, Chief of Party, ACQUIRE Tanzania Project (R.V.) FHI 360/Tanzania Christine Lasway, Associate Director (T.S.) NIMR MMRC Godfather Kimaro, Research Scientist (R.V.) Pathfinder Mustafa Kudrati, Country Director (R.V.) Tanzania Ministry of Health Maurice Hiza, National FP Program Coordinator (R.V.) USAID/Tanzania Tim Manchester, Sr. FP/RH Advisor USAID (T.S.) UGANDA Conservation Through Public Health (CTPH) Dr. Gladys Kalema-Zikusoka, Founder and CEO (R.V.) FHI 360/Uganda Angela Akol, Country Director (T.S.) JHUCCP/Uganda Martin Ninsiima, Program Officer, Advance Family Planning Project (R.V.) USAID/Uganda James Tanu Duworko, Activity Manager (R.V.) USA Bill & Melinda Gates Foundation Monica Kerrigan, Deputy Director, Family Planning/Reproductive Health (T.S.) ChildFund International Sadia Parveen, Program Quality Advisor (R.V) 60 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION FHI 360/North Carolina (an asterisk denotes persons with whom several contacts and individual interviews were made; others made presentations in group meetings) Aurelie Brunie, Scientist, Health Services Research Bill Finger, Associate Director, Research Utilization, PROGRESS Charles Morrison, Senior Director, Clinical Sciences Cindy Geary, Senior Director, Behavioral and Social Sciences Colleen Macko, Administrative Assistant/Budget Lead, PROGRESS David Hubacher, Scientist, Clinical Sciences David Sokal, Senior Scientist, Clinical Sciences Donna McCarraher, Associate Director, Behavioral and Social Sciences Elena Lebetkin, Senior Technical Officer, PROGRESS Eva Canoutas, Associate Director, Research Utilization Gina Etheredge, Technical Advisor, Strategic Information Heather Vahdat, Associate Scientist, Behavioral and Social Sciences Jennifer Headley, Research Associate, Behavioral and Social Sciences Johannes Van Dam, Senior Director, Program Sciences John Bratt, Scientist, Health Services Research *John Stanback, Deputy Director for Research, PROGRESS (Both) Karen Katz, Deputy Director, Health Services Research Kate Plourde, Technical Assistant, Research Utilization Kate Rademacher, Technical Officer, Research Utilization Karin Ganter, Technical Officer, PROGRESS Kelly L’Engle, Scientist, Behavioral and Social Sciences Kirsten Krueger, Technical Advisor, Research Utilization Laneta Dorflinger, Distinguished Scientist, Clinical Sciences Laureen Lopez, Scientist, Clinical Sciences Lauren Hart, Associate Technical Officer, Research Utilization Leigh Wynne, Technical Officer, Research Utilization Lucy Harber, Instructional Design Associate, Technical Training Lucy Wilson, Monitoring and Evaluation Lead, PROGRESS Mackenzie Green, Research Associate, Health Services Research *Maggwa Ndugga, PROGRESS Project Director (Both) Markus Steiner, Scientist, Clinical Sciences Marga Eichleay, Research Associate, Health Services Research PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 61 Marjorie Newman-Williams, Chief Operating Officer Morrisa Malkin, Technical Officer, Research Utilization Paul Feldblum, Scientist, Clinical Sciences *Rose De Buysscher, PROGRESS Deputy Director for Management and Administration (Both) Susan McIntyre, Director, HQ Projects Tracy Orr, Technical Officer, Research Utilization Theresa Hoke, Scientist, Health Services Research Tricia Petruney, Senior Technical Officer, Research Utilization Trinity Zan, Senior Technical Officer, Research Utilization *Ward Cates, President Emeritus, Research (T.S.) Jhpiego Cat McKaig, FP Team Leader, Material and Child Health Integrated Program (MCHIP) (T.S.) National Institutes of Health Trent MacKay, Chief, Contraception and Reproductive Health Branch, Center for Population Research (written response) PATH Sara Tifft, Senior Program Officer (T.S.) Save the Children Winifride Mwebesa, FP and RH Advisor (written response) USAID Office of Population and Reproductive Health, Washington, D.C. Alex Todd, (SDI, Repositioning Family Planning Champion) (Both) Carmen Coles, (PEC, Repositioning Family Planning Champion) (Both) Erika Martin, RTU (Both) Ishrat Hussain, Africa Bureau (Both) Jeff Spieler, Scientific Advisor, OPRH (Both) Judy Manning, RTU, Health Development Officer (T.S.) Megan Matthews, RTU, PROGRESS Technical Advisor (Both) Mihira Karra, Chief, RTU and PROGRESS Agreement Officer’s Representative (AOR). (T.S.) Nandita Thatte, RTU/SDI USAID (Both) Patricia Macdonald, SDI (Both) Patricia Stephenson, RTU, former PROGRESS AOR (Both) Sarah Harbison, Science Advisor, OPRH (Both) Shawn Malarcher, RTU (Both) Victoria Graham, SDI, CBA2I Champion (Both) 62 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION ZAMBIA FHI 360/Zambia Mike Welsh, Country Director (T.S. & written response) USAID/Zambia Dr. Masuka Musumali, FP/MNCH Advisor (written response) PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 63 ANNEX C. REFERENCES DISSEMINATION AND TECHNICAL PRODUCTS Community-Based Access Working Group. “Community-based health workers in Nigeria can safely and effectively administer injectable contraceptives,” September 2011. East, Central and Southern Africa Health Community (ECSA). “Expanding access to family planning services at the community level: findings and recommendations from a regional assessment.” Final report, October 2011. FHI 360. “Building a stronger medical research institute in Tanzania,” 2012. FHI 360. Community-Based Access to Injectable Contraceptives Toolkit: Compilation of CBA2I Toolkit Documents Contributed by FHIO 360. February 2012. FHI 360. “Costed implementation program: Innovative approach in Tanzania helps revitalize family planning programs,” 2012. FHI 360. Fact Sheet. “The Invest-FP Calculator: A planning tool for expanded access to family planning,” February 2012. FHI 360. Informational Booklet. Mobile 4 Reproductive Health (m4RH), 2011. FHI 360. Informational Booklet. PROGRESS. Program Research for Strengthening Services. FHI 360. Research Brief: Étendre l’utilisation de la contraception au Rwanda. FHI 360. Research Brief: Expanding contraceptive use in Rwanda, 2010. FHI 360. Research Brief: Family planning information and referrals at child immunization clinics: study in Ghana and Zambia highlights implementation challenges, December 2010. FHI 360. Research Brief: Feasibility of providing family planning services through an agricultural cooperative field day: Lessons from Rural Kenya, 2012. FHI 360. Research Brief: Introduction of the intrauterine contraceptive device 375 in India: Positive assessment findings help guide national scale up, 2012. FHI 360. Research Brief: Women’s ability to self-screen for COCs compared to a nurse’s assessment: Drug shops in rural and peri-urban Tanzania, 2012. FHI 360. Works in PROGRESS. Numbers 1-4. FHI 360 and MCHIP. “Integration of family planning and immunization services: global summary of current programmatic experiences and research projects,” 2010. FHI 360 and MCHIP. Technical Brief: Integration of family planning with immunization services: a promising approach to improving maternal and child health, 2010. FHI 360 and Rwanda Ministry of Health. “Rwanda takes no-scalpel vasectomy training nationwide,” November 2011. Janowitz B, Stanback J, Boyer B. “Task sharing in family planning.” In Studies in Family Planning, 43: 1-6. 64 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION Kenya Ministry of Health, USAID, FHI 360, Jhpiego, Tharaka District Health Management Team. Kenya Ministry of Health Project Support Greater Access to Contraception Program brief, September 2011. Krueger, K., A. Akol, P. Wamala, and A. Brunie A. “Scaling up community provision of injectables through the public sector in Uganda.” In Studies in Family Planning, 42 (2):117-124. Lasway, C. “The Price of Repositioning a National Family Planning Program.” Presentation Institutionalizing Evidence-Based Practices Technical Meeting. March 13, 2012, Washington. DC. Malarcher S., O. Meirek, E. Lebetkin, I. Shah, J. Spieler, J. Stanback, “Provision of DMPA by community health workers: what the evidence shows.” In Contraception 2011, 83:495- 503. National Coordinating Agency for Population and Development (NCAPD). Managing Population to Achieve Kenya’s Vision 2030: Plan of Action, 2011. Ngugga M. and J. Stanback, “PROGRESS Approach Institutionalizing Evidence-Based Practices.” Presentation Institutionalizing Evidence-Based Practices Technical Meeting. March 13, 2012, Washington, DC. Stanback, J, J. Spielerb, I. Shah, and W. Finger. “Community-based health workers can safely and effectively administer injectable contraceptives: conclusions from a technical consultation.” In Contraception 81: 181–184. Uganda Ministry of Health. Policy Guidelines and Service Delivery Standards for Community Based Provision of Injectable Contraception in Uganda; Addendum to Uganda National Policy Guidelines and Service Standards for Sexual and Reproductive Health. December 2010. USAID, FHI 360, MCHIP. “Integration of family planning with immunization services: A promising approach to improving maternal and child health,” 2010. USAID, WHO, UNFPA, IBP. “Family planning and development: actions for change,” 2010. World Health Organization, US Agency for International Development, Family Health International. Community-Based Health Workers Can Safely and Effectively Administer Injectable Contraceptives: Conclusions from a Technical Consultation. Research Triangle Park, NC: FHI, 2010. Zan, T. “Reveler les Defis: Capacity Building of the Reproductive Health Division in Senegal. Presentation Institutionalizing Evidence Based Practices Technical Meeting.” March 13, 2012, Washington, DC. PROGRAM DOCUMENTS FHI 360. “Activities & Accomplishments under the Seven Thematic Areas”: 1–16, FHI 360. “Activities & Accomplishments within the Major Portfolio Countries,” February 2012: 1–8, FHI 360. “Challenges and Lessons Learned”: 1–8. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 65 FHI 360. “Evolution of the PROGRESS Project from Proposal to Date. PROGRESS towards the 13 Results,” April 2011: 1–5. FHI 360. “Management and Organization of the Project,” February 2012. FHI 360. “Meeting Minutes: PROGRESS/USAID Management Review,” December 15-16, 2009: 1–9. FHI 360. “Meeting Minutes: PROGRESS Management Review—Minutes September 22, 2010: 1–10. FHI 360. “Meeting Minutes: PROGRESS and USAID Review Meeting,” April 18–19, 2011: 1–10. FHI 360. “Meeting Minutes: RU PROGRESS and USAID MEETING,” September 20–21: 1–3. FHI 360. “PROGRESS by the Numbers,” February 2012: 1. FHI 360. “PROGRESS Gap Analysis [Abbreviated],” February 2012: –4. FHI 360. “PROGRESS. Global Review of Manufacturers. Expand Access to Injectable Contraceptives: 1. FHI 360. “PROGRESS Key Results Review RY08”: 1–3. FHI 360. “PROGRESS Key Results Review RY09”: 1–5. FHI 360. “PROGRESS Key Results Review RY10”: 1–6. FHI 360. “PROGRESS Key Results Review RY11”: 1–6. FHI 360. PROGRESS Year 1 Semi-Annual Report: July–December 2008: 1–29. FHI 360. PROGRESS Year 1 Annual Report: July 2008–June 2009: 1–43. FHI 360. PROGRESS Year 1 & 2 Implementation Plan: June 2008–June 2010: 1-42. FHI 360. PROGRESS Year 2 Semi-Annual Report: July–December 2009: 1–92. FHI 360. PROGRESS Year 2 Annual Report (July 2009–June 2010) & Year 3 Interim Workplan (July 2010–June 2011):1–148. FHI 360. PROGRESS Year 3 Semi-Annual Report (July 2010–December 2010) & Year 4 Workplan (January 2011–June 2011):1–137. FHI 360. PROGRESS Year 3 Annual Report (July 2011–June 2012) & Year 4 Workplan (July 2011– June 2012): 1–168. FHI 360. PROGRESS Year 4 Semi-Annual Report (July 2011–Dec. 2011) & Interim Workplan (Jan 2012–June 2012): 1–172. FHI 360. PROGRESS Performance Monitoring Plan, December 2011: 1–14. FHI 360. Report on Activities & Accomplishments within the Major Portfolio Countries: 1–8. FHI 360. Technical Proposal. Program Research for Strengthening Services (PROGRESS). February 2008: 1–51. 66 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION FHI 360. “With the End in Mind: PROGRESS Global Management Review.” December 3–4, 2011, Dakar, Senegal: p. 1–13. FHI 360. “Working with Partners”: 1–8. USAID. Leader with Associates Cooperative Agreement No. GPO-A-00-08-00001-00. Program Research for Strengthening Services (PROGRESS). June 2008: 1–110. OTHER Population Reference Bureau. “2011 Population Data Sheet.” Singh, Susheela; J. E. Darroch, L. S. Ashford, and M. Vlassoff. Adding it up: The Costs and Benefits of Investing in Family Planning and Maternal and Newborn Health. Guttmacher Institute and UNFPA, 2009:1–40. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 67 ANNEX D. MONITORING AND EVALUATION FRAMEWORK (Yellow represents activities where progress is only partially on track to meet.) LEGACY AREA I. MAXIMIZING HUMAN RESOURCES BY TASK-SHIFTING AND ADDRESSING MEDICAL BARRIERS TO FP SERVICES 1. At least two models of task-shifting identified and evaluated. 2. Follow-on research and evaluation conducted to support the introduction of tested models of task-shifting and the reduction of medical barriers. 3. Provide synthesis of lessons learned around task-shifting, from research and programmatic experience. 4. Research evidence on task-shifting and reducing medical barriers synthesized and communicated to at least four countries and one international or regional body. 5. Evidence-based tools and practices on task-shifting and reducing medical barriers developed and introduced into programs and service delivery in at least four countries. 6. Technical assistance provided to Missions, MOHs, and other partners in at least three countries to promote the scale-up of evidence-based tools and practices on task-shifting and reducing medical barriers. LEGACY AREA II. EXPANDING SERVICE DELIVERY OPTIONS WITHIN AND BEYOND THE HEALTH SECTOR 1. At least three models or approaches for expanding service delivery within the health sector identified, assessed for feasibility, and/or evaluated. 2. At least three models or approaches for expanding service delivery beyond the health sector identified, assessed for feasibility, and/or evaluated. 3. Follow-on research and evaluation conducted to support the introduction of tested models of expanding service delivery options. 4. Research evidence on expanding service delivery options synthesized and communicated to at least four countries and one regional or international body. 5. Evidence-based tools and practices on expanding service delivery options developed and introduced into programs and service delivery in at least four countries. 6. Technical assistance provided to Missions, MOHs, and other partners in at least three countries to promote the expansion of service delivery options. LEGACY AREA III. EXPANDING FP METHOD MIX FOR HOME, COMMUNITY, AND LOWER-LEVEL PROVIDER USE 1. At least three studies to inform contraceptive improvements and procurement through increased understanding of the needs and preferences of potential contraceptive users designed and implemented. 2. At least four studies to evaluate and support the introduction of new, improved or more affordable contraceptive options (including new approaches to injectable contraceptives and new implants) designed and implemented. 68 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 3. Research on existing methods to guide contraceptive programming and procurement decisions in resource-poor settings developed and implemented. 4. Research evidence and best practices to inform contraceptive improvements and procurement and expanded method mix through increased understanding of the needs and preferences synthesized and disseminated globally. 5. Lessons learned from introducing new, improved, or more affordable contraceptive options synthesized and communicated (including development and introduction of tools and practices) to at least three countries and one international or regional body. 6. Technical assistance provided to Missions, MOHs, and other partners in at least four countries to inform contraceptive programming and procurement and to expand the contraceptive method mix, including the introduction of Sino-Implant (II). LEGACY AREA IV. INCREASING IN-COUNTRY CAPACITY FOR RESEARCH AND RESEARCH UTILIZATION CAPACITY BUILDING 1. An improved framework for research capacity building developed and implemented within PROGRESS. 2. Capacity building on conducting family planning program research provided to at least two local or regional research institutions. 3. Capacity building on utilization and/or promotion of research results in program decision￾making provided to at least four local institutions or organizations. 4. At least four programmatic research concepts originating from RU capacity building partners developed and implemented by PROGRESS. 5. Capacity building on monitoring and evaluation provided to at least four local institutions or organizations. 6. Underutilized Research Results 7. Technical assistance provided at the country level to Missions, MOHs, family planning technical working groups, and other partners to support the dissemination, introduction, and scale-up of research results and best practices. 8. Facilitate utilization and scale-up of best practices and underutilized research results on a regional level through at least two partnerships. 9. Provide technical leadership to inform global guidance and promote use of research results and best practices. V. CROSSCUTTING 1. Crosscutting activities contributing toward the PROGRESS goal of improving access to family planning among underserved populations. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 69 ANNEX E. FHI 360’S CONSIDERATION OF FUTURE THEMATIC AREAS FHI 360 headquarters’ staff compiled the following information. PROGRESS staff was asked to consider the future thematic areas that will be important for advancing family planning research. FUTURE DIRECTIONS PROGRESS has consolidated its portfolio through its legacy areas and seven thematic areas. Other important areas of research and research utilization certainly exist, and a new project may want to both build on the current work as well as expand to new areas. New areas of work may include research and learning on referrals; linking CHWs with health centers; task-shifting for long-acting methods; integrating FP information into youth services; use of pregnancy testing to reduce medical barriers; and many others. Within the current PROGRESS thematic areas, ideas are as summarized below. POSTPARTUM FAMILY PLANNING THEMATIC AREA 1. Greater initiation of FP in postpartum period needed. With high unmet need in this period, multiple approaches are needed. 2. FP/Immunization. Next project needs to frame messages based on both research findings and programmatic experiences. This requires more information from programmatic experiences (perhaps with a simple survey in PROGRESS Year 4) as well as new operations research. Reporting on the findings from PROGRESS research need to be clear to distinguish between unsuccessful parts of a particular intervention, without giving too negative of an overall message. 3. Expand method options during postpartum, including PPIUD. Support potential for expanding this type of service beyond central hospitals, using health centers and task￾shifting to trained nurses. Add demand creation if possible. 4. Fertility Awareness. Continue to develop behavior change communication (BCC) strategies such as being developed in Rwanda to expand awareness about return to fertility postpartum among clients and providers. Non-Health Thematic Area 1. Research on FP/non-health integration scale-up. To have an impact, the PROGRESS projects, if successful, need to be implemented at scale. What does it take to implement an evidence-based FP/non-health integration program at scale and maintain the effectiveness seen in the pilot project? 2. Expand work in this area to take advantage of country interest in linking FP with development. (FP affects all 8 MDGs). 3. Link with non-health institutions that have built in self-interest, existing infrastructure. This helps ensure sustainability/scale-up (e.g., as PROGRESS is demonstrating with the Green Belt Movement) 4. Design intervention-based studies to show an impact on CPR. 70 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 5. Refine materials used in microfinance/PHE/agriculture interventions and share lessons learned. Regional consultations, guidance documents, packaging of what it takes for successful replication (M&E, sustainability, etc.), development of virtual hub for guidance 6. Advocate for intersectoral funding. This needs to include USAID and other donors, as well as development sectors. MOBILE HEALTH THEMATIC AREA Address challenges that emerged from “first generation” of m4Rh project, including: 1. New technology, not much to go on from implementation or communication stand point. 2. No true control group for research. 3. Financing—e.g., public-private partnerships, MOH budgets, donor funding. 4. Privacy in context of a user database (related to financing point) – Expand m4Rh “base” approach: Add content from continuation component, tailored for different audiences and with different content. – Additional research questions: Impact of communications services like m4Rh on FP behavior, reach of mobile phones, using smart phones/apps vs. dumb phones/SMS, questions of sustainability and costing issues. EXPANDING METHOD MIX THEMATIC AREA 1. Depo-SubQ in Uniject. If current research shows service delivery Depo-SubQ in Uniject to be acceptable and feasible, an important next step for research is self-injection. There is already a growing literature on self-injection with subcutaneous DMPA, including a third of the women in the original Pfizer clinical trial, so examining self-injection in developing countries is an obvious choice, given the potential to increase privacy and convenience, to provide the method via social marketing, and to reach women in remote areas with multiple doses. 2. Sino-Implant (II). Because Sino-Implant (II) suffers in some countries from a bias against Chinese-sourced pharmaceutical products, research utilization efforts should focus on countering such misperceptions. In addition, a future research agenda might include acceptability and other post-marketing research in countries where the product has been approved. 3. LNG-IUS. Because Mirena has become so popular in more developed countries, the LNG￾IUS has the potential to improve the use and reputation of IUDs in developing countries. Research on the acceptability of the LNG-IUS should be undertaken in anticipation of lower-cost hormonal IUDs that are already in development. 4. Implants for younger women. Given the increasing availability of implants, and the demonstrated advantages of offering implants to younger women, future research and research utilization should emphasize this important practice. Besides ad hoc research on acceptability and uptake of the practice, general research on family planning service delivery should include collecting data on the characteristics of implant clients. Approaches could include advocacy among providers about the appropriateness of this method for youth, as well as promotion through mobile technology and youth networks. PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 71 5. Vasectomy. As vasectomy is scaled up in Rwanda, a future option for both research and research utilization is the creation of centers of excellence in Rwanda (and possibly elsewhere) where study tours and training could help expand the role of this effective contraceptive option. DRUG SHOP THEMATIC AREA 1. The next project should continue work on this high-potential area. Research will be important to convince policymakers about safety, but RU will be even more important. Because private sector provision of a popular socially marketed product will be self￾sustaining, policy change in a country such as Nigeria would be a huge success. 2. In the current project, an end-of-project technical meeting, perhaps in Africa, is desirable, but, in the next project iteration, one goal should be a WHO technical consultation. CBFP THEMATIC AREA 1. Address emerging CBFP operational trends toward more integrated services from CHWs, longer training periods, and other issues. 2. Work with MOH on financial/policy guidance using various advocacy tools. 3. Continue to build capacity of partners to take leadership in continued expansion of CBA2I as standard of practice in selected countries— West Africa countries, countries that need continued advocacy and resources, and south-to-south guidance in new countries. 4. Among research issues, consider as priorities: – Paid vs. nonpaid/year-long training vs. short training models – CBFP referral system models – CBFP supervision (minimum amount/alternative approaches, etc.) – M&E of scale-up practices after successful pilots (applies in other thematic areas as well) 72 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 73 ANNEX F. INTERVIEW GUIDE FOR USAID MISSION STAFF Individual’s ability to answer specific questions regarding PROGRESS design and accomplishments will vary. The following interview guide is a comprehensive checklist of possible areas of query to be tailored based on the initial response of involvement with PROGRESS. For individuals with less experience/contact with PROGRESS, more general questions and thoughts on the future will be the focus. For individuals with extensive involvement with PROGRESS, the interviewers will be able to ask more detailed questions on the successes and challenges of design and accomplishment. Date of interview: Name and title of respondent: Name of the interviewer: Introduction The PROGRESS project, implemented by FHI 360, is moving into its final year of implementation. PROGRESS is an operations research project, funded by USAID Bureau for Global Health, Office of Population and Reproductive Health, Research Technology and Utilization Division. This project has undertaken significant research, research utilization, and capacity building activities in many countries to improve access to family planning among underserved populations. USAID has launched an end-of-project evaluation on the PROGRESS cooperative agreement. You have been identified as having important experience to share regarding the PROGRESS project. Your responses to the questions that follow are greatly appreciated and will be invaluable to future USAID-funded program designs. 1. Please describe your engagement with the PROGRESS project and how your role affected the in country implementation of activities: (Assess level of knowledge of PROGRESS and family planning research) Buy-In Process 1. What were the factors that led Mission staff to buy in to PROGRESS, both with field support and supporting core funded activities? 2. How did you learn about the project? 3. How were the activities negotiated between the Mission, the ministry, and the PROGRESS project with FHI 360 staff, Washington personnel, and any other players? 4. Degree to which PROGRESS objectives were aligned with Mission and ministry strategies. 74 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION Implementation 5. As a centrally funded research activity that is intended to support the field, was the design and implementation appropriate to address and influence Mission and ministry needs? 6. Did activities conducted by FHI 360 (PROGRESS project) meet the expectations of the Mission? Recommendations for Improvement, Including Administration and Management 7. Are there some examples of results (in your country or visible in the region) from the project, either local or from the global technical leadership that promote the global learning application of evidence-based practices in local contexts? 8. How has this project helped the country to undertake evidence-based practices in family planning and increased evidence in country? 9. Are there examples of recent research results integrated into bilateral partnerships and other field programs? 10. Are there other relevant research results that could be integrated – what barriers exist to doing so? 11. What might have facilitated research utilization in your country program? By a Washington project? By Washington USAID staff? By other partners? 12. Things that could be changed to improve communication and processes. If USAID/Washington was engaged, was it helpful? 13. If not, were there any challenges and should the USAID/Washington management team have been involved? What needs to be changed? 14. Degree to which project has coordinated activities with other CAs and donor organizations, and satisfaction with the degree of partnership— recommendations for improved team work. 15. Degree to which “hand-off” was successful from PROGRESS project activity to a service delivery or other partner. Project Design 16. Ease of use of mechanism for funding, use, and reporting of use of funds. 17. Perceptions of adequacy of funding for PROGRESS and use of Core and FS funds and of flexibility of use. 18. Degree to which PROGRESS offers skills that are not present in other contracts, in other CAs, or in local institutions. 19. What other skills would be important/useful and are not present? 20. What could USAID/Washington do differently to be more effective in promoting research and research utilization in countries? PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 75 21. Areas in which PROGRESS assistance is perceived to be more important: 1) research 2) research utilization 3) research methods capacity building in their context? 22. PROGRESS has four legacy areas: task-shifting/injectable and CBFP, integration with other health programs and beyond health programs, method mix, and capacity building: Were these identified areas responsive to your needs? Which ones? Were important areas missing? 23. Are there questions or activities you would have liked PROGRESS to address in your country that were not possible due to financial, time, or other constraints? Which? 24. Is there anything you would like to add about the PROGRESS project that might be helpful for the future design of USAID-funded research programs that you have not already mentioned? 76 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 77 ANNEX G. E-MAIL QUESTIONNAIRE Dear Colleague, As per the e-mail sent on April 5 by Megan Matthews, thank you for your willingness to participate in the PROGRESS cooperative agreement’s end-of-project evaluation. PROGRESS is a family planning operations research project, funded by USAID and implemented by FHI 360. PROGRESS, launched in 2008, has undertaken research, research utilization, and capacity building activities in many countries with the objective of improving access to family planning among underserved populations. Please visit the website at www.FHI 360.org/progress for additional information. Unfortunately, as timelines have been condensed due to unforeseeable events, it is not feasible to undertake telephone interviews with all suggested interviewees. Your perspective is invaluable to this review, and we would appreciate your response to the following questionnaire. Lucy Wilson will not contact you as indicated in the previous email. Please send your responses to Tabitha Sripipatana, one of the two independent consultants leading the evaluation effort with Dr. Ricardo Vernon, by Monday April 30 at Tabitha.sripipatana@gmail.com Thank you! Best regards, Tabitha Sripipatana, MPH and Ricardo Vernon, PhD If you have any questions or concerns about your involvement with this evaluation, please feel free to contact the USAID Technical Advisor Megan Matthews (mmatthews@usaid.gov) or the FHI 360/PROGRESS team—Maggwa Baker (bmaggwa@FHI 360.org) or Lucy Wilson (lwilson@FHI 360.org). Please provide a written response to the following questions: 1. Please briefly describe your institutions and your own involvement with the PROGRESS project. Please include country and activity/research title. 2. How was the activity/study concept generated? How was your collaboration brought about? Please discuss the degree to which your organization was involved in planning the activity and any recommendations on how you could have been better engaged. 3. Are there areas of collaboration and support from the PROGRESS project that have been important and valuable in relation to work to achieve national family planning and reproductive health goals and priorities? If yes, please detail the valuable areas of collaboration and support. 4. Has this partnership produced new research results that can be useful in-country beyond the existing parameters of this project and/or how has this activity brought in evidence￾based practices in family planning? Please provide some value added examples for your organization or for the country health program as a whole. 78 PROGRAM RESEARCH FOR STRENGTHENING SERVICES (PROGRESS) END-OF-PROJECT EVALUATION 5. Did working with a research organization like FHI 360 help you to achieve your own institution’s objectives? 6. How easy or difficult has it been for your organization or project to coordinate activities with PROGRESS? Please explain and make recommendations on opportunities for improvement. 7. Did activities conducted by PROGRESS meet the expectations of your organization/agency as negotiated in the activity description? If yes, what objectives were met? If no, please provide a brief description of expectations not met. 8. Looking to the future, do you see a demand for research, research utilization, and/or research capacity building for family planning as a high priority in your institution/agency? If yes, please note if you consider one area of higher priority than the others. How do you perceive that research results can be systematically and sustainably integrated into health programming worldwide by USAID supported partners? 9. What general recommendations would you give PROGRESS to improve? Is there anything you would like to add about PROGRESS that might be helpful for the future design of USAID-funded research programs that you have not already mentioned? For more information, please visit http://www.ghtechproject.com/resources GH Tech Bridge Project 1725 Eye Street NW, Suite 300 Washington, DC 20006 Phone: (202) 349-3900 Fax: (202) 349-3915 www.ghtechproject.com