EVALUATION MEASURE Evaluation Phase IV – Midterm Performance Evaluation FINAL REPORT February 2018 This publication was produced for review by the United States Agency for International Development. It was prepared by Lynne Franco, Svetlana Negroustoueva, Kelsey Simmons, Elisa Knebel, Sabine Topolansky, and Sarah Lunsford of EnCompass LLC through the Policy Planning, and Learning–Learning, Evaluation, and Research contract. Abstract Purpose: The midterm performance evaluation of MEASURE Evaluation Phase IV seeks to inform technical programming and program management activities. Questions: The evaluation responds to three overarching questions: (1) Is the project meeting its stakeholders’ needs? (2) What are the benefits of a health sector–wide versus a health area–specific approach? (3) Are the tools developed useful? Methods: This evaluation used a mixed-methods approach. The team conducted in-person interviews with 117 stakeholders in Cote d’Ivoire, Mali, and Nigeria; 49 virtual or in-person interviews with internal and external stakeholders in Washington, D.C., and other countries; an online survey with 120 internal (U.S. Government) and external stakeholders; and a review of 104 project documents. Findings and Conclusions: MEASURE Evaluation has been successful in meeting many needs at country and global levels, but has been challenged to comprehensively and consistently meet all needs. Stakeholders perceive a unique role for the project in the USAID landscape in strengthening heath information systems (HIS), conducting health impact evaluations, and building evaluation capacity. They view the project’s approach to technical assistance and capacity building as facilitating ownership and sustainability at the country level. Almost all stakeholders see the benefits of a health sector–wide approach to strengthening HIS; however, this is challenging for USAID, given the funding streams it has to manage. Stakeholders appreciate and use the many tools that the project has developed and adapted. Strengthening data quality and data use for decision making and facilitating interoperability of databases remain the most pressing HIS strengthening needs. Prepared for the United States Agency for International Development USAID Contract Number AID-OAA-TO-17-00015 February 23, 2018 Implemented by: EnCompass LLC 1451 Rockville Pike, Suite 600 Rockville, MD 20852 Phone: +1 301-287-8700 Fax: +1 301-685-3720 www.encompassworld.com MEASURE EVALUATION PHASE IV – MIDTERM PERFORMANCE EVALUATION FINAL REPORT Disclaimer The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development (USAID) or the United States Government. Policy Planning, and Learning–Learning, Evaluation, and Research This task order is implemented through the Policy, Planning, and Learning–Learning, Evaluation, and Research (PPL-LER) Indefinite Delivery, Indefinite Quantity contract, funded by the United States Agency for International Development (USAID). USAID’s Bureau for Policy, Planning, and Learning (PPL) awarded EnCompass LLC the 5-year PPL-LER contract to provide technical and advisory services for performance and impact evaluations, evaluation and performance monitoring capacity building, and performance monitoring activities at the mission, bureau, and agency-wide levels. PPL-LER task orders design and implement evaluation studies and assessments based on rigorous evidence sources (both quantitative and qualitative), develop and deliver evaluation and performance monitoring training, and provide technical assistance in performance monitoring for USAID development programs worldwide. Recommended Citation Franco, L., S. Negroustoueva, K. Simmons, E. Knebel, S. Topolansky, and S. Lunsford. 2018. MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report. Prepared for the United States Agency for International Development (USAID) and the MEASURE Evaluation project. Rockville, MD: EnCompass LLC. Acknowledgments The evaluation team appreciates the strong support provided by the USAID project management team and MEASURE Evaluation staff, and offers thanks to all who provided input to the interviews and surveys. CONTENTS LIST OF EXHIBITS.............................................................................................................ii ACRONYMS ....................................................................................................................... iv EXECUTIVE SUMMARY .................................................................................................... v INTRODUCTION...............................................................................................................1 Background and Project Context.............................................................................................1 Evaluation Questions...............................................................................................................3 EVALUATION DESIGN AND METHODOLOGY.............................................................4 Sampling and Data Collection.................................................................................................4 Data Analysis...........................................................................................................................5 Limitations..............................................................................................................................6 FINDINGS...........................................................................................................................7 Is the Project Meeting the Needs of Its Stakeholders? ..............................................................7 What Are the Benefits of a Health Sector–Wide versus a Health Area–Specific Approach to HIS Strengthening and Evaluation Capacity Building?..........................................................19 Are the Tools Developed Useful?...........................................................................................21 Remaining Needs ..................................................................................................................25 CONCLUSIONS................................................................................................................27 RECOMMENDATIONS....................................................................................................30 ANNEX 1: EVALUATION SCOPE OF WORK .................................................................34 ANNEX 2: EVALUATION TEAM PROFILES...................................................................46 ANNEX 3: DOCUMENTS REVIEWED ............................................................................49 ANNEX 4: EVALUATION DESIGN AND METHODS ....................................................55 ANNEX 5: DATA COLLECTION TOOLS (QUALITATIVE INTERVIEW GUIDES).....67 ANNEX 6: DATA COLLECTION TOOLS (ONLINE SURVEY QUESTIONNAIRES)....99 ANNEX 7: COTE D’IVOIRE BRIEF COUNTRY REPORT............................................100 ANNEX 8: MALI BRIEF COUNTRY REPORT...............................................................109 ANNEX 9: NIGERIA BRIEF COUNTRY REPORT ........................................................115 ANNEX 10: ONLINE SURVEY TABLES.........................................................................127 ANNEX 11: CONFLICT OF INTEREST DISCLOSURES..............................................141 February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report i LIST OF EXHIBITS Exhibit 1: MEASURE Evaluation stakeholders ................................................................................ 1 Exhibit 2: Distribution of MEASURE Evaluation funding, years 1–3, by type and source ............... 2 Exhibit 3: Key evaluation questions.................................................................................................. 3 Exhibit 4: Data collection methods by respondents reached ............................................................. 4 Exhibit 5: Project performance ratings from USAID mission respondents who worked with MEASURE Evaluation on a five-point scale (10 missions) ....................................................... 7 Exhibit 6: MEASURE Evaluation’s frequently cited technical successes from country visits............. 8 Exhibit 7: Summary of most-reported success and inhibiting factors in three countries .................... 9 Exhibit 8: Ratings from USG respondents who worked with MEASURE Evaluation on the project’s performance in building capacity on a 5-point scale ............................................................... 10 Exhibit 9: USG respondents’ satisfaction with MEASURE Evaluation on a 5-point scale .............. 11 Exhibit 10: USG/Washington interviewees’ differing perceptions on strengths and weaknesses of MEASURE Evaluation........................................................................................................... 12 Exhibit 11: Country-level survey respondents’ ratings on benefits of participating in groups that MEASURE Evaluation led, with weighted average on a 5-point scale (n=43)......................... 13 Exhibit 12: Survey respondents’ satisfaction ratings with MEASURE Evaluation’s helpfulness to the groups in which it participated on a 3-point scale................................................................... 14 Exhibit 13: USAID missions’ reasons to work with MEASURE Evaluation (n=13) ....................... 15 Exhibit 14: Types of evaluations and related activities in MEASURE Evaluation Phase IV (n=53). 15 Exhibit 15: MEASURE Evaluation’s contributions to sustainability in three countries visited........ 18 Exhibit 16: USAID mission survey respondents’ perceptions of MEASURE Evaluation's encouragement of sustainability.............................................................................................. 19 Exhibit 17: Country visit respondents’ most-cited tools with which MEASURE Evaluation is involved, in order of importance ............................................................................................ 22 Exhibit 18: Comments about specific MEASURE Evaluation supported tools............................... 23 Exhibit 19: MEASURE resource downloads/hits: July 1, 2014–June 30, 2017 .............................. 25 Exhibit 20: Survey respondents’ top five emerging HIS development needs for the near future (next 2–5 years) (n=99) ................................................................................................................... 26 Exhibit 21: Survey respondents’ top emerging evaluation capacity development needs for the near future (next 2–5 years) (n=99)................................................................................................ 27 Exhibit 22: MEASURE Evaluation polarities ................................................................................. 28 Exhibit 23: Evaluation design matrix.............................................................................................. 56 Exhibit 24: Selected countries against USAID criteria .................................................................... 61 Exhibit 25: Online survey sample by stakeholder and method........................................................ 63 Exhibit 26: Sample by stakeholder and method.............................................................................. 64 Exhibit 27: MEASURE Evaluation Phase IV activities for years 1–3............................................ 101 February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report ii Exhibit 28: Most useful tools, as cited by stakeholders ................................................................. 106 Exhibit 29: MEASURE Phase IV activities................................................................................... 109 Exhibit 30: Most useful tools, as cited by stakeholders ................................................................. 113 Exhibit 31: MEASURE Evaluation Phase IV activities ................................................................. 116 Exhibit 32: Data collection breakdown by stakeholder group....................................................... 117 Exhibit 33: Staffing challenges under the HMIS portfolio............................................................ 122 Exhibit 34: Most useful tools, as cited by stakeholders ................................................................. 125 Exhibit 35: Respondents who have participated in a technical working group or community of practice that MEASURE Evaluation has led (Question 3 – long survey) .............................. 128 Exhibit 36: Respondents’ rating of effectiveness of technical working group or community of practice they participate in (Q4 – long survey) ..................................................................... 128 Exhibit 37: Degree to which respondents felt participating in the technical working groups/communities of practice benefited them (Q5 – long survey) ..................................... 129 Exhibit 38: Number of respondents participating in a technical working group or community of practice that MEASURE Evaluation is involved in (Q6 – long survey)................................. 130 Exhibit 39: Rating of the assistance MEASURE Evaluation provided to technical working groups (Q7 – long survey) ............................................................................................................... 131 Exhibit 40: Tools respondents have worked with (Q9 – long survey) (n=55) ............................... 132 Exhibit 41: Respondent rating of utility of MEASURE Evaluation tools (Q10 – long survey) ..... 133 Exhibit 42: Respondent awareness of MEASURE Evaluation tools that have not been produced despite the need, or tools that did not prove to be useful (Q11 and 12 – long survey).......... 134 Exhibit 43: Respondents reasons for buying into MEASURE Evaluation (Q16 – long survey) .... 134 Exhibit 44: Degree to which respondents feel MEASURE Evaluation actions encourage sustainability of efforts after the end of Phase IV (Q22 – long survey).................................. 135 Exhibit 45: Internal and external stakeholder respondents who said their organization plans to use or collaborate with MEASURE Evaluation in the future (Q25 – long survey) .......................... 136 Exhibit 46: U.S. Government Washington-based respondents’ satisfaction with MEASURE Evaluation based on their experiences with the project (Q3 – short survey).......................... 136 Exhibit 47: Internal stakeholder respondent knowledge of USAID-funded projects that work in evaluation and/or health information system (Q17 – long survey; Q10 – short survey)........ 137 Exhibit 48: Respondents’ identification of three top emerging needs in the next 2–5 years for the development of HIS in the country or region they work with/in (Q23 – long survey; Q8 – short survey)......................................................................................................................... 138 Exhibit 49: Respondents’ identification of top three emerging needs in the next 2–5 years for the development of evaluation capacity in the country or region they work with/in (Q24 – long survey; Q9 – short survey).................................................................................................... 139 Exhibit 50: U.S. Government (internal stakeholder) rating of MEASURE Evaluation performance on HIS strengthening, use of information, and conducting evaluation (Q19 – long survey; Q4 – short survey)...................................................................................................................... 140 February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report iii CDC U.S. Centers for Disease Control MOH Ministry of Health and Prevention NCDC Nigerian Center for Disease DATIM Data for Accountability Control Transparency and Impact NMEP National Malaria Elimination DCHA Bureau for Democracy, Conflict, Program and Humanitarian Assistance OGAC Office of the Global AIDS DHIS 2 District Health Information Coordinator Software 2 OVC Orphans and vulnerable children DPRS Department of Planning, Research and Statistics PEPFAR U.S. President’s Emergency Plan for AIDS Relief DQA Data quality assessment PLACE Priorities for Local AIDS Control DREAMS Determined, Resilient, Efforts Empowered, AIDS-free, PMI President’s Malaria Initiative Mentored, and Safe eLMIS/eSIGL Electronic drug logistics PNOEV National OVC Program management information system PRISM Performance of Routine FMWASD Federal Ministry of Women Information System Management Affairs and Social Development RDQA Routine data quality assessment FP/RH Family planning and reproductive RDQA+G Gender-Integrated Routine Data health Quality Assessment GEMNet-Health Global Evaluation and RHIS Routine Health Information Monitoring Network for Health System GH Pro Global Health Program Cycle SIGDEP 2 Management Tool for Electronic Improvement Project Patient Files HIS Health information systems SOAR Supporting Operational AIDS Research HMIS Health management information system TB Tuberculosis M&E Monitoring and evaluation UNICEF United Nations Children’s Fund MCHN Maternal and child health and USAID United Stated Agency for nutrition International Development MEASURE Monitoring and Evaluation to USG United States Government Assess and Use Results WHO World Health Organization MER Monitoring, evaluation, and reporting WSS Water Supply and Sanitation ACRONYMS February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report iv EXECUTIVE SUMMARY BACKGROUND The Monitoring and Evaluation to Assess and Use Results (MEASURE) Evaluation Phase IV project is the flagship mechanism for strengthening health information systems (HIS) in developing countries at the USAID Bureau for Global Health. MEASURE Evaluation supports U.S. Government (USG) offices in Washington, D.C., and works in more than 30 countries. At the global level, the project supports the development of data systems to meet USAID’s monitoring and evaluation (M&E) needs. At the country level, it provides evaluation implementation, capacity building, technical assistance, information sharing, and knowledge management to strengthen country HIS and improve host-country capacity to manage HIS. PURPOSE In August 2017, USAID/Washington contracted EnCompass LLC to conduct a midterm performance evaluation of MEASURE Evaluation to examine how effective the project has been in meeting key stakeholders’ needs. The evaluation’s purpose is to inform technical programming and project management activities for the remainder of the current cooperative agreement and support the design and scope of future global HIS procurements. EVALUATION QUESTIONS AND METHODS The evaluation responds to three overarching questions and associated sub-questions: 1. Is the project meeting the needs of its stakeholders? This question refers to needs related to HIS, evaluation, and learning for internal and external stakeholders, and seeks to examine the project’s comparative advantages and disadvantages in responding to needs, how well the project fits in the current landscape of USAID M&E and HIS mechanisms, and project processes to facilitate the sustainability of quality, performance, and use of HIS and evaluation-related work. 2. What are the benefits of a health sector–wide versus a health area–specific approach? This question examines the key facilitators and barriers the project (and USAID) faces to implement a health sector–wide approach to strengthening country HIS and evaluation. It also explores the extent to which the presence of a health sector–wide portfolio, in addition to health area–specific portfolios, facilitate or hinder the project’s effectiveness in improving HIS performance and evaluation capacity building. 3. Are the tools developed useful? This question examines the tools most frequently used and/or adapted at the country level; factors of success (and barriers) in tool development, February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report v deployment, adaptation, and dissemination; and which tools might need continuous investment for future adaptation. A five-person international team, coordinated from the EnCompass office in Rockville, Maryland, carried out the evaluation, along with three local consultants in Cote d’Ivoire, Mali, and Nigeria, between August 2017 and February 2018. The team used a concurrent, mixed-methods approach that allowed for depth and breadth in data collection and triangulation during analysis and interpretation. The team collected data from internal stakeholders (staff from USAID/Washington, USAID missions, and other USG agencies) and external stakeholders (staff from MEASURE Evaluation, partner organizations of the project, and country governments). In sum, the team interviewed or held group sessions with 117 internal and external stakeholders in country visits to Nigeria, Cote d’Ivoire, and Mali; conducted virtual or in-person interviews with 49 internal and external stakeholders in Washington and globally; conducted an online survey with 120 internal and external stakeholder respondents from Washington and around the world; and reviewed 104 project documents. FINDINGS AND CONCLUSIONS Evaluation findings and conclusions are grouped under the three main evaluation questions, with a fourth section dedicated to remaining needs for HIS strengthening and evaluation capacity building. Findings, grouped under each conclusion, are based on the triangulation of data from country visits, global-level interviews, the online survey questionnaire, and a document review. IS THE PROJECT MEETING THE NEEDS OF ITS STAKEHOLDERS? Conclusion 1: With the wide range of internal and external stakeholders’ needs, MEASURE Evaluation has been challenged to comprehensively and consistently meet all of the needs at the country and global levels. Finding 1: At the country level, internal and external stakeholders in the majority of countries covered in this evaluation perceive that MEASURE Evaluation is meeting their needs, but the project faces challenges with specific activities in a subset of countries. Finding 2: Washington-based internal stakeholders varied in their perspectives of the extent to which MEASURE Evaluation is meeting their needs. Some offices perceived that the project has met their needs as a responsive partner that provides high-quality technical assistance, but others were not satisfied with the degree of technical leadership or quality they have received. Finding 3: All stakeholder groups appreciate MEASURE Evaluation’s collaborative working style and processes, particularly when applied to technical working groups and other collaboration platforms. Conclusion 2: MEASURE Evaluation is playing a unique role in the USAID landscape in HIS strengthening, conducting health impact evaluations, and building evaluation capacity in health, despite differing views among USG stakeholders on the role of evaluation in the project. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report vi Finding 4: Internal and external stakeholders perceive that MEASURE Evaluation has a niche in the USAID landscape for HIS strengthening, and appreciate the project’s work in evaluation capacity building. Finding 5: Internal and external stakeholders perceive that MEASURE Evaluation has a reputation for and expertise in conducting evaluations, and a niche in the current USAID M&E landscape for conducting health impact evaluations. Finding 6: Although USAID/Washington perceives that MEASURE Evaluation is meeting its needs for gender M&E and HIS-related services, USAID missions show limited demand for gender-related support to HIS strengthening beyond incorporating the sex disaggregation of data. Conclusion 3: MEASURE Evaluation’s approach to technical assistance and capacity building has facilitated ownership among country-level stakeholders. Operationalization of the learning agenda and its principles is still at an early stage, but this focus is important for paving the way for more sustainable outcomes. Finding 7: To ensure sustainability, MEASURE Evaluation has facilitated broad stakeholder engagement, institutionalization of technical assistance, and local maintenance of the HIS. However, sustainability is challenged by meeting the ongoing needs for country-level financial and human resources, which most stakeholders perceive as outside the project’s scope and mandate. WHAT ARE THE BENEFITS OF A HEALTH SECTOR–WIDE VERSUS A HEALTH AREA–SPECIFIC APPROACH? Conclusion 4: Almost all stakeholders see the benefits to a health sector–wide approach to strengthening HIS, and USAID and MEASURE Evaluation have been able to leverage this approach with health area–specific approaches to achieve stronger results. However, a health sector–wide approach has been inherently challenging for USAID to manage, given its health area–specific organization and funding structure. Finding 8: Internal and external stakeholders perceive that a health sector–wide approach to HIS strengthening is highly beneficial. Yet, MEASURE Evaluation and USAID also recognize that it presents a challenge in managing demands from multiple funding streams, including health area–specific funds. Finding 9: At the country level, internal and external stakeholders see the combination of health sector–wide and health area–specific approaches as beneficial for creating a single HIS to meet the needs of all programs and stakeholders. MEASURE Evaluation has effectively leveraged multiple funding streams to support a health sector–wide HIS. ARE THE TOOLS DEVELOPED USEFUL? Conclusion 5: Many tools MEASURE Evaluation has supported, developed, and/or adapted at the country level are well-appreciated and used by their specific target audiences, both with and without project support. However, there is potential to serve a wider audience and room for broader dissemination and application, both globally and at subnational levels. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report vii Finding 10: Country-level internal and external stakeholders perceive that tools and resources supported by MEASURE Evaluation Phase IV are useful, particularly those for data quality and use and for assessing HIS. Finding 11: In the countries visited, internal and external stakeholders perceive that MEASURE Evaluation’s participatory process for tool development and adaptation, coupled with its capacity building, have contributed to tool uptake at the country level. Finding 12: Internal stakeholders perceive that tools and resources are not disseminated widely enough, although project data show positive trends for exposure to and uptake of the tools. WHAT ARE THE REMAINING NEEDS? Conclusion 6: Strengthening data quality and data use for decision making and facilitating interoperability of databases remain the pressing needs to address in order to continue progress in strengthening HIS and evaluation capacity building. Finding 13: Internal stakeholders’ suggestions for MEASURE Evaluation’s focus in the remainder of Phase IV emphasize completing current work and strengthening management, while external stakeholders focus on harmonization and sustainability. Finding 14: Internal and external stakeholders identified their top HIS strengthening priorities as improving data quality and strengthening analysis, and use through capacity building, and focusing on the interoperability of databases, especially at the local level. Finding 15: In evaluation capacity building, global and country-level internal and external stakeholders identified improved capacity to use data for decision making and improved quality of evaluations as top priorities. RECOMMENDATIONS These recommendations reflect evidence emerging from the evaluation and the evaluation team’s interpretation of the findings and conclusions. All recommendations stem from the broad set of triangulated data. 1. Given the short remaining timeline and reduced bureau-wide funds for the remainder of Phase IV, MEASURE Evaluation and USAID should streamline communication and set clear priorities and expectations for what must and can be accomplished in the time remaining. 2. MEASURE Evaluation should continue to collaborate effectively with internal and external stakeholders to strengthen HIS and evaluation capacity building to ensure that Phase IV work is completed on time and with high quality. The project should prioritize finalizing the work on the learning agenda related to HIS strengthening and evaluations, and prepare dissemination plans to ensure awareness and use of these results by the bureau, missions, and external stakeholders. 3. MEASURE Evaluation should continue to build on its Phase IV achievements in tool and resource development and adaptation, and USAID and the project should work jointly to raise awareness and disseminate more widely across USAID missions, particularly for health February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report viii area–specific and gender integration tools. At the country level, the project should focus its efforts on facilitating tool dissemination to increase the likelihood of use at subnational levels. 4. USAID and MEASURE Evaluation should work on increasing demand for the gender￾related M&E tools and resources for HIS work, including health area–specific tools, and expand work to better integrate gender considerations into HIS and evaluation data use as the project’s work on data quality and data use for decision making continues to deepen. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report ix Exhibit 1: MEASURE Evaluation stakeholders INTRODUCTION The Monitoring and Evaluation to Assess and Use Results (MEASURE) Evaluation Phase IV project is the flagship mechanism for strengthening health information systems (HIS) in developing countries at the USAID Bureau of Global Health. In mid-2017, USAID/Washington contracted EnCompass LLC to conduct a midterm performance evaluation of the project. A five-person international team, coordinated from the EnCompass headquarters in Rockville, Maryland, carried out the evaluation, along with three local consultants in Cote d’Ivoire, Mali, and Nigeria, between August 2017 and February 2018. BACKGROUND AND PROJECT CONTEXT MEASURE Evaluation seeks to empower institutions and people to identify, collect, analyze, and use technically sound information to improve global health and well-being. Its results framework (see Annex 1) includes (1) strengthened collection, analysis, and use of routine health data; (2) improved country-level capacity to manage HIS, resources, and staff; (3) methods, tools, and approaches improved and applied to address health information challenges and gaps; and (4) increased capacity for rigorous evaluations. MEASURE Evaluation is a Leader with Associates cooperative agreement with a 5- year period of performance (July 1, 2014, through June 28, 2019). It is implemented by the Carolina Population Center, University of North Carolina at Chapel Hill, in partnership with ICF International, John Snow, Inc., Management Sciences for Health, Palladium, and Tulane University School of Public Health and Tropical Medicine. It operates as an integrated, “bureau-wide” project, providing assistance across the Bureau for Global Health’s five technical offices 1 and addressing all of the bureau’s health elements and focus areas. The project is managed from the Office of HIV/AIDS and has a management team representing all bureau offices. As Exhibit 1 illustrates, the project serves and works with many internal stakeholders—U.S. Government (USG) entities that invest resources in MEASURE Evaluation in exchange for project services—and external 1 Offices of Health Systems, Office of HIV/AIDS, Office of Infectious Diseases, Office of Maternal and Child Health and Nutrition, and Office of Population and Reproductive Health. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 1 Exhibit 2: Distribution of MEASURE Evaluation funding, years 1–3, by type and source Core 27% Special Initiative 24% Field 49% Other (<4%) DATIM (SI) 12% 12% Ebola 4% FP/RH 5% Bureau-Wide Programming 9% MCH 9% HIV (without Malaria DATIM) 11% 38% stakeholders—entities outside the USG with which the project works or that are affected by or benefit from its work. MEASURE Evaluation supports a number of offices within USAID and the Office of the Global AIDS Coordinator (OGAC) in Washington, D.C., and works in more than 30 countries. 2 At the global level, the project supports the development of data systems to meet USAID and OGAC monitoring and evaluation (M&E) needs and provides technical collaboration and coordination, capacity building, and information sharing and knowledge management to strengthen country HIS and improve host countries’ capacity to manage HIS. The project has helped countries build systems for decision making, support information technology, evaluate tools, support District Health Information Software 2 (DHIS 2) 3 and Data for Accountability Transparency and Impact (DATIM) 4 software, improve data quality, and provide assistance on data analysis and use. The project received about $155 million in its first 3 years, split about evenly between central (core and special initiative) and field support funds (Exhibit 2). 2 See https://www.measureevaluation.org/countries for a list of countries. 3 DHIS 2 is a tool for collection, validation, analysis, and presentation of aggregate and patient-based statistical data, tailored to integrated health information management activities. This generic, modular, web-based, and open-source software package allows the user to design the contents of a specific information system without the need for programming. DHIS 2 was developed, with donor funding, by the Health Information Systems Programme at the University of Oslo. DHIS 2 has become the de facto global HIS standard, and is used in more than 40 countries. MEASURE Evaluation Phase IV collaborates and coordinates with the University of Oslo to establish and improved electronic HIS and to use the data for M&E, support countries’ adoption of DHIS 2 for HIS, support individual countries with DHIS 2 integration and rollout, and help them streamline data collection and reporting. Specific activities vary with the needs of each country (e.g., training Ministry of Health officials, mentoring on data management and data quality, adapting DHIS 2 tools to the local systems context, and technical support for DHIS 2 users). 4 DATIM is a is a PEPFAR-specific instance of DHIS 2. The platform (developed in 2015) brings together several PEPFAR-specific data streams—Monitoring, Evaluation, and Reporting (MER), Site Improvement through Monitoring Systems (SIMS), Expenditure Analysis (EA), and Evaluation Standards of Practice (ESoP). It is the basis for setting up priorities and targets and tracking progress of PEPFAR country programs toward achieving epidemic control. MEASURE Evaluation has played an important role in DATIM construction and maintenance, leading the data exchange work stream, managing the helpdesk, and contributing to capacity-building efforts for users from USG, implementing partners, and ministries of health, including managing USG investments by subcontracting with other key implementers (e.g., University of Oslo, BAO Systems, and Regenstrief Institute). February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 2 EVALUATION QUESTIONS USAID leadership laid out a set of evaluation questions in Request for Task Order Proposals N SOL-OAA-17-000079 (see Annex 1). The sub-questions, presented in Exhibit 3, reflect minor revisions, based on input from the August 17, 2017, evaluation kick-off meeting and the Septe 7, 2017, evaluation design meeting attended by USAID staff, MEASURE Evaluation staff, and EnCompass evaluation team. o. mber the Exhibit 3: Key evaluation questions 1. Is the project meeting the needs of its stakeholders? 1a. To what extent is the project meeting the HIS, evaluation, and learning needs of key stakeholders? 1b. What do key stakeholders consider to be the project’s comparative advantages and disadvantages in responding to their needs? 1c. To what extent does the project fit into the current landscape of USAID M&E and HIS mechanisms? 1d. What processes are in place to facilitate the sustainability of quality, performance, and use of HIS and evaluation-related work? What barriers to sustainability exist? 2. What are the benefits of a health sector–wide versus a health area–specific approach to HIS strengthening and evaluation capacity building? 2a. What are the key facilitators and barriers MEASURE Evaluation faces with respect to implementing a health sector–wide approach to strengthening country HIS and evaluation? 2b. To what extent does the presence of a health sector–wide portfolio, in addition to health area–specific portfolios, facilitate or hinder the project’s effectiveness in strengthening the collection, analysis, and use of routine health data, improving country-level capacity to manage HIS resources and staff, and building evaluation capacity? 3. Are the tools developed useful? 3a. Which tools are most frequently used and/or adapted at the country level, and how? 3b. What have been the success factors in terms of tool development, deployment, adaptation, and dissemination? 3c. What are the barriers to tool development, deployment, adaptation, and dissemination? 3d. What tools are likely to require continuous investment for future adaptation and use? Why? February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 3 EVALUATION DESIGN AND METHODOLOGY The EnCompass evaluation team (see Annex 2) used a concurrent, mixed-methods approach that allowed depth and breadth in data collection and triangulation during analysis and interpretation. The design included semi-structured interviews with internal and external stakeholders at global level and USAID missions, and with a wide range of internal and external stakeholders during three country visits; an online survey targeting internal and external stakeholders at global and country levels; and extensive document review. Exhibit 4 gives an overview of samples for each method. See Annex 3 for the list of documents reviewed and Annex 4 for details about sampling, data collection, and limitations. Exhibit 4: Data collection methods by respondents reached SAMPLING AND DATA COLLECTION Data collection tools are presented in Annex 5 (qualitative tools) and Annex 6 (online questionnaire). Country visits: The evaluation team conducted country visits to gain a 360-degree view of the project from all stakeholder perspectives (e.g., USAID, government, other USAID implementing partners, other donors, project staff). The USAID Management Team selected Cote d’Ivoire, Mali, and Nigeria for the visits, based on four criteria—diversity of portfolio, work across result areas, implementing partner in-country presence, and a USAID investment of more than $2 million. See Exhibit 24 in Annex 3 for more details on these criteria. During each 2-week visit, an international consultant and one local consultant conducted interviews, focus group discussions, observation, and February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 4 debriefings. For brief reports with more details about the countries visited, see Annex 7 (Cote d’Ivoire), Annex 8 (Mali), and Annex 9 (Nigeria). Semi-structured interviews: The sampling frame for global-level interviews was selected and prioritized in coordination with USAID, balancing the desire for wide representation with time and funding limitations. The sample included key USAID and other USG staff in Washington, USG implementing partners, and external partners. To expand exposure to USAID mission perspectives on other continents, the evaluation team conducted virtual interviews with USAID mission staff in Bangladesh and the Central America region. To gain an in-depth understanding of why missions have not bought into the project, the team reached out to several missions identified by USAID, but were able to speak with only one mission. Online survey: The sampling frame for the online survey (administered via SurveyMonkey) was developed in coordination with USAID, considering geographic reach and the desire for wide representation. Two versions of the online survey questionnaire were developed:  A shorter, more open-ended version for USAID and other USG staff based in Washington. This was sent to 37 individuals (not interviewed), with a 38 percent response rate.  A longer, more closed-ended version, in English and French, for USAID staff at missions with health portfolios (those buying into the project and those that had not) and members of global and country-level technical working groups or other collaboration platforms. The English version was distributed to 229 individuals, with a 45 percent response rate from USAID missions buying into the project (4 percent from those not buying in), and 40 percent for external stakeholders. The French version was distributed to 59 individuals, with a 27 percent response rate. The evaluation team also emailed the link to the main points of contact for each technical working group identified by the project, to share with members of their working groups; this resulted in 5 additional responses. DATA ANALYSIS Semi-structured interviews and focus group discussions: Transcripts of verbatim notes for all interviews and focus group discussions (country visits and phone/Skype/in-person interviews) were coded using an online qualitative data analysis program, Dedoose. Content analysis entailed a combination of deductive codes based on evaluation questions, followed by inductive coding drawing from the data. Online survey data: English and French responses were combined and analyzed in Microsoft Excel and in Stata 14. Sample sizes were too small to disaggregate for statistical analysis, but results are presented in Annex 10 by internal and external stakeholders. Triangulation: The core evaluation team discussed and interpreted the emerging findings together as a means of validation, identification, and testing of rival explanations for key themes. Triangulation of data from country-level interviews, other interviews, the online survey, and the document review led to mutually reinforcing findings, as well as divergent and sometimes conflicting ones. For each point of disagreement, the evaluation team re-analyzed appropriate data to explore factors that may have contributed to the disagreement and came to a reconciliation when possible. In triangulating the evaluation data, it became apparent that there were distinct perspectives on the February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 5 project’s ability to meet stakeholders’ needs, based on interaction with the project and role, indicating that the different data collection processes captured the multitude of opinions and ideas. LIMITATIONS This evaluation has several limitations, some that affect the interpretation of results and some that affected the evaluation process.  Long project history and broad project scope: MEASURE Evaluation is in its fourth 5-year implementation phase, with an extensive history and strong name recognition among stakeholders. Some stakeholders were unable to distinguish among the phases and attribute results appropriately to Phase IV or to be knowledgeable of the range of current activities. The evaluation team tried to address this issue by focusing questions on events and activities during Phase IV and by triangulating data across all sources.  Small number of country visits: Resource and time limitations allowed only three country visits (of the more than 30 countries where the project works), and all on one continent. Nonetheless, the visits provided a balance of information from the range of stakeholder perspectives. Although data from the online survey had the potential to provide information from a similar range (USAID, government entities, other partners), it was not possible to control which stakeholders responded or to ensure the same balance of perspectives from each country. The sample for USAID missions across the evaluation methods covered 34 percent of missions buying into the project, but represents 73 percent of field support funding and 58 percent of missions using HIV funds for MEASURE Evaluation activities.  Representation of USAID missions not buying in to the project in the sample: Data were limited from missions that had not bought into the project. It is important to understand how missions choose to use a mechanism such as MEASURE Evaluation. Other missions not buying in and not responding could have opinions that differ from the two that responded.  Online survey sampling and respondent bias: Response rates to the online survey were within common ranges of response rates for all groups, except among those not buying into the project. However, as with all passive survey administration, response rates are lower than those administered live, and therefore subject to some respondent bias.  Unintended bias toward HIS versus evaluation-related findings: The countries selected for visits offered less data on the project’s evaluation portfolio than the project’s HIS strengthening work, as none of these had significant evaluation portfolios. Virtual interviews were only able to include one mission with a large evaluation portfolio.  Time and financial constraints: This project is large, both in geography and content areas. A longer evaluation time frame would have allowed greater reflection on the design and the possibility to reach a larger pool of respondents and gain an even broader picture of MEASURE Evaluation’s ability to meet stakeholders’ needs. The evaluation budget did not always match USAID’s desires for sample size for interviews.  Team composition changes: Due to unforeseen circumstances, the original team lead had to depart the evaluation at the design stage. Replacement team members brought both evaluation and HIS strengthening expertise, but this shift was challenging in the short time frame. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 6 Exhibit 5: Project performance ratings from USAID mission respondents who worked with MEASURE Evaluation on a five-point scale (10 missions) Very high Above average Average Below average Very low Conducting evaluations (n=10) Collecting and using health information to make strategic decisions (n=10) Managing health information systems (n=7) 0% 20% 40% 60% 80% 100% FINDINGS The evaluation findings are grouped under the three main evaluation questions, with a fourth section dedicated to information related to future needs. Although there are no sub-headings for the evaluation sub-questions, all are addressed in this section. IS THE PROJECT MEETING THE NEEDS OF ITS STAKEHOLDERS? MEASURE Evaluation serves internal and external stakeholders at global and country levels. Its ability to meet this wide range of stakeholder needs has been mixed. At the country level, interna covered in this evaluation p but the project faces challen l and external stakeholders in the majority of countries erceive that MEASURE Evaluation is meeting their needs, ges with specific activities in a subset of countries. The project has been successful in many countries, but has had significant difficulties in at least part of its portfolio in others. Overall, data from the three country visits and interviews with other country missions indicate that the majority of internal stakeholders, and almost all external stakeholders, reported that the project was meeting their needs to a high degree. MEASURE works to fulfill all our needs. Until now, we have not needed to seek expertise outside of MEASURE. The technicians have always been at the highest level, and they are here with us (we are one team). This is extremely important. —Government, Mali MEASURE has been very supportive and a great partner. —USG, Guyana Data from the online survey from an additional 10 missions indicate positive perceptions of the project’s performance with regard to conducting evaluations, strengthening HIS collection and use of data, and systems management. USAID mission respondents rated its performance as very high or above average (Exhibit 5). Internal and external stakeholders in the countries visited cited specific examples of ways the project has led or contributed to technical successes (Exhibit 6). When mission-level USG respondents to the online survey were asked if they planned to use MEASURE Evaluation in the future, 50 percent (7) responded “yes,” 50 percent (7) did not know, February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 7 Exhibit 6: MEASURE Evaluation’s frequently cited technical successes from country visits Mali: • Based on the Performance of Routine Information System Management (PRISM) assessment, advocating with national actors and donors to adopt DHIS 2 as the HIS platform • Providing technical support for DHIS 2 rollout, enabling early release of the 2016 Annual Statistics Report • Adapting the RDQA tool, which enables identification of data gaps, visualization of trends, and real-time decision making • Enabling availability of epidemiological data for decision making Nigeria: • Reactivating National OVC Management Information System • Leading the OVC Technical Working Group • Conducting a PEPFAR OVC Outcome Monitoring Survey • Conducting a secondary Analysis of Nigeria Malaria Indicator Survey • Building Ministry of Health capacity through a malaria surveillance workshop • Inaugurating the Health Data Governance Council • Supporting the HMIS Technical Working Group Cote d’Ivoire: • Implementing DHIS 2 • Supporting rollout of the Electronic Drug Logistics Management Information System (eLMIS) • Supporting the transition from the Electronic Medical Record for HIV Patients to the web-based SIGDEP 2 • Assessing health system performance using HMIS data • Institutionalizing harmonized HMIS data quality assurance and none (0 percent) responded “no.” Among those who did not know, several mentioned the uncertainty of future funding. Among external stakeholders, 81 percent (of 54) said yes, 15 percent said they did not know, and 4 percent said no. Despite primarily positive feedback from country-based stakeholders across data sources, there were instances of dissatisfaction. In Cote d’Ivoire, internal stakeholders noted that the project was sometimes slow in communication with USG implementing partners, and that it had a thinly stretched project team (see the country report in Annex 7). In Nigeria, several internal stakeholders were dissatisfied with specific (and sometimes key) activities, and some external stakeholders noted that staff turnover and insufficient numbers of in-country staff, as well as a shifting focus, have challenged the project in building relationships and ensuring continuity of work. Government partners in Nigeria stated that they felt the project was not transparent enough about funding priorities and was lacking in overall communication (see the country report in Annex 9). MEASURE should just be open and transparent. It is like having 10 things and they show us two, and when we ask for the remaining eight, they tell us their donor said this or that. —Government, Nigeria Factors influencing whether stakeholder needs were met were consistent across the three countries visited. Where facilitating factors existed, MEASURE Evaluation tended to be successful in meeting stakeholder needs in that country. Where these factors did not exist, the project faced challenges. One critical factor for success is strong communication between various stakeholders and the project. For example, continuity of project staff in Mali and Cote d’Ivoire allowed for building relationships with government and other USG partners, as well as the mission, and this facilitated significant collaboration that made MEASURE Evaluation more successful. In Nigeria, the gap in funding February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 8 Exhibit 7: Summary of most-reported success and inhibiting factors in three countries Inhibiting Factors Success Factors Sphere of Influence Project Sphere of Control Sphere of Influence  Limited government  Insufficient project staff  Continuity of project staff  Government leadership and and staff turnover leadership and  Long history, strong ownership ownership for HIS  Slow response time reputation  Inconsistent USAID  USAID mission  Lack of leadership  Expertise of project staff mission interest and interest in and support to provide leadership and  Lack of clear support in HIS for HIS technical guidance communication on  Poor internet connectivity work plans and  Strong collaboration and and availability in certain timelines with coordination with donors, parts of the country government and government, USAID, and  Incomplete coverage of partners others computer and related  Commitment to equipment at facility level government ownership for data entry and  Ability to provide key analysis resources for national  Poor interoperability implementation, while across government mobilizing partners to fill systems gaps at subnational level contributed to loss of local staff and a minimal on-the ground presence, which challenged relationship-building and led to communication breakdowns among the USAID mission, the project, and the government in many instances. Exhibit 7 summarizes these factors, indicating which are within the project’s sphere of influence and which are within its sphere of control. 5 Certain inhibiting factors that internal and external stakeholders reported were both outside of the project’s sphere of influence and outside its sphere of control: changing indicators from the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) and other centrally funded initiatives; short USAID project timelines, limiting the ability to see long-term results; turnover of USAID staff and government counterparts; and project funding limited to the national level versus the subnational level. For example, in all three countries visited, MEASURE Evaluation funding was earmarked only for national-level activities. Though the project used its influence to affect subnational work through coordination and mobilization of efforts across partners, work at the subnational level was ultimately funded and run by other donors and other USAID projects, sometimes leading to inadequate results beyond the project’s control. Washington-based internal stakeholders varied in their perspectives of the extent to which MEASURE Evaluation is meeting their needs. Some offices perceived that the project has met their needs as a responsive partner that provides high-quality technical assistance, but others were not satisfied with the degree of technical leadership or quality they have received. 5 The concept of spheres of control and influence is from Covey’s Seven Habits of Highly Effective People (1992) and is the basis of outcome mapping in the evaluation field. The sphere of control includes those things in a system that a project can change or determine. The sphere of influence includes activities over which a project can have some degree of impact, but does not exercise full control. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 9 Exhibit 8: Ratings from USG respondents who worked with MEASURE Evaluation on the project’s performance in building capacity on a 5-point scale Very high Above average Average Below average Very low Conducting evaluations (n=9) Collecting and using health information to make strategic decisions (n=11) Managing health information systems (n=9) 0% 20% 40% 60% 80% 100% In the online survey and interviews, Washington-based internal stakeholders provided a range of opinions on MEASURE Evaluation’s performance, speaking to their own direct needs, as well as those they heard from mission staff who communicate with them. Respondents from the Office of HIV/AIDS, the office that manages the project, more frequently reported dissatisfaction with project management and technical leadership, and expressed frustration with certain country-level activities. [The USG] was not pleased with the outcome [of the evaluation work]: it was not up to the standard. Speed, efficiency, and flexibility were not there. It needed innovativeness and design. The project’s approach was very cookie-cutter: they pushed back on criticisms, identified limitations without solutions. —USG, HIV Interviewees working on health area–specific work, such as those in malaria, infectious diseases, and population and reproductive health tended to be more satisfied, although they have narrower scopes and interactions with the project. In particular, the project’s work with DATIM, which accounts for 12 percent of its funding overall (see Exhibit 2), was viewed positively. While I emphasized the things that could be strengthened, we do think we are getting exceptionally good value for money [for DATIM], and are very happy with the delivery side of things. —USG, HIV Overall, … for malaria-specific activities, provided by [project consortium partners], they have high marks in technical assistance in the country and core level. —USG, non-HIV MEASURE got rave reviews for [the] Global Development Lab (Liberia)—which [supported] them with Ebola funds … It was not a lot of money, but MEASURE’s work was well regarded, and well appreciated. They had good people to do the work. Everywhere I go, they are appreciated. —USG, non-HIV As Exhibit 8 illustrates, 65 percent of Washington-based USG survey respondents6 rated the project’s performance as above average or very high in building HIS and evaluation capacity. Satisfaction with specific MEASURE Evaluation activities reported from the online survey also showed a range of perceptions. A majority of USG respondents were satisfied, although a few individuals (working in HIV) were very unsatisfied with certain performance aspects (Exhibit 9, with more details available in Exhibit 46 in Annex 10). 6 Individuals completing the online survey questionnaire were different from those interviewed; there was no overlap. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 10 Providing technical leadership (n=13) Coordinating with other stakeholders (n=12) Coordinating with USAID (n=12) Leading technical working groups (n=9) Delivering tools and products in a timely manner (n=13) Delivering useful tools and products (n=12) 0% 20% 40% 60% 80% 100% Exhibit 9: USG respondents’ satisfaction with MEASURE Evaluation on a 5-point scale Very satisfied Satisfied Neutral Unsatisfied Very unsatisfied In interviews, USG respondents cited project weaknesses such as unresponsiveness to USAID feedback, lack of innovation, cost and time inefficiencies, and, as one respondent put it, what felt like a “constant tug-of-war.” Some USAID/Washington interviewees felt the project’s work was “too academic and theoretical” and that it “has not been able to keep up with developments in health informatics and HIS,” particularly related to rapidly evolving data systems for PEPFAR, tuberculosis (TB), and malaria. USG interviewees and online survey respondents acknowledged that some of the project’s challenges are due to its expansive scope and the varying needs and expectations of its many stakeholders (see Exhibit 1). [MEASURE Evaluation’s] mandate/scope and activities seems to span such a wide spectrum of activities that it’s hard to know what they truly excel in anymore, and there is a strong perception that they may have cast their net too wide and therefore have lost their technical edge in any specific area. —USG, Washington Exhibit 10 summarizes Washington-based USG interviewee perceptions of the project’s strengths and weaknesses. As with country-level performance, some factors emerge as both a strength (when it is perceived to exist) and a weakness (when it is perceived as not existing). February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 11 Exhibit 10: USG/Washington interviewees’ differing perceptions on strengths and weaknesses of MEASURE Evaluation Perceived Weaknesses Perceived Strengths Cost/Financial Management  Very costly  Good value for money  Slow and cumbersome financial reporting processes Staffing  Inadequate use of implementing partners’ technical  Consortium of diverse implementing partners with experts recognized technical expertise  Staffing shortages, especially of technical experts  In-country teams staffed with technical experts  Reliance on U.S. consultants (vs. local experts) Processes  Inadequately responsive to USAID needs  Poor coordination with in-country implementing partners  Reactive versus proactive with internal stakeholders  Some instances of poor coordination among consortium partners  Adequate responsive to USAID and OGAC guidance  Good coordination with in-country USG implementing partners  Proactive with external stakeholders  Wide geographic coverage providing opportunities for South-to-South and project-wide learning Technical Assistance  Not a thought leader  Not able to be innovative and relevant  Overly academic; not practical  Insufficient capacity for analysis  Impractical impact evaluations  High-quality technical assistance (management and technical skills)  Tools tailored to different contexts  Rigorous evaluations and evaluation capacity building Vision  Overly activity-oriented  Support to promote country-level vision of strong HIS  Not visibly delivering on the learning agenda through DHIS 2 All stakeholder groups appreciate MEASURE Evaluation’s collaborative working style and processes, particularly when applied to technical working groups and other collaboration platforms. Data from the country visits, interviews, and the online survey indicate most global and country stakeholders’ strong appreciation for MEASURE Evaluation’s collaborative working style and processes. Country-level stakeholders cited the project’s process for collaboration and coordination as fostering buy-in, engagement, and ownership in HIS strengthening efforts. Washington-based USG online survey respondents rated the project highest for its coordination with other stakeholders, with 67 percent “satisfied” or “very satisfied” (see Exhibit 9). The majority of USAID mission interview respondents cited a variety of ways the project is coordinating effectively. In Mali (see Annex 8), internal and external stakeholders both noted that the project has created an environment of collaboration, in which government and other partners feel consulted and feel they can solve problems together. One interviewee remarked that the project “leads from behind,” providing technical expertise and direction in supporting the routine HIS and DHIS 2, while allowing space for capacity strengthening and national actors to take ownership. Stakeholders in Mali also appreciated that although the project’s resources for the DHIS 2 rollout focused at the national level, it coordinated with USAID and the Ministry of Health to mobilize February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 12 Exhibit 11: Country-level survey respondents’ ratings on benefits of participating in groups that MEASURE Evaluation led, with weighted average on a 5-point scale (n=43) Strongly agree Agree Neither agree nor disagree Disagree Strongly disagree Enhanced my job performance Enabled me to locate useful knowledge and resources Enabled me to get advice from others on technical issues 0% 20% 40% 60% 80% 100% Enabled me to learn about similar work of others Enabled me to learn about strengthening evaluation Enabled me to learn about strengthening HIS other partners to fill gaps at the subnational level. In Nigeria (see Annex 9), internal and external stakeholders noted that the project’s work in their orphans and vulnerable children (OVC) portfolio included collaborative work with government implementing partners to support government in the reactivation of the National OVC Management System. Respondents in all three countries reportedly saw themselves owning the system and receiving support they needed from the project, as articulated by one respondent below. MEASURE set up the [DHIS 2] steering committee and the technical committee. I have never seen a partner that has generated so much excitement—people have been so motivated. For example, on equipment issues, UNICEF did not hesitate to commit. The partners shared the responsibilities for the supply of computers, etc. There was great collaboration among the partners, despite their legendary rivalry! —Government, Mali Stakeholders across countries and at the global level agreed that the project’s facilitation of technical working groups, communities of practice, and other groups was beneficial. In the online survey, 45 country-level members of a global or country-level technical working group or community of practice agreed strongly or very strongly with statements about the benefits of participating in these groups (Exhibit 11). Non-government respondents appeared more likely to feel strongly about how they benefited than host-government respondents. Online survey respondents (almost exclusively country-level respondents) participating in a technical working group or other collaborating platform in which MEASURE Evaluation also participated were very positive about the degree to which the project was helpful in assisting these groups (Exhibit 12) particularly related to providing or creating useful tools or resources and helping these groups reach their goals. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 13 Exhibit 12: Survey respondents’ satisfaction ratings with MEASURE Evaluation’s helpfulness to the groups in which it participated on a 3-point scale Very helpful Moderately hepful Not involved Providing or creating useful tools or resources (n=34) Facilitating learning (n=33) Convening stakeholders (n-33) Disseminating knowledge (n=33) Achieving the group's goals (n=34) 0% 20% 40% 60% 80% 100% Internal and external stakeholders perceive that MEASURE Evaluation has a niche in the USAID landscape for HIS strengthening, and appreciate the project’s work in evaluation capacity building. Many internal stakeholders and almost all external stakeholders at the country and global levels recognized the important HIS strengthening role of a project such as MEASURE Evaluation. Internal and external stakeholders highlighted the project’s contributions in HIS governance and leadership, HIS and data management, data quality, information products and dissemination, and HIS performance strengthening. When stakeholders were prompted in interviews and the online survey, few could think of an alternative project to perform the same role in HIS strengthening. Among USG online survey respondents, only 40 percent were familiar with the Digital Health Initiative, which also works in HIS strengthening. If you take out MEASURE Evaluation, I’m not sure any other person, agency, or group that has addressed that gap at all, so whatever progress they have seen, a large proportion of that can be attributed either directly or indirectly [to] MEASURE. —Country-level partner, Nigeria The MEASURE project has great expertise. The USAID team felt comfortable with this team, due to their technical expertise … [Other USAID mechanisms] do not have the same level and depth of technical expertise. —USG, Washington In country-level interviews, internal and external stakeholders listed a number of reasons for the project’s comparative advantage in leading HIS strengthening: its reputation and expertise; its ability to build on successes, such as institutionalization of data quality assessments (DQAs) into government systems and institutionalizing M&E curricula in universities; its global presence; the number of committed technical staff; and its knowledge of the country context. A mapping of 79 USAID-supported M&E mechanisms shows that MEASURE Evaluation is one of the few projects with a specific mandate for M&E capacity building of local and regional partners, and the only one in the impact evaluation arena for the health sector. Many internal and external stakeholders praised the project’s evaluation courses. The work with the GEMNet-Health [Global Evaluation and Monitoring Network for Health] team has helped me design a stronger evaluation proposal for submission. I am able to confidently teach in a five-day workshop on M&E fundamentals, and another workshop on impact evaluation. —External stakeholder, global February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 14 Exhibit 13: USAID missions’ reasons to work with MEASURE Evaluation (n=13) Strong reputation in evaluation capacity building (n=9) Familiarity with past services and performance (n=8) Strong reputation in HIS strengthening (n=7) Ease of buy-in into mechanism (n=5) No better alternative mechanism, project, or… 0% 20% 40% 60% 80% 100% Exhibit 14:Types of evaluations and related activities in MEASURE Evaluation Phase IV (n=53) Performance Impact evaluation evaluation (counterfactual) (outcome, process) 25% 23% Other studies, surveys, MER outcome assessments monitoring survey* 23% 30% We have had several trainings on M&E by MEASURE. … The training was very, very useful, not only for me but everybody that attended; both facilitators and participants from different states … Capacity in evaluation globally has been a strength of the project, related [to] individual capacities and capacity building and transfer. —Government, Nigeria USAID mission online survey respondents most often cited MEASURE Evaluation’s strong reputation in evaluation capacity building and HIS strengthening and its past performance as reasons they chose to work with the project (Exhibit 13). Internal and external stakeholders perceive that MEASURE Evaluation has a reputation for and expertise in conducting evaluations, and a niche in the current USAID M&E landscape for conducting health impact evaluations. Conducting rigorous evaluation is MEASURE Evaluation Result Area 4. The Phase IV evaluation portfolio includes 53 evaluations and related activities, 32 percent of which are complete, 45 percent in implementation, and 23 percent in the planning or design phase. Exhibit 14 breaks out these activities by type. * The Monitoring, Evaluation, and Reporting (MER) OVC Essential Survey Indicators, required under PEPFAR MER guidance, provide a snapshot of project outcomes at a point in time and allow assessment of changes in outcomes among OVC project beneficiaries over time to provide evidence to the U.S. Congress on the key outcomes of OVC programming. See https://www.measureevaluation.org/resources/publications/sr-17-140/ Respondents both praised and criticized the project for its evaluation work. The online survey showed positive views of project performance related to conducting evaluations and related studies (see Exhibit 5 and Exhibit 8), with this aspect rating highest. Information from the country visits February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 15 referred to performance evaluations, which is not representative of the project’s evaluation portfolio. In Nigeria, the PEPFAR OVC Monitoring Outcome Survey was appreciated for providing the mission with a baseline to assess its implementing partners working on OVC (see Annex 9). The project was also recognized for facilitating use of results from a previous malaria indicator assessment to inform the design of its malaria surveillance workshop. USAID online survey respondents were asked to note their awareness of projects offering evaluation services or research that could serve missions or the Bureau for Global Health. Aside from MEASURE Evaluation, respondents were most familiar with Supporting Operational AIDS Research (SOAR), bilateral or regional M&E platforms, the Global Health Program Cycle Improvement Project (GH Pro), and the Digital Health Initiative (see Annex 10 for more detail). USAID interviewees in Washington and at missions noted MEASURE Evaluation’s unique expertise in impact evaluation and their demand for impact evaluations conducted by the project. Nobody ever wanted to do an impact evaluation; now all bilateral projects ask MEASURE Evaluation to do an impact evaluation. The impact evaluation of [project X] gave [USAID] a lot of food for thought [on] what changes are needed … All impact evaluations are done through MEASURE. —USG, Bangladesh [T]he strength of MEASURE is that they do impact evaluation and we don’t have a lot of mechanisms that do that … It is something we need; it is a niche area. —USG, Washington However, some USG online survey respondents (Washington and missions) noted that there have been issues with the project’s evaluation work. In working with MEASURE Evaluation on a large-scale evaluation, it was surprising that the team did not take the initiative to understand the M&E system and indicators already in place for the intervention being delivered. What was more disappointing is that even after repeatedly highlighting this, the team still did not and has not made any real effort to understand these data and utilize these collected data to interpret the evaluation. —USG, Washington Although USAID/Washington perceives that MEASURE Evaluation is meeting its needs for gender M&E and HIS-related services, USAID missions show limited demand for gender-related support to HIS strengthening beyond incorporating the sex disaggregation of data. MEASURE Evaluation has made a concerted effort to give a “gender sweep” to all its work on tool development and integrate gender considerations in assessments or analysis. Most USAID/ Washington respondents who were aware of the project’s work in gender concurred that it was highly attentive to gender concerns in developing tools and resources and other routine work. For the most part they have well integrated [gender] … where gender makes sense, with PLACE [Priorities for Local AIDS Control Efforts] or evaluation tools, monitoring, developing indicators, guidance on M&E … they have done that very well. —USG, Washington However, there seems to be limited country-level demand for gender integration work. Only 5 of the 32 countries where the project works have specified gender activities in their MEASURE Evaluation portfolios, although 42 percent of core (n=12) and 30 percent of special initiatives (n=10) involve gender activities. In the online survey, 13 percent of the 56 respondents had worked with the Guidelines for Integrating Gender into an M&E Framework and Systems Assessment. When asked about February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 16 the project’s gender integration work, interviewees in the country visits primarily mentioned sex disaggregation as the way HIS activities addressed gender issues. In the online survey, when asked to list their top three emerging HIS development needs, only 13 percent of 98 internal and external stakeholder respondents noted an increased use of sex￾disaggregated data from HIS, and only 17 percent cited improving national institutions’ capacity to demand and use evidence from equity-focused and gender-responsive evaluations for planning. Some Washington-based internal stakeholders working in non-HIV health areas noted that they did not see a clear application of gender to their work, or see a demand for it. To ensure sustainability, MEASURE Evaluation has facilitated broad stakeholder engagement, institutionalization of technical assistance, and local maintenance of the HIS. However, sustainability is challenged by meeting the ongoing needs for country-level financial and human resources, which most stakeholders perceive as outside the project’s scope and mandate. In USAID’s Local Systems Framework, sustainability is defined as the “the ability of a local system to produce … valued results and its ability to be both resilient and adaptive in the face of changing circumstances,”7 and the agency’s website leads with a quote from Administrator Mark Green: “The purpose of foreign assistance is to end the need for its existence.”8 MEASURE Evaluation specifies four main project principles for sustainability, in line with these values and definitions. 9 As the sample quotes in Exhibit 15 illustrate, the project’s processes and approaches align well with principles 1, 2 and 3, although there are concerns about the ongoing availability of human and financial resources to maintain the system by host countries (principle 4). Data from the three country visits indicate that the project has been effective in ensuring that the government is in the driver’s seat for country-level HIS strengthening and galvanizing groups of actors to strengthen and monitor HIS strengthening efforts. Respondents across all three countries pointed to participatory processes for tool development and adaptation, and to the cadre of project￾trained government counterparts who feel technically competent to manage and troubleshoot information systems to ensure HIS sustainability. 7 See https://www.usaid.gov/sites/default/files/documents/1870/LocalSystemsFramework.pdf, p.5. 8 See https://www.usaid.gov/ 9 These principles emerged from a 2015 meeting specifically convened to define sustainability for the project. The definition is available in MEASURE Evaluation P4 Key Operational Definitions (provided to the evaluation team on October 9, 2017), building from USAID’s Vision for Health Systems Strengthening: 2015–2019 (https://www.usaid.gov/sites/default/files/documents/1864/HSS￾Vision.pdf). February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 17 Exhibit 15: MEASURE Evaluation’s contributions to sustainability in three countries visited Red text denotes principles not fully addressed. Principle 1. Regional and national leadership with broad stakeholder engagement: The country government is the architect of the health sector response and the designer of its HIS with support from partners. During the DHIS setup workshops … there were additional indicators, but we always found a consensus of what to include, what to leave out. MEASURE, in these discussions, remained neutral and left it to the government to decide. —Government, Mali Principle 2. Institutionalization and routinization: Technical assistance should be demand-driven and embedded in complex national systems through routine and regular processes managed by national actors. The experience with MEASURE Evaluation was made through the setting up of electronic tools (SIGDEP 2, eSIGL) … put in place with the collaboration of MEASURE Evaluation, but the [Directorate of Informatics and Health Information] has taken the lead and this allows a good transition with the delegation of tasks. —Government, Cote d’Ivoire Principle 3. Process and direction: Local stakeholders are proactively and regularly engaged in a process to ensure the maintenance and improvement of the HIS. Putting [DHIS 2] in place is not easy. Now that the system is in place … the National Health Directorate and the Center for Planning and Statistics have taken over the system, and they are able to follow the system and respond to ministry requests for information. In terms of achievements … [in the] the coordination meetings … the analysis of strengths and weaknesses allows the partners and the government to solve the problems, and so everyone is aware. —External partner, Mali Principle 4. Resource mobilization and management: A country’s commitment to finance its own systems as donor financing and support of system operations, maintenance, and development diminishes. I told them the best way is for partners to do advocacy to government to start supporting their own project. … If they keep bringing free money, we think if we don’t do it, that someone else will be there to do it for us. So, the best thing for partners to do is to go back and do advocacy. —Government, Nigeria Data from other USAID missions and the online survey indicate that the project is contributing to sustainability in its work to develop a culture of data use, build national capacity to maintain the system, and encourage ownership of the system (Exhibit 16). These data also indicate that the work is not finished. The project’s learning agenda focuses heavily on understanding the dynamics of developing and operating a sustainable HIS. This is a work in progress. I have scoured USAID literature for priorities and strengthening of HIS. There is not much at all … One sustainable thing we are leaving is that map of the HIS world and I am hoping that going forward, USAID will begin to formally adopt that as the map of where HIS needs to go. —MEASURE Evaluation However, the project faces some countervailing factors in being responsive to internal stakeholder needs and fostering sustainability. There is often a tension between USAID’s need for quick work and the time required to build ownership and sustainability. I don’t think they are doing a good job [internally]. They aren’t paying attention to country ownership—just reacting to mission requests … They don’t define the problem or try to educate the client, to say, “If you want us to do this, there are other things that need to be done, or increase the time duration for this activity. That isn’t the way it works.” … there are limitations but also opportunities to engage their client. —USG, Washington February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 18 Exhibit 16:USAID mission survey respondents’ perceptions of MEASURE Evaluation's encouragement of sustainability Great extent Some extent Contributed to development of a culture of data use at decentralized levels of the health system (n=7) Built adequate capacity of national counterparts to update the HIS as needs evolve (n=6) Implemented activities in ways that encourage true national level ownership and leadership (n=6) Built adequate capacity of national counterparts to conduct rigorous evaluations (n=4) Facilitated coordination and collaboration with other partners supporting HIS strengthening (n=7) Contributed to development of a national level culture of data use (n=10) Co-developed ongoing financing of recurrent HIS system costs (n=11) Neutral Very little Not at all 0% 20% 40% 60% 80% 100% Internal and external stakeholder concerns about the sustainability of HIS strengthening efforts at the country level focused heavily on factors related to meeting stakeholder needs, cited in Finding 1. These factors are generally outside of MEASURE Evaluation’s sphere of control and its (technical and geographic) mandate to support the national level, although sometimes within its sphere of influence: inadequate in-country resources (human, material, and financial, particularly at the implementation level for recurrent costs and ongoing activities); retention of government staff trained by the project; continual changes to indicators from external bodies, requiring significant adaptations that make it hard to stabilize the system; and the short duration of USAID projects or timelines, which sacrifices ownership for speed. [I]n the PEPFAR world, the expectations of what would be collected and used has become quite large and I think that keeping up with those demands has been difficult. Particularly in PEPFAR countries, developing systems for data collection is difficult. … All things considered, MEASURE has done quite a bit to build capacity, but they have had to do it on a shoe string. —USG, Washington [S]ustainability should also be a top plan you should have so that it can continue. [USAID] programs are usually 2 to 3 years … it doesn’t give us time to plan for takeover and continue running with it. It always ends at the time we are trying to gather momentum to take over. —Government, Nigeria WHAT ARE THE BENEFITS OF A HEALTH SECTOR–WIDE VERSUS A HEALTH AREA–SPECIFIC APPROACH TO HIS STRENGTHENING AND EVALUATION CAPACITY BUILDING? MEASURE Evaluation has three funding streams (see Exhibit 2): core funds, negotiated with the USAID Management Team; field support funds, received from and negotiated with country missions; and special initiative funds, in which an activity is defined by emerging USAID priorities February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 19 inside or outside the Bureau for Global Health. A portion of core funds are “bureau-wide” (health sector–wide) funds, generated through a negotiated contribution across Bureau for Global Health offices and used to support health system strengthening efforts in HIS and evaluation. Bureau-wide funds make up about one-third of the project’s core funds and 9 percent of its overall funding. Most country-level funds address evaluation or HIS strengthening across multiple funding sources, although half of project-supported countries have only HIV funding. Bureau-wide funds provide an opportunity to prevent “siloing” of health-specific activities by creating systems, tools, and approaches that are relevant to any health area and translating lessons learned in one area to others. Internal and external stakeholders perceive that a health sector–wide approach to HIS strengthening is highly beneficial. Yet, MEASURE Evaluation and USAID also recognize that it presents a challenge in managing demands from multiple funding streams, including health area–specific funds. Internal and external stakeholders agreed that the project’s ability to support and apply a health sector–wide approach for HIS strengthening, including supporting DHIS 2 rollout, has facilitated its ability to respond to country governments’ priorities, enabling it to strengthen HIS across health areas. The reality in [the] field is that people are not building siloed systems—even if money comes in siloed ways … A cross-bureau project allows for greater flexibility to meet the needs in the field. —USG, Washington In government, the facilities are not sector specific and it is helpful to get ownership with not just being in one area or another, but being across the board. It makes it easier for MEASURE because facilities are not sector￾specific. —USG, Nigeria USAID mission and USAID/Washington respondents acknowledged that although they prefer a health sector–wide approach to HIS strengthening, it is sometimes complicated and challenging to manage, given the need to ensure that activity funding meets mandatory requirements. PEPFAR and PMI [the President’s Malaria Initiative] have specific objectives and targets they have to meet, and those targets aren’t strengthening HIS; it is to put people on treatment. For PMI to support anything, they have to draw a direct line to how they can achieve their targets; otherwise the coordinator … was not going to approve it for funding. —USG, Washington The more integrated [the project is], the more difficult to get across attribution. How do I report to Congress about malaria in a nutrition activity? —USAID mission, Nigeria At USAID/Washington and USAID missions, internal stakeholders acknowledged that multiple funding streams funded across a health sector–wide approach can create challenges for the project in responding to a variety of stakeholder needs. [W]e place a lot of different demands on MEASURE. They get funding from 10 different funding sources and each funding unit has their own ideas and they have to be responsive … [O]verall, I give them 80 percent marks. Some of those things suffer partly because of what we demand of them, partly because we pull them in so many directions, but also partly because they aren’t as responsive as they should be. —USG, Washington February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 20 At the country level, internal and external stakeholders see the combination of health sector–wide and health area–specific approaches as beneficial for creating a single HIS to meet the needs of all programs and stakeholders. MEASURE Evaluation has effectively leveraged multiple funding streams to support a health sector–wide HIS. Country-level internal stakeholders and country-level and global external stakeholders reported that the concurrent portfolio of health sector–wide and health area–specific funds has allowed the project to leverage its sector-specific knowledge and relationships in its HIS efforts. [MEASURE Evaluation] is building the health system by its work on DHIS 2. At the same time as it is working in malaria, it is also paying attention to the health system issue, trying to connect the malaria intervention to the entire health system, so working on both sides. It is a very good approach. —External stakeholder, global At the country level, there was evidence of leveraging learning across the project’s portfolios. MEASURE Evaluation staff in Mali (see Annex 8) and Nigeria (see Annex 9) reported sharing information and insights across funding streams to advance their learning agenda. In Nigeria, for example, project staff working on the National OVC Management Information System reported sharing insights and ideas from their successes with colleagues working to the National Health Management Information Systems (HMIS) and surveillance. It is clear that many missions see the benefit of this approach and are pooling some of their health area–specific funding streams to enable the project to build a strong, integrated (health sector–wide) HIS platform. In Mali, for example, alongside strengthening specific aspects of malaria and disease surveillance data systems and data use, the mission has pooled malaria with other specific funding streams (e.g., Maternal and Child Health and Nutrition [MCHN] and Family Planning and Reproductive Health [FP/RH]) for MEASURE Evaluation’s DHIS 2 support. The mission supports the project by ensuring that the funding streams get their data reported back to PMI, PEPFAR, and other funders. The Nigeria mission has also pooled HIV, MCHN, and FP/RH funds for its work on HIS. There is also evidence of health area–specific activities supporting the broader health sector– wide HIS in Cote d’Ivoire (see Annex 7). The project focus can enhance non-PEPFAR [areas and] until now has integrated [non-HIV areas] more informally. SIGDEP 2 [the electronic patient file management system] used malaria and TB indicators [as well], and MEASURE was able to help with DHIS 2 as a whole. HIV HIS experts have worked for other systems in health: DHIS 2 is multi-health-specific. —USG, Cote d’Ivoire During the 2014 Ebola outbreak, MEASURE Evaluation in Cote d’Ivoire helped the National Public Health Institute strengthen its early-warning epidemiological surveillance system by building capacity in epidemic response data analysis and use at regional, district, and community levels, and by assisting the Ministry of Health to develop and implement an early warning system. These efforts were leveraged to integrate early warning data into DHIS 2 and conduct investigations of other epidemic-prone diseases. Similar examples for disease surveillance and malaria were noted in Mali. ARE THE TOOLS DEVELOPED USEFUL? MEASURE Evaluation has developed a large number of general and country-specific tools as part of its efforts to meet stakeholder needs, including assessment toolkits, data collection tools, guidelines, February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 21 training resources, and indicator compendia. The project reports having developed 77 tools and updated 13 in Phase IV, including country-level tools for 15 countries. In some cases, such as DATIM or DHIS 2, the project did not develop the tool itself, but is working with others in its development or to support its rollout, or both. Country-level internal and external stakeholders perceive that tools and resources supported by MEASURE Evaluation Phase IV are useful, particularly those for data quality and use and for assessing HIS. Across all data sources, internal and external stakeholders identified tools and resources developed and adapted by the project as useful, user-friendly, well-structured, clear, comprehensive, and helpful for building capacity. [W]e will use [the OVC data collection tools] to make decisions. OVC tools changed our reporting … Individual partners used to report but we couldn’t see it on a country-wide scale. … Because of the additional indicators, we review and we see changes over time. … The tools have also created a synergy with OVC implementing partners, who see there is need for capacity building to do better reporting. —USAID mission, Nigeria Data from the three country visits provide a view of the most frequently cited tools across the range of stakeholders, based on country-level activities (Exhibit 17). In some cases, the project developed the tools were directly; in others, it supported their adaptation or interoperability in the country context, building from experiences in other countries. Exhibit 17: Country visit respondents’ most-cited tools with which MEASURE Evaluation is involved, in order of importance Mali Nigeria Cote d’Ivoire  Mali DHIS 2 User Guide  Nigeria Data Quality  Cote d’Ivoire DHIS 2 support tools, Review/DQA SIGDEP 2, eLMIS, OVC database  Mali Health Information Data Management Support  Nigeria Health Information Data and training manuals (RDQA) Monitoring Tool Management Support (RDQA)  HIV Indicators Dictionary Monitoring Tool  Mali HIS standard manual  National Health Indicators Dictionary and procedures  Nigeria RHIS tool  Data Quality Review tool  Mali Surveillance HIS  OVC data collection tools  RDQA Standard and Procedures  Malaria M&E training  Cote d'Ivoire Adapted Data Use/ Manual curriculum Applied Epidemiology Curriculum The online survey questionnaire included a list of tools the project had developed or was involved in supporting rollout and/or application. Fifty-five respondents reported using at least one of the tools listed (see Exhibit 40 in Annex 10). The most frequently cited tools were DHIS 2 (version after 2014), the RDQA and Data Quality Review (version 2.1), the Performance of Routine Information System Management (PRISM) tool, the PLACE update, PEPFAR Dashboard 3.0, PEPFAR M&E and Reporting, and the Results-Based Financing Indicator Compendium. Of these most frequently used tools, PLACE and PRISM were developed by MEASURE Evaluation, whereas the project worked with other stakeholders or partners on DHIS 2 (University of Oslo), Data Quality Review (World Health Organization, Global Fund, and Gavi), and DATIM and PEPFAR Dashboards (PEPFAR and University of Oslo). Among all respondents who claimed to have ever used the tools February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 22 Exhibit 18: Comments about specific MEASURE Evaluation supported tools Mali: [Mali versions of the] RDQA, DHIS 2 User Guide, and the HIS manual and standards … solve the problems in implementation. We participated in the design, validation, and financing in our areas of intervention the training of agents on these. ―External partner (167 instances), 78 percent rated the tools as “very useful.”10 The tools most used and perceived as useful were primarily those addressing overall HIS strengthening needs: DHIS 2, tools related to data quality, and the PRISM for assessing HIS. Exhibit 18 lists examples of tools that interviewees and survey respondents cited. Democratic Republic of Congo: DHIS 2 is slowly transforming how one accesses and analyzes data … while contributing to the improvement of data quality. ―USG Kenya: [PLACE] tools helped define East Africa … venues where patrons meet new sexual partners. This helped map out behavior of priority cross-border populations likely to transmit or acquire new HIV infections. ―USAID mission Zambia: The DQR [Data Quality Review] tool and the RDQA tool were used to conduct an assessment of 4 HIV/AIDS indicators in 23 health facilities in 5 provinces. The DHIS 2 was used to collect, analyze, and share programmatic data from the lowest levels to the national levels. SAVVY tools were used to conduct verbal autopsy interviews … to increase the number of deaths registered with a proceeding ICD [International Classification of Diseases] code. ―Government In the countries visited, internal and external stakeholders perceive that MEASURE Evaluation’s participatory process for tool development and adaptation, coupled with its capacity building, have contributed to tool uptake at the country level. Respondents across all three countries visited pointed to the project’s participatory process for tool development and adaptation as one of its key successes. Respondents in Nigeria highlighted how the process for revising OVC data collection tools involved all OVC Technical Working Group partners to ensure utility and ownership. Similarly, the project’s financial and technical support to the DHIS 2 Technical Working Group in Mali involved government and technical implementing partners in all tool revisions, in ways that gave leadership and ownership to the government. Government counterparts consistently spoke of the tools as their own, not the project’s. External stakeholders in the three countries appreciated the project’s training on data collection tools. MEASURE brought together the stakeholders … to see what tools are being used, what other countries are using [and] projected the tools on the screen to show what could be adapted to the Malian context. MEASURE did all this with [us]. MEASURE helped finance and organize these workshops to make the changes for Mali. — Government, Mali 10 When asked specifically whether the project was ever unable to deliver a tool upon request, 4 respondents (of 54) said “yes,” and 4 (including 3 for the previous question) answered “yes” to whether the project had ever produced a tool that was not useful, relevant, or appropriate. Only 1 of these respondents gave any details, saying it was related to an impact evaluation. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 23 Internal stakeholders perceive that tools and resources are not disseminated widely enough, although project data show positive trends for exposure to and uptake of the tools. Some internal stakeholders perceive that MEASURE Evaluation does not do enough to fully disseminate and promote its tools, noting that tools developed for one health area or country have not often been disseminated elsewhere, limiting the potential for adaptation and application in the wider health community, and for bureau-wide impact. All this stuff coming out is PEPFAR-driven, but it doesn’t get communicated to anyone working outside of the HIV realm, and partly, that is [USAID’s] fault. Our own management team could do a better job at sharing and sensitizing the broader bureau to the products that MEASURE has designed and developed. —USG, Washington Project staff noted that all tools are not created equal; some are designed for a specific setting and topic, and some are continually adapted to the local context and level of experience, making dissemination beyond a particular country less useful. Tools are also being adapted to account for changing disease patterns or other contextual factors; for example, the Routine Health Information System (RHIS) curriculum will always need updating due to changing context. Internal stakeholders suggested expanding dissemination of MEASURE Evaluation tools or raising awareness about them via several pathways and modalities: briefs to introduce tools within USAID, specifically with missions; project technical assistance at the country level to introduce relevant tools; technical working groups at global level to discuss and harmonize efforts on similar types of tools; and promotion to more general audiences through meetings, webinars, peer-reviewed articles, and the project website. Among those interviewed at the country level, most stakeholders were familiar with the tools that had been adapted for their country or sector in Phase IV (e.g., malaria training curriculum materials, PEPFAR M&E and reporting documents), although they had limited awareness of many of the global Phase IV tools. However, country-level respondents were familiar with globally developed tools from previous phases (e.g., PRISM, DQA). In fact, many tools from Phase III (e.g., DQA) have been adopted and institutionalized in government systems and continue to be used. Project statistics on tool downloads indicate wide exposure (see Exhibit 19) under Phase IV, with the malaria M&E training curriculum (English and French), PEPFAR OVC materials, materials on the HIS Strengthening Model, and PRISM 2.0 being the most frequently downloaded. The project uses webinars as one platform to share information about the tools it has developed. Webinar attendance shows increasing participation since the beginning of Phase IV, from 2,218 in 2014 to 2,670 in 2015 and 3,160 in 2016. Country-level participation from Kenya, Nigeria, South Africa, and Tanzania has increased more than five-fold (from 33 in 2014 to 203 in 2016). USG participation has also increased—nine-fold from USAID and thirteen-fold from the U.S. Centers for Disease Control and Prevention (CDC). February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 24 2,595 452 Exhibit 19: MEASURE resource downloads/hits: July 1, 2014–June 30, 2017 Malaria M&E training curriculum (English) PEPFAR MER: Collection of Essential Survey Indicators of OVC Well-Being through Outcome Monitoring: Facilitators' Guide to Data Collector Training Performance of Routine Information System Management (PRISM) 2.0 MEASURE Evaluation’s HIS Strengthening Model Malaria M&E training curriculum (French) MEASURE Evaluation’s HIS Strengthening Model: A Summary RHIS Curriculum on Basic Concepts and Practice: Syllabus Guidance for Evaluating the Impact of National Malaria Control Programs in Highly Endemic Countries 2,000 1,474 1,391 941 790 692 RHIS Curriculum on Basic Concepts and Practice: Facilitators 517 Guide Total Market Approach Practitioners Guide, Module 1 503 0 1,000 2,000 3,000 REMAINING NEEDS Internal stakeholders’ suggestions for MEASURE Evaluation’s focus in the remainder of Phase IV emphasize completing current work and strengthening management, while external stakeholders focus on harmonization and sustainability. All stakeholders identified a range of actions to complete in Phase IV. Most responses from project staff and global interviews focused on practical elements of the work, such as completing deliverables in a timely fashion. Some suggestions focused on improving management and learning aspects; improving communication in ways that give USAID the varied information it needs, but are not too burdensome to the project; increasing communication between USAID and consortium partners; and continuing to refine financial reporting processes to match USG needs. A major focus cited for the remaining time was demonstration of what has been learned and the roadmap for the future. Some tool-specific suggestions focused on finalizing existing tools, conducting a gender review of all tools, promoting wider dissemination in the project’s global networks, investing more in the use of PRISM, and continuing harmonization with tool development by other global partners. Internal and external stakeholders identified their top HIS strengthening priorities as improving data quality and strengthening analysis, and use through capacity building, and focusing on the interoperability of databases, especially at the local level. Respondents from the country visits, online survey, and global interviews all concurred that although the project has done a great deal in setting up and facilitating rollout of DHIS 2 and other February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 25 Exhibit 20: Survey respondents’ top five emerging HIS development needs for the near future (next 2–5 years) (n=99) Internal Stakeholders External Stakeholders Increased use of HIS data for decision making at national and subnational levels Interoperability of databases Improved subnational level government capacity to manage HIS resources, supervision, and training Improved capacity to demand, implement, and use HIS at national and subnational levels Improved quality of routine health information 0% 10% 20% 30% 40% 50% 60% platforms, developing tools, and building capacity, the next stages of in-country HIS strengthening should focus on data quality, analysis, and use, and interoperability of data systems. Trainings are happening for decentralized decision-makers (district directors, chief medical officers), but the use of data at all levels is not yet systematic and much remains to be done. Actors do not yet appropriately do data analysis and use for decision making. —Government, Cote d’Ivoire Exhibit 20 shows online survey respondents’ priorities for HIS strengthening. External stakeholders tended to favor data quality and HIS functioning more frequently than internal stakeholders, who leaned more toward data use and interoperability. (For full results, see Exhibit 48 in Annex 10.) To address these needs, interviewees suggested a greater focus on capacity building, particularly in training regional/state and local actors to ensure improvements in the quality of data filtering up to the national level. Building capacity of government staff, not just at the national level but down to the state level and the local level to be able to take over these functions [the use of the software]. At the national level it is better, but at the lower level, it is still very, very poor. —Global external In evaluation capacity building, global and country-level internal and external stakeholders identified improved capacity to use data for decision making and improved quality of evaluations as top priorities. In online survey responses, internal and external stakeholders’ priorities for evaluation also focused on data quality and use for decision making, as Exhibit 21 indicates. USG and country-level interviewees suggested possible approaches for leveraging academic, private, and networks of evaluation firms and institutions to build capacity, some of which have been operationalized through GEMNet-Health: work with African universities to strengthen M&E pre￾service training capacity; twinning relationships established between institutions (North-to-South as well as South-to-South); creation of mentor/mentee programs; and supporting small grants for February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 26 Exhibit 21: Survey respondents’ top emerging evaluation capacity development needs for the near future (next 2–5 years) (n=99) Internal Stakeholders External Stakeholders Improved national government decision makers' capacity to demand and use evidence from rigorous evaluations Improved national institutions' capacity to design, implement, and facilitate use of rigorous evaluations Improved quality of evaluations: rigor, designs, data, relevance, availability, and accessibility for use Improved national institutions' capacity to demand and use evidence from equity-focused and gender￾responsive evaluations 0% 20% 40% 60% 80% emerging evaluators that include access to funds for evaluation and publishing/presentations and mentoring to design, plan, implement, and publish. CONCLUSIONS These conclusions are based on triangulation of data from country visits, global-level interviews, and the online survey and document review. Specific findings contributing to each conclusion are noted. With the wide range of internal and external stakeholders’ needs, MEASURE Evaluation has been challenged to comprehensively and consistently meet all of the needs at the country and global levels. As noted in Findings 1, 2, and 3, the project has been successful in meeting needs of many stakeholders (both internal and external at the country level, and some internal stakeholders at the USG/Washington level), whereas other stakeholders do not feel their needs are satisfied. Stakeholder needs represent certain polarities11 that, though not always opposites, pull the project’s focus and efforts in different ways and make it challenging to respond to the range of needs. Exhibit 22 summarizes the polarity of needs, with a few of the most important discussed below. Data for USG needs versus data for country needs: Is MEASURE Evaluation primarily to respond to the internal systems and information needs of USAID and other USG agencies (e.g., supporting DATIM), or is it to serve the development objective of better-functioning health systems that are able to meet the health needs of people in countries where USAID operates? These two poles are 11 Polarities are “ongoing, chronic issues which are both unavoidable and unsolvable.” These situations are not problems to solve; they are polarities that require effective management. They appear as seemingly opposite ideas or actions that may be best addressed by “both/and” rather than “either/or” thinking. See http://www.uuce.org/assets/McKandersHandout.pdf. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 27 Exhibit 22: MEASURE Evaluation polarities Left Pole Right Pole Data for USG Data for country reporting HIS needs Focus on national level Focus on local level Majority HIV funding Diverse funding streams Long-term funding Short-term funding required for sustainable demands of Congress HIS outcomes Achieving basic HIS Desire for innovative HIS strengthening in various approaches country contexts obviously interdependent, but they require different time frames and different approaches to meeting stakeholder needs. Proportion of HIV funding and Office of HIV/AIDS management versus proportion of funding from other health programs: The project’s goal is to strengthen HIS to better to serve all health areas. However, because a large proportion of its funding is for HIV/AIDS and because it is managed from the Office of HIV/AIDS, it has focused heavily on HIV, creating a tension with the need for bureau-wide funds and other health area–specific objectives. Health sector–wide versus health area–specific needs: The project’s key results have been best achieved through strengthening an integrated HIS that serves all health-area needs, although its funding often requires a narrower focus. Related to these polarities are the varying perspectives and visions of the project that different stakeholders have (see Exhibit 1). In this context, MEASURE Evaluation’s performance in meeting internal and external stakeholders’ needs is mixed. In some cases, such as its work supporting DATIM at the central level and work in some countries (e.g., Mali and Cote d’Ivoire), the project is seen as a valuable partner and perceived as meeting needs. In these varying circumstances, where the project is valued, it is because it is able to create useful tools that respond to stakeholder needs and is considered a trusted partner with deep technical expertise. For country-level work, stakeholders also appreciate the project’s ability to build capacity and ownership for HIS and evaluation and for the team’s knowledge of the country context. In other areas, internal stakeholders in particular are not satisfied with the services provided due to technical leadership, communication, management, or quality of some specific products and services. MEASURE Evaluation is playing a unique role in the USAID landscape in HIS strengthening, conducting health impact evaluations, and building evaluation capacity in health, despite differing views among USG stakeholders on the role of evaluation in the project. USAID has recognized the need for HIS strengthening and stronger capacity for rigorous evaluations as key contributors to its development goals, objectives, and ability to report on its achievements. As noted in Findings 4, 5, and 6, there are few other mechanisms that provide HIS strengthening and evaluation capacity strengthening at the country or regional levels. And although other mechanisms conduct evaluations, MEASURE Evaluation has a niche in health impact evaluation. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 28 MEASURE Evaluation’s approach to technical assistance and capacity building has facilitated ownership among country-level stakeholders. Operationalization of the learning agenda and its principles is still at an early stage, but this focus is important for paving the way for more sustainable outcomes. As noted in Finding 7, the project’s approach to providing technical assistance is consistent with ensuring sustainability of capacity (skills and competencies) and ownership at the country level, reflecting the results areas of its mandate. There are examples of how MEASURE Evaluation has helped mobilize other partners and resources for capacity and ownership at subnational levels, but internal and external stakeholders are concerned with sustainability of HIS resource availability and infrastructure, and with capacity to maintain the skills of subnational staff over time. The learning agenda, which is still a “work in progress,” provides a maturity model for HIS strengthening and could provide countries with a framework to think through these sustainability issues. Almost all stakeholders see the benefits to a health sector–wide approach to strengthening HIS, and USAID and MEASURE Evaluation have been able to leverage this approach with health area–specific approaches to achieve stronger results. However, a health sector–wide approach has been inherently challenging for USAID to manage, given its health area–specific organization and funding structure. As noted in Findings 8 and 9, stakeholders in Washington see the benefits of a health sector–wide approach to HIS as the most efficient and effective way to ensure the quality and use of data. However, there are needs and pressures from USAID offices for the project to deliver specific results in a short period related to health-area outputs and outcomes. At the same time, country-level HIS are supposed to cut across all health areas. To some degree, these issues are being addressed by applying multiple funding sources to HIS strengthening (including support for DHIS 2 implementation and related capacity building), while strengthening specific health areas, such as malaria, disease surveillance, or comprehensive services for OVC. Many tools MEASURE Evaluation has supported, developed, and/or adapted at the country level are well-appreciated and used by their specific target audiences, both with and without project support. However, there is potential to serve a wider audience and room for broader dissemination and application, both globally and at subnational levels. Findings 10, 11, and 12 illustrate that stakeholders perceive many tools and resources that the project has contributed to, developed, and/or adapted as useful. External stakeholders spoke highly of tools that help them do their jobs and about the way the project contextualized their adaptation, which facilitated ownership among country government. There is still work to be done to ensure full uptake at subnational levels, but the project has worked to fulfill its mandate by training trainers and supporting national-level staff. As dissemination efforts reach ever-increasing audiences, there are many possible additional uses or adaptations in other health areas, and many other potential audiences. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 29 Strengthening data quality and data use for decision making and facilitating interoperability of databases remain the pressing needs to address in order to continue progress in strengthening HIS and evaluation capacity building. As indicated in Findings 13, 14, and 15, internal and external stakeholders have many different perspectives, but there is a collective desire for the project to complete its ongoing activities, namely evaluations, strengthening monitoring of uptake of tools, and facilitating use of data from the HIS and evaluations. Integrated platforms exist at the country level, and stakeholders agree that strengthening data quality, capacity for data use, and interoperability are the key next steps in HIS strengthening. RECOMMENDATIONS Given the polarities acknowledged in the previous section, there are a range of perspectives about MEASURE Evaluation’s goals and achievements. Because the two sides of each polarity are interdependent, it is not possible to choose one as the solution while neglecting the other without negative consequences. Even so, the polarities can be managed effectively. It is in this spirit that the following recommendations are made. These recommendations reflect the evidence emerging from the evaluation and the evaluation team’s interpretation of findings and conclusions. Although there are instances when a recommendation may reflect an individual’s or several respondents’ suggestions, all recommendations stem from the broad set of triangulated data. Given the short remaining timeline and reduced bureau-wide funds for the remainder of Phase IV, MEASURE Evaluation and USAID should streamline communication and set clear priorities and expectations for what must and can be accomplished in the time remaining. From data presented in Findings 1, 2, 6, 7, 12, 13, 14, and 15, there is evidence of multiple and wide-ranging expectations for the project in the remaining year and a half, in terms of meeting stakeholder needs, integrating gender, advancing sustainability, finalizing results to respond to the learning agenda, and disseminating tools. Prioritization will be key. To make use of the limited remaining time and resources, the goal needs to be optimizing project focus and efforts to carry out high-priority actions and achieve high-priority desired results. USAID has approved (or is in the process of approving) work plans for Year 4, and activity planning is underway. USAID and the project should work together to prioritize ongoing and remaining activities, with the criteria of what is most important to accomplish and what is realistic in a year and a half. At the USAID and project headquarters level, this includes establishing clear and focused communication channels between USAID/Washington and the project’s management team, and includes input from USAID health area–specific offices and project consortium counterparts. Because MEASURE Evaluation is funded through a cooperative agreement, there is scope for dialogue and space for the project to push back on USAID suggestions or demands that do not appear feasible or useful, or would not adequately leverage consortium capacities and resources. Both sides need to listen closely to each other to balance priorities in an optimal way. This would require a February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 30 communication forum and atmosphere that facilitates consensus-building, and possibly a compromise about what is most important to accomplish with the resources and time remaining. This forum could take the form of a meeting after this evaluation, and using its results, for which the project and USAID could prepare a list of key results to achieve in the remaining time and actions to achieve them within time and budget constraints. These lists could be compared, discussed, and then revised to achieve a final set of priorities. The final list is not likely to address all stakeholder needs and desires, and the USAID Management Team will need to clarify this with other offices in the Bureau for Global Health. A set of “next-level” priorities can be made, to be addressed only if time and money are available, or through other mechanisms or future procurements. At the country level, if not already completed, a similar process should be repeated with USAID missions and the project’s country office: What activities will or will not continue through the end of the project in 2019, with a focus on establishing transition plans to ensure sustained support for HIS and evaluation capacity-building efforts beyond 2019? Given the limited resources, it would be useful for USAID and the project to consider the most efficient and impactful use of resources devoted to USAID project communications. Currently, the updates require significant preparation time, which might not be the optimal use of resources as the project moves into its final phase. The project and USAID should each propose a small set of actions they feel would be useful to streamline communications. MEASURE Evaluation should continue to collaborate effectively with internal and external stakeholders to strengthen HIS and evaluation capacity building to ensure that Phase IV work is completed on time and with high quality. The project should prioritize finalizing the work on the learning agenda related to HIS strengthening and evaluations, and prepare dissemination plans to ensure awareness and use of these results by the bureau, missions, and external stakeholders. Based on Findings 3, 7, 9, and 11, the project should continue to build on its collaborate working style and success in HIS support and tool development to finish its work in Phase IV; document lessons learned and results, in line with its learning agenda and results framework; disseminate and facilitate use of evaluation results; and develop sustainability plans with counterparts for HIS strengthening and evaluation capacity building. Documentation should include not only project successes and challenges during implementation across HIS and evaluation at the global and country levels, but also lessons learned that other projects and donors can use when looking to implement a health sector–wide approach. USAID and MEASURE Evaluation should discuss the highest-priority areas for strengthening operationalization of the learning agenda and a timeline for its completion (see Recommendation 1 for prioritization). Dissemination plans should be adapted to the various audiences: USAID/Washington, missions, other USG stakeholders, and other external stakeholders. The project needs to keep USAID updated on the learning agenda’s progress and interim results so they can collaboratively determine how to shape and disseminate the final products to facilitate their use and learning within and outside of USAID. The project and the USAID Management Team should work on ensuring that dissemination activities also clearly articulate and provide evidence for why a health sector–wide approach to HIS strengthening is critical in addressing information needs for specific health areas, and document the pathway to a stronger HIS with specific audiences. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 31 MEASURE Evaluation should continue to build on its Phase IV achievements in tool and resource development and adaptation, and USAID and the project should work jointly to raise awareness and disseminate more widely across USAID missions, particularly for health area– specific and gender integration tools. At the country level, the project should focus its efforts on facilitating tool dissemination to increase the likelihood of use at subnational levels. Based on Findings 6, 8, 11, 12, and 14, MEASURE Evaluation and USAID should jointly address the potential for greater awareness of globally developed tools, including gender-specific tools (RDQA+G) and the potential for cross-application of health area–specific tools, considering that different audiences will need different dissemination strategies. The project has devoted significant resources to developing and adapting tools and resources, revising previous versions to respond to changing realities and disseminating the tools at the global and national levels to targeted audiences. As part of the prioritization highlighted in Recommendation 1, any additional materials needed to support awareness-raising about key tools should be included. Tool uptake at the country level may depend on awareness of these tools among a range of individuals (USAID mission staff and external stakeholders); increased awareness can come from many sources and media. The USAID Management Team should complement the project’s efforts by ensuring that USAID staff in Washington and the field are aware of what is available and how it can be useful to support meeting their development objectives. At the country level, internal and external stakeholders highlighted the need for greater focus at subnational levels to ensure that those most likely to input the data have greater and sustained capacity to use the tools. Because MEASURE Evaluation’s work is mostly focused at national level, the project should coordinate with USAID and other in-country partners to strategize how to ensure broader sharing of these tools. While efforts to raise awareness of existing tools and resources should continue, existing mechanisms for South-to-South capacity building could be leveraged in this phase, if prioritized as expressed in Recommendation 1. The evaluation did not generate a specific list of tools for wider application, but reflected articulation of demand for learning how tools from one area might be useful in others. USAID and the project should jointly review the current list and identify tools with applicability across health areas, either directly or with adaptation, and develop a sensitization plan, in line with the prioritization mentioned in Recommendation 1. USAID and MEASURE Evaluation should work on increasing demand for the gender-related M&E tools and resources for HIS work, including health area–specific tools, and expand work to better integrate gender considerations into HIS and evaluation data use as the project’s work on data quality and data use for decision making continues to deepen. As Finding 6 indicates, there is a gap between USAID/Washington’s positive perceptions of the project’s work in gender and missions’ limited demand for project expertise and resources related to incorporating gender into HIS strengthening, beyond ensuring sex disaggregation of data. The project should build on the successes that USAID has appreciated and work with USAID to raise demand and promote its work with internal and external stakeholders, leveraging its global presence. Given the limited time remaining, these efforts need to be considered under the prioritization discussed in Recommendation 1. However, there should be room for the project and the USAID February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 32 Management Team to strengthen attention to gender in scopes of work and work plans, which would increase the likelihood of increasing gender integration, particularly in the analysis and interpretation of HIS data and subsequent decision making. MEASURE Evaluation is well￾positioned to respond to the continuing needs for integrating gender into HIS and health M&E activities, if demand exists. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 33 ANNEX 1: EVALUATION SCOPE OF WORK Source: Section C of the task order contract for this evaluation. 1. PURPOSE The purpose of the Monitoring and Evaluation to Assess and Use Results (MEASURE) Evaluation Phase IV Midterm Performance Evaluation is to:  Understand how effective the project is in meeting key stakeholders needs, and if they are on track to achieve project targets  Inform technical programming and project management for the second half of the Phase IV cooperative agreement  Advise future procurement needs. USAID leadership and MEASURE Evaluation Senior Management will utilize the findings from the midterm evaluation to identify areas for program improvement, and implement recommendations. In addition, findings will inform technical programming and work plan development for the second half of the Phase IV cooperative agreement. USAID leadership will also use the findings and recommendations of the midterm evaluation to identify areas of future procurement. The findings will allow USAID to better determine salient global health information systems strengthening needs, and will inform the design and scope of a potential future project. 2. BACKGROUND 2.1 DESCRIPTION OF THE PROBLEM, DEVELOPMENT HYPOTHESIS, AND THEORY OF CHANGE Limited individual, organizational and system level capacity to collect, analyze and use data for decision making continues to impede the efficiency, effectiveness and sustainability of health programs. Increased global demand for health information and accountability of results has solidified the need for timely and accurate data. International donors and global initiatives such as the Sustainable Development Goals (SDGs), Global Fund for AIDS, TB and Malaria (GFATM), the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR), the President’s Malaria Initiative (PMI) and Family Planning 2020, demand high quality health information. Increasingly, there is a demand from host-country governments and civil society for stronger HIS and evaluation to improve policy and health outcomes. High-quality health information is reliant on a strong HIS. Thus, strengthening health information systems enables countries to access and use data to inform policy formulation, program planning, February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 34 management, and monitoring and evaluation. Furthermore, a robust health information system supports transparency and accountability. Under previous MEASURE Evaluation phases, progress has been made in generating demand for data, improving availability of data, and improving data quality. An emphasis on improvement of data analysis, use and systems is the next pivotal step in the health data use trajectory described in the theory of change below. Demand → Availability → Quality → Analysis → Use This trajectory is not entirely linear; data use can affect data demand in a cyclical pattern, increasing and refining demand. 2.2 RESULTS FRAMEWORK Due to the increasing demand of high quality health information, Phase IV of the MEASURE Evaluation project places particular emphasis on health information systems strengthening and improved capacity of host countries to manage health information systems. In this regard, the activity objectives of MEASURE Evaluation Phase IV are to be accomplished through achievement of the four results listed below: Result 1: Strengthened collection, analysis, and use of routine health data Result 2: Improved country-level capacity to manage health information systems, resources, and staff Result 3: Methods, tools, and approaches improved and applied to address health information challenges and gaps Result 4: Increased capacity for rigorous evaluation. The results framework also includes three input areas, which are:  Technical collaboration and coordination  Capacity building  Information sharing and knowledge management. To achieve these results, MEASURE Evaluation develops new methodologies and tools to improve monitoring and evaluation, builds sustainable capacity at the country-level, participates in collaboration and coordination of activities, and disseminates and shares information, knowledge and best practices. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 35 2.3 PROJECT SUMMARY Activity/Project Name MEASURE Evaluation Phase IV Implementer The University of North Carolina at Chapel Hill Cooperative Agreement/Contract # AID-OAA-L-14-00004 Total Estimated Ceiling of the Evaluated $180 million Project/Activity (TEC) Life of Project/Activity 2014-2019 USAID Office Office of HIV/AIDS MEASURE Evaluation Phase IV is a Leader with Associates Cooperative Agreement. The project is implemented by the Carolina Population Center, University of North Carolina at Chapel Hill (UNC) in partnership with ICF International, John Snow, Inc., Management Sciences for Health, Palladium, and Tulane University School of Public Health and Tropical Medicine. As a Global Health Bureau-wide project, MEASURE Evaluation works across various health elements. Under Phase IV, the project has a ceiling of $180 million. MEASURE Evaluation’s mission is to advance the field of global health Monitoring and Evaluation (M&E) through collaboration at local, national, and global levels. The project works to strengthen country-level capacity to generate and use high-quality health information for global health decision￾making. MEASURE Evaluation has a 19-year history in monitoring and evaluation. The project has gone through three previous phases upon which the current phase is built: Phase I (1997–2003), Phase II (2003–2008) and Phase III (2008–2014). MEASURE Evaluation Phase IV is a 5-year cooperative agreement that extends from July 2014 to June 2019. Progress toward project objectives contributes to improving health systems, programs and policies, which ultimately impacts health outcomes. Project activities are categorized by funding type: field-funded and core-funded. Field-funded activities are requests made and funded by USAID missions, while core-funded activities are requests made and funded by USAID/Washington, often in collaboration with MEASURE Evaluation project staff. Core-funded activities fall into two categories: Global Health (GH) Bureau-wide Activities and Element-specific Activities. Element-specific Activities are project activities that receive February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 36 only one source of funding and are directed to specific health element areas, such as Family Planning and Reproductive Health (FP/RH), HIV/AIDS, Tuberculosis (TB), Malaria, Avian Influenza (AI), Maternal and Child Health (MCH), Nutrition, Neglected Tropical Diseases (NTDs), Water Supply, Sanitation and Hygiene (WASH), Other Public Health Threats (OPHT), Ebola, Highly Vulnerable Children, and Health Systems Strengthening (HSS). Bureau-wide Activities, on the other hand, receive multiple sources of funding and are designed to address issues that cut across multiple health areas. Examples of bureau-wide activities include developing evaluation methodology manuals that can be applied across various health elements and working with regional institutions to develop local capacity to provide HIS strengthening training and technical assistance. The USAID project management team for MEASURE Evaluation Phase IV includes an Agreement Officer Representative (AOR), an alternate AOR, and a Program Assistant as well as representatives from Global Health technical offices. The management team meets regularly, and provides guidance and monitoring by activity, result area, technical area, and country. The team is responsible for monitoring specific activities, from work plan development to implementation and dissemination. The team also provides more general project monitoring and evaluation, including ensuring adherence to the Project Monitoring Plan (PMP) and the Learning Agenda, carrying out annual management reviews, and planning for this midterm evaluation. The management team also manages reporting and information dissemination, budget planning and funding requests. 2.4 SUMMARY OF THE PROJECT M&E PLAN The objectives of MEASURE Evaluation Phase IV’s Project M&E Plan are to (1) allow the project to track the progress, performance and achievement of results and make adjustments along the way in order to improve effectiveness of its interventions; and (2) broaden evidence on effective and efficient interventions that can be used globally for strengthening HIS and M&E systems. These objectives are addressed through two components: a PMP and the Learning Agenda. MEASURE Evaluation Phase IV’s PMP includes indicators and targets to monitor progress and achievement of results. The Learning Agenda, new in Phase IV, is a project-wide system to build evidence of best practices in health information system strengthening. The Learning Agenda also seeks to build evidence on how improved health information systems positively affect health outcomes. MEASURE Evaluation’s Learning Agenda defines the following three questions to develop a theoretical framework for understanding robust health information systems: 1. What are the factors and conditions of HIS performance progress? How should we measure key HIS concepts and functions? 2. What are the stages of progression to a strong HIS? What comes first, typically? Or is there a typical pattern of progression? 3. What are the characteristics of a strong HIS? What seem to be the drivers of HIS improvement and what strategies do those drivers suggest? The evaluation team will be expected to review the PMP and Learning Agenda thoroughly to answer the evaluation questions and better understand fidelity to the program plan and model, as well as to determine if the project is performing as expected. USAID’s Gender Equality and Female Empowerment Policy, adopted in 2012 and whose implementation is articulated in ADS 205, February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 37 requires gender integration throughout the program cycle, including all phases of programming, budgeting, and reporting. The evaluation team should review how the project has addressed gender issues, including how the project’s gender strategy has been operationalized and how it has influenced project outcomes. 3. EVALUATION QUESTIONS The evaluation team must consider the following evaluation questions, which will be refined in collaboration between the USAID MEASURE Evaluation Management Team and evaluators. Evaluators are welcome to look at any aspect of the project that may affect or influence the questions listed below. They are encouraged to talk to a wide range of stakeholders, such as USAID HQ [headquarters] and the missions (both missions currently working with the project and those who are not), other USG agencies such as the Office of the AIDS Coordinator (OGAC) and Centers for Disease Control and Prevention (CDC), other key technical partners such as University of Oslo, country Ministries of Health, local universities, and civil society. The USAID MEASURE Evaluation Management Team is eager to understand the details of how the project is operating, why certain things are working well and others are not, and how the project may make improvements/adjustments in this phase as well as how it could be improved in future procurements. IS THE PROJECT MEETING THE NEEDS OF ITS STAKEHOLDERS? With regard to the project’s technical performance in strengthening the collection, analysis, and use of routine health data (Result 1); improving country capacity to manage health information systems, resources, and staff (Result 2); and building local capacity in rigorous evaluation (Result 4):  To what extent is the project meeting the HIS and evaluation needs of key stakeholders? o What do key stakeholders consider to be the project’s comparative advantages and disadvantages in responding to their needs?  How does the project fit into the current landscape of USAID M&E and HIS mechanisms?  What processes are in place to facilitate the sustainability1 of health information system performance and use and capacity building achievements? What are the key barriers to sustainability? From USAID’s Local Systems Framework: Sustainability refers to the ability of a local system to produce desired outcomes over time. Discrete projects contribute to sustainability when they strengthen the system's ability to produce valued results and its ability to be both resilient and adaptive in the face of changing circumstances. WHAT ARE THE BENEFITS OF A HEALTH SECTOR–WIDE VS. HEALTH AREA–SPECIFIC APPROACH? MEASURE Evaluation is a bureau-wide project with concurrent portfolios of health sector-wide activities and health area-specific activities:  What are the key facilitators and barriers MEASURE Evaluation faces with respect to implementing a health sector-wide approach to strengthening country HIS and evaluation? February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 38  To what extent does the presence of a health sector–wide portfolio in addition to health area–specific portfolios facilitate or hinder the project’s effectiveness in strengthening the collection, analysis, and use of routine health data, improving country-level capacity to manage health information systems, resources, and staff, and building evaluation capacity? What are the advantages and disadvantages of the project having a bureau-wide lens? ARE TOOLS DEVELOPED USEFUL? With regard to methods, tools, and approaches improved and applied to address health information challenges and gaps (Result 3):  Which tools are most frequently used and/or adapted at the country level, and why? o For example, are there common key technical characteristics (e.g., methodology, medium) or facilitating factors (funding source, collaboration with a multilateral, dissemination approach) of the tools that are well utilized?  Are there clear examples of MEASURE Evaluation tools and products impacting health policies, strategies and outcomes? In addition to answering these evaluation questions, USAID encourages the evaluation team to develop recommendations for this phase of the project as well as if a future procurement (or procurements) might be useful to build upon the work of MEASURE Evaluation Phase IV. If future procurement(s) are recommended, it would be useful to have specific recommendations about what format is recommended. For example, if USAID/GH designs one or more implementing mechanisms to follow after MEASURE Evaluation Phase IV ends, does the evaluation team have recommendations as to the structure and organization of the follow-on(s) (i.e., number/size of mechanism[s], type of contract/agreement, and technical and operational organization/division) and why? Are there recommendations for consolidation with other USAID funded projects? How can future procurements be better streamlined for success? 4. EVALUATION DESIGN AND METHODOLOGY The evaluation team will work collaboratively with the USAID Management Team to develop a detailed work plan and data collection strategy for the midterm evaluation. The Contractor will travel to Nigeria, Mali, and Madagascar; however, these countries are subject to change. The USAID Management Team has selected the three countries based on the following criteria:  Significant investment ($2 million or more)  Diverse activity portfolio (investment in more than one disease area)  Work across result area  Implementing partner presence in country. The evaluation team will meet with project staff in Washington DC and Chapel Hill, North Carolina. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 39 The evaluation methodologies for this midterm evaluation includes: (1) a document review, (2) key informant interviews, (3) focus group discussions, (4) surveys to key stakeholders, and (5) direct observation. Evaluators are encouraged to talk to a wide range of stakeholders, such as USAID HQ M&E and HIS technical staff and AOR/CORs of M&E and HIS projects, and USAID missions (both missions currently working with the project and those who are not), other USG agencies such as OGAC and CDC, other key technical partners such as University of Oslo, country Ministries of Health, local universities, and civil society. The following documents should be included in the document review: a. Request for Applications (RFA) b. Cooperative Agreement c. Annual Reports d. Project M&E Plan (including the Learning Agenda and the PMP) e. Result Area Theories of Change and Strategic Approaches f. Capacity statements g. Work plans/SOWs for field-funded and core-funded activities h. Trip reports for the selected countries i. Activity deliverables (reports, tools, etc.) j. Financial databases k. Project-level gender strategy. 5. DELIVERABLES AND REPORTING REQUIREMENTS 5.1 KICK-OFF MEETING/CALL The evaluation team must be available for a kick-off call and/or meeting or both to meet the MEASURE Evaluation management team to review the Statement of Work, review expectations and timeline, and establish rapport. This will also be an opportunity for the evaluation team to discuss, clarify and refine the evaluation questions. The exact date of the call/meeting will be decided after award. 5.2 EVALUATION WORK PLAN Within 2 weeks of the award of the contract, a draft work plan for the evaluation must be completed by the Team Lead and presented to the Task Order Contracting Officer’s Representative (TOCOR). The work plan will include: (1) the anticipated schedule and logistical arrangements; and (2) a list of the members of the evaluation team, delineated by roles and responsibilities. During this presentation, the Team Lead and the USAID management team will also review this SOW, with special attention to the evaluation objectives and questions. 5.3 EVALUATION DESIGN Within 3 weeks of approval of the Evaluation Work Plan, the evaluation team must submit to the TOCOR an evaluation design (which will become an annex to the evaluation report). The evaluation design will include: (1) a detailed evaluation design matrix that links the Evaluation Questions in the SOW to data sources, methods, and the data analysis plan; (2) draft questionnaires February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 40 and other data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, plans as to how the sampling frame will be developed, and the sampling methodology); (4) known limitations to the evaluation design; and (5) a dissemination plan. USAID offices and relevant stakeholders are asked to take up to 3 weeks to review and consolidate comments through the TOCOR. 5.3.1 Revised Evaluation Design Once the evaluation team receives the consolidated comments on the Initial Evaluation Design and Evaluation Work Plan, they must return with a Revised Evaluation Design and Evaluation Work Plan within 3 weeks. 5.4 MIDTERM BRIEFING AND INTERIM MEETINGS The evaluation team is expected to hold a midterm briefing with the USAID MEASURE Evaluation management team on the status of the evaluation, including potential challenges and emerging opportunities. The team will also provide the evaluation TOCOR/manager with periodic briefings and feedback on the team’s findings, as agreed upon during the in-briefing. If desired or necessary, weekly briefings by phone can be arranged. 5.5 DRAFT EVALUATION REPORT The draft evaluation report must be consistent with the guidance provided in Section 5.7, Final Report Format. The report format must use the USAID Evaluation Report template and must address each of the questions identified in this SOW and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the Initial Draft Evaluation Report is submitted, the USAID MEASURE Evaluation management team will have 2 weeks in which to review and comment on the initial draft, after which point the COR will submit the consolidated comments to the evaluation team. 5.5.1 Revised Draft Evaluation Report The evaluation team will then be asked to submit a Revised Final Draft Report 2 weeks hence, and again the USAID MEASURE Evaluation management team will review and send comments on this final draft report within 2 weeks of its submission. The evaluation team is welcome to share an early draft or detailed outline that includes main findings and bullets before finalizing the draft evaluation report. If any recommendations on the final evaluation report are deemed sensitive to a future procurement, the USAID management team may request two versions of the report: an internal USAID document and a public final evaluation report. The Draft and Final Evaluation Reports must exclude any procurement-sensitive and other Sensitive but Unclassified (SBU) information. This information will be submitted in a memo to USAID separate from the Evaluation Report. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 41 5.6 FINAL PRESENTATION The evaluation team is expected to hold two 1-hour final presentations in person to discuss the summary of findings and recommendations. One presentation will be to the USAID project management team in the Washington, DC area, and the other to implementing partner staff in Chapel Hill. The presentations should take place after the revised final draft report is submitted, and will be scheduled as agreed upon during the in-briefing. During the discussion, evaluators will present their recommendations based on their evaluation findings, input and feedback from the USAID MEASURE Evaluation management team and the Implementing Partner before the evaluation report is drafted. A draft of the Final Presentation should be submitted to the MEASURE Evaluation management team prior to finalization. 5.7 FINAL EVALUATION REPORT The evaluation team will take no more than 2 weeks to respond/incorporate the final comments from the USAID MEASURE Evaluation management team. The evaluation team leader will then submit the Final Evaluation Report to the TOCOR. The evaluation team is expected to produce 8 hard color-copies and an electronic copy of the Final Report to the USAID management team. All project data and records will be submitted in full and should be in electronic form in easily readable format, organized and documented for use by those not fully familiar with the project or evaluation, and owned by USAID. The Final Report is to be accompanied by a PowerPoint presentation that aims to debrief selected stakeholders of the results and recommendations stemming from the midterm evaluation. After receipt of the Final Evaluation Report, MEASURE Evaluation and USAID will work together on a management response to agree on a process and timeline for implementing recommendations made in the evaluation report. 6. EVALUATION TEAM COMPOSITION The evaluation team must include one team leader with two additional evaluators and one evaluation assistant supporting the team. All team members will be required to provide a signed statement attesting to a lack of conflict of interest or describing any existing conflict of interest. The evaluation team will become familiar with USAID’s Evaluation Policy and guidance included in the USAID Automated Directive System (ADS) Chapter 201. 6.1 KEY PERSONNEL 6.1.1 Evaluation Lead 6.1.1.1 Position Description The Evaluation Lead will have extensive experience conducting performance evaluations, including evaluations of HIV/AIDS projects. The Evaluation Lead will be responsible for: (1) providing team leadership; (2) managing the team’s activities; (3) ensuring that all deliverables are met in a timely February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 42 manner; (4) serving as a liaison between the USAID and the evaluation/analytic team; and (5) leading briefings and presentations. 6.1.1.2 Education, Experience and Skills  Advanced degree in Public Health or a related field  Minimum of 8 years of international public health evaluation experience  Demonstrated experience leading health sector evaluation/analytics, using both quantitative and qualitative methods  Excellent skills in planning, facilitation, and consensus building  Advanced interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders  Excellent skills in project management, leadership, teamwork and teambuilding  Excellent organizational skills and ability to keep to a timeline  Good writing skills, with extensive report writing experience  Excellent communications, both oral and writing. 6.1.2 Evaluation Members All team members must have extensive experience conducting performance evaluations. Collectively, the team should have experience working in global public health (HIV/AIDS, Malaria, Family Planning, Nutrition, Maternal and Child Health). At least one team member should have very strong evaluation design experience. Other team members should collectively have expertise in health information systems, impact evaluation, organization development/capacity building, program design, and knowledge of French is not required but would be beneficial. 6.2 OTHER STAFF 6.2.1 Evaluation Assistant The Evaluation Assistant will support the evaluation team in the following ways, but not limited to: arranging transportation, calling and setting up appointments, confirming meetings and calls, arranging for meeting space, setting up conference calls, data collection, analysis, and/or entry. Please note that the Evaluation Assistant will collaborate with MEASURE Phase IV, upon arrival in country, to set up transportation and provide translation services. 7. FINAL REPORT FORMAT The evaluation final report must include an executive summary; introduction; background of the local context and the projects being evaluated; the main evaluation questions; the methodology or methodologies; the limitations to the evaluation; findings, conclusions, and recommendations; and lessons learned (if applicable) as described here. The report must be formatted according to the evaluation report template and must address each of the questions identified in the SOW and any other issues the team considers to have a bearing on the objectives of the evaluation. Any such issues can be included in the report only after consultation with USAID. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 43 The executive summary must be 3 to 5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology must be explained in the report in detail. Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). The annexes to the report must include:  The Evaluation SOW  Any significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team  All tools used in conducting the evaluation, such as questionnaires, checklists, and discussion guides  Sources of information, properly identified and listed  Disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts. In accordance with ADS 201 and AIDAR 752.7005, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within 30 calendar days of final approval of the formatted report. 8. CRITERIA TO ENSURE THE QUALITY OF THE EVALUATION REPORT Per the USAID Evaluation Policy and USAID ADS 201, draft and final evaluation reports will be evaluated against the following criteria to ensure the quality of the evaluation report. • The evaluation report should represent a thoughtful, well-researched, and well-organized effort to objectively evaluate what worked in the project, what did not, and why. • Evaluation reports shall address all evaluation questions included in the SOW. • The evaluation report should include the SOW as an annex. All modifications to the SOW—whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline—need to be agreed upon in writing by the CO and/or COR. • The evaluation methodology shall be explained in detail. All tools used in conducting the evaluation—such as questionnaires, checklists, and discussion guides—will be included in an annex in the final report. • The evaluation will assess the extent to which the project has helped transform gender norms and reduced gender gaps; how sex-disaggregated data has been used and how it has affected health information systems; and the extent to which the project has developed capacity to analyze and use data to address gender issues. • Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.). February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 44 • Evaluation findings should be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence. • Sources of information need to be properly identified and listed in an annex. • Recommendations need to be supported by a specific set of findings. • Recommendations should be action-oriented, practical, and specific, with defined responsibility for the action. 9. OTHER REQUIREMENTS All quantitative data collected by the evaluation team must be provided in machine-readable, non￾proprietary formats as required by USAID’s Open Data Policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. All modifications to the required elements of the SOW of the contract/agreement, whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the Task Order Contracting Officer (TOCO) and/or TOCOR. Any revisions should be updated in the SOW that is included as an annex to the Evaluation Report. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 45 ANNEX 2: EVALUATION TEAM PROFILES LYNNE FRANCO, SCD, EVALUATION TEAM LEAD Dr. Franco brings more than 30 years of experience leading complex, multi-stakeholder evaluations; designing, implementing, and managing implementation research in health; and using facilitation and participatory technical assistance and evaluation techniques. Dr. Franco has designed evaluations that have sought to address capacity building, research, scale, institutionalization, and country ownership and implemented methods, such as Appreciative Inquiry, organizational network analysis, online surveys, in-depth interviews, creative group data collection, and household surveys. In her current role, Dr. Franco serves as team leader on long-term evaluation partner contracts, such as the MacArthur Foundation’s Big Bet On Nigeria portfolio and Save the Children’s Saving Newborn Lives Initiative. She has also led shorter-term evaluations, such as the evaluations of the USAID Bureau of Policy, Planning and Learning’s Program Cycle, PEPFAR’s Caribbean Regional Program, and the Bill & Melinda Gates Foundation’s Maternal Health Task Force and African Tobacco Control Consortium. Over her career, Dr. Franco has held long-term positions in Benin, Malawi, and Mali, and worked throughout Africa, Eastern Europe, Latin America, and the Middle East. She has authored peer￾reviewed publications on topics such as quality improvement, health policy reform, programming for orphans and vulnerable children, social participation in and impact of community-based health insurance, and network analysis of organizations working on stillbirth issues. Dr. Franco holds a ScD in International Health Systems and an MHS in Health Planning from Johns Hopkins University School of Hygiene and Public Health. SVETLANA NEGROUSTOUEVA, MPA, EVALUATION TEAM MEMBER Ms. Negroustoueva is a monitoring and evaluation (M&E) and gender professional with 15 years of cross-sectoral experience. She specializes in evaluations and developing and delivering M&E training, assessing M&E systems and building tools, and conducting data quality assessments (DQAs) for projects in health, gender, climate change, and food security. She has worked for the World Bank and on a number of USAID-funded projects, including MEASURE Evaluation (2008–2012). Her engagement involved development and validation of indicators to monitor the U.S. Government’s response to the HIV/AIDS globally. In addition, she co-developed and managed the piloting of the Referral M&E Toolkit in Kenya and Thailand, in collaboration with MEASURE Evaluation resident advisors and PEPFAR-supported sites. She has deep expertise in assessing data quality, having helped build the capacity of USAID staff on DQAs and routine DQAs, also for Sahel Resilience Learning (SAREL) and REGIS projects, contributed to developing their DQA guidance and provided technical assistance to Kenya’s Ministry of Health for the development of their national DQA strategy. Ms. Negroustoueva is a Co-chair of the a EvalGender+ global initiative and of Feminist and Gender Transformative Evaluation Thematic Interest Group at the American February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 46 Evaluation Association (AEA). She holds a Master of Public Affairs from the University of Texas, Austin. She is bilingual in Russian and English, and fluent in French. SARAH CASTLE, PHD, FORMER TEAM LEADER AND PART-TIME EVALUATION TEAM MEMBER Dr. Castle was the original team leader. Although she fell ill, she participated through the design phase as a team member and contributed to the qualitative interview guides and the online survey design. Dr. Castle has more than 20 years’ experience working on public health issues, including extensive experience conducting performance evaluations. Most recently, in January 2017, she worked for Abt Associates on an initiative targeting the private sector in Mali and seeking to identify reasons for weak collaboration between the private and public sectors, as well as opportunities to reinforce the role of the private sector to extend universal health coverage. In 2016, she lent her expertise to the Palladium Group in the design and analysis of qualitative research to inform the rebranding of Condom Protector in Mali. In 2014, she supported the evaluation of USAID’s Maternal and Child Integrated Program in Mali, related to improving national health strategies, policies, and programs. In 2013 and 2014, she served as the Team Leader for USAID’s Support for International Family Planning Programs (SIFPO) funding to Population Services International, and was the Team Lead for a midterm evaluation of USAID/SIFPO funding to Marie Stopes International in 2013. Dr. Castle holds a PhD in Population Studies from the London School of Hygiene & Tropical Medicine. She speaks fluent English, French, and Fulfulde, and intermediate Bambara. KELSEY SIMMONS, MA, EVALUATION TEAM MEMBER Ms. Simmons supports EnCompass’ technical assistance and evaluation work, assisting in resource development and qualitative and quantitative evaluations of large international and domestic projects. She has lived and worked throughout Sub-Saharan Africa, researching, designing, and evaluating programs in HIV/AIDS, water and sanitation, malaria, and health systems strengthening. She also has experience designing M&E frameworks for social enterprises that measure operational efficiency as well as social impact. Ms. Simmons’ most recent portfolio of work includes leading the endline evaluation of Save the Children’s Saving Newborn Lives grant, funded by the Bill & Melinda Gates Foundation. Previously, she managed EnCompass’ evaluation work for the MacArthur Foundation’s portfolio of maternal health accountability grants in Nigeria. She led qualitative analysis of an organizational network analysis of the community of organizations working on stillbirths, supported the evaluation of UN Women’s regional architecture, and led the case study of the PEPFAR-supported Gender￾Based Violence Initiative in Tanzania, which identified effective practices for integrating gender￾based violence programming into existing HIV/AIDS programs. She has presented EnCompass’ work at international conferences such as the American Evaluation Association (2016), the African Evaluation Association (2017), and the USAID Global Health Mini-University (2017). Before joining EnCompass, Ms. Simmons worked for USAID’s Global Health Bureau in the Office of HIV/AIDS and was a community health volunteer with the Peace Corps in Zambia. She holds an MA in International Development Studies, with concentration in Global Health Monitoring and Evaluation, from the Elliott School of International Affairs at The George Washington University. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 47 ELISA KNEBEL, MHS, EVALUATION TEAM MEMBER Ms. Knebel is an evaluator with more than 20 years of experience in a wide range of approaches to help international development organizations measure and maintain meaningful change. She is highly skilled in helping organizations use evaluation results for more effective programming. Ms. Knebel has strong qualitative and quantitative research skills and extensive experience in evaluation design and management, tool design and administration, and conducting multi-country data collection with a range of stakeholders. Ms. Knebel recently performed a performance evaluation of USAID’s Mobile Solutions Technical Assistance and Research (mSTAR) and NetHope projects, which provided new insights for the agency to plan its future funding in mobile health and information and communication technologies. She was a team member on performance evaluations of a Bill & Melinda Gates Foundation–funded maternal and child health network and a Gates-funded family planning project in India. She has extensive experience working in nutrition and HIV/AIDS. Ms. Knebel holds an MHS in International Health from Johns Hopkins University School of Hygiene and Public Health. She is fluent in French and English, and speaks Spanish. SARAH SMITH LUNSFORD, PHD, EVALUATION TEAM MEMBER Sarah Lunsford joined the team in late October to help after Sarah Castle’s full departure from the team. Dr. Lunsford has 13 years’ experience working on public health issues, including extensive experience in research and evaluation. She currently works for EnCompass LLC, seconded to the USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project as a Senior Advisor for Research and Evaluation. For ASSIST, she has been a principle or co-investigator on evaluations and research studies on diverse topics, such as the effectiveness, efficiency, and sustainability of improvement interventions, knowledge management approaches, institutionalization of quality improvement methodologies, and approaches to supporting quality improvement efforts in-country. She has a PhD in Medical Anthropology from the University of Queensland in Brisbane, Australia, and an MPH in Global Health and Development from Tulane University. SABINE K. TOPOLANSKY, MA, EVALUATION ASSISTANT Ms. Topolansky is an international affairs professional with nearly 5 years of experience, including an internship at UNESCO headquarters in Paris, France. In her current role as Evaluation Associate for the Technical Assistance and Evaluation team at EnCompass, she has supported analysis of qualitative and quantitative data using a variety of software programs, conducted document reviews and interviews, contributed to the development of data collection instruments, administered surveys, and helped draft evaluation designs. Her most recent assignments include support to the midline and endline evaluations of the MacArthur Foundation’s maternal health accountability portfolio in Nigeria. She also supports the EnCompass-led evaluation of the U.S. State Department’s programming focused on LGBTI issues in Sub-Saharan Africa, Central America, and Eastern Europe. Ms. Topolansky also has experience working on bioethics, gender issues, HIV, public health, and gender-based violence, and recently co-authored a report related to gender equality. She holds an MA in International Affairs from the George Washington University and speaks Hungarian, Italian and French. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 48 ANNEX 3: DOCUMENTS REVIEWED GLOBAL MEASURE Evaluation. 2014. 2014-2015 Year 1 Annual Report. MEASURE Evaluation. 2014. MEASURE Evaluation Phase IV Executed Cooperative Agreement. MEASURE Evaluation. 2014. MEASURE Evaluation Phase IV Monitoring and Evaluation Plan. MEASURE Evaluation. 2014. MEASURE Evaluation Phase IV Year 1 Approved Work Plan. MEASURE Evaluation. 2014. The Monitoring and Evaluation to assess and use results (MEASURE) Evaluation Phase IV Project (RFA). MEASURE Evaluation. 2015. 2015-2016 Year 2 Annual Report. MEASURE Evaluation. 2015. MEASURE Evaluation Learning Agenda. MEASURE Evaluation. 2015. MEASURE Evaluation Phase IV Annual Report – Year 1 July 2014- September 2015. MEASURE Evaluation. 2015. MEASURE Evaluation Phase IV Year 2 Approved Work Plan. MEASURE Evaluation. 2015. MEASURE Evaluation Phase IV: Monitoring and Evaluation Plan. MEASURE Evaluation. 2016. Gender Integration in and across the MEASURE Evaluation project: Year 2. MEASURE Evaluation. 2016. MEASURE Evaluation Learning Agenda. MEASURE Evaluation. 2016. MEASURE Evaluation Phase IV Year 3 Approved Work Plan. MEASURE Evaluation. 2016. Phase IV Indicator Matrix. MEASURE Evaluation. 2017. MEASURE Evaluation Phase IV: Overview of Historical and Operational Context of the Project. MEASURE Evaluation. 2017. MEASURE Evaluation’s unique role in the global HIS landscape. MEASURE Evaluation. 2017. MEASURE Evaluation’s Work in Health System Strengthening. MEASURE Evaluation. N.d. MEASURE Evaluation Gender Strategy: How MEASURE Evaluation is advancing the monitoring and evaluation of gender in health programs and policies. United States Agency for International Development (USAID). 2014. MEASURE Evaluation Phase IV Executed Cooperative Agreement. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 49 CAPACITY STATEMENTS MEASURE Evaluation. 2015. Health Systems Strengthening. MEASURE Evaluation. 2016. Data Quality for Monitoring and Evaluation Systems. MEASURE Evaluation. 2016. Evaluation. MEASURE Evaluation. 2016. Health Information Systems. MEASURE Evaluation. 2016. Making Gender Count. MEASURE Evaluation. 2017. Evaluation. COTE D’IVOIRE Khatri, U., S. Cloutier, and S. Mullen. 2013. A Case Study of the Integrated Patient Monitoring System in Cote d’Ivoire. MEASURE Evaluation. Mancini, D., G. Stecklov, and J. F. Stewart. 2001. The Effect of Structural Characteristics on Family Planning Program Performance in Côte d’Ivoire and Nigeria. MEASURE Evaluation. MEASURE Evaluation. 2014. Technical Brief: Findings from the Case Study to Measure M&E Systems Strengthening in Cote d’Ivoire and Nigeria. MEASURE Evaluation. 2016. Intégration d’un module de système d’alerte précoce dans le DHIS 2 en Côte d’Ivoire. MEASURE Evaluation. 2016. MEASURE Evaluation Phase IV - Cote d’Ivoire Scope of Work and Country Work Plan Narrative Year 1: January 2015 – December 2015. MEASURE Evaluation. 2016. MEASURE Evaluation Phase IV - Cote d’Ivoire Scope of Work and Country Work Plan Narrative Year 2: October 2015 – September 2016. MEASURE Evaluation. 2016. MEASURE Evaluation Phase IV - Cote d’Ivoire Scope of Work and Country Work Plan Narrative October 2016 – September 2017. MEASURE Evaluation. 2016. Scope of Work and Country Work Plan Narrative Year 1: January 2015 – December 2015: Cote d’Ivoire. MEASURE Evaluation. 2016. Scope of Work and Country Work Plan Narrative: October 2016 – September 2017: Cote d’Ivoire. MEASURE Evaluation. 2017. MEASURE Evaluation Phase IV - Activity Progress Report Cote d’Ivoire. MEASURE Evaluation. 2017. MEASURE Evaluation Phase IV - Approved Activity Work Plans, Cote d’Ivoire Years 1 to 3. MEASURE Evaluation. 2017. MEASURE Evaluation Phase IV - Travel Summary and Trip Reports, Cote d’Ivoire. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 50 Ministère de la Sante et de l'Hygiène Publique, République de Côte d'Ivoire. 2016. Plan de Suivi et d'Evaluation du PNDS 2016-2020. Ministère de la Sante et de l'Hygiène Publique, République de Côte d'Ivoire. 2016. Plan National de Développement Sanitaire 2016-2020. Ministère de la Sante et de l'Hygiène Publique, République de Côte d'Ivoire. 2016. Plan Stratégique National 2016-2020. Nutley, T., L. Gnassou, M. Traore, A. E. Bosso, and S. Mullen. 2014. Moving data off the shelf and into action: an intervention to improve data-informed decision making in Cote d’Ivoire. Global Health Action. United States President’s Emergency Plan for AIDS Relief (PEPFAR). 2014. Cote d’Ivoire Operational Plan Report FY 2013. United States President’s Emergency Plan for AIDS Relief (PEPFAR). 2015. Cote d’Ivoire Operational Plan Report FY 2014. United States President’s Emergency Plan for AIDS Relief (PEPFAR). 2016. Cote d’Ivoire Country Operational Plan (COP) 2016. United States President’s Emergency Plan for AIDS Relief (PEPFAR). 2016. Fact Sheet: Partnering to Achieve Epidemic Control in Cote d’Ivoire. United States President’s Emergency Plan for AIDS Relief (PEPFAR). 2017. Cote d’Ivoire Country Operational Plan (COP) 2016. MALI Economic Community of West African States (ECOWAS/CEDEAO). 2012. Health Information Policy and Strategies in the ECOWAS Region. MEASURE Evaluation. 2014. Analyse Situationnelle du Système d'Information Sanitaire du Mali. MEASURE Evaluation. 2014. Évaluation de l’aide apportée par l’USAID à la prévention du VIH au Mali de 2000 à 2010. MEASURE Evaluation. 2014. Evaluation des Niveaux CSREF, CSCOM et Communautaire du SLIS du Mali a l'Aide de l'Outil PRISM. MEASURE Evaluation. 2015. MEASURE Evaluation Phase IV - Mali Annual Report April – September 2015. MEASURE Evaluation. 2015. MEASURE Evaluation Phase IV - Mali FY 15 Country Scope of Work Phase IV Year 1 January – September 2015. MEASURE Evaluation. 2015. MEASURE Evaluation Phase IV - Mali Monitoring and Evaluation Plan. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 51 MEASURE Evaluation. 2015. MEASURE Evaluation Phase IV - Mali Phase IV Year 1 Quarterly Report January – March 2015. MEASURE Evaluation. 2015. MEASURE Evaluation Phase IV - Mali Quarterly Report April – June 2015. MEASURE Evaluation. 2016. Aligning Stakeholders for Health Information Systems Strengthening: One Step at a Time. MEASURE Evaluation. 2016. MEASURE Evaluation Phase IV - Mali Annual Report April – September 2016. MEASURE Evaluation. 2016. MEASURE Evaluation Phase IV - Mali FY 16 Country Scope of Work Phase IV Year 2 October 2015 – September 2016. MEASURE Evaluation. 2016. MEASURE Evaluation Phase IV - Mali FY 17 Country Scope of Work Phase IV Year 3 October 2016 – September 2017. MEASURE Evaluation. 2016. MEASURE Evaluation Phase IV - Mali Monitoring and Evaluation Plan 2016-2017. MEASURE Evaluation. 2016. MEASURE Evaluation Phase IV - Mali Quarterly Report October – December 2015, Year 2. MEASURE Evaluation. 2016. MEASURE Evaluation Phase IV - Mali Quarterly Report January – March 2016, Year 2. MEASURE Evaluation. 2016. MEASURE Evaluation Phase IV - Mali Quarterly Report April – June 2016, Year 2. MEASURE Evaluation. 2017. Assessment of Mali’s 2016 National Campaign for the Promotion of Family Planning. MEASURE Evaluation. 2017. Évaluation de la Campagne nationale de 2016 pour la promotion de la planification familiale au Mali. MEASURE Evaluation. 2017. MEASURE Evaluation Phase IV - Activity Progress Report, Mali. MEASURE Evaluation. 2017. MEASURE Evaluation Phase IV - Mali Quarterly Report October – December 2016, Year 3. MEASURE Evaluation. 2017. MEASURE Evaluation Phase IV - Mali Quarterly Report January – March 2017, Year 3. MEASURE Evaluation. 2017. MEASURE Evaluation Phase IV - Mali Quarterly Report April – June 2017, Year 3. MEASURE Evaluation. 2017. MEASURE Evaluation Phase IV - Mali Quarterly Report October – December 2016, Year 3. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 52 MEASURE Evaluation. 2017. MEASURE Evaluation Phase IV – Travel Summary and Trip Reports, Mali. MEASURE Evaluation. N.d. Mali Country Context, Activity Summary, and Indicator Reporting Table. MEASURE Evaluation. N.d. MEASURE Evaluation Phase IV - Approved Activity Work Plans, Mali Years 1 to 3. Ministère de la Sante et de l'Hygiène Publique, République du Mali. 2014. Evaluation du Système Local d'Information Sanitaire (SLIS) avec les Outils PRISM: Rapport. Ministère de la Sante et de l'Hygiène Publique, République du Mali. 2014. Plan Décennal de Développement Sanitaire et Social (PDDSS), 2014-2023. Ministère de la Sante et de l'Hygiène Publique, République du Mali. 2014. Programme de Développement Socio-Sanitaire 2014-2018 (PRODESS III). Ministère de la Sante et de l'Hygiène Publique, République du Mali. 2014. Plan Stratégique du Système d'Information Sanitaire et Sociale (PS-SNISS) 2015-2019, September 2015. NIGERIA Federal Ministry of Health Nigeria, Department of Health Planning, Research and Statistics (DPRS). 2014. National Health Information System Strategic Plan 2014–2018. Federal Ministry of Health Nigeria, Department of Public Health. 2010. National Tuberculosis and Leprosy Control Programme (NTBLCP) Workers’ Manual – Revised 5th Edition. Federal Ministry of Health Nigeria. 2010. National Strategic Health Development Plan (NSHDP) 2010 – 2015. Ikamari, L., A. A. Adewuyi, and A. Akinlo. 2007. Decision Maker Perceptions in Kenya and Nigeria: An Assessment of Data Use Constraints. Makinde, O. A., A. Azeez, S. Bamidele, A. Oyemakinde, K. A. Oyediran, W. Adebayo, B. Fapohunda, A. Abioye, and S. Mullen. 2014. Development of a Master Health Facility List in Nigeria. Online Journal of Public Health Informatics (OJPHI) 6(2). MEASURE Evaluation. 2010. Community-Level Program Information Reporting for HIV/AIDS Programs: Introduction. MEASURE Evaluation. 2010. Community-Level Program Information Reporting for HIV/AIDS Programs: Tools and Processes for Engaging Stakeholders Module 1. MEASURE Evaluation. 2010. Community-Level Program Information Reporting for HIV/AIDS Programs: Tools and Processes for Engaging Stakeholders Module 2. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 53 MEASURE Evaluation. 2010. Community-Level Program Information Reporting for HIV/AIDS Programs: Tools and Processes for Engaging Stakeholders Module 3. MEASURE Evaluation. 2010. Community-Level Program Information Reporting for HIV/AIDS Programs: Tools and Processes for Engaging Stakeholders Module 4. MEASURE Evaluation. 2014. A Case Study to Measure National HIV M&E System Strengthening, Nigeria. MEASURE Evaluation. 2014. Assessment of Primary Health Care Facilities for Decentralization of HIV/AIDS Services in Nigeria 2012. Federal Ministry of Health, Abuja, Nigeria. MEASURE Evaluation. 2016. MEASURE Evaluation Phase IV - Scope of Work and Country Work Plan Narrative April 2016 – June 2017, Nigeria. MEASURE Evaluation. 2017. MEASURE Evaluation Phase IV - Health Information System Strengthening Strategy 2017 – 2017, Nigeria. MEASURE Evaluation. 2017. MEASURE Evaluation Phase IV - HMIS Scope of Work and Country Work Plan Narrative Year 3: May 2017 – September 2017, Nigeria. MEASURE Evaluation. 2017. MEASURE Evaluation Phase IV - Scope of Work and Country Work Plan Narrative Year 3: October 2016 – September 2017, Nigeria. MEASURE Evaluation. 2017. MEASURE Evaluation Phase IV – Travel Summary and Trip Reports, Nigeria. MEASURE Evaluation. 2017. Nigeria Country Context, Activity Summary, and Indicator Reporting Table. MEASURE Evaluation. N.d. Nigeria Country Context, Activity Summary, and Indicator Reporting Table. President’s Malaria Initiative (PMI). 2013. Nigeria Malaria Operational Plan FY 2013. President’s Malaria Initiative (PMI). 2014. Nigeria Malaria Operational Plan FY 2014. President’s Malaria Initiative (PMI). 2015. Nigeria Malaria Operational Plan FY 2015. President’s Malaria Initiative (PMI). 2016. Fact Sheet. Nigeria Country Context. President’s Malaria Initiative (PMI). 2016. Nigeria Malaria Operational Plan FY 2016. President’s Malaria Initiative (PMI). 2017. Nigeria Malaria Operational Plan FY 2017. United States President’s Emergency Plan for AIDS Relief (PEPFAR). 2016. Partnering to Achieve Epidemic Control in Nigeria. Ye, Y., E. Patton, A. Kilian, S. Dovey, and E. Eckert. 2012. Can universal insecticide-treated net campaigns achieve equity in coverage and use? The case of northern Nigeria. Malaria Journal. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 54 ANNEX 4: EVALUATION DESIGN AND METHODS EVALUATION APPROACH EnCompass used a mixed-methods approach that allowed depth and breadth in data collection and triangulation during analysis and interpretation, guided by four core principles to ensure focused, relevant, and practical information about MEASURE Evaluation’s achievements and inform implementation of future activities. These principles include:  A whole-systems approach that recognized the complexity of the systems in which MEASURE Evaluation is implemented and the importance of gathering a variety of perspectives from different actors  A collaborative, participatory approach that is utilization-focused, actively engaging USAID (in Washington and at missions), project partners, and other key stakeholders (as relevant) throughout the evaluation process to articulate key evaluation questions and interpret results so findings can effectively inform future decision making and so learning is captured, shared, and built upon  A strengths-based approach, grounded in Appreciative Inquiry, that allowed the team to identify the best of what is being done and build on success to achieve even greater results  An outcome mapping framework that focused on identifying concrete, observable results along the pathway between project outputs and the ultimate results, reflecting on what is in the project’s sphere of control, sphere of influence and sphere of interest. The evaluation design matrix linking evaluation questions to data sources, methods, and the data analysis plan was finalized in collaboration with and approved by USAID. The matrix is presented in Exhibit 23. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 55 Exhibit 23: Evaluation design matrix Questions, Sub-questions, Operational and Areas of Focus Questions Data Sources Methods Evaluation Question 1: Is the project meeting the needs of its stakeholders? Sub-question 1a: To what extent is the project meeting the HIS, evaluation, and learning needs of key stakeholders? AREA OF FOCUS: This question will  How effectively and efficiently has  Documents and databases:  Document review explore two key topics. First, it will the project been meeting HIS needs? o Project M&E plan  Interviews identify how key stakeholders define  How effectively and efficiently has o Annual work plans, annual reports  Focus groups their needs in relation to MEASURE the project been meeting evaluation- Evaluation project activities. Second, it o Management information systems:  Online survey related needs? will explore how, and to what extent, project routine monitoring data  To what extent is implementation of project activities correspond with the  USAID/Washington, missions the MEASURE Evaluation Learning self-identified needs of those  Other USG: PEPFAR, PMI agenda effective? stakeholders.  National Ministry of Health, malaria, TB control, Ebola commissions  Ministries/Departments of child development/social work, other OVC￾related agencies  Global, U.S.-based, and country-level technical working groups: Inter-Agency Working Group, Roll Back Malaria Monitoring, Evaluation Reference Group  Global Evaluation and Monitoring Network for Health (GEMNet-Health) Sub-question 1b: What do key stakeholders consider to be the project’s comparative advantages and disadvantages in responding to their needs? AREA OF FOCUS: This question will  What other M&E and HIS projects  Descriptions of other USAID mission  Document review first identify what other projects and and programs are offering similar and bilateral/regionally funded projects programs are offering services similar to services to MEASURE Evaluation, providing HIS and evaluation support MEASURE Evaluation, in terms of HIS and how?  USAID assistance officer’s and evaluation capacity-building representatives (AORs) of similar  What are MEASURE Evaluation’s  Interviews support at the country and global level. projects comparative advantages and It will then investigate what factors key  Focus groups disadvantages in responding to  USAID missions, other USG: PEPFAR, stakeholders identify as MEASURE stakeholder needs compared to PMI country-level teams Evaluation’s comparative advantages those projects and programs? and disadvantages in relation to these other projects and programs. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 56 Questions, Sub-questions, Operational and Areas of Focus Questions Data Sources Methods Sub-question 1c: To what extent does the project fit into the current landscape of USAID M&E and HIS mechanisms?  To what extent does the project  USAID Bureau for Global Health  Interviews coordinate, collaborate, and  USAID AORs of similar projects  Document review r communicate with other USAID M&E  Documents about other projects m and HIS mechanisms?  To what extent are there  USAID/Washington  Interviews mechanisms at USAID to facilitate  Project staff  Document review coordination, collaboration, and  Other-USAID funded projects communication between USAID M&E and HIS mechanisms? AREA OF FOCUS: This question will explore how well the MEASURE Evaluation project fits into the broade USAID M&E and HIS mechanisms fro the perspectives of project implementation and management. Sub-question 1d: What processes are in place to facilitate the sustainability of quality, performance, and use of HIS and evaluation-related work? What barriers to sustainability exist? AREA OF FOCUS: This question will first identify how key stakeholders define HIS sustainability, as well as the sustainability of evaluation capacity￾building outcomes. The question will take into account how different stakeholders (project staff, USAID staff, non-USAID staff, country-level stakeholders, etc.) define these two issues. The question will then identify the success factors in relation to sustaining MEASURE Evaluation’s achievements in these two areas.  What are facilitating factors for and barriers to sustaining the MEASURE Evaluation project’s achievements?  How aligned is the definition of HIS sustainability among stakeholders?  What are the facilitators and barriers to sustaining achievements related to HIS systems?  USAID/Washington and missions  Project staff  National Ministry of Health, malaria, TB control, Ebola commissions  Global Fund, World Health Organization, UNAIDS, UNICEF  Interviews  Focus groups  Online survey  How aligned is the definition of  USAID/Washington, missions  Interviews evaluation capacity among  Project staff  Focus groups stakeholders?  Ghanaian/Other local universities and  Online survey  What are the barriers to sustaining consulting firms achievements with regard to evaluation? Evaluation Question 2: What are the benefits of a health sector–wide versus a health area–specific approach? Sub-question 2a: What are the key facilitators and barriers MEASURE Evaluation faces with respect to implementing a health sector–wide approach to strengthening country HIS and evaluation? AREA OF FOCUS: We will break this  Internally, what are the facilitators  USAID/Washington, missions  Interviews question into three lines of inquiry, one and barriers MEASURE Evaluation  Project teams at headquarters, country  Focus groups that focuses internally (within the faces in implementing a health level project), one that focuses externally on sector–wide approach? February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 57 Questions, Sub-questions, Operational and Areas of Focus Questions Data Sources Methods how key stakeholders perceive this  Project staff breakdown, and one that focuses on USAID perceptions of the advantages  What are USAID perceptions on the  USAID/Washington, missions  Interviews advantages and disadvantages of a and disadvantages of the approach.  Project teams at headquarters, country  Focus groups health sector–wide approach, based level on their knowledge of and experience  National Ministry of Health, malaria, TB with MEASURE Evaluation? control, Ebola commissions  Externally, what are the perceived  AORs of similar projects advantages and disadvantages of MEASURE Evaluation’s being implemented as a health sector–wide approach? Sub-question 2b: To what extent does the presence of a health sector–wide portfolio, in addition to health area–specific portfolios, facilitate or hinder the project’s effectiveness in strengthening the collection, analysis, and use of routine health data, improving country-level capacity to manage HIS resources and staff, and building evaluation capacity? AREA OF FOCUS: The question will explore how the combination of a health sector–wide approach and health area– specific portfolio facilitates and hinders the project’s implementation and potential contributions to the outcomes of inquiry. We will place special focus on assessing the benefits and disadvantages of this approach internally, within the USAID Global Health Office.  How are outcomes affected by a health sector–wide portfolio approach, in addition to health area– specific portfolio approaches?  What are the advantages and disadvantages of a health sector– wide lens in terms of the project’s ability to meet stakeholder needs?  To what extent does each of the USAID Global Health offices benefit from MEASURE Evaluation project, implemented across the health sector–wide portfolio?  How does MEASURE Evaluation utilize its health sector–wide portfolio for internal learning and to respond to the USAID Learning Agenda?  USAID/Washington, missions (HIV, TB,  Interviews malaria, family planning, Ebola, etc.)  Focus groups  Other USG: PEPFAR, PMI  Project staff  Design and Analysis Toolkit for Inventory and Monitoring (DATIM) stakeholders  National Ministry of Health, malaria, TB control, Ebola commissions  Ministries/Departments of child development/social work, other OVC￾related agencies Evaluation Question 3: Are tools developed useful? AREA OF FOCUS: This question will  Documents and databases:  Document and first identify which subset of tools will be o Project annual reports database review February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 58 Questions, Sub-questions, Operational and Areas of Focus Questions Data Sources Methods the focus of investigation, with input Trend data for tool/guidance Interviews  Which tools are most frequently used o  from USAID and MEASURE Evaluation. downloads and/or adapted at the country level, We will then delve into which tools are by whom, why, and how? o Registration data from webinar used and adapted most frequently at participants  What have been the success factors the country level. Specifically, we will in terms of tool development,  Project knowledge management look into disaggregated use with regard deployment, adaptation, and statistics to health policies, health strategies, and dissemination? project outcomes. Further, we will  Project sample reports on tool/guidance explore what stakeholders use identify as  What are the barriers to tool the success factors in relation to the development, deployment,  PEPFAR, PMI teams, DATIM most-used tools, in terms of adaptation, and dissemination? stakeholders development, deployment, adaptation,  What tools are likely to require  World Health Organization, UNAIDS and dissemination. continuous investment for future adaptation and use? Why?  USAID missions  Interviews  PEPFAR/PMI teams  Online survey  Country-level/ and regional technical working groups  GEMNet-Health Evaluation Question 4: What is the optimal structure, management, and organization of the follow-on(s), and why? AREA OF FOCUS: This question will  What are lessons learned from  Project documents, work plan, annual  Document review explore lessons that have been learned MEASURE Evaluation project reports  Interviews from MEASURE Evaluation’s implementation?  Descriptions of other USAID-funded  Focus groups implementation. We will delve into projects  How can the management of future lessons related to structure and project procurements be streamlined for  USAID Bureau for Global Health, USAID mechanisms, management (of success? AORs of similar projects MEASURE staff, implementing partners, and USAID staff), and  MEASURE Evaluation management and organization of the consortium of implementing partner principal partners working under MEASURE investigators, country offices Evaluation Phase IV. We will then explore, in this question, stakeholders’ suggestions (MEASURE staff, USAID staff, and external stakeholders) for making future projects successful, in terms of structure, management, and organization. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 59 EVALUATION PROCESS The evaluation was carried out by a core team—each bringing specific strengths to the process— coordinated from the EnCompass headquarters office in Rockville, Maryland. Team composition changed during the evaluation after the original team leader fell ill. International team members included Lynne Franco (team leader), Svetlana Negroustoueva, Kelsey Simmons, Elisa Knebel, Sarah Castle, Sabine Topolansky, and Sarah Smith Lunsford (see Annex 2 for team profiles). Three additional team members—one per country visited, each with significant experience in health, project evaluation, and qualitative research—contributed to country data collection and analysis: Mamadou Faramba Camara (Mali), Mabinu Olasumbo Oladipo (Nigeria), and Kouakou Claude Konan (Cote d’Ivoire). The participatory and iterative evaluation process aimed to ensure a focused and useful evaluation report that meets the operational needs of key stakeholders. The evaluation team facilitated a 1-day design meeting on September 7, 2017, with members of the USAID MEASURE Evaluation management team and key MEASURE Evaluation staff to review evaluation questions, key stakeholders and actions to be observed,12 and other key elements of the design. Evaluation design took place mainly in September 2017, with data collection in October and November, and analysis and writing of the draft report in December 2017. DATA COLLECTION TOOLS AND METHODS As the evaluation design matrix (see Exhibit 23) illustrates, the evaluation team used one or more of the following methods to answer each evaluation question and sub-question, tailored to specific project activities and locations:  Document review  Country visits  Semi-structured key informant interviews  Focus group discussions  Observation  Online survey DOCUMENT REVIEW The EnCompass team conducted a document review of project-related documents (e.g., plans, status reports, and knowledge management statistics), as listed in Annex 3. Each evaluation team member, including the local consultants, reviewed and analyzed a selection of MEASURE Evaluation Phase IV design and implementation documents and evaluations. Standardized data extraction forms, organized around the evaluation questions, facilitated a systematic process for the document review and subsequent analysis. The results provided context and key information to triangulate with other sources during later stages of the evaluation. 12 The outcome mapping framework was used to examine project spheres of control, influence, and interest—what the project does, who it is trying to influence to do what, and what key results it hopes will result from those actions. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 60 Exhibit 24: Selected countries against USAID criteria USAID Criteria Cote d’Ivoire Mali Nigeria Diverse activity  Health sector–wide  Health sector–wide:  Health sector–wide: HIV, portfolio  Ebola maternal and child MCH, FP/RH (investment in  HIV, including health (MCH), malaria,  HIV, including orphans more than one prevention of mother-to- Family Planning and and vulnerable children disease area) child transmission/ Reproductive Health  Malaria antiretroviral therapy (FP/RH), nutrition, HIV,  MCH integration water supply and  FP/RH  Health systems sanitation (WSS)  Tuberculosis strengthening  Malaria  Nutrition assessment,  Nutrition assessment,  Disease surveillance counseling, and support counseling, and support (Ebola, Global Health Security Agenda) Results 1, 2, 3 Results 1, 3, 4 Results 1, 2, 3, 4 Inputs 1, 2 John Snow, Inc. John Snow, Inc. Palladium ICF International John Snow, Inc.  FP/RH Work across result areas Input 2 Implementing partner presence in country USAID $3.9 million (years 1–3) $7.3 million (years 1–3) $7.8 million (years 1–3) investment greater than $2 million COUNTRY VISITS To provide an in-depth picture of how MEASURE Evaluation was interacting with the full range of internal and external stakeholders, the evaluation included three country visits. Using the criteria in Exhibit 24, the USAID MEASURE Evaluation management team selected the Cote d’Ivoire, Mali, and Nigeria as representing the range of countries where MEASURE Evaluation was working. Each visit lasted 2 weeks and included individual and group interviews, facilitated focus group discussions, some limited observation of MEASURE Evaluation–supported events, and a debriefing at the end of the visit to validate findings. Specific data collection methods are described below and were used for global-level interviews. SEMI-STRUCTURED KEY INFORMANT INTERVIEWS EnCompass core evaluation team members and local consultant in three countries used semi￾structured interviews with a wide range of informants to elicit participants’ perceptions and capture rich, detailed accounts of their experiences of the MEASURE Evaluation project. The evaluation team developed interview guides (Annex 5), which USAID reviewed and approved. The guides were then adapted in the context of each respondent just prior to the interview process. The guides used an Appreciative Inquiry approach, which began with an exploration of exceptional experiences and ended with wishes for the future of MEASURE Evaluation and beyond. This approach facilitates February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 61 openness with respondents and allows exploration of both what has worked well and what needs to be improved. Stakeholders’ responses are anonymized in the evaluation report. FOCUS GROUP DISCUSSIONS Where possible, the evaluation team organized structured group discussions to enhance sample size and to obtain individual and group responses. These facilitated discussions were conducted during country visits and with MEASURE Evaluation staff (some virtual and some in person). They were intended to be used with some USAID/Washington staff and some external global stakeholders as well, but staff availability was too limited to use this method, so the evaluation team substituted a group interview format, using the semi-structured interview guide. Using active and visual methods, the EnCompass facilitator engaged the group to generate information and reflect on responses, using a variety of facilitation techniques, such as a sticky wall and cards or World Café.13 In-person and virtual group discussions were conducted via Adobe Connect, which offers a range of interactive features. The guides for these sessions can be found in Annex 5. OBSERVATION The EnCompass team observed the following activities with MEASURE Evaluation participation:  The MEASURE Evaluation partners’ meeting with USAID at the University of North Carolina on October 10, 2017  In-country activities during two of the country visits: o Mali: DHIS 2 Steering Committee meeting and health center setup for data entry o Cote d’Ivoire: Work-planning meeting between USAID and the project  Six presentations by MEASURE Evaluation staff at the American Evaluation Association (AEA) Conference, held November 8 through 11, 2017, in Washington, D.C: o Harmonizing evaluation education across continents: GEMNet-Health (panel session). Presentation titled “Professionalizing Monitoring and Evaluation - Experiences on Certification and Competencies: In practice, not just in theory!” o Measuring Key Domains of Reproductive Empowerment: Lessons Learned from a Multicountry Approach (Session #:3139; panel session). Presentation titled “Development of Reproductive Decision-Making Measures: Challenges, Insights, and Learning from a Mixed-Method Study in Nepal” o Measuring intangibles: Methodological challenges in gender analysis (Session #: FIE2; panel session). Presentation titled “Big Data Analytics for Measuring Gender Norms: Too Big to Ignore” o “Getting to an Evaluation Plan: A Five-Step Process from Engagement to Evidence” (demonstration) 13 See http://comm.eval.org/coffee_break_webinars/viewdocument/cfb-supercharge-yo for an AEA Coffee Break on this method. (Requires an AEA login.) February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 62 Exhibit 25: Online survey sample by stakeholder and method Response Target Sent to Responded rate Short survey USAID/Other USG staff based in Washington 37 14 38% Long survey USAID missions  Buying in to MEASURE 44 20 45%  Not buying in to MEASURE 26 1 4% External stakeholders  English survey 159 64 40%  Not buying in to MEASURE 59 16 27% Unlisted members of technical working groups Unknown 5 — o “Building the Ship as It Sails: Challenges and Strategies to Assess and Enhance Evaluation Capacity in the Real World” (think tank panel) o “Qualitative Methods for Evaluation: Moving from Afterthought to Forethought” (panel session) ONLINE SURVEY This evaluation used an online survey tool for three purposes: (1) to reach a wide range of respondents, both geographically and thematically; (2) to obtain quantitative data to validate selected qualitative data; and (3) to facilitate limited disaggregation. The survey targeted three groups: USAID and USG staff based in Washington, D.C.; USAID mission staff in countries not visited (both those engaged with MEASURE and those not); and members of in-country technical working groups and U.S.-based communities of practice. The sampling frame for the online survey (see Exhibit 25) was selected in coordination with USAID, considering geographic reach and a desire for wide representation. For USAID and USG staff, a shorter, more open-ended survey was distributed to 37 Washington-based staff, with a 38 percent response rate (14 individuals). A longer, more closed-ended survey was sent to all USAID missions with a health portfolio (both those buying into MEASURE Evaluation and not) and a wide range of global and country-level technical working group members in which MEASURE Evaluation is engaged. The long survey was distributed to 229 individuals, with a response rate of 46 percent. Of the 106 individuals responded, 20 percent were USAID staff and 80 percent were external stakeholders. A French version was shared with 59 individuals, with 16 responding. MEASURE Evaluation provided a list of contacts for the external stakeholders at the country level and main point of contacts for each technical working group. The evaluation team shared the link to the online survey with the technical working group point of contacts via email, and asked them to share the link with other members of their technical working group. The English survey link was shared with 12 individuals, and 4 individuals completed the survey. The French survey was shared with 13 individuals, with 1 respondent. Before the evaluation team distributed the surveys (via SurveyMonkey), USAID sent an email to targeted respondents announcing it, and issued frequent reminders to those who had not yet responded. See Annex 6 for the long and short versions of the online survey questionnaires. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 63 Exhibit 26: Sample by stakeholder and method Method Data collection events Stakeholder type MEASURE TOTAL Evaluation Internal External USAID or Country Partners USG Gov. Country visits Mali 18 interviews, 3 FGDsa 6 27 13 10 56 Nigeria 23 interviews, 2 FGDs 7 11 6 5 29 Cote d’Ivoire 15 interviews, 5 FGDs 6 11 8 6 31 Virtual/In-person USAID/Washington 16 interviews, 2 FGDs 21 GHb /OHAc 5 N/A N/A N/A GH/PMId 4 N/A N/A N/A GH/PRHe 4 N/A N/A N/A GH/OHSf 3 N/A N/A N/A GH/IDg 2 N/A N/A N/A GH/MCHNh 1 N/A N/A N/A GH Front Office 1 N/A N/A N/A Global Dev. Lab 1 N/A N/A N/A Other USG (OGACi ) 1 interview 1 N/A N/A N/A 1 USAID missions 3 interviews 3 N/A N/A N/A 3 External stakeholders 5 interviews N/A N/A 5 N/A 5 MEASURE Evaluation 5 interviews, 3 FGDs N/A N/A N/A 19 19 headquarters staff Online survey USAID/Washington 14 N/A N/A N/A 14 USAID missions 21 N/A N/A N/A 21 External stakeholders N/A 23 62 N/A 85 TOTAL 79 72 94 40 283 a Focus group discussion; b Bureau for Global Health; c Office for HIV and AIDS; d President’s Malaria Initiative; e Office of Population and Reproductive Health; f Office of Health Systems; g Office of Infectious Diseases; h Office of Maternal and Child Health and Nutrition; i Office of the Global AIDS Coordinator SUMMARY OF SAMPLE Through the various methods of data collection, the evaluation team reached respondents from 17 of the 30 countries where MEASURE Evaluation works, representing 73 percent of field support funds coming to the project, and 14 of the 24 countries with at least some HIV funds. ANALYSIS The EnCompass team applied a multistep process to triangulate and synthesize the results of the evaluation and develop draft findings and conclusions that respond to the evaluation questions. Each data collection method had its own analysis process: country visit data, online survey data, global interviews and group discussion data, and document review. The core evaluation team summarized emerging themes and the evidence behind each theme from each data source, using an internally developed template that was shared across team members for further triangulation. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 64 The evaluation team used Dedoose14 to facilitate coding of qualitative data, including from document extraction, based on evaluation questions and objectives. The team then coded and analyzed all data collected through interviews and focus group discussions, for content and thematic analysis. Initial coding was performed by the team member who conducted the interview. For country and global interview visit data, one member of the team conducted a second round of coding and analysis. Quantitative data from the online surveys were analyzed in Microsoft Excel and in Stata version 14. The EnCompass team convened for a 2.5-day internal data analysis and interpretation session, where they triangulated the emerging themes across sources and developed and mapped the evaluation findings, conclusions, and recommendations. LIMITATIONS This evaluation has several limitations, some of which influence how data should be interpreted, and others that affected the evaluation process. To mitigate the limitations and provide a balanced picture, the evaluation team triangulated data across sources as much as possible. LIMITATIONS POTENTIALLY INFLUENCING THE RESULTS  Long project history and broad project scope: MEASURE Evaluation is in its fourth 5-year phase of implementation, with an extensive history and strong name recognition among stakeholders. Some stakeholders were unable to distinguish among the phases and attribute results appropriately to Phase IV. Because of the project’s broad scope, a number of interviewees and survey respondents could only speak to one very specific aspect (e.g., someone working in malaria could only speak to their malaria work). The evaluation team tried to address this issue by focusing questions on events and activities from the last 3 years (since the start of Phase IV) and by triangulating data across all sources.  Country sample selection: The in-depth picture of country-level work and ability to meet stakeholder needs was an important approach and provided a full view from all perspectives. However, the results from Cote d’Ivoire, Mali, and Nigeria do not fully represent all the perspectives of internal and external stakeholders across MEASURE Evaluation’s 30 countries and its country-level portfolios, despite USAID’s significant effort to choose a representative sample of countries. Again, triangulation with online survey data and additional interviews with other USAID missions helps mitigate this limitation.  USAID mission representation in the sample: In addition to the country visits, the evaluation team targeted five missions with MEASURE Evaluation activities and two without for semi￾structured virtual interviews. Despite repeated outreach, only three of the seven missions were interviewed. Twenty-one USAID mission staff initiated the online survey, representing 14 missions. As perspective on why missions do not buy into MEASURE Evaluation, the sample only included one virtual interview and one online survey respondent who answered the question about why they did not use MEASURE Evaluation. It is possible that those missing 14 Dedoose is an online qualitative data analysis platform that allows multiple users to code and analyze simultaneously. See www.dedoose.com. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 65 from this sample feel strongly about the project, one way or another, and are not appropriately weighed in the results.  Online survey sampling and respondent bias for external stakeholders: Response rates to the survey were acceptable, at 38 percent overall, but may not be fully representative of the populations of project stakeholders. The response rate was low among technical working group members, for whom the evaluation team relied on group leaders to share the link with others in their groups. Because the sample population was not completely known, it is not possible to calculate the response rate.  Unintended bias toward HIS versus evaluation-related findings: The countries selected for visits offered less data on the project’s evaluation portfolio than on the project’s HIS strengthening work, as none had significant evaluation portfolios. Virtual interviews were only able to include one mission with a large evaluation portfolio. Thus, the evaluation team was not able to assess the evaluation portfolio in the same depth as the HIS strengthening portfolio. LIMITATIONS THAT AFFECTED THE EVALUATION PROCESS  Short time frame: This evaluation had a 6-month duration, from contract signing to submission of the final report. For a project this large and comprehensive, the evaluation design requires adequate thought and reflection on the part of the evaluation team and USAID together. Without adequate preparation and, importantly, time for testing an online survey, the evaluation is rushed in design, data collection, and analysis.  Budget constraints: USAID would have liked a larger sample of virtual and in-person interviews from the agency and the broader USG, and with external stakeholders at the global level. However, the budget and time frame could not accommodate this, and prioritization was necessary.  Lack of scheduled time for reflection with the USAID MEASURE Evaluation management team: Although two presentations were included in the evaluation scope of work, there was not time programmed for the evaluation team and the USAID management team to reflect on the full findings, conclusions, and recommendations. Although midterm briefing was included, it was too early in the process, and lacked the benefit of the full written report.  Changes to team composition: The evaluation team underwent two changes. Due to a serious medical incident, the team leader had to step down from that role within the first 3 weeks. EnCompass had to replace the team leader and add another team member to ensure smooth functioning and staffing for all three country visits. Later, again due to illnesses, more help had to be brought in to ensure the timely completion of all the interviews and analysis. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 66 ANNEX 5: DATA COLLECTION TOOLS (QUALITATIVE INTERVIEW GUIDES) MEASURE PHASE IV MIDTERM PERFORMANCE EVALUATION INTERVIEW GUIDE: EXTERNAL AND GLOBAL-LEVEL STAKEHOLDERS Date: Interviewer: Respondent Name/Title/Affiliation: Respondent Contact Information: INTRODUCTION Thank you very much for setting aside time to talk with me/us today. EnCompass LLC has been contracted by USAID to conduct the midterm performance evaluation of the MEASURE Evaluation Phase IV project. The evaluation will focus on the extent to which MEASURE Evaluation is meeting the needs of its key stakeholders. USAID and MEASURE Evaluation will use the findings from this performance evaluation to inform technical programming and project management for the rest of Phase IV, and for the design and scope of future procurements. You have been identified by MEASURE Evaluation as a key stakeholder in its work and therefore a key informant for this study. This study is not an assessment of your organization or your work Before we begin, I want to let you know that any information or examples we discuss and any quotes we use in the report will not be attributed to a specific person or institution, and all identifying information will be removed. You are also free not to respond to any of our questions or to stop the interview at any time. The interview will take about 1 hour. Do you give your consent for this interview? February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 67 If you don’t mind, I would like to record this conversation, solely for the purposes of listening attentively now and taking notes later. Is that all right? If the respondent agrees, BEGIN RECORDING. Before I/we begin, do you have any questions? NOTE TO INTERVIEWER: When you reach 50 minutes, tell the respondent how many questions remain and check that you can complete the questions in the next 10 minutes. If not, ask for additional time. BACKGROUND 1. Please tell me/us briefly how long you have been working with [ORGANIZATION] and what your current role is. Probe: Would you identify your work focused more on evaluation, health information systems, or other monitoring and evaluation and/or technical areas? a. Please tell me in what capacity you have been involved with MEASURE Evaluation Phase IV since it started in 2014? 2. Think about your collaboration with the MEASURE project related to [evaluation and/or health information systems] in the last 3 years. Tell me about an exceptional experience, a high point, or a success—a time when you felt progress had been made in your work. Probe to learn what happened. What were the needs and how did MEASURE Evaluation meet those needs? What were the enabling factors for success in terms of: technical strengths, management and coordination, other? Who else was involved? (Try to get one great quote.) 1. IS THE PROJECT MEETING THE NEEDS OF ITS STAKEHOLDERS? Evaluation Sub-question 1a: To what extent is the project meeting the health information systems, evaluation, and learning needs of key stakeholders effectively and efficiently? NOTE TO INTERVIEWER: Tailor the question based on the response to the background question about whether the respondent’s background is in evaluation or health information systems, or other health-specific areas (HIV, tuberculosis, malaria, etc.). We would like to start by identifying the most urgent needs related to health information systems and evaluation. We will then explore how the project is meeting these needs and potential opportunities for improvement. 3. If you had to identify the top three most pressing needs in relation to health information systems and/or evaluation over the past 3 years, what would they be? February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 68 a. In what ways has MEASURE Evaluation contributed to meeting these needs? Probe separately regarding (1) health information systems needs and (2) evaluation capacity￾building needs, based on the respondent’s background and experience. Are there expectations of contributions that have not materialized? b. How would you describe the utility and the quality of MEASURE Evaluation’s support in meeting these needs? Probe: Is the project producing outputs and deliverables that are of high quality? Are they the right deliverables to meet the needs? Are they useful to their target audience? Are they produced in a timely manner to meet the needs—not only on schedule, but is the timing right? c. What are some of the key factors that have facilitated and inhibited MEASURE Evaluation’s success in meeting these needs? d. What else still needs to be done to meet these needs? Probe for specific activities, processes, focus areas, and so on. Evaluation Sub-question 1b: What do key stakeholders consider to be MEASURE Evaluation’s comparative advantages and disadvantages in responding to their needs? 4. What other mechanisms, projects, or programs are you aware of that are also supporting monitoring and evaluation and health information systems globally? Probe for other donors working in this space, including bilateral partners (e.g., U.K. and German international development agencies [DFID, GIZ]), multilateral partners (e.g., United Nations, Global Fund), foundations (e.g., Bill & Melinda Gates Foundation), and others. What would you consider to be MEASURE Evaluation’s comparative advantage and disadvantage compared with these other monitoring and evaluation or health information systems projects? 5. To what extent is MEASURE Evaluation coordinating, collaborating, and communicating with other global monitoring and evaluation and health information systems mechanisms? Probe for specific examples and assessment. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 69 6. Are you a member of any of the technical working groups in which the MEASURE Evaluation project is involved? If so, which one(s)? What role do you see MEASURE Evaluation playing in these groups (for example, leading/participating/coordinating)? Check if on the list:  Bureau for Global Health Cooperating Agencies M&E Technical Working Group  Community-Based HIS Stakeholder Consultation Meetings  Data Usenet Listserv  Every Newborn Action Plan (ENAP) Working Group  Geographic Information Systems (GIS) Working Group  Global Digital Health Network  Global Evaluation and Monitoring Network for Health (GEMNet-Health)  Health Data Collaborative  Implementing Best Practices in Family Planning Consortium  Ministry of Health Data Alignment  PEPFAR Behavioral/Structural Prevention Technical Working Group  PEPFAR Orphans and Vulnerable Children (OVC) Technical Working Group  Programmatic Mapping and Size Estimation for Key Populations Partnership  Roll Back Malaria Monitoring and Evaluation Reference Group (RBM-MERG)  Routine Health Information Network  World Health Organization (WHO) Routine Health Information System (RHIS) Strengthening Collaboration  Other If not, SKIP the next section and GO TO Evaluation Sub-question 1d. 7. To what extent do you feel MEASURE Evaluation is playing its role effectively, and why? Probe for specific examples of how this coordination has resulted in improvements in the respondent’s work or their organization’s work, or benefited others. Probe for ways MEASURE Evaluation is coordinating—disseminating knowledge, convening stakeholders, technical leadership in health information systems strengthening, technical leadership in evaluation-related capacity building, facilitating learning. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 70 2. WHAT ARE THE BENEFITS OF A HEALTH SECTOR–WIDE VERSUS A HEALTH AREA–SPECIFIC APPROACH? Evaluation Sub-question 2a: To what extent does the presence of a health sector–wide portfolio, in addition to health area–specific portfolios, facilitate or hinder the project’s effectiveness in strengthening the collection, analysis, and use of routine health data; improving country-level capacity to manage health information systems resources and staff; and building evaluation capacity? In this section, we are going to discuss the unique structure of the MEASURE Evaluation project. We would like to explore the effects of this approach on the project’s outputs and outcomes. 8. MEASURE Evaluation concurrently implements portfolios of work that are focused on health sector–wide activities (such as DHIS 2 and Health Management Information System [HMIS]) and health area–specific activities (such as malaria, tuberculosis, and orphans and vulnerable children [OVC]). In what ways has the ability to work both horizontally and vertically enhanced impacts? In what ways has this inhibited progress? Probe for how this dual aspect has played out with regard to meeting the needs discussed above. How has the health sector–wide approach to health information systems enhanced health information systems work in a specific health area? Probe for strengthening the collection, analysis, and use of routine health data; improving country￾level capacity to manage health information systems resources and staff; and building evaluation capacity across the health sector. 3. ARE THE TOOLS THAT HAVE BEEN DEVELOPED USEFUL? NOTE TO INTERVIEWER: To inquire about MEASURE Evaluation tools, this next set of questions focus on the tools and resources the project has developed over the last 3 years. 9. Which of the following tools, methods, curricula, and/or approaches are you aware of, if any? NOTE TO INTERVIEWER: Have a list of tools ready. Health information systems performance RDQA update-Gender-Integrated Routine Data Quality Assessment (RDQA+G) Tool Guidelines Data Quality Review (DQR) Tool, Version 2.1 DataCollTool Routine Data Quality Assessment Tool: User Manual Guidelines Malaria M&E training curriculum (English and French) TrainResource Health information systems governance, strategic planning, and management Routine Health Information Systems: Curriculum on Basic Concepts and Practice TrainResource (Facilitator’s Guide and Syllabus) Tools, methods, and approaches PLACE Update: Tool for Collection of Biologic Specimens DataCollTool Summary of HIV Prevalence and Size Estimates in Key Populations Tool DataCollTool February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 71 Evaluation PEPFAR Monitoring, Evaluation, and Reporting: Collection of Essential Survey Indicators or Orphans and Vulnerable Children Well-being through Outcome Monitoring, Facilitator’s TrainResource Guide to the Data Collector Training; MER OVC Essential Survey indicators FAQ; OVC Survey Toolkit Data Collector's Training and Manual (French Translation) Framework for Evaluating the Impact of National Malaria Control Programs in Highly Guidelines Endemic Countries DATIM ADX Adapter Application Software/App DHIS 2 v2.17-2.28 Software/App PEPFAR Dashboard 3.0 Software/App Data Quality Protocol for DATIM Site Location Guidelines Probe: How have you been involved in the development or use of any of these tools? a. From where you sit, which tools, toolkits, or resources (or types of tools or guidance documents) are most useful to you or for the activities in which you are engaged? b. Which tools have you struggled with using? Probe for causes due to low quality, user-friendliness, adapting to your context, and so on. 10. What have been the facilitating and inhibiting factors in dissemination, use, and/or adaptation of these tools? Probe for tool development, deployment, adaptation, revision, and dissemination (announcement, brown bags, webinars, podcasts). How successful was MEASURE Evaluation in addressing those barriers? Why? a. To what degree have these tools integrated gender considerations in development, adaptation, use, and dissemination? Probe to understand how the respondent defines gender considerations (e.g., data disaggregation, monitoring of impact on girls and women). 11. Can you provide an example of when these tools and products led to decisions related to global or institutional health policies, strategies, or outcomes? If yes, how has it happened? February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 72 Probe: Are you aware of any of these tools being adopted and used by country stakeholders and/or other global stakeholders without USAID’s support? LOOKING FORWARD 12. If you had three wishes for improving health information systems and evaluation capacity globally in the future, what would they be? 13. What recommendations or suggestions would you have for USAID to address these needs in future projects? Probe: Adjust the question to the respondent’s perspective in terms of making progress under PEPFAR, the President’s Malaria Initiative, family planning, Ebola, Zika, the Sustainable Development Goals, and so on. CONCLUSION 14. Is there anything else you feel I/we need to know that you have not yet expressed? 15. Do you have any questions for me/us? Thank you very much for your time and insights. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 73 MEASURE PHASE IV MIDTERM PERFORMANCE EVALUATION INTERVIEW GUIDE: MEASURE EVALUATION HEADQUARTERS STAFF Date: Interviewer: Respondent Name/Title/Affiliation: Respondent Contact Information: INTRODUCTION Thank you very much for setting aside time to talk with me/us today. As you know, EnCompass LLC has been contracted by USAID to conduct the midterm performance evaluation of the MEASURE Evaluation Phase IV project. The evaluation will focus on the extent to which MEASURE Evaluation is meeting the needs of its key stakeholders. USAID and MEASURE Evaluation will use the findings from this performance evaluation to inform technical programming and project management for the rest of Phase IV. You have been selected to speak with us because of your affiliation with MEASURE Evaluation in Phase IV. Before we begin, I want to let you know that any information or examples we discuss and any quotes we use in the report will not be attributed to a specific person or institution, and all identifying information will be removed. You are also free not to respond to any of our questions or to stop the interview at any time. The interview will take about 1 hour. Do you give your consent for this interview? If you don’t mind, I would like to record this conversation, solely for the purposes of listening attentively now and taking notes later. Is that all right? If the respondent agrees, BEGIN RECORDING. Before I/we begin, do you have any questions? NOTE TO INTERVIEWER: When you reach 50 minutes, tell the respondent how many questions remain and check that you can complete the questions in the next 10 minutes. If not, ask for additional time. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 74 BACKGROUND 1. Please tell me/us briefly how long have you been working for MEASURE Evaluation, and what your current role is. Probe if the respondent started before 2014: How, if at all, have your role/responsibilities changed since Phase IV started in 2014? 2. Think about your experience working for MEASURE Evaluation, related to either evaluation or to health information systems, in the last 3 years. Tell me about an exceptional experience, a high point, or a success—a time when you felt progress had been made in making data available for decision making and its use to improve programs. Probe to learn what happened, at what level. What were the needs and how did MEASURE Evaluation meet those needs? What enabled that interaction/project/support to be a success? Who else was involved? (Try to get one great quote.) 1. IS THE PROJECT MEETING THE NEEDS OF ITS STAKEHOLDERS? Evaluation Sub-question 1a: To what extent is the project meeting the health information systems, evaluation, and learning needs of key stakeholders effectively and efficiently? We would like to start by identifying your view of the most urgent needs related to health information systems and evaluation. We will then explore how the project is meeting these needs and potential opportunities for improvements in the future. 3. If you had to identify the top three most pressing needs in relation to health information systems and/or evaluation-related support over the past 3 years, what would they be? NOTE TO INTERVIEWER: Tailor the question based on response to the background questions: evaluation or health information systems, or other. Probe for which of the pressing needs were met. By MEASURE? By others? Which needs remain? What do you anticipate to be pressing needs over the next 2 to 3 years? Probe for needs from other stakeholder perspectives, if necessary:  Internal: USAID mission, country-specific technical working groups, other Bureaus  External: Country ministries and government bodies, in-country implementing partners and subcontractors, and populations who are the ultimate beneficiaries of MEASURE Evaluation’s work (e.g., orphans and vulnerable children, people living with HIV). a. In what ways has MEASURE Evaluation contributed to meeting these needs? Probe regarding (1) health information systems needs and (2) evaluation capacity-building needs, separately if applicable. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 75 b. What are some of the key factors that facilitated and inhibited MEASURE Evaluation’s success in meeting these needs? Probe for internal and external factors. Probe for potential solutions to challenges: What could USAID do differently? What could the project do differently? c. What else can the project do to ensure these needs are met by end of Phase IV? 4. To what extent has the Phase IV Learning Agenda informed priorities and new activities in Phase IV? a. To what extent has the Phase IV Learning Agenda been successful in showing progress toward building strong and sustainable health information systems? Evaluation Sub-question 1b: What do key stakeholders consider to be MEASURE Evaluation’s comparative advantages and disadvantages in responding to their needs? Here, we would like to explore which other projects and programs are offering services similar to MEASURE Evaluation’s, in terms of health information systems and evaluation capacity-building support at the global levels, and how you see MEASURE Evaluation in comparison to these projects. 5. What other USAID and U.S. Government central mechanisms are you aware of that support monitoring and evaluation and health information systems? NOTE TO INTERVIEWER: Listen for:  Global Health Bureau mechanisms, such as Supporting Operational AIDS Research (SOAR), Global Health Program Cycle Improvement Project (GH Pro), Coordinating Implementation Research to Communicate Learning and Evidence (CIRCLE), Accelerating Strategies for Practical Innovation and Research in Economic Strengthening (ASPIRES), Health Evaluation, Research, and Development (HERD), and the Breakthrough-RESEARCH project  USAID/Washington mechanisms, such as Eval-ME (Evaluation IDIQ), Monitoring, Evaluation, Research, and Learning Innovations (MERLIN), Strategic Program for Analyzing Complexity and Evaluating Systems (SPACES MERL), the Developmental Evaluation Pilot Activity (DEPA-MERL), Expanding the Reach of Impact Evaluation (ERIE), and Rapid Feedback MERL. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 76 6. What would you consider to be MEASURE Evaluation’s comparative advantages and disadvantages compared with other USAID and U.S. Government global monitoring and evaluation mechanisms or projects that support health information systems? NOTE TO INTERVIEWER: Specific examples, if needed:  Supporting Operational AIDS Research (SOAR)  Global Health Program Cycle Improvement Project (GH Pro)  Coordinating Implementation Research to Communicate Learning and Evidence (CIRCLE)  Accelerating Strategies for Practical Innovation and Research in Economic Strengthening (ASPIRES)  Health Evaluation, Research, and Development (HERD)  Breakthrough-RESEARCH project  USAID mission-wide country/regional monitoring and evaluation mechanisms (e.g., IBCTI) Evaluation Sub-question 1c: To what extent does the project fit into the current landscape of USAID monitoring and evaluation and health information systems projects? 7. To what extent does MEASURE Evaluation successfully coordinate, collaborate, and communicate with other USAID mechanisms supporting health information systems and monitoring and evaluation globally? 8. What would you consider to be MEASURE Evaluation’s comparative advantage and disadvantage compared with other monitoring and evaluation projects funded by other donors, outside of the U.S. Government, that you know of? Probe: Other donors working in this space, including bilateral partners (e.g., U.K. and German international development agencies [DFID, GIZ]), multilateral partners (United Nations, Global Fund), foundations (Bill & Melinda Gates Foundation), and others. Evaluation Sub-question 1d: What processes are in place to facilitate the sustainability of health information systems quality, performance, and use? Here, we would like to explore how you define sustainability in relation to the MEASURE Evaluation project and what some factors are that may facilitate or inhibit sustainability of achievements by the MEASURE Evaluation project in your country. 9. Imagine that MEASURE Evaluation Phase IV is over and has achieved sustainable results in strengthening health information systems and evaluation capacity. What does that look like for health information systems? For evaluation-related capacities? February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 77 a. What processes and standards is MEASURE Evaluation using to encourage sustainability of health information systems quality, performance, and use? Probe: Have capacity-building activities related to the use of health information systems and evaluation-related capacity building contributed to sustainability? What more could we do? b. What milestones, if any, are you seeing toward sustainable health information systems? To what extent has the MEASURE Evaluation project been able to monitor, assess, and learn about the progress toward achieving these milestones? c. How well has the project been able to capture evidence that stakeholders have used findings from evaluations conducted by, or with support from, MEASURE Evaluation to improve programs, policy, and advocacy? 2. WHAT ARE THE BENEFITS OF A HEALTH SECTOR–WIDE VERSUS A HEALTH AREA–SPECIFIC APPROACH? In this section, we are going to discuss the unique structure of MEASURE Evaluation—a Bureau￾wide project with concurrent portfolios of health sector–wide activities and health area–specific activities. We would like to explore the effects of this approach on project’s outputs and outcomes. Evaluation Sub-question 2a: To what extent does the presence of a health sector–wide portfolio, in addition to health area–specific portfolios, facilitate or hinder the project’s effectiveness in strengthening the collection, analysis, and use of routine health data; improving country-level capacity to manage health information systems resources and staff; and building evaluation capacity? 10. In your opinion, what are the successes and challenges faced by the MEASURE Evaluation project when implementing a health sector–wide project only? a. How well has USAID supported implementation of a health sector–wide approach for the MEASURE Evaluation project? Probe for how well these portfolios are speaking to each other. For example, is the tuberculosis portfolio learning from the HIV portfolio? Similarly, as a global project, how is MEASURE Evaluation helping to share lessons learned in one country for application in other countries? 11. In your opinion, what are the successes and challenges faced by the MEASURE Evaluation project when implementing a health sector–specific project? Why? Probe: How do you think this affects the project’s ability to meet stakeholders’ needs? February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 78 12. What have been advantages and disadvantages in implementing a project with concurrent portfolios of health sector–wide activities and health area–specific activities related to health information systems and evaluation in meeting USAID’s needs? Probe for differences between USAID missions and USAID/Washington and country stakeholders. Probe for examples of how the Bureau-wide portfolio has made a difference across the health sector and how it has affected individual health elements. Are all health elements benefiting equally? Probe for the extent to which the Bureau-wide agenda is not performing well, why not. How could it be improved? How could the project maximize these activities? a. How does this affect MEASURE Evaluation’s ability to meet the needs of external stakeholders? Probe for external outcomes (e.g., the ability to deliver high-quality support and use) and internal processes (e.g., the need to manage multiple funding streams and the time needed for collaboration). 3. ARE THE TOOLS THAT HAVE BEEN DEVELOPED USEFUL? 13. From where you sit, which tools (or types of tools or guidance documents) would you say have been most useful for stakeholders? Probe for epidemic-specific, cross-cutting, gender tools and products. Probe for why they have been useful. 14. How has MEASURE Evaluation facilitated dissemination, use, and adaptation of the tools? Probe for tool development, deployment, adaptation, revision, and dissemination (announcement, brown bags, webinars, podcasts). How successful was MEASURE Evaluation in addressing those barriers? Why? a. In what ways have gender considerations been integrated into tool development, adaptation, use, and dissemination? Probe to see how the respondent defines gender considerations (e.g., data disaggregation, monitoring of impact on girls and women). February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 79 15. Can you provide an example of when MEASURE Evaluation tools and products have contributed to decisions related to national/regional health policies, strategies, and outcomes? If yes, how has it happened? Probe: Are you aware of any of these tools being adopted and used by country stakeholders and/or other global stakeholders without USAID’s support? 16. What have been some lessons learned, derived from MEASURE Evaluation’s experience in tool development at the country and global level, that should be considered for future projects? Probe: What are the characteristics of tools that should be improved and applied globally and at a country level? What tools or types of tools/guidance documents are likely to require continuous investment? LOOKING FORWARD 17. If you had three wishes for improving health information systems and evaluation capacity building in the future, what would they be? 18. What recommendations or suggestions do you have for how USAID could address those needs in future projects? Probe: Adjust the question to the respondent’s perspective in terms of making progress under PEPFAR, the President’s Malaria Initiative, family planning, Ebola, Zika, the Sustainable Development Goals, and so on. CONCLUSION 19. Is there anything else you feel I/we need to know that you have not yet expressed? 20. Do you have any questions for me/us? Thank you very much for your time and insights. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 80 MEASURE PHASE IV MIDTERM PERFORMANCE EVALUATION INTERVIEW GUIDE: USAID AND U.S. GOVERNMENT HEADQUARTERS￾LEVEL STAKEHOLDERS Date: Interviewer: Respondent Name/Title/Affiliation: Respondent Contact Information: INTRODUCTION Thank you very much for setting aside time to talk with me/us today. EnCompass LLC has been contracted by USAID to conduct the midterm performance evaluation of the MEASURE Evaluation Phase IV project. The evaluation will focus on the extent to which MEASURE Evaluation is meeting the needs of its key stakeholders and whether the project is on track to achieve its targets for Phase IV. USAID and MEASURE Evaluation will use the findings from this performance evaluation to inform technical programming and project management for the rest of Phase IV and for the design and scope of future global health information systems procurements. You have been selected to speak with us because of your role in supporting health information systems and evaluation and your work with MEASURE Evaluation Phase IV. The purpose of this data collection is to assess the performance of the MEASURE Evaluation Phase IV project. Before we begin, I want to let you know that any information or examples we discuss and any quotes we use in the report will not be attributed to a specific person or institution, and all identifying information will be removed. You are also free not to respond to any of our questions or to stop the interview at any time. The interview will take about 1 hour. Do you give your consent for this interview? If you don’t mind, I would like to record this conversation, solely for the purpose of listening attentively now and taking notes later. Is that all right? If the respondent agrees, BEGIN RECORDING. Before I/we begin, do you have any questions? NOTE TO INTERVIEWER: When you reach 50 minutes, tell the respondent how many questions remain and check that you can complete the questions in the next 10 minutes. If not, ask for additional time. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 81 BACKGROUND 1. Please tell me/us briefly how long have you been working with [USAID/U.S. Government] and what your current role is. a. Please tell me how you have worked with MEASURE Evaluation since 2014. 2. Think about your experience working with or alongside MEASURE Evaluation during the last 3 years. Tell me about an exceptional experience, a high point, or a success—a time when you felt progress had been made in the project. Probe to learn what happened. What were the needs and how did MEASURE Evaluation meet those needs? What enabled that interaction/project/support to be a success? Who else was involved? (Try to get one great quote.) 1. IS THE PROJECT MEETING THE NEEDS OF ITS STAKEHOLDERS? Evaluation Sub-question 1a: To what extent is the project meeting the health information systems, evaluation, and learning needs of key stakeholders effectively and efficiently? We would like to start by identifying what you, as a representative of USAID/the U.S. Government, see as your key needs related to monitoring and evaluation, health information systems, data quality, and capacity building over the last 3 years. We will then explore how MEASURE Evaluation is meeting those needs and where you see opportunities for improvements in the future. 3. If you had to identify the top three most pressing needs in relation to health information systems and/or evaluation-related support over the past 3 years related to the work you do, what would they be? NOTE TO INTERVIEWER: Tailor the question based on response to the background questions: evaluation or health information systems, or other. Probe for needs from other stakeholder perspectives, if necessary:  Internal: USAID mission, country-specific technical working groups, other Bureaus  External: Country ministries and government bodies, in-country implementing partners and subcontractors. a. In what ways has MEASURE Evaluation contributed to meeting these needs? Probe regarding (1) health information systems needs and (2) evaluation capacity-building needs, separately if applicable. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 82 b. How would you describe the utility and the quality of MEASURE Evaluation’s support in meeting these needs? Probe: Is the project producing outputs and deliverables that are high quality? Are they useful to their target audience? Are activities being implemented on schedule? c. What are some of the key factors that facilitated and inhibited MEASURE Evaluation’s success in meeting these needs? d. What else still needs to be done to meet these needs for the remainder of Phase IV? Evaluation Sub-question 1b: What do key stakeholders consider to be MEASURE Evaluation’s comparative advantages and disadvantages in responding to their needs? Here, we would like to explore which other projects and programs are offering services similar to MEASURE Evaluation’s, in terms of health information systems and evaluation capacity-building support at the country level, and how you see MEASURE Evaluation in comparison to these projects. 4. What other USAID and U.S. Government central mechanisms that support monitoring and evaluation and health information systems are you aware of? If the answer is “none,” SKIP question 5. NOTE TO INTERVIEWER: Listen for:  Global Health Bureau mechanisms, such as Supporting Operational AIDS Research (SOAR), Global Health Program Cycle Improvement Project (GH Pro), Coordinating Implementation Research to Communicate Learning and Evidence (CIRCLE), Accelerating Strategies for Practical Innovation and Research in Economic Strengthening (ASPIRES), Health Evaluation, Research, and Development (HERD), and the Breakthrough-RESEARCH project  USAID/Washington mechanisms, such as Eval-ME (Evaluation IDIQ), Monitoring, Evaluation, Research, and Learning Innovations (MERLIN), Strategic Program for Analyzing Complexity and Evaluating Systems (SPACES MERL), the Developmental Evaluation Pilot Activity (DEPA-MERL), Expanding the Reach of Impact Evaluation (ERIE), and Rapid Feedback MERL. 5. What would you consider to be MEASURE Evaluation’s comparative advantages and disadvantages compared with other projects in the current landscape of USAID and U.S. Government monitoring and evaluation mechanisms? Specific examples, if needed: February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 83  Supporting Operational AIDS Research (SOAR)  Global Health Program Cycle Improvement Project (GH Pro)  Coordinating Implementation Research to Communicate Learning and Evidence (CIRCLE)  Accelerating Strategies for Practical Innovation and Research in Economic Strengthening (ASPIRES)  Health Evaluation, Research, and Development (HERD)  Breakthrough-RESEARCH project  USAID mission-wide monitoring and evaluation mechanisms (e.g., MSI)  USAID/Washington mechanisms, such as Eval-ME (Evaluation IDIQ), Monitoring, Evaluation, Research, and Learning Innovations (MERLIN), Strategic Program for Analyzing Complexity and Evaluating Systems (SPACES MERL), the Developmental Evaluation Pilot Activity (DEPA-MERL), Expanding the Reach of Impact Evaluation (ERIE), and Rapid Feedback MERL. Evaluation Sub-question 1c: To what extent does the project fit into the current landscape of USAID monitoring and evaluation and health information systems projects? 6. [USAID only] To what extent do you see MEASURE Evaluation successfully coordinating, collaborating, and communicating with other USAID mechanisms or projects that include monitoring and evaluation or health information systems strengthening? Probe for specific examples and assessment. 7. [USAID only] What mechanisms and systems are in place to facilitate coordination, collaboration, and communication within USAID related to monitoring and evaluation and health information systems projects globally? Probe: What about coordination and collaboration at the global level? 8. [USAID only] To what extent is MEASURE Evaluation coordinating, collaborating, and communicating with other, more intervention-focused USAID projects? Probe for specific examples and assessment, such as other PEPFAR-funded projects, Maternal and Child Survival Program (MCSP), and other Family Planning and Reproductive Health (FP/RH) projects. 9. What would you consider to be MEASURE Evaluation’s comparative advantage and disadvantage in comparison to other monitoring and evaluation and/or health information systems projects funded by other donors outside of the U.S. Government? February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 84 Probe: Other donors working in this space, including bilateral partners (e.g., U.K. and German international development agencies [DFID, GIZ]), multilateral (United Nations, Global Fund), foundations (Bill & Melinda Gates Foundation), and others. Evaluation Sub-question 1d: What processes are in place to facilitate the sustainability of health information systems quality, performance, and use? Now, we would like to explore how you define sustainability in relation to the MEASURE Evaluation project and what some factors are that may facilitate or inhibit sustainability of achievements by the MEASURE Evaluation project in your country. 10. Imagine that MEASURE Evaluation Phase IV is over and has achieved sustainable results in strengthening health information systems and evaluation capacity. What does that look like for health information systems? For evaluation-related capacities? What might be milestones you would see? a. [U.S. Government MEASURE Management Team Only] To what extent have you seen evidence of MEASURE Evaluation’s contributions to achieving these milestones? b. [U.S. Government MEASURE Management Team Only] What evidence have you seen, if any, that stakeholders have used findings from evaluations conducted by or with support from MEASURE Evaluation to improve programs, policy, and advocacy? 2. WHAT ARE THE BENEFITS OF A HEALTH SECTOR–WIDE VERSUS A HEALTH AREA–SPECIFIC APPROACH? In this section, we are going to discuss the unique structure of MEASURE Evaluation—a Bureau￾wide project with concurrent portfolios of health sector–wide activities and health area–specific activities. We would like to explore the effects of this approach on the project’s outputs and outcomes. Evaluation Sub-question 2a: What are the key facilitators and barriers that MEASURE Evaluation faces with respect to implementing a health sector–wide approach to strengthening country health information systems and evaluation? 11. [USAID only] In your opinion, what facilitates or inhibits USAID’s efforts to support a health sector–wide project? Probe: Are USAID Offices satisfied with their contributions to health sector–wide activities and the benefits they receive from them? February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 85 12. [USAID only] Has your Office provided blended funding to MEASURE Evaluation to support health information systems or other health area–specific funding? If so, what was the reason for this decision? Probe for whether the Office is seeing any learning across the malaria portfolio and other portfolios. Are they getting an advantage from working with a Bureau-wide project, as opposed to a HIV-only project, for example? a. How satisfied were you with the outcome of this decision? 13. In your opinion, what facilitates or inhibits MEASURE Evaluation’s implementation of a health sector–wide project? Probe: How do you think this affects the project’s ability to meet stakeholders’ needs? Evaluation Sub-question 2b: To what extent does the presence of a health sector–wide portfolio, in addition to health area–specific portfolios, facilitate or hinder the project’s effectiveness in strengthening the collection, analysis, and use of routine health data; improving country-level capacity to manage health information systems resources and staff; and building evaluation capacity? 14. In your opinion, what are the advantages and disadvantages of MEASURE Evaluation having a project with concurrent portfolios of health sector–wide activities and health area–specific activities related to health information systems and evaluation in terms of meeting USAID’s needs? Probe for differences between USAID missions and USAID/Washington. Perception of cost savings? a. How does this affect MEASURE Evaluation’s ability to meet the needs of external stakeholders? Probe for external outcomes, such as the ability to deliver high-quality support and use, as well as internal processes, such as the need to manage multiple funding streams and the time needed for collaboration. 3. ARE THE TOOLS THAT HAVE BEEN DEVELOPED USEFUL? This next set of questions will focus on the tools and resources developed by MEASURE Evaluation over the last 3 years. 15. What, if any, tools, methods and/or approaches (and curricula) are you aware of that MEASURE Evaluation has been involved in over the past 3 years? February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 86 NOTE for interviewer: Have a list of tools ready. HIS performance RDQA update-Gender-Integrated Routine Data Quality Assessment (RDQA+G) Tool Guidelines Data Quality Review (DQR) Tool, Version 2.1 DataCollTool Routine Data Quality Assessment Tool: User Manual Guidelines Malaria M&E training curriculum (English and French) TrainResource HIS governance, strategic planning, and management Routine Health Information Systems: Curriculum on Basic Concepts and Practice TrainResource (Facilitator’s Guide and Syllabus) Tools, methods, and approaches PLACE Update: Tool for Collection of Biologic Specimens DataCollTool Summary of HIV Prevalence and Size Estimates in Key Populations Tool DataCollTool Evaluation PEPFAR Monitoring, Evaluation, and Reporting: Collection of Essential Survey Indicators or Orphans and Vulnerable Children Well-being through Outcome Monitoring, Facilitator’s TrainResource Guide to the Data Collector Training; MER OVC Essential Survey indicators FAQ; OVC Survey Toolkit Data Collector's Training and Manual (French Translation) Framework for Evaluating the Impact of National Malaria Control Programs in Highly Guidelines Endemic Countries DATIM ADX Adapter Application Software/App DHIS 2 v2.17-2.28 Software/App PEPFAR Dashboard 3.0 Software/App Data Quality Protocol for DATIM Site Location Guidelines 16. From where you sit, which tools (or types of tools or guidance documents) would you say have been most useful for you or your projects? Probe for examples, epidemic-specific, cross-cutting, gender tools and products. a. In what ways have gender considerations been integrated into tool development, adaptation, use, and dissemination? Probe to see how the respondent defines gender considerations (e.g., data disaggregation, monitoring of impact on girls and women). 17. What have been the facilitating and inhibiting factors in dissemination, use, and/or adaptation of these tools? Probe for MEASURE Evaluation’s contribution to tool development, deployment, adaptation, revision, and dissemination (announcement, brown bags, webinars, podcasts). How successful was MEASURE Evaluation in addressing those barriers? Why? February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 87 18. Can you provide an example of when the use of MEASURE Evaluation tools and products has contributed to decisions related to national/regional health policies, strategies, and outcomes? If yes, how has it happened? Probe: Are you aware of any of these tools being adopted and used by country stakeholders and/or other global stakeholders without USAID’s support? 19. What lessons can be derived from MEASURE Evaluation’s experience in tool development that should be considered for future projects? Probe: What are the characteristics of tools that should be improved and applied globally and at the country level? What tools or types of tools/guidance documents are likely to require continuous investment? LOOKING FORWARD 20. If you had three wishes for improving health information systems and evaluation capacity building for activities you support, what would they be? Probe: Adjust the question to the respondent’s perspective in terms of making progress under PEPFAR, the President’s Malaria Initiative, family planning, the Sustainable Development Goals, and so on. 21. What recommendations or suggestions do you have for how USAID could meet these needs in designing future projects? Probe for optimal structure, management, and organization of the follow-ons; services or activities to add, subtract, or continue to be most successful; and the rationale behind these ideas. 22. [USAID only] If you could design a follow-on project to MEASURE Evaluation, what would it look like in terms of structure, management, and organization? CONCLUSION 23. Is there anything else you feel I/we need to know that you have not yet expressed? 24. Do you have any questions for me/us? Thank you very much for your time and insights. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 88 MEASURE PHASE IV MIDTERM PERFORMANCE EVALUATION INTERVIEW GUIDE: USAID MISSIONS WITHOUT MEASURE EVALUATION Date: Interviewer: Respondent Name/Title/Affiliation: Respondent Contact Information: INTRODUCTION Thank you very much for setting aside time to talk with me today. EnCompass LLC has been contracted by USAID to conduct the midterm performance evaluation of the MEASURE Evaluation Phase IV project. The evaluation will focus on the extent to which MEASURE Evaluation is meeting the needs of its key stakeholders. USAID will use the findings from this performance evaluation to inform technical programming, and for the design and scope of future global health information systems procurements. We recognize that [this] USAID mission has not bought into the MEASURE Evaluation project. You have been selected to speak with us because of your role in supporting health information systems and evaluation-related activities in [this] USAID mission, which has similar portfolio to many USAID missions with MEASURE Evaluation support. The purpose of this interview is to learn about your experience with other health-information systems and evaluation-related mechanisms as well as to explore the opportunities for mechanisms like MEASURE Evaluation to support you in the future. Before we begin, I want to let you know that any information or examples we discuss and any quotes we use in the report will not be attributed to a specific person or institution, and all identifying information will be removed. You are also free not to respond to any of our questions or to stop the interview at any time. The interview will take about 45 minutes. Do you give your consent for this interview? If you don’t mind, I would like to record this conversation, solely for the purposes of listening attentively now and taking notes later. Is that all right? If the respondent agrees, BEGIN RECORDING. Before I/we begin, do you have any questions? February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 89 NOTE TO INTERVIEWER: When you reach 50 minutes, tell the respondent how many questions remain and check that you can complete the questions in the next 10 minutes. If not, ask for additional time. BACKGROUND 1. Please tell me/us briefly how long have you been working with USAID [mission] and what your current role is. Probe: Would you identify your work focused more on evaluation, health information systems, or other monitoring and evaluation and/or technical areas? 2. What mechanism(s) within Global Health Bureau is the USAID mission in [COUNTRY] using to support health information systems and evaluation strengthening services, including monitoring and evaluation tool development and data quality assessments? Probe if necessary:  Supporting Operational AIDS Research (SOAR)  Global Health Program Cycle Improvement Project (GH Pro)  Coordinating Implementation Research to Communicate Learning and Evidence (CIRCLE)  Accelerating Strategies for Practical Innovation and Research in Economic Strengthening (ASPIRES)  Health Evaluation, Research, and Development (HERD)  Breakthrough-RESEARCH project  USAID mission-wide country/regional M&E mechanisms (e.g., IBCTI)  None 3. How did you learn about the existence of these mechanisms? 4. What made you choose this/these mechanism(s) to support your needs in monitoring and evaluation and health information systems? Probe: What goes into their decision-making process in selecting global mechanisms for monitoring and evaluation, evaluation, health information systems, tool development, data quality assessments, and so on. 5. Are you familiar with the MEASURE Evaluation project? If not, SKIP to Sub-question 1a. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 90 6. Did your USAID mission consider using MEASURE Evaluation and then ultimately use another mechanism? If so, which mechanism and why? Probe for global, bilateral, or regional. 1. NEEDS OF STAKEHOLDERS BEING ADDRESSED BY OTHER MECHANISMS Evaluation Sub-question 1a: To what extent is the project meeting the health information systems, evaluation, and learning needs of key stakeholders effectively and efficiently? NOTE TO INTERVIEWER: Tailor the question based on the response to the background question about whether the respondent’s background is in evaluation or health information systems, or other. We would like to start by identifying the most urgent needs related to health information systems and evaluation in [COUNTRY]. 7. If you had to identify the top three most pressing needs in relation to health information systems and/or evaluation-related capacities over the past 3 years in [COUNTRY], what would they be? a. In what ways do you feel that this/these other mechanism(s) have contributed to meeting these needs? Probe separately regarding (1) health information systems needs and (2) evaluation capacity￾building needs, based on the respondent’s background and experience. Are there expectations of contributions that have not materialized? b. What else might still to be done to meet these needs? Probe for specific activities, processes, focus areas, and so on. Evaluation Sub-question 1c: To what extent does the project fit into the current landscape of USAID monitoring and evaluation and health information systems? 8. Are you a member of any of the technical working groups in which MEASURE Evaluation is involved? If so, which one? How do you see the MEASURE Evaluation project’s role in them (leading/participating/coordinating)? Adapt to region and check if on the list:  Bureau for Global Health Cooperating Agencies Monitoring and Evaluation Technical Working Group  Routine Health Information Network February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 91  Community-Based Health Information System Stakeholder Consultation Meetings  Data Usenet Listserv  Every Newborn Action Plan (ENAP) Working Group  Geographic Information Systems (GIS) Working Group  Global Digital Health Network  Global Evaluation and Monitoring Network for Health (GEMNet-Health)  Health Data Collaborative  Implementing Best Practices in Family Planning Consortium  Ministry of Health Data Alignment  PEPFAR Behavioral/Structural Prevention Technical Working Group  PEPFAR Orphans and Vulnerable Children (OVC) Technical Working Group  Programmatic Mapping and Size Estimation for Key Populations Partnership  Roll Back Malaria Monitoring and Evaluation Reference Group (RBM-MERG)  World Health Organization (WHO) Routine Health Information System (RHIS) Strengthening Collaboration If not, SKIP the next section and GO TO Evaluation Sub-question 1d. 9. To what extent do you feel the [technical working group(s) or community(ies) of practice] is/are effective? Probe for specific examples of benefits for the respondent’s work. Probe for specific mechanisms used by technical working groups/communities of practice for coordinating, disseminating knowledge, convening stakeholders, technical leadership in health information systems strengthening, technical leadership in evaluation-related capacity building, and facilitating learning. 2. ARE THE TOOLS THAT HAVE BEEN DEVELOPED USEFUL? This next set of questions will focus on the tools and resources developed by MEASURE Evaluation over the last 3 years, that you may be familiar with. 10. Which, if any, of the following tools, methods, and/or approaches are you aware of? Have any of these been used in [COUNTRY]? If none, SKIP the next section. If yes, probe how they have you been involved in the use of any of these tools? NOTE TO INTERVIEWER: Have a list of tools ready. Health information systems performance RDQA update-Gender-Integrated Routine Data Quality Assessment (RDQA+G) Tool Guidelines February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 92 Data Quality Review (DQR) Tool, Version 2.1 DataCollTool Routine Data Quality Assessment Tool: User Manual Guidelines Malaria M&E training curriculum (English and French) TrainResource Health information systems governance, strategic planning, and management Routine Health Information Systems: Curriculum on Basic Concepts and Practice TrainResource (Facilitator’s Guide and Syllabus) Tools, methods, and approaches PLACE Update: Tool for Collection of Biologic Specimens DataCollTool Summary of HIV Prevalence and Size Estimates in Key Populations Tool DataCollTool Evaluation PEPFAR Monitoring, Evaluation, and Reporting: Collection of Essential Survey Indicators or Orphans and Vulnerable Children Well-being through Outcome Monitoring, Facilitator’s TrainResource Guide to the Data Collector Training; MER OVC Essential Survey indicators FAQ; OVC Survey Toolkit Data Collector's Training and Manual (French Translation) Framework for Evaluating the Impact of National Malaria Control Programs in Highly Guidelines Endemic Countries DATIM ADX Adapter Application Software/App DHIS 2 v2.17-2.28 Software/App PEPFAR Dashboard 3.0 Software/App Data Quality Protocol for DATIM Site Location Guidelines [Other: Country-Specific Tools] 11. From where you sit, which tools, toolkits, or resources (or types of tools or guidance documents) from above would you say have been most useful for country-level stakeholders? Probe for examples: epidemic-specific, cross-cutting, gender tools and products. a. In [Country], what factors have facilitated or inhibited factors in their use and/or adaptation? Probe for tool development, deployment, adaptation, revision, and dissemination (announcement, brown bags, webinars, podcasts). How successful was MEASURE Evaluation in addressing those barriers? Why? b. To what degree do you feel these tools have integrated gender considerations in development, adaptation, use, and dissemination? Probe to see how the respondent defines gender considerations (e.g., data disaggregation, monitoring of impact on girls and women). February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 93 LOOKING FORWARD 12. If you had three wishes for improving health information systems and evaluation capacity building in the future, what would they be? 13. What recommendations or suggestions do you have for how USAID can address these needs in future projects? Probe: Adjust the question to the respondent’s perspective in terms of making progress under PEPFAR, the President’s Malaria Initiative, family planning, Ebola, Zika, the Sustainable Development Goals, and so on. CONCLUSION 14. Is there anything else you feel I/we need to know that you have not yet expressed? 15. Do you have any questions for me/us? Thank you very much for your time and insights. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 94 MEASURE PHASE IV MIDTERM PERFORMANCE EVALUATION VILT GROUP DISCUSSION GUIDE: EXTERNAL STAKEHOLDERS BACKGROUND 1. Please state your name, where you work, and in what capacity you have worked with or interacted with MEASURE in the last 3 years. IS THE PROJECT MEETING THE NEEDS OF ITS STAKEHOLDERS? 2. Please reflect on your dealings with MEASURE Evaluation or the use of MEASURE Evaluation’s services and tools since the start of Phase IV; in other words, the last 3 years. What have you most appreciated and why? 3. Let’s discuss the groups that MEASURE Evaluation has led. Can you tell me a bit about what goals or priorities the groups have achieved to date? a. How did MEASURE help these achievements? Is there anything MEASURE could have done differently? 4. Now let’s discuss the groups that MEASURE participated in. What do you think MEASURE Evaluation’s added value has been in participating in these? WHAT ARE THE BENEFITS OF A HEALTH SECTOR–WIDE VERSUS A HEALTH AREA–SPECIFIC APPROACH? ARE THE TOOLS THAT HAVE BEEN DEVELOPED USEFUL? 5. Can you explain to me in what capacity you have interacted with these tools? For example, were you involved in the use of these tools? Have you heard of others using these tools? 6. Now we will put up two boxes and ask you to reflect on what you have appreciated about these tools and what you wish could have been done differently or what challenged you in working with these tools? February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 95 LOOKING FORWARD 7. Let’s turn to areas for improvement of the project. Again, please reflect on your dealings with MEASURE Evaluation or use of their services and tools since the start of Phase IV, i.e., the last 3 years. If you had to identify any areas of improvement, what would these be? 8. I would like to ask you about the future. If you had to identify the top three most pressing needs in relation to strengthening health information systems and/or evaluation in the countries or regions where you work, what would they be? a. In thinking about these needs, what recommendations or suggestions would you have for USAID to address these needs in future projects? CONCLUSION 9. Is there anything else you feel I/we need to know that you have not yet expressed? Or, do you have any questions for me/us? Thank you so much for your time. Have a great day. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 96 MEASURE PHASE IV MIDTERM PERFORMANCE EVALUATION VILT GROUP DISCUSSION GUIDE: MEASURE BACKGROUND 1. Please state your name, where you work, how long have you been working for MEASURE Evaluation, and what your current role is. IS THE PROJECT MEETING THE NEEDS OF ITS STAKEHOLDERS? 2. I want you to think about your experience working for MEASURE Evaluation, related to evaluation and/or to health information systems, in the last 3 years. I want to go around the small group and have each person share one exceptional experience, high point, or success—a time when you felt progress had been made in making data available for decision making and its use to improve programs or building capacity in monitoring and evaluation. 3. Based on the success stories you just shared, I want you to reflect on the themes you heard and what made these cases a success. If you had to identify key factors that enabled these successes, what would they be? WHAT ARE THE BENEFITS OF A HEALTH SECTOR–WIDE VERSUS A HEALTH AREA–SPECIFIC APPROACH? 4. As you all know very well, MEASURE implements concurrent portfolios of health sector–wide activities (such as DHIS 2 and health management information systems) and activities that are health area–specific (such as work on HIV, with orphans and vulnerable children, and malaria). We would like to hear your feedback on how this structure might have facilitated or inhibited MEASURE Evaluation’s ability to meet key stakeholders’ needs. a. Reflecting on MEASURE Evaluation’s structure, please type advantages and disadvantages you see in implementing a health area–specific AND a health sector– wide project concurrently. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 97 ARE THE TOOLS THAT HAVE BEEN DEVELOPED USEFUL? 5. In thinking about the tools developed during Phase IV of MEASURE, which, if any, have you seen as having the greatest impact? a. Can you provide an example of where any of these tools have had an effect on decision making by external stakeholders, either in policies, strategies, and improved outcomes or by being taken on by other actors, partners, and so on? If yes, why did that happen? What enabled that success? 6. In thinking about the tools MEASURE Evaluation has created in Phase IV, what are some of the key lessons that have been learned in tool creation, dissemination, and use that should be applied to future projects? LOOKING FORWARD 7. In reflecting on MEASURE Evaluation’s areas for improvement and successes and challenges over the last 3 years, what do you think the project can do more of and less of to ensure that the project’s needs are met and MEASURE Evaluation is able to accomplish its goals by end of Phase IV? 8. If you had to identify the most pressing needs for health information systems and evaluation￾related support, what would they be? 9. What recommendations or suggestions do you have for how USAID could address those needs in future projects? CONCLUSION 10. Is there anything else you feel I/we need to know that you have not yet expressed? 11. Do you have any questions for me/us? Thank you so much for your time. Have a great day. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 98 ANNEX 6: DATA COLLECTION TOOLS (ONLINE SURVEY QUESTIONNAIRES) The following three tools are being submitted separately:  Online survey questionnaire for USAID/Washington staff  Online survey questionnaire for USAID mission and other external stakeholders – English  Online survey questionnaire for USAID mission and other external stakeholders – French February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 99 INTRODUCTION EnCompass LLC is conducting the Midterm Performance Evaluation of the MEASURE Evaluation Project - Phase IV (2014- Present). The evaluation focuses on the extent to which the project has been meeting the needs of its key stakeholders the last three years. USAID and MEASURE Evaluation will use the findings to inform technical programming and project management for the remaining two years of Phase IV, and for the design of future global health information systems procurement. You have been selected to complete this survey because of your work in health information systems and evaluation capacity building at the global, regional, or country level. Any information or examples you provide and any quotes will not be attributed to a specific person or institution, and all identifying information will be removed. You are also free not to respond to any of our questions. Please kindly complete and return this survey to us by November 17, 2017. ` A. RESPONDENT 1. What is your sex? Female Male 2. Please select which country you work in: B. TECHNICAL WORKING GROUP/COMMUNITY OF PRACTICE MEMBERS * 3. Have you ever been a member of any of the global, regional, or country Technical Working Groups/Communities of Practice that MEASURE Evaluation Phase IV led? Please check all that apply: N/A I have not been a member of any group that MEASURE Geographic Information Systems (GIS) Working Group Phase IV led Global Evaluation and Monitoring Network for Health Bureau for Global Health Cooperating Agencies Monitoring (GEMNet-Health) and Evaluation (M&E) Technical Working Group (TWG) Routine Health Information Network (RHINO) Data Use Net Listserv I have been a member of another group that MEASURE led (please name the group): B. TECHNICAL WORKING GROUP/COMMUNITY OF PRACTICE MEMBERS Monitoring and Evaluation (M&E) 4. Please rate how effective the Technical Working Group/ Community of Practice has been in achieving its goals: Neither effective Very effective Effective nor ineffective Ineffective Very ineffective N/A N/A I have not been a member of any group that MEASURE Phase IV led Bureau for Global Health Cooperating Agencies Technical Working Group (TWG) Data Use Net Listserv Geographic Information Systems (GIS) Working Group Global Evaluation and Monitoring Network for Health (GEMNet-Health) Guinea National HIS TWG Kenya Data Security for HIV Systems Advisory Group Mali Monitoring & Evaluation Working Group Nepal GIS Listserv Routine Health Information Network (RHINO) Sierra Leone National HIS TWG N/A I have not been a member of any group that MEASURE Phase IV led I have been a member of another group that MEASURE led (please name the group): Please comment on the strengths and weaknesses of the group to date: agree Agree nor disagree Disagree disagree N/A Neither agree Strongly Strongly Enabled me to learn about strengthening HIS Enabled me to learn about strengthening evaluation Enabled me to learn Enhanced my job 5. On average, please indicate to what extent participation in the Technical Working Groups/Commu of Practice has benefitted you: nities about similar work of others Enabled me to get advice from others on technical issues Enabled me to locate useful knowledge and resources performance Please describe to what extent the group has helped your or your organization: B. TECHNICAL WORKING GROUP/COMMUNITY OF PRACTICE MEMBERS * 6. Have you ever been a member of any of the global, regional, or country Technical Working Groups/Communities of Practice that MEASURE Evaluation IV participated in? Please check all that apply: N/A--I was not a member of any group that MEASURE IV Implementing Best Practices in Family Planning Consortium participated in Mali Malaria in Pregnancy Working Group Asia eHealth Information Network (AeHIN) RHIS Subgroup MOH Data Alignment Community-Based Health Information System (HIS) Stakeholder Consultation Meetings Programmatic Mapping and Size Estimation for Key Populations Partnership East Africa Sample Vital Registration with Verbal Autopsy (SAVVY)/Civil Registration and Vital Statistics Collaboration Roll Back Malaria Monitoring and Evaluation Reference Group (RBM-MERG) Every Newborn Action Plan (ENAP) Working Group West African Health Organization Collaboration Global Digital Health Network WHO Routine Health Information System (RHIS) Health Data Collaborative Strengthening Collaboration Other group (please specify): B. TECHNICAL WORKING GROUP/COMMUNITY OF PRACTICE MEMBERS 7. On average, please rate how helpful MEASURE Evaluation was in assisting the groups you were a member of with the following: Very helpful Moderately hepful Not involved N/A Achieving the group's goals Disseminating knowledge Convening stakeholders Facilitating learning Providing or creating useful tools or resources Please comment on the involvement and level of help that MEASURE provided to the group(s): C. TOOLS AND RESOURCES * 8. The following is a subset of tools that MEASURE Evaluation invested in and which have been highly accessed in the last 3 years. Have you ever worked with any of these tools? Côte d'Ivoire Data Collection Tools for Community-Level Indicators for Early Warning System Data Quality Protocol for DATIM Site Location Data Quality Review (DQR) Tool, Version 2.1 DHIS 2 (versions after 2014) Framework for Evaluating the Impact of National Malaria Control Programs in Highly Endemic Countries Guidelines for Integrating Gender into an M&E Framework and System Assessment (English/French) Malaria M&E training curriculum (English/ French) HIS Strengthening Model (HISSM) PEPFAR Dashboard 3.0 PEPFAR M&E, and Reporting: Collection of Essential Survey Indicators of OVC Well-being through Outcome Monitoring Planning Guide for a Total Market Approach to Increase Access to Family Planning PLACE Update: Tool for Collection of Biologic Specimens PLACE Update: Summary of HIV Prevalence and Size Estimates in Key Populations Tool PRISM Tools for Assessing, Monitoring, and Evaluating RHIS Performance Results-Based Financing (RBF) Indicator Compendium Routine Data Quality Assessment (RDQA) Tool: User Manual RDQA update-Gender-Integrated Routine Data Quality Assessment (RDQA+G) Tool Routine Health Information Systems (RHIS): Curriculum on Basic Concepts and Practice (Facilitator’s Guide and Syllabus) Sample Vital Registration with Verbal Autopsy (SAVVY) Yes No C. TOOLS AND RESOURCES * 9. Please select all of the tools that you have worked with: Côte d'Ivoire Data Collection Tools for Community-Level Indicators for Early Warning System Data Quality Protocol for DATIM Site Location Data Quality Review (DQR) Tool, Version 2.1 DHIS 2 (version after 2014) Framework for Evaluating the Impact of National Malaria Control Programs in Highly Endemic Countries Guidelines for Integrating Gender into an M&E Framework and System Assessment (English/French) HIS Strengthening Model (HISSM) Malaria M&E training curriculum (English/ French) PEPFAR Dashboard 3.0 PEPFAR M&E, and Reporting: Collection of Essential Survey Indicators of OVC Well-being through Outcome Monitoring PLACE Update: Tool for Collection of Biologic Specimens PLACE Update: Summary of HIV Prevalence and Size Estimates in Key Populations Tool Planning Guide for a Total Market Approach to Increase Access to Family Planning PRISM Tools for Assessing, Monitoring, and Evaluating RHIS Performance Results-Based Financing (RBF) Indicator Compendium Routine Data Quality Assessment (RDQA) Tool: User Manual RDQA update-Gender-Integrated Routine Data Quality Assessment (RDQA+G) Tool Routine Health Information Systems (RHIS): Curriculum on Basic Concepts and Practice (Facilitator’s Guide and Syllabus) Sample Vital Registration with Verbal Autopsy (SAVVY) C. TOOLS AND RESOURCES 10. Please rate the utility of each of the MEASURE Evaluation tools that you have worked with: Very useful Somewhat useful Not at all useful Côte d'Ivoire Data Collection Tools for Community-Level Indicators for Early Warning System Data Quality Protocol for DATIM Site Location Data Quality Review (DQR) Tool, Version 2.1 DHIS 2 (version after 2014) Framework for Evaluating the Impact of National Malaria Control Programs in Highly Endemic Countries Guidelines for Integrating Gender into an M&E Framework and System Assessment (English/French) Malaria M&E training curriculum (English/ French) HIS Strengthening Model (HISSM) PEPFAR Dashboard 3.0 PEPFAR M&E, and Reporting: Collection of Essential Survey Indicators of OVC Well￾being through Outcome Monitoring PLACE Update: Tool for Collection of Biologic Specimens PLACE Update: Summary of HIV Prevalence and Size Estimates in Key Populations Tool No Yes Very useful Somewhat useful Not at all useful Routine Data Quality Assessment (RDQA) Tool: User Manual RDQA update-Gender￾Integrated Routine Data Quality Assessment (RDQA+G) Tool Routine Health Information Systems (RHIS): Curriculum on Basic Concepts and Practice (Facilitator’s Guide and Syllabus) Planning Guide for a Total Market Approach to Increase Access to Family Planning PRISM Tools for Assessing, Monitoring, and Evaluating RHIS Performance Results-Based Financing (RBF) Indicator Compendium Sample Vital Registration with Verbal Autopsy (SAVVY) Please elaborate the utility of the tools that you have worked with: 11. Do you have any examples of MEASURE Evaluation tools and products having an impact on an organization, a health system, health strategy, or health outcomes? If yes, please elaborate on how this was achieved 12. Please comment on MEASURE Evaluation's ability to produce useful, relevant tools: Yes No Don't Know Are you aware of any instances in which the MEASURE Evaluation project was not able to produce a tool that was requested? Are you aware of any MEASURE Evaluation tools that were not very useful, relevant or appropriate? If you marked Yes for either question, please elaborate: Type of organization * 13. Please indicate which type of organization you work in: I work at a USAID Mission that bought into MEASURE Evaluation I work at a USAID Mission that did not use MEASURE I work at USAID/Washington D. USAID Perspective * 14. Check which one of the following applies to your context: D. USAID PERSPECTIVE 15. What is the reason that your Mission has not used MEASURE? We feel that another USAID/W mechanism better meets our needs We have a regional mechanism that we use for similar activities We have a bilateral mechanism that we use for similar activities Other Please elaborate on any other reasons: D. USAID PERSPECTIVE 16. Why did your Office or Missions buy into MEASURE Evaluation? Check all that apply: Strong reputation in HIS strengthening Strong reputation in evaluation capacity building No better alternative mechanism, project, or program Familiarity with past services and performance Ease of buy in into mechanism Other (please specify): * 17. Which of the following USAID-funded projects that work in evaluation and/or health information systems are you familiar with? Check all that apply Supporting Operational AIDS Research (SOAR) Global Health Program Cycle Improvement Project (GHPro) Coordinating Implementation Research to Communicate Learning and Evidence (CIRCLE) Accelerating Strategies for Practical Innovation and Research in Economic Strengthening (ASPIRES) Health Evaluation, and Applied Research and Development (HEARD) Breakthrough-RESEARCH project USAID bilateral or regional M&E platforms or mechanisms Digital Health Initiative None Other (please specify) D. USAID PERSPECTIVE 18. Please describe MEASURE’s strengths and challenges compared to these projects. D. USAID PERSPECTIVE 19. Based on your experience with MEASURE Evaluation, please rate the project's performance in building capacity to do the following: Very high Above average Average Below average Very low Don't know Managing health information systems Collecting and using health information to make strategic decisions Conducting evaluations Please elaborate on your ratings above: 20. Given your experience with the MEASURE Evaluation project, what are thestrengths of the project that you would you like to see maintained in a future project? 21. Given your experience with the MEASURE Evaluation project, what are theweaknesses of the project that you would like to see changed in future projects? collaboration with other partners supporting 22. To what extent do you feel that MEASURE has implemented the following activities in ways that will encourage sustainability of HIS/Evaluation efforts at the country level after the end of Phase IV? To a great extent To some extent Neutral Very little Not at all Don't know Implemented HIS strengthening activities in ways that encourage true ownership and leadership by national counterparts Facilitated broader coordination and various aspects of HIS strengthening Co-developed on-going financing of recurrent costs of the HIS system (equipment, supervision, training, supplies) Built adequate capacity of national counterparts to update the HIS as needs evolve Built adequate capacity of national counterparts to conduct rigorous evaluations Contributed to development of a culture of data use at national level of the health system Contributed to development of a culture of data use at decentralized levels of the health system Please add any comments on the sustainability of MEASURE's activities: E. CLOSING QUESTIONS * 23. What do you consider top three emerging needs in the next 2 – 5 years for the development of health information systems in the region/country you cover or work in? Please select three: Improved capacity to demand, implement and use HIS at Improved capacities to monitor and evaluate HIS national and sub-national (regional/district) levels. performance Improved capacities of sub-national (regional/district) level Interoperability of databases (DHIS2, logistics databases, governments to manage health information systems human resource databases, other specific interventions, such resources and processes, including supervision and ongoing as OVC, HIV/DATIM, immunizations, etc.) training Increased availability of procedural guidance documents for Increased use of HIS data for decision making at national and software and processes, such as Standard Operating sub-national (regional/district) levels Procedures (SOP) Increased use of sex-disaggregated data from HIS to identify Increased resources for maintenance and replacement of HIS and plan interventions to close gender gaps software and infrastructure (computers, internet access, solar kits, etc) Integration of HIS data to respond to health-related Sustainable Development Goals (SDGs) and respective Improved quality of routine health information targets Other (please specify) * 24. What do you consider top two emerging needs in the next 2 – 5 years for the development of evaluation capacity in the region/country you cover or work in?Please select up to two: Improved capacity of national government decision-makers to demand and use evidence from rigorous evaluations for planning and advocacy Improved capacity of national institutions to design, implement and facilitate use of findings from rigorous evaluations Improved capacities of national institutions to demand and use evidence from equity-focused and gender responsive evaluations for planning interventions Improved quality of evaluations in terms of level rigor, quality of designs and data collected, relevance to answer priority questions, and availability and dissemination for use that are accessible to decision-makers Other (please specify) Yes No 25. Does your organization plan to use or collaborate with the MEASURE Evaluation project, in the future? Don't know N/A If Yes or No, please explain why: 26. Anything else to add? Kindly provide any comments or questions: THANK YOU FOR YOUR PARTICIPATION Thank you for taking the time to respond to this survey. If you have any questions, please do not hesitate to contact stopolansky@encompassworld.com. INTRODUCTION EnCompass LLC is conducting the Midterm Performance Evaluation of the MEASURE Evaluation Project - Phase IV (2014- Present). The evaluation focuses on the extent to which the project has been meeting the needs of its key stakeholders the last three years. USAID and MEASURE Evaluation will use the findings to inform technical programming and project management for the remaining two years of Phase IV, and for the design of future global health information systems procurement. You have been selected to complete this survey because of your work with MEASURE. Any information or examples you provide and any quotes will not be attributed to you. Please kindly complete and return this survey to us by November 17, 2017. A. Intro * 1. Check which of the following applies to your context: I work at the USAID/Washington Bureau of Global Health (BGH) I do not work at the USAID/Washington Bureau of Global Health (BGH) If you do not work at BGH, please specify where you work: Providing technical leadership B. EXPERIENCE WITH MEASURE 2. Please describe your involvement with MEASURE Evaluation Phase IV: 3. Based on your experience with MEASURE Evaluation Phase IV, please rate your satisfaction with the project’s services: Very satisfied Satisfied Neutral Unsatisfied Very unsatisfied Don't know Delivering useful tools and products Delivering tools and products in a timely manner Leading technical working groups Coordinating with USAID Coordinating with other stakeholders Please elaborate on your ratings above: B. EXPERIENCE WITH MEASURE 4. Based on your experience with MEASURE Evaluation, please rate the project's performance in building others’ capacity to do the following: Very high Above average Average Below Average Very low Don't know Managing health information systems Collecting and using health information to make strategic decisions Conducting evaluations Integrating gender in M&E of health programs and HIS Please elaborate on your ratings above: 5. In your view, what have been the project’s main achievements? 6. In your view, what have been the project’s main challenges? 7. Overall, what lessons have been learned regarding the structure, management, and results of this project that USAID should apply in creating a future project? Improved capacity to demand, implement and use HIS at Improved capacities to monitor and evaluate HIS national and sub-national (regional/district) levels. performance Improved capacities of sub-national (regional/district) level Interoperability of databases (DHIS2, logistics databases, governments to manage health information systems human resource databases, other specific interventions, such resources and processes, including supervision and ongoing as OVC, HIV/DATIM, immunizations, etc.) training Increased availability of procedural guidance documents for Increased use of HIS data for decision making at national and software and processes, such as Standard Operating sub-national (regional/district) levels Procedures (SOP) Increased use of sex-disagg er gated data from HIS to identify Increased resources for maintenance and replacement of HIS and plan interventions to close gender gaps software and infrastructure (computers, internet access, solar kits, etc) Integration of HIS data to respond to health-related Sustainable Development Goals (SDGs) and respective Improved quality of routine health information targets C. CLOSING QUESTIONS * 8. What do you consider top three emerging needs in the next 2 – 5 years for the development of health information systems in the region/country you cover or work in? Please select up to three: Other (please specify): * 9. What do you consider top two emerging needs in the next 2 – 5 years for the development ofevaluation capacity in the region/country you cover or work in?Please select up to two: Improved capacity of national government decision-makers to demand and use evidence from rigorous evaluations for planning and advocacy Improved capacity of national institutions to design, implement and facilitate use of findings from rigorous evaluations Improved capacities of national institutions to demand and use evidence from equity-focused and gender responsive evaluations for planning interventions Improved quality of evaluations in terms of level rigor, quality of designs and data collected, relevance to answer priority questions, and availability and dissemination for use that are accessible to decision-makers Other (please specify): C. CLOSING QUESTIONS 10. Which of the following USAID-funded projects that work in evaluation and/or health information systems are you familiar with? Check all that apply: Supporting Operational AIDS Research (SOAR) Global Health Program Cycle Improvement Project (GHPro) Coordinating Implementation Research to Communicate Learning and Evidence (CIRCLE) Accelerating Strategies for Practical Innovation and Research in Economic Strengthening (ASPIRES) Health Evaluation, and Applied Research and Development (HEARD) Breakthrough-RESEARCH project USAID bilateral or regional M&E platforms or mechanisms Digital Health Initiative Other (please specify): 11. In your view, of the projects that you are familiar with, which USAID-funded project is best able to address the emerging HIS and evaluation needs? If none of the projects, why not? C. CLOSING QUESTIONS 12. Anything else to add? Kindly provide any comments or questions: THANK YOU FOR YOUR PARTICIPATION Thank you for taking the time to respond to this survey. If you have any questions, please do not hesitate to contact stopolansky@encompassworld.com. INTRODUCTION EnCompass LLC effectue l'évaluation à mi-parcours du projet MEASURE Evaluation - Phase IV (2014 – à nos jours). L'évaluation se base sur la mesure dans laquelle le projet a répondu aux besoins de ses principales parties prenantes au cours des trois dernières années. L'USAID et MEASURE Evaluation utiliseront les résultats pour informer les programmes techniques et la gestion du projet pour les deux années restantes de la Phase IV, et pour la conception de la prochaine acquisition des systèmes d’information mondiaux sur la santé. Vous avez été sélectionné pour remplir ce questionnaire en raison de votre travail dans les systèmes d'information sur la santé et dans le renforcement des capacités de l'évaluation au niveau mondial, régional ou national. Toutes les informations ou exemples et citations que vous donnerez ne seront pas attribués à une personne ou une institution spécifique, et toutes les informations d'identification seront supprimées. Vous êtes également libre de ne répondre à aucune de nos questions. Merci de bien vouloir remplir et renvoyer ce questionnaire avant le 20 novembre 2017. A. REPONDENT 1. Quel est votre sexe? Féminin Masculin 2. Veuillez sélectionner le pays dans lequel vous travaillez : B. GROUPE DE TRAVAIL TECHNIQUE/MEMBRES DE LA COMMUNAUTÉ DE PRATIQUE * 3. Avez-vous déjà été membre de l'un des Groupes de Technique de Travail (GTT)/communautés de pratique mondiaux, régionaux ou nationaux que MEASURE Evaluation Phase IV a coordonné ? Veuillez cocher tout ce qui s’applique à vous : N/A Je n'ai fait partie d'aucun groupe de travail coordonné par Systèmes d’Information Géographique (SIG) ME pendant la phase IV du projet. Systèmes d’Information Géographique (SIG) Suivi-Evaluation du Réseau mondial pour la santé (GEMNet￾Santé) Suivi-Evaluation (S&E) du Groupe de Technique de Travail des agences de coopération en santé mondiale Réseau d'information de routine sur la santé (RHINO) Utilisation des données Listserv net J'ai été membre d'un autre groupe technique de travail coordonné par MEASURE (Veuillez nommer le groupe) : B. GROUPE DE TECHNIQUE DE TRAVAIL/MEMBRES DE LA COMMUNAUTÉ DE PRATIQUE 4. Veuillez évaluer l'efficacité du groupe de technique de travail/communauté de pratique : Ni efficace ni Très efficace Efficace inefficace Inefficace Très inefficace N/A N/A Je n'ai fait partie d'aucun groupe de travail coordonné par ME pendant la phase IV du projet. Systèmes d’Information Géographique (SIG) Suivi-Evaluation (S&E) du Groupe de Technique de Travail des agences de coopération en santé mondiale Utilisation des données Listserv net Systèmes d’Information Géographique (SIG) Suivi-Evaluation du Réseau mondial pour la santé (GEMNet-Santé) Guinea National HIS TWG Kenya Data Security for HIV Systems Advisory Group Mali Monitoring & Evaluation Working Group Nepal GIS Listserv Réseau d'information de routine sur la santé (RHINO) Sierra Leone National HIS TWG N/A I have not been a member of any group that MEASURE Phase IV led M'a permis de me renforcer dans le SIS me Renforcer dans l'évaluation Ni efficace ni Très efficace Efficace inefficace Inefficace Très inefficace N/A J'ai été membre d'un autre groupe technique de travail coordonné par MEASURE (Veuillez nommer le groupe) : Veuillez commenter les forces et les faiblesses du groupe à ce jour : 5. En moyenne, veuillez indiquer dans quelle mesure vous avez bénéficié de la participation aux groupes Technique de Travail/communautés de pratique : Fortement Ni d'accord ni en Fortement pas d’accord D’accord désaccord Pas d’accord d’accord N/A M'a permis de M'a permis d'apprendre des autres M'a permis de renforcer mes compétences sur d’autres questions techniques M'a permis d’améliorer mes connaissances et d’autres ressources utiles A améliorer les performances de mon travail Veuillez décrire dans quelle mesure le groupe vous a aidé ou votre organisation : B. GROUPE TECHNIQUE DE TRAVAIL/MEMBRES DE LA COMMUNAUTÉ DE PRATIQUE * 6. Avez-vous déjà été membre de l'un des groupes techniques de travail /communautés de pratique mondiaux, régionaux ou nationaux auxquels MEASURE Evaluation IV a participé? Veuillez cocher tout ce qui s'applique : N/A--Je n'ai été membre d'aucun groupe auquel MEASURE Mise en œuvre de bonnes pratiques dans le Consortium IV a participé auquel de planification familiale Sous-groupe RHIS du réseau (AeHIN) d’information eHealth Groupe de travail sur le paludisme en période de grossesse Asie au Mali Système d’information en santé communautaire (HIS) Alignement des données MS Réunions consultatives des parties prenantes Cartographie programmatique et estimation de la taille de Enregistrement vital des échantillons de l'Afrique de l'Est partenariat avec les populations clés avec autopsie verbale (SAVVY)/Collaboration en matière d'état civil et de statistiques de l'état civil Faire reculer le Groupe (RBM-MERG) paludisme Collaboration de l'Organisation Ouest Africaine de la Santé Groupe de travail sur le plan d'action pour le nouveau-né (ENAP) Système d'information de routine sur la santé de l’OMS (RHIS) Réseau mondial de santé numérique Collaboration des données sur la santé Autre groupe (veuillez préciser): B. GROUPE TECHNIQUE DE TRAVAIL/MEMBRES DE LA COMMUNAUTÉ DE PRATIQUE 7. En moyenne, veuillez évaluer l'utilité de l'évaluation de MESURE Evaluation dans l’assistance des groupes où vous étiez membre : Très utile Légèrement utile Pas impliqué N/A Atteindre les objectifs du groupe Diffuser les connaissances Regrouper les parties prenantes Faciliter l’apprentissage Fournir ou créer outils ou ressources utiles Veuillez commenter l'implication et le niveau d'aide que MEASURE a fourni au (x) groupe (s) : C. OUTILS ET RESSOURCES * 8. Ce qui suit est un sous-ensemble d'outils dans lesquels MEASURE Evaluation a investi et qui ont été largement consultés au cours des 3 dernières années. Avez-vous déjà travaillé avec l'un de ces outils? Outils de collecte des données de la Côte d'Ivoire pour les indicateurs communautaires pour le système d'alerte précoce Protocole de la qualité des données pour le lieu du site DATIM Revue de la Qualité des données - Outil de Revue de la Qualité des données (DQR), Version 2.1 DHIS 2 v2.17-2.28 (versions après 2014) Cadre d'évaluation de l'impact des programmes nationaux de lutte contre le paludisme dans les pays très endémiques Guide pour l’intégration du genre dans une évaluation d’un cadre et système de suivi et évaluation Programme de formation sur le S & E du paludisme (anglais et français) Le modèle de renforcement des systèmes d'information sanitaires de MEASURE Evaluation (HISSM) Tableau de bord 3.0 de PEPFAR Suivi, Evaluation, et Rapports (SER) du PEPFAR: collecte des indicateurs d'enquête essentiels ou bien être des orphelins par le biais du suivi des résultats des enfants vulnérables Guide de planification d’une approche globale du marché afin d'améliorer l'accès à la planification familiale Mise à jour PLACE (Priorités pour les efforts locaux de lutte contre le SIDA): Outil pour la collecte de spécimens biologiques PLACE: Résumé de la prévalence du VIH et outil des estimations de la taille des populations clés La performance de la gestion du système d'information de routine (PRISM) 2.0 Financements basées sur les résultats (PBF) Outil d'évaluation de routine de la qualité des données (RDQA): Manuel de l'utilisateur Mise à jour RDQA - Outil d'évaluation de la qualité des données de routine intégrant le genre (RDQA + G) Oui Non C. TOOLS AND RESOURCES * 9. Veuillez sélectionner tous les outils avec lesquels vous avez travaillé : Outils de collecte de données de la Côte d'Ivoire pour les indicateurs communautaires du système d'alerte précoce Protocole de qualité des données pour le Lieu du Site DATIM Revue de la Qualité des données - Outil de Revue de la Qualité des données (DQR), Version 2.1 DHIS 2 (version après 2014) Cadre d'évaluation de l'impact des programmes nationaux de lutte contre le paludisme dans les pays très endémiques Principes pour l'intégration du genre dans un cadre de S&E et évaluation du système (anglais / français) Le modèle de renforcement des systèmes d'information sanitaires de MEASURE Evaluation (HISSM) Programme de formation sur le S & E du paludisme (anglais et français) Tableau de bord 3.0 de PEPFAR Suivi, Evaluation, et Rapports (SER) du PEPFAR: collecte des indicateurs d'enquête essentiels ou bien être des orphelins par le biais du suivi des résultats des enfants vulnérables Mise à jour PLACE (Priorités pour les efforts locaux de lutte contre le SIDA): Outil pour la collecte de spécimens biologiques Mise à jour PLACE: Résumé de la prévalence du VIH et outil des estimations de la taille des populations clés Guide du praticien de l'approche globale du marché, modules 1 et 2 (TMA) La performance de la gestion du système d'information de routine (PRISM) 2.0 Financements basées sur les résultats (PBF) Outil d'évaluation de routine sur la qualité des données (RDQA): Manuel de l'utilisateur Mise à jour RDQA - Outil d'évaluation de la qualité des données de routine intégrant le genre (RDQA + G) Systèmes d'information de routine sur la santé (RHIS) : Programme des concepts de base et les pratiques (Guide de l'animateur et programme d'études) Echantillon de l'enregistrement de l’état civil avec autopsie verbale (SAVVY) C. TOOLS AND RESOURCES 10. Veuillez évaluer l'utilité de chacun des outils de MEASURE Evaluation avec lesquels vous avez travaillé : Très utile Peu utile Pas du tout utile Outils de collecte de données de la Côte d'Ivoire pour les indicateurs communautaires du système d'alerte précoce Protocole de qualité des données pour le Lieu du Site DATIM Revue de la Qualité des données - Outil de Revue de la Qualité des données (DQR), Version 2.1 DHIS 2 (version après 2014) Cadre d'évaluation de l'impact des programmes nationaux de lutte contre le paludisme dans les pays très endémiques Principes pour l'intégration du genre dans un cadre de S&E et évaluation du système (anglais / français) Programme de formation sur le S & E du paludisme (anglais et français) Le modèle de renforcement des systèmes d'information sanitaires de MEASURE Evaluation (HISSM) Tableau de bord 3.0 de PEPFAR Très utile Peu utile Pas du tout utile Suivi, Evaluation, et Rap ports (SER) du PEPFAR: collecte des indicateurs d'enquête essentiels ou bien être des orphelins par le biais du suivi des résultats des enfants vulnérables Mise à jour PLAC E (Priorités pour les efforts locaux de lutte contre le SIDA): Outil pour la collecte de spécimens biologiques Mise à jour PLAC E : Résumé de la prévalence du VIH et outil des estimations de la taille des populations clés Outil d'évaluation de rou tine sur la qualité des données (RDQA): Manuel de l'utilisateur Mise à jour RDQA - Outil d'évaluation de la qualité des données de routine intégrant le genre (RDQA + G) Systèmes d'info rmation de routine sur la santé (RHIS) : Programme des concepts de base et les pratiques (Guide de l'animateur et programme d'études) Guide du praticien de l' approche globale du marché, modules 1 et 2 (TMA) La performance de la gestion du système d'information de routine (PRISM) 2.0 Financements basées sur les résultats (PBF) Echantillon de l'en registrement de l’état civil avec autopsie verbale (SAVVY) Veuillez préciser l'utilité des outils avec lesquels vous avez travaillé : 11. Avez-vous des exemples d'outils et de produits MEASURE Evaluation ayant un impact sur une organisation, un système de santé, une stratégie de santé ou des résultats sur la santé? Non Oui Si oui, veuillez préciser comment cela a été réalisé : 12. Veuillez commenter les capacités de MEASURE Evaluation à produire des outils utiles et pertinents : Oui Non Ne sait pas Êtes-vous au courant des cas où le projet MEASURE Evaluation n'a pas été en mesure de produire un outil qui a été demandé? Connaissez-vous des outils de MEASURE Evaluation qui n'étaient pas très utiles, pertinents ou approprié ? Si vous avez coché Oui pour l'une ou l'autre question, veuillez préciser : TYPE D'ORGANISATION * 13. Veuillez indiquer le type d'organisation dans lequel vous travaillez : D. PERSPECTIVE DE L’USAID * 14. Cocher l’un des éléments suivants qui s'applique à votre contexte : Je travaille dans une mission de l'USAID qui a adhéré MEASURE Evaluation Je travaille dans une mission de l'USAID qui n'a pas utilisé MEASURE Je travaille à l'USAID / Washington D. PERSPECTIVE DE L’USAID 15. Quelle est la raison pour laquelle votre mission n'a pas utilisé MEASURE? Nous pensons qu'un autre mécanisme de l'USAID/W répond mieux à nos besoins Nous avons un mécanisme régional que nous utilisons pour des activités similaires Nous avons un mécanisme bilatéral que nous utilisons pour des activités Autre Veuillez préciser d’autres raisons : D. PERSPECTIVE DE L’USAID 16. Pourquoi votre bureau ou vos missions ont-ils participé à MEASURE Evaluation? Cocher tout ce qui s’applique : Forte compétence dans le renforcement de SIS Forte compétence dans le renforcement des capacités en évaluation Pas de bon mécanisme, projet ou programme alternatif Familiarité avec les services et performances antérieurs Facile à adhérer le mécanisme Autre (à préciser) : * 17. Lequel des projets suivants financés par l'USAID qui travaillent dans l'évaluation et/ou dans les systèmes d'information sur la santé connaissez-vous? Cocher tout ce qui s’applique. Soutenir la recherche opérationnelle sur le sida (SOAR) Projet d'amélioration du cycle des programmes de santé mondiale (GHPro) Coordonner la recherche sur la mise en œuvre pour communiquer l'apprentissage et les données (CIRCLE) Accélérer les stratégies d'innovation pratique et de recherche dans le renforcement économique (ASPIRES) Évaluation de la santé, recherche appliquée et développement (HEARD) Projet Innovation-RECHERCHE/ Plateformes ou mécanismes bilatéraux ou régionaux de S&E de l’Initiative de Santé Numérique de l'USAID Initiative de santé numérique Aucun Autre (à préciser) D. PERSPECTIVE DE L’USAID 18. Veuillez décrire les forces et les défis de MEASURE par rapport à ces projets D. PERSPECTIVE DE L’USAID 19. Sur la base de votre expérience avec MEASURE Evaluation, veuillez évaluer les performances du projet en matière de renforcement des capacités à faire ce qui suit : Au-dessus de la Au-desouss de Très élevé moyenne Moyenne la moyenne Très bas Ne sait pas Gestion des systèmes d'information sur la santé Collecte et utilisation des informations sur la santé pour prendre des décisions stratégiques Réalisation des évaluations Veuillez commenter vos notations ci-dessus : 20. Compte tenu de votre expérience avec le projet MEASURE Evaluation, quelles sont que vous aimeriez voir maintenues dans un futur projet? lesforces du projet 21. Compte tenu de votre expérience avec le projet MEASURE Evaluation, quelles sont lesfaiblesses du projet que vous aimeriez voir changées dans les futurs projets? 22. Dans quelle mesure estimez-vous que MEASURE a mis en œuvre les activités suivantes de manière à encourager la durabilité des efforts du SIS/Evaluation au niveau des pays après la fin de la Phase IV? c a Une certaine En grande partie mesure Neutre Très peu Pas du tout Ne sait pas MEASURE a mis en œuvre des activités de renforcement des SIS de manière à encourager une véritable appropriation et un leadership par les homologues nationaux MEASURE a encouragé t facilité une oordination et une ollaboration plus larges vec d'autres artenaires soutenant ivers aspects du nforcement des SIS e c p d re MEASURE a travaillé avec le gouvernement et d'autres partenaires pour développer le financement continu des coûts récurrents du système SIS (équipements, supervision, formation, fournitures) MEASURE a renforcé les capacités adéquates des homologues nationaux à mettre à jour SIS, selon les besoins MEASURE a renforcé les capacités adéquates des homologues nationaux à mener des évaluations rigoureuses MEASURE a contribué au développement d'une culture de l'utilisation des données au niveau national du système de santé MEASURE a contribué au développement d'une culture de l'utilisation des données au niveau décentralisé du système de santé Veuillez ajouter des commentaires sur la durabilité des activités de MEASURE E. QUESTIONS FINALES * 23. Selon vou développeme s, quels sont les trois principaux nouveaux besoins dans les 2-5 prochaines années pour le nt des systèmes d'information sur la santé dans la région/pays où vous intervenez ou travaillez? Veuillez en sélectionner trois: Amélioration des capacités à identifier, mettre en œuvre et Amélioration des capacités à suivre et évaluer utiliser le SIS au niveau national et les performances infranational (régional/district). Interopérabilité des bases de données (DHIS2, basesde Amélioration des capacités au niveau données logistiques, bases de données des infranational (régional/district) des gouvernements à gérer les ressources humaines, autres interventions spécifiques, telles systèmes d’information sur la santé, les ressources et les que OVC, HIV/DATIM, vaccinations, etc.) méthodes, y compris la supervision et la formation continue Amélioration de la disponibilité des documents d’orientation Amélioration de l’utilisation des données du SIS pour la procédurale pour les logiciels et les méthodes, tels que les prise de décision au niveau national et Procédures Opérationnelles Standard (SOP) infranational (régional/cercle) Augmentation des ressources pour l’entretien et Amélioration de l’utilisation des données désagrégées par le remplacement des logiciels et infrastructures sexe à partir du SIS pour identifier et planifier les interventions HIS (ordinateurs, accès Internet, kits solaires, etc.) afin de combler les lacunes du genre Amélioration de la qualité des informations de routine sur la Intégration des données du SIS pour atteindre les Objectifs santé du Développement Durable (ODD) et les objectifs respectifs sur la santé Autre (à préciser) * 24. Selon vous, quels sont les deux principaux nouveaux besoins dans les 2-5 prochaines années pour le développement des capacités en évaluation dans la région/pays où vous intervenez ou travaillez? Veuillez sélectionner jusqu’à deux : Amélioration des capacités des décideurs des gouvernements nationaux à demander et à utiliser des données issues des évaluations rigoureuses pour la planification et le plaidoyer Amélioration des capacités des institutions nationales à concevoir, mettre en œuvre et faciliter l'utilisation des résultats des évaluations rigoureuses Amélioration des capacités des institutions nationales à demander et à utiliser des données provenant des évaluations basées sur l'équité et sensibles au genre pour la planification des interventions Amélioration de la qualité des évaluations en termes de rigueur des niveaux, de la qualité des plans et des données collectées, de pertinence pour répondre aux questions prioritaires, de disponibilité et diffusion à des fins d'utilisation accessibles aux décideurs Autre (à préciser) 25. Votre organisation prévoit-elle d'utiliser ou de collaborer avec le projet MEASURE Evaluation à l'avenir? Oui Non Ne sait pas N/A Si Oui ou Non, veuillez expliquer pourquoi : 26. Autre chose à ajouter? Veuillez fournir des commentaires ou des questions : MERCI POUR VOTRE PARTICIPATION Merci d'avoir pris le temps de répondre à ce questionnaire Si vous avez des questions, n’hésitez pas contacter au stopolansky@encompassworld.com ANNEX 7: COTE D’IVOIRE BRIEF COUNTRY REPORT This annex provides data for the MEASURE Evaluation Phase IV midterm performance evaluation report, offering a snapshot of the project’s work in Cote d’Ivoire and helping with triangulation of findings, conclusions, and recommendations across all data sources for the broader report. The information presented here is not meant to stand alone as an evaluation of MEASURE Evaluation’s work in Cote d’Ivoire. This brief country report follows the structure of the main midterm evaluation report, including presentation of findings by evaluation question. COUNTRY CONTEXT AND MEASURE EVALUATION FOCUS MEASURE Evaluation has worked in Cote d’Ivoire since 2004, providing support to strengthen the health information system (HIS) and HIV monitoring and evaluation (M&E) systems to increase the capacity of health staff and U.S. Government (USG) implementing partners and improve data collection, quality, and use at all levels of the health system. The project has also worked closely with the AIDS Control Program at the Ministry of Health. MEASURE Evaluation collaborates with Ivoirian government partners to strengthen collection, analysis, and use of routine health data. Phase IV addresses Results 1, 2, and 3, with the goal of contributing to the establishment of a fully functional M&E system and an integrated, high￾performing health management information system (HMIS). Capacity building has focused on harmonizing the HMIS using national and the United States President’s Emergency Plan for AIDS Relief (PEPFAR) monitoring, evaluation, and reporting indicators. During the 2014 Ebola outbreak, MEASURE Evaluation also supported strengthening of the national HIS and epidemic￾response systems through activities to enhance data collection and management and build capacity in data analysis, data use, and epidemiology: the activity has been closed. In the first 3 years of Phase IV, MEASURE Evaluation in Cote d’Ivoire received more than $6 million exclusively for HIV ($6,224,761; Exhibit 27), which included both mission and core funds. The MEASURE Evaluation office in Cote d’Ivoire is run by John Snow, Inc., and comprises six technical staff—a resident advisor, deputy director, and four information technology experts—and six support staff. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 100 Exhibit 27: MEASURE Evaluation Phase IV activities for years 1–3 Stakeholder Main Activities Funding Source by (Focus Area) Activity Health sector–wide National AIDS Control Program, National Malaria Strengthen national HMIS: $4,895,296 (Field HIV) Control Program, National Public Pharmacy, DHIS 2, eLMIS, SIGDEP 2, Directorate of Informatics and Health Information, and OVC database USAID Private Sector Health Program, and HMIS for health sector and OVC community sector (HMIS) National OVC Program, known as PNOEV (OVC) Strengthen national HMIS: $139,465 (Field HIV: DHIS 2, eLMIS, SIGDEP 2, OVC-specific) and OVC database National AIDS Control Program (HIV/AIDS) Prevention of Mother-to- $1,100,000 (Core HIV: Child Transmission/ Special initiative) Antiretroviral Therapy Integration Project Health area–specific National Public Hygiene Institute (Surveillance) Strengthen Ebola and other $500,000 (Core: Ebola) epidemic-prone disease surveillance systems METHODS AND SAMPLE Qualitative data collection was conducted using country-level interview guides adapted for target audiences (see Annex 5). The same guides were used for Cote d’Ivoire, Mali (Annex 8), and Nigeria (Annex 9), with the list of tools and resources adapted for each country before data collection commenced, in consultation with USAID and the MEASURE Evaluation office in the country. In Cote d’Ivoire, a two-person team collected data in two phases—October 16 through 20 and October 30 through November 3, 2017—through in-person and group interviews, review of MEASURE Evaluation and Government of Cote d’Ivoire documents. In addition, the team observed a project work-planning meeting with USAID and the MEASURE Evaluation project leadership team, which had not been pre-planned as part of data collection. The team gathered data from 33 individuals—four from USAID; two from the U.S. Centers for Disease Control and Prevention (CDC); six from MEASURE Evaluation; 11 from the Government of Cote d’Ivoire, Ministry of Health, national HIV/AIDS and malaria control programs, pharmaceuticals, and Ministry of Family, Gender, and Children HIV/AIDS program, and the National OVC Program (PNOEV); two from other international organizations and donors; and eight from USAID and CDC implementing partners. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 101 FINDINGS HOW WELL HAS MEASURE EVALUATION MET STAKEHOLDER NEEDS? MEASURE Evaluation Accomplishments and Remaining Areas: Stakeholders from across activities overwhelmingly indicated that the project has met their needs to a high degree, and recognized that MEASURE Evaluation has leveraged its staff’s expertise and deep knowledge of the health system. MEASURE Evaluation works in the [HMIS] system, for the system and with the system. —Government, Cote d’Ivoire The most-cited success was the 2014 rollout of District Health Information Software 2 (DHIS 2). Although this was a government-led national effort, all stakeholders recognized MEASURE Evaluation’s instrumental role in employing specific participatory engagement processes to guide national stakeholders and implement an approach that met everyone’s needs, including USAID’s. [DHIS 2] was a great experience; they really facilitated things: this is the type of collaboration that USAID is looking for: when things are decided they are done, without supervisions. —USAID DHIS 2 was rolled out nationwide, with full uptake by national stakeholders, due to leadership and technical guidance from MEASURE Evaluation staff, working in collaboration with the government to implement USAID’s vision. All stakeholders expressed the need for subnational support to ensure the system’s proper functioning and utility for decision making across the health system. MEASURE Evaluation’s style of collaboration has facilitated ownership and capacity development of key national actors and USG implementing partners. MEASURE Evaluation has provided the Ivoirian government with technical guidance, while advancing the USG’s goal of strengthening the HMIS, through engagement and leadership in technical working groups for DHIS 2, the Electronic Learning Information Management System (eLMIS), the national HIV patient data management system (known as SIGDEP 2), the orphans and vulnerable children (OVC) database, and indicators and data collection tools for HIV and prevention of mother-to-child transmission. The government particularly praised MEASURE Evaluation’s support to improve the OVC database. Although stakeholders cited challenges, including slow operationalization of the database, many of these issues were addressed during a technical meeting among users in October 2017. OVC database applications have come to fill a void. The implementation process required meetings with stakeholders to have consensus: meeting of explanation, orientation, advocacy. —Government, Cote d’Ivoire Although the majority of databases with which the project has engaged are operational, external stakeholders across the board pointed to the need to achieve full functionality and interoperability, also citing continuing issues with SIGDEP 2 (pharmacy, stock tracking, and management). Government and USAID- and CDC-supported HIV service delivery partners reported that site- and district-level data quality and use were still problematic (especially at the subnational level), and that data use at any level remains aspirational. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 102 MEASURE Evaluation Advantages and Disadvantages: There is no other bilateral M&E mechanism funded by USAID in Cote d’Ivoire, and all stakeholders clearly identified MEASURE Evaluation as providing M&E support for the health sector, especially for USAID and CDC implementing partners. The following advantages came out strongly in discussions with stakeholders:  MEASURE Evaluation is familiar with the HIS and has knowledge of all relevant documents.  MEASURE Evaluation staff’s diplomacy, communication skills, and technical competence— with vast capacity across domains and health-sector areas—have allowed the project to deliver and provide leadership in the HIS space, in collaboration with the government.  MEASURE Evaluation’s flexibility and adaptability to changing needs facilitates rapid responses to emergencies, such as the Ebola outbreak.  MEASURE Evaluation’s global vision and recognition by UNAIDS and the World Health Organization provide extra credibility and validity to the project’s work in Cote d’Ivoire. Stakeholders also cited some disadvantages, expressing a wish for improved operations for smoother and more efficient delivery:  MEASURE Evaluation needs to better transfer knowledge and skills to ensure the sustainability of achievements with the government.  There is a need for improved responsiveness and communication with USAID, and to meet needs of other selected stakeholders, such as CDC and USG implementing partners, even more effectively.  Increasing the frequency of communication about results and using visual technologies will help MEASURE Evaluation showcase its accomplishments (noted by USAID/Cote d’Ivoire). Despite being funded almost exclusively through PEPFAR, MEASURE Evaluation was able to extend its support, leveraging the DHIS 2 work, and is highly regarded for responding effectively to stakeholder needs with technical capacity, contextual knowledge, clear areas of work, and open communication and collaboration. Issues of Sustainability: Stakeholders reported that MEASURE Evaluation’s sustainability measures have been built into some efforts, although key challenges, related to retention of government staff and their overall limited ownership of the processes, impede the sustainability of project achievements in the HMIS arena. There is, however, evidence of certain milestones and processes that increase the likelihood that achievements will be sustained after Phase IV ends. Several stakeholders pointed to service delivery partners’ use of MEASURE Evaluation tools and guidelines as evidence of continued sustainability. Changes in indicator definitions and disaggregation are likely to require further investment as tools and resources are revised, as an important element of HMIS, but MEASURE Evaluation’s tool development work has facilitated the identification of high-priority indicators that are collected at the sites, as well as overall improvement in data quality. Different stakeholders, including the Ivoirian government and USG HIV service delivery implementing partners, are using MEASURE Evaluation–supported tools and resources for data February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 103 collection and data quality assessment (DQA), and have been empowered to conduct DQAs routinely and independently from MEASURE Evaluation. MEASURE Evaluation’s training-of-trainers approach and capacity-building efforts, including training materials related to DQA, DHIS 2, SIGDEP 2, and eLMIS, have enabled Ivoirian government and USG implementing partners to use them independently of the project. However, changes to PEPFAR reporting and revisions to tools are seen as essential services that the project can continue supporting. Stakeholders cited transferability of M&E skills from work under the PEPFAR portfolio, due to the project’s capacity-building efforts, as a personal and professional benefit. Stakeholders considered MEASURE Evaluation’s support for the deployment of DHIS 2, SIGDP 2, and the eLMIS as crucial for the viability of HMIS, given increased efficiency of data collection, transmission, and availability at all levels. Before [DHIS 2] we went regularly to the field to have access to the data … this is no longer the case. —Government, Cote d’Ivoire Stakeholders highlighted data use as a major gap. This gap is not necessarily fully within MEASURE Evaluation’s sphere of control, although the project has provided technical capacity and leadership in this area by reinforcing the value of data quality and creating demand for a culture of data use. Beyond questions of data, three key areas require more work to complete Phase IV in a manner that fully meets stakeholders’ needs and leverages existing processes and partnerships:  Improving the interoperability of DHIS 2, SIGDEP 2, the eLMIS, and the OVC database to facilitate data use and ensure effective use of human and financial resources  Building the epidemiological-surveillance capacity of regional and district-level staff of the Directorate of Informatics and Health Information  Overall human resource availability in the Government of Cote d’Ivoire. The government needs to recruit civil servant staff (computer scientist and statistician) for [DHIS 2]. —Government, Cote d’Ivoire To strengthen evidence-based decision making and ensure sustainability, USAID and national stakeholders have requested development of a framework for showcasing the data and information produced with support from MEASURE Evaluation. HOW HAS A HEALTH SECTOR–WIDE VERSUS A HEALTH AREA–SPECIFIC APPROACH PLAYED OUT IN COTE D’IVOIRE? USAID, the CDC, and other stakeholders reported the DHIS 2 rollout as a key facilitator for a health sector–wide approach. Although Phase IV has been funded almost exclusively through PEPFAR, MEASURE Evaluation’s DHIS 2 support has demonstrated the project’s flexibility and technical capacity to integrate its high-quality HIV/AIDS work into overall HIS strengthening. Much of this has been done informally, without a clear mandate or associated funding, due to the exclusive nature of PEPFAR funding for project activities under Phase IV. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 104 MEASURE Evaluation’s capacity to deliver on the health area–specific portfolio was exemplified during the 2014 Ebola outbreak, when the project supported the National Public Health Institute and the early-warning epidemiological surveillance system by strengthening community-level data collection related to epidemic surveillance and Ebola contact follow-up through SMS. National stakeholders praised MEASURE Evaluation’s assistance in building the capacity of those in charge of epidemic response at the health-district and subnational levels in data analysis, use of information, and investigating health phenomena, as well as the project’s specific contribution in adapting and delivering the “data use module” and integrating it in the training curriculum. Ebola was a desperate need: MEASURE was there … The coordination and the sense of the organization of MEASURE made this activity possible. The training lasted 1 month and was carried out in several phases: theoretical, case study, field investigation, protocol development, data analysis, restitution. —Government, Cote d’Ivoire Recognizing project capacities outside of HIV, and in light of recent approval of the President’s Malaria Initiative (PMI) plan for Cote d’Ivoire,15 internal and external stakeholders specifically requested MEASURE Evaluation’s entry into that space. MEASURE Evaluation can provide supervision at all levels, train M&E experts at all levels, leverage their malaria-related expertise from other countries, and provide program support for studies …, propose methodologies, and streamline processes [during PMI rollout and implementation]. —Government, Cote d’Ivoire The project’s focus can enhance non-PEPFAR [and] until now has integrated [non-]HIV more informally. SIGDEP 2 used malaria [and] TB indicators, and MEASURE was able to help with DHIS 2 as a whole. HIV HIS experts have worked for other systems in health; DHIS 2 is multi￾health-specific. —CDC MEASURE Evaluation’s OVC work, though still funded by PEPFAR, has taken the project beyond HIV to engage with other sectors, such as with the Ministry of Education, and exemplifies USG￾facilitated collaboration at all levels (clinical, social, and community). ARE THE TOOLS THAT MEASURE EVALUATION HAS SUPPORTED USEFUL? All stakeholders indicated a strong recognition of high-quality design and resulting usefulness of the tools and resources that MEASURE Evaluation has developed or adapted in Cote d’Ivoire (Exhibit 28). In addition to support, technical assistance, and mentorship for DHIS 2, SIGDEP 2, the eLMIS, and the OVC database, the following tools were most often reported across all stakeholders:  Cote d’Ivoire HIV Indicators Dictionary (Dictionnaire des indicateurs VIH)  Cote d’Ivoire National Health Indicator Reference Sheets (Dictionnaire des indicateurs non santé)  Training manuals for DHIS 2 and SIGDEP 2. The data quality review and routine DQA tools were also cited as useful and reportedly adopted by CDC partners and other organizations supporting HIV service delivery at the district level. 15 See https://www.pmi.gov/where-we-work/cote-divoire. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 105 Exhibit 28: Most useful tools, as cited by stakeholders MEASURE Evaluation (n=4) Partners (n=5) Government/Ministry of Health (n=11) Registre de Consultation OVC Database/Manual SIGDEP 2/Manual eLMIS/Manual DHIS 2/User Guide Health Indicators HIV Indicators Dictionary 0% 20% 40% 60% 80% 100% A number of MEASURE Evaluation tools and resources were satisfactory for stakeholders working outside of HIV. Stakeholders involved in the Ebola response specifically referenced the Cote d’Ivoire Adapted Data Use/Applied Epidemiology Curriculum (Module de utilisation des données/Formation surveillance épidémiologique). Two tools (Analyse situationnelle du système d’information sanitaire du Mali and Suivi et évaluation des programmes de lutte contre le paludisme – Cours en ligne) were in the top three of those most downloaded in Cote d’Ivoire, further building evidence to address malaria￾related needs of stakeholders. In the beginning, the DHIS was not well understood by national actors and partners. MEASURE Evaluation helped us develop the DHIS 2 manual and train the [users]. Through the manual, the application is used at the district level. —Government, Cote d’Ivoire We [National Malaria Control Program] always reach out to MEASURE to discuss tools, integrate. We ask them to share success stories from other countries as well. —Government, Cote d’Ivoire Many stakeholders cited data use as the key gap in the HMIS discussions; however, a few examples are available to illustrate MEASURE Evaluation’s impact on decision making in the Ivoirian government. The existence of DHIS 2 and other applications (SIGDEP 2, eSIGL) made it possible to have data available for reports. —Government, Cote d’Ivoire [The] user manual for the OVC database … allowed [users] to understand the database and to use it. —Government, Cote d’Ivoire When prompted about the extent to which MEASURE Evaluation has addressed gender considerations in its work, all stakeholders eagerly cited the sex-disaggregation requirement in the February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 106 T tools and databases, which the project facilitated. Nonetheless, consistent with the overall gap reported in data analysis and use, almost no examples of use of sex-disaggregated data were cited. Stakeholders from PNOEV highlighted the value and use of sex-disaggregated data in the context of the global Determined, Resilient, Empowered, AIDS-free, Mentored and Safe (DREAMS) initiative,16 and PNOEV has been anticipating MEASURE Evaluation’s assistance in developing indicators through DREAMS. PNOEV is aware that DREAMS requires the use of sex-disaggregated data for programmatic decision making, for which the project’s assistance would be instrumental. Despite the multitude and technical strength of the tools and resources MEASURE Evaluation has supported, stakeholders brought up issues that have inhibited full use of these tools, including the small size of the team, thinly stretched human resources, the dominance of PEPFAR funding, and frequent changes in PEPFAR requirements. Other obstacles noted included the availability of paper￾based guidance documents for dissemination, internet issues, and data quality and use at the subnational level. However, internal and external stakeholders were almost unanimous in their perception that these issues were outside of the project’s mandate and geographic scope in Phase IV. Many tools have been developed, now the issue is at system level: use and decision-making. It’s not project role to actually deploy tools. … [the Ministry of Health] needs to embrace what is offered to them and institutionalize [the tools after they] are developed and printed. —International implementing partner here is an issue of leadership over data: accessibility, data quality, use. —CDC CONCLUSIONS MEASURE Evaluation in Cote d’Ivoire has succeeded in meeting HIS-related needs of external stakeholders and largely those of internal stakeholders. Although the USG has not prioritized evaluation, stakeholders recognize the project’s ability to meet evaluation-related needs. A combination of facilitating factors has helped the project work effectively with national partners and USG stakeholders, such as the continuity of project leadership, a long history in Cote d’Ivoire, and the prioritization of DHIS 2 the government and USAID, which helped MEASURE Evaluation showcase its ability to work across the health portfolio. Some inhibiting factors have prevented MEASURE Evaluation from fully realizing its potential in Phase IV, such as its primarily PEPFAR funding, the small project team, recent retention issues due to a budget reduction,17 and limited government leadership and ownership of HIS-related processes. Stakeholders made the following recommendations for improvement before the end of the project: using visualizations when presenting data to ministers; improving the interoperability of the different databases; developing a strategy for development and dissemination of online tools; and overall articulation of links between project activities and the sustainability of results, with clarity of roles. 16 See https://www.usaid.gov/what-we-do/global-health/hiv-and-aids/technical-areas/dreams. 17 When the budget was reduced by nearly $570,000 from Year 2 ($1,357,523) to Year 3 ($787,749), the decision was made to reduce the number of staff and continue providing technical assistance to ministries for additional software development and implementation. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 107 Despite some shortcomings, MEASURE Evaluation has a unique role in supporting stronger HIS in Cote d’Ivoire. The project is well-positioned to support the Ivoirian government in addressing emerging needs, such as Ebola, and to provide technical assistance and expertise at the subnational and district levels to improve the quality of routine data before it reaches Abidjan, for local data use, as desired by all stakeholders. The October 2017 work-planning meeting between the project and USAID during this evaluation was a testament to MEASURE Evaluation’s recognition of stakeholders’ needs and its work to make mid-course corrections to meet those needs. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 108 ANNEX 8: MALI BRIEF COUNTRY REPORT This annex provides data for the MEASURE Evaluation Phase IV midterm performance evaluation report, offering a snapshot of the project’s work in Mali and helping with triangulation of findings, conclusions, and recommendations across all data sources for the broader report. The information presented here is not meant to stand alone as an evaluation of MEASURE Evaluation’s work in Mali. This brief country report follows the structure of the main midterm evaluation report, including presentation of findings by evaluation question. COUNTRY CONTEXT AND MEASURE EVALUATION FOCUS In Mali, MEASURE Evaluation Exhibit 29: MEASURE Phase IV activities has provided assistance for Stakeholder Main Activities Funding Source by monitoring and evaluation (M&E) (Focus Area) Activity of health programs since 2010. In Health sector–wide Phase III, MEASURE supported Center for Strengthen the $4,610,992 (MCH) the National Malaria Control Planning and National HMIS $450,000 (Malaria) Program to pilot information Statistics, $400,000 (FP) systems, supported the M&E needs National Health $197,000 (Nutrition) of the USAID/Mali Health Office, Directorate $119,981 (HIV) (HMIS) and conducted a full assessment of USAID M&E support to $120,199 (MCH) the national health management the Health Office $21,381 (Nutrition) and community health information $10,000 (WSS) systems (PRISM assessment). Health area–specific In Phase IV, MEASURE National Health Strengthen Ebola $900,000 (Ebola) Evaluation’s focus has shifted away Directorate virus disease $900,000 (Global /Disease surveillance Health Security from direct M&E support to the Surveillance Strengthen Agenda) mission and toward epidemiological implementation of District Health surveillance information Information Software 2 (DHIS 2), system while continuing support to National Malaria Strengthen $2,387,279 (Malaria) malaria and epidemiological Control Program malaria M&E surveillance. Using the results of National Health Evaluation of $193,900 (MCH) the Performance of Routine Directorate/ family planning $190,000 (FP/RH) Information System Management Reproductive campaign $3,000 (Nutrition) Health February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 109 (PRISM) assessment,18 MEASURE helped pave the way for the Government of Mali to adopt DHIS 2 as its health management information systems (HMIS) platform, and worked with USAID and many other partners to roll out DHIS 2 nationwide. MEASURE Evaluation has received $10.5 million for Phase IV (including Year 4), with funds distributed evenly over the 4 years, though with varied funding sources (see Exhibit 29, previous page): 47 percent from Maternal, Neonatal, and Child Health; 27 percent from the President’s Malaria Initiative (PMI); and less than 10 percent each from Global Health Security Agenda, Ebola, Family Planning (FP), Nutrition, the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR), and Water Supply and Sanitation (WSS). Overall, HMIS support represents 55 percent of funding efforts, followed by malaria program M&E support (23 percent), and disease surveillance (17 percent). MEASURE Evaluation has a team of nine technical staff and a chief of party. METHODS AND SAMPLE Qualitative data collection was conducted using country-level interview guides adapted for target audiences (see Annex 5). The same guides were used for Cote d’Ivoire (Annex 7), Mali, and Nigeria (Annex 9), with the list of tools and resources adapted for each country before data collection commenced, in consultation with USAID and the MEASURE Evaluation office in the country. In Mali, a two-person EnCompass evaluation team collected data between October 9 and October 20, 2017, through in-person and group qualitative interviews, reviews of MEASURE and Government of Mali documents, observation of a DHIS 2 steering committee meeting, and a visit to a health center. The evaluation team gathered data from 56 individuals from USAID (6), MEASURE (10), Government of Mali (27): Ministry of Health (MOH), Statistics, National Health Directorate, National Malaria Control Program, National HIV/AIDS Program, and Pharmaceuticals; and other international and USAID implementing partners (13). FINDINGS HOW WELL HAS MEASURE EVALUATION MET STAKEHOLDER NEEDS? MEASURE Evaluation Accomplishments and Remaining Areas: MEASURE Evaluation has supported an impressive rollout of the DHIS 2, officially launched by the MOH in June 2017, that has resulted in almost complete coverage, with equipment and training for all health facilities, from hospitals to health centers. The decision to use DHIS 2 as the platform for Mali’s routine health information system (HIS) was not easily achieved, but MEASURE Evaluation’s PRISM evaluation paved the way for this decision by providing a strong, clear assessment of the system’s strengths and weaknesses, followed by ongoing leadership from USAID, MEASURE Evaluation, and key officials from the Ministry of Health and the Planning and Statistics Unit.19 18 Ministry of Health and Public Hygiene. 2014. Evaluation du Système Local d’Information Sanitaire (SLIS) avec les Outils PRISM. USAID: MEASURE Evaluation TR-14-104FR. Available at: https://www.measureevaluation.org/resources/publications/tr-14- 104-fr 19 The Planning and Statistics Unit serves three ministries: health, social development, and family protection. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 110 MEASURE Evaluation’s great success over the past 3 years was the rollout of DHIS 2 at the national, regional, district level. This evaluation [with the PRISM approach] was exceptional, as it revealed the shortcomings of the health information system as a whole … MEASURE disseminated the results well, and DHIS 2 was one of the recommendations. —International implementing partner National-level ministry staff, USAID, and other implementing partners that support portions of the rollout all heralded MEASURE Evaluation’s support for its technical leadership, capacity strengthening, financial support for DHIS 2 management structures (steering and technical committees), and stakeholder mobilization. The start was not easy, but over time, we were able to convince others and speak with one voice. The launch demonstrated the true collaboration with MEASURE. They do not make the decisions, they play their role of technical assistance, and all the technical problems are debated. … We had difficulties with other actors, but not with MEASURE. —Malian government Although DHIS 2 is not the only focus of MEASURE Evaluation activities, it is the centralizing one, as the project’s work in specific areas, such as malaria and disease surveillance, is incorporated in the DHIS 2 platform. The project has provided longstanding support to the National Malaria Control Program, much of which laid the groundwork for strengthening the routine health information system (RHIS) overall. MEASURE Evaluation supported USAID/Mali’s M&E needs in the first year of Phase IV, supporting the Health Office to develop the Health Development Objective Performance Management Plan and Performance Indicator Reference Sheets and work with other implementing partners to understand and collect data for these indicators. In 2015, however, the mission awarded a contract for an M&E platform to provide these kinds of services mission-wide, so the project no longer provides these services. MEASURE Evaluation Advantages and Disadvantages: Clearly, the rapid rollout of DHIS 2 is not the success of any one actor, but all stakeholder groups consistently cited the project’s significant contribution to this achievement. Based on a partner mapping, USAID worked with its implementing partners and with other partners (Global Fund and UNICEF) to cover all regions in the country at the operational level, while the project provided technical and financial support at the national level. MEASURE Evaluation worked with USAID and the MOH to convince all actors to work together. The project was able to play its important role for several reasons—the way it chose to work with all stakeholders, its excellent technical staff, and the flexibility in problem-solving it demonstrated with government stakeholders. One of the lessons is to put the government “in front.” MEASURE said, “You are the owner of this system; we can help you based on our knowledge of other countries if you want.” MEASURE and USAID remained behind the scene. —International implementing partner [MEASURE Evaluation staff] are very competent, positively aggressive, and cordial. —Malian government February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 111 MEASURE is the lead—it’s their mandate but they also have the skills needed. Even the MOH is asking for MEASURE’s opinion. —USAID implementing partner Issues of Sustainability: Sustainability rests on multiple factors, including political and social issues, resources, and systems. At this moment, there is clearly political will for DHIS 2, within government and from partners. With MEASURE support, systems have been put in place to manage the system, such as a steering committee and a technical committee with broad participation. Putting this system in place was not easy. Now that the system is in place … DNS and CPS have taken ownership of the system, and they are able to monitor the system and respond to MOH requests for information. —USAID implementing partner Key stakeholders perceive that MEASURE Evaluation has carried out its work in ways that facilitate sustainability—encouraging MOH ownership and decision making, building capacity as DHIS 2 and other technical tasks are implemented, providing other opportunities for training, training ministry and implementing partner staff as trainers and supervisors for DHIS 2, and supporting the management structures (steering and technical committees). All stakeholder groups had concerns about sustainability of DHIS 2 functioning, primarily related to resource needs (financial, human, and infrastructure), and the need to fund recurrent costs of the RHIS: Internet access, regular replacement of equipment and materials, ongoing training for additional staff at the operational level, and financing steering and technical committee meetings. MEASURE Evaluation is working with the MOH to develop a more robust sustainability plan, including resource mobilization and inclusion of ongoing RMIS support in the Health Sector Development Program, which is being developed. HOW HAS A HEALTH SECTOR–WIDE VERSUS A HEALTH AREA–SPECIFIC APPROACH PLAYED OUT IN MALI? The perception across all stakeholder groups in Mali was that MEASURE Evaluation’s work in specific health areas (malaria and disease surveillance) was a benefit for health sector–wide efforts with DHIS 2, and vice versa. The mission managed the varied funding streams and ensured appropriate reporting capacity. Malaria and disease surveillance staff at the MOH felt their work helped open the doors to a common, stronger platform for RHIS, due to their needs for more frequent data compilation and reporting, while benefiting from a real-time system that gives them easier access to data. MEASURE is here in Mali for the fight against malaria with PMI funds. They support the information system through the malaria program. Improving the fight for malaria requires the information system for [the whole health sector]. —Malian government DHIS 2 takes into account all health areas. Before DHIS 2, MEASURE was here just for malaria. Now, it DHIS 2 is the foundation for all USAID health programs The advantage of a sectoral approach is that it encompasses everything and everyone can have its share. Training on a single disease can have effects for other diseases. —USAID February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 112 Exhibit 30: Most useful tools, as cited by stakeholders MEASURE (n=4) External (n=5) Government (n=11) Mali DHIS 2 User's Guide (includes Ebola and Malaria) Mali RHIS Data Management Supervision Tools (including RDQA) Mali RHIS Data Standards and Procedures Manual Mali Action Plan for Strengthening Data Availability, Quality and Use of Epidemic Diseases Mali Data Management Guidance for Ebola Surveillance Mali Data Analysis Curriculum Mali Validated Plan to Address Data Use Barriers Mali Updated List of Indicators for Monitoring Epidemic Diseases 0% 20% 40% 60% 80% 100% ARE THE TOOLS THAT MEASURE EVALUATION HAS SUPPORTED USEFUL? MEASURE supported the development/adaptation and dissemination of a number of tools that stakeholders felt were the keys to improving data collection, capture, and use (see Exhibit 30) for RHIS and specific programs. The Mali adaptations of the DHIS 2 User Guide and supervision tools, especially the routine data quality assessment (RDQA) tools, were the most appreciated and used. Key factors in uptake included a real need for the tool, joint development and MOH ownership of the tool, and financing for printing and training on how to use the tool. MEASURE brought together stakeholders during the workshops to see the tools they are using and those used by other countries. MEASURE projected the tools on the screen to show what could be adapted to the Malian context. MEASURE did all this with the Planning and Statistics Unit … who felt the lack of the right tools, but did not have the resources to develop them. —Donor RDQA, user guide, RHIS standards and procedures manual … These three allow one to solve the problems in [DHIS 2] implementation. The RDQA is very clear and easy to use. We participated in the design, validation, and financing health worker training on these tools in our intervention zone. —USAID implementing partner For the future, most stakeholders spoke of the need for interoperability of DHIS 2 and other key information systems. Other key areas for tool development include electronic patient registers (currently being tested for maternal health services) and electronic community-based data systems. Gender: In discussing gender integration in RMIS, USAID and MEASURE both spoke to sex disaggregation and participation in training events. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 113 CONCLUSIONS MEASURE Evaluation has been exceptionally successful in Mali, due to a confluence of factors. The 2013 PRISM assessment paved the way for uptake of DHIS 2, along with the skillful navigation of current MEASURE staff working in concert with USAID. MEASURE Evaluation’s collaboration style has facilitated ownership and capacity development of key actors. Nationwide rollout of DHIS 2 is almost complete,20 although significant work remains to achieve full functionality: quality and use of data are still problematic, especially at the operational level. Important next steps include addressing the technical and political aspects of interoperability for logistics systems (in process), Expanded Program of Immunization, HIV, community, and human resource information systems. An evaluation of the HMIS, planned for fiscal year 2018, will be an important next step in demonstrating success to date, identifying gaps, and developing consensus among the many partners on the way forward. It will also be critical to think hard about a real plan for sustainability. 20 A few health centers in one extremely unstable northern region remain to be covered with training and equipment. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 114 ANNEX 9: NIGERIA BRIEF COUNTRY REPORT This annex provides data for the MEASURE Evaluation Phase IV midterm performance evaluation report, offering a snapshot of the project’s work in Nigeria and helping with triangulation of findings, conclusions, and recommendations across all data sources for the broader report. The information presented here is not meant to stand alone as an evaluation of MEASURE Evaluation’s work in Nigeria. This brief country report follows the structure of the main midterm evaluation report, including presentation of findings by evaluation question. COUNTRY CONTEXT AND MEASURE EVALUATION FOCUS In Nigeria, MEASURE Evaluation has supported monitoring and evaluation (M&E) and health information systems (HIS) for more than a decade, with activities focusing on malaria, orphans and vulnerable children (OVC), HIV/AIDS, tuberculosis (TB), and disease surveillance. When Phase III ended in 2014, funding ceased from the USAID mission and the Nigeria office closed, although MEASURE Evaluation continued to conduct core-funded in-country activities. Nigeria adopted District Health Information Software 2 (DHIS 2) for routine management of health information years ago, but the process for updating and managing the health facility database was never optimized. In early 2016, the USAID mission requested MEASURE Evaluation support for a portfolio focused on strengthening the national HIS—working with the Department of Planning, Research and Statistics (DPRS) in the Federal Ministry of Health to develop a process for indicator harmonization and institutionalization of a Master Facility List. With an 18-month work plan and limited assurance of funds beyond this one activity, MEASURE Evaluation chose to staff the activity out of its existing Palladium office. However, the need for technical leadership led to a second USAID request, in early 2017, to hire an in-country chief of party. This was done by June 2017, and a new office opened shortly thereafter. As of October 2017, MEASURE Evaluation had a five-person team in Nigeria, with three full-time staff, including the chief of party, and two part￾time staff. Throughout Phase IV, MEASURE Evaluation has received $7.9 million21 for 11 activities that span three of the four result areas across varied funding sources: 65 percent from the Office of HIV/AIDS, 16 percent from the President’s Malaria Initiative (PMI), 6 percent each from Maternal and Child Health (MCH) and Family Planning and Reproductive Health (FP/RH), 4 percent from TB, and 2 percent from Nutrition. The activities with the greatest investment in Phase IV were activities under the health management information system (HMIS) portfolio and the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR) OVC Outcome Monitoring Survey (see Exhibit 31). 21 No funds have been received for Year 4, due to the significant pipeline. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 115 Exhibit 31: MEASURE Evaluation Phase IV activities Stakeholder Main Activities Funding Source by Activity (Focus Area) Health sector–wide Federal Ministry of  Harmonize National Health Indicators $1,700,000 (HIV) Health/DPRS (HMIS)  Develop Health Facility Registry and $170,500 (MCH) Master Facility List $250,000(FP/RH)  Develop standard operating procedures for national routine program data validations and data quality assessments Health area–specific Federal Ministry of Women  PEPFAR OVC Outcome Monitoring $1,502,400 (HIV) Affairs and Social Survey $79,500 (MCH) Development (OVC)  Strengthen Nigeria OVC Management $617,000 (HIV) Information System National Malaria Elimination  National Malaria Surveillance Workshop $320,000 (Malaria) Program (Malaria)  Secondary Analysis of Nigeria Malaria Indicator Survey $250,000 (Malaria)  Malaria Intervention Assessment (closed) $195,000 (HIV) $ 500,000 (Malaria) $ 198,000 (Nutrition) Nigerian Center for Disease  Strengthening Surveillance Capacity $300,000 (TB) Control (TB) National TB and Leprosy  Assessment of the Quality of TB Care $250,000 (Core)22 Control Program (TB) METHODS AND SAMPLE Qualitative data collection was conducted using country-level interview guides adapted for target audiences (see Annex 5). The same guides were used for Cote d’Ivoire (Annex 7), Mali (Annex 8), and Nigeria, with the list of tools and resources adapted for each country before data collection commenced, in consultation with USAID and the MEASURE Evaluation office in the country. In Nigeria, a two-person EnCompass evaluation team collected data between October 16 and October 28, 2017, through in-person and group interviews and reviews of MEASURE Evaluation and Government of Nigeria documents. The team gathered data from 29 individuals from USAID, MEASURE Evaluation, and the government—Federal Ministry of Health, Federal Ministry of Women Affairs and Social Development (FMWASD), Nigerian Center for Disease Control (NCDC), and the National Malaria Elimination Program (NMEP)—as well as other international agencies and USAID implementing partners (see Exhibit 32). 22 Core funding include Bureau-wide, HIV, FP/RH, and TB. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 116 Exhibit 32: Data collection breakdown by stakeholder group Measure Internal: External: Nigerian External: Global External EVALUATION USAID/USG Government Partner  5 interviews  7 interviews  1 group interview (3  1 group interview (3 respondents) respondents) o NMEP o SMILE, Catholic Relief  8 interviews with Services government partners:  3 interviews with other o Federal Ministry of external partners: Health/DPRS (4) o FHI 360 o FMWASD (1) o Academy for Health o NMEP (1) Development (AHEAD) o NCDC (1) o Global Fund Note: Internal stakeholders are the USAID mission, country-specific technical working groups, other Bureaus. External stakeholders are country ministries and government bodies, in-country implementing partners, and populations who are the ultimate beneficiaries of MEASURE Evaluation’s work. FINDINGS These findings cover all MEASURE Evaluation–supported work streams in Phase IV. As Exhibit 31 illustrates, funding and level of effort committed to Phase IV activities varied substantially. Thus, more detail and explanation are committed to certain activities, such as the HMIS portfolio and PEPFAR OVC activity, reflecting the level of investment and to learn from successes and challenges across projects. HOW WELL HAS MEASURE EVALUATION MET STAKEHOLDER NEEDS IN NIGERIA? The team investigated the extent to which MEASURE Evaluation Phase IV has met the needs of internal and external stakeholders across all activities in Nigeria. The findings vary significantly from activity to activity. Internal and external stakeholders working on health area–specific activities in OVC, malaria, and TB concurred that MEASURE Evaluation had met their needs for technical expertise and leadership in strengthening health information and capacity in M&E. However, this was not true for HMIS: All internal stakeholders and the majority of external stakeholders working on the HMIS activity reported that MEASURE Evaluation had not met their needs in completing the HMIS portfolio efficiently and effectively. HMIS Strengthening: Internal and external stakeholders reported that MEASURE Evaluation had mostly been unable to meet their needs related to the Health Facility Registry and Master Facility List activity. This was a frustrating to both groups, as this was not only delaying the DPRS; other government officials and external donors were relying on this activity to move forward. The Master Facility List. MEASURE was tasked to do that work and they are the cog in the wheel of the work that [the U.S. Government] wants to do, that Global Fund wants to do, that other development partners want to do. Any time we have a meeting … the first question is, “Where is MEASURE on the Master Facility List?” —USAID/U.S. Government, Nigeria February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 117 The last meeting we had with DPRS, the same question was asked and the response was still the same: “We are working on it.” So, we are waiting for that to be finalized. —Government, Nigeria Reasons given for why this activity was unable to meet stakeholders’ needs differed slightly, with USAID staff and Nigerian Government stakeholders identifying different needs and therefore different challenges. USAID staff reported that MEASURE Evaluation was unable to meet their needs for a project that could deliver high-quality work, on time and within budget, noting that there was lack of clarity on how the team was planning to accomplish its ambitious work plan, and that work often fell short of their quality standards and was delayed. USAID staff also reported challenges in communication—internally, in communicating their work plan to the mission, and externally, in relying on the mission to engage with other implementing partners as part of finalizing the Master Facility List. Nigerian government staff discussed challenges, but at the same time highlighted some of MEASURE Evaluation’s key contributions in convening key stakeholders, providing useful technical assistance, and continuing to build M&E capacity within the Federal Ministry of Health/DPRS. The most-reported challenges included lacking a clear understanding of the project’s work plan, delays in implementation, and breakdowns in communication. However, these stakeholders attributed the challenges to both MEASURE Evaluation and USAID’s communication styles. The government reported difficulty in getting a clear and consistent message from USAID and MEASURE Evaluation colleagues and continuous changes in the project’s scope of work, which made it extremely challenging for them to plan. The needs of the organization have not been fully met. The [Master Facility List], we have not finished the process; the data quality review, we just left it hanging. By all means I need to see their work plan … so I cannot definitely answer your question well unless I see the work plan and say, “Oh, they have been able to help us with this and that.” —Government, Nigeria For Nigerian government stakeholders, USAID’s shifting of priorities for MEASURE Evaluation’s work was unclear and complicated their own internal planning related to indicator harmonization and the Master Facility List. They reported, the shifting scope created difficulties in completing these activities, planning for future work, and trusting MEASURE Evaluation’s capacity to finish work it had started. We planned a lot of things based on an assurance, and now you are saying, “No, we want to go this way and not that way.” I found that very, very unacceptable for government.” —Government, Nigeria Despite these challenges, all Nigerian government stakeholders working on the HMIS activity also highlighted areas in which MEASURE Evaluation had met their needs for technical assistance and capacity building in M&E and support in convening key stakeholders. Government staff highlighted MEASURE Evaluation’s leadership in inaugurating the Health Data Governance Council—a national HIS governing body comprising stakeholders from across the government and supporting the DPRS in coordinating the HMIS technical working group. Government officials also pointed to MEASURE Evaluation’s approach to integrating capacity building throughout its support, both February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 118 formally, through training and workshops, and informally, in daily interactions with their government counterparts. I can tell you specifically that when it comes to the [Master Facility List] and [Master Facility Registry], which I think is super important, we have gained a lot as a division and as a country in terms of capacity building and helping all of us understand what it should be about and getting knowledge from the states… that is also capacity building for us here. —Government, Nigeria Government stakeholders further highlighted the fact that MEASURE Evaluation’s HMIS work, including indicator harmonization, often required long, tedious processes that provided few tangible results. The MEASURE team we had for the indicators did a lot of work that would be hard for somebody just standing and looking from the outside to appreciate. It was back-breaking and painstaking. —Government, Nigeria Other Health-Specific Areas: Internal and external stakeholders working in the health area–specific portfolio were overwhelmingly positive about MEASURE Evaluation’s ability to meet their needs. Government, USAID staff, and implementing partners all highlighted its work with OVC as relevant and useful to the needs of FMWASD and implementing partners. Almost all internal and external stakeholders highlighted MEASURE Evaluation’s successful collaboration with FHI 360 to support FMWASD in revising OVC data collection tools as part of the process to harmonize reporting in the National OVC Management Information System. These same partners reported that the project was successful in completing system upgrades in early 2017, and highlighted that government agencies and implementing partners are now actively using it. At a point, we were stopped until they came on board again, and this led to the resuscitation of the electronic data base. And they also helped us to review the old tools that stakeholders were complaining about … They have been doing that and training stakeholders on how to use these tools and the database and it has been so wonderful. —Government, Nigeria Government staff and implementing partners highlighted MEASURE Evaluation’s leadership and convening role in bringing together government and implementing partners in the OVC Technical Working Group. Inside and outside of the technical working groups, they reported, MEASURE Evaluation engaged partners to ensure their participation in development and dissemination of tools. USAID and external partners highlighted the project’s important work in conducting the PEPFAR OVC Outcome Monitoring Survey, which provided a picture of the status of in-country OVC projects and provided a baseline for USAID assessment of partners’ work in the future. In terms of baseline survey of OVC, we got a good report. We hadn’t done that in a long while and needed a sense of what was happening with implementing partners, and it gave us a view. Also, it was useful in getting partners on their toes to agree to address gaps, as they know we will do endline survey at the end. —USAID/U.S. Government, Nigeria Internal and external stakeholders working with MEASURE Evaluation on malaria activities reported that the project was able to meet their needs for technical leadership, capacity building, and commitment to government ownership. As an example of the high-quality technical support that MEASURE Evaluation has provided throughout Phase IV, government staff highlighted a recent malaria M&E and surveillance workshop, which was built on the 2015 Malaria Indicator Assessment February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 119 (funded by the President’s Malaria Initiative/Nigeria and conducted with MEASURE Evaluation support). The project collaborated with NMEP to use the results from the 2015 assessment to design a Nigeria-specific workshop that addressed gaps in M&E capacity and incorporated gender for the first time. This type of training had previously been done on a regional basis, allowing only a few individuals to attend, and government partners reported how important it was for more M&E staff to be trained. The quality of [MEASURE Evaluation] support has been fantastic, especially the technical support. This last [workshop], everybody was happy about it. People were meant to do projects for their states and right there they developed annual operations plan on surveillance, monitoring, and evaluation, so technically, they have been very good. —Government, Nigeria Although MEASURE Evaluation’s work in TB is relatively new and small, government partners working on TB activities reported a high degree of satisfaction with the work thus far. NCDC staff reported that having a senior specialist seconded to the NCDC was already proving to be highly valuable in bringing new ideas for how to improve surveillance capacity, while ensuring links to broader HIS strengthening in Nigeria. Comparative Advantages/Disadvantages and Facilitating/Inhibiting Factors: Facilitating and inhibiting factors reported across stakeholders also varied by activity (HMIS versus health area– specific). It was evident that where MEASURE Evaluation was successful, comparative advantages were present and being exploited, whereas where the project was challenged, these factors were often missing. Stakeholders also reported facilitating and inhibiting factors outside the project’s sphere of influence. Some comparative advantages were consistent across the majority of internal stakeholders, and almost all external stakeholders, such as MEASURE Evaluation’s long-term global and country-level reputation for M&E expertise, its knowledge of the local context, and its dedicated technical staff. MEASURE Evaluation is seen as a technical warehouse for monitoring and evaluation, so that [the] name MEASURE Evaluation also sells the work that we do. —Government, Nigeria Internal and external stakeholders working on health area–specific activities also highlighted facilitating factors in MEASURE Evaluation’s success, such as having technical experts involved consistently throughout the project, often engaging with government counterparts on a daily basis. MEASURE staff are hands on. They go and sit in government offices and have meetings; a kind of in-your-face approach. Sometimes things are not very clear when USAID says, “This is what we want” … But [MEASURE Evaluation’s] approach and being involved is very helpful. —USAID/U.S. Government, Nigeria [T]he key facilitating factor has been in the area of staffing … MEASURE has made concerted efforts to identify as key staff persons who have been part of the programming efforts over the years and who understand the operating environment, understand the culture or the established working relationship with government, [implementing partners], with the donors and this made working with them quite easy … that is a major factor for me that contributed to the success. —External stakeholder, Nigeria February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 120 Internal and external stakeholders working in health sector–specific activities highlighted other facilitating factors with regard to how MEASURE Evaluation was organized and managed—some within its sphere of influence and some outside its sphere. Government and USAID staff highlighted a clear work plan and consistent scope of work that is clearly communicated, which was noted as contrary to what happened under the HMIS activity. One way [MEASURE is] successful is that they had clear timelines and bi-weekly or weekly calls to check in on progress and that has been exceptional—when I couldn’t join other partners were on the call and I would get the meeting notes. —USAID/U.S. Government, Nigeria MEASURE Evaluation and external stakeholders highlighted facilitating factors outside of their sphere of influence, such as having government staff who were open and committed to the cause and having USAID staff who supported open communication with MEASURE Evaluation and government colleagues. The aspect of the stability in terms of the key staff in government. Quite a number of those staff have been around for a long time and they have institutional memory that you could easily leverage on and I don’t think that should be taken for granted as a factor that may have contributed to their success. —External stakeholder, Nigeria External stakeholders, Nigerian government stakeholders, and USAID stakeholders all reported that MEASURE Evaluation’s main comparative disadvantage in Nigeria was its in-country staffing challenges. With multiple staffing changes, including the lack of a chief of party until June 2017, there were limited staff who could consistently meet with the government, communicate with USAID, and ensure the timeliness of the project’s work. External stakeholders reported that even for health area–specific activities, the limited staffing structure inhibited MEASURE Evaluation’s ability to carry out its large Phase IV scope of work. [T]he responsibilities are quite enormous … compared with the previous phase that had a more robust staffing mechanism in place and in a sense enabled that particular phase to have a greater penetration with regard to the ability to work with government and the implementing partners. —External stakeholder, Nigeria USAID, MEASURE Evaluation, and Nigerian government stakeholders reported that staffing was a challenge not only for the project, but also within USAID and the government, as Exhibit 33 illustrates. The staffing challenges were caused by different reasons but all resulted in significant miscommunication and perceived lack of transparency across partners. MEASURE should just be open and transparent. It is like having ten things and they show us two and when we ask for the remaining eight, they tell us their donor said this or that so they should put us in the true picture of what they are doing. —Government, Nigeria When I meet the funders and they said we aren’t focusing here anymore and we are changing, that means it is also a problem. Consistency should be the keyword. —Government, Nigeria Government, external stakeholders and MEASURE Evaluation staff also noted another factor outside of the project’s sphere of influence: The Nigerian government is a decentralized system with limited capacity for M&E, which is a challenge for implementing such a large project in such a short time. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 121 Exhibit 33: Staffing challenges under the HMIS portfolio  Two key MEASURE Evaluation staff left shortly after the activity was underway. MEASURE tried to fill this gap by having support from headquarters, feeling that it could not staff a full office for an 18-month contract.  Following USAID’s request for a chief of party, MEASURE Evaluation had difficulty recruiting due to the short timeline (12 months) and internal project considerations, which delayed opening of the office (John Snow, Inc., versus Palladium staff)  By the time the chief of party joined, relationships were strained and the activity was at a very challenging point.  In April 2016, as the HMIS activity was beginning, a corruption inquiry into the DPRS resulted in a complete turnover of staff in the office.  None of the four key DPRS staff interviewed had worked there for more than 16 months.  One of the four staff from DPRS has already left since the start of the HMIS activity  Over the past 2 years, multiple activity managers have been point-persons for the project at USAID.  Project management is split between the Office of HIV/AIDS and the Office of Health, Population, and Nutrition. The technical resources in the government occasionally are not of the top quality that you want. You find maybe the head is very good but the technical people who are supposed to lead these activities are not as strong. Some of us have been supporting them to write, providing them with technical support, but it isn’t an easy task. —MEASURE Evaluation, Nigeria M&E Landscape: The only other USAID-supported M&E mechanism that was present during Phase IV was the Nigeria Monitoring and Evaluation Management Services contract (held by the Mitchell Group), which provided M&E services to the USAID mission. That project has closed.23 Internal and external stakeholders working in health area–specific activities, such as malaria and OVC, reported a high degree of collaboration, both in terms of donor coordination and between MEASURE Evaluation and other implementing partners. However, USAID staff reported being unsatisfied with the project’s coordination with key implementing partners that could have been essential to the success of the HMIS activity. USAID staff expressed a wish that MEASURE Evaluation had been more proactive in coordinating with other implementing partners, also recognizing that although they ended up playing a critical convening role, they could have done so earlier in the process. [MEASURE Evaluation collaborates with the FMWASD … Sometimes it is difficult because the system of government and the system of the international organization is not the same, so they have to bridge the gap by trying to mobilize the respective government agencies to be able to see reasons why meetings should be held, why we should do things the way we do them and also try to … put them in the driver’s seat so that they can move and I think they have been able to show resilience on that issue. —External stakeholder, Nigeria Government, USAID, and MEASURE Evaluation stakeholders focused on the challenges USAID faced in coordinating with the other major donors working in HMIS, including the Bill & Melinda Gates Foundation, the Global Fund, and the World Health Organization (WHO). Although there 23 The follow-on project, Support for Strategic Information and Project Management Services, is managed by DevTech. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 122 is an HMIS Technical Working Group, it has only recently begun making progress in increasing cross-partner communication. Emphasizing this point, three interview respondents reported different stakeholders as leading the HMIS Technical Working Group—the government, WHO, and WHO and USAID jointly. Recognizing the challenge, USAID and other donors have been making a more pronounced effort to coordinate among themselves and their implementing partners. Global Fund partners confirmed that communication and collaboration have continued to improve. Issues of Sustainability: Internal and external stakeholders working on health-specific activities reported several areas as evidence of sustainability—MEASURE Evaluation’s capacity-building efforts built into training, efforts to institutionalize the National OVC Management Information System, supporting technical working groups and governments in taking the lead role with OVC, and ensuring that MEASURE Evaluation tools and guidelines are disseminated. I think that is one of the strength of MEASURE Evaluation, being able to work with FMWASD at the national level to coordinate these [technical working group] meetings and … move the strategies that will enhance the well-being of children and improve, in particular, the area of quality evaluation. —External country partner, Nigeria Stakeholders across activities spontaneously mentioned MEASURE Evaluation Phase III activities that continue to benefit the Nigerian health system. Most reported was the institutionalization of an M&E curriculum at Obafemi Awolowo University and Ahmadu Bello University, both of which continue to produce M&E professionals without Phase IV funding. I read about the CEOs of one of the PEPFAR implementing agencies … I saw clearly that he was an M&E expert trained in Ife [Obafemi Awolowo University]. So, the output now are CEOs of an organization funded by CDC … For me that’s one of the high point. —External stakeholder, Nigeria DPRS stakeholders mentioned MEASURE Evaluation’s Phase III work supporting the development of the National HIS Strategic Plan (2014–2018), the National Strategic Health Development Plan (2010–2015), and the National Health Information Systems Policy (2014) as important groundwork that influenced the focus of HIS activities in Phase IV. OVC stakeholders highlighted that information from the 2008 OVC Situational Assessment Analysis Survey continues to be the most updated information on OVC in Nigeria. Government partners and USAID staff pointed to the importance of the sustainability of the Master Facility List, but were concerned about the lack of systems and processes to ensure its continuous updates and oversight. The rollout of some of these processes originally fell under the HMIS portfolio; however, government, external stakeholders, and MEASURE Evaluation reported that, given the timeline and work plan provided, there was no opportunity to apply a systems focus in rolling out this activity. All three stakeholder groups noted that an activity such as the Master Facility List ideally requires a longer timeline and work plan, including time to build capacity of subnational personnel in the facilities as part of the data collection process that was required to populate the Master Facility List. All internal and external stakeholders also noted government systems that need should be strengthened to ensure continuous updates to this activity. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 123 HOW HAS A HEALTH SECTOR–WIDE VERSUS A HEALTH AREA–SPECIFIC APPROACH PLAYED OUT IN NIGERIA? USAID and Nigerian government stakeholders reported the key facilitator of a health sector–wide approach as the ability encourage government ownership by aligning with the government system. USAID stakeholders reported that this approach enabled MEASURE Evaluation to use more flexible definitions across USAID offices (e.g., not being restricted to PEPFAR definitions for OVC activities). USAID and MEASURE Evaluation stakeholders highlighted the need to align with different USAID Offices’ staff, funding, and reporting requirements as a key barrier to implementing a health sector–wide approach. In government, the facilities are not sector-specific and [the health sector–wide approach] is helpful to get ownership … not just being in one area or another, but being across the board. —USAID/U.S. Government, Nigeria MEASURE Evaluation and USAID staff reported that the project’s concurrent portfolio allowed it to leverage its health sector–specific knowledge and relationships to harmonize information and efforts across its health sector–wide activities. MEASURE Evaluation staff highlighted the benefit of having USAID activity managers, including for the HMIS, to whom they could go for support. That’s what I see in MEASURE … it helps everybody focus and brings out the best in each of their portfolios. The lead person takes over discussing what will happen and advise us. So, it helps them to leverage on skills and knowledge of getting it from the best of the pool. —MEASURE Evaluation, Nigeria Most external stakeholders reported that although they saw this as theoretically beneficial, they did not feel they could adequately answer this question, because they did not have in-depth knowledge of the project’s internal structure. ARE THE TOOLS THAT MEASURE EVALUATION HAS SUPPORTED USEFUL? Overall, respondents reported the most knowledge of tools related to their health area–specific work. Across all stakeholder groups, the most-mentioned tools were the routine data quality assessment tools and guidelines and the Routine HMIS Guidelines, as Exhibit 34 shows. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 124 Exhibit 34: Most useful tools, as cited by stakeholders MEASURE Evaluation (n=5) USAID/U.S. Government (n=5) External (n=3) Nigerian Government (n=9) Framework for Malaria Control Monitoring, Evaluation, and Reporting Essential Survey Malaria M&E Curriculum DHIS 2 OVC Data Collection Tool Routine HMIS Guidelines Routine Data Quality Assessment Tool 0% 20% 40% 60% 80% 100% Government respondents reported institutionalizing the routine data quality assessment to ensure that all data quality would be held to the MEASURE Evaluation standard. [The data quality assessment] tool … is based on a program and if at the program level you discover that is what you need, you are likely to use them because programs were using them. [Data quality assessment] tools help you to evaluate both the system that was generating the data and individuals’ capacity. —Government, Nigeria In terms of sector-specific tools, the newly disseminated OVC data collection tools were reported as having provided the government with an unprecedented, country-wide view across organizations working on OVC activities. Stakeholders also highlighted the Malaria M&E Curriculum and the Framework for Evaluating the Impact of National Malaria Control Programs in Highly Endemic Countries as well known and used. MEASURE Evaluation led a very collaborat[ive] and participatory process of reviewing all the data collection tools for OVC and change management reporting … we all got to a point where there was a sense of agreement as to a set of data collection tools that everyone should use … this is a major achievement on the part of MEASURE Evaluation. —External stakeholder, Nigeria Although few external stakeholders could speak to tool development, some noted that the most successful tools were those developed in response to a need and based on evidence gathered in Nigeria. They also recognized MEASURE Evaluation’s participatory tool development process, which respondent to “the needs at the time.” The only challenge highlighted in tool dissemination and use related to adapting to specific internal country contexts—the need to adapt for local language and for lower-level staff capacity. However, many stakeholders noted that building subnational capacity was outside of MEASURE Evaluation’s mandate and scope. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 125 CONCLUSIONS In Nigeria, most external stakeholders regard MEASURE Evaluation as offering expertise in M&E. However, its ability to meet internal and external stakeholders’ needs has varied. In health sector– specific activities, the project’s malaria, OVC, and TB counterparts highly value its role as a provider of capacity building and HIS support. These stakeholders attributed MEASURE Evaluation’s success to its consistent, committed staff, who provide high-quality technical support and were able to work with USAID, government, and implementing partner colleagues to communicate clear messages about their vision and work plan. Consistency, transparency, and commitment have enabled MEASURE Evaluation to succeed in these health-specific areas. HIS activities have been less successful, due to a confluence of factors related to project history, staffing, management, structure, and communication. A funding gap between Phases III and IV created a gap in HIS work in Nigeria, followed by staff movement within MEASURE Evaluation, USAID, and DPRS, which challenged relationship-building and resulted in unclear communication across the three groups. Delays in hiring a chief of party were attributed to difficulties recruiting for a short-term position, delays from MEASURE Evaluation headquarters regarding where to house the chief of party, and challenges in finding a candidate with the right combination of technical expertise and experience working with Federal Ministry of Health. The lack of consistent in-country staff for the HMIS activity limited collaboration with other implementing partners, compounded by limited donor collaboration in this area. However, USAID and external stakeholders reported signs of improvement in donor coordination and collaboration in the final months of the project. MEASURE Evaluation’s ability to work across the health sector and in specific health areas has allowed its teams to leverage sector-specific work to benefit the broader health system. Tools developed in Phases III and IV continue to be used and highly regarded in Nigeria. Its participatory approach to adaptation has resulted in government partners’ institutionalizing tools such as the routine data quality assessment and OVC data collection tools, highlighting the sustainability of the Phase III tools and their successful dissemination at the country level. For future procurements, most internal and external stakeholders recommended continuing to build capacity in data quality and use, expansion of training to subnational stakeholders, and increasing government ownership. Internal and external stakeholders working on the Master Facility List suggested focusing on the interoperability of HIS as the next priority, once the List is complete. To accomplish these results, secondary stakeholders suggested that USAID-funded projects take a longer-term view of HIS strengthening in project design, management, and implementation, ensuring continued collaboration and coordination across all actors. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 126 ANNEX 10: ONLINE SURVEY TABLES This annex presents a comprehensive set of tables representing the full results from the online surveys, short and long versions. Some of these data are already provided in graphic or narrative form in the Findings section of the main report narrative. All data presented here are organized by MEASURE Evaluation’s internal and external stakeholder groups:  Internal stakeholders are those from USAID, the Office of the Global AIDS Coordinator (OGAC), and other U.S. Government agencies.  External stakeholders are those from country-level ministries of health (and other relevant ministries), other USAID implementing partners, other donors, other implementing partners, academia, and international organizations working at country or global level. Note: Some online survey questions were relevant to only a portion of survey respondents, and not all respondents answered all the questions or fully completed the survey. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 127 Exhibit 35: Respondents who have participated in a technical working group or community of practice that MEASURE Evaluation has led (Question 3 – long survey) Group/Community Internal n External n TOTAL N “I have not been a member of any group that MEASURE Phase IV led [technical working group]” 19 38 57 Bureau for Global Health Cooperating Agencies Monitoring 0 6 6 and Evaluation (M&E) Technical Working Group (TWG) Data UseNet Listserv 0 1 1 Geographic Information Systems (GIS) Working Group 1 0 1 Global Evaluation and Monitoring Network for Health 0 9 9 (GEMNet-Health) Routine Health Information Network (RHINO) 2 14 16 Internal stakeholders: USAID, OGAC, other U.S. Government agencies. External stakeholders: Country ministries, USAID implementing partners, other donors, other implementing partners, academia. Exhibit 36: Respondents’ rating of effectiveness of technical working group or community of practice they participate in (Q4 – long survey) Group Very effective Effective Neutral Ineffective Very ineffective TOTAL 1 n (%) 2 n (%) 1 n (%) 2 n (%) 1 n (%) 2 n (%) 1 n (%) 2 n (%) 1 n (%) 2 n (%) N BGH M&E 0 2 (40%) 0 2 (40%) 0 0 0 0 0 1 (20%) 5 GIS 0 0 1 (100%) 0 0 0 0 0 0 0 1 GEMNet 0 7 (78%) 0 2 (22%) 0 0 0 0 0 0 9 RHINO 0 4 (33%) 1 (100%) 6 (50%) 0 1 (8%) 0 1(8%) 0 0 13 Other 1 (100%) 7 (30%) 0 16 (70%) 0 0 0 0 0 0 24 Respondents can be members of more than one technical working group and/or community of practice. 1 = Internal stakeholders (USAID, OGAC, and other U.S. Government agencies); n=2 2 = External stakeholders (country ministries, USAID implementing partners, other donors, other implementing partners, academia); n=46 February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 128 Exhibit 37: Degree to which respondents felt participati ng in the technical working groups/communities of practice benefited them (Q5 – long survey) Strongly agree Agree Neutral Disagree Strongly disagree TOTAL Enable to: 1 n (%) 2 n (%) 1 n (%) 2 n (%) 1 n (%) 2 n (%) 1 n (%) 2 n (%) 1 n (%) 2 n (%) N Learn about strengthening HIS Learn about 1 (50%) 15 (36%) 1 (50%) 21 (50%) 0 3 (7%) 0 2 (5%) 0 1 (2%) 44 strengthening evaluation Learn about 2 (100%) 18 (43%) 0 19 (45%) 0 3 (7%) 0 1 (2%) 0 1 (2%) 44 similar work of others Get advice from 1 (50%) 12 (29%) 1 (50%) 23 (55%) 0 4 (10%) 0 2 (5%) 0 1 (2%) 44 others on technical issues Locate useful 0 18 (43%) 2 (100%) 18 (43%) 0 4 (10%) 0 1 (2%) 0 1 (2%) 44 knowledge and resources Enhance job 2 (100%) 19 (45%) 0 17 (40%) 0 4 (10%) 0 1 (2%) 0 1 (2%) 44 performance 1 (50%) 21 (50%) 1 (50%) 14 (33%) 0 5 (12%) 0 1 (2%) 0 1 (2%) 44 Respondents can be members of more than one technical working group and/or community of practice. 1 = Internal stakeholders (USAID, OGAC, and other U.S. Government agencies); n=2 2 = External stakeholders (country ministries, USAID implementing partners, other donors, other implementing partners, academia); n=43 February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 129 Exhibit 38: Number of respondents participating in a technical working group or community of practice that MEASURE Evaluation is involved in (Q6 – long survey) Group/Community Internal n External n TOTAL N N/A – Not a member 17 43 60 Asia eHealth Information Network (AeHIN) RHIS Subgroup 1 3 4 Community-Based Health Information System (HIS) Stakeholder Consultation Meetings 1 5 6 East Africa Sample Vital Registration with Verbal Autopsy (SAVVY)/Civil 0 2 2 Every Newborn Action Plan (ENAP) Working Group 0 1 1 Global Digital Health Network 2 3 5 Health Data Collaborative 1 6 7 Ministry of Health Data Alignment 0 3 3 Programmatic Mapping and Size Estimation for Key Populations Partnership 3 3 6 Roll Back Malaria Monitoring and Evaluation Reference Group (RBM-MERG) 1 0 1 West African Health Organization Collaboration 1 2 3 WHO Routine Health Information System (RHIS) Strengthening Collaboration 1 2 3 Other groups* 1 14 15 * Other groups include OVC, SAVVY Implementation, Capacity Building + Data Audit, Bangladesh Maternal Mortality Survey, MEASURE small grants program, Technical Task Team on DHIS, Routine and Survey Data Analysis, Gender-Based Violence Referral System Project, PEPFAR Pivot on Its Effects on the National Health Information System. February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 130 Exhibit 39: Rating of the assistance MEASURE Evaluation provided to technical working groups (Q7 – long survey) Very helpful Moderately helpful Not involved TOTAL 1 2 1 2 1 2 N n (%) n (%) n (%) n (%) n (%) n (%) Achieving the group’s goals 1 (25%) 27 (82%) 2 (50%) 4 (12%) 0 0 34 Disseminating knowledge 3 (75%) 22 (69%) 0 8 (25%) 0 0 33 Convening stakeholders 1 (25%) 20 (61%) 2 (50%) 10 (30%) 0 1 (3%) 34 Facilitating learning 1 (25%) 23 (70%) 2 (50%) 8 (24%) 0 0 34 Providing or creating useful tools or resources 2 (50%) 27 (82%) 1 (25%) 3 (9%) 0 1 (25%) 34 1 = Internal stakeholders (USAID, OGAC, and other U.S. Government agencies); n=3 2 = External stakeholders (country ministries, USAID implementing partners, other donors, other implementing partners, academia); n=32 February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 131 Exhibit 40: Tools respondents have worked with (Q9 – long survey) (n=55) Tools used Internal n External n TOTAL N Cote d’Ivoire Data Collection Tools for Community-Level Indicators for Early Warning System 1 1 2 Data Quality Protocol for DATIM Site Location 2 2 4 Data Quality Review (DQR) Tool, version 2.1 4 13 17 DHIS 2 (version after 2014) 5 20 25 Framework for Evaluating the Impact of National Malaria Control Programs in Highly Endemic Countries 1 4 5 Guidelines for Integrating Gender into an M&E Framework and System Assessment (English/French) 1 6 7 HIS Strengthening Model (HISSM) 1 4 5 Malaria M&E training curriculum 2 5 7 PEPFAR Dashboard 3.0 7 2 9 PEPFAR M&E, and Reporting: Collection of Essential Survey Indicators of OVC Well-being through Outcome Monitoring 3 5 8 PLACE Update: Tool for Collection of Biologic Specimens 2 3 5 PLACE Update: Summary of HIV Prevalence and Size Estimates in Key Populations Tool 6 4 10 Planning Guide for a Total Market Approach to Increase Access to Family Planning 2 2 4 PRISM Tools for Assessing, Monitoring, and Evaluating RHIS Performance 2 10 12 Results-Based Financing (RBF) Indicator Compendium 1 7 8 Routine Data Quality Assessment (RDQA) Tool: User Manual 8 13 21 RDQA update – Gender Integrated Routine Data Quality Assessment (RDQA+G) Tool 0 5 5 Routine Health Information Systems (RHIS): Curriculum on Basic Concepts and Practice (Facilitator’s 1 6 7 Guide and Syllabus) Sample Vital Registration with Verbal Autopsy (SAVVY) 1 6 7 Internal stakeholders: USAID, OGAC, and other U.S. Government agencies External stakeholders: country ministries, USAID implementing partners, other donors, other implementing partners, academia February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 132 Exhibit 41: Respondent rating of utility of MEASURE Evaluation tools (Q10 – long survey) Very useful Somewhat useful Not at all useful TOTAL 1 2 1 2 1 2 N n (%) n (%) n (%) n (%) n (%) n (%) CI Community indicators early warning 0 1 (100%) 1 (100%) 0 0 0 2 Data quality for DATIM 2 (100%) 2 (100%) 0 0 0 0 4 DHIS 2 3 (60%) 17 (89%) 2 (40%) 2 (11%) 0 0 24 Framework for evaluating impact Malaria 0 3 (100%) 0 0 1 (100%) 0 4 Data Quality Review (DQR) 3 (75%) 11 (92%) 1 (25%) 1 (8%) 0 0 16 Integrating gender into M&E framework 0 4 (67%) 1 (100%) 2 (33%) 0 0 7 Malaria M&E training curriculum (Eng/Fr) 2 (100%) 5 (100%) 0 0 0 0 7 HISSM 0 3 (75%) 1 (100%) 1 (25%) 0 0 5 PEPFAR Dashboard 3 (43%) 2 (100%) 4 (57%) 0 0 0 9 OVC indicators 1 (33%) 4 (80%) 2 (67%) 1 (20%) 0 0 8 PLACE biological specimens 1 (50%) 2 (67%) 1 (50%) 1 (33%) 0 0 5 PLACE HIV prevalence and size estimates 4 (67%) 4 (100%) 2 (33%) 0 0 0 10 RDQA user manual 8 (100%) 13 (100%) 0 0 0 0 21 RDQA+Gender 0 5 (100%) 0 0 0 0 5 RHIS curriculum 0 5 (83%) 1 (100%) 1 (17%) 0 0 7 Guide for a Total Market Approach 0 2 (100%) 0 0 0 0 2 PRISM tools 0 9 (100%) 2 (100%) 0 0 0 11 RBF indicator compendium 0 7 (100%) 1 (100%) 0 0 0 8 SAVVY 1 (100%) 4 (67%) 0 2 (33%) 0 0 7 1 = Internal stakeholders (USAID, OGAC, and other U.S. Government agencies) 2 = External stakeholders (country ministries, USAID implementing partners, other donors, other implementing partners, academia) February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 133 Exhibit 42: Respondent awareness of MEASURE Evaluation tools that have not been produced despite the need, or tools that did not prove to be useful (Q11 and 12 – long survey) Yes No TOTAL Awareness of: Internal External Internal External Total Don’t know n (%) n (%) n (%) n (%) responding Instances where MEASURE Evaluation has not 1 (8%) 3 (8%) 11 (85%) 24 (62%) 39 13 produced a requested tool MEASURE Evaluation tools that were not useful, 2 (15%) 2 (5%) 11 (85%) 26 (67%) 41 11 relevant or appropriate 1 = Internal stakeholders (USAID, OGAC, and other USG agencies) 2 = External stakeholders (country ministries, USAID implementing partners, other donors, other implementing partners, academia) Exhibit 43: Respondents reasons for buying into MEASURE Evaluation (Q16 – long survey) Internal (N=13) Reason for buying in: n (%) Strong reputation in HIS strengthening 7 (53%) Strong reputation in evaluation capacity building 9 (69%) No better alternative mechanism, project, or program 2 (15%) Familiarity with past services and performance 8 (62%) Ease of buy in into mechanism 5 (38%) Other 1 (8%) February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 134 Exhibit 44: Degree to which respondents feel MEASURE Evaluation actions encourage sustainability of efforts after the end of Phase IV (Q22 – long survey) To a great extent To some extent Don’t know TOTAL Field n (%) Field n (%) Field n (%) Implementing HIS strengthening in ways that encourage true ownership and leadership by national counterparts 1 (8%) 4 (33%) 2 (17%) 12 Facilitated broader coordination and collaboration with other partners supporting various aspects of HIS strengthening 3 (25%) 1 (8%) 2 (17%) 12 Co-developed on-going financing of recurrent costs of the HIS system (equipment, supervision, training, supplies) 0 1 (8%) 3 (25%) 12 Built adequate capacity of national counterparts to update the HIS as needs evolve 1 (8%) 3 (25%) 2 (17%) 12 Built adequate capacity of national counterparts to conduct rigorous evaluations 1 (8%) 4 (33%) 2 (17%) 12 Contributed to development of a culture of data use at the national level of the health system 1 (9%) 5 (45%) 4 (36%) 11 Contributed to development of a culture of data use at decentralized levels of the health system 2 (17%) 6 (50%) 2 (17%) 12 February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 135 Exhibit 45: Internal and external stakeholder respondents who said their organization plans to use or collaborate with MEASURE Evaluation in the future (Q25 – long survey) Does your organization plan to use of Internal External collaborate with the MEASURE TOTAL n (%) n (%) Evaluation project in the future? Yes 7 (50%) 44 (81%) 51 (75%) No 0 2 (4%) 2 (3%) Don’t know 7 (50%) 8 (15%) 15 (22%) TOTAL 14 54 68 Exhibit 46: U.S. Government Washington-based respondents’ satisfaction with MEASURE Evaluation based on their experiences with the project (Q3 – short survey) Very satisfied Satisfied Neutral Unsatisfied Very unsatisfied TOTAL HIV n (%) Non-HIV n (%) HIV n (%) Non-HIV n (%) HIV n (%) Non-HIV n (%) HIV n (%) Non-HIV n (%) HIV n (%) Non-HIV n (%) N Delivering useful tools and products 0 1 (20%) 3 (43%) 3 (60%) 2 (29%) 1 (20%) 0 0 2 (29%) 0 12 Delivering tools and products in a timely manner 0 1 (20%) 2 (29%) 3 (60%) 3 (43%) 0 1 (14%) 1 (20%) 1 (14%) 0 12 Leading technical working groups 0 2 (50%) 2 (40%) 2 (50%) 2 (40%) 0 0 0 1 (20%) 0 9 Coordinating with USAID 0 1 (20%) 4 (57%) 2 (40%) 1 (14%) 1 (20%) 2 (29%) 1 (20%) 0 0 12 Coordinating with other stakeholders 0 2 (40%) 3 (43%) 3 (60%) 2 (29%) 0 2 (29%) 0 0 0 12 Providing technical leadership 0 3 (50%) 2 (29%) 2 (33%) 2 (29%) 1 (17%) 2 (29%) 0 1 (14%) 0 13 HIV = Those working for the Office of HIV/AIDS or OGAC Non-HIV = Those working for other offices in the Bureau for Global Health February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 136 Exhibit 47: Internal stakeholder respondent knowledge of USAID-funded projects that work in evaluation and/or health information system (Q17 – long survey; Q10 – short survey) Field Washington TOTAL Other USAID-funded projects n (%) HIV Non-HIV n (%) n (%) Field Washington N=13 N=14 Global Health Program Cycle Improvement Project (GH Pro) 4 (30%) 5 (36%) 5 (36%) 4 10 Coordinating Implementation Research to Communicate Learning and Evidence (CIRCLE) 1 (8%) 0 2 (14%) 1 2 Learning and Evidence 0 0 3 (21%) 0 3 Accelerating Strategies for Practical Innovation and Research in Economic Strengthening (ASPIRES) 2 (15%) 2 (14%) 2 (14%) 2 4 Health Evaluation, an Applied Research and Development (HEARD) 1 (8%) 2 (14%) 2 (14%) 1 4 Breakthrough RESARCH project 1 (8%) 1 (7%) 1 (7%) 1 2 USAID bilateral or regional M&E platform or mechanisms 5 (38%) 1 (7%) 4 (29%) 5 5 Digital Health Initiative 2 (15%) 4 (29%) 3 (21%) 2 7 Supporting Operational AIDS Research (SOAR) 0 4 (29%) 4 (29%) 0 8 None 3 (23%) 0 0 3 0 Other 0 0 3 (21%) 0 3 HIV = Those working for the Office of HIV/AIDS or OGAC Non-HIV = Those working for other offices in the Bureau for Global Health February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 137 Exhibit 48: Respondents’ identification of three top emerging needs in the next 2–5 years for the development of HIS in the country or region they work with/in (Q23 – long survey; Q8 – short survey) Internal Stakeholders TOTAL Field Washington Field Washington Top three emerging needs in the next 2–5 years in HIS Internal n (%) External n (%) HIV n (%) Non-HIV n (%) Internal N=18; External N=67 N=14 (HIV: 7; Non-HIV: 7) Improved capacity to demand, implement and use HIS at national and sub-national (regional/district) levels 7 (39%) 24 (36%) 2 (29%) 3 (1%) 31 5 Improved capacities of sub-national (regional/district) level governments to manage health information systems resources and processes, including supervision and ongoing training 6 (34%) 26 (39%) 3 (43%) 3 (1%) 32 6 Increased use of HIS data for decision making at national and sub-national (regional/district) levels 10 (56%) 30 (45%) 1 (14%) 2 (%) 40 3 Increased use of sex-disaggregated data from HIS to identify and plan interventions to close gender gaps 3 (17%) 8 (12%) 1 (14%) 0 11 1 Integration of HIS data to respond to health-related Sustainable Development Goals and respective targets 2 (11%) 17 (25%) 0 0 19 0 Improved capacities to monitor and evaluate HIS performance 3 (17%) 15 (22%) 1 (14%) 0 18 1 Interoperability of databases (DHIS 2, logistics databases, human resources databases, other specific interventions, such as OVC, HIV/DATIM, immunizations, etc.) 9 (50%) 27 (40%) 5 (71%) 3 (43%) 36 8 Increased availability of procedural guidance documents for software and processes, such as standard operating procedures (SOP) 1 (6%) 4 (6%) 0 1 (14%) 5 1 Increased resources for maintenance and replacement of HIS software and infrastructure (computers, internet access, solar kits, etc.) 3 (17%) 7 (10%) 0 0 10 0 Improved quality of routine health information 9 (50%) 31 (46%) 1 (14%) 1 (14%) 40 2 Other 1 (6%) 4 (6%) 2 (29%) 4 (58%) 5 6 HIV = Those working for the Office of HIV/AIDS or OGAC Non-HIV = Those working for other offices in the Bureau for Global Health February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 138 Exhibit 49: Respondents’ identification of top three emerging needs in the next 2–5 years for the development of evaluation capacity in the country or region they work with/in (Q24 – long survey; Q9 – short survey) Stakeholders TOTAL Field Washington (Internal) Field Washington Top three emerging needs in the next 2–5 years for the development of evaluation capacity Internal n (%) External n (%) HIV n (%) Non-HIV n (%) Internal N=18; External N=67 N=14 (HIV: 7; Non-HIV: 7) Improved capacity of national government decision￾makers to demand and use evidence from rigorous evaluations for planning and advocacy 12 (67%) 45 (68%) 1 (15%) 1 (15%) 57 2 Improved capacity of national institutions to design, implement and facilitate use of findings from rigorous evaluations 8 (44%) 31 (46%) 2 (29%) 3 (43%) 39 5 Improved capacities of national institutions to demand and use evidence from equity-focused and gender responsive evaluations for planning interventions 3 (17%) 12 (18%) 1 (15%) 1 (15%) 15 2 Improved quality of evaluations in terms of level rigor, quality of designs and data collected, relevance to answer priority questions, and availability and dissemination for use that are accessible to decision makers 5 (28%) 41 (61%) 2 (29%) 4 (57%) 46 6 Other 4 (22%) 1 (1.5%) 2 (29%) 2 (29%) 5 4 HIV = Those working for the Office of HIV/AIDS or OGAC Non-HIV = Those working for other offices in the Bureau for Global Health February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 139 Exhibit 50: U.S. Government (internal stakeholder) rating of MEASURE Evaluation performance on HIS strengthening, use of information, and conducting evaluation (Q19 – long survey; Q4 – short survey) Very high Above average Average Below average Field Washington Field Washington Field Washington Field Washington TOTAL n (%) n (%) n (%) n (%) n (%) n (%) n (%) N (%) HIV Non-HIV HIV Non-HIV HIV Non-HIV HIV Non-HIV Field Washington Managing HIS 1 2 6 2 2 1 1 1 HIV: 3 0 0 0 0 8 (13%) (40%) (75%) (67%) (40%) (13%) (34%) (20%) Non-HIV: 5 Collecting and using health info 2 2 8 1 3 2 1 2 HIV: 4 to make 0 0 0 0 12 (17%) (40%) (67%) (25%) (60%) (17%) (25%) (50%) Non-HIV: 5 strategic decisions Conducting 5 5 2 2 1 HIV: 3 0 4 (100%) 0 0 0 0 0 12 evaluations (42%) (42%) (67%) (17%) (34%) Non-HIV:4 Integrating gender in M&E 1 1 1 1 HIV: 2 of health — 0 0 — — 0 — 0 — (50%) (50%) (50%) (50%) Non-HIV: 2 programs and HIS HIV = Those working for the Office of HIV/AIDS or OGAC Non-HIV = Those working for other offices in the Bureau for Global Health February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 140 ANNEX 11: CONFLICT OF INTEREST DISCLOSURES This annex presents disclosures of conflicts of interest for the following evaluation team members:  Lynne Franco  Elisa Knebel  Svetlana Negroustoueva  Kelsey Simmons  Sarah Lunsford  Sabine Topolansky  Claude Kouakou Konan  Mamadou Faramba Camara  Mabinu Olasumbo Oladipo February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 141 DISCLOSURE OF CONFLICTS OF INTEREST Name: Lynne Franco Title: Vice President, Technical Assistance and Evaluation Organization: EnCompass LLC Evaluation Position:  Team leader  Team member Evaluation Award Number: Contract or other instrument USAID Contract No. AID-OAA-I-15-00021 Task Order No. AID-612-TO-17-00015 USAID Project(s) Evaluated: Include project name(s), implementer name(s), and award number(s), if applicable MEASURE Evaluation Phase IV, implemented by University of North Carolina at Chapel Hill, Cooperative Agreement No. AID-L-14-00004 I have real or potential conflicts of interest to disclose.  Yes  No Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose project(s) are being evaluated or in the outcome of the evaluation 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular project(s) and organization(s) being evaluated that could bias the evaluation. If yes, I disclose the following facts: I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature: Date: 11/20/2017 February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 142 DISCLOSURE OF CONFLICTS OF INTEREST Name: Elisa Knebel Title: Senior Evaluation and Learning Advisor Organization: EnCompass LLC Evaluation Position:  Team leader  Team member Evaluation Award Number: Contract or other instrument USAID Contract No. AID-OAA-I-15-00021 Task Order No. AID-612-TO-17-00015 USAID Project(s) Evaluated: Include project name(s), implementer name(s), and award number(s), if applicable MEASURE Evaluation Phase IV, implemented by University of North Carolina at Chapel Hill, Cooperative Agreement No. AID-L-14-00004 I have real or potential conflicts of interest to disclose.  Yes  No Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose project(s) are being evaluated or in the outcome of the evaluation 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular project(s) and organization(s) being evaluated that could bias the evaluation. If yes, I disclose the following facts: I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature: Date: November 22, 2017 February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 143 DISCLOSURE OF CONFLICTS OF INTEREST Name: Svetlana Negroustoueva Title: Evaluation Consultant Organization: EnCompass LLC Evaluation Position:  Team leader  Team member Evaluation Award Number: Contract or other instrument USAID Contract No. AID-OAA-I-15-00021 Task Order No. AID-612-TO-17-00015 USAID Project(s) Evaluated: Include project name(s), implementer name(s), and award number(s), if applicable MEASURE Evaluation Phase IV, implemented by University of North Carolina at Chapel Hill, Cooperative Agreement No. AID-L-14-00004 I have real or potential conflicts of interest to disclose.  Yes  No Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose project(s) are being evaluated or in the outcome of the evaluation 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular project(s) and organization(s) being evaluated that could bias the evaluation. If yes, I disclose the following facts: I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature: Date: November 21, 2017 February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 144 DISCLOSURE OF CONFLICTS OF INTEREST Name: Kelsey Simmons Title: Evaluation Specialist Organization: EnCompass LLC Evaluation Position:  Team leader  Team member Evaluation Award Number: Contract or other instrument USAID Contract No. AID-OAA-I-15-00021 Task Order No. AID-612-TO-17-00015 USAID Project(s) Evaluated: Include project name(s), implementer name(s), and award number(s), if applicable MEASURE Evaluation Phase IV, implemented by University of North Carolina at Chapel Hill, Cooperative Agreement No. AID-L-14-00004 I have real or potential conflicts of interest to disclose.  Yes  No Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose project(s) are being evaluated or in the outcome of the evaluation 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular project(s) and organization(s) being evaluated that could bias the evaluation. If yes, I disclose the following facts: I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature: Date: 11/21/2017 February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 145 DISCLOSURE OF CONFLICTS OF INTEREST Name: Sarah Lunsford Title: Sr. Research Advisor Organization: EnCompass LLC Evaluation Position:  Team leader  Team member Evaluation Award Number: Contract or other instrument USAID Contract No. AID-OAA-I-15-00021 Task Order No. AID-612-TO-17-00015 USAID Project(s) Evaluated: Include project name(s), implementer name(s), and award number(s), if applicable MEASURE Evaluation Phase IV, implemented by University of North Carolina at Chapel Hill, Cooperative Agreement No. AID-L-14-00004 I have real or potential conflicts of interest to disclose.  Yes  No Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose project(s) are being evaluated or in the outcome of the evaluation 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular project(s) and organization(s) being evaluated that could bias the evaluation. If yes, I disclose the following facts: I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature: Date: 12/1/2017 February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 146 DISCLOSURE OF CONFLICTS OF INTEREST Name: Sabine Topolansky Title: Evaluation Associate Organization: EnCompass LLC Evaluation Position:  Team leader  Team member Evaluation Award Number: Contract or other instrument USAID Contract No. AID-OAA-I-15-00021 Task Order No. AID-612-TO-17-00015 USAID Project(s) Evaluated: Include project name(s), implementer name(s), and award number(s), if applicable MEASURE Evaluation Phase IV, implemented by University of North Carolina at Chapel Hill, Cooperative Agreement No. AID-L-14-00004 I have real or potential conflicts of interest to disclose.  Yes  No Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose project(s) are being evaluated or in the outcome of the evaluation 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular project(s) and organization(s) being evaluated that could bias the evaluation. If yes, I disclose the following facts: I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature: Date: November 20, 2017 February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 147 DISCLOSURE OF CONFLICTS OF INTEREST Name: Claude Kouakou Konan Title: Evaluation Specialist – Consultant Organization: EnCompass LLC Evaluation Position:  Team leader  Team member Evaluation Award Number: Contract or other instrument USAID Contract No. AID-OAA-I-15-00021 Task Order No. AID-612-TO-17-00015 USAID Project(s) Evaluated: Include project name(s), implementer name(s), and award number(s), if applicable MEASURE Evaluation Phase IV, implemented by University of North Carolina at Chapel Hill, Cooperative Agreement No. AID-L-14-00004 I have real or potential conflicts of interest to disclose.  Yes  No Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose project(s) are being evaluated or in the outcome of the evaluation 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular project(s) and organization(s) being evaluated that could bias the evaluation. If yes, I disclose the following facts: I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature: Date: 11/21/2017 February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 148 DISCLOSURE OF CONFLICTS OF INTEREST Name: Mamadou Faramba CAMARA Title: Consultant Organization: EnCompass LLC Evaluation Position:  Team leader  Team member Evaluation Award Number: Contract or other instrument USAID Contract No. AID-OAA-I-15-00021 Task Order No. AID-612-TO-17-00015 USAID Project(s) Evaluated: Include project name(s), implementer name(s), and award number(s), if applicable MEASURE Evaluation Phase IV, implemented by University of North Carolina at Chapel Hill, Cooperative Agreement No. AID-L-14-00004 I have real or potential conflicts of interest to disclose.  Yes  No Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose project(s) are being evaluated or in the outcome of the evaluation 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular project(s) and organization(s) being evaluated that could bias the evaluation. If yes, I disclose the following facts: I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature: Date: 21/11/2017 February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 149 DISCLOSURE OF CONFLICTS OF INTEREST Name: Mabinu Olasumbo Oladipo Title: Evaluation Specialist – Individual Consultant Organization: EnCompass LLC Evaluation Position:  Team leader  Team member Evaluation Award Number: Contract or other instrument USAID Contract No. AID-OAA-I-15-00021 Task Order No. AID-612-TO-17-00015 USAID Project(s) Evaluated: Include project name(s), implementer name(s), and award number(s), if applicable MEASURE Evaluation Phase IV, implemented by University of North Carolina at Chapel Hill, Cooperative Agreement No. AID-L-14-00004 I have real or potential conflicts of interest to disclose.  Yes  No Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose project(s) are being evaluated or in the outcome of the evaluation 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular project(s) and organization(s) being evaluated that could bias the evaluation. If yes, I disclose the following facts: I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature: Date: November 20, 2017 February 2018 | MEASURE Evaluation Phase IV – Midterm Performance Evaluation: Final Report 150