USAID’S HEALTH EVALUATION AND APPLIED RESEARCH DEVELOPMENT (HEARD) PROJECT EVALUATION REPORT Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) The effectiveness, efficiency, and quality of GH Pro support services and their comparative advantage over other USAID service support mechanisms October 2018 USAID’s Health Evaluation and Applied Research Development Project (HEARD) is funded by United States Agency for International Development (USAID) under cooperative agreement AID-OAA-A-17-00002. The project team includes prime recipient University Research Co., LLC (URC) and sub-recipient organizations. This report was produced for review by the United States Agency for International Development (USAID). It was prepared by University Research Co., LLC (URC) and The University of California, San Francisco’s Global Strategic Information (GSI) Group and was authored by Paul R. De Lay, Samantha Ski, Roger Myrick, Hana Azman, and Neia Prata. Recommended Citation De Lay, P., S. Ski, H. Azman, N. Prata, and R. Myrick. 2018. Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro). Prepared for the United States Agency for International Development (USAID) and GH Pro. Chevy Chase, MD: University Research Co., LLC. Acknowledgements The evaluation team is deeply appreciative of the USAID project management team and of GH Pro staff for their commitment and generous support of evaluation efforts. The team extends a special thanks to survey respondents and interviewees, including clients, consultants, and CORs/AORs of comparable USAID support mechanisms, and to involved project staff from those mechanisms. In addition, the evaluation benefitted greatly from the contributions of Jim Sherry, who was instrumental in the evaluation design and provided strategic guidance throughout; as well as from several Participating Investigators: Danielle Charlet who provided technical oversight; and Abbas Alawieh and Tricia Ryan, who contributed to data collection/analysis and report drafting; and from the wisdom of members of the Quality Reference Group and Strategy Reference Group. USAID’S HEALTH EVALUATION AND APPLIED RESEARCH DEVELOPMENT (HEARD) PROJECT EVALUATION REPORT Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) The effectiveness, efficiency, and quality of GH Pro support services and their comparative advantage over other USAID service support mechanisms CONTENTS Tables and Figures.....................................................................................................ii Acronyms ........................................................................................................................iii Executive Summary...................................................................................................1 Introduction to the Evaluation...........................................................................5 Dates of Evaluation Implementation....................................................................5 Purpose...........................................................................................................................................5 Evaluation Questions.........................................................................................................5 Project Background.............................................................................................................5 Evaluation Question 1...............................................................................................7 Background................................................................................................................................. 7 Evaluation Methods............................................................................................................. 7 Limitations..................................................................................................................................10 Findings.........................................................................................................................................11 Conclusions.............................................................................................................................20 Recommendations............................................................................................................20 Evaluation Question 2...........................................................................................23 Background..............................................................................................................................23 Evaluation Methods..........................................................................................................23 Limitations.................................................................................................................................26 Findings.......................................................................................................................................26 Conclusions............................................................................................................................30 Recommendations.............................................................................................................31 Evaluation Question 3 ..........................................................................................33 Background............................................................................................................................. 33 Evaluation Methods......................................................................................................... 33 Limitations................................................................................................................................ 35 Findings for Each Data Source............................................................................. 36 Overall Findings.................................................................................................................. 38 Conclusions..............................................................................................................................41 Annexes..........................................................................................................................45 Introduction to the Evaluation Annexes........................................................46 0.1 USAID Statement of Work............................................................................46 0.2 GH Pro Midterm Evaluation SOW.......................................................55 0.3 Evaluator Profiles................................................................................................64 0.4. Non-Disclosure and Conflicts Agreements...............................67 1.0 Evaluation Question 1 Annexes..................................................................... 88 1.1 GH Pro Staff Roles................................................................................................ 88 1.2 Assignments Sampled for Document Work Stream Review..90 1.3 Survey Question Types....................................................................................92 1.4 GH Pro Client Survey........................................................................................ 93 1.5 GH Pro Consultant Survey .......................................................................... 98 1.6 GH Pro Staff Survey ..........................................................................................102 1.7 GH Pro Client Interview Guide................................................................107 1.8 GH Pro Staff/Management Interview Guide............................109 2.0 Evaluation Question 2 Annexes...................................................................110 2.1 GH Pro Evaluation Reports Reviewed by QRG ........................110 2.2 List of Quality Reference Group (QRG) Members................113 2.3 Quality Evaluation Standards from PEPFAR, USAID, & Scrivens Meta-evaluation Checklist..........................................................114 2.4 Comprehensive Evaluation Reporting Tools.............................117 2.5 PEPFAR Evaluation Standards of Practice Adherence Checklist..................................................................122 2.6 USAID Evaluation Report Review Template.............................127 2.7 Comprehensive Evaluation Reporting Tool Scores for Evaluations Reviewed.....................................................131 2.8 Qualitative Findings of GH Pro Evaluations Using the Comprehensive Evaluation Reporting Tool, by Evaluation Component: Strengths and Areas for Improvement......................134 3.0 Evaluation Question 3 Annexes................................................................138 3.1 Support Mechanisms Matrix................................................................... 138 3.2 Discussion Guide for Interviews with COR/AORs .............146 3.3 Questionnaire for CORs/AORs of USAID Service Support Mechanisms .......................................................................148 3.4 Strategy Reference Group Participants.........................................151 3.5 Strategy Reference Group: Focused Discussion Questions, with Associated Comments................................................153 ii Evaluation Report TABLES & FIGURES Figure 1. GH Pro’s 515 managed assignments..........................................................................................................................................................................................................7 Figure 2. Staff: Sources of delays in on-time completion............................................................................................................................................................................16 Figure 3. Evaluation methodologies, with the most used noted in red text...............................................................................................................................37 Figure 4. Map of Evaluation Projects.............................................................................................................................................................................................................................39 Table 1. Survey sample sizes and response rates.................................................................................................................................................................................................9 Table 2. Average extent of change for n=8 assignments.............................................................................................................................................................................12 Table 3. Consultant ratings of working with GH Pro........................................................................................................................................................................................15 Table 4. A summary of client ratings on the quality of deliverables and overall satisfaction.......................................................................................17 Table 5. Understanding the variety of evaluation methods used by GH Pro.............................................................................................................................18 Table 6. Overview of GH Pro evaluation reports, and sampling frame by geographic area & programmatic focus...........................24 Table 7. Evaluation quality assessment tools used to score sampled GH Pro evaluation reports........................................................................25 Table 8. Mean and median scores of all evaluations using the Comprehensive Evaluation Reporting Tool, by evaluation component........................................................................................................................................ 27 Table 9. Average cumulative scores of GH Pro evaluations based on the Comprehensive Evaluation Reporting Tool, by various disaggregation categories..........................................................................................................28 Table 10. Summary scores of USAID/non-PEPFAR evaluations using the Evaluation Report Review Template, by USAID evaluation component (N = 17 evaluations)...................................................................................29 Table 11. Summary scores of USAID/PEPFAR evaluations using the PEPFAR Evaluation Standards of Practice Adherence Checklist, by PEPFAR evaluation component (N = 13 evaluations)...........................................................30 Table 12. Comparative USAID service support mechanisms for the Bureau of Global Health.................................................................................34 Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) ACRONYMS ADS Automated Directives System MEASURE Monitoring and Evaluation to Assess and Use AMS Administrative Management Services Results AOR Agreements Officer Representative MEDS Monitoring and Evaluation Design Support APM Assistant Project Manager MERL Monitoring, Evaluation, Research, and Learning ASPIRES Accelerating Strategies for Practical Innovation and Research in Economic Strengthening MERLIN Monitoring, Evaluation, Research, and Learning Innovations ASSIST Applying Science to Strengthen and Improve MICS Multiple Indicator Cluster Surveys Systems MS Mission Support (one of three service areas CCN Cooperating Country National evaluated) CDC US Centers for Disease Control and Prevention P3 Office of Policy, Programming, and Planning CIRCLE Coordinating Implementation Research to PEPFAR US President’s Emergency Plan for AIDS Relief Communicate Learning and Evidence PHIA Population-based HIV Impact Assessment COR Contracting Officer Representative PM Project Manager DEPA MERL Developmental Evaluation Pilot Activity–MERL PMO Project Management Office (related to GH Pro DHIS2 District Health Information System 2 budgeting for core costs) DHS Demographic Health Survey POPTECH Population Technical Assistance Project EPIC Evaluation Process Interest Group PPL Bureau for Policy, Planning, and Learning ERIE Expanding the Reach of Impact Evaluation Project SOAR Project Supporting Operational AIDS Research EVAL–ME Policy, Planning, and Learning– Monitoring and PSA Procurement Support Award Evaluation QRG Quality Reference Group (used to assess evaluations) F&A Finance and Administration RFA Request for Applications FA Facility Access RFP Request for Proposals FACT Foreign Affairs Counter Threat SOW Statement of Work GH Bureau for Global Health SPACES MERL Strategic Program for Analyzing Complexity and GH Pro Global Health Program Cycle Improvement Project Evaluating Systems–MERL GH TECH Global Health Technical Assistance Project SPM Senior Program Manager GHFP–II GHPOD–II GHSI–III Global Health Fellows Program II Global Health Professional and Organizational Development II Global Health Support Initiative III SRG TA TB Strategy Reference Group (used to assess the service support portfolio) Technical Assistance (one of three service areas evaluated) Tuberculosis GSI HEARD Global Strategic Information Health Evaluation and Applied Research Development TCN TDM Third Country National Technical Directive Memo HTSOS High Threat Security Overseas Seminar UCSF University of California, San Francisco IDIQ Indefinite Delivery Indefinite Quantity URC University Research Company IRB Institutional Review Board USAID United States Agency for International Development MCH Maternal and Child Health USN US National iii iv Evaluation Report Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) EXECUTIVE SUMMARY The United States Agency for International Development (USAID) Global Health Program Cycle Improvement Project (GH Pro) is a cost-plus-fixed-fee, five-year contract awarded to prime contractor Dexis Consulting Group with The QED Group, LLC. as the subcontractor. GH Pro has a ceiling value of $84.2 million and a period of performance of July 1, 2014, to June 30, 2019. The purpose of this midterm evaluation was to review GH Pro’s performance to date in the landscape of various support mechanisms funded by USAID’s Bureau for Global Health (GH), with the goal of identifying opportunities to add value, improve program quality and efficiency, and reduce cost. The midterm evaluation concentrated on three areas of service: Mission support (Substitute or Supplemental staff for USAID Missions), technical assistance (technical expertise to conduct assessments or reviews or to support strategic planning, project design, and coordination), and evaluations (independent evaluations of USAID programs and projects, generally at mid-term or end-of-project, in compliance with USAID and PEPFAR evaluation policies and standards). The evaluation team from USAID’s Health Evaluation and Applied Research Development (HEARD) Project has made specific recommendations for each of the three Evaluation Questions encompassed by this evaluation, for corrective action by GH Pro and for a broader readership of those who seek improved methods to evaluate the quality of evaluations. Moreover, there are recommendations for USAID’s ongoing management of GH Pro and the design of a future service support portfolio. EVALUATION QUESTION 1 To what extent is GH Pro effective and efficient in meeting stakeholder needs in three key service areas1 (program and project evaluation, Mission support, and technical assistance2 )? During the defined period for the evaluation (7/1/2014 to 12/31/2017), USAID clients3 in GH and Missions asked GH Pro to complete 515 total assignments. GH Pro completed 328 assignments, all but two of which were in three main service areas: evaluation (18% [54] of the 328 assignments), Mission support (52% [184] completed), and technical assistance (30% [88] completed). To assess effectiveness and efficiency, the evaluation team utilized three integrated methodologies: evaluator-facilitated document review, surveys, and key informant interviews. Major Findings Accessing GH Pro and Finalizing the Statement of Work for an Assignment When USAID GH and Missions required evaluations, technical assistance, or Mission support, those who utilized GH Pro cited three highly prevalent themes that guided their decisions: ease of use, accessibility, and flexibility. While the perspective of surveyed GH Pro staff (n=24) was that statements of work (SOWs) frequently required substantial revision, surveyed USAID clients4 (n=162) disagreed: 16.7% of surveyed staff versus 88% of surveyed clients stated “no” when asked if significant changes were made to the SOW from the initial to final version. 1 A service area refers to a category of assistance (e.g., Mission support, technical assistance, or evaluation services) that GH Pro and similar mechanisms provide for the United States Agency for International Development (USAID). 2 The majority of GH Pro’s assignments have fallen into these three categories, which are therefore the focus of the evaluation. Of the 328 assignments GH Pro completed over the evaluation period, none were in the research coordination category and only two concerned support for conferences/meetings. 3 A USAID “client” is defined in this evaluation as a USAID staff person requesting a specific service from a project mechanism. 4 A USAID client refers to any USAID staff person requesting a specific service from a project mechanism. 1 2 Evaluation Report Implementing Assignments and Quality of Final Products Overwhelmingly, clients surveyed stated that consultants hired through GH Pro were suitably knowledgeable and experienced in their relevant area, and their last GH Pro assignment was completed per specifications outlined in the final SOW. A majority of clients stated that GH Pro recruitment efforts yielded a consultant with the necessary qualifications without being overqualified, while a quarter of those surveyed stated that consultants identified were only somewhat appropriate. When analyzed by the type of services used, GH Pro clients were the least satisfied with evaluation services. Clients reported problems with team member qualifications and/or fit 15% of the time for evaluation services but only 11% of the time for Mission support and 10% of the time for technical assistance. Similarly, clients were less likely to report “no bottlenecks or problems” for evaluation services (27% of the time) compared to Mission support (36%) and technical assistance (34%). An average of 93% of respondents said they would use GH Pro again, with several clients pointing to its unique role among GH mechanisms as a “one-stop shop” for support. Cost of GH Pro Services Most clients rated the cost of the final deliverable provided by GH Pro as very costly. Despite the perception of high costs, clients recognized that GH Pro did offer one cost efficiency over other service mechanisms: GH Pro charged the client only that which was expended when the cost was lower than the full cost stated in the original estimate. GH Pro charges a project-management-office (PMO) cost of 25% on top of each assignment’s costs to cover operational expenses. Because GH Pro does not have core funding, it relies on the PMO to fund senior staff, like the full-time senior recruiter, critical finance and administrative support roles, and standard operating costs of maintaining the office. Major Conclusions Within the broader service support portfolio, GH Pro provides a unique set of operational and technical functions that can be accessed in an efficient and rapid manner. GH Pro can offer temporary Mission staff, targeted technical assistance, and a special niche in delivering low-cost, short-duration, performance evaluations. GH Pro was found to receive strong performance ratings from clients in terms of flexibility and ease of access and use, and from consultants in terms of ease of working with and responsiveness of GH Pro. A majority of clients rated the quality of GH Pro deliverables to be good and assignments were completed per specifications outlined in the final SOW, although due to a number of factors roughly half assignments are not completed in the specified timeframe. GH Pro clients tended to be the least satisfied with the outputs of assignments in the evaluation service area and there was a common perception among clients that GH Pro was “expensive.” Limitations Fewer follow-up interviews with clients were conducted than planned due to low response rates among the sub￾sample of survey respondents contacted, though responses to open-ended survey questions offered ample qualitative data on key evaluation components. While the evaluation team intended to offer a more extensive analysis of the GH Pro’s use of local solutions, the team encountered limitations with determining citizenship and residency designations among assignment team members. Summary of Recommendations Recommendations to GH Pro focused on expanding senior technical and recruitment capacity staffing along with continued and enhanced focus on client engagement to clarify expectations, navigate bottlenecks, and better define costs. Recommendations to USAID suggested the importance of core funding, the challenge of targeting small businesses in GH Pro-like mechanisms, and the importance of strengthening and institutionalizing technical peer review of evaluation SOWs. Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) EVALUATION QUESTION 2 To what extent are the evaluation quality standards set by USAID and the US President’s Emergency Plan for AIDS Relief (PEPFAR) employed and achieved by GH Pro? GH Pro completed 59 evaluations from the beginning of the Project in 2014 to mid-2018. The process to review GH Pro’s evaluations consisted of sampling from the 59 evaluations that GH Pro had completed by mid-2018 to create a subset of 30 evaluations for review. The team then created a Quality Reference Group (QRG) of eight evaluation experts to review the extent to which the sample met evaluation quality standards according to three tools: the current USAID evaluation standards, the PEPFAR guidelines, and a Comprehensive Evaluation Reporting Tool. This last tool consolidated a list of standards derived from Scrivens5 and other leading evaluation experts and served as an overall best practice review instrument. Evaluations were divided into two categories: those conducted according to USAID evaluation standards and those expected to adhere to PEPFAR evaluation standards. Major Findings The evaluations conducted by GH Pro tended to emphasize the measurement of sustainability, quality, scale, and cost. All were performance evaluations, with an average cost of $300,000 and a median duration of 266 days. Findings from these evaluations supported recommendations for course correction and designs for future work. There was a strong emphasis on the identification of effective models of implementation to achieve results. Out of 17 evaluations using USAID/non-PEPFAR standards, almost all met these standards across the following components: introduction and purpose, information and background, evaluation questions, findings and conclusions, and recommendations. The fewest number of evaluations (12) met USAID/non-PEPFAR standards for the component called Annexes. Out of 13 evaluations using USAID/PEPFAR standards, most scored highly in the following components: clearly state evaluation questions, purpose, and objectives; use appropriate evaluation designs, methods, and analyses; construct data collection and management plans; and use findings for program improvement. The components with the fewest number of evaluations meeting PEPFAR standards were the following: address ethical considerations and assurances (5); identify resources and articulate budget (7); and disseminate results (7). QRG members noted several strengths of the GH Pro evaluations. Most notably, each evaluation’s purpose and questions were clearly defined, with the relationship between the questions and the purpose explicit. Additionally, the methodology and approach, as well as the limitations, were clearly described. Overall, evaluators tended to rate the GH Pro evaluations with average scores for quality across most components. This finding was consistent across the three evaluation tools. Cumulative evaluation scores varied little by cost and the year the evaluation was conducted, and scores varied only slightly by methodology. End-term evaluation scores were slightly higher than midterm scores. Major Conclusions The evaluation team concludes from the findings that all GH Pro evaluations that were assessed met the criteria for quality, but very few excelled. A key issue here is the rigor of GH Pro evaluation methods and the level of resources made available (including the budget and the time to complete evaluations). Methodologically, the evaluations tended to use simple, though legitimate, methods and analytic approaches. Funding for the evaluations tended to be somewhat minimal compared to the scope of the projects being evaluated. To increase rigor, evaluators would need to expend more resources. 4 Scrivens, M. (2011). Evaluating evaluations: A meta-evaluation checklist, 6th edition. Retrieved from http://michaelscriven.info/images/ EvaluatingEvals-Checklist.pdf 3 4 Evaluation Report Limitations When using the three review tools, not all of the included components applied to all the evaluation reports that were reviewed. Additionally, each evaluation was assessed by only one QRG member. Finally, we were unable to assess the extent to which data were used to inform program improvement, as this was beyond the scope of the three Evaluation Questions. Summary of Recommendations Recommendations for projects that perform evaluations stress the need for reports to include more information on evaluators, ethical concerns, budgets, and more detailed program improvement plans. Recommendations for USAID include the need for increased focus on dissemination strategies, provision of technical assistance to USAID requesters of evaluations, increased evaluation resources and more routine reviews of evaluation quality. EVALUATION QUESTION 3 What is the added value of GH Pro to USAID’ Global Health Bureau and Missions for the three key service areas (evaluation, Mission support, technical assistance), considering other, existing mechanisms? The goal of Evaluation Question 3 was to describe the role and unique advantages that GH Pro plays within the broader portfolio of 17 service support projects that were reviewed. Each of these projects provided assistance in one or more of the three service areas: Mission support, technical assistance, and evaluations. Data for this Question was collected from four sources: document review, key informant interviews, structured survey, and a strategy reference group. Major Findings and Conclusions GH Pro is seen by USAID clients as flexible, responsive, and able to recruit competent, knowledgeable consultants. It also has the capacity to track and manage financial resources from multiple funding sources. The overall analysis of the broader service support portfolio indicates that the 17 projects provided critical support services to USAID over the past two to five years. Many of these projects were in their second or third iteration, building on previously successful models. Most projects focused on one or two of the specific service areas (GH Pro is one of few exceptions). Among the providers in the evaluation service area, there seems to be a rational spread of mechanisms that have different areas of focus and evaluation capacities—from low-cost, short-duration performance evaluations to high-cost, long-duration impact evaluations. There are a number of sector-specific evaluation providers (e.g., HIV/ AIDS, behavior change) and a set of evaluation providers that have been set up to experiment with more innovative evaluation methods to provide rapid answers to program operation challenges. Summary of Limitations The evaluation had neither the mandate nor the resources to assess the effectiveness, efficiency of operations, or quality of outputs for each of the 17 service support projects. Therefore, assessing comparative advantages across the service support portfolio was limited to defining areas of focus, business model, staffing, and operational systems. Summary of Recommendations Recommendations included suggestions to continue the analysis of the entire service support portfolio, consider establishing a cross-bureau procurement strategy for follow-on projects, increase the detail of information about the various projects available to potential USAID clients, and closely track and engage in the potentially evolving views on the role of evaluations in the field of development. Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) INTRODUCTION TO THE EVALUATION Evaluation Implementation: October 1, 2017–August 31, 2018 PURPOSE The purpose of this midterm evaluation of the United States Agency for International Development (USAID) Global Health Program Cycle Improvement Project (GH Pro) is to review the Project’s performance to date in the landscape of various support mechanisms funded by USAID’s Bureau for Global Health (GH), with the goal of identifying opportunities to add value, improve program quality and efficiency, and reduce cost. (See Annex 0.1: USAID Statement of Work and Annex 0.2: GH Pro Midterm Evaluation SOW.) This evaluation encompasses two distinct but closely related purposes, with greater weight placed on the second of the two: • Purpose 1: Improve the project management and technical support that GH Pro offers for the remainder of its implementation. • Purpose 2: Inform any potential future service support projects. The primary audience for this evaluation is both GH Pro (vis a vis Purpose 1) and USAID GH staff involved in designing the follow on procurement to GH Pro, or other similar service support mechanisms (vis a vis Purpose 2). Secondary audiences include current and future implementing partners of similar service support projects and their USAID focal points in Washington and in Missions. EVALUATION QUESTIONS The evaluation set out to answer three major questions: 1. To what extent is GH Pro effective and efficient in meeting stakeholder needs in three key service areas (program and project evaluation, Mission support, and technical assistance6 )? 2. To what extent are the evaluation quality standards set by USAID and the US President’s Emergency Plan for AIDS Relief (PEPFAR) employed and achieved by GH Pro? 3. What is the added value of GH Pro to the USAID GH and Missions for the three key service areas (evaluation, Mission support, and technical assistance), considering other, existing mechanisms? Distinct methodological approaches were used to address each of these questions, which led the evaluation team to split the evaluation into three operational components, with each component’s methods, findings, conclusions, and recommendations reported in a cohesive section. PROJECT BACKGROUND GH Pro is a cost-plus-fixed-fee, five-year contract awarded to prime contractor Dexis Consulting, LLC (Dexis) with a ceiling value of $84.2 million and a period of performance from July 1, 2014, to June 30, 2019. Dexis implements the GH Pro with a subcontractor, The QED Group, LLC (QED). GH Pro is a similar procurement mechanism to several preceding contracts awarded over the past three decades, including the Global Health Technical Assistance Project (GH TECH), Population Technical Assistance Project (POPTECH), Monitoring and Evaluation Design Support (MEDS), Applying Science to Strengthen and Improve Systems (ASSIST) Project, and the Synergy Project. It is awarded and managed out of USAID GH in the Office of Policy, Programs, and Planning (P3). GH Pro was designed to offer several services to USAID GH Headquarters- and Mission-based clients, including project evaluation, Mission support, technical assistance, global health program and research management, and support for conferences and meetings. Due to lack of demand for the latter two service 6 The majority of GH Pro’s assignments have fallen into these three categories, which are therefore the focus of the evaluation. Few or no assignments were carried out by GH Pro in the other two project areas for which GH Pro was established. Of the 328 assignments GH Pro completed over the evaluation period, none were in the research coordination category and only two concerned support for conferences/meetings. 5 6 Evaluation Report areas, this evaluation only focused evaluation, Mission support, and technical assistance. Annex 1.1 describes the standard operations for submitting and implementing an assignment. These involve: • Evaluation: independent evaluations of USAID programs and project, generally at mid-term or end-of-project, in compliance with USAID and PEPFAR evaluation policies and standards; • Mission Support: short- and medium-term (up to six months) substitute or supplemental USAID staff to provide managerial and technical support to US Missions abroad to fill temporary gaps and provide surge capacity; • Technical Assistance: technical expertise to conduct assessments or reviews or to support strategic planning, project design, and coordination. USAID clients7 wishing to engage GH Pro services submit a Scope of Work (SOW) to GH Pro’s Contracting Officer Representative (COR). Once approved by the COR, the SOW is submitted to GH Pro. The SOW includes the parameters of the work needed, the expertise needed, including number, qualifications, and level of effort of desired team members, timeline, and deliverables. GH Pro reviews each SOW and provides feedback, develops a cost estimate, and recruits appropriate consultant(s) to carry out the work. Once the SOW, cost estimate, and key consultant(s) are agreed to by the client, the COR approves the Technical Directive Memo (TDM), which is the administrative action that allows the money for the assignment to be obligated and transferred to GH Pro, and authorizes GH Pro to begin carrying out the assignment. Once the TDM is approved, GH Pro can enter into contract with consultants and begin logistical arrangements related to assignment execution. Annex 1.1 describes the roles of various GH Pro staff in operating the Project. GH Pro is situated in the constellation of 17 service support organizations available to GH (and in some cases to the agency as a whole). This portfolio of service support mechanisms is discussed in the Evaluation Question 3 section, which examines the comparative advantage of GH Pro. Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) EVALUATION QUESTION 1 To what extent is GH Pro effective and efficient in meeting stakeholder needs in three key service areas (program and project evaluation, Mission support, and technical assistance)? BACKGROUND At the start of this evaluation, GH Pro had completed three of five Project years. This evaluation examined GH Pro performance from its start through December 31, 2017, with the aim to understand GH Pro’s operational effectiveness and efficiency to inform potential process improvements to be implemented in the final phase of the Project, and to help USAID understand what has and has not worked well within the GH Pro mechanism to help inform the design of future mechanisms. During this period, USAID GH and Missions asked GH Pro to complete 515 total assignments. GH Pro completed 328 assignments, all but two of which were in three main service areas: evaluations (18% [54 completed evaluations]), Mission support (52% [184 completed assignments]), technical assistance (30% [88 completed assignments]). Non-completed assignments were either currently active, in planning, or delayed or canceled by the client (see Figure 1). Figure 1. GH Pro’s 515 managed assignments 1% 32% Completed Delayed mission support 14% In planning 14% Cancelled 18% Completed technical assistance 18% Active 0% 11% Completed Completed meetings evaluations EVALUATION METHODS Evaluation Question 1 sought to answer to what extent GH Pro is effective and efficient in meeting stakeholder needs in the three service areas under review. Effectiveness is the ability of the Project to accomplish its objectives. The objective of GH Pro is to provide GH Offices, Regional Bureaus, and Missions with ready access to high-quality, external, technical expertise—ultimately, to support and evaluate programs contributing to the achievement of the US Government’s Global Health Initiative targets. Efficiency is related to how well resources are used to achieve the outcomes. Elements that were examined to evaluate efficiency included the time needed for the planning, launch, and completion of assignments; examination of cost; and client perceptions of both general process efficiency and cost advantage. The evaluation team examined several elements involved in carrying out each GH Pro assignment to assess the ability and effectiveness of GH Pro to fulfill its objectives. These elements were the following: a) Support to the client in further developing and finalizing a clear and actionable statement of work (SOW) to guide accomplishment of deliverables within the time frame and budget, based on the client’s original SOW submission b) Mobilization of internal and external resources to carry out the SOW c) Completion of the SOW per specifications d) Satisfaction of the client resulting from the process and the outputs in a) through c) The evaluation team utilized three integrated methodologies to assess effectiveness and efficiency: evaluator-facilitated document review, surveys, and key informant interviews, which are described in turn in the next three sections of this report. 7 8 Evaluation Report Evaluator-Facilitated Document Review The following documents were reviewed: • Key GH Pro documents, including the assignment tracker (up to date through December 31, 2017), operational templates, contracts, routine end-of￾assignment satisfaction surveys, and quarterly and annual reports—The purpose of analyzing these documents was to understand the operational flow of assignment requests, potential roadblocks, and the adequacy and clarity of guideline information. • Self- Assessment report led by Connie Carrino, with the participation of GH Pro staff, January to May 2017— The Self-Assessment exercise was a useful model for assessing key aspects of the effectiveness and efficiencies of GH Pro. This evaluation built on the Self￾Assessment exercise, with a renewed emphasis on quality, efficiency. • Assignment Document Work Stream – The evaluation team assembled a sample of the full document stream for 32 assignments, in part to understand (in conjunction with key informant interviews) the workflows for individual assignments and the roles of specific GH Pro staff in the assignment process. The team identified areas of potential inefficiency and/or roadblocks, delays, added costs, or impact on the ultimate quality of the products. The documents needed to initiate, approve, fund, and finalize an assignment were the following: – Initial SOW from the USAID client, along with cost estimates – The technical directive memo (TDM), which once approved allows GH Pro to begin assignment implementation, as well as all it includes (finalized SOW, cost estimate, source of funding, time frame, and team member and deliverable descriptions) – Documentation kept by the project manager (PM) or leadership team during the assignment – Any available guidance provided to consultants – Any modifications to the TDM, especially if SOWs were revised after approval – Draft and final deliverables – Satisfaction surveys or consultant surveys, when available Assignment Sample Selection for Document Work Stream Review Extreme Case Sampling (e.g., best and problematic cases) and Purposeful Random Sampling were used to select sample assignments for document review. For the Extreme Case Sampling, GH Pro was asked to identify assignments that represent one best case and one problematic case for each assignment type (i.e., evaluations, Mission support, and technical assistance). The full Assignment Document Work Stream was requested for each assignment selected. Purposeful Random Sampling was used to ensure a range of assignments were examined in the Work Stream review. Assignments were sampled to promote representation from a variety of client types (Mission vs. HQ; geographic regions; frequent, less frequent, and non-repeat requesters), assignment types, and assignment complexity (a combination of cost and duration were used as a proxy for complexity).7 The resulting sample of 32 included 8 evaluations, 15 Mission support, and 9 technical assistance assignments (see Annex 1.2). 7 The process undertaken to determine the sample under Purposeful Random Sampling included the following: (a) All completed assignments listed in the assignment tracker (current as of 12/31/2017) were sorted into three groups by type of assignment (evaluation, technical assistance, or Mission support). (b) Each of those groups was then divided into terciles for cost and duration and then scored as 1, 2, or 3 accordingly. For both cost and duration, those in the lowest terciles were scored 1, those in the middle terciles were scored 2, and those in the highest tercile were scored 3. (c) Cost and duration scores were then added together to produce a total complexity score. Scores of 5 or 6 were rated “high complexity,” scores of 3 or 4 were rated “medium complexity,” and scores of 1 or 2 were rated “low complexity.” (d) The sampling frame was used to determine weights by cost and duration for each type, and those weights were applied to each complexity level to determine the number of each type and complexity level to be included in the sample (with the target of achieving a sample of about 10% of total completed assignments). (e) Assignments were sorted by type and complexity level and randomly sampled within each of those categories by assigning a random number to each assignment, sorting by the random numbers, and taking the first assignments listed up to the number being sampled. Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) - - Analysis of Sampled Evaluation Assignment SOWs As part of the review of the Assignment Document Work Stream, the evaluation team (sometimes referred to collectively as “we” in this report) analyzed the extent to which SOWs changed from when they were initially submitted to GH Pro to when they were included in the approved TDM. We also documented the types of evaluation and methods utilized among sampled assignments. • Extent of changes to SOWs: The extent of change in each of six SOW components (purpose, evaluation questions, methods, analytic plan, products, and expertise/experience) was determined by comparing the SOW that was submitted initially to the final SOW included in the TDM for each evaluation assignment. For the six SOW components, a score of 0, 1, or 2 was assigned. No substantive change (score of 0) meant that either no or minor edits were made. Significant change (score of 1) meant that content was substantially tweaked or expounded upon. Extensive or complete change (score of 2) was characterized either by sections that were completely new or entirely overhauled in the final SOW. Two scorers independently scored each set of SOWs and then compared their work for discrepancies. There were discrepancies in 12 of the 48 assigned scores (i.e., the two scorers did not vary in their scoring of 36 of 48 components), and the final score was determined based on deliberation and consensus. Table 1. Survey sample sizes and response rates • Types of evaluations and methods: Sampled assignment SOWs were also reviewed to chart (a) the type of evaluation selected (i.e., performance or, if PEPFAR-funded, process, outcome, impact, or economic) and (b) methodological categories chosen and described as part of the plan for a given evaluation assignment. Surveys The evaluation included surveys of clients, consultants, and GH Pro staff (Table 1) using QuestionPro, an online survey software application. A specific emphasis was placed on improving response rates and the quality of responses to ensure that findings were clear and actionable. For surveys, the sampling was conducted as follows: • GH Pro Client Survey—sent to all USAID requesters of GH Pro services and to selected “non-requester” Missions; the non-requestor Missions were identified in coordination with USAID • GH Pro Staff Survey—sent to all staff currently working at GH Pro • GH Pro Consultant Survey—sent to all consultants who completed work on GH Pro assignments Total Sent Bounce Backs Total Delivered Total Completed Response Rate* Self Assessment Response Rates** Client 643 128 515 162 31% 16% Staff 26 0 26 24 92% 81% Consultants 484 16 468 279 60% 51% * Total Completed divided by the Total Delivered ** Refers to a 2017 GH Pro Self-Assessment 9 10 Evaluation Report The GH Pro Client Survey was sent to 643 clients by the P3 director to encourage participation, resulting in 128 bounce￾backs (i.e., email addresses that were no longer valid) and 162 completions.8 The majority (90.12%) of responses were from respondents located in the US at the time of the survey. The GH Pro Staff Survey was sent to 26 staff, of which 24 completed the survey.9 The GH Pro Consultant Survey was sent to 484 consultants. With 16 bounce-backs, the survey was completed by 279 people.10 Response rates achieved were generally slightly higher than those achieved through the GH Pro Self-Assessment conducted in 2017. A breakdown of survey question types and their applicability to GH Pro’s effectiveness and efficiency, as well as the surveys themselves, can be viewed in Annexes 1.3–1.6. Key Informant Interviews To understand respondent and stakeholder perceptions of efficiency and effectiveness in greater detail, group discussions or individual interviews were held with a total of 25 GH Pro staff, 5 USAID GH Pro management team members, and a subsample of 4 Client Survey respondents who provided their emails for potential follow-up and either reported very positive or very negative experiences with the mechanism. A standardized questionnaire was developed for each of the three groups and adapted as needed (e.g., to tailor questions based on GH Pro staff functions or client responses). See Annexes 1.7-1.8 for interview guides. Qualitative Analysis of Interview and Focus Group Data One evaluator in each group or interview was assigned to produce notes, and audio recordings were consulted to fill gaps in notetaking. Notes were edited to generate a memo for each discussion. In most cases, memos were sent to group or interview participants to allow participants to provide additional feedback. Final memos were reviewed and coded using an inductive approach to identify recurring themes within the data and to achieve data reduction, reorganization, and representation.11 Conventional data analysis was conducted to derive coding categories directly from the text data. Triangulation was then employed to check the findings of this qualitative analysis with those of the surveys, including qualitative text responses to specific survey questions. LIMITATIONS Most of USAID clients who completed the survey were located in Washington, either because they were permanently based there or had recently returned from Field Mission assignments. This could cause some bias in the overall response analysis. A subsample of nearly 20 Client Survey respondents was contacted to request a follow-up interview; however, only four responded. Of the four who responded, all were based at USAID/Washington, and none had utilized GH Pro for Mission support (only for technical assistance and evaluations). Clients provided extensive responses to open-ended survey questions, which offered ample qualitative data regarding key components of the Evaluation Question. Nonetheless, additional interviews, particularly among field-based clients and clients who had requested Mission support in addition to technical assistance and evaluations would have provided additional insights related to client experiences with GH Pro services. Finally, while the evaluation team intended to offer a more extensive analysis of the GH Pro’s use of local solutions, the team encountered limitations with determining US National (USN)/ Third Country National (TCN)/ Cooperating Country National (CCN) designations among assignment team members. Efforts were made to identify distinctions using curricula vitae and data in cost estimates, but this information was unavailable (GH Pro does not keep lists of consultants by nationality). Improved monitoring would enable a better understanding of the use of local individuals and institutions. It should be noted that the Scope of Work the covers GH Pro did not stipulate that this information would be collected. 8 The GH Pro Client Survey took an average of 17 minutes to complete. 9 The Staff Survey took an average of 48 minutes to complete. 10 The Consultant Survey took an average of 28 minutes to complete. 11 Roulston, K. (2014). Analysing interviews. In Flick, U. (Ed.), The Sage Book of Qualitative Data Analysis (ch. 20). Thousand Oaks, CA: SAGE Publishing. Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) FINDINGS A. Accessing GH Pro Finding 1.1: Overall, clients found GH Pro to be flexible and easy to access for services, with clear templates and the necessary information readily accessible Clients repeatedly praised GH Pro’s ease of use, submitting comments such as the following: “very easy and very quick way to identify technical or administrative assistance,” “it is quick and flexible,” “it’s convenient,” and “quick and easy.” A handful of clients mentioned the “quality” of GH Pro’s work as an advantage, and others mentioned their “pool of expertise and availability.” Another client echoed this sentiment stating, “GH Pro consultants have experience working in the region.” The majority of client respondents (90%) either agreed or strongly agreed that they had the information they needed to engage GH Pro’s services. Nearly 83% of client respondents also agreed (61%) or strongly agreed (23%) that the SOW template was easy to use. Clients (8.5%) who disagreed indicated that the template could be long and difficult to navigate and that the tables were at times cumbersome or difficult to work with. Surveyed clients were asked to select up to three choices indicating why they selected GH Pro for their assignment. The most commonly selected answer was “ease of contracting” (26.9%), followed by “flexibility” (17.4%), “past experience with GH Pro” (16.7%), and “referral from a colleague” (13.9%). Overall, throughout client survey and interview responses, the prevalent themes guiding USAID GH’s and Missions’ decisions to utilize GH Pro included ease of use, accessibility, and flexibility. That 73% of client respondents had utilized GH Pro for two or more assignments (30% for four or more) may speak to these widely held perceptions of GH Pro. In addition to the availability of necessary information and ease of use of templates, accessing GH Pro does not require the same high bureaucratic transaction costs that other mechanisms do (e.g., some require the issuance of task orders and competition for awards). In contrast to other mechanisms, the decision of a USAID HQ or Mission-based client to utilize GH Pro can be as simple as the relevant focal point(s) (including the relevant contracting officer representative or agreements officer representative [COR/ AOR]) developing an SOW and sharing it with the GH Pro COR for review and submission to GH Pro. From the perspective of GH Pro staff, the absence of competition requirements for each assignment has the benefit of allowing for a more collaborative and therefore more efficient process for finalizing SOWs and budgets. B. Finalizing the Scope of Work Finding 1.2: The technical capacity of GH Pro’s staff to support the development of sows was limited Upon receiving an SOW, GH Pro staff strive to support the client in arriving at a final version. GH Pro’s reviews of SOWs through a project management lens focus on ensuring the work can be accomplished within the budget and time frame. GH Pro’s technical review of SOWs, particularly evaluation SOWs, is constrained due to GH Pro’s limited technical staffing (i.e., one staff member with evaluation design expertise) and the high volume of assignments. In addition, the evaluation team’s analysis revealed a wide range of technical content across the SOWs that GH Pro manages; no single person could be an expert in such a wide range. Clients reported limitations in the technical attention that GH Pro could give each SOW prior to assignment initiation. The consultants who eventually serve as the technical experts for each assignment cannot be contracted and formally engaged in the process until the SOW is finalized and the TDM is signed and approved. The absence of core funding prohibits GH Pro from having a broader set of technical experts on staff or retainer to support this process. Both USAID and GH Pro staff noted that engaging technical expertise during SOW development would bring increased value to GH Pro assignments. The inability of GH Pro to solicit broader and deeper technical inputs in the SOW development process limits the use of creative 11 12 Evaluation Report methods, sampling strategies, and other design elements in GH Pro evaluations, which could enhance evaluation quality and rigor. Finding 1.3: The quality of the evaluation sows that GH Pro received varied widely A further challenge GH Pro faces in supporting the development of SOWs is the variation in the quality of the SOWs initially received. The evaluation team found that USAID GH and Missions each have their own process for reviewing and vetting evaluation scopes before they are submitted to the implementer. Some entities, such as GH’s Office for Population and Reproductive Health, have a strong and institutionalized process for committee review, where every evaluation scope is reviewed and given feedback by a six-member committee of evaluation experts. Other offices and Missions are meant to adhere to a minimum standard of review by two peers, but this standard may be inconsistently implemented, and Missions in particular may have inconsistent access to reviewers with evaluation expertise. This variation in review processes means a proportion of SOWs received by GH Pro have deficiencies in the evaluation design (e.g., the evaluation questions cannot be realistically answered by the design laid out). In these cases, GH Pro works with the requesting office, as well as country support teams and evaluation focal points at HQ, to refine and improve the SOW that will be in the final TDM. GH Pro staff reported that this process is a critical and time-consuming component of project initiation, whether GH Pro is providing technical inputs to improve the evaluation design or supporting client efforts to fill in gaps. Interestingly, while the GH Pro staff perspective was that SOWs frequently require substantial revision, the clients surveyed disagreed, with 88% of surveyed clients vs. only 16.7% of staff respondents replying “no” when asked if significant changes were made to the SOW from the initial to final version. The evaluation team analyzed the extent of changes needed to finalize evaluation SOWs. The analysis found that—for several critical, design-related SOW components— “substantial” (score of 1) to “comprehensive” (score of 2) change was needed during the process of SOW negotiation (i.e. from original to approved SOW); see Table 2. The average change score across sampled assignments and SOW components was 1.21. The most significant change in SOW components was seen in components that described who and what was needed for clients to arrive at their goals: Expertise/Experience (1.63) and Products (1.63). The Analytic Plan (1.50) and Methods (1.38) components, which are core to the evaluation design, scored higher on average than the “significant change” score. Finally, components dealing with why the client is requesting an evaluation, Purpose (0.38) and Evaluation Questions (0.75), scored lowest in terms of the extents of change. According to GH Pro staff, the principal reason changes were made to a given SOW was that more specificity was needed (33.3%), followed by incomplete requests (22.2%) and unrealistic expectations (22.2%; e.g., too much was requested in too little time, or the scope was too great for the proposed budget/time frame). All staff stated that the final SOW adequately addressed the reasons changes were needed. Consultant respondents reported that final SOWs were clear and answered all questions about the consultant and GH Pro’s roles and responsibilities, with over 88% strongly agreeing (43.7%) or agreeing (45.5%). Table 2. Average extent of change for n=8 assignments SOW Component Score Purpose 0.38 Evaluation Questions 0.75 Methods 1.38 Analytic Plan 1.50 Products 1.63 Expertise/Experience 1.63 Average across all components for n=8 1.21 assignments Note: As a reminder, for the six SOW components, a score of 0, 1, or 2 was assigned to indicate the extent of change from the original SOW submitted to GH Pro by the COR to the ultimate, approved SOW in the TDM. No substantive change (score of 0) meant that either no or minor edits were made. Significant change (score of 1) meant that content was substantially tweaked or expounded upon. Extensive or complete change (score of 2) was characterized either by sections that were completely new or entirely overhauled in the final SOW. Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) Nonetheless, some consultants expressed concerns with working within SOWs that they did not have the opportunity to co-develop and did not feel were technically sound. In those situations, GH Pro staff suggested to consultants that approved SOWs could be seen as guides that could be modified in consultation with the client when needed. C. Implementing Assignments Finding 1.4: The diverse range and uniqueness of skill sets required by clients made recruiting qualified consultants one of GH Pro’s biggest challenges Recruitment for assignments is a process that takes a great deal of time and energy. Project management staff and the senior international recruiter reported it a challenge to identify the specific and unique/rare skill sets requested by clients, often on short timelines. At times, clients are faced with a trade-off between timeliness and quality of match. Sometimes, they chose to wait when the requested skill set could not be found immediately; other times, they settled for a less perfect fit. Challenging assignments required recruiters to consider a combination of factors—language (French and Portuguese speaking ability can be especially challenging), clinical experience, experience with community-based interventions, and/or specific topical expertise are among the qualifications that clients may require. Further complicating the process, candidates with the appropriate qualifications also had to be willing and able to be deployed to the designated location and along the timeline in question, and they may have also needed administrative qualifications such as facility access (FA) and completion of High Threat Security Overseas Seminar (HTSOS) or Foreign Affairs Counter Threat (FACT) trainings. Deployments for candidates without FA or up-to-date HTSOS or FACT training can be substantially delayed until these qualifications can be met. Another major challenge arose when clients expected senior-level expertise even when the resources available only supported mid-level experience. Additionally, if an assignment was delayed during the planning stage—i.e. a consultant was under contract but unable to work—the consultant may have taken on other work and subsequently have been unavailable or delayed when an assignment got up and running, creating additional challenges. Finding individuals who are both qualified and available can be difficult, but this evaluation found that the hiring of a new senior international recruiter over one year ago resulted in dramatic improvements in GH Pro’s consultant recruiting efforts. With the new recruiter, GH Pro’s JobScience database was expanded to include over 4,200 consultants, and the Project is now able to mobilize more quickly, can frequently meeting short USAID turnaround times, even just one or two days. The evaluation team found that a cadre has been built consisting of experienced GH Pro consultants with frequently needed skill sets and FA and key trainings in place, and GH Pro more regularly employs repeat consultants. Finding 1.5: Clients were largely satisfied with consultants secured via GH Pro; however, many reported utilizing consultants they identified or that were identified by someone other than GH Pro A majority of clients surveyed (69.7%) stated that GH Pro recruitment efforts yielded a consultant with the necessary qualifications without being overqualified, while 26.2% stated that consultants identified were somewhat appropriate, and less than 5% said that consultants were somewhat or very inappropriate. Almost all client respondents, 97.3%, stated that the consultants were suitably knowledgeable and experienced in their relevant area. GH Pro clients indicated that the mechanism worked best, however, when the client already knew a consultant who they wanted to hire and trusted to deliver quality work, rather than when they relied on GH Pro to identify a qualified consultant. Quantitative findings indicated that 38.6% of clients reported prior experience with the consultant hired, and 23.3% reported that consultants were recommended to them by someone other than GH Pro staff. About a fifth (19.1%) of clients reported that it was GH Pro who identified the consultant hired. Clients reported that GH Pro selected the evaluation consultants 23% of the time, Mission support consultants 13% of the time, and 13 14 Evaluation Report technical assistance consultants 16% of the time. One client mentioned, “Quality of services is highly dependent on the consultants hired. You have to rely on your own personal contacts to determine if a particular consultant is a good fit or will do a high quality job.” From GH Pro’s perspective, respondents said recruiting consultants started with the client; if a client had a preference in mind, GH Pro staff usually began there. Even if a consultant is referred by USAID, GH Pro is still required to determine availability , check references, and verify salaries. Also, GH Pro must also submit a total of three potential consultant CV’s for each position. If a preferred consultant was not readily identified, GH Pro considered people in its internal networks, followed by a search of the GH Pro consultant database (e.g. JobScience), Devex.com, and the Project’s monthly LinkedIn subscription. It has been previously noted when the USAID client did not have a specific recommendation for a consultant, that GH Pro’s recruitment of highly competent consultants has improved over time. Finding 1.6: Facility access requirements delayed consultant mobilization and limited the use of local resources Many GH Pro assignments require consultants to gain FA to overseas Missions in order to carry out the work. Findings indicate that, with few exceptions, FA requires consultants to be US citizens or green card holders with three years of residence in the US, which limits candidate pools, often excluding qualified non-US citizen candidates and precluding the ability to use local talent. This represents a potential roadblock in USAID’s efforts to increase local capacity and the use of local resources. Early in the life of GH Pro, obtaining FA for consultants was a difficult and lengthy process, described by one GH Pro staff member as “a super slow, black box.” Often performed in Washington or by regional security officers, it would usually take three to four months. Capacity for FA was something that the Project had to develop rapidly. Following concerted effort, GH Pro staff streamlined the process, which now typically takes four to six weeks, and GH Pro has built up a cohort of qualified, experienced consultants for Mission support who already have FA clearance, which is critical to responding quickly. FA clearances obtained by GH Pro consultants are valid for the life of the Project. The follow-on will not be able to leverage the benefits of the FA-approved cohort GH Pro has developed, and FA will need to be secured anew for each consultant. Finding 1.7: Negotiation of consultants’ rates presented challenges stemming from USAID rules and regulations and gender wage gaps GH Pro staff widely reported that consultant rate determinations were significantly constrained by three-year biodata requirements and the inconsistent application of USAID requirements by the Contract Officer. These issues combined to create significant challenges for GH Pro staff in securing desired consultants, even USAID-preferred consultants, who reported turning down assignments due to low daily rates. A consultant’s refusal of daily rates, in turn, presented significant challenges for GH Pro project management staff who then had to restart recruitment for that position. In addition, Missions wrote to GH Pro pressing for exceptions to the daily rate limitations. In some cases, GH Pro had no room to negotiate, and clients did not always understand the rigidity of their own Agency’s policies around daily-rate determination (in some cases, blaming GH Pro for failing to get approval for the daily rate needed to secure the consultant when GH Pro’s hands were tied by USAID regulations). GH Pro staff observed the limitations in relying on biodata. For example, one limitation was the perpetuation of gender wage gaps between men and women of equal experience who were, in some cases, performing the same roles together on the same teams, with men earning significantly more than women. Another observed limitation was that consultants in low- and middle-income countries were more likely to have difficulty providing sufficient documentation of their salary history, resulting in a disproportionate likelihood of these consultants being offered a rate below what their education and experience would indicate. There existed a sentiment among staff respondents that using past salary to determine current salary, without taking into consideration additional factors of experience and education, was unfair. Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) These issues at times caused tensions once teams came together in the field. Finding 1.8: GH Pro’s use of local solutions was not tracked With the aid of the senior international recruiter, GH Pro improved its capacity for recruiting local consultants, i.e., CCN or TCN consultants. In the context of evaluations, non￾USN consultants are more frequently retained in support roles or logistics roles, though GH Pro staff and USAID aim to fill technical roles with local hires. It is more feasible to hire non-USN consultants in countries with a long and robust US Mission presence. In such cases, GH Pro favors an international hire for the evaluation team leader and tries to fill the remaining technical roles with CCNs first, then TCNs, before resorting to additional international hires. Use of CCNs and TCNs generally has budgetary advantages. As noted previously, assignments requiring FA preclude the use of CCNs and TCNs. Document review of sampled assignments revealed that none of the eight sampled assignments included a CCN or TCN team lead. Furthermore, no CCN or TCN consultants qualified for a rate at or near the USAID maximum. The document review could not be used to determine whether other technical roles were filled with CCNs or TCNs because this information was not tracked nor available within available assignment documentation. One additional challenge to increasing the use of local solutions was that local staff responsible for logistics often had much lower daily rates than other team members, and there was a need to advance funds for logistics to Table 3. Consultant ratings of working with GH Pro be able to plan/prepare for implementation. Wiring these large advances to someone making $45 a day or even less, for example, was risky because the funds could not be recouped at the end if they were not spent. Sometimes, an alternative was used—namely, wiring funds to the team lead to be able to recover unused funds if needed—but the team lead had to be willing to manage the funds. Finding 1.9: Consultant experiences with GH Pro were generally positive, though logistical challenges and payment delays were a problem for some Consultants rated their experience with GH Pro highly on several points on a scale of 1 to 5, with 1 being the lowest possible rating and 5 the highest or best possible rating (Table 3). Almost all consultants (95.3%) reported they would work with GH Pro again, and almost all (95.7%) would recommend GH Pro to a colleague. A minority of consultants reported challenges with logistics and payments. Some consultants (38.4%) reported there was additional information that would have been useful to know at the beginning of their assignment (13.7% strongly agreed, 26.7% agreed). Of those who reported wanting additional information, logistical information was what was most lacking, including documentation required for reimbursements, background on colleagues on the team, USAID Mission contact information, and housing information. Some consultants suggested that payment delays and other logistical issues hindered morale and caused major sources of frustration, though they did not directly delay mobilization or the completion of assignments per the SOWs. Criterion Mean Rating (out of 5) Ease of being hired as a consultant with GH Pro 4.26 Ease of working with GH Pro throughout the duration of the assignment 4.42 Responsiveness of GH Pro during the assignment 4.44 Final satisfaction with the experience of working with GH Pro 4.39 Note: The mean score is presented on a scale of 1 to 5, with 5 being the highest possible rating. 15 16 Evaluation Report Finding 1.10: Roughly half of GH Pro assignments were not completed in the anticipated time frame, with delays in recruitment and finalizing deliverables When asked to think about their most recently completed assignment, GH Pro staff reported that roughly half were not completed in the anticipated time frame (47%). The primary reasons for delays were finalizing deliverables (25.0%), identifying and recruiting the team (25.0%), and receiving final approval (21.4%), as shown in Figure 2. These bottlenecks can often be beyond GH Pro’s control. Delays are common given the many variables at play: the need to identify, mobilize, and recruit the right consultant(s); deploy them in varied contexts for which they will be dependent on numerous local actors; and prepare reports that must undergo numerous review steps. Continuously improving the operational systems (e.g., the growth in the consultant database, improvements to the FA request process), skilled troubleshooting, and responsiveness are key ways GH Pro tries to maintain efficiency in navigating bottlenecks, especially in relation to recruitment. Nearly 85% of GH Pro clients highly rated the timeliness of GH Pro responses. Figure 2. Staff: Sources of delays in on-time completion Finalizing deliverables Identifying and recruiting the team Receiving final approval Mobilizing the team Other Finalizing the technical directive memo Conducting the assignment 0% 5% In alignment with GH Pro staff, client respondents also reported identifying qualified team members as the most frequent source of bottlenecks (12%), followed by securing travel documents/visa (8%). Clients reported bottlenecks to be more prevalent in evaluation assignments compared with Mission support and technical assistance. For example, clients reported issues with team member qualifications and/or fit 15% of the time for evaluations but only 11% of the time for Mission support and 10% of the time for technical assistance. Similarly, clients were less likely to report “no bottlenecks or problems” for evaluations (27% of the time) when compared to Mission support (36%) and technical assistance (34%). Finding 1.11: Project management staff managed nearly every aspect of assignment execution, which required an intensely fast pace; finance staff reported responsibilities that under￾leveraged their training and ability Project management staff handled granular details of project assignments with little room to delegate responsibilities. These details included booking hotels, flights, and other travel logistics; recruitment; management of consultants, clients, and progress against SOWs; project 10% 15% 20% 25% 30% Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) launches; budgets; SOW modifications; funding advances; Finding 1.13: GH Pro evaluations fell exclusively and procurements. Staff reported that even working with in the category of performance evaluations and the travel agent required careful oversight. Finance and used largely traditional methods administrative (F&A) staff were responsible for reviewing final cost estimates during the assignment planning phase and for processing payments throughout but were not involved in cost estimate development or assignment-level financial analysis. PMs were expected to navigate logistics and recruitment in different country settings and were expected to be conversant with USAID processes, forms, procedures, and regulations. Many local logistics coordinators were engaged who were adept at managing local logistics but many were not familiar with USAID reporting requirements and regulations. Effective training in these cases was sometimes difficult from afar. D. Quality and Cost of Final Deliverables Finding 1.12: Clients rated the quality of GH Pro deliverables to be good overall Of clients surveyed, 95.8% reported that their last GH Pro assignment was completed per specifications outlined in the final SOW. Overall, 71.9% said that the final assignment products and reports included informative findings, conclusions, and recommendations that were specific and actionable. The client survey sought client perspectives on both the quality of the final deliverable and overall satisfaction with the GH Pro mechanism on a 5-point Likert scale (Table 4). Essentially, all evaluations GH Pro has conducted have been performance evaluations. GH Pro has not been asked to conduct the other types of evaluations outlined within USAID non-PEPFAR standards and policies (i.e., impact evaluations) or USAID PEPFAR standards and policies (i.e., outcome, impact, and economic evaluations, and implementation science and operations research). GH Pro’s evaluation SOW template lists a range of methods that clients can select to include in their evaluation designs. Methods utilized in the evaluations from the document review sample (n=8, denoted A–H in Table 5) were more basic (increasingly complex methods are listed toward the bottom of the table).12 Finding 1.14: GH Pro evaluations did not achieve the client satisfaction ratings that mission support and technical assistance assignments did, despite project efforts to improve evaluation quality When analyzed by the type of services used, GH Pro clients were the least satisfied with evaluation services (Table 4). In addition, of the 162 Client Survey respondents, 75 requested only one type of assignment through GH Pro, while 87 (53.7%) requested more than one kind of assignment. Of those who stated using GH Pro for evaluation services, 27% did not return as repeat customers, as compared with 11% for MS and 13% for TA. This could indicate higher Table 4. A summary of client ratings on the quality of deliverables and overall satisfaction Client Survey Question Overall Evaluation Requesters Mission support Requesters Technical assistance Requesters How do you rate the quality 4.2 4.1 4.3 4.3 of the final deliverable? How do you rate your overall 4.1 4.0 4.2 4.2 satisfaction with GH Pro? 12 The document review sample of 32 assignments included eight evaluations. The analysis under Evaluation Question 2 of 30 GH Pro evaluations assessed the appropriateness of methods, among other evaluation elements. This analysis was meant to demonstrate the predominant methods employed in GH Pro evaluations. Of the sampled eight evaluations, two were PEPFAR-funded process evaluations, and the remaining six were performance evaluations. 17 18 Evaluation Report Table 5. Understanding the variety of evaluation methods used by GH Pro13 Method A B C D E F G H Document/Data Review X X X X X X X X Secondary Analysis of Existing Data X X X Key Informant Interviews X X X X X X X X Focus Group Discussions X X X X X Group Interviews X X X X X X X Client/Participant Satisfaction or Exit Interviews X Facility or Service Assessment/Survey X Cost Analysis X Verbal Autopsy Survey X X Observations X X X Data Abstraction X X Case Study X Rapid Appraisal Methods Other X X satisfaction among clients of GH Pro’s Mission support and technical assistance services. An average of 93% of respondents said they would use GH Pro again. When clients were given the opportunity to describe why they would or would not use GH Pro again, statements included: “It still remains one of the best options that we have to procure evaluations or technical support;” “They deliver on time, very supportive and responsive to our needs;” and “Excellent quality and ease of use.” It is important to note that several clients who selected yes did so because, as one respondent wrote, “at this point the Bureau does not seem to have any other ‘one stop shop’ mechanisms to fulfill such TA needs in an efficient manner.” Others wrote, “yes, unless a better option is available” and “yes, there is no other mechanism available for short term consultants.” While a few clients stated they would not use GH Pro again, most clients reported that they would work with GH Pro again. Among those who stated they would not use GH Pro again, descriptions included comments critiquing quality. For example, one respondent stated, “The staff at GH Pro and on the evaluation team were very easy to work with, however, the products submitted to us were of sub￾par editorial quality and do not appropriately reflect the tremendous work done by USAID and its implementing partners.” Another reported “poor quality, poor writing and lack of editing.” GH Pro staff cited several efforts undertaken to drive up the quality of evaluations over the course of the Project, including the addition of the communications manager, and a focus on improvements in SOW quality, recruitment, and guidelines. Staff also cited the increased use of a small cadre of excellent consultants, who know what they are doing and are able and knowledgeable enough to follow the guidelines. The need for a specialized technical review processes, across technical areas and parallel to the editing conducted by the communications manager, emerged as a major theme among staff and other key informant 13 Columns in Table 5 represent the eight evaluations that were among the 32 GH Pro assignments included in the document review sample. Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) interviews. Additional technical capacity, it was stated, would improve quality while supporting more innovative evaluations beyond what GH Pro can support now. Finding 1.15: Clients reported that the cost of GH Pro services was high Most clients rated the cost of the final deliverable provided by GH Pro as very costly. Clients gave GH Pro an average rating of 3.82, with 1 being inexpensive and 5 being very costly. Clients rated the value for money spent on GH Pro at 3.56, with 1 being low value and 5 being high value. Through qualitative findings many clients expressed a sense that value for money was low—paying GH Pro’s overhead represented paying a “middle man” that was not worth the cost. This perception was especially true in cases where a client had identified a consultant already, did not need help with the SOW, and/or was not satisfied with GH Pro’s management of recruitment, assignment management and communication, or report editing. When asked about the disadvantages of working with GH Pro in an open-ended survey question, 68 client respondents (42%) referenced high costs. These findings demonstrate that there is room for improvement perceptions of costs and value for money spent. Despite the perception of high costs, clients recognized that GH Pro did offer one cost efficiency over other service mechanisms: GH Pro charged the client only that which was expended when the cost was lower than the full cost stated in the original estimate. If there are unused funds on a given assignment, GH Pro can reprogram them to the client once the assignment is completed and closed. This cost efficiency is advantageous for GH Pro, too. Finding 1.16: The PMO charge lent to perceptions of high costs among clients but was critical for overcoming the absence of core funding GH Pro charges a project-management-office (PMO) cost of 25% on top of actual costs to cover operational expenses, seen by some clients as the reason GH Pro is considered to be so expensive. Because GH Pro does not have core funding however, it relies on the PMO to fund things like the full-time senior recruiter, critical finance and administrative support roles, and standard operating costs of maintaining the office. Because GH Pro cannot charge to assignments prior to TDM approval, all the project management efforts up to approval are also covered by the PMO. A range of senior GH Pro staff highlighted detriments resulting from the absence of core funding. At the outset, Project start-up was difficult because not enough funds were available to rapidly execute the necessary capacity build￾up. There were insufficient funds to hire staff, set up critical systems, or procure equipment. As a small business, GH Pro’s prime, Dexis, could not front these funds. Once the Project started to implement assignments and to slowly build up management reserve costs via the PMO charge, Dexis still could not expand at the rate that was required because the demand was so high. Senior staff indicated that GH Pro did not really catch up with demand (i.e., hiring sufficient staff to manage the assignment workload appropriately) until one￾and-a-half to two years into the Project. GH Pro could either complete assignments or improve capacity/processes; there was no capacity to do both. Senior staff cited other challenges stemming from the lack of core funding, including the following: • Seasonal lulls in the number of assignments cause fluctuations in PMO funds, leading to the need to decrease staff and functionality when funds are low. • As GH Pro nears the end of the Project period, the decreasing number of assignments will limit PMO funds available to cover critical close-down activities. • It is not generally possible to fund PM staff to provide short-term, field-based support when needed. Project management staff navigate complex logistics and recruitment from afar, including training local logistics partners on USAID compliance. For more complex assignments, fielding project management staff to support logistics and compliance at start-up (e.g., review fleet hires/ensure the security of fleets, ensure vendors are legitimate) would be beneficial. 19 20 Evaluation Report CONCLUSIONS GH Pro was found to receive strong performance ratings from clients in terms of flexibility and ease of access and use, and from consultants in terms of ease of working with and responsiveness of GH Pro. A majority of clients rated the quality of GH Pro deliverables to be good and assignments were completed per specifications outlined in the final SOW. Only Half of All Assignments Are Completed in the Anticipated Time Frame While delays may have been due to factors outside of GH Pro’s control, this failure to deliver results in the anticipated time frame was nonetheless problematic and could partially explain client dissatisfaction with the Project. Interviews with clients revealed that, when assignments were not completed when expected, the assignments were extended until they were finished. Clients Have Mixed Feelings about the Quality of GH Pro’s Work Although clients perceived GH Pro as quick and easy to engage, clients reported GH Pro was unable to consistently provide high-quality outputs. Clients Perceive GH Pro as Too Expensive These findings demonstrate that there is room for improvement in the costing associated with the mechanism and in the perceived value for money spent. This perception of high cost may be the result of optics rather than truly elevated fees. Regardless, GH Pro, and subsequently USAID, should reflect on the cost of GH Pro’s services, the rate of assignment completion within the anticipated time frame, and clients’ mixed reviews of the quality of their work. Clients Have the Most Issues with Evaluation Assignments Survey data suggest that GH Pro clients and consultants are the least satisfied with the outputs of assignments in the evaluation service area. Incidentally, evaluation consultants were selected by GH Pro (as opposed to a selection based on prior experience or a personal recommendation from another colleague) more often than consultants for other tasks. RECOMMENDATIONS For each recommendation, we have noted the finding(s) that were the basis for the recommendation. For GH Pro: Staffing 1.1. Consider Increasing Senior Technical Staff Positions: Additional senior technical staff capacity would be useful to assure technical quality during SOW development, adequate briefing of the evaluation consultants, and development of high-quality evaluation reports. Adding technical capacity would allow GH Pro to be more supportive of USAID in ensuring the quality of evaluation designs at the start. It would also enable GH Pro to be more hands-on with the review of deliverables to further improve quality. (Findings 1.2, 1.10, 1.13, 1.14) 1.2. Consider Revising the Responsibilities of Project Managers and Finance Staff: Consider adjusting the model that requires PMs to handle every aspect of assignments. Shifting some responsibility to F&A staff (e.g., cost estimate development, budget tracking, and/or travel logistics) could help to leverage the F&A staff’s financial training while freeing up PMs to concentrate on other responsibilities. In particular, earlier involvement of an F&A specialist—during SOW budgeting, revisions to the SOW, and finalization of the cost estimate—is recommended. This shift may also address existing dissatisfaction among the F&A cadre that all they do is accounts payable and could address professional growth among the PMs so that they have more opportunities to engage technically. (Findings 1.9, 1.11) For GH Pro: Operations 1.2. Improve yhe Technical Review of Evaluations Throughout: Currently, there is limited technical bandwidth at the Project to ensure a high-quality review of the SOW, work plans, and data collection tools. Consider ways to increase this bandwidth (additional technical staff, retainer staff, bringing on team leads earlier) to ensure adequate technical oversight throughout the process, from SOW to final report. (Findings 1.2, 1.3, 1.14) Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) 1.3. Maintain Capacity in Facility Access Processing: GH Pro is strongly encouraged to continue to support a dedicated staff person who is fully conversant with the FA and security clearance processes. (Finding 1.5) 1.4. Improve Tracking of Local and Regionally-Based Staff and Institutional Resources: Track local and regionally-based individuals and institutions that support assignments, including the type of support provided (e.g., team lead, technical advising, data collection support, logistics coordination), as a part of routine monitoring. This will facilitate an understanding of the utilization of local resources, important given Agency priorities in this area. (Findings 1.6, 1.8) For GH Pro: Client Engagement 1.5. Consider Standardized Orientation Materials for USAID Clients on Facility Access and Daily Rates: Many Mission personnel do not understand the rigid policies on FA and daily rate determination, which leads to erroneous expectations and delays in deploying support staff. (Findings 1.6, 1.7) 1.6. Improve Direct Engagement with Field-Based USAID Clients: Project management staff should make more of an effort to engage field-based USAID clients to ensure that the consultants are meeting expectations and to avoid problems later when final products are due. One example of this type of engagement is to clarify when the USAID client would like to see draft products—early in the process or when the document is nearing finalization. (Finding 1.14) 1.7. Continue to Develop Strong Working Relationships with Key USAID Staff: GH Pro should continue to prioritize strong relationships with USAID administrative and support services staff, including Country Support Services, along with USAID technical focal points relevant to each assignment. These connections facilitate timely responses for Mission support, technical assistance, and evaluations. (Findings 1.10, 1.14) 1.8. Consider Developing Updated Information for USAID Clients on How Costs are Estimated: The ongoing perception by USAID staff that GH Pro services cost more than similar mechanisms is often inappropriate. It would be useful to more clearly communicate in cost estimates the funds needed to accomplish the assignment, including the cost areas currently covered by the PMO. (Finding 1.15, 1.16) For USAID: Managing GH Pro and Follow-On Projects 1.9. Consider Core or Transition Funding for Follow on: Consider core funding for key operational and technical staff. This would improve start-up capacity, allow for more robust technical oversight of assignment scoping and deliverables, lend flexibility in dealing with seasonal funding lulls, and improve end-of-project operations. (See expanded recommendation in Evaluation 3) Limited use of core funds to travel PMs to the field would also improve efficiency and effectiveness in managing complex assignments and supporting Missions where GH Pro provides frequent support. When on site, core staff could provide training for local logistics coordinators on USAID rules and regulations. (Findings 1.2, 1.11, 1.15, 1.16) 1.10. Increase Recruiting Capacity – For any follow-on project that comes after GH Pro, consider including more than one dedicated recruiter on the core staff, who can help with Project-wide recruitment. An additional recruiter(s) would allow PMs to spend less time searching databases and networks for candidates and more time managing and backstopping consultants and teams. (Findings 1.4, 1.5, 1.10) 1.11. Strengthen Support and Review Systems to Improve Evaluations: USAID should consider improving systems to assist USAID field and HQ staff with the design and management of evaluation activities, through institutionalized review processes that routinize access to evaluation experts for peer review during SOW development and evaluation design. This may require bolstered staffing of evaluation experts. (Findings 1.3, 1.10) 21 22 Evaluation Report 1.12. Carefully Consider When to Use Small Business Set-Asides: Small business set-asides promote the use of more diverse and potentially more innovative organizations. However, many small businesses do not have the capacity for rapid, large-scale start-ups such as that required by GH Pro (and likely its follow-on). Small business set-asides may be more appropriate for lower-volume support projects or for narrow, well-defined service areas, like Mission support. They may be less suited for evaluation and technical assistance support. (Finding 1.15, 1.16) Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) EVALUATION QUESTION 2 To what extent are USAID and PEPFAR evaluation quality standards employed and achieved by GH Pro? BACKGROUND GH Pro had completed 59 evaluations for which final reports were available from the beginning of the Project in 2014 through April 20, 2018. All of these would be classified as performance evaluations, with an average cost of $300,000 and a median duration of 266 days. EVALUATION METHODS In operationalizing the review process, a Quality Reference Group (QRG) of eight evaluation experts was engaged. Experts independently reviewed and consolidated views on the extent to which evaluations met quality standards, based on USAID, PEPFAR, and existing literature. The evaluation team implemented a two-step review process focused on evaluation quality. Step 1: Sampling Process From among the 59 evaluations that GH Pro had completed to date, 30 were sampled for review by the QRG (see Annex 2.1 for the list of evaluations, and Annex 2.2 for the list of the evaluators in the QRG). Reports for all 59 completed evaluations were accessed from the GH Pro website and were also available via USAID’s Development Experience Clearinghouse website. We used a mixed, purposeful sampling approach,14 where the evaluations were first categorized according to geographic focus: Africa; Asia; Europe and Eurasia; Global/HQ; and Latin America and the Caribbean. From within these categories, to ensure sample diversity, we then generated a list of evaluations to be reviewed by the QRG based on cost, complexity, performance vs. impact, and programmatic focus area. This approach allowed us to be more confident in our reach and scope in capturing the breadth and depth of GH Pro activities to date. We accessed 59 completed GH Pro evaluations, which served as the universe for the sample. Step 2: Review and Scoring Process The QRG was engaged to conduct the review process. These experts independently reviewed and consolidated their views on the extent to which evaluation quality standards were met, according to USAID and PEPFAR guidelines and existing literature. The group consisted of eight evaluation experts. Characteristics of the experts included experience with international development, USAID, Centers for Disease Control (CDC), PEPFAR, and performance evaluation; familiarity with USAID and PEPFAR review standards; experience with evaluating the quality of evaluations; and knowledge of standardized peer review processes, such as National Institutes of Health peer review processes. The group included evaluation experts based in the US and sub-Saharan Africa. Experts who had been involved with GH Pro directly were not eligible to be a member of the QRG to minimize biased measurement and reporting. The list of QRG members and their affiliations are presented in Annex 2.2. Reviewers were convened virtually for two meetings using Zoom audio/video technology. The first meeting included a training on the application of selected sets of quality standards and tools for assessing GH Pro evaluations. This process served to (a) define the sets of quality standards used; (b) review the sampling approach, scoring process, and timeline; and (c) introduce the evaluation tools QRG members would use to assess the selected GH Pro evaluation reports. Reviewers applied criteria adapted from quality evaluation standards from PEPFAR, USAID, and Scrivens’ meta￾evaluation checklists15 (compared in Annex 2.3.) to develop 14 Nastasi, B. Study notes: Qualitative research: Sampling and sample size considerations (Adapted from a presentation). SAGE Publications. 15 Evaluating Evaluations: A Meta-Evaluation Checklist, 6th Edition, by Michael Scrivens, 2011. http://michaelscriven.info/images/ EvaluatingEvals-Checklist.pdf 23 24 Evaluation Report the Comprehensive Evaluation Reporting Tool. It was used scoring processes adapted from the National Institutes of to apply a consolidated list of standards that served as an Health study section reviews.16 overall best practice review. The instrument also reflected Table 6. Overview of GH Pro evaluation reports, and sampling frame by geographic area & programmatic focus Countries and Programs GH Pro Reports Selected Reports for Evaluation Activity Africa 30 17 Cross-cutting* 11 7 Family Planning 3 2 Health Systems Strengthening 9 4 HIV/AIDS 2 1 Malaria 3 1 Maternal and Child Health (MCH) 2 2 Asia 10 4 Cross-cutting* 2 1 Family Planning 1 N/A Health Systems Strengthening 4 2 HIV/AIDS 3 1 Europe and Eurasia 1 1 Tuberculosis (TB) 1 1 Global / HQ 17 7 Cross-cutting* 3 N/A Emerging Pandemic Threats 1 N/A Family Planning 4 1 Health Systems Strengthening 2 1 Malaria 3 2 MCH 2 1 Neglected Tropical Diseases 1 1 TB 1 1 Latin America and the Caribbean 1 1 Health Systems Strengthening 1 1 Grand Total 59 30 Note: * includes the following: Center for Accelerating Innovation and Impact, Health Research, Social and Behavior Change, Digital Health, the Demographic Health Survey Program, the Supply Chain Program, Health Financing, and Private Sector Engagement. 16 National Institutes of Health. Scoring system and procedure. Last reviewed March 18, 2015. Retrieved from https://grants.nih.gov/grants/peer/ guidelines_general/scoring_system_and_procedure.pdf Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) Each reviewer was assigned no more than four evaluation While all tools include similar content, key differences reports for review. The average report length was about include the following: 35 pages, excluding annexes. Experts reviewed each report using standardized scoring tools described below, which were housed in an online data collection platform (Qualtrics). Reviewing and scoring each report took one to two hours. Reviewers used a set of three tools to operationalize the review process and determine whether evaluation standards were met, as presented in Table 7 (see Annexes 2.4.–2.6. for the set of tools). Each evaluation report in the sample was scored using two of the three tools listed above: all evaluations were assessed using the Comprehensive Evaluation Reporting Tool; all evaluations funded (partially or entirely) with PEFPAR funds were evaluated according to PEPFAR evaluation standards using the PEPFAR Evaluation Reporting Tool; all evaluations funded without PEPFAR funds were assessed according to USAID evaluation standards, using the USAID Evaluation Reporting Tool. In reporting findings of this analysis, this report will refer to the evaluations that were funded partially or entirely with PEPFAR funds as USAID/PEPFAR evaluations, and to evaluations that were funded with other non-PEPFAR funding sources as USAID/non-PEPFAR evaluations. Both the PEPFAR and USAID tools were obtained directly from respective PEPFAR and USAID evaluation guidelines. • The PEPFAR and USAID tools use a scoring system of Yes/No, while the Comprehensive tool uses a modified Likert scale, whereby for each criterion, reviewers rated whether evaluations met definitions of quality on scale of 1 to 5 (1 = not addressed, 2 = poorly/partially addressed, 3 = adequately addressed, 4 = more than adequately addressed, and 5 = exemplary). • The PEPFAR Evaluation Standards of Practice Adherence Checklist contains sub-questions corresponding to each scored component, and requests reviewers (a) score each sub-question by answering yes or no and then (b) determine the score for each component based on the scores for the sub-questions within each component. (e.g., If all sub-elements under each standard were rated yes, the overall standard received a rating of yes; if all sub￾elements under the standard were rated no, the overall standard received a rating of no; and if any combination of yes and no existed for the sub-elements, the standard was scored as “partial.”) • The Comprehensive tool includes the following topics that are not included in either of the other tools: Validity, Cost Utility, and Generalizability. • Acknowledging that the SOW for GH Pro evaluations stipulated adherence to either the USAID or PEPFAR Table 7. Evaluation quality assessment tools used to score sampled GH Pro evaluation reports Evaluation Quality Assessment Tool Types Evaluations Scored, by Tool 1. Comprehensive Evaluation Reporting Tool (adapted from All evaluation reports in sample (n=30) Scrivens13) 2. PEPFAR Evaluation Standards of Practice Adherence Evaluations conducted with (any) PEPFAR funding (n=13) Checklist17 3. USAID Evaluation Report Review Template18 Evaluations conducted with non-PEPFAR funding (n=17) 17 US President’s Emergency Plan for AIDS Relief (PEPFAR). (2015). Evaluation standards of practice, version 2.0. Retrieved from https://www.pepfar. gov/documents/organization/247074.pdf 18 USAID, Bureau for Policy, Planning, and Learning. (2016.) Evaluation report checklist and review template. Retrieved from https://USAIDlearninglab. org/library/evaluation-report-checklist-and-review-template 25 26 Evaluation Report guidelines, the evaluators felt that the addition of the Comprehensive tool provided a standardized, comparable approach to all of the evaluations (USAID funded or PEPFAR funded) and provided additional rigor to ascertaining the overall quality of an individual evaluation. In addition to scoring evaluations, the QRG was asked to discuss potential key priorities and considerations concerning the process of the evaluation of evaluations. This discussion capitalized on the opportunity of having evaluation experts convened to synergize major insights and make contributions to the field. Under Evaluation Question 2, using the methodology noted above, we strove to provide a credible examination of the quality of evaluation services performed by GH Pro. We also hoped to incorporate novel thinking as to how future evaluators might assess the value of evaluation activities. Step 3: Data Analysis Quantitative Analysis Using the Comprehensive Evaluation Reporting Tool, we calculated mean and median scores and the associated range for each evaluation component area. In addition, we calculated cumulative scores for each evaluation, based on scoring per the evaluation question stated in the given evaluation. We excluded questions with conditional statements (e.g., “If changes were made to the evaluation plan, were they documented?”) to avoid biasing cumulative scoring results. We disaggregated cumulative scores by the following parameters: USAID/non-PEPFAR vs. USAID/PEPFAR, geographic region, programmatic area, year that the evaluation was completed, cost of evaluation, midterm vs. end-of-project evaluations, and evaluation methods (qualitative methods only vs. quantitative and qualitative methods combined). Using the USAID Evaluation Report Review Template, we calculated the number of evaluations that met (yes) and did not meet (no) USAID evaluation standards, by evaluation component. Similarly, for the PEPFAR Evaluation Standards of Practice Adherence Checklist, we calculated the number of evaluations that fully met (yes), partially met (partial), or did not meet (no) PEPFAR evaluation standards, by evaluation component. Qualitative Analysis We consolidated comments provided by QRG members across different components of the Comprehensive Evaluation Reporting Tool for all 30 evaluations, noting when two or more of QRG members made the same observations. LIMITATIONS Evaluation reporting tools were adapted from three sources (PEPFAR, USAID, and Scrivens), all of which included specific components for the evaluation process. However, not all components applied to all the evaluation reports that were reviewed. Additionally, each evaluation was assessed by only one QRG member. Finally, we were unable to assess the extent to which data were used to inform program improvement, as this was beyond the scope of the three Evaluation Questions. FINDINGS Finding 2.1: GH Pro’s Niche is as a Provider of Performance Evaluations The evaluations conducted by GH Pro tended to emphasize the measurement of sustainability, quality, scale, and cost. The evaluations were focused on performance. Findings from these evaluations supported recommendations for project course correction (in the case of midterm evaluations) and USAID’s designs for future project procurements. There was a strong emphasis on the identification of effective models of implementation to achieve results. The capability of the projects to meet objectives, deliver results, and meet benchmarks were common themes. Methodologically, the GH Pro evaluations that were studied relied on a mix of quantitative and qualitative methods, with reliance on qualitative methods and with some use of primary and secondary sources of data, such as existing program data or national sources. Methodological approaches included the following: focus group discussions, stakeholder interviews, case studies, document reviews, exit interviews, and direct observations. Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) Sampling methods were generally purposive or convenient. An analysis was usually descriptive, with an emphasis on triangulation. This basic approach tended to be determined by resource limitations. Measuring the progress of implementation was often a central purpose of the evaluations that were performed by GH Pro. Implementation was most commonly gauged by assessing project management, level of staffing, service delivery, and service quality. Service quality was defined as a program’s ability to meet objectives and reach stakeholders with useful and effective programming. Health systems strengthening was also a recurring area of focus and was measured through the analysis of financial management, business practices, organizational planning, procurement capacity, and the presence of overall functional processes within health systems. Individual capacity was measured through the number of trainings. The main audience for the evaluation reports produced by GH Pro is USAID, and in the case of midterm evaluations, the implementing partner is a key secondary audience. Finding 2.2: The Quantitative Scores of GH Pro’s Evaluations were ‘Adequate’ on Average, and Most Evaluations were in Compliance with Most Required Components, with Some Exceptions For the review of the 30 sampled GH Pro evaluations, most components of the Comprehensive Evaluation Reporting Tool received median scores of 3 (adequately addressed), with the exception of the following components: resources and budget, score of 1 (not addressed); ethical considerations, score of 2 (poorly/partially addressed); and results dissemination, score of 2 (poorly/partially addressed) (see Table 8). Individual Table 8. Mean and median scores of all evaluations using the Comprehensive Evaluation Reporting Tool, by evaluation component Components Mean score Median score Range Executive Summary 3.5 3.5 (2–5) Program Information & Project Background 3.4 3 (2–5) Evaluation Purpose 3.4 3 (2–5) Evaluation Questions 2.9 3 (1–5) Resources & Budget 1.9 1 (1–5) Methodology & Approach 3 3 (1–5) Team Composition & Stakeholder Engagement 2.4 3 (1–5) Limitations 3.1 3 (1–5) Findings & Conclusions 2.9 3 (1–5) Responsiveness to Evaluation Questions 3.3 3 (1–5) Recommendations 3 3 (1–5) Ethical Considerations 2.3 2 (1–5) Monitoring Planning & Implementation of Evaluation 3.1 3 (1–5) Annexes 2.9 3 (1–5) Results Dissemination 2.1 2 (1–5) Program Improvement 2.5 3 (1–4) Overall Evaluation 3.4 3 (1–5) 27 28 Evaluation Report scores for the reviewed evaluations, per the Comprehensive non-PEPFAR evaluations compared to USAID/PEPFAR Evaluation Reporting Tool, can be found in Annex 2.7. evaluations: 162.3 (112-223) and 154.3 (126-201), respectively Average cumulative scores were slightly higher for USAID/ (see Table 9). Table 9. Average cumulative scores of GH Pro evaluations based on the Comprehensive Evaluation Reporting Tool, by various disaggregation categories Disaggregation N Average Cumulative Score Min. Max. USAID/PEPFAR vs. USAID/non-PEPFAR USAID/PEPFAR 13 156.2 126 201 USAID/non-PEPFAR 17 162.3 112 223 Geographic Area Africa 17 163.4 130 223 Asia 4 149 126 176 Europe and Eurasia 1 155 --- --- Global/HQ 7 156.4 112 170 Latin America and the Caribbean 1 167 --- --- Programmatic Area Cross-cutting 8 167.3 135 223 Family Planning 3 163.0 151 170 Health Systems Strengthening 8 162.8 131 198 HIV/AIDS 2 130.5 126 135 Malaria 3 157.3 150 170 MCH 3 163.7 130 201 Neglected Tropical Diseases 1 112.0 --- --- TB 2 162.0 155 169 Year 2015 4 155 135 170 2016 12 164 130 201 2017 12 161.2 126 223 2018 2 133.5 112 155 Midterm vs. End of Year Midterm 16 154.3 112 181 End of Year 14 165.8 135 223 Methodology Qualitative Only 10 159.6 112 223 Quantitative & Qualitative 20 159.7 126 201 Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) The highest scoring evaluations focused on cross-cutting programmatic areas and the African geographic region (refer back to Table 6). Cumulative evaluation scores varied little by the year that the evaluation was conducted, with the exception of one USAID evaluation conducted in 2018 that received the lowest score of 112 (Table 9). End-of￾year evaluations scored slightly higher than evaluations conducted at the midpoint: 165.8 (121-223) and 151 (99-181), respectively. On average, USAID/PEPFAR evaluations cost approximately $126,000 more (median cost of $347,365 for USAID/PEPFAR evaluations vs. $221,226 for USAID/non￾PEPFAR). USAID/non-PEPFAR evaluations generally scored 6 points higher (Table 9). Out of 17 USAID/non-PEPFAR evaluations, almost all met USAID evaluation standards across the following components: introduction and purpose; information and background; evaluation questions; findings and conclusions; and recommendations (Table 10). The component with the fewest number of evaluations (12) meeting USAID standards is called annexes. Out of 13 USAID/PEPFAR evaluations, most scored highly in the following components: clearly state evaluation questions; use appropriate evaluation designs, methods, and analysis; construct data collection and management plans; and use findings for program improvement (Table 11). The components with the fewest number of evaluations meeting PEPFAR standards are the following: address ethical considerations and assurances (5); identify resources and articulate budget (7); and disseminate results (7). Table 10. Summary scores of USAID/non-PEPFAR evaluations using the Evaluation Report Review Template, by USAID evaluation component (N = 17 evaluations) Components # of Evaluations that Met the Criteria (Yes) # of Evaluations that Did Not Meet the Criteria (No) Executive Summary 14 3 Introduction & Purpose 16 1 Information & Background 15 2 Evaluation Questions 17 0 Methodology 13 4 Limitations 13 4 Findings & Conclusions 15 2 Recommendations 16 1 Annexes 12 5 Gender* 12 4 Overall* 12 4 Ethical Considerations 2.3 2 Monitoring Planning & Implementation of Evaluation 3.1 3 Annexes 2.9 3 Results Dissemination 2.1 2 Program Improvement 2.5 3 Overall Evaluation 3.4 3 Note: *Some evaluations were missing a score for this component. 29 30 Evaluation Report Table 11. Summary scores of USAID/non-PEPFAR evaluations using the Evaluation Report Review Template, by USAID evaluation component (N = 17 evaluations) Components # of Evaluations that Met the Criteria (Yes) # of Evaluations that Partially Met the Criteria (Partial) # of Evaluations that Did Not Meet the Criteria (No) Engage Stakeholders 4 8 1 Clearly State Evaluation Questions, Purpose, & Objectives* 11 2 0 Use Appropriate Evaluation Designs, Methods, & Analysis 10 3 0 Address Ethical Considerations & Assurances19 5 3 5 Identify Resources & Articulate Budget20 6 0 7 Construct Data Collection & Management Plans 10 1 2 Ensure Appropriate Evaluation Qualifications & Evaluation 6 4 3 Independence Monitor the Planning & Implementation of an Evaluation 8 4 1 Produce Quality Evaluation Reports 6 7 0 Disseminate Results21 Note: *One evaluation was missing a score for this component. Finding 2.3: The Qualitative Review of Evaluations Documented Strengths and Weaknesses QRG members noted several strengths of the GH Pro evaluations. Most notably, each evaluation’s purpose and questions were clearly defined, with the relationship between the questions and the purpose made explicit. Additionally, the methodology and approach, as well as the limitations, were clearly described. Finally, original SOWs were available in the evaluations and included general, albeit brief, plans for disseminating results and conducting program improvement. Noted areas for improvement included the programmatic and policy use of evaluation findings, background information on evaluators , plans for 0 6 7 dissemination, and budget and resources. For additional details on qualitative findings, please refer to Annex 2.8.22 CONCLUSIONS Overall, evaluators tended to rate the GH Pro evaluations with average scores for quality across the majority of components. Evaluators agreed that evaluation components that scored the lowest were the following: resource and budget allocations, results dissemination, and ethical considerations. This finding was consistent across all tools that included these three elements (i.e., the Comprehensive and PEPFAR tools). Cumulative evaluation scores varied little by cost or the year the evaluation was conducted, and they varied only slightly by methodology. 19 “Ethical considerations” are defined here as a review of the potential need for human subject protection and a clear statement in the Evaluation report as to whether this is needed or not. 20 “Resources and Budgets” are defined here as the overall costs for the performance of the evaluation and should be included in the report or in the Annex section. 21 “Dissemination plans” for the evaluation report need further detail and should be better defined by the USAID client requester. 22 USAID guidance to include background information on the evaluators was only required after 2017. Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) End-of-project cumulative evaluation scores were slightly higher than midterm scores. We can conclude from this finding that all GH Pro evaluations that were assessed met the criteria for evaluation quality, but very few excelled. A key issue here is the rigor of GH Pro evaluation methods and the level of resources put into the evaluations (including the financial budget and the time allotted to complete evaluations), both of which are points raised by the QRG members. Methodologically, as noted below, the evaluations tended to use simple though legitimate methods and analytic approaches. Funding for the evaluations tended to be somewhat minimal compared to the scope of the projects being evaluated. To increase rigor, evaluators would need to expend more resources. The following provides a summary of key observations and recommendations from the evaluation experts during the review meeting. Evaluator Observations of the Evaluation Process • All three tools used by QRG members to assess the evaluations focused on what was in the report vs. what was actually done. There may be a difference in the quality of the evaluation report vs. the quality of the evaluation itself. It may be hard to get a sense of the quality of implementation from the report alone. • USAID/non-PEPFAR and USAID/PEPFAR evaluations are held to different sets of quality standards, which overlap but are not identical. The tool for USAID/PEPFAR evaluations results in three possible scores (yes/partial/ no) whereas that for USAID/non-PEPFAR evaluations results in only two possible scores (yes/no), making comparison difficult. • Regarding the tools, some questions were best answered with yes/no responses, but some were more appropriate for scale-based measures. In the Comprehensive Evaluation Reporting Tool (using a Likert scale of 1–5), what is considered “adequately addressed” (score of 3)? Many evaluators understood a rating of 3 to mean that the component of the report was acceptable and met minimum standards, which is likely why most responses ranged between 1 and 3. Evaluators had a difficult time differentiating between ratings of 3, 4, and 5 in practice (if the component in question met basic evaluation criteria). • The stakeholder dissemination plans were not always clear. Is dissemination to the funder sufficient? Or, ideally, is dissemination back to the country or project staff preferable? For USAID evaluations, the dissemination strategy seemed to be defined by the specific requester/funder. • Key areas—such as budget, ethical considerations, theories of change, and dissemination—were often underdeveloped or missing. • In the PEPFAR Evaluation Standards of Practice Adherence Checklist, the developers of this tool should rethink the “partial” category. Evaluators found it difficult to assess. RECOMMENDATIONS For each recommendation, we have noted the finding(s) that were the basis for the recommendation. For Service Projects that Conduct Evaluations, including GH Pro 2.1. Provide Evaluator Information: Provide more information on evaluators’ background, experience, and independence, in particular, to clearly exempt any conflicts of interest (or to be transparent with evaluator selection). (Finding 2.2) 2.2. Provide More Information on Ethical Issues: More detailed information is needed on ethical considerations (if there are human subject protection issues) in evaluation reports, including the appropriate use of consent forms. (Finding 2.2,) 2.3. Ensure More Detailed Documentation of Evaluation Budgets and Costs: Consider a more transparent and easily accessible budget and resource allocation breakdown for the evaluation for inclusion in the ANNEX section of any evaluation report. (Finding 2.2) 31 32 Evaluation Report 2.4. Develop Recommendations Focused on Program Improvement: Recommendations for program improvement need more substantive development. Evaluation findings need to be fleshed out more and made more explicit. (Finding 2.1) For USAID 2.5. Detail Dissemination Expectations: USAID client requesters could provide more detailed plans for dissemination (e.g., timeline; target audience, including levels of stakeholders; dissemination format) and require evaluation teams to detail the design, implementation, and descriptions of their dissemination activities. (Findings 2.1, 2.2) 2.6. Provide Technical Assistance to USAID Clients Who Request Evaluations: Consider a recommendation that relevant USAID staff (e.g., in P3 or the Bureau for Policy, Planning, and Learning [PPL]) could further develop appropriate guidelines and tools to help USAID requestors identify the most appropriate types of evaluations to be utilized and the most appropriate methodologies. Ideally, the final decisions would result from a dialogue between the USAID client and the consultant team. (Finding 2.2) 2.7. Furnish Appropriate Resources to Enhance Rigor: Consider providing more time and resources for evaluations to enhance rigor (e.g., sampling design/methods/statistical analysis); consider partnering with academic institutions to help. (Findings 2.1, 2.2) 2.8. Provide Technical Assistance for Theories of Change: Similarly, more technical assistance (or internal technical capacity) is needed to improve the description of (and thereby enable evaluation of performance against) the theories of change. (Findings 2.1, 2.2) 2.9. Increase the Frequency of Assessing the Quality Of Evaluations: The evaluation of evaluation mechanisms should be conducted in a timelier manner (e.g., at designated biannual cycles, when a project has completed a critical number of evaluations). (Finding 2.2) 2.10. Adjust Evaluation Review Tools to Facilitate Quality Comparisons: Consideration should be given to aligning the scoring outcomes for the PEPFAR and USAID review checklists, and to introducing a Likert-style scale for quality standards that are more effectively assessed on a scale. In addition, the Comprehensive tool includes components not included in the other tools (Validity, Cost Utility, and Generalizability), which could add value to the PEPFAR and USAID tools. (Finding 2.2) Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) EVALUATION QUESTION 3 What is the added value of GH Pro to the USAID Bureau for Global Health and Missions for the three key service areas (evaluation, Mission support, and technical assistance), considering other, existing mechanisms? BACKGROUND P3 in USAID GH would like to expand and improve USAID’s access to effective, efficient, and quality support services in the areas of evaluation, Mission support, and technical assistance. The goal of Evaluation Question 3 was to describe the role and unique advantages that GH Pro plays within the broader portfolio of 17 service support projects. Each of these projects provides one or more of the three support services: Mission support (providing substitute or supplemental staff to USAID missions), technical assistance (primarily with operational issues), and evaluations (from performance to impact evaluations). The evaluation team (HEARD) had neither the mandate nor the resources to evaluate the effectiveness, efficiency, or quality of the outputs from the 17 projects. Instead, the focus of Evaluation Question 3 was to examine the 17 USAID service support mechanisms, including GH Pro, using the following parameters: areas of focus, organizational structure, staffing patterns, financial arrangements, the business model, and the use of local solutions. The goal of this analysis was to assess the overall portfolio of service support facilities, including questions such as the following: • Are there gaps in needed services? • Are there duplications of service? • Are these services being optimally delivered? • Are management and oversight functions sufficient? • Are we assured of a consistent quality of products? • Are there more effective approaches for developing future portfolios in these three service areas to ensure effectiveness, efficiencies, and overall quality? EVALUATION METHODS For Evaluation Question 3, we used a mixed-methods approach. Seventeen projects/mechanisms were selected for this exercise in consultation with P3. These are presented in Table 12. The criteria used for selection included the following: • Each of the projects is currently providing support in one or more of the three selected service areas (Mission support [substitute and supplemental USAID staff], technical assistance, and evaluations); • Each of the projects serves USAID GH as identified by P3 staff. Some projects (e.g., MERLIN suite) are Agency￾wide and are available to all Bureaus. Data for Evaluation Question 3 was collected from four sources: document reviews, key informant interviews, surveys, and a Strategy Reference Group (SRG). • Structured document review – The evaluation team ensured review of the most current USAID GH users’ guide for USAID/Washington health programs,23 individual project websites, and other available information that was identified by the respective COR/AOR. A Support Mechanism Matrix for comparing various characteristics of each USAID support mechanism was created (see Annex 3.1.), and information from the documents reviewed was extracted to populate the matrix. In addition, selected comments from the interviews with CORs/AORs were inserted into the Support Mechanism Matrix. 23 USAID GH updates its users’ guide quarterly, and recent versions are available on the GH website at https://www.USAID.gov/what-we-do/ global-health/global-health-users-guide 33 34 Evaluation Report • Key informant interviews were conducted with a subset of CORs/AORs from each of the projects reviewed, using a structured interview guide. (See Annex 3.2.) In addition, interviews were conducted with members of the management team of GH Pro. The interviews were voluntary and considered confidential. It was agreed that there would be no attribution of any specific comment to an interviewee. After the interviews with the CORs/ AORS, notes were transcribed, and these were shared with the interviewee to check for accuracy. • A survey, using QuestionPro software, was provided to the CORs/AORs to pass on to their respective project staff. (See Annex 3.3.) The two primary purposes of the survey were to determine whether local resources were used (staff and organizations) and to identify specific aspects of the evaluation work, including the types of evaluations performed and staff members’ experience with specific evaluation methodologies. Due to a lack of a sufficient response, this data source was dropped from the Evaluation Question 3 exercise. Table 12. Comparative USAID service support mechanisms for the Bureau of Global Health Comparative USAID Service Support Mechanisms for the Bureau of Global Health Primary Services Provided* Monitoring and Evaluation to Assess and Use Results (MEASURE) Evaluation Evaluation, Capacity building, TA Phase IV Monitoring, Evaluation, Research, and Learning Innovations Network (MERLIN) Evaluation MERLIN/Developmental Evaluation Pilot Activity–Monitoring, Evaluation, Evaluation Research, and Learning (DEPA MERL) MERLIN / Rapid Feedback MERL Evaluation MERLIN/ Strategic Program for Analyzing Complexity and Evaluating Systems Evaluation (SPACES) MERL MERLIN/ Expanding the Reach of Impact Evaluation (ERIE) Evaluation Accelerating Strategies for Practical Innovation and Research in Economic Evaluation/TA Strengthening (ASPIRES) Breakthrough Research Evaluation/TA Coordinating Implementation Research to Communicate Learning and Evidence Evaluation/TA (CIRCLE) Global Health Program Improvement Cycle Project (GH Pro) Evaluation/TA/MS Policy, Planning, and Learning–Monitoring and Evaluation (EVAL-ME) Evaluation/ TA Project Supporting Operational AIDS Research (Project SOAR) Evaluation/Operations Research Global Health Support Initiative III (GHSI–III) Staff (primarily for HQ) Global Health Fellows Program II (GHFP–II) Staff (primarily for HQ) Rapid Staff Support Services (formerly Firehouse) MS Procurement Support Award (PSA) MS Global Health Professional and Organizational Development II (GHPOD–II) MS/Conference support/Training Note: * This column lists primary focus within the three service areas. These providers may provide other types of services. For more information, refer to Annex 3.1: Support Mechanisms Matrix. In this table, as with others, MS means Mission support and TA means technical assistance. Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) • An SRG of six experienced, former USAID senior program managers, who have previously been end users of the three service areas that GH Pro offers (technical assistance, Mission support, and evaluations), was convened through video conference on July 16, 2018. The primary focus for this group discussion was to engage in a more forward-looking exercise to critically review the characteristics of the array of support mechanisms utilized by USAID and, then, to develop recommendations for USAID based on the comparative advantage question. A number of tools were created to guide and inform the discussion of the SRG. These included the Support Mechanism Matrix described above and a structured agenda, defining key questions to be addressed to focus the SRG’s deliberations. Agenda topics included reviewing the key questions and proposing recommendations for future actions. It was stressed that these recommendations should be actionable within existing USAID operating procedures and would seek to save costs, increase efficiency, and/or enhance the quality and value of a product. Analytic Approach A structured document review was performed, and information was extracted using the Support Mechanism Matrix as a framework. A descriptive analysis of the survey results was undertaken, addressing questions concerning the use of local solutions, self-identifying capacities for performing different types of evaluations, and utilizing various methodologies. Analysis of the key informant interviews with CORs/AORs and management team members focused on perceived challenges and the strengths and weaknesses of their specific projects. The interviewee comments were kept anonymous to encourage interviewees to share candid information. These three data sources—the structured document review, descriptive analysis of the structured survey, and analysis of the COR/AOR key informant interviews—were then used to create two distinct products, each of which described aspects of the entire service support portfolio. The Support Mechanism Matrix provides information on the following: identifying the COR/AOR, prime contractor and subcontractors, period of performance, business model used, and specifics about services. A Map of Evaluation Projects was also created using the data from the document review, key informant interviews, and the survey tool. All of this data and the two tools mentioned above were used to create a set of discussion questions for the SRG. LIMITATIONS As noted above, the evaluation had neither the mandate nor the resources to assess the effectiveness, efficiency of operations, or quality of outputs for each of these projects. To address Evaluation Question 3, we used standard reviews of available documentation and in-depth interviews with CORs and AORs, who were responsible for the 17 projects subject to review. We did not perform a systematic review of the associated midterm or end-of-project evaluations. We did not speak directly to Project Directors or staff, perform any customer satisfaction surveys, or examine the volume of requests/assignments from USAID clients. There was a limited response to surveys sent to non-GH Pro users. Therefore, this data was not used. Evaluators were unable to review internal USAID websites or data sources. As for the perceived cost of services by USAID clients, we were not able to truly compare costs from one mechanism to another, as this would require adjusting for presence/absence of core funding, how indirect costs are calculated, profit and NICRA variations, contract mechanism, etc. Each of the methods just described would have provided additional information for the Support Services Matrix. For the Map of Evaluation Projects, the data were derived from publicly available documentation and from the interviews with CORs/AORs. We did not vet the data with the project directors. However, the intent of this exercise was to create a set of key questions and background materials for an SRG discussion. The methodology described above did provide us with the saturation needed to identify critical issues and 35 36 Evaluation Report led to a productive discussion of the portfolio of service support projects. This review focused on identifying a range of common issues across the portfolio, including the strengths and weaknesses of each support mechanism, the factors that made each mechanism unique and distinct from the others, and the trade-offs among the various characteristics that make up an effective, efficient, and valuable service support project. Conflicts of Interest The HEARD Project is a USAID service support mechanism, created in 2017, which provides evaluation services. Thus, there was the potential for conflict of interest while addressing the three Evaluation Questions. The evaluation team attempted to address any potential conflicts in the following ways: • The team leader for the evaluation team was an external consultant and was not part of the full-time staff of HEARD. The team leader made every possible effort to ensure that the data collection, analyses, and conclusions were unbiased. • For Evaluation Questions 2 and 3, after data collection and analysis by the HEARD team, selected draft findings were presented to external, independent groups of experts for review and to finalize the conclusions and recommendations. • The HEARD Project was excluded from being considered among the comparative service support mechanisms for the purposes of the Evaluation Question 3 analysis; the HEARD AOR was not interviewed nor requested to complete the survey, and the HEARD Project was excluded from the Support Mechanism Matrix. FINDINGS FOR EACH DATA SOURCE Document Review Information obtained from the latest version of the USAID GH users’ guide for USAID/Washington health programs and from the respective project websites tended to focus on information that described the project’s SOW and the participating organizations. There was generally less information on specific capacities, previous work, and the utilization and value of final products, such as evaluation reports. The information available also generally lacked a discussion of weaknesses or deficiencies. Information gleaned from this review were entered into the Support Mechanisms Matrix, presented in Annex 3.1. Key Informant Interviews Seven CORs/AORs, responsible for managing 11 of the 17 identified USAID support mechanism projects, participated in structured, in-depth interviews. In addition, key informant interviews took place with the USAID management team for GH Pro and with other USAID staff who were familiar with the overall service organization portfolio and interacted with these service organizations on a regular basis. As noted, interviews were considered confidential and no attributions were made. A number of CORs/AORs made comments about their high workload and said it was challenging to provide all of the functions necessary to manage their projects. Essentially, all of the COR/AOR interviewees supported a recommendation to create a process to improve regular communication among the CORs/AORs to discuss the challenges, solutions, and future work of their respective projects. The interviewees consistently stated that more CORs/AORs are needed to handle the workload and that the current certification process should be reviewed and improved. Finally, several CORs/AORs expressed concern that the technical aspects for project oversight were slowly diminishing, as seen in the change of name from “COTRs” to “CORs” (the T that was removed stood for “technical”). Survey Results Five surveys were completed from the 17 projects identified for review representing only a 30% response rate. Among the respondents, 40% reported they used local institutions/ experts “almost all of the time,” and 40% used them “sometimes.” Approximately 20% of overall budgets were used for “local solutions.” Due to the relatively small sample size, it was difficult to analyze evaluation methodologies to Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) generalize across the 12 projects that perform evaluations. However, for the five projects where data were available from this survey, the results are provided in Figure 3. Evaluation methodologies, with the most used noted in red text. Strategy Reference Group The review and analysis of data that were collected in this evaluation generated a set of overarching questions/issues that served as the basis for the SRG discussion. Six USAID ex-senior staff were convened via a video conference call. Figure 3. Evaluation methodologies, with the most used noted in red text 1. Developmental evaluation 2. Participant/direct observations 3. Secondary data analysis (program data, surveillance data, service provision data) 4. Focus group discussions 5. Customer satisfaction surveys/exit interviews 6. Outcomes harvesting 7. Mathematical modeling 8. Contribution analysis 9. Plausibility analysis 10. Cost-benefit analysis 11. Cost-effectiveness analysis 12. Cost-utility analysis 13. Most Significant Change technique 14. Household surveys 15. Facility Surveys 16. Key informant interviews 17. Rapid experiments 18. Rapid Appraisals 19. Theory-based evaluation 20. Appreciative inquiry *Red text indicates that essentially all projects or project CORs state that they utilize this methodology See Annex 3.4 for the list of their names, current positions, and USAID experience. See Annex 3.5 for the list of discussion questions. The SRG met on July 16, 2018, using Zoom video￾conference technology. Background information was provided on the rationale and focus for convening the SRG. Currently, USAID is facing a number of major challenges. This evaluation focused specifically on GH Pro but also examined the broader portfolio of 17 different service support projects that primarily serve USAID GH. Lessons learned from the evaluation of GH Pro proved useful in examining the broader service support portfolio. The following represent specific observation findings from the SRG. Evaluation providers: Should these provide only evaluations or other service areas? There was a general consensus that future projects should probably not be linked with Mission support to protect the integrity and rigor of evaluation work. Instead, combining technical assistance with an evaluation focus can achieve mutual benefits. There was strong support for the concept that quality evaluations came from evaluators who also understood the technical and operational dimensions of the activities that were being evaluated. Thus, when an evaluator also provided technical assistance in a single development area, his or her familiarity with that area bolstered the evaluator’s expertise and helped to craft more high-value and actionable evaluations. Evaluation providers: Provide a full range of types and methodologies? The SRG supported the need for a permanent core group of evaluation experts within any evaluation project—a core group who can work with a range of health sub-sectors and be conversant in a wide variety of methodologies, including new methods for USAID, such as economic evaluations. Retaining this expertise within a specific project would likely require core funding. In addition, the SRG noted an ongoing need to improve evaluation skills among core staff within Missions and among those that support them. It was noted that over the previous five years, increased effort has gone into enhancing the M&E skills among USAID staff. 37 38 Evaluation Report Improving evaluation action plans USAID should consider increasing the support for analyzing, publishing, and disseminating lessons learned from the range of evaluations that are conducted, including through conferences and publications. Should evaluation projects include core funding for permanent technical and operational staff? USAID GH, including the various Offices, needs to be prepared to provide core funding to support in-house expertise and experience, including senior management and technical staff and basic financial tracking operations. This core funding can be used to leverage Mission buy￾in for improved evaluation services and more proactively promote engagement. Core funding will ensure more consistent quality in evaluations and allow for core expertise to be used for rapid reviews and for the design and implementation of full evaluations. Equally important for the design and implementation of development projects, core funding will guarantee that relevant data are collected from project inception and during implementation. One SRG member noted that many evaluations start as impact evaluations, but, because of a lack of existing, useful data, they revert to performance evaluations. Use of small business set-asides SRG members perceived small businesses to be excellent partners, particularly for smaller, targeted projects with a narrow set of defined issues, such as Mission support services. Small businesses were viewed as less useful in large, more complicated projects, where a dramatic scale-up of services is envisioned. Small businesses are also seen as less likely to be nimble and flexible when changes are needed in project implementation. Use of local solutions (indigenous organizations) It was noted that PEPFAR will now require that approximately 70% of project funds go to indigenous organizations. However, USAID’s experience with local solutions has been mixed. While local entities are critical partners in performing rigorous and relevant evaluations, they often lack the capacity and expertise to implement quality evaluations on their own. SRG members reported that the most capable local technical experts are invariably in high demand and often not available for ongoing USAID work. Use of the Broad Agency Announcement This process is a relatively new one for USAID. It is used to solicit proposals from outside groups for certain research and development activities, and it may not be suitable for all projects. In addition, there seem to be very high USAID management costs to review the wide range of responses that are produced as part of the process. The quality of the ultimate products from the Broad Agency Announcement process has yet to be assessed. OVERALL FINDINGS Analysis of the information obtained from the document and website reviews, surveys, and in-depth, structured interviews yielded a number of key products. The first was an expanded Support Mechanism Matrix, as noted previously (Annex 3.1). The second product was the Map of Evaluation Projects (Figure 4), which provides a snapshot of where each of the 12 service support mechanisms that perform evaluations sit relative to two parameters (low vs. high cost and performance vs. impact evaluations). Finding 3.1: The 17 service support projects have provided critical support services to USAID over the past two to five years; though the mechanisms have overlapping functions, they tend to have distinguishing elements in their objectives Findings from the four data sources demonstrate that many of the projects are in their second or third iteration, building on previously successful models, and many provide one or two of the defined service areas (Mission support, technical assistance, or evaluation). However, GH Pro and MEASURE Evaluation are exceptions in providing all three. A relatively new approach to assist with procurements, the Broad Agency Announcement, was used for the initial stages of the procurement of the MERLIN umbrella of projects. This method fosters a collaborative effort between USAID and potential implementing partners during the proposal solicitation process and increases access to local organizations. In the case of the MERLIN project, it encouraged innovation and the use of new tools in evaluation. Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) Among the providers in the evaluation service area, there seems to be a rational spread of mechanisms that have different areas of focus and evaluation capacities—from low￾cost, short-duration performance evaluations to high-cost, long-duration impact evaluations. Between these extremes, there are a number of sector-specific evaluation providers (e.g., HIV/AIDS, behavior change) and a set of evaluation providers that have been set up to experiment with more innovative evaluation methods to provide rapid answers to program operation challenges. There is also a subset of evaluation providers (e.g., MEASURE Evaluation Phase IV) that has a major focus on increasing local capacity to perform evaluations and better utilize the resulting data for improving program effectiveness and efficiency. Figure 4. Map of Evaluation Projects Finding 3.2: The evaluations conducted by various service support projects differ in evaluation complexity and average approximate cost Figure 4 shows a scatter plot that attempts to map evaluation projects using different parameters. This chart primarily serves as a preliminary demonstration of the types of analyses that could be performed and examines performance vs. impact and low and high cost. The X-axis presents a range from performance evaluations (score = 1) to impact evaluations (score = 5). The Y-axis presents the approximate average costs for an evaluation project, from a low of $200,000 to a high of $2,500,000. More extensive 39 40 Evaluation Report data, vetting with the providers, and additional axes (such as the average length from inception to completion for evaluations) would make this a more accurate and useful exercise. These types of analyses would be useful, both for USAID requestors of evaluations and in the strategic planning of the overall service support portfolio. Finding 3.3: Limited information was available about the use of local staff and institutions The survey indicated that evaluation projects used local staff and/or organizations approximately 40% of the time and that approximately 20% of overall budgets were used for funding local groups. The SRG noted that, particularly for evaluations, local staff and institutions provided relevance, context, and access to information more difficult to obtain using international staff. However, for highly technical areas, they noted that only a small number of local groups were often available, and scheduling their participation was challenging. Finding 3.4: Only a small number of projects support short- and long-term mission and institutional staffing—each has their own niche GH Pro primarily focuses on short-term Mission support. This type of support involves providing substitute USAID staff when permanent staff are away and supplemental USAID staff for specific, urgent tasks, such as the development of a PEPFAR Country Operations Plan. Similar mechanisms for short-term staff include Rapid Staff Support Services (formerly Firehouse). GHSI-III and GHFP-II provide longer-term institutional support, primarily for Washington-based offices. All of these projects are often overwhelmed by requests. Interviews with CORs/AORs noted a lack of uniformity across these hiring mechanisms in the areas of performance evaluation, promotion, management, and salaries. There appears to be the need to build some consistency across these hiring mechanisms on these issues of equity, cost management, efficiency, promotions, and performance appraisals. This evaluation indicated that there needs to be fair, transparent practices that manage these challenges and expectations. A stronger, defining identity for each of the hiring mechanisms that could help to assure how staff are treated. Finding 3.5: Our analysis of technical assistance services provided by various mechanisms was limited We were unable to perform an in-depth analysis of this service support area, primarily because technical assistance is so loosely defined, and only a small number of the 17 sampled projects included technical assistance. It was noted that consultants who could provide technical assistance for the implementation of specific activities, such as supply chain management, were often seen as better able to evaluate these activities and more adept at providing actionable recommendations for improvement. Finding 3.6: Short-term performance evaluations have shown progress in using results rapidly Among projects that provide evaluation services, several themes arose during this evaluation. The first is described in this finding, and the remainder in subsequent findings. A relatively small subset of the assessed projects focused on shorter-term (four to six months), relatively low-cost (approximately $200,000) performance evaluations. USAID has made an admirable effort to stimulate innovation in the methodologies for these performance evaluations, with a strong focus on rapid utilization of the results to improve program delivery. Projects under the MERLIN umbrella are a good example. Finding 3.7: Concerns were raised about decreasing support for impact evaluations For those projects that primarily perform impact evaluations, concerns were expressed about decreasing funder (e.g., PEPFAR) support for these studies, due to the lengthy time spans needed, high costs, and perceptions that the primary goal was research oriented rather than Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) results oriented. This evaluation also revealed potential interest to separate research evaluation mechanisms from the work of implementation, which was seen as potentially weakening the research-to-practice process. Finding 3.8: Respondents perceived that USAID increasingly prefers integrated programs that do research and evaluation There is a perception that USAID seems to be moving toward a preference for integrated programs for procurements that do research and evaluation. This integration may lead to a reduced number of awards at the global health research level with an increasing focus on broader, inter-bureau activities. For example, the Office of HIV/AIDS, under a directive from the US Department of State’s Office of the Global AIDS Coordinator, is no longer putting their funds into core activities, creating a new funding landscape. As such, the major challenge is bringing in enough money to help achieve critical momentum and maintain a high enough level of staffing to allow for an evaluation project to focus efficiently. Participants also noted that there had been insufficient cross-Bureau information-sharing when procuring and managing mechanisms that focus on research and primarily impact evaluations. Participants said there needs to be a stronger focus on lessons learned from these projects. Finding 3.9: Respondents perceived a state of flux concerning USAID evaluations The overall environment surrounding the role of evaluations at USAID was described as in a state of flux. Both internal forces (e.g., senior USAID budgeting officers) and external forces (e.g., PEPFAR) are reviewing the overall value of current evaluation activities within USAID. Some concerns revolve around the perception that some evaluations use poorly thought-through evaluation questions and inappropriate methodologies. There is the continued concern that evaluation results are neither widely shared nor likely to influence future activities in a meaningful way. A tension exists between the value of traditional mid-project and end-of-project external evaluations vs. the routine collection of operational data combined with regular population-style surveys. Within USAID, there are perceived ways of improving monitoring and evaluation as a unifying concept. Multiple data sources (e.g., the SRG, COR/AORs) stated that fewer but more strategic evaluations addressing critical questions should become the norm. Every project should not necessarily be required to have either a midterm and end-of-project evaluation. There should be a better use of monitoring systems and routine reviews of program results. When evaluations are needed, then the key questions, methodologies, and analytic processes should be rigorously reviewed by standing internal structures with technical staff (similar to the EPIC—the Evaluation Process Improvement Committee—that has been used by the Population and Reproductive Health (PRH) Office for many years). There was also a perceived need for USAID to increase its capacity to promote and disseminate lessons learned from evaluations through social media, journal publications, etc. CONCLUSIONS Within the broader service support portfolio, GH Pro provides a unique set of operational and technical functions that can be accessed in an efficient and rapid manner. GH Pro can offer temporary Mission support (substitute and supplemental staff), targeted technical assistance, and a special niche in delivering low-cost, short-duration performance evaluations. GH Pro is viewed by USAID clients as flexible, responsive, able to recruit competent, knowledgeable consultants, and able to track and manage financial resources from multiple funding sources. Currently, no other service support project offers all of three of these services. GH Pro is in constant, high demand and consistently responds to a large volume of requests from Mission and HQ USAID clients, with over 500 assignments recorded as of December 2017. Across the entire portfolio of 17 service support projects, there appears to be little overlap in the areas of focus, which may be due partly to a conscious decision to focus on specific service areas when formulating the SOWs for the RFAs or RFPs. The lack of overlap may also be due to the fact that USAID client requesters only request specific 41 42 Evaluation Report services from a project. For example, GH Pro was designed to support five service areas. In addition to evaluation, Mission support, and technical assistance, GH Pro was designed to provide support for research capacity and conference/meeting planning. However, there have been essentially no requests for these services from USAID clients. Thus, the move toward specific areas of focus may have occurred through market forces. Across all of these mechanisms, USAID staff have identified a set of critical operational principles that are required to best serve the needs of the Agency, including the following: • Flexibility: This principle relates to responsiveness in the face of changes in time periods, budgets, SOWs, areas of focus, staffing, and interactions with USAID staff. • Timely responses: USAID staff carry heavy workloads and can operate efficiently only if they receive timely and substantive responses to requests. • Efficiency: All staff respondents noted the quest for “value for money,” the proper and efficient expenditure of all funds, and impeccable financial tracking. • Technical competence: Ideally, expertise and experience within specific development areas are available at all times—whether through permanent core staff or through timely access to external consultants. USAID staff expect competent input in the development, implementation, and evaluation of the work that USAID funds. • Quality products: There was a consensus among respondents that quality, though challenging to define, was a critical goal for all three service support areas. Within USAID there continues to be a significant evolution in the philosophy, goals, operations, and expectations of development work. An excellent example of this is reflected in a recent announcement of the launch of the “Promoting Self-Reliance through USAID’s Program Cycle” initiative, which describes a novel approach to assessing a specific country’s developmental status based on a set of scores that measure self-reliance. The need for routine midterm or end-of-project evaluations is being reassessed. In fact, the overall role of evaluations is under review, with more focus on the quality of the evaluations rather than the number. Major development areas, like the response to the HIV/AIDS epidemic, are increasingly relying on the regular collection of service delivery monitoring data and conducting regular surveys and surveillance activities (e.g., PHIA). The overarching philosophy is that evaluations would be used only for specific performance and impact questions that cannot be addressed through the analysis of monitoring and surveillance data. This approach is not an entirely new concept. During the era when increasing immunization coverage and improving access to modern contraceptives were the major USAID priorities, operational data and major surveys, such as the Demographic Health Survey (DHS) and the Multiple Indicator Cluster Surveys (MICS), were the primary data sources. The challenge, however, is that not all development work lends itself to these monitoring methods. Decreasing the risk of acquiring specific health conditions, increasing equity and access, and the creation of safer and healthier environments all require more expansive and nuanced methods for evaluating their impact. It is unclear at this time whether these views on monitoring and evaluation will be extended to other health and development sectors within USAID. A strategic and thoughtful balance of methodologies will be needed for the USAID of the future. RECOMMENDATIONS For each recommendation, we have noted the finding(s) that were the basis for the recommendation. For USAID on Improving the Process of Designing and Managing the Service Support Portfolio 3.1. Continue the Analysis of the Service Support Portfolio: Due to the lack of a mandate and limited resources, we were not able to pursue further responses to the request for interviews and the use of the survey tool. Nor did we review existing public or internal evaluations of each project that was included. In the future, P3 may wish to expand upon the methods utilized in this evaluation Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) to better understand the extent to which the service support portfolio meets the expectations of USAID clients. Additional data collection would enhance accuracy and precision. (Findings 3.1, 3.2 3.6, 3.8, 3.9) 3.2. Consider Establishing a Cross-Bureau Procurement Approach for Service Support Projects: In the future, GH could establish a Bureau-wide or cross￾Bureau approach to the procurement of service support projects to avoid gaps in services, duplication, unnecessary competition, and inefficiencies. If this approach is not considered realistic, then, at a minimum, consider asking teams to survey other offices and sectors when a new service support project is being procured; the goal would be to better understand existing activities within the intended area of focus and related operational systems. (Findings 3.1, 3.8) 3.3. Facilitate USAID Clients’ Choices of the Most Appropriate Projects: We recommend that relevant USAID staff (e.g., in P3 or PPL) further develop appropriate training, guidelines, and tools to help USAID requestors identify the most appropriate types of service support projects. Ideally, the final decisions would result from a dialogue between the USAID client and the consultants who are performing the specific service area. Capacity￾building could also involve expanding the information available in the USAID GH users’ guide for USAID/ Washington health programs and establishing a reference/ clearinghouse function within USAID (possibly within P3) that could provide further guidance to potential clients on which were the most appropriate support mechanisms for a particular need. (Findings 3.1, 3.2, 3.4) 3.4. Consider Innovative Procurement Processes: The Broad Agency Announcement process was used for the procurement of a set of projects under the MERLIN umbrella. However, there is little evaluation information currently available on the effectiveness, efficiency, and quality of the implementing organizations that have been procured through this process. Monitoring the quality of output from these mechanisms would help in understanding the role of the Broad Agency Announcement in future procurements for service support. (Findings 3.3) 3.5. Increase Oversight and Reconsider COR/AOR Roles and Responsibilities: We recommend a review of COR/AOR staffing. Consider increasing the number of certified CORs and AORs to cover the extensive work required to manage these projects. In addition, consider the curriculum of the certification process, which may not be relevant to all of the tasks at hand. Examine the balance of technical vs. management responsibilities for COR/AOR performance. Develop an ongoing process to allow communication among CORs/AORs about current challenges and solutions concerning the provision of service support to USAID clients. COR/AOR collaboration could involve regular meetings under rigorous supervision to ensure that only critical topics are addressed and that follow-up is mandatory. (Findings 3.4, 3.8) 3.6. Reconsider the Role of Monitoring and Evaluation Within the Context of Development: A number of recommendations suggested by this evaluation’s respondents reflected more forward-thinking attitudes about the role of monitoring and evaluation. These recommendations could be embraced by USAID, including the following: fewer but more strategic evaluations that address critical questions, increased use of monitoring systems, and routine reviews of program results. A reassessment of the Automated Directives System (ADS) would be needed to revise the routine need for midterm or end-of-project evaluations. If evaluations are needed, then the key questions, methodologies, and analytic processes should be rigorously reviewed by defined technical staff as a required part of the approval process. Finally, USAID should consider how to increase its capacity to promote and disseminate lessons learned from evaluations through social media and journal publications. (Findings 3.8, 3.9) For USAID on Designing the Follow-on to GH Pro 3.7. Address the Critical Need gor Core Funding in Follow-on Projects: GH Pro was established without core funding, as such the Project lacked a critical group of core-funded management, technical, and operational staff to establish essential systems. Properly executed core 43 44 Evaluation Report funding as part of the overall budget allows an organization to handle a rapid, high-volume start-up, deal with seasonal lulls in funding, and provide continuous technical expertise that not only ensures continuity but the ability to learn from experience. (Findings 3.1, 3.2) 3.8. Reconsider Multitasking Within Service Support Projects: Combining different service areas into a single project can lead to significant efficiencies, primarily due to the sharing of critical staff, infrastructure, and operational functions. It also can lead to ease of use by clients through one-stop shops. However, when the services provided are too disparate and have unequal demands on resources, then overall project quality will likely suffer. There are strategic combinations that should be considered in new procurements, such as consolidating evaluation with technical assistance or combining Mission staff support functions with medium-term staff hiring services. For example, a consultant who provides technical assistance for supply chain management would potentially conduct a more focused and relevant evaluation of similar projects and their recommendations could be more actionable. (Findings 3.1,3.2,3.7) Midterm Performance Evaluation of the Global Health Program Cycle Improvement Project (GH Pro) ANNEXES 45 INTRODUCTION TO THE EVALUATION ANNEXES The report annexes are organized in the order in which they are referenced in the body of the report and are numbered according to the section of the report in which they are originally referenced. Annexes introduced in the Introduction to the Evaluation section are numbered 0.1 to 0.2. Two annexes that were not referenced in the body of the report that provide evaluator profile information and evaluator conflict of interest disclosures also appear in the ‘zero’ annex section as 0.3 and 0.4, respectively. Annexes referenced in the section of the report addressing Evaluation Question 1 appear as Annexes 1.1-1.8. Annexes referenced in the section of the report addressing Evaluation Question 2 appear as Annexes 2.1-2.8. Annexes referenced in the section of the report addressing Evaluation Question 3 appear as Annexes 3.1-3.5. 0.1. ANNEX: USAID STATEMENT OF WORK PURPOSE OF THE EVALUATION The Global Health Program Cycle Improvement Project (GH Pro) is an $84.2 million, five-year contract that provides USAID operating units working on health-related activitiesin headquarters and the field with short- and medium-termtechnicalservices of consultants with expertise pertaining to health program assessment, design, monitoring and evaluation, and program support through the following five components: 1. Program and Project Evaluation 2. Mission Support 3. Technical Assistance 4. GH Program and Research Management 5. Support for Conferences/Meetings The purpose of this midterm evaluation is to review the project’s performance to date in the context of the landscape of various Global Health Bureau support mechanisms,with the goal of identifying opportunitiesto add value, improve program efficiency and reduce cost. USAID leadership and GH Pro management team will use the findings, conclusions, and recommendationsto improve the technicalsupport the project offers as well as project management for the remainder of its implementation as well as to inform any potential future projects. SUMMARY INFORMATION Project/Activity Title: Global Health Program Cycle Improvement Project Implementing Partner: Dexis Consulting Group Award/ContractNumber: AID-OAA-C-14-00067 Project/Activity Funding: $84.227,086.00 Performance Period August 2014-September 2017 (Awarded 7/2/2014) Active Geographic Regions Global 46 DevelopmentObjective(s) (Dos) Funding Account Source(s) GH/HIDN; GH/C/AIDS;GH/C/POP Requester USAID/Washington/Bureau for Global Health/ Office of Policy, Programs & Planning/ Division of Strategy, Analysis, Evaluation and Outreach AOR/COR: Carl Hawkins BACKGROUND A. Description of the Problem, DevelopmentHypothesis(es), and Theory of Change GH Pro is similar to a series of contracts GH has previously commissioned dating back over thirty years, namely GH TECH, POPTECH, MEDS, ASSIST, and SYNERGY. It was designed to support the GH Bureau’slongstanding commitment to assisting countriesto meet their health and development goals, in line with the 2010 USAID Forward reforms and the Global Health Initiative framework1 to improve health outcomesthrough health systemsstrengthening, by providing GH Bureau headquarters and the field activities with the short- and medium-term technical services through consultantsfor five areas: (1) program and project evaluation; (2) technical assistance; (3) GH program and research management;(4) mission support; and (5) logisticalsupport for meetings/conferences. For evaluation specifically – an important aspect of USAID Forward – GH Pro supports the agency in improving program evaluation methods and increasing implementation of independent evaluations of health projects and programs. B. Summary Strategy/Project/Activity/Intervention to be evaluated The Global Health Program Cycle Improvement Project (GH Pro) is a $84.2 million cost plus fixed-fee, five-year contract. It is a follow-on contract to the 5-year GH Tech Project and GH Tech Bridge contractsI-IV. The prime partner is Dexis Consulting Group with QED as sub-contractor. GH Pro provides USAID operating units working on health-related activitiesin headquarters and the field with the short- and medium-termtechnicalservices of consultants with expertise pertaining to the following five health support components: 1. Program and Project Evaluation ● High quality, transparent, independent, and collaborative evaluations ● Compliantwith USAID and PEPFAR policies ● Mission Support 2. Short- and medium-termstaff assistance ● Managerial and technicalsupport ● Fill gaps from temporary absences and provide surge capacity 3. Technical Assistance ● Technical expertise ● Support for strategic planning, project design, and coordination ● Conduct assessments and reviews 4. GH Program and Research Management 1 GHI target health elementsinclude: HIV/AIDS, Malaria, Maternal Health, Child Health, Nutrition, Family Planning and ReproductiveHealth, Neglected Tropical Disease 47 Number of assignments ● Augment USAID capacity in specialized fields to design, manage, and implement research and policy analysis 5. Support for Conferences/Meetings ● Support to plan, organize, implement, and document meetings, conferences, workshops, and other events GH Pro adheresto the following core operating principles 1. Independence and Impartiality, including a commitment to evidence-informed, transparent, and unbiased decision making, resource investment, learning, and accountability,while avoiding conflict of interest for all activities. 2. Coordination and Collaboration:GH Pro is expected to coordinate and collaborate, as appropriate,with USAID partners (such as the U.S. Centers for Disease Control and Prevention (CDC) and the Global Fund to Fight AIDS, TB, and Malaria (GFATM)) and GH cooperating agency implementing partners (such as U.S. and international non-governmental organizations and private corporations and foundations),to build synergies and improve the effectiveness and efficiency of project activities. 3. Local Capacity Utilization: GH Pro contributesto country health system strengthening by purposefully utilizing (when appropriate) the expertise and capacities of local organizations and individuals within the implementation services. 4. Talent Management and Inclusion: In support of USAID Forward and the Agency’s approach to leadership with inclusion, the GH Pro works to include the participation of a wide variety of stakeholder(USAID staff in Missions and at headquarters; representatives of partner USG agencies;state and local government officials; representatives of other bilateral donors; participantsfrom technical organizations) in its work. C. Summary of the Project/Activity Monitoring, Evaluation, and Learning (MEL) Plan GH Pro Performance Metrics Indicator Measurement Definition Disaggregate Data Source Assignment Management Average time (and range) from SOW submission to GH Pro to TDM signing Total number of assignmentssubmitted to GH Pro for action. Assignments, once entered in the Assignment Tracker, as designated as: 1) Pre-TDM (planning), 2) Active (once TDM is approved/signed), 3) closed (completed), and 4) other,such as delayed or cancelled. Assignments will be disaggregated by type of assignment (TA, Eval, Mission Support, Research, Meeting/Conference Logistic Average number of days from the time GH Pro receives an assignment to the date the TDM is approved/signed, and the assignment becomes active. Assign’t Type: TA, Eval, Mission Support, Research, Mtg Logistic Support Status: planning, active, closed, other Requester: GH, Mission (Region) Assign’t Type: TA, Eval, Mission Support, Research, Mtg Logistic Support Assignment Tracker Assignment Tracker 48 Average time (and Average number of days from the time GH Pro an Assign’t Type: TA, Eval, Assignment range) from approved/signed TDM to the date the assignment Mission Support, Tracker assignment TDM is completed. Assignment completion is defined Research, Mtg approval to as approval of all deliverables (does not include Logistic Support completion posting to the DEC (where applicable) or financial closeout). Evaluations Number of completed evaluationswith quantitative datasets submitted to USAID’s Data Development Library (DDL) The number of evaluation assignmentsthat have quantitative datasets uploaded to USAID’s DDL. Report Tracker Client Satisfaction Average rating score At the close of each assignment, the USAID point Assign’t Type: TA, Eval, GH Pro on performance of GH of contact and key staff consultants will be sent an Mission Support, Performan Pro assignments Assignment performance questionnaire, asking them to rate how the Assignmentwent (GH Pro’s performance and consultant’s performance). Research, Mtg Logistic Support Reviewer Type: Requester (GH, Mission/ Region), Consultant c e Review Survey using Survey Monkey GH Pro Management Total number of GH Pro staff, with number of new hires and number of staff departures Each quarter,the number of current staff working at GH Pro, the number of new hires, and the number of staff who left GH Pro. Project record s Number of consultants in GH Pro Consultant Database Number of Consultantswith a record in the GH Pro Jobscience database. GH Pro Jobscience Consultant database D. Summary of other Projects/Activitieswhich provide GHB and Missions with Evaluation support or TA There are a variety of evaluation mechanisms which GHB and Missions have access to, including: 1. MEASURE Evaluation ( Impact Evaluations) [GHB, Kristen Wares (kwares@usaid.gov)] 2. Eval-ME IDIQ [PPL, Winston Allen, (wallen@usaid.gov)] 3. GH Pro [GHB, Carl Hawkins(chawkins@usaid.gov)] 4. MERLIN [LAB, Sophia van der Bijl (svanderbijl@usaid.gov)] 5. HEARD [GHB, Supriya Madhavan (smadhavan@usaid.gov)] 6. CIRCLE [GHB, Sara Sulzbach (ssulzbach@usaid.gov) 7. SOAR [GHB/OHA] 8. ASPIRES [GHB/OHA] 9. MERLIN ● SPACES MERL ● Rapid Feedback MERL ● DEPA MERL ● Balanced MERL ● ERIE 49 There are severalsupport mechanisms which can provide short-term or longer-term TA to GHB or Mission support, including: 1. GHSI [GHB] 2. GHFP [GHB, Michael Wilburn (mwilburn@usaid.gov)] 3. GH Pro [GHB, Carl Hakwins(chawkins@usaid.gov)] Documentsto review: ● RFA/Q? ● Contract/SOW ● Annual Reports ● Minutes and/or agendasfrom the Project M&E Plan ● Bi-weekly management meeting and monthly evaluation meeting minutes ● Report from the internal review ● Work plans/ SOWs/consultant agreements/consultantCVs for a selection of consultanciesin each of the five health support components ● reports for the selected consultancies ● Financial databases ● Monitoring databases ● Public facing documentsfor other mechanisms ILLUSTRATIVE EVALUATION QUESTIONS The evaluation team should consider the following evaluation questions as a starting point, with the final evaluation questionsto be refined in collaboration between the USAID GH Pro Management Team, the GH Pro Management Team, and evaluators. Evaluators are welcome to look at any aspect of the project that may affect or influence the questions listed below. They are encouraged to talk to a wide range of stakeholders, such as USAID HQ and the Missions(both Missions who have worked with the GH Pro and those who have not), other USG agenciessuch as OGAC and CDC, and consultants who have been engaged by GH Pro. The USAID GH Pro Management Team seeks to identify opportunitiesto add value, improve program quality and efficiency, and reduce cost, and identify how the project may make improvements/adjustments in this phase as well as how it could be improved in potential future procurements. Objective: Identify opportunitiesto add value, improve program quality and efficiency, and reduce cost EvaluationQuestions and Methods: 1. Identify to what extent was the project effective in meeting stakeholder needs for evaluation and TA? 2. Identify areas for improved quality by reviewing the quality of deliverables produced by GH Pro and the extent to which quality assurance processes are incorporated into the project and followed. 50 3. Identify areas for improved efficiency or cost-reduction in key program processes, including a description of unplanned, unanticipated benefits or costs. 4. a) What is the value add of GH Pro to GHB for evaluations considering other, existing mechanisms(e.g. MEASURE Evaluation, CIRCLE, MERLIN, and Eval-ME). b) What is the value add of GH Pro to GHB for TA and Mission Support considering other, existing mechanism (e.g. GHFP and GHSI). The evaluation team will work collaborativelywith the USAID GH Pro management team to develop a detailed workplan, an evaluation matrix linking methods with evaluation questions, data collection strategy, and data collection instrumentsfor the mid-term evaluation The primary methodologiesfor this evaluation could include ● Document Review ● Key Informant Interviews ● Surveys ● Other methodssuggested by the evaluation team ADDITIONAL SOW REQUIREMENTS Additional requirementssuch as a methods matrix, the evaluation team composition, evaluation LOE, final report format, and other evaluation requirements required by ADS 201 will be discussed and documented with HEARD and the evaluation team as appropriate prior to data collection. ANNEX A: DELIVERABLES AND REPORTING REQUIREMENTS Introductory meeting: Within 5 days of being awarded the contract, the evaluation team will have an in-briefing with the USAID GH Pro managementteam and/or COR, for introductions and to discuss the team’s understanding of the assignment, initial assumptions, evaluation questions, methodology, and work plan, to review this Statement of Work (SOW), with special attention to the evaluation objectives and questions. Final Workplan: Within 2 weeks of the award of the contract, a draft work plan for the evaluation shall be completed by the lead evaluator (i.e. the Evaluation Advisor) and presented to the Contracting Officer’s Representative (COR). The work plan will include: (1) the anticipated schedule and logistical arrangements; and (2) a list of the members of the evaluation team, delineated by roles and responsibilities, and (3) a draft outline of the evaluation design. The workplan will be informed by the introductory meeting Evaluation Design: Within 3 weeks of approval of the work plan, the evaluation team must submit to the Contracting Officer’s Representative (COR) an evaluation design (which will become an annex to the Evaluation report). The evaluation design will include: (1) a detailed evaluation design matrix that links the Evaluation Questionsto data sources, methods, and the data analysis plan; (2) draft questionnaires and other data collection instruments or their main features; (3) the list of potential interviewees and sites to be visited and proposed selection criteria and/or sampling plan (must include calculations and a justification of sample size, 51 plans as to how the sampling frame will be developed, and the sampling methodology for quantitative methods); (4) known limitationsto the evaluation design; and (5) a dissemination plan. USAID offices and relevantstakeholders are asked to take up to 10 business days to review and consolidate commentsthrough the COR. Once the evaluation team receivesthe consolidated comments on the initial evaluation design and work plan, they are expected to return with a revised evaluation design and work plan within 10 business days. Midterm Briefing and Interim Meetings: The evaluation team is expected to hold a midterm briefing with the USAID GH Pro management team on the status of the evaluation, including potential challenges and emerging opportunities. The team will also provide the evaluation COR or his/her designee with periodic briefings and feedback on the team’s findings, as agreed upon during the introductory meeting. If desired or necessary,weekly briefings by phone can be arranged. Draft Report Presentation: The evaluation team is expected to hold a presentation to discuss the main findings and recommendationsto USAID. At or before this meeting, the team will share a detailed outline that includes main findings and recommendations. This presentation will be scheduled as agreed upon during the introductory meeting, and should take place before the draft report is submitted. The evaluation team can incorporate feedback from the meeting in the draft report. Draft Report: The draft evaluation reportshould be consistentwith the guidance provided in the USAID Evaluation Report Format, and should describe the findings from the evaluation by separately and comprehensively address each of the objectives and questionslisted in the evaluation design as well as any other issues (discussed in the draft report presentation) the team considersto have a bearing on the objectives of the evaluation. The submission date for the draft evaluation report will be determined in the evaluation work plan. Once the initial draft evaluation report is submitted, the USAID GH Pro management team will have 15 business days in which to review and comment on the initial draft, after which point the COR or his designee willsubmit the consolidated commentsto the evaluation team. The evaluation team will then be asked to submit a revised final draft report 10 business days hence, and again the USAID GH Pro management team will review and send comments on this final draft report within 10 business days of its submission. The evaluation team is welcome to share an early draft or detailed outline that includes main findings and bullets before finalizing the draft evaluation report. Final Report: After receiving the draft report USAID will have 14 business days to respond with a set of consolidated comments. The evaluation team will then submit revisionsto USAID in the form of a PDF file within 10 business days. Unresolved differencesrelated to the evaluation findings or recommendationsshould be included in an annex as a statement of different. Followingsthis, GH Pro staff should also be provided a copy of the final draft report to provide opportunity to comment and to document any significant unresolved differences of opinion in an annex as a statement of difference. The report will be released as a public document on the USAID website dec.usaid.gov. The evaluation final reportshould include an executive summary; introduction; background of the local context and the projects being evaluated; the main evaluation questions; the 52 methodology or methodologies;the limitations to the evaluation; findings, conclusions, and recommendations; and lessons learned (if applicable) as described here. The reportshould be formatted according to the evaluation report template. The executive summary should be 3–5 pages in length and summarize the purpose, background of the project being evaluated, main evaluation questions, methods, findings, conclusions, and recommendations and lessons learned (if applicable). The evaluation methodology shall be explained in the report in detail. Limitationsto the evaluation shall be disclosed in the report, with particular attention to the limitations associatedwith the evaluation methodology (e.g.,selection bias, recall bias, unobservable differences between comparator groups, etc.) The annexes to the report shall include: ● The Evaluation SOW; ● Any statements of difference regarding significant unresolved differences of opinion by funders, implementers, and/or members of the evaluation team; ● All tools used in conducting the evaluation,such as questionnaires, checklists, and discussion guides; ● Sources of information, properly identified and listed; and ● Disclosure of conflict of interest forms for all evaluation team members, either attesting to a lack of conflicts of interest or describing existing conflicts of. In accordance with ADS 201 and AIDAR 752.7005, the contractor will make the final evaluation reports publicly available through the Development Experience Clearinghouse within 30 calendar days of final approval of the formatted report. To ensure the quality of the draft and final evaluation report, and to comply with ADS 201 the documentsshould be evaluated against the following criteria: ● The evaluation reportshould represent a thoughtful,well-researched, and well organized effort to objectively evaluate what worked in the project, what did not and why. ● The evaluation reportshould address evaluation questionsincluded in the final workplan deliverable. ● The evaluation methodology must be explained in detail and all tools used in conducting evaluation should be included in the Annex of the final report. ● Limitationsto the evaluation should be disclosed in the report. ● Findingsshould be specific concise and supported by strong quantitative and qualitative evidence. Data should be presented as facts, not opinions. ● Recommendationsshould be actionable, practical and specific with defined responsibility for the action and should be supported by findings. ● All modificationsto the SOW need to be agreed upon, in writing by USAID. All quantitative data collected by the evaluation team must be provided in machine-readable, non-proprietary formats as required by USAID’s Open Data policy (see ADS 579). The data should be organized and fully documented for use by those not fully familiar with the project or the evaluation. USAID will retain ownership of the survey and all datasets developed. 53 All modificationsto the required elements of the SOW of the contract/agreement,whether in technical requirements, evaluation questions, evaluation team composition, methodology, or timeline, need to be agreed upon in writing by the COR. Any revisionsshould be updated in the SOW that is included as an annex to the Evaluation Report. ANNEX B: LIST OF ADDITIONAL RESOURCES Project Website Other project resources/capacity statements GH Pro Internal Assessment Evaluation Timeframe USAID’s Open Data Policy USAID’s Evaluation Policy USAID’s Assessing and Learning Policy 54 0.2. ANNEX: GH PRO MIDTERM EVALUATION SOW RESPONSE TO STATEMENT OF WORK FOR PERFORMANCE EVALUATION GH PRO Distributed to: Anne Palaia, GH Pro Evaluation Activity Manager, USAID; Neal Brandes, HEARD AOR, USAID Washington, DC Working Draft Updated: October 17, 2017 Health Evaluation and Applied Research Development (HEARD), is funded by United States Agency for International Development (USAID) under cooperative agreement No. AID-OAA-A-17-00002. The project team includes prime recipient, University Research Co., LLC (URC) and sub-recipient research organizations. This draft document was produced for review by the United States Agency for International Development. It was prepared by University Research Co., LLC. The contents of this document are the sole responsibility of University Research Co., LLC and do not necessarily reflect the views of USAID or the United States Government. SUMMARY INFORMATION Project/Activity Title: Global Health Program Cycle Improvement Project Implementing Partner: Dexis Consulting Group Award/ContractNumber: AID-OAA-C-14-00067 Project/Activity Funding: $84.227,086.00 Performance Period August 2014-September 2017 (Awarded 7/2/2014) Active Geographic Regions Global Funding Account Source(s) GH/HIDN; GH/C/AIDS; GH/C/POP AOR/COR: Carl Hawkins Requester USAID/Washington/Bureau for Global Health/ Office of Policy, Programs & Planning/ Division of Strategy, Analysis, Evaluation and Outreach 55 PURPOSE OF THE EVALUATION We believe we understand the purpose of the midterm evaluation: The purpose of this midterm evaluation is to review the project’s performance to date in the context of the landscape of various Global Health Bureau support mechanisms,with the goal of identifying opportunitiesto add value, improve program quality and efficiency and reduce cost. We have noted the intended use of the evaluation findings as described in the Scope of Work (Annex 1). USAID leadership and GH Pro management team will use the findings, conclusions, and recommendationsto: improve the technicalsupport the project offers as well as project management for the remainder of its implementation as well as to inform any potential future projects. Based on our preliminary discussions with USAID, we have demarcated two distinct purposes for the evaluation, with the greater weight of interest on the second of the two: Purpose 1: to improve the project management and the technicalsupport that the project offers for the remainder of its implementation; Purpose 2: to inform any potential future projects. The demarcation of the five key project areas described for GH Pro include; 1. Program and Project Evaluation • High quality, transparent, independent, and collaborative evaluations • Compliant with USAID and PEPFAR policies 2. Mission Support • Short- and medium-term staff assistance • Managerial and technical support • Fill gaps from temporary absences and provide surge capacity 3. Technical Assistance • Technical expertise • Support for strategic planning, project design, and coordination • Conduct assessments and reviews 4. GH Program and Research Management • Augment USAID capacity in specialized fields to design, manage, and implement research and policy analysis 5. Support for Conferences/Meetings • Support to plan, organize, implement, and document meetings, conferences, workshops, and other events We understand based on our preliminary discussions with USAID and GH Pro that three major areas of work essentially encompass a major proportion of the GH Pro Project’s work to date and should be the central focus of the evaluation, namely: 1. To provide short- and medium-term managerial and technical staff assistance support to Missions to fill gaps from temporary absences and address surge capacity requirements 2. To conduct high quality Program and Project Evaluations that are transparent, independent, collaborative and compliant with USAID and PEPFAR policies 3. To provide technical assistance for strategic planning, project and program design, coordination and conduct of assessments and reviews. 56 The evaluation team will also strive to collect all available data as to why the other two Program Areas were undersubscribed. UNDERSTANDING OF THE BACKGROUND OF THE EVALUATION. We note in the Background Section of the SOW: 1. The four GH Pro Operating Principals; • Independence and Impartiality • Coordination and Collaboration • Local Capacity Utilization • Talent Management and Inclusion: We further noted in the Background Section that: • There are a variety other Projects/Activities and evaluation mechanisms listed through which the GHB and Missions have access to evaluation support or TA ; • There are several support mechanisms listed which can provide short-term or longer-term TA to the GHB and Missions; and that • The GHB target health elements include: HIV/AIDS, Malaria, Maternal Health, Child Health, Nutrition, Family Planning and Reproductive Health, Neglected Tropical Diseases We have noted the expectation that: The evaluation team will work collaboratively with the USAID GH Pro management team to develop a detailed workplan, an evaluation matrix linking methods with evaluation questions, data collection strategy, and data collection instruments for the mid-term evaluation. ILLUSTRATIVE EVALUATION QUESTIONS We note from the Evaluation Questions section of the SOW that: • The final evaluation questions will be refined in collaboration among the USAID GH Pro Management Team, the GH Pro Management Team, and evaluators; • The evaluators are welcome to look at any aspect of the project that may affect or influence the evaluation questions; and that • The evaluators are encouraged to talk to a wide range of stakeholders, such as USAID HQ and the Missions (both Missions who have worked with the GH Pro and those who have not), other USG agencies such as OGAC and CDC, and consultants who have been engaged by GH Pro. Following our initial proposal and discussions with USAID and GH Pro colleagues,we have provided some additional edits to the illustrative evaluation questions provided in the SOW and propose three overarching questions for the evaluation with: Q1 focused on project effectiveness and efficiency; Q2 on quality improvement and standards; and Q3 focused on project strategy and comparative value. Q1. To what extent was the GH Pro Project effective and efficient in meeting stakeholder needs in three key project areas? (Program and Project Evaluation, Mission Support and Technical Assistance) 57 Q2. To what extent are USAID and PEPFAR evaluation quality standards employed and achieved by the GH Pro Project? Q3. What is the comparative value of GH Pro to the USAID Global Health Bureau and Missions for the three key project areas considering other, existing mechanisms? SUMMARY OF PROPOSED APPROACH We have summarized our proposed approach in the following table, dividing the evaluation into three parallel components based on a combination of timing, stakeholder and operational considerations. Part I: Project Effectiveness and Efficiency This component of the evaluation is envisioned to build on the regular reporting and self￾evaluation completed by GH Pro with the assistance of an external consultant in 2016. Part 1 will include: a. Document Review, including sampling of products for Part 2 of the evaluation. b. Further analysis of the Self-Evaluation data, where feasible; and A. Follow-up Survey of project stakeholders and potential stakeholders; We will integrate the results of the GH Pro team’s self-assessment with additional insights gleaned from our own review of the available documents and discussions with GH Pro staff. We propose to actively engage the GH Pro team in this part of the evaluation. Several outputs of Part 1 of the evaluation will support the work of the Quality Reference Group and the Strategy Reference Group in Parts 2 and 3 of the evaluation. The new data gathering effort will focus on stakeholder experience/views- and non-stakeholder or ‘potential stakeholders’ views. For the country based component of this work, we propose to utilize evaluation professionals in HEARD regional hubs (West Africa, East Africa, South Asia and Southeast Asia). • The Document Review will elaborate the GH Pro Performance Metrics within the Summary of the Project/Activity Monitoring, Evaluation, and Learning (MEL) Plan to identify the documentation required for the Reference Groups’ subsequent product/process assessments to answer Q2 and Q3. It will include within its review the list of Documents to Review provided, including the Self-Assessment completed by GH Pro with the assistance of a consultant. • We propose to include a strategic reflection by GH Pro on the utility and application of their four operating principals in order to propose early in the process some specific areas of perceived current and future strengths to also inform the evaluation. • We also propose to elicit specific recommendations from the GH Pro team that will illuminate their existing ideas on future design considerations to inform the work of the Strategy Reference Group. Part II: Project Quality We propose to identify from our HEARD Partners a Quality Reference Group of three evaluation experts to independently review and consolidate their views on the extent to which USAID and PEPFAR evaluation quality standards have been achieved by the GH Pro Project. We will also explore with the Quality Reference Group possible ways to assess utility and value of evaluations that have been performed. These discussions will be utilized to inform the work of the Strategy Reference Group described in the next section. 58 Part III: Project Strategy and Comparative Value In consultation with USAID, we propose to convene a Strategy Reference Group of experienced program managers who have previously been ‘end users’ of project evaluations in a more forward-looking exercise to critically review the products and processes of the GH Pro project compiled and analyzed in Parts1 & 2. We will rely on the more strategic and interpretive expertise of the Strategy Reference Group to weigh the findings of the evaluation and to develop specific recommendations to USAID on both purposes of the evaluation, namely to: • to improve the project management and the technicalsupport the project offers as well for the remainder of its implementation; and • to inform any potential future projects and mechanisms for evaluation and capacity strengthening to achieve USAID’s mission goals. 59 Component 1 GH Pro Team Current Stakeholders Potential Stakeholders Quality Ref Group Expert Ref Group Q1. To what extent was the GH Pro Evaluation Project effective and efficient in meeting stakeholder needs in Questions three key project areas? (Program and Project Evaluation, Mission Support and Technical Assistance) 1.1.What areas can be identified for improved efficiency or cost￾reduction in key program processes, including a description of unplanned, unanticipated benefits or costs. 1.2.What was the relative effort, key products/processes in the three key project areas ()? 1.3.How well do GH Pro operating principles apply – and how well have they been applied in GH Pro work? 1.4.What areas can be identified for improved efficiency or cost￾reduction in key program processes, including unplanned, unanticipated benefits or costs? 1. Evaluator-Facilitated Document Review 2. Survey and Key Informant Interviews (and Focus Group?) Approach 3. External-Facilitator Draft Recommendation Process 1. GH Pro Performance Metrics 2. Survey and Interview Guide Required 3. Recommendation Framing Inputs Guide Component 2 Q2. To what extent are USAID and PEPFAR evaluation quality standards employed and achieved by the GH Pro Project? 2.1. Are USAID and PEPFAR evaluation quality standards in general use? 2.2. What areas for improved quality can be identified by reviewing the quality of deliverables produced by GH Pro? 2.3. To what extent are quality assurance processes incorporated into the project and followed? 1. Quality Reference Group Review of a GH Pro Evaluation products 1.Sample of GH Pro evaluation products identified in Part 1 of the evaluation Component 3 Q3. What is the added value of GH Pro to the USAID Global Health Bureau and Missions for the three key project areas considering other, existing mechanisms? 3.1. What do USAID clients of GH Pro see as the unique and comparative value of the mechanism? 3.2. Where do USAID clients see overlap with other USAID (or non-USAID) mechanisms? And where redundancy exists, to what extent is that seen as desirable redundancy or choice? 3.3. To what extent does/should GH Pro collaborate and coordinate its efforts with other USAID mechanisms? 3.4. To what extent has or should GH Pro be positioned as a USAID supported mechanism in support of other USG supported global health efforts (multilateral, NGO)? 1. Strategy Reference Group review of overall evaluation documentation and Outputs of Part 1 and 2 of the Evaluation 2. Focus Group Discussion 1. Outputs of Part 1 and 2 of the Evaluation 2. Recommendation Framing Guide 3. Annotated Draft Outline of Report 60 DELIVERABLE AND REPORTING REQUIREMENTS We have noted the Deliverables and Reporting Requirements in Annex A of the SOW (see 1.1. Annex USAID Statement of Work above) and have translated them into the Milestone Chart below. October November Elements of Scope of Work 2 3 4 5 1 2 3 4 1. Launch/Updated SOW Meeting F a. Presentaton of Revised SOW X X b. (Updated) Evaluation Questions X X F c. (Updated) Key Points for Clarification X X 2. Final Workplan D F a. Schedule X X X X X b. Logistics arrangement X c. Evaluation Team Members, Roles, Responsib X X X X X d. Draft outline of the Evaluation Design X X X 3. Evaluation Design a. evaluation design matrix X b. draft questionnaires/data instruments X c. list of potential interviewees X d. initial list of countries X e. Draft submitted f. USAID Review g. Revised Evaluation Design and Work plan 4. Operations a. Document Review b. Survey c. Key Informant Interviews d. Data Analysis e. Engagement of Quality Reference Group f. Engagement of Strategy Reference Group g. QRG review of preliminary findings h. QRG recommendations i. SRG review of preliminary findings j. SRG recommendations 5. Interim(I) Midterm(M) and Final (F) I I Briefings 6. Draft (D) and Final(F) Reports/ a. Draft Report b. USAID Comment c. Revised Draft Report d. USAID Comment e. Final Report: f. GH Pro Comment g. DE Clearinghouse Submission Key: C = Comments provided; D = draft submission; F = final submission December 1 2 3 4 X X X X X X X X X X X X D X X I 1 X X X January 2 3 4 C X F X X X X X I OPERATIONS February 5 1 2 3 4 1 X X X X X X X X X X X X X X X X X X I March 2 3 X X M 4 X 1 April 2 3 I X X 4 5 I D X 1 X May 2 3 I C X X 4 I R X 1 P X June 2 3 F C X X C 4 F In developing the Milestone Chart we included each of the Annex A deliverables with the preparatory and review times specified for each including: the Introductory Meeting, Final Workplan, Evaluation Design, Midterm Briefing and Interim Meetings, Draft Report Presentation, Draft Report, and Final Report. 61 We have noted that the required ‘launch time’ is two full calendar months and the ‘landing time’ an additional 4 months, creating significant time pressure on the ‘operational time’ squeezed within. For that reason we have proposed (in red) a set of activities ‘2a. Component I and II Evaluation Design” that we would like to move forward within a month of initiation and prior to the completion of the design and review of the later evaluation Components III and IV. We have proposed a highly iterative/interaction approach with both the USAID GH Pro Team and the GH Pro management team to enable an adaptive design process as the evaluation progresses with each subsequent component informed by – and building on – the previous component. FORMATTING, REPORTING AND ADDITIONAL SOW REQUIREMENTS We have noted the various formatting, reporting and additional requirements SOW including that: • Additional requirements such as a methods matrix, the evaluation team composition, evaluation LOE, final report format, and other evaluation requirements required by ADS 201 will be discussed and documented with HEARD and the evaluation team as appropriate prior to data collection; • The evaluation reportshould be formatted per the evaluation report template; • The requirements for a detailed explanation of methodologies and their limitations; • The necessary annexes to the report.; • The requirements in accordance with ADS 201 and AIDAR 752.7005, to make the final evaluation reports publicly available through the Development Experience Clearinghouse within 30 calendar days of final approval; • The requirements of USAID’s Open Data policy (ADS 579) to provide all quantitative data collected by the evaluation team in machine-readable, non-proprietary formats. ILLUSTRATIVE BUDGET We have attached an illustrative budget which includes the assumptions that: • The evaluation can begin rapidly with Components I and II even as the Evaluation Technical Leader and Expert Reference Group are being identified/recruited; • The Evaluation Team Leader will be chosen from a shortlist of three shared with USAID for their ‘no objection’; • The Quality Reference Group and Strategy Reference Group will be assembled in close consultation with USAID employing all appropriate conflict-of-interest safeguards; • The frequent (biweekly) consultations between USAID and the Evaluation Team will be progressively more virtual and brief, prepared and minuted by the Evaluation Team. PLACEHOLDER (ANTICIPATED) RECOMMENDATIONS PROJECT EFFECTIVENESS AND EFFICIENCY 1. On the balance of Core team/Partner/Ad Hoc Technical resources required to best achieve stated mission: • whether or not there is a need for a fixed evaluation team or partnership arrangement • whether or not consultant teams are composed of individuals at appropriate levels - and how this impacts costs) 62 2. On the balance of local talent utilization required to achieve USAID Forward strategic objectives and USAID experienced staff required for efficiently addressing staff substitution functions 3. On the balance of central versus Mission resources required to best achieve stated mission: • implications on speed and availability of resources for study scoping • implications for responsiveness to clients PROJECT QUALITY 4. On the utility of current/future Quality Improvement Strategy 5. On the implications of the structure/function/use mix of technical resources on the capacity for ‘cutting edge’ quality products. • What is the most effective balance to be achieved between agency/situational context and methodological sharpness? 6. On how the adherence to GH Pro Operating Principals (Independence and Impartiality, Coordination and Collaboration, Local Capacity Utilization, and Talent Management and Inclusion) contribute to the quality of project products PROJECT STRATEGY AND COMPARATIVE ADVANTAGE 7. On the comparative advantage GH Pro has or should pursue with respect to its three basic functions. ( e.g. is there a difference between what GH Pro's specialization/repertoire SHOULD be versus what it is?) 8. On the value to USAID of GH Pro collaborating/coordinating/competing its efforts with other USAID mechanisms. 9. On the value to USAID of GH Pro being more or less positioned as a USAID supported mechanism in support of other USG supported (multilateral, NGO) global health efforts. BUDGET TOTAL COST $ 226,894 63 0.3. ANNEX: EVALUATOR PROFILES EVALUATION TEAM LEAD: PAUL DE LAY, MD, DTM&H (LOND) Dr. De Lay is a global health expert who has spent decades contributing to policies, programs, technical information, guidelines and best practices related to all sectors involved in the HIV/AIDS response. Most recently, Dr. De Lay served as Deputy Executive Director for the Joint United Nations Programme on HIV/AIDS. He has held multiple leadership positions including Director, Evidence, Monitoring and Policy, UNAIDS in Geneva, Switzerland, and Chief of the HIV/AIDS Division, Global Bureau for Population, Health, and Nutrition, United States Agency for International Development (USAID) in Washington, DC. Dr. De Lay has also served as a Team Leader/Epidemiologist for the World Health Organization, Global Programme on AIDS in Malawi, where he assisted the Ministry of Health with planning and implementing that country’s AIDS control activities. Dr. De Lay has approximately 30 years of experience in monitoring and evaluation of local, national and international programs. These include: • Author or co-author on multiple articles, book chapters on evaluating complex programs, including a focus on the politics of M&E. • Developing curriculum and training of monitoring and evaluation specialists, (e.g. establishing the UNAIDS M&E Country Advisors posts) • Managing the analysis and writing of major global evaluation reports, including annual reports to the UN Secretary General, UNAIDS Global Reports on the AIDS Epidemic. • Participating in or managing major programmatic evaluations, including evaluating the AIDS Accountability Index, The Global Plan for Elimination of HIV Infections in Children and Keeping Mothers Healthy, the Second Independent Evaluation of UNAIDS (2009), the Five Year Evaluation of the Global Fund for AIDS, TB, and Malaria (2009), and the evaluation of PEPFAR (2007) Earlier in his career, Dr. De Lay was Medical Director of Refugee Medical Services for the City of San Francisco Department of Public Health, where he established medical screening protocols and ongoing care for more than 35,000 refugees who had settled in San Francisco. Dr. De Lay received a BS in biology in 1971 from the University of California at Santa Cruz, California (junior year in neurobiology, University of Sussex, England) and graduated from the University of California at Davis School of Medicine in 1975. He completed his medical internship at the United States Public Health Service Hospital in San Francisco, California. Dr. De Lay received a Diploma of Tropical Medicine and Hygiene from the London School of Hygiene and Tropical Medicine in 1980 followed by a Residency and Fellowship with Board Certification in Preventive Medicine and Public Health at the United States Public Health Service Hospital, San Francisco in 1981. QUALITY REVIEW LEAD: ROGER MYRICK, PHD Dr. Myrick has extensive domestic and international experience and expertise in HIV monitoring and evaluation. As the Deputy Director of Programs and Director of Monitoring and Evaluation (M&E) in the University of California, San Francisco’s Global Strategic Information (UCSF-GSI) group, he is responsible for leading a team of M&E specialists in the development of M&E tools and the provision of capacity 64 building technical assistance in countries. In addition to overseeing UCSF-GSI’s externally-focused global M&E capacity development and technical assistance portfolio for PEPFAR countries, Dr. Myrick provides leadership internally to UCSF-GSI’s technical assistance, capacity development, transition, and country ownership efforts, including developing frameworks, documentation, metrics, and tools to measure externally-focused activities. ABBAS ALAWIEH, MPH Mr. Alawieh served as a Research Associate in support of the data collection and analysis for the evaluation. HANA AZMAN FIRDAUS, MPH Ms. Azman Firdaus’ expertise includes field experience in routine program monitoring, program evaluations, data quality assessments, and capacity building workshops. In addition, she has and continues to be heavily engaged in development, implementation, analyses and utilization of data to improve program performances. DANIELLE CHARLET, MD, PHD Dr. Charlet is Associate Director for Technical Support with USAID’s Health Evaluation and Applied Research Development (HEARD) Project. She is a global health scientist with experience in health program design and management, basic and applied research, and promoting effective and innovative approaches for improving healthcare. Dr. Charlet has conceptualized and designed an integrated child health and nutrition program in a rural region in South India; established effective data collection and monitoring practices for a non-profit organization; and has conducted global health research in a variety of subject areas, including maternal, newborn, and child health; community care; human resources for health; and health financing. She additionally has experience working on measles, malaria, and tuberculosis studies. AMANDA NACE, MPH Ms. Nace is a public health professional with twelve years of experience researching, implementing, and evaluating public health programs, of which five years were based internationally in developing country contexts, and has experience working with the U.S. Centers for Disease Control and Prevention, Peace Corps, and foreign Ministries of Health. Ms. Nace is currently working on her Doctor of Public Health degree from the City University of New York (CUNY) Graduate School of Public Health and Health Policy. 65 NEIA PRATA MENEZES, MPH Ms. Prata Menezes supports various strategic information and monitoring and evaluation activities under the UCSF-GSI portfolio. She provides technical input on the development of instruments, implementation of assessments and evaluations, and analysis of data. TRICIA RYAN, MPH Ms. Ryan served as a Research Associate in support of the evaluation design. JAMES M. SHERRY, MD, PHD Dr. Sherry is currently Professor of Immigrant, Refugee, and Global Health at the CUNY Graduate School of Public Health and Health Policy at the City University of New York. He is also Director of USAID’s Health Evaluation and Applied Research Development (HEARD) Project. Dr. Sherry has over 25 years of experience in global health ranging from policy, political, government, and institutional development experience at UNICEF, UNAIDS, and WFP. In his various roles, he has supported the establishment of the Children’s Vaccine Initiative, the re-establishment of basic health services in postwar Rwanda, negotiations around global health policy by the UN General Assembly and Security Council, and the design of the Ending Child Hunger Initiative (REACH). In his current role as the Director of USAID’s HEARD Project, Dr. Sherry oversees the integration of the Project’s research agenda, which creates science collaborations, research programs, and evidence/data sharing efforts with a wide-range of research partners and multi-country stakeholders in LMICs. Additionally, his experience includes multi-country health monitoring and mapping systems development – in immunization, child and adolescent health promotion; community-based surveillance, disease control/eradication, humanitarian emergencies, and the global HIV/AIDS response. Dr. Sherry’s implementation science expertise focuses on issues related to the large-scale international program implementation. SAMANTHA SKI, DRPH, MA Dr. Ski is an Implementation Research Scientist with USAID’s Health Evaluation and Applied Research Development (HEARD) Project, where she focuses on evaluation design and supporting stakeholder engagement and evidence translation within research systems. She is well-versed in quantitative and qualitative research methods, and has experience designing and executing several studies examining the effects of health policies at the international, national, and institutional levels. Ms. Ski’s research experience includes analysis of PMTCT policy adoption across five countries; testing the effect of policy adoption on PMTCT service delivery outcomes; a budget analysis of diagonal investments in sexual and reproductive health across one country’s Global Fund HIV/AIDS grant portfolio; stakeholder opinion surveys (e.g. exploring the understanding of the global health architecture among country-level practitioners and policy makers). 66 0.4. ANNEX: NON-DISCLOSURE AND CONFLICTS AGREEMENTS 67 UNIVERSITY RESEARCH Co., 5404 Wisconsin Avenue, Suite 800 LLC Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-8427 www.urc-chs.com NON-DISCLOSURE AND CONFLICTS AGREEMENT Project: Health Evaluation Applied Research Development (HEARD) Project Assignment Title: Performance Evaluation of the USAID's Global Health Program Cycle Improvement Project (GH Pro) I. Intending to be legally bound, I hereby accept the obligations contained in this agreement in consideration of my being granted access to sensitive data. As used in this Agreement, sensitive data is marked or unmarked "sensitive but unclassified information" (SBU), including oral communications, that meets the standards set by Office of Management and Budget (0MB) Circular A-130 Appendix 3 and the U.S. Agency for International Development (USAID) Automated Directives System (ADS.) I understand that any data or systems of records protected from unauthorized disclosure by the provisions of Title 5, United States Code Sections 552 (often referred to as "The Freedom oflnformation Act") and 552a ("The Privacy Act") is/are sensitive data. In addition, other categories of information, including but not limited to medical, personnel, financial, investigatory, visa, law enforcement or other information which, if released, could result in harm or unfair treatment to any individual or group, or could have a negative impact upon foreign policy or relations, or USAID's mission. 2. I understand and accept that by 9eing granted access to ·sensitive data, special confidence and trust has been placed in me by the United States Gbvemment. 3. I acknowledge I have been given access_to USAID sensitive data to facilitate the performance of duties assigned to me for compensation. I understand it is my responsibility to safeguard sensitive data disclosed to me, and to refrain from disclosing sensitive data to persons not requiring access for performance of official duties. Before disclosing sensitive data, I must determine the recipient's "need to know" or "need to access" sensitive data for USAID purposes. 4. Any breach of this Agreement may result in the termination ofmy access to Sensitive Data, which, if such termination effectively negates my ability to perform my assigned duties, may lead to the termination ofmy employment or other relationships with the Departments or Agencies that granted my access. 5. [ have reviewed my employment (past, present and under consideration) and financial interests, as well as those ofmy household family members, and certify that, to the best of my knowledge and belief, I have no actual or potential conflict of interest that could diminish my capacity to perform my assigned duties in an impartial and objective manner. 6. I will not use Sensitive Data, while working on this assignment or thereafter, for personal gain or detrimentally to USAID, or disclose or make available all or any part of the Sensitive Data to any person, firm, corporation, association, or any other entity for any reason or purpose whatsoever, directly or indirectly, except as may be required for the benefit USAID. 7. Misuse of government Sensitive Data could constitute a violation, or violations, of United States criminal law, and Federally-affiliated workers (including some contract employees) who violate 68 U n iv e r s it y R e s e a r c h Co., LLC 5404 Wisconsin Avenue, Suite 800 Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-8427 www.urc-chs.com privacy safeguards may be subject to disciplinary actions, a fine of up to $5,000, or both. In particular, U.S. criminal law (18 USC § 1905) protects confidential information from unauthorized disclosure by government employees. There is also an exemption from the Freedom of Information Act (FOIA) protecting such information from disclosure to the public. Finally, the ethical standards that bind each government employee also prohibit unauthorized disclosure (5 CFR 2635.703). 8. All Sensitive Data to which I have access or may obtain access by signing this Agreement is now and will remain the property of, or under the control of, the United States Government. I agree that I must return all Sensitive Data which has or may come into my possession a. upon demand by an authorized representative of the United States Government; b. upon the conclusion of my employment or other relationship with the Department or Agency that last granted me access to The Global Health Program Cycle Improvement Project (GH Pro) documentation, Sensitive Data; or c. upon the conclusion of my employment or other relationship that requires access to Sensitive 9. Notwithstanding the foregoing, I shall not be restricted from disclosing or using Sensitive Data that: Data. a. is or becomes generally available to the public other than as a result of an unauthorized disclosure by me; b. becomes available to me in a manner that is not in contravention of applicable law; or c. is required to be disclosed by law, court order, or other legal process. ACCEPTANCE The undersigned accepts the terms and conditions of this Agreement. Name Title 69 n O ______jo J 5. / a.o/7 Signature ' ' Dai^e / ' R T ) E 1 / 9 */___________ r . , l T * . X 7 ~ University Research Co., LLC 5404 Wisconsin Avenue, Suite 800 Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-8427 www.urc-chs.com NON-DISCLOSURE AND CONFLICTS AGREEMENT Project: Health Evaluation Applied Research Development (HEARD) Project Assignment Title: Performance Evaluation of the USAID’s Global Health Program Cycle Improvement Project (GH Pro) 1. Intending to be legally bound, I hereby accept the obligations contained in this agreement in consideration of my being granted access to sensitive data. As used in this Agreement, sensitive data is marked or unmarked “sensitive but unclassified information” (SBU), including oral communications, that meets the standards set by Office of Management and Budget (OMB) Circular A-130 Appendix 3 and the U.S. Agency for International Development (USAID) Automated Directives System (ADS.) I understand that any data or systems of records protected from unauthorized disclosure by the provisions of Title 5, United States Code Sections 552 (often referred to as “The Freedom of Information Act”) and 552a (“The Privacy Act”) is/are sensitive data. In addition, other categories of information, including but not limited to medical, personnel, financial, investigatory, visa, law enforcement or other information which, if released, could result in harm or unfair treatment to any individual or group, or could have a negative impact upon foreign policy or relations, or USAID’s mission. 2. I understand and accept that by being granted access to sensitive data, special confidence and trust has been placed in me by the United States Government. 3. I acknowledge I have been given access to USAID sensitive data to facilitate the performance of duties assigned to me for compensation. I understand it is my responsibility to safeguard sensitive data disclosed to me, and to refrain from disclosing sensitive data to persons not requiring access for performance of official duties. Before disclosing sensitive data, I must determine the recipient’s “need to know” or “need to access” sensitive data for USAID purposes. 4. Any breach of this Agreement may result in the termination of my access to Sensitive Data, which, if such termination effectively negates my ability to perform my assigned duties, may lead to the termination of my employment or other relationships with the Departments or Agencies that granted my access. 5. I have reviewed my employment (past, present and under consideration) and financial interests, as well as those of my household family members, and certify that, to the best of my knowledge and belief, I have no actual or potential conflict of interest that could diminish my capacity to perform my assigned duties in an impartial and objective manner. 6. I will not use Sensitive Data, while working on this assignment or thereafter, for personal gain or detrimentally to USAID, or disclose or make available all or any part of the Sensitive Data to any person, firm, corporation, association, or any other entity for any reason or purpose whatsoever, directly or indirectly, except as may be required for the benefit ITS ATP 7. Misuse of government Sensitive Data could constitute a violation, or violations, of United States criminal law, and Federally-affiliated workers (including some contract employees) who violate 70 U n iv e r s it y Re s e a r c h Co., LLC 5404 Wisconsin Avenue, Suite 800 Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-8427 www.urc-chs.com privacy safeguards may be subject to disciplinary actions, a fine of up to $5,000, or both. In particular, U.S. criminal law (18 USC § 1905) protects confidential information from unauthorized disclosure by government employees. There is also an exemption from the Freedom of Information Act (FOIA) protecting such information from disclosure to the public. Finally, the ethical standards that bind each government employee also prohibit unauthorized disclosure (5 CFR 2635.703). 8. All Sensitive Data to which I have access or may obtain access by signing this Agreement is now and will remain the property of, or under the control of, the United States Government. I agree that I must return all Sensitive Data which has or may come into my possession a. upon demand by an authorized representative of the United States Government; b. upon the conclusion of my employment or other relationship with the Department or Agency that last granted me access to The Global Health Program Cycle Improvement Project (GH Pro) documentation, Sensitive Data; or c. upon the conclusion of my employment or other relationship that requires access to Sensitive Data. 9. Notwithstanding the foregoing, I shall not be restricted from disclosing or using Sensitive Data that: a. is or becomes generally available to the public other than as a result of an unauthorized disclosure by me; b. becomes available to me in a manner that is not in contravention of applicable law; or c. is required to be disclosed by law, court order, or other legal process. ACCEPTANCE The undersigned accepts the terms and conditions of this Agreement 71 _____ - _________________ t(s D-®/ j L . f ^ ^ Sjjyp&tTreT~ Date v ^ £ \ v ^ - w Name \ Title ' ' University Research Co., LLC 5404 Wisconsin Avenue, Suite 800 Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-8427 www urc-chs.com NON-DISCLOSURE AND CONFLICTS AGREEMENT Project: Health Evaluation Applied Research Development (HEARD) Project Assignment Title: Performance Evaluation of the USAID’s Global Health Program Cycle Improvement Project (GH Pro) 1. Intending to be legally bound, I hereby accept the obligations contained in this agreement in consideration of my being granted access to sensitive data. As used in this Agreement, sensitive data is marked or unmarked “sensitive but unclassified information” (SBU), including oral communications, that meets the standards set by Office of Management and Budget (OMB) Circular A-130 Appendix 3 and the U.S. Agency for International Development (USAID) Automated Directives System (ADS.) I understand that any data or systems of records protected from unauthorized disclosure by the provisions of Title 5, United States Code Sections 552 (often referred to as “The Freedom of Information Act”) and 552a (“The Privacy Act”) is/are sensitive data. In addition, other categories of information, including but not limited to medical, personnel, financial, investigatory, visa, law enforcement or other information which, if released, could result in harm or unfair treatment to any individual or group, or could have a negative impact upon foreign policy or relations, or USAID’s mission. 2. I understand and accept that by being granted access to sensitive data, special confidence and trust has been placed in me by the United States Government. 3. I acknowledge I have been given access to USAID sensitive data to facilitate the performance of duties assigned to me for compensation. I understand it is my responsibility to safeguard sensitive data disclosed to me, and to refrain from disclosing sensitive data to persons not requiring access for performance of official duties. Before disclosing sensitive data, I must determine the recipient’s “need to know” or “need to access” sensitive data for USAID purposes. 4. Any breach of this Agreement may result in the termination of my access to Sensitive Data, which, if such termination effectively negates my ability to perform my assigned duties, may lead to the termination of my employment or other relationships with the Departments or Agencies that granted my access. 5. I have reviewed my employment (past, present and under consideration) and financial interests, as well as those of my household family members, and certify that, to the best of my knowledge and belief, I have no actual or potential conflict of interest that could diminish my capacity to perform my assigned duties in an impartial and objective manner. 6. I will not use Sensitive Data, while working on this assignment or thereafter, for personal gain or detrimentally to USAID, or disclose or make available all or any part of the Sensitive Data to any person, firm, corporation, association, or any other entity for any reason or purpose whatsoever, directly or indirectly, except as may be required for the benefit USAID. 7. Misuse of government Sensitive Data could constitute a violation, or violations, of United States criminal law, and Federally-affiliated workers (including some contract employees) who violate 72 U n iv e r s it y Re s e a r c h Co., LLC 5404 Wisconsin Avenue, Suite 800 Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-8427 www.urc-chs.com privacy safeguards may be subject to disciplinary actions, a fine of up to $5,000, or both. In particular, U.S. criminal law (18 USC § 1905) protects confidential information from unauthorized disclosure by government employees. There is also an exemption from the Freedom of Information Act (FOIA) protecting such information from disclosure to the public. Finally, the ethical standards that bind each government employee also prohibit unauthorized disclosure (5 CFR 2635.703). 8. All Sensitive Data to which I have access or may obtain access by signing this Agreement is now and will remain the property of, or under the control of, the United States Government. I agree that I must return all Sensitive Data which has or may come into my possession a. upon demand by an authorized representative of the United States Government; b. upon the conclusion of my employment or other relationship with the Department or Agency that last granted me access to The Global Health Program Cycle Improvement Project (GH Pro) documentation, Sensitive Data; or c. upon the conclusion of my employment or other relationship that requires access to Sensitive Data. 9. Notwithstanding the foregoing, I shall not be restricted from disclosing or using Sensitive Data that: a. is or becomes generally available to the public other than as a result of an unauthorized disclosure by me; b. becomes available to me in a manner that is not in contravention of applicable law; or c. is required to be disclosed by law, court order, or other legal process. ACCEPTANCE The undersigned accepts the terms and conditions of this Agreement Name A l f a n , | 2 D R Date 1 Title 73 X j Signature University Research Co., LLC 5404 Wisconsin Avenue, Suite 800 Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-8427 www.urc-chs.com NON DISCLOSURE AND CONFLICTS AGREEMENT Project: Health Evaluation Applied Research Development (HEARD) Project Assignment Title: Performance Evaluation of the USAID’s Global Health Program Cycle Improvement Project (GH Pro) 1. Intending to be legally bound, I hereby accept the obligations contained in this agreement in consideration of my being granted access to sensitive data. As used in this Agreement, sensitive data is marked or unmarked “sensitive but unclassified information” (SBU), including oral communications, that meets the standards set by Office of Management and Budget (OMB) Circular A-130 Appendix 3 and the U.S. Agency for International Development (USAID) Automated Directives System (ADS.) I understand that any data or systems of records protected from unauthorized disclosure by the provisions of Title 5, United States Code Sections 552 (often referred to as “The Freedom of Information Act”) and 552a (“The Privacy Act”) is/are sensitive data. In addition, other categories of information, including but not limited to medical, personnel, financial, investigatory, visa, law enforcement or other information which, if released, could result in harm or unfair treatment to any individual or group, or could have a negative impact upon foreign policy or relations, or USAID’s mission. 2. I understand and accept that by being granted access to sensitive data, special confidence and trust has been placed in me by the United States Government. 3. I acknowledge I have been given access to USAID sensitive data to facilitate the performance of duties assigned to me for compensation. I understand it is my responsibility to safeguard sensitive data disclosed to me, and to refrain from disclosing sensitive data to persons not requiring access for performance of official duties. Before disclosing sensitive data, I must determine the recipient’s “need to know” or “need to access” sensitive data for USAID purposes. 4. Any breach of this Agreement may result in the termination of my access to Sensitive Data, which, if such termination effectively negates my ability to perform my assigned duties, may lead to the termination of my employment or other relationships with the Departments or Agencies that granted my access. 5. I have reviewed my employment (past, present and under consideration) and financial interests, as well as those of my household family members, and certify that, to the best of my knowledge and belief, I have no actual or potential conflict of interest that could diminish my capacity to perform my assigned duties in an impartial and objective manner. 6. I will not use Sensitive Data, while working on this assignment or thereafter, for personal gain or detrimentally to USAID, or disclose or make available all or any part of the Sensitive Data to any person, firm, corporation, association, or any other entity for any reason or purpose whatsoever, directly or indirectly, except as may be required for the benefit USAID. 7. Misuse of government Sensitive Data could constitute a violation, or violations, of United States criminal law, and Federally-affiliated workers (including some contract employees) who violate 74 5404 Wisconsin Avenue, Suite 800 Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-8427 www.urc-chs.com privacy safeguards may be subject to disciplinary actions, a fine of up to $5,000, or both. In particular, U.S. criminal law (18 USC § 1905) protects confidential information from unauthorized disclosure by government employees. There is also an exemption from the Freedom of Information Act (FOIA) protecting such information from disclosure to the public. Finally, the ethical standards that bind each government employee also prohibit unauthorized disclosure (5 CFR 2635.703). 8. All Sensitive Data to which I have access or may obtain access by signing this Agreement is now and will remain the property of, or under the control of, the United States Government. I agree that I must return all Sensitive Data which has or may come into my possession a. upon demand by an authorized representative of the United States Government; b. upon the conclusion of my employment or other relationship with the Department or Agency that last granted me access to The Global Health Program Cycle Improvement Project (GH Pro) documentation, Sensitive Data; or c. upon the conclusion of my employment or other relationship that requires access to Sensitive Data. Name Title 75 U n iv e r s it y R e s e a r c h Co., LLC 9. Notwithstanding the foregoing, I shall not be restricted from disclosing or using Sensitive Data that: a. is or becomes generally available to the public other than as a result of an unauthorized disclosure by me; b. becomes available to me in a manner that is not in contravention of applicable law; or c. is required to be disclosed by law, court order, or other legal process. ACCEPTANCE / j The undersigned afccep/s the terms and conditions of this Agreement. _________A j X ^ ^ _______________ 0 3 ( 0 ^ | ip. ^'''Signature* " Date * MU mo m» . ■ University Research Co., LLC 5404 Wisconsin Avenue, Suite 800 Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-8427 www.urc-chs.com NON-DISCLOSURE AND CONFLICTS AGREEMENT Project: Health Evaluation Applied Research Development (HEARD) Project Assignment Title: Performance Evaluation of the USAID’s Global Health Program Cycle Improvement Project (GH Pro) 1. Intending to be legally bound, I hereby accept the obligations contained in this agreement in consideration of my being granted access to sensitive data. As used in this Agreement, sensitive data is marked or unmarked “sensitive but unclassified information” (SBU), including oral communications, that meets the standards set by Office of Management and Budget (OMB) Circular A-130 Appendix 3 and the U.S. Agency for International Development (USAID) Automated Directives System (ADS.) I understand that any data or systems of records protected from unauthorized disclosure by the provisions of Title 5, United States Code Sections 552 (often referred to as “The Freedom of Information Act”) and 552a (“The Privacy Act”) is/are sensitive data. In addition, other categories of information, including but not limited to medical, personnel, financial, investigatory, visa, law enforcement or other information which, if released, could result in harm or unfair treatment to any individual or group, or could have a negative impact upon foreign policy or relations, or USAID’s mission. 2. I understand and accept that by being granted access to sensitive data, special confidence and trust has been placed in me by the United States Government. 3. I acknowledge I have been given access to USAID sensitive data to facilitate the performance of duties assigned to me for compensation. I understand it is my responsibility to safeguard sensitive data disclosed to me, and to refrain from disclosing sensitive data to persons not requiring access for performance of official duties. Before disclosing sensitive data, I must determine the recipient’s “need to know” or “need to access” sensitive data for USAID purposes. 4. Any breach of this Agreement may result in the termination of my access to Sensitive Data, which, if such termination effectively negates my ability to perform my assigned duties, may lead to the termination of my employment or other relationships with the Departments or Agencies that granted my access. 5. I have reviewed my employment (past, present and under consideration) and financial interests, as well as those of my household family members, and certify that, to the best of my knowledge and belief, I have no actual or potential conflict of interest that could diminish my capacity to perform my assigned duties in an impartial and objective manner. 6. I will not use Sensitive Data, while working on this assignment or thereafter, for personal gain or detrimentally to USAID, or disclose or make available all or any part of the Sensitive Data to any person, firm, corporation, association, or any other entity for any reason or purpose whatsoever, directly or indirectly, except as may be required for the benefit USAID. 7. Misuse of government Sensitive Data could constitute a violation, or violations, of United States criminal law, and Federally-affiliated workers (including some contract employees) who violate 76 U n iv e r s it y R e s e a r c h Co., LLC 5404 Wisconsin Avenue, Suite 800 Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-8427 vnvw. urc-chs.com 8. privacy safeguards may be subject to disciplinary actions, a fine of up to $5,000, or both. In particular, U.S. criminal law (18 USC § 1905) protects confidential information from unauthorized disclosure by government employees. There is also an exemption from the Freedom of Information Act (FOIA) protecting such information from disclosure to the public. Finally, the ethical standards that bind each government employee also prohibit unauthorized disclosure (5 CFR 2635.703). All Sensitive Data to which I have access or may obtain access by signing this Agreement is now and will remain the property of, or under the control of, the United States Government. I agree that I must return all Sensitive Data which has or may come into my possession a. upon demand by an authorized representative of the United States Government; b. upon the conclusion of my employment or other relationship with the Department or Agency that last granted me access to The Global Health Program Cycle Improvement Project (GH Pro) documentation, Sensitive Data; or upon the conclusion of my employment or other relationship that requires access to Sensitive Data. c. 9. Notwithstanding the foregoing, I shall not be restricted from disclosing or using Sensitive Data that: a. is or becomes generally available to the public other than as a result of an unauthorized disclosure by me; b. becomes available to me in a manner that is not in contravention of applicable law; or c. is required to be disclosed by law, court order, or other legal process. ACCEPTANCE The undersigned accents the terms arid conditions of this Agreement. 77 ----------- JA JO________ ____ i 7- Signature Date --------------------- \ w c , ____________R d i e c i Q A W v L A Name Title U n iv e r sit y Re s e a r c h Co., LLC 5404 Wisconsin Avenue, Suite 800 Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-8427 www.urc-chs.com NON-DISCLOSURE AND CONFLICTS AGREEMENT Project: Health Evaluation Applied Research Development (HEARD) Project Assignment Title: Performance Evaluation of the USAID’s Global Health Program Cycle Improvement Project (GH Pro) 1. Intending to be legally bound, I hereby accept the obligations contained in this agreement in consideration of my being granted access to sensitive data. As used in this Agreement, sensitive data is marked or unmarked “sensitive but unclassified information” (SBU), including oral communications, that meets the standards set by Office of Management and Budget (OMB) Circular A-130 Appendix 3 and the U.S. Agency for International Development (USAID) Automated Directives System (ADS.) I understand that any data or systems of records protected from unauthorized disclosure by the provisions of Title 5, United States Code Sections 552 (often referred to as “The Freedom of Information Act”) and 552a (“The Privacy Act”) is/are sensitive data. In addition, other categories of information, including but not limited to medical, personnel, financial, investigatory, visa, law enforcement or other information which, if released, could result in harm or unfair treatment to any individual or group, or could have a negative impact upon foreign policy or relations, or USAID’s mission. 2. I understand and accept that by being granted access to sensitive data, special confidence and trust has been placed in me by the United States Government. 3. I acknowledge I have been given access to USAID sensitive data to facilitate the performance of duties assigned to me for compensation. I understand it is my responsibility to safeguard sensitive data disclosed to me, and to refrain from disclosing sensitive data to persons not requiring access for performance of official duties. Before disclosing sensitive data, I must determine the recipient’s “need to know” or “need to access” sensitive data for USAID purposes. 4. Any breach of this Agreement may result in the termination of my access to Sensitive Data, which, if such termination effectively negates my ability to perform my assigned duties, may lead to the termination of my employment or other relationships with the Departments or Agencies that granted my access. 5. I have reviewed my employment (past, present and under consideration) and financial interests, as well as those of my household family members, and certify that, to the best of my knowledge and belief, I have no actual or potential conflict of interest that could diminish my capacity to perform my assigned duties in an impartial and objective manner. 6. I will not use Sensitive Data, while working on this assignment or thereafter, for personal gain or detrimentally to USAID, or disclose or make available all or any part of the Sensitive Data to any person, firm, corporation, association, or any other entity for any reason or purpose whatsoever, directly or indirectly, except as may be required for the benefit USAID. 7. Misuse of government Sensitive Data could constitute a violation, or violations, of United States criminal law, and Federally-affiliated workers (including some contract employees) who violate 78 u n iv e r s it y Re s e a r c h Co., LLC 5404 Wisconsin Avenue, Suite 800 Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-84 www.urc-chs.com privacy safeguards may be subject to disciplinary actions, a fine of up to $5,000, or both. In particular, U.S. criminal law (18 USC § 1905) protects confidential information from unauthorized disclosure by government employees. There is also an exemption from the Freedom of Information Act (FOIA) protecting such information from disclosure to the public. Finally, the ethical standards f not Ki nrl •—v ■. - 1 1 « . ■ . _ . that bind each government employee also prohibit unauthorized disclosure (5 CFR 2635.703). All Sensitive Data to which I have access or may obtain access by signing this Agreement is now and \A/1 I I rpmain fno /x- ______I___it_______ , 1 r .1 VT . . _ _ ^ will remain the property of, or under the control of, the United States Government. I agree that I must rpfn rn oil v < i < I N i.— 1 _ 1 I * ^ return all Sensitive Data which has or may come into my possession upon demand by an authorized representative of the United States Government; upon the conclusion of my employment or other relationship with the Department or Agency that j w it.ii uit, u r a g e n c y int last granted me access to The Global Health Program Cycle Improvement Project (GH Pro) documentation, Sensitive Data; or ' upon the conclusion of my employment or other relationship that requires access to Sensitive Data. 9. Notwithstanding the foregoing, I shall not be restricted from disclosing or using Sensitive Data that: is or becomes generally available to the public other than as a result of an unauthorized disclosure by me; b. becomes available to me in a manner that is not in contravention of applicable law or c. is required to be disclosed by law, court order, or other legal process. ACCEPTANCE The undersigned accepts the terpis and conditions of this Agreement. mokwirinrim r<- h Title 5 ^buctjt 79 —T ^ t -----m— ---------- 1 r/'q \ \ ' u ; ^ Name X U n iv e r s it y Re s e a r c h Co., LLC 5404 Wisconsin Avenue, Suite 800 Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-8427 www.urc-chs.com NON-DISCLOSURE AND CONFLICTS AGREEMENT Project: Health Evaluation Applied Research Development (HEARD) Project Assignment Title: Performance Evaluation of the USAID’s Global Health Program Cycle Improvement Project (GH Pro) 1. Intending to be legally bound, I hereby accept the obligations contained in this agreement in consideration of my being granted access to sensitive data. As used in this Agreement, sensitive data is marked or unmarked “sensitive but unclassified information” (SBU), including oral communications, that meets the standards set by Office of Management and Budget (OMB) Circular A-130 Appendix 3 and the U.S. Agency for International Development (USAID) Automated Directives System (ADS.) I understand that any data or systems of records protected from unauthorized disclosure by the provisions of Title 5, United States Code Sections 552 (often referred to as “The Freedom of Information Act”) and 552a (“The Privacy Act”) is/are sensitive data. In addition, other categories of information, including but not limited to medical, personnel, financial, investigatory, visa, law enforcement or other information which, if released, could result in harm or unfair treatment to any individual or group, or could have a negative impact upon foreign policy or relations, or USAID’s mission. 2. I understand and accept that by being granted access to sensitive data, special confidence and trust has been placed in me by the United States Government. 3. I acknowledge I have been given access to USAID sensitive data to facilitate the performance of duties assigned to me for compensation. I understand it is my responsibility to safeguard sensitive data disclosed to me, and to refrain from disclosing sensitive data to persons not requiring access for performance of official duties. Before disclosing sensitive data, I must determine the recipient’s “need to know” or “need to access” sensitive data for USAID purposes. 4. Any breach of this Agreement may result in the termination of my access to Sensitive Data, which, if such termination effectively negates my ability to perform my assigned duties, may lead to the termination of my employment or other relationships with the Departments or Agencies that granted my access. 5. I have reviewed my employment (past, present and under consideration) and financial interests, as well as those of my household family members, and certify that, to the best of my knowledge and belief, I have no actual or potential conflict of interest that could diminish my capacity to perform my assigned duties in an impartial and objective manner. 6. I will not use Sensitive Data, while working on this assignment or thereafter, for personal gain or detrimentally to USAID, or disclose or make available all or any part of the Sensitive Data to any person, firm, corporation, association, or any other entity for any reason or purpose whatsoever, directly or indirectly, except as may be required for the benefit USAID. 7. Misuse of government Sensitive Data could constitute a violation, or violations, of United States criminal law, and Federally-affiliated workers (including some contract employees) who violate 80 U n iv e r s it y R e s e a r c h Co., LLC 5404 Wisconsin Avenue, Suite 800 Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-8427 www.urc-chs.com privacy safeguards may be subject to disciplinary actions, a fine of up to $5,000, or both. In particular, U.S. criminal law (18 USC § 1905) protects confidential information from unauthorized disclosure by government employees. There is also an exemption from the Freedom of Information Act (FOIA) protecting such information from disclosure to the public. Finally, the ethical standards that bind each government employee also prohibit unauthorized disclosure (5 CFR 2635.703). 8. All Sensitive Data to which I have access or may obtain access by signing this Agreement is now and will remain the property of, or under the control of, the United States Government. I agree that I must return all Sensitive Data which has or may come into my possession a. upon demand by an authorized representative of the United States Government; b. upon the conclusion of my employment or other relationship with the Department or Agency that last granted me access to The Global Health Program Cycle Improvement Project (GH Pro) documentation, Sensitive Data; or c. upon the conclusion of my employment or other relationship that requires access to Sensitive Data. 9. Notwithstanding the foregoing, I shall not be restricted from disclosing or using Sensitive Data that: a. is or becomes generally available to the public other than as a result of an unauthorized disclosure by me; b. becomes available to me in a manner that is not in contravention of applicable law; or c. is required to be disclosed by law, court order, or other legal process. ACCEPTANCE The undersigned accepts the terms and conditions of this Agreement. _________ l^M QL Jjl________ p^r Q,9oi7 ___ Signature Date _____ j^iAyoilp. Ckariej-_____ tiEAf 0 AsocA~t&. Director & lechm<-‘a£. Name Title S C^>p’oTf~' University Research Co., LLC 5404 Wisconsin Avenue, Suite 800 Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-8427 wu-w.urc-chs.com NON-DISCLOSURE AND CONFLICTS AGREEMENT Project: Health Evaluation Applied Research Development (HEARD) Project Assignment Title: Performance Evaluation of the USAID’s Global Health Program Cycle Improvement Project (GH Pro) 1. Intending to be legally bound, I hereby accept the obligations contained in this agreement in consideration of my being granted access to sensitive data. As used in this Agreement, sensitive data is marked or unmarked “sensitive but unclassified information” (SBU), including oral communications, that meets the standards set by Office of Management and Budget (OMB) Circular A-130 Appendix 3 and the U.S. Agency for International Development (USAID) Automated Directives System (ADS.) I understand that any data or systems of records protected from unauthorized disclosure by the provisions of Title 5, United States Code Sections 552 (often referred to as “The Freedom of Information Act”) and 552a (“The Privacy Act”) is/are sensitive data. In addition, other categories of information, including but not limited to medical, personnel, financial, investigatory, visa, law enforcement or other information which, if released, could result in harm or unfair treatment to any individual or group, or could have a negative impact upon foreign policy or relations, or USAID’s mission. 2. I understand and accept that by being granted access to sensitive data, special confidence and trust has been placed in me by the United States Government. 3. I acknowledge I have been given access to USAID sensitive data to facilitate the performance of duties assigned to me for compensation. I understand it is my responsibility to safeguard sensitive data disclosed to me, and to refrain from disclosing sensitive data to persons not requiring access for performance of official duties. Before disclosing sensitive data, I must determine the recipient’s “need to know” or “need to access” sensitive data for USAID purposes. 4. Any breach of this Agreement may result in the termination of my access to Sensitive Data, which, if such termination effectively negates my ability to perform my assigned duties, may lead to the termination of my employment or other relationships with the Departments or Agencies that granted my access. 5. I have reviewed my employment (past, present and under consideration) and financial interests, as well as those of my household family members, and certify that, to the best of my knowledge and belief, I have no actual or potential conflict of interest that could diminish my capacity to perform my assigned duties in an impartial and objective manner. 6. I will not use Sensitive Data, while working on this assignment or thereafter, for personal gain or detrimentally to USAID, or disclose or make available all or any part of the Sensitive Data to any person, firm, corporation, association, or any other entity for any reason or purpose whatsoever, directly or indirectly, except as may be required for the benefit USAID. 7. Misuse of government Sensitive Data could constitute a violation, or violations, of United States criminal law, and Federally-affiliated workers (including some contract employees) who violate 82 University Research Co., LLC 5404 Wisconsin Avenue, Suite 800 Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-8427 www.urc-chs.com privacy safeguards may be subject to disciplinary actions, a fine of up to $5,000, or both. In particular, U.S. criminal law (18 USC § 1905) protects confidential information from unauthorized disclosure by government employees. There is also an exemption from the Freedom of Information Act (FOLA) protecting such information from disclosure to the public. Finally, the ethical standards that bind each government employee also prohibit unauthorized disclosure (5 CFR 2635.703). 8. All Sensitive Data to which I have access or may obtain access by signing this Agreement is now and will remain the property of, or under the control of, the United States Government. I agree that I must return all Sensitive Data which has or may come into my possession a. upon demand by an authorized representative of the United States Government; b. upon the conclusion of my employment or other relationship with the Department or Agency that last granted me access to The Global Health Program Cycle Improvement Project (GH Pro) documentation, Sensitive Data; or c. upon the conclusion of my employment or other relationship that requires access to Sensitive Data. 9. Notwithstanding the foregoing, I shall not be restricted from disclosing or using Sensitive Data that: a. is or becomes generally available to the public other than as a result of an unauthorized disclosure by me; b. becomes available to me in a manner that is not in contravention of applicable law; or c. is required to be disclosed by law, court order, or other legal process. ACCEPTANCE The undersigned accepts the terms and conditions of this Agreement. Signature Date AgBAS .AUUiQtglU Name Z _________ A S ^ C i a -tU Title t 83 UNIVERSITY RESEARCH CO., LLC 5404 Wisconsin Avenue, Suite 800 Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-8427 www.urc-chs.com NON-DISCLOSURE AND CONFLICTS AGREEMENT Project: Health Evaluation Applied Research Development (HEARD) Project Assignment Title: Performance Evaluation of the USAID’s Global Health Program Cycle Improvement Project (GH Pro) 1. Intending to be legally bound, I hereby accept the obligations contained in this agreement in consideration of my being granted access to sensitive data. As used in this Agreement, sensitive data is marked or unmarked “sensitive but unclassified information” (SBU), including oral communications, that meets the standards set by Office of Management and Budget (OMB) Circular A-130 Appendix 3 and the U.S. Agency for International Development (USAID) Automated Directives System (ADS.) I understand that any data or systems of records protected from unauthorized disclosure by the provisions of Title 5, United States Code Sections 552 (often referred to as “The Freedom of Information Act”) and 552a (“The Privacy Act”) is/are sensitive data. In addition, other categories of information, including but not limited to medical, personnel, financial, investigatory, visa, law enforcement or other information which, if released, could result in harm or unfair treatment to any individual or group, or could have a negative impact upon foreign policy or relations, or USAID’s mission. 2. I understand and accept that by being granted access to sensitive data, special confidence and trust has been placed in me by the United States Government. 3. I acknowledge I have been given access to USAID sensitive data to facilitate the performance of duties assigned to me for compensation. I understand it is my responsibility to safeguard sensitive data disclosed to me, and to refrain from disclosing sensitive data to persons not requiring access for performance of official duties. Before disclosing sensitive data, I must determine the recipient’s “need to know” or “need to access” sensitive data for USAID purposes. 4. Any breach of this Agreement may result in the termination of my access to Sensitive Data, which, if such termination effectively negates my ability to perform my assigned duties, may lead to the termination of my employment or other relationships with the Departments or Agencies that granted my access. 5. I have reviewed my employment (past, present and under consideration) and financial interests, as well as those of my household family members, and certify that, to the best of my knowledge and belief, I have no actual or potential conflict of interest that could diminish my capacity to perform my assigned duties in an impartial and objective manner. 6. I will not use Sensitive Data, while working on this assignment or thereafter, for personal gain or detrimentally to USAID, or disclose or make available all or any part of the Sensitive Data to any person, firm, corporation, association, or any other entity for any reason or purpose whatsoever, directly or indirectly, except as may be required for the benefit USAID. 7. Misuse of government Sensitive Data could constitute a violation, or violations, of United States criminal law, and Federally-affiliated workers (including some contract employees) who violate privacy safeguards may be subject to disciplinary actions, a fine of up to $5,000, or both. In 84 5404 Wisconsin Avenue, Suite 800 Chevy Chase, MD 20815-3594 TEL 301-654-8338 FAX 301-941-8427 www.urc-chs.com particular, U.S. criminal law (18 USC § 1905) protects confidential information from unauthorized disclosure by government employees. There is also an exemption from the Freedom of Information Act (FOIA) protecting such information from disclosure to the public. Finally, the ethical standards that bind each government employee also prohibit unauthorized disclosure (5 CFR 2635.703). 8. All Sensitive Data to which I have access or may obtain access by signing this Agreement is now and will remain the property of, or under the control of, the United States Government. I agree that I must return all Sensitive Data which has or may come into my possession a. upon demand by an authorized representative of the United States Government; b. upon the conclusion of my employment or other relationship with the Department or Agency that last granted me access to The Global Health Program Cycle Improvement Project (GH Pro) documentation, Sensitive Data; or " ‘ c. upon the conclusion of my employment or other relationship that requires access to Sensitive Data. 9. Notwithstanding the foregoing, I shall not be restricted from disclosing or using Sensitive Data that: a. is or becomes generally available to the public other than as a result of an unauthorized disclosure by me; b. becomes available to me in a manner that is not in contravention of applicable law; or c. is required to be disclosed by law, court order, or other legal process. UNIVERSITY RESEARCH CO., LLC 85 ACCEPTANCE J The undersigned accepts the terms anchconditions of this Agreement. __MJlL JlL____ Chi z*u___ • Signature ¥ ^ Date — --------------- / W c L -------_________________Q h J u l 4 n * i - ^ N i Title --------------------- NON-DISCLOSURE AND CONFLICTS AGREEMENT Project: Health Evaluation Applied Research Development (HEARD) Project Assignment Title: Performance Evaluation of the USAID’s Global Health Program Cycle Improvement Project (GH Pro) 1. Intending to be legally bound, UCSF hereby accepts the obligations contained in this agreement in consideration of its being granted access to sensitive data. As used in this Agreement, sensitive data is marked “sensitive but unclassified information” (SBU), including oral communications, that meets the standards set by Office of Management and Budget (OMB) Circular A-130 Appendix 3 and the U.S. Agency for International Development (USAID) Automated Directives System (ADS.) UCSF understands that any data or systems of records protected from unauthorized disclosure by the provisions of Title 5, United States Code Sections 552 (often referred to as “The Freedom of Information Act”) and 552a (“The Privacy Act”) is/are sensitive data. In addition, other categories of information, including but not limited to medical, personnel, financial, investigatory, visa, law enforcement or other information which, if released, could result in harm or unfair treatment to any individual or group, or could have a negative impact upon foreign policy or relations, or USAID’s mission. 2. UCSF understands and accepts that by being granted access to sensitive data, special confidence and trust has been placed in UCSF by the United States Government. 3. UCSF acknowledges UCSF has been given access to USAID sensitive data to facilitate the performance of duties assigned to UCSF for compensation. UCSF understands it is UCSF’s responsibility to safeguard sensitive data disclosed to UCSF, and to refrain from disclosing sensitive data to persons not requiring access for performance of official duties. Before disclosing sensitive data, UCSF must determine the recipient’s “need to know” or “need to access” sensitive data for USAID purposes. 4. Any breach of this Agreement may result in the termination of UCSF’s access to Sensitive Data, which, if such termination effectively negates its ability to perform its assigned duties, may lead to the termination of my employment or other relationships with the Departments or Agencies that granted UCSF’s access. 5. UCSF has reviewed its projects and partnerships (past, present and under consideration) and financial interests, and certify that, to the best of its knowledge and belief, UCSF has no actual or potential conflict of interest that could diminish its capacity to perform its assigned duties in an impartial and objective manner. 6. UCSF will not use Sensitive Data, while working on this assignment or thereafter, for personal gain or detrimentally to USAID, or disclose or make available all or any part of the Sensitive Data to any person, firm, corporation, association, or any other entity for any reason or purpose whatsoever, directly or indirectly, except as may be required for the benefit USAID. DocuSign Envelope ID: 3D4154F8-0354-40E5-8A24-E6D5901F3403 86 7. Misuse of government Sensitive Data could constitute a violation, or violations, of United States criminal law, and Federally-affiliated workers (including some contract employees) who violate privacy safeguards may be subject to disciplinary actions, a fine of up to $5,000, or both. In particular, U.S. criminal law (18 USC § 1905) protects confidential information from unauthorized disclosure by government employees. There is also an exemption from the Freedom of Information Act (FOIA) protecting such information from disclosure to the public. Finally, the ethical standards that bind each government employee also prohibit unauthorized disclosure (5 CFR 2635.703). 8. All Sensitive Data to which UCSF has access or may obtain access by signing this Agreement is now and will remain the property of, or under the control of, the United States Government. UCSF agrees that it must return all Sensitive Data which has or may come into its possession a. upon demand by an authorized representative of the United States Government; b. upon the conclusion of my employment or other relationship with the Department or Agency that last granted me access to The Global Health Program Cycle Improvement Project (GH Pro) documentation, Sensitive Data; or c. upon the conclusion of my employment or other relationship that requires access to Sensitive Data. Notwithstanding the foregoing, UCSF may retain for archival purposes only one copy of the Sensitive Data. 9. Notwithstanding the foregoing, UCSF shall not be restricted from disclosing or using Sensitive Data that: a. is or becomes generally available to the public other than as a result of an unauthorized disclosure by me; b. becomes available to me in a manner that is not in contravention of applicable law; c. developed by UCSF independently of knowledge or information obtained by its access to the Sensitive Data; d. already known to me before receipt of the Sensitive Data, as shown by the Institution’s prior written records; or e. is required to be disclosed by law, court order, or other legal process. ACCEPTANCE The undersigned accepts the terms and conditions of this Agreement. ______________________________________________________1/11/2018 ________________________ Signature Date Rachel Sievert ___________________________________________________________ Award Team Manager ___________________ Name Title DocuSign Envelope ID: 3D4154F8-0354-40E5-8A24-E6D5901F3403 87 1.0. EVALUATION QUESTION 1 ANNEXES 1.1. ANNEX: GH PRO STAFF ROLES USAID clients wishing to engage GH Pro services submit a scope of work (SOW) to GH Pro’s COR. Once approved by the COR, the SOW is submitted to GH Pro. The SOW includes the parameters of the work needed, the expertise needed, including number, qualifications, and level of effort of desired team members, timeline, and deliverables. GH Pro reviews each SOW and provides feedback, develops a cost estimate, and recruits appropriate consultant(s) to carry out the work. Once the SOW, cost estimate, and key consultant(s) are agreed to by the client, the COR approves the Technical Directive Memo (TDM), which is the administrative action that allows the money for the assignment to be obligated and transferred to GH Pro, and authorizes GH Pro to begin carrying out the assignment. Once the TDM is approved, GH Pro can enter into contract with consultants and begin logistical arrangements related to assignment execution. GH Pro was operating with 24-26 staff as the evaluation was taking place. As of December 31, 2017, this team was managing 113 assignments in the active or planning phases. While staffing levels within GH Pro can fluctuate based on assignment volume and funding, the Project Director leads a team that includes the following roles:2 • Deputy Director/Technical Advisor for Evaluations: serves as the sole technical advisor, as such provides technical review and input for each evaluation SOW and report; also supervises recruitment. • Deputy Director/Operations Manager: oversees administrative and logistics aspects of operations including supervision of Project Manager work on initial cost estimates and other mobilization activities as needed. • Senior Finance and Administration (F&A) Manager: leads financial tracking and management including contract administration; supervises F&A officers at various levels, who are responsible for a range of activities including processing payments (e.g. consultant payments, travel advances, travel reimbursements, funds for field-based logistics), financial reporting, and creating monitoring budgets, among others. • Contracting Officer (CO): supports contracting and procurement activities. • Program Managers (PMs) at various levels: Senior Program Managers, Program Managers, and Associate Program Managers, or PMs3 , manage assignments from initiation until closeout, a process which includes client communication, recruitment, logistics planning and 2 Role descriptions are not comprehensive, but indicate the key functions of each role in relation to assignment implementation 3 PMs at various levels are grouped when referenced in this report, except when specific insights pertain to one of the three sub-groups. 88 implementation, and closeout activities. Evaluations are nearly all managed by Senior PMs; most other PMs only work on MS and TA. • Senior International Recruiter: grows and maintains consultant database and assists PMs with recruitment for all three assignment types, including identification of potential local consultants. • Communications Manager: brought on to improve evaluation report quality in particular, provides editing for evaluation reports as well as GH Pro’s Quarterly and Annual reports. • Program Assistants: assist PMs with administrative and logistical background support on specific projects (primarily evaluations and occasionally mission support or technical assistance); one PA serves as the point of contact for the Facility Access (FA) request process. 89 1.2. ANNEX: ASSIGNMENTS SAMPLED FOR DOCUMENT WORK STREAM REVIEW TD # Requestor Country/ Office TD Title Requestor Region Type PEPFAR TDM Start Date TDM End Date o/a Approved TDM Cost Estimate 007/008 Tanzania Social Marketing Program (TSMP) & Capacity and Communications Project (TCCP) Combined Evaluations AFR Ev Yes 2/9/2015 11/30/2015 $517,463.00 066 GH/PRH Health Communications Capacity Collaboration (HC3) Project Evaluation DC Ev No 5/29/2015 3/31/2016 $343,081.00 261 Rwanda Health Systems Mid￾term Evaluation AFR Ev No 9/20/2016 10/20/2017 $371,421 284 GH/ID Challenge TB Project Management Review DC Ev No 9/30/2016 4/30/2017 $202,134 343 GH/PRH Evidence Project Evaluation DC Ev 1/17/2017 8/31/2017 $152,444 049 Benin Integrated Family Health Program Evaluation AFR Ev No 2/24/2015 8/7/2015 $316,081.00 146 Philippines PNG MARPS Project Evaluation ASIA Ev Yes 2/8/2016 10/31/2016 $298,391.00 065 Ethiopia MLDM Project Mid￾term Evaluation AFR Ev No 8/12/2015 1/31/2016 $283,711.00 002 GH/OHA OGAC Multilateral Tech Advisor DC TA Yes 10/23/2014 6/30/2015 $161,503.00 282 GH/PRH Reproductive Health Technical Assistance DC TA No 8/31/2016 9/30/2017 $102,529 027 GH/OHA Pediatric HIV Consultant DC TA Yes 7/15/2015 7/31/2016 $176,788.00 015 GH/CII MNCH Country Procurement Analysis DC TA No 1/12/2015 11/15/2015 $300,810.00 414 GH/ID PMI Collaboration Advisor (w/ Gates Foundation) DC TA No 5/23/2017 8/30/2017 $52,293 120 GH/PRH Evidence Project Management Review Follow-up DC TA No 7/2/2015 8/20/2015 $15,529.00 222 AA/GH Personal Assistant Specialist DC TA Yes 4/18/2016 12/31/2016 $26,940 061 GH/HIDN Public Private Partnerships Senior Advisor DC TA No 2/6/2015 9/30/2015 $155,185.00 064 GH/HIDN Immunization Strategy Development DC TA No 2/6/2015 9/15/2015 $129,440.00 164 Uganda M&E Project Mgmt Specialist AFR MS Yes 2/17/2016 9/2/2016 $256,945.00 90 091 Ivory Coast Health Office Support AFR MS Yes 5/5/2015 1/6/2016 $143,034.00 074 Senegal G2G Technical Assistance AFR MS No 3/26/2015 11/30/2015 $180,410.00 306 Burma Program Management Support ASIA MS No 11/29/2016 8/2/2017 $164,179 158 Mozambique SI Team Lead AFR MS Yes 12/2/2015 6/30/2016 $121,848.00 016 Nepal Health Program Review Assistance ASIA MS No 8/4/2015 10/31/2016 $300,000.00 102 Nigeria Gender Analysis AFR MS Yes 9/16/2015 5/3/2016 $222,492.00 054 Cambodia Malaria Program Design Assistance ASIA MS No 2/11/2015 4/30/2015 $33,891.00 081 Ivory Coast COP Support AFR MS Yes 4/7/2015 5/20/2015 $34,024.00 442 Namibia Health Office Support AFR MS Yes 8/2/2017 8/31/2017 $52,718 252 Liberia MOH Technical Assistance AFR MS No 6/8/2016 7/29/2016 $53,481 191 DRC Sr. Health Advisor AFR MS No 3/8/2016 5/15/2016 $87,883 333 Namibia HIV Communications Advisor AFR MS Yes 2/1/2017 6/2/2017 $77,134 239 Nepal Health Systems Strengthening Activity Design ASIA MS No 6/9/2016 10/31/2016 $124,474 371 Bangladesh Health Service Delivery Advisor ASIA MS No 2/28/2017 7/14/2017 $83,707 91 1.3. ANNEX: SURVEY QUESTION TYPES Survey Question Focus Efficient Effective Ease of initiation X Ease of use X Timeliness of completion X Responsive during implementation X Bottlenecks during implementation X Quality of templates/forms X X For Mission support and technical assistance: Appropriateness of consultant X For technical assistance and evaluation: Quality of consultant X For Mission support: Was immediate supervisor satisfied with work? X For technical assistance and evaluation: Were recommendations actionable? X Overall satisfaction X X 92 1.4. ANNEX: GH PRO CLIENT SURVEY At the request of the USAID Bureau of Global Health, Office of Policy, Programs and Planning, the Health Evaluation and Applied Research Development (HEARD) Project is conducting the mid-term evaluation of the Global Health Program Improvement Cycle Project (GH Pro). As a part of this evaluation, we are requesting your input to help us assess the efficiency and effectiveness of GH Pro. As a client who has used GH Pro services, your insights into the project are a crucial part of this assessment. We are asking you to complete the following survey to assist us in our evaluation. All responses will be kept confidential. Your name is requested only to identify whether you have completed a survey. Aggregated results and non-attributed comments will be used to inform the evaluation and will be shared with USAID. The final evaluation report will be publicly available. Your participation in this survey is voluntary; however, please note that your participation is critical to assure the value and validity of this evaluation. We have made every effort to simplify and shorten the survey, so that it will not require a substantial amount of your time. This survey should take approximately 15 minutes to complete. If there are questions where you feel that you do not have adequate information to provide an informed response, you may opt-out of answering. If you have questions about the survey or suggestion as to how to improve the survey, please do not hesitate to contact us (aalawieh@urc-chs.com). 1. Name (This is intended to provide us with information for follow-up and to identify sites where we have received no response.) 2. Country/Office or HQ/Office 3. Title/Position 4. How did you first hear about GH Pro? 1. USAID Contacts 2. USAID Users' Guide 3. GH Pro website 4. From a consultant 5. Other __________ 5. How many assignments have you requested from GH Pro? 1. 1 2. 2-3 93 3. 4-7 4. 7+ 6. What led you to choose GH Pro for the assignment(s)? You may check up to 3. 1. Past experience with GH Pro 2. Referral from a colleague 3. Availability of information about GH Pro 4. Support provided during SOW development and other lead up to the assignment 5. Ease of contracting 6. Flexibility 7. Cost effectiveness 8. Quality of final product 9. Other 7. What are the advantages of using GH Pro? 8. What are the disadvantages of using GH Pro? 9. For which of the following services did you use GH Pro? Choose all that apply. 1. Evaluation 2. Mission Support 3. Technical Assistance The following questions relate to your experience using GH Pro for ${piping_text}. Please answer these questions if you have requested ${piping_text} from GH Pro. 10. Please rate how strongly you agree or disagree with the following statement: I had the information I needed to request GH Pro to start work on ${piping_text}. 1. Strongly Disagree 2. Disagree 3. Agree 4. Strongly Agree 11. Please rate how strongly you agree or disagree with the following statement: The Scope of Work (SOW) form was easy to use. 1. Strongly Disagree 2. Disagree 3. Agree 94 4. Strongly Agree 12. What, if any, kinds of bottlenecks or problems did you experience with ${piping_text}? Please check all that apply OR select "No bottlenecks or problems." 1. No bottlenecks or problems 2. Securing pre-travel documents, such as visas 3. Travel 4. Lodging 5. Transportation 6. Payment or reimbursement 7. Scheduling of appointments 8. Team size 9. Team member qualifications and/or fit 10. Communication with GH Pro 11. Communication with consultant/team 12. IRB approval 13. Report preparation 14. Report finalization 15. Security during assignment period 16. Unexpected events (e.g. natural disasters, medical issues, political unrest) 17. Other (Please specify) __________ 13. Please use this space to add any detail you would like about the bottlenecks or problems you experienced listed above, if applicable. 14. Did GH Pro help with the resolution of any bottlenecks or problems? 1. Yes 2. No 3. Not applicable 15. How appropriate was the staffing that GH Pro provided for your ${piping_text}? For example, did the consultant(s) hired have the necessary qualifications, without being over qualified? 1. Very appropriate 2. Somewhat appropriate 3. Somewhat inappropriate 4. Very inappropriate 16. Was/were the consultant(s) suitably knowledgeable and experienced in the relevant area of ${piping_text}? 1. Yes 2. No 95 17. How did you choose the ${piping_text} consultant(s)? 1. GH Pro selected the consultant(s) 2. Prior experience with the consultant(s) 3. The consultant(s) was/were recommended to you by someone other than GH Pro staff 4. Other __________ 18. From your perspective, were there significant changes to the initial ${piping_text} SOW and the final ${piping_text} SOW? 1. Yes 2. No 19. Was the ${piping_text} completed per specifications laid out in the final SOW? 1. Yes 2. No 20. Did the final ${piping_text} report include informative findings, conclusions, and recommendations that were specific and actionable? 1. Yes 2. No 3. Not applicable 21. Was the immediate supervisor for the ${piping_text} consultant/team satisfied with the consultant’s/team’s performance? 1. Yes 2. No Please rate your experience with the following aspects of the GH Pro ${piping_text} on a scale of 1-5, with 1 the lowest possible rating and 5 the highest possible rating. 1 2 3 4 5 22. How do you rate the ease of use of GH Pro for ${piping_text}? ❏ ❏ ❏ ❏ ❏ 23. How do you rate the timeliness of the response? ❏ ❏ ❏ ❏ ❏ 24. How do you rate the final SOW in terms of addressing your needs for this evaluation? ❏ ❏ ❏ ❏ ❏ 25. How do you rate the responsiveness and flexibility of GH Pro during the assignment? ❏ ❏ ❏ ❏ ❏ 26. How do you rate the quality of the final deliverable? ❏ ❏ ❏ ❏ ❏ 27. How do you rate the final cost of the ${piping_text}? (1-inexpensive, 5-very costly) ❏ ❏ ❏ ❏ ❏ 96 28. How do you rate the value for money you received from GH Pro? (1-low value, 5 high value) ❏ ❏ ❏ ❏ ❏ 29. How do you rate your overall satisfaction with GH Pro for ${piping_text}? ❏ ❏ ❏ ❏ ❏ These final questions relate to your overall experience with GH Pro with any type of assignment. Would you use GH Pro again? 1. Yes 2. No Would you allow us to contact you if we need further information, such as to ask you to participate in a follow up interview? If yes, please provide your email address: Thank you for your time and valuable insights! We will ensure that the final evaluation is made available to all survey respondents. 97 1.5. ANNEX: GH PRO CONSULTANT SURVEY The Health Evaluation and Applied Research Development (HEARD) Project is conducting the mid-term evaluation of the Global Health Program Improvement Cycle (GH Pro) Project. As a part of this evaluation, we are requesting your input to help us assess the efficiency and effectiveness of GH Pro. As a consultant who has worked with USAID clients through GH Pro, we would benefit from your insights into the project. We are asking you to complete the following survey to assist us in our evaluation. Your name is requested only to identify whether you have completed a survey. All responses will be kept confidential, and any comments will not be attributed to you. Aggregated results and non-attributed comments will be used to inform the evaluation and will be shared with USAID. The final evaluation report will be publicly available. Your participation in this survey is voluntary. You may choose not to answer any questions and may stop at any time. This survey should take approximately 15-20 minutes to complete. 1. Name (for tracking purposes only) 2. How did you first hear about GH Pro? 1. The GH Pro website 2. From USAID contact 3. From another GH-Pro consultant 4. Other 3. How many assignments have you worked on with GH Pro? 1. 1 2. 2-3 3. 4-7 4. 7+ 4. Which type(s) of assignments have you completed for GH-Pro? Please check all that apply 1. Mission Support 2. Technical Assistance 3. Evaluation 4. Other (e.g. meetings, research) 5. Have you had similar consultancies with other USAID service support mechanisms? 1. Yes 2. No 6. What was the nature of the work on the other USAID service support mechanisms? 1. Mission Support 98 2. Technical Assistance 3. Evaluation 4. Other (e.g. meetings, research) In the following section, please rate how strongly you agree or disagree with the following statements about working with GH Pro. Please add comments as needed. 7. The consultant agreement, which includes the SOW, answered all of my questions about the roles and responsibilities of both myself and GH Pro. 1. Strongly Agree 2. Agree 3. Neutral 4. Disagree 5. Strongly Disagree 8. There was additional information that would have been useful to know at the beginning of the assignment (please include what kinds of information and how it might have been useful in your comments). 1. Strongly Agree 2. Agree 3. Neutral 4. Strongly Disagree 5. Strongly Disagree 9. Did you face any difficulties or roadblocks at any of the following stages of the assignment? Please check all that apply. 1. Securing pre-travel documents 2. Travel 3. Lodging 4. Transportation 5. Payment or reimbursement 6. Scheduling appointments 7. Team size 8. Team member qualifications and/or fit 9. Communication with USAID client 10. Non-disclosure of identifiable information 11. USAID client attempts to influence findings 12. Report preparation 13. Human subjects 14. Photographing people or sites 15. Report review 16. Security while on assignment 17. Unexpected events (e.g. natural disasters, medical issues, political unrest) 18. I didn’t have any issues. 99 19. Other 10. For any of the difficulties identified in question 9, please provide additional information, including what the problem was and if you were able to handle it within the team or if GH Pro needed to intervene: 11. If GH Pro needed to intervene, what role did they play in solving the issue? 12. Was the client at HQ or in a Mission? 1. HQ 2. Mission In the following set of questions, please rate your experience with GH Pro on a scale of 1-5, with 1 the lowest possible rating and 5 the highest (or best) possible rating. Please add comments as needed. 1 2 3 4 5 13. How do you rate the ease of being hired as a consultant with GH Pro? ❏ ❏ ❏ ❏ ❏ 14. How do you rate the ease of working with GH Pro throughout the duration of the assignment? ❏ ❏ ❏ ❏ ❏ 15. How do you rate the responsiveness of GH Pro during the assignment? ❏ ❏ ❏ ❏ ❏ 16. How do you rate your final satisfaction with the experience of working with GH Pro on the assignment? ❏ ❏ ❏ ❏ ❏ 17. Please add any comments pertaining to questions 13-16 below: 18. Would you work with GH Pro again? 1. Yes 2. No 19. Would you recommend that a colleague work with GH Pro? 1. Yes 2. No 100 Finally, please answer the following open-ended questions. 20. What advice do you have for GH Pro to improve the efficiency and effectiveness of recruiting consultants? 21. What advice do you have for GH Pro to improve the efficiency and effectiveness of GH Pro to provide evaluation support, technical assistance, or mission support? 22. Do you have any recommendations for how GH Pro could improve the experience of working with them? 23. Would you allow us to contact you if we need further information, such as to ask you to participate in a follow up interview? If yes, please provide your email address: Thank you for your time and valuable insights! 101 1.6. ANNEX: GH PRO STAFF SURVEY The Health Evaluation and Applied Research Development (HEARD) Project is conducting the mid-term evaluation of the Global Health Program Improvement Cycle (GH Pro) Project. As a part of this evaluation, we are requesting your input to help us assess the efficiency and effectiveness of GH Pro. As a staff member who has worked with both GH Pro consultants and USAID clients, you have valuable insights. We are asking you to complete the following survey to assist us in our evaluation. Your name is requested only to identify whether you have completed a survey. All responses will be kept confidential, and any comments will not be attributed to you. Aggregated results and non-attributed comments will be used to inform the evaluation and will be shared with USAID. The final evaluation report will be publicly available. Your participation in this survey is voluntary. You may choose not to answer any questions and may stop at any time. This survey should take approximately 15-20 minutes to complete. If any questions are not applicable to your role with GH Pro, please skip and continue with the remaining questions. 1. Name (for tracking purposes only) 2. How long have you worked for GH Pro? 1. 0-6 months 2. 6 months to 1 year 3. 1-2 years 4. 2-3 years 5. 3+ years 3. What is your current position in GH Pro? 4. Have you worked for a different USAID support mechanism, similar to GH-Pro, that provided either Mission Support, Technical Assistance, or Evaluation services? 1. Yes 2. No 5. Please list the name of the other USAID support mechanism(s) for which you worked: 6. How long did you work for the other USAID support mechanism(s)? 1. 0-6 months 2. 6 months to 1 year 3. 1-2 years 102 4. 2-3 years 5. 3+ years 7. Which position(s) did you hold during that work? Considering the GH Pro reference and guidance materials, including guidance emails, please answer the following: 8. Is the guidance and support information up to date? 1. Yes 2. No 3. I don't know 9. Is this information easy for you to access? 1. Yes 2. No 3. I don't know 10. Is this information easy for you to understand? 1. Yes 2. No 3. I don't know 11. Are documents in the GH Pro system (e.g., Egnyte and Office 365) easy to share with consultants and USAID clients? 1. Yes 2. No 3. I don't know 12. Of the forms you ask clients and consultants to fill out, are there specific ones that tend to be returned with errors or questions? 1. Yes 2. No 13. What might improve the efficiency of information flow for you? Please check all that apply. 1. Easier access to online and print resources, including to reference and guidance documents 2. Fewer online sources of information to consult 3. Better connectivity with USAID systems to access USAID information for assignments 4. Better connectivity with USAID systems to share GH Pro information within USAID 103 5. Better knowledge of financial management procedures 6. Other __________ In the following section, please consider the last assignment on which you worked that was completed. 14. Were there significant changes made to the SOW from what GH Pro initially received to what was approved? 1. Yes 2. No 3. I don't know 15. What was the principal reason that significant changes were made to the SOW from what GH Pro initially received? 1. Incomplete request 2. Unclear request 3. More specificity needed 4. Unrealistic expectations (e.g., too much requested in too little time, scope too great for proposed budget or timeframe) 5. Other __________ 16. Do you think the final SOW adequately addressed the principal reason the changes were needed? 1. Yes 2. No In the following section, please consider the last assignment on which you worked that was completed. 17. Was this assignment completed in the anticipated timeframe? 1. Yes 2. No 18. At what point did delays occur? (select all that apply) 1. Finalizing the TDM 2. Identifying and recruiting the team 3. Mobilizing the team 4. Conducting the assignment 5. Finalizing deliverables 6. Receiving final approval 7. Other 19. Please describe the reasons for the delays: 104 In the following set of questions, please rate how strongly you agree or disagree with the following statements about working with GH Pro. Strongly Agree Agree Neutral Disagree Strongly Disagree 20. My work is covered by a designated staff￾member when I am on leave. ❏ ❏ ❏ ❏ ❏ 21. I have the tools I need to optimally serve USAID clients. ❏ ❏ ❏ ❏ ❏ 22. I have the support I need to optimally serve USAID clients. ❏ ❏ ❏ ❏ ❏ 23. I have the tools I need to optimally serve consultants. ❏ ❏ ❏ ❏ ❏ 24. I have the support I need to optimally serve consultants. ❏ ❏ ❏ ❏ ❏ 25. I am fully supported by senior management, when I have problems or need assistance. ❏ ❏ ❏ ❏ ❏ 26. The team is adequately staffed to respond effectively to incoming requests. ❏ ❏ ❏ ❏ ❏ 27. GH Pro effectively orients/educates clients and potential clients about the GH-Pro mechanism. ❏ ❏ ❏ ❏ ❏ 28. Please add any comments pertaining to questions 20-27 below: In the following set of questions, please rate your experience with GH Pro on a scale of 1-5, with 1 the lowest possible rating and 5 the highest (or best) possible rating. 1 2 3 4 5 29. How do you rate your satisfaction working for GH Pro? ❏ ❏ ❏ ❏ ❏ 30. How do you rate the process of working with consultants throughout their assignment? ❏ ❏ ❏ ❏ ❏ 31. How do you rate the process of working with USAID clients throughout the assignment? ❏ ❏ ❏ ❏ ❏ 32. Please add any comments pertaining to questions 29-31 below: 105 Finally, please answer the following open-ended questions. 33. Overall, what does GH Pro do well? 34. What could be improved about GH Pro, and how would one do this? 35. Would you allow us to contact you if we need further information, such as to ask you to participate in a follow up interview? If yes, please provide your email address: Thank you for your time and valuable insights! 106 1.7. ANNEX: GH PRO CLIENT INTERVIEW GUIDE • Introductions/Rationale for Mid-Term Evaluation/Explanation of Interview Format o Our names are Paul De Lay, Samantha Ski, and Abbas Alawieh representing USAID’s HEARD Project. o We are supporting USAID to perform this mid-term evaluation of GH Pro. There are three evaluation questions that are being considered: 1. To what extent was the GH Pro project effective and efficient in meeting stakeholder needs in three program areas (Ev, MS, TA)? 2. To what extent are USAID and PEPFAR evaluation quality standards employed and achieved by the GH Pro Project? 3. What is the comparative value of GH Pro to the USAID Global Health Bureau and Missions for the three project areas, considering other, existing mechanisms. o The interview will take about 30-45 minutes. We will audio record you and take notes. We will provide these to you to edit for accuracy. We will not mention your name, unless you approve. The interview is voluntary, and you can choose to not answer specific questions. If you have any questions after the interview you can contact sski@urc￾chs.com. • Describe your backround, including amount of time with USAID, projects involved in/supported. • When and for which purposes did you request GH Pro services? • What was your level on engagement with GH Pro over the course of the assignment you requested? o What did GH Pro do well? o What did the consultant(s) do well? o What was good about the final report? o Cost? • From your survey responses, I understand your experience on the negative side was [cite specific example from survey and written responses]. Can you tell me more about that? o What could/should GH Pro have done better? o What could/should the consultant(s) have done better? o What could/should have been better in the final report? o Cost? • Were you able to use the evaluation findings to inform future program decisions? • Are there other options besides GH Pro for achieving the support work you utilized GH Pro for? • Have you ever utilized other mechanisms besides GH Pro for TA, MS or evaluations? • What would you recommend to GH Pro to improve effectiveness and efficiency of their services? • What would you recommend to USAID/GH leadership to improve effectiveness and efficiency of the service support mechanisms available to HQ and Missions? 107 108 1.8. ANNEX: GH PRO STAFF/MANAGEMENT INTERVIEW GUIDE • Introductions/Rationale for Mid-Term Evaluation/Explanation of Interview Format o Our names are Paul De Lay, Samantha Ski, and Abbas Alawieh representing USAID’s HEARD Project. o We are supporting USAID to perform this mid-term evaluation of GH Pro. There are three evaluation questions that are being considered: 1. To what extent was the GH Pro project effective and efficient in meeting stakeholder needs in three program areas (Ev, MS, TA)? 2. To what extent are USAID and PEPFAR evaluation quality standards employed and achieved by the GH Pro Project? 3. What is the comparative value of GH Pro to the USAID Global Health Bureau and Missions for the three project areas, considering other, existing mechanisms. o The interview will take about 60 minutes. We will audio record you and take notes. We will provide these to you to edit for accuracy. We will not mention your name, unless you approve. The interview is voluntary, and you can choose to not answer specific questions. If you have any questions after the interview you can contact sski@urc￾chs.com. • Describe your background, including amount of time with GH-Pro, previous work with similar organizations, etc. • What are your current roles and responsibilities? • What have been the major successes of GH Pro? • What have been the challenges the organization has experienced (specific to TA, MS, Ev)? o Are the challenges different for the three work areas? o How do you deal with situations where the USAID input/assistance is lacking? • Possible probes within successes and challenges questions: o Timeliness o Recruitment o GH Pro Capacity o Staff perceptions of the quality of GH Pro’s work, including quality of evaluations and evaluation reports o Staff perceptions of client satisfaction with GH Pro’s work • Describe the quality of available guidelines and systems for performing your work. • What would you keep and what would you change in GH Pro in order to improve effectiveness, efficiency, and job satisfaction? 109 2.0 EVALUATION QUESTION 2 ANNEXES 2.1. ANNEX: GH PRO EVALUATION REPORTS REVIEWED BY QRG Requestor Country/ Office Geograph ic Focus Programmati c Focus PEPFAR /USAID Title Burundi Africa Cross￾cutting* USAID IHPB Evaluation Tanzania Africa Family Planning USAID RESPOND Project Evaluation Uganda Africa MCH PEPFAR Interagency OVC Evaluation Madagascar Africa Cross￾cutting* USAID Integrated Social Marketing (ISM) Program Final Evaluation Liberia Africa Health Systems Strengthening USAID Rebuilding Basic Health Systems (RBHS) Project Final Evaluation Malawi Africa Cross￾cutting* PEPFAR Central Medical Stores Trust Assessment - Second Eval Report Mozambique Africa Health Systems Strengthening PEPFAR FORSSAS Project Evaluation Nigeria Africa Cross￾cutting* PEPFAR ESMPIN Evaluation Rwanda Africa Health Systems Strengthening USAID Health Systems Mid-term Evaluation Nigeria Africa Cross￾cutting* USAID SHOPS Mid-Term Evaluation Uganda Africa Health Systems Strengthening PEPFAR SDS Project Evaluation Nigeria Africa Cross￾cutting* PEPFAR PEPFAR OVC Portfolio Costing Analysis 110 Requestor Country/ Office Geograph ic Focus Programmati c Focus PEPFAR /USAID Title Namibia Africa HIV/AIDS PEPFAR Strengthening HIV/AIDS Responses in Prevention and Protection (SHARPP) Project Mid-Term Evaluation Uganda Africa Cross￾cutting* PEPFAR MEEPP Evaluation Ethiopia Africa Malaria USAID MLDM Project Mid-term Evaluation Benin Africa Family Planning USAID Integrated Family Health Program Evaluation Tanzania Africa MCH PEPFAR Tunajali & LIFE Projects Evaluation Bangladesh Asia Cross￾cutting* USAID Social Marketing Innovations Program Evaluation Bangladesh Asia Cross￾cutting* USAID Smiling Sun Clinics Facility Assessment Bangladesh Asia Health Systems Strengthening USAID NHSDP Evaluation Bangladesh Asia Health Systems Strengthening USAID MaMoni Project Evaluation Cambodia Asia HIV/AIDS PEPFAR HIV Flagship Project Evaluation Kyrgyz Republic Europe and Eurasia TB PEPFAR Defeat TB Project Evaluation GH/PRH Global / HQ Health Systems Strengthening PEPFAR Leadership, Management & Governance (LMG) Project Evaluation GH/HIDN Global / HQ Malaria USAID Malaria Vaccine Development Program Evaluation GH/ID Global / HQ Malaria USAID MalariaCare Project Evaluation GH/CII Global / HQ MCH USAID Saving Lives at Birth 111 Requestor Country/ Office Geograph ic Focus Programmati c Focus PEPFAR /USAID Title GH/MCHN Global / HQ MCH USAID Fistula CarePlus Mid-term Evaluation GH/ID Global / HQ Neglected Tropical Diseases USAID Neglected Tropical Diseases Program Evaluation GH/ID Global / HQ TB USAID Challenge TB Project Management Review GH/PRH Global / HQ Family Planning USAID Evidence to Action (E2A) Project Evaluation Haiti Latin America and the Caribbea n Health Systems Strengthening PEPFAR SSQH Project Evaluation & Project Design Follow on *Includes the following: Center for Accelerating Innovation and Impact; Health Research; Social and Behavior Change; Digital Health; The Demographic and Health Surveys Program (DHS Program); Supply Chain Program; Health Financing; Private Sector Engagement. 112 2.2. ANNEX: LIST OF QUALITY REFERENCE GROUP (QRG) MEMBERS Name Organization Title Deborah Rugg Claremont Evaluation Center; New York Founder, Executive Director, Professor Etelvina Mbalane UCSF/Global Programs; Mozambique Director of HIS/M&E Janet Myers UCSF; SF Professor John Novak Consultant; Wash. DC Global Health Advisor Melinda Hochgesang Project Concern International; San Diego, CA Director, Strategic Information for Impact Starley Shade UCSF; SF Associate Professor Tulli Tuhuma JSI Research & Training Institute; Tanzania Chief of Party Sophia Zamudio Haas Center for AIDS Prevention Studies (CAPS), UCSF Academic Specialist 113 2.3. ANNEX: QUALITY EVALUATION STANDARDS FROM PEPFAR, USAID, & SCRIVENS META-EVALUATION CHECKLIST PEPFAR USAID Scrivens Engages stakeholders Engages stakeholders (any persons, organizations, or institutions that have an investment in what will result from an evaluation and what will be done with the results) from the beginning and throughout the evaluation to ensure the success of the evaluation and implementation of the recommendations. Oriented towards reinforcing local ownership Evaluations will be consistent with institutional aims of local ownership through respectful engagement with all partners, including local beneficiaries and stakeholders, while leveraging and building local evaluation capacity. Validity Evaluations are logically and statistically sound and appropriately address the question at hand, using the necessary methods and sampling frames. Potential biases and limitations are clearly defined. All relevant values are specified in detail, scaled appropriately, measured or estimated reliably, and integrated in a defensible way to reach clear conclusions. Includes clear evaluation questions, purpose, and objectives Explicitly states the evaluation questions, purpose, and objectives. Based on best methods Evaluations will use methods that generate the highest quality and most credible evidence that corresponds to the questions being asked, taking into consideration time, budget, and other practical considerations. Clarity Evaluations are clear in their purpose, goals, objectives, funding, methods, findings, recommendations, and intended audience. Utilizes appropriate evaluation design, methods, and analytical techniques Takes into consideration the program maturity, the questions to be addressed, and the resources available to determine the appropriate evaluation design, methods and analytical techniques. Unbiased measurements and reporting Evaluations are not subject to the perception or reality of biased measurement or reporting due to conflict of interest or other factors. Evaluations conducted to meet evaluation requirements will be external (i.e., led by a third-party contractor or grantee, managed directly by USAID). Credibility Evaluations are credible in that they limit and/or address potential biases and they are free from potential conflicts of interest. Addresses ethical considerations and assurances Addresses human rights protections during planning and implementing phases. Transparent Shares findings, including making available publicly a description of methods, key findings and recommendations within three months of an evaluation’s conclusion. Propriety Evaluations are ethical, legal, and mindful of cultural/conventional appropriateness, including consideration of respect for privacy, informed consent, and avoidance of exploitation of social/class/gender/age/religious/ethnic/sexual orientation groups. 114 Identifies resources and articulates budget Identifies the evaluation budget at the start of program planning. Integrated into the design of strategies, projects, and activities Are designed with clear development hypotheses, realistic expectations of the value and scale of results, and clear understanding of implementation risks. Data is collected using high-quality methods and analyzed to establish a reference point. Data collection should be designed based on a plan for analysis, to ensure that the appropriate variables are obtained and that, the sample size is large enough to permit valid statistical comparisons. Cost utility Evaluations effectively lay out costs and benefits analysis that includes context and environmental/personal/social capital gains and losses. Utilizes data collection and Relevant Generalizability* management plans Addresses the most important and Evaluation design, implementation procedures, Creates data collection and management plans relevant questions about strategies, or results have merit and are generalizable prior to implementation to ensure that data projects, or activities; should include across various contexts. Extent to which are valid, reliable, and accessible. sufficient local and global contextual information so that the external validity and relevance of the evaluation can be assessed; should include information on the cost structure and scalability of the intervention, as well as its effectiveness. evaluation results are sustainable and immune to changes in program context or program variations of the minor kind (e.g. environment or seasonal variations). Ensures appropriate evaluator qualifications and evaluation independence Ensures that an evaluator has appropriate experience and capabilities. Manages any conflicts of interest of the evaluators (or team) and mitigates any untoward pressures that could be applied to the evaluator or evaluation team that would influence the independence of the evaluation process. Monitors the planning and implementation of an evaluation Continuously monitors the evaluation. Produces quality evaluation reports The final evaluation report contains certain elements to ensure the quality and transparency of the evaluation. 115 Disseminates results in a timely manner Evaluation results are disseminated to all stakeholders, the public and funders. Uses findings for program improvement Evaluation findings and recommendations are utilized for decision making and program improvement. 116 2.4. ANNEX: COMPREHENSIVE EVALUATION REPORTING TOOLS Comprehensive Evaluation Reporting Tool Evaluation Title: Evaluation Report Review By: Date: Key: 1 = Not addressed; 2 = Poorly/partially addressed; 3 = Adequately addressed; 4 = More than adequately addressed; 5 = Exemplar. EVALUATION REPORTING CRITERIA Was the standard met? Reviewer Comments 1 2 3 4 5 Executive Summary 1. Does the executive summary provide an accurate reflection of the most critical elements of the report, including the evaluation purpose, questions, background information, methods, limitations, findings, and recommendations? The executive summary should not add new information or contradict the evaluation report. Program Information & Project Background 2. Is the information provided about the country and/orsector context for the strategy/project/activity sufficient to provide a clear understanding of the subject of the evaluation? Is the geographic scope clear (preferably with a map)? 3. Are the basic characteristics of the strategy/project/activity being evaluated adequately described? 4. Are the interventions clearly described, and is the strategy/project/activity’s theory of change sufficiently described (preferably with a graphic and narrative description)? Evaluation Purpose 5. Does the evaluation purpose representthe management intent/reasons(as described in the SOW) for undertaking the evaluation? 6. Is it clear how the evaluation results will be used? 7. Is it clear who the primary and secondary audiences are? Evaluation Questions 8. Are the evaluation questions clearly related to the evaluation purpose? 9. Do the evaluation questionsreflect the evaluation questionsfrom the SOW? 10. If they have been modified, does the reportstate that there was written approval for changes in the evaluation questions? 11. If changed, are the new questions limited, clear, and researchable? Resources & Budget 12. Does the evaluation report include total cost of implementing the evaluation? Methodology and Approach 117 13. Does the methodology section (in report or annex) describe specific data collection and analysis methods in detail? 118 14. Is it clear which methods are used to address each evaluation qu estio n (preferably throug h a design matrix)? 15. Are the methods soun d and appropriate for each of the evalua tion qu estions(e. g., are the methods up to the task set forth by the evalua tio n questions), taking into consideratio n time, budg et, and other practical consideratio ns? 16. Does the report (or methods annex) describe specific data analys is methods the team used? (frequency distribu tio ns, cross￾tabulations, correlation analysis, etc.) 17. Are thedata collection tools(questionnaires, checklists, interview guide s ,andother instruments) used in the evaluation provided in the annex of the report or protocol? 18. Are data collection an d management procedures clearly described in the evalua tio n report? 19. If changes were mad e to the evalua tio n plan/protocol, were they documented? Team Composition & Stakeholder Engagement 20. Does the evalua tio n repor t includ e a descriptio n of the evalua tion team, including evaluator names, each member’s role, an d their background and experiences? 21. Does the evalua tion repor t indicate tha t the evaluatio n team leader was externa l to the funding partner? 22. Does the evalua tio n repor t identify local team members? 23. Does the evalua tio n repor t demo nstrate key stakeholder engagement in informing the design , implementation, and dissemina tio n of evalua tio n results? In the use of evalua tion find ings? Limitations 24. Does the report includ e a descriptio n of stud y limitations(lack of ba seline data; selection bias as to sites, intervie wees, comparison groups; seasonal unavailability of key informants)? 25. Is it clear wha t ha s bee n done to mitigate limitations or to restrict find ing sto what is perm issible given the limitatio ns? 26. Does attention to limitations flow throug h the entir e document, including the executive summary? 27. Are the conclusions an d recomm enda tions explicitly cognizan t of the limitations? Findings & Conclusions 28. Do the find ing s presented appear to be drawn fro m social scie nce data collection an d ana lysis methodsthe team described in its study methodology (including secondary data it assemble d or reanalyzed)? 29. In the presentatio n of find ings, does the team draw on data from the rang e of methods they used rather than answer using da ta fro m primaril y one method? 30. Are the find ing s clearly dist inguished fro m conclusions and recommenda tionsin the report, at le ast by the use of language tha tsignals transitions (“the evalua tio n found that…..”, “the team concluded that …..”)? 31. Are quantitative find ing s reported precisely, i.e., as specific numbers or percentagesrather than generalstatements like “some”, “many”, or “most”? 32. Is it clear which quantitative and qualitative information supports which findings? 33. Does the report present find ing s about unplanned/unanticipated results? 34. Does the report disc uss alternative possible causes of results/outcomesit documents? 119 35. Are evaluation findings disaggregated by sex at all levels(activity, outputs, outcomes) when data are person-focused? 36. Does the report explain whether access/ participation and/or outcomes/benefits were different for men and women when data are person-focused? 37. Are the conclusions directly based on findings and evidence presented in the report? Responsiveness to Evaluation Questions 38. Is the evaluation report structured to present findings in relation to evaluation questions, as opposed to presenting information in relation to program/project objectives, or in some other format? 39. Are all of the evaluation questions, including sub-questions, answered primarily in the body of the report (as opposed to in an annex)? 40. If any questions were not answered, does the report provide a reason why? Recommendations 41. Are recommendationsspecifically and clearly supported by findings and conclusions? (Can a readerfollow a transparent path from findingsto conclusions to recommendations?) 42. Are they clearly separated from findings and conclusions? 43. Are recommendations action-oriented, practical, and specific? 44. Do the recommendations assign or designate the executor of each recommendation? 45. If there are recommendationsincluded, do the evaluators develop orshare the recommendations with key stakeholders? 46. Is the process used to develop the recommendations clear? 47. Is outside expert knowledge or evidence to support a recommendation properly cited? Ethical Considerations 48. Does the evaluation report describe procedures to ensure human rights were protected with respect to privacy, confidentiality, and maintenance of the dignity of participants? 49. Did the evaluation receive IRB approval where applicable or other human-subject review (for non-research evaluation)? 50. If interviews were conducted, does the evaluation report describe informed consent procedures and ensure that participants were informed of the risks and benefits of their participation? Monitoring Planning & Implementation of Evaluation 51. Is there evidence of adequate planning and monitoring of the evaluation implementation? (i.e. work plans, timelines/schedules, deliverables, etc.) Annexes 52. Is the evaluation SOW included as an annex to the evaluation report? 53. Is the listing of sources of information in the annex clear and complete, including documents reviewed, data collection tools, interview guides, individualsinterviewed? 54. If any statements of differences are included, do the statements have merit? Does the evaluation team respond appropriately? (statements prepared by team members, the Mission, the Implementing Partner, or other stakeholder) 55. Are any potential conflicts of interest described, along with how they weremitigated? Results Dissemination 120 56. Does the evaluation report include a plan for disseminating findingsto relevantstakeholders? 57. Was the final evaluation report made available to the respective agency within 90 days of clearance/approvals by all relevant authorities? Program Improvement 58. Does the evaluation include a plan for how the findings will be used for decision-making and program improvement(e.g. mid￾course corrections, new procurements, resource allocation, intervention uptake) and a timeframe? Overall Evaluation 59. Is the report structured effectively, formatted appropriately, well￾written and clear? 60. Is the evaluation a thoughtful, well-researched, and well￾organized effort to objectively evaluate the strategy, project, or activity? 2.5. ANNEX: PEPFAR EVALUATION STANDARDS OF PRACTICE ADHERENCE CHECKLIST PEPFAR Evaluation Reporting Tool – Part II: Adherence Checklist (for completed evaluations) EVALUATION REPORTING TOOL -- Part II ADHERENCE CHECKLIST -- Completed Evaluations Only Title of evaluation Date evaluation report approved by agency Reviewer name Reviewer title and agency Date of review How were evaluation results used? EVALUATION REVIEW FACTOR Was the standard met? Reviewer Comments No Partially Yes [Please include comments to explain if No or Partially met, and reference documents/supporting materials used in making the assessment ] 122 ESoP 1: ENGAGE STAKEHOLDERS ◦ NO ◦ Partially ◦ YES 1a. The evaluation team identified the stakeholders, their information needs, and involved these stakeholders in informing the design, implementing the evaluation, disseminating, and using the results. ESoP2: CLEARLY STATE EVALUATION QUESTIONS, PURPOSE, AND OBJECTIVES ◦ NO ◦ Partially ◦ YES 2a. There is a clear description of the project being evaluated, the purpose of the evaluation, the evaluation questions, and how the evaluation results will be used and by whom. ESoP3: USE APPROPRIATE EVALUATION DESIGNS, METHODS, AND ANALYSIS ◦ NO ◦ Partially ◦ YES 3a. The selected design, methods, and analytical plan are appropriate for the evaluation questions being asked. (Please reference your agency’s protocol processes – as well as the data collection tools referred to in 3b) 3b. The data collection tools (questionnaires, checklists, interview guides, and other instruments) used in the evaluation are provided in the annex of the report or protocol. ESoP4: ADDRESS ETHICAL CONSIDERATIONS AND ASSURANCES ◦ NO ◦ Partially ◦ YES 4a. The evaluation report describes procedures in place to ensure human rights were protected with respect to privacy, confidentiality, and maintenance of the dignity of participants and received IRB approval where applicable or other human-subject review (for non-research evaluation). 123 4b. If interviews were conducted, informed consent procedures were described and documented in the evaluation report to ensure that participants were informed of the risks and benefits of their participation, as well as the lack of consequences in their eligibility to receive services regardless of their participation. ESoP5: IDENTIFY RESOURCES AND ARTICULATE BUDGET ◦ NO ◦ Partially ◦ YES 5a. The evaluation report included total cost of implementing the evaluation. ESoP6: CONSTRUCT DATA COLLECTION AND MANAGEMENT PLANS ◦ NO ◦ Partially ◦ YES 6a. Data collection and management procedures were described in the evaluation report. Changes made to the evaluation plan/protocol were documented. ESoP7: ENSURE APPROPRIATE EVALUATOR QUALIFICATIONS AND EVALUATION INDEPENDENCE ◦ NO ◦ Partially ◦ YES 7a. The evaluation report includes a description of the evaluation team including: evaluator names, each member’s role in the evaluation, and their background and experiences, providing evidence of the teams’ qualifications in the technical areas of the project and in research/evaluation methods. 7b. The evaluation report provides evidence of the management of conflict of interest for both internal and external evaluations, including statements of conflict of interest procedures and declarations to ensure credibility and mitigate bias. ESoP8: MONITOR THE PLANNING AND IMPLEMENTATION OF AN EVALUATION ◦ NO ◦ Partially ◦ YES 124 8a. There is evidence of adequate planning and monitoring of the evaluation implementation such as work plans, timelines/schedules, and deliverables by the team lead and USG staff providing oversight. ESoP9: PRODUCE QUALITY EVALUATION REPORTS ◦ NO ◦ Partially ◦ YES 9a. The evaluation report has all relevant components of a high quality evaluation report including: · cover and title pages; · executive summary; · project background · evaluation purpose and questions; · evaluation design, methods, and limitations; · findings and conclusions · recommendations; · dissemination · references · appendices (evaluation protocol/SOW, data collection tools, informed consent forms, abridged bios of evaluation team members, Conflict of Interest Statements, evaluation costs, data sources, results frameworks/logical frameworks, funding documents 9b. The evaluation report conveys that the evaluation was undertaken in a manner to ensure credibility, objectivity, transparency, and the generation of high quality information and knowledge? 125 9c. Findings are specific, concise, and supported by strong quantitative and/or qualitative evidence from multiple sources, data collection methods, and analytic techniques. If not, an explanation is provided. 9d. Each conclusion in the report is supported by a specific or clearly defined finding. 9e. Each recommendation is supported by a specific or clearly defined set of findings and conclusions, and are feasible, specific, responsive to the purpose, and actionoriented. ESoP 10: DISSEMINATE RESULTS ◦ NO ◦ Partially ◦ YES 10a. The evaluation report includes a dissemination plan for how the findings of the evaluation will be disseminated to relevant stakeholders (e.g. reports, presentations, publications, agency websites, annual reports, policy briefs). 10b. The final evaluation report was uploaded to the respective agency website within 90 days after clearance/approvals by all relevant authorities. ESoP 11: USE FINDINGS FOR PROGRAM IMPROVEMENT ◦ NO ◦ Partially ◦ YES 11a. The evaluation report includes a stated plan for how the evaluation findings will be used for decision-making and program improvement (e.g. mid-course corrections, new procurements, resource allocation, and intervention uptake) and timeframe, if appropriate. 126 2.6. ANNEX: USAID EVALUATION REPORT REVIEW TEMPLATE This Review Template is for use during a peer review of a draft evaluation report for assessing the quality of the report. For each section of the evaluation report, the Template provides a series of questions to prompt considerations of quality during the review. A box is provided to check if the section under review should be revised, and a space is provided for comments. In providing comments during a peer review, reviewers should be familiar with what was asked of the evaluation team in the Evaluation SOW and provide actionable comments appropriate to the drafting stage of the evaluation report. For checking if required elements of an evaluation report are simply present, please see the Evaluation Report Checklist. Evaluation Title: Evaluation Report Review By: Date: Executive Summary Check if revisions needed Does the executive summary provide an accurate reflection of the most critical elements of the report, including the evaluation purpose, questions, background information, methods, limitations, findings, and recommendations? The executive summary should not add new information or contradict the evaluation report. Comments: Introduction and Purpose Check if revisions needed Does the evaluation purpose represent the management intent (as described in the SOW)? Is it clear why the evaluation was conducted and who the primary and secondary audiences are? 127 Comments: Information and Background Check if revisions needed Is the information provided about the country and/or sector context for the strategy/project/activity sufficient to provide a reader without prior knowledge a clear understanding of the subject of the evaluation? Are the basic characteristics of the strategy/project/activity being evaluated adequately described? Is the geographic scope clear (preferably with a map)? Are the interventions clearly described, and is the strategy/project/activity’s theory of change sufficiently described (preferably with a graphic and narrative description)? Comments: Evaluation Questions Check if revisions needed Do the evaluation questions reflect the evaluation questions from the SOW? If they have been modified, does the report state that there was written approval for changes in the evaluation questions? If changed, are the new questions limited, clear, and researchable? Comments: Methodology Check if revisions needed Does the methodology section (in report or annex) describe specific data collection and analysis methods in detail? Is it clear which methods are used to address each evaluation question (preferably through a design matrix)? Are the methods sound and appropriate for each of the evaluation questions (e.g., are the methods up to the task set forth by the evaluation questions)? Are the methods those that would generate the highest-quality and most credible evidence that corresponds to the questions being asked, taking into consideration time, budget, and other practical considerations? Are the methods based on social science methods and tools that reduce the need for evaluator-specific judgments? Does the documentation of the methods offer sufficient expectation that if another team applied the same methods, they would generate the same findings? Limitations Check if revisions needed Are limitations to the methods used presented clearly and fully? Is it clear what has been done to mitigate limitations or to restrict findings to what is permissible given the limitations? Does attention to limitations flow through the entire document, including the 128 executive summary? Are the conclusions and recommendations explicitly cognizant of the limitations? Does the report assume external validity? Comments: Findings and Conclusions Check if revisions needed Are all evaluation questions addressed in the main body of the report? Are findings credible—presented as analyzed facts logically linked to evidence, rather than anecdotes, hearsay, and unverified opinions or documentation (e.g., from strategy, project, or activity monitoring? Are findings specific, concise, and supported by quantitative and qualitative information that is reliable and valid? Is it clear which quantitative and qualitative information supports which findings? Are the findings objective, such that if a different, well-qualified evaluator were to undertake the same evaluation, he or she would arrive at the same or similar findings and conclusions? If normative judgments are presented, is it clear what criteria were used to make those judgments? Are the findings clearly distinguished from conclusions and recommendations? Are the conclusions directly based on findings and evidence presented in the report? Comments: Recommendations (if included) Check if revisions needed Are recommendations specifically and clearly supported by findings and conclusions? Are they clearly separated from findings and conclusions? Are recommendations action-oriented, practical, and specific? Do the recommendations assign or designate the executor of each recommendation? Promising Practice: If there are recommendations included, did the evaluators develop or share the recommendations with key stakeholders in order to ‘ground-truth’ them? Is the process used to develop the recommendations clear? Is outside expert knowledge or evidence to support a recommendation properly cited? Comments: Annexes Check if revisions needed Sources of information: Is the listing of sources of information in the annex clear and complete, including documents reviewed and individuals interviewed? Data collection tools: Are data collection tools included in the annex complete? Do they match what is described in the methods section? 129 Statements of Differences: If any statements of differences are included, do the statements have merit? Did the evaluation team respond appropriately? Evaluation team: Is sufficient information provided about the evaluation team, including disclosure of conflict of interest statements? Are any potential conflicts of interest described, along with how they were mitigated? Comments: Gender Check if revisions needed Do evaluation methods incorporate attention to gender relations in all relevant areas? Do findings and conclusions address gender where relevant and appropriate? If person-level outcome data are assessed, are they sex-disaggregated? Comments: Overall Check if revisions needed Is the report structured effectively and formatted appropriately? Is it well-written and clear? Overall, is the report a thoughtful, well￾researched, and well-organized effort to objectively evaluate the strategy, project, or activity? Comments: 130 EW Resources & Budget Total Score 181 164 151 130 150 158 169 155 176 152 155 223 170 143 198 161 160 159 167 131 163 159 168 167 Executive Summary 1. 4 4 3 2 2 3 4 3 4 2 5 5 4 2 5 3 3 3 5 4 5 2 2 4 Program Information & Project Background 2. 4 4 3. 4 4 4. 4 4 Evaluation Purpose 4 3 3 4 2 2 3 3 2 3 3 4 4 3 2 2 2 3 3 4 4 2 3 3 3 5 5 5 5 5 3 4 4 4 3 3 3 4 4 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 5 4 4 3 3 3 3 3 4 4 4 5. 4 4 6. 4 3 7. 4 4 Evaluation Questions 3 3 3 3 2 2 3 3 3 3 2 3 4 3 3 3 3 3 3 4 4 4 4 3 4 4 3 5 3 3 4 4 4 2 4 3 4 4 4 4 3 3 3 3 3 3 3 3 3 3 5 3 3 3 3 5 5 3 2 5 3 3 3 3 4 3 8. 4 4 3 3 3 3 3 3 4 4 4 5 4 3 4 3 3 3 3 3 3 5 4 4 9. 4 4 3 3 3 3 3 3 4 4 4 5 4 2 4 3 3 3 3 3 3 1 4 4 10. 1 2 1 3 3 1 1 1 1 1 1 3 3 3 3 3 1 11. 1 3 3 3 3 1 1 1 1 1 1 3 3 3 3 3 1 CRITERIA (See Annex 2.4 for criteria) USAID/Burundi Integrated Health Project Mid-Term Performance Evaluation Leadership, Management and Governance Project: End Of Project Evaluation Tanzania Respond: End-Of-Project Performance Evaluation Positioning Programs for Cost Effective Outcomes: The Uganda Interagency OVC USAID/Ethiopia Midterm Evaluation of the Malaria Laboratory Diagnosis and Monitoring and Evaluation of Emergency Plan Progress II (MEEPP II) Performance Mid-Term Management Review of the Challenge TB Project Midterm Evaluation of USAID Defeat TB (DTB) Project in Kyrgyzstan Midterm Performance Evaluation of the Bangladesh Marketing Innovations For External Evaluation of the USAID Malaria Vaccine Development Program USAID/Liberia Rebuilding Basic Health Services Final Project Evaluation Final Performance Evaluation of USAID/Madagascar Integrated Social Mid-term program evaluation of the MalariaCare Project Implementation of the 2012 Joint Strategy for Supply Chain Integration in Malawi Health and Social Welfare Systems Strengthening: (HSWSS) Final Evaluation Midterm Performance Evaluation of the Bangladesh NGO Health Service Delivery Final Evaluation: Saving Lives at Birth Program Evaluation End of Project Evaluation of the Expanded Social Marketing Project (ESMPIN) In Midterm Performance Evaluation of the USAID/Rwanda Strengthening the Capacity Bangladesh MaMoni Health Systems Strengthening (HSS) Project Midterm Strengthening Health Outcomes Through the Private Sector (SHOPS) USAID/Uganda Strengthening Decentralization for Sustainability (SDS) Evidence to Action (E2A) project performance evaluation report Services de sante de qualite pour Haiti (SSQH) Evaluation Report 2.7. ANNEX: COMPREHENSIVE EVALUATION REPORTING TOOL SCORES FOR EVALUATIONS REVI 12. 3 3 3 1 1 3 2 3 3 1 1 1 3 1 1 4 2 1 1 1 1 3 1 Methodology and Approach 13. 4 3 4 3 4 4 4 3 4 3 5 5 4 4 5 3 3 3 3 2 5 5 2 14. 3 3 3 3 3 3 4 2 4 3 3 5 4 4 5 3 3 3 3 3 5 5 4 2 15. 3 3 3 3 3 3 3 2 4 3 3 5 4 4 5 3 3 3 3 3 5 2 3 3 16. 3 4 4 3 3 3 3 2 3 2 2 5 4 4 5 3 3 3 3 2 5 3 2 2 17. 4 3 4 3 3 3 3 3 4 2 5 5 4 1 4 3 3 3 3 1 5 3 4 18. 3 3 3 3 4 3 4 2 3 2 2 4 4 4 5 3 3 3 3 2 3 5 3 2 19. 1 3 1 3 3 1 1 1 1 1 1 3 3 3 3 3 3 Team Composition & Stakeholder Engagement 20. 2 3 2 2 1 3 3 3 1 3 1 5 1 1 3 3 3 3 3 2 2 2 4 2 21. 3 3 3 2 1 4 2 4 1 3 3 3 1 1 1 3 3 3 3 3 1 3 3 3 22. 2 2 2 2 1 3 2 4 1 1 1 1 1 1 1 3 2 3 3 3 2 3 3 3 23. 3 3 3 2 2 2 3 2 3 4 3 3 1 1 1 3 3 3 3 2 1 1 3 3 Limitations 24. 4 3 2 4 3 3 4 3 3 4 4 5 4 3 4 2 5 3 3 3 5 4 2 3 25. 4 2 2 4 3 2 3 3 3 2 3 5 4 3 4 3 3 3 3 2 5 4 2 3 26. 4 3 3 4 3 3 3 2 3 1 3 4 4 3 5 3 3 3 2 2 2 3 2 3 27. 3 3 3 4 3 3 3 3 3 2 2 4 4 4 4 3 3 3 3 2 2 3 2 3 Findings & Conclusions 28. 4 3 3 3 3 3 3 3 4 3 2 5 4 4 5 3 3 3 3 3 4 2 3 3 29. 4 3 3 3 4 4 3 3 4 3 2 5 4 3 5 4 3 3 3 3 4 4 3 4 30. 3 3 3 3 3 3 3 3 4 4 3 4 4 3 4 3 3 3 3 3 4 2 3 4 31. 3 3 3 2 3 4 3 3 4 3 3 5 3 3 4 5 3 3 3 4 4 3 3 4 32. 3 3 2 3 3 3 3 3 4 3 1 5 3 3 5 3 3 3 3 3 3 3 3 4 33. 3 3 2 1 3 1 3 3 3 1 1 1 4 3 3 3 3 3 3 1 1 1 3 3 34. 3 1 2 1 3 3 3 3 3 1 1 5 4 4 3 3 3 3 3 1 1 2 3 3 35. 4 3 3 3 2 1 3 3 3 1 1 5 3 1 1 3 3 3 3 1 1 4 3 36. 4 3 3 3 3 1 3 4 3 1 1 1 1 1 1 3 3 3 3 1 1 4 3 37. 4 3 3 3 3 3 3 3 4 3 3 4 4 3 4 3 3 3 3 2 3 4 3 4 Responsiveness to Evaluation Questions 38. 4 4 3 3 3 4 4 3 4 4 4 5 4 3 4 3 3 3 3 3 5 3 4 5 4 3 5 4 39. 4 4 40. 3 3 Recommendations 41. 4 3 3 3 3 4 3 3 4 4 3 4 4 3 5 4 3 3 3 3 4 4 4 4 42. 3 3 3 3 3 3 3 3 4 5 4 4 4 3 4 3 3 3 3 3 4 4 4 4 43. 4 4 3 3 3 3 4 3 4 4 3 5 4 4 4 3 3 3 3 3 4 4 4 4 44. 3 3 2 2 2 3 4 4 3 3 2 4 2 3 3 3 4 3 3 3 3 3 3 3 45. 2 2 3 1 3 2 3 2 3 1 1 4 1 1 3 3 3 3 3 1 1 5 3 3 46. 3 2 1 1 3 3 3 3 3 3 3 4 3 4 3 3 3 3 3 1 1 1 3 3 47. 3 2 2 2 3 3 3 3 3 3 2 3 2 2 2 3 3 3 3 3 3 3 Ethical Considerations 48. 3 3 3 49. 2 3 3 50. 2 3 3 3 3 4 3 3 1 1 3 3 3 1 3 2 3 1 3 3 3 Monitoring Planning & Implementation of Evaluation 51. 3 3 3 Annexes 52. 3 3 4 53. 3 3 4 54. 1 1 55. 3 3 Results Dissemination 56. 3 2 2 57. 1 1 1 Program Improvement 58. 4 3 Overall Evaluation 59. 3 60. 3 2 3 1 1 3 3 3 3 3 3 3 3 3 3 3 1 3 1 3 1 2 3 3 1 2 1 3 1 3 3 3 1 2 3 3 2 3 4 4 3 3 3 3 3 3 2 3 3 3 3 3 4 2 3 3 3 3 4 3 3 1 3 3 4 4 3 3 3 1 1 4 4 4 1 2 1 2 4 5 5 1 1 3 1 3 4 3 4 1 3 1 4 4 3 5 2 1 3 1 3 3 3 5 3 5 5 5 5 3 5 4 1 3 1 3 4 5 4 1 1 1 1 2 4 4 1 5 1 1 3 4 4 3 1 1 1 1 1 3 1 1 1 4 1 4 3 4 3 1 5 5 4 5 4 4 3 3 3 1 4 4 4 3 3 3 1 3 2 2 3 3 3 3 3 3 2 2 3 3 3 1 3 3 3 3 3 3 3 3 3 3 2 3 3 3 1 3 3 3 3 2 3 5 3 1 3 4 3 5 3 3 5 3 3 3 1 3 3 5 3 3 3 3 2 1 1 3 3 4 1 3 3 1 1 4 4 5 1 1 3 3 5 3 3 1 3 5 4 3 3 1 3 3 3 5 3 4 1 2 4 4 3 1 3 4 4 4 4 3 3 3 1 1 4 4 1 3 1 1 4 4 3 3 4 4 3 1 4 4 2.8. ANNEX: QUALITATIVE FINDINGS OF GH PRO EVALUATIONS USING THE COMPREHENSIVE EVALUATION REPORTING TOOL, BY EVALUATION COMPONENT: STRENGTHS AND AREAS FOR IMPROVEMENT Components Strengths Areas for Improvement 1 Executive Summary • A few evaluation reports had well-written executive summaries with clearly outlined high￾level findings. • In some evaluations, specific details, such as evaluation questions, methods, and limitations, were missing from the executive summary section. 2 Program Information & Project Background • In most evaluation reports, basic characteristics of the strategy/project/activity being evaluated were adequately described. • In some evaluation reports, interventions were clearly described. • A few evaluation reports supplemented their respective geographic scope descriptions with a detailed map. • In many evaluations, the theory of change framework was missing. • In some evaluations, the level of detail was limited (missing country maps, specific country context to problem/issue). • In some evaluations, information was not available in the body of the report; readers had to search in the annex. 3 Evaluation Purpose • In most evaluation reports, evaluation purposes were clearly described. Intended audiences were noted, though some evaluation reports made the distinction (primary vs. secondary) in the SOW itself (in an annex). • In many evaluations, there was limited information on how results will be used for intended audiences. • In some evaluation reports, the potential use of evaluation results was only briefly described. 4 Evaluation Questions • In most evaluation reports, evaluation questions were noted and were related to the respective evaluation purposes. • In a few evaluations, it was observed that evaluation questions may not be applicable to the stated purpose of the evaluation. • In most evaluation reports, the evaluation questions remained the same as the SOW (no modifications). • In a few evaluations, written approvals for modifications were not always specified. 5 Resources & Budget • N/A • None of the evaluations had positive comments on resources and budget. • In most evaluations, information on resources and budget for the evaluation was missing. 6 Methodology & Approach • In most evaluation reports, the methodology was described, with the descriptions ranging from general to very detailed. • In most evaluation reports, the evaluation plan/protocol remained the same (no modifications). • Not all evaluation reports included an annex for more detailed description of methods used for evaluation. • Not all evaluation reports included data collection tools in the annex. • Not all evaluation reports included description of data collection and management procedures. • In most evaluations, there was a very short period of performance to conduct the evaluation. • In many evaluations, there was limited detail on methods (who, what, how, etc.). • In some evaluations, there was limited information on data management and analysis. • In a few evaluations, data collection tools were not provided in the report/annex. 7 Team Composition & Stakeholder Engagement • Some evaluation reports described the presence/engagement of local team members. • In some evaluations, there was missing information on team composition and whether the evaluation was conducted externally. Reviewers felt that more description was important to ensure lack of bias in the evaluation. 135 • In some evaluations, information on team members’ backgrounds and experiences was missing. • In a few evaluation reports, stakeholder engagement was described in detail. 8 Limitations • In some evaluation reports, study limitations were well described. Among these, a few evaluation reports provided detailed description of how each limitation was mitigated. • In some evaluation reports, limitations were described within the body of the report but not in the executive summary. • In some evaluations, the limitations of the evaluation were not explicitly described. 9 Findings & Conclusions • In most evaluation reports, multiple data sources/methods were referenced when presenting findings. • In most evaluation reports, findings were clearly distinguished from conclusions and recommendations. • In many evaluation reports, quantitative findings were presented specifically. • In some evaluations, the findings presented were very general and not necessarily tied clearly to the methods. 10 Responsiveness to Evaluation Questions • In most evaluation reports, all evaluation questions were addressed. • In all evaluations, evaluation questions were clearly responded to. 11 Recommendations • In most evaluation reports, general recommendations were described. • In some evaluation reports, recommendations were specific and action oriented. • In some evaluations, the executor for the recommendations was not always designated. • In some evaluations, the process used to develop the recommendations was not always clear. 12 Ethical Considerations • In some evaluation reports, the SOW contained a description of the procedures to ensure ethical research conduct for the protection of human subjects. • In most evaluations, ethical considerations were not addressed. • In some evaluations, consent tools/forms were not present. 136 • 13 Monitoring, Planning, & Implementation of Evaluation • In many evaluation reports, planning and monitoring plans (including timeline) were included in the SOW. • In most evaluations, documentation of the monitoring, planning, and implementation process was present in the evaluation but not in the actual body of the report. 14 Annexes • In most evaluation reports, a clear and detailed evaluation SOW was included in the annex. • In some evaluation reports, signed conflicts of interest forms were included in the annex. • In many evaluations, Question #54 of the Comprehensive Evaluation Reporting Tool, regarding statements of differences, was not clear. 15 Results Dissemination • In some evaluation reports, a general description of the dissemination plan was provided in the SOW. • In some evaluations, conflicts of interest were not identified. 16 Program Improvement • In most evaluation reports, a general description of the program improvement plan was described. • It was observed that program improvement plans were briefly described but did not include the intended audience, process for program improvement, or evidence that feedback was incorporated to improve programs. Overall Evaluation • It is difficult to comment overall since this was a case-by-case review. • But in general, there seemed to be a consensus that most of the evaluation reports addressed the evaluation purpose/objective. • It is difficult to comment overall since this was a case-by-case review. • But in general, QRG members noted that some evaluations could be more concise. 137 3.0. EVALUATION QUESTION 3 ANNEXES 3.1. ANNEX: SUPPORT MECHANISMS MATRIX A. SUPPORT MECHANISMS MATRIX FOR EVALUATION Mechanism Name USAID Oversigh t Contractors Main Role/ Areas of Focus Period of Perform ance and Annual Budget Description Business Model (Core Funding, Direct/Indirect Costs) Evaluation Types Capacity for Evaluation Methodologies Scoring for MAP Chart MEASURE Evaluation BW UNC with Ev 7/1/14- The objective of MEAUSRE Evaluation Phase IV is to MEASURE Evaluation has experience Large scale, MEASURE Avg cost = Phase IV (MEASURE) (Bureau￾wide) GH/OH/ SIEI Consortium of ICF, JSI, MSH, Palladium, & Tulane University 6/30/19 enable countries to strengthen their national, community, and facility-based systems to generate high￾quality information that is used for decision making at all levels of the health system. To achieve this objective the project concentrates on four results: 1. Strengthened collection, analysis, and use of routine health data; 1. Improved country-level capacity to manage health information systems, resources, and staff; 3. Methods, tools, and approaches improved and applied to address health information challenges and gaps; 4. Increased capacity for rigorous evaluation. MEASURE Evaluation Phase IV is a GH Bureau-wide project that it provides assistance to all health elements and includes a portfolio of integrated, system-wide strengthening approaches. This phase of the project emphasizes strengthening the collection, analysis, and use of routine health information, and overarching country HIS sustainability. Related areas of technical assistance (TA) include system design and conducting a wide range of rigorous evaluations to provide the strongest possible evidence on the evaluation questions in the resources and time available. The project also conducts methodological studies, pilot tests or proof of concept studies, and population￾based surveys and surveillance. Builds monitoring and evaluation of capacity through the development of a cadre of local professionals to conduct rigorous evaluations. Supports opportunities for formal training, through local and regional workshops and by partnering with local universities, incorporating advanced methods in graduate school curricula. complex impact and performance evaluations. Evaluation offers expertise in a full range of evaluation procedures, including the development of rigorous study designs, protocol development, ethical research practices (including access to U.S.- based and in￾country ethics review boards), sample-size estimations, $2,000,000 Perf/Imp = 5 COR: Kristen Wares interoperability, and data quality, security, management, questionnaire kwares@usaid.gov analysis, visualization, and use. Another area of emphasis for the project is increasing local capacity for implementation of rigorous evaluation (including impact evaluation). Related areas of TA include quantitative/qualitative study design, protocol development, gender and ethical considerations, use of geographic information, and mentorship and training. MEASURE Evaluation Phase IV is not intended for the implementation of performance evaluation. development, statistical methods, costing methods, Population based surveys qualitative methods, and biomarker measurement. PPL Monitoring and Evaluation (EVAL-ME) PPL/LER Pangora Ev, TA The Pangora Group offers a range of health and development services and experts, across family planning/reproductive health (FP/RH) and maternal child Utilizes an IDIQ. Women owned, small business Avg cost= $1,800,000 138 COR: Winston Allen wallen@usaid.gov health (MCH), neonatal health, nutrition, HIV/AIDS, TB, and malaria; and in cross-cutting areas such as health systems strengthening, community mobilization, capacity-building, private sector health including public￾private partnerships, and gender/youth mainstreaming. This can be provided as short-term or long-term technical assistance in-country or in Washington, DC through institutional support mechanisms. PPL’s Evaluation and Performance Monitoring Services IDIQ can provide technical and advisory services for evaluation and performance monitoring activities worldwide, at the Mission (Operating Unit), Bureau, or Agency-wide. It can commission performance evaluations, impact evaluations, M&E platforms, synthesis of evaluation lessons, local M&E capacity building, or performance monitoring plan. Perf/Imp 4.75 Project Supporting Operational AIDS Research (Project OHA Prime: Population Council; Subs: Elizabeth Ev OR 9/15/14- 9/14/19 The goal of Project SOAR is to improve the quality and impact of HIV and AIDS program outcomes by strengthening operations research and evaluation activities. Project SOAR supports the PEPFAR Blueprint Core funding, with three main funding sources: PEPFAR core, PEPFAR central and field missions. Avg Cost = $1,800,000 SOAR) Glazer Pediatric AIDS Roadmap for Driving Results with Science and contributes to PEPFAR's Impact Agenda. While Population Council is the prime, they operate like a consortium. Others are UNC Perf/Imp 4 Foundation; Futures Group (Palladium?); Futures Institute; This is an operations research mechanism conducting research and evaluations to generate evidence to improve HIV prevention, care, and treatment services. Includes operations research, capacity building, Chapel Hill, Johns Hopkins University, EGPAF, Palladium, and Avenir Health. They each manage different aspects of the portfolio that they take the lead on. AOR: Allison Cheng Johns mathematical modeling, and impact evaluations. There At Population Council, the project has a acheng@usaid.gov Sarah Sandison Hopkins, UNC Chapel Hill are three main functions: • To conduct operations research and director, two deputies (one operations, one technical). The project also has a Senior Advisor for Data Utilization, a role that ssanderson@usaid.gov implementation science • To strengthen capacity o We coordinate with the project on those activities, and every activity is supposed to work with a local research organization that is both in-country and works with individuals from that country, who should be co-investigators o This component is part of the small grants program, and it is done in a way that ensures quality • To improve data utilization/translation o When research findings become available, they should be used by policymakers and other decisionmakers o In earlier iterations of research awards, a common criticism was that the reports produced were getting put on shelves and not engages in-country stakeholders. Beyond those core positions, other staff are dedicated to producing reports, planning public webinars, coordinating, and branding. They also have a lead point of contact from each of the consortium members, and someone at Johns Hopkins University coordinates all the activities. SOAR does not have core funding for core technical staff, but that funding is built into each study budget, as that was more acceptable to PEPFAR funding requests. This built-in percentage is called the technical leadership percentage. There is a continuous effort to engage in￾country stake holders and partners. This standard approach to local solutions goes well beyond just requiring a data utilization plan. A 139 getting used in programs or policies, so that is an active priority for my team standard percentage of the overall cost goes to the stakeholder engagement, capacity strengthening and data utilization Accelerating Strategies for Practical Innovation and Research in OHA FHI360 Ev TA 7/1/13- 6/30/18 The primary goal of the project is to support gender￾sensitive programming, research, and learning to improve the economic security of highly vulnerable individuals, families, and children infected or affected by ASPIRES focuses on efficient provision of technical assistance (TA) to scale up high￾quality interventions in the areas of consumption support, money management, Economic research Avg Cost = $1,500,000 Perf/Imp Economic HIV/AIDS, as well as key populations at high risk of and income promotion. 3 Strengthening acquiring HIV. (ASPIRES) Provides technical assistance to scale up high quality The project also focuses on design and implementation of rigorous research to COR: Colette Peck interventions in consumption support, money management, and income promotion. Design and evaluate programs and inform a new understanding of best practices in ES. cpeck@usaid.gov implementation of research and evaluation to inform programming in economic strengthening. Global Health Program P3 Prime: Dexis Ev 7/1/14- GH Pro provides consultants for: Technical assistance-- Small business set aside Rapid Surveys, key Avg Cost = Improvement Cycle Project (GH Pro) Sub: QED TA MS 7/1/19 Individuals and teams for strategic planning, project design, assessments/analytical work and other types of technical assistance; Mission/staffing support-- No core funding. Instead, use PMO for core staff and functions performance evaluation at the project level informant interviews, document $200,000 Perf/Imp Managerial and technical support for short-term and medium-term assignments (up to 6 months) to fill gaps from temporary absences and provide surge capacity; review 1 COR: Carl Hawkins chawkins@usaid.gov Logistical support for meetings/conferences--Assistance for planning and implementing meetings, conferences, workshops and other events; Evaluation--Performance assessment and evaluation services through independent consultant teams. Typically, appropriate for examining a single activity (mechanism). Conducts routine mid-line and end-line performance evaluations and provides a mechanism to hire consultants for technical assistance and strategic planning. Includes: performance evaluations, TA support, and conference/facilitation support. 140 Monitoring, Evaluation, P3 Prime: Social Ev 3/2015 – Monitoring, Evaluation, Research and Learning Innovative, rapid performance evaluations of Avg Cost = Research and Learning Innovations Network Impact: Subs: Search for Common 3/2019 Innovations Program (MERLIN) is a USAID endeavor led by the U.S. Global Development Lab and in partnership with the Bureau for Policy, Planning and Learning and country projects. Use of new evaluation approaches. Rapid feedback $300,000 Perf/Imp (MERLIN)/ includes Ground, the the Bureau for Global Health. It aims to source, co-create Utilized Broad Agency Announcement (BAA) 2 DEPA MERL, SPACES William and co-design development solutions that innovate on for procurement, which was a very MERL, RAPID FEEDBACK Davidson traditional approaches to monitoring, evaluation, collaborative and iterative process. BAA MERL, & ERIE (See below) Institute at the University of Michigan research and learning (MERL). It contains SPACES MERL, DEPA MERL, Balanced-MERL, ERIE, and Rapid Feedback MERL (listed below) encourages use of diverse, local sources. Overall project has not yet been adequately evaluated. Will do “rolling client satisfaction” surveys. AOR: Sophia van der Bijl svanderbijl@usaid.gov Each component under the MERLIN umbrella focuses on specific sector, often utilizing innovative methods. Currently there are 20+ pilot projects. MERLIN/Developmenta P3 Ev 3/2015 – DEPA-MERL will test of the effectiveness of a See above l Evaluation Pilot Activity (DEPA-MERL) 3/2019 developmental evaluation approach, as well as accompanying monitoring and evaluation tools and flexible contracting mechanisms, in achieving effective programming for innovative interventions, untested theories of change, and/or implementation in complex COR: Shannon Griswold sgriswold@usaid.gov contexts. DEPA MERL provides an approach to evaluation that is quick, ongoing, and takes an iterative approach to data collection, analysis and feedback that contributes to timely changes throughout the project cycle and allows for system changes as well as changes in target outcomes. MERLIN/Rapid Feedback MERL P3 Prime: R4D; Subs: Abt Associates, Mathematica, Notre Dame Ev 3/2015 – 3/2019 Rapid Feedback MERL is a collaborative approach to learning and adapting. Improved data capture and compressed feedback loops provide decision-makers with timely, actionable evidence. Design and implementation decisions can be optimized to maximize See above AOR: Sophia van der Bijl Initiative for chances of impact and improve prospects for long-term (svanderbijl@usaid.gov Global Development success. MERLIN/SPACES MERL P3 Prime: Global Obesity Prevention Center at Johns Hopkins University; Subs: Global Knowledge Initiative, LINC, Resilient Africa Network at Makarere University Ev 3/2015 – 3/2019 SPACES MERL aims to bring a variety of tools and methodologies that decision makers can use (alone or in combination) to enhance assessment of innovation impact potential and to provide a comprehensive systems analysis. See above 141 AOR: Sophia Van der Bijl svanderbijl@usaid.gov School of Public Health MERLIN Expanding the P3 Ev 3/2015 – Expanding the Reach of Impact Evaluation (ERIE) is an See above Reach of Impact Evaluation (ERIE) 3/2019 approach to conducting retrospective long-term impact evaluations of completed USAID interventions. These evaluations will leverage and build on existing program data to either assess if the observed short-term impacts are sustained, or to investigate results which might only AOR: Sophia van der Bijl be expected to emerge over a long-term horizon. We will svanderbijl@usaid.gov use innovative data collection strategies and methods to identify the appropriate counterfactuals, and generate lessons on how to plan for and conduct these long-term impact evaluations which can be incorporated in USAID bureau and mission planning. Coordinating MCHN Social Ev 2016 - Provides technical, logistical and administrative support Utilized BAA for procurement. Supportive CIRCLE is involved Avg Cost = Implementation Research to Solutions International TA 2021 to USAID’s Health Research Program. CIRCLE’s goal is to facilitate and support systematic research and research translation to accelerate Ending Preventable Child and services contract CIRCLE is seen as an extension of the MCH in developmental evaluation in integrated health $1,500,000 Perf/Imp Communicate Learning Maternal Deaths, an AIDS-Free Generation, and advance health research team. Provides flexibility. bilateral program. 4 and Evidence (CIRCLE) other global health priorities. CIRCLE supports the Health Research Program to advance research-to-use Started out as core, very lean team, but they can expand as needed and secure the kind of Even though it’s called evaluation, COR: Sara Sulzbach ssulzbach@usaid.gov processes through: stakeholder engagement; knowledge management to facilitate communication and collaborative learning; research translation; strategic technical expertise that USAID needs at a given time “Accordion Model” it’s really more embedded, iterative planning; and targeted research and analysis activities. Broad span of activities: convening/ consultative/ technical meetings/logistics. They have a full range of technical and applied research that they can support. Can help implement BAA process. Have Core staff and core funding. Striving for mix of PMs and people with technical capacity as well. Full time Project Director, Deputy Director/Research Lead, RA, additional staff for KM/Comms, and another rooted in MCH research/practice, project assistant, finance person implementation research. Breakthrough Research GH/PRH Population Ev 8/2017 - Focuses on Social and Behavior Change (SBC) in Cooperative Agreement. The bulk of the Avg Cost = Council, TA 8/2021 combination with Breakthrough-ACTION. Breakthrough- funding is core funding that is received from $2,000,000 Avenir Health, RESEARCH can accept funds from any USAID Mission or cross-bureau funds including PRH, MCH, and ideas42, operating unit. It will convene and engage a broad range Zika. And this works since the consortium Perf/Imp Institute for of health and development stakeholders, supporting consists of big, research-focused institutions 4.2 AOR: Hope Hempstone Reproductive them in developing, promoting, and operationalizing who have many research people on staff. hhempstone@usaid.go Health at Georgetown visionary, consensus-driven agendas for SBC research that contribute to measurable global health impact. In Breakthrough-RESEARCH is largely designed v University, addition to designing and implementing high-priority SBC for missions looking to do behavioral research Population studies, Breakthrough-RESEARCH may conduct or to evaluate social and behavioral change Reference performance and impact evaluations of Breakthrough- interventions, including, for example, Bureau, and ACTION activities and bilateral SBC mechanisms. 142 Tulane University Breakthrough-RESEARCH is a technical leader that is housed in the Bureau for Global Health. Most of the project’s work is in global health, but they also do R&D in other sectors, and a lot of social and behavioral change projects are cross-sectoral. These can include typical food for peace, agricultural initiatives pertaining to maternal and child health nutrition, WASH, etc. Breakthrough-RESEARCH can evaluate all the different behavior change elements of a project beyond just the health. behavioral elements of service delivery projects, etc. Breakthrough-RESEARCH currently implements studies that range from 5-year, $3 million impact evaluations to 1-year, $200k evaluations, and everything in between. The project does not focus on performance evaluations. 143 B. SUPPORT MECHANISMS MATRIX FOR MISSION SUPPORT AND TECHNICAL ASSISTANCE Mechanism Name USAID Oversight Contractors Main Role/Areas of Focus Period of Performance and Annual Budget Description Business Model (Core Funding, Direct/Indirect Costs) Staffing Patterns COR/AOR Comments GHSI-III GH/ PDMS Camris MS 12/16 to 12/21 This is a USAID Direct Institutional Primarily Washington based with 2 to 4 Administrative Support Services Contract year timespans. These are considered that provides human resource-based institutional support staff. Almost all are support services to USAID’s health in Washington offices. programs worldwide. Persons hired under this contract will be mid-career or senior Recruiting candidates is challenging professionals in technical areas, with because these are essentially “semi￾support staff, and will supplement USAID’s permanent staff.” Staff can be recruited cadre of health professionals. for general health jobs, or for very specific areas, e.g. Zika infections. Persons hired under this contract may serve Recruitment is a more formal and serious in the Bureau for Global Health, Regional process, since these are not short term Bureaus in Washington, D.C., or in field staff. 250 staff with ongoing needs COR: Tara Lewing missions around the world (excluding support staff), and they will complement Ongoing challenges around equity with tlewing@usaid.gov USAID’s cadre of health professionals in direct hire staff for staff management, technical and professional specialties. Their performance evaluations, promotions duties will focus on supplementing health and salaries. programs with their specific skills and experience. These professionals will be expected to contribute to Agency technical leadership in the health sector. Specific goals that they are expected to embrace include: • Improving global health, including child, maternal and reproductive health; • Reducing disease, especially HIV/AIDS, malaria, tuberculosis, and polio; and • Increasing access to improved drinking water and sanitation services. Global Health Fellows GH/PDMS Public MS 10/1/11- GFP-II helps USAID address its immediate Provide fellows for 2 to 4 years. Many of Currently 125 Program II (GHFP-II) Health Institute 9/18 and emerging human capital needs by these fellows go on to professional developing a diverse group of global health positions in global health. professionals to support and sustain the effectiveness of the Agency's current and fellows AOR: Christiana Mpaka future health programs cmpaka@usaid.gov Rapid Staff Support DCHA/CS3 MS Ongoing CS3 Firehouse members represent a broad 50 PSCs Services (formerly Firehouse) range of international development expertise and are familiar with USAID processes and mechanisms. Firehouse members are available to USAID Missions COR: Peter Henderson upon request to support management of phenderson@usaid.gov field staff, provide technical expertise, temporarily fill newly created positions, or 144 fill vital vacant positions. The Firehouse’s wide array of technical skills ensures that CS3’s response to Mission requests is effective and fully integrated. Procurement Support Jefferson MS 8/1/2013- The PSA supports the full acquisition and Award (PSA) Consulting Group 8/31/18 assistance lifecycle for Global Health￾funded and health-related projects. The Washington Office as well as Missions can request PSA staff for short-term or long￾COR: Kelly Saldana term services. They support for the ksaldana@usaid.gov following activities: Assist in the negotiation and administration of contracts, grants, cooperative agreements, and interagency agreements Global Health Encompass Training/ 2016 - 2021 The GHPOD program is focused on Small Business set aside procurement. Professional and Organizational Social Impact Conf Org improving the effectiveness of USAID's health sector by developing and increasing capacity of its health professionals through Provides in country face to face training online training, and conference and training support to USAID, including the Development II high quality professional and organizational SOTA meetings. Conducts over 40 (GHPOD II) development services worldwide. GHPOD provides a variety of PD and OD activities activities per year. COR: Michael Wilburn mwilburn@usaid.gov that focus on technical, management, leaders, and personal effectiveness. The program offers a broad range of tailored organizational and professional development assistance through training, teambuilding, strategic planning, and meeting facilitation. 145 3.2. ANNEX: DISCUSSION GUIDE FOR INTERVIEWS WITH COR/AORS Date and location of interview_________________________________________________ Name and Title of Interviewee_________________________________________________ Name of Project (Cooperative Agreement, Contract)_______________________________ Introduction and Rationale for Interview Thank you for agreeing to participate in this process. We are: Paul De Lay, Team Leader for this evaluation, and _______________________. This assessment is being conducted by the USAID HEARD Project, under the direction of P3. It is part of a mid-term evaluation of the GH-Pro Project, but is a separate assessment aimed at understanding the range of evaluation and technical support available to USAID Missions Question: Evaluation Question 3: What is the comparative value of GH-Pro to USAID, considering other, existing mechanisms. More specifically, this assessment will assist us in formulating guidance on how USAID Service Support mechanisms can be made more responsive, effective, efficient and able to provide higher quality products. This includes improving the information provided to USAID clients when they are choosing a service support project and would also inform future procurements. We are not evaluating the individual projects performance, but rather looking at their business models, operational systems, staffing patterns and other parameters that could provide insight for future projects. As part of this process, we are interviewing CORs/AORs who manage Global projects with evaluation and field technical support components. This interview is voluntary and should take between 30 and 45 minutes. We will not attribute any comments to you, if you choose to remain anonymous. We will provide you with a summary of the interview for you to review and revise if needed. Background experience and expertise of participant, including time within USAID, serving as COR/AOR for other projects, if also involved in drafting of RFP/RFA Describe your current role as COR/AOR for a USAID Service Support Project What services does your Project provide for USAID?? 146 Describe what you consider are the major successes of your Project in responding to field requests. What have been the major challenges that the Project has experienced and how have these been dealt with? (Specific questions about evaluation and TA related to use of local resources (Local solutions), preparation for the team’s work, usefulness of the results and the specific capacities that your project currently has) For follow-on and future procurements, what would you recommend to improve the effectiveness, efficiency and quality of USAID support mechanisms? (specific questions could include Cooperative Agreement versus Contract, Small Business Designation, Use of Core Funding, Permanent Staff versus use of consultants, Multi￾tasking or focused on one service support area, etc) ______________________________________________________________________________ 3.3. ANNEX: QUESTIONNAIRE FOR CORS/AORS OF USAID SERVICE SUPPORT MECHANISMS The Office of Policy, Planning, and Programs (P3) would like to expand the Agency’s capacity to evaluate program performance and impact. This Questionnaire/Checklist is intended for the CORs/AORs for the current USAID mechanisms that provide evaluation/operations research, and implementation science services. This information will be used to assess the extent that the Agency uses local resources, to expand the information contained in the USAID Program Users Guide and to inform the development of future evaluation project mechanisms. We would like the COR/AOR to work with their respective project management teams to complete this short document and return it to Paul R De Lay at pkdelay11@me.com. Strengths and Challenges What do you consider the primary strengths of your project in the design and implementation of evaluation/operations research/implementation science activities at country, regional and global level? What do you consider the primary challenges that the project confronts when planning and implementing evaluation/OR/IS activities? Local Resources To what extent does this project utilize local institutions and/or local experts when conducting evaluation/OR/IS activities? ____Always (Nearing 100%) ____Sometimes ____Never To what extent does this project sub-contract local institutions (bureaus of statistics, private evaluation and survey firms) when conducting evaluations? ____Always (Nearing 100%) ____Sometimes ____Never If local evaluators are utilized as members of evaluation teams, what types of local resources have been used in executing evaluations/OR/IS? Please indicate the number of evaluations executed under your project that have used local experts in each of the following roles. ____Evaluation leads/managers 148 ____Technical Evaluation experts ____Field Staff ____Translators/Logistics ____Other – please explain: ____________________________________ If local resources are used to implement an evaluation, what approximate percentage of the project’s funds for evaluation/OR/Implementation Science goes to local subawards since the beginning of the project ____% Evaluation Types How many evaluations has the project completed since inception? ________ What proportion of the project's completed evaluations since inception fall into each of the seven categories of evaluation below? ____Performance ____Process ____Outcome ____Impact ____Economic ____Operations research ____Implementation science ____ Other Evaluation Methodologies Does your mechanism have the capacity to utilize the following evaluation methods and approaches? Method Project has the capacity to design and implement this methodology with either current staff resources or through use of consultants Approximate # of activities the project has used this methodology in evaluation/OR/IS implementation since inception Developmental evaluation Participant/Direct observations Secondary data analysis (program data, surveillance data, service provision data) Focus Group Discussions Customer satisfaction surveys/Exit interviews Outcomes harvesting Mathematical modeling Contribution analysis Plausibility analysis Cost Benefit Cost Effectiveness Cost Utility Most significant change Household surveys Facility Surveys Key Informant Interviews Rapid experiments Rapid Appraisals Theory based evaluation Appreciative Inquiry 150 3.4. ANNEX: STRATEGY REFERENCE GROUP PARTICIPANTS NAME CURRENT POSITION USAID EXPERIENCE Barbara Turner Barbara N. Turner is currently Barbara has over 35 years of experience in international (chair) serving as a Senior Advisor at University Research Co., LLC where she served as President from 2005-2016. public health and development, primarily with the U.S. Agency for International Development. For USAID she was in charge of the foreign assistance budget development (over $14 billion) and associated policy. She also led the Global Bureau technical functions, particularly health, and led development activities in the former Soviet Union and the Balkans, served as a Public Health Officer for the Middle East and Asia, and was a leader in introducing innovative public-private partnerships. Carol Peasley Carol is currently working as an independent consultant and serving on the boards of Plan International USA; Enclude (formerly ShoreBank International, a consulting firm focused on inclusive finance); and the USAID Alumni Association. Carol retired from USAID in 2005 after a 35-year career, achieving the rank of career minister. She held senior positions in Washington and overseas, including as USAID mission director in both Malawi (1988-93) and the Russian Federation (1999-2003), and as Counselor to the Agency and Senior Deputy Assistant Administrator for Africa in Washington. This included two years as acting Assistant Administrator for Africa (1996-98). She also had overseas assignments in Nepal, Thailand, and Costa Rica. David Oot Since leaving USAID in 1997, David led the health and nutrition department at Save the Children-US (SC) until 2014, and currently serves as a senior advisor for health and nutrition within SC’s department of global health. Served as a health officer with the US Agency for International Development (USAID) for 25 years, including assignments in Vietnam, Pakistan, Thailand, Kenya, and Nepal, as chief of Population, Health, and Nutrition in USAID’s Bureau for Asia, and as Director of USAID's Global Bureau Office of Health and Nutrition. In my work with Save the Children, frequently interacted with USAID both directly and indirectly, including involvement in identifying and pursuing USAID funding opportunities, and oversight of USAID-funded global and country-level activities. Joy Riggs-Perla In June 2013, Joy joined Save the Children as the Director of the Saving Newborn Lives program supported by the Gates Foundation. Joy was a Foreign Service Officer with USAID for 23 years, serving long-term assignments in Swaziland, Philippines, Indonesia and Egypt. From 1997 to 2001, Ms. Riggs-Perla served as Director of USAID/Washington’s Health office, where she directed global health programs, providing strategic direction for programs in HIV/AIDS, maternal health, child survival and infectious diseases. Cynthia Rozell Consultant for USAID for past 15 years, specializing in project design, project implementation, local solutions and NPA, and performance evaluation. Joined USAID as an FS Direct Hire in July of 1979; Over those years worked in LAC (4 tours overseas), AFR (4 tours overseas), USAID/W (in AFR) and in the Administrator's Office as a Senior Advisor on the design of the Millennium Challenge Corporation - USAID Rep on Interagency Task Force. Anne Peterson Dr. E. Anne Peterson, MD, MPH, directs the delivery of more than $500 million in medical aid and relief supplies to more than 90 countries each year at Americare. Anne oversees Americares Emergency Response, U.S. Medical Assistance and Medical Outreach programs as well as all other ongoing global health programs. Anne served as assistant administrator of the U.S. Agency for International Development's Global Health Bureau. In that role, Anne oversaw of all of USAID's programs in maternal and child health, AIDS, family planning, neglected tropical diseases and health systems. 152 3.5. ANNEX: STRATEGY REFERENCE GROUP: FOCUSED DISCUSSION QUESTIONS, WITH ASSOCIATED COMMENTS GENERAL QUESTIONS: For evaluation providers, should these be the sole task or multi-tasking projects? o Consistent quality of evaluations requires in-house specialized technical expertise and rigor o Does providing other services, such as mission support, research support, or TA diminish the quality of evaluations? For evaluation providers, should projects provide a full range of evaluation types and methodologies? o Currently, most Evaluation projects tend to focus on limited types and methodologies o Is it preferable for Evaluation projects to focus on a narrow set of evaluation types (e.g. only performance evaluations) or possess the capacity to do a wide set of evaluation methodologies Which is preferred: multiple office/sector based service support projects or cross bureau service support projects? o If Cross Bureau, then more efficient, less confusing, less competition o If Office Based, larger number leading to client confusion, less efficient, but better focused on task at hand, more competition. OPERATIONAL QUESTIONS: Should service support projects have core funding? o Implications on speed and availability of resources for initial analyses o Implications for responsiveness to clients o Implications on quality of product o How to deal with perception of increased cost if no core funding provided? How to optimally use small business set asides? o Meets Congressional guidelines o Expands partners, including local partners/resources o May have limited ability to scale up, perform essential financial monitoring tasks What are the ideal staffing patterns for service support projects? o The proportion and expertise of permanent support staff o Are permanent technical staff preferred vs consultant technical staff Use of local solutions? o On the balance of local talent utilization required to achieve USAID strategic objectives and USAID external experienced staff required for efficiently addressing staff substitution function For service support projects, cooperative agreement vs contract? o CAs may be more flexible, more likely to engage USAID staff o Contract more likely to produce consistent high-quality product o Distinctions often blurred by different contract staff Use of broad agency announcement? o May provide more innovation and expand use of local resources o Quality may suffer o What is the balance between agency/situational context and innovative, methodological sharpness