Monitoring and Evaluation of Emergency Plan Progress II (MEEPP II) Performance Evaluation November 2016 This publication was produced at the request of the United States Agency for International Development. It was prepared independently by Rachel Jean-Baptiste, PhD, MPH, Oxford Epidemiology Services LLC, Siobhan Green MS, Sonjara Inc., Malik Jaffer, MPH, and Yusuf Hamba, MSc. Cover photo: A district biostatistician shows the evaluation team PEPFAR indicator data from the District Health Information System (DHIS) 2 in Uganda. Credit: Siobhan Green. Monitoring and Evaluation of Emergency Plan Progress II (MEEPP II) Performance Evaluation November 2016 USAID Contract No. AID-OAA-C-14-00067; Assignment 194 DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. MEEPP II EVALUATION iii This document is available online. Online documents can be located in the GH Pro website at http://ghpro.dexisonline.com/reports-publications. Documents are also made available through the Development Experience Clearinghouse (http://dec.usaid.gov). Additional information can be obtained from: Global Health Performance Cycle Improvement Project 1331 Pennsylvania Avenue NW, Suite 300 Washington, DC 20006 Phone: (202) 625-9444 Fax: (202) 517-9181 http://ghpro.dexisonline.com/reports-publications This document was submitted by GH Pro to the United States Agency for International Development under USAID Contract No. AID-OAA-C-14-00067. MEEPP II EVALUATION v ACKNOWLEDGMENTS This evaluation would not have been possible without the support, cooperation, and sharing of information and experiences, perceptions, and viewpoints of different stakeholders, who provided vital material for this report’s findings and conclusions. The team wishes to acknowledge a debt of gratitude to all those, including health facilities, districts, and national leaders, implementing partners of the United States Government (USG), and other partners of the Government of Uganda, who gave generously of their time and shared their thoughts, at times extensively and in great depth. Special thanks are due to the leadership and staff of MEEPP II for their continuous friendly support, flexibility, and practical help. The USG staff who support the PEPFAR work done by USAID/Uganda deserve special mention for their time and sharing of their insights about the role of MEEPP II with the evaluation team. In addition, we would like to thank Fiona Waata, Solome Sevume, and Rachel Kwezi of USAID/Uganda for their support in coordinating this activity. MEEPP II EVALUATION vii CONTENTS Acknowledgments............................................................................................................................... v ACRONYMS .......................................................................................................................................ix EXECUTIVE SUMMARY .................................................................................................................. xi Evaluation Purpose and Evaluation Questions...................................................................... xi Project Background .................................................................................................................... xi Evaluation Questions, Design, Methods, and Limitations .................................................. xi Findings, Conclusions, and Recommendations .................................................................... xii PEPFAR ........................................................................................................................................ xiii Government of Uganda ........................................................................................................... xiv I. Project Background ......................................................................................................................... 1 II. Evaluation Purpose and Questions ............................................................................................. 2 Evaluation Purpose ....................................................................................................................... 2 Evaluation Questions ................................................................................................................... 2 III. Evaluation Design, Methods, and Limitations ..........................................................................3 IV. Findings ............................................................................................................................................ 6 Evaluation Question 1: PEPFAR Reporting ............................................................................ 6 Evaluation Question 2: Extent of Transition to GoU M&E Systems ................................ 8 Evaluation Question 3: GoU Preparedness..........................................................................12 Evaluation Question 4: Data Quality and Data Usage .......................................................18 V. Conclusions and Recommendations ........................................................................................23 Conclusions (ALL) .....................................................................................................................23 Recommendations .....................................................................................................................24 Annex 1. Scope of Work ................................................................................................................27 Annex 2. Evaluation Methods and Limitations ...........................................................................57 Annex 3. Data Collection Instruments ........................................................................................85 Annex 4. Sources of Information ............................................................................................... 130 Annex 5: Disclosure of any Conflicts of Interest ................................................................... 149 viii MEEPP II EVALUATION TABLES Table 1. Data Collection Methods Employed for Each Evaluation Question .......................................3 Table 2. Government Preparedness ............................................................................................................ 13 Table 3. GoU and Stakeholder Perceptions of Government Preparedness on Scale of 1 (least prepared) to 10 ................................................................................................................................. 14 Table 4. Trainings Provided to IP M&E Advisors by MEEPP II (as reported by IPs) ........................ 20 Table 5. IP Perception of Impact of MEEPP II ............................................................................................ 21 Table 6. Recommendations for the GOU .................................................................................................. 26 FIGURES Figure 1.PEPFAR reporting along the continuum of the data lifecycle ...................................................7 Figure 2. Number of districts reporting at 100% in DHIS 2 .....................................................................9 Figure 3. Sustainability framework for government preparedness to ensure reliability of data from DHIS 2 ....................................................................................................................................... 17 Figure 4. Current state of the two MISs, based on the data lifecycle.................................................. 18 Figure 5: MEEPP-II Results Framework ....................................................................................................... 32 MEEPP II EVALUATION ix ACRONYMS ACP AIDS Control Program ADPs AIDS development partners AHA Anti-homosexuality Act AM Activity manager AOR Agreement Officer's Representative APR Annual program results ART Antiretroviral therapy CDC Centers for Disease Control and Prevention CDO Childcare development organization COP Country Operating Plan COR Contracting Officer's Representative CSF Civil Society Fund CSO Civil society organization DHIS District Health Information System DHO District Health Officer DoD Department of Defense DQA Data Quality Assessments EMR Electronic medical record F&A Finance and administration FGD Focus group discussion FY Fiscal year GHPro Global Health Program Cycle Improvement Project GoU Government of Uganda HIBRID HIV-Based Real Time Integrated Database HIV/AIDS Human immunodeficiency virus infection/Acquired immunodeficiency syndrome HMIS Health management information systems HSS Health systems strengthening IDI Infectious Disease Institute IPs Implementing partners IRCU Inter-Religious Council of Uganda IRs Intermediate results IT Information technology ITS Information technology systems KCCA Kampala Capital City Authority KII Key informant interview M & E Monitoring and evaluation x MEEPP II EVALUATION MEEPP Monitoring and Evaluation of the Emergency Plan Progress MEL Monitoring, evaluation, and learning MER Monitoring, Evaluation, and. Reporting MICT Ministry of Information and Communications Technology MIS Management information systems MMC Medical male circumcision MoGLSD Ministry of Gender, Labor, and Social Development MoH Ministry of Health MRC Medical Research Council MUSPH Makerere University School of Public Health NGIs Next Generation Indicators NGO Nongovernmental organization OGAC Office of the Global AIDS Coordinator OVC MIS Orphans and Vulnerable Children Management Information System OVC Orphans and vulnerable children PCO PEPFAR Coordinators Office PEPFAR Presidents Emergency Fund for AIDS Relief PEPPMIS Presidents Emergency Plan Performance Management Information System PMMP Performance Monitoring and Management Plan PMPs Program Performance Monitoring Plans PMTCT Prevention of mother-to-child transmission PRS Partner Reporting System RC Resource center SAPR Semi-annual program results SI Strategic information SOP Standard operating procedures SSS Social and Scientific Systems, Inc. TASO The AIDS Support Organization TWG Technical working group UAC Uganda AIDS Commission UNAIDS United Nations Programme on HIV and AIDS UNICEF United Nations Children's Fund USAID United States Agency for International Development USG United States Government WHO World Health Organization MEEPP II EVALUATION xi EXECUTIVE SUMMARY EVALUATION PURPOSE AND EVALUATION QUESTIONS GH Pro conducted an evaluation of the Monitoring and Evaluation of the Emergency Plan Progress (MEEPP II), a $17 million USAID-funded activity to support planning and reporting by the President’s Emergency Plan for AIDS Relief (PEPFAR) in Uganda. The purpose of this evaluation was to comprehensively analyze MEEPP II’s performance in supporting PEPFAR data and analytical needs and assess progress in the transition from reliance on a project-managed database to the Government of Uganda (GoU) systems for reporting PEPFAR interventions. The findings of this evaluation are expected to contribute to United States Government (USG) decisions on the level and type of support to further strengthen the GoU data system for PEPFAR reporting and other national needs. Specifically, the evaluation provides an opportunity for USG to identify gaps, including strengths and weaknesses, and gather evidence upon which a post-MEEPP strategy can be based. PROJECT BACKGROUND MEEPP II is a 5-year contract funded by PEPFAR through the United States Agency for International Development (USAID). The project is implemented by Social and Scientific Systems Inc. (SSS). While MEEPP I focused on developing a comprehensive PEPFAR implementing partner-based performance management system, MEEPP II was asked to shift its focus to strengthening existing national and district-level M&E systems to support the data needs of key HIV/AIDS-related stakeholders. The project was to have significantly contributed to strengthened national HIV/AIDS M&E systems that would allow the GoU, PEPFAR, the Global Fund, and other development partners to meet most of their data needs from these systems. EVALUATION QUESTIONS, DESIGN, METHODS, AND LIMITATIONS Using a cross-sectional study design, the following four questions were investigated: 1. PEPFAR Reporting. To what extent has MEEPP II efficiently, consistently, and routinely collected, analyzed, and reported data for PEPFAR planning, project management, and performance reporting? 2. Transition (MEEPP). To what extent have the data collection, data validation, analysis, and reporting systems established by MEEPP effectively transitioned to the national systems pivoted around DHIS 2 and OVC MIS? 3. Transition (GoU). To what extent is the GoU prepared to make the transition to becoming the primary source for PEPFAR data, and to what extent will the data be readily accessible by PEPFAR and other users? 4. Data Quality and Data Use. To what extent has MEEPP II contributed towards timely acquisition of quality data and use of data by health facilities, district health managers, the GoU, and USG implementing partners (IPs)? The evaluation team used several methods, including surveys that targeted all biostatisticians and district health officers in all 112 districts, all IPs, and key informant interviews and focus group discussions with the USG, GoU, and district and health facility staff during site visits, in addition to document reviews. While the design of the evaluation was relatively strong, a limitation is xii MEEPP II EVALUATION that data collection focused more on the District Health Information System (DHIS) 2 than on the Orphans and Vulnerable Children Management Information Systems (OVC MIS). FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS Findings Question 1: PEPFAR Reporting MEEPP II provided indispensable support to the USG for reporting and planning by the Office of the Global AIDS Coordinator (OGAC). The MEEPP II team successfully ensured regular, timely, and accurate data exchanges for PEPFAR planning and reporting both before and after DATIM. The MEEPP II team supported 6 USG/PEPFAR Country Operating Plan (COP) planning processes; 12 Semi-annual Program Results (SAPR) and Annual Program Results (APR) reports were compiled and submitted to PEPFAR/OGAC on time; and 4 HIV/AIDS burden tables were produced to assist in performance review and program planning for several COPs. In 2015, the MEEPP II team also started supporting quarterly reporting. Over 70 PEPFAR IPs were supported and trained in PEPFAR reporting requirements with 100% of USG prime partners receiving monthly feedback on data reported to the HIV-Based Real Time Integrated Database (HIBRID), DHIS 2, and OVC MIS. Further, 100% of all IPs have had reported data reviewed against targets and achievements. In addition, the MEEPP II team responded with speed to any special requests made by the USG team or the GoU for additional analyses or reports to help with decision making. Question 2: Transition to GoU M&E Systems MEEPP II’s transition of PEPFAR reliance for data to national systems is its greatest accomplishment. Significant efforts were made by the MEEPP II team to build GoU capacity to ensure data availability and reporting, and the team also made some progress toward sustainable capacity for the GoU to ensure data reliability, data analysis, or data use. The team found this to be the case for both the DHIS 2 and OVC MIS, though with the latter built-in checks that cannot be overridden have helped to ensure data reliability. In evaluating MEEPP II’s role over time, the team found that it evolved significantly from one focused on collecting data from partners to one focused on improving data availability in the DHIS 2 and OVC MIS so that these two GoU M&E systems could provide the data needed for PEPFAR reporting. This was accomplished for most requirements outlined in the PEPFAR next-generation indicators (NGIs), though there still remained indicator requirements captured only by partners especially those associated with community datasets and the revised Monitoring, Evaluation, and. Reporting (MER) indicators and entered into HIBRID, a PEPFAR-specific MEEPP-supported database. Question 3: Government Preparedness While overall ownership and a sense of responsibility for the data in the DHIS 2 and the OVC MIS were evident, key activities along the continuum of data are not led by the government. In addition, policies and other supportive infrastructure to ensure data reliability, validity, and analysis are in their infancy and still require significant technical assistance. The team concluded that, at this time, the GoU is prepared to make DHIS 2 data available to PEPFAR, but data validation and analysis are still areas that will need to be supported if the DHIS 2 data are to be used for PEPFAR reporting purposes. MEEPP II EVALUATION xiii Question 4: MEEPP II Contribution to Data Quality and Use The team concluded that MEEPP II contributed significantly to ensuring availability of quality data on HIV in Uganda. MEEPP has contributed to the timely acquisition of quality data for HIV in Uganda in several important ways: 1. Provision of a clean data set for PEFAR reports that can be used by the Ministry of Health (MoH), the Ministry of Gender, Labor, and Social Development (MoGLSD), USG, and other donors, setting the “gold standard” for clean data for Uganda. One challenge is that the clean dataset does not have an independently accessed repository, limiting the availability of the data for future use. 2. The number and timeliness of reports from districts has significantly improved since 2012, especially for HIV-related reports in the DHIS 2. 3. IPs reported significant support for PEPFAR reporting and PEPFAR indicators, with some support for other aspects of the data lifecycle. 4. With regard to data use, the team did not find explicit activities implemented or supported by MEEPP tied to data usage, although the districts and sites are using their data for making important management decisions. However, it should be noted that this was not an explicit responsibility of MEEPP II; other IPs are responsible for ensuring data reliability and use at district and health facility levels. For OVC MIS, data usage was more clearly attributed to MEEPP, due to MEEPP II-supported quarterly meetings, which greatly facilitated data analysis and usage at the district and sub-county level. MEEPP II’s main contribution to a data-driven learning environment and decision making within the Uganda HIV/AIDS program community is timely acquisition of quality data and producing the gold standard dataset. Conclusion MEEPP II provided indispensable support to the Uganda PEPFAR team for OGAC planning and reporting. Through diligent work, the program enabled the team to submit high-quality quarterly, semiannual, and annual reports to OGAC on time. It also served an integral role in helping to harmonize indicators between GoU M&E systems and PEPFAR. MEEPP II’s efforts greatly contributed to increasing the availability of data in GoU M&E systems, particularly the DHIS 2 and the OVC/MIS. However, although the GoU forms part of the team, MEEPP II staff continue to lead the bulk of data cleaning and validation activities, with heavy reliance on IPs for follow-up with districts and sites. Also not transitioned were ability to analyze or use the data from the DHIS 2 for decision making. As of now, the GoU is not prepared to ensure that the DHIS 2 is the primary source of data for PEPFAR reporting. To become prepared, there is a need to improve relevant policy, advocacy capacity, organizational infrastructure, physical infrastructure, staffing, and financial viability. Recommendations PEPFAR General xiv MEEPP II EVALUATION  Continue to focus on improving the quality and strategic use of GoU M&E systems following UNAIDS and PEPFAR’s Three Ones Policy: “one national plan, one national coordinating authority, and one national monitoring and evaluation system.” 1  Consider having staff embedded where the work needs to happen. This may be within the various USG offices responsible for PEPFAR reporting, unless that would introduce additional limitations, such as on flexibility of movement and physical infrastructure, that are difficult to surmount. Consider embedding staff within the GoU, in particular the MoH Resource Center (now the Division of Health Information), which is responsible for ensuring the quality of the DHIS 2, and with the MoGLSD, which is responsible for OVC MIS. Embedding staff within the GoU would likely increase efficiency and provide the opportunity for mentoring and shared learning in order to increase capacity within the GoU to improve all aspects of the data lifecycle. Interviews with a number of stakeholders, particularly donors and Ministry personnel, suggested that this same recommendation be part of an MEEPP II follow-on. Additionally, a document review revealed that this same recommendation was made in the MEEPP I evaluation.  Clarify roles and responsibilities delineated for all PEPFAR-funded M&E activities.  Hold IPs explicitly responsible and accountable for GoU M&E strengthening in districts and sites that they support.  Play an integral role in shaping the emerging legal and policy framework around e-data in partnership with the GoU to secure USG’s continued access to data from GoU M&E systems Follow-on Programming  Support planning and reporting needs of the USG Uganda PEPFAR Program, including preparation of COPs and quarterly and annual reporting. Specifically, new programming should continue with interagency coordination and facilitation as directed by the USG.  Ensure coordination between PEPFAR and the GoU and employ management responsive to the ever-changing USG and GoU reporting and planning needs.  As an explicit goal for MEEPP II follow-on, include a component on learning and data use that will explicitly focus on building data-driven decision making in Uganda’s HIV community. This activity would likely include knowledge management, communication, and collaboration and learning for PEPFAR, the GoU, and partners of the GoU.  Build additional software tools that can better assist with data validation and analysis of HIV data for decision making by all stakeholders GOVERNMENT OF UGANDA In order for GoU M&E systems to become reliable, trusted, and sustainable, the GoU should receive support from the USG and other partners to improve the policy environment for health data. Specific recommendations include elaboration of policies, particularly those for cross￾national data exchange and others listed in the eHealth Strategy Draft; production and housing 1 http://www.pepfar.gov/about/c19381.htm. MEEPP II EVALUATION xv of the final dataset accessible to PEPFAR and all other GoU partners at the national level; investments in a sustainable electronic medical record (EMR) system that feeds into the DHIS 2 summary reports; extending capacity for data into the DHIS 2 by health facilities; and increasing the focus on data use and outreach at all levels. MEEPP II EVALUATION 1 I. PROJECT BACKGROUND In 2010, USAID awarded a five-year contract to Social and Scientific Systems Inc. (SSS) to implement Monitoring and Evaluation of Emergency Plan Progress (MEEPP) II, the President’s Emergency Plan for AIDS Relief (PEPFAR) monitoring and evaluation (M&E) to support routine M&E of all PEPFAR-funded HIV/AIDS activities in Uganda. MEEPP II’s overall strategic result is to strengthen national HIV/AIDS management information systems (MIS) to accommodate PEPFAR data needs. MEEPP II has two major strategic objectives: 1. Improve PEPFAR data collection, management, analysis, and use in program planning—Country Operating Plans (COP) and reporting to the Office of the Global AIDS Coordinator (OGAC) and other United States Government (USG) agencies. 2. Strengthen Uganda’s national M&E reporting for selected key indicators in the national HIV/AIDS Strategic Plan’s Performance Monitoring and Management Plan (PMMP) The contract called for MEEPP II to work closely with various USG agencies—USAID, Centers for Disease Control and Prevention (CDC), State Department, and Department of Defense (DoD)—implementing partners (IPs), the Government of Uganda (GoU), and other HIV/AIDS development partners (ADPs) to provide technical assistance, training, and capacity building for M&E systems. Investments made under the MEEPP II contract were aimed at ensuring M&E capacity at national, regional, and service outlet levels for sustainability and to support accountability. To this effect MEEPP II was tasked with continuously analyzing data and providing feedback to the various stakeholders to influence program direction. In addition, it reviewed its own project implementation strategies and adjusted them to meet ever-changing needs. The MEEPP II Results Framework is outlined in Annex 1. In 2015, the MEEPP II contract was extended from September 2015 to May 2016. Under the modification, the two major objectives were clarified to include transition of the PEPFAR program performance management system to the national systems and to support GoU HIV/AIDS and orphans and vulnerable children (OVC) national monitoring systems in the context of the “three ones” principle. The MEEPP II team identified critical assumptions that needed to be enacted as a prerequisite to successful implementation that include the following:  PEPFAR indicators are harmonized with national indicators.  The USG instructs all its IPs to use national reporting tools, where applicable.  A web-enabled M&E reporting system is adopted and embraced as a national solution.  The GoU accepts inclusion of additional PEPFAR next-generation indicators (NGIs) into the web-enabled reporting system.  IPs commit adequate financial resources to ensure information technology (IT) infrastructure and technical support to districts in accordance with the set minimum standards  Alignment with the national reporting timeline and schedule.  District targets are integrated into the COP planning process. 2 MEEPP II EVALUATION II. EVALUATION PURPOSE AND QUESTIONS EVALUATION PURPOSE GH PRO was asked to evaluate the Monitoring and Evaluation of the Emergency Plan Progress (MEEPP II) project, a five-year contract implemented by SSS and funded by PEPFAR through USAID. While MEEPP I focused on developing a comprehensive PEPFAR IP-based performance management system, MEEPP II was asked to shift its focus to strengthening existing national and district-level M&E systems to support the data needs of key HIV/AIDS-related stakeholders. The project was to have significantly contributed to strengthening national HIV/AIDS M&E systems, allowing the GoU, PEPFAR, the Global Fund, and other development partners to meet most of their data needs from these systems. The purpose of this evaluation is to comprehensively analyze MEEPP II’s performance in supporting PEPFAR data and analytical needs and assess progress in the transition from reliance on a project-managed database to GoU systems for reporting PEPFAR interventions. The findings of this evaluation are expected to contribute to USG decisions about the level and type of support to further strengthen the GoU data system for PEPFAR reporting and other national needs. Specifically, the evaluation provides an opportunity for USG to identify gaps, including strengths and weaknesses, and gather evidence upon which a post-MEEPP strategy can be based. EVALUATION QUESTIONS 1. PEPFAR Reporting. To what extent has MEEPP II efficiently, consistently, and routinely collected, analyzed, and reported data for PEPFAR planning, project management, and performance reporting? 2. Transition (MEEPP II). To what extent have the data collection, data validation, analysis, and reporting systems established by MEEPP II effectively transitioned to the national systems pivoted around DHIS 2 and OVC MIS? 3. Transition (GoU). To what extent is the GoU prepared to make the transition to becoming the primary source for PEPFAR data, and to what extent will the data be readily accessible by PEPFAR and other users? 4. Data Quality and Data Use. To what extent has MEEPP II contributed to timely acquisition of quality data and use of data by health facilities, district health managers, the GoU, and USG IPs? The evaluation team pointed out the fact that some of the questions are not within the current scope of work of MEEPP II. This was particularly true for aspects of question 3, as it was not specifically within the scope of MEEPP II’s work to ensure GoU preparedness for the transition but simply to ensure that the transition took place. Similarly, for question 4, MEEPP II’s direct stakeholders were the national level GoU, USG, and USG IPs. MEEPP II did not have direct responsibility for ensuring timely acquisition of quality data and their use by health facilities or district health managers, though it was in a position to influence this through its work with IPs. However, USAID requested that the evaluation answer these questions so that USAID could gain insight into additional needs for a MEEPP II follow-on activity. The Contracting Officer’s Representative (COR) for MEEPP II also communicated a similar message to the MEEPP II team throughout the evaluation. MEEPP II EVALUATION 3 III. EVALUATION DESIGN, METHODS, AND LIMITATIONS Overall Evaluation Design: The evaluation team used a cross-sectional design for this evaluation, with mixed methods for data collection. They used a systems-strengthening approach to guide data collection, analysis, and reporting on evaluation questions 1 and 4 and Oxford Epi’s multi-dimensional Sustainability Framework to answer questions 2 and 3. These are fully described in the Evaluation Workplan (annex 2). Type of Evaluation: This was a performance evaluation. Summary of Stakeholder Engagement: Data were collected from all stakeholders, including MEEPP II staff and subpartner DataCares, the MEEPP II, USG technical working groups (TWGs), PEPFAR-funded IPs (M&E advisors and Chiefs of Party, the GoU, specifically the Ministry of Health (MoH), the Ministry of Gender, Labor, and Social Development (MoGLDS), and the Ministry of Information and Communications Technology (MICT), district biostatisticians and district health officers, and health facility staff. Sampling Strategy: A detailed sampling strategy is included in the Evaluation Workplan (annex 2). In brief, using software developed by Oxford Epi, the team selected all 112 districts for participation. In addition, the team conducted a “deep dive”’ into 8 districts—2 from each region selected for volume, as directed by USG. Convenience sampling was used to identify stakeholders for key informant interviews (KIIs) and focus group discussions (FGDs) stratified by stakeholder type. Data Collection Methods and Rationale Aligned: The team used multiple data collection methods to address each evaluation question. Summarized in Table 1, these included document reviews, FGDs, KIIs, surveys, and site visits with stakeholders from the MoH, the MoGLSD, and MICT, as well as the Uganda AIDS Commission (UAC), USG staff, IP staff, and staff of district and health facilities. Where data from one set of discussions seemed to indicate a finding, team members used subsequent data collection encounters to further validate, invalidate, or broaden/deepen understanding of the finding by discovering new perspectives. Team members debriefed each other daily and compared notes and findings. (A full description of data collection methods can be found in Annex 2.) Table 1. Data Collection Methods Employed for Each Evaluation Question Data Collection Method Evaluation Questions 1: PEPFAR Reporting 2: Transition￾MEEPP 3: Transition￾GOU 4: Data Document Review X X FGD: USG HIV Co-Chairs X X X FGD USG SI TWG X X X FGD USG OVC TWG X X X FGD M&E IP X X X X Survey COP X X X X Survey M&E IP X X X X 4 MEEPP II EVALUATION Data Collection Method Evaluation Questions Survey District Biostatisticians, District Health Officers (DHOs) District Planners X X X KIIs MoH X X X X KIIs MoGLSD X X X X KIIs UAC X X X X KIIs MICT X Visits: Districts X X X X Visits: Health Facilities X X Group Discussion: MEEPP II Team X X X X Group Discussion: DataCares X X X X Database Review (DHIS 2, OVC MIS, HIBRID) X X X X Ethical Considerations and Assurances: The team obtained informed consent from every participant and protected their identities in reporting the findings. The team obtained special additional written permission from participants whose pictures were taken or videoed. Deviations and Adjustments: The team was not able to finalize interviews from the OVC IP in Nakasongola District due to the time constraints of the team and the OVC partner, despite the team visiting the district twice. This did not, however, significantly affect the evaluation findings. Procedures used to ensure that the data are of highest achievable quality: The team developed the tools together, conducted training, and almost always collected qualitative data in pairs. Data collectors underwent two rounds of training on the web-based tool and built in checks to ensure the reliability of collected data were integrated into the web-based tool used by data collectors. Data Analysis Plan Analysis of Question 1: PEPFAR Reporting Data reviewed from MEEPP II-approved Program Performance Monitoring Plans (PMPs) were used to document PEPFAR planning and reporting activities. They were supplemented by data from a survey of IP M&E and staff, FGDs with members of the USG HIV TWG co-chairs and the Strategic Information (SI) and OVC TWGs, and interviews with the MEEPP II team. Analysis of Question 2: Transition KIIs with GoU officials, including representatives from the UAC, the AIDS Control Program (ACP), and the MoGLSD, and with the MEEPP II team provided the initial evidence for the transition. Grounding our analysis along the data lifecycle, notes were reviewed to identify exactly where within the lifecycle the work previously done by MEEPP was replaced by GoU entities. This was further triangulated with field visits to the field where we validated our understanding of the overall process. Our final analysis is coded in color along the continuum, with red being less than halfway, yellow between halfway and up to four-fifths done, and green meaning 80% or more completed across all levels of the GoU. MEEPP II EVALUATION 5 Analysis of Question 3: Government Preparedness Information on government preparedness was gathered in two ways. First, we asked participants in KIIs to rate how prepared they thought the GoU was to become the primary source of data for HIV/AIDs on a scale of 1 to 10. Subsequently, we asked participants to rate the GoU on specific aspects of preparedness using the Oxford Epi Sustainability Framework. Specifically, participants used a scale of 1 to 10 to rate GoU ownership, policy environment, infrastructure, advocacy capacity, staffing, financial viability, and trust. National GoU staff rated the national level, while district and health facility GoU staff rated at their respective levels. We calculated an overall score for government preparedness as the average of all scores. Where the data followed normal distribution, results were captured as the mean +/- standard deviation. Where the data did not follow normal distribution, results were captured as medians. In addition, average scores for each aspect of GoU preparedness were calculated and assigned a color (either red [0–4.9], yellow [5.0–7.9], or green [8.0–10.0]). These data were further triangulated with information obtained from document reviews and vetted through KIIs and/or FGDs. Thus, an aspect with an average score that made it red could also have some yellow to depict that some work was underway. Analysis of Question 4: Contribution of MEEP II to Data Quality and Data Use The evaluation team analyzed responses by M&E advisors and Chiefs of Party from 37 IPs on MEEPP II’s perceived impact on the continuum of data. These data were further triangulated by qualitative data on MEEPP II contribution from KIIs, FGDs, and document reviews. The team reviewed findings for data quality and data use separately. Part one of the question required us to define quality, which, as stated by the MoH, refers to data that are “reliable, timely, complete, and accurate.” This definition includes terms similar to those used by USAID, as guided by the quality standards found in ADS 201.3.5.8, though the ADS also emphasizes data validity, integrity, and precision as integral standards of data quality in addition to timeliness, completeness, and reliability. The team measured timely acquisition of quality data as (1) evidence of quality data available for use by PEPFAR and MoH; (2) the number of reports submitted on time to PEPFAR and through the MoH MIS system; and (3) IP ratings of the perceived impact of MEEPP II on data quality and data use. (4) The team also looked for artefacts of data quality improvement (such as standard operating procedures [SOPs], references to trainings, etc.) that originated from MEEPP II. The second part of the question focused on data usage by GoU, USG IPs, and facilities. The team asked district officials what they used data in the DHIS 2 or OVC MIS for, and to provide examples. The team also asked M&E officers and Chiefs of Party for PEPFAR IPs to rate, on a scale of 1-10, their perception of the impact of MEEPP II on data use. Limitations of the Design and Analytic Methods: The team notes that a disproportionate amount of data collection focused on the DHIS 2 compared to OVC MIS. (While both are GoU M&E systems, the OVC MIS is relatively new compared to the DHIS 2.) This is in part due to the fact that the team had access to more information about the DHIS 2, since MEEPP II has been working with it since 2012, compared to its more recent involvement with the OVC MIS. For example, given the limited timeframe, it was easier to include biostatisticians in the survey, since their phone numbers were known to the Resource Center, than sub-county officials, who are primarily working with OVC MIS. 6 MEEPP II EVALUATION IV. FINDINGS In total, data were collected from 342 people. Web-based and phone surveys reached 105 of 112 districts, interviewing 206 district health officers (DHOs) and biostatisticians. Forty-seven M&E officers and Chiefs of Party from 37 of 53 IPs responded to web-based surveys. In addition, the team conducted KIIs with a total of 93 stakeholders, including USG staff, GoU national officials, and district DHOs, biostatisticians, and staff of selected high￾volume health facilities in 8 districts (Annex 4). Informed consent was obtained for all participants. The team also reviewed 140 documents (Annex 4). EVALUATION QUESTION 1: PEPFAR REPORTING Findings Summary: MEEPP II provided indispensable support to the USG for OGAC reporting and planning. The MEEPP II team successfully ensured regular, timely, and accurate data exchanges with DATIM for PEPFAR planning and reporting. The MEEPP II team worked closely with all USG agencies and the PEPFAR Secretariat. The team supported 6 USG/PEPFAR COP planning processes; 12 SAPR and APR reports were compiled and submitted to PEPFAR/OGAC on time; and 4 HIV/AIDS burden tables were produced to assist in performance review and program planning for several COPs. Over 70 PEPFAR IPs were supported and trained in PEPFAR reporting requirements and 100% of USG prime partners received monthly feedback on data reported to HIBRID, DHIS 2, and OVC MIS. Further, 100% of all IPs have had reported data reviewed against targets and achievements. The evidence for this comes from results documented in the MEEPP II PMP and supported by KIIs and FGDs with USG staff. Reflecting on the dynamic nature and complexity of USG reporting to OGAC, one USG representative was quoted as saying “OGAC reporting is a massive quarterly headache. MEEPP has removed a lot of the pressure and does the work that USG cannot do.” MEEPP II EVALUATION 7 Data Availability and Validity When reviewing elements of the data lifecycle in Figure 1, the MEEPP II team was instrumental in ensuring availability (entry/compilation) of all PEPFAR data. As part of the data review process, MEEPP II staff led the verification and validation of reported data through IPs using various layers of checks, cleaning, and validation protocols that have improved over time. Initially, these checks were done manually and later in Excel using pivot tables and some automation, and hence were not as efficient as using a software model that automates data verification and validation checks. A feedback loop to IPs is also part of the process as well as monthly meetings with stakeholders. Because of this, and the fact that the DHIS 2 was only available to MEEPP II staff at night, a number of stakeholders, including USG representatives, commented on the workload of the MEEPP II staff, noting that they often work very long hours to produce the required PEPFAR reports. Thus, the evaluation team rated the MEEPP II team outstanding (green) for data availability, but also recognized that the lack of automation made the data validation process less efficient (yellow added). Data Analysis and Reporting Data analysis by MEEPP II staff produced special reports per ad hoc requests by USG TWGs. Referring to the MEEPP II team, one USG TWG member was quoted as saying, “We are overwhelmed. There is no way we could have managed the data. I get whatever I want from them.” Some of these analyses were used for USG program changes, such as partner realignment, as well as more routine tasks, such as COP planning and quarterly, SAPR, and APR reporting. MEEPP II played a significant role in supporting the USG team using DATIM to submit data to OGAC. MEEPP II invested time and energy in setting up systems and mapping to use the API Exchange function with DATIM. The use of the exchange is more efficient because of the time savings reaped over manual individual partner entry into DATIM. Review and approval protocols are still in place with partners and USG agencies. Data Use The MEEPP II team produced HIV burden tables. The USG team was able to use these burden tables to assist in HIV programming during the COP process. More specifically, the burden tables were one element used in assisting the USG team with the partner realignment/rationalization process. While MEEPP II produced a number of special reports, the evaluation team did not find any evidence of strategic dissemination of these reports. They were not visible in any of the offices of GoU officials interviewed. When USG staff were asked about these special reports, a common response was “I’ve never seen them.” Furthermore, these reports are perceived by USG staff to be meeting needs outside of PEPFAR or the GoU. Figure 1.PEPFAR reporting along the continuum of the data lifecycle 8 MEEPP II EVALUATION Lessons learned and best practices to inform future M&E capacity-building efforts supporting PEPFAR implementation in Uganda: An overall lesson learned is that the use of additional resources as provided by MEEPP II or future mechanisms is required by the USG to in order to meet with increasing demands for submitting planning and reporting documents to OGAC and for supporting internal USG processes. Any future mechanism will need to work extremely closely and in coordination with other M&E-focused partners in order to maximize efficiency and avoid possible duplication. Process inefficiencies led to greater demands on the time of staff. As a lesson learned, efficiency improvements, such as building additional software tools, streamlining data management and validation processes, and building more proactive and ad hoc analysis approaches and tools for HIV data for decision making by all stakeholders may free up many labor resources for the above additional tasks, especially if the GoU can take over much of the labor-intensive data validation work performed by MEEPP II. These processes will also assist the USG in negotiating and implementing future data requirement needs. The use of an electronic system versus a paper-based system is a best practice. This allows for greater efficiency and timeliness of the data lifecycle. In particular, the use of the GoU data systems versus a parallel reporting system is a best practice. MEEPP II continually documented the majority of its processes and translated these into SOPs. Although some SOPs have not been operationalized as yet, the development and use of SOPs are a best practice. Standardization and harmonization of HMIS tools and timelines is a best practice. Changes to DHIS 2 submission and review timelines requires continuous training. Mapping is essential for continued USG success using the DHIS 2 as a primary source of data and is a best practice. This includes mapping of PEPFAR indicators, mechanism IDs, site IDs, civil society organizations (CSOs), geocoding of sites, etc. The mapping is so successful that we heard from GoU staff that they turn to MEEPP for information on site data when government rezones for new districts. The IP M&E feedback loop is a best practice. This allows partners to track data against sites they support by program area and indicator. This process may be enhanced with partner access to the DHIS 2 for continual review rather than waiting for reports from MEEPP II. Use of the DATIM exchange is a best practice. This process is more efficient and timely than using individual IP entry approach. EVALUATION QUESTION 2: EXTENT OF TRANSITION TO GoU M&E SYSTEMS Findings Summary: MEEPP II transitioned sufficient capacity for data availability and reporting, though capacity for the GoU to ensure data reliability, data analysis, or data use is insufficient. The team found this to be the case for both the DHIS 2 and OVC MIS, though with the latter built-in checks that cannot be overridden have helped to ensure data reliability. MEEPP II EVALUATION 9 Evolution of MEEPP’s Role over Time MEEPP I used PEPMIS, a project management system developed internally to support PEPFAR reporting. IPs were taught to report directly to this system. MEEPP I staff would then initiate data validation processes to identify outliers and communicate with staff from the relevant IPs to resolve the issues. MEEPP II’s role was largely to produce the PEPFAR reports from this database, bypassing the government system. An FGD with members of the HIV TWG shed light on the implications of this vertical reporting process. Misalignment of timelines further exacerbated the issues: the GoU fiscal year runs from July to June, while that of PEPFAR runs from October to September. In addition, the DHIS 2 does not lock and data changes from day to day, resulting in duplication and in some cases double reporting. USG staff confirmed the difficulty that PEPFAR faced when planning or reporting based on numbers that were different from those of the GoU (for example, number of patients on antiretroviral therapy [ART]). To resolve these issues, the GoU and PEPFAR decided on a transition plan that would result in PEPFAR and the GoU using the same data source. MEEPP II created HIBRID, a system designed to bridge the transition between PEPFAR-based partner-driven M&E systems to GoU M&E systems. HIBRID is the repository for the following: 1. Sexual prevention/behavior change indicators that currently are not housed in any GoU M&E system 2. IP data on locations, program focus, and indicators to be tracked 3. Any emerging indicators not currently able to be tracked in GoU systems (such as MER 2.0 fine disaggregation). MEEPP II staff also catalyzed a number of other activities, including harmonization of PEPFAR indicators (mapping of the PEPFAR next generation indicators [NGIs], Monitoring and Evaluation Reporting 1.0 and 2.0) with GoU data to facilitate reporting. In 2013, the MEEPP II team began the process of transitioning from PEPMIS to DHIS 2 using HIBRID. Currently it is estimated that close to 80% of existing PEPFAR indicators are drawn from the DHIS 2 and OVC MIS, although this may change pending new indicators from PEPFAR. This software remains a key element in MEEPP II’s ability to manage and provide quality data to PEPFAR. PEPFAR Indicators and their Alignment with National Information Systems MEEPP II played a great role in both planning and implementing activities that helped in the transition of the source of PEPFAR reporting from its own databases to DHIS 2. The project worked closely with MoH and other stakeholders to harmonize the HMIS tools with PEPFAR Q… Q 0 50 100 Outp atien t ART Regis ter Inpat ient Over all Q4 2012 29 4 30 3.0 Q2 2016 63 79 65 46 … Figure 2. Number of districts reporting at 100% in DHIS 2 10 MEEPP II EVALUATION indicators so that PEPFAR quarterly, semiannual, and annual reports can be based on data from the DHIS 2. Interviews with USG staff confirmed that this has largely been accomplished. “Data for reporting on almost all PEPFAR indicators can now be obtained from DHIS 2 and OVC.” – USG Official There was a substantial increase in the number of reports submitted by districts to the DHIS 2. A USG official had this to say: “MEEPP’s support to DHIS 2 was its greatest achievement.” – USG “Political will has risen to appreciate OVCs.” – MoGSLD As Uganda receives PEPFAR funding to support the determined, resilient, empowered, AIDS￾free, mentored, and safe (DREAMS) activities of women, the reporting burden related to age disaggregation will be increasing as of October 2016. Changes to PEPFAR indicators and their definitions will require realignment with the GoU systems if these systems are to remain central to PEPFAR data reporting. With the OVC MIS, MEEPP II’s sub-partner DataCares strengthened the technology of the OVC MIS system, following the technical lead of MoGSLD officials. Findings from KIIs and FGDs revealed that the MoGSLD staff very much feel that the OVC MIS is their system and are quite pleased to have a better handle on the number of CSOs providing OVC services throughout the country. This again points to improvements in data collection, which effectively transitioned from a MEEPP II-owned system to the OVC MIS. The findings from KIIs and FGDs revealed that indeed all the indicators per OVC Core Program Area can be obtained on the OVC MIS and the districts are able to generate reports on OVC interventions. In the DHIS 2, the greatest improvement was seen in the ART reports, although other reports also improved (Figure 2). The OVC MIS also saw similar improvements in the number of service providers reporting. The PEPFAR report relies also on data pulled from the OVC MIS. These data are considered the gold standard for OVC data in Uganda. Timely submission of data depends on a number of factors, such as the availability of tools and staff skills to document the reports. The evaluation team found that MEEPP II played a significant role by ensuring distribution of HMIS tools to districts and sites. Overall, KIIs and FGDs observed that MEEPP performed very well in improving data collection at the national level through the coordination mechanism that accelerated stakeholder participation in supporting both growth and use of DHIS 2 and OVC MIS in Uganda. However, this was the extent of the transition, and the extent to which this would continue outside of partner support is questionable. Paths for Processes that Are Still in Transition: Data Validation After downloading the data from the DHIS 2, MEEPP II staff begin a thorough process of data validation that starts with analysis of the completeness of the data reported This is a consistent, repeatable process, with the same steps followed each time, and leads to the production of the gold standard database. While staff from the GoU participated in this activity, MEEPP II and the GoU agreed that this is an activity largely led by MEEPP II staff. MEEPP II did develop SOPs for data validation together with national GoU officials, and these are being integrated into district trainings and quarterly review meetings. However, at the national level, key informants revealed MEEPP II EVALUATION 11 several challenges at the MoH Resource Center that may have prohibited capacity transfer. One IP respondent put it succinctly “The Ministry of Health Resource Center is besieged with a number of challenges among which [are] low staffing and insufficient facilitation from the national purse to oversee the task at hand.” – Chief of Party, USG Implementing Partner The team did find that the GoU applies some of the steps. For example, the first step is to verify that data came in from all sites (completeness), and this is largely done; 92.3% of district officials from 105 districts surveyed said they do this. In that same survey, 71% said, yes, there were SOPs. In the survey, among those who said there were SOPs available, only 24.9% said they followed them very closely. However, at the 8 districts we visited, we did not find a single copy of the SOP for data cleaning and validation. The team’s interviews with district officials supported these findings. One district biostatistician told the team: “We don’t have written SOPs for the DHIS 2 but we know the processes and we share in meetings.”– KII with District Official. Inquiries into how data validation is done also varied from one district to another. Some biostatisticians interpreted data validity as double-checking data entry into the DHIS 2: “We select random indicators, and compare the data entered in the DHIS II with hard copies of the reports at the office level.” – Survey of District Officials. Others view it as a cross-checking exercise between what the sites report to the district and the data contained in the site registers. However, methods for doing this varied by district, with almost no two districts doing the exact same thing. For example, one district said: “HMIS 105 is brought, biostatistician looks at it to see realisticness, and the district health team carries out data quality assessments at health facilities.” – Survey of District Biostatisticians Another district described the process as follows: “The biostatistician is facilitated to go to the health facility to check and count the register physically and in case of any error, it is then corrected before the data is entered in the DHIS II.” – Survey of District Biostatisticians In some districts, a sampling strategy is used to select sites for which the data will be cleaned. In other districts, data cleaning may be done for all sites but with specific focus on key indicators and more in-depth focus on HIV indicators. “Data cleaning is done for all sites but at HC 2 we concentrate on malaria and FP. The other 20 sites work on HIV, TB, MCH indicators. When we conduct support supervision we cross check what was reported with what’s found. Support supervision visits are done quarterly, and the visits are to the 20 sites that provide HIV services.” – KII with District Health Officer MEEPP II recently created Excel-based tools designed to automate the logic checks and various other steps in the data validation process for PEPFAR indicators. Because they minimize manual manipulation, these tools could make it easier to transition data validation to the GoU. These tools were being finalized while the evaluation was taking place, and the team did not get a clear timeline for MEEPP II transition of these tools to the GoU. 12 MEEPP II EVALUATION Data Analysis The team found little evidence at national or district level for MEEPP II transitioning of capacity for data analysis. The DHIS 2 as well as the OVC MIS databases all produced some preprogrammed analyses and graphs. Biostatisticians at districts can, and often do, run additional analyses to answer specific questions as requested by the DHO. However, these are not linked to MEEPP II capacity building since biostatisticians receive academic training in data analysis. At the national level, the evaluation team did not find any evidence of data analysis conducted by national staff. KIIs with MoH officials revealed that when they have a need for data analysis, they make requests to MEEPP II: “We either request MEEPP II to do this together with the MoH, or MEEP II does it for the MoH.” – MoH Official. The team did not find any evidence of plans for transition in this area. However, this may be an internal issue, resolvable by having additional qualified staff at the MoH Resource Center. Data Use MEEPP II did not have a strategic plan for data use and did not create demand for use of data outside of PEPFAR reporting among IPs and at the national level. The team did not find evidence that MEEPP II transitioned its capacity for data use to the GoU, for data from either the DHIS 2 or OVC MIS. The team did not find any indication of transition in this area. EVALUATION QUESTION 3: GoU PREPAREDNESS Finding Summary: While overall ownership and a sense of responsibility for the data in the DHIS 2 and the OVC MIS was evident, key activities along the continuum of data are not led by the government. In addition, policies and other supportive infrastructure to ensure data reliability, validity, and analysis are in their infancy and still require significant technical assistance. The team concluded that, at this time, the Government of Uganda is prepared to make DHIS 2 data available to PEPFAR, but data validation and analysis are still areas that will need to be supported if the DHIS 2 data are to be used for PEPFAR reporting purposes. To what extent is the GoU prepared to make the transition into becoming the primary source for PEPFAR data? In evaluating this question, we first sought to establish the importance of the DHIS 2 and OVC MIS for Uganda. Stakeholders confirmed that indeed both of these management information systems (MIS) were meeting a critical need of the Government of Uganda. Among the eight districts we visited, all of the biostatisticians and the DHOs interviewed mentioned that the DHIS 2 was an important management and decision-making tool for them. One DHO succinctly described the importance of the DHIS 2: “It is important in that it gives us the information about performance of our facilities, which information is used to improve health services delivery. It helps us to plan, to prioritize, because we look at where we are not doing well and prioritize. It helps us lobby for funding. It helps us disseminate the right information to technical political and community people.” – District Health Officer, Government of Uganda Finding: Overall Preparedness of the GoU MEEPP II EVALUATION 13 When asked the single question “On a scale of 1 to 10, how prepared is the Government of Uganda to become the primary source of data for HIV/AIDS for PEPFAR and other donors?”, the overall perception, as shared by all stakeholders including the GoU, is that the government is approximately halfway ready to become the primary source of HIV/AIDS data. Average responses were relatively comparable to the average summary responses about specific dimensions of preparedness (Table 2) as analyzed using the Oxford Epi Sustainability Framework. For example, we noted very few differences between average scores using either method for districts or IPs, and national GoU scores did not differ widely between the two methods. In order to better understand what could be improved, specific dimensions of preparedness, namely GoU ownership, policy environment, advocacy capacity, staffing (capacity and numbers), physical infrastructure, organizational infrastructure, and financial viability were analyzed. Table 2. Government Preparedness Finding: GoU Ownership The evaluation team found very high perceptions of government ownership of the DHIS 2 and OVC/MIS among all stakeholders (Table 3). Median scores for ownership at the national and district levels were 8.5, at health facilities 8.8, among USG IPs 8.0, and among donors 9.5. KIIs with the ministries further underscored this perception of strong ownership, as summarized in these quotes: Average Score (1-10) (calculated across dimensions) Average Score (1-10) (spontaneous response) National 4.5 (Median) 5.4 +/- 1.2 District 5.5 +/- 2.2 5.5 +/- 2.6 Health Facility 6.4 +/- 1.3 6.1 (median) 3.5 +/- 2.2 4.0 (Median) USG Implementing Partners 4.9 +/- 2.1 4.8 +/- 2.4 Other Donors 6.0 +/- 1.3 3.8 (Median) 14 MEEPP II EVALUATION Table 3. GoU and Stakeholder Perceptions of Government Preparedness on Scale of 1 (least prepared) to 10 “The OVC/MIS system was built entirely on Government guidelines.” – Staff member, MoGLSD “OVC used to be viewed as PEPFAR but now with OVC MIS, MEEPP has put the Government in the lead. We are now able to include all partners, even non-PEPFAR. The tools are Ministry, not MEEPP. And the political will has risen to appreciate OVCs at national and local levels.” – Staff Member, MoGLSD. However, the evaluation team also found that three of the eight districts visited expressed an indication of lack of complete ownership. Discussions were particularly around districts having more autonomy to make changes to the database, including entering or removing constituents as deemed necessary. This capacity was previously available to districts but was removed by the national level after it was abused repeatedly in attempts to demonstrate greater completeness of reporting (KIIs with MEEPP II, District site visits). Using the Oxford Epi Sustainability Framework, the evaluation team further analyzed each domain of government preparedness and colored each domain red (0-4.9), yellow (5.0-7.9), and green (8.0-10.0) based on the findings (Table 3). The percentages were derived by asking respondents to answer 17 questions on a scale of 1 to 10 and obtaining an average score for each answer. Several similar questions were combined to create a domain, and their scores were averaged. Finding: GoU Advocacy Capacity District and national GoU staff perceived their capacity to advocate successfully to be less than half, per results from surveys and KIIs. The evaluation team found that while a significant amount of advocacy was taking place both at national and district levels using the data in the DHIS, most districts have not yet successfully advocated for additional funding in their budget based on data. At the national level, we did learn that due to advocacy by the Resource Center and others, there has been a reorganization of the Resource Center, now upgraded to the Division of Health Information, led by a commissioner. We further learned that there has been an increase in staff allocation, and for the first time a line item for the purchase of HMIS tools was included in the national health budget for 1 billion Uganda Shillings (approximately US$300,000), though Domain National District Site IP Donors Ownership 8.5 8.5 8.8 8.0 9.5 Policy 3.5 2.3 8.6 7.0 10 Advocacy Capacity 3.0 4.0 5.5 5.3 5.8 Staffing (number and capacity) 3.0 6.5 8.1 4.5 4.0 Physical Infrastructure 5.4 5.2 8.1 5.3 6.1 Organizational Infrastructure 3.0 5.0 5.8 4.7 4.0 Financial Viability 2.0 3.0 4.0 2.8 1.7 MEEPP II EVALUATION 15 this is far from what is needed annually ($1.8 million to $2.5 million). Thus the team considered the perceptions of low capacity for advocacy (3.0 on a scale of 1 to 10, red), with the reality that the MoH did advocate and received 12% to 16% of its required needs. This raised the score from 3.0 to, say, 4.6 (generously speaking), and though not enough to warrant a full change of color from red to yellow, the team concluded that the red could be modified to include some yellow to better recognize the capacity that does exist. Finding: Staffing gaps for data personnel differ by levels and are perceived to be greatest at the national level compared to district or site levels. At the national level, staffing gaps for the Resource Center are great. Not all of the funded positions are staffed, with only two2 MoH￾funded positions with people in them. Though other positions are funded, they are filled by seconded individuals supported by partners. The evaluation team heard requests from GoU for donor-funded staff to be seconded or co-located with GoU in order to facilitate communications, efficiency of work, and mentoring, and as a tangible physical symbol of support to the GoU. At the district level, however, the addition of biostatisticians at each district has helped to create a perception of sufficiency. However, a biostatistician cannot ensure the reliability and validity of the data from the district alone. Some districts also have an HMIS focal person and data entry clerks. These types of people are not uniformly available in all districts. Health facilities in the 8 districts visited said they had enough staff to focus on data. We visited only high-volume sites, and they had dedicated records clerks whose job was to create the DHIS 2 reports. In some instances, these staff are supported by partners. Sites did, however, say they do not have enough clinical staff for the patient load. Finding: The GoU is prepared to provide a significant proportion of the required physical infrastructure needed for the performance of the DHIS 2. This includes office space, desks, and computers. However, there are noticeable challenges, including maintenance and updates of computers and irregular supply of electricity, as well as lack of a sustainable solution for uninterrupted access to HMIS paper tools. The DHIS 2 is Internet-dependent, yet district offices have irregular access to the Internet. “Sometimes it [DHIS 2] goes off when you are in the middle of entering a site report. You have to then wait for days before you can go back. This is disturbing us.” – District Biostatistician By making these additional reflections, the evaluation team was able to introduce additional data (GoU provision of physical infrastructure, for example) to further support or add to the perceptions shared by the interviewers, thus enriching the analysis. Finding: GoU Organizational Infrastructure This domain dealt with documentation on data processes, plans for staff training and retention, communication with MoH, data validation processes, TWGs, coordination, and other activities to ensure the reliability of DHIS 2 data. The evaluation team found that significant additional support to the GoU was needed in this domain. The discussion below focuses specifically on data completeness and validation processes. We should also note that we did not find any TWG s for data, though biostatisticians in the Western Region have decided to create the Southwestern Association of Biostatisticians, with support and patronage from the Resource Center. Their plan is to engage as a group and provide technical support to any of their districts as the need arises. They meet using available opportunities, such as regional workshops where they are all invited, and they discuss issues specific to their districts and obtain support. Nevertheless, organizational capacity is still in its infancy. In addition, the team did not find 16 MEEPP II EVALUATION evidence of rewards for doing data well, or sanctions for inconsistencies, outside of the League Tables. However, the League Tables reward districts largely on data completeness, which does not affect the rest of the data continuum. The team concluded that there is still a significant amount of work required to standardize processes to make DHIS 2 data reliably available to PEPFAR from the GoU M&E system, and retained the color red. Finding: GoU Financial Viability National MoH officials and donors on average had the lowest perception of the GoU’s financial capacity to ensure reliable data from the DHIS 2 (average of 2.0 and 1.7 respectively, on a scale of 1 to 10). The GoU is still not budgeting for maintenance and upkeep of DHIS 2 and for the continual learning of its users. However, upon closer analysis, the team realized that the GoU already makes a number of important financial contributions that provide invaluable support to DHIS 2. For example, the GoU pays for the buildings that house the office space for the district and for the salaries of key staff (including the DHO and the HMIS focal person for the district), and increasingly of the biostatistician. In taking these factors into consideration, the evaluation team added a bit of yellow to the domain for financial sustainability. To what extent will the data be readily accessible by PEPFAR and other users? Finding: Access to DHIS 2 Interviews with the MoH Resource Center revealed that there are set procedures for requesting access to the DHIS 2. The evaluation team obtained access to the DHIS 2 using these rules and found them sufficient. However, there were significant delays in obtaining the initial access, which meant analysis of the DHIS 2 was performed during field visits, compressing the time for analysis and limiting its influence on data collection tools. Finding: Access to OVC MIS All of the data for the OVC MIS is readily accessible online by anyone with Internet access. The evaluation team concluded that data from the OVC MIS will be readily available to the USG. Finding: Policy Environment At the moment, USG has ready access to data from the DHIS 2 and OVC MIS. However, the evaluation team was not able to identify what guarantees this right, either by bilateral relations, law, or a policy of data exchange between the two countries. The evaluation team further investigated the extent to which the policy and legislative environment would enable, or serve as a barrier to, USG having ready access to the data in the DHIS 2 or OVC MIS. Review of the Data Protection and Privacy Bill currently in the Uganda Parliament revealed a provision for data exchange between countries, namely, Article 15 Processing personal data outside Uganda: Where a data processor or data controller processes personal data outside Uganda, the data processor or data controller shall ensure that the country in which the data is processed has adequate measures in place for the protection of the personal data which are at least equivalent to the protection provided by this Act. – Data Protection and Privacy Act (being reviewed), article 15. The bill is currently tabled as the GoU creates a new Parliament. The MICT anticipates the passing of this bill in the new Parliament (KII, MICT official). However, there will then be a need for the development of regulatory frameworks and policies to facilitate and govern data MEEPP II EVALUATION 17 exchange between countries including the United States, as delineated in the eHealth Draft Strategy (KII, MoH official). Given that policies are in the making (as opposed to being nonexistent), the evaluation team concluded that data from the DHIS and OVC MIS will be available to the USG as needed for planning and reporting on PEPFAR activity. Nevertheless, the evaluation team also noted a need for USG involvement in further shaping the legal environment and policies to facilitate data availability. While current laws protect personal data collected, they still do not speak to specific data collected from health facilities and stored digitally within the DHIS 2 and OVC MIS. In a KII with the MoH Resource Center, the evaluation team received a copy of the eHealth Strategy currently being reviewed by stakeholders. This strategy specifically notes the lack of a legal framework to govern the implementation of the growing availability of e-HMIS in Uganda. Finding: Public Image and Trust in the Data Central to all of this is the idea of public image. The team asked districts surveyed to rate their level of trust in the DHIS 2 data, and average ratings were 8.4 +/- 1.4. District officials also rated their perception of the level of trust that the public (anyone outside the district) placed in the DHIS 2 data, and their average rating was 7.6 +/- 2.0. Average ratings of trust in the DHIS 2 data were 6.9 +/- 2.2 for IPs (M&E advisors), 7.0 (median score) for national GoU officials, and 6.0 +/- 2.8 for donors. In all cases, recognition of the need for improvement is there, though for different reasons. District officials note that the public, particularly patients, trust them since they do not have any alternative (KIIs with district officials). One donor noted, however, that given that the DHIS 2 is only four years old, the public does not yet know about it yet, and cannot have a high level of trust. Figure 3 summarizes the final findings of the evaluation team regarding GoU preparedness. Using the domains as building blocks, there is a more urgent need to focus on policy, advocacy capacity, organizational infrastructure, and financial viability, though some focus on physical infrastructure and staffing is still required. Government preparedness of the DHIS 2 does not require additional focus on GoU ownership, but it will be important to continue to build trust in the DHIS 2 data so that it becomes increasingly used by others. Lessons drawn from the transition process for obtaining PEPFAR data for USAID, post￾MEEPP II: The evaluation team notes several lessons learned from this transition to GoU M&E systems. First, MEEPP II staff had a limited interpretation of “transitioned.” Instead of looking at the entire continuum of data, they focused primarily on ensuring that data for PEPFAR reporting would be available through the DHIS 2. This limited interpretation might have hindered their ability and Figure 3. Sustainability framework for government preparedness to ensure reliability of data from DHIS 2 18 MEEPP II EVALUATION vision to support the MoH in strengthening its capacity along other aspects of the data continuum, namely data validity, data analysis, and data use. Second, a related lesson learned was that the time spent on the transition might have been disproportional to the time spent on PEPFAR reporting, thus not giving MEEPP II an opportunity to complete the transition. This was vetted by a KII with USG staff quoted as saying “we kept them too busy responding to OGAC.” USG staff also expressed the sentiment that they should be partly blamed for the lack of focus on the other aspects of the data continuum: MEEPP II is a contract and necessarily followed the direction given to them by the contractor. A third lesson learned is that ensuring reliable data for PEPFAR reporting is a complex, multi-step process, and often the coordination that MEEPP II does behind the scenes is overlooked and its importance underestimated. If not taken into consideration, this may further hinder the transition. EVALUATION QUESTION 4: DATA QUALITY AND DATA USAGE Findings Summary: The team concluded that MEEPP II contributed significantly to ensuring the availability of quality data on HIV in Uganda. MEEPP has contributed to the timely acquisition of quality data for HIV in Uganda in several important ways: 1. Provision of a clean dataset for PEFAR reports that can be used by MoH, MoGLSD, USG, and other donors, setting the gold standard for clean data for Uganda. One challenge is that the clean dataset does not have an independently accessed repository, limiting the availability of the data for future usage. 2. Number and timeliness of reports from districts significantly improved since 2012, especially for HIV-related reports in the DHIS 2. 3. Significant support reported by IPs for PEPFAR reporting and PEPFAR indicators, with some support for other aspects of the data lifecycle. 4. With regard to data use, no explicit activities found implemented or supported by MEEPP that were tied to data usage, although the districts and sites are using their data for making important management decisions. For OVC MIS, data usage was much more clearly attributed to MEEPP due to the quarterly meetings, which seemed to be a major source of data analysis and usage at the district and sub-county level. Finding: MEEPP II Contributed to the Timely Acquisition of Quality Data As discussed earlier, MEEPP II was primarily focused on timely data collection and validation for PEPFAR and its IPs. This focus greatly contributed to the fact that currently health facility data are available and reported through DHIS 2, and OVC MIS data are available in the OVC MIS (Figure 4). Other areas of the data continuum still need to be improved. Through a review of project documents and interviews with MEEPP, MoH, MoGLSD, and IP stakeholders, the team learned that for DHIS 2, the generation of the League Tables, which provide MoH with an easy way to measure performance in submitting data and 0%-49% 50%-79% 80%-100% Figure 4. Current state of the two MISs, based on the data lifecycle MEEPP II EVALUATION 19 helping pinpoint problematic districts, helped to drive on-time reporting. For OVC MIS, regional quarterly review meetings with districts, as well as built-in completeness reports in the software, helped. Additionally, service organizations are able to enter their reports directly into the OVC MIS II. However, a major challenge identified by the team is that not all providers of OVC services have been identified, and districts suspect significant underreporting. The definition of quality, to quote MoH, refers to data that are “reliable, timely, complete, and accurate.” The team interprets this as the first two steps—data availability and data validation— on the system-strengthening assessment. Every month, MEEPP II staff pull data from the DHIS 2 and create a final dataset that is viewed by the MoH and other stakeholders, including other donors, as the gold standard. To create the gold standard dataset, MEEPP performed many tasks behind the scenes to make sure the data to be reported to PEPFAR are reliable and valid. Specifically, the MEEPP team worked with the MoH to create logic checks and validation processes that meet both PEFAR and MoH needs. Up until the evaluation, these checks were largely manual, though the team did see an Excel spreadsheet prototype that will automate the processes (published just as the evaluation finished). One challenge is that the clean dataset does not have an independently accessed repository, limiting the availability of the data for future use. Development of Data Quality Infrastructure MEEPP supported MoH development of SOPs and printed registers. Many of these SOPs and registers were in evidence in district offices, and many for data collection seemed to be well￾understood and followed. However, the same was not the case for SOPs on data validation and cleaning; while MEEPP had detailed data quality assessment (DQA) for PEPFAR data, it is exclusively focused on “USG Cognizant/Agreement Officer’s representatives, activity and project managers, Strategic Information Team, USG M&E contractors, and USG IPs.”2 The team did not find a single SOP on data validation in the 8 districts visited. When asked if they had SOPs for data reporting, 86.3% of biostatisticians surveyed from 105 districts said yes. However, when asked how closely they follow them, only 24.2% said “very closely.” In addition to SOPs, the team discovered many behind-the-scenes activities that help MEEPP II to generate quality data, such as mapping of all facilities, harmonization of the PEPFAR indicators with DHIS 2 definitions, etc. MEEPP II support to the MoGLSD started less than two years ago, after a year-long hiatus in support of the OVC MIS. The team found that a significant amount of work has been done by MEEPP II and its subcontractor DataCares. IPs and probation officers interviewed reported substantial contributions from MEEPP to improve data quality. Most cited the quarterly regional meetings, which at least two to three MEEPP/DataCares employees or consultants attend with district and sub-county GoU staff and CSOs to review data submitted to the OVC MIS. During these quarterly meetings, data are reviewed for accuracy as well as analyzed for impact, gaps, and trends. Formal trainings and informal technical support is provided to CSOs and childcare development organizations (CDOs) who attend, as well as technical support for computer software. 2 Standard Operating Procedure #2, Conducting Data Quality Assessment and Improvement for PEPFAR Indicators, page 7, MEEPP II 2015. 20 MEEPP II EVALUATION Support to PEPFAR IPs Through KIIs, FGDs, and web surveys of IPs, the team learned that MEEPP II regularly provided IPs with feedback on the quality of submissions (incomplete or inaccurate data), so that the IPs can target districts and facilities that have the most need of help. MEEPP also provided a monitoring, evaluation, and learning self-assessment tool to IPs to help guide their self￾improvements. However, survey results show that IPs reported that the most support from MEEPP II was on PEPFAR Indicators and PEPFAR/data reporting, followed by DHIS 2 and data collection, and less on cleaning, analysis, or usage. These findings also demonstrate that MEEPP II tended to focus more on data collection and less on capacity building around validation/cleaning, analysis, and usage. Finding: MEEPP II’s Contribution to the Use of Quality Data Table 4. Trainings Provided to IP M&E Advisors by MEEPP II (as reported by IPs) The evaluation team found mixed results in this area. First, MEEPP II did not have an explicit mandate to ensure data use and, second, its operational mandate was to work with national GoU and with IPs for PEPFAR reporting. The team found scarce evidence of MEEPP II contributing to data use by the national GoU, and the GoU admitted to this as a major internal weakness: “Converting data to usable products is still a weakness.” – KII with GoU Official There was only one example of data use by the GoU supported by MEEPP II. When developing the new guidelines for pediatric HIV/AIDS, a KII with a donor revealed that data from MEEPP II was heavily relied upon. Within PEPFAR, USG staff did say they use HIV burden tables, though modified from what was produced by MEEPP II. The team asked questions at site and district levels to measure the existing level of usage and then attempted to tie the usage back to MEEPP. For DHIS 2, since all the work MEEPP performed at the district level was via IPs, the team found it difficult to trace any particular example of data usage back to MEEPP. In addition, the team found IP support for data use inconsistent; some were excellent (even looking to publish data in peer-reviewed journals), and others seemed Trainings % IPs who say YES (N = 37) PEPFAR indicators 92.1% DHIS 2 60.5% Data collection 57.9% Data cleaning 44.7% Data analysis 42.1% PEPFAR reporting 60.5% Figure 5: Site with a quality improvement program that was implemented by the MoH MEEPP II EVALUATION 21 to not prioritize usage among the sites they support. For OVC MIS, data usage was much more clearly attributable to MEEPP due to its quarterly meetings, which seemed to be a major source of data analysis and usage at the district and sub-county level. However, at the district and site levels, the team found a great deal of evidence of data usage, such as for performance management (against targets, time series), mapping gaps in service delivery (example: midwives vs. facility delivery rates), advocacy (for more resources, support), and planning (especially HR, supplies/consumables). Some of the challenges found included inconsistent use of data—data used for advocacy but not effective at changing budgets or not enough policy impact. Sites and districts involved in quality improvement seemed to have stronger data usage processes and tracking. USG IPs also had a range of data usage abilities. Some were experts already, using data for research as well as planning, resource allocation, advocacy, and performance monitoring. Others were not as sophisticated in their use of data. While MEEPP II did have a monitoring, evaluation, and learning (MEL) assessment tool developed for IPs, no evidence of its usage or impact was discovered by evaluators. Perceptions from IPs also demonstrate that MEEPP II largely focused on strengthening their capacity for PEPFAR reporting and less on data use (Table 5). Among COPs, only 45.5% reported that MEEPP II trained their staff on data use, compared to 100% who reported that MEEPP II trained their staff for PEPFAR data reporting. These results were similar among M&E advisors, who rated training in data use similarly low (43.8%). Furthermore, IPs perceived MEEPP II’s impact along the continuum of data to be lowest on data use (6.97 +/- 1.6; Table 5). The team found that MEEPP II had made minimal contributions to the use of data by health facilities, the GoU, and USG IPs. Its data use was focused almost exclusively on USG to produce formal PEPFAR reports and special reports, but there was no communications or dissemination strategy for these reports so that the reports could effect change. Other data use was incidental to data collection and validation processes. Finding: MEEPP II’s Contributions to Data-Driven Learning and Decision Making The final question focused on MEEPP’s contribution to a data-driven learning and decision￾making environment in Uganda. The team discovered that, outside of improving data quality and availability (a necessary step before data use and data- driven decision making can occur), MEEPP II was more reactive than proactive. The MEEPP team mentioned they had no formal communications or dissemination strategy for the data they collected or the special reports, and their workplans and PMP reinforced this. Some examples of MEEPP contributions to data-driven decision making include OVC MIS regional review meetings facilitated by MEEPP II, anecdotal reports of gap analysis by district/region, performance targeting and improvement, and discussion between CSOs on Table 5. IP Perception of Impact of MEEPP II Activity Mean +/- SD (on scale of 1 to 10) Data collection 7.76 +/- 1.6 Data cleaning 7.67 +/- 1.2 Data analysis 7.60 +/- 1.7 PEPFAR reporting 8.38 +/- 1.3 Data use 6.97 +/- 1.6 22 MEEPP II EVALUATION service provision. All key informants who had attended these regional review meetings mentioned their value in helping them by providing a better sense of what was going on in their district or region to support OVC. Several probation officers mentioned they found them useful to perform gap analysis and identify advocacy and resource mobilization opportunities within their districts and communities. However, the team wants to add the following caveat to the findings: MEEPP II’s activities and PMP did not include an explicit focus on building capacity for data usage or building data-driven decision making in Uganda’s HIV community. In addition, any explicit data usage or data-driven decision making MEEPP II performed was focused on MEEPP II’s own abilities to generate USG reports (COP, OGAC quarterly reports, and special reports). In addition, MEEPP II’s partners for DHIS 2 were exclusively national GoU (MoH ACP and UAC) and support to IPs to support sites and facilities; they did NOT include direct interaction with facilities, districts, or other partner partners. MEEPP II did have more outreach to CSOs and districts in its support to OVC MIS but did not explicitly focus on building capacity for data use or data-driven decision making with these partners. MEEPP II EVALUATION 23 V. CONCLUSIONS AND RECOMMENDATIONS CONCLUSIONS (ALL) Question 1: PEPFAR Reporting MEEPP II provided indispensable support to the Uganda PEPFAR team for OGAC planning and reporting. It enabled that team to submit high-quality quarterly, semi-annual, and annual reports on time to OGAC. Project staff worked diligently with IPs through trainings and other means to ensure understanding of PEPFAR indicators and changes. They also served an integral role in helping to harmonize indicators between GoU M&E systems and PEPFAR. Their efforts helped to continuously improve the availability of data for PEPFAR. They developed useful logic checks that picked up on abnormalities and worked with other PEPFAR-funded IPs to obtain corrections at site level. However, MEEPP II staff could have been much more efficient if their data validation processes were not manually conducted. The evaluation team also notes that Excel-based automation tools have been developed and will be deployed in the near future to assist with data validation. MEEPP II conducted all required analyses for PEPFAR reporting, as well as additional analyses, such as the HIV burden table, and its responses to other requests by the strategic information (SI) TWG were instrumental in PEPFAR planning and decision making. However, the evaluation team noted a lack of strategic use of data beyond PEPFAR reporting. While MEEPP II produced a number of other reports, other stakeholders, including the USG, were largely unaware of them and consequently did not use them. Question 2: Transition to GoU M&E Systems MEEPP II’s efforts greatly contributed to increasing the availability of data in GoU M&E systems, particularly the DHIS 2 and the OVC MIS. Using a series of interventions, from harmonization of indicators to trainings and continual coaching and support, project staff helped the MoH to increase complete reporting to the DHIS 2 from three districts in 2012 to 46 districts in 2016, and among the remaining districts not reporting, only four had less than 80% of their sites reporting. These efforts resulted in MEEPP II being able to use DHIS 2 data for PEPFAR reporting since October 2014. However, activities to clean the data within the DHIS 2 and make it more reliable were not transitioned to the GoU. Although the GoU forms part of the team, MEEPP II staff continue to lead the bulk of data cleaning and validation activities, with heavy reliance on IPs for follow-up with districts and sites. Also not transitioned were ability to analyze or use the data from the DHIS 2 for decision making. Question 3: Government Preparedness The DHIS II is meeting a critical need for the GoU, and there is a strong sense of ownership and responsibility for the DHIS II to become the primary source of data for PEPFAR and other partners. However, the evaluation team concluded that as of now, the GoU is not prepared to ensure that the DHIS 2 is the primary source of data for PEPFAR reporting. Using the Oxford Epi Sustainability Framework to analyze the findings, the team concluded that a focus on improving relevant policy, advocacy capacity, organizational infrastructure, physical infrastructure, staffing, and financial viability would improve government preparedness. 24 MEEPP II EVALUATION Question 4: MEEPP II Contribution to Data Quality and Data Use MEEPP II’s main contribution to a data-driven learning environment and decision making within the Uganda HIV/AIDS program community has been focused on improving the timely acquisition of quality data and supporting USG to create reports for its usage. Usage by the audiences listed in the question has been incidental to data collection and validation activities and varied based on existing M&E and quality improvement programs and capacity within the organization. Another conclusion is that MEEPP II did not include explicit usage promotion activities for the Uganda HIV/AIDS program community in its work plan or PMP; nor did the team feel that this task was an aspect of its mandate. The project audience scope was exclusively USG, GoU national level, and IPs, and the focus was on supporting improved data collection and validation for PEPFAR reporting. This exclusive focus, the evaluation team believes, is partially responsible for the lack of usage that can be directly attributed to MEEPP II outside of the USG. RECOMMENDATIONS PEPFAR General  Based on the findings for all four evaluation questions, continue to focus on improving the quality and strategic use of GoU M&E systems along the continuum of the data lifecycle (collection, validation, analysis, and reporting and use) following UNAIDS and PEPFAR’s Three Ones Policy: one national plan, one national coordinating authority, and one national M&E system. It is likely that as the quality of the data within the GoU M&E system improves, its perceived usefulness will also improve. The specific ways to do this will necessarily evolve from the stakeholders but should be coordinated and integrated within an MEEPP II follow-on activity.  Based on the findings for questions 2 and 3 about the capacity-building support provided by MEEPP II to the various stakeholders, consider having staff embedded within the various USG offices responsible for PEPFAR reporting. The evaluation team heard requests from GoU for donor-funded staff to be seconded or co-located with the GoU in order to facilitate communications, efficiency of work, and mentoring and as a tangible physical symbol of USG support for strengthening the GoU national M&E systems. Staff from USG also made a similar suggestion, and the evaluation team for the MEEPP I had made a similar recommendation. Thus, the current evaluation team recommends that USG consider embedding staff within the GoU, in particular, the MoH Resource Center (now the Division of Health Information) responsible for ensuring the quality of the DHIS 2 and the MoGLSD responsible for OVC MIS. This would place the staff where the work needs to happen but would have to be weighed against other factors, such as flexibility of movement and physical infrastructure. An additional benefit would be potential saving on office space and physical infrastructure funded by the MEEPP II follow-on activity.  Based on the findings for question 1, clarify roles and responsibilities delineated for all PEPFAR-funded M&E activities in order to maximize efficiency and avoid possible duplication. MEEPP II EVALUATION 25  Based on the findings for question 3, hold IPs explicitly responsible and accountable for GoU M&E strengthening in districts and sites that they support. To do this, the Mission might consider including language supportive of M&E strengthening and accountability in all future IP contracts and grants. For currently funded projects where this modification may be logistically difficult, the team would urge the MoH and the Mission to make strong suggestions to partners about their involvement in ensuring that M&E capacity is measurable and increases over time. Team leaders and office directors could consider setting reporting on IP strengthening of GoU M&E system as part of each Agreement Officer's Representative (AOR) and COR annual review and could encourage AORs and CORs to report progress on M&E system strengthening as part of every portfolio review conducted at the Mission. They should also provide a means for IPs to share with PEPFAR and with each other common challenges and issues that arise, as well as flag any potential challenges that PEPFAR may need to be aware of that could compromise data quality and reporting.  Based on the findings for question 2, play an integral role in shaping emerging legal and policy framework around e-data in partnership with the GoU to secure USG’s continued access to data from GoU M&E systems. PEPFAR should explicitly look for any possible challenges to access to data or issues around data security and privacy. Follow-on Programming  Based on the findings for question 1, support planning and reporting needs of the USG Uganda PEPFAR program, including preparation of COPs and quarterly and annual reports. Specifically, new programming should continue with the interagency coordination and facilitation as directed by the USG.  Based on the findings for question 2, ensure coordination between PEPFAR and the GoU and employ management responsive to ever-changing USG and GoU reporting and planning needs. This coordination would also help identify and address emerging challenges, such as what occurred with the USG’s response to the Anti-Homosexuality Act.  Based on the findings for question 4, if PEPFAR wishes to contribute to data quality and usage in Uganda, it needs to make those expectations explicit in the SOW for a follow￾on project., Such a focus should include a component on learning and data use that will explicitly focus on building data-driven decision making in Uganda’s HIV community. This activity would likely include knowledge management, communication, and collaboration and learning for PEPFAR, the GoU, and partners of the GoU. In addition, activities should be reoriented from producing formal printed reports to dissemination of the content via partners, social media, and other formats that may be a better return on investment.  Based the findings for question 1, the follow-on project should include efficiency improvements, such as building additional software tools, streamlining data management and validation processes, and building more proactive and ad hoc analysis approaches and tools for HIV data for decision making by all stakeholders. The team believes these efficiency improvements will free up many labor resources for the above additional tasks, especially if the GoU can take over much of the labor-intensive data validation 26 MEEPP II EVALUATION work performed by MEEPP II. These processes will also assist the USG in negotiating and implementing future data requirement needs. GoU Based on the findings for questions 2 and 3, in order for the GoU M&E systems to become reliable, trusted, and sustainable, the data reveal the need for the GoU, with support from the USG and other partners, to elaborate relevant policies, continue to build ICT capacity, expand access to existing databases like the DHIS, and create demand for data, thereby promoting data use. There are certain activities that will necessarily take place at the national level and others at the district and site levels. Table 6 provides a summary of these. Table 6. Recommendations for the GOU Recommendation Nat’l District Site Elaborate relevant policies for cross-national data exchange within the eHealth Strategy.  Produce and house the final dataset that all partners will use with a high level of confidence.  Invest in EMRs and “offline digital” data collection.    Increase access to DHIS.2.   Increase the focus on data analysis, usage, and outreach.    Improve capacity to mentor sites and facilities.    Improve user experience and access policies in DHIS 2 based on user groups.    Ensure ICT capacity is built.    Continue quarterly data quality review meetings.    Disseminate, train, and support compliance with eHealth policies.    MEEPP II EVALUATION 27 ANNEX 1. SCOPE OF WORK Assignment #: 194 [assigned by GH Pro] Global Health Program Cycle Improvement Project – GH Pro Contract No. AID-OAA-C-14-00067 EVALUATION OR ANALYTIC ACTIVITY STATEMENT OF WORK (SOW) Date of Submission: 1/20/2016 Last update: 5/5/2016 I. TITLE: Evaluation of the Monitoring and Evaluation of Emergency Plan Progress (MEEPP II) Project II. Requester / Client  USAID Country or Regional Mission Mission/Division: Uganda / III. Funding Account Source(s): (Click on box(es) to indicate source of payment for this assignment)  3.1.1 HIV  3.1.2 TB  3.1.3 Malaria  3.1.4 PIOET  3.1.5 Other public health threats  3.1.6 MCH  3.1.7 FP/RH  3.1.8 WSSH  3.1.9 Nutrition  3.2.0 Other (specify): IV.Cost Estimate: $374,171 (Note: GH Pro will provide a cost estimate based on this SOW) V. Performance Period Expected Start Date (on or about): May 1, 2016 28 MEEPP II EVALUATION Anticipated End Date (on or about): October 14, 2016 VI.Location(s) of Assignment: (Indicate where work will be performed) Uganda VII.Type of Analytic Activity (Check the box to indicate the type of analytic activity) EVALUATION:  Performance Evaluation (Check timing of data collection)  Midterm Endline  Other (specify): Performance evaluations focus on descriptive and normative questions: what a particular project or program has achieved (either at an intermediate point in execution or at the conclusion of an implementation period); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to program design, management and operational decision making. Performance evaluations often incorporate before-after comparisons, but generally lack a rigorously defined counterfactual.  Impact Evaluation (Check timing(s) of data collection)  Baseline Midterm Endline  Other (specify): Impact evaluations measure the change in a development outcome that is attributable to a defined intervention; impact evaluations are based on models of cause and effect and require a credible and rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. Impact evaluations in which comparisons are made between beneficiaries that are randomly assigned to either a treatment or a control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured. OTHER ANALYTIC ACTIVITIES  Assessment Assessments are designed to examine country and/or sector context to inform project design, or as an informal review of projects.  Costing and/or Economic Analysis Costing and Economic Analysis can identify, measure, value and cost an intervention or program. It can be an assessment or evaluation, with or without a comparative intervention/program.  Other Analytic Activity (Specify) MEEPP II EVALUATION 29 PEPFAR EVALUATIONS (PEPFAR Evaluation Standards of Practice 2014) Note: If PEPFAR funded, check the box for type of evaluation  Process Evaluation (Check timing of data collection)  Midterm Endline Other (specify): Process Evaluation focuses on program or intervention implementation, including, but not limited to access to services, whether services reach the intended population, how services are delivered, client satisfaction and perceptions about needs and services, management practices. In addition, a process evaluation might provide an understanding of cultural, socio￾political, legal, and economic context that affect implementation of the program or intervention. For example: Are activities delivered as intended, and are the right participants being reached? (PEPFAR Evaluation Standards of Practice 2014)  Outcome Evaluation Outcome Evaluation determines if and by how much, intervention activities or services achieved their intended outcomes. It focuses on outputs and outcomes (including unintended effects) to judge program effectiveness, but may also assess program process to understand how outcomes are produced. It is possible to use statistical techniques in some instances when control or comparison groups are not available (e.g., for the evaluation of a national program). Example of question asked: To what extent are desired changes occurring due to the program, and who is benefiting? (PEPFAR Evaluation Standards of Practice 2014)  Impact Evaluation (Check timing(s) of data collection)  Baseline Midterm Endline Other (specify): Impact evaluations measure the change in an outcome that is attributable to a defined intervention by comparing actual impact to what would have happened in the absence of the intervention (the counterfactual scenario). IEs are based on models of cause and effect and require a rigorously defined counterfactual to control for factors other than the intervention that might account for the observed change. There are a range of accepted approaches to applying a counterfactual analysis, though IEs in which comparisons are made between beneficiaries that are randomly assigned to either an intervention or a control group provide the strongest evidence of a relationship between the intervention under study and the outcome measured to demonstrate impact.  Economic Evaluation (PEPFAR) Economic Evaluations identifies, measures, values and compares the costs and outcomes of alternative interventions. Economic evaluation is a systematic and transparent framework for assessing efficiency focusing on the economic costs and outcomes of alternative programs or interventions. This framework is based on a comparative analysis of both the costs (resources consumed) and outcomes (health, clinical, economic) of programs or interventions. Main types of economic evaluation are cost-minimization analysis (CMA), cost-effectiveness analysis (CEA), cost-benefit analysis (CBA) and cost-utility analysis (CUA). Example of question asked: What is the cost-effectiveness of this intervention in improving patient outcomes as compared to other treatment models? VIII.BACKGROUND If an evaluation, Project/Program being evaluated: Project/Activity Title: Evaluation of the Monitoring and Evaluation OF Emergency Plan Progress (MEEPP II) Project Award/Contract Number: AID-617-C-10-00008 Award/Contract Dates: 9/8/2010 - 5/29/2016 Project/Activity Funding: $17,099,654 Implementing Organization(s): Social Science Systems Inc. (SSS) Project/Activity AOR/COR: Joseph Mwangi 30 MEEPP II EVALUATION Background of project/program/intervention: The overall goal of the 5-year MEEPP II project is to strengthen HIV/AIDS-related national monitoring and evaluation systems and support the collection of complete, accurate, and timely data that can meet government of Uganda, PEPFAR, Global Fund and other development partner data requirements. The primary objective of MEEPP II is to shift from a vertical and comprehensive PEPFAR Implementing partner (IP)-based program performance management system and strengthen the existing national and district-level M&E systems to support data needs of the key HIV/AIDS related stakeholders. These stakeholders include the Ministry of Health (MOH), Ministry of Gender, Labour and Social Development (MGLSD), PEPFAR, Global Funds, UNAIDS, WHO, UNICEF and other development partners. To achieve this primary objective, MEEPP II has developed and implemented collaborative strategies engaging PEPFAR USG agencies and their respective implementing partners (IPs), the USG activity managers, as well as GOU national-level stakeholders, including MOH/Resource Center (RC), MOH/National AIDS Control Program (NACP), MGLSD/ OVC National Implementation Unit and Uganda AIDS Commission (UAC). Since the launch of MEEPP II in September 2010, the project has centered its strategies on the following current USG Strategic Information (SI) pivots:  Alignment of USG-supported systems with the national information systems;  Support robust M&E systems at service delivery points and districts;  Increase use of data for evidence-based planning and decision making at all levels; and,  Promote technological innovations to track referrals, linkages and retention of clients. It is expected that by August 2015, MEEPP-II project will have significantly contributed to a strengthened national HIV/AIDS M&E systems allowing government of Uganda (GOU), PEPFAR, the Global Fund and other development partners to meet most of their data needs from these systems. The program is scheduled to end in May 2016. Conceptual Framework for Transition Plan In order to achieve its primary objective of shifting from a vertical PEPFAR Implementing Partner (IP)-based performance management system to using existing district-based national M&E systems, Social Science Systems Inc (SSS), the MEEPP- II Implementing Partner, has developed and is in the process of implementing a transition plan. This plan aims at transferring key MEEPP data management responsibilities to national and district level stakeholders. To facilitate a responsible transition MEEPP II will keep the IP-based performance management system (PEPMIS) later modified to the HIV Information Based Real Time Integrated Database (HIBRID) developed to mirror the national data management information system (the District Health Information System (DHIS 2) for a short time after PEPFAR starts using the DHIS-2 for reporting. The HIBRID was used as a platform to train PEPFAR IPs on the national data management system, while DHIS 2 and OVC MIS were being rolled-out and strengthened to take the place of PEPMIS. HIBRID will be redundant, and will MEEPP II EVALUATION 31 be phased out once DHIS-2, OVC MIS and other national data management systems can meet data needs of stakeholders at an acceptable quality. Strengthening of the national M&E systems to adequately accommodate national data needs requires a major shift in the way USG, national and district stakeholders operate, embodied by a change in the standard operating procedures. MEEPP-II will organize increased support to strengthen national M&E systems and improvements in data quality, stakeholder data needs as illustrated in figure 1 below. Figure 1. Purpose of the Transition Plan Strategic or Results Framework for the project/program/intervention (paste framework below) If project/program does not have a Strategic/Results Framework, describe the theory of change of the project/program/intervention. The results framework has been modified to reflect changes that have occurred over the past 3 years with the original Intermediate Result 2 being consolidated into intermediate result 1 and intermediate result 3 to reflect the emphasis placed on building capacity and providing M&E support to districts as opposed to PEPFAR IPs. The flow of the activities and out puts has also been revised to improve clarity and logic given changes that have occurred over the last three years especially the emphasis that has been placed on alignment with the national system and M&E support to districts in addition to supporting the collection of complete, accurate, and timely data to meet GOU and PEPFAR reporting requirements. The performance monitoring plan (PMP) will consequently be updated to reflect these changes. 32 MEEPP II EVALUATION National HIV/AIDS Related M&E Systems Strengthened to Accommodate National and Most PEPFAR Data Needs Intermediate Result 1 PEPFAR Data Quality Management for use in Program Planning & Reporting Improved Intermediate Result 2 Monitoring of District M&E capacity and performance strengthened Intermediate Result 3 Use of programmatic knowledge for decisions strengthened Data reported to PEPFAR of acceptable quality Quality and Utilization of HIV/AIDS related data by districts improved Feedback loop established for review of important programmatic insights and recommendation Standard operating procedures for PEPFAR data management and target setting strengthened PEPFAR data collection procedures aligned with national processes Monitoring of completeness and timeliness of HIV/AIDS related data submitted by districts improved IPs proficient in the use of national data collection tools Quality of special studies improved MEEPP Historical data standardized to serve USG and national analytical needs Figure 5: MEEPP-II Results Framework What is the geographic coverage and/or the target groups for the project or program that is the subject of analysis? National and district-level HIV M&E systems; MEEP supports selected districts through USAID and CDC implementing partners. IX.SCOPE OF WORK A. Purpose: Why is this evaluation or analysis being conducted (purpose of analytic activity)? Provide the specific reason for this activity, linking it to future decisions to be made by USAID leadership, partner governments, and/or other key stakeholders. The purpose of this evaluation is to comprehensively analyze MEEPP-II’s performance towards supporting PEPFAR data and analytical needs, and assess progress in the transition from reliance on a project-managed database, to the Government of Uganda (GoU) systems for MEEPP II EVALUATION 33 reporting PEPFAR interventions. The findings of this evaluation are expected to contribute to USG decision on the level and type of support to further strengthen the GOU data system for PEPFAR reporting and other national needs. Specifically, the evaluation provides an opportunity for USG to identify gaps, including strengths and weaknesses, and gather evidence upon which a post-MEEPP strategy can be based. B. Audience: Who is the intended audience for this analysis? Who will use the results? If listing multiple audiences, indicate which are most important. The primary users of the evaluation findings are PEPFAR Program managers, Implementing Partners, GoU and other national stakeholders with interest in HIV/AIDS data collection, reporting, and analysis. C. Applications and use: How will the findings be used? What future decisions will be made based on these findings? The findings of this evaluation are expected to contribute to USG decision on the level and type of support to further strengthen the GOU data system for PEPFAR reporting and other national needs. The evaluation provides an opportunity for USG to identify gaps, including strengths and weaknesses, and gather evidence upon which a post-MEEPP strategy can be based. D. Evaluation/Analytic Questions & Matrix: a) Questions should be: a) aligned with the evaluation/analytic purpose and the expected use of findings; b) clearly defined to produce needed evidence and results; and c) answerable given the time and budget constraints. Include any disaggregation (e.g., sex, geographic locale, age, etc.), they must be incorporated into the evaluation/analytic questions. USAID policy suggests 3 to 5 evaluation/analytic questions. b) List the recommended methods that will be used to collect data to be used to answer each question. c) State the application or use of the data elements towards answering the evaluation questions; for example, i) ratings of quality of services, ii) magnitude of a problem, iii) number of events/occurrences, iv) gender differentiation, v) etc. 34 MEEPP II EVALUATION Note: The Mission views the ‘Areas to consider’, listed below each question, as critical to providing essential information significant to this evaluation. Evaluation Question Research Methods Application or Data Use 1 PEPFAR Reporting. To what extent has MEEPP-II efficiently, consistently and routinely collected, analyzed, and reported data for PEPFAR planning, project management and performance reporting? Areas to consider:  Lessons learned and best practices to inform future M&E capacity building efforts supporting PEPFAR implementation in Uganda 2 Transition (MEEPP). To what extent have the data collection, data validation, analysis and reporting systems established by MEEPP effectively transitioned to the national systems pivoted around DHIS-2 and OVC MIS? Areas to consider:  Evolution of MEEPP’s role over time  Paths for processes that are still in transition  PEPFAR indicators and their alignment with the national information systems 3 Transition (GOU). To what extent is the GoU prepared to make the transition into becoming the primary source for PEPFAR data, and to what extent will the data be readily accessible by PEPFAR and other users? Areas to consider:  Lessons drawn from the transition process for obtaining PEPFAR data for USAID, post MEEPP MEEPP II EVALUATION 35 Note: The Mission views the ‘Areas to consider’, listed below each question, as critical to providing essential information significant to this evaluation. Evaluation Question Research Methods Application or Data Use 4 Data Quality and Data Use. To what extent has MEEPP-II contributed towards timely acquisition of quality data and use of data by health facilities, district health managers, GOU and USG Implementing Partners? Areas to consider:  MEEPP’s contribution to a data driven learning environment and decision making within the Uganda HIV/AIDS program community 5 Other Questions [OPTIONAL] (Note: Use this space only if necessary. Too many questions leads to an ineffective evaluation or analysis.) E. Methods: Check and describe the recommended methods for this analytic activity. Selection of methods should be aligned with the evaluation/analytic questions and fit within the time and resources allotted for this analytic activity. Also, include the sample or sampling frame in the description of each method selected. General Comments related to Methods: Evaluation Team will collect and analyze both secondary and primary data in answering the evaluation questions. The evaluation should use a mix method approach, collecting both quantitative and qualitative data, including site visits and interviews with key stakeholders, knowledgeable of MEEPP’s performance and future data needs of PEPFAR and the GoU in addressing the HIV/AIDS epidemic. Evaluation team will design all data collection instruments and submit to GH Pro and USAID for review prior to beginning fieldwork. The following are suggested methodologies that should be considered in designing the evaluation: (1) review of relevant program related documents, (2) in-depth interviews of key informants and/or focus groups, (3) direct observation, assessment and review of MEEPP-II and national health information systems, and (4) mini survey of potential users of data that will be supported by the GoU.  Document and Data Review (list of documents and data recommended for review) This desk review will be used to provide background information on the project/program, and will also provide data for analysis for this evaluation. Documents and data to be reviewed include technical reports, annual and quarterly reports, etc. The results of the analysis of 36 MEEPP II EVALUATION these reports should contribute to lessons learned from MEEPP’s implementation during the period covered by the evaluation.  MEEPP contract  MEEPP Annual Workplans  MEEPP Annual Reports  Miscellaneous MEEPP reports  National HIV/AIDS M&E Plan  Uganda DHIS-2 data and/or reports  OVC MIS  HMIS Resource Center, Uganda Ministry of Health (http://health.go.ug/hmis/)  Uganda MERS and SIMS data that will be accessed from DATIM (will secure access rights for team – view only)  Uganda Demographic Health MIS data 2014-2015 http://dhsprogram.com/what-we￾do/survey/survey-display-484.cfm  Uganda DHS AIS data 2011 (http://dhsprogram.com/publications/publication-AIS10-AIS￾Final-Reports.cfm)  PEPMIS to access all PEPFAR Indicators over the course of MEEP  Other docs and/or data reports  Secondary analysis of existing data (This is a re-analysis of existing data, beyond a review of data reports. List the data source and recommended analyses) Data Source (existing dataset) Description of data Recommended analysis  Key Informant Interviews (list categories of key informants, and purpose of inquiry) Using a semi-structured questionnaire comprising primarily of open ended questions, the evaluation team will conduct in-depth interviews a with:  CDC, USAID and other USG Agencies  GOU MOH AIDS Control Program (ACP) and Resource Center (RC)  Uganda AIDS Commission (UAC)  PEPFAR Implementing Partners (IPs), including Uganda Health Supply Chain project and a sample of comprehensive implementing partners  Other stakeholders in the Uganda HIV/AIDS community including UNAIDS, WHO, UNICEF The evaluation team will propose the data collection tools and work closely with PEPFAR SI￾TWG to identify appropriate respondents. These interviews are expected to provide insights into how MEEPP has performed and expectations of how GoU can best support the HIV/AIDS community as the central source of sector data. MEEPP II EVALUATION 37  Focus Group Discussions (list categories of groups, and purpose of inquiry) Optional: It may be useful to conduct focus group discussions among GoU representatives and other stakeholders to provide insights into how MEEPP can support the GoU and the HIV/AIDS community through the central source of sector data and use of these data.  Group Interviews (list categories of groups, and purpose of inquiry) Optional: Some of the key informant interviews can be clustered, as long as there are no power differentials, and all respondents feel comfortable in voicing their opinions within the group. (See list and description above under KII.)  Client/Participant Satisfaction or Exit Interviews (list who is to be interviewed, and purpose of inquiry)  Facility or Service Assessment/Survey (list type of facility or service of interest, and purpose of inquiry)  Cost Analysis (list costing factors of interest, and type of costing assessment, if known)  Survey (describe content of the survey and target responders, and purpose of inquiry) Optional: A mini-survey among potential contributors and users of HIV/AIDS program data that will be produced through the system. The survey may be administered online (e.g., Survey Monkey), and as needed by phone or in-person (where connectivity is limited), using a structured questionnaire. The evaluation team will work closely with PEPFAR SI-TWG to facilitate a list of respondents with contact (email & phone) information. The evaluation team should conduct statistical analysis, and the results should provide insights to both USAID and the GoU on a post MEEPP strategy targeted at customer satisfaction in accessing sector data through the government system.  Observations (list types of sites or activities to be observed, and purpose of inquiry) Evaluation Team will plan to access and assess MEEPP-II and national health information systems. The team is expected to include senior M&E, IT, database and MIS experts with sufficient experience to materially comment on these systems. The team will directly observe and assess the process of transition from the MEEPP-II database to the DHIS-2, and OVC 38 MEEPP II EVALUATION MIS; as well as determine the readiness of GoU to efficiently manage DHIS-2 as the primary source of HIV/AIDS data, and the type of technical support that may be required for strengthening operations of the data system. This is not an assessment of how appropriate DHIS-2 is, but the team is expected to comment on how well the functions of data collection, analysis and reporting and likely to transition.  Data Abstraction (list and describe files or documents that contain information of interest, and purpose of inquiry)  Case Study (describe the case, and issue of interest to be explored)  Verbal Autopsy (list the type of mortality being investigated (i.e., maternal deaths), any cause of death and the target population)  Rapid Appraisal Methods (ethnographic / participatory) (list and describe methods, target participants, and purpose of inquiry)  Other (list and describe other methods recommended for this evaluation/analytic, and purpose of inquiry) If impact evaluation – Is technical assistance needed to develop full protocol and/or IRB submission?  Yes No List or describe case and counterfactual” Case Counterfactual MEEPP II EVALUATION 39 X. HUMAN SUBJECT PROTECTION The Evaluation Team must develop protocols to insure privacy and confidentiality prior to any data collection. Primary data collection must include a consent process that contains the purpose of the evaluation, the risk and benefits to the respondents and community, the right to refuse to answer any question, and the right to refuse participation in the evaluation at any time without consequences. Only adults can consent as part of this evaluation. Minors cannot be respondents to any interview or survey, and cannot participate in a focus group discussion without going through an IRB. The only time minors can be observed as part of this evaluation is as part of a large community-wide public event, when they are part of family and community attendance. During the process of this evaluation, if data are abstracted from existing documents that include unique identifiers, data can only be abstracted without this identifying information. XI.ANALYTIC PLAN Describe how the quantitative and qualitative data will be analyzed. Include method or type of analyses, statistical tests, and what data it to be triangulated (if appropriate). For example, a thematic analysis of qualitative interview data, or a descriptive analysis of quantitative survey data. PEPFAR SI-TWG expects this evaluation to involve rigorous data analysis to support findings and conclusions; hence a detailed data analysis plan is requested as part of the Team’s workplan. All data analyses and presentation of key data findings should also address important data disaggregation /categorization relevant to performance of MEEPP and anticipated performance of GoU as the primary source of HIV/AIDS sector data nationally. All analyses will be geared to answer the evaluation questions. Additionally, the evaluation will review both qualitative and quantitative data related to the project/program’s achievements against its objectives and/or targets. Quantitative data will be analyzed primarily using descriptive statistics. Data will be stratified by demographic characteristics, such as sex, age, and location, whenever feasible. Other statistical test of association (i.e., odds ratio) and correlations will be run as appropriate. Thematic review of qualitative data will be performed, connecting the data to the evaluation questions, seeking relationships, context, interpretation, nuances and homogeneity and outliers to better explain what is happening and the perception of those involved. Qualitative data will be used to substantiate quantitative findings, provide more insights than quantitative data can provide, and answer questions where other data do not exist. Evaluators will analyze whether answers to questions are different for different groups, particularly questions on the access and use of the current, and post-MEEPP data. Use of multiple methods that are quantitative and qualitative, as well as existing data (e.g., project/program performance indicator data, DHS, HMIS data, etc.) will allow the Team to triangulate findings to produce more robust evaluation results. 40 MEEPP II EVALUATION The Evaluation Report will describe analytic methods and statistical tests employed in this evaluation. XII.ACTIVITIES List the expected activities, such as Team Planning Meeting (TPM), briefings, verification workshop with IPs and stakeholders, etc. Activities and Deliverables may overlap. Give as much detail as possible. Background reading – Several documents are available for review for this analytic activity. These include the Uganda MEEPP project proposal, annual work plans, M&E plans, quarterly progress reports, and routine reports of project performance indicator data, as well as survey data reports (i.e., DHS and HMIS). This desk review will provide background information for the Evaluation Team, and will also be used as data input and evidence for the evaluation. Team Planning Meeting (TPM) – The TPM will begin virtually, prior to the Evaluation Team convening in Uganda. This virtual planning will result in a draft of the Evaluation Workplan and Protocol (see below). The TPM will continue in-country, once the Team has convened in Uganda. During the TMP, before the data collection begins, the Team will:  Review and clarify any questions on the evaluation SOW  Clarify team members’ roles and responsibilities  Establish a team atmosphere, share individual working styles, and agree on procedures for resolving differences of opinion  Review and finalize evaluation questions  Review and finalize the assignment timeline  Develop data collection methods, instruments, tools and guidelines  Review and clarify any logistical and administrative procedures for the assignment  Develop a data collection plan  Draft the evaluation work plan for USAID’s approval  Develop a preliminary draft outline of the team’s report  Assign drafting/writing responsibilities for the final report Evaluation Workplan and Protocol draft will be submitted to USAID/Uganda prior to the Evaluation Team convening in Uganda, with a final workplan and protocol due at the close of the TPM that includes:  Evaluation design matrix including key sub-questions and or indicators, data collections methods,  Sampling frame and sample for each method  Data collection tools with statement to obtain oral consent, where appropriate  Data management plan to insure data integrity that covers data collection, data storage, data entry and cleaning (as needed)  Data analysis plan  Timeline for field work and deliverables USAID/Uganda will circulate to PEPFAR SI-TWG for review and send back comments to the team within 5 working days of receipt of the draft. During this time GH Pro will also review MEEPP II EVALUATION 41 the data collection instruments. The Team should not commence data collection until they have received approval from USAID. Briefing and Debriefing Meetings – Throughout the evaluation the Team Lead will provide briefings to USAID. The In-Brief and Debrief are likely to include the all Evaluation Team experts, but will be determined in consultation with the Mission. These briefings are:  Evaluation launch, a call/meeting among the USAID, GH Pro and the Team Lead to initiate the evaluation activity and review expectations. USAID will review the purpose, expectations, and agenda of the assignment. GH Pro will introduce the Team Lead, and review the initial schedule and review other management issues. Following this, there will be a launch call with MEEPP and the Evaluation Team Lead to set expectations.  In-brief with USAID, as part of the TPM. This briefing may be broken into two meetings: a) at the beginning of the TPM, so the Evaluation Team and USAID and PEPFAR SI TWG can discuss expectations and intended plans; and b) at the end of the TPM when the Evaluation Team will present an outline and explanation of the design and tools of the evaluation (Evaluation Workplan and Protocols). Also discussed at the in-brief will be the format and content of the Evaluation report(s). The time and place for this in-brief will be determined between the Team Lead and USAID prior to the TPM.  In-brief with MEEPP to review the evaluation plans and timeline, and for the project to give an overview of the project to the Evaluation Team.  The Team Lead (TL) will brief the USAID weekly to discuss progress on the evaluation. As preliminary findings arise, the TL will share these during the routine briefing, and in an email.  A final debrief between the Evaluation Team and PEPFAR Country Team will be held at the end of the evaluation to present preliminary findings. During this meeting a summary of the data will be presented, along with high level findings and draft recommendations. For the debrief, the Evaluation Team will prepare a PowerPoint Presentation of the key findings, issues, and recommendations. The evaluation team shall incorporate comments received from PEPFAR Country Team during the debrief in the evaluation report. USAID may request a separate debrief just with the management team prior to the PEPFAR Country Team debrief. They will inform the Evaluation Team during the TPM, if this is needed. (Note: preliminary findings are not final and as more data sources are developed and analyzed these finding may change.)  Stakeholders’ debrief/workshop will be held with the MEEPP staff and MOH. These will likely be two separate debriefs, but USAID will confirm with the Evaluation Team during the TPM. These debrief workshops will occur following the final debrief with the Mission and PEPFAR Country Team, and will not include any information that may be deemed sensitive by USAID. Fieldwork, Site Visits and Data Collection – The evaluation team will conduct site visits for data collection. Selection of sites to be visited will be finalized during TPM in consultation with USAID. The evaluation team will outline and schedule key meetings and site visits prior to departing to the field. It is recommended that the Evaluation Team split into sub-teams to cover the district visits in an efficient amount of time. 42 MEEPP II EVALUATION Evaluation/Analytic Report – The Evaluation/Analytic Team under the leadership of the Team Lead will develop a report with findings and recommendations (see Analytic Report below). Report writing and submission will include the following steps: 1. Team Lead will submit draft evaluation report to GH Pro for review and formatting 2. GH Pro will submit the draft report to USAID 3. USAID will review the draft report in a timely manner, and send their comments and edits back to GH Pro 4. GH Pro will share USAID’s comments and edits with the Team Lead, who will then do final edits, as needed, and resubmit to GH Pro 5. GH Pro will review and reformat the final Evaluation/Analytic Report, as needed, and resubmit to USAID for approval. 6. Once Evaluation Report is approved, GH Pro will re-format it for 508 compliance and post it to the DEC. The Evaluation Report excludes any procurement-sensitive and other sensitive but unclassified (SBU) information. This information will be submitted in a memo to USIAD separate from the Evaluation Report. MEEPP II EVALUATION 43 XIII.DELIVERABLES AND PRODUCTS Select all deliverables and products required on this analytic activity. For those not listed, add rows as needed or enter them under “Other” in the table below. Provide timelines and deliverable deadlines for each. Deliverable / Product Timelines & Deadlines (estimated)  Launch briefing May 10, 2016  Launch briefing with MEEPP May 12, 2016  Draft Workplan with timeline Draft (before arrival in country): May 24, 2016  Evaluation protocol with data collection tools Draft (before arrival in country): May 24, 2016  In-brief with Mission June 2-6, 2016  In-brief with MEEPP June 8, 2016  Routine briefings Weekly  Out-brief with Mission with Power Point presentation July 6, 2016  Findings review workshop with stakeholders with Power Point presentation July 7, 2016  Draft report Submit to GH Pro: July 21, 2016 GH Pro submits to USAID: July 27, 2016  Final report Submit to GH Pro: August 17, 2016 GH Pro submits to USAID: August 23, 2016  Raw data (cleaned datasets in CSV or XML) with Data Dictionary August 17, 2016  Dissemination activity  Report Posted to the DEC October 14, 2016  Other (specify): Estimated USAID review time Average number of business days USAID will need to review deliverables requiring USAID review and/or approval? 10 Business days XIV.TEAM COMPOSITION, SKILLS AND LEVEL OF EFFORT (LOE) Evaluation/Analytic team: When planning this analytic activity, consider:  Key staff should have methodological and/or technical expertise, regional or country experience, language skills, team lead experience and management skills, etc.  Team leaders for evaluations/analytics must be an external expert with appropriate skills and experience.  Additional team members can include research assistants, enumerators, translators, logisticians, etc.  Teams should include a collective mix of appropriate methodological and subject matter expertise. 44 MEEPP II EVALUATION  Evaluations require an Evaluation Specialist, who should have evaluation methodological expertise needed for this activity.  All team members will be required to provide a signed statement attesting that they have no conflict of interest, or describing the conflict of interest if applicable.  All team members sign a non-disclosure agreement as part of their GH Pro contract. Team Qualifications: Please list technical areas of expertise required for this activities List the key staff needed for this analytic activity and their roles. You may wish to list desired qualifications for the team as a whole, as well as for the individual team members. PEPFAR SI-TWG expects an evaluation team that includes a senior evaluator who may also serve as Team Leader, a HIV/AIDS Program SI Specialist, and a MIS/Database Management/IT Expert. Key Staff 1: Team Lead and Evaluation Specialist Roles & Responsibilities: The team leader will be responsible for (1) providing team leadership; (2) managing the team’s activities, (3) ensuring that all deliverables are met in a timely manner, (4) serving as a liaison between the USAID and the evaluation/analytic team, and (5) leading briefings and presentations. As the evaluation specialist, this person will also provide quality assurance on evaluation issues, including methods, development of data collection instruments, protocols for data collection, data management and data analysis. S/He will oversee the training of all engaged in data collection, insuring highest level of reliability and validity of data being collected. S/He is the lead analyst, responsible for all data analysis, and will coordinate the analysis of all data, assuring all quantitative and qualitative data analyses are done to meet the needs for this evaluation. S/He will participate in all aspects of the evaluation, from planning, data collection, data analysis to report writing. Qualifications:  Master’s Degree or higher  Minimum of 10 years of experience in public health, which included experience in implementation of health activities in developing countries  Demonstrated experience leading health sector project/program evaluation/analytics, utilizing both quantitative and qualitative methods  Had a significant role in a minimum of ten evaluations/assessments, preferable covering health, IT and M&E activities/programs  Led a minimum of five evaluation/assessment  Demonstrated knowledge of data needs for PEPFAR reporting  Excellent skills in planning, facilitation, and consensus building  Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders  Excellent skills in project management  Excellent organizational skills and ability to keep to a timeline MEEPP II EVALUATION 45  Good oral communication and writing skills, with extensive report writing experience  Experience working in the region, and experience in Uganda is desirable  Familiarity with USAID and PEPFAR policies and practices  Evaluation policy & standards of practice  Results frameworks  Performance monitoring plans  At least 8 years of experience in USAID M&E procedures and implementation  At least 5 years managing M&E, including evaluations  Strong knowledge, skills, and experience in qualitative and quantitative evaluation tools  Experience implementing and coordinating other to implements surveys, key informant interviews, focus groups, observations and other evaluation methods that assure reliability and validity of the data.  Experience in data management  Able to analyze quantitative, which will be primarily descriptive statistics  Able to analyze qualitative data  Experience using analytic software  Demonstrated experience using qualitative evaluation methodologies, and triangulating with quantitative data  Able to review, interpret and reanalyze as needed existing data pertinent to the evaluation  Strong data interpretation and presentation skills  Familiarity with USAID and PEPFAR health programs/projects, particularly PEPFAR M&E is preferred Key Staff 2 Title: HIV/AIDS Program SI Specialist Roles & Responsibilities: Serve as a member of the evaluation team, providing expertise in HIV SI systems and processes, particularly related to PEPFAR and national routine information systems. S/He will participate in planning and briefing meetings, data collection, data analysis, development of evaluation presentations, and writing of the Evaluation Report. Qualifications:  Master’s degree or higher in Epidemiology, Biostatistics or related field  At least 10 years’ experience working in M&E of HIV/AIDS prevention and treatment programs  Demonstrated excellent knowledge of indicators and data requirements for all PEPFAR Technical Areas and a good understanding of service cascades.  Familiar with DHIS-2 and similar national routine heath information systems is desirable  Experience working with projects and at the local and national levels on reporting processes and data flow, and use of data 46 MEEPP II EVALUATION  Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders  Proficient in English  Good writing skills, specifically technical and evaluation report writing experience  Experience in conducting USAID evaluations of health programs/activities is desirable Key Staff 3 Title: Management Information Systems (MIS)/Database Management Expert Roles & Responsibilities: Serve as a member of the evaluation team, providing expertise in the transition process, and the functionality of the GoU system, and assessing the preparedness of the DHIS-2 and OVC MIS to serve as the primary source of data for PEPFAR activities. The ICT/MIS Expert will also be responsible for identifying information needed to understand shifts in data quality and assess data use at district and facility levels, and serve as the team IT expert. S/He will participate in planning and briefing meetings, data collection, data analysis, development of evaluation presentations, and writing of the Evaluation Report. Qualifications:  At least 8 years of experience working with ICT, MIS, including a minimum of six years developing and/or managing health information systems  Familiar with DHIS-2 and similar national routine heath information systems  Experience working with projects and at the local and national levels on data management, data quality, and use of data  Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders  Proficient in English  Good writing skills, specifically technical and evaluation report writing experience  Experience in conducting USAID evaluations of health programs/activities is desirable  Experience working with projects and at the local and national levels on reporting processes and data flow, and use of data  Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders  Proficient in English  Good writing skills, specifically technical and evaluation report writing experience Key Staff 3 Title: Evaluation Specialist Roles & Responsibilities: Serve as a member of the evaluation team, providing quality assurance on evaluation issues, including methods, development of data collection instruments, protocols for data collection, data management and data analysis. S/He will oversee the training of all engaged in data collection, insuring highest level of reliability and validity of data being collected. S/He is the lead analyst, responsible for all data MEEPP II EVALUATION 47 analysis, and will coordinate the analysis of all data, assuring all quantitative and qualitative data analyses are done to meet the needs for this evaluation. S/He will participate in all aspects of the evaluation, from planning, data collection, data analysis to report writing. Qualifications:  At least 8 years of experience in USAID M&E procedures and implementation  At least 5 years managing M&E, including evaluations  Experience in design and implementation of evaluations  Strong knowledge, skills, and experience in qualitative and quantitative evaluation tools  Experience implementing and coordinating other to implements surveys, key informant interviews, focus groups, observations and other evaluation methods that assure reliability and validity of the data.  Experience in data management  Able to analyze quantitative, which will be primarily descriptive statistics  Able to analyze qualitative data  Experience using analytic software  Demonstrated experience using qualitative evaluation methodologies, and triangulating with quantitative data  Able to review, interpret and reanalyze as needed existing data pertinent to the evaluation  Strong data interpretation and presentation skills  An advanced degree in public health, evaluation or research or related field  Proficient in English  Good writing skills, including extensive report writing experience  Familiarity with USAID and PEPFAR health programs/projects, particularly PEPFAR M&E is preferred  Familiarity with USAID and PEPFAR M&E policies and practices  Evaluation policies and standards of practice  PEPFAR indicators  Performance monitoring plans Other Staff Titles with Roles & Responsibilities (include number of individuals needed): Local Evaluation Logistics /Program Assistant will support the Evaluation Team with all logistics and administration to allow them to carry out this evaluation. The Logistics/Program Assistant will have a good command of English and local language(s). S/He will have knowledge of key actors in the health sector and their locations including MOH, donors and other stakeholders. To support the Team, s/he will be able to efficiently liaise with hotel staff, arrange in-country transportation (ground and air), arrange meeting and workspace as needed, and insure business center support, e.g. copying, internet, and printing. S/he will work under the guidance of the Team Leader to make preparations, arrange meetings and appointments. S/he will conduct programmatic administrative and support tasks as assigned 48 MEEPP II EVALUATION and ensure the processes moves forward smoothly. S/He may also be asked to assist in translation of data collection tools and transcripts, if needed. Evaluation Coordinator (1 local consultants) to assist the Evaluation Team with data collection, analysis and data interpretation. S/He will have basic familiarity with health topics, as well as experience conducting surveys interviews and focus group discussion, both facilitating and note taking. Familiarity with DHIS-2, OVC-MIS is desirable. S/He will also assist in translation of data collection tools and transcripts, as needed. The Local Evaluator will have a good command of English and local language(s). S/He will also assist the Team and the Logistics Coordinator, as needed. S/He will report to the Team Lead. Will USAID participate as an active team member or designate other key stakeholders to as an active team member? This will require full time commitment during the evaluation or analytic activity.  Yes – If yes, specify who:  Significant involvement – If yes, specify who: The USAID Program Office Monitoring and Evaluation Specialist will have primary management role for the evaluation process while the PEPFAR SI-TWG through the two SI-TWG Co-chairs will have technical responsibility and make the necessary arrangements for USG inputs and briefings. The Evaluation Team will have various consultations with PEPFAR TWGs and the PEPFAR Coordination Office. Some staff from USAID or the PEPFAR SITWG may join the evaluation team to participate in the field work.  No Staffing Level of Effort (LOE) Matrix (Optional): This optional LOE Matrix will help you estimate the LOE needed to implement this analytic activity. If you are unsure, GH Pro can assist you to complete this table. a) For each column, replace the label "Position Title" with the actual position title of staff needed for this analytic activity. b) Immediately below each staff title enter the anticipated number of people for each titled position. c) Enter Row labels for each activity, task and deliverable needed to implement this analytic activity. d) Then enter the LOE (estimated number of days) for each activity/task/deliverable corresponding to each titled position. e) At the bottom of the table total the LOE days for each consultant title in the ‘Sub-Total’ cell, then multiply the subtotals in each column by the number of individuals that will hold this title. Level of Effort in days for each Evaluation/Analytic Team member MEEPP II EVALUATION 49 Activity / Deliverable Evaluation/Analytic Team Team Lead / Evaluation Specialist MIS/ Database Managemen t Expert HIV SI Specialist Evaluation Coordinat or Logistics/ Program Assist 1 Launch Briefing 0.5 2 Desk review 5 5 5 5 Virtual Team Planning Meeting 3 3 3 3 Draft and submit Evaluation Workplan and Protocol prior to arrival in Uganda 2 2 2 2 3 Preparation for Team convening in￾country 2 4 Travel to country 2 2 2 5 Team Planning Meeting 3 3 3 3 3 6 In-brief with Mission 1 1 1 1 1 7 In-brief with project 0.5 0.5 0.5 0.5 0.5 8 Data Collection DQA Workshop (protocol orientation for all involved in data collection) 2 2 2 2 9 Prep / Logistics for Site Visits 0.5 0.5 0.5 0.5 2 10 Data collection / Site Visits (including travel to sites) 15 15 15 15 15 11 Data analysis 5 5 5 5 3 12 Debrief with Mission and PEPFAR Country team, with prep 1 1 1 1 1 13 Stakeholder debrief workshops with MEEPP and MOH, with prep 1 1 1 1 1 14 Depart country 2 2 2 15 Draft report(s) 5 4 4 4 1 16 GH Pro Report QC Review & Formatting 17 Submission of draft report(s) to Mission 18 USAID Report Review 19 Revise report(s) per USAID comments 3 2 2 2 20 Finalize and submit report to USAID 21 508 Compliance Review 22 Upload Eval Report(s) to the DEC Total LOE per person 52 49 49 45 30 Total LOE 52 49 49 45 30 If overseas, is a 6-day workweek permitted Yes No Travel anticipated: List international and local travel anticipated by what team members. MEEPP Project office is in Kampala and there will be site visits conducted in a sample of districts all over Uganda. It is anticipated the Team will split into two groups to visit 8 districts, covering high and low volume facilities in scale up and aggressive scale up districts. This will include USAID and CDEC supported facilities/districts and IPs. XV.LOGISTICS Note: Most Evaluation/Analytic Teams arrange their own work space, often in their hotels. However, if Facility Access is preferred GH Pro can request it. GH Pro does not provide Security Clearances. Our consultants can obtain Facility Access only. Check all that the consultant will need to perform this assignment, including USAID Facility Access, GH Pro workspace and travel (other than to and from post). 50 MEEPP II EVALUATION  USAID Facility Access Specify who will require Facility Access:  Electronic County Clearance (ECC) (International travelers only)  GH Pro workspace Specify who will require workspace at GH Pro:  Travel -other than posting (specify): GH Pro Evaluation team will arrange their own in-country transportation needed for data collection and meetings  Other (specify): XVI.GH PRO ROLES AND RESPONSIBILITIES GH Pro will coordinate and manage the evaluation/analytic team and provide quality assurance oversight, including:  Review SOW and recommend revisions as needed  Provide technical assistance on methodology, as needed  Develop budget for analytic activity  Recruit and hire the evaluation/analytic team, with USAID POC approval  Arrange international travel and lodging for international consultants  Request for country clearance and/or facility access (if needed)  Review methods, workplan, analytic instruments, reports and other deliverables as part of the quality assurance oversight  Report production - If the report is public, then coordination of draft and finalization steps, editing/formatting, 508ing required in addition to and submission to the DEC and posting on GH Pro website. If the report is internal, then copy editing/formatting for internal distribution. XVII.USAID ROLES AND RESPONSIBILITIES Below is the standard list of USAID’s roles and responsibilities. Add other roles and responsibilities as appropriate. USAID Roles and Responsibilities USAID will provide overall technical leadership and direction for the analytic team throughout the assignment and will provide assistance with the following tasks: Before Field Work  SOW. o Develop SOW. o Peer Review SOW o Respond to queries about the SOW and/or the assignment at large.  Consultant Conflict of Interest (COI). To avoid conflicts of interest or the appearance of a COI, review previous employers listed on the CV’s for proposed consultants and provide additional information regarding potential COI with the project contractors evaluated/assessed and information regarding their affiliates.  Documents. Identify and prioritize background materials for the consultants and provide them to GH Pro, preferably in electronic form, at least one week prior to the inception of the assignment.  Local Consultants. Assist with identification of potential local consultants, including contact information. MEEPP II EVALUATION 51  Site Visit Preparations. Provide a list of site visit locations, key contacts, and suggested length of visit for use in planning in-country travel and accurate estimation of country travel line items costs.  Lodgings and Travel. Provide guidance on recommended secure hotels and methods of in-country travel (i.e., car rental companies and other means of transportation). During Field Work  Mission Point of Contact. Throughout the in-country work, ensure constant availability of the Point of Contact person and provide technical leadership and direction for the team’s work.  Meeting Space. Provide guidance on the team’s selection of a meeting space for interviews and/or focus group discussions (i.e. USAID space if available, or other known office/hotel meeting space).  Meeting Arrangements. Assist the team in arranging and coordinating meetings with stakeholders.  Facilitate Contact with Implementing Partners. Introduce the analytic team to implementing partners and other stakeholders, and where applicable and appropriate prepare and send out an introduction letter for team’s arrival and/or anticipated meetings. After Field Work  Timely Reviews. Provide timely review of draft/final reports and approval of deliverables. XVIII.ANALYTIC REPORT Provide any desired guidance or specifications for Final Report. (See How-To Note: Preparing Evaluation Reports) The Evaluation/Analytic Final Report must follow USAID’s Criteria to Ensure the Quality of the Evaluation Report (found in Appendix I of the USAID Evaluation Policy). a. The report must not exceed 25 pages (excluding executive summary, table of contents, acronym list and annexes). b. The structure of the report should follow the Evaluation Report template, including branding found here or here. c. Draft reports must be provided electronically, in English, to GH Pro who will then submit it to USAID. d. For additional Guidance, please see the Evaluation Reports to the How-To Note on preparing Evaluation Draft Reports found here. Reporting Guidelines: The draft report should be a comprehensive analytical evidence￾based evaluation/analytic report. It should detail and describe results, effects, constraints, and lessons learned, and provide recommendations and identify key questions for future consideration. The report shall follow USAID branding procedures. The report will be edited/formatted and made 508 compliant as required by USAID for public reports and will be posted to the USAID/DEC. The findings from the evaluation/analytic will be presented in a draft report at a full briefing with USAID and at a follow-up meeting with key stakeholders. The report should use the following format:  Executive Summary: concisely state the most salient findings, conclusions, and recommendations (1-3 pages);  Table of Contents (1 page);  Acronyms  Evaluation/Analytic Purpose and Evaluation/Analytic Questions (1-2 pages)  Project [or Program] Background (1-3 pages)  Evaluation/Analytic Methods and Limitations (1-3 pages)  Findings 52 MEEPP II EVALUATION  Conclusions  Recommendations  Lessons Learned  Annexes - Annex I: Evaluation/Analytic Statement of Work - Annex II: Evaluation/Analytic Methods and Limitations - Annex III: Data Collection Instruments - Annex IV: Sources of Information o List of Persons Interviews o Bibliography of Documents Reviewed o Databases o [etc] - Annex V: Disclosure of Any Conflicts of Interest - Annex VI: Statement of Differences (if applicable) The evaluation methodology and report will be compliant with the USAID Evaluation Policy and Checklist for Assessing USAID Evaluation Reports -------------------------------- The Evaluation Report should exclude any potentially procurement-sensitive information. As needed, any procurement sensitive information or other sensitive but unclassified (SBU) information will be submitted in a memo to USIAD separate from the Evaluation Report. -------------------------------- All data instruments, data sets (if appropriate), presentations, meeting notes and report for this evaluation will be provided to GH Pro and presented to USAID electronically through the GH Pro Program Manager. All datasets will be in an unlocked, in an electronic format (CSV or XML). XIX.USAID CONTACTS Primary Contact Alternate Contact 1 Alternate Contact 2 Name: May Mwaka Fiona Waata Solome Sevume Title: Monitoring and Evaluation Specialist program Mangement Specialist – Monitoring and Evaluation Health Office Monitoring and Evaluation Specialist USAID Office/Mission Uganda/Program Office Uganda Uganda Email: mmwaka@usaid.gov fwaata@usaid.gov ssevume@usaid.gov Telephone: +256 414 306 518 +256 414 306 707 +256 414 306 706 Cell Phone: +256 772 138 529 +256 772 138 450 +256 772 138 525 List other contacts who will be supporting the Requesting Team with technical support, such as reviewing SOW and Report (such as USAID/W GH Pro management team staff) MEEPP II EVALUATION 53 Technical Support Contact 1 Technical Support Contact 2 Name: Diana Harper Title: Senior Evaluation and Program Advisor USAID Office/Mission Office of Policy, Planning and Programs USAID Bureau for Global Health Email: dharper@usaid.gov Telephone: 571-551-7086 Cell Phone: 571-228-3619 XX.REFERENCE MATERIALS Documents and materials needed and/or useful for consultant assignment, that are not listed above 54 MEEPP II EVALUATION ANNEX 1.1. SAMPLE EVALUATION DESIGN MATRIX Evaluation Question Sub question (will help you answer the key evaluation question) Indicator/ Performance Measure (information needed to answer the question Data Source (primary and or secondary) Data Collection Instrument Data Analysis Plan Comments Q1. Q2. Q3. Q4. MEEPP II EVALUATION 55 ANNEX 1.2. ILLUSTRATIVE EVALUATION REPORT OUTLINE Cover page (Title and date of the study, names of recipients and the evaluation team). Table of Contents List of Acronyms Executive Summary [Stand-Alone, 1-3 pages, summary of report. This section shall not contain any material not found in the main body of the report] Main Part of the Report USG prefers smaller documents approx. 20-25 pages. Additional details can go to annexes Introduction/Background and Purpose: [Overview of the evaluation. Covers the purpose and intended audiences for the study and the key questions as identified in the SOW) Evaluation Approach and Methods: [Brief summary. Additional information, including instruments should be presented in an Annex]. Findings: [This section, organized in whatever way the team wishes, must present the basic answers to the key evaluation questions, i.e., the empirical facts and other types of evidence the study team collected, including the assumptions]. Conclusions: [This section should present the team’s interpretations or judgments about its findings]. Recommendations: [This section should make clear what actions should be taken as a result of the evaluation]. Lessons Learned: [In this section the team should present any information that would be useful to people who are designing/manning similar or related new or on-going programs in Uganda or elsewhere. Other lessons the team derives from the study should also be presented here]. Annexes: [These may include supplementary information on the evaluation itself; further description of the data collection/analysis methods used; data collection instruments; summaries of interviews; statistical tables, and other relevant documents]. 56 MEEPP II EVALUATION ANNEX 1.3. CRITERIA TO CHECK THE QUALITY OF THE EVALUATION REPORT  The evaluation report should represent a thoughtful, well-researched and well organized effort to objectively evaluate what worked in the project, what did not and why.  Evaluation reports shall address all evaluation questions included in the scope of work.  The evaluation report should include the scope of work as an annex. All modifications to the scope of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer.  Evaluation methodology shall be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists and discussion guides will be included in an Annex in the final report.  Evaluation findings will assess outcomes and impact on males and females.  Limitations to the evaluation shall be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).  Evaluation findings should be presented as analyzed facts, evidence and data and not based on anecdotes, hearsay or the compilation of people’s opinions. Findings should be specific, concise and supported by strong quantitative or qualitative evidence.  Sources of information need to be properly identified and listed in an annex.  Recommendations need to be supported by a specific set of findings.  Recommendations should be action-oriented, practical and specific, with defined responsibility for the action. MEEPP II EVALUATION 57 ANNEX 2. EVALUATION METHODS AND LIMITATIONS Acronyms ADPs HIV/AIDS Development Partners CDC Centers for Disease Control and Prevention COP Country Operating Plan DHO District Health Officer DO Development Objective DOD Department of Defense FGD Focus group discussion GoU Government of Uganda HIV/AIDS Human immunodeficiency virus infection/acquired immunodeficiency syndrome IP Implementing Partner IR Intermediate Result KII Key informant interview M&E Monitoring and Evaluation MEEPP Monitoring and Evaluation of the Emergency Plan Progress MGLSD Ministry of Gender, Labour and Social Development MoH Ministry of Health NGO Non-governmental organization OGAC The Office of the Global AIDS Coordinator OVC Orphans and Vulnerable Children PEPFAR U.S. President’s Emergency Plan for AIDS Relief PI Performance improvement PM Performance management PMP Performance management plan PMMP Performance Monitoring and Management Plan PNFP Private not for profit SOW Scope of Work SSS Social and Scientific Systems TWG Technical working group UAC Uganda AIDS Commission USG United States Government WHO World Health Organization 58 MEEPP II EVALUATION 1. INTRODUCTION The Monitoring and Evaluation of the Emergency Plan Progress (MEEPP II) project is a 5-year contract funded by the President’s Emergency Plan for AIDS Relief (PEPFAR) through the United States Agency for International Development (USAID). The project is implemented by Social and Scientific Systems (SSS) Inc. Following the successful completion of MEEPP I (2005- 2010), SSS was awarded the second MEEPP contract which is implemented as MEEPP II. The overall goal of the 5-year MEEPP II project is to strengthen the human immunodeficiency virus infection/acquired immunodeficiency syndrome (HIV/AIDS) -related national monitoring and evaluation (M&E) systems and support the collection of complete, accurate, and timely data that can meet government of Uganda, PEPFAR, Global Fund and other development partner data requirements. The primary objective of MEEPP II is to shift from a vertical and comprehensive PEPFAR Implementing partner (IP)-based program performance management system and strengthen the existing national and district-level M&E systems to support data needs of the key HIV/AIDS related stakeholders. These stakeholders include the Ministry of Health (MOH), Ministry of Gender, Labour and Social Development (MGLSD), PEPFAR, Global Funds, UNAIDS, WHO, UNICEF and other development partners. To achieve this goal, MEEPP II has developed and implemented collaborative strategies engaging PEPFAR USG agencies and their respective implementing partners (IPs), the USG activity managers, as well as GOU national-level stakeholders and Uganda AIDS Commission (UAC). MEEPP II’s original contract was underwent two cost extensions, and is now expected to end by December 2016. It is expected that by December 2016, MEEPP-II project will have significantly contributed to a strengthened national HIV/AIDS M&E systems allowing Government of Uganda (GOU), PEPFAR, the Global Fund and other development partners to meet most of their data needs from these systems. To this extent, USAID requested a performance evaluation of MEEPP II. The purpose of this evaluation is to comprehensively analyze MEEPP-II’s performance towards supporting PEPFAR data and analytical needs, and to assess progress in the transition from reliance on a project￾managed database, to the Government of Uganda (GOU) systems for reporting PEPFAR interventions. The findings of this evaluation are expected to contribute to USG decision on the level and type of support to further strengthen the GOU data system for PEPFAR reporting and other national needs. Specifically, the evaluation provides an opportunity for USG to identify gaps, including strengths and weaknesses, and gather evidence upon which a post-MEEPP II strategy can be based. 2. CONTEXT – MEEPP II PROJECT OVERVIEW In 2010, a five-year contract was awarded to Social and Scientific Systems Inc. to implement the MEEPP II project. MEEPP II is the President’s Emergency Plan for AIDS Relief Monitoring and Evaluation (M&E) implementing partner contracted by USAID to support routine monitoring and evaluation of all PEPFAR funded HIV/AIDS activities in Uganda. MEEPP II’s overall strategic result is to strengthen national HIV/AIDS management information systems to ensure that most PEPFAR data needs are accommodated by the end of the project period. MEEPP II EVALUATION 59 MEEPP II’s strategic approach is guided by the current USG strategic information (SI) pivots, and the transition plan designed to guide the process towards one national M&E system. The SI pivots focus on:  Alignment of USG-supported systems with the national information systems;  Support robust M&E systems at service delivery points and districts;  Increase use of data for evidence-based planning and decision making at all levels; and,  Promote technological innovations to track referrals, linkages and retention of clients. MEEPP II has two major strategic objectives; 1. Improve PEPFAR data collection, management, analysis, and use in program planning Country Operating Plan (COP) and reporting to the office of the Global AIDS Coordinator (OGAC) and USG Agencies 2. Strengthen Uganda’s National M&E reporting for selected key indicators in the national HIV/AIDS Strategic Plan’s Performance Monitoring and Management Plan (PMMP) The contract called for MEEPP II to work closely with the various USG agencies; USAID, Centers for Disease Control and Prevention (CDC), State Department, and Department of Defense (DoD), implementing partners (IPs), Government of Uganda (GoU) as well as other HIV/AIDS development partners (ADPs) to provide technical assistance, training, and capacity building for monitoring and evaluation systems. Investments made under the MEEPP II contract are aimed at ensuring M&E capacity at both the national, regional, and service outlet levels for sustainability and to support accountability. To this effect MEEPP II was tasked with continuously analyzing data and providing feedback to the various stakeholders to influence program direction. In addition, it reviews its own project implementation strategies and adjusts them to meet ever changing needs. In 2015, the MEEPP II contract was modified extending the initial contract period from September 2015 to May 2016. Under the modification, the two major objectives were clarified to include transition of the PEPFAR program performance management system to the national systems, and to support GoU HIV/AIDS and orphans and vulnerable children (OVC) national monitoring systems in the context of the “three ones” principle. In May 2016, MEEP received a second extension through December 2016. 60 MEEPP II EVALUATION Figure A2.1: Results Framework National HIV/AIDS Related M&E Systems Strengthened to Accommodate National and Most PEPFAR Data Needs Intermediate Result 1 PEPFAR Data Quality Management for use in Program Planning & Reporting Improved Intermediate Result 2 Monitoring of District M&E capacity and performance strengthened Intermediate Result 3 Use of programmatic knowledge for decisions strengthened Data reported to PEPFAR of acceptable quality Quality and Utilization of HIV/AIDS related data by districts improved Feedback loop established for review of important programmatic insights and recommendation Standard operating procedures for PEPFAR data management and target setting strengthened PEPFAR data collection procedures aligned with national processes Monitoring of completeness and timeliness of HIV/AIDS related data submitted by districts improved IPs proficient in the use of national data collection tools Quality of special studies improved MEEPP Historical data standardized to serve USG and national analytical needs Evaluation Team A team of four experts, three international and one Ugandan professional will conduct the Performance Evaluation of MEEPP II as follows: Dr. Rachel Jean-Baptiste: Team Leader will be responsible for 1) providing team leadership, 2) managing the team’s activities, 3) ensuring that all deliverables are met on time, 4) serve as liaison between USAID and the evaluation/analytic team, and 5) leading briefings and presentations. She will provide the overall quality assurance on evaluation issues, including methods, development of data collection instruments, protocols for data collection, data management and data analysis. She will oversee the training of all engaged in data collection, insuring highest level of reliability and validity of data being collected. She is also the lead analyst, responsible for all data analysis, assuring that all qualitative and quantitative data are analyzed accordingly to meet the needs of this evaluation. Mr. Malik Jaffer: HIV/AIDS Program SI Specialist, will provide expertise in HIV SI systems and processes, particularly related to PEPFAR and national routine information systems. He, too, will conduct key informant interviews with stakeholders, including the Ministry of Health and other government officials, the USAID and MEEPP project management team, district management MEEPP II EVALUATION 61 team, and health facilities. He will be part of all planning and briefing meetings, data collection, data analysis, development of evaluation presentations, and writing of the evaluation report. Mr. Yusuf Hamba will be responsible for assessing the project’s activities and results within the greater Ugandan context. He will assess the project’s activities and Performance Management Plan as they relate to any HIS or HMIS Strategic Plans and other related documents, and will examine MEEPP’s interaction with and support to the private sector. Ms. Siobhan Green is the Management Information Systems (MIS)/Data Management Expert on the team. She will provide expertise in the transition process, and the functionality of the GoU system, and assessing the preparedness of the DHIS-2 and OVC MIS to serve as the primary source of data for PEPFAR activities. She is also responsible for identifying information needed to understand shifts in data quality and assess data use at district and facility levels, and serve as the team IT expert. She will participate in all briefing meetings, data collection, data analysis, development of evaluation presentations, and writing of the Evaluation Report. All four (4) team members will conduct Key Informant Interviews with stakeholders from PEPFAR, IPs, and the Government of Uganda, and will be involved in data collection in eight (8) districts. 3. PERFORMANCE EVALUATION PURPOSE EVALUATION PURPOSE This evaluation will analyze MEEPP II’s performance on supporting PEPFAR data and analytical needs, assess progress, gaps, and note strengths and weaknesses in the transition from PEPFAR’s reliance on project-managed databases to systems managed by the Government of Uganda, and provide recommendations to the USG for a post-MEEPP II strategy. EVALUATION QUESTIONS Specific questions that will guide this Performance Evaluation are stated in the Evaluation and were discussed in detail with USAID/Uganda and with the MEEPP II team, and are understood to be the following: 1) PEPFAR Reporting: To what extent has MEEPP II efficiently, consistently and routinely collected, analyzed and reported data for PEPFAR planning, project management and performance reporting? Areas to consider: o Lessons learned and best practices to inform future M&E capacity building efforts supporting PEPFAR implementation in Uganda 2) Transition (MEEPP II). To what extent have the data collection, data validation, analysis and reporting systems established by MEEPP II effectively transitioned to the national systems pivoted around DHIS-e and OVC MIS Areas to consider: o Evolution of MEEPP II’s role over time o Paths for processes that are still in transition 62 MEEPP II EVALUATION o PEPFAR indicators and their alignment with the national information systems 3) Transition (GOU). A. To what extent is the GOU prepared to make the transition into becoming the primary source for PEPFAR data? B. To what extent will the data be readily accessible by PEPFAR and other users? Areas to consider: o Lessons drawn from the transition process for obtaining PEPFAR data for USAID, post MEEPP II 4) Data Quality and Data Use. To what extent has MEEPP-II contributed towards: Timely acquisition of quality data? Use of data by health facilities, GOU, and USG Implementing Partners? Areas to consider: o MEEPP II’s contribution to a data driven learning environment and decision making within the Uganda HIV/AIDS program community 4. METHODOLOGY GUIDING PRINCIPLES This Performance Evaluation is taking place during an extension period of the MEEPP II project. As such, it will focus on the goals, objectives and subsequent implementation of the overall project over the past five years. The intent is not to document statistical impact of the project, since we do not have baseline with which to compare. Instead, our overarching guiding principle will be to focus on analyzing the efforts made by MEEPP II, the experience and engagement of stakeholders at the level of USG, GOU, and IPs, and to provide a critical analysis of efficiencies in what was done and what needs to be done to sustain and build on gains. We understand our task to be largely analytical in nature. As such, we propose two overarching principles to guide this evaluation: the systems strengthening principle, and the sustainability principle. The first, M&E systems strengthening, will analyze how MEEPP II operated within itself so that it delivered results to PEPFAR, as well as the various levels of support that MEEPP II provided to the GOU. Components of this system will include services that MEEPP II delivery and the requirements to provide such services including human resources and their capacity, logistics, technology and capacity building for its use, critical information needs, and accountability mechanisms that helped ensure quality of the data reported to PEPFAR and successful implementation of the project. This principle will guide our work to answer questions 1 and 4. The second principle, sustainability, will analyze the transition of MEEPP II’s services to the GOU. Building on existing work on sustainability done by USAID and the World Bank, the team will use a tool developed by Oxford Epi to assess the following components: This principle will guide our work to answer questions 2 and 3. MEEPP II EVALUATION 63 While MEEPP II may have had direct impact on certain recipients of its services, specifically the USG, IPs, and the Central GOU, and their results can be assessed using direct methods. However, MEEPP II’s work with the GOU at the level of district and below were through IPs. Thus the guiding principle of analyzing MEEPP II’s work at the level of district and below will be an analytic focus, though largely qualitative, on documenting the likely contribution or added value of MEEPP II towards the accomplishment of data use and improvement of quality. Lastly, the work of MEEPP II, particularly the transition of PEPFAR’s obtainment of data from MEEPP II’s project databases to the infrastructures of the GOU, and in particular the DHIS II, was largely a systems strengthening activity. Thus in analyzing the transition for what has happened and how well it was done, the guiding analytic principles will be systems strengthening (multiple components) and sustainability (multiple dimensions). Our team brings extensive experience using these principles and their tools to guide the successful evaluation of other USG funded projects. EVALUATION DESIGN The design of this evaluation is cross-sectional with regional stratification with multiple methods of data collection. The Team’s approach is inclusive and comprehensive. A systems strengthening framework will guide data collection, analysis and reporting on evaluation questions 1 and 4. Data collection will include the following: Review of Project documents, including MEEPP quarterly and annual reports, work plans, Performance Management Plans, MEEPP budget monitoring tool, evaluation briefing guide, among others. MEEPP II reports and planning documents have been provided. As additional relevant documents are identified, the evaluation team will request them and they will be reviewed. Review of these documents will allow the evaluation team to obtain a clear picture of what MEEPP has done, and some indication into strengths and weaknesses of the project. Building from this understanding, the Evaluation team will then conduct key informant interviews with MEEPP leadership and staff to document and analyze the MEEPP project from an M&E systems lens. Specifically, the interviews will allow MEEPPP staff to articulate the following for Question 1: 1) What services does MEEPP provide, to whom, and how (this includes DQAs); what is the added value? 2) Logistics and technologies required by MEEPP to provide the service(s) 3) Types of Human Resources MEEPP needed to provide the service, how they are recruited, retained, managed and funded 4) Type of information they find critical to their daily operations (e.g. PEPFAR indicator changes, etc.) and mechanisms in place to obtain it on a regular basis 5) Internal governance/accountability processes that allow them to succeed (or hinders more success) 64 MEEPP II EVALUATION This will allow the evaluation team to be clear on the structures and processes that enable or hinder success of MEEPP II. For Question 4, the following will be evaluated: A. Added value of MEEPP II towards timely acquisition of quality data for PEPFAR, IPs For this we will look at activities such as routine data collection, DQA, and training and capacity building provided by MEEP in the area of data quality and data use. B. Evidence of interaction with the data By looking at user logs for each system, we will be able to see who logs into the DHIS II, and when is this done. We may also be able to conduct a frequency analysis to see if the database is used more often around certain times of the year (potential indication of data use for planning). C. When data is used, what data is used, and how is it used for decision making By discussing with various types of users of DHIS II throughout the country, we will collect this information through key informant interviews. To answer Questions 2 and 3, data will be collected based on a sustainability framework with the following key pillars: A. A.SHARED UNDERSTANDING of the goals and objectives of the transition from MEEPP II to GoU B. B. OWNERSHIP AND RESPONSIBILITY of the GoU for achieving the goals of the transition C. C.POLICY ENVIRONMENT that supports the transition, or governmental responsibility as primary source of data for all, including donors; that highlights expected quality of data; D. ADVOCACY CAPACITY (including capacity for strategic planning and decision making, access to decision makers, etc.) E. STAFFING (number and type required to deliver the data services; capacity building plans; etc.) F. ACCESS TO TECHNOLOGY NECESSARY TO THE AIM (includes meds, supplies, as well as IT for data management) G. FINANCIAL VIABILITY (outside of support from MEEPP II or other donor funding) H. FINANCIAL MANAGEMENT SYSTEMS (ability to fundraise, account for and manage funds on their own) I. RECOGNITION AND SUPPORT J. INFRASTRUCTURE (Organizational, Physical, Informational) K. PUBLIC IMAGE/TRUST (both from the Government's perspective, and from private sector) This Sustainability Framework builds on previous work done by USAID in the area of governance. Oxford Epi has successfully used data from this framework to other USAID MEEPP II EVALUATION 65 evaluations. Given these evaluation questions, and the Team’s understanding of the needs of USAID/Uganda Mission, this framework will facilitate the collection of the right data. These may include but are not limited to review of a) costed strategic plan for the implementation of the GOU HIS; b) HIS or HMIS policy guidelines; c) district-level plans for HIS and supervision of related workforce performance; supportive supervision; and, d) training plans for improving quality and efficiency of data collection and analysis at all levels within the GOU. These may also include policy reviews. In order to answer the questions around future access to data, we need to understand the current and potential future policy frameworks, including any MOUs signed between US Government and GoU, legal frameworks around access to data (such as the Access to Information Act), and international agreements (such as Uganda’s non￾involvement in the Open Government Partnership and the International Aid Transparency Initiative) Database reviews Using a mix of qualitative and quantitative methods, following three checklists (attached), the team will review the two databases (DHIS2 and OVCMIS) and any other relevant systems used by the MOU, MEEPP, or other IPs, looking at the following as how the factor impacts sustainability: 1. Software quality 2. Privacy, security, and risk management 3. Database management (planning and execution) 4. Interactions between data collection systems 5. Data modeling within the databases 6. Data report output Table 1 provides an overview of data collection, methods, tools and stakeholders involved in answering each question. 66 MEEPP II EVALUATION Table A2.1: Overview of Data Collection Question Measuring Method Tools Stakeholders to be interviewed To what extent has MEEPP II efficiently, consistently and routinely collected, analyzed and reported data for PEPFAR planning, project management and performance reporting? Efficiency: -processes that facilitated data collection, validation, and analysis (data consolidation, aggregation, manipulation, number of systems used/steps taken to get to reportable data); and procedures or steps taken to develop reports (routine and special reports) -Logistics and technologies required by MEEPP to provide the service(s) -Types of Human Resources MEEPP needed to provide the service, how they are recruited, retained, managed and funded -Type of information MEEPP II found critical to their daily operations (e.g. PEPFAR indicator changes, etc.) and mechanisms in place to obtain it on a regular basis - Internal governance/ accountability processes that allow them to succeed (or hinders more success) Routinely: establishment of regular procedures for data services; SOPs; Consistency: Timeliness of Project Reports Project document review Key Informant Interviews Document Checklist Structured interview guides MEEPP, USAID MEEPP STAFF IPs USG 2. Transition (MEEPP). To what extent have the data collection, data For DHIS II: Project document review Key Informant Interviews Document Checklist MEEPP II staff IPs MEEPP II EVALUATION 67 Question Measuring Method Tools Stakeholders to be interviewed validation, analysis and reporting systems established by MEEPP effectively transitioned to the national systems pivoted around DHIS-II and OVC MIS 1) What was done: catalogue of activities done by MEEP to transition data services to GoU 2) Was it completed (all PEPFAR indicators can be obtained from DHIS)? 3) Does it meet standards for efficiency, consistency, and routine data collection, analysis and reporting to serve the needs of PEPFAR and other donors? 4) Does it meet standards for data consistency and usefulness for the GoU? 5) Are stakeholders satisfied with MEEPP performance during the transition? For MIS OVC: 1) What was done: catalogue of activities done by MEEP to transition data services to GoU 2) Was it completed (all PEPFAR indicators can be obtained from DHIS)? 3) Does it meet standards for efficiency, consistency, and routine data collection, analysis and reporting to serve the needs of PEPFAR and other donors? 4) Does it meet standards for data consistency and usefulness for the GoU? Focus Group Discussions (DHIS II TWG; USG M&E TWG) Database review (DHIS, OVC MIS) Web logs Online Survey on satisfaction of stakeholders Direct observation of improved processes by districts and sites, matched with what MEEPP II says should happen; structured interview guides Focus group Discussion guide Use and User Analysis protocol Software quality Checklist Risk Management Checklist Data quality and output checklist Stakeholder satisfaction survey GoU (central GoU District Health facility data staff) Private sector health facilities DHIS II TWG USG M&E TWG Technologies: -DHIS II -OVC MIS 68 MEEPP II EVALUATION Question Measuring Method Tools Stakeholders to be interviewed 5) Are stakeholders satisfied with MEEPP performance during the transition? 3.Transition (GOU). To what extent is the GOU prepared to make the transition into becoming the primary source for PEPFAR data and others? A. SHARED UNDERSTANDING of the goals and objectives of the transition from MEEPP II to GoU B. OWNERSHIP AND RESPONSIBILITY of the GoU for achieving the transition C. POLICY ENVIRONMENT that supports the transition, or governmental responsibility as primary source of data for all, including donors; that highlights expected quality of data; D. ADVOCACY CAPACITY (including capacity for strategic planning and decision making, access to decision makers, etc.) E. STAFFING (number and type required to deliver the data services; capacity building plans; etc.) F. ACCESS TO TECHNOLOGY NECESSARY TO THE AIM (includes meds, supplies, as well as IT for data management) G. FINANCIAL VIABILITY (outside of support from MEEPP II or other donor funding) For GoU: -At central level: Document review (HMIS policies and strategic plans, web logs for DHIS II and OVC MIS, SOPs) - Database review (DHIS II, OVC MIS) -Key Informant Interviews At Region: Key informant interviews At Districts: Focus Group Discussions -direct observations of District HMIS using DHIS II At health facility -direct observation of preparation of report for DHIS II Key informant interviews Document Checklist Policy review checklist -Software Quality Checklist -Risk Management Checklist -Use and User Analysis protocol -Data quality and output checklist Structured interview guides Structured interview guides Focus group Discussion guide Observation checklist GoU (central GoU District Health facility data staff) MEEPP II staff IPs MEEPP II EVALUATION 69 Question Measuring Method Tools Stakeholders to be interviewed H. FINANCIAL MANAGEMENT SYSTEMS (ability to fundraise, account for and manage funds on their own) I. RECOGNITION AND SUPPORT J. INFRASTRUCTURE (Organizational, Physical, Informational) K. PUBLIC IMAGE/TRUST (both from the Government's perspective, and from private sector) (districts HIS, USG M&E TWGs) Survey online -target all District HMIS and biostatistics personnel within the regions of work to explore preparedness and transfer; Online survey of all Medical directors of district hospitals and health centers to explore data use For PEPFAR: -Key informant interviews with USG -Key informant interviews with IPs -Focus group discussion with M&E TWG or equivalent For other donors in the community Focus group with DHIS II TWG (assuming multi-donor representation) 70 MEEPP II EVALUATION Question Measuring Method Tools Stakeholders to be interviewed Key informant interviews with Global Fund and other donors 4. Data Quality and Data Use. To what extent has MEEPP-II contributed towards Timely acquisition of quality data? Use of data by health facilities, GOU, and USG Implementing Partners? M&E Systems Strengthening -Delivery of data services -HR capacity -Technologies DHIS II -Added value of MEEPP II towards timely acquisition of quality data for PEPFAR, IPs -routine data collection -DQAs -training Evidence of interaction with the data (user logs from each system) -when data is used, what data is used, and how is it used for decision making -extent to which MEEPP contributed to meeting the data needs of health facilities, GoU, USG and IPs being met -extent to which MEEPP ensured that data was available in usable format to health facilities, GoU, USG and IPs being met Qualitative summary of stakeholder perceptions of added value of MEEPP towards timely acquisition of quality data Quantitative analysis of data use (who uses the data, what data are they using, where is data use happening, when does it happen, how does it happen, and for what reason is data used) MEEPP II EVALUATION 71 Question Measuring Method Tools Stakeholders to be interviewed -extent to which MEEPP ensured ease of integrating new indicators and other changes OVC MIS --Added value of MEEPP II towards timely acquisition of quality data for PEPFAR, IPs -routine data collection -DQAs evidence of interaction with the data (user logs from each system) -when data is used, what data is used, and how is it used for decision making -extent to which MEEPP contributed to meeting the data needs of health facilities, GoU, USG and IPs being met -extent to which MEEPP ensured that data was available in usable format to health facilities, GoU, USG and IPs being met -extent to which MEEPP ensured ease of integrating 72 MEEPP II EVALUATION Question Measuring Method Tools Stakeholders to be interviewed new indicators and other changes -evidence of interaction with the data (user logs from each system) -when data is used, what data is used, and how is it used for decision making MEEPP II EVALUATION 73 DATA COLLECTION PLAN Data collection tools, including Key Informant Interviews, FGD guides, questionnaires and checklists designed for this evaluation will examine and verify the completeness and effectiveness of certain project activities along with country ownership of these activities. These instruments include:  Document review checklist: A checklist of key evaluation elements to check for in document reviews  Policy Review Checklist: for the review of privacy and data use policies that may currently or in the future impact the ability of PEPFAR or other partners to access DHIS2 or other relevant data collected by GoU.  Key Informant Questionnaires (customized by audience) o GOU o PEPFAR o IP o MEEPP o Health Facility o District Management Team  Software Quality Checklist (for DHIS2 and OVC MIS)  Risk management checklist (for DHIS2 and OVC MIS) Data Collection, Storage, and Reporting Checklist (for DHIS2 and OVC MIS) Use and User Analytics protocol: (for DHIS2 and OVC MIS) to see who is actually using the system and for what purpose. Key Information Interviews (KII) using structured interview guides: These will be tailored to the group of individuals being interviewed to elicit information to: a) validate and, where possible, verify project approaches, interventions, and achievements and their current technical and strategic appropriateness; b) secure opinions and perceptions of project implementation effectiveness and issues, and identifying gaps in project activities; c) obtain first-hand reports on training received, data systems changed, and overall capacity building in data collection, analysis, validation, reporting and use; d) determine how stakeholders and beneficiaries interact with the project, regarding issues of leadership, ownership, partnership, and collaboration; and, e) determine how the project has enabled change in the area of use of quality data for decision making at IP and at all levels of the GOU. KIIs will be conducted with the following groups of people: 1. MoH officials; 2. District level health officers, management teams, service commissions; 3. MEEPP II staff in Uganda; 4. Technical support partners; 5. USAID; CDC, DOD, Peace Corps 6. Other donors, including the Global Fund Focus Group Discussions (FGDs) also using structured guides will focus on aspects of their work environment as it relates to data collection, validation, analysis, reporting and use. To strengthen understanding of these issues with the DHIS II, focus groups are anticipated for health facilities, district management HMIS teams, DHIS II TWG, USG M&E TWG. We will also conduct FGDs with the OVC SI TWG within districts to document their perceptions on the 74 MEEPP II EVALUATION functionality of the OVC MIS database and its ability to support their role in enhancing quality of care for the OVCs. Online survey data entry will be web-based, and in some instances (i.e. interviews with district officers), the interview will be conducted by phone. The following surveys are anticipated: chief of party survey (including perceived added value of MEEPP II), and a survey of district planners and biostatisticians to assess preparedness at the district level for DHIS II to be the primary source of data for PEPFAR and other donors. We will also conduct the survey for CSO’s that provide care and support to OVCs in the different regions since these are the direct beneficiaries of the OVC MIS data. For data collected using the web-based tools, all (100%) eligible bodies (IPs and districts) will be invited to participate. Data collection instruments are included in the Annex. Data Map: The Team will also map all aspects in data lifecycle, from collection to storage to output and measure sustainability along the lifecycle in order to identify holes or promising practices. Figure A2.2. Example of high-level data mapping of typical USAID M&E data SAMPLING STRATEGY Selection Criteria for regions: We selected two districts from each of the four main regions of Uganda: Central, Western, Eastern, and Northern regions in order to ensure geographic coverage. MEEPP II EVALUATION 75 Selection criteria for districts for online surveys: All districts included; Selection Criteria: Districts and sites for in-depth review Based on guidance from USG during in-brief, we initially selected all high volume districts (n = 9). From those, we selected four that were priority districts for PEPFAR. We then looked at all other districts, and divided them into sustained and transitioning, based on whether or not the labeling of the majority of health facilities within that district. From this list, we selected districts with the highest number of health facilities – two for sustained, and two for transitioned. Within each district, we selected the highest volume sites, limiting ourselves to three sites per district. Using this methodology, a sample of 18 sites include a mix of USG donors (8 USAID, 8 CDC, 1 State, and 1 DOD), a mix of priority sites (12 priority, 4 transitioned and 2 sustained sites), and 8 hospitals, 2 health center 4’s, 6 health center III’s, and 2 health center IIs that are a mix of private, NGO, and government sites. These are summarized in Table 2. Table A2.2: Data Collection Regions, Districts, and Sites Region District DHIS 2 Health Facility DHIS2 Health Facility level DHIS2 Health Facility Ownership High or Low Volume site (Based on the criteria of high volume site is one which has 500 or more Care Clients AGENCY Priority/transition/Maint Clin Care IP Names Eastern Region Jinja District Family Hope Center Jinja HC III Private Not For Profit High Volume CDC PRIORITY Children AIDS Fund (CAF) Eastern Region Jinja District TASO Jinja CLINIC HC II NGO High Volume CDC PRIORITY TASO/HIV/AIDS & TB/PCT Eastern Region Jinja District Jinja Regional Ref HOSPITAL RR Hospi tal Government High Volume USAI D PRIORITY URC/SUSTAIN (TREAT FOLLOW ON) Eastern Region Bulambul i District Muyembe HC IV HC IV Government High Volume USAI D TRANSITI ON MSH/STAR-EASTERN Central Region Nakason gola District Nakasongola HC IV HC IV Government High Volume CDC MAINTE NANCE/S USTAINE D MILDMAY/COMPREHENSIV E CENTRAL Central Region Kampala District Kisenyi HC III HC III Government High Volume CDC PRIORITY IDI/COMPREHENSIVE KAMPALA CITY COUNCIL Central Region Kampala District Mulago National Hospital- MJAP ISS Clinic Clinic NGO High Volume CDC PRIORITY MUFM/MJAP Central Region Kampala District Alive Medical Services HC III HC II Private Not For Profit High Volume CDC PRIORITY Children AIDS Fund (CAF) Wester n Region Isingiro District Kabuyanda HC IV HC IV Government High Volume USAI D PRIORITY EGPAF/STAR-SW Wester n Region Isingiro District Rwekubo HC IV HC IV Government High Volume USAI D PRIORITY EGPAF/STAR-SW 76 MEEPP II EVALUATION Wester n Region Isingiro District Nakivale HC III HC III Government High Volume STAT E PRIORITY STATE - UNHCR Wester n Region Kisoro District Kisoro HOSPITAL Gene ral Hospi tal Government High Volume USAI D MAINTE NANCE/S USTAINE D EGPAF/STAR-SW Northe rn Region Kitgum District St. Joseph'S Kitgum HOSPITAL Gene ral Hospi tal Private Not For Profit High Volume CDC PRIORITY Uganda Episcopal Conference (Uganda Catholic Medical Bureau) UCMB Northe rn Region Kitgum District New Life Health Centre HC II HC II Private Not For Profit High Volume CDC PRIORITY Uganda Protestant Medical Bureau Northe rn Region Kitgum District Kitgum HOSPITAL Gene ral Hospi tal Government High Volume USAI D PRIORITY URC/QAP/HCI (ASSIST) Northe rn Region Pader District 5Th Military Division HOSPITAL Gene ral Hospi tal Government High Volume DOD TRANSITI ON DOD/UPDF Northe rn Region Pader District Atanga HC III HC III Government High Volume USAI D TRANSITI ON URC/QAP/HCI (ASSIST) Northe rn Region Pader District Puranga HC III HC III Government High Volume USAI D TRANSITI ON URC/QAP/HCI (ASSIST) Selection criteria for data collected through Key Informant Interviews: To conduct the qualitative interviews of key informants at the central level, sampling will be by convenience, as the list of key informants will be generated by USAID. If, during the course of our interaction, we learn of an additional person we should interview, their name will be added to the list. Persons involved in key informant interviews outside of Kampala will be selected using a multi￾tiered stratified random sampling method based on available data of PEPFAR supported sites. We will begin by randomly selecting one region each in the north, south, east and western parts of the country. Within each region, we will randomly select two districts using probability proportional size sampling strategy (PPS), for a total of eight districts. Within each district, we will collect data from the district management team and the district hospital. We will then randomly select 1 health center in each level (II, III, and IV). In total, the sample will include 4 regions, 8 districts management teams, 8 district hospitals, 16 health center IIs, 16 health center IIIs, and 16 health center IVs, for a total of 48 sites. This sampling strategy will allow us to have geographic representation that is selective towards higher volume facilities, as requested by USAID. DATA ANALYSIS The data will be analyzed on the basis of the four major questions of the evaluation. Using the frameworks described above, the data analysis of each question will be summarized by key themes or domains within the framework that applies to them. Both quantitative and qualitative methods will be used, and wherever possible, the data will be disaggregated by gender. Quantitative data will be captured and analyzed in real time using DatAdventure, a software developed by Oxford Epi for such work. Real time analyses to be produced will include graphs and pie charts of questions. Access to the real time data will be given to the evaluation COR at MEEPP II EVALUATION 77 USAID, and she will be able to review results in real time. The raw data will be exportable into Excel tables. The Team will use DatAdventure to provide graphs of descriptive statistics. The data will be exported into excel, and imported into SPSS to conduct additional analyses, including calculations of statistical differences where necessary by region, and by central vs. district. Qualitative data will be analyzed using a Qualitative Data Analysis Matrix. The team will analyze the emergence of opinions, perceptions, and issues. The data will be synthesized to determine recurrent themes and issues. Where appropriate these data will be presented in tables. Quantitative and qualitative data will be analyzed on the basis of the four (4) major questions of the Performance Evaluation. This analysis will be further enhanced by examining the data within the context of the domains within the four evaluation questions. Table A2.3. Overview of Data Analysis Evaluation Question Analytic Framework Key Themes Data Analysis Method 0. Descriptive characteristics of respondents in evaluation data collection Geographic equity -age -gender -geographical spread -type of stakeholder -geographic spread of data resources (HR, logistics, etc.) Quantitative descriptive data: -mean age -% female -% per regions (east, west, north, south) -%Stakeholder: USG, IP, GoU Central, GoU district, GoU health facility (type), private health facilities 1. To what extent has MEEPP II efficiently, consistently and routinely collected, analyzed and reported data for PEPFAR planning, project management and performance reporting? Systems Strengthening Efficiency: processes, like data collection and data analysis (data consolidation, aggregation, manipulation, number of systems used/steps taken to get to reportable data); procedures or steps taken to develop reports (routine and special reports) Consistency: Timeliness of Project Reports Routinely: maintaining the schedule Quantitative measure of number of on time reporting Quantitative stakeholder satisfaction data -Qualitative summary by theme, triangulated with quantitative data 2. (MEEPP). To what extent have the data collection, data validation, analysis and reporting systems established by Sustainability Framework With focus on service provision What did MEEP do to transition Service provision capacity -- GoU ability to provide data-related services (is it being done, how well?) Qualitative summary Qualitative summary describing whether or not this is being done 78 MEEPP II EVALUATION Evaluation Question Analytic Framework Key Themes Data Analysis Method MEEPP effectively transitioned to the national systems pivoted around DHIS-e and OVC MIS -data collection -data validation -data analysis -reporting systems Processes GoU engages in order to provide services Status, strengths and weaknesses of DHIS-II and OVC MIS Consistency of GoU processes to maintain Quality of service delivery by DHIS II and OVC MIS Efficiency of processes with focus on resource allocation￾consistency and sufficiency Use and User Analysis protocol (Siobhan) Quantitative measures, based on checklists, of DHIS II and OVC MIS Qualitative summary documenting consistencies, and highlighting reasons for inconsistencies, or inefficiencies if any are found Summary of resource requirements vs. resource gaps 3. Transition (GOU). To what extent is the GOU prepared to make the transition into becoming the primary source for PEPFAR data? Sustainability Framework Is the Govt Prepared? How prepared? geographic equity of preparation A. SHARED UNDERSTANDING of the goals and objectives of the transition from MEEPP II to GoU B. OWNERSHIP AND RESPONSIBILITY of the GoU for achieving the transition C. POLICY ENVIRONMENT that supports the transition, or governmental responsibility as primary source of data for all, including donors; that highlights expected quality of data D. ADVOCACY CAPACITY (including capacity for strategic planning and decision making, access to decision makers, etc.) V. STAFFING (number and type required to deliver the data services; capacity building plans; etc.) Analysis of similarities and deviations between USG and GoU understanding Quantitative measure of extent of agreement Qualitative summary of findings, noting differences, if any by central, regional, district, and health facilities Qualitative Policy Gap analysis to identify types of policies available and those missing but could help strengthen the policy environment for government leadership in data Quantitative analysis of results of survey questions on MEEPP II EVALUATION 79 Evaluation Question Analytic Framework Key Themes Data Analysis Method VI. ACCESS TO TECHNOLOGY NECESSARY TO THE AIM (includes meds, supplies, as well as IT for data mgt) VII. FINANCIAL VIABILITY (outside of support from MEEPP II or other donor funding) VIII. FINANCIAL MANAGEMENT SYSTEMS (ability to fundraise, account for and manage funds on their own) X. RECOGNITION AND SUPPORT XI. INFRASTRUCTURE (Organizational, Physical, Informational) XII. PUBLIC IMAGE/TRUST (both from the Government's perspective, and from private sector) advocacy capacity from producers, managers, and users of UG health data, separated by type of user. Triangulated with qualitative data from interviews with relevant key informants; HR Gap Analysis that identifies type of staff needed, and summarizes current vs. numbers and types needed Quantitative analysis of status, strengths and weaknesses of DHIS-II and OVC MIS Qualitative summary of explanatory factors Analysis of current funding sources, noting donor and amount. This analysis will include contributions from the GoU Analysis of current flow of funds for data management services (documenting, for example funds to support data HR, DHIS II software updates, physical computers, lap tops, etc., distribution of the same to districts; funds for internet access, etc.) Qualitative summary that 80 MEEPP II EVALUATION Evaluation Question Analytic Framework Key Themes Data Analysis Method documents existing recognition of good work with data (quality data collection, analysis, reporting and use) within the health system Quantitative summary of survey data on extent to which DHIS II and OVC MIS are trusted sources of data for making decisions, disaggregated by health facility, district, provincial, and central levels, by IPs and USG, by members of the DHIS II Technical Working Group, and by other donors. Qualitative summary of reasons for trust or lack of trust 4. Data Quality and Data Use. To what extent has MEEPP-II contributed towards timely acquisition of quality data? Use of data by health facilities, GOU, and USG Implementing Partners? Systems Strengthening -Data service deliver: data acquisition -HR capacity -Technologies DHIS II -Added value of MEEPP II towards timely acquisition of quality data for PEPFAR, IPs -routine data collection -DQAs -training Evidence of interaction with the data (user logs from each system) -when data is used, what data is used, and how is it used for decision making -extent to which MEEPP contributed to meeting the data needs of health facilities, GoU, USG and IPs being met -extent to which MEEPP ensured that data was available Qualitative summary of stakeholder perceptions of added value of MEEPP towards timely acquisition of quality data Quantitative analysis of data use (who uses the data, what data are they using, where is data use happening, when does it happen, how does it happen, and for what reason is data used) MEEPP II EVALUATION 81 Evaluation Question Analytic Framework Key Themes Data Analysis Method in usable format to health facilities, GoU, USG and IPs being met -extent to which MEEPP ensured ease of integrating new indicators and other changes OVC MIS --Added value of MEEPP II towards timely acquisition of quality data for PEPFAR, IPs -routine data collection -DQAs evidence of interaction with the data (user logs from each system) -when data is used, what data is used, and how is it used for decision making -extent to which MEEPP contributed to meeting the data needs of health facilities, GoU, USG and IPs being met -extent to which MEEPP ensured that data was available in usable format to health facilities, GoU, USG and IPs being met -extent to which MEEPP ensured ease of integrating new indicators and other changes -evidence of interaction with the data (user logs from each system) -when data is used, what data is used, and how is it used for decision making The final report will list each evaluation question followed by Findings, Analysis, Conclusions, and Recommendations section. The end summary will focus on priority issues for USAID to address and major lessons learned based on the answers provided in examining the four Evaluation Questions. This approach should aid the Team in finding gaps in the current activities and processes. Specifically, the Team will: 82 MEEPP II EVALUATION 1) Review MEEPP II reported achievements against the PMP and work plan; 2) Summarize commonalities related to the topics covered in the KIIs. Main topics will cover the project implementation process as depicted by the frameworks used; including: ownership; practicality; effectiveness; gaps; suggestions for improvements; and lessons learned. 3) Develop qualitative analysis tables highlighting results of discussions on key themes; 4) Develop quantitative analysis tables highlighting results from surveys 5) Develop summary descriptive statistics of all participants 6) Develop recommendations based on findings for each of the four questions ETHICAL CONSIDERATIONS The evaluation team will implement a policy of informed consent for all key informant interviews and focus group discussions (see Annex 3 for an example) and all interviews will be done on a voluntary basis. Interviewees will be given the option to opt-out of particular questions or the whole interview, if at any time they believe a response would contain sensitive information. Survey takers will also be provided a similar option for informed consent and participation. The information provided as part of these interviews and discussions will not be linked to any specific person in the Final Report and all information provided will be kept confidential and used for planning purposes only. Only general identifying information (organization, geographical unit, gender, and age if reported voluntarily) will be utilized. Any information that could be directly linked to an individual will not be used. Only members of the Evaluation Team will have access to the transcripts and raw data. The Final Report will be a synthesis of the Team’s analysis drawn from interviews from numerous respondents. Any included quotes to highlight particular issues will not include names. We will not collect data from minors, or directly from patient records. We will not collect any data with personally identifiable information throughout this process. LIMITATIONS OF THE EVALUATION Selection only of districts and health facilities that are geographically accessible is a limitation to understanding the extent to which MEEPP II transitioned activities equitably throughout the country. This is due to limited resources and short time frame, and as such, the data collected will not be generalizable to Uganda. However, several efforts have been made to correct for this shortcoming, including ensuring geographic spread of randomly selected districts using PPS (two each from east, west, north, southern parts of the country). In addition, the online surveys transcend boundaries due to geographical accessibility, and may provide additional insights. 5. PREPARATIONS FOR FIELD WORK The evaluation will be carried out by the Evaluation Team in cooperation with USAID and MEEPP II teams. To ensure quality of data collection, the evaluation team leader will establish clear guidelines for data collection, specifically for how to conduct interviews. The SI expert will brief team members on PEPFAR indicators, and processes for their analysis. The MIS expert will lead the review of databases for DHIS and OVC MIS. The Team is supported by a logistics specialist who will work from day 1 to set up appointments, and facilitate travel outside of Kampala. Given the few days available for data MEEPP II EVALUATION 83 collection, the Team is likely to split into two groups to more efficiently manage time and collect maximum data. The team may also employ part time consultants who may help with conducting phone interviews for the online data collection. Accordingly, the Performance Evaluation will include the following steps: 1) finalization of data collection tools; 2) formation of two data collection teams who will conduct visits to two regions each; 3) review and training on tools; 4) conducting the data collection with quality control checks interspersed; 5) data entry and analysis; and, 6) report writing. 6. TIMELINE AND DELIVERABLES Date Tasks and Deliverables 5/16/2016 Launch briefing with USAID/Uganda 5/19/2016 Launch briefing with MEEPP II 5/16-5/24/2016 Preparation of Evaluation Protocol and Timeline 5/24/2016 Submission of Evaluation Protocol and Timeline 5/25-5/30/2016 Desk review and preparation for travel 5/31-6/1/2016 Travel to Uganda 6/6/2016 Inbrief with Mission 6/6/2016 Inbrief with MEEPP II 6/7-6/8/2016 Incorporate comments and finalize evaluation protocol and timeline 6/8/2016 Submit revised protocol, timeline and tools to USAID 6/9/2016 Internal team preparation for DQA Workshop (AM) 6/9/2016 Hold DQA Workshop (PM) 6/1 – 6/11/2016 Logistics preparation for field visits and interviews, launching of online surveys, plans for follow up, etc. 6/13- 6/24/2016 Data collection 6/24-7/1/2016 Data analysis, drafting of the report 7/2-7/4/2016 Preparation for Mission debrief 7/5/2016 Mission debrief 7/6/2016 Stakeholder debrief workshop 7/7/2016 MIS experts leave Uganda 84 MEEPP II EVALUATION 7/7-7/10 Draft report, incorporate suggestions from Mission and stakeholders 7/9/2016 TL leaves Uganda 7/10/2016 SI Expert leaves Uganda 7/12-7/19 Draft report 7/20/2016 Submit report to GH PRO 7/21-7/27/2016 GH PRO reviews report 7/27/2016 GH PRO submits report to USAID 8/3/2016 USAID provides comments on report 8/4-8/10/2016 Revise report per USAID comments 8/11/2016 Submit revised report to GH PRO 8/15/2016 GH PRO submits final report to USAID 8/16/2016 USAID approves final report 8/25/2016 Finalized formatted report submitted to USAID 9/1/2016 USAID provides final approval on formatted report 9/16/2016 508 Compliance Evaluation report completed 9/30/2016 Upload Evaluation report to the DEC * This timeline does not include the weekly updates which will be provided by the Team to USAID. These weekly updates will also provide the Team the opportunity to request any reasonable assistance from USAID, if needed. MEEPP II EVALUATION 85 ANNEX 3. DATA COLLECTION INSTRUMENTS Key Informant Interviews with MEEP II Staff Question/Information Required 00 Date of interview: _____________/____________/_____________ 00 END LINE PERFORMANCE EVALUATION OF MEEPP II INFORMED CONSENT Thank you for giving us the time to speak with you today. My name is _________________, and my colleagues/notetaker: _____________________. As you know, we are here to as part of a team that is conducting an evaluation of the Monitoring and Evaluation of the Emergency Plan Progress II in Uganda. The purpose of this evaluation is to analyze MEEPP II’s performance supporting PEPFAR data and analytical needs, to assess progress, gaps, and to note strengths and weaknesses in the transition from PEPFAR’s reliance on project-managed databases to systems managed by the Government of Uganda. Our role will be to summarize our findings, and to provide recommendations to the USG for a post-MEEPP II strategy. The purpose of our discussion is to gain a better understanding of the databases you use and the role of MEEP II in providing you support as you use them. Using the evaluation questions as a guide, we will be asking you about 1) PEPFAR reporting, 2) transition of MEEPP II activities to GoU systems, 3) preparedness of the GoU for becoming the primary source of data for PEPFAR, and 4) MEEPP II contributions towards data quality and data use. If there are staff members who are particularly suited for specific questions, we would appreciate the opportunity to include them in the interview as part of the group or separately. All of the answers you provide will remain confidential, and will be summarized and included in our report. No information shared will be connected to an individual. Your participation is completely voluntary. You can choose to stop the interview at any time, or you may refuse to answer any questions. There will not be any negative consequences whatsoever. Do you have any questions before we begin? Do we have your consent to conduct the interview? [ ] Yes [ ] No 00 Name of Respondent____________________________ Age ______________ Gender [ ] Male [ ] Female Job Title______________________________________ Email: ____________@_____________.___________ Phone number: ________________________________ Name of Respondent____________________________ Age ______________ Gender [ ] Male [ ] Female 86 MEEPP II EVALUATION Job Title______________________________________ Email: ____________@_____________.___________ Phone number: ________________________________ Name of Respondent____________________________ Age ______________ Gender [ ] Male [ ] Female Job Title______________________________________ Email: ____________@_____________.___________ Phone number: ________________________________ Name of Respondent____________________________ Age ______________ Gender [ ] Male [ ] Female Job Title______________________________________ Email: ____________@_____________.___________ Phone number: ________________________________ Name of Respondent____________________________ Age ______________ Gender [ ] Male [ ] Female Job Title______________________________________ Email: ____________@_____________.___________ Phone number: ________________________________ Name of Respondent____________________________ Age ______________ Gender [ ] Male [ ] Female Job Title______________________________________ Email: ____________@_____________.___________ Phone number: ________________________________ Let’s begin by talking about the role of MEEP II. 01 What was the role of MEEP II in the beginning of your contract? (probe – what services were MEEP II offering, to whom, and how?) MEEPP II EVALUATION 87 02 Can you tell us a bit about the requirements of the transition? How did it come about? How did the processes for PEPFAR reporting change after the transition? 1 03 Since the requirement to transition, how has MEEP II’s role change over time with the transition? (probe: Did your services change with transition? How?) (Confirm transition changed contract core competency from M&E vs. Capacity building (by data lifecycle?) 1 04 What did MEEP II have to do in order to accommodate changes? (probe—management, human resources (recruitment, type, management, etc.), technologies, financial, logistics, internal QA, learning mechanisms, communications with PEPFAR, etc.) What type of information did MEEPP II find critical to their daily operations (e.g. PEPFAR indicator changes, changes in staffing at IP, district, or national etc.) and what mechanisms did they institute to obtain it on a regular basis? 1 88 MEEPP II EVALUATION What types of Human Resources did MEEPP II need to provide the service, how were they are recruited, retained, and managed? How did MEEP II maintain the capacity level of its staff? What internal governance and accountability processes has enabled/facilitated MEEP’s performance to date? 05 What is MEEP II’s relationship with IPs, districts and national levels? Who do you deliver services to, how? Frequency? (look at the diagram and dig in and probe changes in staffing and sites) 1 MEEPP II EVALUATION 89 06 What did MEEP II do during the transition to ensure that capacity for DHIS II was sufficiently built in districts not directly under your supervision? (Probe—role of other partners-PEPFAR or others??) 2 07 What aspects of the transition are not yet fully complete? (probe—plans/paths for processes that are still in transition) How much of HYBRD has transitioned to DHIS II? What parts are still in transition? And what is the plan? Are there aspects HYBRID or MEEP functions that cannot/will not be transitioned to the government (along the data lifecycle)? 2 90 MEEPP II EVALUATION 08 In your view, to what extent have you transitioned processes, or ways of work to the GoU and Districts? Examples? Which additional ‘ways of work’do you think would need to be transferred in order for the GoU and Districts to maintain the quality of the DHIS and OVC MIS? 09 To round off this conversation, how many of PEPFAR indicators are now able to be obtained through government systems? DHIS? _________________ OVC MIS? ______________ 2 10 On a scale of 1 to 10, with 1 being absolutely NOT, to 10 being full confidence, how much do you trust the data obtained in government systems compared to prior MEEPP II systems? |---|---|---|---|---|----|---|---|---| 1 2 3 4 5 6 7 8 9 10 2 11 Now let’s talk about the GoU systems, starting with the DHIS. How has MoH/DHIS II role in PEPFAR reporting evolved over the past 5 years? How engaged is the government in the transition? List by data cycle (collection, cleaning/validating, analyzing, reporting, etc) 3 MEEPP II EVALUATION 91 Starting from the primary data collection points (health facility records clerk?), What is the current data flow (via DHIS) to PEPFAR? 12 What processes are in place for the GoU to integrate new indicators (or change/delete old ones) based on PEPFAR changes? 13 How prepared do you think the GoU is to be the primary source of data for PEPFAR? Ministry of Health (DHIS): Ministry of Gender (OVC/MIS): On a scale of 1-10, with 1 being not at all, to 10 being complete, do you think that the GoU (Ministry of Health for DHIS, and Ministry of Gender for OVC/MIS): Share the same understanding with PEPFAR regarding their role in data? ___________ (MOH for DHIS) _________________ (MinoGen for OVC/MIS) Take full ownership and responsibility for the transition into becoming primary source of data for PEPFAR and other donors? ___________ (MOH for DHIS) _________________ (MinoGen for OVC/MIS) Has a policy environment supportive of their role as primary source of data for all stakeholders? ___________ (MOH for DHIS) _________________ (MinoGen for OVC/MIS) Has capacity to advocate for additional resources to support/maintain it as primary source of data for stakeholders? 3 92 MEEPP II EVALUATION ___________ (MOH for DHIS) _________________ (MinoGen for OVC/MIS) --in particular, what is the advocacy capacity within MIS teams/Levels to support MIS? Has sufficient staff numbers and types to ensure that it can successfully serve as the primary source of data for PEPFAR? Numbers: ___________ (MOH for DHIS) _________________ (MinoGen for OVC/MIS) Types: ___________ (MOH for DHIS) _________________ (MinoGen for OVC/MIS) Are staffing levels are appropriate to manage MIS systems and data collection needs? Yes/No, ___________ (MOH for DHIS) _________________ (MinoGen for OVC/MIS) why/why not? what changes to staffing are needed/have been completed recently to support the HMIS needs? What are the plans and processes for continuous recruitment, education, orientation, retention, and evaluation? Has sufficient access to technology to facilitate its role as primary source of data for PEPFAR? ___________ (MOH for DHIS) _________________ (MinoGen for OVC/MIS) - What is the access to computer, electricity, hardware/software, internet? And Who pays? MEEPP II EVALUATION 93 Has adequate financial resources outside of donor funding for staffing, Internet, training, other resources for HMISto allow it to become or remain the primary source of data for PEPFAR and others? ___________ (MOH for DHIS) _________________ (MinoGen for OVC/MIS) -who is paying the salary of the Biostatistician at each district? Has capacity to fundraise for additional finances if/when necessary to fulfill its role as primary source of data for PEPFAR and others? ___________ (MOH for DHIS) _________________ (MinoGen for OVC/MIS) Have mechanisms to recognize and reward good performance with regards to data ___________ (MOH for DHIS) _________________ (MinoGen for OVC/MIS) Has the physical infrastructure to be the primary source of data for PEPFAR? __________ ___________ (MOH for DHIS) _________________ (MinoGen for OVC/MIS) Has the organizational infrastructure (documented mechanisms for continual learning, staff recruitment and retention, communications with PEPFAR, internal QA, data validation processes, etc.) ___________ (MOH for DHIS) _________________ (MinoGen for OVC/MIS) ----what data validation techniques are being used? ---- What routine data analysis and reports are performed/created? 94 MEEPP II EVALUATION What is the level of public trust of GoU data? ___________ (DHIS) _________________ (OVC/MIS) Why do you think this perception is accurate/not accurate? DHIS: OVC/MIS: Now let’s talk about MEEPP II’s contribution to the quality and use of the DHIS 14 -what role did MEEPP II play to enhance the quality of: DHIS OVC/MIS A) Data collection B) Data validation C) Data analysis D) PEPFAR reporting E) Special reports 4 15 Is the DHIS data being used? for what? By whom? What has been MEEPP II’s role in promoting this use? 4 MEEPP II EVALUATION 95 16 Is the OVC/MIS data being used? For what? By whom? What has been MEEPP II’s role in promoting this use? 4 17 How would you describe MEEPP II’s added value to: IPs: Districts GoU: Central GoU: 4 96 MEEPP II EVALUATION 18 What have been major challenges experienced by MEEPP II in implementing its mandate? gen 19 Now that you have lived through the implementation of MEEP II and the transition, what would you have done differently if you could? gen 20 Do you have any other thoughts you would like to share? gen Thank you very much! If you think of anything else, please do not hesitate to get in touch with us. MEEPP II EVALUATION 97 MEEP II Endline Performance Evaluation￾Software Quality KII Guide System: [HYBRD, DHIS2, OVC MIS] Sources: [HISPUganda, DataCares, GoU MIS team, MEEP IT Team] Summary of System Platform: For all – who does what? (Government vs. contractor vs. other partner) Questions Answer History and contract: When was this project started and what are the contract terms? – by different functions? How do you interact with other partners (MEEP II, GoU MIS team, Contractors)? Permission model: How is access managed? What levels? Who controls access? How easy is this to change? Transition: What is the plan to transition between HYBRD and DHIS/OVC MIS? How do these systems interoperate? Interoperability: What other systems connect to this one? Data entry tools (data collection)? Report tools? What data structure standards do you follow? Can I see a data dictionary/taxonomy? Customer Support: Are there designated customer support personnel? How do users contact and communicate with customer support? Training and technical support: What training, guides, and tools are available? Report and data extraction: what are the standard reports and what are the types of data extraction? GIS? Flexibility: How easy is it to add new indicators? New report formats? Extract data in new ways? What is the data model/database format (i.e. star, flat, relational)? Usage and Usability tracking: can I get access to usage logs – looking at patterns of report downloads, data entry? Do you review those regularly? Software Maintenance: Who manages software maintenance? Walk me through your maintenance infrastructure QA: Is there an explicit process for quality assurance (of data and of code)? Who performs QA? Security/Privacy Protection: is there an explicit plan for security and privacy monitoring and responsiveness? 98 MEEPP II EVALUATION MEEPP II Endline Performance Evaluation HMIS Database Assessment Observation Checklist Updated: June 5, 2016 System: [HYBRD, DHIS2, OVC MIS] Sources: Site Demonstration/Usability testing Summary of System Platform: DATA ENTRY Data Entry YES NO Source Disaggregation Results Ease – how easy is it to enter data into the system from different formats 1 Formats for data entry (web from, excel/csv, third party) Document review, database review KII Role 2 Is there online help or guidance to provide support? Role 3 Is the user interface intuitive? Role 4 Ability to work offline or save and return later? Role Validation – is there data validation built into the data entry process 1 Acceptable ranges Document review, database review KII Role 2 Autocomplete/select boxes Role 3 Calculations Role 4 Preview before submission Role MEEPP II EVALUATION 99 DATA STORAGE Data Storage YES NO Source Disaggregation Results Integrity – Does data retain sufficient context, disaggregation, and associated information in the database to represent accurately in reports 1 How is the data model constructed? (flat, hierarchical, relational, star?) Document review, Database review, KII N/a 2 Outline data fields related to data collection and reporting N/a 3 List disaggregation and classification structure N/a DATA REPORTING Data Reporting YES NO Source Disaggregation Results Accuracy – Is the data presented in the reports an accurate reflection of data collected? 1 Quantitative (numbers are calculated accurately, aggregated data is “aggregatable”, confidence in numbers is accurately displayed 2 Visualizations accurately demonstrate assumptions and analysis (i.e. does not mislead) 3 Data quality addressed (reports do not contain data collection errors) Usefulness – Is the data in the report useful to the specific users 100 MEEPP II EVALUATION Data Reporting YES NO Source Disaggregation Results 1 Meets the data quality standards for the usage? 2 Used by users in other places (up stream report, official counts, performance management? Usability – 1 Does the data in the report come in different formats? 2 How easy is the data to use, as identified by users? 3 Does it require a lot of manual manipulation before it can be used? 4 Does the user need to contact software to get the data Flexible –How easy is it to update the reports? 1 To reflect new requirements 2 Ad hoc reporting 3 To reflect new indicators or structure MEEPP II EVALUATION 101 Key Informants Questionnaire- Explores Questions 1-4 (Various stakeholders including IPs, PEPFAR, GoU, DHIS II TWG members and donors) Question/Information Required 0 0 Date: ___________________________ Organization __________________ Name of respondent :___________________________________________ Age _____ Gender: M F Designation/Job Title: _______________________________________ 0 1 Let’s start the Interview by talking about the MEEPP II Project. They have been supporting PEPFAR reporting, and have been transitioning to national systems such as the DHIS2 and OVC MIS data bases in Uganda. Has your organization worked with or received support from MEEPP II? Yes No If YES, Please provide details of the type of interaction: _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ 0 2 To what extent has MEEP efficiently, consistently and routinely provided support to your organization in the following results areas? Please share some examples to support your views Result Area 1: Data quality management _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ Result Area 2: Monitoring of District M&E capacity and performance _________________________________________________________________________________________ _________________________________________________________________________________________ ________________________________________________________________________________________ Result Area 3: Use of programmatic knowledge for decision making __________________________________________________________________________________________________ __________________________________________________________________________________________________ __________________________________________________________________________________________________ __________________________________________________________________________________ 0 3 In your view, what are the three main achievements of MEEPP II? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ________________________________________________________________________________________ [ [ ] GoU Central [ [ ] GoU District [ ] GoU Health Facility (type______) [ [ ] IP [ [ ] USG/PEPFAR [[ [ ] MEEP Staff 102 MEEPP II EVALUATION 0 4 Are you aware of the transition required by PEPFAR from MEEPP be national systems pivoted around DHIS II and OVC MIS? [ ] Yes [ ] No In your view what do you think will enable the smooth operation and utilization of these two data bases by all stakeholders? What are some of the long term benefits of using the DHIS2 and OVC MIS data base? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ______________ 0 5 How has MEEP transferred capacity to national systems (DHIS II and OVC MIS teams) in the areas below and what suggestions do you have on how this can be improved? A) Data collection _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________________________________ B) Data validation __________________________________________________________________________________ __________________________________________________________________________________ _________________________________________________________ C) Data analysis __________________________________________________________________________________ __________________________________________________________________________________ __________________________________________________________________________________ __________________________________________________ D) Data reporting___________________________________________________________________________ __________________________________________________________________________________ __________________________________________________________________________________ _________________________________________________________ 0 6 What processes did the GoU put in place in order to provide the above services using DHIS II? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ __________________ 0 7 In your opinion, what are the strengths of DHIS II? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ What are the weaknesses of DHIS II? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ MEEPP II EVALUATION 103 0 8 What processes did the GoU put in place in order to provide the above services using OVC MIS? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ 0 9 In your opinion, what are the strengths of OVC MIS? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ What are the weaknesses of OVC MIS? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ __________________ 1 0 What can government do to strengthen the operation and utilization of DHIS2 and OVC MIS data base by the stakeholders in Uganda? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ 1 1 What do you think is the role of stakeholders in strengthening the operation and utilization of DHIS2 and OVC MIS data base? What suggestions do you have to enable stakeholder’s access to the DHIS2 and OVC MIS data in program planning and management? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ 1 2 On a scale of 1-10, how well do you understand the goals of PEPFAR for M&E in Uganda, and the intention of the transition? ___________ 104 MEEPP II EVALUATION What is your current understanding? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ 1 3 1 4 1 5 1 6 1 7 1 8 1 9 2 0 2 1 Can you share examples of how the GoU demonstrated ownership and responsibility for being the primary source of data for PEPFAR and other donors? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ Can you name GoU policies and strategic plans that position the GoU as the primary source of data for PEPFAR and other donors? Can you share thoughts on other policies that are needed to further strengthen the GoU in the area of data? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ Can you name GoU policies that speak specifically to data quality? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ How much of your data are you obtaining from the GoU national systems? [ ] All indicators [ ] Some indicators [ ] None of our indicators Can you share examples of how data collection, validation, analysis and reporting has featured in your strategic plans? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ If you are not satisfied with the data from the DHIS II or OVC MIS, who do you go to advocate? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ Let’s talk about staffing to facilitate and maintain the DHIS II and OVC MIS. What type of staff does the GoU need for the DHIS II? OVC MIS? Does the GoU currently have enough of each staff type? Are they spread out equitably across all districts? _________________________________________________________________________________________ MEEPP II EVALUATION 105 2 2 2 3 2 4 2 5 2 6 2 7 2 8 2 9 _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ Do the staff have the technology and support they need to access, use, and maintain the DHIS II? (e.g. laptops, internet (who pays?), smart phones, etc.?) Supervision of persons responsible for the data at health facilities? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ In the absence of any technical assistance, training or any other indirect financial support from MEEP, to what extent will the DHIS II continue to be useful? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ [district only] In the event that additional funding was made available to the districts, can you explain your funding management process at this district? What are the strengths? Weaknesses? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ Does the GoU have the skills to provide the same data services as MEEPP II? With the same efficiency? And achieve the same level of quality? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ [district only] What additional activities do you wish MEEP would do to further enhance skills and capacity so that you could continue to receive the same quality data from DHIS II once MEEP II is over? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ____________________ Is there a formal recognition system that the GoU uses or promotes to reward employees or districts who do well in the area of data (timely collection, validity, correct analysis, timely reporting, data use? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ___________________ 106 MEEPP II EVALUATION Within your organization, are the roles and responsibilities regarding DHIS II activities clear? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ___________________ Who does what? (and what are their qualifications?) _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ___________________ In your opinion, do you have sufficient office space, office equipment, etc to carry out activities related to DHIS II and reporting? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ___________________ -Do you share information with across other districts, or within the various groups in your district? [ ] Yes [ ] No -Why, why not? _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ ___________________ 3 0 3 1 Describe the three main data related challenges that Uganda health sector faces and for each propose some solutions Challenge 1:________________________________________________________ Solution:____________________________________________________________ Challenge 2:________________________________________________________ Solution:___________________________________________________________ Challenge 3:________________________________________________________ Solution:____________________________________________________________ Describe the three main data related challenges that Uganda OVC sector faces and for each propose some solutions Challenge 1:________________________________________________________ Solution:____________________________________________________________ Challenge 2:________________________________________________________ Solution:____________________________________________________________ Challenge 3:________________________________________________________ Solution:____________________________________________________________ 3 2 Please provide us with any other additional information on ways in which the data situation in Uganda can be improved. _________________________________________________________________________________________ _________________________________________________________________________________________ _________________________________________________________________________________________ MEEPP II EVALUATION 107 District Biostatistician – Web-based Tool Question/Information Required Date _______/_______/______ District Name Personnel type [ ] Biostatistician [ ] District Health Planner Thank you for giving us the time to speak with you today. We are here to learn about how the DHIS is providing data to donors. For PEPFAR, DHIS became the primary data source in 2013. We would like to know some information about how this is going. All of the answers you provide will be kept confidential, and will be summarized and included in our report. We will also be conducting interviews and focus group discussions with other stakeholders, and will be triangulating all findings. The final report will be shared with you through PEPFAR. Whenever we mention PEPFAR, we mean either USAID, CDC, DoD, State, or US Embassy. At any point, if you do not understand a question, please feel free to ask us for clarification. Do you have any questions before we begin? [ ] Yes [ ] No May we continue with the interview? [ ] Yes [ ] No Do you work with the District Health Information System (DHIS) [ ] Yes [ ] No Gen What is your role with the DHIS? ___________________________________________ Gen Is the data from your district’s DHIS reported to PEPFAR? [ ] Yes [ ] No [ ] Don’t know 2 Who is responsible for that? [ ] Me [ ] Someone else Who: _______________________ [ ] Don’t know 2 Are there standard operating procedures in place for preparing data for export? [ ] Yes [ ] No [ ] Don’t know IF YES, what are they? ___________________________________________ IF YES, do you use them? [ ] Yes [ ] No Who can give permission to access the data from your District? ___________________________________________ 2 108 MEEPP II EVALUATION What is the process for obtaining permission to access the DHIS data for your district? ___________________________________________ 2 How does PEPFAR access data from your district DHIS? [ ] Sent by district staff [ ] Downloaded by PEPFAR [ ] Data pulled directly via API 2 What is done to the data in the DHIS before it is ready to be used by PEPFAR? [ ] Data cleaning [ ] Data completeness analysis [ ] Data Quality Assessment (DQA) [ ] Other ______________________________ 2 Are you the one responsible for this? [ ] Yes [ ] No [ ] Don’t know 2 Do you receive any technical assistance from PEPFAR to help you get the DHIS data ready? [ ] yes [ ] No [ ] Don’t know 2 Do you receive any technical assistance from any other DONOR to help you get the DHIS data ready? [ ] Yes [ ] No [ ] Don’t know 2 IF YES, Who? [_____________________________________] 2 Is there a deadline by when the data needs to be ready for PEPFAR? [ ] Yes [ ] No [ ] Don’t know 2 IF YES, is this deadline the same as for the District or the national Government of Uganda? [ ] Yes [ ] No [ ] Don’t know 2 Do you know of any laws or policies that govern the DHIS/HMIS data in Uganda? [ ] Yes [ ] No 2 IF YES, can you tell us which laws or policy(ies) you know? ___________________________________________ 2 Have you received any training in Uganda data law or policies? [ ] Yes [ ] No 2 IF YES, Who provided the training? ___________________________________________ 2 Have you been trained in the Health Management Information Systems Procedures Manual? [ ] Yes [ ] No 2 Do you have a copy of the HMIS Procedures Manual at the district? [ ] Yes [ ] No 2 On a scale of 1 to 10, with 1 being absolutely NOT, to 10 absolutely YES, how would you rate the following: The district’s trust in the data obtained in the DHIS for making decisions for the district? [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 3 The district’s understanding that the DHIS is the primary source of data for PEPFAR [ ] 1 [ ] 2 [ ] 3 3 MEEPP II EVALUATION 109 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 District ownership and responsibility for the DHIS as the primary source of data [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 3 Policy environment at district is supportive of the DHIS as primary source of data [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 3 District has capacity to advocate for additional resources to support/maintain DHIS [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 3 The district has sufficient number of staff to ensure DHIS reliability as the primary source of data for PEPFAR [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 3 The district has sufficient type of staff to ensure DHIS reliability as the primary source of data for PEPFAR [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 3 110 MEEPP II EVALUATION You/your team have sufficient access to technology to ensure DHIS reliability as the primary source of data for PEPFAR [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 3 --Internet [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 3 -- computers [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 3 -- electricity [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 3 -- hardware [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 3 -- software [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 3 MEEPP II EVALUATION 111 [ ] 7 [ ] 8 [ ] 9 [ ] 10 The district has adequate financial resources outside of donor funding for staffing, Internet, training, other resources to ensure DHIS reliability as the primary source of data for PEPFAR [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 3 The district has capacity to fundraise for additional finances if/when necessary to ensure DHIS reliability as the primary source of data for PEPFAR [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 3 The district Has mechanisms to recognize and reward good performance with regards to the DHIS [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 3 The district has the physical infrastructure to ensure DHIS reliability as the primary source of data for PEPFAR [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 3 The district has the organizational infrastructure (documented mechanisms for continual learning, staff recruitment and retention, communications with PEPFAR, internal QA, data validation processes, etc.) to ensure DHIS reliability as the primary source of data for PEPFAR [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 3 The district has the informational/communications [ ] 1 [ ] 2 3 112 MEEPP II EVALUATION infrastructure to ensure DHIS reliability as the primary source of data for PEPFAR (communication channels between the district and PEPFAR are known, clear, and easy to access) [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 Do you perform routine data analysis [ ] Yes [ ] No 4 IF YES, what data validation techniques are being used? 4 Do you produce routine reports? [ ] Yes [ ] No 4 IF YES, what reports are produced routinely? 4 On a scale of 1 to 10, how much trust does the public place in the DHIS data and reports? [ ] 1 [ ] 2 [ ] 3 [ ] 4 [ ] 5 [ ] 6 [ ] 7 [ ] 8 [ ] 9 [ ] 10 4 And finally, to end the interview, I would like to ask you some questions about yourself Gender [ ] Male [ ] Female Age [ ] 18-25 [ ] 26-30 [ ] 31-35 [ ] 36-40 [ ] 41-45 [ ] 45-50 [ ] 51+ Education [ ] High School [ ] BA/University [ ] Master’s [ ] PhD Where educated [ ] Uganda [ ] Outside Uganda, but in Africa [ ] Europe [ ] US [ ] Other _______________________________ That was the last question! Thank you so much for your time today. MEEPP II EVALUATION 113 Implementing Partner Chief of Party – Web-based Survey Program Area : [ ] OVC [ ] ART [ ] PMTCT [ ] HTC [ ] Other Do you support[ ] Districts[ ] Sites[ ] Both districts and sites For data, do you rely on [ ] DHIS II[ ] OVC MIS Involvement with MEEPP II 1. Did MEEP II provide you/your staff training in the use of national data systems, particularly DHIS? [ ] Yes [ ] No 2. On a scale of 1 to 10, with 1 being absolutely low quality, to 10 being highest quality, how would you rate the quality of the training you received from MEEP II in the use of national data systems? [________________] 3. On a scale of 1 to 10, with 1 being absolutely low quality, to 10 being highest quality, how would you rate the quality of continual support you received from MEEP II? [________________] 4. On a scale of 1 to 10, from 1 being no reliance, to 10 being absolute reliance, how reliant are you on the DHIS II for reporting to PEPFAR? [________________] 5. Please tell us about areas where MEEPP II added value to your project? [TEXT] 6. In your opinion, are there missed opportunities for improving: a. Data collection? [ ] Yes[ ] No b. Data validation[ ] Yes[ ] No c. Data storage[ ] Yes[ ] No d. Data analysis[ ] Yes[ ] No e. Data reporting[ ] Yes[ ] No f. Data use for decision making?[ ] Yes[ ] No Partnership with Local Medical institutions 1. Describe the main activities of your project: _______________________________________________________________ _______________________________________________________________ _______________________________________________________________ 2. How many sites do you work with? __________[number] 114 MEEPP II EVALUATION 3. Does your project work with medical centers managed by: [checkbox] a. Government[ ] b. Religious organizations[ ] c. Private non-profit, not religiously affiliated[ ] d. Private for profit, , not religiously affiliated[ ] e. Military[ ] 4. On average across your sites, for PEPFAR data, which organization does what? [scale 0 to 2, 0 none, 1 is some, 2 is primary] Task MEEPP II Implementing Partner District Health Facility Not Done Oversight, training & capacity building around DHIS II Data collection Validation/QA Data entry in DHIS II Extract raw data from HMIS Analysis of data Formal reporting 5. How would you rate the following challenges to data management at your sites Staffing (numbers): [ ] Not a challenge at all [ ] A little challenging [ ] Somewhat challenging [ ] Very Challenging [ ] Super challenging, makes it impossible to manage along the continuum of data (from collection to reporting and use) Staffing (skills) [ ] Not a challenge at all [ ] A little challenging [ ] Somewhat challenging [ ] Very Challenging [ ] Super challenging, makes it impossible to manage along the continuum of data (from collection to reporting and use) Regular access to IT hardware [ ] Not a challenge at all [ ] A little challenging [ ] Somewhat challenging [ ] Very Challenging [ ] Super challenging, makes it impossible to manage along the continuum of data (from collection to reporting and use) Regular access to internet: [ ] Not a challenge at all [ ] A little challenging [ ] Somewhat challenging [ ] Very Challenging [ ] Super challenging, makes it impossible to manage along the continuum of data (from collection to reporting and use) Which aspect of data management is most challenging at the majority of your sites? [ ] Data collection [ ] Data validation [ ] Data analysis [ ] Data reporting [ ] Data archiving MEEPP II EVALUATION 115 6. Are there any outliers or patterns you see within your sites you wish to note? (i.e. one site is particularly strong, or sites with particular features have additional challenges) [Text] Data Usage 1. Does your project use data pulled from the DHIS II? [ ] yes [ ]No – if yes a. What for? _________________________________________ _________________________________________ _________________________________________ b. What decisions are made based on the data? _________________________________________ _________________________________________ _________________________________________ c. What is your trust level in the data? [ Likert scale – 1 = none, 10 = strong] [________________] d. How much manual manipulation or validation does your project have to do before the data is useful [ ] No validation [ ] A little validation [ ] Some validation [ ] A lot of validation 2. Does your project use data from any other sources outside the DHIS II? – If yes, which sources? ________________________________________________________________________________________________ ____________________________________________________________ Government involvement 1. Do you see the GoU involvement in improving DHIS II in Uganda? [ ] Yes [ ] No 2. How? ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ 116 MEEPP II EVALUATION 3. What are the areas you have seen improvements by the GoU in the last 5 years? [Scale (1-10)] Task National Level District Level Oversight, training & capacity building around HMIS Data collection Validation/QA Data entry in HMIS Access to HMIS Analysis of data Formal reporting Other comments and recommendations ____________________________________________________________________________ ____________________________________________________________________________ ____________________________________________________________________________ MEEPP II EVALUATION 117 FGD Guide for IP M&E Staff {Introduction & demographics} (DHIS/OVC MIS) Involvement with MEEPP II 1. What is your organization’s involvement with MEEPP II [Confirm findings from survey] 2. What are the areas where MEEPP II added value to your project? 3. What are some missed opportunities? 4. How has MEEPP II changed over time? Partnership with Local Medical institutions [Review findings from survey – who does what and challenges] 1. Do you agree with this summary? 2. Are there any outliers or patterns you see you wish to note? Data Usage 1. What do you use data from the HMIS for? 2. What decisions are made based on the data? Government involvement 1. What have you observed is the GoU involvement (National and District level) in improving data collection and HMIS in Uganda? Other comments and recommendations ____________________________________________________________________________ ____________________________________________________________________________ 118 MEEPP II EVALUATION Software Quality Assessment Checklist System: DHIS2 Sources: Document review and key informant interviews Summary of Systems: Software Implementation Quality (framework- these elements speak to the sustainability of quality of the MIS) Standard Importance Benchmarks Measurement method Results Impact Completeness: How complete is the implementation (including installation, configuration, testing, and deployment) compared to the requirements? low Rollout Plan: Is there an explicit plan for roll out to users, including explicit release cycles, user communications & training, and technical support specific to different user groups? Are future functionality scheduled and planned for? Customer Support: is there an explicit plan for customer support? Are there designated customer support personnel? Is there an infrastructure for tracking customer support issues? How do users contact and communicate with customer support? Content Maintenance: Is there an explicit plan for content maintenance (adding new, updating existing)? Is there a content review process? Is there a schedule for content? MEEPP II EVALUATION 119 Standard Importance Benchmarks Measurement method Results Impact Site facilitation: for sites which have user engagement functionality (blogs with comments, discussion forums), is there an explicit plan to monitor and facilitate discussion? Are there explicit staff dedicated to these tasks? Are there policies and guidelines posted for users and are staff trained in them? (please see “Collaboration Tool Protocol” for additional criteria) Usage and Usability tracking: is there an explicit plan for tracking usability, user satisfaction, and user experience? Does the team routinely look at analytics to measure usability (such as user surveys, software logs, and google analytics)? (see Web Analytics Protocol for additional criteria) Software Maintenance: Is there an explicit plan for maintenance for the system while in production? Is there an SLA? Is there issue tracking processes? Are there standard site aspects being tracked routinely (such as performance, usage, security logs, etc)? Is there a separate maintenance branch? How long does it take on average to make small changes? Is there a plan and a budget for updating the site based on ecosystem changes (such as new responsive guidelines or protocol changes)? High QA: Is there an explicit process for quality assurance before releasing fixes, improvements, and ongoing? Is QA integrated fully into the software development process, including rolling out of fixes and new releases? 120 MEEPP II EVALUATION Standard Importance Benchmarks Measurement method Results Impact Security/Privacy Protection: is there an explicit plan for security and privacy monitoring and responsiveness? Are staff trained in privacy and security requirements? Is there a security or privacy breach reporting process? (Also see “Technology Risk Assessment Checklist” for deeper evaluation criteria) Accuracy: For sites which have data collection as functionality, are there explicit reports that need to be generated (automatically or manually)? Are they sufficient to the needs of the users? Are they tested periodically for accuracy? (see data quality checklist for additional criteria) MEEPP II EVALUATION 121 OVC MIS Database Assessment Checklist Updated: May 24, 2016 DATA ENTRY Data Entry YES NO Source Disaggregation Results Ease – how easy is it to enter data into the system from different formats 1 Formats for data entry (web from, excel/csv, third party) Document review, database review KII Role 2 Is there online help or guidance to provide support? Role 3 Is the user interface intuitive? Role 4 Ability to work offline or save and return later? Role Validation – is there data validation built into the data entry process 1 Acceptable ranges Document review, database review KII Role 2 Autocomplete/select boxes Role 3 Calculations Role 4 Preview before submission Role 122 MEEPP II EVALUATION DATA STORAGE Data Storage YES NO Source Disaggregation Results Integrity – Does data retain sufficient context, disaggregation, and associated information in the database to represent accurately in reports 1 How is the data model constructed? (flat, hierarchical, relational, star?) Document review, Database review, KII N/a 2 Outline data fields related to data collection and reporting N/a 3 List disaggregation and classification structure N/a DATA REPORTING Data Reporting YES NO Source Disaggregation Results Accuracy – Is the data presented in the reports an accurate reflection of data collected? 1 Quantitative (numbers are calculated accurately, aggregated data is “aggregatable,” confidence in numbers is accurately displayed 2 Visualizations accurately demonstrate assumptions and analysis (i.e. does not mislead) 3 Data quality addressed (reports do not contain data collection errors) Usefulness – Is the data in the report useful to the specific users 1 Meets the data quality standards for the usage? MEEPP II EVALUATION 123 Data Reporting YES NO Source Disaggregation Results 2 Used by users in other places (up stream report, official counts, performance management? Usability – 1 Does the data in the report come in different formats? 2 How easy is the data to use, as identified by users? 3 Does it require a lot of manual manipulation before it can be used? 4 Does the user need to contact software to get the data Flexible –How easy is it to update the reports? 1 To reflect new requirements 2 Ad hoc reporting 3 To reflect new indicators or structure 124 MEEPP II EVALUATION Software Quality Assessment Checklist System: OVCMIS2 Sources: Document review and key informant interviews Summary of Systems: Software Implementation Quality (framework- these elements speak to the sustainability of quality of the MIS) Standard Importance Benchmarks Measurement method Results Impact Completeness: How complete is the implementation (including installation, configuration, testing, and deployment) compared to the requirements? low Rollout Plan: Is there an explicit plan for roll out to users, including explicit release cycles, user communications & training, and technical support specific to different user groups? Are future functionality scheduled and planned for? Customer Support: is there an explicit plan for customer support? Are there designated customer support personnel? Is there an infrastructure for tracking customer support issues? How do users contact and communicate with customer support? Content Maintenance: Is there an explicit plan for content maintenance (adding new, updating existing)? Is there a content review process? Is there a schedule for content? MEEPP II EVALUATION 125 Standard Importance Benchmarks Measurement method Results Impact Site facilitation: for sites which have user engagement functionality (blogs with comments, discussion forums), is there an explicit plan to monitor and facilitate discussion? Are there explicit staff dedicated to these tasks? Are there policies and guidelines posted for users and are staff trained in them? (please see “Collaboration Tool Protocol” for additional criteria) Usage and Usability tracking: is there an explicit plan for tracking usability, user satisfaction, and user experience? Does the team routinely look at analytics to measure usability (such as user surveys, software logs, and google analytics)? (see Web Analytics Protocol for additional criteria) Software Maintenance: Is there an explicit plan for maintenance for the system while in production? Is there an SLA? Is there issue tracking processes? Are there standard site aspects being tracked routinely (such as performance, usage, security logs, etc)? Is there a separate maintenance branch? How long does it take on average to make small changes? Is there a plan and a budget for updating the site based on ecosystem changes (such as new responsive guidelines or protocol changes)? High QA: Is there an explicit process for quality assurance before releasing fixes, improvements, and ongoing? Is QA integrated fully into the software development process, including rolling out of fixes and new releases? 126 MEEPP II EVALUATION Standard Importance Benchmarks Measurement method Results Impact Security/Privacy Protection: is there an explicit plan for security and privacy monitoring and responsiveness? Are staff trained in privacy and security requirements? Is there a security or privacy breach reporting process? (Also see “Technology Risk Assessment Checklist” for deeper evaluation criteria) Accuracy: For sites which have data collection as functionality, are there explicit reports that need to be generated (automatically or manually)? Are they sufficient to the needs of the users? Are they tested periodically for accuracy? (see data quality checklist for additional criteria) MEEPP II EVALUATION 127 Background Client: PEPFAR/USAID (GHPRO) Purpose of analysis: Measure the use and users of M&E data (DHIS2 and OCV MIS) in Uganda as part of the MEEPS II project Change Log Change made Date Author Creation 5/24/2016 Siobhan Green Overview Name of applications: DHIS2 and OVC MIS Dates Analyzed: June 2016 Access Model {TBD Chart the access model based on document review and initial interviews} Role Read Write Other Admin ALL ALL Administer system 128 MEEPP II EVALUATION Defined User Groups {TBD Chart the user groups based on document review and initial interviews} User Group DHIS2 or OVC System Permissions (based on the access model) Access details (location, frequency, media) Their needs (what they need from the application) Application need (what the app needs from the user) Approximate # of people District managers DHIS2 only Manager At office, weekly, desktop Review sites data entry. Pull reports for performance management Quality control over site data entry, 200 Access to Application The following are the use and user metrics used to measure current User composition by different factors. # Item Xref? Metrics 2nd dimension Source Results Total DHIS2 or OVC Total number of users Broken out by user groups Gender DHIS2 or OVC User composition by gender Is a second dimension Status/seniority DHIS2 or OVC User composition by status Is a second dimension Organization DHIS2 or OVC User composition by organization type (e.g. facility, IP, District, Ministry, PEPFAR) Is a second dimension Role/Job position DHIS2 or OVC User composition by role (e.g. Doctor, Administrator, Biostatistician, Project manager). Is a second dimension Health area DHIS2 or OVC User composition by focus area Is a second dimension Locations DHIS2 or OVC User composition by locations Is a second dimension MEEPP II EVALUATION 129 Usage of Application The following are the web metrics used to measure how the current User engages with the application, such as their engagement, common behaviors on the application # Item Xref? Metrics 2nd dimension Look at in particular Source Results Log in average DHIS2 or OVC User engagement by log in average By location, theme, User group. Over time Are people logging in with the frequency needed to complete tasks? Web logs Tasks completed DHIS2 or OVC User engagement by tasks completed By location, theme, User group, over time The typical tasks completed by users. Web logs Most popular Reports DHIS2 or OVC What reports are the most popular by group By user group, location, Format of reports downloaded? Web logs 130 MEEPP II EVALUATION ANNEX 4. SOURCES OF INFORMATION Persons Interviewed Key Informant Category Broad category Priority Type Organization & Location Organization & Location District/L ocation Email Address Phone Number/Skype District Health District 1 Group KII Isingiro Isingiro HQ Isingiro emmaxsam@gmail.com 0782 521344 District Health District 1 Group KII Isingiro Isingiro HQ Isingiro amosnamara@yahoo.com 0782 052028 District Health District 1 Group KII Isingiro Isingiro HQ Isingiro etumusherure@gmail.com 0701 391888 District Health District 1 Group KII Kisoro Kisoro HQ Kisoro s.nsabazoo@gmail.com 0772 601131 District Health District 1 Group KII Kisoro Kisoro HQ Kisoro irankundanathan@yahoo.c om 0779 773433 District Health District 1 Group KII KCCA Kampala dokello@kcca.go.ug 794661091 District Health District 1 Group KII KCCA Kampala pkiggudu@kcca.go.ug 794661139 District Health District 1 Group KII KCCA Kampala rmpirirwe@kcca.go.ug 794661087 District Health District 1 Group KII KCCA Kampala ogwaljim@gmail.com 772352405 District Health District 1 Group KII Bulumbuli District Bulumbuli wanialastephen@yahoo.co m District Health District 1 Group KII Bulumbuli District Bulumbuli fssukuky@gmail.com District Health District 1 Group KII Bulumbuli District Bulumbuli mulongom@yahoo.com District Health District 1 KII Kitgum Kitgum patkidega@yahoo.com 0782 355266 District OVC District 1 Group KII Isingiro Isingiro atujustine@yahoo.com 0782 665666 District OVC District 1 Group KII Kisoro Kisoro HQ Kisoro canaan800@gmail.com 0775 655304 MEEPP II EVALUATION 131 Key Informant Category Broad category Priority Type Organization & Location Organization & Location District/L ocation Email Address Phone Number/Skype District OVC District 1 Group KII Kisoro Kisoro HQ Kisoro nyonzimasam@gmail.com 0772 486806 District OVC District 1 Group KII Kisoro kisoro HQ Kisoro nzbnmpdavid@yahoo.com 0776 017882 District OVC District 1 KII Pader Pader HQ jaddwong@gmail.com 0772 527570 District OVC District 1 KII Bulumbuli District Bulumbuli dnabwire@yahoo.com 782140300 District OVC District 1 KII Jinja District Jinja opioouma@yahoo.com District Total 20 GF Donor 1 KII Global Fund Global Fund Kampala jim_ari@yahoo.co.uk 0772 404985 UNAIDS Donor 1 KII UNAIDS UNAIDS Kampala mubangizij@unaids.org 0772 419770 WHO Donor 1 KII World Health Organisation World Health Organisation Kampala kaggwam@who.int 0772 423207 Donor Total 3 Health and OVC Site Facility 1 Group KII MJAP Kampala kaksliv@yahoo.com 0755 553091 Health and OVC Site Facility 1 Group KII MJAP Kampala jnansubuga@mjap.or.ug 0755 553353 Health and OVC Site Facility 1 Group KII St Josephs Kitgum wynfra@gmail.com 0784 779224 Health and OVC Site Facility 1 Group KII ST Josephs Hospital kitgum kinyerakinyera@gmail.com 0785 429469 Health and OVC Site Facility 1 Group KII ST Josephs Hospital Kitgum medinasalimayubu@gmail. com 0786 042568 / 0794 380829 Health and OVC Site Facility 1 Group KII TASO Jinja Jinja Health and OVC Site Facility 1 Group KII TASO Jinja Jinja Health and OVC Site Facility 1 Group KII ALIVE Kampala pasquine@amsuganda.org 0774 704646 Health and OVC Site Facility 1 Group KII ALIVE Kampala davomagical@gmail.com 0774 831510 132 MEEPP II EVALUATION Key Informant Category Broad category Priority Type Organization & Location Organization & Location District/L ocation Email Address Phone Number/Skype Health Site Facility 1 Group KII Kabuyanda HC IV Kabuyanda HC IV Isingiro tusiimefortunate1@gmail.c om 0752 933284 Health Site Facility 1 Group KII Kabuyanda HC IV Kabuyanda HC IV Isingiro mestudo@hotmail.com 0779 794084 Health Site Facility 1 Group KII Kabuyanda HC IV Kabuyanda HC IV Isingiro alisonkemigisha@gmail.co m 0776 998208 Health Site Facility 1 Group KII Kisoro Hospital Kisoro Hospital Kisoro byishimojolly@yahoo.com 0772 658574 / 0701 658574 Health Site Facility 1 Group KII Kisoro Hospital Kisoro Hospital Kisoro munezero@gmail.com 0778 460454 Health Site Facility 1 Group KII Nakivale HC III Nakivale HC III Isingiro bwambalejackson@gmail.c om 0772 545519 Health Site Facility 1 Group KII Nakivale HC III Nakivale HC III Isingiro gopong@medicalteams.org / georgewilliamodwe@gmail.co m 0774 575902 Health Site Facility 1 Group KII Rwekubo HC IV Rwekubo HC IV Isingiro tgam2014@gmail.com 0774 240707 Health Site Facility 1 Group KII Rwekubo HC IV Rwekubo HC IV Isingiro salimubarak85@gmail.com 0789 548100 Health Site Facility 1 Group KII Rwekubo HC IV Rwekubo HC IV Isingiro kyohairwejuliet@gmail.com 0779 546066 Health Site Facility 1 Group KII Nakasongola HC IV Nakasongol a 0772 981341 Health Site Facility 1 Group KII Nakasongola HC IV Nakasongol a seruyangerobert@yahoo.c om 0772 990266 Health Site Facility 1 Group KII Kitgum Hospital Kitgum geofferyokwera@gmail.co m 0777 480263 Health Site Facility 1 Group KII Kitgum Hospital Kitgum okotcharles2014@gmail.co m 0782 828985 Health Site Facility 1 Group KII New life Health Centre II Kitgum bodong@fh.org 0772/0756 777995 Health Site Facility 1 Group KII New life Health Centre II Kitgum mokot@fh.org 0774 922079 / 0794 000073 Health Site Facility 1 Group KII Family Hope Center Jinja mmpirirwe@gmail.com MEEPP II EVALUATION 133 Key Informant Category Broad category Priority Type Organization & Location Organization & Location District/L ocation Email Address Phone Number/Skype Health Site Facility 1 Group KII Family Hope Center Jinja jayo.caf@gmail.com Health Site Facility 1 Group KII Family Hope Center Jinja kebinunaizk@yahoo.com Health Site Facility 1 KII Pader Pader HQ labesaalvinrichard@gmail.c om 0756 512301 / 0780 121513 Health Site Facility 1 KII 5TH DIVISION MILITARY HOSPITAL Pader spayaojil@gmail.com 0392 670791 Health Site Facility 1 KII Nakasongola Nakasongol a sanyuirene@yahoo.com 0772 574538 Health Site Facility 1 KII Nakasongola Nakasongol a karahukayo@gmail.com 0774 266755 Health Site Facility 1 KII Muyembe HC IV Bulumbuli gonyanga@gmail.com 772871280 Facility Total 33 MOGLSD GoU National 1 KII Ministry of Gender Labour & Social Development Ministry of Gender Labour & Social Development Kampala kashemeire.obadiah@gmail.co m 0772 549125 MOGLSD GoU National 1 KII Ministry of Gender Labour & Social Development Ministry of Gender Labour & Social Development Kampala kayebazibwe@mglsd.go.ug 0774 185458 MoH GoU National 1 KII UAC skhanakwa@uac.go.ug 0772 515840 MoH GoU National 1 KII ACP nmusoba@uac.go.ug MOH - ACP GoU National 1 KII AIDS Control Programme AIDS Control Programme Kampala lkisaakye@yahoo.co.uk 0772-447963 MOH - ACP GoU National 1 KII AIDS Control Programme Kampala 0772 484380 MOH - HI GoU National 1 Group KII Division of Health Information Division of Health Information Kampala emukooyo@yahoo.com;emuk ooyo@gmail.com 0772 400641 MOH - QA GoU National 1 KII Quality Assurance Quality Assurance Kampala sarahbyakika@hotmail.com 0772 423358 MoH - QA GoU National 1 KII MOH Kampala jamumpe@gmail.com 0773 717636 134 MEEPP II EVALUATION Key Informant Category Broad category Priority Type Organization & Location Organization & Location District/L ocation Email Address Phone Number/Skype MOH - TB GoU National 1 KII National TB & Leprosy Programme National TB & Leprosy Programme Kampala upenytho.george@gmail.com 0772 564798 UAC GoU National 1 KII Uganda AIDS Commission Uganda AIDS Commission Kampala nmusoba@uac.go.ug 0772 455254 GoU National Total 11 Health Site IP 1 M&E FGD IDI/KCC IDI/COMPREHENSIVE KAMPALA CITY COUNCIL Kampala rmwondha@idi.co.ug 0772 494432 Health Site IP 1 M&E FGD Mildmay Center MILDMAY/COMPREHE NSIVE CENTRAL Kampala moses.bwambale@mildmay.o r.ug 0782 014160 Health Site IP 1 M&E FGD MJAP MUFM/MJAP Kampala imusoke@mjap.or.ug 0772 992383 Health Site IP 1 M&E FGD HHIWA Workplace Program - HHIWA Kampala erasmus_tanga@wvi.org 0772 490 961 Health Site IP 1 M&E FGD IDI/KCC IDI/COMPREHENSIVE KAMPALA CITY COUNCIL Kampala moses.bwambale@mildma y.or.ug 0782 014160 OVC Site IP 1 Group KII I-DO I-DO Isingiro eturyamureba@gmail.com 0701 521595 OVC Site IP 1 Group KII I-DO I-DO Isingiro tadanah@gmail.com 0701 332299 IP Total 7 Data Cares MEEPP 1 Group KII Data Cares MGLSD Kampala ovcmissupport@mglsd.go. ug 0783 257000 Data Cares MEEPP 1 Group KII Data Cares MGLSD Kampala akiwanuka@dcaresug.com 0776 750005 Data Cares MEEPP 1 KII Data Cares MEEPP Kampala pkagenda@dcareug.com 0752 463808 Data Cares MEEPP 1 KII Data Cares Data Cares Kampala pkagenda@dcaresug.com MEEPP MEEPP 1 Group KII MEEPP Kampala skyokusingura@s-3.com 0772 488105 MEEPP MEEPP 1 Group KII MEEPP Kampala mmugorwa@s-3.c0m 0772 195786 MEEPP II EVALUATION 135 Key Informant Category Broad category Priority Type Organization & Location Organization & Location District/L ocation Email Address Phone Number/Skype MEEPP MEEPP 1 Group KII MEEPP Kampala snabukera@s-3.com MEEPP MEEPP 1 KII MEEPP MEEPP Kampala bamuron@s-3.com 0772 542235 MEEPP MEEPP 1 KII MEEPP MEEPP MEEPP 1 KII MEEPP MEEPP Kampala skununka@s-3.com 0772 905255 MEEPP Total 10 CDC USG 1 USG FGD CDC CDC Kampala CDC USG 1 USG FGD CDC CDC Kampala CDC USG 1 USG FGD CDC CDC Kampala PCO USG 1 KII PEPFAR Coordinator's office PEPFAR Coordinator's office Kampala kamogajx@state.gov 0772 138-323 USAID USG 1 USG FGD USAID USAID Kampala cmuwanga@usaid.gov 0772-138-505 USAID USG 1 USG FGD USAID USAID Kampala jmmwangi@usaid.gov 0772-138-506 USAID USG 1 USG FGD USAID USAID Kampala joruut@usaid.gov 0772 138523 USAID USG 1 USG FGD USAID/GHPRO USAID/GHPRO Kampala USAID USG 1 USG FGD USAID USAID Kampala USG Total 9 Grand Total 93 136 MEEPP II EVALUATION SNo Region District Biostatistician DHO DP Data Clerk Actual Interviews done 1 Central Region Buikwe 1 1 2 2 Central Region Bukomansimbi 1 1 2 3 Central Region Butambala 1 1 2 4 Central Region Buvuma 1 1 2 5 Central Region Gomba 1 1 2 6 Central Region Kalangala 1 1 2 7 Central Region Kalungu 1 1 2 8 Central Region Kampala 1 1 2 9 Central Region Kayunga 0 1 1 10 Central Region Kiboga 1 1 2 11 Central Region Kyankwanzi 1 1 2 12 Central Region Luweero 1 0 1 13 Central Region Lwengo 1 1 2 14 Central Region Lyantonde 1 1 2 15 Central Region Masaka 0 0 0 16 Central Region Mityana 1 1 2 17 Central Region Mpigi 1 1 2 18 Central Region Mubende 1 1 2 19 Central Region Mukono 1 1 2 20 Central Region Nakaseke 1 1 2 21 Central Region Nakasongola 1 1 2 22 Central Region Rakai 1 1 2 MEEPP II EVALUATION 137 SNo Region District Biostatistician DHO DP Data Clerk Actual Interviews done 23 Central Region Sembabule 1 1 2 24 Central Region Wakiso 1 1 2 Central Region Total 22 22 25 Eastern Region Amuria 1 1 2 26 Eastern Region Budaka 1 1 2 27 Eastern Region Bududa 1 1 2 28 Eastern Region Bugiri 1 1 2 29 Eastern Region Bukedea 1 1 2 30 Eastern Region Bukwa 0 0 0 31 Eastern Region Bulambuli 1 1 2 32 Eastern Region Busia 1 1 2 33 Eastern Region Butaleja 1 1 2 34 Eastern Region Buyende 1 1 2 35 Eastern Region Iganga 1 1 2 36 Eastern Region Jinja 0 0 0 37 Eastern Region Kaberamaido 1 1 2 38 Eastern Region Kaliro 1 1 2 39 Eastern Region Kamuli 1 1 2 40 Eastern Region Kapchorwa 1 1 2 41 Eastern Region Katakwi 0 0 0 42 Eastern Region Kibuku 1 1 2 43 Eastern Region Kumi 1 1 2 138 MEEPP II EVALUATION SNo Region District Biostatistician DHO DP Data Clerk Actual Interviews done 44 Eastern Region Kween 1 1 2 45 Eastern Region Luuka 1 1 2 46 Eastern Region Manafwa 1 1 2 47 Eastern Region Mayuge 1 1 2 48 Eastern Region Mbale 1 1 2 49 Eastern Region Namayingo 1 1 2 50 Eastern Region Namutumba 1 1 2 51 Eastern Region Ngora 1 1 2 52 Eastern Region Pallisa 1 1 2 53 Eastern Region Serere 0 0 0 54 Eastern Region Sironko 1 1 2 55 Eastern Region Soroti 1 0 1 2 56 Eastern Region Tororo 0 0 0 Eastern Region Total 27 26 57 Northern Region Abim 1 1 2 58 Northern Region Adjumani 1 1 2 59 Northern Region Agago 1 1 2 60 Northern Region Alebtong 0 0 0 61 Northern Region Amolatar 1 1 2 62 Northern Region Amudat 0 0 0 63 Northern Region Amuru 1 1 2 64 Northern Region Apac 1 1 2 MEEPP II EVALUATION 139 SNo Region District Biostatistician DHO DP Data Clerk Actual Interviews done 65 Northern Region Arua 1 1 2 66 Northern Region Dokolo 1 1 2 67 Northern Region Gulu 1 1 2 68 Northern Region Kaabong 1 1 2 69 Northern Region Kitgum 1 1 2 70 Northern Region Koboko 1 1 2 71 Northern Region Kole 1 0 1 2 72 Northern Region Kotido 1 1 2 73 Northern Region Lamwo 1 1 2 74 Northern Region Lira 1 1 2 75 Northern Region Maracha 1 1 2 76 Northern Region Moroto 1 1 2 77 Northern Region Moyo 1 1 2 78 Northern Region Nakapiripirit 1 1 2 79 Northern Region Napak 1 1 2 80 Northern Region Nebbi 1 1 2 81 Northern Region Nwoya 1 1 2 82 Northern Region Otuke 1 1 2 83 Northern Region Oyam 1 1 2 84 Northern Region Pader 1 1 2 85 Northern Region Yumbe 1 1 2 86 Northern Region Zombo 1 1 2 140 MEEPP II EVALUATION SNo Region District Biostatistician DHO DP Data Clerk Actual Interviews done Northern Region Total 28 27 87 Western Region Buhweju 1 1 2 88 Western Region Buliisa 1 1 2 89 Western Region Bundibugyo 1 1 2 90 Western Region Bushenyi 1 1 2 91 Western Region Hoima 1 1 2 92 Western Region Ibanda 1 1 2 93 Western Region Isingiro 1 1 2 94 Western Region Kabale 1 1 2 95 Western Region Kabarole 1 1 2 96 Western Region Kamwenge 1 1 2 97 Western Region Kanungu 1 1 2 98 Western Region Kasese 1 1 2 99 Western Region Kibaale 1 1 2 100 Western Region Kiruhura 1 1 2 101 Western Region Kiryandongo 1 1 2 102 Western Region Kisoro 1 1 2 103 Western Region Kyegegwa 1 1 2 104 Western Region Kyenjojo 1 1 2 105 Western Region Masindi 1 1 2 106 Western Region Mbarara 1 0 1 2 MEEPP II EVALUATION 141 SNo Region District Biostatistician DHO DP Data Clerk Actual Interviews done 107 Western Region Mitooma 1 1 2 108 Western Region Ntoroko 1 1 2 109 Western Region Ntungamo 1 1 2 110 Western Region Rubirizi 1 1 2 111 Western Region Rukungiri 1 1 2 112 Western Region Sheema 1 1 0 2 142 MEEPP II EVALUATION DOCUMENTS REVIEWED Folder Document Source 2011 2_MEEPP July - September 2011 Quarterly Report.pdf MEEPP 2011 MEEPP April June 2011 Quarterly Report-final.pdf MEEPP 2011 MEEPP II Quarterly Report_October_December_2011.pdf MEEPP 2011 MEEPP JAN MARCH 2011 Quarterly Report.pdf MEEPP 2012 APRIL JUNE 2012 QUARTERLY REPORT.pdf MEEPP 2012 Jan-Mar 2012 Quarterly Report.pdf MEEPP 2012 MEEP II Annual Report October 2011-September 2012 Print only MEEPP 2012 MEEPP II Quarterly Report_October_December_2011.pdf MEEPP 2012 Quarterly report July-Sept 2012 Final.pdf MEEPP 2013 5_MEEPP II Quarterly Report_Jan-March_2013.pdf – Print and PDF MEEPP 2013 7_MEEPP II Quarterly Report_July-Sept_2013.doc – Print and PDF MEEPP 2013 Attachment 1- Final SAPR Narrative Report_2013.pdf PRINT and digital MEEPP 2013 Final_MEEPP II Quarterly Report_October_December_2012.doc – Print and PDF MEEPP 2013 MEEPP II annual report October 2012-September 2013 Print only MEEPP 2013 MEEPP II Quarterly Report_April June_2013 Final.pdf – Print and PDF MEEPP 2014 2_MEEPP II Quarterly Report_July to Sept 2014.docx – Print and PDF MEEPP 2014 3_MEEPP II Quarterly Report_April June 2014.docx – Print and PDF MEEPP 2014 5_MEEPP II Quarterly Report_Jan-March 2014.docx– Print and PDF MEEPP 2014 5_MEEPP II Quarterly Report_Oct-Dec_2013.docx – Print and PDF MEEPP 2014 MEEPP DHIS2 District League Tables FINAL A4.pdf Print and pdf MEEPP 2015 3_MEEPP II Quarterly Report_Jan-March 2015.pdf – Print and PDF MEEPP 2015 3_MEEPP II Quarterly Report_Oct to Dec 2014.pdf – Print and PDF MEEPP 2015 4_MEEPP II Quarterly Report July to Sept 2015 final.pdf – Print and PDF MEEPP 2015 MEEPP II Quarterly Report April-June 2015.pdf – Print and PDF MEEPP Additional documents 2016-print only MEEPP MEEPP II EVALUATION 143 Additional documents CONDUCTING DATA QUALITY ASSESSMENT AND IMPROVEMENT FOR PEPFAR INDICATORS Oct 2015-print only MEEPP Additional documents DISTRICT HAND BOOK-USER GUIDE Nov 2015-print only MEEPP Additional documents Estimating Incidence of HIV with Synthetic Cohorts and Varying Mortality in Uganda-print only MEEPP Additional documents Generating PEPFAR Reports from the National eHMIS/DHIS2 System June 9, 2015- print only MEEPP Additional documents GUIDELINES FOR OVC IDENTIFICATION, PRIOTITIZATION, MONITORING AND GRADUATION Dec 2015-print only MEEPP Additional documents Health Monitoring, Evaluation and Learning Assessment Tool Oct 2015-print only MEEPP Additional documents Key insights from the national HIV and AIDS assessment exercise –October 2011 Print only MEEPP Additional documents Lessons learned from the Uganda HIV and AIDS country response progress reporting, 2015 (October 2015) – Print only MEEPP Additional documents Lot Quality Assurance Sampling Surveys in Uganda Dec 2015-print only MEEPP Additional documents LQAS COMMUNITY SURVEYS-2015 Bududa District Report-print only MEEPP Additional documents LQAS COMMUNITY SURVEYS-2015, Apac District Report-print only MEEPP Additional documents MEEPP II Close out Plan – Print only MEEPP Additional documents MONITORING, EVALUATION AND LEARNING SYSTEMS GUIDE FOR USAID UGANDA ACTIVITIES/IMPLEMENTING MECHANISMS Oct 2015-print only MEEPP Additional documents National Data Sets October 2014-September 2015 – Print only MEEPP Additional documents ORIENTATION MANUAL FOR ROLLOUT AND DISSEMINATION OF OVCMIS IMPROVEMENT TOOLS Dec MEEPP Additional documents Orphans and Other Vulnerable children household vulnerability prioritization toolkit Dec 2015 PRINT only MEEPP Additional documents OVCMIS INDICATOR BOOKLET Nov 2015-print only MEEPP Additional documents OVCMIS REGIONAL DATA REVIEW MEETINGS ON IMPROVING QUALITY OF OVC DATA￾CONCEPT NOTES 16TH MAY, 2016-PRINT ONLY MEEPP Additional documents OVCMIS SERVICE PROVIDER PROCEDURE MANUAL Nov, 2015-print only MEEPP Additional documents Prevention of Mother to Child transmission of HIV in Uganda - print only MEEPP Additional documents REQUEST FOR PROPOSALS (RFP) INVITATION NOTICE-PFP No TGF-16-O8O, Consultancy Program Quality and Efficiency June 17th 2016-print only MEEPP Additional documents STANDARD OPERATING PROCEDURES FOR OVCMIS DATA MANAGEMENT Nov 2015- PRINT ONLY MEEPP Additional documents THE GLOBAL FUND TO FIGHT AIDS, TUBERCULOSIS AND MALARIA, Terms and Conditions of Purchase of Services, Sep 15th 2015-print only MEEPP 144 MEEPP II EVALUATION DQA Reports 8_Data Quality Assessment for Clinical Care Treatment HCT and Nutrition.PDF MEEPP DQA Reports Final ART Cohort DQAI Report_2012-updated.PDF MEEPP DQA Reports MEEPP DQA MMC 2012 Final.PDF Print and PDF MEEPP DQA Reports MEEPP II PMTCT DQAI Final.PDF Print and PDF MEEPP DQA Reports Quality of TBHIV Data Reported by Health Facilities in Uganda.PDF Print and PDF MEEPP Evaluation Briefing Book and Burden Tables 2_HIV Burden table template with Weighted Seroprevalance _2013.xls MEEPP Evaluation Briefing Book and Burden Tables 2_HIV Burden table template with Weighted Seroprevalance _National_2014.xls MEEPP Evaluation Briefing Book and Burden Tables 2_HIV Burden table template with Weighted Seroprevalance _National_2015.xls MEEPP Evaluation Briefing Book and Burden Tables 2_HIV BURDEN TABLES_APR2010.xls MEEPP Evaluation Briefing Book and Burden Tables 2_HIV BURDEN TABLES_APR2011.xls MEEPP Evaluation Briefing Book and Burden Tables 2_HIV BURDEN TABLES_APR2012 with Weighted seroprevalence data 2012.xls MEEPP Evaluation Briefing Book and Burden Tables 2_MEEPP Evaluation briefing book_Jan 2016.PDF print and PDF MEEPP Evaluation Briefing Book and Burden Tables BURDEN TABLES COVER PAGE.DOC MEEPP none 1_USG SUPPORTED SITES BY TYPE OF SUPPORT BASED ON APR 2014.xls MEEPP none 2_FY 2016 Quarter 1_HIVCare_ART_High Volume Sites.xls MEEPP PEPFAR Report 1_APR 2015 by TA by IP.xlsx MEEPP PEPFAR Report 2_APR_2012_Explanatory Notes.docx print and digital MEEPP PEPFAR Report 3_APR_2013_Explanatory notes_Dec 2013.doc print and PDF MEEPP PEPFAR Report APR2010 Explanatory Notes Final.doc Print and digital MEEPP PEPFAR Report COVER PAGE for Technical Area Summary.doc MEEPP PEPFAR Report Explanatory Notes 2011.pdf MEEPP PEPFAR Report Final APR_2014_Narrative_Report_Dec 17 2014.doc MEEPP PEPFAR Report Final APR_2015_Narrative_Report_Jan 2016.docx MEEPP PEPFAR Report SeroSurvey regions_2.pdf MEEPP MEEPP II EVALUATION 145 Reports on Special Studies 1_BASELINE HFQA REPORT_SIMS_STAR-E-Final.pdf MEEPP Reports on Special Studies 2_Using PMTCT Program Data_Surveillance_HIV Prevalence in Uganda 2015.pdf MEEPP Reports on Special Studies 4 MEEPP II Special Study OVC.pdf print and pdf MEEPP Reports on Special Studies 5 MEEPP II Vulnerability Index Report.pdf print and pdf MEEPP Reports on Special Studies Design Effects for OVCs in Uganda.pdf Print and PDF MEEPP Reports on Special Studies Estimating Incidence from Synthetic Cohorts.pdf Print and pdf MEEPP Reports on Special Studies Final PMTCT Narrative Report_APR2011.pdf print and pdf MEEPP Reports on Special Studies HIV Prevalence Based on PMTCT Data Updated 2014-01-22-Copy.pdf MEEPP Reports on Special Studies HIV Prevalence FINAL A5.pdf MEEPP Reports on Special Studies KEY INSIGHTS IN IMPLEMENTATION OF M&E ACTIVITIES AT DISTRICT LEVEL 2011 print and digital MEEPP Reports on Special Studies MEEPP II Comparison of HIBRID and DHIS 2.pdf print and pdf MEEPP Reports on Special Studies MEEPP II Secondary Analysis AIS 2011 A5 FINAL.pdf MEEPP Reports on Special Studies PIASCY ACTIVITIES SURVEY REPORT - JULY 2011.pdf – PRINT and Digital MEEPP Reports on Special Studies Review_updated_March 2016.pdf MEEPP Reports on Special Studies Sexual Prevention Final Report_July 2012.pdf (Behavioral risk prevention interventions) Print and PDF MEEPP Reports on Special Studies Summary Report_MEL.pdf Print and PDF MEEPP Reports on Special Studies Systematic Review and Meta-analysis of Condom Promotion.pdf Print and PDF MEEPP Reports on Special Studies Uganda PMTCT Summary Brief final.pdf MEEPP 146 MEEPP II EVALUATION Reports on Special Studies Using Routine PMTCT Program Data for Surveillance of HIV Prevalence in Uganda.pdf 2013 – Print and PDF MEEPP Standard Operating Procedures 1_Guidance Appreciate Approach to DQAI.doc MEEPP Standard Operating Procedures 1_SOP for Reporting Data from DHIS2 - June 11.doc Print and PDF MEEPP Standard Operating Procedures 1_SOP on DQAI_Draft13102015.docx Print and digital MEEPP Standard Operating Procedures 2_Example-Overall DQA Guidelines.docx MEEPP Standard Operating Procedures 2_Summary of HIV-AIDS Data Quality Checks March 2015.docx MEEPP Standard Operating Procedures 4_Mapping MER Indicators to National Data Systems.xlsx MEEPP Standard Operating Procedures 4_Tier 1 MEL Systems Assessment Tool_Oct12.xls Print and pdf MEEPP Standard Operating Procedures 5_Ministry of Health Data Access Guideline.pdf MEEPP Standard Operating Procedures 5_Tier 2 DQAI_Site Tool.xls MEEPP Standard Operating Procedures 6_ExampleTrainingSchedule MEEPP Standard Operating Procedures 6_Flowchart_PEPFAR Reporting from National Systems MEEPP Standard Operating Procedures 7_Guidelines for extracting DHIS2 datasets using Web API MEEPP Standard Operating Procedures 7_USAID Data Quality Form MEEPP Standard Operating Procedures 8_Table N1a_Health Facility Reporting schedule MEEPP Standard Operating Procedures Algorithms for Data Exchange July 28 2015.pdf –Print/Digital MEEPP Standard Operating Procedures COVER PAGE MEL Assessment ToolFOR THE EXCEL SHEET.docx print and digitalpre MEEPP Standard Operating Procedures Data Accountability platform for Type of Support (DAPTS) to PEPFAR-Funded facility and community sites, country operational plan 2015 print only MEEPP MEEPP II EVALUATION 147 Standard Operating Procedures MEL Systems Assessment Tool_Annexes_Nov26.xlsx print and digital MEEPP Standard Operating Procedures OVCMIS SOPS- Hardware and Software Enviroment.pdf MEEPP TECHNICAL AREA SUMMARY - APR 2010.xls Annual Program Results 2010. Data gathering, aggregation and analysis explanatory notes Print only MEEPP TECHNICAL AREA SUMMARY - APR 2010.xls Annual Program results 2011, Data gathering, aggregation, and analysis explanatory notes January 2012 – print only MEEPP Workplans 1_MEEPP II PMP_Final_June 25062011.docx Print and digital MEEPP Workplans 4_Summary_FY 2012 MEEPP Work Plan in a Nut Shell_last modified 5 Jan 2012.xlxs MEEPP Workplans MEEPP II _Workplan_Extension_April 2015-May 2016.docx Print and digital MEEPP Workplans MEEPP II PMP_Final_December 2013_Updated.docx MEEPP Workplans MEEPP II PMP_Revised_2015-2016.doc Print and PDF MEEPP Workplans MEEPP II__Workplan FY 2012_Oct2011_Sep2012.pdf MEEPP Workplans MEEPP II_Work Plan FY 2015_Oct 14_Sep2015.pdf - print and pdf MEEPP Workplans MEEPP II_Work Plan_FY2014_Oct13_Sep2014.dpf print and pdf MEEPP Workplans MEEPP II_Workplan FY 2011_Sept2010_Sept2011.pdf print and pdf MEEPP Workplans MEEPP II_Yr 3_Workplan FY 2013_Oct2012_Sep2013.pdf print and digital MEEPP Workplans PMP original Print only MEEPP Uganda SS Documents Health Sector - Annual Monitoring Report FY2013-14 Outside Privacy and security Strengthening_district_based_health_reporting_through_DHIS_Uganda Outside Privacy and security DHIS2 Uganda presentation Outside Privacy and security ALB_Background_Paper-Health_Information_Confidentiality_in_Uganda Outside Privacy and security 2015_08_Fundamentals_health_law_Uganda Outside Privacy and security 2015_08 Outside PEPFAR Copy of MER 2 0 Summary targets 2016_02_25 Outside PEPFAR MER 2 0 Indicator Reference Guide for COP16 target setting 2 25 2016 Outside METS CDC-RFA-GH15-1611 Outside G Policies Approved-National-Social-Protection-Policy-09-11-2015-3 Outside 148 MEEPP II EVALUATION G Policies GOU Social protect policy-summary Outside G Policies GOU ICT policy-summary Outside G Policies NATIONAL EMPLOYMENT POLICY Outside G Policies National_ICT_Policy_for_Uganda_2012 Outside G Policies NATIONAL_IT_POLICY_2012_1_1 Outside G Policies unpan035420 Outside Donor info NakivaleFactSheet2014 Outside DATABASES MEEPP II EVALUATION 149 ANNEX 5. DISCLOSURE OF ANY CONFLICTS OF INTEREST 150 MEEPP II EVALUATION MEEPP II EVALUATION 151 152 MEEPP II EVALUATION MEEPP II EVALUATION 153 154 MEEPP II EVALUATION MEEPP II EVALUATION 155 156 MEEPP II EVALUATION MEEPP II EVALUATION 157 158 MEEPP II EVALUATION MEEPP II EVALUATION 159 160 MEEPP II EVALUATION MEEPP II EVALUATION 161 162 MEEPP II EVALUATION MEEPP II EVALUATION 163 Global Health Performance Cycle Improvement Project 1331 Pennsylvania Avenue NW, Suite 300 Washington, DC 20006 Phone: (202) 625-9444 Fax: (202) 517-9181 http://ghpro.dexisonline.com/reports-publications