USAID Pakistan May 2012 This publication was produced for review by the United States Agency for International Development. It was prepared by Rodney Bennett, Erica Holzaepfel, Mildred Howard, and Elizabeth Preble through Social Impact, Inc. EVALUATION John Snow, Inc., Enhancing Strategic Information Project: End-of-Project Evaluation Final Report JOHN SNOW, INC., ENHANCING STRATEGIC INFORMATION PROJECT: END￾OF-PROJECT EVALUATION Final Report May 2012 This publication was produced for review by the United States Agency for International Development. It was prepared by Rodney Bennett, Erica Holzaepfel, Mildred Howard, and Elizabeth Preble through Social Impact, Inc. JOHN SNOW, INC., ENHANCING STRATEGIC INFORMATION PROJECT: END-OF-PROJECT EVALUATION FINAL REPORT Rodney Bennett, Erica Holzaepfel, Mildred Howard, and Elizabeth Preble DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. This document is intended for limited internal distribution and is not yet available in printed or online versions. Documents will be made available through the Development Experience Clearing House (http://dec.usaid.gov). Additional information can be obtained from Social Impact, Inc. 2300 Clarendon Boulevard Arlington, VA, 22201 Tel: (703) 465-1884 Fax: (703) 465-1888 info@socialimpact.com This document was submitted by Social Impact, Inc., to the United States Agency for International Development under USAID Contract No. RAN-I-00-09-00019-00, Task Order AID￾674-TO-12-00004. ESI End-of-Project Evaluation ii CONTENTS Table of Contents ACKNOWLEDGMENTS ................................................................................................................................. vi EXECUTIVE SUMMARY ................................................................................................................................. vii GLOSSARY ........................................................................................................................................................xiv ACRONYMS ......................................................................................................................................................xv I. INTRODUCTION .........................................................................................................................................1 II. PROJECT DESCRIPTION ............................................................................................................................3 III. SUMMARY OF EVALUATION DESIGN AND METHODOLOGY .................................................5 IV. STRATEGIC DESIGN ISSUES ...................................................................................................................8 V. PERFORMANCE MANAGEMENT (MANAGING for RESULTS) .................................................. 10 A. OVERVIEW .......................................................................................................................................... 10 B. PROJECT PLANNING AND MANAGEMENT .......................................................................... 10 1. Responding to a Changing SOW................................................................................................ 10 2. Planning and monitoring tools .................................................................................................... 11 C. PROJECT STAFFING AND ORGANIZATION .................................................................... 12 1. Organization of staff and LOE .................................................................................................... 12 2. Quality of staff ................................................................................................................................ 13 3. ESI staff deputed to DOH offices ............................................................................................... 13 D. MANAGEMENT OF SUBCONTRACTORS .......................................................................... 14 E. FINANCIAL MANAGEMENT ......................................................................................................... 15 1. Financial reporting and accounting ............................................................................................ 15 2. Burn rate analysis ........................................................................................................................... 16 3. Cost saving measures .................................................................................................................... 16 F. USAID MANAGEMENT OVERSIGHT ......................................................................................... 17 1. Ongoing management by USAID COTRs and the Contracts Office ................................ 17 2. Standard Contractor Performance Reports ............................................................................ 17 3. ESI in Lesotho and Swaziland ...................................................................................................... 18 4. PEPFAR Partners M&E Meetings ................................................................................................ 18 G. AUDIT RECOMMENDATIONS ON THE DATA WAREHOUSE ................................... 19 VI. PROJECT ACHIEVEMENTS – TRAINING AND CAPACITY BUILDING ................................. 22 A. TRAINING PROGRAM PLANNING ............................................................................................ 22 B. TRAINING COVERAGE .................................................................................................................. 23 C. RELEVANCE AND SCOPE OF TRAINING CURRICULA ................................................. 24 D. BUILDING CAPACITY IN STRATEGIC INFORMATION IN HIV AND AIDS PROGRAMS FOR PMTCT, PALLIATIVE CARE, AND OVC .......................................................... 25 E. CAPACITY BUILDING IN DATA QUALITY AND ENHANCED USE OF DATA .......... 26 F. RESPONSIVENESS TO CAPACITY BUILDING NEEDS ......................................................... 27 VII. PROJECT ACHIEVEMENTS – THE PARTNERSHIP INFORMATION MANAGEMENT SYSTEM AND GEOGRAPHIC INFORMATION SYSTEM .................................................................... 30 A. INTRODUCTION ............................................................................................................................. 30 ESI End-of-Project Evaluation iii B. ACHIEVEMENTS AND SHORTFALLS IN SYSTEMS CAPACITY ......................................... 30 C. SHORTFALLS IN DATA AND INFORMATION CONTENT .......................................... 32 D. INHERENT RISKS IN PIMS DEVELOPMENT ........................................................................ 33 E. SUGGESTION FOR ALTERNATIVE PIMS CONSTRUCTION ............................................. 34 F. INFORMATION TECHNOLOGY CHANGE MANAGEMENT PROCESS ........................ 35 1. Change control meetings ............................................................................................................. 35 2. Formation of an Information Technology Change Management Structure ..................... 36 VIII. SUMMARY OF CONCLUSIONS AND RECOMMENDATIONS ......................................... 37 APPENDICES .................................................................................................................................................... 38 APPENDIX A. KEY ESI CONTACTS AND PROJECT ..................................................................... 39 APPENDIX B. ESI KEY EVALUATION QUESTIONS ...................................................................... 42 APPENDIX C. LIST OF PERSONS CONTACTED ............................................................................ 44 APPENDIX D. EVALUATION METHODOLOGY AND WORKPLAN ...................................... 47 APPENDIX E. DOCUMENTS REFERENCED ..................................................................................... 58 APPENDIX F. ESI EVALUATION TOOLS ........................................................................................... 61 APPENDIX G. KEY ESI MANAGEMENT-RELATED EVENTS 2008-2012 ................................... 71 APPENDIX H. JSI/ESI PROJECT PLANNING AND MONITORING DOCUMENTATION .. 72 APPENDIX J. MANAGEMENT OF JSI/ESI SUBCONTRACTORS ................................................. 74 APPENDIX K. ESI BURN RATE ANALYSIS ......................................................................................... 75 APPENDIX L. ANALYSIS OF ONLINE SURVEYS ............................................................................. 78 APPENDIX M. SCHEMATICS OF CURRENT AND FUTURE PIMS ARCHITECTURE AND MODEL DASHBOARD ............................................................................................................................. 85 APPENDIX N. SUMMARY OF CONCLUSIONS AND RECOMMENDATIONS…………...91 ESI End-of-Project Evaluation iv ESI End-of-Project Evaluation v Map 1. South Africa and its Provinces (Geographic concentration of ESI Evaluation was in North West, Gauteng, KwaZulu Natal, and Western Cape Provinces) ESI End-of-Project Evaluation vi ACKNOWLEDGMENTS The Evaluation Team wishes to thank everyone who put aside pressing duties to share their thoughts and insights with us. The United States Government (USG), President’s Emergency Plan for AIDS Relief (PEPFAR) strategic information team and the United States Agency for International Development (USAID) and United States Department of Health and Human Services, Centers for Disease Control PEPFAR Activity Managers generously met with us during the initial days of the evaluation – some more than once – and helped our Evaluation Team focus on issues of critical importance to them. We also talked with a large number of PEPFAR implementing partners who described a wide-range of experiences with their internal management information systems, PEPFAR reporting, and their strategic information interests and needs. Special thanks go to James Maloney, PEPFAR Deputy Coordinator and the USG PEPFAR Provincial Liaisons (PPLs) with whom we met in each of the four provinces visited (North West, Gauteng, KwaZulu Natal, and Western Cape). PPLs had valuable information to share with us regarding their work and strategic information needs. Mr. Maloney was particularly helpful in facilitating an online survey that required use of a secure mailing list. Our Evaluation Team had long visits with two key South African Government (SAG) officials: Chris van Rooyen, Deputy Director: Programme Implementation at the Department of Social Development, and Thulani Masilela, Chief Director, Strategic Planning at the Department of Health. Both gave our Evaluation Team in-depth accounts of the productive, bilateral, working relationships and technical support received in the areas of strategic information systems development. The SAG officials we met with in North West and KwaZulu Natal Provinces were equally knowledgeable and informative. The John Snow, Inc., Enhancing Strategic Information Project Director and his team are due a special word of thanks. They made themselves completely available to the evaluators. They were generous with their time and responsive to our many requests for clarification and additional information. Finally, the Evaluation Team wishes to acknowledge the outstanding support received from USAID’s Charles Mandivenyi, Senior Monitoring and Evaluation Specialist, and John Kuehnle, Development Leadership Initiative Health Officer. The ESI Evaluation Team May 2012 ESI End-of-Project Evaluation vii EXECUTIVE SUMMARY Introduction. This evaluation results from a request by USAID/South Africa (USAID/SA) for Social Impact, Inc. (SI) to evaluate the John Snow, Inc. (JSI) project, Enhancing Strategic Information (ESI). The ESI task order was originally awarded for the period July 2008-2013 under terms of the Population, Health and Nutrition Technical Assistance and Support Indefinite Quantity Contract Three – Global Health (TASC3IQC). The task order was a Cost-Plus- Fixed￾Fee (CPFF) term type at a level of $22.9 million including budgetary provisions for sub￾contracting with 3 other organizations. Due to accelerated demands from the Mission, the level of effort (LOE) was reached earlier than originally planned, in March 2012. The Mission still had unmet requirements; this additional LOE and associated additional tasks would normally have been added to the task order under the Logical Follow-on Exception to Competition; however, in this case the TASC3IQC had reached its ceiling preventing the Mission from modifying the task order. Thus, an additional 7-month contract was awarded for additional LOE to accomplish illustrative tasks for the period March 2012 through September 2012. Scope of ESI Project. ESI was specifically concerned with information systems that measure and inform the United States Government (USG) PEPFAR program in South Africa, and the South African Government (SAG) HIV and AIDS program activities. The ESI Project (and thus this evaluation) focused on two distinct areas: 1) refinements to the web-based information system for program reporting and performance monitoring; 2) building the capacity of PEPFAR and SAG implementing partners to collect, report, and use high-quality data and strategic information for purposes of program management at the implementation level. The Project was designed to put in place a technical assistance organization/contractor with the technical scope and capacity to refine the information systems structure to improve its user interaction, and analytical functionality and tools; to serve as the information systems administrator and technical support office for users; and to simultaneously build the awareness, motivation and capacity PEPFAR and SAG managers to place high value on data quality and use of data/information for performance management purposes. A substantial amount of the information systems upgrades involved building from the first generation PEPFAR Data Warehouse to establish a second generation PEPFAR results reporting and strategic information system with improved user interaction and analytical functionality1 . Several local subcontractors were brought in to augment the prime contractor’s capacity in the area of information systems development, while training/capacity building activities were primarily handled by the prime contractor’s staff. Direct beneficiaries of the ESI Project were USG PEPFAR strategic information managers, PEPFAR implementing partners and staff throughout the country, the SAG Department of Health (DOH) and Department of Social Development (DSD) at the national level, and SAG provincial and district HIV and AIDS-related programs and services in 18 designated priority districts. Evaluation Scope and Methods. This evaluation was both summative and formative in nature. It covered the period from inception of ESI in 2008, through March 2012, and considered key areas of ESI Project performance including: the appropriateness of the Project’s 1 The second generation system was named, the Partnership Information Management system (PIMS). ESI End-of-Project Evaluation viii development hypothesis; the adequacy of the Project’s management; and the Project’s performance including key achievements and gaps. The evaluation Scope of Work also requested the evaluators to make recommendations that would inform the future direction of similar projects. This evaluation therefore looked slightly beyond the recently completed Project (March 2012), and generally considered what is likely to be accomplished in terms of additional improvements to the strategic information system by the end of the new 7-month contract (September 2012). Beyond that, this evaluation also envisioned options and opportunities for a possible next generation of PEPFAR strategic information systems development. The primary data collection methods employed in this evaluation included document review, key informant interviews, focus group discussions, on-line surveys, and direct observation. Through document review and lengthy discussions with members of the PEPFAR USG team and the JSI/ESI team, four provinces (Gauteng, North West, KwaZulu Natal, and Western Cape), were selected as the focus of this evaluation because of the high concentrations of PEPFAR implementing partners and ESI/SAG activities. As discussed below (and presented in separate sections of the full report), findings, conclusions and recommendations are grouped into four priority areas of inquiry as follows:  Strategic design issues concerned with assessing the appropriateness of the Project’s development hypothesis and theory of change;  The Project’s performance management practices (managing for results), and recommendations for the future;  Outputs and outcomes of the Project’s performance in the areas of training and capacity building, and recommendations for the future;  Project outputs to date and opportunities for the future in terms of improvements in the functionality and construction of the PEPFAR strategic information system. Findings, Conclusions, and Recommendations – Strategic Design Issues. The problem to be addressed by the ESI Project was that the existing PEPFAR results reporting database, the Data Warehouse (DW), was viewed by users as not being user-friendly, and the DW did not have features that would encourage managers to analyze and use data (from the DW and other data sources) for program planning, monitoring and evaluation purposes. Based on the Evaluation Teams’ understanding of the ESI contract Scope of Work, designers of the Project believed that since the Data Warehouse was an upward reporting system with limited analytical functionality, the data and information contained in the DW was not seen by users at various levels as relevant to their broader programming and performance management information needs. The DW was perceived as not being an adequate source of data/information for programming and performance management purposes, and implementers and managers were perceived as not being motivated and/or lacking in the skills and tools to seek/use strategic data and information (from the DW and other data sources) to strengthen management of their programs. Because of its limited relevance to their management needs, it was believed by designers of the ESI Project that implementers and managers were additionally not highly motivated (and/or lacked skills and tools) to ensure collection and reporting of high quality data into the DW; the quality of data in the DW was an underlying concern. The Project’s development hypothesis was that strengthening the capacity of individuals and ESI End-of-Project Evaluation ix institutions to collect, analyze, and use strategic information improves the quality and sustainability of HIV and AIDS programs and outcomes. The Project’s development hypothesis was specifically based on USAID’s vision for ESI as stated in the introductory section of the ESI Contract. It states in part, “…The U.S. Government (USG) has undertaken activities to improve information use for policy, planning, and practice, and has strengthened the capacity of health systems to conduct data collection, analysis, information use and decision making. Especially under PEPFAR, USG and its implementing partners have made substantial progress in developing the capacity of local organizations and government counterparts to establish, manage, and sustain these activities. These activities have been developed on the premise that generating demand for and improving the use of data in policy formulation, program planning, monitoring and evaluation improves health services and consequently, health outcomes.” The Evaluation Team found that the Project’s development hypothesis was logically linked to the Project’s theory of change as articulated in ESI’s vision and mission statements. The change theory is paraphrased here as follows: a) There is need for improvements in the functionality of the information system such that this, b) leads to improvements in the analysis and use of strategic information for decision making and performance management thereby, c) improving users’ performance management skills, vested interested in producing and using high quality strategic data and information to improve program performance thus, d) contributing to the overarching USAID objective of improving the sustainability and quality of HIV and AIDS programs, services, and outcomes. However, a critical PEPFAR development that occurred after inception of the ESI Project is likely to be of significant importance to future PEPFAR strategic information systems project designs. The 2009 PEPFAR 5-Year Strategy2 introduced a strategic PEPFAR policy shift that required PEPFAR implementing partners to substantially move away from primarily private provider emergency assistance in the fight against AIDS to technical assistance and greater emphasis on improving the sustainability of HIV and AIDS programs through government-run public sector services. As reported by several key informants to this evaluation, this strategic PEPFAR shift, generally referred to as PEPFAR-II, is now at an advanced stage of implementation, with many PEPFAR implementing partners involved in phasing out their direct service delivery programs and shifting emphasis to capacity building and technical support to provincial, district or sub-district public sector implementers. Informants specifically expressed concern that the current PEPFAR indicators do not allow them to adequately report on their actual activities and achievements. The evaluators concluded that the development hypothesis and theory of change for the ESI Project were sound and consistent with USAID development guidelines, but that future PEPFAR information systems may need to specifically address two programmatic implications of PEPFAR-II. For the future, this evaluation recommends that consideration be given to 2 PEPFAR 5-Year Strategy, 2009, http://www.pepfar.gov/documents/organization/13303.pdf ESI End-of-Project Evaluation x modifying PEPFAR reporting formats, indicators, etc., to allow PEPFAR-II implementing partners to better reflect their capacity building and technical support roles. In addition, the evaluators recommend that consideration be given in the future for PEPFAR strategic information systems to be more dependent upon and closely aligned with SAG information systems as a primary source of HIV and AIDS-related data and information, particularly the Department of Health Information System (DHIS) and the National Health Information Centre (NHIC). Findings, Conclusions and Recommendations – Performance Management (managing for results). Shortly after inception of the ESI Project, the SAG called for a more concerted effort and closer collaboration with external partners such as the Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM) and PEPFAR. As a result, during the four years of ESI operation, a number of activities that were originally planned to be achieved over a five￾year period were significantly accelerated, some at the specific request by USAID, others in response to independent requests by DOH and DSD. This evaluation found that the Project’s demonstrated flexibility and responsiveness to DOH requests for capacity building and technical support, helped to facilitate a strong overarching relationship between SAG and USG regarding HIV and AIDS strategic information. This relationship was forged at a critical juncture both in PEPFAR’s programming in South Africa and to the SAG’s receptivity to increased collaboration. The evaluation also found that consistent with a recent Inspector General’s audit recommendation, a Memorandum of Understanding between USAID and CDC has been drafted by USAID with the goal of better defining working relationships in support of PEPFAR results reporting and strategic information systems designs and developments. The memo is currently in Atlanta awaiting CDC approval. The Evaluation Team found that while the Project’s agility in responding to changing requirements was commendable, management practices that would ensure the most strategic and efficient use of resources were inadequate in the first two years of operation. For example, the Project never produced an overall ESI Project M&E Plan (as required by the contract), nor a standard Results Framework to use for work-planning and project monitoring. In the absence of such tools in those critical, fast-moving first two years of the Project, ESI had no way to: a) reflect on the evolving scope of work, b) track expenditures against activities, c) monitor performance, or d) more efficiently guide the activities of the four subcontractors. While the line-item budgeting format used by JSI is consistent with USAID requirements, it would have been prudent for ESI to also have a way of analyzing and monitoring expenditures by functional program categories and sub-categories in order to track expenditures from all line-item categories, for instance, on DHIS training. Particularly in the crucial earlier years of the Project, the absence of an ESI financial system that could track and allow monitoring of Project expenditures by activity area was a major weakness in the Project’s management practices. USAID oversight of the ESI Project was not always sufficiently consistent or rigorous to ensure that the Project adhered to contractually-specified performance management practices such as completing an M&E plan and other sound management practices such as development of rigorous technical specifications for modifications/improvements to the PEPFAR results reporting and strategic information system. ESI End-of-Project Evaluation xi For the future, this evaluation recommends that several specific actions be undertaken by USAID in the interest of more efficient management of future PEPFAR results reporting and strategic information systems developments. The Evaluation Team recommends that:  USAID develop a change management policy and internal management structure for future PIMS developments.  That the scope, functions and composition of the Strategic Information Unit need to be reviewed, and a PEPFAR results reporting and strategic information task force needs to be reconstituted with clear, board-like authorities and limits of involvement in implementation details within the framework of a defined change management policy and structure.  That an in-house, information technology and systems expert (director/administrator) be appointed to manage all external technical developments including change control meetings. Findings, Conclusions and Recommendations – Training and Capacity Building. In the area of training, the Evaluation Team found that ESI planned its training program systematically in consultation with SAG, that the overall training covered workers at different levels of the system, and that there were procedures in place for post-training assessments and follow-up. The Project trained a significant number of individuals in a variety of course subjects, the majority of whom were SAG personnel. ESI training was generally useful/relevant and well￾received by SAG, PEPFAR implementing partners, and the individuals trained, and there appears to be a continuing demand for related training, particularly in the areas of DHIS, Evidenced￾based Health Management, and data quality. ESI additionally undertook effective efforts to address a program objective of involving local training institutions and resources in the conduct of sustainable training/capacity building in strengthening results reporting and strategic information systems. As relates to capacity building in development and use of results reporting and strategic information systems, the Evaluation Team noted that securing USG and SAG agreement and actually creating and fully implementing a single database in cooperation with the DSD for the USAID-supported Orphans and Vulnerable Children (OVC) program represented an ESI best practice. The approach used by USAID and ESI in developing a productive USG/SAG working agreement contributed to success of the OVC collaboration. On another front, the ESI Project expended considerable time and effort in designing and introducing the Routine Data Quality Assessment tool. In addition, in North West Province, the evaluation found that ESI’s approach of secondment of trained M&E specialists within the SAG structure was reported by SAG officials to be very effective, although more needs to be understood about the factors that led to success of this model in that district. For the future, the evaluation recommends that:  ESI’s work in the area of building sustainable training capacity is continued.  That training of SAG personnel, as determined by training needs assessments at the time, should be part of any future technical assistance to SAG in use of HIV and AIDS￾related results reporting and strategic information.  That work accomplished by the ESI Project in the area of integration of PEPFAR and the SAG information systems for OVC might be considered a model for replication. ESI End-of-Project Evaluation xii  That consideration should be given to replicating the model for secondment of M&E specialists in SAG district management positions, giving close attention to successful cases such as the one found in North West Province. Findings, Conclusions and Recommendations – The Partnership Information Management System (PIMS). The functionality of PIMS as developed by ESI was given special attention. The information system development challenges as stated in the ESI Contract were twofold: 1) to further strengthen the reporting functions of the system; 2) to further strengthen the M&E and analytical functions of the system. The Evaluation Team found that the PIMS was generally reported by users to be more intuitive and user friendly than the predecessor DW. The reporting functionality of PIMS appeared to adequately meet USG PEPFAR results report needs. The Evaluation Team found that the ESI Project had a good reputation among users as being “customer-friendly” and responsive to inquiries about technical problems in using the system (such as system access problems, etc.). Beyond this, the Evaluation Team found that current functionality of PIMS falls short of the M&E and analytical capabilities that were envisioned and specified in the ESI Contract. The technical developers at the ESI Project (who agreed with and understood the original intent of PIMS development in the ESI Contract), and the Evaluation Team were in basic agreement that the system as developed/improved to date, cannot meet the criteria set forth in the ESI Contract, for fairly robust, user-managed data analysis and M&E capacity, and will not have achieved this contractually-defined level of analytical functionality by the end of the ESI Project in March 2012. It was also agreed that the Project is unlikely to achieve this contractual standard by the end of the additional 7-month contract (September 2012). In conjunction with this, some system users, particularly Activity Managers, were found to have negative attitudes towards the use of PIMS and that the user-initiated, intuitive instructions now built into PIMS are not adequate to improve these perceptions and attitudes. To counter these concerns, the Evaluation Team recommends that the USG PEPFAR Secretariat, USAID, and ESI organize one or more orientation meetings that present current and future PIMS developments, and that the USG PEPFAR partner agencies exercise authority to ensure that the appropriate levels of participation in, and support for, PIMS is not optional. The evaluators also observed that there is a wide range of data and information that is not now being captured in PIMS but that will need attention (with guidance of the USG PEPFAR strategic information team), in future PIMS developments. Future PIMS developments should consider modifications to PEPFAR reporting indicators in line with PEPFAR-II guidelines, and should consider further alignments and linkages with other SAG strategic information systems. Further, for the future, the Evaluation Team recommends that USAID consider transitioning PIMS to a modular information system construction. The envisioned system would need to set out database links, network requirements and restrictions, data user agreements, and use of the DHIS. The envisioned system would use facility-level technology and web-based servers, and would have a means of controlling data security, access rights, and confidentiality requirements on shared systems. The Geographic Information System (GIS) functionality within PIMS should also be further developed as part of other “next generation” developments of the system. ESI End-of-Project Evaluation xiii Overarching ESI Evaluation Conclusions. While the Evaluation Team found the logic behind ESI’s development hypothesis to be sound, the Project provided little evidence to demonstrate the provision of improved health services or to substantiate improved health outcomes resulting from the Project activities. The primary Evaluation findings indicate that the greatest degree of change due to the Project occurred at lower-levels of the hypothesis at the output and outcome levels as evidenced within the training and capacity building task areas. Lesser degrees of change were evident at the objective and goal levels of the Project regarding data use, data quality, health services and health outcomes. The ESI Project suffered from inadequate, inconsistent, and poor management and oversight at all levels of administration, starting at the top with USAID, trickling down through JSI headquarters in Boston, and most prominently with the ESI Project Team in South Africa under the leadership of the Project’s first COP through early 2011. ESI Project performance demonstrated measurable success across some of the Task areas, particularly within the area of Orphans and Vulnerable Children (OVC) and more limitedly within Training and Capacity Building. While the evaluation findings underscore the sustainability of the OVC work carried out by ESI, the Team did not find significant evidence to indicate the sustainability of the Training and Capacity Building initiatives. ESI End-of-Project Evaluation xiv GLOSSARY Data quality Data are deemed of high quality if they correctly represent the real-world construct to which they refer. Data Warehouse (DW) A 1st generation PEPFAR results reporting database for PEPFAR decision makers and policy makers. Development hypotheses Rational explanation for an event or phenomenon based on what is observed (but not proven). Information system architecture The management processes and rules, systems structure, technical framework, and product technologies for a business or organizational information system. Logic model A conceptual framework that shows the relationships between parts of a program (such as, goals, objectives, inputs, outputs, outcomes, etc.); a Results Framework; a Logical Framework (LogFrame). Partnership Information Management System (PIMS) A 2nd generation PEPFAR results reporting and strategic information system for PEPFAR implementers, managers, decision makers and policy makers. Software Computer software, or just software, is a collection of computer programs and related data that provides the instructions for telling a computer what to do and how to do it. Software refers to one or more computer programs and data held in the storage of the computer for various purposes. Strategic information system A system (preferably computerized) that helps an organization monitor and change its strategies and structures for greater efficiency and effectiveness. It is typically utilized to streamline and quicken the reaction time to environmental changes and to aid in achieving goals and objectives. Strategic information Data and information to support planning, programming, performance monitoring, assessments, and evaluations. Could include geographic, demographic, user and service statistics (disaggregated as needed). Theory of change An outgrowth of repeated successful proofs of a hypothesis; the process by which an intervention or a series of interventions changes a situation from one condition to another. ESI End-of-Project Evaluation xv ACRONYMS AIDS Acquired Immune Deficiency Syndrome AM Activity Manager AMREF African Medical Research Foundation CCM Change Control Meetings CCMT Comprehensive HIV and AIDS Care Management and Treatment CDC Centers for Disease Control (USG) CMMB Catholic Medical Mission Board COTR Contracting Officer’s Technical Representative COP Chief of Party CPPF Cost-Plus-Fixed-Fee (contract) CTO Cognizant Technical Officer CV Curriculum Vitae DHIS Department of Health Information System DSD Department of Social Development DOE Department of Education DOH Department of Health DOJ Department of Justice DQ Data Quality DW Data Warehouse EBHM Evidence-based Health Management ELRC Education Labor Relations Council ESI Enhancing Strategic Information Project FGD Focus Group Discussion FPD Foundation for Professional Development GIS Geographic Information System GFATM Global Fund to Fight AIDS, Tuberculosis and Malaria HAST HIV/AIDS, STIs and TB Programme HISP Health Information Systems Programme HIV Human Immunodeficiency Virus HPCA Hospice Palliative Care Association HQ headquarters IT Information Technology JSI John Snow, Inc. KII Key Informant Interview LOE Level of Effort MATCH Maternal, Adolescent and Child Health (University of Witwatersrand) M&E Monitoring and Evaluation MOU Memorandum of Understanding NGO Non-Governmental Organization NHIC National Health Information Centre NHLS National Health Laboratory Services NW North West Province OVC Orphans and Vulnerable Children PEPFAR President’s Emergency Plan for AIDS Relief PIMS Partnership Information Management System PMP Performance Monitoring Plan PMTCT Prevention of Mother to Child Transmission PCI Positive Community Impact PPL PEPFAR Provincial Liaison PY Program Year ESI End-of-Project Evaluation xvi Q1 Quarter 1 RDQA Routine Data Quality Assessment (process and tool) RFP Request for Proposal RFTOP Request for Task Order Proposal RHAP Regional HIV and AIDS Program RTI Research Triangle Institute SA South Africa SACTWU South African Clothing and Textile Workers Union SAG South African Government SI Social Impact, Inc. SOW Scope of Work STI Sexually Transmitted Infection SKII Structured Key Informant Interview (process and tool) SWOT Strengths, Weaknesses, Opportunities, Threats (program analysis method) USAID United States Agency for International Development USG United States Government WRHI Wits Institute for Reproductive Health and HIV ESI End-of-Project Evaluation 1 I. INTRODUCTION This evaluation results from a request by USAID/South Africa (USAID/SA) for Social Impact, Inc. (SI) to evaluate the John Snow, Inc. (JSI) project, Enhancing Strategic Information (ESI). The ESI Contract was originally awarded for the period July 2008-2013 under terms of a Cost-Plus Fixed Fee (CPFF) contracting mechanism at a level of $22.9 million including budgetary provisions for sub-contracting with 3 other organizations. The following three major changes occurred during the life of the Project that affected its scope as well as the scope of this evaluation:  ESI was originally established as a three-country project covering Lesotho, Swaziland, and South Africa. In May 2011, ESI activities in Lesotho and Swaziland were formally suspended and moved to alternative arrangements with the USAID/SA Regional Health Office. This evaluation is only concerned with ESI in South Africa. The earlier Lesotho and Swaziland components of ESI are only mentioned in this report as they affected ESI project management.  Due to accelerated level-of-effort, the Project ended earlier than planned, in March 2012. A primary focus of this evaluation is a review of the Project’s performance during the period July 2008 through March 2012.  An additional 7-month contract with additional scope of work related to further refinements in the PEPFAR information system was awarded for the period March 2012 through September 2012. Because this evaluation is also charged with making recommendations for future projects, this evaluation also notes ESI activities during the period March-September 2012, and looks beyond September 2012 to a possible new phase of development. ESI in South Africa (referred to variously throughout this report as, “ESI”, the “ESI Project”, and “the Project”), was specifically concerned with information systems that measure and inform the United States Government (USG) President’s Emergency Plan for AIDS Relief (PEPFAR), and the South African Government (SAG) HIV and AIDS program activities. The ESI Project (and thus this evaluation) focused on two distinct areas: 1) refinements to the web-based information system for program reporting and performance monitoring; 2) building the capacity of PEPFAR and SAG implementing partners to collect, use, and report high-quality program data and information. Direct beneficiaries of the ESI Project were USG PEPFAR strategic information managers, PEPFAR implementing partners and staff throughout the country, the SAG DOH and DSD at national level, and SAG provincial and district personnel in 18 designated priority districts. This report begins with an overview of the strategic framework of the ESI Project and an analysis of the Project’s development. Next, the methodology used in this evaluation is discussed, highlighting the scope and limits of the data collection methods applied. The Findings, Conclusions, and Recommendations sections of the report are divided into four parts: 1) The section on Strategic Design Issues discusses the underlying rationale for the ESI Project, and identifies factors that might be considered in any follow-on projects. 2) The Performance Management (Managing for Results) section focuses on how well ESI was managed both internally and in relationship to its sub-contract implementers and USAID contract management; ESI End-of-Project Evaluation 2 3) Project Achievements – Training and Capacity Building section describes what the Project achieved in this programmatic area; 4) Similarly, Project Achievements - the Partnership Information Management System (PIMS) and Geographic Information System section describes what the Project has achieved in the area of enhancing the PEPFAR results reporting and strategic information system. Not all findings and conclusions lead to specific recommendations. However, in instances where recommendations appear, these should be understood as intending to inform the future direction of similar projects. ESI End-of-Project Evaluation 3 II. PROJECT DESCRIPTION The purpose of ESI in South Africa was to improve the quality of HIV and AIDS data as well as monitoring and evaluation of HIV and AIDS programs with a view to better informing South Africa’s policy makers, reducing the prevalence of HIV and AIDS, and improving the quality of life for those afflicted with, and affected by, HIV and AIDS. Funded by PEPFAR, ESI was intended to address PEPFAR reporting and program information needs of a wide range of USG PEPFAR managers and implementers. A very large number of individuals and organizations were direct or indirect beneficiaries of the ESI Project. As administrator of the PEPFAR results reporting and strategic information system, PIMS, (formerly the Data Warehouse), the ESI Project had the authority to interact with reporters into the PEPFAR system which included about 4,000 PEPFAR service facilities, 330 sub-prime and 150 prime PEPFAR implementing partners nationwide, with geographic concentration in four key PEPFAR provinces (North West, Gauteng, KwaZulu Natal, and Western Cape), Activity Managers within USAID and CDC, and the USG PEPFAR Secretariat. The Project also provided training and capacity building assistance to two key SAG departments, the Department of Health (DOH), and the Department of Social Development (DSD)), interacting with a wide variety of SAG managers at national, provincial and district levels. The Project additionally established a province-wide Memorandum of Understanding (MOU) for technical assistance and support activities for HIV and AIDS-related M&E in North West Province. A large staffing component for the Project reflected the wide geographic coverage and technical scope of project activities. A list of the Project’s key organizational clients and a Project Organogram are in included in this report as APPENDIX A – Key ESI Contacts and ESI Project Organogram. The ESI Project, as conceptualized by USAID in its RFTOP and responded to in the JSI technical proposal, was designed with what appears to be a reasonably clear conceptualization of the development problem and interventions to be undertaken. ESI’s technical proposal (dated April 2008) discusses the SAG National HIV/AIDS/STI Strategic Plan, 2007-2011, which notes that partnerships with key donors such as GFTAM and PEPFAR are crucial for the successful implementation of the (SAG) plan, and further recognizes M&E as an important policy and management tool. ESI’s technical proposal goes on to state that, “Many (PEPFAR) partners have dedicated staff to assist in the M&E of their programs. However, as in many other countries faced with increasing reliance on M&E systems to justify (government and donor) performance￾based resource allocations, the M&E sector is an emerging development sector in South Africa. Scarcity of staff at all levels of the Health Information System continues to result in data quality and data management problems. Further, the ESI technical proposal states that the current M&E efforts in government, NGOs and the private sector cannot accommodate the ever-expanding demand for M&E personnel, despite increasing resource allocations for the sector. Prior to inception of the ESI Project, the PEPFAR information reporting system, then called the “Data Warehouse” (DW), had been developed such that it could produce aggregated data for upward reporting, but could not display disaggregated data such as might be useful for monitoring or purposes of making comparisons from reporting units at various levels of the ESI End-of-Project Evaluation 4 reporting network. For example, data were aggregated at the prime PEPFAR implementing partner level such that the contributions of some 200 sub-prime partners could not be identified within the data set. The system also had limited functionality to respond to user queries and requests for analysis and sorting of geo-specific information. The system was found to be non￾user-friendly, and could not promote an information culture where managers and implementers routinely use information for program monitoring, analysis, and decision making. As discussed in ESI Project documents, the development problem around which the ESI Project was designed was a situation where: 1) more robust functionality of the HIV and AIDS data and information systems was needed, and 2) the capacity of users, particularly those with M&E responsibilities, needed to be strengthened. While a clear and rational development problem statement was easily discernible in ESI’s technical proposal, a logic model that would move conceptually from a statement of the development problem to an articulation of objectives (results), sub-objectives (sub-results), and tasks (activities), was not in evidence. Based on a thorough examination of project documents and discussions with ESI staff during this evaluation, it was noted that a formal Results Framework, or logic model, was never fully developed by the ESI Project. Instead, ESI planners referenced the Project objectives prescribed in the original RFP and contract SOW, and further developed the rationale and approaches for accomplishing these objectives. The following six objectives (or “Tasks” as they are commonly referred to in ESI vernacular) were intended to provide an overarching strategic framework for project implementation:  Task 1: Enable partners to build capacity for use of strategic information in health facilities to inform their evidence-based management systems  Task 2: Achieve comprehensive information system development to implement strategic information for OVC, prevention of mother to child transmission (PMTCT), HAST, and other program areas  Task 3: Design and implement action plans for the enhanced use of data within the DOH as well as PEPFAR partners  Task 4: Provide technical assistance for maximizing data quality in results reporting, particularly targeting managerial levels  Task 5: Develop and maintain a USG Data Warehouse and results reporting databases  Task 6: Create high-quality multivariate Geographic Information Systems (GIS) using mapping applications to highlight critical data patterns Although it was not explicitly articulated as a Results Framework or logic model, a change hypothesis could be deduced from Project documents. ESI’s implied theory of change was to improve the functionality of the information system and the data management skills of information system users, thereby improving users’ program management skills, thereby improving the sustainability and quality of HIV and AIDS programs, services, and outcomes. ESI End-of-Project Evaluation 5 III. SUMMARY OF EVALUATION DESIGN AND METHODOLOGY Objectives of the ESI Evaluation were to address the following cross-cutting issues:  To determine the appropriateness of the Project’s development (change) hypothesis;  To determine the adequacy of the Project’s management;  To assess program performance, achievements and gaps;  To recommend approaches for future consideration of USG PEPFAR results reporting and strategic information system planners. These cross-cutting issues were further developed into a list of Key Evaluation Questions. (See APPENDIX B – ESI Key Evaluation Questions). The Evaluation Team used a combination of descriptive and analytical research methods to answer strategic design, project management, and performance assessment questions (i.e., what happened and why), and used a design research approach (i.e., design considerations from a user’s perspective) in answering questions related to design, development, and use of the strategic information system. Mirroring the JSI/ESI Project, this evaluation was national in scope, and included review of both the ESI Project and the results reporting and strategic information system environment the Project was designed to enhance. The evaluation, therefore, included informants at all levels of the extensive PEPFAR operating network in South Africa. The following categories of informants were contacted during the Evaluation:  Level 1-- USG PEPFAR partners and SAG collaborating departments  Level 2-- JSI/ESI and ESI sub-contractors  Level 3-- ESI’s key collaborators among PEPFAR prime implementers  Level 4a-- Field-based prime and sub-prime PEPFAR implementing partners in 4 key PEPFAR Provinces (North West, Gauteng, KwaZulu Natal, and Western Cape)  Level 4b-- SAG provincial, district and sub-district DHIS personnel trained by and/or receiving technical assistance from ESI The list of persons contacted at various levels is included in an appendix to this report. (See APPENXIX C – List of Persons Contacted). The Evaluators were able to identify and contact a relatively large number of people (as listed in APPENDIX C). The maximum numbers of people were identified at each level within the constraints of time and availability. The aggregate of persons contacted therefore represents a built-in selection bias and given this, the Evaluation does not draw generalized statistical “inference” from the data gathered and analyzed. Instead, the Evaluators used a triangulated approach to formulating findings and conclusions based on synthesis of a wide range of quantitative and qualitative data, impressions, and expert opinions. The following data and information-gathering methods were used throughout the Evaluation: ESI End-of-Project Evaluation 6  Review and analysis of information from background documents and other data sources  Structured and open-ended key informant interviews  Focus group discussions  Direct field observations  Online surveys Full details of the methodology used in this Evaluation are presented as an appendix to this report. (See APPENDIX D - Social Impact’s Evaluation Methodology and Work-plan). What follows is a summary of the approaches, constraints, and limitations of the data and information￾gathering methods applied in the Evaluation. Review of background documents and data. An extensive array of background documents from USAID/SA and the JSI/ESI Project files was shared with the Evaluation Team (See APPENDIX E – Documents Referenced). The Team not only used these materials during preparatory phases of the Evaluation, but also occasionally needed to reexamine information contained in a key document for clarification and confirmation. Additional documents were also requested, and these were provided from USAID, ESI, or JSI headquarters. However, delays in receiving many of these critical documents (some received in the closing days of the evaluation), presented a challenge to the evaluation process. Open-ended Key Informant Interviews. Interviews based on the Structured Key Informant Interview Guideline (See APPENDIX F – Evaluation Instruments) were conducted with a total of 52 individuals from 12, Level 1 and 2 key informant organizations/offices. Level 1 and 2 informants included USG PEPFAR results reporting and strategic information agency coordinators, Activity Managers, SAG officials in the DOH and DSD, and ESI Project staff and subcontractors. With few exceptions, interviews held in Pretoria were attended by all four members of the Evaluation Team. During the data synthesis phase of the evaluation in Weeks 5-6, specific time was set aside for team members to review their open-ended interview notes. In this way, information gathered during open-ended interviews was systematically incorporated into Team deliberations on findings, conclusions, and recommendations. [Note: Consensus on impressions was further reinforced through group discussions, preparations for the evaluation debriefing, and the team’s internal review of report drafts]. Focus Group Discussions. The evaluators made a decision to take advantage of the large number of prime PEPFAR implementers clustered in Gauteng Province by conducting 2 FGDs on consecutive days in Pretoria. A variation of the Structured Key Informant Interview instrument was used to guide the first day’s discussion. The second day’s discussion was directed more specifically to gaining a user’s perspective on the Partnership Information Management System. The two sessions were attended by a total of 15 individuals representing 11prime partner organizations. The Evaluators found that both sessions generated frank and open dialogue from participants who voiced a genuine interest in the state-of-the art of M&E in PEPFAR programs. Online Surveys. The use of online surveys was not intended by the Evaluators as a rigorous means of gathering information about project outcomes. Rather, online surveys were envisioned as a supplement to other more direct means of obtaining information (such as key informant interviews) and was intended to be applied to one or more subject areas to augment information obtained from other information sources. Three online surveys were undertaken during the course of the Evaluation. These were: ESI End-of-Project Evaluation 7 Survey 1 Target Group: Purpose: 1,511 Persons ever-trained in an ESI training course To rate trainees’ perception of the usefulness of training to trainees’ work Survey 2 Target Group: Purpose: 95 Project Directors and/or M&E Specialists in PEPFAR Implementing Partner organizations To rate respondents’ knowledge/valuation of the results reporting and strategic information systems design/developments and support services provided by ESI Survey 3 Target Group: Purpose: 100 PEPFAR results reporting and strategic information systems users (including USAID and CDC Activity Managers) To assess user’s results reporting and strategic information systems interests and needs for T/TA in the use of the Partnership Information Management System The 3 online surveys were set-up and administered through Survey Monkey, an online service. Questionnaires were designed, administered, and analyzed according to standard questionnaire research design methodology. Examples of Questionnaires used in the surveys are included as exhibits in an appendix to this report. (See APPENDIX F – ESI Evaluation Instruments). Specific results of the surveys are discussed in relevant findings sections of this report. In general, the online survey method did not produce significant findings, particularly survey 3, which targeted USAID and CDC Activity Managers, as it had an exceptionally low response rate. Threats to Validity. In spite of the availability of a large quantity of Project information, the team encountered an information constraint. From the inception of the evaluation, the Team sought accurate information on the locations and programmatic foci of provincial/district-level prime and sub-prime implementing partners – information required to systematically identify key informants and prepare an orderly itinerary for field visits with appropriate advance notices. In the absence of complete and timely organizational information (and appropriate advance notice to key informants well in advance of start-up of the evaluation exercise), the evaluators were not able to follow the reasonably systematic procedures needed for a balanced well-organized itinerary of interviews and field visits. As a result, with some notable exceptions, the list of organizations and individuals contacted is biased in favor of those most readily accessible and interested in the ESI Project and the evaluation, and does not include a broader cross-section of informants whose views may have differed from those contacted. ESI End-of-Project Evaluation 8 IV. STRATEGIC DESIGN ISSUES Findings. This evaluation was specifically tasked with assessing the appropriateness of the development hypothesis, theory of change, and logic model that were the basis for design of the ESI Project. The problem to be addressed by the ESI Project was that the existing PEPFAR results reporting database, the Data Warehouse (DW), was viewed by users as not being user-friendly, and the DW did not have features that would encourage managers to analyze and use data (from the DW and other data sources) for program planning, monitoring and evaluation purposes. Based on the Evaluation Teams’ understanding of the ESI contract Scope of Work, designers of the Project believed that since the Data Warehouse was an upward reporting system with limited analytical functionality, the data and information contained in the DW was not seen by users at various levels as relevant to their broader programming and performance management information needs. The DW was perceived as not being an adequate source of data/information for programming and performance management purposes, and implementers and managers were perceived as not being motivated and/or lacking in the skills and tools to seek/use strategic data and information (from the DW and other data sources) to strengthen management of their programs. Because of its limited relevance to their management needs, it was believed by designers of the ESI Project that implementers and managers were additionally not highly motivated (and/or lacked skills and tools) to ensure collection and reporting of high quality data into the DW; the quality of data in the DW was an underlying concern. The Project’s development hypothesis was that strengthening the capacity of individuals and institutions to collect, analyze, and use strategic information improves the quality and sustainability of HIV and AIDS programs and outcomes. The Project’s development hypothesis is directly drawn from USAID’s vision for the ESI Project as stated in introductory sections of the ESI Contract No. GHS 1-03-07-00002-00. It states in part, “…The U.S. Government (USG) has undertaken activities to improve information use for policy, planning, and practice, and has strengthened the capacity of health systems to conduct data collection, analysis, information use and decision making. Especially under PEPFAR, USG and its implementing partners have made substantial progress in developing the capacity of local organizations and government counterparts to establish, manage, and sustain these activities. These activities have been developed on the premise that generating demand for and improving the use of data in policy formulation, program planning, monitoring and evaluation improves health services and consequently, health outcomes.” The Evaluation Team found that the Project’s development hypothesis was logically linked to the Project’s theory of change as articulated in ESI’s vision and mission statements, paraphrased here as follows: a) There is need for improvements in the functionality of the information system such that this, b) leads to improvements in the analysis and use of strategic information for decision making and performance management thereby, c) improving users’ performance management skills, vested interested in producing and using high quality strategic data and information to improve program performance thus, d) contributing to the overarching USAID objective of improving the sustainability and quality ESI End-of-Project Evaluation 9 of HIV and AIDS programs, services, and outcomes. Conclusion. The development hypothesis and theory of change for the ESI Project were found to be sound and consistent with USAID programming guidelines. ESI End-of-Project Evaluation 10 V. PERFORMANCE MANAGEMENT (MANAGING for RESULTS) A. OVERVIEW As requested by the Evaluation Task Order, this section of the evaluation report examines how effective ESI management systems, project structure, and USAID contract management were in implementing Project deliverables. To put the ESI Project management and evolution into context, the Figure below provides a list of critical events that changed the direction, cost, and management of the Project over the period 2008-2012. Figure V-1. Major Management Events in the ESI Project DATE EVENT January 2009 New SAG Minister of Health –openness to collaboration with donors in HIV/AIDS & ESI facilitated actual project shifts from focus on PEPFAR partners alone to focus on working with DOH as well April 2009 North West Province MOU signed – beginning of extensive training activities and staff deployment July 2009 USAID requests that Data Warehouse be transferred from Khulisa to ESI December 2009 Health Information Systems Programme (HISP) subcontract terminated by ESI September 2010 Country Manager appointed for JSI/ESI South Africa February 2011 USAID requests ESI to develop PIMS software system May 2011 Former COP departs, Country Manager for South Africa appointed as COP USAID issues stop work order for JSI/ESI in Lesotho & Swaziland March 2012 Khulisa subcontract ends. JSI/ESI project funds ($22,890,925) exhausted and USAID issues new 7-month follow-on contract (March – September 2012) for $2.8 million Other important events that both affected Project management, and were affected by Project management, are listed in the detailed Project Timeline in APPENDIX G – Key ESI Management￾related Events, 2008-2012, and are discussed in greater detail in the subsection following subsections. B. PROJECT PLANNING AND MANAGEMENT 1. Responding to a Changing SOW Findings. During the four years of ESI operation, a number of activities that were originally planned to be achieved over a five year period were significantly accelerated, some at the specific request by USAID, others in response to independent requests by DOH and DSD. This acceleration resulted in part from USG’s shift from the rapid scale-up of sustainable and high quality HIV and AIDS programs implemented by PEPFAR partners, to more intensive support to the SAG to strengthen their results reporting and strategic information for HIV and AIDS programs. ESI was asked to offer large-scale support to SAG in areas of training, capacity building, data quality improvement, and M&E staffing. Examples of activities that accelerated ESI’s ESI End-of-Project Evaluation 11 Level of Effort (LOE) include: 1) intensive capacity-building work in North West Province, 2) support to an 18-month PMTCT project, and 3) creation of a Data Inventory and development of PIMS. Conclusion. ESI was consistently flexible and responsive to the changing priorities of USAID, SAG and PEPFAR. JSI accelerated its LOE, in response to numerous requests and with USAID’s concurrence and thus the contract funding came to an end earlier than originally planned. 2. Planning and monitoring tools Findings. APPENDIX H – JSI/ESI Planning and Monitoring Documentation provides a detailed summary of various planning and monitoring processes and the documentation used in the Project. These are further discussed below.  M&E Plan. The ESI Project never produced an overall ESI Project M&E Plan (which was a contractual requirement), or a Results Framework (which would have additionally been a useful and fairly standard management practice) to use for work-planning or the monitoring of performance. Despite the fact that PEPFAR has no specific level one results reporting and strategic information indicators, an M&E plan could have been developed for internal management purposes based on Tasks and major activities. Without such a plan, ESI had no way to: a) reflect the evolving scope of work, b) track expenditures against activities, c) monitor performance, or d) more efficiently guide the activities of the four subcontractors.  Work Plans, Semi-annual Project Reports and Annual Project Reports. Although the ESI Cost-Plus Fixed Fee Contract included many requirements for work plans, semi-annual project reports and annual project reports, the documents submitted by ESI that were reviewed by this Evaluation Team sometimes contained inadequate detail for appropriate Project planning and monitoring and were not sufficiently quantified. In addition, the reports did not follow a consistent format that would enable comparison over time. The level of detail and inconsistencies found in ESI reporting documents were accepted by USAID. The table found in APPENDIX H outlines all ESI contractual requirements for planning and monitoring documentation, and the degree to which the Project complied.  Specifications for Database Systems Development. Much of the ESI Project focus centered on the development of complex database systems (Data Warehouse, Data Inventory, and PIMS). However, technical user specifications documents were not developed at the beginning of ESI’s work on any of these systems to guide ESI’s software work over the four years of ESI’s operation or to clarify stakeholders’ expectations of the systems. A number of stakeholders (PEPFAR, USAID, CDC and PEPFAR partners) made a range of demands on the systems, creating additional work for ESI and its subcontractors, and contributing to considerable delays in delivering satisfactory software systems. ESI End-of-Project Evaluation 12  Other Project Documentation. As part of the preparation for the new 7-month contract (March -- September 2012), ESI did develop a draft User Requirements Specification Document in collaboration with USG’s strategic information colleagues3. A strategic planning workshop was held in September 2009 with assistance from an external facilitator and was attended by JSI/HQ staff and staff from two ESI subcontractors. A draft Project Strategic Framework4 was developed at this workshop and included a general description of ESI’s vision, mission, values, and a SWOT analysis. However, the Framework did not detail any specific activities or Project targets. A Performance Monitoring Plan (PMP)5 was only developed by ESI in November 2011, less than six months before the ESI Project ended. While the purpose of the PMP is stated as describing targets by year and key indicators that will be monitored over the next five years, it does not accomplish the former, and the time frame for the latter is not feasible. Conclusion. Although there are recent indications of improvements in program planning and management, such as developing the User Requirements Specification Document, which is a positive development and should help avoid confusion about realistic expectations for the balance of the Project in terms of software completion, there were several, major gaps as discussed above. ESI’s unevenness in timely use of program monitoring and evaluation tools with sufficient detail and consistency of format made it difficult for: a) ESI or USAID to adequately track performance over time, and b) ESI, USAID or this Evaluation Team to measure achievements against plans. C. PROJECT STAFFING AND ORGANIZATION 1. Organization of staff and LOE Findings. The ESI Project Organogram shown in APPENDIX A of this report (current as of 2012), is an organizational arrangement that evolved several times over the life of the Project. The six Tasks outlined in the original Scope of Work around which ESI staff were organized, turned out to be a somewhat artificial conceptualization of the Project as it evolved ESI responded to the changing needs by revising staff organization, and continued to re-adjust its staffing pattern (including subcontractor staff) to accommodate ever-changing Project needs. The Project experienced several staffing configurations over its life, as was observed when the Evaluation Team compared the original and revised organization charts. As an example of these structural changes in the Project, JSI/ESI closed Task 3, Data Use, in January 2011 and reallocated Task 3 activities to Tasks 1 and 4. By March 2012, the largest number of ESI staff were placed in Task 4 (data quality), and Task 5 (decision support/databases). In Task 4, the Task Lead sits in JSI/ESI and eight staff members are placed in DOH. For Task 5, seven of the nine staff members, including the Task Lead, are provided to ESI by the subcontractor Khulisa. 3 Partnership Information Management System. Draft User Requirements Specification for a system to monitor and report on the PEPFAR program in South Africa and progress towards implementation of the Partnership Framework. 07 March 2012. Prepared by Derek Kunaka, JSI/ESI. 4 Enhancing Strategic Information. Strategic Framework August 2009. 5 Enhancing Strategic Information Project: South Africa. Performance Monitoring Plan. 4 November 2011. ESI End-of-Project Evaluation 13 By March 2012, 37 full time staff members, including the Chief of Party, were being carried by the prime contract. There is evidence from correspondence in Project files that these various re-organizations caused internal Project tensions and may have presented a challenge to staff productivity. While staffing changes were driven by program demands, ESI management may have been better able to manage these organizational adjustments and re-allocations of LOE if it had had a management tool available to assist with strategic staffing projections and decisions. However, ESI’s system of LOE reporting did not enable the Project to track LOE effort by various tasks (or even by broad categories of activities as required by its Contract. . Conclusion. The constant shifts in strategy, policy, and direction throughout the life of the ESI Project significantly impacted the management and outcomes of the Project in a negative way, evidenced in loose and poor oversight of the project and failure to fully meet all project objectives. ESI’s system of LOE reporting did not enable the Project to track LOE effort by various tasks or even by broad categories of activities as required by its Contract. Such a tool may have facilitated ESI’s monitoring of LOE and management of periodic re-organizations. 2. Quality of staff Finding. Overall, the Evaluation Team was told by individuals interviewed that ESI staff members (including staff provided by subcontractors) were generally competent and were always very responsive to requests for assistance. This comment was reiterated by the senior DOH official for strategic information during an interview with the Evaluation Team. A review of the CVs of staff in key positions in relation to their job descriptions confirmed that senior staff members were well-qualified for their positions. Concerning leadership and senior management, significant management problems were reported to have existed under the first COP, who, while technically qualified, lacked any significant or relevant management experience. The Evaluation Team could not ascertain the degree to which JSI/HQ may have tried to supervise the first COP. Despite his poor management performance, the first COP remained on the Project until early 2011.6 Stakeholders interviewed by the Evaluation Team assured the team that the quality of management improved dramatically when the second COP was put in place, and instituted a more rigorous system of planning, budgeting, and monitoring. Conclusions.  ESI and subcontractor staff members were generally perceived by stakeholders as being well-qualified for their positions.  The Project experienced management problems at the COP level in its early years, but the quality of management has dramatically improved since the first COP was replaced. 3. ESI staff deputed to DOH offices 6 For detailed information regarding the Project timeline, please refer to Appendix G. ESI End-of-Project Evaluation 14 Findings. The ESI project funded HIV and AIDS strategic information positions in selected SAG DOH offices at national, provincial and district levels. Health officials in North West Province, where ESI posted a number of staff, told the evaluators that relationships between themselves, the ESI Project, and the seconded ESI staff were excellent. Health officials in Dr. Kenneth Kaunda District (based in Klerksdorp) in North West Province indicated that ESI secondment of the staff was an approach that fit will with their long term vision of upgrading the M&E and performance management skills of health workers in the district. They were very sure that the overall culture for generating and using high quality data and information for planning and decision making had been substantially improved at all levels of the health service delivery system within the district as a result of the highly focused, systematic and conscious efforts of the two ESI seconded staff. The evaluators were told that the district was in the process of open recruitment for these staff positions and that had recently been budgeted for within the government structure (for which the seconded staff were free to apply). As the evaluators heard, “We found that ESI had no hidden agenda. They made decisions jointly with government, respecting government’s wishes and needs, keeping in mind that not all international partners work that way. We’ve had an open flow of communication. The seconded staff had been fully integrated into the district office. People on the ground didn’t even know the staff members were actually employed by ESI” Dr. Uma Nagpal, Chief Director, Dr. Kenneth Kaunda District Health Services Conclusion. In North West Province, specifically in Klerksdorp, ESI’s approach of secondment of trained M&E specialists within the SAG structure was reported by SAG officials to be very effective. 7 D. MANAGEMENT OF SUBCONTRACTORS Findings. Three subcontractors were initially included in the ESI proposal to work under the prime contractor, JSI: Khulisa Management Services, Health Information Systems Programme (HISP), and Tulane University School of Public Health and Tropical Medicine (later named Tulane International). A fourth subcontractor, Manto Management, was added for a limited period early-on in the Project. By the end of the project, only one of the subcontractors, Khulisa Management Services, was still engaged with ESI. Manto Management was terminated by ESI in June 2010; HISP was terminated in December 2009, and Tulane University was moved out of the ESI Project in consultation with USAID to a separate USAID funding arrangement in September 2011. A summary of subcontractor roles by ESI Project Tasks, as originally envisioned in the Contract, appears in Figure V-2, below.8 7 More needs to be understood about the factors that led to success of this model in that district. 8 APPENDIX J – JSI/ESI management of subcontractors provides a detailed of summary of subcontractor financial management. A review of the initial ESI Contract reveals that subcontractor budgets comprised almost half of the original ESI budget, so their roles in the ESI Project were intended to be significant. At the end of the ESI Project, March 2012, ESI financial data show that HISP had spent $594,831, only 14 percent of its original budget; Khulisa had spent $4,666,718, 97 percent of its original budget; and Tulane had spent $1,757,467, 100 percent of its original budget. Manto Management had spent $104,206 (no original budget was available). ESI End-of-Project Evaluation 15 Figure V-2. ESI Subcontractor Roles by Task TASK SUBCONTRACTORS 1 Capacity building -- 2 Orphans and vulnerable children (OVC) Manto, Tulane 3 Data use HISP 4 Data quality HISP, Khulisa 5 Decision Support Systems (Data Warehouse and PIMS) and training Khulisa 6 Geographic Information Systems Khulisa Relationships between ESI and two key information system developers, Khulisa and HISP, were seen as contentious and acrimonious, as reported by ESI staff, USAID staff, and subcontractor staff. In spite of what appeared to the Evaluation Team to be an appropriate level of intervention by JSI headquarters, issues were never satisfactorily resolved with either subcontractor. The HISP subcontract was terminated before completion, and the Khulisa subcontract ended along with the ESI contract in March 2012, but Khulisa was not carried over to the new 7-month contract award. These problematic subcontractor relationships created a somewhat dysfunctional atmosphere for accomplishment of the intended ESI work in the area of information systems development. There was disagreement on a range of issues including staffing levels, intellectual property, compliance (particularly on invoicing), personality conflicts, deadlines, and the technical quality of work performed. This appeared to be particularly true in the areas of information systems development and support to the DOH and DSD, going back to the change in PEPFAR orientation from an emergency approach to a more sustained approach; and the change from USAID assisting only PEPFAR partners to assisting SAG as well. This resulted in some of the original tasks and skills envisioned for and required by subcontractors changing, further contributing to confusion of roles and expectations of the subcontractors. Conclusions. Ongoing conflicts between ESI and two of its subcontractors distracted the Project from its goals and impeded the Project’s performance. Factors that led to conflict included lack of clear direction from USAID and ESI about database scopes and deadlines as well as the shift in PEPFAR Policy. E. FINANCIAL MANAGEMENT 1. Financial reporting and accounting Findings. The systems that were put in place by ESI South Africa and approved and supported by JSI/HQ in Boston to track financial expenditures (and LOE reporting) only enabled financial tracking and reporting by line item categories (salaries; indirect costs/overhead; consultants; travel, transportation and per diem; equipment, etc.), subcontractor; and specific country (South Africa, Lesotho and Swaziland). ESI End-of-Project Evaluation 16 ESI’s financial system could not track expenditures by the six Project Tasks or activity (i.e., data quality, training, capacity building, OVC, database development and management, etc.). During the first two years of the Project, budgets were prepared for large, free-standing activities such as the Data Warehouse, data inventory, and the PMCT project. Budgets appeared to have been largely illustrative and were not used rigorously for financial planning or monitoring. Even after the creation of some activity budgets, expenditures were still not tracked by activity. Quarterly financial reports were submitted to USAID, as required by contract, following only the line item breakdown in the original contract. ESI was accorded 100 percent line item flexibility. Conclusions. Throughout the Project, the absence of an ESI financial system that could track and allow monitoring of Project expenditures by activity area was a major failing of ESI Project management 2. Burn rate analysis Findings. The Evaluation Team learned that the ESI Project exhausted its funding in March 2012, more than a year before the planned end of Project (July 2013), and sought to determine more specifically: a) what kinds of activities the funds were spent on, b) when funds were spent, c) by whom (ESI and/or subcontractors), and d) in what countries (South Africa, Lesotho and/or Swaziland) funds were spent. An analysis of ESI’s burn rate was provided to the Evaluation Team by JSI/HQ, and was annotated in greater detail by the Evaluation Team based on information gathered from interviews with ESI Project management. APPENDIX K illustrates the trends in ESI expenditures over time and the factors that affected the rates of expenditures. (See APPENDIX K – ESI Burn Rate Analysis). It can be seen from the burn rate analysis that most expenditures are consistent with the general acceleration of Level of Effort in the Project, particularly the rapid expansion of technical assistance support activities with the DOH. A few other high-expenditure items were noted. For instance, when the Data Warehouse was transferred from Khulisa to ESI in Program Year (PY) 2, the transfer entailed unusually high overhead costs. Of note also, ESI’s inputs in Lesotho and Swaziland were more costly than originally envisioned because, USAID later requested ESI to establish country offices in these two countries rather than operating via periodic TA visits from the ESI South Africa office. In addition, after USAID issued a stop work order for activities in Lesotho and Swaziland (May 2011), close out costs were high in Lesotho due to Lesotho labor laws that required buying out country-based salaries. Conclusion. ESI was able to produce financial information that demonstrated that JSI and the Project were keeping an appropriately close watch on Project expenditures and, with a few exceptions, expenditure rates were consistent with the accelerated pace and Level of Effort of the Project. 3. Cost saving measures Findings. The ESI Project instituted several cost-saving measures during its four years of operation to stretch remaining funds as far as possible. These included: ESI End-of-Project Evaluation 17  ESI instituted cost saving training measures: reasonable per diem rates, modest lodging, having attendees (both implementing partners and SAG) assume costs of venues for meetings, etc. As a result, ESI staff reported, participants who came to trainings were motivated to come for the training itself, not for the ‘perks’. The cost savings allowed many more people to be trained.  ESI outsourced printing costs for training materials, which was less expensive than printing in-house.  ESI also reduced travel costs for North West Province by mandating car-pooling and by consolidating field visits. Conclusion. In light of its need to accelerate LOE and expenditures, JSI/ESI demonstrated that it took appropriate efforts to conserve and stretch its financial resources. F. USAID MANAGEMENT OVERSIGHT 1. Ongoing management by USAID COTRs and the Contracts Office Findings. USAID never formalized the changes in ESI’s activities that were either not specifically outlined in the original SOW or that were greatly accelerated due to changing conditions in South Africa. USAID did recognize that funds were being spent faster than anticipated, but could not increase the Project’s financial ceiling because the overall IQC ceiling had already been reached. The Evaluation Team found that the Project’s accelerated burn rate did not present itself as a concern to USAID. Because of the pace at which the Project was moving, priorities and expectations were not always clear, and there were difficulties in tracking how rapidly funds were being spent and on exactly which activities. JSI reported that at times, repeated requests had to be made to USAID to secure the next financial obligation, causing ESI at one point to nearly close operations (January 2011). At another point, an email sent by USAID to JSI requested ESI to “localize” all three country projects. The directive to “localize” was generally understood to mean that JSI should turnover management of ESI to a South African company. How such a change would have been accomplished legally and contractually and at very short notice was unclear, and in any event, the directive was never actually enforced. Four different USAID COTRs managed the ESI Project over the course of four years. While relations between all COTRs and JSI/ESI were reported to be collaborative in most cases, the rigor of engagement and oversight varied among the four. Conclusion. USAID Management of the ESI Project was very erratic and did not exercise appropriate oversight and documentation of the rapidly changing conditions and opportunities in South Africa, within the Mission, or across the PEPFAR Policy landscape of South Africa. USAID placed considerable demands on JSI/ESI to demonstrate flexibility and responsiveness in a dynamic development environment and for the most part, JSI/ESI performed adequately in meeting this challenge. 2. Standard Contractor Performance Reports ESI End-of-Project Evaluation 18 Finding. The ESI Task Order Contract stipulated that an annual review of ESI’s performance on this Task Order was to be undertaken jointly by the Cognizant Technical Officer (CTO)9 and the Contracting Officer. This annual review was to evaluate ESI’s overall performance against the targets and indicators established in the Task Order; and to evaluate work quality, cost control and effectiveness, timeliness, customer satisfaction by USAID, customer satisfaction by end-users, and the effectiveness of key personnel and subcontractors. The Evaluation Team was only able to locate one such annual evaluation, a Standard Contractor Performance Report, which covered the period October 1st, 2008 – September 30th, 2009. ESI was rated “excellent” for quality of product or service, “good” on cost control, “good” on timeliness of performance, and “fair” on business relations. Further, USAID commented that the Contractor “substantially exceeded the contract performance requirements without commensurate additional costs to the Government”. Conclusion. USAID did not conduct annual reviews of ESI’s performance as was stipulated in the ESI Task Order Contract. 3. ESI in Lesotho and Swaziland Findings. Initially, the ESI project included Lesotho and Swaziland, as well as South Africa, with the HIV/AIDS strategic information components of Lesotho and Swaziland being managed by USAID’s Regional HIV/AIDS Program (RHAP). Having Lesotho and Swaziland as components of a project based in South Africa with a resulting heavy emphasis on South African requirements, appeared to work satisfactorily at the beginning of the Project. However, for reasons the Evaluation Team could not determine, USAID management of that period decided to suspend the JSI/ESI Lesotho and Swaziland components of the Project, and to shift those country activities to the USAID Regional Health Office and different contract arrangements. The rationale and timing of this move, and USAID’s process of identifying alternative contractors to replace JSI for the continuation of work in Lesotho and Swaziland, was not clear. There were also additional costs to ESI in South Africa for closing out its involvement in the two countries (discussed above in Section E.3). To a large extent, the localization aspect of the ESI Project was suggested by USAID to be outside the scope of this evaluation. Conclusion. Separation of the Lesotho and Swaziland components of the original project from the JSI/ESI in South Africa may have been awkward and costly for JSI/ESI at the time, however, in the final analysis, all parties with whom the evaluators spoke, appeared satisfied that this separation was achieved. It is not clear whether the localization of the Lesotho and Swaziland components of the project led to an improvement in project performance and attainment of project outcomes and objectives. 4. PEPFAR Partners M&E Meetings Findings. Convening of PEPFAR partners’ meeting is the responsibility of the PEPFAR Secretariat, USAID or CDC, and not ESI, although ESI has helped facilitate PEPFAR partner M&E meetings by finding space and hiring facilitators. In the earlier years of the Project, PEPFAR convened regular meetings of the PEPFAR partners’ M&E staff to review current issues related 9 Older term for a USAID Contracting Officer’s Technical Representative (COTR). ESI End-of-Project Evaluation 19 to their reporting requirements. These meetings were reported to be useful; however, there has not been a PEPFAR partners’ meeting in the past two years. Input from participants in this Evaluation’s Focus Group Discussions confirmed that there is keen interest among PEPFAR partners not only in PEPFAR-related reporting issues but in the broader field of M&E technology in development assistance. The Evaluation Team observed that the rapidly-evolving requirements for PEPFAR reporting under PEPFAR-II10 as well as similar concerns of PEPFAR strategic information systems planners and managers, would also probably be facilitated through regular partner M&E forums. Conclusion. Meetings of PEPFAR partners’ M&E staff have, in the past, been useful and should be continued. Recommendation. The PEPFAR Secretariat, USAID and CDC should designate an individual who will be responsible for convening PEPFAR partners’ M&E meetings at regular intervals. These meetings would serve the purpose of providing a forum for interested PEPFAR partners to engage in professional information sharing on M&E methodology, PEPFAR reporting, and use of data for strategic decision-making. Aside from reporting into the PEPFAR results reporting and strategic information system, such regular meetings might be an additional way that the PEPFAR Secretariat can reinforce a sense of shared purpose among PEPFAR partners. G. AUDIT RECOMMENDATIONS ON THE DATA WAREHOUSE Findings. A special audit was undertaken in July 2011 by the Regional Inspector General/Pretoria of USAID Southern Africa’s HIV/AIDS treatment activities. This audit intended to determine whether the USAID Mission’s HIV/AIDS treatment activities were achieving their main goals. Although this audit was not intended to focus specifically on the ESI project, a significant portion of its inquiry centered on the Data Warehouse, which was at that time in the process of being improved and integrated with other information systems functionalities by the ESI Project. Major conclusions of the Audit that concerned the Data Warehouse are:  There were a number of errors in data generated by the Data Warehouse and JSI could not explain the discrepancies  As of December 31, 2010, USAID had spent over $4.6 million developing and maintaining a Data Warehouse that has not provided consistently reliable information.  There were problems related to co-management of the Data Warehouse by USAID and CDC. Specifically related to the Data Warehouse, the Inspector General’s report recommended that USAID/Southern Africa:  Conduct an assessment of the effectiveness of the Data Warehouse, and  Draft and approve a mutually agreeable co-management plan for the Data Warehouse with the Center for Disease Control and Prevention. 10 Refer to PEPFAR Next Generation Indicators Reference Guide, http://www.pepfar.gov/documents/organization/81097.pdf. ESI End-of-Project Evaluation 20 In response to the second recommendation, the Evaluation Team was informed that an agreement is in progress. USAID responded to the first recommendation by including an assessment of the Data Warehouse as part and parcel of this Evaluation of the ESI Project. The enhancements to the PEPFAR strategic information systems being undertaken by ESI at the time of the audit were based upon, and were an outgrowth of, the earlier generation Data Warehouse. According to this Evaluation’s scope of work, a review was made of reporting and analytical functionality of PIMS (formerly, the Data Warehouse). This Evaluation found that, based on the general impressions offered by key informants, PIMS is perceived by users as generating timely data and information for upward reporting. PEPFAR prime implementing partners who report into PEPFAR felt that data are being entered routinely without major concerns for the accuracy of data. The USG PEPFAR Secretariat in the embassy reported that the quality of data and reports now being generated by PIMS are acceptable for upward reporting to OGAC. A systematic data quality audit (entailing rigorous sampling and verification of data validity and reliability and tracing the causes and possible ways of correcting any data discrepancies found) was beyond the scope of this Evaluation. However, the Evaluation Team did investigate the issue of data reliability of PIMS and learned that procedures to control and correct for data and reporting errors are now in place. As reported by ESI, as well as system reporters/users, there are currently systematic verification procedures at each level of data input, from facilities level, through sub-prime and prime levels to the USAID/CDC Activity Manager level that are in place and working. Activity Managers reported that they use information reported in PIMS as one source of information to monitor the field activities, and cross-check reported data if there appear to be discrepancies. The data quality control procedures that are in place do not guarantee that the data discrepancy problems observed by the Inspector General’s audit have now been corrected. Conclusions.  USAID responded to the recommendation of the Inspector General’s audit on the Data Warehouse by including an assessment of the Data Warehouse as part and parcel of this Evaluation of the ESI Project  Procedures are now in place to control for the reliability of PIMS data, but the accuracy and reliability of PIMS data should be reviewed in the future after some reasonable period of time has elapsed to allow these procedures to work effectively.  A Memorandum of Understanding between USAID and CDC in response to the Inspector General’s recommendation regarding working relationships in support of PEPFAR results reporting and strategic information systems (formerly the Data Warehouse) has been drafted by USAID and is in Atlanta awaiting approval from CDC. Recommendations. (1) At some time in the future, perhaps within the next year, and perhaps in conjunction with the design or implementation of any new information system projects, a systematic data quality audit of PIMS should be undertaken. This audit would entail rigorous sampling and ESI End-of-Project Evaluation 21 verification of data validity and reliability, and trace the causes and possible ways of correcting any data discrepancies found. (2) Finalize an MOU between USAID and CDC on strategic information, as recommended by the recent Inspector General audit. This should be done in conjunction with development of an overall policy and structure for development of future PEPFAR results reporting and strategic information systems as discussed in Section VII-F. 2 of this report. (3) Once a reasonable period of time has elapsed (perhaps another year), a systematic data quality audit of PIMS may be indicated as part of the design or implementation of any new information system projects. ESI End-of-Project Evaluation 22 VI. PROJECT ACHIEVEMENTS – TRAINING AND CAPACITY BUILDING This section of the Evaluation report discusses the extent to which the ESI Project met its programmatic objectives concerned with training and capacity building, including efforts to strengthen the quality and use of strategic information for improved program management. Findings were mainly derived from reviews of Project documents and anecdotal accounts and impressions as reported by key informants11. In an effort to complement qualitative information gathered in this evaluation, the Evaluation Team conducted three on-line surveys of ESI stakeholders, two12 of which are discussed in APPENDIX L – Analysis of Online Surveys, and one of which was targeted to a pool of 1,511 (out of a total of 2,278) persons trained in the ESI Project. A discussion of findings, conclusions, and recommendations in this section covers ESI Project Task Areas 1-4 (See the Project Description section of this report for a list of Task Areas)), and responds to issues raised in this evaluation’s key questions. A. TRAINING PROGRAM PLANNING Findings. ESI’s Task 1: Capacity Building was primarily operationalized in the form of training courses covering various subjects related to the development and use of HIV and AIDS-related strategic information. The Project was tasked with building upon previous PEPFAR-funded strategic information capacity building efforts using a more robust capacity building program that included a graduated set of varied and higher-level training. The contractual requirements underscored the importance of sustainability, requiring the training to be provided by local training institutions as well as incorporating a more rigorous system of evaluating training effectiveness, with a focus on creating cohorts of learners and aggressively recruiting and involving SAG employees. ESI undertook a systematic approach to identifying training needs. As described by ESI training staff, the overall objective the ESI training program was to identify a gradation of the most essential skills needed in PEPFAR related programs to improve uses of strategic information. Priorities set in consultation with SAG officials thus covered a range of skill sets primarily targeted to public sector workers. At the lowest level of the system, data capturers were given high priority in the training program since they were considered to be critical to improving the quality of reporting at the data entry level. Information system users were also targeted through orientation training for managers in the use of the DHIS. At yet another level of capacity building, customized small group training and on-the-job mentoring were used to support a special cadre of ESI personnel who were seconded to work inside the SAG system as M&E specialists. 11 Descriptions contained in this section are illustrative and not intended to be exhaustive. For more details on ESI Project activities, the reader is referred to quarterly and annual ESI Project reports. 12 A third survey targeted to PEPFAR information system users received only a few responses and was not analyzed. ESI End-of-Project Evaluation 23 ESI documents show that it routinely conducted post-training assessments. In addition, ESI also held two follow-up workshops in which previously trained individuals were invited to assess the extent to which their training experiences had helped them in their work. As reported by ESI training staff, post-training assessments and follow-ups were important ways ESI used to ensure that training programs were being continuously refined for maximum effectiveness. Conclusion. There was evidence that ESI planned its training program systematically in consultation with SAG. The overall training covered workers at different levels of the system and there were procedures in place for post-training assessments and follow-up. B. TRAINING COVERAGE Findings. The evaluators conducted several interviews with ESI training staff and undertook an extensive review of ESI Project records and documents pertaining to training activities. According to ESI records, ESI’s team of 4 specialized trainers trained a total of 2278 individuals over the life of the ESI Project including, 1782 individuals from DOH and 996 individuals from PEPFAR partner agencies. In reference to the DSD, although no evidence could be found in records, it was reported to the Evaluation Team in interviews with JSI and DSD staff that ESI also directly trained staff within DSD. Of the total individuals reached with ESI training, 568 individuals were trained in Gauteng Province, 27 were trained in Mpumalanga Province, and 84 were trained in Northern Cape Province. Although the training program overall was intended mainly to complement and support ESI’s capacity building efforts with SAG personnel, some of the training events did also accommodate attendance of individuals from private sector PEPFAR implementing partners. More than one implementing partner reported that in the beginning days of ESI training, announcements for training opportunities used to come from the PEPFAR Secretariat, but that at some point those announcements stopped. Thus there was a feeling that they had been overlooked for ESI training. It appears that ESI’s focus on training of SAG personnel was consistent with the Project’s priorities, but that the rationale for emphasis on public sector training had not been adequately communicated to PEPFAR implementing partners. For purposes of this evaluation and the possible need for follow-up contact with trainees, the evaluators culled the total list of 2278 persons ever-trained by ESI down to a list of 182913 individuals trained one or more times who also had working e-mail addresses. Table VI-1 below, provides a breakdown of those 1829 attendees by type of training received and shows that nearly half (48%) of ESI’s training for these 1829 attendees was for DHIS training. Table VI-1. Representative sample of persons trained by type of training Training course Attendees Advanced Import/Export, CCMT, Data analysis, Training of Trainers 75 Data capturer(s) 109 Department of Health Information Systems (DHIS), combined with 47 13 Four hundred forty-nine (449) individuals were lost to follow-up due to invalid e-mail addresses. ESI End-of-Project Evaluation 24 Geographic Information Systems (GIS) DHIS Level 1 542 DHIS Level 2 280 Evidence-based health management (EBHM) 536 Monitoring and Evaluation (M&E) 73 Prevention of Mother to Child Transmission (PMTCT) 65 Routine Data Quality Assessment (RDQA) 103 TOTAL 1829 Conclusion. The Project trained a significant number of individuals in a variety of course subjects. Due to the fact that the majority of persons trained were SAG personnel, PEPFAR implementing partners did not receive equal access to training opportunities. C. RELEVANCE AND SCOPE OF TRAINING CURRICULA Findings. A review of the training materials suggested that the curricula developed by ESI were clear, relevant, accessible, and practical for applied use across a range of SAG and PEPFAR Partner roles and responsibilities. ESI developed a series of training curricula covering a range of technical areas relating to strategic information, monitoring and evaluation, and data quality. ESI developed 3 training curricula specifically focused on improving the quality and use of the SAG Department of Health Information System (DHIS). In keeping with ESI’s strategy to give greater priority to SAG strategic information training needs, ESI developed a capacity building model for training delivery, which incorporated real DHIS datasets and drew from SAG employee and PEPFAR partner experiences with typical data challenges in order to reinforce the utility and application of the training to the work setting. In addition to information gathered through informant interviews and reviews of ESI documents, the evaluators also conducted a supplementary online survey to provide some additional information about trainees’ perceptions regarding the relevance and usefulness of training received. (See APPENDIX L – Analysis of Online Surveys). A list of 1829 persons trained in one of more ESI training courses who also had a working e-mail address, was culled down to a sample of 1511 to eliminate duplications, (that is, names that appeared more than once in the list). The online survey of 1,511 persons trained through ESI courses showed that trainees were generally satisfied with the training received, and that they felt that the training had assisted them in their work. As reported by key informants, there is continuing demand for additional training opportunities, particularly in the areas of DHIS 2, Evidence-based Health Management, and Data Quality. In terms of an ESI program objective of transferring training capacity to local training resources, ESI documents show that it presented Data Quality, Data Use and GIS courses in partnership with the University of Pretoria twice a year and 3 of these courses were accredited by the University for Continuous Professional Development points. In another effort, an ESI Capacity Building Model and Curriculum employed a training-of-trainers approach intended to encourage the sustainability of training capacity and development of a capacity building network for data quality and use among SAG employees and PEPFAR Partners. ESI staff identified and recruited ESI End-of-Project Evaluation 25 these “network facilitators” who underwent Facilitation and Mentoring courses as well as co￾facilitation coaching provided by ESI trainers until they met ESI standards. One prime PEPFAR partner, the University of Witwatersrand, Wits Institute for Reproductive Health and HIV (WRHI) was enthusiastic about its ESI training in Evidenced-based Health Management, DHIS, and ESI Capacity Building Training, and described to the Evaluation Team how, as a result of its ESI training, WRHI was continuing to expand and strengthen its work in mentoring and capacity building support to its network of PEPFAR sub-prime implementing partners. It should be noted, however, that not all evaluators were in agreement about the effectiveness of the ESI Capacity Building Model and Curriculum. As was related to one evaluator, at least some of the trained trainers had expected that once they were deemed to be “official trainers” capable of delivering ESI training, they would be engaged, not in their own organizations, but by ESI to conduct ESI training. In reality, all training organized and conducted by the Project was primarily provided by permanent ESI training staff, and did not substantially involve the use of the ESI trained trainers. At least some of the “official trainers” may have been disappointed by their experience with the ESI Capacity Building Model and Curriculum due to a misunderstanding about the program’s purpose and intent. Conclusions.  ESI training was generally useful/relevant and well-received by SAG, PEPFAR implementing partners, and the individuals trained, and there appears to be a continuing demand for related training, particularly in the areas of DHIS 2, Evidenced-based Health Management and Data Quality.  ESI undertook effective efforts to address a program objective of involving local training institutions and resources in the conduct of sustainable training/capacity building in strengthening strategic information systems. Recommendation. Development of sustainable training capacity, training- of-trainers (within SAG or external local institutions), and support for training of SAG personnel, as determined by training needs assessments at the time, should be part of any future technical assistance to SAG in use of HIV and AIDS-related strategic information. D. BUILDING CAPACITY IN STRATEGIC INFORMATION IN HIV AND AIDS PROGRAMS FOR PMTCT, PALLIATIVE CARE, AND OVC Findings. ESI’s Project Task 2 required the Project to develop appropriate opportunities to assist the SAG and PEPFAR implementing partners with strengthening their use of HIV and AIDS-related strategic information. This resulted in several actives discussed here.  Prevention of Mother to Child Transmission (PMTCT). ESI responded to the DOH request for support in the roll-out of capacity-building efforts across the 18 priority districts for PMTCT and was a catalyst in the country-wide alignment and standardization of PMTCT indicators, as well as demonstrating the merits of improving data quality.  Palliative Care. Although ESI did not provide any substantial or special support to the area of palliative care, Hospice Palliative Care Association (HPCA) of South Africa is a ESI End-of-Project Evaluation 26 PEPFAR prime implementing partner, and as such, had access to and received assistance from ESI in use of the Data Warehouse and PIMS. HPCA, an NGO that provides technical assistance to some 280 hospice service providers who are seeking to obtain South African hospice care accreditation, has its own internal information system but has also received occasional training and technical assistance from ESI. The M&E Director at HPCA found ESI to be “responsive and helpful” in meeting their agency’s needs for occasional information systems support.  Orphans and Vulnerable Children (OVC). In the area of OVC, as reported to the Evaluation Team by a wide range of key informants including USAID, SAG and PEPFAR partners, the achievements ESI realized through a MOU between USAID and DSD and supported by ESI, demonstrated an unprecedented, productive, and highly positive relationship. ESI’s M&E Specialist was assigned to work onsite with DSD, and reportedly accomplished many objectives. Of note, the ESI M&E Specialist facilitated alignment of the PEPFAR and DSD OVC indicators, which are now used nationally by all PEPFAR partners. ESI additionally worked with DSD to design and implement a baseline assessment of DSD staff capacity for M&E at the district, provincial, and national levels as well as a comprehensive assessment of PEPFAR partners’ needs for information management systems. DSD anticipates that the OVC information system that was assisted by the ESI Project should be fully operational by autumn of 2012. An additional component of support to the OVC program which was of benefit both to USAID and SAG personnel involved with OVC programs was carried out by the Tulane University School of Public Health and Tropical Medicine. Tulane researched successful models for OVC service delivery and published an extensive series of in-depth case studies and reports documenting best practices across a range of OVC programs in South Africa. Conclusions.  Securing USG and SAG agreement on, and actually creating and fully-implementing a single database for OVC represented an ESI best practice.  The approach used by USAID and ESI in developing a productive USG/SAG working agreement was fundamental to the success of the OVC collaboration. Recommendation. For the future, the approach used by USAID and ESI to establish working agreements (, i.e., a MOU) to guide the contractor’s work might be considered as a model for replication. E. CAPACITY BUILDING IN DATA QUALITY AND ENHANCED USE OF DATA Findings. Early on, ESI’s Project Task 3, designed to develop action plans for data use among PEPFAR partners and DOH to improve data quality for HAST and PMTCT programs for monitoring and decision-making, was effectively dismantled and various aspects were absorbed into Tasks 1 (training) and 4 (data quality). According to ESI staff, re-configuration of Task 3 was based on strategic considerations and efficiencies. For example, “enhanced use of data” was covered conceptually in the Evidenced-based Health Management training course, and it was felt that more individuals could be reached through training programs than through customized ESI End-of-Project Evaluation 27 assistance to individual organizations. The aim of ESI Project Task 4 was to develop a strategic and comprehensive approach for improving data quality within the DHIS as well as the PEPFAR reporting and information system. Specifically, the ESI contract called for the establishment of integrated data quality enhancement techniques with approaches to promoting and ensuring widespread data use as well as maximizing the validity and reliability of reported results. ESI demonstrated several accomplishments in this task area, primary through the technical assistance relationship that was developed with the North West Province Department of Health, where ESI placed Data Quality Improvement Mentors and HIV M&E Coordinators in each of the 4 districts. The assistance provided by these seconded staff produced a national routine data quality audit (RDQA) tool and methodology for both training purposes and routine implementation among DOH facilities and PEPFAR Partner agencies. ESI provided extensive data quality trainings for PEPFAR healthcare practitioners as well as key members of the DOH and DSD within their standard capacity building course offerings. ESI seconded staff helped to align the RDQA tool to the South African Statistical Quality Assurance Framework and it has reportedly been adopted as the new standard. However, interviews with key informants revealed that the tool is only able to assess timeliness and completeness of data, and falls short of measuring more sensitive elements of data quality. Following the development of this new tool, ESI worked collaboratively with the NW Province through the MOU for strategic information capacity building, to conduct RDQA baseline assessments in practically every facility across the province. The Evaluation Team was unable to verify whether improvements have been measured in data quality within the NW Province since the baseline assessment was conducted. Interviews with key informants in the Provincial offices of DOH and DSD in NW Province found widespread recognition for the RDQA tool as well as acknowledgement of its importance, but little evidence that it is actually being used to track data quality. Interviews with key informants in Dr. Kenneth Kaunda District (based in Klerksdorp) in North West Province revealed that ESI seconded staff were familiar with the RDQA tool, but had not yet implemented it as a regular practice within the facilities of their district. The reasons why there had been delay in applying the RDQA tool were unclear. While these findings indicate Project accomplishments at the outcome level in cultivating and expanding the culture of data quality within the DOH, they do not indicate cumulative Project accomplishments at the objective level through application of the tool and improvement in data quality. Conclusion. The ESI Project demonstrated considerable achievements in their efforts to build capacity for the adoption and standardization of data quality audits within the DOH, particularly in the NW Province. Significant time and effort was expended in designing and introducing the Routine Data Quality Assessment tool and key members of the Provincial Government were familiar with both the tool and the concept of data quality however, the Evaluation Team did not find evidence that this tool was being used by implementers, nor that the actual quality of data had been improved as a result of ESI’s work. F. RESPONSIVENESS TO CAPACITY BUILDING NEEDS Findings. The Project was presented with many opportunities to rapidly build working ESI End-of-Project Evaluation 28 agreements with national departments and provincial/district sites many of which appeared to have clearly-defined and articulated human resource development needs. A good example of SAG-driven initiatives was described in discussions with officers in North West Province. They reported to the Evaluation Team that they had a clear understanding of what was needed to strengthen M&E capacity in the area of HIV and AIDS in their province, and initially sought assistance from CDC who then referred them to the ESI Project. As in the case of ESI’s use of a MOU for work with DSD (see previous section of this report), ESI’s partnership with the North West Province demonstrates a systematic approach to developing a comprehensive MOU detailing the activities required to effectively build the capacity of provincial and district health officials in strategic information, data quality, and monitoring and evaluation. The MOU was implemented through “secondment” of local staff trained by the ESI Project and deployed to each of the 4 districts within the province. In addition, ESI worked closely with the DOH and DSD in NW Province and within the DOH and DSD at the national level to ensure continuity and a smooth hand-over between ESI’s support of seconded staff to SAG. It appeared to the Evaluation Team that seconded personnel were well trained by ESI, were competent change agents within the system where they were seconded, were well received and integrated with other workers, and able to demonstrate the benefits of their positions to policy decision makers. The secondment model appeared to have worked very well in the one district visited by the Evaluation Team, and according to ESI, the Project’s experience in other districts in North West Province was positive, with the possibility of seconded positions being absorbed by most – if not all – participating districts in the province. The DOH District Health Officer interviewed by the evaluators was appreciative of the assistance provided by ESI and the seconded staff, and felt that the approach had led to sustainable improvements in health management and health services in the district. (Also see Section V-C. 3. of this report). In contrast, the success of similar approaches in other settings may have been more problematic, and the Evaluation Team was cautioned that the ESI Project model for secondment may have had features that made it more successful than other approaches. Specifically, secondments of technical or clinical staff have also been used widely by PEPFAR implementing partners in other provinces/districts to augment SAG/DOH staffing shortages. Several implementing partners voiced concern to the Evaluation Team regarding the inability of the DOH to permanently absorb many of the positions. In these cases, the approach cannot be considered as leading to sustainable capacity building since there will potentially be significant human resource gaps once positions supported by PEPFAR funds are phased out. Conclusions.  ESI was flexible and responsive to DOH requests for capacity building and technical support, which helped to facilitate a strong overarching relationship between SAG and USG regarding HIV and AIDS strategic information. This relationship was forged at a critical juncture both in PEPFAR’s programming in South Africa and in SAG’s receptiveness to increased collaboration.  ESI’s approach of secondment of trained M&E specialists within the SAG structure was reported by SAG officials to be very effective, but more needs to be understood about the factors that led to success of this model in North West Province. ESI End-of-Project Evaluation 29 Recommendation. Consideration should be given to replicating the model for secondment of M&E specialists in SAG district management positions, giving close attention to factors that led to successful collaborations such as ESI’s work in Klerksdorp in North West Province. ESI End-of-Project Evaluation 30 VII. PROJECT ACHIEVEMENTS – THE PARTNERSHIP INFORMATION MANAGEMENT SYSTEM AND GEOGRAPHIC INFORMATION SYSTEM A. INTRODUCTION The ESI contract called for the creation of a USG results reporting database and strategic information system that would build upon the considerable depth and range of data currently stored in the Data Warehouse and its unlimited potential to store additional data from other secondary sources. ESI was to develop a database platform that would encourage individual users to depend on these data for decision-making by enhancing a user-driven analysis package. The remodeled database would overcome the relative inability of the (then existing) Data Warehouse to present users with an attractive, efficient, and powerful analysis functionality that would allow the generation of customized reports through a menu-driven environment. This contractual expectation of the ESI Project’s Tasks 5 and 6 provided a very clear framework that was both comprehensive and forward thinking. It looked for innovation, involvement of stakeholders, including the SAG, and articulated the need for a user-friendly system that would encourage data use for planning and management, including spatial functions such as GIS, as well as for reporting in USG agencies, partner organizations, and SAG departments. The Evaluation Team wishes to emphasize that its evaluation task was to determine the extent to which ESI’s information system developments extended beyond the PEPFAR reporting function to additional capacity of the system to provide users with M&E and analytical tools. Findings, conclusions and recommendations are made within this context. B. ACHIEVEMENTS AND SHORTFALLS IN SYSTEMS CAPACITY Findings. An inventory of PEPFAR prime and sub-prime implementing partners was integrated/linked with the DW and then combined with a GIS functionality to create the PIMS, which became partially operational for Q1 PEPFAR reporting (March 31, 2012). PIMS is expected to go fully operational in July 2012. The overhauled DW, renamed PIMS, was generally reported by users with whom the evaluators spoke to be more intuitive and user-friendly than the first generation Data Warehouse version of the system. As the systems administrator, ESI also was found to have a generally good reputation for being “customer-friendly” and responsive to inquiries about technical problems in using the system (such as access problems, etc.). The above notwithstanding, PIMS still does not have the full M&E and analytical functionality as initially envisioned in the ESI Contract. The Evaluation Team undertook an extensive review of the current system using observations, long working sessions in which ESI technical developers made presentations and demonstrations and answered questions, and interviews with key informants. Based on these reviews, the technical developers at the ESI Project and the Evaluation Team evaluators were in basic agreement that, in terms of the M&E and analytical capacity of the system, PIMS will not have achieved the contractually-defined level of ESI End-of-Project Evaluation 31 functionality (as stated above) by the end of the ESI Project in March 2012, and is also unlikely to achieve this standard by the end of the additional 7-month contract (September 2012). PIMS is also perceived by some users as being limited in providing data for decision-making and user-driven analysis packages. According to a Focus Group Discussion (FGD) conducted with a selected, small group of prime implementers, there were also few expectations that PIMS will be able to achieve full analytical functionality to meet the needs of their senior M&E counterparts within USAID and CDC. By comparison to what they believe is possible with PIMS, some partners are reported to have put in place information systems they believe can meet their own internal managerial and M&E needs as well as to support their PIMS reporting tasks (suggesting that it is possible, in their view, to combine the reporting and analytical functions within a strategic information system). Additionally, as reported and observed by the Evaluation Team, PIMS is perceived, particularly by USG Activity Managers, as lacking powerful, efficient, user-friendly, analytical capabilities, and these perceptions discourage user engagement (again, particularly Activity Managers) even for whatever new, but limited functionality PIMS now possesses. It was reported to the Evaluation Team by USAID and CDC officers that some Activity Managers have been slow and resistant to registering as PIMS users and completing the user orientation training sessions. As of September 2012, PIMS will still lack the following: a sub-prime inventory, DOH sites, district-specific data for analytical functions, demographics, and inbuilt spatial analytical functions. The Evaluation Team wishes to emphasize none of these features are important for basic PEPFAR reporting, and the ESI Project has not been requested by the USG PEPFAR strategic information team or USAID to include them in the systems functionality. However, the Evaluation Team believes, based on the characteristics of the enhanced system as described in the ESI Contract, that these are the type of features that would be critical for the generation of strategic information, user-driven M&E, and analytical tasks. Please refer to APPENDIX M – Schematics Showing Current and Recommended Future PIMS Architecture and Model Dashboard, for further detailed description about how the current (and recommended future) PIMS is/might be organized, and how data input, reporting and access are/might be managed. Conclusions.  PIMS as developed by ESI, functions well as a basic reporting system for PEPFAR, but by the project end date did not have the user-driven M&E and analytical capability called for in the ESI contract, and is not likely to have this capability by the end of the new 7-month ESI contract (September 2012).  The negative attitude of some systems users, particularly Activity Managers, is undesirable and a constraint to development of a user-driven and user-responsive system as might be defined by users themselves. The user-initiated, intuitive instructions now built into PIMS are not adequate to improve these attitudes or engagement of users. ESI End-of-Project Evaluation 32 Recommendations. (1) The USG PEPFAR Secretariat, USAID, and ESI should organize one or more orientation meetings with key users of PIMS (including Activity Managers), that provide an overview of systems features and uses of PIMS, and opens dialogue on ways in which PIMS can be further responsive to users’ M&E and program analysis needs. (2) The USG PEPFAR Secretariat and USAID should ensure that appropriate levels of participation in and support for PIMS is not optional. C. SHORTFALLS IN DATA AND INFORMATION CONTENT Findings. In addition to limitations of PIMS functionality as discussed above, in order for the information to serve the analytical and M&E purposes originally envisioned in the ESI Contract, PIMS must have the capacity to collect/receive, store and display on demand specific data and information required by users. This analytical capacity would need to be in place in addition to the system’s basic capacity to collect, store, retrieve, and report data and information. As the analytical and strategic information needs of users change and evolve, so also must the data and data analysis capability of the information system evolve. Some of the more obvious gaps in the current PIMS information database are discussed below. These gaps were not viewed by the evaluators as being a fault of failure of PIMS as it has been developed to date, but rather, these gaps were viewed as areas that will need attention (with guidance of the USG PEPFAR strategic information team), in future PIMS developments. The Evaluation Team notes that in being responsive to PEPFAR guidance on indicators14, the USG PEPFAR strategic information team should ensure that PIMS is set up to capture information that is relevant to OGAC, PEPFAR Activity Managers, and other users. The Evaluation Team observes that in the absence of a way to continuously update the information being input and generated by PIMS, the relevance of the systems is diminished. As reported in discussion with many PEPFAR-II implementers, these implementing partners are expending most of their PEPFAR budgets on PEPFAR-authorized activities such as technical assistance, but these activities are not required to be reported and are not captured in PEPFAR reporting (except as quarterly narratives). Several PEPFAR-II implementers with whom the Evaluation Team discussed this matter are very concerned that their important work, authorized and funded by PEPFAR, is not adequately coming to the attention of PEPFAR decision makers (such as Activity Managers) and higher level authorities that set program and funding policy. Furthermore, there is a substantial amount of HIV and AIDS data generated in SAG Departments (especially the social cluster which includes DSD, DOE, DOJ, CS, etc.) that is relevant to the strategic interest of USG PEPFAR but that PIMS currently does not have a means of capturing. An overarching development on the SAG side is the current, on-going development of the National Health Information Repository and Data Warehouse (also known as the National 14 PEPFAR Next Generation Indicators Reference Guide, http://www.pepfar.gov/documents/organization/81097.pdf. ESI End-of-Project Evaluation 33 Health Information Centre - NHIC) which in the future may emerge as a critical link to SAG for HIV and AIDS statistics. The latest version of the national health information system (DHIS 2), which is currently in use in a large number of countries outside South Africa is free, open source software, is web based and reportedly has greater analytical capability than PIMS. Since this has users and ongoing support from its developers and the SAG is committed to it, it may be advantageous to upgrade PIMS to similar SAG (and international) standards. Conclusion. In the future PIMS may need to specifically address programmatic implications of PEPFAR-II. Recommendations. (1) In the future, consideration should be given to modifying PEPFAR reporting formats, indicators, etc., to allow PEPFAR-II implementing partners to better reflect their capacity building and technical support roles. (2) In the future, consideration should be given to making the PEPFAR results reporting and strategic information system more dependent upon and closely aligned with SAG information systems as a primary source of HIV and AIDS-related data and information, particularly the Department of Health Information System (DHIS) and the National Health Information Centre (NHIC). D. INHERENT RISKS IN PIMS DEVELOPMENT Findings. PIMS evolved from earlier constructions of the Data Warehouse to its current design. This was of immediate concern to the Evaluation Team because user needs can change rapidly over time, and a system that was designed more than 10 years ago may not be consistent with current needs. Also, once systems specifications are built into an older version (such as the Data Warehouse), they become rigid features of how the system works, and once built upon, can’t be undone. Thus, whatever may have been wrong with the first generation Data Warehouse is likely to have been carried over into PIMS. Building a customized software system may take years, but the user – in this case USG PEPFAR strategic information system planners -- can only really know that the system is working the way it was intended once construction is completed (now projected for September 2012). As described to the Evaluation Team by the ESI system developers, the Data Warehouse software system was customized and built from scratch with a team of programming experts from Khulisa that constructed the system using their own individual computer language and coding. This was a highly individual process that could not be readily understood, duplicated, or modified by anyone (other than Khulisa programming experts), who were not part of the original construction. Once Khulisa built a custom software system from scratch, whoever picked up on further development or enhancements to the system, in practical terms, had no control over the intellectual property. This caused ESI to be generally bound to continue using the services of Khulisa regardless of any technical or contractual conflicts they may have had. Conclusions.  There are risks that whatever technical problems may have existed in the original Data ESI End-of-Project Evaluation 34 Warehouse construction, these are likely to have been carried over into PIMS.  The approach of using customized software development should be avoided if possible. Recommendation. Future PIMS developments should take an approach that would eliminate any technical problems that might have carried over from the Data Warehouse and avoid the risk of customized software development. E. SUGGESTION FOR ALTERNATIVE PIMS CONSTRUCTION Findings. As part of this evaluation, the Team conducted an investigation of the state-of-the-art of strategic information systems developments in the southern Africa region and in South Africa. It was found that a large number of modular designs are on the market, many in the public domain. Third party applications that come with guarantees of systems compatibility, adaptability, and data sharing protocols are available. These cutting-edge technologies are often available at low/no cost to non-profit organizations. The Evaluation Team held discussions with two different PEPFAR implementing partners that have moved to modular information systems construction and also learned from the DOH that they are in the process of converting the national DHIS to a similar modular modality. An additional advantage of third party applications is data security. Currently PIMS only has very rudimentary security capacity (log-in passwords). Modular designs and other applications frequently include state-of-the-art features such as biometric identification and bar-coding technology for individual level data quality and security. The Evaluation Team also noted PIMS is already using a modular application for GIS, although it is not fully-integrated into PIMS. Although GIS functionality has been built into PIMS, for anything other than the simplest spatial mapping ESI uses a separate linked external platform (MapShare), to construct geo-physical analysis using PIMS data, so that it can be shared with PIMS users. PIMS will necessarily need to continue to rely on 2 separate platforms to produce limited maps and reports. This raises the question as to why this duplication has been created, as this process is unnecessarily cumbersome and alternative approaches exist. A proposed configuration of a fully-integrated modular strategic information system with links to other national databases is outlined in APPENXDIX M – Schematics Showing Current and Recommended Future PIMS Architecture and Model Dashboard. Conclusions.  Modular information systems constructions are available and in use in South Africa.  Developing a sophisticated GIS mapping function within the PIMS architecture is desirable and achievable. Recommendations. (1) For the future, USAID should consider transitioning PIMS to a modular information system construction. The envisioned system would need to set out database links, network requirements and restrictions, data user agreements, and use of the DHIS-2. The ESI End-of-Project Evaluation 35 envisioned system would use facility-level technology and web-based servers, and would have a means of controlling data security, access rights and confidentiality requirements on shared systems. (2) The GIS functionality within PIMS should be further developed as part of other “next generation” developments of the system. F. INFORMATION TECHNOLOGY CHANGE MANAGEMENT PROCESS 1. Change control meetings Findings. Change control meetings (CCMs) were the major forum for coordination of issues related to the ever-changing expectations and specifications for the enhanced PEPFAR results reporting and strategic information system. Participants included USAID and CDC’s strategic information team, ESI staff, and often staff from the ESI subcontractor responsible for information systems development. Individuals interviewed for this evaluation expressed a wide range of expectations for, and satisfaction with, these meetings. It was noted that the participation of CDC was of particular importance to the CCMs. Although CDC had no formal management oversight of the ESI Project, CDC was a significant technical contributor in providing guidance to ESI’s work in enhancing the PEPFAR results reporting and strategic information system. While relationships between USAID, CDC and ESI were generally collaborative, both USAID staff and CDC staff had varying expectations and placed numerous demands on the developing system, and demands were not always coordinated or prioritized. One prior ESI COTR commented to the Evaluation Team that the meetings were excellent and even constituted a best practice. However, other meeting attendees commented that since there was not a clear sense of purpose for the meetings, and because the technical capabilities of the attendees varied considerably in terms of knowledge of software development and Information Technology (IT) systems engineering, the meetings were “disappointing.” Problems with the effectiveness and productivity of the CCMs appeared to the Evaluation Team to be structural in nature. One change control management function calls for a forum for systems requirements and technical specifications to be determined at the strategic information policy level, while another change control management function focuses more narrowly on approval/disapproval of incremental technical decisions15 that may affect adherence to requirements and specifications. These two functions provide checks and balances to the change control process and should not be undertaken in the same venue. The ESI change control meetings as described to the Evaluation Team were evidently an inappropriate co-mingling of these two functions. However, problems with the CCMs appear to go deeper than just the need to redefine the purpose and composition of these meetings. As further discussed below, (See Section VII- F. 2.), the overall information technology change management oversight of the ESI Project information system development lacked an effective structure and process. 15 Typical examples of change management issues in the computer and network environments are patches to software products, installation of new operating systems, upgrades to network routing tables, or changes to the electrical power systems supporting such infrastructure. ESI End-of-Project Evaluation 36 Conclusion. Not only were change control meetings poorly organized, but also the entire USAID information technology change management process for the ESI Project was not properly planned for, structured, or managed. Recommendation. For future information system development projects, terms of reference for change control meetings should be established in keeping with a USAID information technology change management policy and structure. 2. Formation of an Information Technology Change Management Structure Findings. The Evaluation Team observed that the ESI Project was tasked with enhancing a complex PEPFAR results reporting and strategic information in the absence of a formal USG PEPFAR information technology change management policy and structure. The evaluators were therefore prompted to explain what is meant by “change management” and the implication of this concept for future PEPFAR information systems developments. Change control within the context of Information Technology (IT) systems development is a formal process used to ensure that changes to a product or system are introduced in a controlled and coordinated manner. It reduces the possibility that unnecessary changes will be introduced to a system without forethought, introducing faults into the system or undoing changes made by other users of software. For Information Technology (IT) systems, change management is a distinct discipline and area of expertise. Typical examples of change management in the computer and network environments are patches to software products, installation of new operating systems, upgrades to network routing tables, or changes to the electrical power systems supporting such infrastructure. In software development, a high level board, task force, or steering committee should be in place to make decisions regarding whether or not proposed changes to a software project should be implemented. The change control task force is usually constituted of project stakeholders or their representatives. The authority of the change control task force should be defined by a written change control policy. It is important for span of control purposes to make a clear distinction in the roles and authorities of the change control task force decision makers compared to the technical developers who will actually implement the changes once authorized. Establishing an effective change control environment for information technology projects requires that specialized expertise be brought into the client organization to management the process. Conclusion. Effective management of IT projects such as ESI requires that the contracting organization (in this case, USAID) have formal internal management structures and capacity to oversee the activity. Recommendations. In connection with future phases of PEPFAR strategic information systems developments: (1) USAID should develop an IT change management policy and structure for future PIMS developments. The policy and structure should be included in the recently drafted MOU for ESI End-of-Project Evaluation 37 improved USAID/CDC coordination of USG PEPFAR strategic information systems as recommended by the 2011 Inspector General’s audit16. (2) Composition and functions of the Strategic Information Unit need to be reviewed, and a PEPFAR strategic information task force needs to be reconstituted with clear board-like authorities and limits of involvement in implementation details within the framework of a defined IT change management policy and structure. (3) An in-house information technology and systems expert (director/administrator) should be appointed to manage all in-house strategic planning and technical developments including overseeing and coordinating the work of external contractors and such activities as change control meetings. VIII. SUMMARY OF CONCLUSIONS AND RECOMMENDATIONS17 Overarching ESI Evaluation Conclusions. 1. While the Evaluation Team found the logic behind ESI’s development hypothesis to be sound, the Project provided little evidence to demonstrate the provision of improved health services or to substantiate improved health outcomes resulting from the Project activities. The primary Evaluation findings indicate that the greatest degree of change due to the Project occurred at lower-levels of the hypothesis at the output and outcome levels as evidenced within the training and capacity building task areas. Lesser degrees of change were evident at the objective and goal levels of the Project regarding data use, data quality, health services and health outcomes. 2. The ESI Project suffered from inadequate, inconsistent, and poor management and oversight at all levels of administration, starting at the top with USAID, trickling down through JSI headquarters in Boston, and most prominently with the ESI Project Team in South Africa under the leadership of the Project’s first COP through early 2011. 3. ESI Project performance demonstrated measurable success across some of the Task areas, particularly within the area of Orphans and Vulnerable Children (OVC) and more limitedly within Training and Capacity Building. While the evaluation findings underscore the sustainability of the OVC work carried out by ESI, the Team did not find significant evidence to indicate the sustainability of the Training and Capacity Building initiatives. 16 Please see Audit recommendations on the Data Warehouse, Section V-G, above. 17 See APPENDIX N for a matrix that re-states all conclusions and recommendations. ESI End-of-Project Evaluation 38 APPENDICES ESI End-of-Project Evaluation 39 APPENDIX A. KEY ESI CONTACTS AND PROJECT ORGANOGRAM Organizations Project Interactions Clinton Health Access Worked together to provide technical support to the DOH for the 3-Tiered ART Strategy development and implementation Dept. of Social Development Provided direct Technical Assistance on OVC M&E, data management and MIS evaluation and presented research paper DOH Eastern Cape Provided Training, Mentorship and support for Evidence-Based Health Management, DHIS, M&E, Data Capturing DOH Free State Provided Training, Mentorship and support for Evidence-Based Health Management, DHIS, M&E, Data Capturing DOH North West Pilot province; requested Data Quality support through CDC and an MOU was established to support the province DOH Western Cape Technical Assistance for Routine DQ Assessment tool institutionalization, provided DHIS training and support WFPD Co-trained on the Evidence-based Health Management and DHIS courses HIV 911 Mapping Provide them all the maps for the HIV 911 annual provincial directory and collaborated on refining geo-coordination HLSP Worked together on the NDOH 18 Priority Districts PMTCT project HST Provided ad hoc TA, co-wrote the HIS section of the South African Health Review 2011 and sit together on DOH HI MRC Contracted to assess electronic patient management systems in the country and recommend standard system NDOH Strategic Health Provided them with an National M&E Advisor who worked with all program managers; position is now filled using NDOH - Health information Commissioned ESI to conduct Rapid Information Needs Assessment as forerunner to HIS strategic development ESI End-of-Project Evaluation 40 NDOH - M&E Jointly developed a customized DQA tool based on national standards for Data Quality NDOH - PMTCT Commissioned ESI to implement PMTCT 18 Priority Districts Project in conjunction with HLSP Population Council Mapping of their training footprint for Post Exposure Prophylaxis (PEP) in Limpopo, Free State and North West Province Right to Care Co-trained on the Evidence-based Health Management and DHIS courses. Requested joining in ESI Capacity Building SAMEA Assisted with logistics and have a team member responsible for Associations communications South to South South Trained on the Evidence-based Health Management and DHIS courses. Requested joining in ESI Capacity Building Stats SA Worked together with SAATCA, DOH, Khulisa and others to establish first Data Quality Auditor accreditation University of Pretoria Trained on GIS, Data Quality and OVC modules 3 times a year on M&E courses WRHI Trained on the Evidence-based Health Management and DHIS courses Requested joining in ESI Capacity Building Training HISP Sub-contractor on the project to provide support for Data Quality and Data Use and improvement activities Tulane University Sub-contractor on project providing research and evaluation support to Dept. of Social Development Khulisa Management Services Sub-contractor supporting and maintaining Data warehouse for USG results reporting; developed the Inventory system ESI End-of-Project Evaluation 41 ESI End-of-Project Evaluation 42 APPENDIX B. ESI KEY EVALUATION QUESTIONS Strategic Design Issue Overall program - What were the objectively verifiable development problem and assumptions of ESI? Was there consistency and agreement among planners regarding the problems being addressed and the program design? - To what extent did the ESI project meet all project objectives as defined in its original Statement of Work? - Was the issue of capacity transfer and sustainability considered in design of the project? - How should future strategic information projects be structured to ensure that the best features of the predecessor systems are retained while appropriately expanding to meet evolving needs? - For the future, how can information systems developments involving bilateral and interagency stakeholders be better coordinated and managed? Training and Capacity Building - Did ESI capacity building interventions (training and technical assistance) provided organizations/individuals with practical and implementable capacities (knowledge and skills) that were relevant to their day-to-day operational challenges? - How might future capacity building interventions be designed for greater effectiveness and impact? Strategic Information Systems - Was ESI program as designed in sync with the Government’s priorities and approaches to information systems development? How could this alignment be strengthened in the future? - What USG planning and coordination mechanisms were in place to facilitate design and change control monitoring of information systems developments? Did monitoring include clearly defined targets against which to measure progress? - How effective were monitoring and change control mechanisms and what changes could be made in the future to improve their functionality and effectiveness? - Does the strategic information system developed under the ESI adequately meet end user requirements? How could this be improved in the future? - How effective has ESI been in improving systems-wide data quality and use of data for decision making? How could this be improved in the future? Project Management - What project management structures and processes were put in place to ensure that customized/on-going training and technical assistance needs assessments were conducted? - Were adequate needs assessments actually carried out and were related ESI capacity building interventions sufficiently customized and flexible to meet the particular and specific training and technical assistance needs of various beneficiaries? - What was ESI’s project management relationship with JSI regional and headquarters management? How critical was this backstopping to ESI/SA field operations? Was backstopping adequate; effective? - To what extent were the program and technical personnel able to transfer sustainable capacities (knowledge and skills) to beneficiaries? - Did ESI maintain a personnel management system that included regular personnel performance reviews? - Did JSI/ESI encounter any noteworthy problems in funds disbursement and management? ESI End-of-Project Evaluation 43 Results Management - What were the verifiable and documentable achievements (quantitative and qualitative measures) of ESI performance in terms outputs and outcomes in each of its project tasks areas? - Have ESI deliverables been produced and has the project met its contractual obligations? - What management procedures were in place to monitor the program? What targets and indicators were used, who measured progress against indicators, on what schedule, and how were results verified? - What have been the major sustainable achievements of the ESI? ESI End-of-Project Evaluation 44 APPENDIX C. LIST OF PERSONS CONTACTED SAG NATIONAL DEPARTMENT OF HEALTH Thulani Masilela, Chief Director, Strategic Planning, Department of Health DEPARTMENT OF SOCIAL DEVELOPMENT, PRETORIA Chris Van Rooyen, Deputy Director: Programme Implementation NORTH WEST PROVINCE GOVERNMENT OFFICES Provincial HIV/AIDS Department Cornelius P. Lebeloe, Director, HIV & AIDS Management Keolebogile Joan Lesetedi, M&E Officer Tuekiso Isaacs, PEPFAR Provincial Liaison Dr. Uma Nagpal, Chief Director, Dr. Kenneth Kaunda District Health Services Keitumeste Mlambo, HAST Programme Manager Keith Themba, Data Quality Manager KWAZULU NATAL PROVINCE GOVERNMENT OFFICES Jaime Harry, Provincial Department of Health Nthabi Mapera, Provincial Department of Health Jubulani Tembe, Provincial Department of Health Darren Kuppusame, Ilembe District Department of Health Uthraj Lekhram, Ethekweni District Department of Health PEPFAR SECRETARIAT James Maloney, Deputy Coordinator PEPFAR PROVINCIAL LIAISONS (PPLs) Thekiso Isaacs, North West Province Masilo Marumo, Gauteng Province Chalone Savant, KwaZulu Natal Province Jessica Rebert, Western Cape Province USAID Jeff Borns, Mission Director Catherine Moore, Deputy Mission Director John Kuehnle, DLI Health Officer Charles Mandivenyi, Senior Monitoring and Evaluation Specialist Wendy Githens Benzerga, HIV Prevention Advisor Malik Jaffer, Senior Technical Advisor Nelly Sibanyoni, Project Management Assistant, Cross-cutting Team Anita Sampson, Prevention and OVC Team Leader Karen Kasan, Regional Health Officer Karin Taljaard, Program Assitant Cecilia Khupe, Regional Agriculture Program Manager Thapedo Maofoe, Health Office Shelagh O’Rourke, Health Office Olga Masuie, Health Office Mashudu Mashamba, Health Office CENTERS FOR DISEASE CONTROL AND PREVENTION Heidi O’Bra, CDC PEPFAR Coordinator Neil Jacobs, HSS/HMIS Lead ESI End-of-Project Evaluation 45 Rachel Lovik, HMIS Specialist JOHN SNOW INCORPORATED – ENHANCED STRATEGIC INFORMATION PROJECT Mukund Vassan, Project Administrator Christa van den Bergh, Task 1 Lead: Capacity Building Natashe Taljaard, Project Assistant Derek Kunaka, Chief of Party Mandla Motshweni, Task 2 Lead: OVC and M&E Anzel Schonfeldt, Task 4 Lead: Data Quality Ken Olivola, Director, International Division and Senior Advisor for ESI KHULISA MANAGEMENT SERVICES Jennifer Bisgard, Director New Initiatives & Business Development Michael Ogawa, Director Rebecca Rishty, Senior Manager Corporate Affairs Mary Pat Selvaggio, Director of Research and Health Peter Capozza, Managing Director Salome Omolo, Associate Director, Public Health Wernher Friedrich, ESI Task 5 Lead: Decision Support Systems Brennan Walsh, ESI GIS Specialist HEALTH INFORMATION SYSTEMS PROGRAMME (HISP) Vincent Shaw, Senior Consultant TULANE UNIVERSITY Tanya Thurman, Principal Investigator PEPFAR IMPLEMENTING PARTNERS Barbara Franken, M&E Manager, Right to Care, Johannesburg Mmaja Motjale, Project Manager, LifeLine, Mafikeng, North West Province O’veray Oduaran, H.R. Manager, LifeLine, Mafikeng, North West Province Ndivhuho Mangale, Data Quality Mentor, WHRI, Mafikeng, North West Province Pearl Govender, data Capturer, St. Mary’s Hospital, Marianhill, KwaZulu Natal Tainos Zinwoni, Medical Manager, St. Mary’s Hospital, Marianhill, KwaZulu Natal Eoghan Hilson, IT/Data manager, St. Mary’s Hospital, Marianhill, KwaZulu Natal Leslie Alfreds, Data Supervisor, St. Mary’s Hospital, Marianhill, KwaZulu Natal Tony Lott, Finance Manager, St. Mary’s Hospital, Marianhill, KwaZulu Natal Catherine Searle, Director: Operations and Development, MATCH, Durban Dr. Arthi Ramkissoon, Divisional Head, MATCH, Durban Ian Hove, Catholic Medical Mission Board (CMMB), Pretoria Nkanyiso Ndovu, Society for Family Health, Pretoria Virginia Francis, Research Triangle Institute (RTI), Pretoria Michelle Layte, Research Triangle Institute (RTI, Pretoria Boniface Hlabano, AMREF, Pretoria John Kigozi, JHPIEGO, Pretoria Suzanne Johnson, Foundation for Professional Development (FPD), Pretoria Erika van Vollenhoven, Education Labor Relations Council (ELRC), Pretoria Marisa Wille, M&E Project Manager Catholic Relief Services, Pretoria Ivy Selepe, Business Manager: Grants, National Health Laboratory Services, Pretoria Sharon Marambu, Program Administrator, National Health Laboratory Services, Pretoria Patience Mhuorreno, Grant Accountant, National Health Laboratory Services, Pretoria Kibogile Mokwena, Principal investigator, MEDUNSA, Pretoria Rachel Meyer, ME& Officer, World Vision, Pretoria Zak Kaufman, Founder, Solution Architect, Verasolutions, (and Advisor to Grassroot Soccer) ESI End-of-Project Evaluation 46 Zinhle Nkosi, Grassroot Soccer Shungu Gwarinda, Mothers2Mothers Johnathan Cocburn, SACTWU Worker Health Program Theunis Hurter, ANOVA Health Institute Mervyan Konjore, Hospice Palliative Care Association Gil Lang, Positive Community Impact Clara Eder, Positive Community Impact ESI End-of-Project Evaluation 47 APPENDIX D. EVALUATION METHODOLOGY AND WORKPLAN Social Impact’s Evaluation Methodology and Work-plan “Evaluation of JSI Enhanced Strategic Information Capacity Project” RFTOP No. RFTOP-674-12-000005 Submitted to USAID/South Africa Social Impact, Inc. 2300 Clarendon Blvd., Suite 300 Arlington, VA 22201 Social Impact Contact: Rolf Sartorius, 703.465.1884, rsartorius@socialimpact.com Project Director: Dr. Richard Blue, 703.465.1884, rblue@socialimpact.com, richardblue@earthlink.net ESI End-of-Project Evaluation 48 Table of Contents INTRODUCTION .............................................................................................. 49 EVALUATION METHODOLOGY ................................................................. 49 Scope of the ESI project ................................................................................ 49 Evaluation scope and key questions. .......................................................... 50 Key informants. ............................................................................................... 50 Data gathering instruments and tools. ....................................................... 51 The ESI Evaluation Framework. ................................................................... 51 Data synthesis and analysis. .......................................................................... 51 Oral presentation. .......................................................................................... 52 Reporting and dissemination of results. .................................................... 52 WORK SCHEDULE ............................................................................................ 52 ANNEX A – Key Evaluation Questions ANNEX B – A Matrix of the Evaluation Framework ANNEX C – Work Schedule Calendar ESI End-of-Project Evaluation 49 INTRODUCTION Social Impact, Inc. (SI) has been requested by USAID/South Africa (USAID/SA) to evaluate the John Snow, Inc. (JSI) project, Enhancing Strategic Information (ESI), which began in 2008 and is now scheduled to end in September 2012. The SI technical team, herein referred to as the “SI Team,” is comprised of 4 technical experts who are responsible for design and conduct of the evaluation in South Africa. The evaluation will be undertaken in 4 phases as follows:  Phase 1 – Planning, Document Review, Finalization of Evaluation Strategy  Phase 2 – Conducting the Evaluation and Data Analysis  Phase 3 – Oral Presentation (including preparation)  Phase 4 – Reporting and Dissemination of Results Phase 1 of the evaluation officially began on March 5, 2012. Documentation relevant to the evaluation was accumulated and reviewed. Upon arrival in South Africa, the SI Team met with USAID/SA and other USG PEPFAR partners for guidance and clarification on the purpose and scope of the evaluation. Issues of particular importance to USG stakeholders were covered, and a list of key evaluation questions was finalized. An USAID/SA in-briefing took place on March 15, 2012 which covered review of the SI Team’s draft Evaluation Methodology and Work-plan and Mission guidance to the SI Team. By the end of Phase 1 on March 17, 2012, preliminary meetings with key USG stakeholders, design of data collection tools and protocols, and the itinerary for field visits have been completed. This document, Evaluation Methodology and Work-plan, is a SI project deliverable representing the completion of Phase 1. EVALUATION METHODOLOGY The Evaluation Methodology serves to define what issues and questions will be investigated and answered; who will provide the needed information (key informants); and how (that is, by what means) data and information will be gathered and analyzed. Scope of the ESI project. Construction of the evaluation framework began with developing an accurate and complete understanding of the scope of the project to be evaluated. Within the broader context of PEPFAR guidelines for strategic information systems development, the project selectively addresses country-specific areas of assistance (described as “tasks areas”), that focus on stakeholder interests such as reporting requirements, information systems software developments, systems consolidations and alignments, and improving the quality and utilization of strategic information. The ESI Strategic Framework (as interpreted within a Logical Framework or Results Framework18 format), the ESI project might be described as follows: Purpose level: To improve the HIV/AIDS status of the South African population Goal level: To strengthen the effectiveness of HIV/AIDS service delivery through the enhanced use of information for evidenced-based decision making Objective/ Result level: To strengthen integrated, sustainable information systems in order to make relevant and credible information widely available to HIV/AIDS program policymakers, planners, managers, and implementers Sub-objectives/ Sub-results Level: Task 1: Build capacity for strategic information in communities 18 Logical Frameworks and Results Frameworks formats are commonly used USAID programming tools. ESI End-of-Project Evaluation 50 Task 2: Achieve comprehensive information system development for OVC and other program areas such as PMTCT and HAST Task 3: Design and implement action plans for enhanced use of data among PEPFAR implementing partners Task 4: Provide TA to maximize data quality in results reporting Task 5: Develop and maintain an USG results reporting data warehouse Task 6: Create high quality, multivariate GIS mapping applications Evaluation scope and key questions. Given the above described scope of the ESI project, the objective of the ESI Evaluation is to address 4 cross-cutting issues (as described in the SI Technical Proposal). These are:  Strategic design issues concerned with the appropriateness of the development hypothesis in the project overall and in specific task areas;  Adequacy of project management in the project overall and in specific task areas;  The extent to which outputs translate into meaningful and intended outcomes in the project overall and in the specific task areas;  Lessons learned and strategic programmatic and managerial considerations that should be taken into account when developing any ESI-type follow-on. Each of these cross-cutting issues have been broken down into Key Evaluations Questions that will need to be addressed to different levels of key informants through one or more data gathering methods. Key Evaluation Questions pertaining to each of the 4 cross-cutting evaluation issues and 6 ESI project task areas is shown in APPENDIX B. Key informants. Each of the key ESI programmatic task areas listed above are referred to extensively in ESI documents and reports and in discussions with ESI partners. Tasks can be further broken down into sub-tasks or activities which are distributed amongst ESI implementing partners. ESI’s main implementing partners to date have been: Khulisa Management Services, Tulane University School of Public Health, the Health Information Systems Programme (HISP), and Manto Management. At the next level are the prime PEPFAR implementing partner organizations that are information hubs for data processing and upward reporting in the PEPFAR information system. The evaluation will target program managers and M&E specialists within this category. At lower levels of the health care system, the evaluation will target a limited sample of field-based prime and sub-prime implementers, as well as SAG provincial, district/sub-district managers and personnel who have participated in ESI technical assistance and training (mainly geared to improving uses of strategic information). In sum, the following categories of informants will be included in the Evaluation: Level 1 – USG PEPFAR partners and SAG collaborating departments Level 2 – JSI/ESI and ESI sub-contractors Level 3 – ESI’s key collaborators among PEPFAR prime implementers Level 4 – a. Field-based prime and sub-prime PEPFAR implementing partners b. SAG provincial, district and sub-district DHIS managers plus personnel trained by ESI ESI End-of-Project Evaluation 51 An attempt will be made to contact the maximum possible Level 1, 2 and 3 informants. A selected sample of field-based PEPFAR implementing partners in North West, Gauteng, KwaZulu Natal, and Western Cape provinces will be visited. During the same visit, SAG provincial, district and sub-district managers in those same provinces who have participated in ESI training and capacity building activities will be contacted. In addition, two different samples will be drawn for Online Surveys targeting a) prime and sub-prime PEPFAR implementing partners (not covered in site visits); and b) the pool of SAG personnel who participated in ESI training and capacity building activities (not covered in site visits). Data gathering instruments and tools. The SI Team will use 4 formal data and information gathering methods during the evaluation as follows:  Review and analysis of information from background documents and data  Structured Key Informant Interviews (SKII)  Direct Field Observations through site visits  Online Surveys All instruments are in the design process, and will be finalized and ready for use from March 19th. It is likely that two versions of SKII will be needed – one for each informant Levels 1 and 2, because each level of informants is expected to bring substantially different perspectives. The Online Survey will utilize a simple rating scale or Y/N response format structured in such a way as to yield numerical scores. The Online Survey using the Survey Monkey service will be distributed in the week of March 19th with a return date of March 30th. In general, answers to strategic design questions will rely heavily on document reviews, augmented by informant interviews. Answers to project management questions will rely heavily on informant interviews. Answers to program management questions will rely heavily on review of project documents and reports, augmented by direct field observations with particular focus on improvements in data quality and use that can be attributed to ESI, and further augmented by an online survey focused on determining the value as perceived by beneficiaries of ESI training and technical support. The ESI Evaluation Framework. All of the above evaluation components together describe the what, who, and how of the ESI evaluation process. These components and their relationships one to another are depicted in the matrix shown in ANNEX A. Data synthesis and analysis. Data analysis will employ a parallel, mixed-data approach in which quantitative data from the Online Surveys are independently analyzed from qualitative data derived from informant interviews and observations. A parallel, mixed-date approach takes the findings and analysis from each data set and uses it to inform and explain findings from the other data set. As this evaluation methodology applies a mixed-methods approach to strengthen and validate findings for the same question through a triangulation design, the analysis involves comparing the findings on each data set to determine whether or not there is a convergence of findings. Qualitative data obtained from the site observations and SKIIs will be analyzed using a separate approach to elucidate emergent themes, contextual factors, and trends. To ensure full coverage of issues, maximum data synthesis and triangulation during the evaluation’s data analysis phase, each SI Team member has been assigned a technical lead for one of the 4 cross-cutting evaluation questions (discussed above). Thus, while all team members will be engaged in gathering information using standard formats ESI End-of-Project Evaluation 52 and protocols, the team will also be communicating closely to share impressions and reach consensus on findings and implications. Oral presentation. Upon completion of the data collection and analysis, the SI Team will produce a draft evaluation report as well as prepare a PowerPoint slideshow and formal, oral debriefing presentation for USAID. The SI Team recognizes the importance of the draft report, as it will be distributed to key stakeholders and discussed during the final oral debriefing and PowerPoint presentation to USAID. In order to make the most of stakeholders’ time, it is crucial that major findings and initial conclusions are presented in a clear and concise document, even at the draft stage. The SI Team Leader, Mildred Howard, will facilitate the oral debriefing and presentation in such a way as to maximize input from USAID staff and stakeholders. Reporting and dissemination of results. Upon submission of the draft report and presentation of the oral debriefing to USAID, the SI Evaluation Team will return to their locations of origin. The SI Team will then await final comments from USAID for a period of 14 days or less. Once the SI Team receives comments from USAID on the draft evaluation report, the SI technical and headquarters teams will take up to 7 working days to revise the report based on comments. An electronic and hard-copy Final Evaluation Report will be submitted to USAID no later than April 30, 2012. WORK SCHEDULE As described, the ESI Evaluation will be carried out in 4 phases. The estimated timeframes for completion of each phase are: Phase 1 – Finalization of Evaluation Strategy March 5 – March 17 (12 work days) Phase 2 – Conducting the Evaluation and Data Analysis March 19 – April 10 (14 work days) Phase 3 –Oral Presentation (including preparation) April 11 – April 12 (2 working days) Phase 4 – Reporting and Dissemination of Results April 16 – April 24 (7 working days) This work schedule is shown in calendar format as ANNEX C. ESI End-of-Project Evaluation 53 ANNEX A Key Evaluation Questions Strategic Design Issues Overall program - What were the objectively verifiable development problem and assumptions of ESI? Was there consistency and agreement among planners regarding the problems being addressed and the program design? - To what extent did the ESI project meet all project objectives as defined in its original Statement of Work? - Was the issue of capacity transfer and sustainability considered in design of the project? - How should future strategic information projects be structured to ensure that the best features of the predecessor systems are retained while appropriately expanding to meet evolving needs? - For the future, how can information systems developments involving bilateral and interagency stakeholders be better coordinated and managed? Training and Capacity Building - Did ESI capacity building interventions (training and technical assistance) provided organizations/individuals with practical and implementable capacities (knowledge and skills) that were relevant to their day-to-day operational challenges? - How might future capacity building interventions be designed for greater effectiveness and impact? Strategic Information Systems - Was ESI program as designed in sync with the Government’s priorities and approaches to information systems development? How could this alignment be strengthened in the future? - What USG planning and coordination mechanisms were in place to facilitate design and change control monitoring of information systems developments? Did monitoring include clearly defined targets against which to measure progress? - How effective were monitoring and change control mechanisms and what changes could be made in the future to improve their functionality and effectiveness? - Does the strategic information system developed under the ESI adequately meet end user requirements? How could this be improved in the future? - How effective has ESI been in improving systems-wide data quality and use of data for decision making? How could this be improved in the future? Project Management - What project management structures and processes were put in place to ensure that customized/on￾going training and technical assistance needs assessments were conducted? - Were adequate needs assessments actually carried out and were related ESI capacity building interventions sufficiently customized and flexible to meet the particular and specific training and ESI End-of-Project Evaluation 54 technical assistance needs of various beneficiaries? - What was ESI’s project management relationship with JSI regional and headquarters management? How critical was this backstopping to ESI/SA field operations? Was backstopping adequate; effective? - To what extent were the program and technical personnel able to transfer sustainable capacities (knowledge and skills) to beneficiaries? - Did ESI maintain a personnel management system that included regular personnel performance reviews? - Did JSI/ESI encounter any noteworthy problems in funds disbursement and management? Results Management - What were the verifiable and documentable achievements (quantitative and qualitative measures) of ESI performance in terms outputs and outcomes in each of its project tasks areas? - Have ESI deliverables been produced and has the project met its contractual obligations? - What management procedures were in place to monitor the program? What targets and indicators were used, who measured progress against indicators, on what schedule, and how were results verified? - What have been the major sustainable achievements of the ESI? ESI End-of-Project Evaluation 55 ANNEX B A Matrix of the Evaluation Framework Program Levels Partner Categories Strategic Design Project Management Results Management Capacity Building Information Systems 1 USG and SAG (CDC, USAID, DOD, PC, State, DoH, and DSD) Document reviews and discussions with key representatives Document reviews and discussions with key representatives Document reviews and discussions with key representatives 2 JSI/ESI and ESI subcontractors (Tulane, HISP, Khulisa, Manto) Structured Key Informant Interviews Structured Key Informant Interviews Structured Key Informant Interviews 3 ESI’s key collaborators among PEPFAR prime implementing partners Structured Key Informant Interviews Structured Key Informant Interviews Structured Key Informant Interviews 4a Field‐based prime and sub‐ prime PEPFAR implementing partners Site visits/observations and Online Survey#1 Site visits/observations and Online Survey #1 Site visits/observations and Online Survey #1 4b SAG provincial, district and sub‐district DHIS managers plus personnel trained by ESI Site visits/observations Online Survey #2 Site visits/observations Online Survey #2 Site visits/observations Online Survey #2 ESI End-of-Project Evaluation 56 March 2012 SUNDAY MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Team arrives in Pretoria a.m. Team planning meetings p.m. Meet with USAID All day All team members ‐ key informant interviews in Pretoria All day All team members ‐ key informant interviews in Pretoria All day All team members ‐ key a.m. USAID in‐briefing p.m. All team members ‐ key informant interviews in Pretoria All day All team members ‐ key informant interviews in Pretoria All day Team work day and planning at hotel in Pretoria 18 19 20 21 22 23 24 Team rest day in Pretoria All day All team members ‐ key informant interviews in Pretoria All day All team members‐ key informant interviews in Pretoria All day team members‐ key informant interviews in Pretoria All day All team members‐key informant interviews in Pretoria a.m. All team members‐ key informant interviews Pretoria p.m. Team travels to NW Province Team planning and work day at hotel in NW province 25 26 27 28 29 30 31 Team rest day in NW Province All day Team visits to provincial health headquarters, PEPFAR Partners, and district health offices in NW Province All day Team visits to provincial health headquarters, PEPFAR Partners, and district health office in NW Province a.m. Team visits district health office in NW Province p.m. Team drives to airport in Joburg and flies to Durban All day Team meets with PEPFAR Partners and key informants in Durban All day Team meets with PEPFAR Partners and key informants in Durban a.m. Milly and Erica fly to Cape Town Ro d and Beth work in Durban Evaluation Schedule ANNEX C ESI End-of-Project Evaluation 57 April 2012 SUNDAY MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY SATURDAY 1 2 3 4 5 6 7 a.m. Rod and Beth rest in Durban Erica and Milly rest in Cape Town p.m. Beth flies back to Pretoria Rod travels to PMB Erica and Milly interview PEPFAR Partners in Cape Town Rod meets with district health offices in PMB Beth meets with key informants in Pretoria Erica and Milly interview PEPFAR Partners in Cape Town Rod meets with district health offices in PMB Beth meets with key informants in Pretoria Erica and Milly interview a.m. Erica, Milly and Rod fly back to Pretoria Beth meets with key informants in Pretoria p.m. All team meets with final key informants in Pretoria All day team data sharing and analysis in Pretoria All day team data sharing and analysis in Pretoria All day team data sharing and analysis in Pretoria Begin work on report drafting and preparation of presentation 8 9 10 11 12 13 14 All day team report drafting and preparation of presentation in Pretoria All day team report drafting and preparation of presentation in Pretoria All day team report drafting and preparation of presentation in Pretoria All day team report drafting and preparation of presentation in Pretoria a.m. Team presents Power Point presentation to USAID p.m. Team members start to depart SA All team members depart SA 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 ESI End-of-Project Evaluation 58 APPENDIX E. DOCUMENTS REFERENCED Documents Referenced  ART Monitoring and Evaluation National Department of Health Workplan  Babatunde, Dr. Sanni, Summary of Objectives of Senior M&E Advisor Strategic Health Programmes National Department of Health  Capacity Building Planning for 2009-2010  Capacity Building Timesheet_Rentia_2011  Department of Social Development, Monitoring and Evaluation Framework: National Action Plan on Orphans and Vulnerable Children Affected by HIV and AIDS, 2009-2012, October 2009  Department of Social Development, Monitoring and Evaluation Plan Draft 1: M&E Capacity Building for the National Action Plan for OVC July 1, 2010-July 1, 2015, March 2010  Department of Social Development, Strategic Plan Update 2009/10-2011/12  Developing a Comprehensive Monitoring and Evaluation framework for the HIV & AIDS and STIs National Strategic Plan 2007-2011, November 2007  DHIS_PMTCT Workshop Assessment  DSD Support Workplan and Budget  ESI Capacity Building Retainment of Skills 2012  ESI Data Warehouse Transition Budget, March 2010  ESI Inventory Budget, March 2010  ESI PMTCT Budget, March 2010  ESI Task 1 Feedback – March and Work Plan April 2011  ESI Task 1 Feedback – May and Work Plan June 2011_Rentia  ESI Technical Team, Change Control, USG/ESI/Partners PIMS, November 2011  ESI Technical Team, Communication Plan for PIMS  ESI Technical Team, PIMS Functional Specification, September 2011  ESI Technical Team, PIMS Permission Grid.  ESI Technical Team, PIMS Reporting Proposed Policy for User Viewing Privileges, November 2011.  ESI Technical Team, Proposed District Technical Assistance Model for PEPFAR Support, November 2011.  ESI Year 3 Workplan July 2010- June 2011  ESI Year 3 Summary Budget, March 2010  ESI Year 4 Workplan Jan-March 2012  ESI Year 4 Workplan Jun-Dec 2011  Hurkchand, Hitesh, Budget Submission Email, July 2010  John Snow International, ESI Full Contacts List.  John Snow International, Enhancing Strategic Information – Project Organization Chart  John Snow International, Subcontract No. 13062-01, July 2008  John Snow International, Subcontract Modification No. 1, Subcontract No. 13062-01 under USAID Contract No. GHS I-03-07-00002-00, April 2009.  JSI/ESI, Annual Technical Proposal Report 1 Oct 2008 – 30 Sept 2009 ESI End-of-Project Evaluation 59  JSI/ESI, ESI South Africa Year 3 Abbreviated Workplan  JSI/ESI, ESI South Africa Year 4 Abbreviated Workplan  JSI/ESI, ESI Strategic Framework, August 2009  JSI/ESI, FY 09 COP JSI Enhance SI  JSI/ESI, FY 09 Semi-Annual Performance Report 1 October 2008 – 31 March 2009  JSI/ESI, FY 09 Semi-Annual Performance Report 1 Oct 2010 – 31 Mar 2011  JSI/ESI, Introduction to ESI North West Province Support, April 2010JSI/ESI, Report of ESI Best Practice in South Africa 2010  JSI/ESI, NDoH Workshop: ESI Course Overviews, November 2011  JSI/ESI, Quarterly Reporting Form: Fourth Quarter 2010  JSI/ESI, Project Annual Report Oct 2010-Sept 2011  JSI/ESI, Task 1 Strategic Framework 2008-2013, August 2008  JSI/ESI, Terms of Reference for Project: Optimise PMTCT Reporting in 18 prioritised sub-districts in South Africa, June 2009.  JSI/ESI, Timeline of Activities for ESI – North West Province Mar 2009- Feb 2010  Khulisa Management Services Ltd, ESI Project Task 4 (Data Quality) Khulisa Close-out Report for ESI-Task 4, February 2012  Kunaka, Derek, JSI/ESI, PIMS Draft User Requirements Specification, March 2012  Manto Management, Developing an M&E System: Establishing the Baseline  Map of Priority Districts  Mhlanga, Dr. Eddie, PMTCT Indicators Letter, March 2010.  NAP Core Indicators  NAP M&E Capacity Building Calendar 2011  NAP M&E Operational Plan  NAP Results Matrix – Mandla Revision v3  National Department of Health, Strategic Plan 2010/11-2012/13  North West Provincial Department of Health and John Snow International, Addendum to Memorandum of Agreement  North West Provincial Department of Health and John Snow International, Memorandum of Agreement, March 2009  PEPFAR Provincial Liaisons  Planned Activities for September – December 2010  PMTCT Monitoring and Evaluation  PMTCT Monitoring and Evaluation – Christa Inputs 16 March  Podges, S., C. Van den Bergh, S. Thela, N. Mzana, and R. Voormolen, National PMTCT Training Report, November 2011  Potgieter, Melinda and Walsh, Brennan, Map for South Africa, August 2009  Republic of South Africa, Government Gazette, South Africa National Health Act 2003, July 2004.  SAG PEPFAR Indicator Matching February 2011  Shrull, Jennifer, JSI/ESI, Enhancing Strategic Information Task 1: Capacity Building  Shrull, Jennifer, JSI/ESI, Enhancing Strategic Information Task 2: Orphaned and Vulnerable Children  Shrull, Jennifer, JSI/ESI, Enhancing Strategic Information Task 3: Data Use  Shrull, Jennifer, JSI/ESI, Enhancing Strategic Information Task 4: Data Quality ESI End-of-Project Evaluation 60  Shrull, Jennifer, JSI/ESI, Enhancing Strategic Information Task 5: Decision Support  Shrull, Jennifer, JSI/ESI, Enhancing Strategic Information Task 6: GIS  Shrull, Jennifer, JSI/ESI, ESI Model of Operation  Shrull, Jennifer, JSI/ESI, History of Enhancing Strategic Information  Statistics South Africa, Data Quality Policy 001: Policy on Informing Users of Data Quality, Feb 2006  Strengthening M&E Systems for NAP 2009-2012  Support to PMTCT – Background  Task 1: Capacity Building – goal, objectives, activities, and indicators  The Capacity Project, Promising Practices to Build Human Resources Capacity in HIV Strategic Information, December 2009  Thela, Samuel, EBHM Workshop Trip Report, February 2012  Thurman, Tanya Renee, Tulane University School of Public Health and Tropical Medicine, Final Report for Task 2 of the Enhancing Strategic Information Project South Africa: Compiling an Evidence-Base for Orphans and Vulnerable Children Programming, November 2011.  Timeline of Activities for the ESI Project, August 08-July 09  Timeline of Activities for the ESI Project, September 08 – July 09, September 2008  USAID, Enhancing Strategic Information Contract, July 2008  USAID, PIMS Quick Guide to Data Capturing  USAID, Terms of Reference for the JSI ESI Project to support the National Department of Health in South Africa to: Strengthen District Health Information System monitoring, evaluation, reporting, information management use of Prevention of Mother to Child Transmission data in 18 Priority Districts of South Africa, July 2010-June 2011  Van den Bergh, Christa, ESI DHIS PMTCT MER Project Feedback, January 2011  Van den Bergh, Christa, PMTCT Project Introduction Letter  Van den Bergh, C., S. Thela, N. Mzana, R. Voormolen, and S. Podges, PMTCT Indicator Progress – Financial Year 2010-1, November 2011  Van den Bergh, C., S. Thela, N. Mzana, R. Voormolen, and S. Podges, DHIS Resource Assessment Report, October 2011  Van den Bergh, C., S. Thela, N. Mzana, R. Voormolen, and S. Podges, JSI ESI Project Report: Strengthen DHIS PMTCT Monitoring and Reporting in 18 Priority Districts, July 2010 to June 2011, November 2011  Voormolen, Rentia, DHIS PMTCT Progress Report – June 2010 – March 2011, Free State Province – Thabo Mofutsanyane District Draft 2011  Voormolen, Rentia, Rapid DHIS PMTCT Situational Assessment, Free State Province, February 2011  Workplan Report July 2009-June 2010  Year 3 Budget_USAID Queries  Year 3 Workplan Task 2, August 2011  Year 4 Workplan – Risk Management Matrix ESI End-of-Project Evaluation 61 APPENDIX F. ESI EVALUATION TOOLS ESI Evaluation FOCUS GROUP DISCUSSION 4-5 April 2012 Questions for Discussion A. INFORMATION SYSTEMS 1. In the past did individuals within your organization receive training or guidance and instructions on how to use the Data Warehouse for reporting? 2. Have individuals within your organization received training or guidance and instructions on using PIMS for reporting and analysis? 3. What is your current understanding about how PIMS is expected to assist your organization in: Program planning? Resource allocation? Gap analysis? Data input and reporting? Identifying, locating and contacting other partners? Other? B. PROJECT MANAGEMENT 1. As PIMS was being developed over the past two years, how well did JSI/ESI keep you updated on new features and requirements of the strategic information software systems? 2. Have you ever encountered problems in the past in using the Data Warehouse? In using the newly introduced PIMS? If so, was assistance provided by JSI/ESI and/or Khulisa staff in a timely fashion? Was assistance useful? 3. How do you use strategic information in management of your HIV and AIDS activities? 4. How and to whom do you communicate any emerging user needs or concerns you might have related to strategic information requirements in your organization? 5. What do you understand to be involved in alignment of PEPFAR and government information systems and reporting requirements? How could PIMS better integrate with other SAG data management and reporting systems? ESI End-of-Project Evaluation 62 C. RESULTS MANAGEMENT 1. What is your understanding of the objectives of the JSI/ESI project, and how were you originally informed about JSI/ESI? 2. What were your expectations about how the JSI/ESI would be able to assist your organization and your work? 3. How have you/your organization interacted with JSI/ESI over the past 3 years? Has JSI/ESI provided your organization with support or training, or otherwise impacted the work of your organization? 4. What, in your view, have been major successes/weaknesses of JSI/ESI? 5. What recommendations can you make for future strategic information technical assistance or support projects? ESI End-of-Project Evaluation 63 Samples of Online Survey Forms ESI End-of-Project Evaluation 64 ESI End-of-Project Evaluation 65 ESI End-of-Project Evaluation 66 ESI End-of-Project Evaluation 67 ESI End-of-Project Evaluation 68 STRUCTURED KEY INFORMANT INTERVEWS Respondent category: Name of Organization Respondent(s) Names/Titles Date E-mail contact Length of time in current position RESPONSE TO QUESTIONS AS APPLICABLE: PART I (LEVEL 3 and 4a PIPs and Sub-PIPs) D. INFORMATION SYSTEMS 4. In the past did individuals within your organization receive training or guidance and instructions on how to use the Data Warehouse for reporting? Click here to enter text. 5. Have individuals within your organization received training or guidance and instructions on using PIMS for reporting and analysis? Click here to enter text. 6. What is your current understanding about how PIMS is expected to assist your organization in: Program planning? Resource allocation? Gap analysis? Data input and reporting? Identifying, locating and contacting other partners? Other? Click here to enter text. E. PROJECT MANAGEMENT 6. As PIMS was being developed over the past two years, how well did JSI/ESI keep you updated on new features and requirements of the strategic information software systems? Click here to enter text. 7. Have you ever encountered problems in the past in using the Data Warehouse? In using the newly introduced PIMS? If so, was assistance provided by JSI/ESI and/or Khulisa staff in a timely fashion? Was assistance useful? Click here to enter text. 8. How do you use strategic information in management of your HIV and AIDS activities? Click here to enter text. Assessor’s comments/ rating INTERNAL USE ONLY ESI End-of-Project Evaluation 69 9. How and to whom do you communicate any emerging user needs or concerns you might have related to strategic information requirements in your organization? Click here to enter text. 10. What do you understand to be involved in alignment of PEPFAR and government information systems and reporting requirements? How could PIMS better integrate with other SAG data management and reporting systems? Click here to enter text. F. RESULTS MANAGEMENT 6. What is your understanding of the objectives of the JSI/ESI project, and how were you originally informed about JSI/ESI? Click here to enter text. 7. What were your expectations about how the JSI/ESI would be able to assist your organization and your work? Click here to enter text. 8. How have you/your organization interacted with JSI/ESI over the past 3 years? Has JSI/ESI provided your organization with support or training, or otherwise impacted the work of your organization? Click here to enter text. 9. What, in your view, have been major successes/weaknesses of JSI/ESI? Click here to enter text. 10. What recommendations can you make for future strategic information technical assistance or support projects? Click here to enter text. PART II (LEVEL 4b District DOH) 1. Have you received any training or technical assistance from JSI/ESI and if so, what assistance was provided and was the assistance useful? Click here to enter text. 2. If training or technical assistance was received did this help you to improve the quality of data/reports? Timeliness of reporting? Use of data/reports for planning and management purposes? ESI End-of-Project Evaluation 70 Click here to enter text. 3. When was your last Routine Data Quality Assessments (RDQA) completed? Can you share results with us? Click here to enter text. ADDITIONAL KEY POINTS FROM NOTES: Click here to enter text. ESI End-of-Project Evaluation 71 APPENDIX G. KEY ESI MANAGEMENT-RELATED EVENTS 2008-2012 TIME PERIOD MANAGEMENT-RELATED EVENTS