This publication was produced at the request of The QED Group LLC-implemented USAID/Uganda Monitoring, Evaluation, and Learning Contract, Order No.: AID-617-C-13-00007. It was prepared by independent consultants Dr. Kiran Kamble, Dr. Charlotte Muheki, Janno Cazemier, Deogratious Sebuwufu, and George Ssali, with assistance from Augustine Wandera and Jordan Rief. DISCLAIMER The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government. Evaluation of the USAID/Uganda Private Health Support (PHS) Program June 2018 II Table of Contents 1.0 EXECUTIVE SUMMARY..............................................................................................................................................1 2.0 EVALUATION PURPOSE ...........................................................................................................................................3 3.0 PROJECT BACKGROUND........................................................................................................................................4 4.0 METHODS AND LIMITATIONS ..............................................................................................................................5 4.1 Evaluation Design Matrix...................................................................................................................................5 4.2 Evaluation Methodology.....................................................................................................................................5 4.3 Limitations/Challenges........................................................................................................................................7 5.0 FINDINGS.......................................................................................................................................................................9 5.1 Implementation ....................................................................................................................................................9 5.2 IR1: How did program activities help to expand the availability of health services by private providers?..........................................................................................................................................................................10 5.3 IR2: How was increased affordability of private health services and products achieved by the Implementing Partner? ...................................................................................................................................................25 5.4 IR3: How did the program result in improvements to quality of private health facilities and services?.............................................................................................................................................................................33 5.5 Were there possible integrations of PHS Program Goals?.....................................................................37 6.0 CONCLUSIONS ........................................................................................................................................................ 42 7.0 RECOMMENDATIONS ........................................................................................................................................... 44 Annex 1: PHS Scope of Work.......................................................................................................................................... 45 Annex 2: PHS SWOT Analysis......................................................................................................................................... 56 Annex 3: Lessons Learned from the Evaluation ........................................................................................................... 57 Annex 4: Evaluation Design Matrix ................................................................................................................................. 58 Annex 5: Detailed Evaluation Methodology .................................................................................................................. 74 Annex 6: Evaluation Sampling Framework..................................................................................................................... 79 Annex 7: List of Sampled Sites.......................................................................................................................................... 83 Annex 8: Data Collection Instruments........................................................................................................................... 85 Annex 9: List of secondary data reviewed..................................................................................................................... 86 Annex 10: Additional Evaluation Questions (HSS & OVC)....................................................................................... 91 Annex 11: Gaps in OP records at PHS HFs.................................................................................................................. 96 Annex 12: Select Program Indicators and Data ........................................................................................................... 97 III List of Tables Table 1: Sample selected.......................................................................................................................................................6 Table 2: DCA achievements.............................................................................................................................................. 12 Table 3: Summary OP data for the 56 PHS health facilities visited ......................................................................... 24 Table 5: Overall sampling framework............................................................................................................................. 79 Table 7: HFs missing OP data ........................................................................................................................................... 96 List of Figures Figure 1: Number of PHS supported Health Facilities by Year................................................................................ 11 Figure 2: Changes in OP attendance for males and females by year (Source: Sampled HFs) ........................... 11 Figure 3: OPD attendance (Source: DHIS2) ................................................................................................................. 12 Figure 4: Percent of available DCA allocations used per bank ................................................................................. 14 Figure 5: Loans from DCA and non-DCA Banks......................................................................................................... 15 Figure 6: Gender-disaggregated loan allocation............................................................................................................ 15 Figure 7: Urban/rural loan allocation & Breakdown by loans to female clients.................................................... 16 Figure 8: Existing vs. New clients..................................................................................................................................... 17 Figure 9: Centenary Bank interest rate variance.......................................................................................................... 18 Figure 10: Implementation of business trainings........................................................................................................... 20 Figure 11: Number of HFs offering CSP services......................................................................................................... 23 Figure 12: Trend of OP capacity (Source: sampled registrars at HFs).................................................................... 23 Figure 13: Partnerships with pharma firms.................................................................................................................... 26 Figure 14: HFs that have adopted regulatory practices.............................................................................................. 28 Figure 15: Comparison in commodity prices in HFs................................................................................................... 29 Figure 16: Comparison in commodity prices in pharmacies..................................................................................... 29 Figure 17: Variance in service prices at PHS HFs......................................................................................................... 30 Figure 18: Effect of partnerships on sales/prices.......................................................................................................... 32 Figure 19: Partnerships with professional councils...................................................................................................... 36 Figure 20: Client satisfaction to services........................................................................................................................ 38 Figure 21: Quality standards/policy at PHS HF............................................................................................................. 39 IV ACRONYMS AIDS Acquired Immunodeficiency Syndrome ART Anti -Retroviral Therapy ARV Anti -retroviral CBO Community Based Organization CDCS Country Development Cooperation Strategy CSO Civil Society Organizations CWD Children with Disabilities DCA Development Credit Authority DHMT District Health Management Team DHO District Health Officer DO Development Objective eMTCT elimination of Mother -To -Child Transmission FBO Faith Based Organization FB -PNFP Facility -Based Private Not -For Profit FP Family Planning GoU Government of Uganda HaaB Health as a Business HCB Health Care Business HCT HIV Counseling and Testing HF Health Facility HIPS Health Initiatives for the Private Sector HIV Human Immunodeficiency Virus HMIS Health Management Information System HSS Health Systems Strengthening IP Implementing Partner IR Intermediate Result IRB Institutional Review Board JMS Joint Medical Stores JRM Joint Review Mission MCH Maternal and Child Health MeTA Medicines Transparency Alliance MoH Ministry of Health MoLG Ministry of Local Government NDA National Drug Authority NFB -PNFP Non -Facility -Based Private Not -For -Profit NGO Non -Governmental Organization OP Out Patient OVC Orphans and other Vulnerable Children PEI Patient Exit Interview PEPFAR President’s Emergency Plan for AIDS Relief PFP Private For -Profit PHP Private Health Practitioner PHS Private Health Support PLWA People Living with AIDS PMP Performance Management Plan PMTCT Prevention of Mother -To -Child Transmission PNFP Private Not -For -Profit PPPH Public Private Partnerships in Health V PROMETRA The Association for Promotion of Traditional Medicine SIDA Swedish International Development Agency SMMC Safe Medical Male Circumcision SOW Statement of Work SQIS Self-regulatory Quality Improvement System TB Tuberculosis UAHPC Uganda Allied Health Professional Council UCMB Uganda Catholic Medical Bureau UHF Uganda Healthcare Federation UMDPC Uganda Medical and Dental Practitioners Council UMMB Uganda Muslim Medical Bureau UNACOH Uganda National Association of Community Occupational Health UNCST Uganda National Council of Science and Technology UNMC Uganda Nurses and Midwives Council UNMHCP Uganda National Minimum Health Care Package UOMB Uganda Orthodox Medical Bureau UPC Uganda Pharmaceutical Council UPHA Uganda Private Health Unit Association UPMA Uganda Private Midwives Association UPMB Uganda Protestant Medical Bureau UPMPA Uganda Private Medical Practitioners Association USAID United States Agency for International Development USG United States Government VHT Village Health Team VMMC Voluntary Medical Male Circumcision WHO World Health Organization 1 1.0 EXECUTIVE SUMMARY This report presents the findings of the evaluation of the USAID/Uganda Private Health Support Program (PHS). The intent of this evaluation was to comprehensively examine the PHS program over its implementation period and to determine to what extent the results were achieved and what factors enabled or hindered the achievement of these results. The original scope of work for PHS focused on the private-for-profit (PFP) sector with the original aim of strengthening PFP service delivery to provide better healthcare options for the population of Uganda. The PHS program conceptualized strengthening of the PFPs by 1) increasing availability of services; 2) increasing affordability of services and products; and 3) improving quality of services. The initial PHS contract was for a 5-year period starting in June 2013 with an estimated cost of US$18,649,973. The contract was modified in September 2015 adding on the Private-Not-For-Profit (PNFP) Health Systems Strengthening (HSS) component and extending the original programming. Another modification in September 2016 occurred increasing the estimated funding to US$ 37,396,654 and the period of performance extended to June 2018. This modification extended the PNFP HSS component and also added on an Orphan and Vulnerable Children (OVC) element. The PHS program evolved several times throughout the length of the program, through contract modifications, a large increase in funding, and differing funding streams causing changing prioritization and collection of data. Although PHS was very successful in meeting their annual targets and in service delivery targets specifically, PHS fell short in aligning the added components to their original Theory of Change (TOC) and results framework (RF), and these numerous changes hindered PHS’s ability to track their progress towards the IRs because they were increasingly tasked with collecting data around Malaria, HIV/AIDS, Family Planning, OVCs, and other areas that did not necessarily span the life of the project. PHS’s quarterly and annual reports show good progress towards many of their defined indicators, but progress towards the larger IRs was not conceptualized or tracked as thoroughly as it should have been. PHS was successful in several areas, including in fostering relationships with the councils, in establishing SQIS protocols, in working with the medical bureaus, in providing OVC support, and in meeting their service delivery targets. The program fell behind in lowering the prices of drugs, commodities, and services to the consumer, and in documenting progress or shortcomings towards the IRs across the life of the program. The evaluation team found that PHS had mixed results in achieving the IRs, but the larger picture shows that PHS also struggled with the learning and adaptation needed to be able to manage the competing components as a cohesive program. The following are the key recommendations from the evaluation: • USAID/Uganda should ensure that modifications and changes in scope are undertaken in a manner that does not fundamentally alter the theory of change (TOC) of the program. If the modification/changes in the scope do alter the TOC, then the program should revise the TOC including the results framework (result areas) and indicators to respond to these changes. • Implementing Partners (IPs) should periodically pause and reflect on whether their activities and the results being achieved are in sync with their TOC and the program objectives as set out in the task order, including program modifications. • IPs should undertake investigations (learning) during implementation intended to establish impediments to achieving results and objectives with intend to support adaptive management and decision-making, and not wait for mid and endline evaluations. • A CLA plan should be developed at the beginning of the program and hiring of a CLA specialist should be prioritized to ensure continuous application and integration of CLA approaches. 2 • Access to financing for the private health sector requires a phased approach- PHS should first prepare smaller HCBs to better situate themselves as candidates for loans, through interventions such as HaaB trainings, business skills development, and financial management, before linking the HCBs to the banks. Similarly, on the banking side, the IP should prepare partner banks in the functioning of the DCA mechanism and offer greater incentives for working with infant business entities. This approach should also be given a realistic timeframe, as these initiatives would take longer than the initial span of the program. • IPs should initiate other modalities to incentivize HFs to deliver comprehensive health services and to add additional services to clients, following the cancellation of use of “performance-based grants” by COP 15 guidance for Program engagement with HIV care and treatment services in the private sector. IPs need to adaptively manage changing strategic priorities in the private health sector. 3 2.0 EVALUATION PURPOSE The purpose of this evaluation is to comprehensively examine USAID/Uganda’s Private Health Support (PHS) Program to determine the extent to which expected results were realized, and what program￾related factors may have facilitated or hindered the implementation of the project and/or achievement of results. The following evaluation questions and sub-questions were examined, as per the scope of work provided by USAID to guide the evaluation. The scope of work is attached in Annex I for more details. 1. IR1: How did program activities help to expand the availability of health services by private providers? 1.1. Assess to what extent this project improved access to financial resources (e.g., loans) for the partner facilities. 1.2. What is the evidence showing that without the DCA support, banks would still be able to lend to private health sector clients? 1.3. Did business skills training employ successful and unique approaches or innovations that can be scaled up? What were they? 1.4. What changes have there been in comprehensive service availability and provider outpatient capacity? 2. IR2: How was increased affordability of private health services and products achieved by the Implementing Partner? 2.1. How have functional partnerships been established between health facilities and pharmaceutical firms? 2.2. To what extent have price regulation / standardization of commodities and services been established? 2.3. What is the relationship between newly established partnerships to changes in prices, sales or distribution of medicines? Have pharmaceutical prices and/or availability changed, and has this affected the consumer? What is the most important change and how has this made a difference in their lives? 3. IR3: How did the program result in improvements to quality of private health facilities and services? 3.1. How have public perceptions of the private health sector changed? And what evidence is there that the program has promoted this shift through enhanced the professionalism and technical capacity of providers? 3.2. Have private sector entities improved partnerships with professional councils and supervisory systems to enhance capacity to ensure professional conduct? 3.3. Have acceptable voluntary accreditation standards been developed and adopted by institutions such as the Medical and Dental Council, Nursing and Midwifery Council, Allied Health Council and Pharmacy Council of Uganda? 4. Were there possible integrations of PHS Program Goals? 4.1. What evidence is there that training provided in regard to professionalism and technical capacity influenced the public’s satisfaction with services? 4.2. Have improvements in self-regulatory functions and policy help to improve professionalism, or lead to supporting greater capacity in the sector? 4.3. Is there evidence that improved professionalism and technical capacity contributed to changes in public satisfaction with services? 4 3.0 PROJECT BACKGROUND The USAID/Uganda Private Health Support (PHS) Program is USAID’s flagship program in the private health sector in Uganda, built on the successes of USAID’s Health Initiatives for the Private Sector (HIPS) Project. The initial PHS contract was for the period for a 5-year period starting June 2013 with an estimated cost of US$18,649,973. The contract was modified in September 2016 adding on the PNFP HSS component and extending the original programming, and the estimated cost was increased to US$37,396,654. The purpose of this program is to contribute towards establishing a viable, cost-effective private sector option for health services in Uganda by: 1. Improving the credibility and cohesiveness of the private health sector; 2. Improving the competency and expanding the capacity of private sector providers. The focus of support is to provide technical expertise, enhance quality standards, improve access to capital, support accreditation, and provide leadership in the private sector. To achieve this, the Program had three primary objectives: ▪ Expanded availability of health services by private providers; ▪ Increased affordability of private health services and products; and ▪ Improved quality of private health sector facilities and services. PHS had an initial mandate to work with the Private-for-Profit (PFP) healthcare providers, including pharmacies and drug stores. The intent was to leverage the private health sector, which provides close to 50 percent of health services in Uganda, to advance key aspects of USAID/Uganda’s HIV/AIDS, Health, and Education (HHE) strategy and expand the private sector contribution to achieving national health goals. Implementation was to be in the USAID/Uganda HHE focus districts, 1 to complement District Development Plans (DDPs) and priorities. Activities under the Inter Religious Council of Uganda (IRCU) were folded into PHS and the Private-not-for-Profit (PNFP) component was added to the PHS portfolio after 1 year and work with Medical Bureaus and their respective 133 HFs were added in Year 4. Thus, components of Health Systems Strengthening (HSS) with the 4 Medical Bureaus (MBs) and their 130 select health facilities (HFs) and support to Orphans and other Vulnerable Children (OVCs) and their caregivers/households were added to PHS. 1 Amuru, Amolotar, Apac, Budaka, Bududa, Bugiri, Bukwo, Bushenyi, Busia, Butaleja, Dokolo, Gulu, Ibanda, Iganga, Isingiro, Kabale, Kalangala, Kaliro, Kamuli, Kamwenge, Kanungu, Kapchwora, Kasese, Kayunga, Kiruhura, Kisoro, Kitgum, Kumi, Kyenjojo, Lira, Luwero, Mayuge, Mbale, Mityana, Mpigi, Nakasongola, Namutumba, Ntungamo, Oyam, Pader, Pallisa, Rukungiri, Sironko and Ssembabule 5 4.0 METHODS AND LIMITATIONS This section provides evaluation methods that were used to answer the evaluation questions. The section covers evaluation design, sampling frame, and both sources of qualitative and quantitative data used in the evaluation. A quasi-experimental approach, using comparative technique – control vs intervention was applied. The reason for using this approach was the lack of adequate baseline data for the expected outcomes and specific areas of intervention. 4.1 Evaluation Design Matrix The detailed Evaluation Design Matrix with evaluation questions and sub-questions further informed by specific indicators and corresponding data sources, data collection instruments, and data analysis plan is attached as Annex 4. 4.2 Evaluation Methodology The evaluation team used a mixed methods approach for this evaluation combining both quantitative and qualitative methods to get answers to the evaluation questions, as described in detail below. Quantitative methods Quantitative methods used in the evaluation mainly consisted of descriptive measures and these included percentages/proportions, trend lines, pie charts, and bar charts. Specific measures and indicators were identified for each of the evaluation questions and sub-questions and due diligence was exercised in acquiring data for each of the indicators from primary data and from secondary data review. Qualitative methods The evaluation team applied qualitative methods for assessing the strategy, processes, achievements, challenges and risks in the PHS program. Qualitative methods were used to contextualize the primary and secondary quantitative data. Qualitative methods entailed review of context and PHS program information, reports and documents, collection of necessary information from the sampled sites, and acquiring relevant information from key stakeholders using semi-structured Key Informant Interview (KII) tools. The intent behind all the above was to gather process information to support the quantitative data for each indicator. The qualitative methods used in the evaluation included coding and thematic and context analysis. The evaluation team used Atlas.ti software to analyze qualitative data that was collected from key program stakeholders. Sampling Approach A purposive sampling approach was adopted to select the sites for primary data collection. The reason behind using this approach was to ensure adequate representation of each of the PHS private sector sub￾groups (PFP and PNFP health facilities, Civil Society Organizations (CSOs), pharmacies/drug stores/outlets, training institutes) while also ensuring adequate geographical distribution. PHS provided a list of partner sites – 71 health facilities (54 PFPs and 17 PNFPs), 15 pharmacies and 40 OVC CSOs. Table 1 details the number and percentage of sites sampled by the evaluation team. 6 Table 1: Sample selected Type Total No. of Units Sample PHS Units Percent Non-PHS (Control) Health Facilities PFP 54 362 67% 22 PNFP 17 12 71% 5 Medical Bureaus HFs (PNFP) 130 83 6% NA Pharmacy 15 9 60% 14 Medical Bureaus 4 4 100% NA Training Institutes 3 3 100% NA OVC-CSOs 40 6 15% NA Health Facility Clients NA 285 (5 per HF) NA NA CSO Households NA 30 (5 per CSO) NA NA For detailed evaluation methodology and design and sampling approach, refer to Annexes 5 and 6 respectively. Primary Data Collection Data collection tools were designed to collect appropriate primary data for each of the indicators. Secondary data was used to verify and validate primary data using the descriptive/subjective information and reports from the program. Provide and Equip (P&E) was contracted to undertake the primary data collection. 27 data enumerators, including 7 supervisors, were trained over 2 days. Field pilot testing of the tools was conducted with the 7 teams in Kampala across 7 sites. Additional training after the pilot testing was conducted to finalize the tools. Primary data collection was conducted from December 15, 2017 through January 31, 2018 across 37 sampled PHS program districts. All soft copies of the data files submitted were manually verified with the hard copies submitted by the field data collection team. To gather information on client perspective, 5 clients were selected from each of the sampled HFs and administered a Patient Exit Interview (PEI) tool. Every 2nd/3rd client exiting the Out-Patient Department (OPD) at the HF was selected using purposive sampling to ensure adequate gender representation and selection of repeat clients and not new ones. Ultimately 275 clients were interviewed. For OVC caregivers, five caregivers/OVC households were randomly selected from each of the CSOs for interviews from each of the 6 CSOs, thus a total of 30 OVC caregivers/households were selected. All the 4 Medical Bureaus (UCMB, UOMB, UMMB and UPMB) were interviewed using KII tools to assess the HSS initiative. Other key stakeholders were interviewed as well, and out of the 40 PHS key partner organizations identified for KIIs, 34 (74%) were interviewed. 2 The original sampling sought to include 37 PFP HFs, but 1 HF that was selected as part of the sampling could not be found in the field. This was Matama Medical Clinic in Kasese. 3 The PFP, PNFP, and Medical Bureau PNFP HFs add up to a total of 56 PHS-supported HFs that were visited for data collection. 7 In person meetings were held with each of the PHS activity teams to ensure the accuracy of all the data and information received. All primary data collection was done under the purview of ethics approval by TASO Institutional Review Committee (IRC) and Uganda National Council for Science and Technology (UNCST). Data Preparation and Analysis Plan All the primary and secondary data was first verified and validated. The secondary data was also triangulated between different program reports, information, and data. Excel was used to collate and analyze the quantitative data, while Atlas.ti software was used to collate and analyze the qualitative data. 4.3 Limitations/Challenges Some of the limitations and challenges that the evaluation team encountered during data collection and verification were: ▪ Training institutes refused to grant interviews citing absence of any collaboration with PHS o Mitigation measure: Interview was scheduled with the Uganda Private Health Training Institutes’ Association (UPHTIA) but the interview could not be conducted due to scheduling conflicts ▪ Some control HFs received HaaB training and this may have positively influenced their results, although they are categorized in this evaluation as controls since they were not included on the list from PHS o Mitigation measure: This is stated clearly in the report so that analysis can take this into consideration ▪ Inability to secure correct contact information for the PHS partner facilities o Mitigation measure: The evaluation team sought the assistance of PHS field supervisors to locate the facilities and get introduced to the facility in-charge ▪ Delays in securing visit dates with the facilities in regions and districts o Mitigation measure: Assistance was sought from the PHS senior management to secure visit dates from the facilities ▪ Physically finding the sites in the districts due to incorrect location information and getting correct contact information for the person in charge at the HFs, for example Matama Medical Clinic (MMC) in Kasese District, which was sampled from the PHS list – could not be traced o Mitigation measure: Strategy was adopted wherein the PHS field supervisors would help the data collectors to locate the facility and also introduce the data collectors to the facility in-charge; PHS supervisors were not allowed to be present during the actual interviews with the informants. Despite this PHS support the MMC in Kasese could not be traced ▪ Selecting matching control HFs o Mitigation measure: The data collectors were advised to randomly select non-PHS HFs with broader guidance on selecting matching level of facilities (HC II, III, IV and Hospitals), wherever possible ▪ Getting sector-wise loan disbursement data from DCA and non-DCA banks o Mitigation measure: The evaluation team tried to secure this data from the banks by providing an excel template using PHS team facilitation, but the required data was not provided 8 ▪ Denial of the administrator at Katureebe Pharmacy in Ntungamo District to be interviewed stating that the pharmacy had not benefitted from PHS intervention o Mitigation measure: The evaluation team sought PHS support, but was still denied ▪ Getting information from the MBs on implementation of various trainings and HSS initiatives across their HFs was not possible o Mitigation measure: Lack of this information did inhibit the team’s ability to understand the extent of HSS through each MBs HF network but it didn’t affect the conclusions drawn as both PHS and the MBs informed that many of the trainings and initiatives were being rolled out/planned to be rolled out in the HFs as it has been just 7 – 9 months since the inception of PHS’s HSS initiative with the MBs 9 5.0 FINDINGS 5.1 Implementation The PHS program started as a smaller initiative at its onset in 2013, focused on PFP HFs mostly in Wakiso and Kampala districts. But in 2014 the program was modified to support access to HIV/AIDS services through PNFP providers, previously supported by USAID through the Inter-Religious Council of Uganda (IRCU) which was folded into the PHS program. In 2016 another modification took place. PHS integrated PFP sector interventions designed to strengthen regulatory systems, improve reporting and financing, and build public-private partnerships, into the faith-based sector. This modification added on a large OVC component in addition to previous PFP and PNFP programming, increased the funding of the program from its initial US$18,649,973 to US US$37,396,654. Although the program always had an OVC component, this modification increased this area. There were several modifications over the life of the program, showing different priority areas based on funding streams. These baskets of funding gave additional resources to Malaria, HIV/AIDS, Family Planning, and others but did not necessarily cohesively fit into the program’s strategic plans, resulting in fragmented data collection, incoherent tracking of the IRs, changing reporting mechanisms, lack of focus, and lack of management. The evolution of PHS shows a program that expanded outside of its manageable interests and was tasked to take on activities based on availability of funding streams rather than alignment to its original objective but was successful in achieving its service delivery targets. The changes to the original scope were significant, and the evaluation team did not find documentation to show that these changes were accompanied by a revised theory of change or revisions to realign the IRs and program goal to the newly specified indicators and funding streams. The two major modifications (Modification in 2014 adding on the PNFP component and Modification in 2016 adding on much larger OVC and HSS components) shifted the focus of the program to service delivery, as is seen in the program indicators focused mainly on service delivery areas. These changes and the shifting priorities to meet PEPFAR and other specific funding requirements were met with success in achieving annual targets, but seemed to have caused the program to lose sight of their result areas. This made it difficult to evaluate PHS because the evaluation team was tasked with evaluating the program from their original scope and against their results framework, and not against their point-in-time requirements based on funding streams or specified indicators. The program indicators found in quarterly and annual reports (select indicators and data can be found in Annex 12) show progress based on quarterly and annual indicators, focused mainly on service delivery targets. These indicators show that the focus was on meeting service delivery targets, and not necessarily on how those indicators contributed to the IRs. This skewed measurement of progress toward the results because although the program reports show positive progress towards targets, the contribution of those targets to the IRs is missing. The evaluation team notes that management, strategy, and prioritization were important parts of the program that are not easily captured when answering the evaluation questions and sub-questions as presented. The findings of the evaluation are presented below, organized by Program IR and by the Evaluation Questions and Sub-Questions. 10 5.2 IR1: How did program activities help to expand the availability of health services by private providers? Overview of PHS Activities Under IR1 It is clear when reviewing PHS’ annual and quarterly reports that they undertook many different activities and were mostly successful in achievement of their programmatic targets. Under IR 1, the PHS program focused on strengthening service delivery at all of their supported PFP and PNFP HFs, with particular focus paid to strengthening and sustaining access to HIV/AIDS prevention, testing, and treatment services. In Year 1, the program focused on identifying private sector companies to participate in HIV prevention services, among other interventions. The program sought to support the delivery of comprehensive quality services by integrating HIV, tuberculosis (TB), voluntary medical male circumcision (VMMC), family planning (FP), malaria, maternal, newborn, and child health (MNCH) and prevention of mother-to-child transmission (PMTCT), and other services. Integration of these services was achieved through on-site training, mentorship, and coaching which emphasized identification of patients as well as quality treatment. PHS also did OVC household economic strengthening activities under this Result area, though it is unclear how these fold up into the IR to improve availability of health services specifically. The program conducted trainings to OVC caregivers on business management, financial management, and conducted household vulnerability assessments. There are many activities within this sub-IR, including trainings on nutrition and food security, trainings on child protection and legal support services, payments for school fees, and support for HIV-positive OVCs to access care and treatment services. Evaluation of PHS’ Progress Toward IR1 The program originally defined expanded availability of health services by private providers in terms of growth in number of HFs offering comprehensive health services, resulting in increased total outpatient capacity at these HFs. This is also how the evaluation scope of work defined this result area. To enable these HFs to expand the health services they provided, their access to financing (commercial loans) was enabled through the global USAID DCA mechanism. The HFs were also supported in strengthening their business and finance management through business strengthening trainings under Health as a Business (HaaB) strategy. PHS did not have any specific interventions to improve physical infrastructure and/or expand human resource capacity at the partner HFs. The DCA mechanism was supposed to have encouraged rural HFs to improve their infrastructure, and the trainings provided on business management and clinical skills were supposed to have improved the capacity to expand the availability of health services provided at these HFs. The DCA mechanism was met with both successes and shortcomings, and there is a lack of evidence to show that it improved banks’ attitudes towards lending to smaller HCBs and/or HCBs outside of the Central Region, which was a result area of the program. The global USAID DCA mechanism was introduced in Uganda with Centenary Bank and Ecobank, and the focus was to provide easier access to loans to HCBs outside of the Central Region from these banks. The DCA mechanism also provided a risk￾sharing safety net to banks to encourage them to loan to smaller HCBs which were seen as higher risk. Centenary Bank has used much more of their available loan allocation than Ecobank, but neither fulfilled the intent of the project. Although it should be noted that Centenary Bank performed much better than Ecobank to the point where the evaluation team was informed that Ecobank will no longer be participating in the program. Centenary Bank had strong utilization of its DCA and reportedly expanded its overall lending to the health sector. Loans to new clients – an indicator to show change in attitude of the banks to entertain HCBs beyond the established entities – has increased by only 9% over the life of the program. Female borrowers were 51% in a 2015 Centenary Bank study and have reduced to 13% in 2018, of which 4% are HCBs jointly owned with males. Loan disbursement to rural HFs has been 54% but only 14% of 11 these have been new clients. Since it was difficult to obtain data from the individual banks, data collected by the evaluation team was used as a proxy. This is true everywhere except where noted otherwise. To respond to Evaluation Question 1, we reviewed the trend in the number of HFs and pharmacies supported by the Program, the number/nature of trainings offered by PHS, the changes in OP capacity of the HFs, and access to financing by the PHS facilities. Where possible we make comparisons between PHS facilities and non-PHS facilities. Figure 1: Number of PHS supported Health Facilities by Year The chart shows a decline in the number of PHS partner facilities signifying contraction in capacity to offer quality services as per program specification, but also a reduction in the number of facilities supported by the program. It has to be noted that the 56 HFs sampled by the evaluation team were selected from the 71 remaining PHS HFs, hence they were under the program since the beginning. One of the reasons given for the reduction in the PHS number of facilities was a move under PEPFAR to reduce the number of low yielding facilities and to concentrate on the higher yield facilities. Data on changes in OP capacity at the PHS HFs was not directly tracked by PHS (PHS relied on OP attendance in the DHIS2) and neither were these partner HFs encouraged by PHS to record OP data other than through PHS’s distribution of materials. So, the OP data acquired by the evaluation team during its field visits had gaps in terms of monthly OP and gender-disaggregated. Available incomplete data suggests an increase of 8% in overall OP capacity from 2013 until 2017 at the 56 HFs visited. It has to be noted that these 56 HFs have been a part of PHS project since 2013 and are among the 71 HFs that have remained part of PHS from the initial 155 in year 1. The number of Out Patient (OPD) clients 155 129 90 71 0 20 40 60 80 100 120 140 160 180 Year 1 Year 2 Year 3 Year 4 Number of PHS HFs has declined to about half from Y1 to Y4 327,957 403,360 435,208 511,621 520,953 818,154 662,645 793,985 784,117 886,714 0 200,000 400,000 600,000 800,000 1,000,000 Year 1 Year 2 Year 3 Year 4 Year 5 Change in OPD attendance by Gender Males Females Figure 2: Changes in OP attendance for males and females by year (Source: Sampled HFs) 12 increased over the years, after an initial decline in Year 2, with PFP HFs driving the growth in OPD clients. OPD data from the 56 HFs visited shows that there has been an increase of 8% in total OPD from Year 1 to Year 5. Male OPD attendance across PFPs and PNFPs increased by 59% and female attendance by 8%. Overall PNFP had 3% OPD increase while PFP show a 10% increase. In terms of data quality, some HFs did not disaggregate the data by gender, therefore the estimate for gender-specific growth are based on only those HFs that did provide gender disaggregated data. Note that this change in male and female OPD attendance does not reconcile with the total OPD attendance change because not all HFs recorded gender disaggregated OPD data. The data collected by the sample is different than the data captured by the Program through DHIS2 but is similar in nature. OPD attendance data from DHIS2 show similar trends as compared to the evaluation survey; specifically, DHIS2 total OPD attendance data shows that total OPD attendance of the 71 PHS high yielding sites grew by about 17% over the period 2014 to 2017 as depicted by the chart below. Figure 3: OPD attendance (Source: DHIS2) Of the 56 sampled HFs, only 2 implemented all the services considered under the Comprehensive Service Package (CSP). 4 Of the 56 HFs, 32 (57%) have added at least one service to their basket of services through the program period. Comparing PHS and non-PHS facilities, we observe that the percentage of PHS facilities offering at least six (6) of the services under the comprehensive services is higher compared to the non-PHS facilities. More information is found in section 1.4 below. Table 2: DCA achievements Access to Finance Goals Baseline 2015 2017 Percent change Percent of loan portfolio to rural HCBs Not available 54% N/A Percent of loan portfolio to new clients 28% 37% + 9% Percent of loan portfolio to female owned HCBs 51% 13% - 38% Centenary Bank DCA utilization 63% 94% + 31% Ecobank DCA utilization 7% 17.5% + 10.5% 4 1) HIV prevention, 2) HIV Counseling and Testing, 3) HIV Care and Support, 4) ART, 5) PMTCT/eMTCT, 6) VMMC, 7) Malaria, 8) TB, 9) MNCH, 10) ANC, and 11) Nutrition 1,053,588 1,144,292 1,177,635 1,232,329 Oct 2013 to Sep 2014 Oct 2014 to Sep 2015 Oct 2015 to Sep 2016 Oct 2016 to Sep 2017 Total OPD attendance for 71 PHS supported sites grew by about 17% 13 Loans to new clients increased by only 9% overall since 2015 as per DCA loan disbursement data, but there were major differences between the performance of Centenary Bank and Ecobank. 37% of those provided loans were new clients in 2018 as compared to 28% new clients in 2015. 5 The DCA data also show that rural loans constituted 54% of all loans disbursed, but to determine whether adequate rural HFs were reached, we need data on the total number of rural HFs, which is not available. As of 2018, only of borrowers were female compared to 51% in 2015, indicating a reduction in loans to female borrowers.6 Details of DCA implementation have been discussed below under section 1.1. HaaB Training It was noted from among the 56 HFs visited that 34 (61%) were trained in Health as a Business (HaaB) with only 20 (59%) of those trained implementing new financial and management systems. The program expanded availability of services to clients through the service delivery platforms. The changes in OP data was more significant for men. From program reports we see that PHS introduced four private sector partners including large tea and sugar cane plantations, including New Forests Company Ltd, Kakira Sugar Limited, Mabale Growers Tea Factory Limited, and Rwenzori Commodities Ltd, to implement HIV/AIDS prevention activities, thus resulting in reaching larger numbers of men since the labor force of these plantations is male dominated. The program data for these service delivery areas seems to explain how the program increased availability, by offering services that led to an increase in OP capacity. PHS performance in each of the aforementioned activities, including the detailed findings and qualitative information gathered through Key Informant Interviews (KIIs) have been discussed under each sub￾question below. Sub-question 1.1: Assess to what extent this project improved access to financial resources (e.g., loans) for the partner facilities. In order to improve access to financial resources for partner facilities, the global USAID Development Credit Authority (DCA) mechanism was established in Uganda under PHS. The DCA was jointly supported by USAID and the Swedish International Development Agency (Sida), and Ecobank and Centenary Bank were involved under this DCA mechanism. This DCA provided a risk sharing platform to the banks by assuring guarantee of 50% - 60% of the amount of loans provided to rural Health Care Businesses (HCBs). By providing easier access to loans for rural HCBs, it was expected that the HCBs would be able to build/expand existing physical infrastructure, purchase new equipment, and hire additional human resources by seeking additional working capital. The business management trainings were aimed at building the capacity of the HCBs to better manage their business and efficiently utilize the loans. Based on the data provided by PHS, Centenary Bank had utilized 94% of its DCA allocated limit while Ecobank has utilized 17.5%. According to PHS, “The objective of the DCA is to encourage banks to lend to sectors which they perceive as having elevated risk, and therefore lending is generally expected to take place at market rates/conditions. In fact, the DCA increases the overall cost of financing to the bank, which they often will pass on to the borrower, at least initially, resulting in slightly higher interest rates. The DCA seeks to act as a catalyst to market entry. To ensure the long-term sustainability of health sector lending it is important for TA providers to let the bank set its own collateral requirements and interest rates and not distort the market, which will ultimately be unsustainable. The hope is that once the banks have success in lending to 5 A Credit Bureau Insight into Health Sector Borrowing in Uganda; Prepared for Centenary Rural Development Bank Ltd; Final Report - 24 March 2015 6 A Credit Bureau Insight into Health Sector Borrowing in Uganda; Prepared for Centenary Rural Development Bank Ltd; Final Report - 24 March 2015 14 a health customer, they will lower their estimation of risk, which would lead to a reduction in interest rate and potentially collateral on future.” Selection of the initial 2 banks was outside of the scope of PHS. Figure 4: Percent of available DCA allocations used per bank Of the 56 sampled HFs, 14 HFs (25%) had applied for a loan with a DCA bank (Centenary Bank or Ecobank) and of these, 7 (50%) obtained the loan. 11% (1 of 9) pharmacies sampled had applied for a loan with a DCA bank and 1 pharmacy obtained a loan. Of the 27 non-PHS (control) HFs visited, 6 (22%) obtained a loan from a DCA bank, whereas 7% (1 of 14) non-PHS (control) pharmacies had obtained a loan from a DCA bank. Of the 56 PHS HFs, 13 (23%) obtained loans from non-DCA banks. Of the 27 non￾PHS (control) HFs visited, 4 HFs (15%) obtained loans from non-DCA banks. 14 non-PHS (control) pharmacies were visited and of those 4 (29%) obtained loans from non-DCA banks. From among the DCA loans allotted, only 9.4% were to sole female-owned Health Care Businesses (HCBs), 3.4% to jointly owned HCBs, whereas 87.3% of the loans were given to male-owned HCBs. Note that this data is from sampled HFs and not from the total number of DCA-supported HFs. But PHS created an umbrella mechanism to also train non-DCA banks to increase uptake of loans, and to link referrals from non-DCA banks to DCA banks. This may explain the number of HFs that obtained loans outside of Centenary and Ecobank. 63% 66% 69% 73% 75% 80% 84% 90% 7% 7% 7% 7% 14.50% 17.50% 17.50% 17.50% Oct-Dec 15 Jan-Mar 16 Apr-Jun 16 July-Sept 16 Oct-Dec 16 Jan-Mar 17 Apr- Jun 17 July-Sept 17 Centenary Bank utilized almost all its DCA allocation as opposed to Ecobank Centenary Bank Ecobank 15 Figure 5: Loans from DCA and non-DCA Banks Figure 6: Gender-disaggregated loan allocation A higher percent of the 56 PHS HFs and 27 non-PHS HFs visited have secured loans from non-DCA banks. Outreach for loans outside of the Central Region was a key component of the project design. The results show that there was less expansion in the rural areas than expected, with 13.9% of loans allocated to new rural clients and 39.7% of the DCA loans allocated to existing rural clients, compared to 22.4% of loans allocated to new urban clients, and 24% of loans allocated to existing urban clients. The PHS Access to Finance (A2F) team stated that Centenary Bank had better utilization of DCA allocation because it had better rural presence as opposed to Ecobank which had minimal rural branches. The team also shared that before being involved under DCA, Ecobank was averse to allocating loans to rural HCBs and new HCBs without prior loan history. Ecobank also had more stringent mortgage requirements than Centenary Bank. The PHS team noted that it was not mandated to undertake any negotiations with the banks to reduce their interest rates and mortgage requirements, which was one of the major reasons for the poor uptake of these bank loans by the HCBs. During the KII with Ecobank representatives, they clarified that Ecobank requires every client who applies for a loan have 120% collateral coverage of the requested loan amount, as per Ecobank’s own stipulations which were not adjusted as part of the program. The bank itself notes 12.3 20 17.1 19.5 0 5 10 15 20 25 DCA Non DCA More PHS and non-PHS facilities obtained loans from non-DCA banks than from DCA banks PHS facilities Non PHS facilities 9.4% 3.4% 87.3% More loans were allocated to male than female clients Female Partnership (Female & Male) Male 16 that smaller facilities have not been able to benefit from these loans because the collateral requirement is too high. The evaluation survey results are in line with results obtained by analyzing DCA banks credit data as at October 2017 presented in the chart below. Figure 7: Urban/rural loan allocation & Breakdown by loans to female clients DCA bank credit data shows that both Centenary and Ecobank extended credit more to existing clients than to new clients, as depicted in the chart below. 17 Figure 8: Existing vs. New clients A KII conducted with a Sida representative noted that the criteria for selecting banks to be involved under DCA could have been better with adequate pre-assessment and alignment of the DCA intent of improving rural healthcare access with the banks’ commercial interests. The Sida representative also noted the importance of monitoring and follow up of specific outcomes focusing on access to health and to some extent gender equality/equity. This is something that could have been taken into account when designing the intervention and where Sida believes there is a gap in the current monitoring and follow up of the DCA. To get a better understanding of this, Sida is currently in the process of conducting a specific evaluation focusing at specific health related outcomes. A USAID Private Sector Unit – Economic Growth Office representative observed that Ecobank did not perform well as it did not have adequate mechanisms to connect with potential borrowers from the rural areas. This, according to the representative, could have been improved by better training of the bank officials and ensuring more loan allocations to HCBs for creating infrastructure (civil, equipment) and hiring human resources. Based on the preceding findings, the evaluation team concludes that PHS was only marginally able to improve access to financial resources for the partner HCBs, with less than desired loan allocations to new and female owned HCBs. One of the main reasons for this is the lack of tracking of loan allocation data to these two crucial entities by the banks and PHS. In addition, PHS and/or the banks have not routinely analyzed the purpose for which the loans were sought by the HCBs. Regular tracking of the aforementioned data points would have helped the banks and PHS to identify the gaps in achieving the DCA intent of increasing access to finances for rural HCBs and institutionalize corrective measures accordingly. The terms of the loans were not adjusted so barriers for smaller HCBs to accessing loans, such as high collateral requirements, were still noted to be reasons for low uptake. PHS notes that the DCA mechanism is meant to encourage lending to what the banks perceive as a sector with “elevated risk” and to not distort the market. Also, the lack of rural presence of the banks adversely affected their outreach ability. The system of equipment buy-back by the vendors and vendor financing by the banks established by PHS working closely with the banks, equipment vendors and HCBs was successful and helped establish stable and sustainable relationships between this crucial healthcare sector value chain. But it should be well noted that Centenary Bank performed much better than Ecobank. 71% 29% 75% 25% Existing Clients New Clients Both banks extended credit to more existing clients Eco Centenary 18 Sub-question 1.2: What is the evidence showing that without the DCA support, banks would still be able to lend to private health sector clients? The baseline survey of DCA borrowers (October 2014) showed that the banks in Uganda were not enthusiastic in allocating loans to HCBs, especially rural HCBs, due to lack of trust in the HCBs owing to their inadequate business and financial management systems. The DCA mechanism sought to establish trust between the banks and the HCBs which was to be manifested by banks allocating more loans to HCBs without the DCA risk-sharing support. Achievement of this outcome would have been determined by number of loans given to HCBs by the banks outside of the DCA mechanism. An increase in the number of such loans by the banks over the years would have indicated increased readiness of the banks to provide loans to the private health sector clients. Unfortunately, such data was not tracked by PHS and the banks have not been able to provide data on loans given to HCBs outside the DCA mechanism. Review of the available7 secondary data and primary data suggests that despite DCA support, only 36% of those provided loans were new clients and the majority of them are from urban areas (Figure 7 above). 75% (42 of 56) of HFs stated that banks (both DCA and non-DCA) would not be their first choice for loans due to high mortgage and interest rates. PHS organized workshops in 3 districts for 62 potential borrowers and 3 non-DCA banks to help foster relationships between the banks and the HCBs to expand financial out-reach through non-DCA banks. As a result, 10 deals worth US$184,571 have been generated through these non-DCA supported banks. Based on the loan disbursement seen by the evaluation team, more PHS than non-PHS HFs acquired loans from non-DCA banks. This suggests that in the absence of DCA, HFs would still get loans from the banks, provided that interest rate and mortgage requirements are favorable and manageable by the HFs General market inquiry did not indicate significant differences in the interest rates between the DCA and non-DCA banks. Moreover, an Ecobank representative noted that the majority of the clients provided loans under DCA by Ecobank were clients of the bank before DCA. Ecobank did state during the KII that there was a 20% increase in its loan portfolio due to involvement in DCA, but most of these loans were to larger, already established HCBs The PHS A2F team did not track data on loans provided by the banks to HCBs outside of the DCA mechanism but they did state that they observed increased favorability within the banks to provide loans to HCBs. The evaluation team, based on the available secondary data and interviews with the banks and the HFs, concludes that DCA has not been able to change the attitude of banks towards providing loans to HCBs without the DCA support. The banks have not undertaken any specific initiatives to target the HCBs, and high interest rates and high mortgage (more than 100%) requirements have not been negotiated or relaxed by the banks. 7 Data is awaited from Centenary and Ecobank on how many non-PHS vs. PHS loans have been given by both banks each year since 2013. Any increase in non-PHS loans over the years would indicate increased favorability of banks to lend to private health sector clients. 13% 34% 9% 44% 0 - 23% 24 - 26% 27 - 29% Not Sure/ Donot Know Interest rates variation at Centenary Bank Clients Figure 9: Centenary Bank interest rate variance 19 20 Sub-question 1.3: Did business skills training employ successful and unique approaches or innovations that can be scaled up? What were they? This initiative by PHS was successful in its conceptualization and initial execution but only partially successful in its end results. PHS applied the CLA approach to these trainings – after noticing some gaps in the first batch of HaaB trainings, PHS launched the HaaB II trainings with improved modules and both included post-training on-site counseling sessions. The gap here was the inability of the trained HFs (only 20% of the HFs sampled did so) to establish new financial and business management systems. This data collected by the evaluation team does not match with PHS survey findings of the HaaB 1 activity which found that 50% of the 62 trained HCBs prepared financial statements, budgets, and other business reports for the first time after the training. These results cannot be rectified based on the evaluation team’s field findings. Thus, the HaaB I and II trainings worked well as activities due to the due diligence exercised by PHS in ensuring that the training modules were relevant and addressed the business management gaps. The HFs visited overwhelmingly expressed satisfaction with these trainings, but just over half had adopted new business and financial management systems, indicating a gap between training and implementation. PHS did not track the institutionalization of new financial and business management practices by the trained HFs regularly but did track it during the Haab II endline survey. Regular tracking would have helped to identify the reasons for non-modification of systems and to adapt programming. As stated in the previous question, weak business and financial management systems of HCBs made them a high-risk sector for the banks to allocate loans to as noted in the DCA borrowers baseline assessment (October 2014). Based on this finding, PHS developed the Health as a Business (HaaB) training strategy, the first of its kind for HCBs. Under HaaB 1,192 HFs and under HaaB II, 82 HFs were trained in business practices followed by post-training, on-site counseling sessions. The following training approaches were administered: ▪ Mix of business skills and financial management linked with QI/SQIS ▪ Adoption of sales estimates and profit margins in inventory management systems ▪ Post-training, on-site counseling sessions; 3 to 5 sessions per HF 60.7% 50.0% 9.5% 58.8% 25.9% 29.6% 13.3% 71.4% Trained in HaaB I & Haab II BC/TA > 5 sessions Defaulted on loan repayment Implemented new financial and management systems Training approached employed by PHS and Implemention of new financial management systems PHS facilities non-PHS facilities Figure 10: Implementation of business trainings 21 ▪ Training/support to the HFs and Banks by the same entity (PHS) ensured a standardized approach 39 (70%) of the 56 HFs visited were trained in business strengthening and financial management. In all 39 there was satisfaction in its sampled clientele towards the availability, quality, and professionalism of services. Of these 39 trained HFs, 26 (67%) received post-training counseling and technical assistance sessions, with 8 HFs receiving 5 or more sessions. Of the 27 control HFs, 7 (26%) received HaaB I and HaaB II trainings and of these 7, 3 received post-training counseling and TA sessions. Of the 39 PHS HFs trained in business strengthening and financial management, 20 HFs (51%) implemented new financial and management systems in their HFs after the trainings. It needs to be noted that some of the non-PHS HFs (as classified in this evaluation because they were not on the list of supported facilities provided by PHS) also received HaaB training and this may have positively influenced their quality of care and financial management procedures. Of the 7 control HFs that received HaaB I and HaaB II trainings, 5 HFs (71%) implemented new financial and management systems after the trainings. None of the 3 trainings institutes sampled agreed to meet the evaluation team so information on other technical and capacity building training approaches could not be obtained. It was difficult for the evaluation team to tie the trainings to the clientele satisfaction that was expressed during the exit surveys undertaken during the evaluation exercise because of a lack of baseline or comparison data. The trainings undertaken by PHS employed unique and innovative approaches that can be scaled up. Marie Stopes, who was supported by PHS to identify some of their supported clinics to received financial training, observed that many of their service providers improved profits to enhance the development of their business either through consolidation of savings or through loans provided by financial institutions. The partner also noted that HFs benefited from training observed in HF financial records and patient records. The HFs interviewed expressed satisfaction with the HaaB trainings and stated that these trainings have helped them better manage their HCBs. The evaluation team concludes that the business skills training developed and provided by PHS employed unique approaches that need to be scaled up, such as – 1) Linking business and finance management with QI initiatives; 2) Strengthening inventory management systems by adopting sales estimates and profit margin concepts; 3) One-on-one post-training counseling sessions; and 4) The same entity (PHS) supporting the HCBs and the banks ensuring cohesive approach. These business skill trainings are expected to improve profit-oriented business management practices at the HCBs. Such efficient business management is expected to drive high quality service delivery aimed at meeting the needs of the community and in turn bolstering increase in business. Sub-question 1.4: What changes have there been in comprehensive service availability and provider outpatient capacity? This question aims to determine two important outcomes under IR1 – changes in comprehensive service availability and changes in the provider out-patient (OP) capacity at the partner HFs. The evaluation team understood changes in provider OP capacity as the changes (increase or decrease) in the number of patients catered to by the partner HFs. Specific interventions targeting changes in OP capacity were not undertaken by the program but it was assumed that increase in the number of services provided by the HFs would lead to an increase in the number of clients accessing services at these HFs. The program objective to increase CSP at partner HFs fell short, though the PHS HFs did offer more services in general which is a success in and of itself. Only 2 of the 56 HFs sampled were offering the full package of all 11 services from the CSP, although at least 10 HFs from among those sampled did have the 22 capacity to implement all the 11 services. Furthermore, only 57% of the HFs managed to add just 1 or 2 services to their basket of services with PHS intervention over 5 years. One reason found to explain the low uptake was that in a Quarterly report, PHS stated that “In Year 2, the Program had proposed (subject to financial capacity and risk assessments of potential grantees) to use performance-based grants for the high-volume sites. Unfortunately, the Program has cancelled this grant as a result of COP 15 guidance for Program engagement with HIV care and treatment services in the private sector and the need to transition clinical sites.” PHS had proposed to increase the services offered through performance-based grants, but because of changing PEPFAR priorities and guidelines this was not possible. But mitigation measures or other ideas for incentivizing HFs to offer additional services could not be found in program documentation. There was no data on OP collected by PHS, despite this being listed as an outcome area. The OP data collected during the evaluation was the only data available for analysis and it cannot be solely relied on, as in many HFs there was a lack of source documents for the OP data, e.g. OPD registers, daily patient registers, etc. In addition, many HFs did not have gender disaggregated OP data and in some cases even total OP data for some years and months. The data does show a slight overall increase in OP over the years, but a larger increase for men. Comprehensive services package has been defined by PHS as including the following - 1) HIV prevention, 2) HIV Counseling and Testing, 3) HIV Care and Support, 4) ART, 5) PMTCT/eMTCT, 6) VMMC, 7) Malaria, 8) TB, 9) MNCH, 10) ANC, and 11) Nutrition. As a part of improving availability of health services by the private sector, PHS developed training modules for different cadre of staff in the area of clinical skills for the 11 aforementioned areas, supply chain management, health management information systems (HMIS), business management, etc. PHS did not support the HFs in creating/expanding their infrastructure or hiring additional human resources. There were more PHS facilities offering more elements of the CSP compared to the non-PHS facilities services. For example, 63% of PHS facilities offered 6-11 elements of the CSP compared to 21% of the non-PHS that offered 6-11 elements. Figure 11 below shows that the PHS facilities offer more elements of the CSP package compared to the non-PHS, which demonstrates availability of more of the CSP elements in the PHS facilities. Increasing the number of services offered by a facility is a process and a function of several factors, i.e. human resources, space, equipment and financial resources. Looking at the shape of the bar chart in Figure 11 below seems to suggest that the PHS facilities are moving to the right, i.e. having several facilities offering more or all of the services under the CSP. 23 Figure 11: Number of HFs offering CSP services Figure 12: Trend of OP capacity (Source: sampled registrars at HFs) PHS did not collect OP data even though this was an outcome of the project. The program notes that this is available in DHIS 2, but the evaluation team did not find analysis or internal tracking in PHS records. The evaluation team therefore had to rely on OP data collected from the visited HFs, and there were incomplete and missing data which undermine the quality of the data. This also explains why the gender￾disaggregated data seen in Figure 2 above does not reconcile with the totals seen here in Figure 12. Many of the HFs did not collect gender-disaggregated data so the changes in male and female attendance are drawn only from the available data. But as was previously mentioned, PHS worked with tea and sugar plantations whose workers are mostly men, which seems to explain the significant increase in male OP attendance. Of the OP data collected from the 56 HFs visited, the data shows that there has been an increase of 8% in total OP from Year 1 to Year 5 in all the HFs, and PNFP HFs have shown 3% increase in 973643 757455 914694 932032 1073087 360902 264327 287576 344215 372897 Y1 OP Y2 OP Y3 OP Y4 OP Y5 OP Trend in OP capacity shows an increase over the five years for PFP PFP PNFP 24 OP while PFP HFs show a 10% increase. A summary of the numbers collected at each HF sampled are listed below in Table 3. These numbers should be compared to Figure 3 above, which shows the OPD data that is available in DHIS2 and was reported by each of the HFs. There are some discrepancies in the numbers from the sample and those reported in DHIS2, but the trends are similar and show an upward swing, indicating that the HFs are making progress in this area. These sources should be triangulated and analyzed together to better understand the use of distributed registrars and how/why these discrepancies exist. Table 3: Summary OP data for the 56 PHS health facilities visited Year 1 Year 2 Year 3 Year 4 Year 5 Increase year 1 to year 5 Percent increase year 1 to year 5 PFP 973,643 757,455 914,694 932,032 1,073,08 7 99,444 10% PNFP 360,902 264,327 287,576 344,215 372,897 11,995 3% Total 1,334,54 5 1,021,78 2 1,202,27 0 1,276,24 7 1,445,98 4 111,439 8% Gender-disaggregated data was not recorded at all HFs (Refer to Annex 11 for details) Males 327,957 403,360 435,208 511,621 520,953 192,996 59% Females 818,154 662,645 793,985 784,117 886,714 68,560 8% None of the 56 HFs had any evidence of formal referral linkages established with higher/lower level HFs. In addition, all the 32 HFs (18 PFPs and 14 PNFPs) that added at least 1 service during project implementation showed an increase in OP. The most common services added were Maternal and Child Health (MCH) and HIV services – Counseling and Testing and ART. The evaluation has seen lower than expected adoption of all elements of the CSP, with 63% of PHS facilities providing more than half (6-11) of the services in the CSP. There is high partial adoption in the PHS facilities compared to the non-PHS facilities. The evaluation team was not in position to establish the drivers of CSP adoption, but this is something worth investigating. The PHS Health Services Team noted during the KII that HFs that already had the necessary infrastructure and human resource availability were provided trainings to include specific services in their basket. PHS did not provide support in expanding infrastructure or hiring additional human resources. There were no specific interventions supporting the HFs in increasing their OP capacity. Despite the data on number of HFs providing specific services under CSP, the evaluation team has not been able to determine changes in comprehensive service availability at the HFs due to absence of any baseline data on the number of CSP services provided by these HFs at the beginning of the program. Some HFs did inform the evaluation team of certain services added/deleted during the program period but there was no documentation of these changes. Although the evaluation team did collect data on changes in OP capacity at the visited HFs, there were gaps in this data in terms of completeness and gender-disaggregated data. PHS did not track the changes in OP capacity so, although, 77% (43 of 56) of HFs stated that there have been changes in service availability which led to an increase in OP capacity, this could not be verified as comprehensive OP data has not been 25 recorded over the project period. As discussed above, there has been an increase in the overall OP capacity noted at the 56 HFs visited, however, due to lack of specific HF data on changes in service availability, it is difficult to attribute this increase in OP capacity to PHS interventions promoting expansion of service availability. It was also noted that the number of PHS partner facilities has decreased by nearly 50% over the project period. At the inception of the project, there were 155 PHS partner HFs that came down to 71 HFs at the end of the project. There is no documentation/analysis of the reasons for this reduction in the number of PHS HFs. KIIs with the PHS team informed that PHS removed some HFs from the list of partners due to non-performance in the area of quality improvement and some HFs opted out as they expected financial support and did not get it. Documentation of reasons for this partner HF reduction would have helped understand the challenges in working with HCBs and refine future such initiatives. Thus, the evaluation team has been able to determine an increase in OP capacity through the project period at the 56 partner HFs visited but could not ascertain significant changes in comprehensive service availability. 5.3 IR2: How was increased affordability of private health services and products achieved by the Implementing Partner? Overview of PHS Activities Under IR2 In reviewing PHS’ quarterly and annual reports, it can be seen that they undertook many activities under IR2. PHS conducted a pharmacy and drug shop census in Kampala working with KCCA, and conducted an awareness campaign on rational use of medicine in Year 4. PHS also worked to develop a recommended pricing list for essential medicines in Years 2 and 3, and disseminated professional fees guidelines. PHS also worked to promote Community Based Health Insurance (CBHI). VSLA groups were selected and trained, and a grantee- Integrated Community Based Initiatives (ICOBI)- rolled out a low-cost health care plan in rural communities. This community health insurance scheme sought to reduce out-of-pocket expenses to the consumer. PHS also worked directly with MOH to update national referral guidelines, using lessons learning from a study commissioned through PHS. Evaluation of PHS’ Progress Toward IR2 The PHS program defined affordability as a minimum 30% reduction in unit sale prices of essential health commodities and service packages in program supported areas, as compared to the general market. This was planned to be achieved through establishment of price regulation and standardization of health commodities and services as policies and disseminating these throughout the partner facilities (HFs and pharmacies) network. Coupled with establishment of functional partnerships between HFs and pharmaceutical firms, these initiatives were also supposed to have led to increased sales/distribution of commodities and services at PHS partner facilities. As a sustainability measure it was anticipated that PHS partner facilities would eventually “graduate” to function independently of PHS support based on the current model. Although some important initiatives were implemented under this IR area, the ultimate objective of reduction in prices for consumers at the PHS partner HFs was not achieved, though partner pharmacies did show better results in price reduction than the HFs. Two different approaches were applied as a part of this IR implementation, including a policy level intervention. 26 PHS has been only partially able to increase affordability of private health services and products. 64% of PHS HFs have partnerships with pharmaceutical firms, compared to a similar rate of 63% of the non-PHS HFs visited that also have partnerships with pharma firms. The percentage of PHS and non-PHS HFs having established partnerships with pharmaceutical firms is similar and this seems to be driven by market forces dictating alliance for increased business and profitability. This shows that the program did not have a meaningful impact in this area. There has been a 55% increase in sales/distribution of commodities and services at the PHS HFs, which can be attributed to these partnerships. However, this has not led to cost savings for the beneficiaries, as seen in section 2.2 below. Despite these partnerships and the increase in sales/distribution, PHS HFs have higher prices for 12 of 15 tracer products (7% to 158%) than the control HFs. But in contrast, PHS pharmacies have reduced prices for 10 of the 15 tracer products (32% to 100%) compared to the control pharmacies. PHS HFs also showed an increased costing (3% to 494%) for 5 of the 6 services and reduced costing (41%) for only the HTS service as compared to non-PHS HFs. This is a deviation from the expected outcome of PHS partner facilities having a minimum of 30% reduction in prices of services and commodities as compared to the market prices. One of the root causes for this deviation seems to be the price regulation/standardization adoption data, wherein only 36% of PHS HFs have adopted MOH price regulation standards, while 56% of PHS pharmacies have adopted these standards. Sub-question 2.1: How have functional partnerships been established between health facilities and pharmaceutical firms? As a part of improving affordability of private health services, PHS aimed at establishing partnerships between HFs and pharmaceutical firms. Such partnerships were expected to reduce costs of commodities and pharmaceuticals procured by the HFs from these pharma firms, which in turn would have led to reduced prices to be paid by the clients. During discussions with the PHS team, it was stated that functional partnerships were defined as those that have led to such price reductions. 64% of the PHS HFs (36 of 56) have established functional partnerships with pharmaceutical firms, of which 18 are PFPs and 18 are PNFPs. Of these 36, 25 HFs (69%) (10 PFPs and 15 PNFPs) have a formal partnership with a signed MoU, while 11 (31%) (8 PFPs and 3 PNFPs) are informal. Of the 27 non-PHS HFs visited, 17 (63%) have established partnerships with pharmaceutical firms - 13 being PFP HFs and 4 PNFPs. Thus, within the small sample of control (non￾PHS) HFs there is better institutionalization of pharma partnerships, showing a lack of success on behalf of the PHS program. Most of these partnerships have been based on some criteria according to reports and interviews, 64% 36% 69% 31% 63% 37% 100% 0% Established Partnerships No Partnerships Formal Partnerships Informal Partnerships Non-PHS HFs have more formal partnerships with pharmaceutical firms than the PHS facilities PHS Non PHS Figure 13: Partnerships with pharma firms 27 although there is no documented set of specific criteria that was available with any of the HFs. The PHS team informed the evaluation team that the criteria were developed by each HF and the pharma firm based on their mutual needs. There is no formal mode of communication established between the HFs and pharma firms in any of the partnerships. All the partnered HFs, however, informed the evaluation team that a regular coordination takes place through phone calls, e-mails and sometimes in-person meetings, though there was no documentation of these interactions. PHS has not formally documented the learning of collaboration of the partnership between facilities and pharmacies. It has been difficult to answer this question of how such functional partnerships have been established between the HFs and pharmaceutical firms as the processes undertaken to establish such partnerships were not documented by the concerned HFs and pharma firms or PHS. The interviewed HFs stated that the partnerships were established with pharmaceutical firms with whom the HF was already conducting business. No specific strategy or change in operational aspect of the partnership had been undertaken. When the established partnerships data between the PHS (64%) and non-PHS (63%) HFs is compared, it is difficult to attribute these partnerships to PHS interventions as they seem to be driven by commercial benefits for the HFs as well as the pharmaceutical firms. Sub-question 2.2: To what extent have price regulation / standardization of commodities and services been established? PHS has worked with MoH to establish standards for price regulation/standardization of commodities and services, and professional fee guidelines to ensure affordability of services through the private sector. These guidelines and standards define a framework for ensuring service quality standards based on international norms and to reduce the high costs of medicines due to high retail mark-ups levied in country. PHS support to MoH in establishing prices for essential medicines and standardization/regulation of commodities and services has worked well. However, the adoption of price regulation by PHS supported HFs was inadequate with only 37% adopting it. The adoption by partner pharmacies was better with 56% adopting these regulations. The program aimed to reduce the prices of services and commodities by a minimum of 30% at the partner facilities (HFs and pharmacies) as compared to the general market. PHS did not track the price variations despite it being an important outcome. PHS stated that they sent M&E officers out annually to a few sites to collect data on this indicator, but it was not regularly tracked over a wider catchment area. Thus, an opportunity to look at the trend in prices over the years and identify and analyze the reasons was missed, and therefore changes and adaptations to counteract these price increases was not initiated by PHS. Based on the field data collected around the average price of 15 tracer products, price reduction was largely achieved by the partner pharmacies but not by the PHS-supported HFs. The PHS HFs showed an increase in prices compared to the non-PHS HFs for 12 of the 15 products and reduced prices for only 3 products as compared to non-PHS HFs. In contrast, PHS pharmacies showed reduced prices compared to non-PHS pharmacies for 10 of the 15 products, with an increase in prices for 5 products. Of the 6 service areas identified, PHS HFs showed increased prices for 5 areas, up to 494% increase for 1 service area, as compared to non-PHS HFs, while reduced prices were seen at PHS HFs only for HIV Testing Services (HTS). But it must be noted that the evaluation team collected the price of the highest and lowest products and used the average as the price for the HF or pharmacy. This may have caused inconsistencies if PHS HFs offered a wider variety of products or higher quality products as compared to the non-PHS HFs. A reduction in price was not seen at HFs, but the methodology needs to be considered as a factor 28 when analyzing the results. It should also be noted that PHS targeted all pharmacies across the country, but only targeted select districts for HFs. Figure 14: HFs that have adopted regulatory practices Of the 56 HFs visited, 21 HFs (37.5%) have adopted these norms for price regulation/standardization of commodities and services; of these 8 are PNFPs and 13 are PFPs. Of the 27 control HFs, 6 (22.2%) have adopted these standards for price regulation/standardization of commodities and services; of these 2 are PNFPs and 4 are PFPs. The evaluation survey finding above show that about 38% PHS and 22% of non-PHS facilities have adopted price regulation/standardization of commodities and services. Without a baseline value, it is difficult to gauge the shift in adoption. If we assume that the difference between the PHS and non-PHS is a measure of PHS contribution to this process, then 16% can be contributed to PHS intervention. Slightly more PHS HFs adopted price regulations, but this did not reflect in their overall pricing, as is discussed below. The highest and lowest prices for 6 services8 and 15 commodities9 offered were collected from the 56 PHS HFs, 27 non-PHS HFs, 9 PHS pharmacies, and 14 non-PHS pharmacies, and were analyzed to verify whether there has been a minimum of 30% reduction in prices for these services and commodities at the PHS HFs and pharmacies as compared to the market – non-PHS HFs and pharmacies, as was an objective 8 1) Intrauterine Device (IUD) insertion, 2) 1 Ante-Natal Care (ANC) visit, 3) HIV Testing Services (HTS), 4) Consultation for Sexually Transmitted Infections (STI), 5) Diagnosis of TB using microscopy, and 6) Diagnosis of malaria using microscopy 9 1) 1 strip of Combined estrogen progesterone oral contraceptive pills, 2) 1 strip of Progestin-only contraceptive pills, 3) Combined estrogen progesterone injectable contraceptives, 4) 1 female condom, 5) 1 IUD, 6) 1 strip of emergency contraceptive pill, 7) 1 Benzathine benzylpenicillin powder for injection, 8) 1 strip of Co-trimoxazole cap/tab (Oral antibiotic), 9) 1 strip of Metronidazole cap/tab, 10) 1 strip of Aspirin cap/tab, 11) 1 strip of Paracetamol tab, 12) 1 vial of Epinephrine injection, 13) 1 strip of Omeprazole tab/cap, 14) 1 full course of ACT (Artemisinin Combination Therapy), and 15) 1 full course of adult dose Chloroquine tab 5.4% 10.7% 14.3% 30.4% 37.5% Involved in pooled procurement Prices recommended for essential medicines for the private sector Adopted PHS developed professional fee guidelines Facilities included in the maternal health voucher program Adopted standards for price regulation/standardization of commodities and services 0.0% 0.0% 3.7% 7.4% 22.2% Involved in pooled procurement Facilities included in the maternal health voucher program Adopted PHS developed professional fee guidelines Prices recommended for essential medicines for the private sector Adopted standards for price regulation/standardization of commodities and services PHS facilities Non-PHS facilities 29 of the project. Note that the average was taken and used for analysis. The number of products offered was not collected, and this should be taken into account when analyzing the results presented. It is unknown whether PHS HFs and pharmacies offered a larger selection of products or higher quality products. Figures 15, 16, and 17 below illustrate the difference in prices. Of the 15 tracer commodities, there was an increase of 7% to 158% in the average prices of 12 products at PHS HFs as compared to the non-PHS HFs. 3 products showed 6% to 95% reduced average prices at PHS HFs than the non-PHS HFs. Figure 15: Comparison in commodity prices in HFs Figure 16: Comparison in commodity prices in pharmacies 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 PHS HFs registered higher prices for 12 out of 15 commodities PHS HF Control HF 0 1000 2000 3000 4000 5000 6000 7000 Non-PHS pharmacies registered higher prices in 10 of the 15 commodities PHS Ph Control Ph 30 In contrast to the HFs, PHS pharmacies showed 32% to 100% reduced average prices for 10 of the 15 tracer commodities as compared to non-PHS pharmacies, whereas 5 commodities had 12% to 97% higher average prices at PHS pharmacies as compared to non-PHS pharmacies. Price comparison for 6 services showed that the PHS HFs had higher prices ranging from 3% to 494% for 5 of the services as compared to non-PHS HFs. PHS HFs showed reduced price (41%) for HIV testing services (HTS) as compared to non-PHS HFs. Of the 56 HFs visited, only 8 HFs (14%) stated that there has been a price recommendation for essential medicines; of these 7 are PNFPs and 1 PFP. In addition, 4 of the 9 pharmacies (44%), 9 of 27 control HFs (33%), and 6 of 14 control Pharmacies (43%) also had price recommendations for essential medicines. Of the 56 HFs visited, only 8 (14%) have adopted the professional fee guidelines; of these 4 are PNFPs and 4 PFPs, whereas 7 of 27 control HFs (26%) have adopted these professional fee guidelines. Of the 56 HFs visited, only 3 (5%) are involved in pooled procurement. All 3 are PNFPs and they have undertaken 3 pooled procurements till date. There has however been no documentation of these activities by the HFs. Of the 56 HFs visited, 17 (30%) are included in the maternal health voucher program; of these 9 are PNFPs and 8 PFPs. Although PHS has set up regulations/guidelines with MOH, the adoption of these by PHS HFs has been low. There is a disconnect between PHS’ work at the policy level and actual adoption and implementation on the ground. This is also seen in that fact that 12 of the 15 commodities at the PHS HFs registered higher prices than the control facilities, and higher prices for 5 of the 6 services as compared to the control HFs. Although there was some success seen at PHS pharmacies, where prices were lower for 10 of 15 commodities, the program did not meet its intended target overall to provide commodities and services at a minimum of 30% cheaper than the market rate. The Coalition for Health Promotion and Social Development (HEPS) Uganda, was commissioned by PHS to undertake a costing and pricing study for various clinical services in 2013. HEPS, based on the study, proposed median price list for treating different conditions and diseases. These recommendations have not been formally accepted by the National Drug Authority (NDA)/MOH or UMDPC. PHS team informed that these recommendations are under review by NDA/MOH. PHS also supported an independent study to develop user fee guidelines which are currently under review by the Governing Council of UMDPC. On approval by UMDPC, these user fee guidelines would be widely disseminated among all HCBs for adoption. But one KII involved in the survey noted that this change will take more time to come into effect, and support should be extended to working with Bureaus to help them encourage the uptake of these guidelines. The National Medicines Policy (2015) lays down guidelines for medicine imports and quality control but specific pricing for essential medicines have not been prescribed. A medicines price study (Medicine Price Components in Uganda) was conducted by Medicines Transparency Alliance (MeTA) in 71% 494% -41% 95% 3% 19% 251% IUD ANC HTS STD TB Sputum Malaria Diag Chest X￾ray Percentage diference in prices for services accross PHS HFs as compared to non-PHS HFs Figure 17: Variance in service prices at PHS HFs 31 October 2015, supported by MOH, WHO, HEPS and HAI Africa. This study shows that the retail mark-up in Uganda is as high as 115% for some pharmaceuticals. The challenge the evaluation team faced in specifically answering this question is the lack of any baseline and periodic program data on price regulation and standardization of services and commodities at the HFs. PHS did not track any data/document any information on the status of adoption of price regulation at the partner HFs and the effect of such regulation on prices of services and commodities at these HFs. Despite crucial studies on costing and pricing undertaken by PHS, there has not been a serious effort to regulate the exorbitant pharmaceutical pricing noted in Uganda. PHS only tracked the pricing of services and commodities at its partner facilities (HFs and pharmacies) annually and never compared any such pricing with that in the general market despite it being an important outcome under this Affordability IR. Similarly, there is no data on the status of standardization of services and commodities at the partner HFs at the beginning of the program and through the program period. Target for price regulation or number of services/commodities to be standardized have also not been defined under the program. So, despite the evaluation team having collected data on adoption of price regulation and adoption of standardization of services and commodities at the partner facilities, it is difficult to determine to what extent these have been established. In absence of such information to enable determination of the extent, the evaluation team analyzed the variability in prices of certain commodities and services between the partner and non-PHS facilities (HFs and pharmacies) to determine the adoption of price regulation. The assumption was that the facilities having adopted price regulation ought to ensure affordable medicine/service prices. Such affordability was defined by the program as a minimum 30% reduction in prices of services and commodities at the partner facilities as compared to non-PHS facilities. Additionally, as medicines and services prices/fees have not been defined, the extent of price regulation could only be determined by comparing prices between PHS and non-PHS facilities. According to PHS, the program only focused on 3 districts to lower prices (Jinja, Kamuli, and Mukono). But this is not reflected in the IR outcome and evaluation question posed which sought to explore affordability overall to the client. It should be noted that in these 3 districts PHS notes success in lowering prices, but the evaluation team found that these savings were not uniform countrywide. Thus, based on the above, the evaluation team concludes that PHS has not been able to operationalize the activities under this IR to achieve the desired price regulation outcomes and the extent of price regulation could not be determined. Standardization of services have been established but in absence of baseline and targets, the extent could not be ascertained. Sub-question 2.3: What is the relationship between newly established partnerships to changes in prices, sales or distribution of medicines? Have pharmaceutical prices and/or availability changed, and has this affected the consumer? What is the most important change and how has this made a difference in their lives? This evaluation sub-question aims to determine the following – 1) the relationship between established partnerships between the PHS HFs and pharmaceutical firms and if/how this resulted in changes in prices, sales, and/or distribution of medicines at these HFs; 2) if these changes in prices and availability of medicines occurred at the partner HFs, how have they affected the consumer (positively or negatively); and 3) the most important change noted by the consumer and how this made a difference in their lives. The evaluation team defined ‘differences in the lives of consumers’ as the change in the out-of-pocket (OOP) expenditure borne by the consumer. A sizeable number (64%) of the PHS HFs sampled had established partnerships with local pharmaceutical firms/franchises although it is not clear as to how many of these partnerships could be attributed to PHS, 32 as 63% of the non-PHS HFs visited also had established such partnerships. The HFs with partnerships did show a 55% increase in volume of sales/distribution of commodities and services, but this did not translate into reduced prices for the clients. Despite this, 60 % of the clients did express satisfaction to changes in availability and volume of commodities and services. We have discussed the strategy of establishing functional partnerships between PHS partner HFs and pharmaceutical firms under 2.1. PHS partner HFs and/or PHS did not track changes in volume of sales/distribution of services and commodities, so the evaluation of this particular change relies on primary data, which shows that 64% of PHS HFs that established pharma partnerships reported a 55% increase in sales/distribution. Availability and pricing of pharmaceuticals was not tracked by PHS partner HFs or PHS, hence it is not possible to quantify whether the change in sales/distribution has led to any change in prices of pharmaceuticals. The price variation at PHS HFs and pharmacies as compared to general market has been used by the evaluation team as a proxy for change in prices at the partner HFs. It was also assumed that a decrease in prices at partner HFs would positively affect the consumer at these HFs and vice versa. Thus, based on the price variation it seems that the consumer has been negatively affected at the partner HFs as the consumer had to pay higher prices compared to the general market, while the consumer seems to be positively affected at the partner pharmacies as the consumer had to pay less as compared to the general market. But it should be noted again that the evaluation team collected the highest and lowest product prices and used the average for analysis. This may have skewed the results if PHS HFs and pharmacies offered more products or higher quality (and thus more expensive) products. Further analysis should be done. Figure 18: Effect of partnerships on sales/prices The primary data shows that of the 36 PHS-supported HFs who established partnerships with pharma, 33 HFs (92%) stated that these partnerships led to changes in prices of commodities/medicines; of these 33, 32.1% 28.6% 26.8% 37.8% 32.1% 30.4% 28.6% 22.5% functional partnerships between facilities and pharma firms % of respondents reporting changes in availabiltiy and prices of services and commodities % of respondents reporting changes in sales/distribution of services and commodities % of consumers satisfied changes in prices and availability of services and commodities Effect of newly established facility partnerships to changes in prices of services and commodities PFP PNFP 33 16 were PFPs and 17 PNFPs. However, as specific prices were not recorded over time, this claim of change cannot be supported by data, but is rather a single point in time proxy measurement. Of the 275 clients interviewed across 56HFs, 161 (59%) informed the evaluation team of being satisfied with the changes in prices and availability – 101 were from PFP HFs and 60 from PNFP HFs. Of the 275 clients interviewed, 48 (18%) claimed to have paid less than what they would have paid at other HFs; of these 32 were from PFPs and 16 from PNFPs. 88 clients (32%) felt that they would have paid the same price for services at other HFs; of these 56 were from PFPs and 32 from PNFPs. A KII with a representative from the Joint Medical Stores (JMS) recorded a successful initiative to link JMS with private HFs, who they noted were previously not a part of their supply chain and related benefits. The PHS team is working with JMS to establish private HF associations that can pool their medicine and health supply needs, equating to greater economies of scale and lower prices and easier logistics for them. Private HFs were previously not ordering directly from JMS because of the small scale of their demands, but by forming these associations and pooling their orders, they are able to benefit from reduced pricing and easier logistics. This was noted and appreciated by JMS. The PHS team noted that partnerships between the partner HFs and the pharmaceutical firms did lead to increase in volumes of sales/distribution of commodities at the HFs, which resulted in reduction in prices of these commodities and reduced out-of-pocket expenditure for the consumer. However, PHS could not provide any data to substantiate this achievement. The interviewed clients did inform the evaluation team that the most important change for them was availability of medicines and some services that were not delivered before PHS. 5.4 IR3: How did the program result in improvements to quality of private health facilities and services? Overview of PHS Activities Under IR3 Under IR 3, PHS conducted several different activities over the life of the program. In Year 1, PHS worked with MOH to implement a public-private partnership for health (PPPH) policy to enable PFP providers to deliver quality services. PHS also worked to develop and roll out QIS tools during Years 2-4. This included working with a contracting firm to develop the initial tool, piloting the tool, trainings, and then rolling out and scaling up roll-out of the tool. PHS also worked to create a laboratory network that was better regulated and more accessible. Labnet offered consumers of all income levels a clearly identifiable place to obtain accurate diagnostic tests at an affordable price, with opportunities for standardizing quality, lowering prices, mitigating competition from unregulated informal market players, and overall better health outcomes for patients. Together with the Uganda Medical Laboratory Technology Association (UMLTA) and the professional councils, PHS supported the creation of Labnet, which is a network of trusted private sector laboratories operating under a franchise. This will ensure quality of services under this Network brand. Evaluation of PHS’ Progress Toward IR3 Improvements in the quality of private HFs and services has been defined under PHS as improved functioning of private health sector regulatory and supervisory systems, development and adoption of voluntary accreditation standards, clinic manuals, improved quality standards by a wide range of providers. Strengthening of the HMIS systems in the private health facilities was also aimed at data being used for 34 efficient decision-making. All these initiatives were supposed to have led to improved public perception of the private health sector. To a large extent, PHS has been able to improve quality of private health facilities/services as defined by the program. Professional councils have been strengthened and voluntary accreditation standards have been developed and adopted by 84% of HFs. Clinical standards and manuals were developed and 90% of the HFs have established SQIS. 86% of the HFs reported their service data to DHIS-2. The following training/capacity building initiatives undertaken by PHS helped improve quality across the private sector: ▪ Support to Marie Stopes to scale up training of all the HFs supported by them ▪ Capacity building of Uganda National Association of Community and Occupational Health (UNACOH) on Public Private Partnerships and financial management ▪ Training of 63 HFs in the use of the web-based ARV ordering system through JMS ▪ Training of community on rational medicine use (RMU) ▪ Mentoring and training the facility staff in ART record keeping and management Sub-question 3.1: How have public perceptions of the private health sector changed? And what evidence is there that the program has promoted this shift through enhanced the professionalism and technical capacity of providers? PHS aimed at improving the availability and quality of services in the private health sector in Uganda and an expected outcome towards the end of PHS was a favorable public perception of the private health sector. The evaluation team used client satisfaction assessed using Patient Exit Interviews (PEI) as a proxy for public perception, although a change in perception was not possible due to a lack of baseline or comparison data. 275 clients attending OPD at the 56 sampled PHS partner HFs were selected to administer service satisfaction related questions as a part of the Patient Exit Interview (PEI), and the vast majority (92%) were either satisfied or highly satisfied with the private health sector. The vast majority (92%) also perceived a positive change in the quality of services offered. PHS provided trainings to the partner HFs in the areas of business and financial management, clinical skills, and QI and SQIS. It was expected that these trainings would build/improve technical capacity and professionalism at these HFs, which would make public perception of these private HFs and the private sector as a whole, more favorable. PHS did impart needs based quality trainings in all the targeted areas but lack of a baseline satisfaction survey or a comparison group makes it difficult to attribute client satisfaction to these trainings. That said, during the patient exit interviews, clients were asked to rate various services they accessed at the HFs. Based on those responses, 253 (92%) gave a favorable rating to the private sector, with 127 (46%) being highly satisfied and 126 (45%) satisfied with the private sector￾provided health services; 255 (92%) said the quality of services at the HF has improved; 266 clients (96%) reported that they were handled sensitively by the different cadres of staff involved in patient management; 251 (91%) reported that they were satisfied with the medical care at the HFs; and 231 (84%) expressed that they were satisfied with the laboratory services. Table 4 below shows the various trainings provided by PHS through the 3 medical training institutes, and the percentage of the sampled HFs that had received these trainings. 35 Table 4: Number of Trainings by HF Type Training PFP PNFP Total Family Planning 28 (78%) 18 (90%) 46 (82%) ANC 18 (50%) 15 (75%) 33 (59%) PMTCT 28 (78%) 19 (95%) 47 (84%) HTS 30 (83%) 18 (90%) 48 (86%) HIV Care 29 (81%) 19 (95%) 48 (86%) ART 21 (58%) 18 (90%) 39 (70%) Ped HIV 17 (47%) 15 (75%) 32 (57%) CD4 15 (42%) 9 (45%) 24 (43%) Immunization 18 (50%) 16 (80%) 34 (61%) TB 19 (53%) 18 (90%) 37 (66%) Malaria 26 (72%) 17 (85% 43 (77%) Nutrition 13 (36%) 15 (75%) 28 (50%) SMC 17 (47%) 17 (85%) 34 (61%) Different stakeholders and PHS partners when interviewed did mention that PHS has been able to bring together the private sector and help establish crucial quality initiatives. The business and financial management trainings undertaken by PHS were also considered very innovative and successful by these partners, including the MOH. The HFs also expressed satisfaction at the quality and utility of the trainings provided by PHS and stated that these have helped them provide highly professional and better quality services to their clients. As stated above, the evaluation team also noted high levels of satisfaction to various services provided by the private sector and the evaluation team has determined that the trainings imparted by PHS have led to improved professionalism and technical capacity at the partner HFs and the clients are satisfied with the services at these partner HFs. Sub-question 3.2: Have private sector entities improved partnerships with professional councils and supervisory systems to enhance capacity to ensure professional conduct? As a part of improving private sector quality of services, PHS undertook the activity of strengthening/building capacity of the professional councils that represent various sections of the private health sector. The intent was also to strengthen partnerships within these councils and between the councils and their members from the private sector. The councils can then monitor the quality of services provided by their members without any external/donor support. The evaluation team found that the partnerships with professional councils have improved (98% of HFs reported), but regulatory supervision by the councils is still weak. More of the PHS-supported HFs than non-PHS HFs established relationships with councils, and 89% have partnerships with local DHMTs. 36 Major councils (UMDPC, NMC, and AHPC) have been provided with systems strengthening support by PHS. 98% of the sampled HFs (55 of 56) have established partnerships with professional councils, of these 35 are PFPs and 20 are PNFPs. Of the 27 control (non-PHS) HFs visited, 74% (20 HFs) have established partnerships with professional councils; of these 17 are PFPs and 3 PNFPs. Of the 56 HFs visited, 51 (91%) have been provided favorable rating by the clients – of these all 51 are registered with a council – of these 31 are PFPs and 20 PNFPs. 94% of the HFs reported having had formal supervision in 2017, of these 34 are PFP HFs and 19 PNFP; but only 63% of the HFs have these supervisory visits documented. 55% of pharmacies (5 of 9) have developed partnerships with the Uganda Pharmaceutical Council (UPC). More PHS HFs are reporting into DHIS-2 than non-PHS HFs. Of the 56 HFs visited, 48 HFs (86%) are reporting service data to DHIS-2; of these 30 are PFPs and 18 PNFPs. Of the 27 control (non-PHS) HFs visited, 20 (74%) are reporting to DHIS-2; of these 16 are PFPs and 4 PNFPs. Most of the supported HFs (89%) have also established partnerships with local District Health Management Teams (DHMTs). All the partners who worked with PHS in strengthening the councils – ASSIST, BTC, UIA, Marie Stopes, UNACOH, UHF, PSI PACE, CEDO, HEPS Uganda, NDA, Labnet, UMA and MOH – were unanimous in their praise for PHS in this activity of strengthening the councils and their supervisory systems. The councils also expressed satisfaction at PHS support in establishing accreditation standards and web-based platforms. The PHS team and the councils noted that the dissemination of the accreditation standards within each council members was planned and the delay in implementation was due to financial constraints. The data shows that private sector entities have improved partnerships with professional councils and supervisory systems. The professional councils did indicate that with PHS’s systems strengthening support, they were in a better position to supervise private health sector providers. The evaluation team thus concludes that standards and processes were strengthened at the professional councils with PHS support, and that partnerships between the private sector facilities and these councils were in the process of being strengthened. 3.3 Have acceptable voluntary accreditation standards been developed and adopted by institutions such as the Medical and Dental Council, Nursing and Midwifery Council, Allied Health Council and Pharmacy Council of Uganda? This activity, in line with the previous activity, aimed at strengthening the organizational bodies/councils representing various sections of the private sector in order to improve private sector service quality. 62.5% 63.0% 55.6% 85.7% 35.7% 11.1% 0.0% 0.0% PHS HFs non PHS HFs PHS Ph non-PHS Ph Over 50% of PFPs have establised functional partnerships with professional councils PFP PNFP Figure 19: Partnerships with professional councils 37 PHS has supported 3 of the major councils (UMDPC, UNMC, and AHPC) in developing and establishing acceptable voluntary accreditation standards and their adoption by the HFs has been good. All the 4 major councils have been supported with development of Continuing Professional Development (CPD) guidelines. The uptake of CPD guidelines has been weak with only 14% (8) of HFs adopting these – 6 PFPs and 2 PNFPs. Of the 56 HFs visited, 84% have been accredited, of these 30 are PFPs and 17 PNFPs. It is worth noting that 40 of these 47 HFs (85%) were accredited before the PHS program – of these 25 are PFPs and 15 PNFPs, indicating that the program did not lead to a large increase in accreditation. 5.5 Were there possible integrations of PHS Program Goals? PHS defined integration as an outcome of program implementation resulting in different program activities bolstering each other. Specific program activities targeted for integration under PHS were: 1) Improved self-regulatory functions and sector supervisory systems; 2) Enhanced professionalism and technical capacity, improved business relationships and increased affordability of services/commodities; and 3) Improvements in public satisfaction in private sector services. An adequate number of HFs have undergone professional and quality trainings and institutionalized quality initiatives. This has led to an improvement in the quality of services delivered and an increase in services/commodities being provided. Primary data shows high level of client satisfaction to specific and overall services but as baseline data on client satisfaction is not available, it is difficult to attribute this satisfaction to the trainings and subsequent quality of services. Moreover, this client satisfaction has not translated into any significant change in OP capacity as indicated by the available OP data. Thus, except lack of significant change in OP capacity, PHS has achieved integration of program goals as per the program definition. Looking at integration from a program implementation and management point of view, the evaluation team noted a lack of coordination and sharing of information between different program activities under PHS. Most of the activity areas under the program performed well but a synergy of implementation was largely not achieved. Absence of information sharing between the Access to Finance (A2F) and HSS teams led to both developing their own business strengthening trainings rather than doing it together. In addition, lack of indicator tracking (from program inception to end), lack of tracking of important data like prices at HFs and OP capacity, and lack of a management dashboard prevented efficient management and timely corrective actions in the program. From this perspective, integration of program goals was not achieved. The vast majority of PHS HFs have adopted SQIS. 89% of the HFs (50 of 56) have adopted SQIS and of these 84% (42 HFs) regularly undergo SQIS self-assessment, and at least 70% of the HFs (39) have undergone some form of quality/technical training. The evaluation team assessed the implementation processes and gaps were noted in integration of different program activities. HaaB trainings under DCA had no coordination with the business skills trainings under HSS. Similarly lack of information sharing/coordination were noted between other PFP and HSS initiatives. Absence of sharing of information also existed between the HSS and the Access to Finance (A2F) teams within PHS regarding business strengthening trainings to HFs. One positive example of integration has been the coordination between HaaB I and II trainings and QI/SQIS trainings. Sub question 4.1: What evidence is there that training provided in regard to professionalism and technical capacity influenced the public’s satisfaction with services? 38 Here the assumption was of a cause and effect relationship between the trainings imparted to the partner HFs and public satisfaction to services provided at these HFs. Though not formally defined, evidence of such cause and effect was expected as a part of program integration under PHS. Public perception of the private health sector is high, but it is difficult to link this perception directly to trainings. Although the assumption by the evaluation team is that the trainings undertaken by PHS and The evaluation findings show that quality of services at the HFs, professional conduct by the HF staff, sensitive handling of clients by the HF staff, and the overall private sector has been rated highly by the clients. Client satisfaction to various services is high, as seen in Figure 20 below. Primary data also shows that most of these HFs have undergone various technical, capacity building, business and financial management, and quality improvement trainings as detailed in Figure 18 above. It is, however, difficult to attribute the satisfaction levels to the trainings due to absence of any baseline data for comparison. Client satisfaction is an important indicator across all the 3 IRs but it was not tracked by the program, so satisfaction measured at the time of the evaluation was used as a proxy across the IRs. Figure 20: Client satisfaction to services 255 (93%) of the 275 clients interviewed reported that their respective HFs have shown improvement in quality of services over the past years; of these, 160 clients were from PFP HFs and 95 from PNFPs. 266 (97%) of the 275 clients interviewed felt that they were handled sensitively at their respective HFs; of these, 169 were from PFP HFs and 97 from PNFPs. Overall, public perception of the private health sector is very high. Of the 275 clients interviewed, 253 (92%) provided favorable rating to the private sector (with 127 (46%) being highly satisfied and 126 (45%) satisfied with the private sector). As a sector-wise breakdown, the PFPs have 162 clients (90 satisfied and 72 highly satisfied) giving favorable rating to the private sector and with PNFPs 91 clients (36 satisfied and 55 highly satisfied) gave the private sector a favorable rating. Under IR3, we have noted all the trainings provided by PHS to its partner HFs. The sampled HFs and various PHS stakeholders interviewed did state that they perceived the trainings imparted by PHS have helped the partner HFs deliver better quality of services. 52% 72% 86% 90% 91% 91% 95% 95% Radiology experience Billing experience Pharmacy experience Administrative services Laboratory experience Registration experience Nursing care Medical care There is high client satisfaction for most services provided by PHS health facilities 39 Sub-question 4.2: Have improvements in self-regulatory functions and policy helped to improve professionalism, or lead to supporting greater capacity in the sector? Figure 21: Quality standards/policy at PHS HF The implementation of SQIS has been a success. 94% of HFs (48 of 56 visited) have implemented SQIS and 82% (42 of 48) of HFs have undergone at least one SQIS self-assessment in the past year. Also 92% of HFs have established QI committees in their HFs. In addition, 84% of HFs are reporting to DHIS-2. SQIS initiative was systematically implemented and managed by the responsible PHS team. Adequate data was regularly collected, analyzed and discussed with the HFs by the PHS team. The HFs were also supported in undertaking regular SQIS self-assessments. All these led to sustainable implementation of SQIS at the partner HFs. Two different program integration goals have been expected from the program under this question. One is that improvements in self-regulatory functions and policy have helped improve professionalism. The evaluation team understood ‘Professionalism’ as the ability of the HFs to deliver services as per prescribed in national/international clinical standards upholding the highest quality and ethical standards and sensitive handling of the clients. The second part examines whether improvements in self-regulatory functions and policy have supported greater capacity in the sector. The evaluation team defined ‘capacity’ as the number of Out-Patients (OP) catered to by the HFs, assuming that improvements in self-regulatory functions and policy at the HFs would have led to an increase in the number of OP being catered to by these HFs. With respect to self-regulatory functions and policy, it has been noted that of the 56 HFs visited, 50 (84%) have established QI/SQIS, of these 33 are PFP HFs and 17 are PNFP HFs. 42 (84%) of these 50 HFs have undergone at least one SQIS self-assessment in the past year. From the 27 non-PHS HFs visited, 16 (59%) (15 PFPs/1 PNFP) have established some form of QI systems. Of these 15 are PFPs and 1 PNFP. So, PHS support has helped the majority of the partner HFs establish SQIS and undertake regular SQIS self￾assessment. The Uganda Healthcare Federation (UHF) was supported by PHS through a grant and through technical assistance to establish SQIS protocols. A representative noted during a KII that this initiative has been a ‘game changer,’ and stated that UHF conducted their own survey that showed that customer satisfaction 63.0% 37.0% 77.8% 66.7% 48.1% 18.5% 7.4% 18.5% 18.5% 7.4% Accredited Quality manual Registered with GOU Functional HMIS HMIS # A good proportions PHS HF have standards in place PFP PNFP 40 was linked to SQIS implementation. 10 One challenge of the SQIS initiative though was reaching rural HFs, as most of their efforts focused on HFs within Kampala. With regard to professionalism, the evaluation team tried to determine professionalism from the perspective of the provider, beneficiary, and PHS stakeholders. Interviews with the partner HFs showed that all of the HFs who have undergone various clinical, business and financial management trainings expressed that the trainings have helped them deliver client-centered services complying with the highest quality and ethical standards. More than 90% of the clients interviewed expressed satisfaction at the quality and professionalism of services delivered at the HFs. All PHS program stakeholders and partners also stated that the trainings provided by PHS did help the partner HFs deliver good quality services, with sensitive client handling, and ethical consideration. With regards to the effect of SQIS at HFs on the OP capacity at the HFs, all the 42 HFs with SQIS together show a 9% increase in OP from Year 1 to Year 5. Of the 42 HFs with SQIS, 26 PFP HFs with established SQIS show a 10% increase overall in OP from Y1 to Y5, whereas 16 PNFPs with QI/SQIS show an overall OP increase of 2% increase from Y1 to Y5. Disaggregating the OP trend based on gender, 26 PFPs show 29% increase in male clients while 3% increase in female clients from Y1 to Y5 and 16 PNFPs show an 8% increase in male clients and a 0.8% increase in female clients from Y1 to Y5. All the 42 HFs with SQIS in total show a 42% increase in male clients and a 4% increase in female clients from Y1 to Y5. The evaluation team concludes that SQIS improved greatly amongst the PHS-supported HFs, and although it is difficult to attribute the effect of this on professionalism, it seems as though there is a positive link between the efforts of PHS in this area and improvements in perception of professionalism. Sub-question 4.3: Is there evidence that improved professionalism and technical capacity contributed to changes in public satisfaction with services? Under this question, three elements have been considered – first is improved professionalism (Evaluation team’s understanding of professionalism has been defined under 4.2); second is improved technical capacity (considered here as ability to provide standardized clinical services); and third is changes in public satisfaction with services. As a part of program integration, it was anticipated that the first and second would lead to the third. 3 of the major councils (UMDPC, UNMC, and AHPC) were provided support in strengthening their systems and establishing voluntary accreditation systems and CPD guidelines. PHS commissioned business and financial management, various technical/clinical capacity building, and professional conduct improvement trainings through several training agencies. During HF visits and interactions with the HF staff and clients it was seen that the staff were happy with the trainings and 92% of clients expressed a favorable opinion of the private sector and were satisfied with the quality of services provided. 96% of clients expressed that they were handled sensitively by various HF staff. Professionalism was discussed above in more detail in Section 4.2. As for technical capacity, the evaluation team relied on interviews with the partner HFs, PHS team and PHS partners and program stakeholders. The HFs interviewed stated that various clinical skill trainings provided by PHS helped them deliver medical services as per established national and international standards, thereby improving their technical capacity. The PHS team noted that trainings were imparted based on needs assessments and have led to improved technical capacity at the partner HFs. PHS stakeholders also confirmed that the trainings imparted by PHS on clinical skills have enabled the partner HFs in delivering quality services due to improved technical 10 This document was not shared and therefore this could not be verified 41 capacity. This qualitative evidence shows a perceived improvement in technical capacity among all vested stakeholders. With respect to public satisfaction to services, as mentioned under preceding questions, the evaluation team has considered client (population accessing services at the partner HFs) satisfaction as a proxy for public satisfaction. From the data collected through PEIs with 275 clients across 56 HFs, there is a significant satisfaction to various clinical and non-clinical services among the clientele. Of the 275 clients interviewed across 56 HFs, 242 (88%) were satisfied with registration services, 214 (78%) with administrative services, 185 (67%) with billing services, 252 (92%) with nursing care, 251 (91%) with medical care, 231 (84%) with laboratory services, 73 (27%) with radiology services, and 222 (81%) were satisfied with pharmacy services. It should be noted that the low satisfaction levels seen are due to the fact that all the services were not available at all the HFs. Although improvement in public satisfaction is difficult to determine, satisfaction at this point and time is very high. The evaluation team also posed two additional evaluation questions for 5) Health Systems Strengthening (HSS), and 6) Orphans and other Vulnerable Children (OVC) activities that weren’t included in the original scope. These activities have only been implemented for the past 9 months so conclusions and lessons learned cannot be confidently drawn, but primary and secondary data were reviewed to assess their implementation so far. Further analysis of these two areas can be found in Annex 10. 42 6.0 CONCLUSIONS The PHS program had many successes, and also some shortcomings. When tasked to evaluate the program against its original results framework and its progress in achieving its IRs, it is noted that contract modifications that changed the original scope and direction of the program and expanded PHS hindered the program in achieving its larger, original intermediate results. The service delivery targets were clear and were mostly achieved as found in quarterly and annual reports, but the program was unable to realign the new, added priorities to the IRs and to modify the theory of change or to clearly map out how the new initiatives contributed to it. The expansion of PHS was large and followed differing funding streams, signifying that PHS was tasked to expand their activities based on the availability of earmarked funds, rather than based on their programmatic goals (as is seen by the selected indicators and program data found in Annex 12). This is evident in their quarterly and annual reports where some indicators are only tracked for short periods of time and then dropped, and not throughout the life of the project, and several indicators start and stop in the middle of the project. Though this is not uncommon in projects, especially those that receive PEPFAR funds, it also reflects competing interests and pressure to report on indicators that were not part of the original mandate. The additional modifications seemed to have been added without a clear framework of how all the components worked together to make a cohesive program or how the different activities contributed to the overarching IRs. This lack of lack of continuity tasked the PHS team with balancing immediate needs and keeping the overarching goal in sight, sometimes to the detriment of the larger objectives as seen in this report; and did not allow for long-term planning. Though in service delivery and in reaching their annual targets, PHS did very well. The most successful initiatives under the PHS program were capacity building initiatives that led to local ownership and in reaching their service delivery targets. PHS’s support of organizations to strengthen 3 of the main councils was overwhelmingly recorded as a success during interviews. The councils were strengthened to develop voluntary accreditation systems, and a major success of PHS was in building the capacity of the councils to play a more prominent role in supervising HFs. PHS’s support to UHF was also successful in that UHF helped to develop SQIS guidance, and SQIS systems were adopted by the vast majority of the sampled HFs which may have played a role in the vast majority of clients expressing a positive opinion of the private health sector. Another success was that prices at PHS pharmacies were lower than the general market, showing a successful mechanism probably due to more stringent regulations by the NDA in the pharmaceutical sector. Successful initiatives under the PHS program came as a result of joint ownership between PHS and partners, and in areas where PHS was able to work closely with partners to implement activities designed towards their result areas. This is why the SQIS and business trainings were successful- because PHS was able to work closely with partners to design and implement these initiatives. The PHS program fell short in lowering the prices of drugs, commodities, and services for the consumer, which was a major goal of the program. Limitations in the methodology, such as using the average price of available tracer commodities, are described under IR1 above. Average prices for the majority of sampled drugs and commodities were higher at PHS-supported HFs, and services were also found to be higher for all but 1 of the sampled services. This means that policy level price regulations and work undertaken by PHS with the MoH and other government entities did not trickle down to actual savings for the client, signifying continuing high out-of-pocket expenditure due to healthcare. The lack of mandatory regulations or oversight meant that HFs were able to determine their own costing structure, and either due to lack of competition or higher overhead costs, the goal to reduce prices for drugs and commodities at the PHS￾supported HFs was not successful. The DCA mechanism was also a mixture of successful and unsuccessful elements. Centenary Bank was very successful in its use of DCA and increasing lending to the health sector. But Ecobank fell short overall because of the high collateral requirements that were not adjusted for smaller health care businesses, hindering loan allocations to less established, rural facilities and 43 businesses. But it should be noted that Centenary Bank was much more successful than Ecobank, and that these two entities should be observed separately. The DCA mechanism did nothing to change or relax the banks’ policies regarding high interest rates and high collateral demands, which still prevented smaller HCBs from accessing these loans. The incentives for the large banks chosen weren’t high enough for them to take on the risk and extra work of lending to very small HCBs. The evaluation team found that lack of coordination between the components was a contributing factor to the program’s mixed results. Several parts of the program relied on each other in the TOC, such as the assumption that increased access to finance through loans and improved business practices through trainings would lead to expanded availability of health services. This would require the program to implement interventions simultaneously and deliberately in order to achieve the larger outcome. In contrast, there was a lack of coordination amongst the PHS team which led to an inability of these separate activities to leverage one another to achieve a higher result. The program successfully contributed to the service delivery indicators outlined in their quarterly and annual reports, but was not able to use CLA principles to adaptively manage how each of the individual components complemented each other and contributed toward the overarching results. A major lesson that should be learned from this program is that programs should be adapted throughout the life of the program to ensure that the results framework is modified as new components are added, and any changes or modifications to the program have to be accompanied by revisiting the theory of change and mapping out how and where new activities fit into the larger objectives. 44 7.0 RECOMMENDATIONS Based on the findings presented in this report, the evaluation team has several recommendations: • USAID/Uganda should ensure that modifications and changes in scope are undertaken in a manner that does not fundamentally alter the theory of change (TOC) of the program. If the modification/changes in the scope do alter the TOC, then the program should revise the TOC including the results framework (result areas) and indicators to respond to these changes. • Implementing Partners (IPs) should periodically pause and reflect on whether their activities and the results being achieved are in sync with their TOC and the program objectives as set out in the task order, including program modifications. • IPs should undertake investigations (learning) during implementation intended to establish impediments to achieving results and objectives with intend to support adaptive management and decision-making, and not wait for mid and endline evaluations. • A CLA plan should be developed at the beginning of the program and hiring of a CLA specialist should be prioritized to ensure continuous application and integration of CLA approaches. • Access to financing for the private health sector requires a phased approach- PHS should first prepare smaller HCBs to better situate themselves as candidates for loans, through interventions such as HaaB trainings, business skills development, and financial management, before linking the HCBs to the banks. Similarly, on the banking side, the IP should prepare partner banks in the functioning of the DCA mechanism and offer greater incentives for working with infant business entities. This approach should also be given a realistic timeframe, as these initiatives would take longer than the initial span of the program. • IPs should initiate other modalities to incentivize HFs to deliver comprehensive health services and to add additional services to clients, following the cancellation of use of “performance based grants” by COP 15 guidance for Program engagement with HIV care and treatment services in the private sector. IPs need to adaptively manage changing strategic priorities in the private health sector. 45 Annex 1: PHS Scope of Work STATEMENT OF WORK FOR EVALUATING USAID’S PHS PROGRAM A. Background The USAID/Uganda Private Health Support (PHS) Program is USAID’s flagship program in the private sector in Uganda, built on the successes of USAID’s Health Initiatives for the Private Sector (HIPS) Project. The purpose of this program is to contribute towards establishing a viable, cost-effective private sector option for health services in Uganda by: 1) Improving the credibility and cohesiveness of the private health sector – Addressing issues within the private sector that limit its effectiveness as a whole. For example, mixed impressions about the quality of private sector services can have a negative impact on its credibility. Similarly, the lack of cohesiveness among private providers undermines their ability to leverage the sector’s vital role in health service delivery. 2) Improving the competency and expanding the capacity of private sector providers – Enhancing professional competence and the ability of individual providers to meet the demand for quality health services in Uganda while improving availability and accessibility of health services. Essential to enhanced professionalism is the concept of accredited private providers. Accreditation will help clients make more informed health care choices and increase confidence in the quality of care provided. The USAID/Uganda PHS Program uses a variety of approaches and vehicles to leverage and strengthen local organizations to ensure sustainability. Specifically, the program aims to strengthen, organize and mobilize the private sector to provide Ugandans with the option of obtaining high-quality health services from Private for Profit (PFP) providers. PHS supported USAID/Uganda’s 2011- 2015 Country Development Cooperation Strategy (CDCS), in particular, Development Objective 3 (DO3): Improved health and nutritional status in focus areas and population group. Three of the four sub-results under Intermediate Results (IR) 3.1 (IR3.12, IR3.1.3, and IR3.1.4) aligned well with leveraging and expanding the existing capacity of the private sector. The focus of support is to provide technical expertise, enhance quality standards, improve access to capital, support accreditation and provide leadership in the private sector. To achieve this, the Program has three primary objectives: > Expanded availability of health services by private providers; > Increased affordability of private health services and products; and > Improved quality of private health sector facilities and services Purpose The purpose of this evaluation is to comprehensively examine USAID/Uganda Private Health Support Program (PHS) to determine extent to which expected results were realized, and what program related factors may have facilitated or hindered the implementation of the project and/or achievement of results. Audience The primary audience will be USAID/Uganda, USAID Global Health managers, PEPFAR managers, the Implementing Partner, the GOU and other stake holders interested in health systems functioning. 46 Application and Use The findings of this evaluation are expected to contribute to USAID decision making on the strengths of this program and its ability to strengthen the private sector health system in Uganda. The evaluation provides USAID with opportunity to identify gaps, opportunities to learn and strong practices that should be adopted and utilized. PHS Results Framework The following summarizes the anticipated intermediate results and the expected outcomes for each IR that the program was expected to achieve under DO3. IR 1: The anticipated results of IR 1 are; expanded availability of private health services through building the capacity of providers to improve geographical coverage; expand service offerings; and access to capital through providing financial technical assistance to improve business relationships; and utilization of the new financing mechanisms. Building on previous work under the HIPS program, business skills training will have assisted private providers in developing essential business and expansion plans with financial programs. The program will have also provide technical assistance to financial institutions, including monitoring utilization of DCA funds by banks, and maintaining a database on usage by participating banks. Additionally, the contractor should also have provided technical assistance with loan guarantee documents, reviewing loan applications for compliance with loan guarantee facility terms, generating demand for DCA facility, and monitoring loan portfolio quality. The PHS program will have also offered technical assistance to banks to help develop financial products suitable to the health sector, and to address banks’ perception of risk in the health sector and other areas identified. IR2 Under IR2, the project will have improved the affordability (i.e., financial access) of private health services and products by reducing market cost inefficiencies and by leveraging private sector resources through public-private partnerships. If private health sector facilities can access discounted rates on health commodities, they will be able to lower the price of service and products. The contractor is expected to have provided continued technical assistance to at least 40 company clinics that have yet to graduate and function independently, offering comprehensive services. Additionally, they should have provided continued support to all 100 clinics, and approximately 50 more PFP clinics assigned to USG under the ART rationalization process, helping them to broker access to affordable quality HIV/AIDS and TB services and products such for employees, their dependents, and surrounding communities. An emphasis on addressing women’s health issues should be evident. IR3: By providing continuous assistance to enhance technical skills for service provision; strengthen supervision and monitoring practices; improve accreditation and regulation of facilities and providers, and fostering a policy-enabling environment for private health sector the program will have enhanced the quality of private health sector facilities and services. Although more than 50 percent of Ugandans use private health care, the private sector has in the past suffered from serious credibility issues related to the quality of services. Similarly, historically there has been a lack of collaboration among the private health sector providers, which undermines its ability to understand, leverage, and advocate for its role in the delivery of health care in Uganda. Thus, the program will have supported regulatory private sector entities by building strong partnerships with the professional councils to enhance their capacity to ensure professional conduct and adherence to a code of ethics among private clinics. 47 IR 1 Outcome IR2 Outcomes IR3 Outcomes • Growth in number of private facilities offering comprehensive health services • Expanded private sector capacity, e.g. increased total outpatient capacity • Increased demand for and access to credit by private providers, e.g., increased number of commercial loans issued to private health providers • Improved business management practices in private facilities • Improved private health sector participation in district service delivery coordination • Reduction in the unit sale price of essential health medicines and service packages in program supported areas, as compared to the general market. (Minimum target: 30% reduction) • Improved price regulation/ standardization of commodities and services • Functional partnership agreements established with private sector pharmaceutical firms/franchises • Increased sales/distributions volume of essential commodities and services from private sector facilities • Percentage increase in partner company clinics that “graduate” to function independently based on current model • Improved public perceptions of the health private sector as a viable service provider • Improved functioning of private health sector regulatory and supervisory systems (annual milestones driving towards sustainability) • Acceptable voluntary accreditation standards adopted by a wide range of providers • Adoption of clinic manuals defining functions and procedures • Improved quality standards regulation • HMIS systems in private sector in place and data utilized for reporting and decision making 48 Evaluation Questions The evaluation questions seek to identify evidence of systems strengthening and capacity building that enabled increased provision of services, greater affordability, improved care, and improved public perception of private health sector services, where PHS was implemented. Although, there is no baseline available addressing these precise questions, combining various reports, and inquiries into perception, might help inform the questions having to do with change that resulted from the project. IR1: Expanded Availability of Health Services by Private Providers: 1. Assess to what extent this project improved access to financial resources for the partner facilities 2. Is there evidence showing that without the DCA support, banks would still be able to lend private health sector clients? 3. Did business skills training employ successful and unique approaches or innovations that can be scaled up? 4. What changes have there been in comprehensive service availability and provider outpatient capacity? IR2: Increased Affordability of Private Health Services and Products: 1. How have functional partnerships been established between health facilities and pharmaceutical firms? 2. Is there a relationship between newly established partnerships to changes in prices, sales or distribution of medicines? What have been the benefits to the consumers? 3. Have pharmaceutical prices and/or availability changed, and has this affected the consumer? What is the most important change and how has this made a difference in their lives? 4. To what extent have price regulation/standardization of commodities and services been established? IR3: Improvements to Quality of Private Health Sector Facilities and Services: 1. How have public perceptions of the private health sector changed? And what evidence is there that the program has promoted this shift through enhanced the professionalism and technical capacity of providers? 2. Have private sector entities improved partnerships with professional councils and supervisory systems to enhance capacity to ensure professional conduct? 3. Have acceptable voluntary accreditation standards been developed and adopted by institutions such as the Medical and Dental Council, Nursing and Midwifery Council, Allied Health Council and Pharmacy Council of Uganda? Examining Integrations of PHS Program Goals Where it is possible collected data should be used to examine program integrations. While these questions are optional, the issue of integration is of great interest to USAID/Uganda and analyses of areas of integration are strongly encouraged. Below are some suggested areas of interest, see Figure 1. a. What evidence is there that training provided in regard to professionalism and technical capacity influenced the public’s satisfaction with services? b. Have improvements in self-regulatory functions and policy help to improve professionalism, or lead to supporting greater capacity in the sector? 49 c. Is there evidence that improved professionalism and technical capacity contributed to changes in public satisfaction with services? Figure 1. Program Integration Methods General Comments related to Methods: The Evaluation Team will collect and analyze both primary and secondary data to answering the evaluation questions. The evaluation should use a mix method approach, collecting both quantitative and qualitative data, suggested sources and methods include: quarterly monitoring reports, patient surveys assessing perceived quality of care or changes in quality of care, focus group discussions, and key informant interviews with those knowledgeable of IPs performance, as well as relevant GOU members. Evaluation team will design or identify all data collection instruments and submit to QED and USAID for review prior to beginning fieldwork. The following are suggested methods that should be considered in designing the evaluation: (1) review of relevant program related documents, (2) in-depth interviews of key informants and/or focus groups, (3) a survey of public perception of private health sector, in select areas, assessing reported changes in attitude among the public and (4) direct observation, assessment and review of PHS and national health information systems. Document Data and Review This desk review will be used to provide background information on the project/program, and will also provide data for analysis for this evaluation. Documents and data to be reviewed include technical reports, annual and quarterly reports, etc. The results of the analysis of these reports should contribute to lessons learned from PHS’s implementation during the period covered by the evaluation. • PHS contract • PHS Annual Workplans • PHS Annual Reports • Miscellaneous PHS reports • National HIV/AIDS M&E Plan • Uganda DHIS-2 data and/or reports • Uganda MERS and SIMS data that will be accessed from DATIM (will secure access rights for team – view only) • Uganda Demographic Health Survey • Access to HIBRID all PEPFAR Indicators over the course of PHS • Ministry of Health Policy documents • Other docs and/or data reports 50 Key Informant Interviews Using a semi-structured questionnaire comprising primarily of open ended questions, the evaluation team will conduct in-depth interviews with: • GOU MOH • Private Public Partnership Office (MOH) • Implementing Partners (IPs), including Uganda Health Supply Chain project and a sample of comprehensive implementing partners • Participants from financial institutions and partner banks (Centary Bank, DFCU, Eco Bank) • Stakeholder/participants involved in private health sector regulatory and supervisory systems. • Private clinics and pharmacies • Other stakeholders The evaluation team will propose the data collection tools and work closely with SI and PHS the implementing partner to identify appropriate respondents. These interviews are expected to provide insights into how PHS has performed and expectations of how GoU can best support the private health sector going forward. Focus Group Discussions Optional: Focus group discussions are optional, and up to the discretion of the contractor. However, the USAID team feels that they are useful. Survey Optional: One form of data collection that may be useful is a survey conducted among patients to try to gain an understanding of changes in patient satisfaction and perception of services, as related to probable areas of impact of the PHS program. A mixed methods survey is recommended. This would allow for the quantification of perceived changes in satisfaction, while open ended questions would provide contextual information. Observations Optional: Observational data may be used to enhance context, but should not be a primary source of information. Human Subjects The Evaluation Team must develop protocols to insure privacy and confidentiality prior to any data collection. Primary data collection must include a consent process that contains the purpose of the evaluation, the risk and benefits to the respondents and community, the right to refuse to answer any question, and the right to refuse participation in the evaluation at any time without consequences. Only adults can consent as part of this evaluation. Minors cannot be respondents to any interview or survey, and cannot participate in a focus group discussion without going through an IRB. The only time minors can be observed as part of this evaluation is as part of a large community-wide public event, when they are part of family and community attendance. During the process of this evaluation, if data are abstracted from existing documents that include unique identifiers, data can only be abstracted without this identifying information. Analytic Plan USAID expects this evaluation to involve data analysis to support findings and conclusions; hence a detailed data analysis plan is requested as part of the Team’s work plan for both qualitative and quantitative data. The evaluation will review both qualitative and quantitative data related to the project/program’s achievements against its objectives and/or targets. Appropriate statistical analyses 51 should be identified in the work plan. Quantitative data will be analyzed primarily using descriptive statistics. Thematic review of qualitative data will be performed, connecting the data to the evaluation questions, seeking relationships, context, interpretation, nuances and homogeneity and outliers to better explain what is happening and the perception of those involved. Qualitative data will also be used to substantiate quantitative findings, provide more insights than quantitative data can provide, and answer questions where other data do not exist. All data analyses and presentations of key data findings should also address important data disaggregation /categorization relevant to performance of PHS, and analyses will be geared to answer the evaluation questions. Data will be stratified by demographic characteristics, such as sex, age, and location, whenever feasible. Use of multiple methods that are quantitative and qualitative, as well as existing data (e.g., project/program performance indicator data, DHS, HMIS data, etc.) will allow the Team to triangulate findings to produce more robust evaluation results. The Evaluation Report contain a Methods section will describing data collection, analytic methods and statistical tests employed in this evaluation. Deliverables 1) Inception report showing the evaluation design, a detailed evaluation plan with timelines and data collection tools. A filled evaluation design matrix following the template attached as Annex 1 should be included in the Inception Report. The report should also provide an overview of the methodology that will be used to select areas to be visited and respondents/participants. 2) Oral Presentation: Power Point presentation (including hand-outs) to: i. USAID alone ii. USAID and PHS project iii. USAID, PHS project and other selected stakeholders. The main presentation will normally take 45 minutes covering the major findings, conclusions and lessons learned, allowing for approximately 45 minutes of discussion and feedback. 3) First Draft Evaluation Report: The first draft should be at most 20-25 pages excluding annexes. The content should cover all the main elements of the report including major findings, conclusions, lessons learned, and relevant annexes. The input from the oral presentation sessions should also be incorporated in the report. 4) Final Draft Evaluation Report: A complete report presented in the agreed-upon format and incorporating comments from USAID and other stakeholders; including annexes. 5) Cleaned labeled and ready to use electronic copies of datasets collected through fieldwork and cleaned ready to use electronic copies of KI or FGD analyses if any. In addition, copies of all instruments used in data collection must be separately delivered to USAID. 6) Final Report: The team leader will submit a final report within one week of receiving final comments from USAID including those from other stakeholders. The core report should be 20-25 pages excluding annexes Team Composition General Team Characteristics • Key staff should have methodological and/or technical expertise, regional or country experience, language skills, team lead experience and management skills, etc. 52 • Team leaders for evaluations/analytics must be an external expert with appropriate skills and experience. • Additional team members can include research assistants, enumerators, translators, logisticians, etc. • Teams should include a collective mix of appropriate methodological and subject matter expertise. • Evaluations require an Evaluation Specialist, who should have evaluation methodological expertise needed for this activity. • All team members will be required to provide a signed statement attesting that they have no conflict of interest, or describing the conflict of interest if applicable. • All team members sign a non-disclosure agreement as part of their QED contract. Team Lead This person will be selected from among the key staff, and will meet the requirements of both this and the other position. The team lead should have significant experience conducting project evaluations/analytics. Roles & Responsibilities: The team leader will be responsible for (1) providing team leadership; (2) managing the team’s activities, (3) ensuring that all deliverables are met in a timely manner, (4) serving as a liaison between the USAID and the evaluation/analytic team, and (5) leading briefings and presentations. Suggested Qualifications: • A minimum of a Master’s Degree in health management or public health. • Minimum of 10 years of experience in public health, which included experience in implementation of health activities in developing countries • Demonstrated experience leading health sector project/program evaluation/analytics, utilizing both quantitative and qualitative methods • Had a significant role in a minimum of ten evaluations/assessments, preferable covering health, IT and M&E activities/programs • Led a minimum of five evaluation/assessment • Demonstrated knowledge of data needs for PEPFAR reporting • Excellent skills in planning, facilitation, and consensus building • Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders • Excellent skills in project management • Excellent organizational skills and ability to keep to a timeline • Good oral communication and writing skills, with extensive report writing experience • Experience working in the region, and experience in Uganda is desirable • Familiarity with USAID • Familiarity with USAID and PEPFAR policies and practices  Evaluation policy & standards of practice  Results frameworks  Performance monitoring plans Program Strategic Information Specialist Roles & Responsibilities: Serve as a member of the evaluation team, providing expertise in HIV SI systems and processes, particularly related to PEPFAR and national routine information systems. S/He will participate in planning and briefing meetings, data collection, data analysis, development of evaluation presentations, and writing of the Evaluation Report. Will be responsible for mining and analyzing data from DATIM and DHIS2 53 Suggested Qualifications: • Master’s degree or higher in Epidemiology, Biostatistics or related field • At least 10 years’ experience working in M&E of HIV/AIDS prevention and treatment programs • Demonstrated excellent knowledge of indicators and data requirements for all PEPFAR Technical Areas and a good understanding of service cascades. • Familiar with DHIS-2, DATIM and similar national routine heath information systems is desirable • Experience working with projects and at the local and national levels on reporting processes and data flow, and use of data • Excellent interpersonal skills, including experience successfully interacting with host government officials, civil society partners, and other stakeholders • Proficient in English • Good writing skills, specifically technical and evaluation report writing experience • Experience in conducting USAID evaluations of health programs/activities is desirable Evaluation Specialist Roles & Responsibilities: Serve as a member of the evaluation team, providing quality assurance on evaluation issues, including methods, development of data collection instruments, protocols for data collection, data management and data analysis. S/He will oversee the training of all engaged in data collection, insuring highest level of reliability and validity of data being collected. S/He is the lead analyst, responsible for all data analysis, and will coordinate the analysis of all data, assuring all quantitative and qualitative data analyses are done to meet the needs for this evaluation. S/He will participate in all aspects of the evaluation, from planning, data collection, data analysis to report writing. Suggested Qualifications: • An advanced degree in public health, evaluation or research or related field • At least 8 years of experience in USAID M&E procedures and implementation • At least 5 years managing M&E, including evaluations • Experience in the design and implementation of evaluations. Experience in public health or health sector evaluations preferred. • Strong knowledge, skills, and experience in qualitative and quantitative evaluation tools • Experience implementing and coordinating other to implements surveys, key informant interviews, focus groups, observations and other evaluation methods that assure reliability and validity of the data. • Familiarity with USAID and PEPFAR health programs/projects, particularly PEPFAR M&E is preferred • Familiarity with USAID and PEPFAR M&E policies and practices o Evaluation policies and standards of practice o PEPFAR indicators o Performance monitoring plans o DHIS-2, DATIM and similar national routine heath information systems • Experience in data management • Able to analyze quantitative, which will be primarily descriptive statistics • Able to analyze qualitative data • Experience using analytic software • Demonstrated experience using qualitative evaluation methodologies, and triangulating with quantitative data • Able to review, interpret and reanalyze as needed existing data pertinent to the evaluation 54 • Strong data interpretation and presentation skills • Proficient in English • Good writing skills, including extensive report writing experience Private Health Sector Advisor This individual will serve as an advisor to the evaluation team by using firsthand knowledge of the private health sector in Uganda to facilitate meaningful and useful data collection to inform the performance evaluation questions. S/he will work closely with the evaluation team providing insight into topics such as key informant and survey questions. S/he will also help to ensure that the evaluation team identifies the proper opportunities for collecting data to inform the evaluation. Preferred qualifications include Public Health training and experience working in the private health sector. Duration The evaluation team will have 6 months to complete the evaluation. The allotted time will begin once the contract is signed. Evaluation Criteria 1. Technical Approach (50%) Sub criteria in order of importance • Extent to which the proposed technical approach is clear, logical, well-conceived, technically sound and accurately interprets the evaluation questions provided in this Statement of Work • Extent to which the proposed technical approach demonstrates an understanding of the implementation context 2. Key Personnel (50%) • Extent to which the proposed key personnel meet the required qualifications demonstrating the Offeror’s ability to conduct the evaluation QED Roles and Responsibilities QED will coordinate and manage the evaluation/analytic team and provide quality assurance oversight, including: • Review SOW and recommend revisions as needed • Provide technical assistance on methodology, as needed • Develop budget for analytic activity • Recruit and hire the evaluation/analytic team, with USAID POC approval • Request for country clearance and/or facility access (if needed) • Review methods, work plan, analytic instruments, reports and other deliverables as part of the quality assurance oversight • Data entry, analysis and interpretation • Report production - If the report is public, then coordination of draft and finalization steps, editing/formatting, 508ing required in addition to and submission to the DEC and posting on QED website. If the report is internal, then copy editing/formatting for internal distribution. USAID Roles and Responsibilities Below is the standard list of USAID’s roles and responsibilities. Add other roles and responsibilities as appropriate. 55 56 Annex 2: PHS SWOT Analysis Strengths Weaknesses ▪ Strong team and good management system ▪ Understanding of the health system in Uganda ▪ Inclusive consultation through staff meetings ▪ Good HR management ▪ Advocacy with the MBs ▪ Change management processes for MBs ▪ Responsiveness/compliance to Work Plans ▪ Timely reporting from PHS ▪ USAID support for innovative ideas ▪ Lack of strong coordination within PHS ▪ Lack of documentation and feedback loops in supervision ▪ IRCU merger led to some staff dissatisfaction ▪ HSS with MBs not based on baselines ▪ Poor quality reporting from PFPs ▪ Variance within USAID regarding PHS scope ▪ Weak documentation Opportunities Threats ▪ Receptive PFPs ▪ PNFPs receptive to donor-funded activities ▪ Good learning and knowledge base for future PFP involvement ▪ Good local technical resources generated ▪ MBs able to identify HSS gaps within their HFs ▪ Social capital built through HSS and OVC ▪ Getting PFPs on board ▪ Micro plan for transition not initiated ▪ Capacity of other IPs to take over activities ▪ Inadequate financial management by the PNFPs ▪ Resistance of the MBs to change ▪ Dependency of OVC CSOs on PHS funding ▪ OVCs/families-caregivers not graduating to independence as envisaged ▪ Staff turnover at HFs 57 Annex 3: Lessons Learned from the Evaluation Following are some salient lessons learned from this program to inform similar future initiatives: ▪ The biggest operational challenge faced by the HFs was the lack of flexible financing, so any new project needs to incorporate elements of regulating/ensuring low interest rates, low mortgage, and long tenure loans ▪ Another operational challenge was the inability of clients to pay, therefore to strengthen quality health services a strong community health insurance scheme should be encouraged ▪ Banks need to develop loan products tailored for the health sector based on a needs assessment; equipment, construction, buying land for expansion, operational costs etc. ▪ Banks need to track loans given sector-wise for sector specific purposes; also, region- and HF- level wise ▪ Ecobank did not have necessary rural presence in Uganda which affected its loan portfolio performance; rural outreach of the bank should be an important criteria while selecting a DCA bank ▪ The program should have a logframe and a management dashboard collating crucial information on a monthly/quarterly basis for effective decision-making ▪ Integration of program goals has to be formalized by developing specific outcomes and performance indicators ▪ CLA needs to be incorporated as an activity within the program from the beginning ▪ Collaboration between all the stakeholders – internal and external – is critical for the success ▪ Persistent advocacy with and support to MOH/GOU is important to maintain their private sector focus ▪ Collaboration with MOH/DHMTs should be prioritized for sustainability ▪ A program activity entailing support to medical/nursing/pharmacy schools on establishing business and financial management curriculum should be included in any private sector program ▪ Roll out of QI activities in the PFP and PNFP sector should to be sustained ▪ Documentation is crucial and an analytical approach to data management is required ▪ Training/capacity building on supply chain management of health commodities and pharmaceuticals should be a part of private sector strengthening ▪ Any initiatives with PFPs, PNFPs and MBs should be based on individual assessments and baselines ▪ For sustainable HSS, it is pertinent that MBs are involved in activities to be implemented within their HF network ▪ Encouraging and disseminating the concept of OVC HHs graduating from donor dependence to independence from the start of the program ▪ Priority to be given to economic empowerment of OVCs and OVC households through business management trainings and providing start-up kits support 58 Annex 4: Evaluation Design Matrix No . Evaluation question No. Evaluation Sub-question No. Measure / Indicator Data Source Sampling Data Collectio n Instrume nt Data Analysis 1 IR1: How did program activities help to expand the availability of health services by private providers? 1.1 Assess to what extent this project improved access to financial resources (e.g., loans) for the partner facilities. 1.1.1 # and % of PHS facilities that applied for loans from DCA banks. Primary data; Key informant interviews (KIIs) with; 1. PHS-supported and 2. non-PHS supported private health facilities 3. Banks that were part of the DCA and 4. A sample of other banks NOT part of the DCA. 1. Sample of PHS￾supported facilities 2. Sample of non￾PHS facilities 3. All banks that were part of the DCA 4. A sample of banks that were not part of the DCA. 5. Cross-sectional survey; comparing "intervention" facilities and banks that were part of the PHS program, with "control" facilities and banks were NOT part of the PHS. 1. Interview guide for PHS￾supporte d facilities 2. Interview guide for non-PHS facilities 3. Interview guide for banks. 1. Quantitative data to be analyzed using appropriate statistical methods and software 2. Qualitative data to be thematically analyzed, also using appropriate analytical tools. 1.1.2 # and % of PHS facilities that obtained loans from DCA banks. 1.1.3 # and % of PHS facilities that obtained loans from non-DCA banks. 1.1.4 # and % of PHS facilities that were trained/counselled on financial management and business strengthening since 2013. 1.1.5 # and % of non-PHS facilities that obtained loans from non-DCA banks 1.1.6 # and % of non-PHS facilities that obtained loans through the DCA banks. 1.2 What is the evidence showing that without the DCA support, 1.2.1 Business bank loans given to PHS facilities as % of total business loans given by selected banks in Uganda since 2013. 1.2.2 Business bank loans given to non-PHS facilities as % of total business loans 59 banks would still be able to lend private health sector clients? given by selected banks in Uganda since 2013. 1.2.3 # and % increase in non-DCA loans given by the DCA banks to private health sector since 2013. 1.2.4 # and % of loans given by non-DCA banks to private health sector since 2013 1.3 Did business skills training employ successful and unique approaches or innovations that can be scaled up? What were they? 1.3.1 Scaled up innovative/successful skills training approaches as a proportion of total number of skills training approaches used for training partner facilities. 1. Secondary data; Document/Rep orts review 2. Primary data; KIIs with PHPs with training entities, with banks and with PHS staff. 1. Sample of PHS￾supported facilities 2. Sample of non￾PHS facilities 3. All training entities 4. Key PHS staff involved in the BDS aspects. 5. Cross-sectional survey; comparing "intervention" facilities and banks that were part of the PHS program, with "control" facilities and banks that were NOT part of the PHS program. 1. Interview guide for PHS￾supporte d facilities. 2. Interview guide for non PHS facilities. 3. Interview guide for trainers; Interview guide for PHS staff. 1. Qualitative data to be thematically analyzed, and quantitative (if applicable) to be analyzed using appropriate analytical tools. 1.3.2 # and % of PHS facilities trained in HaaB I; # and % of PHS-supported facilities trained in HaaB II. 1.3.3 # and % of non-PHS facilities trained in HaaB I; # and % of non-PHS facilities trained in HaaB II. 1.3.4 # and % of PHS facilities that received more than 5 business counseling/technical assistance sessions after HaaB training. 1.3.5 # of non-PHS facilities that received more than 5 business counseling/technical assistance sessions after HaaB training. 1.3.6 # and % of PHS facilities that defaulted on loan repayment. 1.3.7 # of non PHS facilities that defaulted on loan repayment. 1.3.8 # and % of trained (in HaaB I/HaaB II) PHS facilities that implemented new 60 financial and management systems after the training. 1.3.9 # of non-PHS facilities that implemented new financial and management systems after HaaB training. 1.4 What changes have there been in comprehensi ve service availability and provider outpatient capacity? 1.4.1 Changes in comprehensive service availability aggregated for each subsector group facilities (PNFP, PFP). 1. Secondary data: Quantitative (observational trend data): i) PHS reports; ii) DHIS-2 and HMIS data; iii) PEPFAR DO3 data, iv) OP data from all facilities 2. Primary data: Health facility survey; i) Observations at PHS￾supported facilities and non-PHS facilities (possibly for some indicators), ii) Key informant interviews with heads of PHS￾supported 1. All PHS facilities for secondary data review 2. Sampled PHS facilities for primary data collection 3. Sample of non￾PHS facilities 4. All Heads of regulatory councils/bureaus 5. Relevant official from MOH; 6. Cross-sectional survey; comparing "intervention" facilities that were part of the PHS program, with "control" facilities that were NOT part of the PHS program. 1. Section 5 from the Health Facility Assessm ent Instrume nt. 2. Interview guides for other key stakehold ers. 1. Trend analysis for key indicators by year, age, gender, etc.; observational study using secondary data 2. Segregate and analyze findings per private sector group (PNFP, PFP, etc.) Both quantitative and qualitative analytical tools and methods. 1.4.2 Changes in OP capacity aggregated for each subsector group facilities (PNFP, PFP). 1.4.3 Changes in OP capacity in the facilities showing changes in comprehensive service availability. 1.4.4 # and % of PHS facilities with comprehensive services 1.4.5 # and % of visited non-PHS facilities providing comprehensive service package? 61 facilities and non-PHS facilities; iii) Key informant interviews with other key stakeholders. 2 IR2: How was increased affordabilit y of private health services and products achieved by the implement ing partner? 2.1 How have functional partnerships been established between health facilities and pharmaceutic al firms? 2.1.1 # and type of partnerships between health facilities and pharmaceutical firms. 1. Secondary data: i) PHS reports, ii) Partnership documents 2) Primary data: KIIs with i) PHS, ii) non￾PHS facilities, iii) pharma firms, iv) MOH (dist and national) and v) PHS staff. 1. Sample of PHS facilities with partnerships. 2. Sample of non￾PHS facilities with partnerships. 3. All concerned PHS staff and relevant MOH staff. 1. KII Tool. 2. Health Facility Assessm ent instrume nt. 1. Content analysis. 2. Analyze findings per private sector group (PNFP, PFP, etc.). 2.1.2 # of non-PHS facilities having functional partnerships with pharmaceutical firms. 2.1.3 Criteria and standards developed for selecting facilities and pharmaceutical firms to establish partnerships between the two. 2.1.4 # of such partnerships with formal MoU/contract. 2.1.5 System of regular meetings and sharing of information (regular reports) between the two partners. 2.1.6 Successes and failures have been documented to inform implementation strategy. 2.2 To what extent have price regulation / standardizati on of commodities and services 2.2.1 PHS support to GOU-MOH to establish/modify normative standards for price regulation/standardization of commodities and services. 1. Secondary data: i) PHS reports, ii) Financial records from facilities (if allowed access to) 1. Secondary data from all PHS facilities. 2. Sample of PHS facilities with partnerships. 3. KIIs with 1. KII Tool. 1. Cross￾sectional data analysis. 2. Trend analysis (wherever costing data available from facilities). 2.2.2 # of PHS facilities having adopted these standards for price regulation/standardization of commodities and services. 62 been established? 2.2.3 # of non-PHS facilities having adopted these standards for price regulation/standardization of commodities and services. 2. Primary data: KIIs with i) facilities, ii) National Drug Regulatory Authority, iii) MOH (dist and national) and iv) PHS staff. relevant MOH officials 3. Analysis per private sector group (PNFP, PFP, etc.). 2.2.4 Min 30% reduction in prices of services and commodities in PHS facilities as compared to market rates - at the time of assessment. 2.2.5 Prices recommended for essential medicines for the private sector. 2.2.6 # of facilities having adopted PHS developed professional fee guidelines. 2.2.7 # of facilities involved in pooled procurement. 2.2.8 # of pooled procurements undertaken 2.2.9 # of facilities included in the maternal health voucher program 2.3 What is the relationship between newly established partnerships to changes in prices, sales or distribution of medicines? Have pharmaceutic al prices 2.3.1 # of functional partnerships between PHS facilities and pharma firms/franchises. 1. Secondary data: i) PHS reports, ii) Financial records from facilities (if allowed access to) 2. Primary data: KIIs with i) facilities, ii) National Drug Regulatory Authority, iii) 1. All PHS facilities for secondary data 2. Sample of PHS facilities with partnerships 1. KII Tools. 2. Patient Exit Interview (PEI) Tool. 1. Cross￾sectional data analysis. 2. Trend analysis (wherever costing data available from facilities). 3. Analysis per private sector group (PNFP, PFP, etc.). 4. Causal inference 2.3.2 Changes in availability and prices of services and commodities in PHS facilities with partnerships and those without (comparative). 2.3.3 Changes in sales/distribution of services and commodities for PHS facilities with partnerships and changes in prices to those without. 2.3.4 Consumer satisfaction to the above changes in prices and availability of services and commodities. 63 and/or availability changed, and has this affected the consumer? What is the most important change and how has this made a difference in their lives? 2.3.5 Increase in OP capacity of PHS facilities with above changes and those without. MOH (dist and national), iv) PHS staff, and v) beneficiaries. analysis between partnerships and change in prices and increase in sales . 2.3.6 Reduction in OOP for beneficiaries. 2.3.7 Change in well -being (consumer perspective). 3 IR3: How did program activities result in improvem ents to quality of private health sector facilities and services? 3.1 How have public perceptions of the private health sector changed? And what evidence is there that the program has promoted this shift through enhanced the professionalis m and technical capacity of providers? 3.1.1 # and % of patients having favorable opinion of the private sector. 1. Secondary data: i) PHS reports and data on training, ii) OP capacity reports 2. Primary data: i) Health facility assessment tool, ii) Patient Exit Interview (PEI) tool. 1. Sample of PHS facilities (PFPs, PNFPs, CBOs) 2. Sample of Pharmacies and drug stores. 3. 5 patients from each facility (every 2nd /3rd patient exiting the OP). 1. Patient Exit Interview (PEI) Tool. 2. Health facility assessme nt tool. 1. Cross - sectional analysis of the primary data. 2. Trend analysis - contingent upon availability of baseline data. 3. Analysis per private sector sub -group. 4. Causal inference between trainings, quality improvement and public satisfaction. 3.1.2 # and % of patients rating private sector as having better quality services in the past yr. 3.1.3 # and % of patients noting that they were handled sensitively by the different cadres of staff involved in patient management. 3.1.4 Of the facilities having received favorable ratings from the patients, how many have undergone professional training? 3.1.5 Of the facilities having received favorable ratings from the patients, how many have undergone HaaB I and HaaB II trainings? 3.1.6 Of the facilities having received favorable ratings from the patients, how many have implemented SQIS? 64 3.1.7 Of the facilities having received favorable ratings from the patients, how many have undergone SQIS facility self-assessment? 3.2 Have private sector entities improved partnerships with professional councils and supervisory systems to enhance capacity to ensure professional conduct? 3.2.1 # and % of professional councils strengthened with PHS support 1. Secondary data: i) PHS reports and data on partnerships 2. Primary data: KIIs with i) Professional councils, ii) relevant MOH officials, iii) relevant PHS staff 3. Primary data: Health facility assessment tool 1. Sample of PHS facilities (PFPs, PNFPs, CBOs) 2. Sample of Pharmacies, drug stores. 3. All pharmaceutical firms 4. All relevant MOH and PHS staff. 1. Health facility assessme nt survey. 2. KII Tool. 1. Analysis per private sector group (PNFP, PFP, etc.) 2. Causal inference analysis between partnerships with professional councils and supervisory systems and enhanced professional conduct capacity. 3.2.2 # and % of PHS facilities that have established functional partnerships with professional councils (GoU￾MoH, UMDPC, UNMC, AHPC), by each year in the program. 3.2.3 # and % of non-PHS facilities that have established functional partnerships with professional councils (GoU-MoH, UMDPC, UNMC, AHPC), by each year in the program. 3.2.4 Public perception of enhanced professional conduct between PHS facilities having registration with a council and those without. 3.2.5 # and % of PHS facilities undergoing formal supervision (nature, frequency of supervision, feedback loop system) in 2017. 3.2.6 Public perception of enhanced professional conduct between PHS facilities undergoing supervision and those without. 3.2.7 # and % of facilities reporting to the DHIS-2 system (directly and through PHS). 3.2.8 # of facilities having partnerships with local DHMTs. 65 3.3 Have acceptable voluntary accreditation standards been developed and adopted by institutions such as the Medical and Dental Council, Nursing and Midwifery Council, Allied Health Council and Pharmacy Council of Uganda? 3.3.1 # of acceptable voluntary accreditation standards developed and institutionalized under PHS 1. Secondary data: i) PHS reports 2. Primary data: KIIs with i) Key office bearers from all the councils, ii) relevant MOH official, iii) relevant PHS staff 1. All the councils. 1. All the key stakeholders. 1. KII Tool for all the identified stakehold ers. 1. Content analysis. 3.3.2 # of councils having established web - based platforms. 3.3.3 # of councils having developed and reviewed their CPD guidelines. 3.3.4 Barriers and facilitators for developing and adopting standards. 4 Analysis of Possible Integratio ns of PHS Program Goals. 4.1 What evidence is there that training provided in regard to professionalis m and technical capacity influenced the public’s satisfaction 4.1.1 # of PHS facilities having undergone professionalism and technical capacity training. 1. Secondary data: i) PHS reports/data on trainings, ii) Baseline on public satisfaction. 2. Primary data: PEI tool. 1. Sample of PHS facilities (PFPs, PNFPs, CBOs) having undergone training. 2. Sample of facilities (PFPs, PNFPs, CBOs) without training. 3. 5 patients from each facility (every 2nd /3rd patient exiting the OP). 1. PEI Tool. 1. Causal inference analysis 2. If there is any baseline a pre - post can be done. 3. If not, a cross -sectional assessment, by using PEI tool; comparing the findings between 4.1.2 Patient satisfaction with provider 'professionalism' (will be difficult defining 'professionalism' from a layperson perspective) and services in the PHS facilities having undergone training and in those without training. 66 with services? facilities having undergone training and those that haven't. 4.2 Have improvement s in self￾regulatory functions and policy help to improve professionalis m, or lead to supporting greater capacity in the sector? 4.2.1 What self-regulatory function and policies (QI, SQIS) have been developed? 1. Secondary data: i) PHS reports/data on SQIS and QI initiatives and policies, ii) Comprehensiv e service package availability data for PHS and non-PHS (control) facilities 2. Primary data: i) Health facility assessment tool, ii) KIIs with key stakeholders from the private sector and PHS. 1. All relevant PHS staff 2. All relevant key stakeholders 3. Sample PHS facilities 4. Sample of non￾PHS facilities 1. KIIs with PHs and key stakehold ers 2. Health facility assessme nt tool. 1. Causal inference analysis 2. If there is any baseline a pre￾post can be done. 3. If not, a cross-sectional assessment, by using health facility assessment tool. 4.2.2 # and % of non-PHS facilities with self-regulatory activities undertaken since 2013. 4.2.3 # and % PHS facilities that have undergone SQIS self-assessment. 4.2.4 Have improvements in self-regulatory functions and policy helped increase OP capacity in the PHS facilities? 67 4.3 Is there evidence that improved professionalis m and technical capacity contributed to changes in public satisfaction with services? 4.3.1 Similar to 4.1.2 in this section above Similar to 1 (row 15) in this section above Similar to 1 (row 15) in this section above Similar to 1 (row 15) in this section above Similar to 1 (row 15) in this section above 5 HSS: Has the implement ation of health systems strengthen ing (HSS) within the faith￾based health sub-sector led to sustained scale up of services for people living with and 5.1 What is the evidence that leadership and governance systems in the faith￾based PNFP sub-sector has been strengthened ? 5.1.1 # and % of Bureaus, Hospital Boards and Health Unit Management committees trained in corporate governance since 2016. 1. Secondary data: i) PHS reports, ii) Baseline data on pre-PHS availability of services for PLHAs with the PNFPs. 2. Primary data: i) Health Facility Assessment tool, ii) KIIs with Medical Bureaus, iii) KIIs with relevant PHS staff. 1. Key staff from all medical bureaus. 2. All relevant PHS staff. 3. Sample of PNFP facilities (from all bureaus) 4. Sample of non￾PHS PNFP facilities. 1. Health Facility Assessm ent tool. 2. KII with i) Medical Bureau staff, ii) relevant PHS staff. 1. Trend analysis of the secondary data (validity contingent upon availability of baseline and data through the years of implementation. 2. Analysis of the cross￾sectional primary data. 5.1.2 # and % of Medical Bureaus, Hospitals and Health Centers supported with development of Board Governance Manuals since 2016. 5.1.3 # and % of hospital managers trained in strategic planning since 2016 5.1.4 # and % of monitoring and evaluation missions with the 4 medical bureaus since 2016. 5.1.5 # and % of hospitals and health facilities supported by development of Customer Relations Management (CRM) guidelines since 2016. 5.1.6 # and % of bureaus and hospital staff trained in CRM TOTs since 2016. 68 affected by HIV/AIDS 5.1.7 # and % of Bureau Boards and Management trained in advocacy and lobbying since 2016. 5.1.8 # and % of medical bureau staff trained to improve their skills in Operational Research since 2016. 5.1.9 # and % of medical bureau staff trained in DHIS2 to enable them support health facilities in reporting. 5.1.10 # and % of information officers and records assistants trained in health data management to support health facilities adapt information systems such as HRIS, DHIS with the national systems. 5.2 Have the Human Resources for Health (HRH) strengthened in the faith￾based PNFP sub-sector? 5.2.1 # and % of Medical Bureaus, Hospitals and Health Centers supported with development of HR Manuals. 1. Secondary data: i) PHS reports. 2. Primary data: i) Health Facility Assessment tool, ii) KIIs with key staff from all medical bureaus, and relevant PHS staff. 1. Key staff from all medical bureaus. 2. All relevant PHS staff. 3. Sample of PNFP facilities (from all bureaus) 4. Sample of non￾PHS PNFP facilities. 1. Health Facility Assessm ent tool. 2. KII with i) Medical Bureau staff, ii) relevant PHS staff. 1. Trend analysis of the secondary data (validity contingent upon availability of baseline and data through the years of implementation. 2. Analysis of the cross￾sectional primary data. 5.2.2 # and % of workshops conducted for supervisors and health unit managers on the use of performance management tools since 2016 . 5.2.3 # and % of medical bureau managers and health facility managers trained in Work Force Indicator of Staff Needs since 2016. 5.2.4 # and % of medical bureau, hospital and health facility board members trained in Strategic Human Resources Management (SHRM) since 2016. 5.2.5 # and % of medical bureau and health facility staff trained in iHRIS system of the MoH since 2016. 69 5.3 Have the health financing systems strengthened in the faith￾based PNFP sub-sector? 5.3.1 # and % of Medical Bureaus, Hospitals and Health Centers supported with development of Finance Manuals since 2016. 1. Secondary data: i) PHS reports. 2. Primary data: i) Health Facility Assessment tool, ii) KIIs with key staff from all medical bureaus, and relevant PHS staff. 1. Key staff from all medical bureaus. 2. All relevant PHS staff. 3. Sample of PNFP facilities (from all bureaus) 4. Sample of non￾PHS PNFP facilities. 1. Health Facility Assessm ent tool. 2. KII with i) Medical Bureau staff, ii) relevant PHS staff. 1. Trend analysis of the secondary data (validity contingent upon availability of baseline and data through the years of implementation. 2. Analysis of the cross￾sectional primary data. 5.3.2 # and % of medical bureaus and health facilities and staff trained in business management and entrepreneurship since 2016. 5.3.3 # and % of target facilities' technical staff trained in internal audit systems and processes since 2016. 5.3.4 # and % of target health facilities supported in procurement of accounting software and the staff trained in its use since 2016. 5.3.5 # and % of medical bureau board members, medical bureau technical staff and hospital staff trained in Performance Based Financing Mechanism (PBFM) since 2016. 5.3.6 # of trainings conducted on grants application/and proposal development skills for hospital managers and boards since 2016. 5.3.7 # and % of hospital managers and hospital board finance sub-committee members trained in innovative strategies for fundraising and local resource mobilization. 5.3.8 # of orientation meetings conducted for health facility management teams in the guidelines for pricing of select common disease conditions since 2016. 70 6 OVC: How has the PHS performed in strengthen ing family - based approach to care and support to OVCs using sub - granting mechanis m 6.1 What has been the contribution to 90 -90 -90 strategy? 6.1.1 # of OVC grantees. 1. Secondary data: i) PHS reports. 2. Primary data: i) CSO assessment/inte rview tool. 1. All CSOs for secondary data. 2. Sample of OVC CSOs . 3. Sample of non - OVC CSOs. 4. All relevant PHS staff. 1. CSO assessme nt/intervi ew tool. 1. Content and trend analysis of secondary data, comparing with baseline data, if available. 2. Analysis of the cross - sectional primary data, to validate/inform above analysis. 6.1.2 # of partnerships between OVC grantees and health facilities in respective communities to Support HIV testing for OVC and caregivers (outreaches and home based HTS), referral to care and treatment services, follow up support for adherence/viral load suppression 6.1.3 # of HIV+ve OVCs provided transportation to access ART. 6.1.4 # of community volunteers, VHTs and grantee staff involved in counselling (also on hygiene, sanitation, malaria and TB prevention) and adherence support. 6.2 How has PHS strengthened the socio - economic capacity of families to meet basic needs of life? 6.2.1 # of caregivers trained in Financial Literacy, Business startup and management. 1. Secondary data: i) PHS reports. 2. Primary data: i) CSO assessment/inte rview tool, ii) OVC family interview tool, iii) KIIs with relevant PHS staff. 1. All CSOs for secondary data. 2. Sample of OVC CSOs. 3. Sample of non - OVC CSOs. 4. 5 OVC caregivers/families within the vicinity of the CSO site, from each CSO list. 5. All relevant PHS staff. 1. CSO assessme nt/intervi ew tool. 2. OVC caregiver /family interview tool. 1. Content and trend analysis of secondary data, comparing with baseline data, if available. 2. Analysis of the cross - sectional primary data, to validate/inform above analysis. 6.2.2 # of Village Saving and Loans Associations (VSLAs) established and trained. 6.2.3 # of OVC Households and VSLAs linked with micro -finance support services and Government poverty alleviation/wealth creation programs such as CDD, Youth Livelihoods Program. 6.2.4 # of out of school OVCs trained in different vocational skills courses such as tailoring, hair dressing. 6.2.5 # of out of school OVCs provided with startup kits to enable them to begin small scale business enterprises. 71 6.2.6 # of artisan trained in the use of Directorate of industrial Training (DIT) apprenticeship manuals to improve pedagogical, testing and assessment skills. 6.2.7 # of out of school OVCs trained in business skills to enable them manage their businesses enterprises better. 6.2.8 # of community volunteers/ VHTs trained to conduct routine nutrition assessment and referral malnourished children to Health facilities for further management. 6.2.9 # of caregivers trained to improve household food and nutrition security. 6.3 How has PHS ensured safety, protection and psychosocial wellbeing of all OVCs? 6.3.1 # of Grantee staff linked with child protection structures (Probation Officer, CDOs, Police, Child protection committees etc) in respective districts to create awareness about child abuse/violence against children. 1. Secondary data: i) PHS reports. 2. Primary data: i) CSO assessment/inte rview tool, ii) OVC family interview tool, iii) KIIs with relevant PHS staff. 1. All CSOs for secondary data. 2. Sample of OVC CSOs. 3. Sample of non￾OVC CSOs. 4. 5 OVC caregivers/families within the vicinity of the CSO site, from each CSO list. 5. All relevant PHS staff. 1. CSO assessme nt/intervi ew tool. 2. OVC family interview tool. 1. Content and trend analysis of secondary data, comparing with baseline data, if available. 2. Analysis of the cross￾sectional primary data, to validate/inform above analysis. 6.3.2 # of child abuse cases referred for handling/management. 6.3.3 # of teachers trained in providing psychosocial support for children infected and affected by HIV/AIDS in school and community settings. 6.3.4 # of OVCs having undergone birth registration to enable them access public essential services like education. 72 6.3.5 # of children with disabilities (CWDs) categorized for their disabilities. 6.3.6 # of grantees conducting Centre days for children recreation activities, counselling and life skills development. 6.4 How has PHS ensured OVC schooling? 6.4.1 # of OVCs from critically vulnerable families provided with scholastic materials to ensure retention and completion of critical levels of education 1. Secondary data: i) PHS reports. 2. Primary data: i) CSO assessment/inte rview tool, ii) OVC family interview tool, iii) KIIs with relevant PHS staff. 1. All CSOs for secondary data. 2. Sample of OVC CSOs. 3. Sample of non￾OVC CSOs. 4. 5 OVC caregivers/families within the vicinity of the CSO site, from each CSO list. 5. All relevant PHS staff. 1. CSO assessme nt/intervi ew tool. 2. OVC family interview tool. 1. Content and trend analysis of secondary data, comparing with baseline data, if available. 2. Analysis of the cross￾sectional primary data, to validate/inform above analysis. 6.4.2 # of schools fees top ups payments made for OVCs, especially for adolescent girls in candidate classes and those transitioning from primary to secondary education 6.4.3 # of school visits conducted to check on academic progress of OVC in schools and providing career guidance and PSS support where necessary 6.4.4 # of peer school clubs formed 6.4.5 # of adolescent girls provided with sanitary pads 6.4.6 # of adolescent girls trained on how to locally make re-usable sanitary towels 6.5 How was capacity building of OVC 6.5.1 # of private sector engaged to mobilize matching funds for OVC activity implementation 1. Secondary data: i) PHS reports. 2. Primary data: i) CSO 1. All CSOs for secondary data. 2. Sample of OVC CSOs. 3. Sample of non￾1. CSO assessme nt/intervi ew tool. 1. Content and trend analysis of secondary data, comparing with baseline data, if 6.5.2 # of OVCs supported through the above mobilization 73 grantees undertaken? 6.5.3 # of technical grantee staff review meetings for a assessing progress, planning, sharing of updates and promising practices assessment/inte rview tool, ii) KIIs with relevant PHS staff. OVC CSOs. 4. All relevant PHS staff. available. 2. Analysis of the cross￾sectional primary data, to validate/inform above analysis. 74 Annex 5: Detailed Evaluation Methodology Evaluation purpose The purpose of this evaluation is to comprehensively examine USAID/Uganda’s Private Health Support (PHS) Program to determine the extent to which expected results were realized, and what program related factors may have facilitated or hindered the implementation of the project and/or achievement of results. Evaluation questions The evaluation questions seek to identify evidence of systems strengthening and capacity building that enabled the programs objectives of increased provision of services, greater affordability, improved care, and improved public perception of private health sector services, where PHS was implemented. Although there is no baseline available addressing these precise questions, combining various reports with inquiries into perception might help inform the questions having to do with change that resulted from the project. A. IR1: How did program activities help to expand the availability of health services by private providers? a. Assess to what extent this project improved access to financial resources (e.g., loans) for the partner facilities b. What is the evidence showing that without the DCA support, banks would still be able to lend private health sector clients? c. Did business skills training employ successful and unique approaches or innovations that can be scaled up? What were they? d. What changes have there been in comprehensive service availability and provider outpatient capacity? B. IR2: How was increased affordability of private health services and products achieved by the implementing partner? a. How have functional partnerships been established between health facilities and pharmaceutical firms? b. To what extent have price regulation/standardization of commodities and services been established? c. What is the relationship between newly established partnerships to changes in prices, sales or distribution of medicines? Have pharmaceutical prices and/or availability changed, and has this affected the consumer? What is the most important change and how has this made a difference in their lives? C. IR3: How did program activities result in improvements to quality of private health sector facilities and services: a. How have public perceptions of the private health sector changed? And what evidence is there that the program has promoted this shift through enhanced the professionalism and technical capacity of providers? b. Have private sector entities improved partnerships with professional councils and supervisory systems to enhance capacity to ensure professional conduct? c. Have acceptable voluntary accreditation standards been developed and adopted by institutions such as the Medical and Dental Council, Nursing and Midwifery Council, Allied Health Council and Pharmacy Council of Uganda? D. Were there possible integrations of PHS Program Goals? The issue of integration is of great interest to USAID/Uganda, thus examining interactions of the various program components is relevant to the Mission. Analyses of the areas outlined below are designed to help USAID understand how the different pieces of the program may or may not have bolstered one another. Below are the areas of interest, see Figure 2 above; a. What evidence is there that training provided in regard to professionalism and technical capacity influenced the public’s satisfaction with services? 75 b. Have improvements in self-regulatory functions and policy help to improve professionalism, or lead to supporting greater capacity in the sector? c. Is there evidence that improved professionalism and technical capacity contributed to changes in public satisfaction with services? In addition to the above evaluation questions provided by the USAID team, the evaluation team, based on its initial review of the PHS program documents, identified the following two evaluation questions. E. Health Systems Strengthening (HSS): Has the implementation of health systems strengthening (HSS) within the faith-based health sub-sector led to sustained scale up of services for people living with and affected by HIV/AIDS? a. What is the evidence that leadership and governance systems in the faith-based PNFP sub-sector has been strengthened? b. Have the Human Resources for Health (HRH) strengthened in the faith-based PNFP sub-sector? c. Have the health financing systems strengthened in the faith-based PNFP sub-sector? F. Orphans and other Vulnerable Children (OVC): How has the PHS performed in strengthening family-based approach to care and support to OVCs using sub-granting mechanism? a. What has been the contribution to 90-90-90 strategy? b. How has PHS strengthened the socio-economic capacity of families to meet basic needs of life? c. How has PHS ensured safety, protection and psychosocial wellbeing of all OVCs? d. How has PHS ensured OVC schooling? e. How was capacity building of OVC grantees undertaken? Indicators Quantitative and qualitative indicators have been identified for each of the proposed sub-questions. PHS did not have a results based / logical framework and so a set of activities informing the program outcomes were requested from PHS. The indicators have been based on these activities to ensure that no major component of the PHS program is overlooked in this evaluation. The evaluation has focused particular attention on the development of indicators for processes, outputs, and outcomes. The indicators were selected on the basis of their neutrality and measurability. Particular attention was also paid to choosing accessible, useful, unbiased and cost-effective indicators. Evaluation methodology The intent of this evaluation is to understand how PHS has been able to leverage USAID resources at its disposal to improve availability, affordability and quality of healthcare services in the private sector in the target areas. This evaluation will be evidence-based and conducted using both qualitative and quantitative methods. The aim is to identify the strengths, weaknesses, opportunities and threats in the PHS program. Through the evaluation questions and corresponding indicators, the evaluation shall determine what worked and why, and what didn’t work and why in the PHS program. The evaluation team understands that the findings from this evaluation will inform future USAID strategy on working with the private sector in Uganda. Quantitative methods Quantitative methods will be used to determine the outputs/outcomes and ascertain causality between interventions and expected outputs/outcomes associated with the PHS program. Quasi-experimental approach: Comparison (Control vs. Intervention / Before and After analysis) The evaluation will consider a quasi-experimental, cross-sectional approach, which involves reviewing and/or collecting data on specific aspects of the program at given points in time (baseline, mid-term and end-term), with the view to determine the outputs/outcomes of the program. The evaluation will rely on existing secondary information from reports for baseline and mid-term 76 evaluations, where such reports exist. For this end-term evaluation, primary data collection will be conducted and analyzed, and all relevant information from secondary sources will be taken into consideration. Noting that the PHS program has not been implemented in a vacuum, there is need to determine whether the observed changes in the PHS-supported facilities are really a result of the PHS support. Therefore, we propose to collect relevant data from non-PHS supported private facilities (considered to be control sites that have not received PHS support), with the view to make comparisons between PHS site and non-PHS supported sites. The proposed quasi-experimental approach will be triangulated through comparisons that follow the before-and-after analysis. In this approach, comparisons within the intervention group alone will help in examining results on the outcome of interest prior to and after the intervention in the sites of interest. If end-term results differ significantly from baseline information, a case can be made that PHS support was the cause of the change in the intervention sites. Intervention sites: these will include a sample of health facilities or other stakeholders that received (financial and non-financial) support from the PHS program. A detailed description of the sample and sampling approach are presented elsewhere in this report. Control sites: since it is difficult to find a true “control” site within the setting of this evaluation, we propose the selection of a non-equivalent control sample of facilities, but one which permits relevant and reasonable comparisons between sites for the key indicators of interest. In this approach, sites in the control group will be intentionally matched to sites in the intervention group on characteristics that are associated with the outcome of interest (in the Results Areas). In some instances, this matching will be done at the individual level (e.g. to take into consideration the geographic spread of the selected intervention sites), resulting in a one-to-one match of sites in the two groups. Alternatively, and where necessary, we shall consider aggregate matching, in which we select a control group with the same general composition of relevant characteristics as the intervention group. Overall, the combined quasi-experimental approach that involves both the intervention-control comparisons, as well as the before-and-after comparisons, will provide reasonable time-series data, that are sufficient to draw conclusions about the relative impact of the PHS program, in the different results areas. This combined approach is a more robust approach compared to the single before￾and-after design, which is unable to demonstrate long-term effects. It is important to highlight, however, that by nature quasi-experimental designs are susceptible to internal validity and there is need for special care that addresses this as much as possible during the evaluation design. For this reason, we propose the use of additional secondary relevant data to substantiate findings from primary data. Qualitative methods In addition to the proposed quantitative approach, we propose the use of qualitative methods for assessing the “process” aspects of implementing the PHS program. We proposed the use of semi￾structured interview guides to collect information from key stakeholders (a list of these guides is provided in Annex 2). Specifically, qualitative methods will be used in the collection and analysis of information from key stakeholders to address evaluation questions that are qualitative in nature, within the scope of this evaluation. The PHS Program Document details the scope and objectives of the program, the intended areas of intervention, and the expected outputs. This end-term evaluation intends to examine the processes associated with the implementation of the PHS program. As such, the evaluation will assess the manner in which the planned activities were implemented. This assessment will be extended to the areas of implementation that are part of the changes in the scope of the PHS program, especially with the additional implementation areas that result from bringing on board the PNFP-support aspects. 77 Qualitative data obtained through key informant interviews will be analyzed using appropriate qualitative analysis methods. For this evaluation, we propose coding and thematic grouping and analysis of responses, coupled with a critical review, exploration and interpretive techniques. Coding is an interpretive technique that both organizes the data and provides a means to introduce the interpretations of it into certain quantitative methods. Where deemed necessary, computer assisted qualitative data analysis software will be considered. For the qualitative aspects of the evaluation, care will be taken to ensure trustworthiness of the information. This means that special attention will be given to processes of both data collection and data analysis to ensure the credibility of the findings. Some of the steps taken to ensure this will include: team check and triangulation, interviewer corroboration, confirmability, negative case analysis, among others. Data sources Secondary data For both the qualitative and quantitative aspects of the evaluation, extensive reviews of key literature will be undertaken. This will include, among others: key PHS program documents such as, the proposal, the annual work plans, the baseline reports for the different results areas, reports from mid-term reviews of different aspects of the program, relevant information on monitoring and evaluation of key program activities, databases of different supported entities, and others. The full list of documents to be reviewed is presented in Annex 3. Other relevant documents obtained during the course of interactions with key stakeholders will be reviewed and included in the list of key documents. Primary data Intervention sites The PHS program has provided support to a wide range of private-for-profit (PFP) and private-not￾for-profit (PNFP) health facilities and civil society organizations (CSOs) across the country. In addition, they have provided technical support to stakeholders that work with, oversee or manage private sector/CSO entities. As noted earlier, this evaluation seeks to obtain information from a sample of PFPs and PNFPs supported by the PHS program, and information from all stakeholders supported by the PHS program. The sample size and sampling approach considered for the evaluation are presented in the next section. The list of stakeholders is presented in Annex 4 in this report. Control sites The quasi-experimental design proposed for this evaluation articulates the need for studying a sample of control sites, with the view to examine the extent to which observed changes in the intervention sites can be attributed to PHS support. Control sites will include PFPs, PNFPs and CSOs that have NOT been supported by the PHS program. The selection of control sites will be guided by a number of technical considerations. Largely, control sites will be selected from districts where interventions sites are located. To the extent feasible, one-on-one matching will be considered in the selection of control sites, and where this is not feasible, general matching will be purposively considered. Care will be taken to ensure that a control facility is located within the same geographical location as a selected intervention facility. Control facilities have not yet been selected by submission of this inception report, because the team seeks to first get approval of the sample selected for the intervention sites. Once this approval is granted, matching of the control sites will be undertaken and also shared with the relevant parties for approval. 78 Stakeholder interview ▪ Key PHS staff ▪ GOU MOH: HIV, TB and malaria program heads; HMIS staff; other relevant MOH divisions ▪ Private Public Partnership Office (MOH) ▪ Implementing Partners (IPs), including Uganda Health Supply Chain project and a sample of comprehensive implementing partners (to get a list of relevant IPs from USAID) ▪ Participants from financial institutions and partner banks (Centenary Bank, DFCU, Ecobank) ▪ Medical and Dental Council, Nursing and Midwifery Council, Allied Health Council and Pharmacy Council of Uganda ▪ Stakeholder/participants involved in private health sector regulatory and supervisory systems. 79 Annex 6: Evaluation Sampling Framework Sampling for this evaluation will ensure adequate representation of each of the private sector sub￾groups – Private For-Profit (PFP), Private Not-For-Profit (PNFP), Civil Society Organizations (CSOs), drug stores/outlets – health facilities at different levels of service provision, and geographical distribution. While selecting clients/beneficiaries for interviews, adequate gender and age representation will be ensured. A purposive sampling approach will be followed for selection of all the facilities and entities for primary data collection. As this is a program evaluation, the evaluation team believes that all the service delivery and intervention areas of PHS need to be covered to ensure a comprehensive evaluation bringing out all the best practices, weaknesses and challenges in the PHS program. This will enable the evaluation team to provide sound, evidence-based and forward-looking recommendations on promoting the private sector for healthcare in Uganda. Following this approach, the evaluation team has collected data on implementation status of various service areas from PHS. From among each service area, facilities have been randomly sampled ensuring that all private sector sub-groups (PFPs, PNFPs, CSOs, medical outlets/drug stores and training schools) have been adequately represented. Attention was also paid to the geographical distribution of the sampled facilities, making sure that as many districts as possible have been covered. Overall 57 different institutions involved under the PHS program were selected across 36 districts. Of these 21 were PFP health facilities, 17 PNFPs, 6 CSOs implementing OVC activities, 9 pharmacies/drug stores, and 3 training institutes. Table 5: Overall sampling framework Total Implementing PFPs Sampled PNFPs Sampled Total Sampled Providing Care and Treatment Services 71 12 13 25 Trained in Quality Improvement (QI) 71 12 13 25 Voluntary Medical Male Circumcision (VMMC) 21 3 6 9 SQIS participating 42 10 7 17 Providing Community Health Insurance services 5 0 2 2 Engaged in Medicine Prices Dissemination 29 2 0 2 OVC 46 0 6 6 DCA 79 20 2 22 HaaB 185 24 6 30 Facilities 35 facilities have been selected from a total of approximately 71 facilities across all the service areas, based on the information provided by PHS. Approximately 20 facilities have been selected from the facilities from outside of PHS core service areas, which were targeted for DCA and HaaB activities. Thus, the total facilities sampled are 55 across 30 districts. PHS has informed the team that its core service areas are being implemented in 40 districts. So, our sample of 30 renders adequate geographical representation. In addition to the PHS intervention facilities, non-PHS facilities will also be sampled as controls. The listing of private facilities will be acquired from various sources such as local medical associations, councils, and government records. In cases where a listing is not available, private non-PHS facilities will be selected on an ad-hoc basis while in the district. In both the preceding control site selection approaches, due diligence will be exercised in selecting facilities comparable to the intervention facilities. 80 Clients/Beneficiaries 5 clients/beneficiaries will be selected from each of the 55 facilities. Every 2nd or 3rd client exiting the Out-Patient Department (OPD) will be randomly selected ensuring adequate gender and age representation. Thus, 275 clients/beneficiaries in total will be interviewed and this should provide a comprehensive perspective of public perception of the changes in availability, affordability and quality of services. Medical Bureaus The senior management of 3 of the Medical Bureaus – UCMB, UMMB, and UOMB – will be interviewed to understand their experience of implementing PHS activities and also the benefit they have derived from the HSS activities. 4 PNFP facilities from these bureaus have been sampled for the facility assessment. CSOs working with OVCs Of the 46 CSOs involved in OVC support activities, 6 CSOs have been sampled representing from 6 different districts. OVC caregivers/families 5 OVC caregivers/families will be selected under each of the 6 CSOs sampled; totaling 30 OVC caregivers/families. These will be purposively selected from among the caregivers/families in close vicinity of the CSO site to avoid long distances to be covered. Data collection tools The following data collection tools have been appended under Annex 2. 1. Health facility assessment tool 2. Patient exit interview tool 3. CSOs interview/assessment tool 4. OVC caregivers/families interview tool 5. Stakeholder interview tools Data cleaning and collation The collected data will be cleaned and categorized under the various appropriate evaluation questions and indicators. Both Qualitative and Quantitative data will be collated separately. The Evaluation Team has decided to use Excel as the data analytical tool. The primary data numerators and denominators are relatively straightforward for the different research questions. Due diligence will be practiced to ensure valid statistical inferences and logical conclusions based on sound data. Analytic plan The detailed qualitative and quantitative data analysis plan will include the scope of work for the data collectors. The data collectors will visit the 61 selected sites in 5 teams of 2 people. 5 Team supervisors will be responsible for the initial data collation and monitoring. The Evaluation Team will review the incoming data on a regular basis. The data will be stratified by demographic characteristics, such as sex, age, and geographical location, whenever feasible with the objective to review both qualitative and quantitative data related to PHS achievements against its objectives and targets. Based on the research questions, data collection tools are designed and appropriate statistical analyses identified for: 1. Quantitative data – analyze using descriptive statistics; 2. Qualitative data – use thematic review connecting to evaluation questions. Qualitative data will be used to substantiate quantitative data and provide answers where data is not available. All data analyses and presentations of key data findings will also address important data disaggregation /categorization relevant to the performance of PHS. The following classification will be used for the data collection and analysis: PFP vs PNFP (& CSOs). With the following facility categories: 1. Health Facility; 81 2. Medical Training School; 3. Retail Medicine Outlet; 4. OVC Facility. The Analytic Plan will contain the lay out of the tables, the unit(s) of analysis and the unit(s) of measurement, and a detailed lay out of the graphs, as well as the data variables that are required to produce the specific output tables and what percentage calculations are deemed most useful. Based on the volume and quality of the incoming data the evaluation team will decide whether there is a need for a more advanced descriptive analysis (including frequencies, means, medians and percentages). Data sources All the available secondary data sources have been inserted under Annex 3 and the primary data collection tools are under Annex 2. Limitations of the methodology and planned mitigations ▪ The quasi-experimental approach applied here is subject to concerns regarding internal validity, because the intervention and control groups may not be comparable at baseline. This limitation would be mitigated by using two-way assessment to establish casual effect of the PHS program, especially with the PFPs and PNFPs. By combining both the before-and￾after approach (that compares baseline, mid-term and end-term information) and the intervention-control comparison, augmented with a detailed review of relevant secondary data. Additionally, the proposal to interview all stakeholders of the PHS program adds to the robustness and comprehensiveness of the data collection. Lastly, the consideration of both quantitative and qualitative aspects of the program makes the proposed method more robust. ▪ As with most evaluations, the limitations of the proposed methods lie with the fact that we must study only a sample of facilities supported by the PHS program. We have a resource constraint (of both time and financial resources) that limits us to only studying a selected sample of facilities. While care has been taken in ensuring that we achieve the best selection of a sample, representativeness of the sample may not be fully reflective of the geographic spread of the scope of support given through the PHS program. ▪ At inception of the PHS program, although baseline studies were conducted for the different aspects of the program, no baseline data were collected from non-PHS health facilities. As such, collection of data from control sites will only be useable for cross-sectional comparisons with interventional facilities, but we shall not be able to do a time-series analysis for the control sites. This limitation is minimized, because the time-series analysis will be possible for the intervention sites for all the interventions where good baseline data is available. ▪ PHS did not have a results based / logical framework and so a set of activities informing the program outcomes were requested from PHS. As a result, development of an appropriate evaluation methodology has been complex, especially with the need to include the evaluation of aspects that were added to the scope of the program later in the program tenure. ▪ We propose a patient exit survey, because a household survey was more complex, time consuming and expensive. Relying on patients’ opinions about services in the private sector, immediately after the use of a service, has both advantages and disadvantages. The biggest advantage is the it minimizes recall bias. The biggest disadvantage is that perceptions provided by the respondent may only consider the immediate experience which may be a view based on a limited scope of services accessed on that day. ▪ Overall, the limitations of the proposed evaluation methodology are minimal, if the anticipated data sources will provide the expected data (of high quality). If data provided are of poor quality or if some important data are not provided, then the quality of the overall result of the evaluation will be compromised. 82 Ethical considerations The scope of evaluation requires that we assess public perceptions about the private sector. For this, we shall need IRB clearance. Our data collection tools for the patient exit survey will have a component that seeks the consent of the respondent, as is required by all ethics standards. Additionally, the entire assessment requires general IRB clearance from an IRB and also from Uganda National Council of Science and Technology. These ethics clearances will be sought once this Inception Report has been approved. 83 Annex 7: List of Sampled Sites District Facility Type Kampala Abii Clinic Clinic Alex Kampikaho & Annette Tumuheirwe Training Institute Case Hospital Hosp Galilee Community General Hospital Hosp Ikan Medical Centre HC International Medical Group Clinic - Watoto HC Kadic - Nakulabye HC Kireka SDA Health Centre HC Kitante Medical Centre HC Kyadondo Medical Centre HC Mengo Hospital Hosp (Med Bureau) Mirembe Medical Centre HC Namungoona Holy Cross HC III Hosp (Med Bureau) Paragon Hospital Hosp SAS Clinic - Bugolobi Clinic Span Medical Centre, HC II HC St. Joseph clinic Clinic Yamwe Pharmaceuticals Ltd Pharmacy Arua Arua Medical Laboratory Training School Limited Training Institute Kuluva HOSPITAL Hosp (Med Bureau) Zion Medical Clinic Clinic Bugiri Gloria Drug Shop Pharmacy Buikwe Living water community medical centre Clinic Lugazi Scoul Hospital Hosp Buikwe St. Francis Hosp (Nkokonjeru) Hosp (Med Bureau) Bushenyi Ishaka Adventist (SDA) Hosp Hosp (Med Bureau) Gulu Comboni Samaritans of Gulu OVC CSO Ibanda Family medical centre HC Iganga Iganga Islamic Medical Center HC III HC (Med Bureau) Jinja Good Samaritan drug shop (Mafubira) Pharmacy Kakira Sugar Works Hosp Akim Drug Shop Pharmacy Kabarole Ngombe Community Health Project OVC CSO Kamuli Kamuli Medical Centre HC Maama Peace Drug Shop Pharmacy St. John Drug Shop Pharmacy Kasese Community Medical Centre HC Matama Medical Clinic Clinic Kibaale Kagame Maternity Home HC Kiruhura Engari Community HC III HC Family Health Resource Centre Clinic Kitgum Kitgum Maternal & Medical Centre HC 84 Kumi Kumi COU Diocese OVC CSO Kyenjojo Bringing Hope to the Family OVC CSO Lamwo Pearl Medical and Maternity Center HC Lira Care Medical Clinic And Lab Services Clinic King James Nursing Training School Training Institute Lira Medical Center Hosp Luwero Nankyama Foundation OVC CSO Masaka St. Martin's Clinic Clinic Mbale Mbale General Clinic Zam-zam HC Mt. Elgon Hospital Ltd. Hosp Mbarara Diisi Medical Center HC Gord Medical Clinic Clinic Mityana Santa Maria Midcare-Mityana HC Mpigi Double Cure Medical Center HC Mubende St Luke Health Care Centre HC Mukono Prestige Drug Shop Pharmacy Sir Albert General Clinic & Nursing Home Clinic D&G Drug Shop Pharmacy Nebbi Caritas Nebbi OVC CSO Ntumgamo Katureebe Pharmacy Ltd Pharmacy Oyam Morning Glory Clinic Clinic Pader Labora Clinic HC Ultimate Choice Clinic Clinic Rakai Kyotera Med. Centre HC II Hosp Rukungiri Nyakishenyi Catholic HC III HC (Med Bureau) Ssembabule Bamu Hospital Hosp Tororo Tororo General Clinic Clinic Wakiso NRSL Health Unit HC Saidina Abubakar Muslim Hosp Hosp (Med Bureau) SOS Hermann Gmeiner HC (Med Bureau) St. Apollo Health Centre, HC III HC Triam Medical Centre Clinic Clinic Zombo Got Kamba Clinic Clinic 85 Annex 8: Data Collection Instruments A. Facility Assessment Tool B. Patient Exit Interview Tool C. OVC CSO Interview Tool D. OVC Caregiver/Family Interview Tool E. Pharmacy/Drug-Store Interview Tool F. Medical Training School Interview Tool G. Control Health Facility Interview Tool H. Control Pharmacy/Drug-Store Interview Tool I. District Community Development Officer (DCDO) Interview Tool J. MoH/PPPH Desk Key Informant Interview (KII) Tool K. Uganda Pharmacy Council KII Tool L. Uganda Medical and Dental Practitioner’s Council (UMDPC) KII Tool M. Uganda Nurses and Midwives Council (UNMC) KII Tool N. Allied Health Professionals Council (AHPC) KII Tool O. Uganda Private Midwives Association (UPMA) KII Tool P. Federation of Uganda Employers KII Tool Q. Uganda Health Federation (UHF) KII Tool R. Uganda Community Based Association KII Tool S. Uganda Insurance Association KII Tool T. Joint Medical Stores (JMS) KII Tool U. Uganda Health Marketing Group KII Tool V. Uganda Private Medical Practitioner’s Association KII Tool W. Uganda Private Training Institutions Association (UPHTIA) KII Tool X. Uganda National Association of Community and Occupational Health (UNACOH) KII Tool Y. Uganda Manufacturers Association KII Tool Z. African Center for Global Health and Social Transformation (ACHEST) KII Tool AA.Coalition for Health Promotion and Social Development (HEPS Uganda) KII Tool BB. Labnet Uganda KII Tool CC.Belgian Development Agency (BTC) KII Tool DD. Programme for Accessible health, Communication, and Education (PACE) Uganda KII Tool EE. Marie Stopes Uganda KII Tool FF. Uganda Catholic Medical Bureau (UCMB) KII Tool GG. Uganda Protestant Medical Bureau (UPMB) KII Tool HH. Uganda Orthodox Medical Bureau (UOMB) KII Tool II. Uganda Muslim Medical Bureau (UMMB) KII Tool JJ. Development Credit Authority (DCA) Banks (Centenary, Ecobank, DFCU) KII Tool KK.Control Banks KII Tool LL. Swedish International Development Cooperation Agency (SIDA) MM. 1-USAID Team – COR PHS Andrew Kyambadde MM. 2-USAID Team – Patricia Habu – David Rogers (Pvt Sector Unit – Economic Growth Office) NN. 1-USAID/Uganda Private Health Support (PHS) Program – PD (Lali Chania) NN. 2-USAID/Uganda Private Health Support (PHS) Program – COP (Dithan Kiragga) OO. USAID/Uganda Private Health Support (PHS) Program – DCOP (Joy Batusa) PP. USAID/Uganda Private Health Support (PHS) Program – DCOP (Johnson Masiko) QQ. USAID/Uganda Private Health Support (PHS) Program – A2F Team (Francis Zikusooka) RR. USAID/Uganda Private Health Support (PHS) Program – Health Services Team SS. USAID/Uganda Private Health Support (PHS) Program – M&E team TT. USAID/Uganda Private Health Support (PHS) Program – Quality team UU.USAID/Uganda Private Health Support (PHS) Program – HSS team VV. USAID/Uganda Private Health Support (PHS) Program – OVC team 86 Annex 9: List of secondary data reviewed 1. Baseline Survey Report of DCA Borrowers; Oct 22, 2014 2. HEALTH FACILITY CENSUS IN THE KAMPALA CAPITAL CITY AUTHORITY (KCCA) DIVISIONS 3. CENSUS REPORT (PHS), July 2017 4. THE KAMPALA CAPITAL CITY AUTHORITY PHARMACY AND DRUG SHOP FACILITY CENSUS 5. Cost and Pricing: An assessment of private health facilities in Uganda; May 2014 6. Dissemination report for the EMHSL medicines prices information; Jan 20, 2017 7. Fee guidelines for Medical and Dental Practitioners in Uganda, 2017 8. Second Dissemination Report for the EMHSL Medicines Price Information, June 2017 9. CONSUMER AWARENESS CAMPAIGN; CAPACITY BUILDING WORKSHOP FOR JOURNALISTS 10. TRAINING REPORT by PHS – March 15-16, 2017 11. GUIDELINES ON CONTINUING PROFESSIONAL DEVELOPMENT FOR THE ALLIED HEALTH PROFESSIONALS, August 2017; supported by PHS funding 12. Health as a Business (HaaB); Technical Guide, June 2016 13. Medicine Clients Charter, July 2017 14. Medicines Consumer Awareness Campaign Project Report, 2017 15. National Referral Guidelines, 2017 16. Baseline Capacity Assessment, November 2013 17. Provision of Actuarial Consultancy Services to Select Private Health Insurers in Uganda Literature Review, March 2017 18. Report on Existing Medicines; Price Monitoring Policies, July 2017 19. Rwanda Medical and Dental Council committed to Monitoring and enforcing standards for quality medical Practice (No dates) 20. Self-Regulatory Quality Improvement System (SQIS): Tool for Quality Improvement in the private sector, Uganda, 3rd Edition, February 2017 By MOH, GoU; funded by PHS 21. SITUATIONAL ANALYSIS OF REGISTRATION AND LICENSING PROCEDURES IN THE MEDICAL COUNCILS, 2017; PHS 22. 2015Self-Regulatory Quality Improvement System (SQIS): Tool for Quality Improvement in the Private Sector, Uganda August 2015; USAID, UHF, MOH-GoU 23. STAKEHOLDER CONSULTATION TO SUPPORT ALLIED HEALTH PROFESSIONAL COUNCIL TO COMPLETE THEIR CPD GUIDELINES, 14TH JUNE 2016 24. Training Centre for Male Circumcision Assessment Report: The Infectious Disease Institute (IDI) Aug 14-15, 2012 25. USAID/UGANDA PRIVATE HEALTH SUPPORT PROGRAM BASELINE SURVEY REPORT Nov 12 2013 26. PHS Year 1 Reports 27. PHS Year 1 PMP 28. PHS Year 1, Quarter 0 report: June – Sept 2013 29. PHS Year 1, Quarter 1 report: Oct – Dec 2013 30. PHS Year 1, Quarter 2 report: Jan – Mar 2014 31. PHS Year 1, Quarter 3 report: Apr – June 2014 32. PHS Year 1 (and Quarter 4) report: July 2013 – Sept 2014 33. USAID/UGANDA PRIVATE HEALTH SUPPORT PROGRAM PNFP ACTIVITY: SUSTAINING COMPREHENSIVE HIV/AIDS AND OVC SERVICES THROUGH PRIVATE NOT FOR PROFIT PROVIDERS IN UGANDA, AUGUST-SEPTEMBER 2014 34. PHS Year 2 Reports 35. PHS Year 2 PMP 36. PHS Year 2, Quarter 1 report: Oct – Dec 2014 37. PHS Year 2, Quarter 2 report: Jan – Mar 2015 38. PHS Year 2, Quarter 3 report: Apr – June 2015 87 39. PHS Year 2 (and Quarter 4) report: Oct 2014 – Sept 2015 40. USAID/UGANDA PRIVATE HEALTH SUPPORT PROGRAM PNFP ACTIVITY: SUSTAINING COMPREHENSIVE HIV/AIDS AND OVC SERVICES THROUGH PRIVATE NOT FOR PROFIT PROVIDERS IN UGANDA, OCTOBER –DECEMBER 2014 41. USAID/UGANDA PRIVATE HEALTH SUPPORT PROGRAM PNFP ACTIVITY: SUSTAINING COMPREHENSIVE HIV/AIDS AND OVC SERVICES THROUGH PRIVATE NOT FOR PROFIT PROVIDERS IN UGANDA, JANUARY –MARCH 2015 42. USAID/UGANDA PRIVATE HEALTH SUPPORT PROGRAM PNFP ACTIVITY: SUSTAINING COMPREHENSIVE HIV/AIDS AND OVC SERVICES THROUGH PRIVATE NOT FOR PROFIT PROVIDERS IN UGANDA, APRIL-JUNE 2015 43. PHS Year 3 Reports 44. PHS Year 3, Quarter 1 report: Oct – Dec 2015 45. PHS Year 3, Quarter 2 report: Jan – Mar 2016 46. PHS Year 3, Quarter 3 report: Apr – June 2016 47. PHS Year 3 (and Quarter 4) report with PMP: Oct 2015 – Sept 2016 48. USAID/UGANDA PRIVATE HEALTH SUPPORT PROGRAM PNFP ACTIVITY: CONSOLIDATING AND SUSTAINING COMPREHENSIVE HIV/AIDS AND OVC SERVICES THROUGH PRIVATE NOT FOR PROFIT PROVIDERS IN UGANDA, OCTOBER-DECEMBER 2015 49. PHS Year 4 Reports 50. PHS Year 4, Quarter 1 report: Oct – Dec 2016 51. PHS Year 4, Quarter 2 report: Jan – Mar 2017 52. PHS Year 4, Quarter 3 report: Apr – June 2017 53. PHS Year 4 (and Quarter 4) report: Oct 2016 – Sept 2017 PNFP Reports 54. ST Lawrence Otumbali HEALTH CENTRE III 55. Zia Angelina HEALTH CENTRE III 56. Adumi Oje HEALTH CENTRE III 57. Aripea HEALTH CENTRE III 58. Benedict medical CENTRE 59. Bishop Ceasor Asili hospital 60. Bitooma HEALTH CENTRE III 61. Buhara HEALTH CENTRE III 62. Bujumbura HEALTH CENTRE III 63. Comboni Hospital-Kyamuhunga 64. EDIOFE HEALTH CENTRE III 65. St Jacob Ggoli HEALTH CENTRE III 66. Good Samaritan HEALTH CENTRE III 67. UCMB-Holly Family Nyapea Hospital_ Zombo District 68. ST Kizito HEALTH CENTRE III 69. ST LUKE Namaliga HEALTH CENTRE IV 70. Takajjunge HEALTH CENTRE Iii 71. ST LUKE IRUHURA HEALTH CENTRE III 72. Virika Hospital III 73. WARR AGIERMACH HCIII 74. Yerya HEALTH CENTRE III 75. Kakatunda HEALTH CENTRE III 76. KAKORE Health Centre III 77. Karoli Lwanga Hospital Nyakibale 78. Kasawo Mission HEALTH CENTRE III 79. Kavule HEALTH CENTRE III 80. Kisenyi Good Shepherd HEALTH CENTRE III 81. OUR LADY OF CONSOLATA-Kisubi Hospital 82. Kitanga HEALTH CENTRE III 88 83. Kyakatara HEALTH CENTRE III 84. KYATIRI HEALTH CENTRE III 85. Kyembogo HEALTH CENTRE Iii 86. Muko Parish HEALTH CENTRE III 87. Nabingo Parish HEALTH CENTRE III 88. Nandere HEALTH CENTRE II 89. Ngetta HEALTH CENTRE III 90. Nkozi hospital 91. Nkuruba HEALTH CENTRE III 92. St Francis Nsambya Hospital 93. Rubanda PHC HCIII 94. Rushoroza HEALTH CENTRE IV 95. St. Therese of Lisieux HEALTH CENTRE III- Rwibaale 96. St. Anthoney Mitala Maria HEALTH CENTRE III 97. St. Charles Lwanga Buikwe Hospital 98. St. Francis Nkokonjeru Hospital 99. St. Francis Ocodri HEALTH CENTRE III 100. St. Joseph Nswanjere HEALTH CENTRE III 101. St. Mary’s HEALTH CENTRE III 102. St. Matia Mulumba HEALTH CENTRE III 103. St. Adolf Butiti HEALTH CENTRE III 104. St. Gabriel Mirembe Maria HEALTH CENTRE III 105. St. Joseph HEALTH CENTRE III-Madudu 106. St. Francis Hospital-Buluba 107. St. Francis Hospital Nyenga 108. St. Francis Naggalama Hospital 109. St. Francis of Asis HEALTH CENTRE III 110. St. Joseph’s Buyege HEALTH CENTRE III 111. St. Jude Uleppi HEALTH CENTRE III 112. St. Kizito HEALTH CENTRE III-Nattyole 113. St. LUKE KONGE HEALTH CENTRE (HEALTH CENTRE III 114. St. Martin’s HEALTH CENTRE III 115. St. Matia Mulumba HEALTH CENTRE III 116. St. Monica HEALTH CENTRE III-Katende 117. St. Ulrika HEALTH CENTRE III 118. St. Kizito HEALTH CENTRE III 119. St. LUKE Namaliga HEALTH CENTRE IV 120. St Assumpta Alivu HEALTH CENTRE III 121. St. FRANCIS HEALTH CENTER III NJERU 122. CAPACITY ASSESSMENT REPORT; BUIKWE DISTRICT, NJERU TOWN; March 2017 123. Takajjunge HEALTH CENTRE III 124. St. LUKE IRUHURA HEALTH CENTRE III 125. Virika Hospital III 126. WARR AGIERMACH HCIII 127. Yerya HEALTH CENTRE III 128. Comprehensive Assessment Report for Uganda Catholic Medical Bureau (UCMB); May 2017 129. Al-Hijra HEALTH CENTRE III 130. Arahmah Medical CENTRE 131. BUWENGE HOSPITAL AND MEDICAL CENTRE 132. BWEYOGERERE HEALTH CENTRE III 133. CRESCENT MEDICAL CENTRE 134. IGANGA ISLAMIC MEDICAL CENTRE 89 135. Jinja Islamic HEALTH CENTRE III 136. KAKUNGUBE HEALTH CENTRE III 137. KIBULI MUSLIM HOSPITAL 138. Kyotera Muslim HEALTH CENTRE III 139. Lugazi MUSLIM HEALTH CENTRE 140. Lugo Muslim HEALTH CENTRE III 141. LYANTONDE MUSLIM HEALTH CENTRE III 142. St. LUKE IRUHURA HEALTH CENTRE III 143. Njovu Islamic HEALTH CENTRE III 144. Saidina Abubakar Muslim Hospital 145. Saidina Abubakar Nursing Home 146. Comprehensive Assessment Report for Uganda Muslim Medical Bureau (UMMB); May 2017 147. TAQWA HEALTH CENTRE III 148. Taqwa HEALTH CENTRE III Nkonge 149. Comprehensive Assessment Report for Uganda Muslim Medical Bureau (UMMB); May 2017 150. AKONYIBEDO HEALTH Centre 151. ANASTASIAS MWEBAZA HEALTH CENTRE III 152. HOLY CROSS HOSPITAL NAMUNGOONA 153. MPIGI ORTHODOX HEALTH CENTRE III 154. ST PANTELEIMON MONDE 155. Comprehensive Assessment Report for Uganda Orthodox Medical Bureau (UOMB); June 2017 156. All Saints Health Services-Kagoma 157. Amuca HEALTH CENTRE III 158. Anyiribu HEALTH CENTRE III 159. Azur Christian HEALTH CENTRE IV 160. Boroboro HEALTH CENTRE III 161. Bugema university HEALTH CENTRE III 162. Bulyansime HEALTH CENTRE III 163. Double Cure Medical Centre 164. Engari community HEALTH CENTRE III 165. FAMILY CARE HOSPITAL 166. Ishaka SDA Hospital 167. IVUKULA Health Centre II 168. J. O. Y. Medical CENTRE III 169. Kabarole hospital 170. Kairos Health CENTRE IV 171. Kaluba HEALTH CENTRE III 172. Kirema HEALTH CENTRE III 173. KISIIZI HOSPITAL 174. Kiwoko Hospital 175. Kuluva hospital 176. KUMI hospital 177. Kyetume HEALTH CENTRE III 178. MAKONGE COMMUNITY HEALTH CENTRE III 179. MASINDI –KITARA HEALTH CENTER (IV) 180. NEEDS ASSESSMENT REPORT; Masindi district; April 2017 181. MENGO HOSPITAL 182. Mitandi HEALTH CENTRE III 183. Mukono hospital 184. Noah’s Ark HEALTH CENTRE III 185. North Kigezi Diocese Maternal & Child HEALTH CENTRE IV 90 186. PAG HEALTH UNIT LIRA HEALTH CENTRE IV 187. Rugarama hospital 188. Rushere Community Hospital 189. ST Stephen hospital 190. St Apollo Health Centre III 191. ST LUKE IRUHURA HEALTH CENTRE III 192. St Luke Katiyi HEALTH CENTRE III 193. Luke Medical Centre II, LUGALA 194. ZUMBO HEALTH CENTRE III 195. Comprehensive Assessment Report for Uganda Protestant Medical Bureau (UPMB); DRAFT; May 2017 196. Uganda’s National Strategy for Public-Private Partnerships in Health, 2016 – 2020 197. UHF Interim Annual Report, May 2015; USAID (PHS), UHF 198. UHF Grant Year 2, Final Technical Report, November 2016 199. Amendment # 8 to the contract – 06/12/2013 200. CONTRACT No. AID-617-C-13-00005; USAID/Uganda Private Health Support Program 201. USAID/UGANDA PRIVATE HEALTH SUPPORT PROGRAM; ACTUARIAL SERVICES 202. Product Development Report, June 2017; Final Project Report, July 2017 91 Annex 10: Additional Evaluation Questions (HSS & OVC) 8. Health Systems Strengthening (HSS): Has the implementation of health systems strengthening (HSS) within the faith-based health sub-sector led to sustained scale up of services for people living with and affected by HIV/AIDS? After the closure of the Inter-Religious Council of Uganda (IRCU) project in 2016, the team and most of its activities were transferred to PHS. Thus, the HSS initiative was established under PHS focusing on the PNFP sector, specifically the Medical Bureaus. Key HSS activities implemented were: 1. Health governance and management 2. Human Resources for Health 3. Health Financing 4. Quality improvement and Information systems management 5. Coordination - with the public sector and other stakeholders at national, subnational and district levels The HSS initiative has been implemented only for the past 9 months and the focus of the activities have been on systems strengthening at the 4 Medical Bureaus (MBs) and 130 of their network HFs. Evidence of adoption of administrative, finance, and HR standards were noted at the MBs – these are being disseminated to their HFs. It will realistically take some time for these strengthening initiatives to translate into scaled up services for PLHIVs/PAHIVs. 8.1 What is the evidence that leadership and governance systems in the faith-based PNFP sub-sector have been strengthened? The 4 MBs have been strengthened in leadership and governance systems but the initiatives have yet to percolate down to all their respective HFs. Of the 4 Medical Bureaus (MBs), all 4 have been trained in corporate governance. Of the 56 HFs visited, 9 HFs (16%) confirmed that their Hospital Management Committees have been trained in corporate governance; of these 9, 2 are PFP HFs and 7 are PNFPs. 11 HFs (20%) have been supported with development of Board Governance Manuals; of these 11, 3 are PFP HFs and 8 are PNFPs. 10 HFs (19%) informed that their hospital managers have been trained in strategic planning; of these 10, 3 are PFP HFs and 7 PNFPs. All 4 MBs have had supervisory visits from the PHS HSS team, however there is no documentation of such visits or any feedback provided after these visits. Of the 56 HFs visited, 18 HFs (32%) had received a total of 72 supervisory visits from PHS since 2016; of these 8 PFPs received a total of 44 supervisory visits while 10 PNFPs received a total of 28 supervisions. None of the HFs who informed of supervisory visits from PHS had any documentation of such visits nor did they receive any informal/formal feedback from PHS. Of the 56 HFs visited, 9 HFs (16%) confirmed that they have been supported by development of Customer Relations Management (CRM) guidelines; of these 3 are PFP HFs and 6 are PNFPs. None of the 4 MBs have undergone CRM Training of Trainers (TOT) yet. Of the 56 HFs visited, 10 HFs (18%) informed that they have undergone CRM TOTs; of these 5 are PFP HFs and 5 PNFPs. None of the 4 MBs have undergone trainings in advocacy and lobbying yet. Of the 56 HFs visited, none of the HFs were trained in advocacy and lobbying. None of the 4 MBs or the 56 HFs visited have been trained to improve their skills in Operational Research. Of the 4 MBs, all 4 have been trained in DHIS2 to enable them to support health facilities in reporting. Of the 56 HFs visited, 48 HFs (86%) have been trained in DHIS2 reporting; of these 48, 30 92 are PFPs and 18 PNFPs. While of the 56 HFs visited, 48 (86%) have been trained in health data management to support health facilities to adapt information systems such as HRIS, DHIS to comply with the national systems; of these 30 are PFPs and 18 PNFPs. 8.2 Have the Human Resources for Health (HRH) been strengthened in the faith-based PNFP sub-sector? The 4 MBs have been strengthened in HRH but the process of expanding these HRH strengthening initiatives to their HFs has been stalled due to the forthcoming closure of PHS. The activities have been implemented for only 9 months so ascertaining strengthening may not be possible as institutionalization of the HRH initiatives may take longer. Of all the 4 MBs, all 4 have been supported with development of HR Manuals. Of all the 56 HFs visited, 9 HFs (16%) have been supported with development of HR Manuals; of these 9, 3 are PFP HFs and 6 are PNFPs. Of the 56 HFs visited, 11 HFs (20%) stated that workshops were conducted for supervisors and health unit managers on the use of performance management tools; of these 4 are PFP HFs and 7 are PNFPs. All 4 MBs have been trained in Work Force Indicator of Staff Needs (WISN). Of the 56 HFs visited, 10 HFs (18%) have been trained in WISN; of these 10, 3 are PFP HFs and 7 PNFPs. None of the 4 MBs have been trained in Strategic Human Resources Management (SHRM). Of the 56 HFs visited, 8 HFs (14%) have been trained in Strategic Human Resources Management (SHRM); of these 8, 2 are PFP HFs and 6 PNFPs. None of the MBs have yet been trained in iHRIS system of the MoH, whereas, of the 56 HFs visited, 7 HFs (12%) have been trained in iHRIS system of the MoH; of these 2 are PFP HFs and 5 PNFPs. 8.3 Have the health financing systems strengthened in the faith-based PNFP sub-sector? The 4 MBs have been strengthened in health financing systems but the initiatives should be institutionalized in their respective HFs. All 4 MBs have been supported with development of Finance Manuals. Of the 56 HFs visited, 9 HFs (16%) have been supported with development of Finance Manuals; of these 3 are PFP HFs and 6 are PNFPs. None of the MBs have yet been trained in business management and entrepreneurship, while of the 56 HFs visited, 11 HFs (20%) have been trained in business management and entrepreneurship; of these 11, 3 are PFP HFs and 8 are PNFPs. Of the 56 HFs visited, 8 HFs (14%) were trained in internal audit systems and processes; of these 8, 4 are PFP HFs and 4 are PNFPs. And 3 MBs (except UCMB) have been supported in procurement of accounting software and the staff trained in its use. Of the 56 HFs visited, 4 HFs (7%) have been supported in procurement of accounting software and the staff trained in its use; of these 4, 2 are PFP HFs and 2 PNFPs. None of the MBs have been trained in Performance Based Financing Mechanism (PBFM). Of the 56 HFs visited, 8 HFs (14%) have been trained in PBFM; of these 8, 1 is a PFP HF and 7 PNFPs. Of the 56 HFs visited, 5 HFs (9%) informed that trainings have been conducted on grants application/and proposal development skills for hospital managers and boards; of these 5, 1 is a PFP HF and 4 PNFPs. Of the 56 HFs visited, 10 HFs (18%) informed that hospital managers and hospital board finance sub-committee members were trained in innovative strategies for fundraising and local resource mobilization; of these 10, 4 are PFP HFs and 6 PNFPs. Of the 56 HFs visited, 4 HFs informed that in total 14 orientation meetings were conducted for health facility management teams in the guidelines for pricing of select common disease conditions; of these 4 HFs, 2 are PFPs and 2 are PNFPs. 9. Orphans and other Vulnerable Children (OVC): How has the PHS performed in strengthening family-based approaches to care and support to OVCs using sub-granting mechanism? Under the initiatives for the OVCs, the PHS program focused on supporting the OVC households, with some activities specifically targeting the OVCs but under the purview of the OVC family structure. The aim of the program was to – 1) Improve access of the OVCs and their households to HIV services in line with the 90-90-90 strategy; 2) Strengthen socio-economic capacity of the families 93 – focus on out-of-school OVCs – to meet basic needs to life; 3) Ensure safety, protection and psychosocial wellbeing of all OVCs; 4) Support schooling of OVCs, especially adolescent girls; and 5) Achieve all the above through capacity building of OVC sub-grantees (CSOs). PHS has performed well in strengthening family-based approaches to care and support to OVCs using sub-granting mechanism. In total, 46,572 OVCs from 13,993 households are being catered to under PHS, with PFPs catering to 14,674 and PNFPs to 31,898. 9.1 What has been the contribution to the 90-90-90 strategy? PHS had aimed to contribute to 90-90-90 through providing easy access to HIV Testing Services (HTS) for the OVCs and their caregivers, providing easy access to ART for the OVCs and their caregivers, and ensuring linkages to CD4 testing for those on ART to monitor viral load. Specific data on any of the aforementioned activities is not available to ascertain contribution of PHS to the 90-90-90 strategy. All 6 of the OVC CSOs visited informed of having established partnerships with local HFs to support HIV testing for OVC and caregivers through outreaches and home based HIV Testing Services (HTS), referral to care and treatment services, and follow up support for adherence/viral load suppression. Of the 30 OVC caregivers interviewed, 27 caregivers (90%) from all 6 CSOs visited informed that HIV testing Services (HTS) was provided. Of the 30 OVC caregivers interviewed, 21 (70%) informed that they received facility based HTS, while 16 (53%) informed of having received home based HTS. The 6 OVC CSOs visited informed that they had provided transportation to 144 OVCs to access ART. Of the 30 OVC caregivers interviewed, 9 (30%) informed that they had ever received transportation support from the CSO to access ART. The 6 CSOs also informed that in total 114 community volunteers, VHTs and grantee staff were involved in counselling (also on hygiene, sanitation, malaria and TB prevention) and adherence support. Of the 30 OVC caregivers interviewed, 21 (70%) informed that home-visits were conducted to provide counseling and adherence support; 6 caregivers informed that these visits were conducted by VHTs, 28 by CSO staff, and 15 by volunteers. Of the 30 OVC caregivers interviewed, 24 (80%) informed that they had been sensitized on primary healthcare (hygiene, sanitation, malaria prevention, TB prevention and symptoms). 9.2 How has PHS strengthened the socio-economic capacity of families to meet basic needs of life? Based on the secondary data review and primary data collection, PHS seems to have met its targets in strengthening the socio-economic capacity of families to meet basic needs through the available funding and also mobilizing additional funding from the private sector. The 6 CSOs visited informed the evaluation team that of a total of 1500 OVC caregivers, they had trained 1153 (77%) in Financial Literacy, Business Startup, and Management. Of the 30 OVC caregivers interviewed, 25 (83%) stated that they have been trained in the same activity. The 6 CSOs visited informed that in total 132 Village Saving and Loans Associations (VSLAs) have been established and trained. Of the 30 OVC caregivers interviewed, 25 (83%) stated that VSLA groups have been established and trained in their village. The 6 CSOs informed that 132 VSLAs and 707 OVC households have been linked with micro-finance support services and Government poverty alleviation/wealth creation programs such as Community Driven Development (CDD), Youth Livelihoods Program. Of the 30 OVC caregivers interviewed, 14 (47%) formed of ever being referred to micro-finance support services. Of the 30 OVC caregivers interviewed, 11 (37%) informed of ever being referred to Government poverty alleviation/wealth creation programs like CDD, Youth Livelihood programs. The 6 CSOs informed that in total 389 out-of-school OVCs have been trained in different vocational skills courses such as tailoring and hair dressing. Of the 30 OVC caregivers interviewed, 16 (53%) informed the evaluation team of having been trained in different vocational skills courses such as 94 tailoring and hair dressing. The 6 CSOs informed that in total 161 out-of-school OVCs have been provided with startup kits to enable them to begin small-scale business enterprises. Of the 30 OVC caregivers interviewed, 9 informed of having been provided with startup kits to enable them to begin small scale business enterprises. The 6 CSOs additionally informed that in total 32 Artisans have been trained in the use of Directorate of Industrial Training (DIT) apprenticeship manuals to improve pedagogical, testing, and assessment skills. Of the 30 OVC caregivers interviewed, 9 (30%) informed that artisan have been trained in the use of Directorate of industrial Training (DIT) apprenticeship manuals to improve pedagogical, testing and assessment skills. The 6 CSOs informed that 360 out-of-school OVCs have been trained in business skills to enable them to manage their business enterprises better. Of the 30 OVC caregivers interviewed, 19 (63%) stated that OVCs have been trained in business skills to enable them to manage their business enterprises better. The 6 CSOs also stated that 173 community volunteers/ VHTs have been trained to conduct routine nutrition assessment and refer malnourished children to HFs for further management. The 6 CSOs stated that 859 OVC caregivers have been trained to improve household food and nutrition security. Of the 30 OVC caregivers interviewed, 24 (80%) have been trained to improve household food and nutrition security. 9.3 How has PHS ensured safety, protection and psychosocial wellbeing of all OVCs? Primary data from the field visits and secondary data review indicate that PHS have put measures in place to ensure the safety, protection and psychosocial wellbeing of OVCs catered by it. All the 6 CSOs informed that their staff have been linked with child protection structures (Probation Officer, CDOs, Police, Child protection committees etc.) in respective districts to create awareness about child abuse/violence against children. Of the 30 OVC caregivers interviewed, 15 (50%) confirmed having ever been visited by the child protection entities such as Probation Officer, Police, or Child Protection Committees. The 6 CSOs informed that in total 52 child abuse cases have been referred for handling/management. The 6 CSOs informed that in total 197 teachers have been trained in providing psychosocial support for children living with and affected by HIV/AIDS in school and community settings. The 6 CSOs also stated that in total 525 OVCs have undergone birth registration to enable them to access public essential services like education. Of the 30 OVC caregivers interviewed, 21 (70%) informed the evaluation team that their OVCs have undergone birth registration to enable them to access public essential services like education. According to the 6 CSOs, in total 67 children with disabilities (CWDs) have been categorized for their disabilities. Of the 30 OVC caregivers interviewed, 10 had OVCs with disabilities and all 10 (100%) informed of having been categorized for their disabilities. Of the 6 CSOs visited, 3 (50%) informed that they were conducting Centre days for children’s recreation activities, counselling, and life skills development. Of the 30 OVC caregivers interviewed, 13 (43%) stated knowledge of any Centre days for children’s recreation activities, counselling, and life skills development having ever been conducted. 9.4 How has PHS ensured OVC schooling? PHS has worked to ensure schooling of its OVCs as suggested by the primary and secondary data. The 6 CSOs visited stated that 1765 OVCs from critically vulnerable families were provided with scholastic materials to support retention and completion of critical levels of education. Of the 30 OVC caregivers interviewed, 23 (77%) stated that scholastic materials were ever provided. The 6 CSOs visited informed that schools fee top up payments were made for 127 OVCs. Of the 30 OVC caregivers interviewed, 18 (60%) informed that schools fees top up payments were made for their OVCs. The 6 CSOs visited stated that 250 school visits were conducted to check on academic 95 progress of OVCs in schools and to provide career guidance and Psychosocial Support (PSS) support where necessary. The 6 CSOs informed that 21 peer school clubs have been formed. Of the 30 OVC caregivers interviewed, 6 (20%) stated that they had knowledge of peer school clubs in their village. The 6 CSOs stated that in total 1467 adolescent girls were provided with sanitary pads. Of the 30 OVC caregivers interviewed, 17 stated that their adolescent girl OVC was provided with sanitary pads. The 6 CSOs informed that in total 1181 adolescent girls were trained on how to locally make re-usable sanitary towels. Of the 30 OVC caregivers interviewed, 14 informed that their adolescent girl OVC was trained on how to locally make re-usable sanitary towels. 9.5 How was capacity building of OVC grantees undertaken? The OVC grantee staff have been trained in all OVC initiatives planned by PHS and the grantees have also been supported in mobilizing funds from the private sector to ensure sustainability of OVC support initiatives – generating US$ 201,329 for OVC support from the private sector. Of the 6 CSOs visited, 2 (33%) informed of having engaged 9 private sector entities to mobilize matching funds for OVC activity implementation. These 2 CSOs informed that 611 OVCs have been supported through this mobilization. Regular staff review meetings within each OVC CSO has been encouraged as a part of capacity building of these OVC grantees. The 6 CSOs stated that in total 187 staff review meetings were conducted for assessing progress, planning, and sharing of updates and promising practices. Each of the 6 OVC CSOs on an average had conducted 31 staff review meetings over 2 years. 96 Annex 11: Gaps in OP records at PHS HFs The table also shows that the OP data record keeping has been improving over the years – HFs with gender wise missing data has reduced from 57% to 18% and those missing total OP data have reduced from 41% to 7%. Table 6: HFs missing OP data Year HFs missing gender wise OP data Percentage HFs missing gender wise OP data HFs missing total OP data Percentage HFs missing total OP data Year 1 32 57% 23 41% Year 2 31 55% 21 38% Year 3 20 36% 12 21% Year 4 18 32% 10 18% Year 5 10 18% 4 7% 97 Annex 12: Select Program Indicators and Data * Please note that some of the indicator fields are blank because indicators were dropped during the program implementation period (which is a common practice). This Annex is also a selection of only some indicators (those used during the Country Operational Plan planning period) and is used to show both the continuity and lack of continuity that existed during data collection. Although some of the information for the blank fields can be found from other sources, this shows that the Program itself was not tracking these indicators during these time periods, therefore it was not used in decision-making. Period Indicators Aug￾Sep 14 Oct￾Dec 14 Jan￾Mar 15 Apr￾Jun 15 Jul - Sep 15 Oct￾Dec 15 Jan - Mar 16 Apr - Jun 16 July - Sep 16 Oct - Dec 16 Jan - Mar 17 Apr - Jun 17 Jul - Sep 17 HCT Indicator 1: Number of individuals who received testing and counselling services and received their test results 26,891 55,952 57,934 66,793 67,823 40,796 45,855 34,895 41,618 41368 44784 53067 48645 Indicator 2: Total number of individuals who received HCT services for HIV and were found HIV Positive 1,197 1,960 1,978 3256 2135 1,353 1,626 1,483 1,478 1,409 1,568 1,713 1,411 Indicator 3: Number of individuals who received testing and counselling services and received their test results as couples. 1,391 2,559 2,904 2506 2446 SMC Indicator 4: Number of clients counselled, tested and received HIV results as part of SMC package 837 5,048 6812 6048 4793 306 1,327 466 780 Not tracked during the period Indicator 5: Number of males circumcised as part of the minimum package of SMC for HIV prevention services disaggregated by age (quality indicator) 747 4,575 6,273 5,495 4,547 302 1,321 456 739 940 3321 8696 7778 98 Indicator 6: Number of circumcised clients experiencing at least one moderate or severe adverse event (AE) during or following surgery, within the reporting period 28 68 63 67 31 29 63 52 121 Post Exposure Prophylaxis (PEP) Indicator 7: Number of Program supported sites that are capable of providing PEP 16 16 16 16 16 73 73 74 74 Indicator 8: Number of persons provided with PEP disaggregated by exposure type 50 71 106 97 77 150 168 142 157 PMTCT Indicator 8: Number of pregnant women with known HIV status (includes women who tested for HIV and received their results) Indicator not tracked in this period 7,334 8,033 8,443 9,306 6,011 5,596 6190 5826 Indicator 9: Number of women attending 1st ANC visit at PNFP facilities 3,401 4,726 4,927 4,959 4,817 7,236 7,947 8,143 7,912 Indicator 10: Number of pregnant women tested for HIV at 1st ANC visit 3,401 4,726 4,451 4,959 4,817 4,321 4,914 4,990 5,358 4950 5224 5092 1480 Indicator 11: Number of HIV+ pregnant women identified during the reporting period 186 472 98 241 345 139 205 177 140 103 98 93 29 Indicator 12: Number of HIV+ women receiving ARVs to reduce MTCT 109 393 67 214 344 121 218 162 95 410 391 385 319 Indicator 13: Number of deliveries to HIV+ women at PNFP facilities 180 253 276 227 222 196 243 231 250 234 285 234 118 99 Indicator 14: Number of exposed babies given ARVs (Nevirapine Syrup) 179 236 261 227 222 196 238 229 220 224 194 232 245