i Submitted by: Performance Evaluation of USAID/UGANDA’s 2009-2014 STRIDES for Family Health Program Department of Social Work & Social Administration, Makerere University- January 2015 ii Report Details Evaluation report Order No: AID-617-BC-14-00008 Submitted by the Department of Social Work and Social Administration Evaluation Team Dr. Eddy J Walakira Team Leader Dr. Denis Muhangi M&E Specialist Dr. Daniel Murokora MCH/FP Expert John E Ssengendo PBC Specialist Mr. Joshua Kayiwa Statistician Support Team Dr. David Kaawa Mafigiri Technical Evaluation Support Dr. Yovani Lubaale Support Statistician Mr. Jaffer Byawaka Statistical and Secondary Analysis Support Mr. Ismael Ddumba Nyanzi Project Management/Administrative Support iii Contents Contents ___________________________________________________________________ iii Executive summary ___________________________________________________________ v STRIDES Intervention Districts in Uganda ________________________________________ x List of Tables and Figures _____________________________________________________ xi 1. Introduction _____________________________________________________________ 1 1.1 Background and Context _________________________________________________________________1 1.3 Purpose of the Evaluation ________________________________________________________________2 1.4 Evaluation Methodology _________________________________________________________________3 1.5 Limitations ____________________________________________________________________________3 2. Appropriateness and Effectiveness of the Fully Functional Service Delivery Model _____ 4 2.1 Introduction ___________________________________________________________________________4 2.2 Expectations from the FFSDS Approach _____________________________________________________4 2.3 Adaptations to the FFSDS Model __________________________________________________________4 2.4 Appropriateness of the FFSDS Model _______________________________________________________5 2.5 Effectiveness of the FFSDS Model ................................................................................................ 7 2.5.1 Functionality of Health Services ................................................................................................. 7 2.5.2 Making Services More Accessible ............................................................................................ 11 2.5.3 Increasing Utilization of Services ............................................................................................. 12 2.5.4 Quality Improvement and Overall Quality of Care ................................................................... 13 2.5.5 Comprehensiveness of Services ................................................................................................ 14 2.5.6 Sustainability of Services .......................................................................................................... 14 2.6 Conclusions on FFSD __________________________________________________________________ 15 2.7 Challenges and Unexpected Outcomes _____________________________________________________ 15 2.8 Lessons Learned from Implementing FFSD _________________________________________________ 16 2.9 Recommendations on FFDS _____________________________________________________________ 16 3. Effectiveness of the Performance Based Financing Model ________________________ 17 3.1 Introduction __________________________________________________________________________ 17 3.2 Description of the PBF Model ____________________________________________________________ 17 3.3 Effectiveness of the PBC Model __________________________________________________________ 17 3.3.1Trends in Access and Availability of RH/FP Services ............................................................... 18 3.3.2 Trends in Access and Availability of Child Survival Services.................................................. 19 3.3.3 Factors contributing to the Success of PBC .............................................................................. 19 3.4 Potential for PBC Model Expansion _______________________________________________________ 21 3.5 PBC’s Limitations and Challenges ________________________________________________________ 21 3.6 Lessons Learned from using the PBC Model ________________________________________________ 21 3.7 Recommendations on the PBC approach ____________________________________________________ 22 4. Addressing the Unique Needs of Young People ________________________________ 23 4.1 Introduction __________________________________________________________________________ 23 4.2 Unique RH/FP Needs of Youth ___________________________________________________________ 23 4.3 Description of STRIDES Interventions to Address Youth’s Needs _______________________________ 23 Performance in relation to delivery of Youth Friendly Services _____________________________________ 25 4.4 Appropriateness of STRIDES’ Interventions for Youth ________________________________________ 25 4.5 Effectiveness of Youth Interventions _______________________________________________________ 27 Youth awareness of RH/FP issues ____________________________________________________________ 28 4.6 Limitations and Challenges in Addressing Youth’s Needs ______________________________________ 28 4.7 Conclusions, Lessons and Recommendations on Youth Interventions _____________________________ 29 ANNEXES ______________________________________________________________ xxxi Annex A: Scope of Work _________________________________________________________________ xxxi Annex B: Background Information ________________________________________________________ xxxiii iv Annex B-1: Maternal, New Born and Child Health Indicators for Uganda .................................. xxxiii Annex B-2: Strides for Family Health Partners ............................................................................ xxxiii Annex B-3: STRIDES Interventions by District ........................................................................... xxxiv Annex C: Evaluation Methodology _________________________________________________________ xxxv C–1: Evaluation Methodology ....................................................................................................... xxxv Approach to Answering the Evaluation Questions ......................................................................... xxxv Annex C-II: Documents Reviewed ..................................................................................................... xl Annex D: Other Annexes to Section One _____________________________________________________ xli Annex D-1: Number and Type of Health Facilities supported by STRIDES ..................................... xli Annex D-2: Breakdown of Funds Spent by Strides by October 2014 ................................................ xli Annex E: Annexes to Section Two _________________________________________________________ xliii Annex E-1: Health Facilities Renovated/Remodeled ....................................................................... xliii Annex E-2: Examples of Facilities Constructed/Renovated ............................................................ xliii Annex E-3: Summary of Health Facilities by Level that Received Equipment from STRIDES by District xliv Annex E-4: Percentage of Health Facilities Submitting Timely HMIS Reports to Health Sub-District and MoH xlv Annex E-5: Proportion of Facilities Meeting Minimum Requirements: Data from 2013 Star EC Health Facility Assessment (HFA) Reports ................................................................................................. xlvi Annex E-6: Percentage of Villages with Functional VHTs (PY1-PY5) ............................................... l Annex E-7: Trends in Service Utilization under the Fully Functional Service Delivery System ......... l Annex E-8: Number of Health Service Providers Trained by STRIDES ............................................ lii Annex E-9: Proportion of assisted Health Facilities providing RH, and FP services at Baseline and End of Project liii Annex F: Annexes to Section Three _________________________________________________________ liv Annex F-1a: List of PBC Sub Contractors ......................................................................................... liv Annex F-1b: List of PBC Contractors and RPF 002 ........................................................................... lv Annex F-2: Contractors that met Quarterly Targets by Type of Contract ......................................... lvii Annex F-3: Contractors that met Annual Targets by Type of Contract ............................................ lvii Annex F-4: Contribution of PBC Contractors to 4th ANC and IPT Utilization ................................ lviii Annex F-5: Cumulative Contribution of PBC Contractors to Nutrition and DPT Vaccination Services lviii Annex G: Annexes to Section Four __________________________________________________________ lix Annex G-1: Matrix on identified Young People’s Unique Needs ...................................................... lix Annex G-2: Functionality Status of YFS in STRIDES supported HFs ............................................... lx Annex G-3: Number of Health Workers trained in Youth Friendly Services by District 2011 & 2012lxi Annex G-4: Number of Health Workers trained in Youth Friendly Services by District .................. lxi Annex G-5: Knowledge gain for health Workers trained in YFS ...................................................... lxi Annex H: Consultants’ Work-plan and Schedule ______________________________________________ lxiii Annex I: Travel Schedule to Districts (May 25th – June 8th) ______________________________________ lxv Annex J: Evaluation Design Matrix _________________________________________________________ lxvi Annex K: Evaluation Tools _______________________________________________________________ lxxi v Executive summary STRIDES for Family Health was a five year $48 million USAID funded program aimed at increasing the use of Reproductive Health, Family Planning, and Child Survival services at the facility and community levels in fifteen districts in Uganda’s Eastern, Central and Western regions. The program served as many as 588 facilities providing health services to an estimated 5.6 million beneficiaries. Management Sciences for Health (MSH) in partnership with Communication for Development Foundation Uganda (CDFU), the Uganda Private Midwives Association (UPMA), Jhpiego, and Meridian International implemented the program from January, 2009 to end of February 2014. The program employed three major strategies namely: 1) A health systems strengthening approach, also known as the Fully Functional Service Delivery System (FFSDS); an approach that directs improvements across a whole range of health service system components1 ; 2 . 2) Performance-based financing (PBF) with non-governmental organizations (NGOs) and private sector health providers to expand access to essential health services; and; 3) Development of the management and leadership capacity of local institutions, to enhance individuals’ clinical skills, and establish or increase community health accountability. Makerere University’s Department of Social Work and Social Administration (DSWSA) conducted this evaluation over a period of 12 weeks, between May and August 2014. The purpose of this evaluation was to generate insights into the performance of the STRIDES project to determine the extent to which the core program strategies may have been effective in achieving expected results. The key questions that the evaluators sought to answer as provided under the Terms of Reference were;  To what extent was the ‘Fully Functional Service Delivery model’ under STRIDES an appropriate and effective approach to achieve intended results? To what extent does this model link and ensure that the facility, human resources for health (HRH), service delivery and community components work together?  To what extent was the Performance Based Financing model under STRIDES an appropriate and effective approach for improving private / NGO sector service delivery and thereby increasing access and availability of RH/FP and CS services? What is the potential for scale up?  To what extent do reproductive health approaches and interventions implemented by STRIDES address the unique needs of youth within the 15-25 age group in Uganda in comparison to the other age groups? What are the unique factors affecting uptake and utilization of reproductive health Services within the 15-25 age group? METHODOLOGY This evaluation collected data from eight of the fifteen project districts selected from Uganda’s East, Central and Western regions. The evaluation also selected three control districts (one from each of the regions) to gain a comparative picture between STRIDES and non-STRIDES districts. Data limitations, 1 See Management Sciences for Health (2009) STRIDES for Family Health Annual Report, January - September 2009, Cooperative Agreement No. 617-A-00-09-00005-00. Kampala: MSH 2 For further clarity See; Health system strengthening - current trends and challenges. Executive Board 128th session, Geneva, 17-25 January 2011. (EB128/37). Available at: http://apps.who.int/gb/ebwha/pdf_files/EB128/B128_37-en.pdf vi however, affected the extent to which comparison could be made across selected indicators. The evaluation employed a mixed methods design. Qualitative data was collected through in-depth interviews and focus group discussions with key project stakeholders. Quantitative data was generated from the monitoring and evaluation data base of STRIDES, previous studies and reports, and the Ministry of Health’s Management Information System. KEY FINDINGS Fully Functional Service Delivery System Overall, the FFSD model was an appropriate approach to improve health services in the Ugandan context, given the systemic challenges that had for decades afflicted Uganda’s health services systems. USAID/Uganda’s STRIDES project aimed to direct improvements across a range of Uganda’s health system features more particularly — building the capacity of human resources, improving infrastructure, equipment, drugs and supplies, and staff motivation. This approach was based on the view that improving the quality of service delivery is a result of interrelated processes. The approach was also appropriate in as far as it utilized pre-existing health structures and community resources, and adopted strategies suitable to community needs and local contexts. The FFSD approach registered modest performance in key areas such as improving access and utilization of RH/FP services by pregnant mothers with; i) 38% received 4 ANC visits , but representing only an 8% improvement against the baseline but still well below the 60% EOP target; ii) 51.7% received 2+ doses of IPT, well short of 60% EOP target, but above 35% at baseline, iii) 42% live births registered in health facilities compared to 27% at baseline, but still well short of 60% EOP target and; iv) a cumulative number of women using implants 42% compared to 27% at baseline but still well below the 60% EOP target. In comparison to control districts, STRIDES districts, in fact, performed only better and primarily during only the last year of project implementation on some of the indicators (like deliveries in health facilities). Such ‘modest performance’ thus raises the question as to whether STRIDES’ reported improvements could have been as a result of other factors. However, STRIDES registered impressive (some being input-oriented) performance in many areas namely; i) 45% of the targeted villages in 15 districts had functional VHTs exceeding the 40% EOP target, ii) improvements in infrastructure –health facilities complying with national norms and standards increased from 15% at baseline to nearly 70% in PY5, iii) staffing levels where for districts like Kasese, staffing for health workers increased from 21% at baseline to 56% in Performance Year (PY)5—albeit, inability to retain staff in health facilities remained a critical challenge for many health facilities. The relative success of the FFSD approach appears to have been realized or more assured in the short and medium term, and may not project more sustainable, systems-based enhancements. Extraneous factors associated with fostering sustainable, health systems strengthening such as funding by government, infrastructure development, and harmonization of human resource policies and practices at local government level, were not fully addressed because they were primarily beyond the scope of both the project and USAID’s plan to influence Uganda’s actual health policy investments. Some of these constraints were not fully considered during STRIDES’ project design. STRIDES’ design features also overlooked in many cases unrealistic assumptions about the practical functioning of the Government of Uganda’s health systems and in particular, Uganda’s political will and resource availability to meet its obligations under STRIDES and the international commitments vii such as the Millennium Development targets on maternal and child mortality and Accra Agenda for Action 2008. Indeed, Government of Ugandan was not able to make investments expected for key components including infrastructure, staffing, motivation of health workers, and maintaining a consistent flow of drugs and supplies. Finally, what design features STRIDES had for promoting sustainability were never well articulated, and what implementation of sustainability plans was undertaken, was almost too late and, among the health facilities visited, there were those where services had stalled immediately after USAID/Uganda program funding ended. Effectiveness of Performance Based Contracting The evaluation’s findings revealed that the PBC model yielded mixed results. Starting nearly two years after the program’s award, between 2010 and 2013, PBC contributed to STRIDES’ district service targets. PBC’s contribution ranged from 6% to 11% —to the overall realization of selected STRIDES’ performance targets. The contribution of the 53 PBC contractors was well acknowledged though among VHTs and health facility workers. PBC contractors also reported that their capacities were strengthened leading to improved service delivery within their health facilities, and an increase of clients. Notwithstanding, STRIDES’s actual planning and realizing PBC-oriented services experienced a mixed record, possibly because neither STRIDES nor USAID provided particularly strong guidance about the actual features and methods as to how to pursue and develop PBC. Hardly 40% of the contractors achieved their quarterly targets while only about 50% achieved annual targets. The full application of PBC was not fully experimented under government health facilities during STRIDES’ lifetime as USAID may have anticipated. Addressing the Unique Needs of Youth STRIDES interventions amongst the youth were in response to the fact that Uganda’s youth continue to have limited access to and utilization of available RH/FP services, and existing services often do not address youths’ unique needs. STRIDES made considerable effort to address youths’ needs by training selected health workers (153) on handling the youth and through an extensive IEC campaign. Thirty five percent of 588 STRIDES supported health facilities had corners/spaces created to provide Young People Friendly Services (YPFS) against the 45% target. In spite of such efforts, this evaluation found neither statistical data to suggest that USAID’s STRIDES generated an increase in demand for services among the youth, nor, an increase in provision of youth friendly services in the facilities. While interventions designed to reach the youth were well intended, the greatest limitation lay with the program’s design and non-adaptive implementation; and absence of guidance from USAID/Uganda which itself had still not developed a stronger strategic commitment and practical, operational, and youth-sensitive programming principles, while USAID/Washington has only developed USAID’s Youth in Development Policy and programming principles as recently as 2012. In addition, the evaluation shows that young people’s unique needs were never properly investigated, understood and therefore not adequately addressed. Questions regarding the more appropriate location for YFS services – considering that a health facility may not be the best place for young people -, flexibility of access (for different categories of youth) and when services might best be provided needed to be considered more carefully. viii RECOMMENDATIONS Fully Functional Service Delivery System  USAID’s future health systems strengthening projects should include a considerably stronger component for the USG’s engagement and development diplomacy within projects’ design and at high levels such as with Parliament, MoH and Development Partners to ensure that project inputs are accompanied by commensurate/matching resource contributions from Government.  Programs for Health worker training should be more carefully considered and coordinated given expected, impending MoH staff deployment/transfers under local (district) governments.  As the close out of STRIDES draws close, USAID should discuss with STRIDES, MoH and districts to examine what measures could be put in place to promote the continued, enhanced performance of the health services to avoid a reversal of the gains made.  More realistic targets should be set taking into account constraints surrounding the implementation of a project of this nature particularly, limitations that the project could not address on account of being outside of its scope and that of USAID support. Performance Based Contracting  USAID/Uganda should better understand various models and applications of PBC and whether and how it could become a feasible approach to enhancing Uganda’s public health facilities. Given the potential for expansion of PBC revealed in this evaluation, Government facilities with private wings could be used as one point of entry, one more amenable to PBC arrangements. Better understanding the nature of the relationships that exist between contractors and government owned health facilities through community outreaches could also provide a basic starting point.  Provide timely feedback to contractors, and work with them to address identified gaps. In addition, consider within the project design a mechanism through which local actors’ and citizens, and not just the prime USAID recipient, could provide feedback for quality of services rendered. Targeting Youth’s Unique Needs  USAID/Uganda’s health office should provide adequate direction, including project design features, as to what ‘youth-effective’ programming could look like.  Future youth projects require developing a far wider appreciation of youth’s many obstacles, and articulating a wide range of measures needed to successfully mobilize the youth so as to improve their RH/FP service seeking behavior. Referencing and applying the youth programming principles found within USAID’s 2012 Youth in Development would be a reasonable start.  USAID—as well as DO3’s Implementing Partners like MSH—should study examples of other youth development initiatives that have been considered as successful, such as Naguru Teenage Information and Health Centre and Restless Development to learn from their approaches.  USAID and implementing partners need to develop a much more urgent, consultative approach to involving and earnestly engaging and listening to young people to meet their unique needs.  In the interest of promoting and honoring local solutions, USAID could pilot the use of performance based contracting for delivery of services to the youth. ix List of Acronyms and Abbreviations AMSTL Active Management of Third Stage Labor ENC Essential Newborn Care ANC Antenatal Care BCC Behavior Change Communication BEmNOC Basic Emergency and Newborn Care CAO Chief Administrative Officer CDFU Communication for Development Foundation Uganda CEmONC Comprehensive Emergency Obstetric and Newborn Care CS Child Survival DHMT District Health Management Team DHO District Health Officer EOP End of Project FFSDS Fully Functional Service Delivery System FP Family Planning GoU Government of Uganda HC Health Center HF Health Facility HMIS Health Management Information System IEC Information Education Materials IYCF/ENA Infant Young Child Feeding/Essential Nutrition Actions IP Implementing Partner IMAM Integrated Management of Acute Malnutrition LAFPM Long acting family planning methods LLIN Long Lasting Insecticide Treated Nets MCH Maternal and Child Health MNCH Maternal, Neonatal and Child Health M&E Monitoring and Evaluation MoH Ministry of Health MoU Memorandum of Understanding MoES Ministry of Education and Sports MSH Management Sciences for Health MSU Marie Stopes Uganda NGO Non-governmental Organization OPD Out Patient Department PBC/F Performance-Based Contracting/Financing PMCT Prevention of Mother to Child Treatment of HIV PFP Private-for-Profit PMP Performance Monitoring Plan PNFP Private-Not-for-Profit RH Reproductive Health UPMA Uganda Private Midwives Association UNFPA United Nations Population Fund USAID United States Agency for International Development VHT Village Health Team x STRIDES Intervention Districts in Uganda KALANGALA Eastern region: Bugiri , Kaliro , Kamuli , Kayunga , Kumi, Mayuge Western region: Kyenjojo, Kasese, Kamwenge Central región: Luwero, Mityana, Mpigi, Nakasongola, Sembabule, Kalangala xi List of Tables and Figures Figure 2: Percentage of SDPs complying with national Norms and Standards (PY1-PY5) .................................... 10 Figure 3: Percentage of Pregnant Women who received 4 ANC Consultations ...................................................... 12 Figure 4: Effects of FFSDS and PBCs on Uptake of Implants and IUDs ................................................................ 17 Figure 5: Trends in Access and Utilization of RH/FP Services: Contribution of PBC Contractors ........................ 18 Figure 6: Trends in Access and Availability of Child Survival Services overtime .................................................. 19 Figure 7: Health Workers trained in Youth Friendly Services ................................................................................. 23 Figure 8: Knowledge gain among trained Health Workers ...................................................................................... 24 Figure 9: Percentage of Targeted Health Units offering Young People Friendly Services (PY1-PY5) ................... 25 1 1. Introduction 1.1 Background and Context The Government of Uganda has often stated its commitment to improving Maternal, New Born and Child Health (MNCH), reflected in the numerous MNCH related polices, plans and guidelines that have been developed over the years including the National Health Policy 2010 and Health Sector Strategic Plan 2010/11 – 2014/15, the Road Map for Accelerating the Reduction of Maternal and Neonatal Mortality and Morbidity, 2007-2015, the National Child and Newborn Survival Strategy (2010/11 – 2014/15), the National Advocacy Strategy for Maternal, Newborn and Child health (2012- 2015), Newborn Health Implementation Framework: Standards for Newborn Health Care Services (2010), and more recently the GoU’s Sharpened Reproductive Maternal, New Born and Child Health (RMNCH) Plan (2013/14 - 2016/17). Though some notable improvements in MNCH indicators have been observed over the past few years, and in spite of years of substantial donor support, the country is not on schedule to achieve Millennium Development Goals (MDGs) 4 and 5 targets. For example, between 2006 and 2011, the maternal mortality ratio increased from 418 deaths per 100,000 births to 438 deaths per 100,000 (UBOS & ICF International Inc, 2012). The poor maternal health outcomes are the result of the low rate of ANC attendance and facility-based deliveries3 , high total and adolescent fertility rate (TFR and AFR), and still high unmet need for family planning (see Annex B-1). Infant and child mortality rates also remain unacceptably high - at 54/1,000 and 90/1000 live births (UBOS & ICF International Inc., 2012), well short of the MDG target of 31/1,000 and 56/1,000 respectively. Research, over a number of years and many sources, still indicate that a disproportionate burden of infant mortality occurs during the neonatal period, usually within a few days of birth (MoH, 2010; UBOS & ICF International Inc, 2012). Newborn mortality contributes to more than a third (38 percent) of all infant deaths and at least (40%) of deaths among under-fives (MoH, 2010, 2013). The main causes of neonatal death are prematurity, birth asphyxia or injury, and neonatal infections (MoH, 2008). An estimated 25% of all maternal deaths are also still associated with teenage pregnancies. USAID Uganda’s Country Development Cooperation Strategy 2011-2015 contributes to realizing Uganda’s improved health goals through its Development Objective 3 (DO3) aimed at improving the health and nutrition status in selected areas and population groups4 . For a number of years, even well before the current 2011-2015 CDCD, USAID has directed its support through various activities to improve access, availability and quality of MNCH services and adoption of healthy behaviors and practices. The STRIDES for Family Health program was one of such initiatives, implemented between January 2009 and January 2014 for a total budget of $48 million covering 15 districts of Uganda, supporting 588 health facilities (see Annex D-1), and providing health services to an estimated 5.6 million beneficiaries. 3 The low rate of facility-based delivery is attributable to a number of serious challenges women face in accessing the health facilities, including the limited availability of medicines and supplies. 4 At the time of authoring USAID/Uganda’s CDCS, the types of populations and areas where USAID would direct attention to improved health were not well defined. 2 1.2 The STRIDES for Family Health Program STRIDES for Family Health was a USAID funded program aimed at increasing the use of Reproductive Health (RH), Family Planning (FP), and Child Survival (CS) services at the facility and community levels in fifteen selected districts. STRIDES worked in partnership with the Ministry of Health (MOH), local governments (LGs) and civil society organizations to achieve the following specific objectives: (i) increase the quality, and provision of routine RH/FP and CS services at facility level; (ii) improve and expand access to and demand for RH/FP and CS services at the community level; (ii) strengthen supportive systems to advance the use of RH/FP and CS services. STRIDES was implemented by Management Sciences for Health (MSH) in partnership with Communication for Development Foundation Uganda (CDFU), the Uganda Private Midwives Association (UPMA), Jhpiego, and Meridian International (see Annex B-2). In addition, MSH was working with 15 districts5 , their communities and local organizations to increase contraceptive use, healthy timing and spacing of pregnancy (HTSP), reducing maternal and child morbidity and mortality, and creating a sustainable, scalable nationwide intervention by 2014. STRIDES used a multi-pronged strategy to achieve its objectives:  A health system strengthening approach, also known as the Fully Functional Service Delivery System (FFSDS). The definition of a FFSDS is mainly based on the Ministry of Health (MOH) guideline/norms and standards, and health system building blocks6 identified by the World Health Organization (WHO, 2010) (see section 2 for details).  Performance-based financing (PBF) with non-governmental organizations (NGOs) and private sector health providers to expand access to essential health services (see details in section 3).  Development of the management and leadership capacity of local institutions, enhance the clinical skills of individuals, and establish or increase community accountability for health. 1.3 Purpose of the Evaluation The purpose of this evaluation was to generate insights into the performance of this $48 million USAID Maternal New Born and Child Health (MNCH) intervention to determine the extent to which the core program strategies were effective in achieving expected results. Evaluation Questions 1. To what extent was the ‘Fully Functional Service Delivery’ model under STRIDES an appropriate and effective approach to achieve intended results? To what extent does this model link and ensure that the facility, human resources for health (HRH), service delivery and community components worked together? 5 STRIDES Collaborating Districts: Bugiri, Kalangala, Kaliro, Kamuli, Kamwenge, Kasese, Kayunga, Kumi, Kyenjojo, Luwero, Mayuge, Mityana, Mpigi, Nakasongola, and Sembabule. 6 http://www.who.int/healthinfo/systems/WHO_MBHSS_2010_full_web.pdf 3 2. To what extent was the Performance Based Financing model under STRIDES an appropriate and effective approach for improving private / NGO sector service delivery and thereby increasing access and availability of RH/FP and CS services? What is the potential for scale up? 3. To what extent do reproductive health approaches and interventions by STRIDES address the unique needs of youth within the 15-25 age group in Uganda in comparison to the other age groups? What are the unique factors affecting uptake and utilization within the 15-25 age group? In order to operationalize the above evaluation questions, the evaluators derived specific sub-questions given in the Evaluation Design Matrix attached in Annex J. 1.4 Evaluation Methodology The evaluation employed a mixed methods design. The evaluators collected data from 8 intervention districts namely: Kumi, Kamuli and Mayuge in the Eastern region; Mityana, Nakasongola and Sembabule in Central; and Kyenjojo and Kasese in the West. Three comparison districts were also selected to gain a comparative picture between STRIDES and non-STRIDES districts. These were Namayingo, Nakaseke and Kyegegwa in the East, Central and West respectively. Factors including representation of different ethno-geographical characteristics, scope of activities implemented under STRIDES support and district performance in health services (see MoH 2013) were taken into account during the selection of the districts. The evaluation collected Qualitative data through in-depth interviews and focus group discussions with key project stakeholders including health workers within health facilities, key officials from the Ministry of Health, district health staff, implementing partners, community, male and female consumers; the youth and key informants at the central and district levels. The evaluation team conducted over 50 individual in-depth interviews with such informants; as well as 65 FGDs and 18 group interviews with selected participants. Quantitative data was primarily derived from the monitoring and evaluation data base of STRIDES as well as from previous studies and reports including health facility surveys conducted by STAR-EC and STAR-E LQAS. These various data sources were triangulated to arrive at a better understanding of STRIDES’ strategies and achievements. For details of the methodology, including study participants and tools used see Annexes C, J and K. 1.5 Limitations This evaluation faced some limitations, including consistent data availability. Whereas some secondary data exist for project districts at different points in time (2009, 2012, and 2013), much of it was not comparable because either different indicators were assessed or different measurements were undertaken. Closely related to the above, whereas STRIDES tracked monitoring data on its PMP￾specified outcome indicators, the current evaluation also sought data on process indicators relating to improved health system functionality. Data on these were not systematically kept by STRIDES since they were not part of their PMP. As a result, the evaluators were in some cases not able to present actual numbers or proportions of indicator-associated progress realized as evidence of the effectiveness of the FFDS or PBC approaches. Whereas the evaluation attempted to compare STRIDES and non-STRIDES districts as control groups, we were nonetheless unable to get satisfactory insight between the two sets of districts owing to non￾availability of statistical data on comparable indicators. 4 2. Appropriateness and Effectiveness of the Fully Functional Service Delivery Model 2.1 Introduction This section focuses on the fully functional services delivery (FFSD) model used by STRIDES. FFSDS is based on the WHO systems strengthening approach that particularly places emphasis on “initiatives and strategies that improves one or more of the functions of the health system and that leads to better health through improvements in access, coverage, quality, or efficiency”7 . 2.2 Expectations from the FFSDS Approach According to the STRIDES Program Description document8 , USAID/Uganda envisioned that by the program’s end, targeted health facilities in the 15 selected districts would be fully functional and delivering quality, integrated RH/FP and CS services. USAID/Uganda designed the STRIDES project in response to the various problems and challenges that characterized Uganda’s health sector at the time. According to the STRIDES program description document, there was a high unmet need for FP. Only 19 out of 677 health facilities surveyed at baseline offered permanent, long acting FP methods and another 36 offered long acting but not permanent methods such as implants and intra-uterine devices (IUD) (STRIDES Baseline Study, 2009). The report further noted that ‘even the “best” performing districts had low rates of contraceptive use, deliveries under skilled attendance, of antenatal care (ANC) attendance, and of child health’. Low service utilization by the youth was also identified. Other challenges noted included poor quality and functionality of health systems and limited services, inadequate management capacity ranging from the use of the HMIS for work, and resource planning, to drugs, commodity and equipment management. These are the challenges that the FFSDS was meant to address. 2.3 Adaptations to the FFSDS Model STRIDES’ conceptualization of the FFSDS was mainly based on the Ministry of Health (MOH) guidelines/norms and standards, and health system building blocks identified and introduced by the World Health Organization (WHO, 2010). As such, STRIDES activities were designed to provide directed assistance to all levels of the health system - community, health facility, and district leadership - to improve health care access, quality, delivery, and health outcomes. Whereas the FFSDS was initially meant to address all the health care system elements, interviews with the STRIDES senior management team indicated that the conceptualization of the FFSDS model was rethought mid-way into STRIDES. This rethinking resulted from the realization that some of the elements of a fully functional service delivery system were beyond STRIDES’s mandate (i.e., manageable interest) and depended upon more active, Ministry of Health action. Vital components such as investing in infrastructure development, staff motivation, and adequacy of drugs were deemed to be the Ministry of Health’s primary responsibilities. STRIDES could not guarantee that it could improve these and STRIDES’s design did not state that USAID/Uganda’s health team, perhaps in collaboration with other USG and Ugandan health actors, would need to actively promote stronger development diplomacy for realizing GoU health objectives. STRIDES’s mandate/agreement, for instance, only allowed it to refurbish or renovate health facility buildings but not to construct new ones. This therefore limited the extent to which STRIDES could improve health facility infrastructure, as in some health 7 Health system strengthening - current trends and challenges. Executive Board 128th session, Geneva, 17-25 January 2011. (EB128/37). Available at: http://apps.who.int/gb/ebwha/pdf_files/EB128/B128_37-en.pdf 8 See Project Description Document (undated) CA 617-A-00-09-00005-00 Reproductive Health, Family Planning and Child Survival services. ATTACHMENT B. 5 facilities, existing infrastructure was almost collapsing and needed completely new structures. Similarly, determination and payment of salaries for health workers is the GoU’s primary responsibility and STRIDES could not especially influence or mitigate this aspect that critically touches on health workers motivation and performance9 . These factors led to some narrowing of the scope of FFSDS components to which STRIDES directed its efforts. 2.4 Appropriateness of the FFSDS Model The evaluation findings indicate that the FFSDS model was an appropriate approach to improve health services in Uganda. This owes to Uganda’s health system which for over three decades has been characterized by systemic dysfunctionality, poor infrastructure, inadequate and unmotivated health workers, frequent drug stock outs, and inadequate financing, among other challenges. In light of these multi-faceted challenges, USAID/Uganda considered it appropriate, if also ambitious, to employ a holistic approach (FFSD), directing measures to improve all system components. The FFSDS could also be judged as having been consistent with the broader thinking of the time, since it was rooted in the health systems strengthening approach promoted by WHO and already embraced (at least in writing) by the Government of Uganda. By adopting a FFSDS approach, STRIDES enabled stakeholders to utilize and strive to strengthen existing community and district structures, namely; Village Health Teams (VHTs), health facilities, health workers, district teams and private sector health providers. Utilizing the already existing structures and resources is of course often less costly and has more potential for sustainability, such as VHTs who in particular, provide a first line of response for community residents in need of health services. The approach did not seek to develop new or parallel systems. Instead, it strengthened those already in place. For instance by Year 5, STRIDES had trained a total of 2,062 health workers in Family Planning, 764 in Reproductive Health, and 1,223 in Child Survival, training an estimated 600 health workers/annum. The beneficiaries included both facility based health workers as well as VHTs. These trainings were appropriate since they targeted pre-existing health workers and addressed existing needs. STRIDES also endeavored to maintain the appropriateness of the training modalities. When after the first set of trainings it was realized that trainings were taking health workers away from their stations, STRIDES started to offer on the job trainings. This, of course, had its own limitations in a sense that only few health workers could be trained at a time, on the other hand, the time health workers spend away for ‘classical training’ is well-known and often criticized. In addition, the FFSDS approach, in principle allowed collaborating with and leveraging other existing interventions, including other USAID supported projects, as well as a range of civil society and private sector service delivery interventions. All the above features made the FFSDS approach consistent with the existing needs, resources, systems and aspirations of the Ugandan health care system, as well as the internationally upheld principles or norms of pursing effective, institutionally-wide health outcomes. However, as already noted earlier, improving the level of health facilities’ functionality anticipated by STRIDES could not be fully realized given that the Government of Uganda, in spite of national policy 9 This is not to forget other interventions by STRIDES such as trainings and equipping of facilities that could have contributed to staff motivation. 6 commitments, continued not to make health sector investments expected by USAID and other development partners. While government made an effort by recruiting more health workers and slightly improving their pay, these challenges continued to afflict health centres in the STRIDES districts and the rest of Uganda. In light of this, USAID’s designers of STRIDES either overlooked or made unrealistic assumptions that the Uganda government would match the resources and the time expected to ensure a fully functional delivery system. Consistent with USAID programming policy, STRIDES did not make direct investments into recruitment of staff for districts or health facilities and while a critical component of a fully functional service delivery system, STRIDES interventions were not instrumental in improving health staff recruitment itself. Rather than STRIDES’ contribution, there was improvement in the staffing and skilling of staff attributed more to other government’s intervention. Some of STRIDES’ districts with support from the Ministry of Health recruited new staff in health facilities. In Kasese district, staffing for health workers increased from 21% by the time when STRIDES came to the district, to 56% by the time of this evaluation. Recruitments were also confirmed in Kyenjojo districts which boosted staffing with the implementation of the USG’s new “Saving Mothers, Giving Life” (SMGL) project. This SMGL project also recruited five medical doctors and 15 mid wives and paid their salaries during STRIDES’ period. During STRIDES’ duration, health workers cited an increase in the number of deliveries taking place with skilled assistance including those who received caesarian sections. While USAID devoted as much as $48 million to STRIDES (see Annex D-2 for breakdown of STRIDES expenditures), it could be argued that for such a short project period, it was still overly ambitious for USAID to expect to make 588 health facilities fully functional (ranging from HCIIs to Hospitals)10. Reducing the number of facilities and possibly districts, and more importantly, prioritizing which ones to focus on could have made it possible to invest more resources in underfunded areas. STRIDES’ design and implementation features did not seem to have anticipated enough to prepare for how improvements to districts’ services would be sustained. In the STRIDES Consolidated Sustainability Action Plan (2013) which contains the district sustainability plans, it is acknowledged that “sustainability was not explicitly addressed in the STRIDES project’s initial design, though it [was] implicit in the key strategies”. This evaluation found that just after a few months after phase out, districts were not equipped to continue to provide services at the level and quality as reached during the course of STRIDES implementation—for example, to ensure supply of nutritional supplements, to sustain outreaches, to facilitate VHTs, and to ensure availability of HMIS forms. Some of these cessations of services may have been avoided through more serious engagement between STRIDES, MOH, districts and USAID. Whereas STRIDES had recently (in Performance Year 4 and 5) worked with districts to develop sustainability plans, this process should have started earlier. By design, a sustained engagement involving STRIDES, USAID and MoH would have been essential for promoting broader systems and services sustainability. Looking at the district sustainability plans, the evaluation team noted that the districts may not afford to sustain some of the activities listed without continued external funding, although STRIDES leadership reported that the spirit underlying these plans was to identify the activities that districts thought they could more easily sustain. 10 Of the total 588 supported facilities, HCIIs were 356 (61%); HCIIIs were 188 (32%); HCIVs were 27 (5%); and hospitals were 17 (3%). 7 Other assumptions and factors affected FFSDS implementation. Whereas STRIDES had planned to provide rewards to health facilities that met targets (a form of performance based financing), it only did this once after USAID’s SDS project started in 2010. STRIDES had planned to establish private wards at hospitals and HCIVs to generate funds that would top up health workers’ remuneration. It was then learnt that MSH/STRIDES agreement with USAID did not allow for construction of new wards. With respect to new constructions and refurbishments, USAID actors stated that they expected a World Bank supported project would reconstruct and refurbish hospitals. However the project experienced delays and did not do much during the lifetime of STRIDES. Thus many of the hospitals under STRIDES remained in poor physical state. The envisaged FFSD approach therefore could not work well realizing that different, critical health system elements could not be improved. STRIDES nevertheless mitigated some of the above limitations by trying to promote an innovative approach to collaborations with local and international private sector actors, through which it raised, through a corporate social responsibility component, an additional $11 million, both in kind and in cash, used primarily to support health facilities’ equipment and supplies. STRIDES also entered into collaborations with local Ugandan companies which supported specific health infrastructural and equipment improvements, including buying of ambulances, and erecting waiting shades at some health facilities, such as the one at Mpigi HCIV built with financial support from Uganda Baati Ltd. A review of program records confirms that STRIDES FFSDS-oriented approach to systems strengthening and services delivery reached as many as 588 public health facilities11 (see also Annex D￾1) serving an estimated 5.6 million of beneficiaries in 15 districts. 2.5 Effectiveness of the FFSDS Model Effectiveness of the FFSDS model was evaluated in terms of the extent to which the model achieved its goal of making the different health system components work properly, to deliver quality, accessible, comprehensive health services, and meet clients’ needs. 2.5.1 Functionality of Health Services STRIDES contributed to the functionality of health services by supporting improvements in infrastructure, equipment, supplies, drugs, training health workers, support supervision and strengthening facility relationships with communities. Infrastructural Improvements With respect to infrastructure, all the 15 districts selected for STRIDES support had at baseline health facilities with inadequate infrastructural facilities, with an average compliance with national infrastructural standards rate of 48%, this being as low as 10% for health facilities in Kalangala district and 20% for Kamwenge (STRIDES Baseline Study Report, 2009). To address these shortcomings, STRIDES supported the renovation of 10 selected health facilities in 6 districts (Bugiri, Kamwenge, Kasese, Luwero, Mayuge, and Nakasongola) at a total cost of UGX 793,717,739 (approx. $317,487). STRIDES made renovations or refurbishments on health facility premises such as wards and laboratories, accompanied by installation of solar power and piped water. Installation of running water 11 These included 17 hospitals, 27 HCIVs, 188 HCIIIs, and 356 HCIIs. 8 in facilities such as Rwesande HCIV meant that these facilities could now make use of hand-washing facilities to support infection prevention and support to quality improvement efforts. Other health centers where renovations of premises were done are shown in Annex E-1. However, renovations and refurbishments were probably inadequate given the massive need for infrastructural improvements in the 15 districts. STRIDES Year 5 Annual Report indicates that the percentage of service delivery points complying with national norms and standards had increased to nearly 70%. This positive improvement, however, certainly includes the infrastructural improvements supported by other agencies or initiatives. Nevertheless, in spite of STRIDES’ contributions, our findings show that the needs for infrastructural improvements remain in most of the health facilities where many still lack adequate staff housing, transport facilities, power, and ward rooms. Some health centres reported that because of lack of staff houses, some staff posted there had not reported, while others had reported and left. In other cases, lack of staff housing was reported to contribute to health worker absenteeism. Equipment STRIDES provided health facilities with equipment reportedly worth more than $5,523,000. This included those that STRIDES procured at a cost of about $1,072,000 and equipment received through STRIDES’ collaboration with IMEC, valued at about $4,451,000. STRIDES provided health facilities with general equipment, reproductive health and new born equipment, and child survival equipment to 433 out of the 588 health facilities. USAID provided no specific targets, or even financing considerations, for STRIDES to provide equipment, but STRIDES reported that it would have been ideal to reach all the health facilities. The equipment distributed also depended on the health facility’s level. Annex E-3 shows the number and type of equipment distributed by district. Health workers interviewed reported that the equipment received was very useful in their work and had helped to improve the quality of care provided. The support helped a lot …for example to monitor pregnant mothers using the Pantograph, and because of this, together with the quality improvement programme we have implemented; we have not had any maternal death for the last two years. In-charge, Rwesande HCIV, Kasese District Drugs and other Supplies Although STRIDES did not have full responsibility for providing drugs, the project provided some drugs and other supplies to fill shortages where they were identified. Such stop gap supplies were reported to be often useful to offset the shortages health facilities experienced. Still, when the evaluation team visited some health facilities, they had some drugs but were missing others. Kyankaramata HCII (Kyenjojo district), for example, did not have emergency contraceptives, POPs (Progestrogen Only Pills) or some basic antibiotics. This facility had Coartem tablets, but no Paracetamol and no Septrine. DOH and STRIDES staff also reported delays attributed to National Medical Stores such as delivery of HIV testing kits. Thus, in terms of measurably improving the functionality of the health service delivery systems in STRIDES districts, regularly providing drugs and others supplies remained not fully realized, though this cannot not be blamed on STRIDES. 9 STRIDES also provided supplies such as Mama Kits, mosquito nets, family planning commodities, jerry cans with dispensing taps, metallic cups and water purification tablets to ANC facilities to improve the delivery of reproductive health, family planning and other services. For instance, during Year 4, STRIDES in collaboration with UHMG distributed a total of 1,400 jerry cans with dispensing taps, 9,075 metallic cups and 1,440 packets of water purification tablets to ANC facilities in Bugiri, Mayuge, Kaliro, Kalangala, Kyenjojo, Kamwenge and Kasese districts. During Year 4 and Year 5, STRIDES distributed a total of 110,757 Long Lasting Insecticide Treated Nets (LLINs) free of charge. STRIDES intended for these supplies to encourage expectant mothers to complete four recommended ANC visits and increase uptake of IPT to reduce incidence of malaria episodes during pregnancy. During Year 5, STRIDES also contributed to a stable supply of Family Planning commodities by monitoring of stock-outs and internal re-distribution of these commodities from facilities with surpluses to those that were stocked out. The redistribution of FP commodities, it is believed, contributed to increase in the number of new FP users by 5% and repeat users by 38% between Year 4 and Year 5 reported in their 5th year annual report. This support notwithstanding, many still reported shortages of supplies for example for Family Planning: … We have the capacity to offer quality FP services but sometimes we run out of stock … can take some time when we do not have implants and that definitely affects our capacity to offer quality service. Also for pills they can be out of stock because they are supplied by NMS. For the injectables and Depo-Provera we have them in plenty. Interview, In-Charge, Butiiti HCIII, Kyenjojo District Similar shortages of certain types of FP methods were reported in Kamuli and Kumi districts. Similarly, STRIDES’ reports indicated that shortages of vaccines for immunization affected DPT3 achievements in remote districts such as Sembabule and Kalangala. Health Management Information System (HMIS) Reporting STRIDES trained staff in HMIS reporting and provided printed HMIS Reporting forms. STRIDES support not only improved the extent/coverage and timeliness of reporting but its accuracy as well. By Year Five, the percentage of health facilities making timely reporting to health sub-districts/districts was 88%, having increased from 72% at baseline, and those reporting timely to MoH were 90%, up from 78% at baseline as shown in Annex E-4. However, some health facilities lack record-keeping staff which partly contributes to poor reporting. Support Supervision STRIDES supported routine supervision, performance appraisal and recognition strategy (SPARS 5) in the supported health facilities. This was in response to a pre-existing situation where support supervision had been irregular of poor quality (STRIDES Baseline Study 2009). During the supervision visits, STRIDES restocked health facilities with Essential Medicines and Health Supplies manuals and dispensing logs. The support supervision teams also mentored and trained health facility staff in making orders for drugs and supplies; handling data, filling in HMIS forms; and providing hands-on support on issues of infection control and quality improvement. STRIDES conducted some of the quarterly supervision jointly with the MoH resource centre staff. STRIDES also participated in district integrated 10 support supervision and provided technical guidance during extended DHMT and micro planning meetings held in the collaborating districts. Health workers interviewed stated they valued the support supervision and mentorship STRIDES provided to health workers, some, however, , felt that the support supervisions were led more from the STRIDES teams from Kampala rather than the districts which limited opportunities for cultivating broader district ownership and responsibility. In many cases, the health workers thought that STRIDES support achieved success in making services functional. We used to receive some cases and you would have nothing to do about them because we did not have equipment and required skills. We used to refer many of them to Masaka regional referral hospital and yet most of the people were poor and could not afford the cost of transport. But these days you identify someone, you give him what is required and he comes back when he is happy. … you get satisfied … You would find cases where you had to send someone back home without getting the services. Although we had the knowledge to do certain things, we did not have what to use but now we have both. FGD with Health Workers, Sembabule HCIV According to the STRIDES Year Five annual survey, nearly 70% of service delivery points complied with national norms and standards, surpassing both the annual and EOP targets of 32% and representing improved performance compared to Performance Year Four (PY4). A service delivery point is counted as having complied with national norms and standards when at least 80% of the individuals observed have been provided with adequate counseling. Adequate counseling means: all methods are discussed with the clients; jobs aids are used in the counseling session; and FP commodities are in stock. Certainly one limitation of the survey data referred to here is that it is based on data from a one-time visit to the health facilities and does not take into account what happens most other times, for instance when FP commodities are out of stock as already reported to have often been the case. Figure 1: Percentage of SDPs complying with national Norms and Standards (PY1-PY5) In many cases, however, In-charges of some facilities especially HCIIIs and IIs felt that their facilities did not have the capacity to deliver quality RH, FP and CS services. They reported stock outs of drugs and supplies, inadequacies in staffing, poor infrastructure, lack of equipment, lack of power and other 17 17 10 58 68.9 17 22 28 32 32 0 10 20 30 40 50 60 70 80 % SDPs complying with national standards Actual (%) Target (%) 11 challenges that made their services not functioning well. Whether actually, technically meeting ‘national norms and standards’ or not, most health facilities were not very sensitive to unique gender-associated needs particularly where many had mixed wards for males and females. Annex E-5 also shows results from STAR EC Health Facility Assessment in four of the STRIDES Eastern Uganda Districts, which indicate very low levels of conformity against set norms and standards. 2.5.2 Making Services More Accessible STRIDES supported health centers to take services such as immunization, ANC, HIV testing, maternity and cervical cancer screening nearer to the users through integrated community outreaches. Clients who tested HIV positive were started on option B. In some facilities such as Kyankaramata HCII (Kyenjojo district) where such services were provided, informants reported that outreaches helped to solve the problem of men not escorting their wives for services—because services are taken to where such women live. However, whereas the idea was to visit each outreach site once a month, many health facilities were and are still not able to do this, due to lack of necessary logistical and human resources. STRIDES trained community health workers such as VHTs and other volunteers in a range of skill areas to enable them participate in service provision. STRIDES used VHTs as the most reliable structure for continuous provision of education, community mobilization to build demand for health services. As such, they were trained and equipped with bicycles, gum boots and other basics to enable them work in the community. STRIDES also provided bicycles to nearly 1,600 VHTs across the 15 districts for facilitating their transport for providing community level, service delivery. In addition, VHTs STRIDES replenished tool kits for reportedly more than 6,550 VHT members during quarterly facility based VHT meetings in 170 health facilities in 11 districts. In some facilities, such as Balawoli HCII in Kamuli, VHTs operated tri-cycle ambulances which bring patients from the community for treatment or for onward referral. Ambulance services, both vehicles and motorcycle transport provided to pregnant women were well appreciated by community members in Kyenjojo district. By end of Year Five, 45% of targeted villages in the 15 districts had functional VHTs, exceeding the end of project target of 40% as shown in Annex E-6. Despite their contribution, sustaining the involvement of VHTs is closely associated with offering adequate motivational incentives. It was reported that their participation in meetings and health work often reduces considerably when motivational incentives are no longer provided. By Year Five, a total of 503 service delivery points were providing Family Planning counseling or related services12, this achievement exceeded the end of project targets by 98%.STRIDES support also helped more health facilities provide modern contraceptive methods. By end of Year 5 of STRIDES, 96.8% of service delivery points (SDPs) were providing at least one modern contraceptive method. This achievement exceeds the annual and end of project target by more than 5%. However, given the stock￾outs of FP commodities already reported, many health facilities often had one method and not others, which limited the range of choices available to clients. In addition, 74% of health facilities (HC III and above) offered long acting methods (LAM), exceeding the annual and EOP targets of 60% by 23%. However, the facilities offering permanent methods were hardly 25%, falling well short of the annual and EOP 50% target. 12A service delivery point is considered to provide the service when the following conditions are all met: 1) at least one staff member who has been trained in the service; 2) the required equipment is available; 3) the SDP has offered the service in the last 3 months; and 4) contraceptives have been in stock for at least 2 of the past 3 months. 12 2.5.3 Increasing Utilization of Services The utilization of RH, FP and CS services increased – including 4th ANC attendance, but health centers still reported a discrepancy between ANC attendance and those seeking skilled assistance at delivery. Uptake of services was slow, in some cases even falling (as in the case of ANC consultations), long after STRIDES’ project start. Only after STRIDES third year of performance did STRIDES’ service utilization data show increasing trends of service utilization for ANC4 attendance, as shown in Figure 2.5.2. Results for trends in other indicators are shown in Annex E-7. According to the results in the figures in Annex E-7, STRIDES did not realize improvements in majority of the indicators in STRIDES supported districts until Year 4. Indeed, the performance of the STRIDES districts for Year 4 was higher for almost all indicators compared to the non-STRIDES districts. However, the non-STRIDES districts also show an upward trend in performance for most indicators during Year 5 (see Annexes E-7), which casts questions as to whether the further increase in achievement in Year 5 for the STRIDES districts was due to STRIDES support or other general factors that probably applied to the country as a whole. According to STRIDES own analysis of performance for Year Four, district performance analysis for deliveries assisted by a skilled worker showed that only 9 out of 15 districts improved in performance from Year Three. The districts that improved included Kamwenge (69%), Kyenjojo (53%), Kaliro (37%) and Sembabule (33%), and the least improved districts are Luwero and Kayunga which actually declined by 11% and 9% respectively. According to the best performing districts i.e. Kamwenge and Kyenjojo, strong performance was actually attributed to SMGL partners’ (relatively input-intensive) interventions including but not limited to; the provision of mama kits, provision of community ambulances to transport expectant mothers to health facilities, provision of motivational incentives to health workers and implementation of STRIDES/Midas Touch Voucher system that entitled pregnant women to free Comprehensive Emergency Obstetric and Newborn Care (CEmONC) services. No reasons were given for the declines registered in Luwero and Kayunga. Figure 2: Percentage of Pregnant Women who received 4 ANC Consultations Intervention Districts Control Districts 13 While starting only as late as PY3, STRIDES appears to have contributed to increase in ANC attendance and IPTp2 by cultivating demand side interventions that attracted mothers to attend ANC. These interventions included mobilization of pregnant women for ANC, provision of LLINs at health facilities and provision of demand side driven incentives such as hygiene kits and shoes. According to Figure 2.5.2, however, ANC attendance also experienced comparable increases in Control Districts. 2.5.4 Quality Improvement and Overall Quality of Care STRIDES supported 46 health facilities in 10 districts to expand delivery of high-impact practices such as Partograph use, active management of third stage labor (AMSTL) and essential newborn care (ENC). STRIDES also trained health workers in these 10 districts in Quality Improvement (QI) and formed QI teams which at the health facility level, monitor quality by identifying problems and finding solutions. Most health facilities reported having fairly functional QI teams that, by project’s end, were meeting regularly to address quality issues. STRIDES trained health workers to provide family planning services (both short term and long term acting methods), Comprehensive emergency in obstetric care, Basic emergency of obstetric care and New born care. In some cases, STRIDES trained even health workers at HC IIs to provide both short term and long term acting methods, however, some health facilities reported not receiving any FP or other supplies for this – and so at best they could only provide counseling and referral of clients to other health facilities. Other trainings provided by STRIDES to health facility staff included; Integrated Management of Acute Malnutrition (IMAM)—including the use of tools for assessment of malnourished children, data management, community outreach by VHTs, provision of youth friendly services, cervical cancer screening and waste management. Annex E-8 shows the number of health service providers and community members trained by STRIDES in different skill areas. Evaluation findings indicated that staff transfers affected some health facilities whose staff had been trained. For example Rwesande HCIV in Kasese District had all staff trained by STRIDES transferred and none of them was present at the time of this evaluation. Staff turn overs were also singled out as one of the factors that undermined the expected benefits from the trainings. Even in cases where only one or two health workers left a health facility, the effect of this was heavily felt if these were the same health workers that had received STRIDES training. It should be remembered that STRIDES would train only about two health workers from each health facility for each type of training. Overall therefore, the training of health workers by STRIDES would have been more useful if there had been improved opportunities or mechanisms to keep them on their jobs. There were other factors that affected the effectiveness of the trainings, including health workers in some cases lacking the tools and equipment to put their newly acquired skills to use. Some informants from health facilities in Mayuge, Kumi and Kasese district pointed out the key challenges they faced in providing quality care, including lack of a functioning laboratory and antibiotics (Bugoye HCIII, “The challenge we have is the big staff turnover …. Most of the health workers that we trained left their facilities after one or two years and we have since recruited new ones without these skills. … This happens because people keep moving out of the district either for further training or in search for greener pastures.” STRIDES Focal Person, Nakasongola district 14 Kasese); client overload (Atutur hospital, Kumi District, Kasese and Mayuge districts); and lack of privacy due to mixed gender wards (Mayuge). In other cases, health workers cited the lack of equipment such as sterilizers which compromises the quality of care. Others reiterated that they are not able to put into practice the knowledge and skills gained from STRIDES trainings due to lack of the necessary tools and equipment. 2.5.5 Comprehensiveness of Services STRIDES support enabled some health facilities to introduce new services that before were not provided. These include nutrition assessments, long term methods of family planning, and in some cases cervical cancer screening and ultra sound scans. In this respect, STRIDES was effective in increasing the range/scope/comprehensiveness of services available at health centres. Services such as the ultra sound scan at every 4th ANC visit were credited for attracting mothers to complete ANC attendance as recommended. In some cases, STRIDES also introduced complete nutrition units and youth friendly services. Annex E-9 shows the proportion of facilities providing some of these services at baseline and at evaluation. These data show that for instance, the proportion of health facilities providing family planning counselling or services increased from 104 at baseline to 503 by Project Year 5, exceeding the target of 254. Targets were also exceeded for facilities offering long acting family planning methods, and those offering any contraceptive method. However achievements were below target for facilities offering youth friendly services and permanent family planning methods. Certainly, the data in Annex E-9 showing the proportion of facilities stating that they were providing different services as at end of Year Five mask the fact that these services were sometimes not operational due to different reasons such as stock outs of supplies and drugs, temporary absence of the skilled providers, and in the case of youth friendly services – due to low turn up of the youth, as will be discussed ahead in this report. Functionality of health facilities was also augmented through outreach services conducted at the health centres by some contractors engaged by STRIDES under the PBC Arrangement. For instance Marie Stopes conducted outreaches to provide some of the long term acting family planning methods such as sterilization and vasectomy. This arrangement was reported in several facilities – including HCIIIs - levels that would ordinarily not offer such services. In this way, STRIDES enabled service users to access services that they would otherwise not get at their health facility. 2.5.6 Sustainability of Services Some of the informants were optimistic that they would sustain some of the STRIDES-supported elements of health care improvement: Others however pointed to scenarios where services such as management of severe cases of nutrition have already stalled, even only a few months after the cessation of STRIDES support. Both STRIDES project staff and district stakeholders concurred that the sustainability plans should have been Yes we shall sustain them [services] like cancer screening we have been getting the supplies from the government and not from the program. So I believe the program gave us the knowledge which they cannot take away. As for the outreaches, the government provides us with PHC funds and I hope we shall continue using part of that money to conduct community outreaches. The VHTs who are in the community will continue to work even without the support of STRIDES because before it came the VHTs were already working. The program just supported them to do additional activities. Health workers, Butiti HC III, Kyenjojo. 15 prepared earlier in the life of STRIDES, or better during its design, rather than at the end. 2.6 Conclusions on FFSD To a larger extent, the STRIDES intervention, if only after two or three years of performance, improved the functionality of nearly two/thirds of the health facilities. The increase in the demand for services including RH/FP, CS and maternal health illustrates improvements both in terms of quality and quantity of services delivered. It also bears testimony to the effectiveness of using grassroots structures such as the VHTs in improving access and demand for services. Collectively, improvements in human resource skills, service delivery, IEC Materials, HMIS reporting, supplies for family planning, and links with the community contributed to the functioning of the service delivery system. However, this evaluation was unable to determine which aspects of FFSD systems strengthening became most valuable to improving health outcomes. Furthermore, large gaps remained in infrastructure, equipment, supplies, financing and staff motivation; male involvement in ANC and provision of gender and youth friendly services (further elaboration in section 3). As cited earlier, FFSD would have been more successful if it received matching resource support from the Government of Uganda. STRIDES support towards the development of health workers should have better corresponded with the GoU’s carrying through on policy promises for providing matching support from government including staff transfers, performance management and motivation. The same applies to investments made in the area of equipment, supplies and infrastructural improvements, though STRIDES itself, based on the finding that relatively few investments were made even under ‘general equipment’ may have provided deeper, broader equipment provision. The implication is that while government should have committed itself to meeting its obligations, the design of STRIDES should also have better provided for resource mobilization and sustainability considerations in case of government’s failure to meet its obligations. 2.7 Challenges and Unexpected Outcomes Turnover of key trained staff remained a key problem, made more adverse by the fact that for each type of training, in spite of training an estimated 600 HWs/annum, STRIDES would have trained only about two staff from each selected health facility. This necessitated re-training of other staff and change of strategy regarding training: They did more on on-job training, whereby health workers did not have to leave their stations. However this demanded more time and resources since only a few health workers would be trained at a time. Shortcomings related to motivation of health workers, shortages of contraceptive commodities and vaccines, and poor infrastructure, all already discussed, in some of the health facilities undermined the comprehensive building of a FFSD. Unexpected Outcomes Improvements in service delivery resulted, in some cases in increased demand leading to client overload. Demand for services exceeded the available capabilities including staff and supplies. HC II facilities were demanding wards and labs and other facilities that should be for HCIII to meet HCII generated demands. Some are asking to be elevated to a higher status to be better positioned to respond. More encouraging, Kyenjojo district reported that the STRIDES support helped the district to move from 63rd position on the national league table to the 10th position. 16 2.8 Lessons Learned from Implementing FFSD Some of the lessons learned from implementing a FFSD approach are: i. Where holistic/system wide improvements are made, they can positively influence health service delivery. Critical, if not still well-understood relationships exist between information, demand for services, quality of services and consumer satisfaction. ii. Thinking beyond unnecessarily narrow design considerations and initial project funding boundaries should better enable such similar projects to raise substantial additional resources through more effective development diplomacy dialogue as well as other private sector actors and initiatives. iii. Project interventions towards improving a FFSD system requires a stronger appreciation of which elements of such a system are most valuable to support as well as matching resources from the government to advance the government’s mandate. 2.9 Recommendations on FFDS i. Future health systems strengthening projects should include a considerably stronger component for engagement and development diplomacy within STRIDES’s design and at high levels such as with Parliament, MoH and Development Partners to ensure that project inputs are more effectively delivered whether accompanied or not by commensurate/matching resource contributions from Government. ii. Future interventions using FFSD should ensure more tangible commitments from central and local governments to meet GoU counterpart contributions to agreed inputs, support and sustain changes; as well as more effective, USG policy dialogues for promoting stronger government accountability to deliver on such commitments. iii. While the close out of STRIDES is still recent, USAID should discuss with STRIDES, MoH and districts to ensure that a mechanism is in place to guarantee continued performance of the health services previously supported by STRIDES, and to avoid a reversal of the gains made through STRIDES support. This should have happened earlier but is nevertheless still possible, iv. Programs for Health worker training should be coordinated with their deployment/transfers to guarantee that those trained will be of service to their stations or others judged to be in need of similar skills. 17 3. Effectiveness of the Performance Based Financing Model 3.1 Introduction This section describes the evaluation findings regarding the appropriateness and effectiveness of the Performance Based Financing Model (PBC), providing a description of the model, an assessment of effectiveness, trends in service delivery, challenges faced and recommendations going forward. 3.2 Description of the PBF Model The PBC model involved a contracting arrangement between STRIDES and a wide range of contracting agencies namely; Non-Governmental Organizations, private health facilities and drug shops. All payments under PBC were based on the achievement of pre-determined performance targets set by STRIDES, and deemed to contribute to the overall realization of STRIDES program targets. STRIDES hired a total of 53 organizations between 2010 and 2013. For a full list of PBC contractors and their areas of coverage, see Annex F-1a and F-1b. While the Scope of Work varied among the agencies contracted, the performance indicators over which they made contribution included; i) increasing the number of women using family planning; ii) number of clients receiving counseling on family planning; iii) number receiving 4th ANC visit; iv) number of live births delivered; v) number of children receiving Vitamin A supplements; vi) number of children receiving DPT; and v) number of young people who receive RH information, among others. Contractors were required to submit quarterly reports and afterwards assessed and assigned a performance score using HMIS form 105 and STRIDES narrative reporting template. 3.3 Effectiveness of the PBC Model The evaluation revealed that the PBC model made substantial contribution between 2010 (PY 2 and 2013 (PY 5), towards contributing to STRIDES and district targets in relation to FP, RH and CS services as indicated in figure 3.3 below. Figure 3: Effects of FFSDS and PBCs on Uptake of Implants and IUDs Adapted from Management Sciences for Health, 201313 13MSH 2013.Health Systems Strengthening and Integration of Family Planning/Maternal, Neonatal and Child Health 18 When PBCs were introduced (see Figure 3.3), even if late in the third performance year, the uptake of Intra Uterine Devices and implants increased significantly. Discussions with VHTs, health facility workers and the PBC contractors indicated that PBCs’ contribution is well acknowledged. Reference is also made to the increased uptake of family planning. Notably, outreaches involving community meetings with health educators helped to clarify FP misconceptions and also enabled access to needed counseling. However, further analysis shows that overall, about half of the contractors were able to achieve predetermined targets (see Annex F-2 and F-3) suggesting a mixed picture of the overall performance of PBC approach. 3.3.1Trends in Access and Availability of RH/FP Services The data in the figures below and Annexes F-4 and F-5 show the contribution of PBCs to the overall STRIDES Project performance using selected indicators in reproductive health namely: Ante natal care attendance, prevention of malaria in pregnancy, safe deliveries in health facilities, and use of long term family planning methods. Figure 4: Trends in Access and Utilization of RH/FP Services: Contribution of PBC Contractors % live births delivered from a health facility Cumulative number of implants and IUDs inserted Fig. 3.3.1 (and Annexes F-4 and F-5) shows PBC contractors’ contributions to STRIDES’ performance. Even with fewer facilities covered by Contractors, improvements were seen, even if modest, with respect to live births delivered in STRIDES-supported health facilities in the fourth Performance Year; number of implants and IUDs inserted to mothers for family planning; mothers attending 4 ANC visits - the percentage contributed by PBCs increased to nearly 11% in the 4th year (see Annex F-4). However, and not long after starting PBCs, the percentage contributed by PBCs to the overall STRIDES program fell to 6.4% in the final year (2013, PY5)—against an overall program output of 38%. The contribution of contractors though in outreach was not accounted for even though outreach results appear to have contributed to an increase in client turn up in public facilities. PBCs’ contribution reduced as soon as 19 PY5 because STRIDES renewed fewer contracts in the program’s final year, likely contributing to the reduction in PBCs’ r overall contribution to STRIDES program’s performance. 3.3.2 Trends in Access and Availability of Child Survival Services This section highlights PBCs’ contribution to STRIDES’ overall project performance in the provision of child survival services. Specifically, the section reports on number of children under 5 years of age who received 1st and 2nd doses of Vitamin A from USG supported programs. Figure 5: Trends in Access and Availability of Child Survival Services overtime As shown in figure 3.3.2, cumulatively, 123,000 children under 5 years of age received the 1st dose of Vitamin A while STRIDES cumulatively attained 1,692,000 with the overall project target being 1,816,000; an indication of strong performance though slightly below end of project target. PBCs also provided nutrition support to 53,000 children aged under 5 years in the second Performance Year. This increased to 140,000 in the fifth year of STRIDES while the number of children who at 12 months had received three doses of DPT vaccination was cumulatively 56,000 in the 5 Performance Year (see Annex F-5). 3.3.3 Factors contributing to the Success of PBC There were a number of factors that tended to explain the successes of the PBC model: The system of payment based on performance indicators and payment of bonus (worth 10% of the contractors’ annual budget) for exceeding set targets made contractors to consistently strive for greater performance and excellence in service delivery. Another factor that made PBC model register some success was the utilization of both local and international NGOs/service providers. This mix leveraged varying advantages including utilization of better technologies, knowledge of local needs and capacity to mobilize the local population. Use of service providers already operating in their localities—particularly Cumulative number of children under 5 who received 1st dose of Vitamin A from USG￾supported programs Cumulative number of children under 5 who received 2nd dose of Vitamin A from USG￾supported programs 20 hard to reach areas (for example Subi, serving in Islands in Mayuge District, St Fulemena in Kalangala Islands, and Good Samaritan around Lake Katwe in Kasese) was also another strength. This helped increase service coverage and outreach to-hard to-reach areas. This was in part a condition for contractors to serve hard to reach areas, as reported by some informants: By going to these remote communities through outreaches …. the number of our clientele immediately increased [they came for] services like [HIV] testing, ANC, among others. KII with the Director, Bachi Medical Centre, Mayuge District). The delivery of low cost services (due to heavy subsidization) also made services provided by contractors very attractive to the communities: [There was huge demand for FP services] … for example by buying a voucher of U.Shs 2,000 (0.81 US $) a woman would be entitled to an Intra Uterine Device insertion, review and withdrawal of an Intra Uterine Device—and this was affordable to many mothers. KII with the Director, Bachi Medical Centre, Mayuge district The Contractors worked with health workers within public health facilities in their catchment area to conduct outreaches; worked with VHT to mobilize and follow up patients; and also worked with District Health Offices for coordination and reporting of deliverables through the HMIS. These collaborations enhanced their ability to contribute to system-wide benefits. The PBC model made business sense for the contractors as it resulted in increased community awareness of their services, and enrollment of new patients especially following outreaches. Many clients have remained with the facilities after the end of the free services provided under STRIDES. The bonus paid to the Contractors provided the much needed capital for expansion of services in addition to the donations of medical equipment and furniture which, for many contractors was a unique gesture from STRIDES: Our clinic had existed for many years but with hardly any equipment for the medical ward. With the money paid to us, we used it to buy medical equipment. KII with Director, Bachi Medical Centre On the other hand, whereas it could be argued that the prospect of contract renewal motivated contractors to perform to their best; discussions held with STRIDES staff revealed that contract renewal was based on several factors, the contractors’ performance being just one of them. The others were: i) the contractor’s contribution to the core STRIDES indicators such as those for FP, ANC and immunization among others, and ii) the sustainability of the contractor’s work in the geographical area. Thus, sustainability was considered in light of district presence involving ownership of health facilities in the area of coverage. By owning health facilities in the districts covered, it was assumed that the contractor would or could strive to continue to deliver health services to the target group beyond the lifetime of STRIDES. Our analysis (See Annexes F-2 and F-3) shows that in all cases, one-off contractors achieved more than the repeat contractors, and one-off contractors had outstanding performance (70%) as compared to repeat contractors (61 %), implying that renewal of contracts was based on other factors and not necessarily performance. 21 3.4 Potential for PBC Model Expansion The PBC model registered some success in with regard to contributing to realization of targets under family planning, reproductive health, and child survival services. Notwithstanding its potential for expansion on account of factors such as presence of contractors and their great interest to participate; overall, the performance of PBC was not consistent with high performance levels. This suggests that the potential for its expansion while apparent will require careful reflection, and building support mechanisms that would improve the performance of private sector actors in ensuring timely and quality delivery of health services. Depending on whether costs for respective services and potential benefits could also be better planned and estimated in any next generation USAID/Uganda effort in relation to PBC, the potential for PBC expansion may thus only be relatively promising. Alongside the interest of contractors to participate, more particularly on account of the attractive incentives; there is donors’ broader interest in advancing Results-Based Financing models of assistance. Apart from appreciating the mixed evidence of the merits of results/performance-based contracting, now well-stated in the literature, including estimating whether citizens’ views themselves about ‘performance’ could also be considered when entering into such arrangements; ventures of this nature requires a more careful examination of contracting arrangements that could still be applied successfully within the modalities of government health facilities. Government facilities with private wings could be used as one point of entry, one more amenable to PBC arrangements. Better understanding the nature of the relationships that exist between contractors and government owned health facilities through community outreaches could also provide a basic starting point. 3.5 PBC’s Limitations and Challenges i. There was a reported variation in the definition of some indicators which would result in reduction in some reported outputs. It was the contractors’ view that in some cases there was a difference in interpretation between what STRIDES designed and what was actually assessed by STRIDES although only a few respondents stated this. Payment for specific outputs made service providers concentrate on performing activities without providing equal attention to related processes that are not paid for yet important for realizing quality of care improvement. For example Marie Stopes reported to have counseled patients on both short and long term family planning, but would only be paid for provision of long term family planning. ii. Some of the RH product distributors had a limited financial base that made it difficult to regularly re-stock the UHMG supplied products, even though they were highly subsidized. iii. STRIDES’s actual planning and realizing PBC-oriented services experienced a mixed record, possibly because neither STRIDES nor USAID provided particularly strong guidance about the actual features and methods as to how to pursue and develop PBC. iv. Whether well understood, executed, or not, the full application of PBC nonetheless was not fully experimented under government health facilities during STRIDES’ lifetime as USAID may have anticipated. 3.6 Lessons Learned from using the PBC Model 1. PBC demonstrated that it is possible for private sector actors to work with community and government structures particularly during outreaches where the contractor provided education on a range of RH/FP services, while government facilities sent staff to provide immunization and nutritional support services. 22 2. Neither the use of private contractors under PBC nor the strengthening of government facilities in providing BCC was able to solve lingering misconceptions about some RH/FP services. The implication is that there is still a communication gap and mistrust between service users and service providers regarding FP/RH services. This would therefore call for a reflection on the best way to deal with this gap and building trust among health service users and service providers. 3. Building the capacities of private contractors under PBC is still critical so as to improve the quantity and quality of health services delivery. For example, contractors revealed that they had their capacities built in areas such as record keeping and budgeting for services. This in turn helped them to do realistic costing for health services and it was expected that it would reduce the risk of incurring losses due to poor cost calculation. Other areas of capacity building included training on provision of long acting family planning, screening of cervical cancer and award of sizable contracts that enabled them to raise funds to buy some crucial equipment like ultra-scanners. 4. Strong quality assurance mechanisms are critical. STRIDES put in place a strong mechanism for monitoring services of private contractors and ensuring quality delivery. This was very instrumental in ensuring that contractors performed their work to the standards expected by STRIDES. In turn Contractors put in place self-assessment and monitoring mechanisms which hitherto were not strictly followed. It was on the basis of good performance that some Contractors received bonus payments. Involving contractors in community outreaches changes their outlook to service provision, notably; they realized that it was not good enough to wait for clients to come to your facility. Instead, reaching out to them helps to market your services and builds more solid relationships with clients. 3.7 Recommendations on the PBC approach  Broadly improving PBC: USAID/Uganda should better understand various models and applications of Performance and Results Based Contracting and whether and how it could become a feasible approach to enhancing Uganda’s public health facilities—these being the main stay of health care delivery in local governments. Even under the arrangement of working with private health providers STRIDES contracting partners conducted at least relatively successful outreaches and were able to secure the cooperation and bring on board Government health workers which suggest the potential for expansion.  Better Feedback: Provide timely feedback to contractors, and working with them to improve capacity so that gaps identified are worked on instead of discontinuation as was the case. Discontinuation negatively affects the trust built between providers and clients and undermines efforts to ensure delivery of sustainable services. In addition, consider within the project design a mechanism through which local actors’ and citizens, and not just the prime USAID recipient, could provide real feedback for quality of services rendered; and having citizens’ voices and views taken into account.  Considering PBC contracting length: Contracts could be increased to between 2-3 years so that the momentum and interest for staying engaged is maintained.  Determining which services might better be realized through PBC: Services that are critical in the successful delivery of RH/FP and CS services that were not catered for within the performance indicators such as counseling clients could be more carefully considered in future project design.  Building a component of capacity building to a still fledging private sector involved in the delivery of health services (RH/PF) services should in subsequent design of projects of this nature be prioritized. 23 4. Addressing the Unique Needs of Young People 4.1 Introduction This section describes qualitative findings related to interventions that sought to address young Ugandans’ unique needs. The evaluation examined the extent to which STRIDES interventions delivered youth friendly services. Successes and challenges faced are presented as well as some recommendations going forward. 4.2 Unique RH/FP Needs of Youth Due primarily to long-standing cultural and social norms as well as policy and institutional constraints, Uganda’s youth continue to have limited access to and utilization of available RH/FP services. Available services often do not address youths’ unique needs such as the failure among girls and boys to consistently adopt protective sexual behavior practices, including parents’ reticence to provide progressive, sexual education to their children, being faithful to one sexual partner, and use of condoms. Young people hardly access friendly antenatal care and HIV counseling and testing, thus hindering their knowledge and disclosure of HIV status and timely access to ART. In addition, they experience difficulties in prevention and management of pregnancies and associated stigma, while knowledge of appropriate child care practices among teenage mothers is limited. The limited involvement or disinterest of parents, teachers and leaders either elected or traditional undermines the supportive role that these actors would have played in improving knowledge of protective behavior and health service seeking behavior. Examples of the unique needs mentioned by young people are summarized in the matrix in Annex G-1. 4.3 Description of STRIDES Interventions to Address Youth’s Needs STRIDES intervened to address some of the needs of youth by equipping health workers with the knowledge and skills for handling the youth in a friendly manner, increasing awareness among youth about RH/FP services, and provision of supplies and equipment to support service delivery. With regards to training, STRIDES trained 153 health workers over the life time of the project in provision of youth friendly services as shown in the figure below: Figure 6: Health Workers trained in Youth Friendly Services Bugiri Kalangal a Kaliro Kamuli Kamwen ge Kasese Kayunga Kumi Kyenjojo Luwero Mayuge Mityana Mpigi Nakason gola Sembabu le Female 18 3 10 20 9 15 4 6 9 6 10 9 3 6 3 Male 230005260100012 0 5 10 15 20 25 Number of health workers trainned in youth friendly services by District Only fourteen percent (22) of the health workers trained by STRIDES in YFS during STRIDES implementation period were male. Figure 4.3.2 illustrates the contribution of the STRIDES training to 24 knowledge gain among health workers for provision of youth friendly RH/FP services. Overall, there was on average a 32-point gain in the level of knowledge following training in all STRIDES districts, with the lowest gain recorded in Kasese (22%) and Luwero (25); while the highest gain was realized in Mityana (48%) and Mayuge (40%). The evaluation was nonetheless unable to correlate improvements in knowledge gain with the improved delivery of RH/FP services to the youth. In particular, there was limited evidence to suggest that in the majority of health facilities, health workers used the knowledge and skills gained to provide youth friendly services. For example, it was repeatedly mentioned that health facilities lacked privacy —for example there was no room designated for handling youth, which undermined privacy and efforts to provide services designated for youth. Health workers suggested an improvement in the quality of services in the health facilities generally on account of improved delivery of medical equipment and supplies specifically meant for reproductive health and child survival services. While such reasoning cannot be discounted just for the sake, there was no data captured on RH/FP services utilization by youth. Any reference to improved health services seeking by youth was mostly assumed or implied by the health workers. Figure 7: Knowledge gain among trained Health Workers More efforts to improve the provision of youth friendly services were symbolized with procurement of sports materials. More specifically, the STRIDES Annual Performance Report (2013:27) reveals that in PY5, STRIDES procured and provided sets of games [equipment] to 58 health facilities targeted for the establishment of youth friendly corners across the 15 STRIDES collaborating districts. Thirty Seven (37) of the 58 facilities received audio-visual equipment (TV’s and DVDs) for edutainment of the youth. Having been distributed as late as STRIDES’ last year, this limited youth’s utilization of these services. 25 Performance in relation to delivery of Youth Friendly Services As more evidence of earlier, narrow project design, despite health professionals knowing that young Ugandans face particularly RH/FP challenges, USAID and STRIDES introduced the notion of providing youth friendly services after commencement of project implementation. However, little supplemental information about youth’s-associated issues was collected during STRIDES’ baselines. Without discounting the effort made to access the youth with services, STRIDES planned to have 45% of the health facilities it served, actively deliver young people friendly services (YPFS)14 by the end of the project life. Figure 8: Percentage of Targeted Health Units offering Young People Friendly Services (PY1-PY5) By the end of STRIDES, 35 percent of the targeted health facilities were recorded by STRIDES to have been equipped to provide YPFS. The evidence suggested that at the time of the evaluation, some of the centers were not necessarily providing YFS due to several limitations including but not limited to; a) lack of space to provide privacy needed to access youth with services, b) the inability to use the sports equipment offered to the youth due to lack of supervisory oversight from the facilities, c) complaints that health staff were not consulted on the choice of sports equipment, d) lack of a designated person to handle issues of the youth, e) the possibility that health facilities did not represent the most suitable avenues for reaching out to the youth. Some secondary data also indicates that only 12 out of 15 health facilities visited by STRIDES staff during PY4 had functional YFS. This in actual practice, the youth friendly corners were not functional. In general it appears that that the requirements for establishment of youth friendly corners were underestimated by STRIDES thus affecting the performance. 4.4 Appropriateness of STRIDES’ Interventions for Youth The evaluation results showed that in spite of marked increase in awareness of several FP services among the youth, there remain misperceptions and poor attitudes about RH services including fears of 14Characteristics of Young People-Friendly Services (YPFS): (1) Providers trained in YRH issues; (2) Providers trained in communication; (3) Respectful; (4) Non-judgmental attitude; (5) Confidentiality; (6) Privacy; (7) Convenient hours. Young people are those aged between 12 and 24 years. A health facility will be considered to be offering YPFS if it meets at least 5 out of the 7 aspects of the service. Numerator: Number of targeted health units offering young people-friendly services. Denominator: Total number of USAID IP (STRIDES) supported health units. PY1(Baseli ne) PY2 PY3 PY4 PY5 EOP Target Actual 9 9 13 40 35.1 Target 9 25 35 45 45 0 5 10 15 20 25 30 35 40 45 50 % of Facilities offering YPF Services 26 potential side effects of some FP methods. These included fears related to excessive bleeding, inability to conceive, or even death as noted by the participants below: Sometimes I fear that I will die if I take those family planning methods. Female youth FGD, Kumi Sometimes when you go for these family planning methods like injections, you bleed so much. So I fear to go [back] because I am scared that I may not be able to produce in future. Female youth FGD, Kumi Some girls also tell me that there are so many mothers who are complaining that they have lost their uterus because they used family planning methods. Sometimes I wonder what those drugs do, or if they kill all the eggs. I fear to lose my uterus. Female youth FGD, Kumi For me I am a married man, so I will speak on behalf of my wife. My wife came here for Depo-Provera as a family planning method, but she bled for about a month after it was administered. So I don’t know what causes that. Male Youth FGD, Mayuge Persistent negative attitudes towards condom use as well as poor knowledge and skills to use condoms were further noted by youth participants: Some don’t like condoms. They say that they are not 100 percent safe.” Female FGD Mayuge Some youths don’t know how to use the condoms. Female FGD Mayuge Some say that live sex is much more fun, so they don’t want to use condoms.” Female FGD Mayuge Some men tell us that if you insist on using the condom, it shows that you don’t trust him and that he is HIV positive. Female Youth FGD, Mayuge Knowledge about potential side effects of FP methods and how to deal with them was also reported to be low among youths. Despite a notable increase in available contraceptives and other essential RH/FP commodities, youths still indicated that there is need for more: Sometimes when we go to hospital, some drugs are out of stock so we have to go and buy. Sometimes condoms are not there. Female Youth FGD, Kumi STRIDES youth friendly interventions had, at best, mixed results. On the one hand, youth friendly corners were reported to have initially realized an increase in participation and demand for services among youth. However, some informants reported limitations to youth friendly corners including lack of appropriate time, space and privacy for providers to truly easily receive and attend to youth as well as inadequate staff training or even interest to work more intensively with youth. . This suggests that there was limited understanding and dialogue among the youth and service providers. Further, the youth friendly corners were perceived as inappropriate due to hindrances such as limited youth’s time and their inability to attend given in-school youths needed to attend their student schedules. In other cases, materials were not adequate or of the choice of the youth (the youth and the health staff 27 mentioned they needed to have been consulted); while most youth corners were based inside health units/facilities which some youth never considered comfortable visiting. Similarly, youths mentioned that antenatal care services were less likely to be perceived as youth friendly, due to the associated stigma surrounding youth pregnancy as noted below: Some are too shy, and too scared to move around while they are pregnant, because they don’t want their schoolmates to see them in that state. Female FGD Mayuge There is some evidence that community outreach campaigns were and can be effective in reaching out to the youth. Utilization of VHTs to implement RH/FP services has led to strengthening of integration between CHW and HC staff. This in-turn was reported to have improved service delivery for the youth in the communities. Youths, traditionally regarded to have poor health seeking behaviors were reported to have responded positively and demanded for services. At the same time VHTs felt more accommodated and welcomed as community health workers due to their frequent interface and their collaboration with the HC staff during outreaches and at health facilities. Thus, outreach activities strengthened team work and fostered mutual trust between community health workers (VHTs) and health facilities’ staff. This in turn strengthened the health care system. 4.5 Effectiveness of Youth Interventions Youths reported a noticeable increase in uptake of RH/FP services over STRIDES’ past three years, however, such reported uptake could not be validated due to lack of data15. Rather than the practical effectiveness of establishing ‘youth friendly services’ this evaluation attributes any such uptake to VHTs’ role and other government and partners’ health interventions. I think I started noticing these changes about three years ago. But what I can say is that the quality has gone up. These days’ people come for health services in large numbers compared to three years ago. Male Youth FGD, Mayuge The use of VHTs to reach youth was augmented with training in skills for handling the youth and improved supply of essential commodities to deliver a range of RH/FP services, a fact that was well noted by this participant: I also think that the government is working so hard to come up with educative programs, especially with the VHT’s. In fact, the VHT’s have worked so hard in providing us with health education, and mobilizing us to go for health services. Male Youth FGD, Mayuge STRIDES’ training and IEC efforts provided health education about sexual and reproductive health matters such as abstinence, protected sex, avoiding unwanted pregnancies at health centers as well as to the community through both print and radio media respectively. Youths appeared to appreciate the novelty and flexibility attributed to media campaigns as one participant noted: 15 USAID had not issued any supplemental, youth-sensitive, disaggregated data reporting requirements when STRIDES had started. 28 These days even if you are not able to go to the hospital, we are sensitized over the radio on antenatal issues and health education in genera. Back then we didn’t have such radio programs. Female youth FGD, Kumi While at least basic education and information campaigns appeared to become effective, the introduction of youth friendly corners at health centers usually did not elicit the desired outcomes. Some youths indicated that they were not comfortable with the health center venues while others felt that they did not have enough time to engage in all the activities offered at the youth friendly corners. Some youths perceived the services at health centers to be unfriendly, and in spite of STRIDES’ reporting it had well met the characteristics of ‘youth friendly services’, given the often, apparent unwelcoming health workers’ attitudes especially towards youth as indicated below: Sometimes when we go for the family planning methods, some health workers are harsh. They are not friendly. Female Youth FGD, Kumi Sometimes when you go for family planning methods, the health workers just pass you and attend to other patients, they sometimes ask ‘what has this young girl come to do?’ They do not give us attention. Female Youth FGD, Kumi Some health workers attributed the low success rates of youth friendly activities to inadequate training about their functionality or poor perception and attitudes about the concept. Youth awareness of RH/FP issues There was an increase in youth’s awareness of RH/FP issues as noted across all FGDs. However, the awareness or increased knowledge could not be collaborated with an increase in use of RH/FP among the youth. Youth participants however, noted that they had received knowledge from a number of sources including the VHTs, radio programs focused on youth health services, IEC materials at the health centers and education sessions delivered by health workers when they sought care at the health units. Some of the programs could have been delivered by other partners outside of STRIDES. Still participants noted that the increased knowledge/awareness coupled with modest improvement in availability of contraceptives in some centers, and other medicines and supplies like ‘Mama kits’, boosted some confidence in the health services by the youth. 4.6 Limitations and Challenges in Addressing Youth’s Needs Shortage or stock outs of supplies and medicines e.g. condoms, pills and other essential commodities and supplies remained a reoccurring limitation to youths’ utilization of RH/FP services as a participant noted below: Sometimes you go to the hospital and you want Norplant but it is out of stock. Female Youth FGD, Kumi In addition, youths reported that when services were not available at the health centers they resorted to seeking them from other sources, usually private-for-profit providers. However, some youths perceived such RH/FP services to be prohibitively costly when sought privately especially in light of having expected them initially to be free of charge at the health centers as explained below: 29 They are usually free of charge, but they do run out of stock often so we have to go and buy in private clinics. Female Youth FGD, Kumi There were also instances of delayed provision of materials essential for BCC activities. Similar to several key interventions already cited, some IEC materials were delivered late in STRIDES years and likely hindered the potential success of interventions aimed at increasing youth’s awareness of RH/FP services. Findings indicate that while VHTs’ outreach efforts contributed significantly in increasing youth access to RH/FP services, their understanding about youth’s role in development and skills set still remained limited, a situation exacerbated by few training opportunities and very low levels of informed, national dialogues about this subject. 4.7 Conclusions, Lessons and Recommendations on Youth Interventions Key Conclusions Based on this evaluation’s findings, there appears to be a substantial awareness of several FP services among the youth. The role of print and media programs, particularly through FM radio in improving knowledge and awareness in STRIDES areas, was visibly important. However, in spite of considerable ‘training’ efforts, there still remained significant negative perceptions and attitudes about RH/FP services among youth which contributed to poor service seeking behavior, illustrated by the negative perceptions towards several methods of family planning, including using condoms. STRIDES understanding and provision of Youth Friendly Services with health facilities was an untested idea that was not broadly vetted and faced significant bottlenecks. Not only did health clinic staff not easily, truly adopt and honor most of the principles of youth friendly services, but clinics’ limited HRH resources led to long lines, shorter periods to attend to clients, and perceived time spent away from clients’ other productive activities. Apart from not carefully questioning the appropriate, potential effectiveness, or not, of whether health clinics were particularly well positioned to provide YFDS, inadequate staff training and follow up entrusted to develop and use these youth corners was also reported to undermine their appropriateness. The role of VHTs while commendable has been constrained by limited training opportunities and in some cases stock outs of essential commodities and supplies. Sustainability, therefore remains a challenge particularly regarding the essential commodities/supplies and equipment that VHTs need to fulfill their tasks. Prohibitive service fees or cost of essential commodities and supplies continue to limit youths’ utilization of RH/FP services. This limitation is especially exacerbated when free services are not available at the health facilities yet cost of similar services from private entities remains prohibitive to some youths. Key Lessons In general, while the interventions targeting the youth—aimed at improving access to youth friendly services were well intended--STRIDES’ greatest program limitation lay with USAID’s own under- 30 appreciation of what characteristics truly feature “youth friendly services” and STRIDES subsequent implementation approach. This evaluation’s investigation has made it clear that STRIDES did not have a well-understood perspective of youth’s special needs or a well-designed process of consultation involving young people and laying a strategy for meeting their needs. Across all study districts, youth-friendly corners provoked more questions than answers as young people barely utilized them. Questions regarding service packaging, location of services, flexibility of access and timing or other, youth-in development programming principles all remained inadequately considered or answered. The result is that youth’s unique needs were ill-defined and youth involvement in the planning and delivery of services to their peers were never well prioritized. STRIDES’ differentiation in the with respect to background and profiles of the youth also needed to be better considered, and the types of youth served: –youth in and out of school, teenage mothers, sexually active, those who are not, the nature of peer groups the youth normally associate with, as well as how parents or other respected community elders could have become better involved to support youth. Recommendations  USAID/Uganda’s health office should provide adequate direction as to what ‘youth-effective’ programming could look like in its health service delivery programming.  Implementing a project of this nature in future requires fully appreciating and articulating a wide range of measures needed to successfully mobilize youth to improve their RH/FP service seeking behavior.  USAID should to study examples of other youth initiatives that have been considered as successful, such as Naguru Teenage Information and Health Centre and Restless Development to learn from their strategies and approaches employed in working with youth.  Interventions for reaching youth should not start so late in any such program and a more immediate, earnest consultation process involving young people to meet their unique, changing needs should be better prioritized during planning and implementation.  USAID should support the Uganda government and also encourage its contracting agencies, to incorporate the indicator on youth friendly RH/FP services in HMIS reporting.  USAID should also consider using youth-oriented organizations for piloting the use of performance based contracting for delivery of youth-oriented services.  The cost of accessing RH/FP services among the youth remains a hindrance. Future project design should continue to consider approaches that not only reduce on the cost of accessing the RH/FP services to the youth, but also those that empower young people to effectively demand for the services. xxxi| STRIDES PERFORMANCE EVALUATION REPORT ANNEXES Annex A: Scope of Work xxxii| STRIDES PERFORMANCE EVALUATION REPORT xxxiii| STRIDES PERFORMANCE EVALUATION REPORT Annex B: Background Information Annex B-1: Maternal, New Born and Child Health Indicators for Uganda PROGRESS STATUS Indicator 2001/2 2006 2011 2015 MDG Target￾MDG 4.1. Under-five mortality rate (per 1,000 live births) 152 137 90 56 MDG 4.2. Infant Mortality rate (per 1,000 live births) 88 76 54 31 MDG5.1. Maternal mortality ratio (per 100,000 live births) 524 418 438 131 MDG5.2. Proportion of births attended by skilled health personnel 39.0% 42.2% 58.0% 100% MDG5.3. Contraceptive prevalence rate 22.8% 23.7% 30.0% MDG5.4. Adolescent birth rates 178 152 135 MDG5.5. Antenatal care coverage At least one visit 92.4% 93.5% 94.9% At least four visits 41.9% 47.2% 47.6% MDG5.6. Unmet need for family planning 24.4% 40.6% 34.3% Annex B-2: Strides for Family Health Partners Internal Partner Coordination and Relations In its bid for the STRIDES project, MSH assembled a group of two international (JHPIEGO corporation and Meridian Group International) and two local Ugandan organizations (Communication Development Foundation Uganda and Uganda Private Midwives Association) to implement STRIDES. Each of these four internal partners were subcontracted and given specific responsibilities for project implementation while working together as a single integrated project team. The sub-awards are amended annually to add each approved work plan which clearly defines the sub-contactor’s scope of work each year. All sub-award team members work in concert as part of the STRIDES project team and are expected to closely coordinate their activities with other members on the team, and under the overall supervision of the STRIDES management team. STRIDES team is also expected to work closely with relevant local partners, USAID programs, STRIDES collaborating districts authorities, Uganda Ministry of Health and other relevant implementing organizations. Communications Development Foundation Uganda (CDFU): CDFU’s primary role as a sub-award is to support STRIDES in its efforts to increase demand for FP/RH and CS services at facility and community level through the development and implementation of effective IEC/BCC and social marketing strategies as appropriate. With respect to the later, CDFU works in close consultation with Meridian International which is another sub-award to that is taking lead responsibility for the expansion of social marketing activities through STRIDES. CDFU has nine permanent members of its staff assigned to all three regions who function as integrated members of the STRIDES team. Jhpiego Corporation (Jhpiego): is primarily responsible for strengthening provider capacity and assisting in facility improvement. Jhpeigo’s role includes the review and update of the existing training manuals, an assessment of provider needs, the identification of potential trainers, conducting training of trainers and of providers in the specific areas of basic FP an and long acting and permanent methods and supporting the ongoing coaching of trainers and providers post–training. Jhpiego has no full-time staff in country but contributes regular Short Term Technical Assistance (STTA) according to plans and agreements with the STRIDES technical team in country. Meridian: Provides specialized STTA in developing and implementing the social network program. xxxiv| STRIDES PERFORMANCE EVALUATION REPORT Uganda Private Midwives Organization (UPMO): This was on board to take the lead in involving private midwives. It was meant to increase of the number of members of the organization and help organize trainings work towards improving the quality of services. The organization failed to comply with expectations as agreed in its subcontract. Despite technical support from STRIDEs, the organization failed to improve its performance and the subcontract was terminated following the appropriate procedures. However, to allow continuity of the work, MSH did employ one of their staff who had been working on STRIDES to continue the work she had started. Annex B-3: STRIDES Interventions by District STRIDES Districts Interventions Total # of RH/FP/CS Interventions & Shoe distribution Nutrition Malaria Control Water for Health Drug Seller Initiative Leadership Development Quality Improvement Kumi √ √ √ 3 Kamuli √ √ √ √ 4 Kaliro √ √ 2 Mayuge √ √ √ √ 4 Bugiri √ √ √ √ 4 Kayunga √ √ √ 3 Kalangala √ √ √ √ 4 Mityana √ √ √ √ 4 Luwero √ √ 2 Nakasongola √ √ √ √ √ 4 Mpigi √ √ √ 3 Sembabule √ √ √ √ 4 Kyenjojo √ √ √ √ √ 5 Kamwenge √ √ √ √ √ 5 Kasese √ √ √ √ √ 5 xxxv| STRIDES PERFORMANCE EVALUATION REPORT Annex C: Evaluation Methodology C–1: Evaluation Methodology Evaluation Design The evaluation employed a mixed methods design. Qualitative data was collected through in-depth interviews and Focus Group Discussions with key project stakeholders at the central and district levels. Quantitative data was primarily derived from the monitoring and evaluation data base of STRIDES as well as from previous studies and reports including health facility surveys conducted by STAR EC. These various data sources were triangulated to arrive at a better understanding of STRIDES strategies and achievements. Data was collected from 8 intervention districts selected from East, Central and Western regions. Three control districts were also selected to gain a comparative picture between STRIDES and non-STRIDES districts. We were nonetheless only able to gain insights between STRIDES and non SRIDES districts on services utilization owing to non-availability of statistical data on comparable indicators. Qualitative comparisons where possible, have been provided. Approach to Answering the Evaluation Questions In order to answer the evaluation questions, the evaluation team conceptualized the research questions and made specific analysis that was expected to provide answers. The conceptualizations and analyses made in respect of each research question are outlined below. Appropriateness and Effectiveness of the ‘Fully Functional Service Delivery’ model and Links between facility, human resources for health (HRH), service delivery and community components The fully functional health service delivery system was conceptualized as one that adheres to the MoH/WHO norms and standards, covering elements of service delivery, human resources (sufficient trained & motivated staff), health information systems, infrastructure, equipment, availability and management of medicines and supplies, functional referral and network systems, and leadership and governance. Appropriateness of the FFSD model was assessed by examining its assumptions, key elements and how feasible or practical they were given the realities on ground and the context. Appropriateness was conceptualized as the extent to which the model was suited to address existing needs and challenges, whether it was feasible, and how well it was aligned to the context. The evaluation team examined the challenges that existed in the delivery of RH, FP and CS services at the time STRIDES started and whether the FFSD model was adequately designed to address them. In addition, the team looked at any changes in the model over time in response to emerging needs and lessons learnt over time. Effectiveness of the FFSD model was conceptualized as the ability to deliver the intended outcomes. To assess effectiveness, the team compared the baseline status of health facilities (as the primary service delivery points) to their status at the time of the evaluation. The evaluation team based on the district baseline studies conducted by STRIDES in 2009 and compare with the status at the time of the evaluation, based on health facility assessments conducted by LQAS STAR E in 2013, as well as some primary data collected during the present evaluation. To assess the extent to which the facility, human resources, service delivery and community components worked together, the evaluators looked at the extent to which improvements in one aspect were optimized by improvements in another. Specifically, the focus was on whether health workers trained under xxxvi| STRIDES PERFORMANCE EVALUATION REPORT STRIDES were retained; whether the health workers trained under STRIDES have adequate tools and equipment to work with; linkages between health facilities and communities in terms of the extent of community awareness of services, extent to which VHTs/communities are able to provide RH/FP and CS services, the trends in uptake of services, and the functionality of referral systems from VHTs. Appropriateness and effectiveness of the Performance Based Financing model and its potential for scale up To assess the appropriateness of the PBF/PBC model, the evaluators assessed its assumptions, whether the approach was well suited to solve the gaps and challenges in service delivery at the time, whether the rewards and sanctions system was motivating to the sub-contractors, and whether the approach as a whole was suitable to the communities, the providers and the context. Effectiveness of the PBC model was assessed by examining whether the approach achieved its intended its outcomes, namely, increasing availability and supply of services, and increasing uptake and utilization of services. Extent to which reproductive health approaches and interventions implemented by STRIDES address the unique needs of youth within the 15-25 age group in comparison to the other age groups and the unique factors affecting uptake and utilization within this age group Youths are a unique socio-demographic category with unique needs and behaviors compared to the older population. The STRIDES Program endeavored to put in place strategies that specifically targeted to reach the youth aged 15-25 and to meet their reproductive health needs. In this evaluation, attention was paid to the unique needs of the youth, both from the perspective of the youth themselves and other stakeholders. The evaluation looked at the strategies that were put in place to meet those needs, and the extent to which these strategies worked; whether they reached the youth, whether they attracted more youth to come for services, what worked well and what did not work well; what barriers the youth were still facing in accessing RH services and what their preferences would have been in relation to what STRIDES has been doing. Selection of sample sites for Data collection A sample of 8 districts was selected out of the 15 districts where STRIDES was implemented. The sample districts are shown in table C-1 below. Selection of sample districts was based on a number of considerations including representation of different ethno-geographical characteristics; scope of activities implemented under STRIDES support; and district performance in health services delivery as per the MOH District league Tables for 2012/2013. Priority was also placed on those districts with a bigger number of STRIDES interventions (See Annex B-3) undertaken during the life of STRIDES to make it possible to collect data on all aspects of project interventions. In addition, three non-STRIDES districts were studied to gain insight into their situation relative to STRIDES supported districts. Unfortunately, data gained from control districts could not be well aligned with comparable indicators. As such, the control districts have not been well catered for in the report. The districts selected for control did not have support from STRIDES sister projects such as STAR EC or STAR SW. They were selected to represent middle-level performance according to the District League table assessments for 2012/2013 (See in Table 2). Choice of control districts also deliberately excluded the very old districts such as Jinja, Masaka, Mukono and Mubende which may have particular advantages in terms of health infrastructure and services. The sample districts were Kamuli, Kumi and Mayuge for eastern region, with Namayingo as the control. Those for central region were Mityana, Nakasongola and Ssembabule, with Nakaseke as the control. For western region they included Kyenjojo and Kasese, with Kyegegwa as the control. The sampled STRIDES districts included Kyenjojo in western region, which had the Savings Mothers, Giving Life xxxvii| STRIDES PERFORMANCE EVALUATION REPORT project implemented by Baylor Uganda, also focusing on matters of maternal health that STRIDES was focusing on. Table C-1: Sample Districts Studied Region All STRIDES Districts Sample Districts Control Districts Eastern Kamuli Kamuli Namayingo Kumi Kumi Kayunga Bugiri Mayuge Mayuge Kaliro Central Luwero Nakaseke Mpigi Mityana Mityana Nakasongola Nakasongora Kalangala Ssembabule Ssembabule Western Kyenjojo Kyenjojo Kyegegwa Kamwenge Kasese Kasese Data Collection Review of Program documents, Project Database and National HMIS The evaluation team reviewed relevant program documents including quarterly and annual reports, work plans, Performance Management Plans, Memorandum of Understanding with District local governments, PBC/F contracts, program status reports, and other policy and working documents (see list of documents reviewed in Annex C-2 (ii). The evaluation team also reviewed STRIDES project monitoring data to derive analyses of project outputs and outcomes. Some specific information relating to the STRIDES program districts and control districts was also extracted from the National HMIS at MoH. In-Depth Interviews (IDIs) with Key Informants IDIs were conducted with several stakeholders and partners (see Table C-2 below). At the national level IDIs were conducted with selected staff of USAID/Uganda, STRIDES senior management and program staff, Ministry of Health (MOH) officials, and staff from contractors and partners. At the district level, interviews were conducted with District Health staff, health facility managers/in-charges, local organizations involved in the implementation of STRIDES, and groups of STRIDES clients/beneficiaries. These interviews helped to generate new information, verify and validate information from other sources, as well as identifying gaps and good aspects of the program’s strategies and activities. Focus Group Discussions FGDs or group interviews were conducted with clients attending ANC or FP clinics, Community health workers/VHTs, women of reproductive age, and male and female youths aged 15-25 years. xxxviii| STRIDES PERFORMANCE EVALUATION REPORT Health Facility Interviews Semi-structured interviews were conducted with the In-charges of sample health facilities. Data was collected on the extent and forms of STRIDES support, issues of management and leadership, key challenges in realizing full functionality, and lessons learnt. Quantitative data on the status and functionality of health facilities was generated from secondary sources. Similar interviews were conducted in the non-STRIDES (control) districts in order to assess the functionality of the health services delivery system in those districts. The table below summarizes the interviews and group discussions conducted. Table C-2: Summary of Interviews and Group Discussions conducted Method Category of Informants Total Intervention districts Control districts In-depth interviews DHO/Assistant DHO (MCH)/ STRIDES focal person 14 12 2 Facility In-charge (Hospital, HCIV, HCIII, HCII) 30 22 8 PBC contractors/sub-contractors 10 10 N/A SMGL coordinator 1 1 N/A FGD Male Youth 15-19 7 4 3 Male Youth 20-25 7 6 1 Female Youth 15-19 8 6 2 Female Youth 20-25 7 5 2 FGD with women of reproductive age (15-49) 16 16 0 FGD with VHTs 20 16 4 Group interviews STRIDES senior management team 1 N/A N/A PBC contractors 1 1 N/A Group interview with Health workers (HCIII, HCIV) 16 16 0 Data Collection Tools Different data collection tools were designed for purposes of collecting data from the different sources outlined above. The tools consisted of the following:  KII guide for national level informants (staff of STRIDES, MOH and USAID)  KII Guide for partner organizations and sub-contractors  KII guide for district staff (DHO/DHMT) in STRIDES districts  KII guide for district staff (DHO/DHMT) in control districts  Interview Guide for Health Facility Managers  Interview Guide for Health Facility Managers (In Control Districts)  Interview Guide for ANC/FP clients xxxix| STRIDES PERFORMANCE EVALUATION REPORT  Interview Guide for Community Health Workers/VHTs  FGD guide for women of reproductive age  FGD guide for youths  FGD guide for health workers  Secondary Data Extraction Guide (See Annex K for Tools Used) Data Analysis Quantitative and qualitative data were analyzed based on the Performance Evaluation questions. Qualitative data was analyzed using an inductive approach to derive meanings and issues pertaining to program design, implementation, impact and issues of sustainability among others. Quantitative data collected from secondary sources was analyzed mainly using Ms excel to derive percentages, trends and other descriptive measures to depict programme performance. To compare STRIDES and control districts, comparisons were made based on secondary data on key indicators of RH/FP/CS, indicators of FFSD and interview data regarding the general performance and quality of RH/FP and CS services in the two sets of districts. Ethical Considerations Basic ethical standards and considerations were observed for all interviews and discussions held. Evaluation participants were interviewed after providing oral informed consent. Information that could be directly linked to an individual has been anonymized and quotes included in this report do not include names. Where minors were interviewed, consent from their parents was also sought before the interviews. Limitations and Challenges One of the limitations of this evaluation is that some of the assessments that the evaluation sought to make could only be done using objectively verifiable indicators. For instance the appropriateness of the FFSD and PBF models is a qualitative measurement and largely depended on the Consultants’ as well as stakeholder’s interpretation of the data and the context. The resulting conclusions therefore may not be backed by hard data and objective evidence. To minimize this challenge, the consultants have triangulated information from different sources. STRIDES PMP focused only on outcome indicators and did not capture process indicators that would have been essential to assess the FFSD approach, such as infrastructure and human resource improvement. As a result, it was not been possible in this evaluation to provide quantitative comparisons of FFSD indicators before and after STRIDES. Whereas the evaluators hoped to find good secondary data to compare the functionality of health facilities before and after STRIDES, the existing data for different periods of time was sometimes not very comparable, having focused on different indicators and used different measures, while in other cases, previous studies covered only a few of the STRIDES districts. As a result, some of the analyses are based on a limited set of districts for which data was available. Finally, whereas the evaluation included some control districts, no uncontaminated control cases could be found. Whereas the selected control districts did not have STRIDES interventions and support, they certainly had support from other sources such as NGOs. Thus the absence of STRIDES support does not necessarily mean that these districts are worse off in terms of services. The control districts, though from the same geographical locations could also be subjected to different contextual factors that affect service delivery, hence making them difficult to compare with the STRIDES districts. Moreover, the absence of xl| STRIDES PERFORMANCE EVALUATION REPORT baseline data linked to the indicators for comparison made it difficult to make conclusions depicting a picture between control and intervention districts. Annex C-II: Documents Reviewed Strides Specific Documents:  Original STRIDES Proposal  District baseline reports and consolidated baseline report  STRIDES PMP, project work plans  STRIDES district selection document  STRIDES Program description document  STRIDES Training Strategy (2012)  STRIDES Quality Assurance Plan  Sample MoU with respective Districts  Sample PBC/F  STRIDES Communication Plan (Jan 2012)  STRIDES M&E Manual (Revised 2013)  Uganda Joint BCC Survey Report (Oct 2012)  Quarterly and annual PROJECT reports to USAID,  STRIDES health facility assessment report (2012)  District Sustainability Plans  Program status reports, and other policy and working documents Other Documents  Health Facility Assessments by STAR EC (2011-2013)  LQAS Community Survey Reports by STAR EC (2011-2013)  Uganda Health System Assessment (MoH, 2011)  Annual Health sector review Report 2012/2013  Uganda Health Workforce Study (March 2007) xli| STRIDES PERFORMANCE EVALUATION REPORT Annex D: Other Annexes to Section One Annex D-1: Number and Type of Health Facilities supported by STRIDES District Number of Health Facilities supported HC II HC III HC Grand Total IV HOSPITAL Bugiri 32 9 1 1 43 Kalangala 3 6 2 0 11 Kaliro 13 5 1 0 19 Kamuli 33 15 2 2 52 Kamwenge 24 8 2 0 34 Kasese 56 37 3 3 99 Kayunga 22 8 2 1 33 Kumi 11 5 1 3 20 Kyenjojo 11 12 1 1 25 Luwero 37 27 3 1 68 Mayuge 33 6 2 1 42 Mityana 36 17 2 1 56 Mpigi 10 17 1 1 29 Nakasongola 21 9 2 1 33 Sembabule 14 7 2 1 24 Grand Total 356 188 27 17 588 Annex D-2: Breakdown of Funds Spent by Strides by October 2014 Total amount obligated to MSH/STRIDES from USAID was $48,128,563. A total of $ 46,348,473 was spent by October 2014, as below: Item USD Direct salaries 1,431,146 HSV applied 236,866 Direct overhead 1,339,803 Local profession 5,206,590 Local prof OVHD 1,989,439 Consultants 994,096 Consult OVHD 383,399 Local staff sal 2,436,485 Allowances 409,253 Travel and trans 3,376,406 Training 5,155,745 Sub contracts 5,740,429 Grants 10,656,611 Outside services 1,164,403 Communications 462,802 xlii| STRIDES PERFORMANCE EVALUATION REPORT Postage/shipping 84,851 Rent/utilities 680,725 Copying/printing 476,256 Supplies/materials 957,778 INS/CONF/other 1,757,937 Equipment 1,407,451 Total 46,348,473 xliii| STRIDES PERFORMANCE EVALUATION REPORT Annex E: Annexes to Section Two Annex E-1: Health Facilities Renovated/Remodeled District Facility Name Unit/Block Renovated Nakasongola Kikoiro HCII OPD block Kazwama HCIII Maternity block and OPD block Luwero Luwero-Kasana HCIV Theatre block and General ward block Nyimbwa HCIV Maternity block Mayuge Mayuge HCIII Maternity block Kigandalo HCIV General block Kamwenge Rukunyu HCIV Theatre, OPD, labor/maternity block Rwamanja HCIII Maternity block Kasese Rwesande HCIV OPD, maternity block Bugiri Kayunga HCIII MCH block Annex E-2: Examples of Facilities Constructed/Renovated Kayogera health center. Left, the old structure; right, the refurbished block with STRIDES support Clients waiting shed at Mpigi HCIV constructed by STRIDES in partnership with Uganda Baati Ltd xliv| STRIDES PERFORMANCE EVALUATION REPORT Annex E-3: Summary of Health Facilities by Level that Received Equipment from STRIDES by District a) Facilities that Received Equipment procured by STRIDES Facility Type District Total Kumi Bugiri Kaliro Mayuge Kamuli Kayunga Luwero Nakasongola Mpigi Mityana Sembabule Kalangala Kasese Kyenjojo Kamwenge Hospital 3 1 0 1 2 1 0 1 1 1 1 0 1 0 0 13 Health centre IV 0 1 1 2 2 1 3 1 2 2 2 1 2 2 2 24 Health centre III 7 7 5 7 7 5 9 4 4 6 5 2 13 6 4 91 Health centre II 6 14 13 10 23 4 9 8 9 6 15 4 30 5 6 162 TOTAL 16 23 19 20 34 11 21 14 16 15 23 7 46 13 12 290 b) Facilities That Benefited From IMEC Donated Equipment Facility Type District Total Kumi Bugiri Kaliro Mayuge Kamuli Kayunga Luwero Nakasongola Mpigi Mityana Sembabule Kalangala Kasese Kyenjojo Kamwenge Hospital 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 2 Health centre IV 1 1 1 2 2 2 3 2 1 3 2 1 1 1 2 25 Health centre III 6 6 5 5 9 9 5 5 9 9 5 2 7 9 8 99 Health centre II 1 1 1 2 3 1 1 0 0 2 0 1 1 2 1 17 TOTAL 8 8 7 9 14 12 9 7 10 14 8 4 10 12 11 143 GRAND TOTAL 24 31 26 29 48 23 30 21 26 29 31 11 56 25 23 433 xlv| STRIDES PERFORMANCE EVALUATION REPORT c) Number of Equipment Distributed by STRIDES to Health Facilities by District General equipment Reproductive health including newborn Child survival Bugiri 303 489 173 Kaliro 336 436 60 Kamuli 717 723 364 Kamwenge 270 460 231 Kasese 928 1177 736 Kayunga 279 543 232 Kumi 220 444 206 Mayuge 329 473 586 Ssembabule 575 626 366 Total 3,957 5,371 2,954 d) Number of Equipment Distributed by STRIDES to Health Facilities by District Annex E-4: Percentage of Health Facilities Submitting Timely HMIS Reports to Health Sub-District and MoH Indicator Project Year 1 (Baseline) Year 5 Performance End of Project Planned Target EOP Achievement against Target (%) % facilities submitting timely HMIS reports to HSD/district 72 88 90 98 % districts submitting timely HMIS reports to MoH 78 90 93 97 Bugiri Kaliro Kamuli Kamwe nge Kasese Kayung a Kumi Mayuge Sembab ule GENERAL EQUIPMENT 303 336 717 270 928 279 220 329 575 RH INCLUDING NEWBORN 489 436 723 460 1177 543 444 473 626 CHILD SURVIVAL 173 60 364 231 736 232 206 586 366 0 200 400 600 800 1000 1200 1400 Number of Items xlvi| STRIDES PERFORMANCE EVALUATION REPORT Annex E-5: Proportion of Facilities Meeting Minimum Requirements: Data from 2013 Star EC Health Facility Assessment (HFA) Reports Table E-5-1: Proportion of health facilities meeting minimum process requirements (2013) Bugiri % (N=12) Kaliro % (N=6) Kamuli % (N=15) Mayuge % (N=10) Number of facilities with all components for minimum infrastructure 0.0 0.0 0.0 20.0 Number of facilities that offer all essential services 50.0 16.7 33.3 30.0 Number of facilities where clients are knowledgeable about diagnosis, dose, frequency and duration of admin of drugs 0.0 0.0 0.0 0.0 Number of facilities where all elements of proper record￾keeping and stock control are adequate for latex gloves 50.0 16.7 53.3 60.0 Number of facilities that can show a monthly service report that is less than three months old and can show evidence of data use 66.7 73.3 50.0 Table E-5-2: Number of health facilities meeting minimum IMCI requirements (2013) Bugiri % (N=12) Kaliro % (N=6) Kamuli % (N=15) Mayuge % (N=10) Number of facilities where a qualified staff provides IMCI care 58.3 83.3 66.7 80.0 Number of facilities with all essential equipment for provision of IMCI services 16.7 0.0 0.0 10.0 Number of facilities with all relevant IMCI drugs in stock on the day of the survey 0.0 0.0 6.7 0.0 Number of facilities with all child vaccines valid and stored in working refrigerator or cold box on the day of the survey 33.3 66.7 80.0 40.0 Number of facilities that have all relevant guidelines for IMCI readily available 41.7 33.3 13.3 10.0 Number of facilities where the health worker who provides IMCI care has received training or updates (ever) in all of the following: IMCI, ACTs, RDT use 0.0 73.3 40.0 Number of facilities where the health care worker who provides IMCI care has received in-service training in any of the areas of competency in the past 12 months 75.0 33.3 73.3 20.0 xlvii| STRIDES PERFORMANCE EVALUATION REPORT Number of Health Facilities where the health worker who provides IMCI care received external comprehensive supervision in the last 3 months 16.7 33.3 26.7 30.0 Number of facilities where registers are consistently filled out completely and properly 25.0 33.3 60.0 30.0 Number of facilities where health workers consistently undertook all assessment tasks 0.0 0.0 0.0 0.0 Number of facilities where diagnosis receives appropriate treatment in all cases 0.0 0.0 0.0 0.0 Number of health facilities where the clinician described how to administer ACTs, other anti-malarials, antibiotics and ORS 0.0 0.0 0.0 0.0 xlviii| STRIDES PERFORMANCE EVALUATION REPORT Table E-5-3: Number of health facilities delivering ANC and Delivery services against national standards (2013) Bugiri % (N=12) Kaliro % (N=6) Kamuli % (N=15) Mayuge % (N=10) Number of health facilities where the health worker consistently completes all essential assessment tasks during the ANC visit 0.0 0.0 0.0 0.0 Number of health facilities where the health worker completes all essential health education tasks 0.0 0.0 0.0 0.0 Number of health facilities that prescribe / administer appropriate drugs and vaccines relevant for ANC 0.0 0.0 0.0 0.0 Number of Health facilities offering all essential ANC laboratory tests 0.0 0.0 6.7 10.0 Number of health facilities where ANC services are offered by qualified health worker. 91.7 100.0 86.7 70.0 Number of health facilities with all essential ANC supplies 8.3 0.0 20.0 10.0 Number of health facilities with all essential ANC drugs available and valid on day of assessment. 0.0 0.0 13.3 0.0 Number of health facilities with at least 3 relevant visible IEC materials on ANC 25.0 0.0 0.0 10.0 Number of health facilities with complete up to date ANC records 25.0 16.7 40.0 40.0 Number of health facilities where health workers received comprehensive external supervision in the last 3 months 8.3 0.0 26.7 20.0 Number of health facilities where Health workers offering ANC have ever received in-service training in all relevant service areas. 25.0 16.7 26.7 20.0 Number of health facilities where health workers offering ANC have received training in at least one of the relevant service areas in the past twelve months 91.7 50.0 93.3 50.0 Number of health facilities(level IV and above) with delivery services offering blood transfusion 8.3 0.0 26.7 30.0 Number of health facilities with all functional essential delivery equipment and supplies 0.0 0.0 0.0 0.0 Number of health facilities with essential drugs in maternity wards (present and valid). 0.0 0.0 0.0 0.0 Number of health facilities where the most recent labor or labor at time of assessment was monitored correctly using Partograph. 16.7 33.3 100.0 80.0 Number of health facilities with complete consistent record of deliveries 25.0 0.0 6.7 10.0 Number of health facilities where health workers have ever received training in EmONC 75.0 83.3 60.0 70.0 xlix| STRIDES PERFORMANCE EVALUATION REPORT Table E-5-4: Number of health facilities delivering FP/PNC & Post-Abortion services against national standards(2013) Bugiri % (N=12) Kaliro % (N=6) Kamuli % (N=15) Mayuge % (N=10) Number of health facilities with family planning methods in stock at the time of the assessment 8.3 0.0 13.3 0.0 Number of health facilities where reproductive health clinical services for youth are offered in a private area 16.7 6.7 40.0 Number of Health facilities where health worker offering family planning services have ever had comprehensive training to offer family planning services 83.3 50.0 60.0 40.0 Number of health facilities offering post abortion care. 8.3 100.0 53.3 20.0 Number of health facilities with all equipment and supplies essential for post abortion care 8.3 0.0 6.7 0.0 Number of health facilities where all essential postnatal services are offered to mothers 66.7 100.0 80.0 90.0 Table E-5-5: Number of health facilities meeting minimum process requirements for Nutrition (2013) Bugiri % (N=12) Kaliro % (N=6) Kamuli % (N=15) Mayuge % (N=10) Number of health facilities with qualified and trained staff Providing nutrition services 83.3 83.3 80.0 100.0 Number of health facilities with all essential equipment for the management of acute under￾nutrition 0.0 16.7 26.7 0.0 Number of health facilities offering nutrition assessment services 58.3 50.0 86.7 30.0 Number of health facilities with essential supplies for the management of acute under-nutrition including stocks of Ready To Use Therapeutic Feed (RUTF) 25.0 50.0 26.7 0.0 l| STRIDES PERFORMANCE EVALUATION REPORT Annex E-6: Percentage of Villages with Functional VHTs (PY1-PY5) Annex E-7: Trends in Service Utilization under the Fully Functional Service Delivery System Fig E=7=1: % pregnant women who received 2+ doses of IPT 21 21 25 32 45 0 0 21 35 35 40 40 0 5 10 15 20 25 30 35 40 45 50 PY1 (Baseline) PY2 PY3 PY4 PY5 EOP Target % Villages with functional VHTs Actual (%) Target (%) li| STRIDES PERFORMANCE EVALUATION REPORT Fig. E-7-2: % of live births delivered from a health facility Fig E-7-3: % live births with low birth weight lii| STRIDES PERFORMANCE EVALUATION REPORT E-7-4: Cumulative children immunized against DPT and given Vitamin A Annex E-8: Number of Health Service Providers Trained by STRIDES Area of Training NUMBER TRAINED BETWEEN YEAR 2 – 5 Target by End of Project Achievement against EOP target (%) Male Female Total Number of service providers trained in FP, RH and CS FP 345 1,717 2,062 1,855 111% RH 147 617 764 819 93% CS 490 733 1,223 1,165 105% Number of people trained in Child Health and Nutrition IMAM 362 893 1,255 1,390 90% IYCF/ENA 305 774 1,079 360 300%16 PD/Hearth 828 1,568 2,396 2,790 86% 16 STRIDES has initially planned to conduct IYCF/ENA trainings at regional level, targeting health workers from hospitals and HCIVs, but later changed the strategy and conducted them at district level, covering health workers from all levels, hence reaching more health workers than initially planned. liii| STRIDES PERFORMANCE EVALUATION REPORT Annex E-9: Proportion of assisted Health Facilities providing RH, and FP services at Baseline and End of Project Health Facilities Baseline Year 5 End of project Target Achievement against EOP target (%) % targeted health units offering Young People￾Friendly Services 9 35 45 78% % health facilities (HC III & above) providing Basic Emergency Obstetric Care (BEmONC) 10 20 (at Year 4) 40 -- % health facilities (HC IV & above) providing Comprehensive Emergency Obstetric Care 9 24 (at Year 4) 25 -- # of USG-assisted Service Delivery Points providing FP counseling or services 104 503 254 198% % health facilities (HC III & above) offering long acting methods (LAM) 37 74 60 123% % health facilities (HC III & above) offering permanent methods (PM) 30 25 50 51% % USAID supported Service Delivery Points offering any modern contraceptive method 46 97 92 105% liv| STRIDES PERFORMANCE EVALUATION REPORT Annex F: Annexes to Section Three Annex F-1a: List of PBC Sub Contractors Contractor Type of Organization District Teso Rural Development Organization (Vision TERUDO) Local FBO Kumi FHI 360 International NGO Bugiri, Luwero, Nakasongola, Mayuge Bufumira Islands Development Association (BIDA) Local NGO Kalangala Chain Foundation Uganda Local NGO Kayunga Holy Family, Virika Hospital Local faith based hospital Kyenjojo Family Life Education Program (FLEP) Busoga Local FBO Kaliro, Kamuli, Mayuge Marie Stopes Uganda National NGO All 15 districts Program for Accessible Health, Communication and Education (PACE) Social marketing National NGO organization Kamuli, Kaliro, Bugiri, Kumi Uganda Health Marketing Group (UHMG) Social marketing National Local NGO organization Kasese, Mityana, Mpigi, Kamwenge, Sembabule, Kyenjojo, Kalangala, Nakasongola Midas Touch Medical Services - Kumi Local Health Facility Kumi Uganda Manufacturers’ Association (UMA) Leading manufacturers’ association in Uganda Mityana, Kyenjojo and Kayunga Ernest Cook Ultrasound Research and Education Institute (ECUREI) Private training organization Mpigi lv| STRIDES PERFORMANCE EVALUATION REPORT Annex F-1b: List of PBC Contractors and RPF 002 Contractor Type of Organization District Midas Touch Medical Services￾Kyenjojo Local Health Facility Kyenjojo ECUREI Private training organization Mpigi, Luwero Aids Information Centre National NGO Mpigi, Mityana, Mayuge, Kaliro, Kumi Pathfinder International Uganda International NGO Kasese, Kyenjojo, Kamwenge Lutheran World Federation Uganda International NGO Sembabule FHI360 International NGO Bugiri, Luwero, Nakasongola, Mayuge International Baby Food Action Network (IBFAN) Uganda Local NGO Bugiri, Kamwenge, Kasese, Mpigi Midas Touch Medical Services￾Kumi Private Health facility Kumi DSSD Caritas Fort Portal Faith Based NGO Kyenjojo Family Life Education Program (FLEP) Busoga Diocese Faith Based local NGO Bugiri, Kaliro, Kamuli & Mayuge Community Empowerment Initiative Uganda Local NGO Nakasongola Health Office, Fort Portal Diocese Private-not for-profit faith based department under Fort Portal Catholic Diocese Kyenjojo & Kamwenge Act4Africa- Uganda Local NGO Mayuge Community Integrated Development Initiatives National NGO Luwero Contractor Type of Organization District of Operation Good Samaritan Care Small clinic and maternal home Kasese Alleluia Medical Centre (AMC) Small maternal home and clinic with lab services Kasese Kaneka Rural Clinics Advisor Clinics and Nutritional Centre (KARUCAN) small NGO Kasese St. Luke Medical Center Small clinic with lab services Kamwenge St. Edward health unit small clinic and maternal home Kyenjojo Kyenjojo Pharmacy Retail and wholesale pharmacy Kyenjojo The Potter Health Care Clinic Small clinic Kyenjojo lvi| STRIDES PERFORMANCE EVALUATION REPORT St Emmanuel Healthcare Centre Small clinic with lab services Kyenjojo St Mary's Domiciliary and Maternity Home Clinic and maternal home Kamuli Subi Medical Centre Small clinic Mayuge Bachi Medical Centre Small clinic Mayuge Kaluba Medical Centre Clinic with lab services Mayuge Countryside Health Care Clinic Clinic and maternal home Kamuli St. Paul Domiciliary Clinic Clinic and maternal centre Kayunga Awebwa Maternity Centre for profit clinic Luwero Nakasongola Medical Center Clinic and maternal home Nakasongola St. Charles Drug Shop Small Clinic Mityana Banda Health Centre Health Centre Mityana Community Centre, Mityana Small clinic with youth centre Mityana Kisa Kyamukama Medical Clinic Small clinic with lab services Mpigi Cranmedic Medical Services Small clinic with lab services Mpigi Family Drugs Shop Small drug shop Mpigi D&D drug Shop Small drug shop Sembabule Walter Clinic Small clinic with lab services Kalangala St Philomera Drug Shop Small clinic and maternal home Kalangala lvii| STRIDES PERFORMANCE EVALUATION REPORT Annex F-2: Contractors that met Quarterly Targets by Type of Contract Annex F-3: Contractors that met Annual Targets by Type of Contract One off contract Repeat contract Overall Achieved pre-determined targets 49.3 32.3 40.9 Pre-determined Targets not achieved 50.7 67.7 59.1 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 Percent One off contract Repeat contract Overall Achieved pre‐determined targets 57.5 41.5 49.6 Pre‐determined Targets not achieved 42.5 58.5 50.4 0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 Percent lviii| STRIDES PERFORMANCE EVALUATION REPORT Annex F-4: Contribution of PBC Contractors to 4th ANC and IPT Utilization % of pregnant women who receive 4 ANC Visits % pregnant women who received 2+ doses of IPT Annex F-5: Cumulative Contribution of PBC Contractors to Nutrition and DPT Vaccination Services Cumulative number of children under five years reached by USG supported nutrition programs Cumulative number of children who at 12 months have received three doses of DPT vaccination from a USG- supported immunization program. lix| STRIDES PERFORMANCE EVALUATION REPORT Annex G: Annexes to Section Four Annex G-1: Matrix on identified Young People’s Unique Needs Unique youth need STRIDES RH/FP approach/Intervention Narratives illustrating unique need - challenges There are still remain negative attitudes about condoms; some youth still need to learn how to use condoms, others need to change their attitudes and perceptions about condoms; to cope with distrust arising from previous failed attempts to appropriately use condoms Set up youth friendly corners Trained VHTs and equipped them to engage more with youth (see figures below) Outreach services by VHTs “Some don’t like condoms. They say that they are not 100%.” Female FGD Mayuge “Despite relatively high awareness of condoms, some misperceptions and negative attitudes about them persists” “Some men tell us that if you insist on using the condom, it shows that you don’t trust him and that he is HIV positive.” Female Youth FGD, Mayuge Expectant youth suffer stigma which makes it difficult for them to receive antenatal care Instituted youth friendly corners Peer service providers – training some youth to engage those in community, some became VHTs “Some are too shy, and too scared to move around while they are pregnant, because they don’t want their schoolmates to see them in that state” Female FGD Mayuge Despite a notable increase in contraceptives availability, youths still experience unwanted pregnancies Moreover, not all youth access the products – cost, distance to point of access Increased awareness and provision of: -Supplies – stocks in some districts were increased Provided several FP methods through health facilities though many never provided a youth friendly environment Antenatal and postnatal care services “Sometimes when we go to hospital, some drugs are out of stock so we have to go and buy. Sometimes condoms are not there.” Female Youth FGD, Kumi There remains some negative perceptions about FP services including potential side effects of some FP methods Health education and IEC to improve knowledge (IEC) and attitudes towards FP/RH e.g. - condoms as well as how to use them/apply them - side effects of FP methods and dealing with perceived dangers “Some say that live sex is much more fun, so they don’t want to use condoms” Female FGD Mayuge “Some youths don’t know how to use the condoms” Female FGD Mayuge lx| STRIDES PERFORMANCE EVALUATION REPORT Annex G-2: Functionality Status of YFS in STRIDES supported HFs District Name of Health Facility YFS Functional, Y/N Mityana Sekanyonyi HCIV Yes Kyantungo HCIV Yes Mwera HCIV No Mayuge Mayuge HCIII Yes Baitambogwe HCIII No Kamuli Namwendwa HCIV Yes Nankandulo HCIV No Kaliro Namugongo HCIII Yes Budiini HCIII No Kayunga Bbaale HCIV Yes Galiraaya HCIII No Mpigi Mpigi HCIV Yes Muduma HCIII Yes Luwero Zirobwe HCIII Yes Kasana HCIV Yes Nakasongola Kakooge HCIII Yes Fransciscan HCIV Yes Nakasongola HCIV No Sembabule Sembabule HCIV No Lwebitakuli HCIII No Mateete HCIII No Ntuusi HCIV No Kasese Bugoye HCIII No Kyenjojo Kyarusozi HCIV No Kamwenge Rukunyu HCIV No lxi| STRIDES PERFORMANCE EVALUATION REPORT Annex G-3: Number of Health Workers trained in Youth Friendly Services by District 2011 & 2012 Female Male Overall 2011 11 5 16 2012 120 17 137 Total 131 22 153 Annex G-4: Number of Health Workers trained in Youth Friendly Services by District Female Male Overall Bugiri 18 2 20 Kalangala 3 3 6 Kaliro 10 0 10 Kamuli 20 0 20 Kamwenge 9 0 9 Kasese 15 5 20 Kayunga 4 2 6 Kumi 6 6 12 Kyenjojo 9 0 9 Luwero 6 1 7 Mayuge 10 0 10 Mityana 9 0 9 Mpigi 3 0 3 Nakasongola 6 1 7 Sembabule 3 2 5 Total 131 22 153 Annex G-5: Knowledge gain for health Workers trained in YFS District Pre-Test Post-Test Knowledge Gain Bugiri 64.4 94.0 29.7 Kalangala 52.2 85.2 31.0 Kaliro 56.6 89.2 32.6 Kamuli 57.7 88.6 31.0 Kamwenge 52.2 89.7 37.4 Kasese 48.1 69.9 21.9 Kayunga 54.8 87.8 33.0 Kumi 58.8 76.2 32.7 lxii| STRIDES PERFORMANCE EVALUATION REPORT Kyenjojo 53.3 91.7 38.3 Luwero 65.1 90.6 25.4 Mayuge 52.5 92.5 40.0 Mityana 38.8 88.9 48.5 Mpigi 50.7 90.0 39.3 Nakasongola 54.3 91.4 37.1 Sembabule 59.4 85.6 26.2 Overall 55.1 86.4 32.3 lxiii| STRIDES PERFORMANCE EVALUATION REPORT Annex H: Consultants’ Work-plan and Schedule Activity Timing Outputs/ April May June deliverables 7-11 & 14-17 April* 22-25 April* 28 April – 2 May 5-9 May 12-16 May 19-23 may 26-30 May 2-6 June 9-13 June 16-20 June 23-27 June 30 June – 14 July Signing of award; internal consultants’ meeting; identification & assembling of documents Desk review of documents, inception meeting with USAID/STRIDES; preparation and submission of draft inception report and tools; Submit Inception Report & Tools X Inception report Receive feedback on draft inception report & tools; discussion of inception report & tools; Finalize inception report & tools; submission of final inception report & tools; Recruitment of RAs. Training of team members; assemble logistics; make appointments with stakeholders Field data collection in Kampala lxiv| STRIDES PERFORMANCE EVALUATION REPORT Field data collection in districts & finishing up Kampala Data Processing & Analysis Oral Presentation to USAID; X Analyze feedback; incorporate in draft report Submit Draft Report X Draft Report Presentation to USAID/STRIDES X PP presentation Revision and production of Final Report Submit Final Report X Final report Presentation to USAID/STRIDES and other Stakeholders *18th and 21st April are holidays (Good Friday and Easter Monday respectively) lxv| STRIDES PERFORMANCE EVALUATION REPORT Annex I: Travel Schedule to Districts (May 25th – June 8th) Sun May 25 Mon May 26 Tue May 27 Wed May 28 Thu May 29 Fri May 30 Sat May 31 Mon June 2 Tue June 3 Wed June 4 Thu June 5 Fri June 6 Sat June 7 Team 1: Eastern (TL=Dr John Ssengendo ) Trave l Mayug e Mayug e Mayug e + travel Kamuli Kamuli Kamuli + Travel Kumi Kumi Kumi + travel Nama - yingo Nama - yingo Nama - yingo Team 2: Central (TL=Dr David Mafigiri) Trave l Semba￾bule Semba- bule Semba- bule + Travel Mityana Mityana Mityana + travel Nakaso￾ngola Nakaso- ngola Nakaso- ngola + travel Naka￾seke Naka- seke Naka- seke Team 3: Western (TL=Dr Denis Muhangi) Trave l Kasese Kasese Kasese + travel Kyenjoj o Kyenjoj o Kyenjoj o + travel Kyegegw a Kyegegw a Kyegegw a Trave l Note: Saturdays have been included in the travel schedule considering that health facilities are open on Saturdays, and that community interviews such as FGDs with youths can also be conducted on Saturdays. lxvi| STRIDES PERFORMANCE EVALUATION REPORT Annex J: Evaluation Design Matrix Evaluation Question Sub question (will help answer the key evaluation question) Indicator/ Performance Measure (information needed to answer the question Data Source (primary and or secondary) Data Collection Instrument Data Analysis Plan Q1a. To what extent was the ‘Fully Functional Service Delivery’ model under STRIDES an appropriate and effective approach to achieve intended results? Appropriateness of the FFSD Model What were the assumptions underlying the FFSD model and how realistic were they in the Ugandan and district-specific contexts? What challenges that existed in the delivery of RH, FP and CS services at the time STRIDES started and how was the FFSD model designed to address them? Were there any changes in the model, and what were the reasons and results of these changes? What are the stakeholders’ perceptions about the appropriateness of the FFSD model. Did the suggested solutions prove to be suitable and feasible? Effectiveness of the FSSD Model To what extent and in what ways were the different providers’ (health facilities, community actors…) capacities strengthened and supported to enhance the provision of RH/FP and CS services? Is there improved availability of essential commodities at the facility level? Alignment between assumptions, strategies and needs on ground Outcome indicators as stated in PMP, specifically; %ge of HFs complying with national norms & standards; %ge of HFs offering modern FP methods; %ge of HFs offering EmNOC; %ge of HFs offering CEmONC; %ge of HFs offering FP counseling and/or services; %ge of HFs offering LAPM; improved client satisfaction with services. KIIs with STRIDE program staff, member of the DHMT, officials from MoH, other key stakeholders Health facility interviews Review of program documents Project M&E system data, KIIs with STRIDE program staff, member of the DHMT, officials Semi￾structured interview guide Health facility interview tool Document review checklist Data extraction form Semi￾Content and thematic analysis of qualitative data Statistical analysis. lxvii| STRIDES PERFORMANCE EVALUATION REPORT Evaluation Question Sub question (will help answer the key evaluation question) Indicator/ Performance Measure (information needed to answer the question Data Source (primary and or secondary) Data Collection Instrument Data Analysis Plan What are the trends with regard to the demand or utilization of RH/FP and CS services at community and facility level? Extent to which the initiative’s intended results (outputs or outcomes) have been achieved What are the challenges and lesson learned from the implementation of the model? Improved coordination between actors, availability of supplies, community involvement, better financing etc. from MoH, other key stakeholders Health facility interviews Focus group discussions with health workers structured interview guide Health facility interview tool Q1b. To what extent does the FFSD model link and ensure that the facility, human resources for health (HRH), service delivery and community components worked together? Whether health workers trained under STRIDES were retained Whether the health workers trained under STRIDES have adequate tools and equipment to work with Linkages between health facilities and communities in terms of; — the extent of community awareness of services — Extent to which VHTs/communities are able to provide RH/FP and CS services — the trends in uptake of services — the functionality of referral systems from VHTs What are the trends with regard to the Proportion of staff trained by STRIDES still at their work stations %ge of HFs adhering to norms with regard to equipment and essential commodities %ge of VHTs providing RH/FP and CS services %ge of VHTs making referrals to HFs Review of program documents Project M&E system data, Health facility interviews Document review checklist Data extraction form Semi￾structured interview guide Health facility interview tool Content and thematic analysis of qualitative data Statistical analysis. lxviii| STRIDES PERFORMANCE EVALUATION REPORT Evaluation Question Sub question (will help answer the key evaluation question) Indicator/ Performance Measure (information needed to answer the question Data Source (primary and or secondary) Data Collection Instrument Data Analysis Plan demand or utilization of RH/FP and CS services at community and facility level? Q2. To what extent was the Performance Based Financing model under STRIDES an appropriate and effective approach for improving private / NGO sector service delivery and thereby increasing access and availability of RH/FP and CS services? What were the assumptions underlying the PBF approach and how realistic were they in the Ugandan and district-specific contexts? To what extent were the key design features of the approach suited to the nature and character of sub￾contractors? What challenges that existed in the delivery of RH, FP and CS services at the time STRIDES started and to what extent was the PBF model adequately designed to address them? What changes if any, have taken place in the design of the PBF approach, for what reasons and with what outcomes? What lessons have been learnt from the implementation of PBF approach? Did PBF contribute improved health care delivery and quality of healthcare? What are the other determinants of success for PBF – from communities Alignment of the institutional framework and set up of PBF with national policies and legal framework Extent of achievement of targets by PBF contractors Increased uptake of services as measured in outcome indicators in the PMP E.g. number of clients using FP methods. Perception about quality of care provided by NGOs/private sector (as compared to what baseline status Review of program documents Review of monitoring data, KIIs with STRIDE program staff, DHMTs, officials from MoH; NGOs/private sector in and those not in the program FGDs with service users Interview guides FGD guides Content and thematic analysis of qualitative data lxix| STRIDES PERFORMANCE EVALUATION REPORT Evaluation Question Sub question (will help answer the key evaluation question) Indicator/ Performance Measure (information needed to answer the question Data Source (primary and or secondary) Data Collection Instrument Data Analysis Plan to national level? What is the potential for scale up of PBF? To what extent is there a supportive legal/policy framework? What would be the resource requirements for scaling up the approach nationwide? How did the contractors under PBFM (NGOs, private transporters, health facilities private wings etc.) perform? To what extent did they meet set targets? What are the trends in service in utilization of RH/FP and CS services in private wings? Q3.To what extent do reproductive health approaches and interventions implemented by STRIDES address the unique needs of youth within the 15-25 age group in Uganda in comparison to the other age groups? What are the unique factors affecting uptake and utilization within the 15-25 age group? What are these unique needs of the youth? How is STRIDES program targeting the unique needs of youth –males and females aged 15-25? Are the youth accessing and utilizing RH/FP services? Factors are affecting uptake and utilization of Reproductive Health, Trends in health care service utilization, by youths Positive changes in making health services youth friendly %ge of health units offering youth friendly services Youth client satisfaction with services KIIs with STRIDE program staff, member of the DHMT, officials from MoH, other key stakeholders Focus group discussions Health facility interviews Interview guides FGD guides Content and thematic analysis of qualitative data lxx| STRIDES PERFORMANCE EVALUATION REPORT Evaluation Question Sub question (will help answer the key evaluation question) Indicator/ Performance Measure (information needed to answer the question Data Source (primary and or secondary) Data Collection Instrument Data Analysis Plan Family Planning, and Child Survival (RH/FP/CS) services among youths aged 15-25 years Review of STRIDES M&E database Review of documents lxxi| STRIDES PERFORMANCE EVALUATION REPORT Annex K: Evaluation Tools Tool 1: Interview guides for Key informants at National Level Categories a. Selected staff of USAID/Uganda b. Selected STRIDES program staff c. Ministry of Health (MOH) officials d. Selected managers and program staff from contractors and partners e. Other USAID-funded implementing partners working on similar issues Questions About STRIDES  Tell us more about the STRIDES project? (probe for: goal, key result areas, project strategies- FFSDS, PBC, management and leadership (M&L), activities undertaken by STRIDES etc.)  What is your role in relation to the implementation of the STRIDES project?  What were the differences across districts in the scope of project activities/interventions? ‘Fully Functional Service Delivery’ model  Do you know about the STRIDES Fully Functional Service Delivery’ (FFSD) model? Tell me a little bit more about it  What changes if any have taken place in the design features of the model? Why and what have been the outcome of this?  Perceptions about the effectiveness of the FFSD model Probe: — Extent to which it has improved the capacity of health facilities to deliver quality RH/FP and CS services — Comment about the contribution of the model in improving access to essential commodities, infrastructure, human resources, information management etc. — Extent to which program has strengthened health facility based and community based referral/service networks — What are some the unexpected effects or outcomes of FFSD model? — Do you think the FFSDS has contributed towards the improvement of the Ugandan health system? How?  To what extent does this model link and ensure that the facility, human resources for health (HRH), service delivery and community components worked together? lxxii| STRIDES PERFORMANCE EVALUATION REPORT — In particular, how has the model strengthened communities’ capacities to provide RH/FP and CS services (e.g. their support to VHT?) — How has the model supported the districts and facilities to better collect and share information?  What are the challenges and lesson learned from using the ‘Fully Functional Service Delivery’? (strengths, weaknesses of this model/) Performance Based Contracting/Financing  How does the STRIDES performance based contracting (PBC) model work? Tell me a little bit more about it? Probe for: o Who are the actors involved? o Systems in place to monitor the quality of services, check for clarity in roles and responsibilities, incentives  What kind of coordination is there between the STRIDES subcontractors (NGOs under PBC – Name the existing subcontractor is the region/district) and the District, service providers, communities, and stakeholders?  To what extent was the Performance Based Financing model under STRIDES an appropriate and effective approach for improving private / NGO sector service delivery and thereby increasing access and availability of RH/FP and CS services? Probe for: o STRIDEs experiences with the PBC model e.g. the establishment of private wings in the government facilities—views on service utilization in the private wings; quality of services , what have been the outcomes? o Experience with hiring private transporters—their views on transport service uptake, trends, efficiency of service etc. o What has been the impact of the PBC model on access to health care delivery and quality of healthcare? (Probe: Extent to which the PBC model is an appropriate and effective approach for improving health care service delivery) o What are some the unexpected effects or outcomes of PBC? o What are the other benefits in contracting private providers and/or Civil Society Organization for provision of health services? o Does PBC make a difference in terms of equity and targeting the poor and most vulnerable to receive RH/CS and FP services? o Factors that influence the effectiveness PBC model– from communities to national level?  Challenges encountered in implementation of the PBC  Adaptations that can be made to the PBC strategy used by STRIDES for more effective outcomes  What is the potential for scaling up of PBC model used by STRIDES? Probe for: lxxiii| STRIDES PERFORMANCE EVALUATION REPORT o To what extent can PBC model be mainstreamed into the wider health system? Should PBC be seen as a permanent or temporary approach for engagement of private sector/ NGOs to financing/organizing a health system? o What is the feasibility of replicating the PBC approach within Uganda health care delivery system? Does it align well with existing systems and approaches? o What resources would be required? o Is the legal/policy framework supporting to scaling up this model? o How can the systemic approach best be operationalized to ensure institutional embedding of PBC? What are appropriate exit strategies for actors to explore? o How can PBC best be scaled up in view of sustainability and the required capacities at decentralized as well as national level, while maintaining autonomy and responsibilities for results at a local level? Addressing Unique needs of Youth15-25 o What are their unique needs as compared to other age groups? o What are the factors affecting uptake and utilization of reproductive health within the 15-25 age group?  How are STRIDES’s reproductive health approaches and interventions addressing these needs? What is the difference for males and females aged 15-25?  Has the STRIDES program effectively targeted the unique needs of youth –males and females aged 15- 25? Are the youth accessing and utilizing RH/FP services? Other Questions Do you have any other comment(s) or suggestions that you would like to make related to our discussion? Tool 2: Interview guide for selected managers and program staff from contractors and partners About STRIDES  How does your organization work with STRIDES (Probe for activities, strategies, etc.)  What is your role in relation to the implementation of the STRIDES project? ( Probe for ...)  About the PBC model Probe for: o Institutional framework and set up of PBC including the actors involved o Systems in place to monitor the quality of services  To what extent was the Performance Based Financing model under STRIDES an appropriate and effective approach for improving private / NGO sector service delivery and thereby increasing access and availability of RH/FP and CS services? lxxiv| STRIDES PERFORMANCE EVALUATION REPORT Probe for: o STRIDES experiences with the PBC model o What has been the impact of the PBC model on access to health care delivery and quality of healthcare? (Probe: Extent to which the PBC model is an appropriate and effective approach for improving health care service delivery) o What are some the unexpected effects or outcomes of PBC? o What are the other benefits in contracting private providers and/or Civil Society Organization for provision of health services? o Factors that influence the effectiveness PBC model– from communities to national level?  How do the partners view the overall partnership with MSH and STRIDES?  What kind of coordination is there between the STRIDES subcontractors (NGOs under PBC – Name the existing subcontractor is the region/district) and the District, service providers, communities, and stakeholders?  Has this collaboration improve the demand of services? How? lxxv| STRIDES PERFORMANCE EVALUATION REPORT DISTRICT LEVEL Categories  District Health Officer and other members of the District Health Management Team (DHMT)\  Selected health facility managers/in-charges,  representatives of local organizations involved in the implementation of STRIDES  Selected STRIDES project clients / beneficiaries. Tool 3a: Interview Guide District Health Officer and other members of the District Health Management Team (DHMT) About STRIDES  Do you know the STRIDES Project? What do you know about it? (probe for: goal, key result areas, activities undertaken by STRIDES etc.)  What are the activities undertaken by STRIDES project? (probe for the activities RH/FP Child Survival and Nutrition)  What are some of the key strategies/approaches used under the STRIDE program to improve access and utilization of Reproductive Health (RH), Family Planning (FP), and Child Survival (CS) services at the facility and community levels o Probe for: FFSDS, PBC, M&L  Whom does the STRIDES project work within the district? About STRIDES support  What support have you as DHMT received from the STRIDES project? (probe: -If trainings, what type of trainings, what topics, how many people trained? -If equipment – what and how many? -How useful have been the trainings received?  Who determined/determines the support you get?  Do you think the support from STRIDES helped to strengthen the capacity/functionality of the district heath system? Why? Why not?  To what extent has the STRIDEs programme contributed to improvements in access to and quality of RH/CS/FP services provide in the district? — Extent to which it has improved the capacity of health facilities to deliver quality RH/FP and CS services — Comment about the contribution of the model in improving access to essential commodities, infrastructure, human resources, information management etc. lxxvi| STRIDES PERFORMANCE EVALUATION REPORT  Comment of the quality of your HMIS data. Has STRIDES done anything to help improve the quality of your HMIS data?  Tell us about the coordination of programs between DHMT, providers/Facilities, local authorities … please provide examples.  What kind of coordination is there between the STRIDES subcontractors (NGOs under PBC – Name the existing subcontractor is the region/district) and the District, service providers, communities, and stakeholders? Performance Based Contracting/Financing  How does the STRIDES performance based contracting (PBC) model work? Tell me a little bit more about it? Probe for: o Institutional framework and set up of PBC including the actors involved o Systems in place to monitor the quality of services  To what extent was the Performance Based Financing model under STRIDES an appropriate and effective approach for improving private / NGO sector service delivery and thereby increasing access and availability of RH/FP and CS services? Probe for: o To what extent has the PBF model increased service delivery? o What makes PBC A good approach? What are the limitations? o What has been the impact of the PBC model on access to health care delivery and quality of healthcare? (Probe: Extent to which the PBC model is an appropriate and effective approach for improving health care service delivery)  E.g. the establishment of private wings in the government facilities—views on service utilization in the private wings; quality of services, what have been the outcomes?  Experience with hiring private transporters—their views on transport service uptake, trends, efficiency of service etc. o What are some the unexpected effects or outcomes of PBC? o What are the other benefits in contracting private providers and/or Civil Society Organization for provision of health services? o Factors that influence the effectiveness of the PBC model– from communities to national level?  Challenges encountered in implementation of the PBC  Adaptations that can be made to the PBC strategy used by STRIDES for more effective outcomes  What is the potential for scaling up of PBC model used by STRIDES? o What is the feasibility of replicating the PBC approach within Uganda health care delivery system? o What resources would be required? lxxvii| STRIDES PERFORMANCE EVALUATION REPORT o Is the legal/policy framework supporting to scaling up this model? Other questions  Which other organization (s)/projects are supporting you in similar/related activities like STRIDES in the area? (Probe for what they do similar or different from STRIDES, how do they relate to STRIDES, how they can collaborate. Tool 3b: Interview Guide District Health Officer/DHMT (in Control Districts)  What is the status of health services in this district with respect to access to and quality of RH/FP/CS services? — What is the capacity of health facilities to deliver quality RH/FP and CS services? — What is the status of availability of:  essential commodities,  infrastructure,  human resources,  information management etc.  quality of care  Comment of the quality of your HMIS data. — Are all your health facilities able to report on time? — What proportion report on time?  Apart from health facilities, who else is involved in providing RH/FP/CS Services in your district? — Probe for NGOs, private sector — How do you coordinate with/monitor the private providers? — Comment of the contribution of these providers in ensuring accessibility and availability of services  Which organization(s)/projects are supporting you in RH/FP/CS work? — Probe for what they do? — What contribution have they made to the improvement of services?  Do you know the STRIDES Project in the neighboring districts of ………..(name them)? — How different are health services in your neighboring districts supported by STRIDES compared to your own district? Tool 4a: Interview Guide for Health Facility Managers/In-Charges ‘Fully Functional Service Delivery’ model lxxviii| STRIDES PERFORMANCE EVALUATION REPORT  What is the status of health services at your facility with respect to functionality and quality of RH/FP/CS services? — Comment on your capacity to deliver quality RH/FP and CS services? — What is available/what is missing in terms of:  essential commodities,  infrastructure,  human resources,  information management  IEC/BCC Materials  quality of care FFSDS checklist  Adequate Infrastructure  Equipment  Trained and motivated staff  IEC/BCC  Job Aides  Community support  Administrative Support  Client satisfaction  Gender sensitiveness  Medicines/contraceptives  Supplies About STRIDES support  In what ways has STRIDES supported this HC/hospital? (TA, Equipment, training, etc.). Please give examples (or show equipment) of this support …name trainings  Do you think STRIDES has helped strengthen the capacity of this HC? Why? Why not?  Do you think this health facility is fully functional? Why? Why not? What needs to be done? (Please refer to the FFSDS checklist – try to assess “globally” some of the components through questions and observation of the facility)  How is the HC reaching out to the community with health programs? Which ones?  Are there any programs/training addressing quality of services? Any QI tools applied (COPE – Client oriented – provider efficient; Performance Improvement; Facilitative supervision; etc.). Have QI and client satisfaction been assessed? Any survey applied?  Has STRIDES helped improve the quality of services? How? (training, introducing tools, Quality Improvement approaches – which ones?)  How does the HC coordinate programs/activities with the DHMT? STRIDES? NGOs? Communities? INGOs? lxxix| STRIDES PERFORMANCE EVALUATION REPORT  Is there any “referral” system between community agents (NGOs, VHTs, CHW, CBDs, TBAs, etc.) and the HC?  What would you recommend STRIDES to be focus on in the next two years? Other comments/recommendations Performance based financing  Do you know the strategy PERFORMANCE-BASED CONTRACTING (PBC)? Please explain?  Perceptions about the PBC  Do you think  What kind of coordination is there between the STRIDES subcontractors (NGOs under PBC – Name the existing subcontractor is the region/district) and the District, service providers, communities, and stakeholders?  Has this collaboration improve the demand of services? How?  Do you have any kind of recommendations to improve this coordination? Addressing Unique needs of Youth, 15-25 o In your opinion what are the unique factors affecting uptake and utilization of reproductive health within the 15-25 age group?  To what extent do reproductive health approaches and interventions implemented by STRIDES address the unique needs of youth–males and females aged 15-25?  Has the STRIDES program effectively targeted the unique needs of youth –males and females aged 15- 25? Are the youth accessing and utilizing RH/FP services? Tool 4b: Interview Guide for Health Facility Managers/In-Charges (In Control Districts)  What is the status of health services at your facility with respect to functionality and quality of RH/FP/CS services? — Comment on your capacity to deliver quality RH/FP and CS services? — What is available/what is missing in terms of:  essential commodities,  infrastructure,  human resources,  information management  IEC/BCC Materials  quality of care FFSDS checklist lxxx| STRIDES PERFORMANCE EVALUATION REPORT  Adequate Infrastructure  Equipment  Trained and motivated staff  IEC/BCC  Job Aides  Community support  Administrative Support  Client satisfaction  Gender sensitiveness  Medicines/contraceptives  Supplies  Do you think this health facility is fully functional? Why? Why not? What needs to be done?  Comment of the quality of your HMIS data. — Are you able to report on time every month? — Out of 12 months, how many are you on time?  Are there trained VHTs in your catchment area? — How do you work with the VHTS? — What proportion of them are active (make referrals, submit reports) — How else is the HC reaching out to the community with RH/FP/CS services? — Who else is involved in providing RH/FP/CS Services in your catchment area? — Probe for NGOs, private sector — Comment of the contribution of these providers in ensuring accessibility and availability of services  Which organization(s)/projects are supporting you in RH/FP/CS work? — Probe for what they do? — What contribution have they made to the improvement of services?  Do you know the STRIDES Project in the neighboring districts of ……….. (name them)? — How different are health services in your health facility compare to those supported by SRIDES? — Do you think STRIDES has helped strengthen the capacity of those Health Centres? Why? Why not? Addressing Unique needs of Youth, 15-25 o In your opinion what are the unique factors affecting uptake and utilization of reproductive health within the 15-25 age group? lxxxi| STRIDES PERFORMANCE EVALUATION REPORT  To what extent do your reproductive health approaches and interventions address the unique needs of youth–males and females aged 15-25?  Are you able to effectively target the unique needs of youth –males and females aged 15-25? Are the youth accessing and utilizing RH/FP services? Tool 5: Focus Group Discussion Guide for Clients Attending ANC OR FP Clinics 1. How would you describe the state of health services in your community? (Good/poor; reasons for this description. Where else do people go to seek health services? Are there other health centres?) 2. What health services are provided at this health centre? 3. What do you think about the services provided here? a. Do they meet your needs? b. Are they affordable? 4. Have FP/RH/CS services at this centre changed over the last five years? How would you rate the health services provided at this health centre compared to five years ago? 5. (If improved) What could be the reasons? 6. What do you like most about the services here? 7. What don’t you like about the services here? 8. Are you satisfied with the quality of services that the program provides? (Please explain reasons for your answer) 9. What challenges do you find in accessing services at this health centre 10. What improvements in services would you like to see at this health centre? Tool 6: Focus Group Discussion Guide for Community Health Workers including VHTs [This tool is to be administered to community health workers such as VHTS. VHTS have been trained with support from STRIDES to strengthen the link between health facilities and communities. They were expected to be involved in providing services such as immunization, ANC, FP and others] Introduction 1. How would you describe the general state of Family Health in your area (district or Parish)? 2. Background information – when was VHT formed/selected, how many members, male/female composition 3. Did you receive training? Who trained you? When? Roles and Involvement 4. What are your roles (roles you are supposed to do?) 5. What roles do you do in practice (probe: are there roles you are supposed to do but not able to do? Which ones and why?) 6. What roles are specifically played in the areas of FP/RH, CS, 7. What motivates/drives you to do this work? lxxxii| STRIDES PERFORMANCE EVALUATION REPORT 8. What support do you get from the health centres? 9. What support do you get from the community? 10. What support do you get from the local government (sub-county, parish, LC1 committee/chair person)? 11. Where else do you get support from (any NGOs, CBOs, and private sector)? 12. Do you enjoy your work? Why/why not? 13. Is your work useful to the community? How has the community benefitted? 14. Is the community happy with your work? 15. Do you know the STRIDES project? 16. How were you involved with it? Service Delivery 17. Are there any activities/services that the program has introduced in this area? 18. How did STRIDES improve your work? 19. How has STRIDES work improved the availability of health services in your community? Quality 20. Has the STRIDES for Family Health Program improved the quality of health services in your community? 21. Are you satisfied with the quality of services that the program provides? (Please explain reasons for your answer) 22. How would you describe the quality of health services provided by this program? 23. How would you rate the quality of services provided by this program compared to the ones you had before? Access 24. Has the program increased health care access in this area? Please explain (probe for what ways and services have increased compared to pre-STRIDES) 25. What are the strengths of this program? 26. What are the weaknesses of the program? Probe for challenges faced in implementing STRIDES 27. Overall, what is there to show that the health of community members in your community has improved? Sustainability 27. Will you continue serving your community as a VHT? Why/Why not? 28. How can your work be improved? Tool 7: Focus Group Discussion Guide for Women of Reproductive Age 1. How would you describe the state of health services in your community? (Good/poor; reasons for this description. Where do people go to seek health services? Which is the most preferred source of health services? Why?) 2. What health services are provided at [refer to health centre supported by STRIDES]? 3. What do you think about the services provided at that health centre? a. Do they meet your needs? b. Are they affordable? 4. Have the services at this centre changed over the last five years? How would you rate the health services provided at this health centre compared to five years ago? lxxxiii| STRIDES PERFORMANCE EVALUATION REPORT 5. (If improved) What could be the reasons? 6. What do you like most about the services at that centre? 7. What don’t you like about the services at that centre? 8. Are you satisfied with the quality of services that the program provides? (Please explain reasons for your answer) 9. What challenges do you find in accessing services at this health centre 10. What improvements in services would you like to see at this health centre? Tool 8: Focus Group Discussion Guide for Male And Female Youths Aged 15-25 Years 1. What health services are available for the youth in this community? (Who provides them?) (probe for both facility-based and community-based services) 2. Where do young people like you go to seek health services? Which is the most preferred source of health services? Why?) 3. What do you think about the services provided (at that health centre, in the community, etc.)? a. Do they meet the needs of the youth? b. Are they friendly to the youth? c. Are they affordable to the youth? 4. Have the services at this centre changed over the last five years? How would you rate the health services provided at this health centre compared to five years ago? 5. (If improved) What could be the reasons? 6. What do you like most about the services at that centre? 7. What don’t you like about the services at that centre? 8. Are you satisfied with the quality of services that the program provides? (Please explain reasons for your answer) 9. What challenges do you find in accessing services at this health centre 10. What improvements in services would you like to see at this health centre? Tool 9: Focus Group Discussion/Group Interview Guide for Health Workers 1. How would you describe the general state of Family Health in your area (Sub-county or Parish)? 2. What do you know about the STRIDES for Family Health Program? Service Delivery 3. What support has the STRIDES for Family Health Program provided to your facility? a. Are there any new services that the program has introduced in this facility/community? b. Has the program improved health care infrastructure? Please explain c. Has the program supported any trainings for health workers? Quality 4. Do you find the support provided by the Strides for Family Health Program to your facility appropriate to your needs? lxxxiv| STRIDES PERFORMANCE EVALUATION REPORT 5. Has the STRIDES for Family Health Program improved the quality of health services provided by your facility? 6. Are you satisfied with the quality of services that you provide? (Please explain reasons for your answer) 7. How would you rate the quality of services provided by your facility compared to five years ago (i.e. before STRIDES support) Access and Service Delivery 8. Has the program support helped to increase health care access in this community? Please explain (probe for what ways and services have increased compared to pre-STRIDES) 9. Has the program improved on the availability of essential drugs and commodities 10. Has the program improved on community outreach and participation? 11. What plans do you have to sustain the activities/services of this program? 12. What are the strengths of this program? 13. What are the weaknesses of the program? Probe for challenges faced in implementing activities under the STRIDES program. Tool 10:Secondary Data Extraction Guide - Quantitative indicators to extract from secondary sources 1 Maternal and Child Health Indicators (From 2007-2013)  Trends in %ge of pregnant women who receive 4 ANC consultations over the last 5 years (for STRIDES supported health facilities; analysis by district and region, comparison with national indicators where possible)  Trends in %ge of pregnant women who received 2+ doses of IPT over the last 5 years (same as above)  Trends in %ge of live births delivered from a health facility over the last 5 years (same as above)  Trends in %ge of underweight children at measles vaccination over the last 5 years (same as above)  % live births with low birth weight (same as above)  Number of children who at 12 months have received three doses of DPT vaccination from a USG￾supported immunization program  Number of children under 5 years of age who received Vitamin A from USG-supported programs.  Percentage of clients satisfied with health services received 2 Reproductive Health and Family Planning Indicators  Number of USG-assisted Service Delivery Points providing FP counseling or services  Percentage of health facilities (HC III & above) offering long acting and permanent methods (LAPM)  Number of service providers trained by STRIDES in FP/RH/CS  Percentage of USAID supported Service Delivery Points offering any modern contraceptive e method  Percentage of Service Delivery Point complying with national norms and standards (relevant to FP counseling)  Number of implants and IUDs inserted 3 Fully Functional Service Delivery Systems (FFSDS): Number of STRIDES – supported health facilities with: I) Adequate infrastructure to support RH/FP & CS services; II) Sufficient medical equipment and supplies; III) Availability of all essential drugs and commodities; lxxxv| STRIDES PERFORMANCE EVALUATION REPORT IV) Presence of sufficient human resources, who are trained and motivated; V) Adequate health financing, availability of sufficient information; VI) A functional referral system; VII) Community outreach and participation; VIII) Client satisfaction; and quality of care; 4 Linking the Community Component with the FFSDS Quantitative or qualitative data about the work of VHTs Expected results Year 3 target End of project target % villages with functional VHTs 35 50 % VHTs with stock-outs of FP tracer commodities 35 20 5 Performance Based Financing Model  Number and percentage of contractors who achieved pre-determined performance targets, milestones or results  Number and percentage of contractors with outstanding performance  Number and percentage of contractors who failed to achieve the agreed upon deliverables  Number of private wings established in HCIVs and functionality of private wings  Service utilization/uptake in established private wings (trends since establishment) in HC IVs to generate revenue that can be used by the public wing to purchase necessary supplies  Number of partnerships established with private transport providers to improve access to MCH services (how many transporters per region; trends in clients uptake of transport services—could ask M&E system, private transporters)  Number of partnerships established with health related NGOs to facilitate referral of moderate to severe cases of malnutrition to the selected facilities; (trends in numbers of persons referred; and follow up the rehabilitation of treated cases) (How many identified; number of children and mothers referred?).  Number of clients receiving services from a USAID-affiliated private sector service 6 Meeting the Unique Needs of Youth aged 15-25  Percentage of targeted health units offering Young People-Friendly Service