JUNE 2007 This publication was produced for review by the United States Agency for International Development. It was prepared by Peter Barron, Andrew Balyeku, Allan Kebba, Christine Lalobo and Justine Nankinga, Management Systems International. FINAL EVALUATION OF UPHOLD PROGRAM FOR HUMAN AND HOLISTIC DEVELOPMENT(UPHOLD) MANAGEMENT SYSTEMS INTERNATIONAL 3rd Floor, Cotton House Tel: 256-41-253978/9 15 Clement Hill Road Fax: 256-41-253980 P.O. Box 6959 Kampala, Uganda E-mail: info@mems.co.ug FINAL EVALUATION OF UPHOLD Funded by the United States Agency for International Development under the terms of the award number 617-A-00-02-00012-00 June 2007 Final Report Evaluation Team Peter Barron Andrew Balyeku Allan Kebba Christine Lalobo Justine Nankinga Contents EXECUTIVE SUMMARY ..................................................................................................................i ACRONYMS.......................................................................................................................................v 1 INTRODUCTION .......................................................................................................................1 1.1 PROJECT BACKGROUND .........................................................................................................1 1.2 EVALUATION PURPOSE ...........................................................................................................1 1.3 EVALUATION METHODOLOGY................................................................................................2 1.4 LIMITATIONS OF THE EVALUATION.........................................................................................2 2 EVALUATION FINDINGS........................................................................................................4 2.1 PROGRAMME DESIGN .............................................................................................................4 2.2 TECHNICAL COMPETENCY EVALUATION................................................................................5 2.3 CAPACITY BUILDING EVALUATION ........................................................................................6 2.4 EVALUATION OF EXTERNAL FACTORS....................................................................................7 2.5 PERFORMANCE EVALUATION IN SPECIFIC INTERVENTIONS ....................................................8 2.5.1 Child Health Interventions.............................................................................................9 2.5.2 Integrated Reproductive Health ...................................................................................10 2.5.3 Malaria .........................................................................................................................11 2.5.4 Tuberculosis Prevention and Control...........................................................................17 2.5.5 HIV/AIDS ....................................................................................................................18 PIASCY...........................................................................................................................18 Abstinence Being Faithful and other HIV prevention.....................................................20 HIV Counseling and Testing ...........................................................................................20 Prevention of Mother to Child HIV transmission (PMTCT)...........................................22 Palliative care...................................................................................................................25 RH in War Affected Districts..........................................................................................26 2.5.6 Quality Assurance........................................................................................................26 2.5.7 Supervision ..................................................................................................................27 2.5.8 Monitoring and Evaluation ..........................................................................................28 2.5.9 LQAS ...........................................................................................................................29 2.6 DEVELOPMENT AND MANAGEMENT OF RELATIONSHIPS .......................................................30 2.6.1 UPHOLD Relationships with Central Government Partners.......................................30 2.6.2 UPHOLD Relationships with its Consortium and USAID Partners............................30 2.6.3 UPHOLD Relationships with Local Governments and CSOs.....................................30 2.7 UNINTENDED RESULTS.........................................................................................................34 3 KEY LESSONS LEARNED .....................................................................................................35 4 RECOMMENDATIONS...........................................................................................................37 5 CONCLUSIONS........................................................................................................................38 APPENDIX 1: Scope of Work for UPHOLD.......................................................................................I APPENDIX 2: Evaluation Framework...........................................................................................VIII APPENDIX 3: Schedule of Activities............................................................................................. XII APPENDIX 4: Persons Involved ..................................................................................................... XV APPENDIX 5: Interview tools......................................................................................................... XX APPENDIX 6: Bibliography......................................................................................................... XXII i EXECUTIVE SUMMARY UPHOLD is a 5 year integrated social services program, implemented under USAID’s strategic objective 8, which aims to improve human capacity. UPHOLD was contracted under a cooperative agreement, originally scheduled to end in September 2007. The evaluation took place between April 16th and June 22nd 2007. The evaluation team of five carried out a range of interviews, undertook a number of district and site visits and read through a wide range of documents. The details of these are contained in the appendices of this report. The evaluation report has a number of limitations. These include the fact that no previous evaluations were done so there was no baseline to which the evaluation team could refer; some of the key program objective changed during the duration of the programme (e.g. introduction of PIASCY); the funding climate in USAID changed fundamentally during the lifetime of the program which had considerable impact on UPHOLD (e.g. introduction of PEPFAR). The heart of the report deals with answering the question: “To what extent has UPHOLD achieved its overall HIV/AIDS and health goals and results”? The report highlights the extent to which external factors such as the delay in the selection of districts, the process of “re-districting” impacted on the operations of UPHOLD. It also highlights the many issues in the design of the programme, (e.g. constraints placed upon, and the prescriptive nature of the funding), that impacted on the ability of UPHOLD to maneuver. The report also gives details of the achievements, or lack thereof, of a number of direct activities carried out by UPHOLD. These include activities of child health including EPI and growth promotion, reproductive health, malaria (including the function of CMDs and the distribution of ITNs), TB, HCT and PMTCT, AB and other HIV prevention, palliative care and PIASCY. A number of cross-cutting activities that were important in ensuring capacity development were also detailed. These were the areas of grant making and decentralization, improvements in quality assurance and monitoring and evaluation. A number of lessons have been learned and a number of recommendations have been made. These are listed below. 1) There is a danger that cooperative agreement arrangements leave programme goals ‘too loose’. This was the case in UPHOLD where the programme tried to ‘do everything, everywhere’ with limited and finite resources. The consequence was that some activities were very thin on the ground. It took two years before there was a “Focus for Impact”. This resulted in a narrowing and deepening of activities. Future programmes should ensure that the parameters are more targeted and that interventions should have sufficient depth to make meaningful changes on the ground. 2) The strategic decision making in UPHOLD was not as clear as it could have been. The major decisions around funding flows were made by USAID and did not leave the UPHOLD management with a huge amount of discretion around implementation. In addition UPHOLD management did not have a steering committee or reference group which could provide an oversight role in terms of strategic direction. Finally UPHOLD did not adequately tap into the intellectual resources available in its consortium partners with regard to strategic direction. It should be built into the design of future programmes that there should be a formal annual strategic decision making meeting where key stakeholders’ input is provided and which is based on a strategic review of performance, direction and focus. 3) The relationship between the donor, USAID, and the key stakeholder, the GoU, was not as clear as it should have been and resulted in a loss of 6 months at the commencement of ii the project through a delay in district selection. Large programs should be endorsed within the GoU prior to commencement with clear goals and objectives. 4) UPHOLD was not subject to any formal evaluation during the five years of operation. There was no mid-term review nor were there any formal annual reviews. As a result it was difficult to assess whether the programme was ‘on track’ in relation to the big picture. All stakeholders were thus deprived of the opportunity to engage with the strategic direction of the programme. For a program of this size, a formal mid-term review is imperative, and annual reviews would also ensure that the programme is heading in an appropriate direction. 5) Without systems strengthening to link with the services programme, UPHOLD ‘walked on one leg’, trying to implement a district based programme but always having to consider systems issues at the delivery (LG) level as well as at the central MoH level. Programs that are trying to improve service delivery need to take into account constraints in the formal health system. 6) Central MoH involvement in large programs is essential. This involvement can: a) assist the strategic direction in ensuring that program goals are consistent with health system policy b) help with sustainability by getting MoH participation in and buy-in to activities of the program (e.g. LQAS) c) open doors to the program through providing it with continued endorsement Programs should build into their design a more formal relationship with the central MoH, where roles and responsibilities are clearly delineated. ƒ Many of the excellent activities implemented by UPHOLD have question marks against their sustainability after the end of the program. Examples of this include the future of the CMDs and the LQAS. Sustainability strategies should be transparent and a critical consideration in the design and implementation of large programs. ƒ Although UPHOLD had the laudable aim of integration of activities at the district level, this is difficult to achieve when the health systems are running different projects in a vertical fashion (e.g. PMTCT). Programmes need to design their activities around the realities of the health system and feed back to central level well documented policy briefs to encourage systemic change of service innovations. ƒ Although this program was designed as a multi-sectoral program (health and education) to be delivered at district level, there was no prototype or pilot on which the design was based. Large scale programs should be based on tested methodologies. Innovative activities have an important role but they should be tested in small scale pilots before going up to scale. ƒ The efficiency and effectiveness of interventions and activities by grantees was compromised by the uncertainties around the funding flows in terms of the length of time of these and the amounts. The funding used generally by UPHOLD, to a large extent determined by its own funding streams, meant that grants were made for a single year or less; and that the amounts of the grants were uncertain. This resulted in pulsed, uncertain and inefficient grant making to CSOs. There should be more certainty in programme funding flows and more flexibility given to the program to make grants for periods longer than one year. Where circumstances are appropriate grants should be made to allow for continuity of activity. ƒ The differing financials years of UPHOLD, USAID and the LGs often meant that funding did not fit into the financial framework of LGs and did not coincide with budgeting cycles. Funding should plug in the existing planning frameworks and cycles as much as possible. iii ƒ The RFA method of grant making ensures that CSOs have sufficient technical ability to write grant proposals and demonstrate their likelihood of success and so results in the choice of CSOs who are likely to deliver. It however limits the grant making to certain kinds of CSOs and results in much CSO time being spent on writing proposals at the expense of service activity. The best features of the RFAs should be coupled with the targeted selection and capacity development of particular CSOs, in order to ensure sustainability and overall long-term development of these CSOs. ƒ There is a tension between being focused on short term achievements and taking on a developmental role through improving human capacity (which is generally a softer and longer term achievement). Programs need to be clear, transparent and realistic as to what their mission is and what they are trying to do. ƒ UPHOLD has an excellent monitoring system where all program activities were tracked. However, over-prescriptive target setting, together with unrealistic expectations of the accuracy of data, can result in too much attention being placed on the numbers. Monitoring has an important role to play but it should be complemented by strategic analysis and overview, where the big picture is constantly kept in mind. ƒ One of the positive achievements of the program was the use of CSOs who in certain instances made important and large contributions (e.g. in home mobilization of HCT). Future programs should look at the positive ways in which CSOs were harnessed and build upon these. UPHOLD itself should write short policy briefs on the success of this (and other interventions) and the lessons that it learnt in the process. ƒ UPHOLD encouraged and facilitated the relationships between CSOs and LGs. Future programs should further encourage in all ways, including financial, the closer working together of these parties. ƒ Some activities of UPHOLD took a very long time to mobilize, e.g. start-up of TB programme; others were started and then curtailed because of lack of resources e.g. growth promotion. There needs to be a balance between the depth and breadth of the range of activities and also the geographical areas in which they work. Programs cannot be all things to all people. Activities taken on by programs need to be carefully thought through in terms of financial, human and other resources required prior to their commencement and there needs to be a balance between the number of activities and how widespread they are implemented. ƒ There were a number of activities of UPHOLD that resulted in spin-offs that were not anticipated in the original plan. Examples of these include: o Distribution of LLINs through CMDs exposed a channel that had not been used before. This resulted in the CMDs becoming a point of reference for health issues in the villages. This model of CMD distribution has been taken up by the MoH. o LQASs have been well accepted as a means of getting population based information. o PIASCY has resulted in community and household involvement in education. On the other hand it has been reported that some schools not involved in the model school process, have had negative reactions. o Use of CSOs has resulted in high mobilization of decentralized and household HCT and also led to increased utilization of family planning. o Although CMDs and CSOs have been highly successful in certain activities, they now face an uncertain future and their sustainability is not assured. The overall conclusions of the evaluation team are that most of the operational targets set by UPHOLD have been attained and especially in the last year UPHOLD seems to have moved into top gear. At the operational level much has been achieved and CSO capacity has been increased as has LG capacity, though to a lesser extent. Although the evaluation team obviously could not assess the operation of UPHOLD in the initial years 2002-2004, it iv appears in our assessment that UPHOLD in 2007 is a well functioning, hard working, efficient and effective organisation. However, UPHOLD suffered from being an over-ambitious project with inadequate strategic direction, both from without and from within, in the initial years. This resulted in the project having too much breadth and too little depth. The changing of key implementation objectives, (e.g. through PEPFAR, PIASCY implementation), did not help this process. Our overall assessment is that UPHOLD has successfully evolved over time and is making successful and useful interventions. However, with more strategic direction and focus initially, this program could have achieved more. v ACRONYMS ABC Abstinence, Being faithful, and Condoms ACT Artemesinin-based Combination Therapy AIC AIDS Information Centre AIDS Acquired Immune Deficiency Syndrome AIM AIDS Integrated Model District Programme ANC Antenatal care ARVs Anti-Retroviral Drugs BCC Behavior-Change Centered Communication CAO Chief Administrative Officer (local government) CB TB Community Based TB treatment CB TB DOTS Community Based TB Directly Observed Therapy Short Course CBGP Community-based growth promotion CMDs Community Medicine Distributors COP Chief of Party (UPHOLD) CPD Continuous Professional Development CPTCs Core Primary Teacher Colleges CSO Civil society organization CV Curriculum Vitae DCOP Deputy Chief of Party (UPHOLD) DDHS District Director Health Services DDs Drug Distributors DEO District Education Officer DHACs District HIV/AIDS Committees DHT District Health Team DOTS Directly Observed Therapy, Short course EPI Expanded Programme on Immunization FP Family planning FPP Focal Point Person FSG Family Support Group FY Fiscal Year Global Fund The Global Fund Against AIDS, Tuberculosis and Malaria GOANC Goal-oriented antenatal care GoU Government of Uganda HBMF Home Based Management of Fever HCT HIV Counseling and Testing HIV Human Immunodeficiency Virus HMIS Health Management Information System HSSP II Health Sector Strategic Plan II HUMC Health Unit Management Committee IDP Internally displaced persons IEC Information, education, communication IPT Intermittent preventive treatment IRH Integrated Reproductive Health ISSP Integrated Social Services Programme (UPHOLD) ITNs Insecticide-Treated Nets JCRC Joint Clinical Research Centre JSI JSI Research and Training Institute, Inc. LCs Local Councils LG Local Government LLINS Long-Lasting Insecticide Treated Nets LQAS Lot Quantity Assurance Sampling M&E Monitoring and Evaluation vi MEEPP Monitoring and Evaluation of the [US President’s] Emergency Plan for AIDS Relief Progress Project MIP Malaria in pregnancy MIS Management Information System MoES Ministry of Education and Sports MoH Ministry of Health MoLG Ministry of Local Government MoUs Memorandum of Understanding NMCP National Malaria Control Programme NMS National Medical Stores NTLP National TB and Leprosy Programme OVC Orphans and Vulnerable Children PEAP Poverty Eradication Action Plan PEPFAR President’s Emergency Plan for AIDS Relief PIASCY Presidential Initiative on AIDS Strategy for Communication to the Youth PLHA People Living with HIV/AIDS PMI President’s Malaria Initiative PMP Performance Monitoring Plan PMTCT Prevention of Mother-to-Child Transmission PTC Post-test clubs PTC Primary Teachers Colleges PY Programme Year QA Quality Assurance RFA Request for Application RH Reproductive Health SNIDS Sub-National Immunization Days SO8 USAID Strategic Objective 8: Improved human capacity STI Sexually Transmitted Infections TA Technical Assistance TASO The AIDS Support Organization TB Tuberculosis TPI Training and Performance Improvement TPI/QA Training and Performance Improvement/Quality Assurance UAC Uganda AIDS Commission UACP Uganda HIV/AIDS Control Project UDHS Uganda Demographic and Health Survey UNAHCO Uganda National Health Consumers Organization UNEPI Uganda National Expanded Programme for Immunization UNICEF United Nations Children’s Fund UPE Universal primary education UPHOLD Uganda Program for Human and Holistic Development UPMA Uganda Private Midwives Association USAID United States Agency for International Development US$ United States Dollars USG United States Government VCT Voluntary Counseling and Testing VHT Village Health Team YSP Yellow Star Programme Final Evaluation of UPHOLD programme 1 INTRODUCTION 1.1 PROJECT BACKGROUND UPHOLD is a 5 year integrated social services program, implemented in terms of USAID’s strategic objective 8, which aims to improve human capacity. UPHOLD was delivered under a cooperative agreement, originally scheduled to end in September 2007 but extended to September 2008. The aim was contribute to USAID/UGANDA strategic objective of improved human capacity intermediate results (IR) of effective use of social services (IR1); increased capacity to sustain social services (IR2) and; strengthened enabling environment for social sector services (IR3). Within IR2, UPHOLD has 5 key performance areas namely Public service delivery, quality of public service delivery, support to public service delivery, private sector delivery and Grants Programme. The project was thus designed in the RFA to deliver a package of high quality technical assistance, targeted training and capacity building activities, and financial and material support to 20 district, their sub-counties and selected private organizations in public health and education systems. The theory was to achieve success by (1) learn from the processes adopted by AIM for selecting districts and rapidly cover all 20 target project districts with innovative approaches within the first project year; (2) direct most effort toward specific vulnerable groups in every district; (3) building cooperation and collaboration especially between Health, Education and HIV/AIDS sectors at every level; (4) going national by replication and expansion to achieve nationally measurable improvements. UPHOLD activities were thus implemented in seven broad technical domains some strictly within one of the three sectors: (1) Health, (2) Education, (3) HIV/AIDS and the remaining four areas integrated these sectors: (4) Education/Health, (5) Education/HIV/AIDS, (6) Health/HIV/AIDS and, (7) Education/Health/HIV/AIDS. An Integrated Coordinating Committee (ICC) that includes major stakeholders from public (MOH, MOES and MoLG), partners, private and professional sectors and USAID was planned to maximize breadth and depth in joint strategic and annual planning exercises and regular meetings among stakeholders especially in unfolding Services Agreements and the Contracts. 1.2 EVALUATION PURPOSE This evaluation was awarded under the Scope of Work shown in Appendix 1. The evaluation purpose was to extract lessons that would benefit the USG/Uganda and GOU partner institutions with future programming and inform implementing partners of what worked and what did not work during implementation of the program. Four key evaluation questions guided the process: 1) “To what extent has UPHOLD achieved its overall HIV/AIDS and health goals and results? How did the following factors contribute to the achievement of goals and results? • Program design strengths and limitations 1 Final Evaluation of UPHOLD programme • Technical competency to undertake the planned activities • Technical support and capacity building for districts and grantees • External factors 2) How did UPHOLD develop and manage relationships with USAID implementing partners, GOU partners and the civil society? 3) Did UPHOLD yield any unintended positive/negative results 4) What are the key lessons learned from the design and implementation of this large multi￾sectoral, district based program?” The evaluation provides both quantitative and qualitative assessments of the approach UPHOLD applied and determines whether UPHOLD is on track to achieve its impact objectives. USAID showed interest in continuing support to HIV/AIDS and health service delivery in a decentralized environment. While learning from old and ongoing activities within and outside the USG portfolio, USAID seeks to garner lessons learned regarding design and implementation of large, multisectoral programs as one means of rapidly scaling up HIV/AIDS and health services in a decentralized setting. The evaluation provides answers both at program and strategic level by addressing the question of whether UPHOLD is on track to achieve the intended goals of its HIV/AIDS and health interventions. The evaluation also distills lessons learned about program implementation that have a bearing on scaling up HIV/AIDS and health interventions nationwide. 1.3 EVALUATION METHODOLOGY A team of five evaluators composed of four external consultants, two of whom were public health doctors, one with experience of local government and decentralization, the fourth seconded from the Ugandan Ministry of Health and the fifth, a senior staff member seconded from UPHOLD provided a wide range of work and background experience and included. The evaluation took place between April 16th and June 22nd 2007, implemented in two phases. The team leader and one of the team spent a week in Kampala from April 16th to April 21st 2007 to set the platform for the subsequent in-depth phase of interviews and field trips that took place between May 20th and June 9th 2007. Given the time and logistical constraints, the district review was confined to 2 regional offices, 2 districts in depth and, another 2 districts more superficially. The report was written from June 10th to 21st 2007. Based on an initial review of the documentation, the time available and the geographic constraints of visiting districts, the evaluation team prepared an evaluation framework and proposed field trip, including the selection criteria for the districts visited. These documents, shown in Appendix 3, comprised the essential components of the inception report. This inception report was presented at a meeting held on the 23rd May 2007, attended by representatives of USAID, the GoU, and other stakeholders. The team carried out in-depth interviews with a range of national and district stakeholders, undertook a number of district and site visits and took part in a number of activities in the districts visited. The details of these are contained in Appendix 2. The interview questionnaires are contained in Appendix 5 and the documents reviewed are listed in Appendix 6. The evaluation was limited to the Health and HIV/AIDS domains thus excluding Education which was undergoing a separate evaluation. 1.4 LIMITATIONS OF THE EVALUATION UPHOLD, is reportedly the largest USAID programme in the whole of Africa in terms of monetary value. It is also a very complex programme in terms of the range of interventions and the geographical coverage. The time allocated to this evaluation was relatively small and therefore, by definition, the evaluation could not be in-depth and cover every aspect of UPHOLD over a 5 year period. 2 Final Evaluation of UPHOLD programme There were no previous evaluations, such as a mid-term review, which could have been expected in a programme of this scale, with which to compare this evaluation. Neither were there formal annual reviews with which to compare and therefore the evaluation team had to cast their eyes back over a period of 5 years without any real landmarks or baselines upon which they could rely1 . Over the lifetime of the programme, some of the key implementation objectives changed. So in fact what UPHOLD does in 2007, is quite different in a number of ways from what was envisaged in 2002. (For example there are very little child health or reproductive health activities in 2007 and there is a massive activity around PIASCY, which was non-existent, nor even conceptualized at the commencement of the programme). During the lifetime of the programme the funding climate considerably changed and PEPFAR, with its emphasis on results, numbers and monitoring and evaluation had a significant impact on how UPHOLD implemented its activities and how it relates to its various partners. Notwithstanding the above comments on the limitations of this evaluation, the team feels confident that with the information obtained through our interviews, document reviews and field visits that we are able to give a balanced assessment of UPHOLD, based on our collective judgement of the programme. The remainder of the report follows generally along the lines of the structure of the key evaluation questions. 1 There was reportedly an internal JSI management review approximately midway through the programme, following which there was a sharper focus of UPHOLD and its activities. This review was not made available to the evaluation team. 3 Final Evaluation of UPHOLD programme 2 EVALUATION FINDINGS “How did the programme design’s strengths or limitations contribute to the achievement of goals and results?” 2.1 PROGRAMME DESIGN The original RFA put out by USAID was broad in concept and was extremely ambitious in envisaging the integration of the three broad areas of work. The range of interventions spanned across a very wide scope of work and the number of districts chosen was also large for the range of interventions proposed. The concept had not been piloted prior to the RFA and in the opinion of the evaluation team was too broad both in the number and range of interventions and in the geographic area. Although designed as a district-based programme the design was not altogether coherent and over one third of the programme budget was for UPHOLD to act as a funding conduit for channeling USAID funds to two large national CSOs, namely TASO and AIC. It appeared to the evaluation team that for this component of the budget, as if USAID was using UPHOLD to carry USAID’s management responsibility. In the original RFA, it was envisaged that a “sister” programme strengthening support systems would be awarded. UPHOLD and this systems programme were supposed to be complementary to each other, with the systems programme focusing on central and national levels. The systems programme was never awarded, for reasons that were not made clear to the evaluation team. As a result of this gap, UPHOLD needed to be more involved at the central level. Consequently, UPHOLD has faced a constant balancing act in trying to meet its district level commitments with trying to meet some of the demands of central level ministries and departments. The programme was awarded as a cooperative agreement, which allows for flexibility in the deliverables. However, the actual implementation objectives were too open-ended and this resulted in too many activities being implemented (e.g. each of the specific components in health such as malaria and child health had a variety of different activities). As a result the programme had different deliverables in different geographical areas and was subsequently thin on the ground. The programme design was highlighted by very specific and earmarked donor funding. This resulted in the overall strategic direction of the programme being dependent on the decision of the funding allocation by the donor (e.g. US$ 2.3million, equivalent to 3% of the total programme budget was allocated to reproductive health. Spread over 5 years and 202 districts this is equal to around US$ 23,000 per district per year; and is probably insufficient to have had a significant impact on overall reproductive health.) This funding affected the strategic decision making around the key implementation objectives and gave the programme management limited scope for maneuverability. The programme was designed to be implemented in a structurally integrated manner. However, in practice it was difficult to implement in this manner. For example in the Ministry of Health, many of its HIV/AIDS programmes are managed in a vertical manner such as PMTCT. In addition a major component of UPHOLD, added at an advanced stage of the programme implementation, PIASCY, was designed as a national programme implemented country-wide in all districts, not only in the UPHOLD designated ones. Further evidence of the lack of an integrated programme is evidenced by the fact that this evaluation team has specifically had the education component excluded from the evaluation. 2 Redistricting increased districts to 34 4 Final Evaluation of UPHOLD programme 2.2 TECHNICAL COMPETENCY EVALUATION How did the programme team’s technical competency to undertake the planned activities contribute to the achievement of goals and results? Assessment of the impact of the technical competency to undertake the planned activities on the achievement of the goals and results of UPHOLD was extremely difficult to evaluate. There were two main reasons for this. The first is that the staff composition has changed over the five years of the programme and the second is that the evaluation team generally had to rely on indirect assessments of technical competency. The evaluation team found that UPHOLD in 2007 to be an organization with an extremely high work ethos with the key personnel clearly on top of their work. UPHOLD’s organizational ability was very efficient and was exemplified by flawlessly organized field trips and presentation of documents for the evaluation team. UPHOLD was found to have impressive financial and administrative standards. UPHOLD appeared to have a culture of sharing among staff and it also appeared to be an organization where staff learned on the job and it has a pro-active approach to staff development. UPHOLD generally seemed to have recruited a high caliber of skilled staff that have been competent to do the work expected of them. Over the lifetime of the programme the number of expatriates employed by UPHOLD has decreased from twelve to three. In essence the staff of UPHOLD have appeared to have high technical competency and have been in control of the areas of work for which they have been responsible. However, there have been certain exceptions where the technical competence has not been as high as it could have been and this has influenced the activities. Examples of this have been in tuberculosis and at times in reproductive health. Although UPHOLD has made use of the partners in the consortium, who have all contributed to the staff complement, it was reported that UPHOLD did not make optimal use of the available expertise in the consortium partners. A corollary to this is that generally the operational technical competency has been good and operational implementation carried out in a thorough and expert manner. However, insufficient attention was given to the strategic direction of the programme, and consortium partners were not adequately utilized to assist in this most important aspect of the programme. 5 Final Evaluation of UPHOLD programme 2.3 CAPACITY BUILDING EVALUATION “How did the programme’s technical support and capacity building for districts and grantees contribute to the achievement of goals and results?” As part of its overall implementation framework, UPHOLD adopted the grants strategy as a modality to achieve programme goals and objectives. In the programme start up period it became increasingly clear that LGs and CSOs grantees would require capacity building to shore up existing capacities and enhance achievement of results, especially considering the fact that resources channeled through both represented a significant proportion of UPHOLD grants. While capacity building inputs targeted assistance to all grantees, a significant proportion targeted CSOs specifically focusing on core technical competencies areas such as planning, budgeting, programme implementation (in technical areas), partnering (writing MoUs and mediation skills), reporting related to activities and finances, as well as monitoring and evaluation. In the case of LGs, targeted capacity building efforts occurred at three levels. There was specific training that focused on imparting skills and knowledge to functionaries in technical areas to facilitate improvement in access to and use of services; there was on the job training, hands on support and other forms of focused training to familiarize financial and technical functionaries with activity and financial reporting templates; while the third level focused on training elected officials in the area of advocacy. The general assessment is that these capacity building efforts targeted at LGs may not have resulted in the magnitude of expected impact on implementation performance. However, the LGs that were interviewed did report that the capacity development efforts made some contribution to achievement. In general, the evaluation reveals that the tailored capacity building efforts significantly improved core and technical competencies of CSOs. This was manifested in the documented and observed implementation performance. In respect of LGs, the achievements were more modest but there were noticeable improvements in respect of implementation performances. While the rigorous RFA process would account for the CSO performance, it is worth noting that the tailored training delivered by UPHOLD also contributed significantly to this outcome. The results of a CSO follow up capacity exercise reveals that as far as core and technical capacities were concerned there was great improvement in planning, reasonable improvement in financial and technical reporting as well as M&E and partnership relationships. However, the evaluation also revealed that the least improvement was in the area of budgeting, which has proved to be a challenging area for all CSOs. The evaluation team found it difficult to find a linear link between improved capacity and improved results. The only other issue with regard to capacity building was the extent to which it focused on achieving results. As a result of the focus on numbers, capacity development efforts were oriented to enhance capacities in that direction. As a result there were diminished efforts and resources placed on systems and institutional strengthening as a process. 6 Final Evaluation of UPHOLD programme 2.4 EVALUATION OF EXTERNAL FACTORS “How did external factors contribute to the achievement of goals and results?” UPHOLD was essentially designed as a district based programme where there would be integrated capacity development in the three areas of health, education and HIV/AIDS. The programme was designed to be implemented in 20 districts and the selection of these districts was fundamental to UPHOLD initiating activities. However, for a number of reasons that are not clear to the evaluation team, the final selection of districts was delayed for a period of 6 months, from November 2002 until May 2003, when the districts were agreed upon by GOU and USAID. In effect, the delayed decision making around the selection of districts resulted in a loss of the essential implementation component by around 10% (6 months out of 5 years) of the total programme time. There were a number of other issues related to the districts that had an adverse effect on the functioning of UPHOLD. Over time the number of districts has increased from 20 to 34, in a process known as “redistricting”. Although the geographical area in which UPHOLD is working has remained the same, the number of LGs, CAOs, DDHSs and DHTs, with which UPHOLD has to interact have multiplied, resulting in increased work for UPHOLD staff especially the regional office teams. The conflict areas in which UPHOLD has worked have changed over the lifetime of the programme. The intensity of the conflict has changed and the geographical areas in which the conflict has occurred have also changed. As a result of this UPHOLD has had to constantly adjust its work to take into account the impact of the conflict on the people and organizations in these areas. Districts have been heavily reliant on the graduated tax for raising revenue to pay for aspects of service delivery in the districts. This tax was abolished leaving districts short of essential funding. As a result there was pressure on UPHOLD to use some of its resources to plug the gaps left by the abolition of the graduated tax. In 2006 there were national and presidential elections. As a result of political activity there was a marked reduction in service delivery and activity throughout the districts. This reduced the ability of UPHOLD to function effectively for a period of around 2-3 months. During the lifetime of the programme the funding climate of USAID changed radically with the introduction of PEPFAR and PMI. These presidential initiatives had far-reaching effects on UPHOLD in terms of narrowing the focus of the programme, prescribing the deliverables and influencing the monitoring and evaluation components of UPHOLD. 7 Final Evaluation of UPHOLD programme 2.5 PERFORMANCE EVALUATION IN SPECIFIC INTERVENTIONS To what extent has UPHOLD achieved its overall HIV/AIDS and health goals and results? The USAID’s Integrated Services Cooperative Agreement (ISCA) called for a careful crafted package of high-quality technical assistance, training and capacity building, as well as financial and material support to districts and non-governmental organizations. Strategically focused to increase access, utilization, quality support and sustainability of services in Education, Health and HIV/AIDS through an integrated approach, UPHOLD works in six broad technical areas and which include 1) Child Health; (2) Adolescent Health (Youth Friendly Services); (3) Integrated Reproductive Health; (4) Communicable Disease Control (Malaria and TB); (5) HIV/AIDS and; (6) Primary School Education. The overall effort was assessed through level of funding for the UPHOLD programme by area of intervention is shown in Figure 1. This table clearly indicates the main funding thrusts with education taking one fifth of the total budget; HIV central level grants to TASO and AIC, where UPHOLD was used as a conduit for funding with limited responsibility for the outcomes of the use of the funds, took one third; HIV/AIDS under the guise of PEPFAR used just over 30%; whilst all of the health related activities took up 16%. Figure 1: Funding for UPHOLD by USAID by intervention area In the health section it can be seen that malaria took the lion’s share with 6% (including PMI) of the total and the remainder was thinly spread across the major areas of reproductive health, child health and tuberculosis with around 3% each. The funding flows had wide and far￾reaching implications for the activities and results obtained in each of the specific areas of intervention. There were a couple of general issues which impacted on all of UPHOLD’s activities. One of the key issues was to develop UPHOLD’s annual plans so that these simultaneously reflected UPHOLD’s priorities as well as national and district priorities and consequently ensured stakeholder buy-in. This required much flexibility on UPHOLD’s part and as a result much of the impact seen in UPHOLD districts was an accrual of efforts from many actors and is difficult to attribute to UPHOLD alone. As mentioned in the section on design issues impacting on UPHOLD, with the systems programme not being funded, a number of health sector challenges had to be overcome. 8 Final Evaluation of UPHOLD programme These included a shortage of health workers, a lack of facilities, and inadequate supply of basic equipment and supplies. UPHOLD tried to address some of these issues in some of the districts by providing grants to the district local governments and CSOs to improve delivery of services in health units. 2.5.1 Child Health Interventions UPHOLD’s provided 4% of its budget to improve child health focusing on immunization and a less extent on support to community-based growth promotion (CBGP) as an entry point to integrated child health care. A. Childhood Immunization UPHOLD’s child health component focussed on disease prevention through supplementing (1) Routine immunization, including strengthening delivery in hard to reach areas; (2) National Bi-annual Child Days; (3) Sub-National Immunization Days and (4) Mass Measles, Polio and Tetanus campaigns. UPHOLD support included national level coordination and planning for the mass campaigns; training and follow up of service providers; technical assistance in programme design; planning, monitoring and evalution; funding for social mobilization; BCC and job aids to improve the functioning of health support personnel in districts where activities were carried out. UPHOLD’s support to the EPI contributed to the following : • Coverage of children between 12 and 23 months who received their 3rd dose of DPT by 12 months increased from 50.8% to 72.2% to 84% in 2004, 2005 and 2006 respectively, in UPHOLD districts.3 • Achieved more than 100% coverage during the 2006 mass measles campaign in targeted areas. The high percentage was because some children over the target age were provided immunization and vitamin A supplementation. • Coverage of 2003 Mass Measles, Polio and Tetanus Campaign – achieved more than 98% coverage in UPHOLD districts. • Coverage of Vitamin A supplementation for children 6 to 59 months reached 82% in 2006 Child Days Plus4 Since UPHOLD is only one of the many contributors to EPI, it is difficult to single out success areas and apportion these directly to UPHOLD. • Radio programmes for Radio Listening Clubs on 10 stations in 9 languages produced and boadcast (35,000 active club members; topics included immunization, child feeding, fatherhood) • BCC support to SNIDS in North and North West – flyers, posters, radio spots, banners in 3 languages • Mr. Immunizer poster in 6 languages and Mr. Immunizer sticker for children • Radio spots on 10 stations, instructions and support, • Posters for Child Days, Illustrative examples of BCC materials produced for Child Health. • ‘Health Worker Matter’ newsletter on routine immunization with MOH (10,000 copies for 34 districts), 3 LQAS 2004, 05 and 06 4 Based on health facility data 9 Final Evaluation of UPHOLD programme B. Growth Promotion Community-Based Growth Promotion was the principal strategy for promoting and contributing to integrated child health services at the local and household level. In line with the MoH Village Health Team (VHT) approach, UPHOLD trained and facilitated community volunteers to empower communities to take responsibility for their own health. Through monthly village-level weighing sessions for young children, these volunteers identify children with sub-optimal feeding and care practices. These problems are then addressed through individual counseling to help mothers to improve on these and to re￾establish or maintain adequate growth. The monthly weighing sessions are also an opportunity to address issues of immunization, antenatal care, childbirth practices, and other child and maternal health issues. Since 2005, UPHOLD focused these efforts in 6 districts. These were chosen based on child health needs, the likelihood of sustainability due to district priorities and geographic location. Complete training and counseling materials as well as tools for record keeping and reporting have been developed and community volunteers have been trained to implement the strategy. Results of this intensive work and prototype materials are intended for adaptation and use throughout Uganda in conjunction with other activities to improve child health. Some of the selected results are shown below: ƒ 80 district and sub county level trainers and 1,225 Community Growth Promoters trained to cover on-going growth monitoring and promotion in 500 villages in 6 districts. ƒ 15,000 children attended the monthly village level growth promotion sessions for weighing and individual counseling. ƒ Discussions indicated a decline in missed immunization as a result of Community-Based Growth Promotion. 2.5.2 Integrated Reproductive Health UPHOLD’s primary interventions in IRH focus on maternal health, particularly GOANC and family planning. UPHOLD supported reproductive health interventions are primarily delivered through CSOs5 . These organizations carry out advocacy activities and community mobilization for increased use of services, with some of them providing services as well. Additionally, UPHOLD supports the Uganda Private Midwives’ Association (UPMA), whose members offer integrated reproductive services at their private clinics. Availability of RH services IRH activities were implemented in 14 UPHOLD-supported districts with varying approaches in response to local district priorities. These were mainly through strengthening existing RH service delivery points by training health workers in better logistics management for provision of commodities. UPHOLD supported the promotion of service utilization through BCC. By using CSOs, the hard to reach populations especially in conflict areas were reached. 5 These include Rakai AIDS Information Network (RAIN), Rukungiri Women Development Company (RWODEC), Tooro Kingdom, Bushenyi Medical Centre, Straight Talk Foundation and Rural Health Concern (RUHECO). 10 Final Evaluation of UPHOLD programme Quality of RH service delivery By engaging CSOs with expertise and experience in specific IRH areas, quality was maintained. Furthermore, UPHOLD assisted in the expansion of youth-friendly sexual and reproductive health programming and participated in the national policy dialogue about these issues with the MoH and with other partners. Training and education focused on increasing capacity in the health system, extending networks for better private care and supporting community reproductive health workers. Extensive community mobilization and advocacy were also important components. Of the 55 UPHOLD supported sites for PMTCT, health workers in 52 of the sites have received training in GOANC. Utilization of RH services Utilization of FP services has been high with over 300,000 women counseled on FP during their 1st ANC visit in 2006. ANC attendance exceeded the annual targets and was expected to surpass the end of programme target of 2,900,000 visits. Over 11,000 mama kits were procured and distributed to IDP camps in 5 conflict affected districts. Coverage of RH services GOANC training for service providers in UPHOLD districts played a part in improving the percentage of women who completed their 4th ANC visit to 53%, up from an original 48%. There has been a 28.8% increase in the number of women accessing clean and safe delivery kits in UPHOLD districts. The percentage of women receiving IPT1 and IPT2 during their last pregnancy has risen to 35.8% since 20056 . Impact UPHOLD largely achieved its stated targets in the districts were IRH was implemented. In FP over 265,000 couple years of protection were reached. In the last two years the percentage of women who have delivered at a health unit has risen to 50% in UPHOLD districts, up from 41% in 2004. In UPHOLD districts 20.5% of women aged 15-49 use modern methods of contraception. Issues Though ANC attendance of at least four times in UPHOLD supported districts may be higher than the national average the coverage remains low at 53%. The uptake of IPT2 is still low. The IRH intervention was thinly spread across the UPHOLD districts and the resources made available were insufficient to achieve meaningful changes in reproductive health. 2.5.3 Malaria Malaria activities were allocated 6% of the total programme resources made available by USAID. Malaria interventions were strengthened in FY2005/6 when UPHOLD was tasked by USAID, under the PMI, to focus on selected interventions7 in specific districts. Prior to this, resources were provided to districts across many activity areas and while results were realized, the scale and depth of these was mainly seen in the training of CMDs. Consequently, most of the malaria intervention impact is seen after FY2005/6. Eight of the twenty districts supported by UPHOLD were in areas where conflict and insecurity presented significant challenges to the implementation of core interventions. Two of the districts, Kitgum and Gulu, had major population displacements with over 600,000 IDPs, comprising 80% and 45% of their populations respectively. UPHOLD’s approach to 6 LQAS 2006. 7 Distribution of ITNs, net re-treatment, supporting home-based management of fever and increasing utilization of intermittent preventive treatment of malaria in pregnancy. 11 Final Evaluation of UPHOLD programme these districts was adaptive and flexible with a sharing of effort with other humanitarian community stakeholders including USAID and other USAID-funded partners. Modifications made to UPHOLD from the start of PMI included procurement and distribution of 260,000 long lasting ITNs in four districts with IDP Camps, and in five non￾conflict districts which on the basis of LQAS surveys had the lowest bed net coverage. UPHOLD continued its intensive support to the HBMF strategy and purchased new Homapak®8 stocks, and also used existing MoH stocks of Homapak®, throughout the life of the programme even though the government’s malaria treatment policy at facility level changed to ACT. A. Home Based Management of Fever The HBMF component targeted 4,949,795 children below 5 years of age. Implementation was through the existing HBMF structures and geared to support district efforts within the parameters of the national malaria control strategic plan of 2005/10. HBMF Service delivery HBMF interventions were initially started in 14 districts in 2003 and were later expanded to all UPHOLD districts by the end of 2004. During FY2006/7 UPHOLD was still training CMDs to cover the gaps created by attrition. At the same time UPHOLD was still supporting the existing CMDs. The programme had a target of ensuring increased accessibility to services at the community level by having two trained CMDs in each of the 19,203 villages which covered most of the UPHOLD supported districts. By training 25,570 CMDs out of the targeted 38,406, the programme reached two thirds of its stated target. This is approximately equivalent to 1 CMDs per 130 children under 5, though it was much higher in the designated districts. These CMDs served as community based outlets for malaria treatment and they also were a source of prevention against malaria messages. The CMDs provided an additional resource to the alleviation of malaria and supported the work of other facility based health providers. Quality of HMBF Services The programme improved the quality of malaria services. This was achieved especially through supporting the sharing of experiences at quarterly CMDs meetings, provision of logistics such as bags and, supporting the integration of CMDs reports into the routine district reporting systems. Use of registers by CMDs increased from 60% in 2003 to 71% in 2006. One of the reasons for failing to increase the use of registers further is attributed to the weakness in supervision of CMDs by facility staff and the appropriate use of CMDs information contained in the registers. Discussions also revealed that, in some cases, parallel systems for distributing medicines to CMDs by UPHOLD still persisted and is indicative of low integration of the CMDs into the community and facility management systems. UPHOLD did however attempt to improve the data management and utilization in all the health units in the districts by training health workers on the HMIS, incorporating the record-keeping by the CMDs as well as information on Intermittent Preventive Treatment (IPT). As evidenced by their reporting and programme records, on average, 71% of the CMDs are active. 8 HOMAPAK is an antimalarial tablet formulation for children under 5 years combining Chloroquine and Sulphadoxine/Pyremethamine (CQ/SP) 12 Final Evaluation of UPHOLD programme UPHOLD was also involved in more sophisticated training around malaria and was involved in the training of 62% of facility workers in management of severe and complicated malaria to support referred cases. Utilization of HBMF Services Though use of home treatment declined nationally from 13% in 2002/3 to 2.4% in 2005/6, UPHOLD supported districts showed a doubling in the use of Homapak® from 8% in 2005 to 16% in 20069 . Particular efforts were made to obtain emergency supplies, especially when the national medicine supply system failed. The percentage of households reporting that they had ever used Homapak® (or other home based treatment package) was 26% in 2006. Available records indicate that over 2.5 million episodes of fever in children under five were treated by the trained CMDs in the 29 districts. Sixty five per cent of the children with fever were treated within 24 hours with 92% recovering on Homapak® treatment alone without needing referral. Coverage of HBMF Services The programme managed to reach most of the ill children. The target was to increase the proportion of children under five years old who get appropriate anti-malarial treatment within 24 hours of onset of fever from 40% to 45% in 2005 and by 2006 the programme had reached a coverage target of 77%10. Impact of HBMF services It is estimated that by end of 2006, in UPHOLD districts around 92% of children with malaria recovered without requiring referral to next level. The 2004/05 HBMF results show that 94% of the almost 1.6 million children who utilized community medicine distributor services fully recovered. A review of the community based LQAS surveys shows that in 2006, 77% of children with fever received recommended treatment compared to 31% in 2004. There were a number of contributing factors in programme design to achievements in the HMBF interventions. Working at community level ensured that existing systems of CMDs were used. These included building capacity among the facility health workers to train and support the CMDs, supporting the CMDs quarterly review meetings with transport allowances, supporting the integration of CMDs reports into the district reporting system on malaria and, promoting the utilization of CMDs through messages in the media. There was a high attrition rate among CMDs. After national training, UPHOLD had to fill a gap of 62% to restore coverage to 2 CMDs per district. Based on reporting only 49% of those trained reported back on their activities by end of PY3 but this increased steadily from 67% in PY4 to 71% in PY5 to program efforts. More study is needed to look at those factors that reduce attrition of community volunteers However, reliance on UPHOLD programme funding for meetings undermines their sustainability. Support provided by UPHOLD for review meetings of CMDs, including support supervision and refresher training, will only continue into part of FY07/08 when these allowances will be stopped with the programme winding down its activities. To ensure the continuity of CMDs will require extra resources and is dependent upon funding from external donors such as the Global Fund and PMI. Systems for support supervision of health workers, delivery of medicines to different levels in the district and data retrieval especially at community level remained weak. This is increasingly vulnerable with the roll-out of the new malaria treatment policy. 9 Based on LQAS 2006 results. 10 Based on LQAS results. 13 Final Evaluation of UPHOLD programme Little support was given to the CMDs by the private for profit providers despite their training in malaria case management. (This was seen especially in Mbarara) A number of external factors hindered achievements in this technical area. These included lack of CSOs working in the area of malaria lead to few implementing partners, with little implementation of HBM through CSOs; irregular and delayed distribution of Homapak through the NMS system, parallel activities with other partners in the north, especially the Malaria Consortium and UNICEF and poor linkages developed through district and sector working groups. B. Intermittent Preventive Treatment UPHOLD supported an increase in IPT uptake especially under the PMI. Training in GOANC was the main vehicle of support between 2003 and 2005. Under PMI, the focus is on whole site support supervision of health workers to improve service delivery including communication to clients and BCC activities, including job aides and radio media articles developed with Health Communications Partners. ANC Service delivery In an attempt to promote ANC services UPHOLD supported training of 168 health workers in GOANC covering IPT as well as PMTCT. Under PMI, focused whole site supervision was also carried out to 236 health workers in 79 health facilities in 17 districts. IPT uptake tracked over time shows that targets were met up to FY 2004/05 when IPT coverage was 65% but show a decline in FY 2005/06, where there was coverage of 36%. This latter figure is similar to the national figures of 34% obtained in HSSP II. The peak coverage in 2004/5 is probably due to the fact that this is when initial activities were at their highest level. Quality Activities included improving service delivery quality by building skills and improving performance using on the job training and on-site support supervision. Technical assistance for promoting IPT within the GOANC involved orientation of 43 mentors from 12 districts each to supervise 3-5 facilities to provide on-site support. There was additional training of 31 facility managers who supervise health workers and promote outreach visits. The introduction of the Performance Improvement Approach (PIA) via supervision showed that service quality continued to be dogged by inadequate and irregular supplies of IPT drugs, poor record keeping, poor organization of work flow in the facilities to support IPT and limited skill among health workers. This latter is likely to have resulted in suboptimal impact of BCC activities. However, the 2006 LQAS shows that 72% of households reported hearing at least 1 IEC radio message on malaria prevention in the 12 months preceding the interview. Impact The impact of IPT intervention is expected to be low. This is partly due to low utilization of ANC and compounded by the low proportion of facilities providing IPT, by 2006. Challenges and way forward The continuing challenge is to increase early ANC attendance, which remains low. Just over a half (53%) of women attended ANC at least 4 times.11 There is a need to promote the quality of ANC further by ensuring that health workers offer the full range of antenatal services, including IPT, to clients. 11 LQAS 2006 14 Final Evaluation of UPHOLD programme Supervision tools have been developed and much information is collected. However, supervision needs to be more widely practiced. There has been some resistance to the implementation of supervision. Approaches that promote provider change should be studied more and documented. UPHOLD later shifted to concentrate on GOANC with more emphasis on the context of PMTCT than IPT. However, systemic problems that have limited the availability of PMTCT services to all ANC clients certainly affect the IPT uptake as well. C. Support National Malaria Treatment Policy UPHOLD supported the MoH, the NMCP and districts to strengthen the implementation of the new malaria treatment policy of the Artemisinin-based combination therapy through two rounds of on-site support supervision of health workers in the 34 UPHOLD supported districts Results show that ACT was prescribed in only 41% of cases of malaria despite the fact that over 90% of health workers had been oriented and given the new policy guidelines on management of uncomplicated malaria. Over 80% of the health workers in the UPHOLD supported districts had adequate skills in malaria case management. However over 70% of these health workers were not communicating effectively with patients regarding malaria and its treatment. Other problems included stock outs of ACT in 27% of the supervised facilities. It is difficult to make an assessment of the impact when many health workers are not being adequately trained and supervised. In addition, despite the change in policy at facility level to ACT, the policy at community level (with the exception of the conflict districts of Gulu, Kitgum and Amuru) has not changed and CMDs are still using Homapak for treatment of uncomplicated malaria in the rest of the country. D. Insecticide Treated Nets The re-treatment of ITNs increased over the duration of the programme. The start of the PMI grant stretched the ITN distribution coverage in the northern districts beyond the UPHOLD designated districts in 2006, but in view of the real need in these areas this was seen as an important and necessary deviation from the initial UPHOLD targets. ITN Availability UPHOLD procured and distributed over 311,000 ITNs and thus met the targets for making ITNs available. As a jump start to the PMI activities, 260,000 long lasting insecticide treated nets (LLINs), secured with PMI funding, were distributed and these formed the bulk of the nets distributed. Most of this was in 2006 when there was a 10-fold increase in the volume of ITNs distributed. This distribution was largely supplementary to ITNs provided by other partners, especially the Malaria Consortium. UPHOLD also supported re￾treatment of 144,615 bed-nets in six UPHOLD districts in 2004, 144,869 in 2005 and 505,573 in 27 PMI supported districts (including 12 in UPHOLD) in 2006. Overall number of ITNs procured and nets retreated met targets. Accessibility The accessibility for each designated target group was improved through specific delivery channels. Before the introduction of PMI, relatively few ITNs were distributed to pregnant women through facility-based ANC clinics. This occurred in only 3 districts. People living with HIV/AIDS (PLWA) were reached through post-test clubs and psychosocial support groups formed under HCT and PMTCT interventions. 15 Final Evaluation of UPHOLD programme The nets obtained with PMI funding were ably distributed through using 5,768 CMDs as end-point distributors of LLINs for the nine earmarked districts. These included the 5 districts where usage of bed nets in 2004 was shown by LQAS to be below 10%, and 4 districts in conflict situations (viz Gulu, Katakwi, Kitgum and Lira). The distribution of ITNs through the CMDs was unique and proved effective in quantifying and identifying needy households. Subsequent surveys later found that there was a 95% net retention in households with beneficiary children, more than six months after the net distribution. Utilization ITNs were distributed to the project target groups children below five years (243,1882 through CSOs in 2005; 224,183 in 2006); 25,612 pregnant women; 25,690 people affected by HIV/AIDS and 200 orphans and vulnerable children. A follow on study in the nine districts, six months after the net distribution, showed that 87% of children below five years slept under an insecticide treated net (ITN) the night before the study. Impact The percentage of children under five years who slept under a treated net the night before the LQAS surveys increased from 11.7% in 2004 to 17.2% in 2005. The national level12 shows an increase in the under 5 population falling sick from malaria from 43% in 2002/03 to 56% in 2005. This contrasts with the reduction in under 5 malaria prevalence in the UPHOLD supported districts in the last 2 weeks preceding the survey from 56% in 2004 to 44% in 2006 (LQAS). This might be attributed to the increased availability of preventive services; in particular the ITNs distributed in the 9 districts identified as high risk. An illustrative example is given for Lira district where UPHOLD supplemented on malaria control activities. Data from the district HMIS shows a marked drop in malaria cases though this could have been also influenced by disbandment of the IDP settlements among other factors. Since the figures show a huge drop in prevalence, there is need for further investigation by the project. Figure 2: Incidence of Malaria in Lira District Malaria Incidence Lira District (Source: District HMIS data base) - 5,000 10,000 15,000 20,000 25,000 30,000 July Aug Sept Oct Nov Dec Jan Feb Mar April May June July Aug Sep Octt Nov Dec Jan Feb Mar April 2005 2006 2007 Number of Cases Reported LLIN Distribution 12 UNHS 2005/6 figures 16 Final Evaluation of UPHOLD programme 2.5.4 Tuberculosis Prevention and Control TB is a growing problem with at least 50% of those adults with active TB co-infected with HIV. The detection rate of new smear positive cases (53%) and treatment success were low (60%) at the start of interventions13. This was attributed mainly to limited recording and reporting, poor compliance to DOTS, poor access to health care services, insufficient number of skilled staff, inadequate diagnostic facilities and weak record keeping and reporting systems. Availability UPHOLD’s response to the TB problem was to improve the quality of CB TB DOTS services through supporting refresher trainings on TB diagnosis and management for health workers and laboratory personnel and facilitation of supervisory activities. Originally implementation was focused in the 8 UPHOLD districts that did not overlap with the AIM Programme. However, after the AIM Programme ended in June 2006, TB activities were extended to the former AIM supported districts as well. Facility-based TB services supported UPHOLD’s integrated HIV/AIDS /TB/STI/Malaria supportive supervision initiative that covered 61 health facilities and 325 service providers. In some districts where this was done, marked improvements were observed as early as 2004 (e.g. CBDOTS coverage was high in Rukungiri (87%) and Rakai (99%) districts. Quality Up to fairly recently, UPHOLD did little to improve facility based TB service provision. Although the reason for the lack of support stated by UPHOLD was that health facilities were being supported by other partners, this support was present in only 32% of the health facilities that subsequently received technical support from UPHOLD for provision of TB services. Impact The case detection rate, for districts whose reports were available, (measured by the case notifications) has remained lower than targeted. Only a cumulative total of 671 PLHAs were provided with treatment for HIV and TB up to 2006. The results show that community involvement in HIV and TB prevention and care through training of health workers and volunteers in the provision of community based directly observed tuberculosis treatment (CB-DOTS) has improved TB treatment success rate. This has been seen especially in some districts like Rakai (88%) which are above the national target of 85%. However, the achievement in respect of the TB detection rates however remained lower than the targets set, with an increasing gap every year. Plans in 2006/7 to reach 13,000 health workers with support supervision and expand CB TB DOTS in 34 districts may not improve detection rates up to the targeted levels within the lifespan of the programme. Issues The late start was attributed to low prioritization of TB programmes by districts. This resulted in few requests for support by districts around TB and consequently there was minimal activity around TB. This prioritization changed after a stakeholders meeting was held. However, the expansion of the TB activities only took place in October 2006, close to the UPHOLD project to wind up time. Local governments did not prioritize UPHOLD support in TB, mainly because earlier there were multiple programmes and partners supporting TB interventions. Recently this 13 UPHOLD annual Report 2004 17 Final Evaluation of UPHOLD programme number has decreased in UPHOLD supported districts and requests for support by LG have increased. At this late stage, however, activities are not being rolled out intensively and additional funding is scarce. It is not clear how this revamping, including making local government and CSO grants, and facilitating training via zonal NTLP programme staff support through the UPHOLD Regional Offices to the local governments will affect performance. Current monitoring data shows that case notification rates remain low. Nevertheless efforts have concentrated on training of health facility supervisors and workers, facilitating support supervision activities that strengthened links from the district to the community (in only 6 districts) and provision of 10,000 copies of the Health Workers Matter newsletter on TB in the 34 districts. 2.5.5 HIV/AIDS The HIV/AIDS interventions included HIV/AIDS Counselling and Testing (HCT), Prevention of Mother to Child Transmission (PMTCT), General Palliative Care, integrated HIV/TB Services, Abstinence and Faithfulness, Other Prevention and Orphans and Vulnerable Children (OVC). It was feasible to evaluate a few of these as agreed during the inception report discussions. A. PIASCY President Museveni’s Presidential Initiative on AIDS Strategy for Communication to Youth (PIASCY) was a unique component of the UPHOLD response to PEPFAR and has effectively been implemented in all primary schools throughout Uganda and not just in UPHOLD districts. The activity has been in existence since 2004 with training of 3 teachers in each of the 15,000 government aided primary schools. Coverage PIASCY I, implemented in 2004, facilitated the orientation of 45,000 primary school teachers nationwide, over a six month intensive period, on the delivery of the PIASCY programme. PIASCY II strategy is through school-based interventions focusing on the adoption of behavior change among pupils and the practice of supportive behaviors by teachers and peers. The comprehensive PIASCY II has been in existence for less than 1 year and as a result the implementation experience is limited and an evaluation of outcomes is therefore difficult. PIASCY II was designed as a set of comprehensive and holistic activities at the school and household level to build on PIASCY I. It is an integral part of School Health and Nutrition programming. The approach was to reach all schools through targeting the centralized system of teachers training colleges, under the MoES. A selected number of schools were targeted for an intensive pupil-centered training intervention, so-called “model schools”. So far, 1,078 primary schools, representing approximately 2 schools in each of the 570 coordinating centers, have been reached. It is intended that these schools act as sites of excellence and mentor the surrounding schools, but the effect of this intervention has not yet been tested. These activities included classroom support through radio programming for teachers; children's clubs activities to support life skills and critical thinking about gender and violence; school and community activities to protect girls from defilement; and home support by promoting parent and child communication about delaying and abstaining from sex. 18 Final Evaluation of UPHOLD programme Utilization The percent of primary school children, aged 6-12, who regularly attended school, including all five days preceding the LQAS surveys, increased from 76.9% in 2004 to 82.2% in 2006. This offers a large potential to reach many young people with messages of abstinence. Quality PIASCY is a government programme supported by UPHOLD. Though UPHOLD supported the implementation of the programme, PIASCY largely remains a vertical MoES activity with little control at district local government level. It is envisaged that once training has extended to cover most primary schools, efforts will be made to transform it into a district managed activity. Supervision and monitoring are still weak with almost all the indicators generated in the PIASCY proposal not being used. Impact Education impact is slow and little data are available to offer trends. However, qualitative information shows that enthusiasm has been built. A number of positive stories qualify improvement in school environments, and abilities of teachers, parents and pupils. “I was surprised that all teachers had gone for HCT” MoES Official “We no longer have school drop outs and absenteeism by girls ever since we started this PIASCY programme last year” Headmistress Model School Lira District It was difficult for the evaluation team to use the LQAS results because at the time the survey was done, all schools were engaged in Performing Arts Festivals and this may have been responsible for the high rates found. Issues At this early implementation stage, many operational problems are expected to occur. For example, it has been realized that schools lacked trained HIV/AIDS counselor teachers to provide effective counseling services to pupils infected and affected by HIV/AIDS. This will require closer collaboration with specially trained HIV/AIDS counselors for the age group. New programmes tend to add a burden of work to already over stretched staff. PIASCY has greatly increased the burden of recording and technical reporting on PIASCY school based activities. Many schools have allocated teachers to positions of focal point persons in implementing primary schools. Through dialogue, resistance initially posed by religious leaders and cultural biases are being tackled. This is being done especially through the community participation components in the school programmes. To facilitate continuity and integration of PIASCY, supervision should be integrated within the district, municipality and sub-county school inspectorate system. Efforts should be made to embrace private schools within the PIASCY reach. PIASCY has not been integrated within the district education school program. The large effort applied in getting and harnessing the support from parents and teachers and the 7 day training of all teachers in the country will require enormous resources. 19 Final Evaluation of UPHOLD programme The District Education Departments were found weak in conceptualizing PIASCY objectives and planned activities14. More effort was needed in supporting them, especially in rolling out PIASCY as a district activity. Though indicators were drawn, there is no baseline information for several PIASCY indicators. Implementation of PIASCY is vertical with little connection to other UPHOLD activities. It was felt that UPHOLD districts were not performing any better, in relation to PIASCY, than other districts. B. Abstinence Being Faithful and other HIV prevention UPHOLD supports awareness and encourages positive behavior change through abstinence and faithfulness as a means to HIV prevention. Other activities targeted the use of condoms in the most at risk populations. These include commercial sex workers, long distance truck drivers, urban motorcycle riders (commonly referred to as ‘boda boda’ riders), discordant couples, fishermen and the communities living at the landing sites, IDPs and other mobile populations. Coverage Half (17 out of the 34) of the UPHOLD supported districts implemented AB and other prevention activities. Activities in the former AIM supported districts are expected to commence in 2007. The targeted number of youths and adults for abstinence and be-faithful messages was reached by end of 2006. Similarly, targets for reaching school going children were met. The multiple message channels used managed to reach the 1,078 model schools. Activities have also reached a few non-model schools and together, have managed to establish the conspicuous school abstinence clubs and “talking compounds”. Use of CSOs (e.g. TUKO) has improved the reach of abstinence, be-faithful and gender based violence reduction messages to out-of-school youth, couples and the general community. Increased coverage was also obtained through the use of the national CSOs viz AIC and TASO. Over 50 local music, dance and drama troupes were mobilized by CSO grantees to propagate messages on abstinence and mutual faithfulness in communities. The abstinence script entitled The Clever Dancer, was developed by UPHOLD, and adapted to local contexts. Other methods included using community based opinion leaders to increase and facilitate discussions on key HIV prevention behaviors. Impact After performances, the troupes engaged their audiences in interactive dialogues on the topic of abstinence and innovative solutions were arrived at. Abstinence commitment cards were also another innovation promoted by CSOs for youth who wished to make commitment. C. HIV Counseling and Testing HCT is one of the most important HIV preventive activities and is an entry point into HIV care. UPHOLD worked in partnership with thirteen districts, Local Governments and CSOs to provide HCT services. The UPHOLD HCT strategy aimed at creating demand, 14 Annual report 2003 pp 12 20 Final Evaluation of UPHOLD programme increasing availability and improving the quality of HCT services. Particular emphasis was placed on serving populations vulnerable to infection, such as TB-infected individuals and persons with STIs, adolescents, internally displaced persons, pregnant women and their partners, couples, as well as people identified by districts and CSOs as engaging in high risk behaviors. UPHOLD targeted to increase HCT delivery sites from 32 to 500, train 300 service providers and reach 71,600 clients with HIV counseling, testing and results within the programme period. Service Delivery UPHOLD started operating in Uganda at the point where HIV Counseling services were provided at voluntary basis, via Voluntary Counseling and Testing (VCT). Both the coverage and utilization of services were very low. UPHOLD worked within the framework and policies of the MOH, supporting where gaps existed that may have impacted on district health service delivery. UPHOLD, for example, supported the MoH to review the HCT National Policy guidelines, and the HCT counselor training manual. UPHOLD supported 13 districts to provide HCT services and by 2006 there were a total of 683 sites providing the service. The HCT package included pre-test counseling and information giving, HIV testing, giving results, post-test counseling and follow–up support. Follow up support includes counseling on positive living, linkage to basic HIV care which includes Cotrimoxazole for prophylaxis, referral for antiretroviral therapy and psychosocial support. Clients are supported to form or join post-test clubs for peer psychosocial support. Post–test clubs members also participate in community sensitization and counseling on HCT. However, 88.9% (607/683) of these are outreach centers where a team of counselors and a laboratory person are sent out to provide services to lower level health facilities without staff or infrastructure to provide static services. Such activities are heavily dependent on the availability of funds to pay for fuel and allowances for the outreach team. Although the Health Sector Strategic Plan II recommends provision of static HCT services at Health Centre III level, only 35% of the HC IIIs in UPHOLD HCT geographical area are providing static services. Quality of services To ensure provision of quality HCT services, training was conducted for service providers in public and private health facilities on counseling, HIV testing, and logistics and data management. A total of 887 health workers were trained and service outlets increased from 32 in 2004 to 683 in 2006. Health facilities and CSOs were supported in accessing HIV test kits from NMS and provided with HIV test kits in times of shortage. Regular support supervision was provided to CSOs and health facilities in collaboration with the LGs and MoH. Utilization UPHOLD supported districts in the development and dissemination of key messages on the benefits of HCT through a variety of communication channels including drama and radio talk shows which helped to create awareness and demand for HCT services. UPHOLD supported the formation and functioning of 1700 radio listening clubs reaching about 11 million people. Behavior change communication addressed stigma, gender based violence and other factors that hinder uptake of HCT services. Community Owned Resource persons (CORPS) trained on HCT and members of the post test clubs participated in these and other HCT community initiatives. This facilitated informal and formal dialogue at community level. From 2004 to 2006, a cumulative total of 228,475 clients had HCT and received results through UPHOLD support exceeding the 2007 target 21 Final Evaluation of UPHOLD programme of 150,000. Most of the clients were reached through community mobilization and provision of outreach HCT services by CSOs. Challenges ƒ Human resource and infrastructure challenges greatly affected access to and quality of HCT services. High client to service provider ratios made it difficult to provide HIV testing and same day results especially during outreaches where there was good turn up in response to community mobilization. In response to infrastructure limitations, UPHOLD supported the renovation of health facilities to create counseling rooms and laboratory space. ƒ Another challenge included frequent stock out of HIV test kits resulting partly from stock-outs at national level, delayed or non reporting on part of the health facilities districts, or shortfall in the distribution system of the NMS. ƒ HIV related stigma is still prevalent particularly in Nakapiripirit district in Karamoja sub￾region. ƒ Some areas are very inaccessible, especially in parts of the conflict districts. ƒ There is also limited access to HIV care and treatment services partly due to low coverage of ART. ƒ The procurement and distribution system for free medical supplies e.g. HIV test kits does not adequately cater for CSOs. The CBOs rely on the neighboring health facilities or the districts for these supplies. This has implications on reporting of consumption data and therefore re-supply of commodities to health facilities. Impact There has been a progressive increase in access to HCT services. Based on the annual LQAS, the proportion of adults aged 15 years and above who know their results has increased from 16% in 2004 to 28% in 2006. Sustainability Most of the HCT service outlets were outreaches whose operations are dependent on availability of funds. D. Prevention of Mother to Child HIV transmission (PMTCT) UPHOLD works in partnership with local governments and CSOs in 18 districts to increase access and utilization of PMTCT. Efforts are geared towards working in line with the MoH guidelines to support hospitals and health centers in the private and public sectors and CBOs. According to the national guidelines, the PMTCT package includes the provision of proper antenatal care, counseling and testing for pregnant women, use of antiretroviral drugs to prevent transmission of HIV, safe delivery practices, counseling and support for safe infant feeding practices, and counseling and support for future family planning practices and follow up for the mother and baby. The revised PMTCT guidelines15 indicate a shift from single dose Nevirapine to combined ARVs for PMTCT, provision of routine HIV testing as opposed to voluntary testing and early diagnosis of HIV among infants among other changes. Service delivery Working through the public and private sector, UPHOLD has supported the scale up and provision of PMTCT to 55 service outlets in supported districts. Health workers in these service outlets have been trained on integration of PMTCT in GOANC, HIV counseling and rapid HIV testing. A total of 17 family support groups were formed and facilitated to provide peer psychosocial support, promote adherence to PMTCT and linkage to basic HIV care and ART when needed. 15 Policy Guidelines for Prevention of Mother to Child HIV Transmission, August 2006 22 Final Evaluation of UPHOLD programme Quality UPHOLD provide technical and financial support to improve the quality of PMTCT services. This included training and supervision of service providers, renovation of facilities to create suitable counseling rooms, waiting areas for ANC clients and laboratory rooms, community mobilization, sensitization and addressing gender based violence related to PMTCT in order to increase service uptake. UPHOLD supported the MoH to launch the revised PMTCT policy which emphasizes provision of comprehensive family centered HIV care and use of combined ARV regimens for PMTCT, follow up of mother-baby pairs and infant testing. In this spirit, support is provided to HIV positive mother networks and CBOs to follow up mothers, identify HIV exposed children and link infected mothers and children to ART. A good example was seen in Gulu where CBOs work together with health facility staff to identify and follow up HIV exposed children and link them to JCRC for infant HIV testing. Malaria prevention is a key intervention in GOANC and basic HIV care. A total of 4,690 LLINs were distributed to members of the FSG as one of the measures for prevention of malaria during pregnancy in addition to IPT routinely provided in ANC. Figure 2: Cascade of clients - PMTCT services in UPHOLD supported districts. 1745 22001 42509 1198 14958 30268 924 6498 19795 55 652 1621 17 251 871 3 153 422 0 5000 10000 15000 20000 25000 30000 35000 40000 45000 2004 2005 2006 New ANC Counselled Tested and recieved results Tested HIV positive HIV positive mum ARVs Babies receiving ARVs Although the number of clients utilizing PMTCT services is increasing progressively, the cascade shows a high drop out at every level. This is clear evidence of missed opportunities. In 2006, 47% (19,795/42,509) of clients accessing PMTCT sites were tested and 53% (871/1,621) of diagnosed HIV positive pregnant women received ARVs for PMTCT; an increase from 30% (6498/22001) and 38% (251/652) of the same indicators in 2005. Although this is a massive improvement it is still far below the national average of 61% (244,956/402,125) testing rate for clients accessing PMTCT sites and 64% (12,353/19,509) ARV uptake for HIV positive mothers in 200516. Utilization UPHOLD has been instrumental in creating an awareness of and demand for PMTCT services. Community education programmes have been conducted to create awareness of 16 MoH: Programmeme for prevention of Mother to Child Transmission of HIV, 2005 annual report 23 Final Evaluation of UPHOLD programme PMTCT, its benefits and the roles of the community and family members. Family support groups of HIV positive clients and their spouses have been formed, trained and supported to provide education on PMTCT at health facilities and conduct community mobilization. CBOs have been supported to conduct health education and engage communities in dialogue on PMTCT and male involvement. A good example was given in World Vision, Gulu where regular dialogues are held on PMTCT in the camps and the FSG formed has many male spouses participating in the group. Activities are ongoing in the community to address gender based violence and promote male involvement. These are two of the key factors identified that affect PMTCT uptake. Knowledge of MTCT has improved and 91% of respondents (LQAS 2006) know that it is possible to transmit HIV from the mother to the baby and 42% know all the three modes of MTCT. Knowledge of personal HIV sero-status is an entry point into care. The proportion of mothers who have been tested for HIV and received results during pregnancy has increased over time from 11% in 2004 to 26% in 2006. This shows a similar trend as that of the national average of 34.5% of pregnant women who were tested in 200617. Cumulatively 27,217 pregnant women have received HIV testing against a target of 41,900 and 1,139 HIV positive women received ARVs for PMTCT against a target of 6,950. Although the targets have not been cumulatively reached in the 3rd year, 2006, the performance was markedly improved. Contributors to achievements The increase in PMTCT sites from 9 in 2004 to 55 in 2006 has led to increased access and utilization of PMTCT services. Other contributing factors include community mobilization; the gender based violence prevention campaigns and increased access to ARVs in the country generally. Challenges ƒ Low level of scale up of PMTCT services ƒ PMTCT service delivery is greatly affected by general health system challenges of lack of adequate human resources, infrastructure and other factors that affect health facility delivery rates. ƒ Other contributing factors include a low male involvement in PMTCT; stigma and lack of community support for HIV positive clients. ƒ Frequent stock outs of HIV test kits and Nevirapine at national level compounded with poor data and logistics management and reporting contribute to irregular provision of services. Sustainability Integration of PMTCT into GOANC through capacity building for health workers and use of existing health facilities will contribute to sustainability of services. In summary, UPHOLD contributed significantly to increase the access to and utilization of PMTCT services. Although some targets set were not achieved, the strategy of integration of PMTCT into GOANC is evidence based, can be replicated and contributes to sustainability 17 UACP 2006 24 Final Evaluation of UPHOLD programme E. Palliative care Service delivery Palliative care aims to provide comprehensive care and support services to PLHA. UPHOLD provides financial and technical support to both public and private health facilities to strengthen the delivery of comprehensive and integrated clinical services so that health facilities become a one stop centre for holistic HIV care. The package of palliative care includes clinical services for management of malaria, STIs and TB, pediatric AIDS management; and provision of Cotrimoxazole prophylaxis. Other support includes psychosocial, spiritual and terminal care. Although provision of antiretroviral therapy forms part of the package in other programmes, it was outside UPHOLD’s scope of work. Partnering with 13 local governments and 8 CSOs, UPHOLD supported these organizations to provide general palliative care services. By the end of 2006 a total of 42 sites were providing palliative services. A family approach to delivery of palliative care services was promoted. Community based structures including post-test clubs, psychosocial support groups for HIV+ mothers and spouses. Religious leaders, faith-based organizations and volunteers were supported and utilized in carrying out community mobilization and in the delivery of care and referral services. The links between the health units and community support groups were also strengthened to ensure effective referral and follow up for PLHA. Quality UPHOLD undertook activities to improve quality of palliative care service provision at community, health facility and MoH level. UPHOLD supported the MoH in development and dissemination of “Guidelines for Providing Home Based Care” and training health workers in management of opportunistic infections. A total of 509 health workers were trained against a target of 222. At community level, home based care providers were trained in basic facts about HIV, basic counseling skills and home based management of common opportunistic infections. In addition support supervision to health facilities and community workers providing home based care was conducted. Utilization To improve utilization of palliative care services UPHOLD conducted community mobilization to create awareness of availability of services and efforts were made to address stigma and discrimination against PLHA. Community campaigns through multi￾media activities were conducted to improve knowledge, attitudes and practices related to care and support, stigma and discrimination against PLHA. Family support groups and post-test clubs were formed and members supported to access care and support. A cumulative total of 16,438 PLWHA were provided with treatment for opportunistic infections by the end of 2006 rising from 2,785 in 2004. The cumulative target was 33,000. Though Insecticide treated bed nets were distributed to PLWHA, it was difficult to assess performance because no targets were set. Impact The GoU, with support from partners, has over the period of time improved comprehensive HIV care in the country. In 2004 the MoH launched provision of free ARVs for treatment. In addition, other organizations are providing ARVs at subsidized prices. The annual LQAS survey shows that 14% of households had a person who was very sick or bedridden for three or more months in 2006 compared to 21% in 2005. This change is attributed to increased availability of ART and prophylaxis against opportunistic infections. 25 Final Evaluation of UPHOLD programme F. RH in War Affected Districts During 2005, UPHOLD extended HIV/AIDS services to 18 camps for IDPs in Kitgum and Gulu with plans to reach more than double this number of camps in 2006. This was done through a number of CSO grants to carry out HIV/AIDS related activities. Straight Talk Foundation (STF) was funded to set up the Gulu Youth Centre with the purpose of attracting, through innovative approaches, the large number of scholars and out of school youth in the Gulu district who have a heightened vulnerability to HIV/AIDS and STIs. The Uganda Sero-Behavioral Survey 2004/05 reported an HIV prevalence of 8.3% in the northern part of the country, higher than the national average of 6.4%. At the Youth Centre, information, education and communication (IEC) materials are distributed, educational films are available, and there is a drama group. Test kits and contraceptives are also available, and peers are now involved in delivering services, combining information with recreation and entertainment. One of the shortfalls of this activity was that it was anticipated that UPHOLD would document and disseminate the results of this approach in Gulu. This does not appear to have been done in a systematic manner so that lessons could be learnt and shared with other stakeholders. 2.5.6 Quality Assurance UPHOLD’s QA component was a cross-cutting component involving training and performance improvement, behavior change communication, and the private sector, to inculcate a culture of quality at all levels of the education, health and HIV/AIDS interventions. QA focused on developing and establishing systems to monitor and improve performance standards at service delivery level, including scaling up existing best practices. There was also an element of community participation and involvement in the development of standards, and the development and monitoring of an incentive system and achievement of standards. Three key activities were undertaken. ƒ Scaling up the MoH’s Yellow Star Programme (YSP) that covers improvement in 35 basic standards of service delivery through supervision and a reward system such as certification and presents. ƒ Community dialogue to increase participation and incorporate client perspectives through the ‘Stars in Progress’ initiative and promotion of active HUMCs ƒ Integration of facility HIV/AIDS services with tuberculosis, malaria and sexually transmitted diseases through the development of a supervision tool Coverage UPHOLD revitalized the YSP in all 34 districts and 68% of the facilities received yellow star supervision during last quarter before the 2006 LQAS survey. Coverage of ‘Stars in Progress’ covered only 6 districts in 2006. Consumer advocacy is still being piloted in two districts of Mayuge and Bugiri under a CSO grant to Uganda National Consumers/Users’ Organization (UNHCO) and is working with Community Development Workers to develop an advocacy programme for health and HIV/AIDS services. Quality UPHOLD combined standards developed by other partners to develop and pretest a more user friendly version of YSP tool. This is now being adopted by the MoH. To further strengthen demand, utilization and quality of health services, UPHOLD worked with UNHCO to develop a training manual for district trainers and consumer advocates at 26 Final Evaluation of UPHOLD programme lower levels for consumer advocacy. Further BCC activities were developed and placed on radio spots and posters to support the advocacy initiative. Impact Slow progress has been made to improve standards. The major low scoring areas in YSP in LQAS 2004 included growth monitoring, management, and the availability of basic examination equipment, drug stock-out management and the functional referral systems. YSP reporting has now been incorporated within the routine HMIS reporting form. Nearly one third (30%) of facilities in UPHOLD-supported districts achieved 75% of basic health care standards. Only 30% of the facilities had at least 4 HUMC meetings in the 12 months preceding the 2005 LQAS. A few examples exist (e.g. Pallisa district) where low scores have prompted establishment of new HUMCs. However, this was led more by the technical district officials than the community. Working with CDWs in 3 districts also promoted dialogue especially in informing facilities providers to understand why health facilities were not being utilized. Issues Sustainability of reward systems, especially involving gifts and sponsored celebrations, has been a problem for many health promotion programmes in the country. These are unlikely to continue unless other donors continue the process. Whereas the YSP tool serves as a good quality improvement tool, a number of system level factors seem to hinder government owned facilities from attaining and maintaining high scores. In districts visited, it was mainly the private not for profit facilities that scored higher. This was attributed to their ability to quickly respond to recommendations. Implementation of most of the health quality assurance activities started late in the life of UPHOLD and achieved a very small coverage. It is difficult to envisage how they will cover all the UPHOLD districts by closure of the programme. 2.5.7 Supervision The districts’ supervision systems were weak. Only 56% of health facilities received regularly supervision from the DDHS's Office on a monthly basis. Despite training activities carried out by UPHOLD to provide health workers with the technical knowledge and skills to perform on the job support supervision, several factors constrained the supervision system in districts. These included the limited allocation and availability of funds for supervision activities under Primary Health Care grants from Central Government to the districts and poor motivation and low demand for service data or supervision reports from managers. This is compounded by the fact that the YSP is performed parallel to the routine district integrated quarterly support supervision. Efforts are needed to integrate these different models of supervision. Though the system of community dialogue through UNACHO have been instrumental in developing community demand, it will be necessary to ensure that this is institutionalized. Efforts were needed to ensure that there was district buy in. 27 Final Evaluation of UPHOLD programme 2.5.8 Monitoring and Evaluation The M&E strategy was an important and central component of UPHOLD. The importance of M&E to UPHOLD was indicated by the upgrading of its status in the organization to be part of the executive management with the leader of the M&E cluster being promoted to a DCOP level and reporting directly to the COP. UPHOLD’s M&E results are essentially contained in the program’s Performance Monitoring Plan (PMP) that is produced annually. The PMP highlighted the annual targets set by UPHOLD in its annual work plans as well as the actual results obtained. The PMP provided a regular concise monitoring of activities by geographic area by year. To display a summary of activities in a complex program in a clear and simple manner was a great achievement of UPHOLD. Targets ƒ One aspect of the M&E process that was confusing for the evaluation team was the way in which PEPFAR targets were handled. As far as the evaluation team could understand, targets were set for a wide range of activities on an annual basis. The actual results obtained for the activities by UPHOLD were then compared against these targets. However, it appeared that because of differing financial years and PEPFAR having country targets that were allocated to USAID-funded projects and other reasons that were not made clear to the evaluation team, that the original targets set by UPHOLD were then retrospectively adjusted by MEEPP, which does the overall M&E for USG. This resulted in several targets, which were at variance with each other, being published in formal publications, which caused confusion in UPHOLD and also in the minds of the evaluation team. ƒ Another aspect of target setting that could be improved was the fact that these were determined centrally by UPHOLD’s M&E cluster and then given to the regional offices to implement. The setting of targets could have been done in a more participatory manner and could also have been set done more transparently. This applied in relation to targets required by CSOs as well. The CSOs were given targets to achieve that were prescriptive and over-precise. ƒ Targets were also fine-tuned to a level of precision that was unwarranted by the data available and by the accuracy needed. So for example instead of saying that say 5000 people would be tested for HCT (which would be an estimate of what could be achieved based on previous performance and baseline), UPHOLD would set a target of say 4987. All of these targets were categorized against the USAID’s strategic objective 8, which aims to improve human capacity. Both the targets and the results have been sub-categorized into the following objectives: 8.1 Effective use of social services 8.1.1 Improved quality of social services 8.1.2 Increased access and availability of social services 8.1.3 Improved behaviors adopted 8.2 Increased capacity to sustain social sector services 8.2.1 Improved decentralized planning, management and monitoring systems 8.2.2 Increased private sector role in service delivery 8.3 Strengthened enabling environment for social sector services 8.3.1 Increased community participation and advocacy 8.3.2 Effective sectoral policies and advocacy 28 Final Evaluation of UPHOLD programme This framework provided a very useful way of ensuring that the numerous activities of the programme were easily accessible, and is a model of how a programme should represent its monitoring activities over time. This framework was then replicated down to district level where it formed the basis for data collection by the various UPHOLD staff, and associated partner organizations. The level of detail required by the monitoring process, as well as the time constraints to collect this data, placed considerable pressure on district and local partner institutions, and on occasion it seemed that there was more emphasis placed on the collection of numbers than the actual activity itself. UPHOLD contributed to the national HMIS system by providing training of district staff in HMIS modules, supporting the MoH to develop a training manual aimed at improving HMIS data quality and training district Focal Persons for HMIS in supportive supervision. However the pressure within UPHOLD to obtain data “speedily”, and also external pressure from PEPFAR to show results, resulted in duplicate extraction of data from the HMIS by UPHOLD. This was costly to the program in terms of using program staff resources to carry out this extraction. It also had the effect of not boosting the HMIS and it is unhelpful to the HMIS that an external program like UPHOLD has better quality data than does the MoH. 2.5.9 LQAS In an attempt to focus on population information, which is where the impact of activities is measured, UPHOLD introduced an innovative method to obtain household data through surveys at the household level called Lot Quality Assurance Sampling. Over 3 successive years from 2004 to 2006 these LQASs were carried out. Issues ƒ Capacity was built at the district level to carry out surveys. ƒ Districts appeared to accept the results as useful in planning. ƒ The results have been well-accepted at a range of national meetings. ƒ However, the sustainability of LQAS has been questioned as the analysis, interpretation and reporting of results has not been institutionalised. ƒ The central level MoH has not been as actively involved with the strategic aspects of LQAS as it could have been. So for example the latest LQAS of 2006 was analysed, interpreted and reported upon by UPHOLD without any input from the HMIS unit of the MoH. ƒ There are cost implications of carrying out annual LQASs. Districts do not have sufficient resources to undertake these without external funding. 29 Final Evaluation of UPHOLD programme 2.6 DEVELOPMENT AND MANAGEMENT OF RELATIONSHIPS How did UPHOLD develop and manage relationships with USAID implementing partners, GOU partners and civil society. 2.6.1 UPHOLD Relationships with Central Government Partners UPHOLD was designed to be a district based programme. It had not been anticipated at the design phase that the programme would have to establish formal relationships with Ministries although this eventually happened. This had been a designated role for the USAID Systems programme which was later abandoned. GoU and the ministries were not involved in the decision making and planning processes on strategic issues or directions of UPHOLD other than their involvement in the selection of the districts which took an inordinately long time (6 months). UPHOLD’s relationships with central government Ministries evolved around, and were defined by, the various mechanisms used to share information along the regular communication channels especially with particular sections in the MoH (e.g. PMTCT, Malaria). However it is important to point out that UPHOLD was considered a very responsive partner especially when approached to consider ‘ad hoc’ requests to address and jointly support national issues like mass immunization and implementation of the PIASCY interventions. The relationship between the donor, USAID, and the key stakeholder, the GoU, was not as clear as it should have been and resulted in a loss of 6 months at the commencement of the project through a delay in district selection. 2.6.2 UPHOLD Relationships with its Consortium and USAID Partners The strategic decision making in UPHOLD was not as clear as it could have been. The major decisions around funding flows were made by USAID and did not leave the UPHOLD management with a huge amount of discretion around implementation. In addition UPHOLD management did not have a steering committee or reference group which could provide an oversight role in terms of strategic direction. Finally UPHOLD did not adequately tap into the intellectual resources available in its consortium partners with regard to strategic direction. UPHOLD was not subject to any formal evaluation during the five years of operation. There was no mid-term review nor were there any formal annual reviews. As a result it was difficult to assess whether the programme was ‘on track’ in relation to the big picture. All stakeholders, including USAID implementing partners, were thus deprived of the opportunity to engage with the strategic direction of the programme. 2.6.3 UPHOLD Relationships with Local Governments and CSOs These relationships with local governments and civil society were evaluated through (1) grants (2) decentralised CSO and LG capacity building strategies and activities UPHOLD used. A. Grants Management The UPHOLD grant strategy features 6 funding mechanisms with different stakeholders accessing resources through different UPHOLD channels. (1) National e.g. AIC and TASO, (2) Central level with multi-district reach e.g. UPMA, Tuko Club, (3) Local Government, (4) 30 Final Evaluation of UPHOLD programme RFAs and competitive funding to CSOs, (5) Non-RFAs with solicited funding proposals e.g. Straight Talk Foundation, Madrasa and, (6) Core PTC type grantees. The various grants were guided by fairly elaborate annual MoUs between grantees and UPHOLD and in the case of the CSOs, a manual defining the rules of engagement. In all cases grantees were required to submit activity and financial reports following prescribed UPHOLD templates. Accessing the grants was usually subsequent to approval of submitted proposals or applications following a prescribed format. A main feature of the grants strategy implementation was the provision of tailored capacity building for the different grantees in the areas of planning, budgeting and proposal writing. CSOs specifically benefited from this. This capacity building effort covering additional areas was also extended to all grantees during programme implementation. Capacity building also focused on technical and financial reporting although LGs and CSO grantees reported that the new templates resulted in additional pressure placed on resources. A number of challenges were associated with timely technical reporting. In addition, CSO proposals submitted in response to RFAs underwent a rigorous approval process that resulted in the sieving out of a significant proportion of CSO bids (42 out of 625 CSOs who applied in the first selection process). The criteria also required CSOs to have prior experience in target intervention domains and demonstrated partnership with LGs and other CSOs. There is no doubt that the sieving process, by ensuring a minimum threshold of CSO capacity, also increased the likelihood of achieving desired results. This is borne out by the decision by UPHOLD to increase resource flows to CSOs instead of LGs in PY 5. The issue of liquidation or absorption by grantees and especially by LGs consistently stands out as one of the biggest challenges in the grants strategy. Other challenges included the creation of new districts with the emergence of a whole new set of LG and CSO functionaries with limited capacities to implement UPHOLD activities and comply with UPHOLD requirements. An internal study18, to identify bottlenecks to the flow of funding and implementation, commissioned internally by UPHOLD concluded that ‘weak or inadequate coordination’ arising from unclear terms of reference of the LG focal point persons was one of the main reasons for bottlenecks. It appears that increasing pressure to deliver on the results strongly influenced UPHOLD responses to grantees. These responses varied from ‘more intensive planning’ (e.g. increased involvement in activity planning with LGs) sessions with grantees to the more ‘drastic’ change in funding mechanism through UPHOLD regional office arrangements. The routing of funding via regional offices appears to have led to the affected LGs’ misunderstanding of the UPHOLD regional office role as having become that of an implementer. A close analysis of the various scenarios that prevailed in LGs reveal that UPHOLD had limited options other than to take over the joint responsibility for activity reporting and full responsibility for financial reporting and leaving implementation of activities to LGs. There are however three issues requiring some further discussion with regard to the entire grants strategy design and implementation. The first is that from PY 1 to PY 5 LGs, irrespective of whether they had demonstrated minimum competencies or registered whatever level of performance during the PY of implementation, were ‘always’ assured of receiving an indicative planning figure for each PY for identification and prioritization of interventions. In 18 Factors influencing flow of funds and implementation of programmes in Local Governments supported by UPHOLD 2005/06 31 Final Evaluation of UPHOLD programme addition, initial disbursements to LGs in the first FYs appear to have been made on an exact division of the total LG envelope. In the later years, demonstrated ‘burn rate’/ liquidation capacities influenced individual LG funding levels suggesting a degree of linkage to utilization of funds and performance. The second issue is to do with strengthening systems and institutions. It is clear that UPHOLD had limited room to maneuver in this regard. The ‘need’ to introduce USAID formats for activity and financial reporting, as well as for preparation of proposals, had both negative as well as positive spin offs from a systems strengthening perspective. On one hand grantees received training and support to enable them to implement activities while achieving a satisfactory rating on filling a USAID programme proposal template. This implied increased programme proposal development capacity reflecting an acceptable standard. On the other hand, the training and support to LGs tended to focus more on ensuring successful UPHOLD activity implementation and not necessarily the LG Health sector work plan activities. Again the USAID templates increased the LG periodic reporting ‘burden’. The third issue is that during implementation, due to the nature of the short one year grants, there were periods when grantees were uncertain of subsequent funding. This occurred especially when there was no communication on indicative planning figures from UPHOLD. This problem was linked to the varying financial years of the various sources of funding of UPHOLD (e.g. USAID, PEPFAR, PMI) as well as PEPFAR rules that pre-determined the funding period. However UPHOLD came up with innovative responses to ensure that funding flows were smooth to the extent possible. B. Decentralized CSO and LG Capacity Building One of the key strengths that influenced the UPHOLD implementation mode in 2003 was the decentralized system of governance at local government and service delivery level. This applied especially to the attendant systems and institutions that would define and influence the UPHOLD programme rollout. During the evaluation, it emerged that UPHOLD engagement with local government focused strongly both at higher LG as well as lower LG levels. This engagement was defined by the relationships established with key technical, management and political functionaries that included the CAO and nominated UPHOLD Focal Point Persons, DDHS, members of the DHT and the DEO. The relationships between UPHOLD and LGs evolved initially in the preparatory processes necessary for rollout and were nurtured during key LG routine annual processes like planning, budgeting, prioritization, monitoring and evaluation. The relationships were also cemented by the annual provision of indicative planning figures conveying ceilings in which LGs would prioritize UPHOLD activities. They were also cemented by the periodic arrangements in which coordination with other stakeholders in the health sector exchanged information on planned activities and level of efforts. This resulted in strong efforts to avoid duplication and ensure LGs and communities benefited from appropriately planned complimentary interventions. Particular outcomes arising from LG planning and rollout of programme activities also defined the nature of engagement with UPHOLD. For example, after occurrences of unfocused prioritization (“LGs wish to do something of everything”), there was eventually more prescriptive UPHOLD guidance and development of the ‘Focus for Impact’ strategy that shrunk the potential activity menu. 32 Final Evaluation of UPHOLD programme Delayed activity and financial reporting by LGs resulted in routing the funding of activities through the regional offices in some LGs19. In addition, country wide ‘emergencies’ arising from problems associated with national procurement systems, resulting in UPHOLD procuring supplies through MoH systems (e.g. to address HIV test kit stock outs). A large proportion of UPHOLD responses was also embodied in provision of significant technical assistance or support either in the form of ‘on job training’ or other appropriate capacity building interventions to address critical LG capacity needs. Changes in the programme environment like ‘redistricting’, implementation of the LG restructuring report20 commissioned by MoLG, and specific central government policies that negatively impacted on LG local revenue generation also influenced the relationships in the districts. In all these cases, UPHOLD should be commended for the studies conducted to analyze and understand the changes and assess various options with a view to identifying practical ‘win-win’ alternatives. Some of these included the decision to expand interventions from 20 to 34 districts and amending existing MoUs, while in effect maintaining the same geographical area, as well as training and providing hands on support to the few new staff in the created districts. The expansion to 34 districts served to further redeem the UPHOLD regional arrangement as this provided the most appropriate modality to deal with the increase in ‘body corporate’ entities, more functionaries with limited core functional capacities and limited knowledge of UPHOLD goals, interventions and implementation frameworks. Issues of coordination were significant especially in LGs (e.g. Acholi and Lango) with high CSO development support presence in the health and HIV/AIDS sector. Inevitably, UPHOLD interventions also increased the already high transaction costs and the administrative and financial reporting burden that LGs were already carrying. This was not helped by the fact that different support partners, including UPHOLD’s capacity building efforts, focused on strengthening (only) those systems that enhanced implementation of ‘own programme’ activities and not necessarily the LG systems21. It emerged that there is a clear recognition by all stakeholders of the fact that the variety and complexity of donor procedures associated with various support instruments places a substantial burden on LGs especially in multi-donor aid effort settings. Secondly, the management capacities of LGs tend to be overstretched, which hinders efforts to strengthen existing beneficiary institutions and systems. However, the reality seems to weigh heavily in favor of the status quo prevailing. Although there was a commendable effort in Gulu district to attempt to get this issue22 on the agenda, this was not seen through or ‘led’ by the LGs. The issue of the ‘abolition’ (suspension) of graduated tax and its effect on LGs, especially with regard to LGs disablement in their ability to generate and utilize revenues to address district priorities, needs some thought. While it is fact that LG locally generated own resources gave them complete discretion in terms of decision making on expenditure areas, it is also fact that the greater proportion of these resources were used on council activities that were not necessarily always the most appropriate with regard to impact on service delivery23. It is true however that these resources did at one point and for a period meet salaries for a selected cadre of health workers (health facility support staff) and council activities sittings. But there is limited precedent to inform the supposition that these resources if present or available would have in the prevailing circumstances been utilized in an optimal manner to impact positively on LG service delivery. 19 This was mis-understood by these LGs to mean UPHOLD regional office transformation into a direct implementer of activities which was not the case as UPHOLD’s only involvement was to pay for activities directly 20 The report recommended three ‘model’ appropriate and affordable district structures which LGs would choose to suit local situations 21 In Lira DLG the health department accountant reported she had different formats for UNICEF, UPHOLD, DANIDA, WHO, SHSSPP 22 Harmonization and alignment to the ‘extent possible’ taking existing donor aid requirements in consideration but focusing on reducing LG burden 23 First Joint Annual Review of Decentralization 2004, thematic paper 5: Sustainability of LG, issues and the importance of local revenues 33 Final Evaluation of UPHOLD programme UPHOLD has been very responsive to the decentralized arrangement in LGs in the areas of operation. This is evidenced by: ƒ the integration of UPHOLD activities in DDPs and DHSSPs ƒ reported increases in volumes (both access and use) of health and HIV/AIDS services ƒ increases in the quality of services in an increased number of health facilities and in Lira DLG, the Yellow Star intervention being a case in point. ƒ LG functionaries as well as members of the VHT all reported having benefited from the training and other forms of capacity building interventions provided through UPHOLD ƒ UPHOLD focus at the LG level further enabled the fostering of ‘working’ relationships between LGs 2.7 UNINTENDED RESULTS Did UPHOLD yield any unintended positive/negative results? There were a number of activities of UPHOLD that resulted in spin-offs that were not anticipated. Examples of these include: o Distribution of LLINs through CMDs exposed a channel that had not been used before. This resulted in the CMDs becoming a point of reference for health issues in the villages. This model of CMD distribution has been taken up by the MoH. o LQASs have been well accepted as a means of getting population based information. o PIASCY has resulted in community and household involvement in education. On the other hand it has been reported that some schools not involved in the model school process, have had negative reactions. o Use of CSOs has resulted in high mobilization of decentralized and household HCT and also led to increased utilization of family planning. o Although CMDs and CSOs have been highly successful in certain activities, they now face an uncertain future and their sustainability is not assured. 34 Final Evaluation of UPHOLD programme 3 KEY LESSONS LEARNED What are the key lessons learned from the design and implementation of this large multi￾sectoral district based programme. a) There is a danger that cooperative agreement arrangements leave programme goals ‘too loose’. This was the case in UPHOLD where the programme tried to ‘do everything, everywhere’ with limited and finite resources. As a result, the programme was not sufficiently focused. The consequence was that some activities were very thin on the ground. It took two years before there was a “Focus for Impact”. This resulted in a narrowing and deepening of activities. Where cooperative agreements are used, the lower level goals should be made more specific than they were in this programme so that all stakeholders are clear as to what the programme is trying to achieve. b) Without systems strengthening to link with the services programme, UPHOLD ‘walked on one leg’, trying to implement a district based programme but always having to consider systems issues at the delivery (LG) level as well as at the central MoH level. c) Central MoH involvement in large programs is essential. This involvement can: i) assist the strategic direction in ensuring that program goals are consistent with health system policy ii) help with sustainability by getting especially MoH and Uganda AIDS Commission participation in and buy-in to activities of the program (e.g. LQAS). This was evident with PIASCY in the Education sector. iii) open doors to the program through providing it with continued endorsement d) Many of the excellent activities implemented by UPHOLD have question marks against their sustainability after the end of the program. Examples of this include continued motivation of the CMDs and funding of district LQAS. e) Although UPHOLD had the laudable aim of integration of activities at the district level, this is difficult to achieve when the health systems are running different projects in a vertical fashion (e.g. PMTCT). f) Although this program was designed as a multi-sectoral program (health and education) to be delivered at district level, there was no prototype or pilot on which the design was based. g) The efficiency and effectiveness of interventions and activities by grantees was compromised by the uncertainties around the funding flows in terms of the funding schedules and the amounts. The funding used generally by UPHOLD, to a large extent determined by its own funding streams, meant that grants were made for a single year or less; and that the amounts of the grants were uncertain. This resulted in pulsed, uncertain and inefficient grant making to CSOs. h) The differing financials years of UPHOLD, USAID and the LGs often meant that funding did not fit into the financial framework of LGs and did not coincide with planning and budgeting cycles. i) The RFA method of grant making has both positive and negative results. It ensures that CSOs have sufficient technical ability to write grant proposals and demonstrate their likelihood of success and so results in the choice of CSOs who are likely to deliver. It 35 Final Evaluation of UPHOLD programme however limits the grant making to certain kinds of CSOs and results in much CSO time being spent on writing proposals at the expense of service activity. j) There is a tension between being focused on getting results, achieving targets and ensuring that the ‘numbers’ are improving (which are generally short term achievements) and taking on a developmental role through improving human capacity (which is generally a softer and longer term achievement). k) UPHOLD has an excellent monitoring system where all program activities were tracked. However, over-prescriptive target setting, together with unrealistic expectations of the accuracy of data, can result in too much attention being placed on the numbers. l) One of the positive achievements of the program was the use of CSOs who in certain instances made important and large contributions (e.g. in home mobilization of HCT). m) UPHOLD encouraged and facilitated the relationships between CSOs and LGs. n) Some activities of UPHOLD took a long time to mobilize, e.g. start-up of TB programme; others were started and then curtailed because of lack of resources e.g. growth promotion. There needs to be a balance between the depth and breadth of the range of activities and also the geographical areas in which they work. Programs cannot be all things to all people. o) Some intervention specific lessons include: i) Mass immunization presents a good opportunity to publicize and make popular the use of EPI services. There has been a general improvement in utilization of immunization services following on the mass campaigns. ii) Targeting of hard to reach areas enabled filling of gaps that other district funds were not able to cover. iii) Programme reports found that programme action on gender based violence is associated with increased PMTCT uptake. iv) The use of CMDs, who are conversant with their own communities, to target the beneficiaries is helpful in minimizing misallocation of the nets. This was adopted as strategy by the MoH and is a lesson for future approaches for the distribution of the 1.8 million ITNs purchased via the Global Fund. 36 Final Evaluation of UPHOLD programme 4 RECOMMENDATIONS As a result of this end evaluation, the following actions should be taken by USAID in its future programmes: a) Future programmes should ensure that the parameters are more targeted and that interventions should have sufficient depth to make meaningful changes on the ground. b) Large programs should be endorsed within the GoU prior to commencement with clear goals and objectives. c) For a program of this size, a formal mid-term review is imperative, and annual reviews would also ensure that the programme is heading in an appropriate direction. d) There should be a formal annual strategic decision making meeting where key stakeholders’ input is continuously provided and which is based on a strategic review of performance, direction and focus. Programs should build into their design a more formal relationship with the central MoH, where roles and responsibilities are clearly delineated. e) Sustainability strategies should be transparent and a critical consideration in the design and implementation of large programs. f) Programmes need to design their activities around the realities of the health system and feed back to central level well documented policy briefs to encourage systemic change of service innovations. Programs that are trying to improve service delivery need to take into account constraints in the formal health system. g) Large scale programs should be based on tested methodologies. Innovative activities have an important role but they should be tested in small scale pilots before going up to scale. h) Future programs should further encourage in all ways, including financial, the closer working together of these parties. i) There should be more certainty in programme funding flows and more flexibility given to the program to make grants for periods longer than one year. Where circumstances are appropriate grants should be made to allow for continuity of activity. j) Funding should plug in the existing planning frameworks and cycles as much as possible. k) Programs need to be clear, transparent and realistic as to what their mission is and what they are trying to do. l) Monitoring has an important role to play but it should be complemented by strategic analysis and overview, where the big picture is constantly kept in mind. m) The best features of the RFAs should be coupled with the targeted selection and capacity development of particular CSOs, in order to ensure sustainability and overall long-term development of these CSOs. n) Future programs should look at the positive ways in which CSOs were harnessed and build upon these. UPHOLD itself should write short policy briefs on the success of this (and other interventions) and the lessons that it learnt in the process. o) Activities taken on by programs need to be carefully thought through in terms of financial, human and other resources required prior to their commencement and there needs to be a balance between the number of activities and how widespread they are implemented. 37 Final Evaluation of UPHOLD programme 5 CONCLUSIONS The conclusions are arranged according to the evaluation questions. 5.1 Most of the operational targets set by UPHOLD have been attained and especially in the last year UPHOLD seems to have moved into top gear. UPHOLD start was much slower than designed but has successfully evolved over time and is making successful and useful interventions Although the evaluation team obviously could not assess the operation of UPHOLD in the initial years 2002-2004, it appears in our assessment that UPHOLD in 2007 is a well functioning, hard working, efficient and effective organisation. However, with more strategic direction and focus, this program could have achieved more. On the basis of the main objectives of the UPHOLD’s grants strategy, there is no doubt UPHOLD program expanded access and use of ‘comparatively improved’ quality services, registered a fair achievement in fostering the concept of integration across health and HIV/AIDS and increased the capacities of local CSOs to utilize grants. UPHOLD has also increased the capacities of LGs and especially CSOs to implement (UPHOLD) activities although while these capacities increased, LGs and CSO did not always implement and monitor activities in a manner consistent with the 'high' standards of transparency and accountability that was expected of them after the capacity building. 5.2 UPHOLD yield a number of unintended positive and negative operational results. Success stories, and lessons learnt in the project, need to be adequately marketed and publicized through targeted, short briefs produced by UPHOLD. 5.3 UPHOLD programme design suffered from being an over-ambitious project with inadequate strategic direction, both from without and from within, in the initial years. This resulted in the project having too much breadth and too little depth. The changing of key implementation objectives, (e.g. through PEPFAR, PIASCY implementation), did not help this process. Despite plans to start quickly in a few districts and “go national” quickly, it takes a long time for large multi-sectoral, district based program to start off and focus the implementation especially when the system strengthening is actually needed. There are a number of tradeoffs with annualized grants method used especially when they focus on reaching generally short term achievements rather than the developmental concept. Also, the grant process did not adequately propel UPHOLD programme to facilitate building capacity of Ministries and local governments to collaborate with CSO in implementation management. 5.4 UPHOLD management did not have a steering committee or reference group which could provide an oversight role in terms of strategic direction. Though UPHOLD developed good relationships with USAID implementing partners in the beginning, this waned as implementation progressed. Relationships with MoH did attain levels of collective ownership of achievements, which is needed for going national, in only a few innovative areas such as yellow star and net distribution through CMDs. There is no doubt that UPHOLD has been responsive to the decentralized arrangement in LGs in the areas of operation. However, the relationship between the donor, USAID, and the key stakeholder, the GoU, was not as clear and UPHOLD did not adequately tap into the intellectual resources available in its consortium partners especially with regard to strategic direction in successive annual plans. 38 ANNEXES: Final Evaluation of UPHOLD programme APPENDIX 1: Scope of Work for UPHOLD 1. BACKGROUND Uganda Programme for Human and Holistic Development (UPHOLD) is funded through a Cooperative Agreement (CA) with the United States Agency for International Development (USAID) for five years, with a completion date of September, 2007. UPHOLD is funded by USAID under the President’s Emergency Plan for AIDS Relief (PEPFAR), President’s Malaria Initiative (PMI), Basic Education, Health (Child Survival/Maternal Health, Reproductive Health/Population) and Infectious resources. UPHOLD is implemented by JSI Research and Training Institute, Inc., with support from World Education (WE), Education Development Center (EDC), The Constella Futures Group, Malaria Consortium, and The Manoff Group. UPHOLD was designed to contribute to the “Improved Human Capacity” strategic objective of USAID/Uganda‘s Integrated Strategic Plan (ISP) for the period 2002 -2007. The objective aimed at assisting Uganda to improve its human capacity through addressing constraints to improving health and education status. Following a major review and reform of the United States Government (USG) foreign assistance strategy in 2006, the USAID/Uganda 2002- 2007 ISP was ended in 2006. Under the new strategy, human capacity strengthening is now programmed under the new programme area, “Investing in People”. USAID/ Uganda human capacity strengthening approaches remain the same as under the ISP integrating Health, Education and HIV/AIDS sectors and UPHOLD contributes to the new programme area. Focusing on service delivery at the district level, UPHOLD’s main goal is to increase the utilization, quality, support and sustainability of services in education, health and HIV/AIDS through an integrated multi-sectoral approach that seeks to: • improve educational status; • reduce the spread of HIV/AIDS and sexually transmitted infections; • decrease child and maternal mortality; and • stabilize population growth UPHOLD activities are implemented in 34 districts (up from 20) covering about 42 percent of Uganda’s s population. The program’s core technical interventions cover: • Primary School Education: improving teacher effectiveness; facilitating dialogue and consensus-building between families, communities, teachers and other stakeholders; building capacity in planning, management and supervision at decentralized levels; increasing parental involvement; and facilitating the use of innovative tools and approaches to improve children’s learning in school and at home. • HIV/AIDS: strengthening services for Voluntary Counseling and Testing; Preventing Mother to Child Transmission of HIV; managing Sexually Transmitted Infections; supporting People Living with HIV/AIDS; and innovative approaches to empower and involve young people and address gender and culture. • Integrated Reproductive Health: promoting improved antenatal and postnatal care; safe and clean deliveries; essential obstetric care; post abortion care; gender-sensitive approaches; effective dialogue and decision-making in families and communities; increased access to and utilization of quality family planning services and methods; youth-friendly services; and efficient synergies with the control of HIV/AIDS, sexually transmitted infections, and malaria. • Child and Adolescent Health: key interventions include the Integrated Management of Childhood Illnesses (IMCI) and Community IMCI; Community-Based Growth Promotion; improved strategies and indicators for Nutrition; Vitamin A & Micronutrient I ANNEXES: Final Evaluation of UPHOLD programme supplementation; promoting Exclusive Breastfeeding; childhood immunizations; youth￾friendly services; and innovative strategies promoting peer support groups, child-to-child activities, and parent-child communication. • Communicable Disease Control: Malaria control activities include the effective Home Based Management of Fever, Intermittent Preventive Treatment of malaria during pregnancy, promoting use of Insecticide-Treated Nets. In addition, UPHOLD works to control tuberculosis, and other communicable diseases in a multi-sectoral approach. In addition, UPHOLD has five cross-cutting technical interventions. These are: • Performance Improvement • Quality Assurance • Management and Strategic Planning • Behavior Change (BC) Communication and other BC Strategies • Community Ownership and Involvement 2. PURPOSE OF THE EVALUATION The Automated Directive System (ADS) 203.3.6.1 requires that an evaluation is conducted when there is distinct and clear management need to address an issue. USAID/Uganda is planning to evaluate the results of the HIV/AIDS and Health components UPHOLD programme. The purpose of this evaluation is to extract lessons that would benefit the USG/Uganda and GOU partner institutions with future programming. Secondly, implementing partners, have interest in this evaluation in terms of knowing what worked and what did not work during implementation of the programme. The evaluation will provide quantitative and qualitative assessment of the approach that UPHOLD has applied and determine whether UPHOLD is on track to achieve its impact objectives. USAID is interested in continuing support to HIV/AIDS and health service delivery in a decentralized environment. While we continue to learn from old and ongoing activities within and outside the USG portfolio, USAID seeks to garner lessons learned regarding design and implementation of large, multisectoral programmes as one means of rapidly scaling up HIV/AIDS and health services in a decentralized setting. The evaluation will be able to provide answers both at programme and strategic level by addressing the question of whether UPHOLD is on track to achieve the intended goals of its HIV/AIDS and health interventions. The evaluation will also distill lessons learned about programme implementation that will have a bearing on scaling up HIV/AIDS and health interventions nationwide. 3. KEY EVALUATION QUESTIONS This evaluation entails examining a number of questions: 1) To what extent has UPHOLD achieved its overall HIV/AIDS and health goals and results? How did the following factors contribute to the achievement of goals and results? • Programme design strengths and limitations • Technical competency to undertake the planned activities • Technical support and capacity building for districts and grantees • External factors 2) How did UPHOLD develop and manage relationships with USAID implementing partners, GOU partners and the civil society? 3) Assess whether UPHOLD yielded any unintended positive/negative results 4) What are the key lessons learned from the design and implementation of this large multi-sectoral, district based programme? II ANNEXES: Final Evaluation of UPHOLD programme 4. METHODOLOGY The evaluation team will be required to come up with a clear methodology to answer all the evaluation questions. However, it is expected that data collection will be conducted using a combination of the following: • Review of relevant documents • Key informant interviews with relevant staff of USG/Uganda, UPHOLD and other key partners, and other key informants • A survey or group interviews with a representative number of beneficiaries to assess their level of satisfaction with UPHOLD HIV/AIDS/ and health services • Direct observations of service delivery sites The evaluation team is expected to use a mixed methods approach. In order to achieve rapid analysis of the data, it is anticipated that the evaluation team will have working familiarity with both quantitative and qualitative data analysis tools and skills. The evaluation team is expected to be familiar with USAID data quality standards (ADS 203 and 578), and able to apply them in the final report, by identifying such data limitations as may exist with respect to these standards. For the Document review, USAID will provide the evaluation team with several documents including: • GOU: National frameworks, policies and implementation guidelines from Uganda AIDS Commission, Ministries of Health, Local Government and Local Government Development Plans and reports • The Role of Evaluation in USAID http://www.dec.org/pdf_docs/PNABY239.pdf • UPHOLD: o Cooperative Agreement and amendments o JSI Proposal o Annual and quarterly reports o Annual work plans, results framework and performance monitoring plan o Website o Special study reports by UPHOLD etc. o Grantee stories, lessons learned, case studies o Internal assessments and reviews 6. EVALUATION TEAM COMPOSITION AND REQUIRED QUALIFICATION 6.1 Team Members A team of international and Uganda experts is proposed. It is essential that all team members understand the development context and, while previous experience in Uganda is definitely advantageous, it is essential that team members have previous Africa experience in countries with high HIV prevalence. The team should number no more than five persons who should possess the skills and experience below: International Lead Consultant (Team Leader): • Demonstrated experience (7 years) in evaluation design, implementation and leadership experience and excellent writing skills is required. • Practical experience (5 years) supporting or evaluating HIV/AIDS and health service delivery in a multi-sectoral decentralized, environment. • Public/Private partnerships with a focus on local governments and civil society • Knowledge of capacity building of key leadership and service delivery • Knowledge of USG and USAID presidential initiatives III ANNEXES: Final Evaluation of UPHOLD programme • Knowledge of Local Government service delivery systems in a decentralized setting. This will bring in international experience in this area Local consultants (4) should collectively cover the following fields or experience: • Capacity development in context of decentralization • HIV/AIDS and health service delivery in a multi-sectoral, decentralized, environment • Public/Private partnerships with a focus on local governments and civil society • Evaluation experience 6.2 Roles and responsibilities The key roles and responsibilities for the Evaluation Team Leader, USAID, and GOU and UPHOLD during the evaluation are: USAID Select and contract the evaluators • Manage the evaluation process • Provide briefing to team • Review draft report and provide feedback • Sign off the final report • Submit evaluation report to USAID/PPC/CDIE Government of Uganda (Ministry of Health (MOH), Ministry of Local Government (MOLG), and Uganda Aids Commission (UAC) • Provide concurrence on the evaluation scope of work • Serve as key points of reference and information, including key documents, for review • Provide concurrence with the evaluation team’s inception report and work plan • Participate in oral debriefing • Review and comment on draft report UPHOLD • Review inception report and work plan • Serve as key points of reference and information, including key documents, for review • Provide logistical support for the evaluation team including office space, assistance with setting up meetings, interviews and providing transport Evaluation Team Leader’s roles and responsibilities:- • Guide and manage evaluation exercise • Responsible for all deliverables to USAID (Work plan, draft and final reports) • Coordinate preparation of the final schedule of meetings and field trips • Ensure that the schedule, as agreed with USAID is adhered to • Lead debriefing meetings 7. SCHEDULE It is anticipated that the evaluation can be completed in eight weeks. The evaluation will begin on or about January 29th 2007 illustratively, it is proposed that team members spend approximately 18 days doing field work. It is not expected that the team will visit all IV ANNEXES: Final Evaluation of UPHOLD programme UPHOLD districts but will select a representative sample with the input of USAID. The Team Leader will provide a final report to USAID no later than March 19th 2007. There will be an Initial conference (team planning meeting) with USAID/Uganda staff, UPHOLD, UAC and MOH representatives during which the team will present an inception report for undertaking the evaluation for input and finalization. In this meeting USAID will also clarify to the team the scope of the evaluation. There will also be oral debriefing meetings during which the team will present an initial draft of the evaluation report, highlighting key findings and recommendations while receiving clarification and input from stakeholders. An illustrative time line for the evaluation is as follows: • 1 week – Document review and preparation for field work • 3 weeks – Field Work (UPHOLD districts and Kampala) • 1 week – Preparation of initial draft report and oral presentation to USAID • 1 week - Final draft report writing and submission to USAID • 1 week – Review of final draft report by USAID and key stakeholders and comments sent to Evaluation Team • 1 week – final report writing and submission to USAID 8. DELIVERABLES Five (5) deliverables are required under this contract: 1) An Inception report including a proposed detailed methodology for carrying out the evaluation including draft copies of data collection instruments and a sampling framework. Detailed work plans for the evaluation complete with timelines for various activities that are envisaged i.e. fieldwork, document analysis, report writing etc. 2) Weekly e-mail progress reports to USAID on status of the evaluation 3) IA report and oral presentation of initial draft report highlighting key findings, conclusions and recommendations five working days of the conclusion of the fieldwork 4) Final draft report that should be submitted to USAID for review and written comments in both hard copies (5) and one electronic copy. It should be submitted within five working g days of submission of the initial draft report. 5) Final Report that should be submitted in both hard copies (5) and one electronic copy within five working days of receipt of USAID’s comments o the final draft report. The report should conform to the following specifications:- • Should not exceed 40 pages of text in the body of the report (excluding an executive summary and annexes) • Must conform to report structure contained in Attachment A • Should mainly focus on questions posed by this TOR/SOW and should include specific recommendations • Must be processed using Microsoft Word 98 or higher and be in Times New Roman 12 point font 6) Data sets and copies of all the instruments used in the evaluation: cleaned labeled and ready to use electronic copies of datasets collected through fieldwork (preferable SPSS – PC format) and cleaned ready to use electronic copies of Focus Group Discussion responses if any. In addition, electronic copies of all the instruments used in data collection must be delivered to USAID V ANNEXES: Final Evaluation of UPHOLD programme 9. BUDGET 9.1 Budget in (person days) for Evaluation Team The budget in person days for the Evaluation Team is shown in the table below. Tasks Team Leader Consultants (4) 1) Document Review and preparation of Inception Report 3 3 2) Initial Team Planning Meeting with USAID and selected stakeholders and preparations for field work 2 2 3) Data collection24 18 18 4) Initial draft Report Preparation 5 5 5) Final draft report writing 5 5 6) Final report writing 5 3 Total 38 36 9.2 Financial budget for Evaluation Team The tentative financial budget for the evaluation is in Attachment B. 10. TERMS OF PAYMENT The consultants will be paid in accordance with their individual contract with USAID, but in no case will final payment be issued prior to USAID’s acceptance of the final report. 24 It is anticipated that the team will cover 6 UPHOLD districts i.e. 1 per UPHOLD administrative/geographical region. Each team consisting of 2 persons would cover 3 districts working 5 days in each district = 15 person days. The balance i.e. 3 days would be spent in Kampala to interview key stakeholders at national level. The team leader may switch days across activities, but the total numbers of days should not exceed 36 days for local consultants and 38 days for the team leader. VI ANNEXES: Final Evaluation of UPHOLD programme VII ILLUSTRATIVE STUDY REPORT OUTLINE Cover page (Title of the study, the date of the study, recipient’s name, name(s) of the evaluation team. Preface or Acknowledgements (Optional) Table of Contents List of Acronyms Lists of Charts, Tables or Figures [Only required in long reports that use these extensively] Executive Summary [Stand-Alone, 1-3 pages, summary of report. This section may not contain any material not found in the main part of the report] Main Part of the Report Introduction/Background and Purpose: [Overview of the final evaluation. Covers the purpose and intended audiences for the study and the key questions as identified in the SOW) Study Approach and Methods: [Brief summary. Additional information, including instruments should be presented in an Annex]. Findings: [This section, organized in whatever way the team wishes, must present the basic answers to the key evaluation questions, i.e., the empirical facts and other types of evidence the study team collected including the assumptions] Conclusions: [This section should present the team’s interpretations or judgments about its findings] Recommendations: [This section should make it clear what actions should be taken as a result of the study] Lessons Learned: [In this section the team should present any information that would be useful to people who are designing/manning similar or related new or on-going programmes in Uganda or elsewhere. Other lessons the team derives from the study should also be presented here.] Annexes [These may include supplementary information on the evaluation itself; further description of the data collection/analysis methods used; data collection instruments; summaries of interviews; statistical tables, and other relevant documents.] ANNEXES: Final Evaluation of UPHOLD programme APPENDIX 2: Evaluation Framework FRAMEWORK FOR EVALUATION OF KEY QUESTIONS Evaluation Questions Analysis of the questions Methods for Data Collection Method Data Source Data Analysis and Interpretation 1. To what extent has UPHOLD achieved its overall HIV/AIDS and health goals and results? The UPHOLD conceptual formulation has 5 key performance areas (KPAs) viz. ƒ Public service delivery ƒ Quality of public service delivery ƒ Support to public service delivery ƒ Private sector delivery ƒ Grants Programme Within each of these 5 KPAs there are multiple: ƒ specific objectives ƒ geographical areas of focus (districts). For the programme as a whole (central level) for each specific objective there will be a comparison between actual and baseline indicators Central level review Central level review Review of baseline and periodic performance information collected against the programme Performance Monitoring Plan (PMP) Review of other primary data pertinent to UPHOLD’s result areas Interviews with key UPHOLD staff responsible for specific objectives e.g. -Health specialist plus manager of focal areas -Senior specialist HIV -Yellow star manager -Private sector specialist UPHOLD programme records and primary sources used by UPHOLD for PMP data National type surveys e.g. DHS, antenatal sero￾prevalence surveys Interview schedule and questionnaires Quantitative comparisons of targets and actual performance. Comparison of UPHOLD results with externally obtained results. Qualitative review and interpretation of interviews VIII ANNEXES: Final Evaluation of UPHOLD programme Evaluation Questions Analysis of the questions Methods for Data Collection Method Data Source Data Analysis and Interpretation Central level review Central level review UPHOLD districts will be compared with non-UPHOLD districts District Level review The district level review will add color and flavor to the central level overview. Given the time and logistical constraints this review will be confined to reviewing 2 districts in depth and another 2 more superficially. -Grants manager Review UPHOLD focus area specific reports Interview key stakeholders. Ministry of Health counterparts and central level beneficiaries of UPHOLD activities Review of comparable data from UPHOLD and non￾UPHOLD districts e.g. tracking of relevant league table positions over time Review of district profiles and activities Interview regional office staff including regional director & community participation officer Focus Group with technical officers from local government and the health sub-district in￾charges Interviews with CSO grantees Visits to facilities Community type visits and Focus area specific reports Interview schedule and questionnaires MOH annual reports UPHOLD central office and regional office records Interview Schedule Focus group discussions Interview schedule Observation of relevant activities Interpretation of both quantitative and qualitative information contained in reports Qualitative interpretation of interviews Quantitative analysis. Interpretation and discussion of comparisons. Understand the reasons for the specific interventions; analyze and interpret the specific needs and reasons for these Interpret the results of the interview Interpret results of focus group Interpret and analyze interviews Interpret results of visits IX ANNEXES: Final Evaluation of UPHOLD programme X Evaluation Questions Analysis of the questions Methods for Data Collection Method Data Source Data Analysis and Interpretation communication with beneficiaries e.g. community medicine distributors and their clients Interview/discussion with psychosocial groups for HIV Observation of relevant activities; Discussion with community members where possible Write up of discussions Interpretation of visit and discussions Interpretation of visit and discussions 1a How did the programme design’s strengths or limitations contribute to the achievement of goals and results? Identification of design (and implementation) factors most critical for success, and relevant for scaling-up programme elements to the district or national level. Also identification of any design factors that had negative effect on the programme results Interviews with: ƒ UPHOLD staff ƒ Key stakeholders ƒ USAID staff Review of programme design documents and any modifications. UPHOLD programme team and programme records Programme design documents and modifications Analysis of interviews with questions specifically focused on this area Critical review, analysis and interpretation of programme design and modifications 1.b. How did the programme team’s technical competency to undertake the planned activities contribute to the achievement of goals and results? Description of the level of competency (or lack thereof) of key UPHOLD staff. Interviews with UPHOLD staff Interviews with key informants Interviews with USAID staff Interview schedules/questionnaire results Analysis and interpretation of interviews 1.c. How did the programme’s technical support and capacity building for districts and grantees contribute to the achievement of goals and results? Description of capacity building activities and results and grantee support Interviews with UPHOLD staff Interviews with key informants Interviews with USAID staff Case studies from districts/interviews at district level Interview schedules/questionnaire results Reviews and write up of district visits Analysis and interpretation of interviews Reviews and write up of district visits 1.d How did external factors contribute to the achievement of goals and results? Review of the external environment, (global, national and regional) and analysis of factors that impacted on the programme Key informant interviews Review of UPHOLD reports Review of other reports Questionnaires UPHOLD annual reports e.g. MOH annual report, WHO annual reports Interpretation of interview schedules Interpretation of reports Interpretation of reports 2. How did UPHOLD Analytic descriptions of relationship Interviews with UPHOLD staff Interview Analysis and interpretation of ANNEXES: Final Evaluation of UPHOLD programme XI Evaluation Questions Analysis of the questions Methods for Data Collection Method Data Source Data Analysis and Interpretation develop and manage relationships with USAID implementing partners, GOU partners and civil society. between UPHOLD and: ƒ USAID ƒ MOH ƒ Civil society ƒ Consortium partners ƒ Grantees, specifically including local government Interviews with key informants Interviews with USAID staff Interviews with consortium members Interviews with grantees Review of programme documents schedules/questionnaire results Minutes of key meetings/documentation of decisions taken interviews Analysis and interpretation of documentation 3. Assess whether UPHOLD yielded any unintended positive/negative results? Review un-anticipated results of the programme “What else” questions integrated into all key informant interviews including partner and beneficiary interviews Inclusion of broader questions about “what’s different” and “why” in district, partner and beneficiary situations Document reviews Review of Interviews UPHOLD annual reports Analysis and interpretation of interviews Analysis and interpretation of reports 4 What are the key lessons learned from the design and implementation of this large multi-sectoral district based programme. This question requires conclusions based on answers developed for previous questions. Note: given that only the HIV/AIDS and health care aspects of UPHOLD are to be covered by this evaluation, lessons learned will not be multi-sectoral, i.e., lessons will not be derived concerning the education component of this programme. N/A N/A Analytic examination of findings on all previous questions. ANNEXES: Final Evaluation of UPHOLD programme APPENDIX 3: Schedule of Activities A. Central (Kampala) Interviews and activities Date Person interviewed 21 Ekaru Stephen- National TB and Leprosy Programme st June 2007 Commissioner for Quality Assurance, Ministry of Health, Emmanuel Kaijuka Assistant Commissioner, Health Promotion and Education Ministry of Health, Paul Kaggwa Senior Medical officer and Focal Person for Malaria Case Management, Malaria Control Programme Ministry of Health, Kato Frederick Chief of Party, UPHOLD Presentation Deputy Chief Of Party Technical, UPHOLD Lucy Shillingi, Senior Health Adviser UPHOLD Margaret Kyenkya Deputy Chief of Party Regional UPHOLD Barbara Durr, Senior HIV/AIDS Specialist UPHOLD Alexander Mugume, Deputy Chief of Party Monitoring, Evaluation and Dissemination UPHOLD Elizabeth Ekochu 22nd June 2007 Senior Communicable Disease Specialist, UPHOLD Betty Mpeka Inception report and discussions with Stakeholders UPHOLD CTO, USAID Andrew Kyambadde , HIV/AIDS Team Leader, USAID Elise Ayers Senior Malaria Technical Adviser, USAID Gunawardena Dissanayake May 23rd 2007 Education Specialist, USAID Sarah Mayanja Ag. Principal Education Officer, Ministry of Education and Sports, (PIASCY)-Kusemererwa E. Araali Programme Director, Monitoring and Evaluation of Emergency Plan Progress Programme, Vincent Owarwo Mugumya AIDS Information Centre Ag. Executive Director, Samali T. Lubandi, Monitoring and Evaluation Manager Cissy Kirambaire, Programme Director Francis Nahamya Ministry of Local Government Assistant Commissioner, Patrick Mutabwire Uganda Private Midwives Association (Grantee) Executive Director Micheal Matsiko, Branch Coordinator Mary Musisi, President Sakina Kiggundu Assistant Commissioner, HIMS and Resource Centre Ministry of Health, Eddie Mukoyo May 24th 2007 Deputy Director Programme Management, The AIDS Support Organization - Robert Ochai Sekimpi –Kirwana Lawrence , Programme Manager Tuko Club Programme Manager, STD/AIDS Control Programme, Ministry of Health, Elizabeth Madraa Assistant Commissioner, Health Services Ministry of Health, (EPI) Makumbi Issa , Principal Assistant Secretary Ministry of Education and Sports (PIASCY)- Aggrey D Kibenge May 25th 2007 Behavioral Change and Communication Specialist, Kenneth Mulondo XII ANNEXES: Final Evaluation of UPHOLD programme Date Person interviewed Programme Management Specialist USAID, Jessica Kafuko Senior Technical Adviser USAID, Serene Thaddeus Deputy Chief of Party Finance and Administration, UPHOLD Katrina Kruhm June 4th 2007 Malaria Consortium Africa Director Graham Roots, Vector Control and Emergency Specialist, Kate Kolaczinski June 5th 2007 Chief of Party, UPHOLD Samson Kironde June 7th 2007 Presentation to USAID, MEMS and UPHOLD. Teleconferences President, Manoff Group Marcia Griffiths World Education Inc. Vice President, Africa Division Burchfield Shirley, Senior Technical Adviser Education Inc. Gill Garb June 8 Presentation of findings to Stakeholders th 2007 Principal Medical Officer, Ministry of Health Zainab Akol B. District Visits The evaluation of UPHOLD had two main components, viz central level activities and district level activities. Although it was desirable that as many districts and as many activities as possible be reviewed in the field, where the ultimate impact is felt, this was constrained by (1) size of review team of 4 to 5 people (only two teams possible) and, (2) need to meet and interview key central level stakeholders It was planned that the Evaluators visit and review UPHOLD activities and impact in 4 selected districts in reasonable breadth and coverage. The review team also considered going to non-UPHOLD districts to try and compare differences, if any between UPHOLD and non-UPHOLD districts. However this option was discarded for a number of reasons: • It would be unlikely that in the short time available that quantifiable differences would be found in a field visit • Time is of essence and reviewing activities and impact in UPHOLD districts should be maximized • Central level review of key indicators relevant to UPHOLD activity could be based on available information (e.g. MOH HMIS data, DHS data) and that comparisons between UPHOLD and non￾UPHOLD districts, would be done based on this information. In the light of the above factors a number of selection criteria were made and based on these, 4 districts were selected and agreed by both UPHOLD and USAID. Each team visited two districts. District Selection Criteria 1. Regional representation 2. Coverage of technical areas 3. Status (conflict/Post Conflict/Normal) 4. Security situation 5. New/Old district 6. Location in relation to other choices XIII ANNEXES: Final Evaluation of UPHOLD programme Selected Districts 1. GULU • Northern region • Conflict transition • Has both ITN and HBMF(using Coartem) • Non AIM district • Has been split to form Gulu and Amuru 2. LIRA • Northern region, Post conflict • No UPHOLD supported HIV/AIDS interventions, • Has both Child Heath IRH CSOs • AIM district • Next to Gulu (time and travel consideration) • Has been split to form Lira and Dokolo 3. KYENJOJO • Western region • All activities apart from Growth promotion • Old district and not split • Has been supported by UPHOLD since the beginning • Presents in depth assessment 4. ISINGIRO • Western region • New district • All activities apart from Growth Promotion • Non AIM district TEAM 1 DATE DISTRICT TIMING ACTIVITY COMMENT Sunday May 27th 2007 GULU Depart KLA 2pm and arrive 6 pm Travel Stay in Gulu Monday 28th May 2007 GULU 8:30 am meet with regional office Continuation of meetings and visits Stay in Gulu Tuesday 29th May 2007 GULU Whole day Continuation Stay in Gulu Wednesday 30th May 2007 GULU Work up to 3:00 pm then leave for Lira Continuation and travel Depart for LIRA Thursday 31st May 2007 LIRA Whole day Regional Office Visit LG Visit CSOs Stay in Lira Friday 1st June 2007 LIRA Whole day Visit LG Visit CSOs Stay in Lira Saturday 2nd June 2007 Depart for Kampala TEAM 2 DATE DISTRICT TIMING ACTIVITY COMMENT Sunday May 27th 2007 KABAROLE Depart 12:00 a.m.- Arrive Fort Portal Travel Stay in Fort Portal Monday 28th May 2007 KABAROLE and KYENJOJO Meet with Regional office Travel to Kyenjojo Visit LG Visit CSO Stay in Fort Portal Tuesday 29th May KYENJOJO Whole day Continuation Stay in Fort Portal Wednesday 30th May 2007 KYENJOJO CSO visit and travel to Mbarara Depart at 4:00 pm for and stay in Mbarara Thursday 31st May 2007 MBARARA and ISINGIRO 8:30 am Afternoon travel to Isingiro Visit Regional Office Visit LG Visit CSO Stay in Mbarara Friday 1st June 2007 ISINGIRO Whole day Continuation Stay in Mbarara Saturday 2nd June 2007 Depart for Kampala XIV ANNEXES: Final Evaluation of UPHOLD programme APPENDIX 4: Persons Involved Stakeholder’s meeting May 23, 2007 - Registration Name of guest Organization Title 1 Adoko George Lira District DCAO /FPP 2 Aggrey D. Kibenge MOES Principal Asst Sec 3 Alex Mugume UPHOLD S/HIV/AIDS Adv 4 Alice Mugoya Ibale UPHOLD PIASCY Prog Mgr 5 Allen Kebba Evaluation Team 6 Andrew Balyeku Evaluation Team 7 Andrew Kyambadde USAID UPHOLD CTO 8 Anthony K. Mugasa UPHOLD IRH Specialist 9 Augustine Wandera MEMS M&E 10 Barbara Durr UPHOLD DCOP 11 Betty Mpeka UPHOLD SCDS 12 Christine Oryema-Lalobo UPHOLD Regional Director North 13 Cissy Kirambaire AIC M&E Mgr 14 E. Mukooyo MOH / RC ACHS/KC 15 E.F. Katumba MOH/RH PMO 16 Elise Ayers USAID H/A Team Leader 17 Elizabeth Ekochu UPHOLD DCOP / M&E 18 Elizabeth Madraa MOH/ACP/STD PM 19 Francesca Akello UPHOLD CPC North 20 I. Makumbi MOH ACHS 21 J. C. Okello UPHOLD Regional Director 22 Katrina UPHOLD DCOP / FA 23 Kusemererwa Araali E. MOES Ag. PEO/PPE 24 Luwa John Charles Gulu DLG HIV/AIDS FPP 25 Margaret Kyenkya UPHOLD Snr. Health Adv 26 Matsiko Michael UPMA Executive Director 27 Musisi Geoffrey UPHOLD Grants Mgr 28 Namusisi Mary UPMA Branch Coordinator 29 Nyehangane W Isingiro DLG DHO 30 Owarwo Vincent MEEP Programme Director 31 Patrick K. Mutabwire MOLG C/LCD 32 Peter Barron Evaluation Team Leader 33 R. Walusimbi USAID M&E Specialist 34 Robert Ochai TASO D. ED 35 Samali T. Lubandi AIC Ag. ED 36 Samson Kironde UPHOLD CEO 37 Saul Onyango MOH SMO 38 Sekimpi-Kirwana Lawrence TUKO Programme Mgr 39 Specy Kakiiza UPHOLD CPC / Ag. RD 40 Tumukurate Espilidon UPHOLD Regional Director XV ANNEXES: Final Evaluation of UPHOLD programme Evaluation Debrief June 08, 2007 Participant list Name Organization Title 1. Adoko George Lira DLG DCAO (FP) 2. Akol Zainab MOH PMO 3. Alice M. Ibale UPHOLD Prog Manager-PIASCY 4. Allen Kebba Evaluation Team Evaluation Team Member 5. Andrew Balyeku Evaluation Team Evaluation Team Member 6. Anthony Kihika Mugasa UPHOLD IRH Specialist 7. Apollo Nkwake UPHOLD M&E Specialist 8. Barbara Durr UPHOLD DCOP 9. Buzaalirwa Lydia UPHOLD HIV/AIDS Specialist 10. Byaruhanga Raymond TASO Executive Director 11. C.Okello UPHOLD Regional Director, 12. Christine Oryema-Lalobo Evaluation Team Evaluation Team Member 13. Dorothy Aanyu Angura UPHOLD Snr. Ed Advisor 14. Francesca Akello UPHOLD CPC North 15. Geoffrey Musisi UPHOLD Grants Manager 16. Jessica Kafuko USAID Prog Manager Specialist 17. John Kyakulaga UPHOLD Private Sector Specialist 18. Josephine B. Kasaija UPHOLD OD Specialist 19. Joshua Kakaire Kibedi UPHOLD Data Manager 20. Justine Nankinga MOH Evaluation Team Member 21. Kenneth Mulondo UPHOLD BCC Specialist 22. Kikaffunda Richard UPHOLD CPC 23. Kusemererwa Araali Emmanuel MOES Ag. PEO/PPE 24. Lucy Shillingi UPHOLD DCOP Tech 25. Mugume A. B. UPHOLD Snr. HIV/AIDS Advisor 26. Mutebi Perez TUKO Club Member 27. Nyehangane William Isingiro DLG DHO 28. Opwonya John Odong Gulu DLG DTLS/UPHOLD 29. Paul Kagwa MOH ACHS HP&E 30. Peter Barron Evaluation Team Evaluation Team Leader 31. Rhona Walusimbi USAID M&E Specialist 32. Robert Ochai TASO Deputy ED 33. Samson Kironde UPHOLD COP 34. Souleymane Barry MEEPP Advisor 35. Specy Kakiiza UPHOLD Regional Director 36. Xavier Nsabagasani UPHOLD Snr. Action Research Specialist XVI ANNEXES: Final Evaluation of UPHOLD programme Internal debrief June 07, 2007 – Participant list Name Organization Email 1. Alex Mugume UPHOLD amugume@upholduganda.org 2. Alice M. Ibale UPHOLD aibale@upholduganda.org 3. Allen Kebba ET arkconsult@siticable.co.ug 4. Andrew Balyeku ET abalyeku@gmail.com 5. Augustine Wandera MEMS awandera@mems.co.ug 6. Barbara Durr UPHOLD bdurr@upholduganda.org 7. Christine Lalobo UPHOLD clalobo@upholduganda.org 8. Elise Ayers USAID eayers@usaid.gov 9. Geoffrey Musisi UPHOLD gmusisi@upholduganda.org 10. Justine Nankinga MOH nankingaj@yahoo.co.uk 11. Lucy Shillingi UPHOLD lshillingi@upholduganda.org 12. Pascal O. Okello UPHOLD pokello@upholduganda.org 13. Rhona Walusimbi USAID rwalusimbi@usaid.gov 14. Roy Thompson MEMS roy@mems.co.ug 15. Samson Kironde UPHOLD Skironde@upholduganda.org 16. Xavier Nsabagasani UPHOLD xnsabagasani@upholduganda.org List of People Interviewed in Districts Kyenjojo District Kyenjojo Local Government Peter Nsugura Ruhweeza: Ag. Deputy Chief Administrative Officer/UPHOLD Focal Point Julius Bahinda: District Health Officer Musinguzi: DCCA Kimara H.E: DRS Ruhweza F. Senior nursing Officer/District Health Visitor Mujaasi David District Health Inspector Asiimwe S. D Senior Education Officer Tibakamya G District Education Officer Inkasiimire A. Peter District Inspector of Schools Bwerere George District Health Educator Mbabazi Jullet Community Medicine Distributor Kyegagwa Health Centre 11 Dr. Mucunguzi Medical Officer in Charge Akello Tabitha Nursing Officer Busibye Lawrence Senior Clinical Officer Kyakatara Health Centre and Fort Portal Diocese Fr. Kaahwa Leopold HIV/AIDS Focal Point Rev. Kalyebara Stephen Exec. Director DEFORA Diocese Baguma Monic PSS Group member Marunga Beatrice Administrative assistant Kobisingye Doreen Accountant Kamaningo I PSS Group member Bagonza Joseph PSS Group member Mbabazi Consolata PSS Group member Kanunyazi Jennifer PSS Group member Isoke Tom PSS Group member Ayebala Josephine PSS Group member Tooro Kingdom Mirembe Justine Programme Officer Tumuhairwe Raymond Programme Manager Mawambi Wilson Programme Finance and Admin. Officer Rwamuhokya Robert Programme Field Officer Hunhhilda Angela Nursing Officer XVII ANNEXES: Final Evaluation of UPHOLD programme Basemera Esther Enrolled Midwife RWIDE Mubiru Vincent Technical Adviser Mukasa Robert Finance and Administrative Officer Kanyambe Godfrey Agricultural Programme Officer Byansi Ben Coordinator KAAC )RWIDE partner Asiimwe Charles OVC officer (Volunteer) Post Test Club Members RWIDE Mwesige Kantagomba Asiimwe Beth Asiimwe Catharine Musa Kemigisa Grace Kobusinge Veronica Tuhaise Rest Mwanuttya Erick Kumhendo Rose Mbabazi Maga Tumuhibmise Kristina Kajumba Leya Kiza Asa Nyangireki Sisiria Kemigisha S Kobusinge M Fibasaga F Banaura Joana Banobere E. Kirokimu Felista Akelo Katalina Manyindo Mbabazi Tibakanya Felista Kanyunyuzi Keeti Kemigisha Violet Razigata Mary Kansiimwe Juliet Katusiimwe Mwesige Kasukale Gres Kabosomi Margaret Kobosinge Macureti Kabakidi Mary Kaddina Kabagenyi Sitela Kagaba Oliver Tibakanya Efurazia Kabajuma Erinora Ahebirungi Robati Kimanya Jona Kajumba Kikitoliya Kasembo Jennifa Kembabazi Olivia Kengozi Mary Asiimwe Feluku Sunday Jasi Edikana Kamuhendo Kisembo Asa Katusabe Grace Kabajwara Oliver Kezabu Sarah Kabagombe Rest Akugizibwe B Tibasage Fatima Abas Kasaija Kabosinguzi H Kabonera G Ahugizibwe Oliver Be Faithful Couple Counselors Kaganda George Kaganda Grace Rukooba Angelica Rukooba Abwooli John Bosco Nkoko William Nkoko Gorett Baguma K. Richard Baguma Stella Basaliza Wilson Basaliza Jamima Kaya Beatress Kaya William Rwaheeru Taddeo Rwabogo Annet Kugonza Enid Mwebesa Moses Asaba K Faith Asaba K Arthur Zainabu Karungi Mujaasi Abdul Noor Kyembogo Holy Cross Mission Health Centre Tusiimwe Florence Chairperson PLWA Ayabale Owen Deputy Director Sr. Angelica Birungi Director AIFIA Post Test Club Tinka Mary Halwima Y Musiime E Sayuni M Katusabe R Atuzarirwe B Tungumisirize O Bazimazikyi M Kyesirikora M Kazigati Beatrice Tumwesigye M Shera Kabaganda Katushaba Cozi Banyenzaki Amos Kabanyaka Marga Kusenene Edward Majara Mary Grace Kabasambu Enid Kobusinge Hope Twesigye Robert Nsungwa Anazia Maliya Murungi Yazefu Amanyire Rusia Ahaisibwe Yovani Isingiro District Isingiro District s Byakatunda Tom Ag. Chief Administrative Officer Nyehanganew District Health Officer/ UPHOLD Focal point Monday Justus District Immunization Focal Point Byaruhanga Grace Ag. DEO Oyesigye Fred District Malaria Focal Point Mugisha Godson District TB/Leprosy Supervisor Maturity Audio Visual Namara Joseph Chairman Maturity Audio Visuals Tumesigye Wycliff Coordinator Rev Canon Francis Mutatwine Greater Mbarara Zimbeiha Joseph Programme Manager Mayanja Memorial Hospital Foundation Mugerwa Benon Hospital Director Yvonne Natukunda Programme Manager Oniel Patrick Data Manager Tumwebaze Henry Homebased HCT and care supervisor Mugisha Naboth Finance Officer XVIII ANNEXES: Final Evaluation of UPHOLD programme Baketunga Johnson Counselor Aisha Muslim Girls School Abstinence Club Nakatoogo Nazifah Chairperson Mpeirwemukama Dorothy Assistant Chairperson Namuddu Mariam Secretary Nabakooza Amisah Assistant Secretary Nankabirwa Rose Choir Mistresss Akandanaho Privah Assistant Choir Mistress Linday Kamay Speaker Gulu District Otto Langoya Sahid Chief Administrative Officer, Gulu Opwonya John Odong UPHOLD Focal Person Luwa Jogn Charles HIV/AIDS Focal Point for Gulu District Ojul Lakop Christine Yellos Star Focal Point Person Ojok Richard Napthali HIMS In Charge Awuru Felix Odwar Santa Assistant Chief administrative Officer Samalie Nabossa Programme manager, World Vision Christine Lamuno Councilor Bungatira LC3 Catherine Aromorach HIV/AIDS Coordinator, world Vision, Gulu Beatrice Acan Clinical Officer, Bungatira Health Centre 11 Ayoo Phoebe Ocan Community Development Facilitator, World Vision Sanya Gilbert Community Development Facilitator, World Vision Charles Wasswa Community Development Facilitator, World Vision Abwola Sunday Technical Adviser, ACCORD Jack Walter Staff member ACCORD Charles Otto Staff Member ACCORD Obutu Francis Health Alert Ojok walter Redeemed Bible Way Church Organization Auma Lily Community AIDS Resources Opira George William Accountant, Redeemed Bible Way Church Organization Luom Timothy Secretary, Agonga Youth Alliance Awany John Treasurer, Agonga Youth Alliance Akullu Jane ACCORD Geoffrey Okello Treasurer District Network PHAS Kitara Nelson Field assistant CARPP George Opeluk Chairman District Network PHAS Aouny Michael Chairperson, Lacwec Ber Akwero Jacinta Programme Officer, ACCORD Geoffrey Binaisa Opiri Accountant, Gulu Youth Centre Lanyero Sarah Medical Clinical Officer, Gulu Youth Centre Kibwola Denis Centre Manager, Gulu Youth Centre Lira District Benard Otim Vector Control Officer, Lira District Local Government Issac Ogwal Chain Technician Officer, Lira District Local Government Ocao John Bosco HMIS in charge Lira District Local Government XIX ANNEXES: Final Evaluation of UPHOLD programme APPENDIX 5: Interview tools Interview Questionnaire - Standard Introduction Evaluation team commissioned by USAID to do end of programme review. 5 people ƒ What do you know about UPHOLD? Probe for specific area ƒ What has been the nature of the relationship with UPHOLD ƒ What contribution has UPHOLD made? Probe specific KPAs What was done where and when? Service delivery quality Public sector Private sector ƒ How has your department engaged with UPHOLD? ƒ What areas has UPHOLD achieved/contributed? Probe for limitations ƒ Has UPHOLD done a good job? Probe for what should have been done differently/weaknesses/limitations Probe yes/no ƒ What has UPHOLD achieved? Illustrate with examples ƒ Why was this achieved? Illustrate with examples ƒ Should there be another UPHOLD? Probe yes/no ƒ Is there anything else about UPHOLD that we haven’t asked? THANK YOU XX ANNEXES: Final Evaluation of UPHOLD programme Interview Questionnaire UPHOLD staff Introduction ƒ Describe your job in UPHOLD ƒ What is your understanding of UPHOLD? Probe what is nature of programme; what is the big picture? What are the interventions? Was the intention to have integrated interventions a success or not? What were the problems related to integration and multi-sectoral programme? What was the influence of external factors? ƒ What have been the achievements of your section/department/area made? Probe specific KPAs What was done where and when? Service delivery quality Public sector Private sector ƒ How have these achievements contributed to the overall health goals of Uganda? ƒ Has UPHOLD done a good job? o UPHOLD generally o Your Department/section specifically Probe for what should have been done differently/weaknesses/limitations Probe for what activities/targets have not been achieved and why? ƒ How has UPHOLD managed relationships? Probe for implementing partners/MOH/civil society/USAID (for DCOPs) Probe communication/interaction with stakeholders/focus/centre vs district (relationship with LG as well) Probe for other implementing agencies (e.g. UNICEF) ƒ What could have been done differently/better? ƒ UPHOLD generally ƒ Your Department/section specifically ƒ Were there any unanticipated activities? Did these result in positive/negative outcomes/results ƒ What are the main lessons that you have learnt? ƒ Should there be another UPHOLD? Probe why? Why not just SWAPS where money given to the government? ƒ Have you personally benefited/been capacitated by working for UPHOLD Is there anything else about UPHOLD that we haven’t asked which you think is important? XXI ANNEXES: Final Evaluation of UPHOLD programme APPENDIX 6: Bibliography • A Review Of The Local Government Budget Framework Papers 2003/04 – 2005/06: Summary Of Objectives And Priorities 2005 • A situation analysis for UPHOLD’s strategic planning: sample districts: Mbarara and Kamuli 2004 • Abstinence and be faithful Project Evaluation Report, Maturity Audio visuals – Uganda 2006/2007 • An evaluation of UPHOLD’s Long Lasting Insecticide Treated Nets (LLINS) distribution exercise: a process evaluation report, UPHOLD 2006 • An evaluation of UPHOLD’s long lasting insecticide treated nets (LLINS) distribution exercise: a process evaluation report UPHOLD (undated) • Analysis of Possible effects of Graduated Tax, Staff restructuring and redistricting on UPHOLD programs, UPHOLD 2006 • Annual Health Work plan Report Lira District Local Government 2006/2007 • Annual Report 2003 UPHOLD • Annual Report 2004 UPHOLD • Annual Report 2005 UPHOLD • Annual Report 2006 UPHOLD • Assessing private clinics readiness/ potential for providing HIV counselling and testing, UPHOLD 2004 • Assessment Of The Financial And Operational Management Of UPHOLD Local Government Grants 2005 • Assessment of UPHOLD districts reports for the financial year 2003 – 2004 • Best practices in community participation and gender mainstreaming: literature review and documentation UPHOLD 2003 • Brief about the achievement for the partnership of Isingiro District and UPHOLD, Isingiro District Local Government (Office of the District Health Officer) 2007 • Child Day and SNIDS communication campaign evaluation UPHOLD 2005 • Cooperative Agreement USAID 2003 • CSO Family and Community Action Grants Request For Applications 2004 • CSO Follow-Up Capacity Report JSI Research & Training Institute, Inc (UPHOLD) • CSO M & E capacity assessment and tools: Phase one dissemination report. Findings from consultant’s field visits UPHOLD • District Training and data collection report LQAS 2006: Arua, Maracha-Terego, Yumbe & Koboko districts UPHOLD 2006 • Evaluation Brief Mayanja Memorial Hospital Foundation 2007 • Evaluation Brief UPHOLD Rwenzori Region 2007 • Evaluation Report The Family and Community Action Grant • Factors influencing flow of funds and implementation of programs in Local Government supported by UPHOLD – 2005/2006, UPHOLD • Family and Community Action Grants Application JSI Research & Training Institute, Inc UPHOLD • Feasibility study of the mobile van for voluntary counselling and testing (VCT) for HIV/AIDS, UPHOLD 2004 • Feed Back Mbarara and Isingiro Community Volunteer Counsellor Association 2007 XXII ANNEXES: Final Evaluation of UPHOLD programme • Gathering Input to Develop Appropriate Responses to HIV/AIDS - Gulu District, February 2005 Boda Boda Riders • Health Sector Report on UPHOLD supported activities 2003/2004 financial Years Kyenjojo District Local Government • Integrated reproductive Health Project Brief Tooro Kingdom 2007 • LQAS survey report 2004, results from 20 districts of Uganda UPHOLD 2006 • LQAS survey report 2005: results from 20 districts of Uganda UPHOLD 2006 • National LQAS conference report: enhancing evidence based planning at district level, the LQAS experience, UPHOLD 2005 • Overview of national policies and priorities related to UPHOLD’S strategic framework 2004 • PMI Annual Report UPHOLD • PMP UPHOLD 2007 • PMP UPHOLD with Outputs UPHOLD 2005 • Post campaign communication evaluation report for the national measles immunization program UPHOLD (undated) • Post Campaign communication evaluation report for the national measles immunization program, UPHOLD 2004 • Pretest Report For Proposed Logo And Name For The Uganda Program For Human And Holistic Development (UPHOLD) 2003 • Process documentation and evaluation of partner defined quality methodology (PDQ), UPHOLD 2006 • Rapid analysis of AIC’s market environment for strategic planning, 2004 • Rapid assessment of monitoring and evaluation in AIDS Information Center (AIC), UPHOLD 2004 • Rapid Assessment of the new Malaria treatment Policy Implementation, UPHOLD 2006 • Report Kyegegwe Sub Health District 2007 • Report on LQAS data collection and supervision in Gulu, Koboko and Pallisa districts • Report on TV and print materials pre-testing for Ugandan national measles immunization campaign UPHOLD 2003 • Request for Application, Family and Community Action: A Competitive Grants Programme for Civil Society Organisations working in Health and HIV/AIDS JSI Research & Training Institute, Inc (UPHOLD) • Response to Technical Questions JSI Research & Training Institute, Inc (UPHOLD) 2003 • Response to USAID/Uganda about questions concerning MEMS Proposal for the UPHOLD Evaluation Monitoring and Evaluation Management Services 2007 • Response to USAID’s Scope of work for the final Evaluation of UPHOLD MEMS 2007 • Results of the 2006 net Mass treatment campaign in 27 districts of Uganda, UPHOLD 2006 • Review Meeting Report Kyambogo Holy Cross Family Centre 2007 • Review of implementation of the home-based management of fever strategy in UPHOLD supported districts, UPHOLD 2005 • RWIDE Briefs 2007 • Team’s Technical Approach JSI Research & Training Institute, Inc (UPHOLD) 2003 • Trials of Improved Practices (TIPS) UPHOLD 2004 • Uganda Demographic and Health Survey 2006 Preliminary Report UBOS XXIII ANNEXES: Final Evaluation of UPHOLD programme • Uganda Health Facilities Survey 2006: Performance of HIV/AIDS and Family Planning Commodity Logistics Systems JSI Deliver • UPHOLD annual work plan: April 2003 – March 2004 • UPHOLD draft annual work plan: October 2004 – September 2005 • UPHOLD draft annual work plan: October 2005 – September 2006 • UPHOLD draft annual work plan: October 2006 – September 2007 • UPHOLD Explanations for sector performance against targets, JSI Research & Training Institute, Inc (UPHOLD) 2007 • UPHOLD Extensions (Activity Implementation Status) JSI Research & Training Institute, Inc UPHOLD • UPHOLD Grantee Agreement ACCORD – GULU 2006 • UPHOLD Grantee Agreement Gulu District 2006 • UPHOLD Grantee Agreement Ibanda Child Development Centre Program 2006 • UPHOLD Grants Manual (UPHOLD) 2005 • UPHOLD strategic framework: working document for dialogue with district stakeholders (April 2003 – March – 2004) • Using a simple survey method to scale up evidence-based decision making at the district level: the UPHOLD project • Voluntary counselling and testing services (VCT) for HIV/AIDS national campaign baseline study, UPHOLD 2004 • Voluntary counselling and testing situation assessment for Kayunga district, UPHOLD 2004 XXIV