New Partners Initiative Technical Assistance Project (NuPITA) November 2012 Exploring the Effects of Holistic Capacity Building: An Evaluation of NuPITA’s Technical Assistance on NPI Partners’ Service Delivery and Sustainability The New Partners Initiative Technical Assistance (NuPITA) project is funded by the United States Agency for International Development (USAID) and implemented by John Snow, Inc. and Initiatives Inc., contract GHS-I-00- 07-00002-00. This document is made possible by the generous support of the American people through USAID. The contents are the responsibility of John Snow, Inc. and do not necessarily reflect the views of USAID or the United States Government. © November 2012 John Snow, Inc. CONTENTS Acknowledgments.............................................................................................................. i List of acronyms and abbreviations................................................................................. ii EXECUTIVE SUMMARY.................................................................................................. 1 CHAPTER ONE: INTRODUCTION............................................................................... 3 1.1 New Partners Initiative and New Partners Initiative Technical Assistance ................................... 3 1.1.1 NuPITA objective and core tasks............................................................................................... 4 1.2 Evaluation scope of work........................................................................................................................... 4 1.2.1 Purpose of the evaluation............................................................................................................. 4 1.2.2 Evaluation objectives...................................................................................................................... 4 1.2.3 Evaluation questions....................................................................................................................... 4 1.2.4 Evaluation team and duration ...................................................................................................... 5 1.3 Organizational Capacity Assessments..................................................................................................... 5 1.3.1 Major gaps identified during the TOCA.................................................................................... 5 1.3.2 Key NuPITA inputs provided to address identified gaps..................................................... 7 CHAPTER TWO: METHODOLOGY ............................................................................. 9 2.1 Evaluation design.......................................................................................................................................... 9 2.2 Study sites and population......................................................................................................................... 9 2.3 Data collection methods.......................................................................................................................... 11 2.3.1 Document review......................................................................................................................... 11 2.3.2 Key informant interviews ........................................................................................................... 11 2.4 Quality assurance, analysis, and presentation ..................................................................................... 12 2.5 Ethical considerations............................................................................................................................... 12 2.6 Evaluation limitations................................................................................................................................ 12 CHAPTER THREE: FINDINGS..................................................................................... 13 3.1 Effects of capacity building on service delivery results..................................................................... 13 3.1.1 NPI partners' views about the usefulness of NuPITA's capacity-building inputs........... 14 3.1.2 Changes in number of beneficiaries reached ......................................................................... 14 3.1.3 Changes in scope and approach in service delivery ............................................................. 19 3.1.4 Institutionalization of internationally recognized standards and procedures................. 19 3.1.5 Changes in the quality of support supervision....................................................................... 20 3.1.6 Use of program data to monitor service outcomes and inform management decisions.................................................................................................................................................... 20 3.1.7 Use of the Child Status Index (CSI)......................................................................................... 21 3.1.8 Changes in the functionality of referral systems/linkages................................................... 22 3.2 Effects of capacity building on the sustainability of NPI partner organizations........................... 22 3.2.1 Special case: Sustainability with change of staff and program ............................................ 29 3.3 NuPITA: Successes and areas for improvement ................................................................................ 29 3.3.1 Strengths......................................................................................................................................... 30 3.3.2 Areas for improvement............................................................................................................... 31 CHAPTER FOUR: DISCUSSION OF RESULTS AND CONCLUSION .................. 33 4.1 Discussion.................................................................................................................................................... 33 4.1.1. Service Delivery ........................................................................................................................... 34 4.1.2 Sustainability................................................................................................................................... 34 4.2 Conclusion................................................................................................................................................... 35 Annexes............................................................................................................................ 38 Annex A: Evaluation Framework – Service Delivery and Sustainability....................................................... 38 Annex B: Terms of Reference for the NuPITA Partners’ Impact Evaluation.............................................. 43 Annex C: Trends in Average OCA Scores vs. Program-Related Domains Scores................................... 46 Annex D: Support Summary Table........................................................................................................................ 47 Annex E: Individual Partner Reports..................................................................................................................... 48 Annex F: List of Persons Interviewed................................................................................................................... 69 Annex G: Schedule for Field Work....................................................................................................................... 71 Annex H: List of Documents Reviewed............................................................................................................... 73 Annex I: Interview Guides....................................................................................................................................... 74 LIST OF TABLES AND FIGURES TABLES Table 1: Selection Criteria for the Sampled Partner NGOs.................................................................. 10 Table 2: Summary of Findings on Service Delivery-Related Indicators for Each NPI Partner....... 15 Table 3: Summary of Findings on Sustainability-Related Indicators Disaggregated by NPI Organization....................................................................................................................................................... 26 FIGURES Figure 1: Aggregated Average Scores for Program Management, Project Performance Management, and Average Organizational Capacity.................................................................................................................... 6 Figure 2: Comparison of Targets and Results for PEPFAR Indicators.................................................18 Figure 3: Aggregated Average Scores for Selected Sustainability Sub-Domains................................ 23 i ACKNOWLEDGMENTS The evaluation was made possible with funding from the United States Agency for International Development (USAID) through the NuPITA project, which is implemented by John Snow, Inc., and partner Initiatives Inc. The evaluation team would like to recognize the planning and logistics support it received from NuPITA staff in Boston, Kampala, and Nairobi. We would like to call out Mark Kowalski, Deputy Director, and Naome Wandera, Senior Monitoring and Evaluation (M&E) Specialist, who provided day-to-day support for planning, arrangements, and clarifications. The team would also like to thank all who participated in interviews. (Organizations and individuals are listed in Annex F.) Respondents were thoughtful and engaged and spent considerable time providing the team with both data and insight about this distinctive partnership. Finally, we applaud the achievements of the NPI partners, whose commitment and determination allowed them to take full advantage of what USAID, through NuPITA, offered. ii LIST OF ACRONYMS AND ABBREVIATIONS ABC Abstinence, Be Faithful, and Condoms strategy AIDS Acquired Immune Deficiency Syndrome AMURT Ananda Marga Universal Relief Team APHIAplus AIDS Population and Health Integrated Assistance ARC American Refugee Committee BCC Behavior change communication CBO Community-based organization CDC Centers for Disease Control and Prevention CERI Children’s Emergency Relief International CLOCA Close-out organizational capacity assessment CSI Child Status Index FAO Food and Agriculture Organization FBO Faith-based organization FXB François-Xavier Bagnoud International GBV Gender-based violence GOU Government of Uganda GRACE Grassroots Alliance for Community Education HBC Home-based care HCT HIV counseling and testing HIV Human Immunodeficiency Virus HQ Headquarters HR Human resources ICOBI Integrated community-based initiatives IGA Income-generating activity JSI John Snow, Inc. KNH Kindernothilfe M&E Monitoring and evaluation MOU Memorandum of understanding NGO Nongovernmental organization NPI New Partners Initiative NuPITA New Partners Initiative Technical Assistance Project OCA Organizational capacity assessment OD Organizational development OVC Orphans and other vulnerable children iii PEPFAR U.S. President's Emergency Plan for AIDS Relief PLWHA People living with HIV and AIDS PMTCT Prevention of mother-to-child transmission of HIV SJCC St. John’s Community Center SOP Standard operating procedure TA Technical assistance TA-NPI Technical Assistance to the New Partners Initiative TCA Technical capacity assessment TOCA Technical and organizational capacity assessment UNFPA United Nations Population Fund USAID United States Agency for International Development USG United States Government 1 EXECUTIVE SUMMARY The New Partners Initiative Technical Assistance Project (NuPITA) was funded by the United States Agency for International Development (USAID) for the period 2008–2012 and implemented by John Snow, Inc. (JSI) in collaboration with Initiatives Inc. The project was designed to provide technical assistance to 15 organizations in eight countries working in HIV and AIDS that received funding from USAID under the New Partners Initiative (NPI) of the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR). In support of NPI, NuPITA aimed to increase the quality of program implementation and strengthen the institutional capacity of NPI partners, supporting PEPFAR’s objective to improve and expand HIV programming through community- and faith-based organizations (CBOs and FBOs). Its intervention focused on six tasks: 1) provide new partner orientation; 2) conduct needs assessment; 3) develop action plans; 4) provide technical assistance; 5) coordinate and communicate with stakeholders; and 6) support project close-out. The evaluation described in this report sought to show the effects of NuPITA’s capacity-building efforts on HIV service delivery results for partner NPI projects as well as on their overall institutional sustainability, and to link these to capacity-building inputs provided by NuPITA. The evaluation was a cross-sectional descriptive and analytical process employing rapid participatory assessment techniques. A purposive sample of eight of the 15 partner agencies emerged from selection criteria that included geographic accessibility, type and range of services provided, and type of nongovernmental organization (FBO, CBO, and national and international NGOs). Given that at least four of the 15 partners had already lost significant institutional project memory due to staff turnover and/or termination of HIV programming, a ninth partner, American Refugee Committee (ARC), was included as a special case to explore whether systems and service standards supported by NuPITA endured as intended. The evaluation methodology included primary data collection and analysis through key informant interviews with staff from partner organizations, NuPITA, and USAID missions and USAID/Washington; secondary data analysis of organizational capacity assessment (OCA), technical and organizational capacity assessment (TOCA), close-out organizational capacity assessment (CLOCA), and NPI project performance data from nine NuPITA partner organizations during the life of the NPI project; and review of documents and records of NuPITA technical assistance provided to partner organizations. Proxy indicators were identified for assessing both service delivery and sustainability and measured during individual interviews with the sampled partner organizations. This evaluation demonstrated that overall, the provision of technical assistance by NuPITA had significant effects on both the quality of services being delivered by sampled recipients and their progress toward sustainability. The capacity-building inputs offered by NuPITA helped build core strengths in governance, administration, organizational management, human resources management, financial management, project performance management, and program management including monitoring and evaluation (M&E). It was also evident that all support provided to the NPI partners was based upon careful needs assessments and negotiated with each partner. Concerning service delivery indicators, most of the sampled NuPITA partners were able to surpass planned targets for the three-year period of their NPI cooperative agreements, in some cases by a large margin. The number of prime service beneficiaries reached by the nine partners was 305,884; the target 2 was 227,857. Services provided by each partner may have included: prevention (abstinence/be faithful [AB], abstinence, be faithful, use condoms [ABC], and behavior change); prevention of mother-to-child transmission of HIV (PMTCT); confidential HIV counseling and testing (HCT); support to orphans and other vulnerable children; and care and support. All sampled partners reported they were using standard operating procedures (SOPs) and guidelines based on international standards that, in most cases, either did not exist or were not being implemented prior to NuPITA’s input to the NPI project. The evaluation team found that NPI implementing sub￾partners had extended use of SOPs and guidelines created since NPI to their non-NPI projects and programs. All sampled partners reported that use of these technical documents produced better quality services. Based on sampled partner descriptions of past and current approaches to supervision, support supervision quality showed significant improvement and was found to be more structured and better documented. There was widespread improved utilization of project performance data to inform management decisions. Some improvements were also achieved in the functionality of referral networks and linkages with other HIV and AIDS service providers. Six of seven sampled partners implementing activities targeting orphans and other vulnerable children (OVC) had adopted and/or adapted the Child Status Index (CSI) tool to monitor children’s well-being outcomes at individual, household, and project levels. In terms of sustainability, all but one of the sampled partners showed an increased and diversified funding base. Some of the recent funding opportunities consisted of multi-year funding, which was unusual before the NPI project. Most NPI partners developed a resource mobilization strategy and strengthened their links with the USAID mission in their respective countries. This evaluation found that NPI partners managed and complied with USG regulations and requirements due to NuPITA support. It was also noted that improved partner recognition was coupled with increased visibility through strengthened technical networks, including participation in specific policy forums and working groups. Four of eight sampled partners succeeded in hiring new staff in addition to those hired under NPI, and six of eight were found to have a costed strategic plan in place. Partners reported the most beneficial aspects of NuPITA support to be the following: trainings (including M&E, referral networks, and close-out); organizational capacity assessments and technical capacity assessments; tailored, on-site technical assistance; and on-site placement of advisors. Finally, all partners recommended the continuation of technical assistance as a crucial component for any future program involving direct service delivery. 3 CHAPTER ONE: INTRODUCTION 1.1 New Partners Initiative and New Partners Initiative Technical Assistance The New Partners Initiative (NPI), created under the U.S. President’s Emergency Plan for AIDS Relief (PEPFAR), sought to expand the reach of HIV and AIDS prevention and care programs by providing funding opportunities to organizations with an established presence in local communities but no prior experience managing United States government (USG) funds. A key objective of the New Partners Initiative was to build the capacity of these organizations to: 1) manage grants from the USG and comply effectively with USG regulations and requirements; 2) develop their organizations into stronger entities better able to access USG and other funding and continue and/or expand programs after the end of the NPI grant; and 3) strengthen the ability of the organizations to implement high-quality HIV and AIDS programs. To support the selected organizations, the U.S. Agency for International Development (USAID) awarded a four-year contract (2008–2012) to John Snow, Inc. (JSI), assisted by Initiatives Inc. NuPITA served to strengthen the organizational, administrative, and technical capacity of 15 partners providing HIV prevention and care services in Ethiopia, Kenya, Nigeria, Rwanda, South Africa, Tanzania, Uganda, and Zambia. 1.1.1 NuPITA objective and core tasks By the end of the NPI partners’ three-year program period, JSI expected all organizations to have achieved at least 90 percent of their NPI implementation plans and targets, and to be able to clearly document changes and the actions taken to achieve organizational capacity improvements. To support this objective, NuPITA’s activities focused primarily on the following six tasks: • Task 1: Provide new partner orientation • Task 2: Conduct needs assessment • Task 3: Develop action plans • Task 4: Provide technical assistance • Task 5: Coordinate and communicate with stakeholders • Task 6: Support project close-out The NuPITA project team worked in concert with a sister project funded by the U.S. Centers for Disease Control and Prevention (CDC); the Technical Assistance to the New Partners Initiative (TA￾NPI), also implemented by JSI and Initiatives Inc. 4 1.2 Evaluation scope of work 1.2.1 Purpose of the evaluation1 In accordance with USAID’s evaluation policy2 and also as part of the program cycle, an evaluation of the NuPITA project was considered in order to look at the previous phases of the cycle to understand why service delivery and sustainability results were achieved (or why they were not), and to generate information to inform development of similar projects. USAID has initiated a local capacity-building initiative that will be amongst the programs that benefit from the results of this evaluation. Results should also be useful for USAID under its USAID FORWARD Initiative as it seeks to increase the number of awards to local partners, an effort that will, in many cases, require intensive capacity-building efforts. 1.2.2 Evaluation objectives As NuPITA approached close-out, an evaluation was considered in order to determine the effects of the capacity-building efforts on HIV service delivery results of partner projects as well as their overall institutional sustainability, to link these to capacity-building inputs provided by NuPITA, and identify lessons learned in capacity building. The two specific objectives of the evaluation were to: a) Determine the effect that NuPITA’s capacity-building technical assistance has had on HIV service delivery results through the work of its partner organizations; and b) Determine the effect that NuPITA’s capacity-building technical assistance has had on the sustainability of its partner organizations. 1.2.3 Evaluation questions The following questions guided the evaluation team during their investigation: a) Have NuPITA partners changed their HIV service delivery approaches over the life of the project? If so, how and what effect has this had on service delivery results? b) Have these changes, if any, to partners’ HIV service delivery approaches been influenced by capacity￾building inputs provided by NuPITA? If so, how? c) Have NuPITA partners experienced changes in their overall institutional sustainability (as measured by the type and quality of linkages with stakeholders and type and quality of systems that allow them to better access funding or to address funding gaps) since the start of their NPI cooperative agreements? If so, what are these changes? d) Have these changes in sustainability been influenced by capacity-building inputs provided by NuPITA? If so, how? e) What factors have contributed to NuPITA achieving or not achieving its intended results? 1 Evaluation is the systematic collection and analysis of information about the characteristics and outcome of programs and projects to improve effectiveness and/or inform decisions about current and future programming. 2 As prescribed in the Automated Directive System (ADS) 203.3.5.5 (revised February 1, 2012). 5 1.2.4 Evaluation team and duration A multidisciplinary team, including Ms. Joan Haffey (team leader and organizational development [OD] specialist), Dr. Luigi Cicciò (team member and public health specialist), and Dr. Daniel Kibuuka Musoke (team member and M&E specialist), conducted the evaluation between May 24 and June 30, 2012. The first week involved the finalization of the methodology and development of data collection tools. The second through the fourth weeks were used to collect information from the different respondents, and the last two weeks were used for data analysis and report writing. 1.3 Organizational Capacity Assessments Because their results were used to frame the evaluation, it is useful to understand how organizational capacity assessments (OCA) were used in the NuPITA project. For each partner, support started—at both the headquarters (HQ) and field levels—with an analysis of current capability measured through an OCA and conducted in partnership with the implementing organization. The OCA used an interactive tool to help partner organizations identify their status on seven management elements: governance, administration, human resource management, financial management, organizational management, program management, and project performance management. The tool measured organizational strengths and challenges and ability to comply with USG administrative, financial, and programmatic regulations and procedures. The outcome was a quantitative baseline to guide planning to strengthen those systems and procedures. An accompanying action plan also pinpointed key areas in which organizations requested technical assistance from the NuPITA project. The OCA was repeated annually for the three years of the NPI project to track progress over time. Although the tool is self￾administered and not meant to measure precise improvement levels, the OCA was useful in monitoring overall the progress that partners made in developing the skills, systems, and documentation needed to sustain high-quality HIV prevention and care programs and plan for future improvements. After the first round of OCAs, it was clear that for all partners the weakest program-related components were program management and project performance management, as illustrated in Figure 1 below. (See OCA program-related result trends by sampled partner in Annex E.) 6 Figure 1: Aggregated Average Scores for Program Management, Project Performance Management, and Average Organizational Capacity In response to the technical needs identified during the initial OCAs, NuPITA developed the technical capacity assessment (TCA) to build on the OCA. The TCA documents the organization’s technical capacity to implement HIV and AIDS programs and comply with national and international HIV and AIDS standards, and also identifies key technical areas within the organization that require strengthening. The TCA is designed to assess five thematic areas: 1) comprehensive prevention; 2) HIV counselling and testing (HCT); 3) prevention of mother-to-child transmission of HIV (PMTCT); 4) care and support, and; 5) orphans and other vulnerable children (OVC). Each tool was further subdivided into relevant domains. The combined OCA and TCA, or TOCA, was implemented with each partner in its second project year. In the final project year, a close-out OCA, or CLOCA, was implemented using a slightly modified OCA tool that included a discussion of partners’ experiences under NPI. Due to time constraints, USAID and NuPITA decided to offer a second TCA on an optional basis. 1.3.1 Major gaps identified during the TOCA Gaps in technical capacity of the NPI partners were identified during the TOCA. Among all domains, gaps across partners were most evident in the following key components of technical capacity: Trend in average organizational capacity score vs. program-related domains' scores 0 1 2 3 4 YEAR 1 YEAR 2 YEAR 3 Program Management Project Performance Management Average Organizational Capacity Score 7 • Prevention: service standards, supervision, guidelines and standard operating procedures (SOPs) • OVC: guidelines and SOPs, service standards, referral systems • Data collection systems • General use of data for decision making • HCT: service standards 1.3.2 Key NuPITA inputs provided to address identified gaps Most of the sampled NPI partners reported to have received the following forms of support to enhance their organizational and technical capacity to successfully implement the NPI program. Human resource (HR) management The NuPITA project assessed the human resource capacity and needs of NPI partners through the OCAs and on-site visits and supported them to develop plans for addressing HR gaps. The following strategies were recommended in their entirety or in part to the NPI partners in order to improve HR management: systems for staff training, job analysis and job description development, performance appraisal systems, retention strategies such as incentives, and staff recruitment and orientation. Financial management All NPI partners were supported to strengthen accounting procedures, establish financial controls, and develop audit plans and processes. Effective reporting All NPI partners were supported to fulfill their reporting requirements through clarifying them under the terms of their cooperative agreement and with PEPFAR, and by helping them to establish systems for regular, effective reporting. Program management and strategic planning systems NuPITA provided support to all NPI partners to develop and track annual workplans for program management. Building M&E systems NuPITA supported all its NPI partners to monitor and evaluate their projects effectively. Using both long- and short-term consultants, partners were supported to develop M&E plans and systems and to improve data management, collection, entry, and analysis. All partners were advised on how to use the PEPFAR M&E strengthening tools to ensure that NPI partners’ programs have the appropriate systems to respond to national, USAID, and PEPFAR requirements. As a cross-cutting element, all partners were engaged in various group and individual trainings and on￾site mentoring and support using short-term experts from within NuPITA and external consultants. The NuPITA project also provided technical assistance to all NPI partners to strengthen the quality and effectiveness of their HIV and AIDS interventions, specifically in the following intervention areas; prevention (including behavior change communication), PMTCT, confidential HCT, OVC, and care and support. 8 Prevention (AB/ABC and behavior change) All partners received customized support in order to create prevention interventions that are based on local context as well as scientific evidence. Technical support was provided to develop HIV prevention messages, tools, and skills to reduce HIV risk and also to improve communication to reinforce practice of key preventive behaviors. Interventions followed strategic behavior change communication (BCC) and USAID’s ABC prevention guidelines. Prevention of mother-to-child transmission of HIV (PMTCT) The NuPITA project supported most of the NPI partners to develop initiatives that generate demand for and increase utilization of services, including greater support for pregnant women living with HIV. Technical support was provided to develop and expand context-appropriate, high-quality PMTCT services. Confidential HIV counseling and testing (HCT) Most partners were supported to tailor HCT services to individual, family, and community needs, with links to treatment and support services. Emphasis was placed on strategies to increase the number of people who know their HIV status, and to facilitate links to appropriate prevention, treatment, and care services. Orphans and other vulnerable children Technical support was provided to expand the scope of services offered. NPI partners who served OVC were supported to design and implement activities that strengthened life skills, improved livelihoods, and addressed stigma, discrimination, and other forms of marginalization. Care and support The NuPITA project supported most of the NPI partners to strengthen the implementation of multisectoral interventions that addressed health and well-being and ensured equitable access and involvement of people living with HIV and AIDS (PLWHA). These activities reflected the approaches in national palliative care strategies. As a cross-cutting element for all partners that were supported in HIV prevention, PMTCT, HCT, OVC, and care and support, technical assistance focused on the following: • Development of SOPs • Verification that service delivery was carried out in compliance with existing national guidelines and standards • Improvement in the quality of services offered • Strengthened support supervision to make it more structured and documented • Strengthened referral systems through use of referral directories and other essential elements of networked services (referral slip/form, network coordination, and feedback on referred clients) • Training in knowledge and skills needed for better service delivery. Different forms of training used included group trainings, individual trainings, and on-site mentoring and support using short-term experts and resident NPI advisors • Monitoring and evaluation systems strengthening • Assistance with the recruitment process to identify professional staff 9 CHAPTER TWO: METHODOLOGY 2.1 Evaluation design This evaluation used multiple evidence sources to obtain a comprehensive and in-depth understanding of NuPITA’s complex and diverse assistance, control the errors implicit in any chosen research method, support sound analyses, arrive at practical conclusions, and make accurate inferences. This was a cross￾sectional descriptive and analytical process and outcomes-based evaluation employing rapid participatory assessment techniques. The methodology for the evaluation combined secondary data analysis of HIV data from NuPITA partner organizations during the life of the NPI cooperative agreements on various implementing areas (such as the number of OVC reached, the number of people infected or affected by HIV who received care and support services, etc.); document review (annual workplans and reports, reports from T/OCAs, mid-term client satisfaction survey); review of the 2011 end-of-project partner survey results, records of NuPITA technical assistance provided to organizations; and primary data collection and analysis through key informant interviews with partner organization, NuPITA, and USAID Washington and mission staff. 2.2 Study sites and population The 15 organizations that were supported by NPI are diverse in size, geographic representation, technical scope, and organizational experience. They represent experienced international and US-based organizations with a global reach through a variety of country programs, such as the American Refugee Committee International (ARC); large faith-based organizations with partners across the developing world, such as Tearfund, UK, and Ananda Marga Universal Relief Team (AMURT); and locally-focused CBOs and FBOs based in Africa, such as integrated community-based initiatives (ICOBI). Therefore, multiple criteria were used to select nine NGOs that fairly represented the unique and diverse characteristics of all the supported NGOs. Selection criteria included the following: being indigenous, international, and/or faith-based NGO; easy access to the project sites; broad range of services; unique menu of services (such as an OVC community development program, support to street children, or provision of early childhood education) and limited project memory and partnerships with community NGOs. Table 1 shows the specific selection criteria for each of the nine NGOs that were visited by the evaluation team.3 3 The NuPITA performance monitoring system did not compare performance among the partner NGOs during the entire project period. The evaluation did not consider performance in its selection criteria. 10 Table 1: Selection Criteria for the Sampled Partner NGOs SN Partner Round Type Country Selection criteria 1 ARC 2 INGO Uganda (Gulu) Limited project memory 2 ICOBI 2 NGO Uganda (Mbarara, Bushyeni) Included as the only indigenous NGO. Field area relatively accessible. 3 Woord en Daad/Mfesane 2 INGO/FBO South Africa (Cape Town) Included for broad range of service implementation. 4 AMURT 3 INGO/FBO Kenya (Nyanza, Central, Coast) Included as representative of FBOs. Country office in Nairobi. 5 FXB 3 INGO Uganda (Kyenjojo, Wakiso, Bundibugyo), Rwanda (Gitarama, Gikongoro, Gisenyi) Included for the component based in Uganda focusing on OVC/community development program. 6 GRACE 3 INGO Kenya (Nyanza, Central, Eastern) Included as project on OVC & early childhood education; HQ based in Nairobi. 7 KNH/SJCC 3 INGO/NGO/FBO Kenya (Western, Nyanza, Central, Rift Valley, Nairobi) International organization partnering with a local NGO; broad scope of services offered. 8 Retrak 3 INGO/FBO Uganda (Kampala), Ethiopia (Addis Ababa) International FBO working with street children in Kampala and Addis Ababa; Kampala selected for ease of access. 9 WellShare 3 INGO Uganda (Mubende, Ssembabule) International NGO and easy access to country office in Kampala. 11 A number of partners lost significant project memory due to staff turnover (American Refugee Committee [ARC], Camfed); termination of the agency’s HIV program (ARC); or because they had closed the office and/or their country program (Children’s Emergency Relief International [CERI], Tearfund Kenya), when this evaluation was conducted. As a special case, the evaluation sample therefore included one partner—ARC—that had lost staff and closed HIV programming at the end of the NPI project. This helped the evaluation team understand if at least one NPI partner without a continuing HIV and AIDS program and with a new senior management team maintained the practices, procedures, and systems that resulted from the NuPITA support beyond the cooperative agreement period. Throughout the report, the sample size of ‘8’ refers to all sampled NGOs except ARC, and a sample size of ‘9’, when specifically mentioned, includes ARC. Additionally, for the international NGOs selected, the evaluation team focused on the host country programs in order to determine the effect of capacity building on service delivery results and institutional sustainability at the level of project implementation. The major evaluation respondents were managerial, technical, administrative, and financial staff of the NPI partners. Complementary informants included NuPITA staff, including NPI advisors, and USAID staff both at the local missions and in Washington, D.C. 2.3 Data collection methods The evaluation comprised a mix of qualitative and quantitative data collection methods. The quantitative data was obtained from the desk review of key project documents including service statistics. The evaluation team used participatory approaches to collect and analyse data, draw conclusions, and make recommendations. The following qualitative methods of data collection were used during the evaluation. 2.3.1 Document Review The evaluation team reviewed different project documents (Annex H) and summarized findings in order to obtain answers to key evaluation objectives as shown in the evaluation framework (Annex A). Any unclear sections were noted and followed up by NuPITA staff. 2.3.2 Key informant interviews Key informant interviews were conducted to gain a deeper understanding of the context for the project achievements and challenges. Key informants were chosen for their expert knowledge of the subject being explored. They were interviewed with the main purpose of understanding their views and opinions on the support provided by NuPITA and how it translated into improving the service delivery results and institutional sustainability of the partner organization. For each interview, at least two consultants participated, with one taking the lead on asking relevant questions while the other took notes, in order to minimize the duration of the interview. One consultant also took responsibility for reviewing and making a summary of key OCA, TOCA, and CLOCA results for each sampled partner and reviewing with other evaluators prior to each interview. Interviews were audio-taped (with permission) to complement the note-taking. Interviews were conducted using the guides included in Annex I. In some cases, scheduling mandated that one consultant conducted the interview and took the notes and was supported by the audio recording of the interview. 12 2.4 Quality assurance, analysis, and presentation After each day of data collection, the team reviewed, edited, cleaned, and summarized key findings. Missing information or inconsistencies were identified and followed up accordingly. Textual data in interview transcripts were explored using content analysis. The analysts read and re-read data transcripts in order to identify emerging themes. All data relevant to each theme were identified and examined using the process of constant comparison, in which each item is checked or compared with the rest of the data in order to establish analytical categories. 2.5 Ethical considerations The evaluation team explained the goals and objectives of the evaluation and confidentiality safeguards to all interviewees and obtained informed consent from all. Furthermore, the team provided assurance that it would utilize the information for the study’s purposes only. No names of individual informants were used in this report without consent. 2.6 Evaluation limitations The project was characterised by a range of interventions and was implemented in several countries in different parts of Africa. Because the time allocated to this evaluation was relatively short, the review team could not cover every aspect of the project, including visiting all NPI partner NGOs. Additionally, service delivery data prior to NuPITA was not available for the majority of the organizations due to their having adopted new programming areas and lack of systematic documentation. Therefore the evaluation team could not compare the situations or performance of the NGOs before and during the NuPITA project. Due to time constraints, the team could not use project documentation to explore possible links between organizational gaps, specific NuPITA inputs, and outcomes. They did, however, discuss these relationships during key informant interviews. Although respondents were able to report generally which NuPITA inputs they most valued, they were not make specific links to gaps or outcomes. In spite of the above limitations, the evaluation team believes that the information obtained through the interviews, document reviews, and field visits, was sufficient to give a balanced assessment of the project. 13 CHAPTER THREE: FINDINGS 3.1 Effects of capacity building on service delivery results The first evaluation objective is consistent with the overall goal and objective of the NuPITA project. The project goal was to increase the quality of program implementation and to strengthen the institutional capacity of NPI partners. The project objective was to have all supported NPI partners achieve at least 90 percent of their implementation plans and PEPFAR targets, and to be able to clearly document changes and the actions taken to achieve organizational capacity improvements. (Please refer to Figure 2 and Annex E for more information on achievements against targets.) In order to determine the effect of NuPITA’s capacity building on the HIV service delivery results of the partner organization, the evaluation team assessed: • The extent to which the NuPITA partners have changed their HIV service delivery approaches over the life of the project, and the effect it has had on their service delivery results. • The extent to which the changes in the NuPITA partners’ service delivery approaches have been influenced by the capacity-building inputs provided by NuPITA. The extent to which the NuPITA partners have changed their service delivery approaches was assessed through the following indicators: • Major gaps identified during the T/OCAs and key inputs offered to address identified gaps • Institutionalization of internationally recognized standards and procedures • Changes in the quality of support supervision • Use of program data to monitor service outcomes and inform management decisions • Changes in the functionality of referral systems/linkages • Use of the Child Status Index • Changes in scope and approach in service delivery • Changes in number of beneficiaries reached Table 2 summarizes the major findings for each sampled NPI partner on service delivery-related indicators and provides a snapshot of performance. More details on the findings in the summary table can be found in the narrative and in Annex E: Individual Partner Reports. 3.1.1 NPI partners’ views about the usefulness of NuPITA’s capacity-building inputs “Our [country program] has been transformed from a traditionally managed organization to a modern one, thanks to NuPITA support… We never dreamt we could get [EU funding]. We are competing with very big international organizations and…are being grouped with these giants now…We built these systems through NuPITA, which has taken us several levels up. We are poised to become an international giant.” –Key informant interview respondent, Kenya 14 During the partner satisfaction survey and the evaluation, most sampled NPI partners reported the following capacity-building inputs as the most useful and beneficial to the success of their projects: Specific trainings Most of the sampled NPI partners reported that NuPITA trainings greatly enhanced their knowledge and skills and helped in building overall organizational capacity. The trainings also helped to strengthen the organizational capacity for resource mobilization and compliance with USAID and PEPFAR regulations and reporting requirements. They also allowed partners to share experiences. AMURT reported that the NuPITA trainings helped them upgrade the quality of the organizational human resources, financial and M&E systems, and in networking with other organizations to learn about their management and programmatic practices. OCA/TOCA/CLOCAs All partners noted that OCA/TOCA/CLOCAs were key in advancing their organizational growth. They provided structure and helped identify needs and improve capacity. All sampled partners reported steady improvements in their projects after implementing activities to address the identified gaps. Those with sub-partners were able to help them address their capacity problems in a more systematic manner. On-site technical advisors All the sampled NPI partners reported that the on-site technical advisors helped them address partner￾specific problems, improve M&E systems, improve training and facilitation methods, and provide M&E visits and assessments. All partners with NPI advisors reported that they were excellent technical references who provided invaluable customized support to project M&E, documentation, compliance, HR. and resource mobilization. 3.1.2 Changes in number of beneficiaries reached “I can tell you that we are a totally new organization now [because of NuPITA support]. We adopted better approaches to service delivery (especially in behavior change communication), expanded geographical scope from two to six South African municipalities, and also expanded target groups to also include [populations at risk]. As if that is not enough, some of our services are reaching people with disabilities.” — Key informant interview respondent, South Africa The evaluation showed that all sampled NPI partners had a positive change for all the PEPFAR indicators. There was a significant increase in the number of beneficiaries reached with key services. Overall, the total number of primary beneficiaries reached for all nine sampled partners was 305,884 (compared to a target of 227,857), with a variance4 of 34 percent. Figure 2 shows that, as a whole, the sampled NPI partners who were provided with technical assistance (TA) by NuPITA exceeded their targets, with variances ranging from 13 to 82 percent. 4 LOP Achieved-LOP Target]/LOP Target 15 Table 2: Summary of Findings on Service Delivery-Related Indicators for Each NPI Partner* * Tables 2 and 3 provide a snapshot of partner achievements related to service delivery and sustainability. For more detailed information, refer to the narrative and Annex E: Individual Partner Reports. Retrak WellShare FXB ICOBI SJCC GRACE Mfesane AMURT Scope/range Scope and coverage now include life skills activities for young boys No change Prevention expanded to include segmented messages for AB/ABC and home￾to-home approach for reaching out of school youth Scope expanded to include health/medical and better approaches PS support New approach: CHWs disseminate HIV prevention messages, introduced expanded community fostership Changed from mass campaigns to small groups and repetitive sessions Adopted better approaches and scope expanded Expanded to include HIV prevention, IGAs, PMTCT, vulnerable groups, HCT # of beneficiaries Surpassed most of their targets Surpassed most of their targets Surpassed most of their targets Surpassed most of their targets Half of the targets surpassed Surpassed targets for PEPFAR indicators and underachieved on others Surpassed most of their targets Surpassed most of their targets 16 SOPs & guidelines Improved SOPs and guidelines and increased use Improved and increased use Improved and increased use Improved and increased use & transferred to sub￾partners Improved and increased use & transferred to sub-partners Improved and increased use Improved and increased use Improved and increased use Support/ Supervision Supervision tools and guidelines now in place and used Supervision tools and guidelines now in place and used Supervision tools and guidelines now in place and used Supervision tools and guidelines now in place and used Supervision tools and guidelines now in place and used Supervision tools and guidelines now in place and used Supervision tools and guidelines now in place and used Supervision tools and guidelines now in place and used Referral system More functional referral network (with MOUs and many collaborating partners) Now have and disseminated referral directory More functional referral network (referral directories and referral slips in place and used) More functional referral network (referral directories and referral slips in place and used) More functional referral network with tools in place and referral processes being institutionalized Limited functionality due to lack of funding More functional referral network with tools in place and referral processes being institutionalized More functional referral network (with MOUs and many collaborating partners) Data use for decision making Increased use of data to inform management decisions Increased use of data to inform management decisions Increased use of data to inform management decisions Increased use of data to inform management decisions Increased use of data to inform management decisions Increased use of data to inform management decisions Increased use of data to inform management decisions Increased use of data to inform management decisions 17 CSI Was introduced and used at household and individual level to direct support Was introduced and used at household and individual level to direct support Was introduced and used at household and individual level to direct support Was introduced and used at household and individual level to direct support Attended training but the CSI not yet used – although there is a plan to roll them out Was introduced and used at household and individual level to direct support Was introduced and used at household and individual level to direct support Not applicable 18 Figure 2: Comparison of Targets and Results for PEPFAR Indicators The evaluation found evidence that the sampled NPI partners were able to surpass their targets because of changes in scope and approaches in service delivery—which were made possible because of the technical support provided by NuPITA—as described in subsequent sections of the evaluation report. In general, results presented in this report emphasize increased numbers of beneficiaries reached by project services, perhaps the most relevant achievements by NPI partners. Analysis of proxy indicators related to service quality and scope and organizational sustainability is included in the discussion and summarized in Tables 2 (services) and 3 (sustainability) as well as Annex E (Individual Partner Reports). 19 3.1.3 Changes in scope and approach in service delivery “Before NPI, we were focusing purely on treatment, now we support OVC care, HIV prevention, economic strengthening [income generation]…We have added PMTCT, which is now a major project for us. We are also now targeting [at risk populations].” — Key informant interview respondent, Kenya The evaluation showed that seven of the eight sampled NPI partners changed the scope and approaches they were using in service delivery as a result of the technical assistance received from NuPITA. For example, the staff of Mfesane, South Africa now believe they are running “a totally new organization.” The organization adopted better approaches in service delivery, especially in BCC, expanded geographical scope from two to six South African municipalities, and expanded target groups to include vulnerable groups and people with disabilities. ICOBI in Uganda has been able to adopt better approaches in psychosocial support; the organization was able to organize an exchange visit with the National Community of Women Living with HIV and AIDS and adopted a ‘kids’ club’ approach for psychosocial support of vulnerable children. Initially the organization focused on supporting OVC education but NuPITA’s technical support helped them expand services to the provision of medical services and block payment of tuition fees. FXB/Uganda was able to include child participation in the menu of services using participatory tools provided by NuPITA. As a result, child protection has now become a core component of their work. 3.1.4 Institutionalization of internationally recognized standards and procedures All the sampled NPI partners reported that the support received from the NuPITA project led to a significant professionalization in their methods of work. Under NuPITA, most had developed and, at the time of the evaluation, were using the following SOPs and guidelines: project implementation manuals; support supervision plans; implementation guides for key activities such as children’s psychosocial clubs; selection guidelines for apprentice service providers; project performance monitoring plans; data flow charts; reporting formats; customized project quality standards for different project activities; timesheets; performance appraisals; asset inventories; and branding guidelines. All the SOPs and guidelines were found, either through key informant interviews or demonstration, to be documented. NPI partners even used them with their implementing sub-partners and in other non-NPI projects. The use of SOPs and guidelines was reported to have translated to better quality of services offered to the project beneficiaries. 20 3.1.5 Changes in the quality of support supervision “Our support supervision to CBOs is now better organized and more systematic. Reports are compiled, the approach is more collaborative and participative than before, when a more directive policing attitude was used.” — Key informant interview respondent, Uganda All nine sampled NPI partners reported significant improvements in the quality of support supervision that they are conducting. By the time of the evaluation, all nine reported that they had developed comprehensive periodic support supervision reports that clearly identify issues to be addressed and evidence of actions taken to address them. All these improvements were associated with the assistance received in developing a support supervision strategy and tools, in addition to the trainings that they attended. NuPITA provided a range of group and organization-specific trainings and addressed the gamut of managerial and technical needs of high-quality HIV and AIDS programming. As just one example of how trainings and direct technical assistance reinforced each other, elements of support supervision were contained in the following illustrative trainings (in addition to training on support supervision itself): Quality Assurance and Quality Improvement; Team Building and Communication; Change Management; Performance Appraisal; and Building Effective Referral Systems. 3.1.6 Use of program data to monitor service outcomes and inform management decisions All the sampled NPI partners reported significant improvements in the use of project performance data to inform management decisions. This is associated with support received from NuPITA in the form of M&E trainings, use of consultants and resident advisors who mentored the NPI partners to develop M&E frameworks/plans, reporting formats, and alignment of reporting tools to performance indicators. Some of the NPI partners reported conducting monthly monitoring and quarterly and annual reviews of their program and service delivery data, which were not done before the NPI cooperative agreements. These reviews allowed them to change implementation strategies when some interventions were under￾achieving. 21 3.1.7 Use of the Child Status Index “Monitoring and evaluation was not in our culture before NuPITA, but now it is second nature. NuPITA helped us develop an M&E plan and now we have data collection tools and conduct regular review meetings to assess progress…[For instance,] we realized that very few people were using our home-based HIV counseling services because of stigma and decided that instead of testing for HIV only, we would offer a cocktail of tests (including blood sugar, malaria, etc.) as a way of reducing the stigma and encouraging people to take up the HIV testing that was later offered as part of the cocktail.” — Key informant interview respondent, South Africa All the NPI partners who were serving individual OVC were trained and provided with the CSI tool for use in monitoring the well-being of OVC and their households. All NPI partners who had started using the CSI tool reported that it was very useful, especially to: a) Identify the specific needs of a child and his/her household and to translate these needs into intervention strategies. ICOBI in Uganda was able to use the CSI tool to identify service delivery gaps among served children. As a result the organization was able to direct interventions to address their needs. b) Monitor staff problems and benefits in efforts to serve children. Additionally, the tool is now being used to advocate for resources and improvements in service quality. The staff of WellShare International reported to have acquired skills and competence to assess vulnerabilities of children and their households following the CSI trainings. They also reported using analyzed CSI data to assess benefits of their activities and advocate for more resources from UNICEF and the Uganda Civil Society Fund. c) Raise awareness among frontline staff (such as community health workers and caregivers) about the multiple dimensions of child well-being to help them understand and address these areas routinely in their work. The staff of Mfesane in South Africa reported that all community health volunteers oriented in the use of the CSI consider it to be a useful tool that constantly reminds them of the multiple dimensions of child well-being that should be routinely assessed. Although Retrak, ICOBI, GRACE, and FXB all collected electronic CSI data, it would have been helpful to more systematically use a CSI database for internal monitoring, analysis of partner use of the tool, and external dissemination of results. 22 3.1.8 Changes in the functionality of referral systems/linkages Most (6 of 8) of the sampled NPI partners reported improvement in the functionality5 of their internal referral networks/linkages and with external service providers to support delivery of comprehensive HIV and AIDS services. This ranged from basic awareness of a need to develop a referral system (GRACE and ARC) to documenting complete referrals (Mfesane, Retrak, FXB, AMURT). At a minimum, there is a referral directory of service providers that was disseminated to network members. Where fully functional referral systems were reported, a referral slip is used whenever clients are referred for a needed service and a feedback loop exists, where a referred client reports back to the referring organization about services received. Only one of the nine sampled partners still experiences limited functionality of referral activities in spite of the support from NuPITA. This group reported lack of a referral directory, referral documentation, and feedback on referred clients and coordination meetings for network members. In addition, a number of partners noted that their clients were expected to pay for referral services—a serious barrier to use—making the referral link ineffective. 3.2 Effects of capacity building on the sustainability of NPI partner organizations The team specifically explored whether NuPITA partners experienced changes in their overall institutional sustainability since the start of their NPI cooperative agreements, and whether those changes had been influenced by capacity-building inputs provided by NuPITA. Similar to the first evaluation objective, this objective was consistent with the overall goal of NuPITA to increase the quality of program implementation and strengthen the institutional capacity of NPI partners. ‘Institutional sustainability’ can be defined as the measure of an organization’s ability to fulfill its mission and serve its stakeholders over time.6 Achieving institutional sustainability is a goal of all organizations. A further distinction was made between financial and programmatic sustainability, as defined by the evaluation team based on its review of NuPITA program documents. ‘Financial sustainability’ was defined as the ability to secure the resources from reliable and diverse sources necessary to achieve organizational goals. ‘Programmatic sustainability’ was defined as the ability of the organization to secure and manage sufficient resources (not necessarily limited to financial resources) to effectively and consistently deliver services or sustain benefits after the donor's technical, managerial, and financial support has significantly decreased or ended. Various indicators were identified and chosen to determine any change or progress in NPI partners’ institutional sustainability. These indicators provided a quantifiable measure in specific areas of sustainability and allowed for comparison over time and across similar institutions. In the area of financial sustainability, the main indicators selected were: 5 Functionality of referral linkages for networked services was assessed based on the presence of established critical elements of networked services such as: MOU with other service providers, use of referral forms/slips, presence of a coordinating organization, organization of meetings of network members, and presence of a feedback loop. 6 Hal Swerissen. June 2007. Understanding the Sustainability of Health Programs and Organisational Change. Melbourne:La Trove University. 23 • Existence of different funding sources than USAID • Existence of different collaborating partners • Existence and use of a costed strategic plan • Existence and use of a resource mobilization strategy Indicators to assess programmatic sustainability were the following: • Existence of networks/linkages • Existence of and adherence to organizational policies, procedures, and practices • Retention of trained and skilled human resources relevant to the services offered • Existence of a plan to sustain critical activities beyond the life of the project A brief summary of the main findings by those proxy indicators included in the interview checklist disaggregated by NPI organization is displayed in Table 2. (See Annex I for interview guides.) Figure 3: Aggregated Average Scores for Selected Sustainability Sub-Domains “I am what I am because of NuPITA, I have acquired a lot of knowledge and skills from the NuPITA trainings and right now I think I can manage any HIV project.” 24 A vital component of financial sustainability is funding diversification, referring to the number of sources that provide the organization’s financial resources. All but one of the NPI partners assessed during this study showed that their funding base had increased with more sources contributing to it – some of which had not been accessible to them before the NPI project and the technical assistance received by NuPITA. Some of these new funding opportunities consisted of multi-year funding, which was reported as a significant achievement for organizations used to receiving funding for just one year. Remarkably, six of the eight sampled organizations whose information was comprehensively collected during the evaluation received additional funding from USAID either as prime beneficiaries or sub-grantees; while Mfesane received grants from different South Africa government departments, namely health, education, and social development. To put it in a respondent’s words, “we have almost become irresistible to local donors.” One of the factors widely reported to have played a pivotal role in this funding diversification—though not directly and objectively verified by the evaluation team—was having structured and sound systems of financial, administrative, and program procedures and policies in place. Documentation of management systems and efficient procedures for administration and finances were largely recognized as keys to sustainability. This was deemed to have been a strong indicator of the overall organization’s competence and strength, which in turn attracted new donors and qualified the NPI partners for further and diversified funding. As stated by one of the respondents, “donors are very happy with our systems and that is all because of NuPITA.” The case of Mfesane is illustrative because they were asked by the South African government to share documentation of their financial systems and job descriptions with other organizations and public-sector entities, a validation of the government’s strong recognition and appreciation of their administrative capabilities. This recognition translated directly into the government’s decision to issue seven grants to Mfesane, according to interviews with Mfesane representatives. All sampled organizations were concerned about the need to diversify their funding base and were aware that an adequate resource mobilization plan is an essential requirement for this. The evaluation team found that six of the eight organizations established measures to address resource mobilization, with a seventh under development. In addition, three of the five NPI partner INGOs had decentralized at least partial responsibility for fund raising from headquarters to the country offices, as part of their NuPITA-supported resource mobilization strategy. NuPITA helped these groups register as local NGOs and all had either received or were being considered for local grants for which they had previously been ineligible. Local fundraising had significantly improved for four of the organizations. At FXB, for instance, all staff now know that they are responsible for the whole resource mobilization process, including the identification of partners and linkages. Some respondents expressed satisfaction at having developed a considerable degree of confidence and ability when going out to market their projects in different arenas and when talking to high-profile donors. A quote from one of the respondents demonstrates that improved confidence. “We have knowledge, confidence, and data. We can contribute to the discussion now.” Additionally, NuPITA-supported documentation of each partner’s NPI project achievements, in the form of printed success story booklets, DVDs, or technical briefs helped organizations disseminate their results and, eventually, its marketability and attractiveness to donor agencies. This was seen as a result of the specific NuPITA-led training on documentation strategies and how to showcase project successes (“a wonderful, wonderful input,” as one respondent defined it). 25 As part of this striving for visibility and in line with NuPITA’s advice, many of the NPI partners used various avenues to strengthen their links with the USAID mission in their respective countries. Three partners reported having held meetings at their mission office to share their project’s successes. AMURT now participates regularly in chief-of-party breakfast meetings at the mission. On World AIDS Day, the USAID mission in Uganda included an account of FXB’s achievements in the Embassy’s electronic newsletter. At least two interviewees mentioned (not asked as an interview question) that field sites had received visits from both USAID and CDC officials to cultivate relations. Respondents largely credited improved relations with USAID, a donor agency previously seen as inaccessible or “intimidating,” to NuPITA’s efforts to facilitate presentations and interactions with the missions. ICOBI, the only local organization featured in the evaluation sample, reported never having been invited to a technical meeting prior to their partnership with NuPITA. By the time their three-year project ended, they were part of the forum for developing the National HIV and AIDS Strategic Plan II, a member of the Regional AIDS Training Network, and had become the regional reference organization on home-to-home HIV counseling and testing. Participation in technical working groups and other fora significantly improved the visibility of NPI partners that do so. For example, Retrak Uganda took part in the national process for the development of a National Strategic Programme Plan of Interventions for OVC, giving this small, independent NGO greater involvement and credibility on the national scale. An additional observation on improved linkages was the formation of a “group of five” NPI partners in Kenya in a quest to share information on forthcoming projects and grant opportunities. These five organizations, previously supported by NuPITA or TA-NPI, are working to submit a joint proposal to the European Union, demonstrating—as reported by sampled NuPITA partners—improved ability to partner with other agencies. The presence of a costed strategic plan (or of a financial or business plan) was another indicator of financial sustainability assessed during the evaluation. Strategic planning is the mechanism that organizations use to clarify their mission, objectives, and the activities needed to accomplish them. Six of the eight responding organizations reported having a strategic plan in place, while another was in the process of developing it. The development of strategic plans was largely attained through the inputs received from NuPITA, including individualized trainings on strategic planning and targeted follow-up support, and in most cases plans were found to be linked to a strategy to sustain current programmatic activities beyond the life of the NPI agreement. 26 Table 3: Summary of Findings on Sustainability-Related Indicators Disaggregated by NPI Organization*∗∗ Indicators/Organizations Retrak WellShare FXB ICOBI SJCC GRACE Mfesane AMURT Funding sources Increased and diversified (USAID incl.) More USAID funding More USAID funding More USAID funding Increased and diversified (USAID incl.) Most program activities came to an end Increased and diversified (Gov’t incl.) Increased and diversified (USAID incl.) Collaborating partners Increased and diversified Increased and diversified Increased and diversified Increased and diversified Increased and diversified Stagnant Increased and diversified Increased and diversified Strategic plan Under development No strategic plan in place Have a costed strategic plan Have a costed strategic plan Have a costed strategic plan Have a costed strategic plan Have a costed strategic plan Have a costed strategic plan Resource mobilization Fund raising decentralized to field office Fund raising decentralized to field office Fund raising partially decentralized to field office Resource mobilization committee in place Resource mobilization strategy developed Fundraising committee in place Resource mobilization strategy developed Resource mobilization strategy developed Linkages Improved, more visible, in many TWG Improved, more visible, in many TWG Improved, more visible, in many TWG Participating in several forums Improved, more visible, in many TWG Limited Improved, more visible, in many TWG Improved Standards and procedures system All in place, made them marketable All in place, made them marketable Many policies & procedures improved Many policies & procedures improved All in place, made them marketable Systems in place, not enough for more funding Gov’t asked to share their systems with other NGOs & public entities All in place, made them marketable Skilled HR Retained and hired more staff Retained most staff; plan to hire more Field staff laid off, managers retained Retained key staff and hired more Retained and hired more Laid off some and retained others Retained most staff and hired NPI advisor Retained all key staff; hired NPI advisor *∗∗ Tables 2 and 3 provide a snapshot of partner achievements related to service delivery and sustainability. For more detailed information, refer to the narrative and Annex E: Individual Partner Reports. 27 Indicators/Organizations Retrak WellShare FXB ICOBI SJCC GRACE Mfesane AMURT Sustaining activities Activities sustained with new funding Through districts and CBOs Improved Developed a sustainability plan for every project Developed a sustainability plan Trying strategy for long-term funding Developed a sustainability plan for HBC activities Developed a sustainability plan 28 Another critical area for sustaining activities is the availability of adequately trained human resources staff, who are vital for maintaining the intended intervention at the required level of service quality. Five of the eight sampled organizations retained their core staff at the end of the NPI project, while one had to lay off the field staff but was able to keep its managerial staff. Remarkably, four of the responding organizations reported having hired new staff, and another was planning to do so if funds allowed. In the case of Mfesane and AMURT, among the newly hired staff were the NPI advisors earlier deployed by NuPITA within their technical assistance package. The opportunity to participate in NuPITA-organized training activities was reported as a contributing factor to staff motivation and ultimately, retention. Skills gained during these trainings also played a role not just in developing staff careers but also in the overall quality of the services rendered. Some of the relevant TA inputs that partners have received from NuPITA to enhance institutional sustainability include individual and group trainings on cost share, documentation, leadership, resource mobilization, financial sustainability, and strategic planning. Specifically, the two-day training on essentials of financial sustainability dealt with those financial issues that affect organizations’ abilities to achieve their missions and objectives. The workshop on resource mobilization—attended by seven of the nine NPI organizations sampled for this evaluation—focused on donor assessment, donor expectations, strategic networking, and resource mobilization systems and included practical sessions on donor funding searches and how to conduct face-to-face interactions with potential donors. Nearly all project trainings took place between 2009 and 2011. Most accounts from the respondents indicated that these training opportunities were useful, enriching, and specific to the topics they intended to cover. However, the timing of specific trainings was mentioned by a number of them as a constraint. In particular, a number of respondents felt the training on resource mobilization should have been scheduled much earlier in the course of the project for its obvious repercussions on the partners’ ability to prepare for their own sustainability. A different criticism of the NuPITA support for resource mobilization was raised by another respondent, who would have preferred more direct, agency-specific links with funding agencies and alerts in case of any request for application released by USAID and other important donors; expressed, in the words of this respondent, as: “There is this funding opportunity and you are well placed to access this. Here are your gaps; let’s fix those.” (Although resource mobilization may seem to take on disproportionate weight as the NuPITA project was closing, a number of partners also requested accelerated attention to the topic and agency-specific support for donor approaches during the mid-project client satisfaction survey; requests that became a key recommendation of the evaluators.) Specific criticism of timing and the approach to resource mobilization aside, all respondents felt that other NuPITA TA largely contributed to the steps partners had taken over the course of NPI to become more sustainable. The NuPITA approach of working alongside its partners rather than prescribing and dictating solutions was widely appreciated and cited several times during the interviews (“Support was tailor-made to improve on our weaknesses.” “They didn’t say ‘you have to do it this way’”). 29 3.2.1 Special case: Sustainability with change of staff and program As described in Section 2.2, during the partner sampling process it became clear that at least four of the 15 NuPITA partners had likely lost significant project memory at the time this evaluation was conducted, either due to staff turnover or because they had closed the country office or HIV program. Assessing what, if anything, such partners retained from NPI and NuPITA would contribute to findings on program sustainability. The team therefore decided to select at least one such partner to determine what, if any, program changes due to NuPITA may have been sustained. The team selected the American Refugee Committee (ARC), as it has both an office in Kampala and accessible new senior management staff who had not been involved in the NPI or NuPITA efforts. (There were staff with institutional memory at HQ but the purpose of the interview was to assess what remained in place in the field program.) ARC/Uganda has eliminated its HIV programming due to funding vagaries rather than a strategic programming decision. Given significant reductions in recent years in the numbers of refugee/internally displaced persons (ARC’s primary beneficiaries), in Uganda, the agency is in the process of reassessing its overall country mandate. ARC currently focuses almost exclusively on gender-based violence (GBV) mitigation. In one large UNFPA Joint Programme on GBV begun a year ago, ARC is an implementing partner at the level of UN Women and FAO and has eight NGO sub-recipients that include both smaller CBOs and experienced international NGOs like CARE. It was not possible for managers to confirm or negate links between any NuPITA inputs, including systems improvement, to securing this new funding or central project role. The new managers were able to describe improvements in ARC’s M&E systems at both the global and national level and acknowledged NuPITA’s contribution to these improved systems. “NuPITA helped us from the national level up to our board of directors.” (ARC had significant support for M&E at both HQ and in Uganda. Each site had an NPI M&E advisor.) The managers were aware of, and use, the documented support supervision system put in place under NuPITA. They could also describe ARC’s organization-wide efforts to standardize some indicators, such as cost per beneficiary for cross-country analysis. Another technical area supported by NuPITA that appeared strong in the GBV programs was referral pathways. However, it is unclear whether these are a result of work done on HIV referrals under NPI or if, since referral systems are central to GBV mitigation programs, they were sufficiently developed as a matter of course. 3.3 NuPITA: Successes and Areas for Improvement “There are so many things that worked well, I have to think about what worked best.” All respondents (NPI partners, NuPITA staff, and USAID representatives) were asked a summary interview question about what they thought worked well and what did not in the NuPITA program and if they had any recommendations for future capacity-building programming. Responses are summarized below, with respondent groups noted when not already indicated. 30 3.3.1 Strengths Quality and responsiveness of NuPITA assistance Respondents in all categories considered the strongest aspect of NuPITA to be the responsive, non￾judgmental, and respectful approach that JSI and Initiatives Inc. took in implementing the program, such as tailoring all inputs to the needs of each organization and including both HQ and field staff, where applicable, in the change process. All respondent categories cited the breadth and high quality of the technical assistance and the professionalism of the diverse NuPITA team. Sustainable improvements in organizational systems and processes All respondent categories highly valued lasting improvements made within partner organizations, with M&E, accounting, and technical SOPs specifically mentioned. Separation of capacity building from contractual oversight Staff felt that being able to provide capacity building without a contractual or funding relationship with the partner was initially daunting and ultimately a key advantage in building trusting open partnerships. Staff and partners appreciated the fact that partners could access any kind of relevant TA at no cost (except time). (NuPITA staff and advisors, NPI partners.) Preference for NuPITA staff support over consultants Some NPI partners found that consultants, due to their lack of an ongoing relationship and understanding of the partner’s context, were not as helpful as NuPITA staff. Easy access to global resource network Commenting on their ability to manage the wide range of requests, NuPITA staff credited JSI’s flat structure, which allowed efficient access to JSI’s global resources. (NuPITA staff and advisors.) Early staff orientation to cooperative TA methods Staff credited early orientation to collaborative TA approaches for their ability to establish good working relations with partners. (This was not an entirely smooth process because of difficulties identifying and retaining appropriate OD staff at the outset.) The quality of diverse staff, training, tools, and other resources provided were considered key assets. (NuPITA staff and advisors.) Other strengths, mentioned less frequently: • Commitment of the advisors • Partner-specific action planning after every T/OCA or training • Improved partner linkages with other NPI partners and within their own organization through inclusion of HQ staff in field TA • Particular usefulness of structured learning visits and documentation (SOPs, procedure manuals and public relations materials) 31 3.3.2 Areas for improvement Program design: short length of NPI Although beyond NuPITA’s control, the short length (three years) of the partners’ NPI cooperative agreements was universally mentioned as limiting partner ability to understand the NPI and NuPITA programs and effectively absorb the TA offered. Program design: centrally funded versus bilateral Although USAID mission staff saw the advantage of multi-country centrally-funded programs for cross￾fertilization, mission-supported capacity building would bring greater contextual understanding and support. The need for future capacity building was questioned in one case, given the tremendous support USAID has given over the years, as it diverts needed funding from direct service delivery. Need for reinforcement of key messages at startup NPI partners and NuPITA staff and advisors all mentioned that, in retrospect, more focused, repeated communication at the beginning might have helped clarify the relationship of NuPITA with both USAID and the NPI partners. New USG partners were often overwhelmed at the start, sorting out NPI and NuPITA partners and what each required and had to offer. In addition, the visibility of the NPI project and expectations that it have an early and successful start contributed pressure. These elements affected partner ability to absorb all information offered at the outset. Sequencing of NuPITA and NPI activities Respondents in all interview categories felt the short NPI project length and the time needed to establish working relations led to TA being scheduled in late in the project, which was stressful. Although it takes all projects time to settle into a workable rhythm, staff and advisors felt that staggering project start so the TA provider begins before its partners would allow research partners to prepare some generic aspects of the program and might foster efficient start up and smoother sequencing of TA. Partner and project staff turnover Although the negative effect of turnover was mentioned by all, USAID, in particular, noted that continuity of the project management team was helpful in mitigating early staffing challenges. Improvements in organization of training program NPI partners all preferred direct, organization-specific TA to group trainings (despite appreciation of training quality and efforts to tailor actions in one-on-one planning meetings at the training venue). In retrospect, almost all would have liked to have an annual schedule of upcoming trainings for planning purposes and better timing and sequencing of trainings. There were many more trainings at the end of NuPITA, some of which (client targeting, financial management for non-finance staff, resource mobilization, closeout) happened after they would have been most useful. Several partners would have liked more inclusion of community partners in NuPITA trainings. (Despite this preference, all sampled partners reported successfully rolling out training received to their partners.) (NPI partners, USAID.) Earlier and more support for resource mobilization Unsurprisingly, given the fact that the project was closing at the time of the evaluation, the need for earlier and agency-specific support for resource mobilization was mentioned by at least half of the 32 sampled partner respondents. They felt the resource mobilization training came too late, despite early requests made in the mid-project client satisfaction survey report). They also felt more information on how to access funding and structured introductions to non-USG donors—as NuPITA did with USAID— would have helped. Although this may be an unrealistic expectation for the short program timeframe, it is useful information for future programming. NuPITA staff mentioned that, in retrospect, they would have categorized many activities (e.g., program documentation, systems improvement) as directly contributing to ability to mobilize resources. Electronic ticketing system This system to manage TA requests was not mentioned either positively or negatively by any partners. JSI HQ staff still considers it a useful system for capturing, sorting, and reporting assistance, but the NuPITA field staff almost uniformly considered it a project element that did not work well, as it only functioned sporadically in the NuPITA field offices due to unresolved internet server problems. Cost-benefit of the NuPITA approach Although all USAID respondents appreciated the NuPITA approach and its accomplishments, the need to assess the cost-benefit of the approach was considered important for balancing these elements of capacity building. 33 CHAPTER FOUR: DISCUSSION OF RESULTS AND CONCLUSION 4.1 Discussion The overall goal of the NuPITA project was to increase the quality of program implementation and strengthen the institutional capacity of NPI partners, supporting PEPFAR’s objective to improve and expand HIV programming through CBOs and FBOs. The project results prescribed in the performance monitoring plan for the project focused on organizational development (OD), which is considered a proxy for both service delivery and sustainability. 4.1.1 Service Delivery The project objective was to have all supported NPI partners achieve at least 90 percent of their implementation plans and targets, and to be able to clearly document changes and the actions taken to achieve organizational capacity improvements. During the evaluation, the success of the NPI partners who were supported by the NuPITA project was measured by their achievement of the PEPFAR indicators and also their ability to work effectively under USG procedures, rules, and regulations, and secure an institutional presence that would likely sustain their involvement in the HIV programming arena after graduation from the NPI program. The sampled NPI partners are diverse in size, country of operation, and technical scope. The services these organizations offer are also varied and include prevention, care, and support. Some of them support and oversee smaller local organizations within their areas of operations. Targets surpassed by all partners through expansion of activities This evaluation demonstrated that all supported partners were able to surpass, in some cases by a large margin, their NPI targets on primary service beneficiaries. Despite some isolated cases of under￾targeting when setting objectives to account for a new donor and program, targets were met and surpassed through a gradual and systematized expansion of activities that in some cases could only start at full scale during the second year of implementation. The higher number of prime service beneficiaries reached was also helped by the changes NPI partners adopted in their service delivery approaches, many of which were in turn initiated through NuPITA TA. Key Achievements in Service Delivery Because of the technical support provided, most NPI partners were able to: • Widen their technical areas of implementation to also include services for beneficiaries not targeted initially, such as income-generation activities and comprehensive care for OVC, and prevention activities for the general population. • Effectively reach their target populations and in some cases reach out to new groups (e.g., HIV prevention, girls among OVC). • Strengthen monitoring and evaluation function at project sites, which eventually resulted in better recording and reporting of project outputs. • Improve referral linkages that enabled most organizations to confidently direct their clients to access other services whenever needed. 34 • Improve the quality of support supervision, with especially strong results among those NPI partners that did not implement activities directly but through sub-grantees that were often smaller community-based organizations. Improvements in reach and quality of services Ultimately, the changes above led to increased numbers of beneficiaries who were reached with project services, which represented perhaps the most relevant achievements by NPI partners. Additionally, the services provided have improved, as shown by the analysis of the service quality proxy indicators including improved referral systems, better and more structured support supervision, use of international guidelines, and standards (see Table 2). Improvements in organizational capacity Following the OCA and TCA exercises, all NPI partners were found to be in need of significant organizational capacity building and technical support. None of the sampled NPI partners was fully skilled in areas such as strategic planning, human resources management, financial management, procurement, technical programming, monitoring and evaluation, indicator setting, data collection, reporting, or resource mobilization for program sustainability when the OCAs and TCA were conducted. Use of the Child Status Index This evaluation has also shown that the CSI tool has been adopted by most of the OVC programs to monitor the support provided and improvements in child well-being at an individual and household level. The widespread use of the CSI tool was due to its roll out though a specific NuPITA-led training on this topic, coupled with M&E strengthening and attended by NPI partners implementing OVC programs. The interviews revealed a different degree and varied pattern of utilization of this tool by the implementing NPI partners. However, there was a way the CSI index could have been used for internal monitoring and external dissemination of results, including establishing an electronic CSI database and systematically assessing the utilization of the CSI tool among supported partners. The evaluation confirmed that NPI partners experienced remarkable improvement over the course of the program in the following areas: organizational capacity, leadership and growth; financial management and systems; expansion of technical HIV and AIDS services in both scope and people reached; strengthening of HR systems and structures; and building of referral networks. 4.1.2 Sustainability Improvements in organizational systems and processes This evaluation also showed that the capacity-building inputs that were provided by NuPITA to the NPI partners enabled them to make significant improvements in their organizational capacity and the quality of NPI programs that they implemented. These inputs enabled the NPI partners to streamline their organizational structures and addressed the gaps and priorities that were identified through the OCAs/TCA. Partners have been able to build core strength in HR management, financial management and M&E. Some of them are now in a better position to transfer adequate TA to their sub-partners and local implementing partners. All assistance tailored to partner-specific need Additionally, it was evident that all support provided to the NPI partners was based upon needs assessments and negotiated with each NPI partner, even within the context of general trainings. All 35 respondents generally recognized this as a very effective yet unique approach to capacity building. It helped them develop a favorable relationship with NPI partner staff and contributed to creating the mutual trust necessary to make capacity building better used and genuinely appreciated. Respondents reported taking advantage of more NuPITA technical assistance opportunities because they saw it would be relevant to their organization. Success obtaining follow-on funding from expanded donor base An obvious consequence of this substantial organizational growth and change was partner ability to obtain further funding from USAID and other sources. The rigorous systems put in place to comply with USAID-required procedures made them more credible organizations and attracted the interest of other donors who were impressed by their overall organizational development, financial discipline, and functional administrative policies. In a competitive field where implementing organizations have to vie for limited available funds, NPI partners have already qualified for funding, even from donors they had considered inaccessible before the NPI project. This diversification of funding sources—often in multi-year funding they had not obtained before—likely laid a strong foundation for future sustainability of the organizations themselves and their programs. Improved organizational confidence and visibility Improved partner marketability moved in tandem with gradually established technical networks, participation in relevant policy fora, and working groups to share experiences and successes and improve the organizations’ visibility to development agencies and potential donors and partners. This is likely to generate a cycle in which more funds give further credit and visibility, while the gained visibility makes the organizations more competitive for further funding. Most of the supported partners state they are more externally confident and visible within their countries of operation than before, including participation in national coordination groups and USAID information-sharing fora. Ability to comply with requirements of USG funding This evaluation showed that the NuPITA project effectively supported the NPI partners to successfully manage and comply with USG regulations and requirements during the life of their cooperative agreements. The supported partners are now stronger entities that can implement high-quality HIV and AIDS programs and also competitively source USG and other funding. 4.2 Conclusion In conclusion, the provision of technical assistance was found to have significant effects on both the quality of services being delivered by the recipient organizations and the institutionalization of systems and processes needed for organizational sustainability. This evaluation has shown that technical assistance was a favorable factor not only in improving the technical capacity of recipient partners, but also more broadly for the overall development of the targeted organizations. Implications for USAID programming Although the evaluation findings may inform the planning of various donors, agencies, and program types, they may have special relevance for USAID, which funded the NPI and NuPITA programs as well as this evaluation. Mission staff from Kenya and Uganda and USAID/Washington program staff also contributed to the evaluation through individual and group interviews. 36 USAID seeks to apply lessons from NuPITA to future USAID capacity-building initiatives, including how Washington can best support country-specific efforts, such as the Local Capacity Development Initiative, which will be managed primarily by USAID missions. NuPITA’s strong, positive relations with its partners, senior staff consistency, and broad technical competencies were seen as advantages that directly affected program results. USAID staff representatives acknowledge the need to ensure some continuity of technical assistance and in-depth knowledge of partner abilities and needs in future capacity-building efforts. They also acknowledge this will be a challenge. More effort will likely be made in the future in USAID programming to both use and support regional TA networks. The NuPITA model may inform new initiatives including Local Capacity Development and USAID FORWARD, “an ambitious reform effort…to change the way the Agency does business—with new partnerships, an emphasis on innovation and a relentless focus on results. It gives USAID the opportunity to…unleash its full potential to achieve high-impact development.”7 NuPITA focused on mentoring new partners for success with USG programming scope and requirements; organization versus project-directed TA; fostering a culture of “learning organizations” through the adoption of iterative, self-assessment processes and networks of information-sharing partners; and data-driven decision-making. All these activities are applicable to USAID FORWARD and Local Capacity Development goals. General implications for future programming The findings of this evaluation are directly relevant to future capacity-building programming as there seems to be strong evidence linking TA provision with successful and long-term abilities to implement high-quality HIV and AIDS programs. For future capacity-building partnerships, NuPITA brings lessons of particular importance. These include: • Transformative organizational change is labor-, time-, and resource-intensive. All respondents acknowledge the intensity of effort needed to achieve long-term systemic changes in organizations. In looking to be both cost-efficient and effective, donors and their partners need to be clear on institution-building objectives and the inputs needed to achieve them. • Capacity-building support is highly valued by all sampled partners, to the point of willingness to forego some direct funding in the future in exchange for capacity-building support. All partners felt strong capacity building was essential for any new USG partners (e.g., USG compliance, systems documentation), and that all partners could benefit from updates in HIV programming and continued support for improving their management skills. They acknowledge that they have internalized iterative self-assessment processes and are now better able to use data to identify problems and resolve them. Despite these improved skills, they feel they would continue to benefit from high￾quality support for organization-specific development, particularly for senior managers. • Separating technical assistance support from contractual and administrative oversight of the recipient helps: 1) focus the TA provider on offering support of direct value to partners who can refuse it; and 2) generate mutual trust and open, collegial relationships. • Partners appreciated the breadth and quality of the overall TA package and of the autonomous process of requesting it. The evaluators attempted to link specific NuPITA inputs to outputs. Partners consistently declined to highlight a particular input when discussing progress, 7 www.forward.usaid.gov, accessed online September 5, 2012. 37 focusing instead on the complementary nature of the support including OCAs and TCAs; tailored, on-site TA provided by the NuPITA team and on-site placement of NPI advisors; structured learning visits and trainings (including but not limited to training-of-trainers, monitoring and evaluation, referral networks, and close-out). • Open, trusting and ongoing relationships that allow a deep knowledge of partner capacities and provide the ability to tailor assistance to specific needs contribute strongly to efficient change. • Sequencing project startup so that the TA provider has even a brief head start on staffing and fundamental support designs before NPI partner agreements begin would have given more time for sequencing of TA and perhaps a smoother project startup. 38 ANNEXES Annex A: Evaluation Framework – Service Delivery and Sustainability Evaluation objectives Evaluation questions Study variables Data collection method Data collection tool Source of information Analysis plan/strategy To determine the effect that NuPITA’s capacity building technical assistance has had on HIV service delivery results through the work of its partner organizations Have NuPITA partners changed their HIV service delivery approaches over the life of the project? If so, how and what effect has this had on service delivery results? • Changes in scope and approach in service delivery (disaggregated by PY and gender/age-group) – over the cooperative agreement period • Document review • Key informant interviews • Document review template • Key informant interview guide • End-of￾project reports for partner NGOs • Staff of partner NGOs • Simple tabulation • Content analysis • Changes in number of beneficiaries reached (disaggregated by PY and gender/age-group) - over the cooperative agreement period • Document review • Document review template • End-of￾project reports for partner NGOs • Simple tabulation • Gaps identified during the TCAs being addressed by the partner NGOs over the cooperative agreement period • Document review • Key informant interviews • Document review template • Key informant interview guide • NuPITA TCA reports • Staff of partner NGOs • Simple tabulation • Content analysis • Institutionalization of internationally recognized standards and procedures • Document review • Key • Document review template • NuPITA TCA reports • Staff of • Simple tabulation • Content 39 Evaluation objectives Evaluation questions Study variables Data collection method Data collection tool Source of information Analysis plan/strategy over the cooperative agreement period informant interviews • Key informant interview guide partner NGOs analysis • Changes in the quality of support supervisions (availability of written comprehensive periodic reports, identification of issues to be addressed, evidence of actions taken to address them) - over the cooperative agreement period • Document review • Key informant interviews • Document review template • Key informant interview guide • Staff of partner NGOs • Field reports for partner NGOs • Content analysis • Use of program data to inform management decisions (periodic review meetings, evidence of internal data dissemination, etc.) • Document review • Key informant interviews • Document review template • Key informant interview guide • NuPITA TCA reports • Staff of partner NGOs • Content analysis • Changes in the functionality of referral systems/linkages • Document review • Key informant interviews • Document review template • Key informant interview guide • Staff of partner NGOs • Field reports for partner NGOs • Content analysis • Change in the aggregate domain scores for the Child Status Index (CSI) – • Document review • Key • Document review template • Staff of partner NGOs • Simple tabulation • Content 40 Evaluation objectives Evaluation questions Study variables Data collection method Data collection tool Source of information Analysis plan/strategy for selected partners – for the period for which it was administered informant interviews • Key informant interview guide • Field reports for partner NGOs analysis Have these changes, if any, to partners HIV service delivery approaches been influenced by capacity building inputs provided by NuPITA? If so, how? • Extent to which the changes prescribed above have been attributed to specific NuPITA inputs by the project stakeholders (NuPITA staff and partner NGOs). Examples of NuPITA inputs include, NuPITA staff, training, consultancy, learning field visits, etc. • Document review • Key informant interviews Develop a listing of inputs received by each organization and find out in interviews how much they thought each contributed to changes in the way they provide services • Key informant interview guide • NuPITA annual report s • Reports of partner NGOs • Staff of partner NGOs • Content analysis To determine the effect that NuPITA’s capacity building technical assistance has Have NuPITA partners experienced changes in their overall institutional sustainability (as measured by the • Number of funding sources between inception and close out of NuPITA • Document review • Key informant interviews • Document review template • Key informant interview guide • Staff of partner NGOs • Reports for partner NGOs Simple tabulation and content analysis 41 Evaluation objectives Evaluation questions Study variables Data collection method Data collection tool Source of information Analysis plan/strategy had on the sustainability of its partner organizations. type and quality of linkages with stakeholders and type and quality of systems that allow them to better access funding or to address funding gaps) since the start of their NPI cooperative agreements? If so, what are these changes? • Number of collaborating partners between inception and close out of NuPITA • Document review • Key informant interviews • Document review template • Key informant interview guide • Staff of partner NGOs • Reports for partner NGOs Simple tabulation and content analysis • Functionality of networks/linkages (participation in national discourses, technical working groups, advocacy networks, etc.) between inception and close out of NuPITA • Document review • Key informant interviews • Document review template • Key informant interview guide • Staff of partner NGOs • Reports for partner NGOs Simple tabulation and content analysis Have these changes in sustainability been influenced by capacity building inputs provided by NuPITA? If so, how? • Extent to which the changes prescribed above have been attributed to specific NuPITA inputs by the project stakeholders (NuPITA staff and partner NGOs). Examples of NuPITA inputs include, NuPITA staff, training, consultancy, learning field visits, etc. • Document review • Key informant interviews • Document review template • Key informant interview guide • Staff of partner NGOs • Reports for partner NGOs Content analysis What factors have contributed to NuPITA achieving or not achieving its intended results? • Descriptive analysis of what worked and what did not work with the NuPITA project (sources being NuPITA staff and partners) • Key informant interviews Key informant interview guide • Staff of partner NGOs Content analysis 42 Evaluation objectives Evaluation questions Study variables Data collection method Data collection tool Source of information Analysis plan/strategy • Other contributing factors to the changes in service delivery and/or institutional sustainability 43 Annex B: Terms of Reference for the NuPITA Partners’ Impact Evaluation Background and rationale The New Partners Initiative Technical Assistance Project (NuPITA) was funded by the United States Agency for International Development (USAID) for the period April 2008 to September 2012. NuPITA was implemented by John Snow, Inc. (JSI) in collaboration with Initiatives, Inc. The project was designed to provide technical assistance to fifteen organizations working in HIV and AIDS that received funding from USAID under the President’s Emergency Plan for AIDS Relief (PEPFAR). Partners worked in eight countries (Ethiopia, Kenya, Rwanda, South Africa, Tanzania, Nigeria, Uganda and Zambia). As part of the New Partners Initiative (NPI), the project aimed to build the capacity of organizations to work at the community level, supported long-term responses to HIV and AIDS, and increased the number and diversity of organizations working with the United States Government (USG) to address HIV and AIDS around the world. Purpose of the evaluation The USAID Evaluation Policy defines evaluation as the systematic collection and analysis of information about the characteristics and outcome of programs and projects to improve effectiveness and/or inform decisions about current and future programming. The Automated Directive System (ADS) 203.3.5.5 (revised February 1, 2012) stipulates that as part of a program cycle, the evaluation should look back over the previous phases of the cycle to understand why results were achieved, or why they were not; and looks forward through informing future development and project designs with this evidence. USAID has also initiated the Local Capacity Building Initiative that would benefit from results of this evaluation. As NuPITA approached project closeout, we wanted to show the effect of our capacity-building efforts through a mixed-methods evaluation. The purpose of the evaluation was to determine the effect of NuPITA’s capacity building efforts on partner HIV program service delivery results as well as their overall sustainability (as measured by the type and quality of linkages with stakeholders and type and quality of systems that allow them to better access funding or to address funding gaps), and to link these to capacity-building inputs provided by NuPITA and identify lessons learned in capacity building. Evaluation objectives The two key objectives of the evaluation were to: 1. Determine the effect that NuPITA’s capacity-building technical assistance had on HIV service delivery results through the work of its partner organizations 2. Determine the effect that NuPITA’s capacity building technical assistance has had on the sustainability of its partner organizations. Sample/suggested evaluation questions included: 1) Have NuPITA partners changed their HIV service delivery approaches over the life of the project? If so, how and what effect has this had on service delivery results? 2) Have these changes, if any, to partners HIV service delivery approaches been influenced by capacity-building inputs provided by NuPITA? If so, how? 3) Have NuPITA partners experienced changes in their overall institutional sustainability (as measured by the type and quality of linkages with stakeholders and type and quality of systems 44 which allow them to better access funding or to address funding gaps) since the start of their NPI cooperative agreements? If so, what are these changes? 4) Have these changes in sustainability been influenced by capacity-building inputs provided by NuPITA? If so, how? 5) What factors have contributed to NUPITA achieving or not achieving its intended results? Scope of the evaluation The methodology for the evaluation will be multifaceted, combining secondary data analysis of HIV data from NuPITA partner organizations (both historical—if available—and data available during the life of the NPI cooperative agreements on issues such as the number of OVC reached, or the number of people infected or affected by HIV who received care and support services), document review, review of the 2010 partner survey results, records of NuPITA technical assistance provided to organizations, and primary data collection and analysis through key informant interviews with partner organization staff, their beneficiaries and stakeholders, NuPITA staff, and USAID Washington and mission staff. It is expected that the evaluation will be conducted over a six-week period in approximately May and June, 2012. The first week will involve finalization of the evaluation methodology. This will be done collaboratively by NuPITA and the evaluation team. The second week will primarily consist of a document review and secondary data analysis of selected partners, the third and fourth weeks will be a continuation of the second week’s activities, as well as key informant interviews through both in person and telephone interviews (depending on the location of the key informants), as well as analysis of these interviews. The final weeks will be focused on writing of the evaluation findings into a report and debriefing NuPITA on the results of the evaluation. NuPITA is looking for a small team of evaluators—approximately 2-3 people—to conduct this work. Throughout the course of the evaluation, the evaluators will remain in regular contact with a designated focal person on the NuPITA team. The evaluation team will be able to conduct significant portions of the evaluation (such as the primary and secondary data analysis, document review, and writing up of findings) remotely. However, it is anticipated that at least some time, approximately 2 weeks will need to be spend in Africa (likely Uganda and possibly other countries. This will be determined prior to the start of the evaluation). Qualifications NuPITA is looking for a team of three evaluators with combined (i.e. not all evaluation team members need to be experts in all three areas) expertise in evaluation, organizational development, HIV and AIDS programming, public health, and health systems strengthening. In addition, each of the evaluators will need to be experienced in both qualitative and quantitative data collection and analysis. Specific qualifications and experience include: - MPH or other relevant graduate degree - Experience designing, conducting and analyzing qualitative interviews - Experience triangulating qualitative and quantitative data - Knowledge of USAID data quality standards - Experience analyzing quantitative data using statistical software (SPSS, SAS, Stata, etc.) - Experience in implementing public health programs, including the areas of HIV, health systems strengthening, behavioural change communication - Experience in organizational development 45 - Excellent oral and written communication skills - Experience in health systems strengthening Deliverables - Weekly telephone updates with the NuPITA focal person, accompanied by a 1-page brief on progress to date - A final evaluation report submitted in draft form by the end of the fourth week of the evaluation, and a final version submitted six weeks after the start of the evaluation. The exact deliverable due date will be determined based on the start date for the evaluation. 46 Annex C: Trends in Average OCA Scores vs. Program-Related Domains’ Scores 47 Annex D: Support Summary Table 48 Annex E: Individual Partner Reports Although this evaluation focuses on the overall effects NuPITA capacity-building technical assistance has had on service delivery and sustainability of NPI partners, a summary of program achievements by partner is included here for reference. For each partner, a summary of service delivery achievements against the pre-set project targets for selected PEPFAR indicators is presented. The OCA domains most closely related to service delivery are program management and project performance management. Trends for these domains over the three years are plotted against the average OCA scores for each partner. The average OCA score is itself a good proxy of overall organization capacity for sustainable programming. It should be noted that the OCA gives a snapshot of self-reported capacity that can be used for planning and monitoring. Nevertheless, OCA scores over time provide a useful gauge of overall progress on key organizational capacity indicators. In general, scores for program management and project performance management—the two service￾related domains—fell short of the other OCA scores, indicating a need for NuPITA to focus more attention on these areas. This led to the development and implementation of the TCA tools. TCA scores directly related to service delivery and sustainability are also presented. These were reviewed in partner interviews to assess progress on addressing gaps since the TOCA in Year 2. Finally, several highlights of each partner’s achievements are given. Ananda Marga Universal Relief Team (AMURT) AMURT Kenya works in the coastal, central, and western provinces to provide affordable HIV and AIDS treatment and health care to poor communities. It supplies foodstuffs, clothing, medicines, water pumps, and other relief materials to victims of famine and political violence. The organization conducts project activities with development partners, the government, CBOs, local NGOs, and beneficiaries. Under NPI, AMURT implemented its project in the three provinces where it operates. Its project objectives were to create sustainable support structures for OVC; to improve treatment, care, and livelihoods for PLWHA in Nyanza Province; to provide opportunities for behavior change, leadership development, and vocational training to out-of-school youth; and to launch a mass education prevention and HIV awareness program. 49 Trend in Organizational Capacity Score vs. Program-related domains' Scores 0 1 2 3 4 Year 1 Year 2 Year 3 Program Management Project Performance Management AVERAGE Organizational Capacity Score 50 AMURT: score for selected TOCA areas 2 2 2 2 2 2 2 3 3 3 3 0 1 2 3 4 OVC - Guidelines/SOPs OVC - Referral Systems Prev - Guidelines/SOPs Prev - Service Standards Prev - Referral Systems Prev - Supervision C&S - Guidelines/SOPs OVC - Service Standards C&S - Referral Systems Data Collection Data Use for Decision Making 51 Illustrative NPI Achievements AMURT Links to NuPITA: • Accessed new USAID funding with different APHIAplus partners in all three regions • Used improved data both to identify a serious prevention program deficit and guide a solution (linking messages to their most popular services) American Refugee Committee (ARC) The American Refugee Committee is an international nonprofit, nonsectarian organization that has provided humanitarian assistance and training to refugees, displaced persons, and host communities since 1981. ARC provides these groups with shelter, clean water and sanitation, health care, skills training, microcredit education, and protection. It builds the capacity of communities to survive conflict and crisis and rebuild lives of dignity, health, security, and self-sufficiency. Since 2006, ARC Uganda has implemented projects that support internally displaced persons and provide services to mitigate gender-based violence. ARC implements the NPI program in Northern Uganda. The project’s overall goal is to contribute to reduced incidence of HIV and AIDS among displaced and returnee populations in Pader, Gulu, and Amuru Districts by providing HIV and AIDS prevention, care, and support services. ARC’s objectives are to work with national partners to increase knowledge and facilitate positive behavior change; enhance access to and utilization of quality HIV and AIDS prevention, care, and support services, and improve the coordination and capacity of national HIV and AIDS responses/organizations. Achievements against project targets 52 Trend in Organizational Capacity Score vs. Program-related domains' Scores 0 1 2 3 4 Year 1 Year 2 Year 3 Program Management Project Performance Management AVERAGE Organizational Capacity Score ARC: score for selected TOCA areas 1 1 1 2 2 2 2 2 2 2 3 3 3 3 3 3 0 1 2 3 4 Care - Data Collection Care - Data Use for Decision Making Prev - Referral systems Care - Referral systems Prev - Data Collection Prev - Data Use for Decision Making PMTCT - Referral systems PMTCT - Data Collection PMTCT - Data Use for Decision Making HCT - Program Specific Protocols Care - Program Specific Protocols Prev - Program Specific Protocols PMTCT - Program Specific Protocols HCT - Referral systems HCT - Data Collection HCT - Data Use for Decision Making 53 Illustrative NPI Achievements ARC Links to NuPITA: • Functional M&E system • Global standardization of some M&E indicators and processes François-Xavier Bagnoud International (FXB) François-Xavier Bagnoud International (FXB) supports vulnerable children and families or caregivers struggling with poverty and HIV and AIDS. FXB is active in 15 countries where it operates 100 programs, including 49 FXB villages. FXB’s village model provides a three-year package to a cluster of 80 households that become increasingly self-sufficient each year. Eighty-five percent of families in the program become financially self￾sufficient by the end of the three years. FXB has been in Uganda since 1990 and currently operates 11 FXB villages, each of which provides 80– 100 families with a comprehensive package of health, education, and psychosocial support and income￾generating activities. PEPFAR supports eight of the FXB villages. Summary performance data against targets was not available from FXB/Uganda at the time of the evaluation. The Rwandan genocide of 1994 orphaned countless children and enabled the spread of HIV and AIDS. In response, in 1995, FXB developed and launched the FXB Village program. There are currently 22 in Rwanda directly reaching approximately 12,000 people, mostly children. Twelve of these programs are funded through PEPFAR. Summary figures below are for the FXB/Uganda program, as Uganda and Kenya were the focus of the team’s in-depth partner visits. Achievements against project targets 54 Trend in Organizational Capacity Score vs. Program-related domains' Scores 0 1 2 3 4 Year 1 Year 2 Year 3 Program Management Project Performance Management AVERAGE Organizational Capacity Score FXB: score for selected TOCA areas 2 2 2 2 3 3 3 3 3 0 1 2 3 4 OVC - Program Specific Protocols OVC - Service Standards Prev - Service Standards Data Collection OVC - Referral Systems Prev - Program Specific Protocols Prev - Referral Systems Prev - Supervision Data Use for Decision-Making 55 Illustrative NPI Achievements FXB/Uganda Links to NuPITA: • Building on NuPITA’s inputs related to strengthening referral networks, exchanged capacity building of partners for free services for FXB clients, greatly extending range and reach of services • Accessed funding available to local NGOs exclusively after NuPITA helped them register in Uganda Grassroots Alliance for Community Education (GRACE) GRACE is dedicated to helping orphaned children, their families, and communities. The organization’s mission is to enhance the capacity of CBOs for self-determined, high-impact, and sustainable initiatives leading to better health and development. Since 2001, GRACE has established functional partnerships with 105 local grassroots organizations in Kenya, including CBOs, faith-based organizations, and youth groups. GRACE builds partner capacity to implement locally determined and sustainable initiatives in response to the HIV and AIDS epidemic. Under the NPI program, GRACE worked with 15 partners in three Kenya provinces (Nyanza, Eastern, and Central) to start early childhood development centers and with four partners to provide education grants to OVC, to train OVC caregivers, and to mobilize the community to better care for their OVC. GRACE also provides prevention messages to the community to build knowledge and practices to reduce risky behavior and to increase acceptance of people who are infected. Achievements against project targets 56 Trend in Organizational Capacity Score vs. Program-related domains' Scores 0 1 2 3 4 Year 1 Year 2 Year 3 Program Management Project Performance Management AVERAGE Organizational Capacity Score GRACE: score for selected TOCA areas 1 1 1 1 1 2 3 3 3 0 1 2 3 4 OVC - Program Specific Protocols Prev - Program Specific Protocols Prev - Service standards Prev - Referral Systems Prev - Supervision OVC - Referral Systems OVC - Service standards Data Collection Data Use for Decision Making 57 Illustrative NPI Achievements GRACE Links to NuPITA: • NuPITA training in structured play therapy was a valued improvement in psychosocial support to OVC • NuPITA technical assistance passed to all 15 of their NPI partners Integrated Community Based Initiatives (ICOBI) Integrated Community Based Initiatives (ICOBI) is an indigenous non-governmental organization established in 1994, whose mission is to improve the quality of life of people living in rural communities. Under the NPI project, ICOBI partnered in Mbarara and Bushenyi districts with Ankole Diocese to reach out to OVC and their households with a comprehensive care package. The partnership resulted in synergies that contributed to the implementation of quality interventions and sharing of experiences and responsibilities. The goal of the project was to enhance the ability of OVC and their households to fully enjoy their rights and aspirations to their full potential. Achievements against project targets 58 Trend in Organizational Capacity Score vs. Program-related domains' Scores 0 1 2 3 4 Year 1 Year 2 Year 3 Program Management Project Performance Management AVERAGE Organizational Capacity Score ICOBI: score for selected TOCA areas 2 2 3 3 0 1 2 3 4 Program Specific Protocols Referral Systems Data Collection Data Use for Decision Making 59 Illustrative NPI Achievements ICOBI Links to NuPITA: • Strengthened systems and processes have led to five new grants and solicitation to be a sub-grantee • Now participates in national fora and technical working groups on OVC Mfesane Mfesane is an independent, ecumenical South African development organisation, registered in 1975. Its mission is to reach out to the poor and the disabled, with the purpose of helping them to achieve their independence and participate fully in society. In 1988, the organization’s focus shifted from the homelands to urban areas, in anticipation of the large-scale urbanization of black South Africans. Under NPI, Woord en Daad, a Dutch FBO, provides Mfesane with financial and technical support to implement a comprehensive HIV response program covering HIV counseling and testing, orphans and vulnerable children, prevention, and care and support components. These activities are operational in: the Eastern Cape and the Western Cape. Mfesane now operates through supporting partner organizations and is implementing its own USAID￾funded HIV and AIDS program. 60 Achievements against project targets 61 Trend in Organizational Capacity Score vs. Program-related domains' Scores 0 1 2 3 4 Year 1 Year 2 Year 3 Program Management Project Performance Management AVERAGE Organizational Capacity Score Mfesane: score for selected TOCA areas 2 2 2 3 3 3 3 3 3 3 4 4 4 4 0 1 2 3 4 HCT - Service standards HCT - Supervision C&S - Program Specific Protocols OVC - Service standards HCT - Program Specific Protocols Prev - Program Specific Protocols Prev - Service standards Prev - Referral Systems Data Collection Data Use for Decision Making OVC - Program Specific Protocols HCT - Referral Systems C&S - Referral Systems Prev - Supervision 62 Illustrative NPI Achievements Mfesane Links to NuPITA: • Expanded from 2 to 6 municipalities to include vulnerable populations, add economic strengthening, HIV and AIDS education awareness, and HBC. • Received 7 grants from GOSA, which uses them as a reference center for systems, documentation and services. “You are in a class of your own now.” • Expanded linkages and networks, including becoming chair of Coalition of NGOs in HIV and Development. Retrak/Uganda Retrak is a faith-based NGO working with OVC on the streets in Africa, and is committed to providing each child with an individual route back to family and community. Retrak/Uganda began working with OVC in 1996. Lessons learned have been successfully transferred to a project in Ethiopia that opened in 2007. The main outreach activities are football training, feeding, and a clinic. Most activities take place at the drop-in center or clubhouse near the Mengo-Kisenyi slum area of Kampala City. Retrak/Uganda also runs a transitional home and training farm outside Kampala in Lubowa, Wakiso District). Vocational training is offered to older OVC who cannot fit into foster care and cannot be resettled. The staff team has a strong focus on social work and consists of social workers, teachers, residential care workers, a nurse, and a sports coach. Retrak/Uganda’s NPI project aims to provide quality care and protection to OVC on the street by meeting their psychosocial, health, and education needs. Additionally, Retrak helps to resettle children in a loving family environment either with their biological parents or, if this is not possible, a foster caregiver. Achievements against project targets 63 Trend in Organizational Capacity Score vs. Program-related domains' Scores 0,0 1,0 2,0 3,0 4,0 Year 1 Year 2 Year 3 Program Management Project Performance Management AVERAGE Organizational Capacity Score RETRAK: score for selected TOCA areas 2 2 2 3 3 3 3 3 4 0 1 2 3 4 OVC Referral Systems Prev. Referral Systems Prev. Supervision OVC Service Standards Prev. Guidelines/SOPs Prev. Service Standards Data Collection Data Use for Decision Making OVC Guidelines/SOPs 64 Illustrative NPI Achievements Retrak/Uganda Links to NuPITA: • Services expanded to include girl OVC living on streets • Expanded staff and program post-NPI with new funding • Have standard operating procedures for all key HIV programs, aligned with GOU and PEPFAR St. Johns Community Center (SJCC) St. Johns Community Center (SJCC), an initiative of the Anglican Church of Kenya, Diocese of Nairobi, was established in 1959. SJCC programs include care and support for people living with AIDS, orphans, and vulnerable children; HIV counseling and testing; HIV prevention; pastoral services; economic empowerment; non-formal education; community health and education; youth development; information empowerment (linked to advocacy and human rights); child rescue; and child development. The programs are implemented in six villages in Nairobi’s informal settlements, and in four regions of Kenya. Because of its extensive experience in community development work, SJCC partnered with Kindernothilfe (KNH) in 2005 to implement an OVC project. Under the NPI project, SJCC built the capacity of communities to provide the following services to OVC: care and support; access to basic medical services; school and community clubs that offer psychosocial support; apprenticeship training for livelihoods; access to revolving fund schemes for OVC households; and community foster care for child-headed households. SJCC does not directly implement OVC activities but builds the capacity of communities through small projects. Within the NPI project, 29 local partners managed the implementation of project activities. Achievements against project targets 65 Trend in Organizational Capacity Score vs. Program-related domains' Scores 0 1 2 3 4 Year 1 Year 2 Year 3 Program Management Project Performance Management AVERAGE Organizational Capacity Score SJCC: score for selected TOCA areas 1 1 2 2 2 2 2 2 3 3 3 3 3 0 1 2 3 4 HCT - Supervision OVC - Referral systems HCT - Referral Systems Prev - Referral systems Prev - Supervision OVC - Program Specific Protocols OVC - Service Standards Data Collection HCT - Guidelines/SOPS HCT - Service Standards Prev - Program Specific Protocols Prev - Standards Data use for decision making 66 Illustrative NPI Achievements SJCC Links to NuPITA: • Documented all HIV programs to GOK and PEPFAR standards • Extending improved M&E system and SOP development to all SJCC programs • Passed NuPITA TA to 29 grassroots CBOs WellShare International WellShare International is a nonprofit organization started in 1979 as Minnesota International Health Volunteers. WellShare implements community-based health programs to improve the lives of women, children, and their communities around the world. The organization now supports domestic programs for Somali immigrants and refugees, as well as field-based programs in Uganda and Tanzania. Under the NPI grant, WellShare works in Uganda, serving the districts of Ssembabule and Mubende where HIV prevalence is high and services to people living with HIV and OVC inadequate. The program goal is to improve HIV and AIDS care, prevention, and testing by increasing the capacity of local organizations to implement high-quality, community-based services for persons living with HIV and AIDS, orphans and other vulnerable children, and youth. The program objectives were to increase the ability and sustainability of CBOs to provide care and support to PLWHA and OVC and implement prevention activities; promote behavior change to reduce new HIV infections; reach eligible adults and children and their families with “umbrella” care services; and strengthen linkages among care activities, counseling, prevention, and treatment programs as part of a comprehensive national response to HIV and AIDS. Achievements against project targets 67 Trend in Organizational Capacity Score vs. Program-related domains' Scores 0 1 2 3 4 Year 1 Year 2 Year 3 Program Management Project Performance Management AVERAGE Organizational Capacity Score WELLSHARE: score for selected TOCA areas 1 1 2 2 2 2 2 3 3 3 3 0 1 2 3 4 C&S - Guidelines & SOPs Prev - Service Standards OVC - Guidelines & SOPs OVC - Service standards OVC - Referral Systems Data Collection Data Use for Decision Making C&S - Referral Systems Prev -Guidelines & SOPs Prev - Referral Systems Prev - Supervision 68 Illustrative NPI Achievements WellShare Links to NuPITA: • New financial and admin systems in place led to consideration as a finalist (at the time of the evaluation) for a civil society fund grant that will be awarded to potentially one of two of 30 bidders • Partnerships and networks substantially improved: bids pending with new partners, participation in national technical working groups, including OVC 69 Annex F: List of Persons Interviewed NuPITA Staff Kampala Mark Kowalski, Deputy Director, Programs Christine Lalobo, OD Advisor (past) Josephine Watuulo, M&E Specialist NaomeWandera, Senior M&E Specialist Agnes Barungi, Compliance Specialist Milly Katana, HIV and AIDS/Public Health Specialist Nairobi Barbara Durr, Project Director US Katrina Kruhm, Deputy Director Donna Bjerregaard, Senior Technical Advisor, Initiatives, Inc. NPI Advisors Andrew Kigozi, Compliance Specialist Josephine Tusingwire, OVC Advisor, Retrak USAID Missions Catherine Muwanga, Program Management Specialist/OVC, USAID/Uganda Jennifer Wasianga, USAID/Nairobi Washington Ken Sklaw Laurie Rushton Uchechi Roxo Elizabeth Baldwin Maria A. Carrasco American Refugee Committee (ARC) Ralf NicoThill, Country Director Ashenafi Eminu Habtewold, Finance Controller Paul Seshadri, Senior Program Coordinator AMURT, Nairobi Dr. Edward K, Kinyanjui, Programs Coordinator Dr. Jitendra Kumar, Executive Director FXB, Kampala Marion Ariongei, HIV and AIDS Prevention Officer Jane Naluwooza, Data Officer Raymond Tumuhairwe, Child Protection Officer Nathan Kaziba, Finance Officer Robert Sekadde, Program Manager 70 FXB, Kigali Damascene Ndayisaba, Country Director Emmanuel Habyarimana, Programs Manager GRACE/Africa Pascal Masila Mailu, CEO Rebecca Mawade, Project Officer ICOBI BoscoTuryamureeba Noel Mwebaze Boniface Mutatina Mfesane Yolandi Velentyn, Program Manager Mark Walker, Senior Program Manager Buyiswa Mpini, Project Manager St. John’s Community Centre (SJCC), Nairobi Peter Njuguna, Director Sarah Karanja, Deputy Director and Springs of Life Project Coordinator Ann Wahinya, Hope Project (previously on OVC project), Community Development Program Officer Joan Watiri, Project Accountant Damaris Njoroge, OVC Programme Coordinator Retrak, Kampala Dinah Mwesigye, Deputy Country Director Moses Wangadia, Health of Social Work Department Florence Soyekwo, Finance Manager Retrak, Ethiopia Joanna Wakia, Regional Development Officer Lynn Kay, Country Director WellShare, Kampala Laura Wando, Country Director Joel Ssevume, Operations Manager Beatrice Bainomugisha, Program Coordinator 71 Annex G: Schedule for Field Work WEEK 1 Monday Tuesday Wednesday Thursday Friday Saturday Sunday 4-Jun-12 5-Jun-12 6-Jun-12 7-Jun-12 8-Jun-12 9-Jun-12 10-Jun-12 Morning Team to finalize interview guide, analyze available data Team to review documents Team to visit Retrak (KLA) Wellshare (8:30- 10:00) 9:30 Josephine T/Andrew Kigozi (NPI Advisors) Document review (cont'd) Afternoon Team to interview NuPITA staff (KLA) NPI Advisors discussion Draft interview write-up Joan USAID noon-1:00 Team to visit FXB (KLA) (2:30) Draft interview write￾up WEEK 2 Monday Tuesday Wednesday Thursday Friday Saturday Sunday 11-Jun-12 12-Jun-12 13-Jun-12 14-Jun-12 15-Jun-12 16-Jun-12 17-Jun-12 Morning Daniel/Josephin e to meet ICOBI HQ & field staff in KLA - Luigi, Naome Joan travel to NBI GRACE/NBI (all) USAID mission (Joan) AMURT (all) Daniel to visit Mfesane Joan and Luigi Interview NuPITA PD and OD specialist in NBI Draft interview write-up 9:00 Christine Lalobo (Luigi/Joan) 4 pm Interview with Katrina Kruhm (Luigi/Joan) 9 pm Interview with Donna Bjerregaard, Initiatives, Inc. (Joan) Draft interview write-up Afternoon 2:30-4:00 SJCC (Joan/ Luigi, and Naome) Daniel travels to NBI Daniel travels to SA Luigi and Joan telephone interviews with Ethiopia(Retrak 2:30) Luigi Joan Naome travel to KLA WEEK 3 Monday Tuesday Wednesday Thursday Friday Tuesday 72 18-Jun-12 19-Jun-12 20-Jun-12 21-Jun-12 22-Jun-12 10-Jul-12 Morning 9:00-10:30 ARC 1:00-2:00 pm FXB/Rwanda Team to analyze collected data Team to continue analyzing and structuring report Team to continue analyzing and structuring report Joan interviews USAID/W team Afternoon Team to present preliminary findings to NuPITA 2:00-3:30 73 Annex H: List of Documents Reviewed  NuPITA annual reports (PY1, PY2, PY3, PY4)  NuPITA semi-annual report (PY5)  NuPITA Performance Monitoring Plan  NuPITA Partner Survey August 2010  NuPITA workplans (PY1, PY2, PY3, PY4, PY5)  NuPITA case studies  USAID NPI TA - RFTOP 2007  JSI Technical Proposal for NuPITA 2007  Original NuPITA Contract (Round 2) 2008  NuPITA OCA Score Guide 3-30-09  Reports from the OCAs, TOCAs, and CLOCAs of the sampled NPI partners  TA-NPI end-of-project survey 2011, analysis report  End-of-project reports of the sampled NPI partners  Final evaluation reports of the sampled NPI partners  Annual reports and summaries of annual data submission to PEPFAR of the sampled NPI Partners (as available)  Selected results from end-of-project partner survey December 2011 74 Annex I: Interview Guides Partner NGO Questionnaire Evaluation Question 1: Have NuPITA partners changed their HIV service delivery approaches over the life of the project? If so, how and what effect has this had on service delivery results? Changes in scope and approach in service delivery (disaggregated by PY and gender/age￾group) – over the cooperative agreement period Pre-NuPITA settings • Range of services offered • For HIV prevention and care services, what service delivery approaches were used (probe for community engagement, partnerships, service awareness, indicator settings, data collection, management and reporting practices) • Probe for availability of service delivery data (HIV prevention and care) that may be compared with PEPFAR data captured during the cooperative agreement (assess for trends during and after NuPITA) • Major challenges (both local and systemic) that constrained service delivery Post-NuPITA settings • Change in the range of services offered • For HIV prevention and care services, what service delivery approaches were/are being used (probe for community engagement, partnerships, service awareness, indicator settings, data collection, management and reporting practices) • Availability of service delivery data (HIV prevention and care) • Major changes in the operating environment that facilitated or hindered achieved of desired results Changes in number of beneficiaries reached (disaggregated by PY and gender/age-group) - over the cooperative agreement period Pre-NuPITA • Where data has not been submitted prior to NuPITA, ask for service delivery data for pre-NuPITA period or for Year One 75 Post-NuPITA settings • Where data has not been submitted prior to NuPITA, ask for service delivery data for PY 1-3 • Assess for achievement of targets (LOP); probe for reasons for over achievement or under achievement of targets • Assess for any other contextual which are pertinent to performance Gaps identified during the TCAs being addressed by the partner NGOs over the cooperative agreement period • Gaps identified during the TCA (refer to TCA reports by implementation area by individual partner (to be garnered from document review prior to the interview) • What were the major weaknesses that were identified during the TCA of your organizations? • What kind of support was received from NuPITA to address the identified weaknesses? • What actions were taken by your organizations to address the identified weaknesses? • What is your view about the usefulness, appropriateness of the support that was received? • In your view, were the identified weaknesses responded to adequately? • How would you rate the usefulness of each mode of TA from NuPITA – probe using a Likert scale – as very useful, useful, somehow useful, or not useful? Institutionalization of internationally recognized standards and procedures over the cooperative agreement period • Refer to TCA reports by implementation area by individual partner (to be garnered from document review prior to the interview) – to identify the standards and procedures that are applicable to the services offered by the partner NGO • Assess for the following; o Extent to which the identified standards and procedures were being utilized at the time when the TCA was performed o Ascertain the extent to which the identified standards and procedures are being utilized at the time of the final evaluation 76 o Any support provided by NuPITA in this area o Usefulness and appropriateness of the support that was received Changes in the quality of support supervisions (availability of written comprehensive periodic reports, identification of issues to be addressed, evidence of actions taken to address them) - over the cooperative agreement period • Refer to TCA reports by implementation area by individual partner (to be garnered from document review prior to the interview) – to identify the weaknesses related to the quality of support supervisions at the time when the TCAs were performed • Assess for the following: o Extent to which the identified weaknesses were addressed during the time of the cooperative agreement o Extent to which quality support supervisions are being conducted at the time of the final evaluation o Probe for any support provided by NuPITA in this area o Probe for usefulness and appropriateness of the support that was received Use of program data to inform management decisions (periodic review meetings, evidence of internal data dissemination, etc.) • Gaps (related to data management and use) identified during the TCA (refer to TCA reports by implementation area by individual partner (to be garnered from document review prior to the interview) • What were the major weaknesses (related to data management and use) that were identified during the TCA of your organizations? • What kind of support was received from NuPITA to address the identified weaknesses? • What actions were taken by your organizations to address the identified weaknesses? • What is your view about the usefulness, appropriateness of the support that was received? • In your view, were the identified weaknesses responded to adequately? • How would you rate the usefulness of each mode of TA from NuPITA – probe using a Likert Scale – as very useful, useful, somehow useful, or not useful? 77 Changes in the functionality of referral systems/linkages • Gaps (related to referral systems and linkages) identified during the TCA (refer to TCA reports by implementation area by individual partner (to be garnered from document review prior to the interview) • What were the major weaknesses (related to referral systems and linkages) that were identified during the TCA of your organizations? • What kind of support was received from NuPITA to address the identified weaknesses? • What actions were taken by your organizations to address the identified weaknesses? • What is your view about the usefulness, appropriateness of the support that was received? • In your view, were the identified weaknesses responded to adequately? • How would you rate the usefulness of each mode of TA from NuPITA – probe using a Likert Scale – as very useful, useful, somehow useful, or not useful? Change in the aggregate domain scores for the Child Status Index (CSI) – for selected partners – for the period for which it was administered • Support received from NuPITA in rolling out the CSI • Obtain information about the aggregate domain scores (6 month period) for the last one and half to two years. • Ascertain the extent to which data collected using the CSI tool is being used for the following purposes: o Used by staff/caregivers to monitor problems and benefits in their efforts in serving children. o Used to identify the specific needs of a child and his/her household and to translate these needs into intervention strategies. o Used by decision-makers plan, implement, and modify child services based on aggregate information about child well-being over time. o Used to support advocacy for resources and improvements in service quality. o Used to raise awareness among frontline staff (such as community health workers and caregivers) about the multiple dimensions of child well-being to help them understand and address these areas routinely in their work. 78 o Used to compare performances within and across programs, and regions Evaluation Question 2: Have these changes, if any, to partners HIV service delivery approaches been influenced by capacity building inputs provided by NuPITA? If so, how? Extent to which the changes prescribed above have been attributed to specific NuPITA inputs by the project stakeholders (NuPITA staff and partner NGOs). Examples of NuPITA inputs include, NuPITA staff, training, consultancy, learning field visits, etc. Evaluation Question 3: Have NuPITA partners experienced changes in their overall institutional sustainability (as measured by the type and quality of linkages with stakeholders and type and quality of systems which allow them to better access funding or to address funding gaps) since the start of their NPI cooperative agreements? If so, what are these changes? Financial sustainability • Assess for the presence of different funding sources at inception and project close-out • Assess for the presence of different collaborating partners at inception and project close-out • Assess for the presence of costed strategic plan (or a financial plan or business plan) that includes some funds that are not restricted for cost recovery - at inception and project close-out • Assess for the presence of resource mobilization strategy- at inception and project close-out Program sustainability • Assess for networks/linkages at inception and project close-out (probe participation in national discourses line technical working groups, forums, etc.) • Assess for the presence and adherence to organizational policies, procedures and practices at inception and close-out • Assess for the presence of trained and skilled human resources (relevant to the services offered) at inception and project close-out • Assess for the presence of a sustainability plan for project activities beyond the life of the project – at inception and project close-out 79 Evaluation Question 4: Have these changes in sustainability been influenced by capacity building inputs provided by NuPITA? If so, how? • In your view, has there been a change in institutional sustainability? If yes, then what is the change in institutional sustainability that happened? • In your view, do you think that the changes above are related to the support from NuPITA? If yes, what are those inputs? Assess the extent to which the changes in financial and program sustainability have been attributed to specific NuPITA inputs by the project stakeholders (NuPITA staff and partner NGOs). Examples of NuPITA inputs include, NuPITA staff, training, consultancy, learning field visits, etc. Evaluation Question 5: What factors have contributed to NUPITA achieving or not achieving its intended results? Basing on your experience in this project and your long lasting interaction with the New Partners Initiative, and in your view, what are things that worked well and those that did not work well? Probe: • For the things that worked well, describe why they were done well and what was key in achieving project success • For the things that did not work well, describe why they did not work well and what was key in hindering project success • If you were to replicate the project, what would you recommend to retain and/or to exclude? TOCA Items for the Interview Guide Organizational strategy 1. Does the organization have capacity to scale up? Guidelines & standard operating procedures 2. Are there organizational guidelines and standard operating procedures for XXX service delivery? 3. Are the guidelines and protocols up to date and in line with National Guidelines and or international guidelines? 80 4. Is there a checklist to guide project implementers and volunteers in their daily work? 5. Are there measures to ensure consistent adherence to project guidelines/SOPs? 6. Are all the implementers applying the guidelines and SOPs in service delivery? Service standards 1. Does the project have documented XXX service standards in place? 2. Are the service standards in line with national guidelines? 3. Have staff and project implementers been oriented on the standards? 4. Are project implementers applying and following the service standards? 5. Is there a standards checklist that project implementers and volunteers can apply in their daily work? Referral system 1. Is there a referral system set up? 2. Have referrals been made to other providers for services not provided by this organization? [Look for referral notes, client return forms, list of other providers, etc. that show existence of a referral relationship] 3. Is there a list of all the organizations providing XXX services in project areas? [Check for a list of other organizations providing related services in the project area] 4. Is there a standardized referral form? 5. Is there documentation of referred clients? 6. Is there means of verifying whether services were received by referred clients? [This may include a feedback loop to track referrals, documentation of referrals] 7. Are there periodic meetings of network providers? Sustainability 1. What efforts are being undertaken to ensure services continue beyond the life of the project? 81 2. Does the organization have a sustainability plan for its activities beyond the life of the project? 3. Which sustainability approaches are being implemented by the organization? 4. Is the organization involved in strengthening networks and coordination? Data collection 1. Are there tools for collecting data at the various levels? 2. Are there mechanisms in place to avoid double counting (for example, unique client identification)? 3. Does the organization have standardized tools across sub-partners and service delivery points? 4. Have staff been trained in the use of the tools? Data use for decision making 1. Does the organization have historical (or baseline) data against which reports can be compared to help in decision making? 2. Does the organization have a process for comparison of achievement of goals and past progress that result in plans to modify action or approach/tools? 3. Do the organization’s management and staff follow a procedure of time-bound corrective action and tracking achievements against plans in all areas of the intervention? 4. Does the organization use data collection and analysis to inform non-data collection/analysis members of the implementation team and the partner community, if relevant? USAID Questionnaire NPI Partners Supported by NuPITA Uganda: ARC, Retrak, FXB, GOAL, Wellshare, ICOBI Kenya: AMURT, Tearfund, G.R.A.C.E, KNH/SJCC NuPITA evaluation objectives: • Determine the effect that NuPITA’s capacity building technical assistance has had on HIV service delivery results through the work of its partner organizations; and 82 • Determine the effect that NuPITA’s capacity building technical assistance has had on the sustainability of its partner organizations (as measured by the type and quality of linkages with stakeholders and type and quality of systems which allow them to better access funding or to address funding gaps). Although this evaluation focuses on NuPITA, we understand that the project is one of many mechanisms USAID used to support all NPI partners and NuPITA contributed to this overall effort. 1. What is your overall impression of the NuPITA project’s performance and results? 2. Do you think that the partners NuPITA supported have improved the range of their HIV services under NPI? If not, why not? If so, how? 3. Do you think they improved the quality of their HIV services? If not, why not? If so, how? 4. Do you think these results are sustainable? Why or why not? 5. Do you think NuPITA inputs contributed to these results? If so, how did they contribute? 6. Was USAID satisfied with the following aspects of NuPITA TA: o Responsiveness: To USAID needs and requests and to partner needs and requests • Effectiveness • Combination of capacity-building inputs 7. In retrospect, do you think the NPI-NuPITA design was useful? Probe: • Rationale for choice of NPI partners • Project length 8. The NPI and TA partnerships were perhaps a unique effort to launch new USG partners effectively. How do you envision future support of this kind to USG partners? How do you think the lessons of this project might best be incorporated into future USG capacity building efforts? NuPITA Staff Interview Guide Repeat the focus of evaluation on service delivery and sustainability Objectives: 83 The two key objectives of the evaluation are to: • Determine the effect that NuPITA’s capacity building technical assistance has had on HIV service delivery results through the work of its partner organizations • Determine the effect that NuPITA’s capacity building technical assistance has had on the sustainability of its partner organizations. 1. The intended results of the project (per project documents) did not focus directly on service delivery and sustainability but rather on organizational development. Did you have service delivery and sustainability in mind from the very beginning? If so, in what ways? 2. In your view, to what extent do you think different inputs had an effect in improving SD and sustainability [advisors, coaching/mentoring, leadership seminar, south to south TA (Tearfund)]? [Probe: did these inputs translate directly into improving scope, coverage and quality of services? How?] 3. Which inputs weren’t demanded/used? 4. What challenges or obstacles did you face related to implementing the project? [Probe: did these limit SD/sustainability] 5. Now that the project is ending, what do you think remains to be done to maximize gains in SD and sustainability? 6. In retrospect, do you feel the T/OCA tools were adequate to provide a solid basis for assessing and improving service delivery and sustainability? Were there domains you would have expanded to give more detail? Was one TOCA adequate? 7. In the area of relations with partners; how did NuPITA coordinate with the NPI contracting NGOs (INGOs) to provide TA to their sub-recipients? (Give example of Woord en Daad/Mfesane). Did the NPI contracting NGOs need capacity improvements around service delivery and sustainability? 8. What is your view of the NuPITA program design? (Probe: for appropriateness for improving services and sustainability of NPI partners? Would you have designed it differently?) 9. In retrospect, do you feel the package of inputs per partner was appropriate? 10. Based on your experience in this project and your long lasting interaction with the New Partners Initiative, and in your view, what are things that worked well and those that did not work well? Probe: 84 • For the things that worked well, describe why they were done well and what was key in achieving project success • For the things that did not work well, describe why they did not work well and what was key in hindering project success • If you were to replicate the project, what would you recommend to retain and/or to exclude NPI Advisors Interview Guide Explain the focus of evaluation on service delivery and sustainability (SD/S) Objectives: The two key objectives of the evaluation are to: • Determine the effect that NuPITA’s capacity building technical assistance has had on HIV service delivery results through the work of its partner organizations • Determine the effect that NuPITA’s capacity building technical assistance has had on the sustainability of its pPartner organizations. 1. Did the NPI organization for which you were an advisor change its HIV service delivery approaches over the life of the project? If so, how and what effect has this had on service delivery results? 2. Have these changes, if any, to partners’ HIV service delivery approaches been influenced by capacity building inputs provided by NuPITA? If so, how? 3. Did your host agency experience changes in its overall institutional sustainability (as measured by the type and quality of linkages with stakeholders and type and quality of systems which allow them to better access funding or to address funding gaps) since the start of their NPI cooperative agreements? If so, what are these changes? 4. Have these changes in sustainability been influenced by capacity building inputs provided by NuPITA? If so, how? 5. For F&A advisors: any comments on improvements in SD/S brought about by improvements in F&A systems. 6. Based on your experience in this project and your long lasting interaction with the New Partners Initiative, and in your view, what are things that worked well and those that did not work well? 85 Probe: • For the things that worked well, describe why they were done well and what was key in achieving project success • For the things that did not work well, describe why they did not work well and what was key in hindering project success • If you were to replicate the project, what would you recommend to retain and/or to exclude? 7. Relation of NuPITA, prime, and sub-recipients. What worked? What didn’t work? 8. What challenges or obstacles did you face related to service delivery and sustainability? 9. Did you receive adequate support from NuPITA for your work? Please give examples. 10. In retrospect, do you feel the combination of NuPITA inputs (e.g., advisor, group training, one￾on-one TA) for your partner organization was appropriate? 11. In your view, to what extent do you think different inputs had an effect in improving SD/S [advisors, coaching/mentoring, leadership seminar, south to south TA (Tearfund)]? [Probe: did these inputs translate directly into improving scope, coverage and quality of services? How?] 12. In retrospect, do you feel the T/OCA tools were adequate to provide a solid basis for assessing and improving service delivery and sustainability? Were there domains you would have expanded to give more detail? 13. Now that the project is ending, what do you think remains to be done to maximize gains in SD and sustainability? 86